I'm building a social app that will allow profile-picture uploads and eventually image sharing in group chats. I need to handle child sexual abuse material responsibly, but I'm unsure how to structure the scanning process. I'm hesitant to send every uploaded image directly to a third-party service such as AWS Rekognition for explicit-content detection before addressing CSAM concerns. I've investigated Google's Content Safety API and PhotoDNA, and I'm also considering open-source detection tools, although I'm concerned about privacy, licensing, and whether sending potentially illegal content to an inference provider could violate its terms. What is a budget-conscious and legally responsible way to screen uploads for CSAM before using general-purpose image-moderation services?
3 Answers
There probably isn’t a single inexpensive API that removes all of the legal and operational responsibility. A typical architecture is to use a specialized CSAM hash-matching service as early as possible, before permanently storing the file or sending it to general image moderation. If the upload is not matched, you can continue with your normal safety checks, such as explicit-content classification. Services such as Project Arachnid or specialized commercial providers may be worth investigating, but pricing and eligibility can be significant. Before launch, consult a lawyer and the relevant national reporting authority, and confirm how each vendor handles suspected CSAM, retention, reporting, access controls, and false positives. Also verify directly with AWS what happens when its service encounters suspected CSAM and what obligations remain with you; do not rely on assumptions about automatic reporting.
A known-image hash database can be one layer of the upload pipeline. The client or server can calculate a cryptographic or perceptual hash and compare it with an approved database before accepting the upload. A match should stop the upload and trigger the documented safety and reporting process. However, hash matching only catches previously identified material and can be bypassed by modified or new images, so it cannot replace a specialized provider, human review procedures, access restrictions, and a complete incident-response plan. Avoid storing or forwarding suspected material unnecessarily, and confirm the exact legal and contractual requirements in every country where the service operates.
Look into Pathways, which may help eligible organizations obtain access to PhotoDNA or connect with suitable providers. Their resources also point toward established guidance for designing CSAM-prevention systems. Eligibility and licensing vary, so treat this as a starting point rather than assuming access is guaranteed.
Thanks—I’ve already registered to see whether they can provide guidance for this design. I’ll wait to hear what options they suggest.
That makes sense. Some commercial options appear to cost tens of thousands of dollars per year, which is difficult for an early-stage product, so I’ll investigate nonprofit or lower-cost programs and get legal advice before launching.