How Can I Verify File Hashes When Using S3 Presigned Upload URLs?
Asked by
MellowCedar42
I'm building an image upload service that sends files directly to S3 with presigned URLs, so the files don't have to pass through my backend. Before creating the upload URL, the client currently sends the file size, content type such as image/png, and a hash that I use for deduplication. However, none of that hash information is trustworthy: the client can submit one value to the backend and then upload a completely different file, or provide a random string as the hash. What's the best way to ensure the uploaded object actually matches the expected checksum while keeping the direct-to-S3 upload flow?
Answers
Another option is to treat the client-provided hash as untrusted until after the upload. Let the file upload directly to S3, then calculate or verify its checksum from an upload-triggered function or another S3 checksum process. Use the verified checksum for deduplication and delete or quarantine objects that don’t pass validation.
Reply to this answer
Have S3 validate the checksum during the upload. When generating the presigned PUT request, include the expected SHA-256 checksum with the object command, such as `ChecksumSHA256`, and sign the corresponding `x-amz-checksum-sha256` header. The client must then send that header along with the file. S3 will reject the upload if the checksum doesn’t match. You should also make sure the client can’t omit the checksum header, for example by enforcing it with an IAM policy.
Reply to this answer
You don’t necessarily need to calculate the hash in your backend. S3 can store and validate checksums provided with the upload, and it can expose checksum metadata for objects afterward. The important part is not to use a client-supplied hash for deduplication until S3 has validated it.
Reply to this answer
Configure the upload permissions so requests without a checksum header are denied. Then tell clients to calculate the checksum locally, include it in the signed request, and upload the file using that exact value. This keeps the data path direct while preventing an arbitrary file from being associated with an unrelated hash.
Reply to this answer
Add your answer
Answers are reviewed before publication.
MellowCedar42: That makes sense for post-upload validation, but I’d prefer enforcing the checksum during the upload if S3 can do that.