How can I process billions of tiny S3 files as they change?

November 16, 2026 5 answers
Asked by Mirex+4p

The bucket has billions of small JSON objects in one flat prefix. Objects are added or overwritten continuously, and a lifecycle policy removes them after 90 days. I can't change how they're delivered, but I can ignore the existing files.

I want to process new and updated objects for analytics. Would S3 notifications, a queue, and workers that batch the JSON into partitioned Parquet files be a reasonable setup? Ordering matters between versions of the same key.

Answers

7h!ctrlspool_352

That architecture makes sense. Use notifications to trigger a worker that reads the JSON content, then send the records to Firehose. It can buffer them, convert them to Parquet, and write date-partitioned objects to a separate analytics bucket.

Lambda is one option for the reader, though containers or EC2 workers could work too. Check the schema and partitioning requirements for Firehose, and size the buffers around how fresh the analytics data needs to be. The aim is to avoid making Athena open a huge number of tiny objects.

m.624!fluxpin:

Copying each JSON object into a partitioned layout would help with organization, but it would still leave the small-file problem. Batching the contents is the useful part here.

Reply to this answer
4t+socketwhistle_926

The ordering requirement is the tricky bit. S3 notifications can arrive more than once and out of order. Putting them through a FIFO queue doesn't restore the order in which the overwrites happened.

Decide whether you need every version or just the latest state for each key. For latest state, make processing idempotent and use an appropriate per-key ordering signal to prevent an older event from replacing newer state. S3 event sequencers can be compared for events on the same key. If every version matters, versioning and reading the version identified by the event are worth investigating. A read by key alone can return a newer overwrite than the one that triggered the notification.

That still leaves recovery and reconciliation to design. A queue and retry policy help with processing failures, but don't assume the notification stream by itself is a complete audit trail.

Mirex+4p:

It's only the versions of the same key that need ordering, not events across different keys. Latest state may be acceptable if processing every version isn't practical.

Reply to this answer
Rilven-2s

Before picking services, estimate the rate, not just the total object count. Each change can mean a notification, a read, processing, and a write or stream record. At this scale, request charges and any KMS calls can matter as much as compute.

Test with a representative burst and measure queue backlog, processing latency, Parquet file sizes, and cost per million changes. Also check the queue and downstream service throughput limits before committing to FIFO.

Reply to this answer
_e61~claspflute.3

Another option is to investigate S3 Metadata journal tables as a way to discover recent changes, then process them in batches. A query engine such as DuckDB could select a time window and feed the resulting keys into your loader.

Leave a delay before processing the newest window and design for overlapping windows and deduplication. Check the journal's availability and retention for your setup, and whether it provides the version information you need before relying on it for this pipeline.

Reply to this answer
2n~teagrommet_463

Think separately about the analytics output and the final database state. Parquet is useful for scanning batches in Athena. If the end goal is a current-state table in Postgres, you'll also need a way to apply updates per key without older changes overwriting newer ones.

You could keep the batched analytics stream and build that database view downstream, rather than making each tiny S3 object a direct database write.

Reply to this answer

Add your answer

Answers are reviewed before publication.

Related Discussions