I receive billions of very small JSON objects—usually a few bytes to 10 KB—in a flat S3 prefix that I cannot change. Existing objects may be overwritten with newer versions, new objects are added continuously, and objects older than 90 days are removed by a lifecycle policy. Because the files are tiny and unpartitioned, querying them directly with Athena creates more metadata overhead than useful work. I can ignore the existing data, but I need a scalable way to process newly added or updated objects and make them analytics-friendly. My initial idea is to capture S3 events, buffer batches with SQS or another service, consolidate the JSON records, convert them to Parquet, and write them to a partitioned analytics bucket. Is that a sensible design, and how should updates and event ordering be handled?
5 Answers
Do not rely on FIFO queues to establish global ordering. S3 notifications are at-least-once and are not guaranteed to arrive in order, so a FIFO queue cannot recover ordering that was never guaranteed upstream. For most processing, use a standard queue, make the consumer idempotent, and deduplicate using the object key plus version ID or another event identifier.
Watch the cost of per-object operations, especially KMS requests if encryption uses customer-managed keys. Reading and rewriting huge numbers of tiny files can make encryption and request charges surprisingly high. Batch work where possible, consider S3 Bucket Keys when compatible with your security requirements, and measure Athena scan time and request costs before committing to the architecture.
Your general approach is sound: use object events to trigger processing, buffer records by time or count, and write properly sized, partitioned Parquet files to a separate analytics location. Lambda can read the small objects and send their contents to Kinesis Data Firehose, which can buffer, batch, convert to Parquet, and apply dynamic partitioning. A container or ECS consumer is another option if the event rate or processing logic is too heavy for Lambda.
That would let the source bucket remain unchanged while giving analytics a much more efficient layout. I’ll need to account for retries and duplicate events in the consumer.
The underlying issue is the file layout. S3 is not a database, and billions of tiny objects create excessive listing, metadata, request, and query overhead. Keep the incoming files if required, but compact them into larger compressed Parquet files—partitioned by useful fields such as date or event type—and query that curated dataset with Athena or another analytical engine. Glue, Firehose, or a custom ECS job can perform the compaction.
Eventually some data will also be loaded into a database, but I still need a pipeline that consumes the source objects and produces that more useful downstream format.
If you only need recent analytics, an S3 metadata journal can provide a more complete change listing than notifications alone. You could query the recent entries, build a list of affected objects, and let DuckDB read the JSON files in batches. This can be useful for reconciliation or scheduled micro-batches, but it still does not make billions of tiny objects an efficient long-term analytics format.
If versions of the same key truly must be processed sequentially, that becomes a per-object ordering problem rather than a global one. You may need state tracking and last-writer-wins logic, because safely waiting for every earlier version can be difficult when notifications are delayed or duplicated.