I have an S3 bucket containing billions of very small JSON objects, ranging from a few bytes to about 10 KB. Objects are continuously added or overwritten, and files older than 90 days are removed with an S3 Lifecycle policy. Unfortunately, the objects arrive in a flat prefix such as s3://bucket/prefix/file1, file2, and so on, and I have no control over the naming or layout.
Because the files are tiny and not partitioned, querying them directly with Athena is impractical: listing and metadata operations take more time and resources than reading the data. I can ignore the existing objects, but I need an analytics pipeline for new and updated objects going forward.
My current idea is to capture S3 events, send them to a queue, process objects in batches on ECS, EC2, or a similar service, consolidate the JSON into larger files, convert the batches to Parquet, and write them to a partitioned analytics bucket. Is this a sensible design, or is there a better AWS-native approach?
5 Answers
If the workload is only occasional or exploratory, S3 Metadata journal tables combined with DuckDB could help identify recently changed objects without repeatedly listing the entire prefix. You can build a list of recent keys from the journal and query those JSON files in one DuckDB session. This is more suitable for incremental or ad hoc work than for a high-volume long-term analytics architecture.
If you prefer ECS or EC2, send notifications to a standard SQS queue and have workers consume them in count- or time-based batches. The workers can read the JSON objects, compact them, write partitioned Parquet, and record which source versions were processed. Add retries, a dead-letter queue, and idempotent output handling because notifications can be duplicated or arrive late.
For a mostly serverless pipeline, S3 notifications can trigger Lambda, which reads the small JSON payload and sends the records—not necessarily the individual files—to Kinesis Data Firehose. Firehose can buffer records, dynamically partition them, convert them to Parquet, and write larger objects to a separate analytics bucket. Make sure the buffering and partition strategy produces reasonably sized files rather than one output object per event.
The general shape is right: treat the incoming objects as a landing area, then compact them into properly sized, partitioned Parquet files for analytics. S3 event notifications are at-least-once and are not globally ordered, so a FIFO queue does not guarantee the ordering you need. Use a standard queue or stream, make processing idempotent, and track the object key plus version ID or another source revision. For multiple versions of the same object, use an explicit last-writer-wins rule unless strict per-object sequencing is absolutely required.
Ordering may matter between versions of one object, even if ordering across different objects does not. In practice, enforcing that reliably is difficult, so deduplication and last-writer-wins processing are probably safer than relying on event order.
Billions of tiny objects are an awkward fit for object storage analytics. Parquet, a table format such as Iceberg or Delta Lake, or another compaction layer is the important part of the solution. Also check encryption costs carefully: per-object KMS operations can become unexpectedly expensive at this scale, so review bucket keys and the overall encryption design before committing to the architecture.
A Lambda or container can also copy the objects into a better layout if you need to preserve the raw files. Keeping the original bucket as an immutable or near-immutable landing zone and maintaining a compacted analytics zone is usually easier to operate.