The bucket has billions of small JSON objects in one flat prefix. Objects are added or overwritten continuously, and a lifecycle policy removes them after 90 days. I can't change how they're delivered, but I can ignore the existing files.
I want to process new and updated objects for analytics. Would S3 notifications, a queue, and workers that batch the JSON into partitioned Parquet files be a reasonable setup? Ordering matters between versions of the same key.
m.624!fluxpin:
Copying each JSON object into a partitioned layout would help with organization, but it would still leave the small-file problem. Batching the contents is the useful part here.