How should I build an analytics pipeline for billions of tiny, unpartitioned S3 files?
I receive billions of very small JSON objects—usually a few bytes to 10 KB—in a flat S3 prefix that I cannot change. Existing objects may be overwritten with…
tiny files
How should I build an analytics pipeline for billions of tiny, flat S3 files?
I have an S3 bucket containing billions of very small JSON objects, ranging from a few bytes to about 10 KB. Objects are continuously added or overwritten, and…