Should I Structure S3 Bucket Names by Client or Topic?

Don't forget to consider your exit strategy. You'll need to prove that data has been deleted at some point. Think about whether the data is user-specific or client-specific. Setting lifecycle rules can really help with that.

If you're planning for long-term storage, having a unique bucket for each client is a solid choice. It simplifies cost tracking using AWS Cost Allocation Tags, which only work at the bucket level. Plus, if you won’t have more than a million customers, you won’t hit any account limits.

I have a similar setup where I route data like this: ingress facade -> bucket -> queue -> consumer to handle events effectively using S3's object created notifications to fan out the data.

There are a couple of key technical points to think about. First, consider permissions—it's easier to set up clear access policies if the client identifier comes first. Second, remember that S3 operates on prefixes. If Databricks needs to process all userdata, it'll be simpler if that prefix appears first, while client-specific processing would also benefit from having the client ID first.

If clients will be writing directly to the bucket, I recommend structuring it with a client prefix for easier management. But I advise against giving them direct access to Databricks. Creating an ingest bucket where you control data movement into your processing bucket is safer to prevent issues with bad data formats. If you're primarily dealing with one Databricks consumer, a single bucket approach might reduce the management overhead of adding new schemas whenever a new client joins.