I'm looking for advice on the best way to handle very large JSON documents when sending them from my backend to the frontend. The documents I'm working with can range significantly in size, from around 500 to 30,000 lines. They are static, and I expect around 1000 users will access them. Should I be using something like Firebase storage, S3 buckets, or are there better strategies?
6 Answers
It really depends on your data and how you plan to use it. Sending data is different from storing it. You could break it into smaller endpoints to fetch only what’s needed or use pagination. Remember, both streaming and compression let you manage larger datasets, but you have to download everything before parsing, which is challenging with massive JSON files.
Here are a few strategies to consider:
1. JSON streaming – send data in chunks instead of one giant payload.
2. Compression – as mentioned, gzip or brotli can make a big difference.
3. Selective data fetching – break your responses into smaller, targeted chunks rather than one massive response.
4. Explore binary formats for even more efficiency.
5. Finally, think about refactoring your approach to avoid such large responses in the first place.
Storing JSON directly might not be the best choice. You should deserialize it and keep the actual data in a database. What you're doing sounds a bit unoptimized.
To give you some solid advice, can you tell us if these documents are static or dynamic? Also, how large are we talking about when you say 'very large'? The traffic you're expecting could help determine the best solution. If it's just about cheap file hosting, B2 or BunnyCDN could be great options, but depending on your scale, the price differences might not be huge.
I'm quite new to this, but I want to understand why you need to send such large JSON files? Is it related to a specific feature you're building?
One simple yet effective method is to compress your JSON data. Using gzip or brotli can significantly reduce the size of the data you're transferring.
Related Questions
Cloudflare Origin SSL Certificate Setup Guide
How To Effectively Monetize A Site With Ads