I'm looking at FastAPI, Litestar, Django REST Framework, Django Ninja and Bolt for APIs with database queries and authentication. What should I look for in a benchmark besides raw JSON throughput?
How should I compare Python API frameworks beyond JSON benchmarks?
Answers
Look at each endpoint separately rather than picking a winner from the fastest JSON response. In one test with a one CPU container and PostgreSQL, Bolt led the 1 KB JSON and simple database read tests, while Litestar with Granian led the paginated articles test. Those rankings don't tell you which will be fastest for your particular queries.
The test used 100 connections for 10 seconds per endpoint and reported the median of five container starts. That's useful context, but I'd also want to see latency and longer runs before using the numbers for capacity planning.
Reply to this answer
Check what's actually being compared. The frameworks used different authentication libraries, and Litestar and Bolt used native msgspec serialization. Even with identical response bytes, the work needed to produce them can differ.
It would be interesting to rerun the JSON tests with orjson where supported, then compare the database and auth endpoints too. Faster serialization might matter a lot less when queries or other I/O dominate. Likewise, compare ORM queries with raw SQL and manually built dictionaries before assuming how much the ORM costs.
Reply to this answer
Server choice matters too. The reported DRF results differed substantially between Gunicorn and Granian, so calling one number the framework's speed is a bit misleading. Keep the server configuration visible when comparing results, and test the combination you would actually deploy.
Reply to this answer
Throughput isn't the whole framework decision. DRF can be convenient for CRUD and business APIs, while a lighter framework may be a better fit for a smaller service. Check the integrations and development tradeoffs you need, then benchmark the realistic endpoints that matter to you.
Reply to this answer
There's room to extend the comparison. Robyn would be interesting to test against the same database and auth workloads, not just JSON. For Bolt, I'd also check its Django integration and deployment setup before treating a fast result as enough reason to switch.
Reply to this answer
Add your answer
Answers are reviewed before publication.
_e93+thunderpocket.6:
For the auth tests, I'd also check that each implementation does equivalent user lookup and validation work. Different libraries can make a throughput comparison tricky.