Suppose a codebase already handles about 1 million daily users and 300,000 concurrent users comfortably. If the target increases to 10 million daily users and 3 million concurrent users, would teams normally change the application code and architecture, or rely mainly on cloud autoscaling and horizontal scaling? The usual advice is "if it works, don't fix it," so I'm wondering whether adding more instances is generally enough or whether major bottlenecks usually require redesigning parts of the system.
4 Answers
There isn’t a universal answer because “10 million users” doesn’t describe the actual workload. The important details include request rates, read/write volume, traffic spikes, latency requirements, payload sizes, database usage, and how predictable the load is. You need load testing and measurements to find the real bottleneck before deciding whether to scale, optimize, or redesign.
Think of vertical scaling as buying a larger vehicle and horizontal scaling as adding more vehicles. More instances improve capacity and fault tolerance, but only if the work can be divided among them. If one request requires a single large process, a shared database transaction, or state stored locally, adding instances may not help. Sometimes the data and application have to be redesigned so the workload can be split safely.
Horizontal scaling often just moves the bottleneck elsewhere. Adding application servers may cause the database, cache, network, connection pool, or a slow query to become the limiting factor. In practice, teams usually combine autoscaling with code and schema improvements, better queries, connection management, caching, and load testing.
Autoscaling works well for a stateless application tier when extra instances can handle requests independently. But it won’t automatically solve every problem, and some systems or components can’t scale that way. You may need vertical scaling, horizontal scaling, caching, queues, database replicas, partitioning, or other architectural changes.
That makes sense—so the first step would be measuring each layer under a realistic workload instead of assuming the application servers are the only limit.