I'm genuinely curious if I'm overlooking something, or if this is just the norm in cloud services. Here's what I need to do: I want to handle an HTTP request, run some code asynchronously, and push a message to a broker. For instance, I'm using AWS and it's a bit complex:
1. API Gateway to provide the HTTP endpoint.
2. Lambda to execute the code.
3. EventBridge for event routing.
4. SQS for queuing and retries.
5. CloudWatch for logging.
6. Then I'm the one tying everything together.
This setup is pretty similar across other platforms like Azure or GCP, just with different names.
I'm facing two major issues:
1. The costs are outrageous since each service bills separately. So, for every request, I get charged for API Gateway, Lambda, EventBridge, SQS, and CloudWatch. As my traffic grows, I end up paying more for connecting services than for the actual processing.
2. There are too many components to manage. I have six different dashboards to monitor, retries set in three different locations, and debugging involves checking multiple services, each with its own limits.
For a straightforward "run code on HTTP request" task, I feel burdened by this numerous services. Is this just part of how things work in the cloud? Do others accept this kind of complexity, or is there a simpler solution I'm missing? I've noticed some people handle this setup or revert to old-style EC2 applications. Is there a middle path? What approaches do you guys take?
5 Answers
Cloud infrastructure does tend to stack services like this. That's how AWS is designed—to be an ecosystem. But if you want something less complex and more straightforward, consider using a single EC2 application to handle everything. You might let go of some of the easily scalable features, but you’ll end up with fewer moving parts.
I see where you're coming from, but you don’t necessarily need all those services. Technically, EventBridge, SQS, and CloudWatch logs aren’t mandatory unless you're building a complex system. They add functionality but can be unnecessary depending on your use case. Think about routing directly within your Lambda instead of relying solely on EventBridge for some actions.
Honestly, it sounds like you've over-engineered your setup a bit. If all you need is to trigger a function, you could simplify it to just API Gateway pointing directly to Lambda. For more complex processes, like if you need better scaling or handling long-running tasks, consider API Gateway -> SQS -> Lambda. It's really not that different from traditional systems—it's just that in the cloud, you can offload a lot of management.
Also, consider using a lambda function URL directly if you don’t need the API Gateway overhead. It simplifies things!
Remember though, IAM and logging via CloudWatch will still be necessary for governance.
The costs can indeed balloon—it’s common. But remember that AWS services charge per use, so for low traffic, it could be affordable. But if you expect significant traffic, don’t overlook traditional setups, especially if you prefer less complexity and are okay with early investments in infrastructure.
Absolutely! Sometimes going back to basics with just a server setup is the best route if you anticipate a high volume.
You don’t need all those services to get things done. If simplifying is your goal, cut out SQS and maybe even EventBridge if they’re not necessary. Sure, every service has its purpose, but if you’re looking for a quick, simpler solution, it can get cumbersome managing too many services for minor tasks.
Yeah, but then you're back to managing everything manually, which is what a lot of people are trying to avoid with the cloud.