Hey everyone! We're working with a pretty standard tech stack—AWS, EKS, ALB, argocd, and a Java HTTP API service. We're looking to implement load shedding to drop a percentage of requests when our service gets overloaded and becomes unresponsive. I'm considering two main options: First, using AWS metrics (like Prometheus or CloudWatch) to trigger a Lambda function that will blackhole a percentage of requests to a different target group. However, this seems somewhat specific to AWS and doesn't align perfectly with our GitOps setup. Second, there's the option of deploying an Envoy sidecar for every service pod to use an admission control filter or something similar. This feels more native to Kubernetes, but it also raises concerns about reliability, like what happens if Envoy itself goes down? I'm leaning towards the second option, but I'm worried I'm overlooking some important aspects. Would love to hear your thoughts! Cheers.
3 Answers
Your choice really depends on how your application handles sessions and stickiness. If it's stateless and can manage requests bouncing between different target groups, then using AWS metrics is a solid approach. But if you have session management that relies on maintaining state with specific pods, the sidecar option might be better. It sounds like you're already considering what fits best with your current workload, so just keep things simple and go for what feels right for your setup!
Have you thought about how your ingress is set up? Envoy can work with HTTPRoute without needing sidecar configuration, and it provides built-in rate limiting and circuit breaker features. It might be easier than managing a sidecar for every pod.
Right now, it's just an ALB created with the ALB controller, nothing complicated. I know deploying Envoy separately is an option, but I think a sidecar would be more straightforward for scaling.
Load shedding is definitely a real term! It's a strategy to manage overload by intentionally dropping requests rather than letting your service crash. This can be crucial when you’re facing a situation where your current resources can't handle all the requests—like if you have 20 pods and they all start restarting due to too much load. Autoscaling alone might not solve the problem if there aren't enough resources to scale up quickly. Check out AWS’s insights on load shedding; it’s fascinating how it can prevent services from becoming unrecoverable.
Totally get that! I've seen it used effectively to prevent cascading failures, especially in scenarios like with electrical grids.
Yeah, for sure! I used to work in energy, and it's a key term there as well, especially during outages.
Exactly! Simplicity really is key, especially if your architecture is already complex.