I'm currently consulting for a large AWS customer in Europe, and they have this extensive distributed system designed as a modular microlith mainly using Node.js. Their application consists of a collection of microservices, each comprising different business units organized into modules. The workload they're handling is sensitive to latency, leading them to group modules that frequently interact within the same microservice. Right now, they're operating around 5,000-6,000 Fargate instances, with interservice HTTP latency in the same availability zone reported between 8-15 ms. Is this latency typical? What latency numbers do you see across your own containers? Are there any straightforward strategies to reduce this latency? Unfortunately, implementing changes can be a challenge in a large organization, like a recent ticket regarding placement groups that has been stalled for two years. How would you approach solving this latency issue?
1 Answer
The latency you're seeing sounds reasonable, especially considering the traffic can vary between intra-AZ (availability zone) and inter-region responses. Make sure you’re averaging your metrics correctly. Also, how do you measure latency? If you're running large-scale traffic, like you said, this can definitely impact latency, especially with Node.js where CPU cycles can be a factor. So just knowing that number without additional context makes it tough to say if your services are underperforming or not.
I'm only looking at latency within the same region, so that's not a factor for me.