My Go application uploads the same in-memory file to two Azure Blob Storage paths concurrently from an AKS pod. One upload sometimes finishes in a couple of minutes while the other takes around 20 minutes. Storage logs show much slower requests for the second upload, and pod network output drops during that window.
The upload configuration is the same, but the destination paths and network ports differ. The cluster also uses Istio. Where should I start looking?
Start with node-level network metrics around the upload window. VM sizes have network bandwidth limits, and other workloads on the same node may be using part of that capacity. Pod output dropping tells you what the pod sent, but not by itself what constrained it.
If you can get help from the cluster team, compare the same workload on a larger node or a less busy node. Check the Istio proxy and any egress gateway metrics too, so you can see whether the slowdown appears before or after that hop.
Check the storage response status codes and retry history before calling this a bandwidth cap. Concurrent uploads can run into storage-side limits, but the slow request timings alone don't prove that's happening. Look for throttling responses and compare request timing for each destination.
Try the two uploads separately, then together. You could also stagger their start times and see whether the slow upload follows the start order or a particular destination. A change in behavior would be a clue, not confirmation of a specific throttling mechanism.
I'd trace the connections as well as the goroutines. Check whether the uploads use separate TCP connections, whether they resolve to the same endpoint, and whether the paths result in different storage-side behavior. Compare connection and request timings with the SDK's retry information enabled. Two client objects don't necessarily tell you what happened on the wire.
Map out the egress path from the pod to Blob Storage. If traffic goes through an Istio egress gateway, check its load and configuration during the slow upload. It's also worth checking whether a private endpoint is available for the storage account and how that would change the route.
Answers are reviewed before publication.
Veslor-7n:
I can't choose which node the pod lands on, so I'll need the cluster team to check the node metrics and help with that comparison.