I have a Go application running in an AKS pod that uploads the same in-memory file to two different Azure Blob Storage paths at the same time. Each upload uses a separate Azure Blob client and runs in its own goroutine. The file is about 30 MB and is uploaded in 1 MB chunks.
Usually both uploads finish within a few minutes, but occasionally one completes in 1–2 minutes while the other takes around 20 minutes. This has happened often enough to be a recurring issue, although it does not occur on every run. When the problem occurs, Azure Storage logs show that individual API requests for the slow upload take roughly a minute, while requests for the fast upload complete in a few seconds. Azure indicated that this looks like a network-throughput problem rather than a storage write problem.
Pod network metrics show the output rate dropping from approximately 300 KB/s to 30 KB/s during the slow upload. The two transfers use different destination paths and network ports, but otherwise use the same code and configuration. Both uploads run in the same pod, and the pod uses Istio for networking.
What could cause one concurrent upload to be throttled while the other remains fast? Could the limitation be at the AKS node, SNAT or egress layer, Istio/Envoy, or somewhere in the Azure Storage path? What tests would help isolate the problem?
5 Answers
The consistent pattern does not necessarily mean one destination path is faulty. A 30 MB file split into 1 MB requests gives the network stack many opportunities to expose a connection-specific problem. Log each chunk's start time, duration, retry count, response status, and connection identity, then compare the fast and slow uploads. Also test the same paths separately and from a pod without Istio; that should help distinguish destination-specific behavior from concurrency or shared-network contention.
Separate Go clients do not necessarily prove that the transfers have completely independent network paths. Check the underlying HTTP transports, connection limits, proxy settings, and whether the Azure SDK is using a shared or intercepted route. Capture request timing and TCP statistics for both uploads, including retransmissions, congestion-window growth, connection reuse, and resets. A packet capture or network observability trace during a slow run could show whether one connection is suffering loss or being delayed.
Start by checking the network limits of the node or VM hosting the pod. Cloud instances have maximum throughput, connection, and sometimes SNAT limits, so two transfers can compete even if the application itself is configured correctly. Since you do not control pod placement, ask the infrastructure team to check node-level network metrics and try scheduling the workload on a larger node as an experiment. Running the uploads sequentially is another useful test: if both are fast sequentially but one slows down when concurrent, that strongly suggests shared network contention.
Because the pod uses Istio, check the sidecar and Envoy connection pools before assuming Azure Blob Storage is the problem. Both transfers may be sharing an upstream pool or egress path, and a large in-flight upload could delay the other stream through connection-pool limits or head-of-line blocking. As a controlled test, temporarily exclude the Blob Storage endpoint from sidecar interception or run the same pod without the mesh. If the timing imbalance disappears, the issue is likely in the service-mesh egress path.
Also investigate egress and SNAT behavior. Both uploads are originating from the same pod and may leave through the same node, NAT gateway, firewall, or egress gateway even though they use different ports. MTU problems, asymmetric routing, network policies, and endpoint routing can produce very different throughput between two connections. The infrastructure team can compare node-level and egress-level metrics with the Blob request timestamps.
That seems worth testing. The uploads use separate clients and leave through different local ports, but I still need to verify whether the sidecar combines or limits their upstream connections.