Dagster jobs running on EKS are intermittently failing to start after Karpenter provisions new nodes. We use s3ComputeLogManager, and the jobs need AWS credentials from the EKS Pod Identity Agent. On newly created nodes, credential requests sometimes time out or disconnect for several minutes, even though the agent logs show that it receives the requests. Dagster then reports that the job could not start because AWS credentials were unavailable.
This only started happening recently and is not consistent. The new nodes have startup taints so workloads are held back until Istio is ready, but the identity agent can still take more than five minutes to respond. That makes me wonder whether the agent is being starved of CPU or otherwise competing with image pulls, Istio initialization, and other node-startup activity. Has anyone seen this behavior or found a reliable way to diagnose or prevent it?
3 Answers
A readiness gate or startup taint could prevent workloads from landing before the identity agent is available, but it may not solve this particular symptom. Since the agent is already receiving requests, it appears to be scheduled and running; the problem may be that it is too CPU-constrained to process them promptly. Check the agent’s CPU and memory usage, throttling, node pressure, and recent changes to its DaemonSet or resource limits before adding more scheduling gates.
It’s also worth isolating Istio. Temporarily exclude the Dagster jobs from sidecar injection, or enable Istio’s option to hold application startup until the proxy is ready. If the delays disappear, proxy initialization or startup resource contention is likely contributing to the credential timeouts.
The resource-starvation theory is worth testing first. The identity agent may have fairly small resource requests, while a new Karpenter node is also pulling images and starting Istio. Compare the agent’s CPU throttling metrics—especially container_cpu_cfs_throttled_seconds_total—on a node during its first few minutes versus a node that has been running for an hour. Throttling would fit the intermittent delays and could explain why the issue appeared after a workload or DaemonSet change.
That makes sense—if requests are reaching the agent, simply waiting for the DaemonSet to become ready probably won’t address a process that is already running but being starved during node startup.