We run Dagster jobs on EKS with S3 compute logs. When Karpenter adds a node for a job, the job sometimes fails to start because it cannot obtain AWS credentials from the EKS Pod Identity Agent. The failures are intermittent and began recently. The agent logs show credential requests arriving, but in some cases it does not respond for more than five minutes, resulting in connection errors or timeouts from Dagster.
The node also has startup taints so workloads should wait until Istio is ready, although the Pod Identity Agent still appears unusually slow. The sensor successfully triggers the run, but the actual job is scheduled onto a newly created node. There are no obvious agent errors, so resource starvation during node startup is one possibility. What should we check, and how can we prevent jobs from starting before the identity agent is actually responsive?
4 Answers
The five-minute delay could indicate that the agent is being heavily throttled while the new node is pulling images and starting system components. Compare the agent container's CPU throttling metrics—especially `container_cpu_cfs_throttled_seconds_total`—on freshly created nodes versus nodes that have been running for a while. Check its CPU and memory requests and limits as well. If throttling is much higher during node initialization, increasing the agent's resources or reducing startup contention may resolve the intermittent credential timeouts.
Make sure workloads cannot land on a new node until the Pod Identity Agent is ready, not just until the service mesh is running. A startup taint or scheduling gate can be removed only after a node-local readiness check confirms that the agent is listening and successfully handling a credential request. This is more reliable than assuming that a running DaemonSet pod is ready immediately.
Istio may be contributing during node startup. Temporarily exclude the Dagster job pods from sidecar injection and compare the results. If injection is required, configure the proxy to remain in the init sequence until it is ready, such as enabling `holdApplicationUntilProxyStarts`. Also check whether the proxy or its init container is consuming enough CPU during startup to starve the identity agent.
Check the timeline on the affected nodes: node registration, DaemonSet scheduling, identity-agent readiness, Istio startup, and the first credential request. Also verify that the agent version and its resource settings have not changed recently. Since IAM role propagation would not normally explain the agent receiving requests but failing to answer them, local node contention or startup ordering seems more likely than an IAM policy problem.
That makes sense for preventing early scheduling, but the agent logs already show that it receives requests and then stalls. I’ll still test a readiness-based taint removal to separate startup ordering from resource pressure.