I recently joined an organization that runs GitHub Actions on self-hosted runners using Actions Runner Controller (ARC) in Amazon EKS. The first job pushed to the workflow remains queued and is not picked up until a second job is submitted, at which point the first job starts running. What should I check first to identify whether the issue is with ARC, the listener, runner startup, or the EKS environment?
5 Answers
If runner images are pulled from ghcr.io, slow or unreliable registry access can make every scale-up look like a queueing problem. Check image-pull events and pod startup times, and consider an approved registry mirror or an internal image cache if your organization allows it. Also review whether EKS nodes have enough capacity and whether network or security policies are delaying access to the registry and GitHub endpoints.
Start with the ARC controller and listener logs while submitting a job. The important questions are whether ARC notices the queued workflow, creates a runner pod, and whether that pod successfully registers with GitHub. If the pod is never created, investigate the scale-set configuration and controller. If it is created but stays pending, check EKS scheduling, permissions, image pulls, and available capacity. If it starts but does not register, focus on the runner image, network access, and listener configuration.
Since the behavior is repeatable, I would look beyond GitHub's queueing and focus on the infrastructure between the workflow and the runner. A stuck runner, delayed pod scheduling, failed image pulls, or a listener that is not maintaining its connection can all produce this symptom. Capture controller, listener, and runner-pod logs during the first job rather than only checking them afterward.
That pattern often points to a cold-start or listener race. Watch the runner namespace while triggering a job with kubectl get pods -n -w and see whether ARC creates a pod immediately. If the pod takes a while to pull its image and register, the second job may simply arrive after the first runner is finally ready. Check the AutoscalingRunnerSet configuration too: if minRunners is 0, there is no warm runner waiting. Setting it to 1 can confirm whether keeping one runner available eliminates the delay. Also inspect the listener logs and verify that it acknowledges the first queued job, that the scale set is connected to the intended runner group, and that the listener is not only reacting when another job arrives.
If there is no specific compliance, networking, cost, or customization requirement for self-hosting, compare the operational effort with managed runners. ARC on EKS adds controller, listener, pod scheduling, image, and networking failure points. That may not be the right immediate fix, but it is worth considering if the organization inherited this setup and does not need its additional control.
Self-hosting may still be required for compliance, private network access, or build dependencies, so I would confirm those constraints before recommending a migration. Even then, measuring the delay between job creation, pod scheduling, image pull, and runner registration should make the root cause much clearer.