I'm deploying a Selenium Grid across 91 Docker Swarm workers and one manager. The servers were cloned from the same VM image and all have joined the swarm, but 11 workers fail to start their services. A rejected task reports: "node is missing network attachments, ip addresses may be exhausted."
The affected overlay network uses 10.0.2.0/24. Its inspection output shows 256 IPs in use and no dynamic addresses available, even though the swarm contains only 92 nodes. The service has a virtual IP of 10.0.2.156/24.
Why can the network report all 256 addresses as used? Are these stale task or endpoint allocations that can be safely cleaned up, or should I create a larger overlay network?
2 Answers
The more durable solution is to create a new overlay network with a larger subnet, such as a suitably planned /16, and update the stack to use it. For example:
`docker network create --driver overlay --attachable --subnet 10.10.0.0/16 selenium-overlay-v2`
Then add the new network to the service or stack and remove the old network after all tasks have migrated. Make sure the subnet does not overlap with your host, VPN, or other Docker networks. If this is deployed from a stack file, update the stack definition rather than changing only one service, otherwise the next stack deployment may restore the old network.
The /24 overlay is exhausted. The IP count is not just the number of swarm nodes: every task and service endpoint attached to that network can consume an address, and failed or rapidly replaced tasks may keep allocations around until the swarm finishes cleaning them up. A /24 also has fewer usable addresses than the nominal 256.
You can temporarily scale the affected service to zero, wait for the old tasks and endpoints to disappear, and then scale it back up. Also inspect which services and containers are attached to the network before removing anything. This may recover leaked allocations, but it is only a short-term fix if the network is shared by many services.
I’m going to try scaling the service down and back up first, then check whether the IP usage drops after the old task records are removed.
A larger network is probably the right long-term fix here. I’ll verify the address ranges first and migrate the stack instead of relying on manual service updates.