We are setting up more than 90 identical servers for Selenium Grid load testing. All 91 workers are joined to the same Docker Swarm, but 11 of them fail to deploy services.
For one affected service, `docker service ps --no-trunc` shows repeated task failures with:
`node is missing network attachments, ip addresses may be exhausted`
The affected worker's Docker logs show the same fatal task error. Inspecting the service and overlay network shows that the service has a virtual IP of `10.0.2.156/24`, while the network reports:
`IPsInUse: 256` and `DynamicIPsAvailable: 0`
The overlay subnet is `10.0.2.0/24`, so it has only 256 addresses. However, the Swarm contains just one manager and 91 workers. Why can the network have all 256 addresses in use? Are stale task or VIP allocations safe to clean up, or should the overlay network be replaced with one using a larger subnet?
2 Answers
The count is for IP allocations on the overlay network, not the number of Swarm nodes. Every service task, network endpoint, and some service-level virtual IPs can consume an address. Failed and restarted tasks can also leave allocations around temporarily, and other services may be using the same network. A `/24` can therefore be exhausted well before you have 256 workers.
First inspect which services are attached to the network and how many tasks they have. Removing obsolete services or unused networks is the proper cleanup method; there is no safe command to manually prune individual VIPs. You can also temporarily scale an affected service down, wait for task cleanup, and scale it back up, but that only helps if the allocations are genuinely stale.
Use a larger overlay subnet for this workload. A `/16` provides far more room, but an existing overlay network generally cannot simply be resized in place. Create a replacement network with a non-overlapping subnet, update or recreate the services to use it, verify that everything is running, and then remove the old network once nothing depends on it.
Also account for every task replica and every service sharing the network—not just the number of physical nodes. If each worker has its own service and tasks are repeatedly rejected and recreated, the address demand can grow quickly.
Before migrating, confirm that the new subnet does not overlap with the host, container, VPN, or other Swarm networks. Repeated task failures can also indicate stale Docker state, so check the affected workers and managers after removing obsolete services.
Scaling down one service may release its task endpoints, but it will not reclaim addresses belonging to other services or currently running tasks. Check the whole network before assuming the failed tasks are the only cause.