Hey everyone! I have two Docker hosts set up and configured somewhat differently. Both of them are running containers with the `restart: unless-stopped` policy, but after shutting down and rebooting them, only one host restarted its containers automatically. The other host required me to run `docker compose up` manually to get things going again. Does anyone know what might be causing this discrepancy?
I just noticed that the host that failed has its logging set to use a GELF endpoint, which is another container running on the same host, as specified in `daemon.json`. I'm wondering if that could be the issue since the logs showed an error indicating it couldn't connect to the GELF endpoint. Could this misconfiguration be the culprit? Also, is there a way to make the logging work with a local container globally?
3 Answers
Here's the thing: when Docker restarts or the host itself reboots, all containers are stopped first before the daemon restarts, so they won't automatically come back online if they were previously stopped. For your case, you might want to switch to using the `always` restart policy instead of `unless-stopped`. That could help ensure they boot up automatically after a reboot.
That’s not completely accurate. Docker's restart policy gets reset when the daemon restarts. So, any container set to `always` will start when the daemon does, regardless of whether they were stopped before. Just something to keep in mind when configuring your containers!
Have you checked your `daemon.json` file? It seems like it might be a configuration error or possibly something like a firewall blocking access to your logging server.
It sounds like you created a loop with your logging setup. The Docker daemon is trying to send logs to a GELF endpoint that’s actually a container on the same host. When Docker starts up, it attempts to connect to that logging service before it's even up, which causes it to fail.
A good solution would be to run your logging endpoint on a different host or even directly on your machine. You might also consider not applying the GELF logging driver to every container on that host. You can define different logging drivers per service in your docker-compose file. For example, keep the default logging driver for the daemon and only use GELF for the specific containers that need it, which should help avoid your startup issues.
Thanks for the tip! I removed the logging config from `daemon.json`, and everything started up properly now.
Actually, that’s not quite right. Based on my experience, with `unless-stopped`, as long as Docker wasn't actively stopping those containers, they should come back on after a reboot just fine. I’ve seen this on my laptop: containers that were running come back online after restart, but those that were manually stopped stay down.