Hey everyone! I just got my hands on a cheap RTX 3090 and added it to my home server for some AI experiments. The issue I'm running into is that this new GPU takes way longer to initialize than my old GTX 1650. As a result, my GPU-dependent Docker containers fail to start right after a reboot. It's a bit of a hassle, especially when my family is home and they start bugging me because Jellyfin is down.
I found that I can add a startup dependency for mount points in the docker.service using systemctl, but I'm looking for a way to implement a similar delay specifically for the NVIDIA CDI. I'm fine with a simple workaround like adding a 30-second wait if that helps. I'm hoping someone here has faced a similar situation and knows how to tackle it. I've tried searching online but didn't come up with much useful info. By the way, I'm using a Debian 13-based distro.
1 Answer
If you're using Docker Compose, you might want to try implementing a health check. It can help ensure that your containers start only after the NVIDIA CDI is ready. Just look into how health checks work, and you could set it up to check for the specific CDI output that indicates it's running. That way, you won't have to rely on a fixed wait time!
That sounds like a solid plan! I’ll try to gather more info on health checks. I found some documentation but most health checks seem geared towards endpoints. I might just have to write a simple script to check the CDI status instead.