I've got a setup with three identical DigitalOcean VMs, each running over 500 containers for a Node.js app. While VM1 is working flawlessly, both VM2 and VM3 are facing a TypeError: fetch failed (undici) to Supabase HTTPS along with other sources. The issue seems to kick in when I hit around 510-530 containers, even though VM1 can easily handle up to 900 without a hitch. Here's the scoop:
- All VMs are running Ubuntu with Docker version 26.1.3.
- The network setup is using the default docker0 bridge with UFW active and FORWARD=DROP.
I've tried various troubleshooting steps like switching DNS servers and removing custom bridge networks, plus I staggered container starts which provided minor improvements. I can connect to the containers, and DNS resolution is fine. However, I'm on the verge of cloning VM1 to fix VM2 and VM3, which would mean an annoying downtime for my clients. Anyone have insights or solutions to avoid this?
3 Answers
I noticed you mentioned using Docker 20.x, but from your version, it looks like you are up to date. Have you confirmed whether the fetch issue is tied to the networking or something specific to VM2 and VM3? Seems like there could be a config issue since VM1 is performing well while the others aren't.
First off, let's check the resources of your VMs. 500 containers could be a lot depending on the specs of your VMs. You mentioned CPU at 23% and memory at 57%, which doesn't scream overload, but traffic profiles can change things. Have you considered using Kubernetes? It might help manage things better at this scale.
I hear you on the resource checks! Everything's been steady until a freak restart just recently. I was running 800 containers smoothly before without any hiccups.
What exactly are these containers doing? The resource usage seems okay, but can they be optimized further? Are there any performance metrics you can share?
They're executing strategies for a web3 crypto game, making API calls and handling transactions here and there. It was all fine until those sudden VM restarts messed with the performance.
Thanks for pointing that out! I was hoping it wasn't just a docker hiccup. I guess I'll have to look more closely at the networking setup on those other VMs before I make drastic changes.