I'm rebuilding the networking on a Docker host with addresses on VLAN-10 and VLAN-30. I need every service to be reachable from my LAN through one of the host's IP addresses, while keeping two independent Compose projects. Containers in either project should be able to reach services in the other by name, such as http://navidrome, without hard-coded container IPs.
Some containers must use 192.168.30.1, a VPN gateway on 192.168.30.0/24, for Internet access. Others should use the normal gateway at 192.168.10.1. The configuration should survive host and container restarts and ideally be managed mostly from the Compose files.
My current design creates bridge networks with custom IPAM settings and uses policy routing plus separate masquerading rules for the two egress paths. It mostly works, but startup is unreliable because Docker networks and their bridge interfaces do not exist when the host routing rules are initially applied. I'm considering redesigning this rather than continuing to patch route persistence.
What would be a clean, maintainable architecture for this? In particular, how should the shared service-name DNS and the two egress paths be configured while minimizing custom host routing and firewall maintenance?
2 Answers
Manage the special networks outside Compose and mark them external in both files. Create them once with docker network create, then let Compose only attach containers to them. This avoids Compose files racing with boot-time routing setup and makes the network names and subnets stable.
For example, use one external shared bridge for cross-project DNS, plus one external bridge for normal egress and another for VPN egress. A container joins the shared network and whichever egress network it needs. Configure the host’s policy-routing tables, firewall marks or source-based rules, and masquerading through a boot service that runs after Docker has started and waits until the expected bridge interfaces exist. Ansible or another configuration-management tool can create the networks and install those rules repeatably.
Health checks and dependency ordering can help applications start in the right order, but they do not replace routing configuration. If possible, putting the two gateways and policy routing on a dedicated router or firewall is cleaner than maintaining custom rules directly on the Docker host.
Create one shared user-defined bridge network for the services that need to communicate across both Compose projects. Declare it as an external network in each file, for example a network named shared-services, and connect the relevant containers to it. Docker’s embedded DNS will then resolve service and container aliases across both projects.
Give each project its own additional private networks as needed. Publish LAN-facing services through the host’s VLAN addresses with normal ports or host-specific bindings; a bridge network does not give every container a separate address on your physical LAN.
For egress, attach a container to a dedicated network for each routing policy. The actual choice between 192.168.10.1 and 192.168.30.1 still has to be implemented on the host or by a dedicated router/firewall container. Compose can define the network attachments, but it cannot by itself make host policy routing and masquerading restart-safe.
That makes sense for the shared DNS network, but I’m still unclear about the egress networks. Would each Compose file define a separate bridge with its own subnet, and would the host then route each subnet through the desired gateway?