I'm a junior engineer designing a production-oriented reference architecture for on-premises Kubernetes and implementing it as reproducible Ansible code. The goal isn't just to automate an installation; it's to begin with failure domains and operational requirements, encode those decisions, and make unsafe topologies fail preflight checks before deployment.
The current design accounts for physical rack boundaries, distributes etcd across independent failure domains, separates control-plane, etcd, ingress, egress, and worker roles, and uses taints to keep workloads on the intended nodes. North-south traffic is handled through dedicated ingress capacity instead of exposing every Kubernetes node. Cilium provides eBPF networking, BGP, network policy, and egress gateway capabilities. Traefik is intended for operational and internal applications, while Istio would be reserved for workloads that need features such as canary releases, weighted routing, blue-green deployments, or mTLS.
Stateful systems such as PostgreSQL, Redis, and object storage are treated as separate availability concerns rather than assuming that spreading pods across nodes automatically provides high availability. The plan also includes testing node and rack failures, BGP convergence, egress gateway failures, MTU behavior, storage and latency characteristics, and other operational assumptions before production use.
I'd appreciate practical criticism of the design. Are rack-aware etcd assumptions valid? What hidden failure domains or single points of failure should I look for? Is separating Traefik and Istio justified, or is it unnecessary complexity? Are there pitfalls with Cilium BGP and egress gateways? Which parts would you change for large-scale operation, and what tends to make this kind of architecture impractical in an enterprise? For a multi-cluster environment with high metric volume, would Thanos be a sensible option for long-term storage and cross-cluster observability? What additional failure tests or measurements should be performed before deployment?
5 Answers
The technically optimal design is often not the design an enterprise can actually adopt. Existing contracts, security policies, procurement, vendor support, legacy networks, storage platforms, and team ownership frequently determine the architecture. A service mesh can be excellent technically but still fail because no single team can operate the interactions between networking, security, the platform, and applications.
Physical rack awareness is valuable, but it should integrate with the capabilities of the storage system too. Platforms such as Ceph may need their own failure-domain configuration, and Kubernetes pod placement alone does not protect stateful data. Include the second site or disaster-recovery location in the design as well.
Running both Traefik and Istio may create more operational burden than value. They overlap in routing and Gateway API functionality, so choose based on concrete requirements rather than assigning one to internal traffic and one to advanced workloads by default. Istio’s ambient mode and gateway components may already cover the routing and policy needs you have in mind.
Also consider moving away from the older Ingress model and designing around Gateway API. Be careful when combining service-mesh policies with ordinary Kubernetes NetworkPolicies: mesh traffic can be encapsulated or redirected, so policies that work for plain traffic may not behave as expected.
The scope is probably too broad for one universal reference architecture. Rack layout, dedicated ingress nodes, service mesh, storage, and network integration all depend heavily on the use case. It may be better to define a small baseline and then provide variants for regulated workloads, high-ingress workloads, stateful systems, and simpler clusters.
Also make the infrastructure integrations explicit: how LoadBalancer services receive addresses, how IPAM and routing are managed, where storage comes from, whether storage follows workloads, and how the cluster connects to the rest of the environment. Minimizing the number of moving parts will usually make the platform easier to operate.
That makes sense. I was considering NodePort on dedicated ingress nodes to simplify firewall rules, but I’ll compare that with a proper load-balancer implementation and document the trade-offs rather than assuming NodePort is automatically simpler.
Before calling this production-ready, define the business requirements and test the boring infrastructure pieces: IP address allocation, router state, storage-class behavior, data locality, storage-network failures, node replacement, hypervisor or bare-metal lifecycle, backup restoration, certificate expiry, and recovery at a second site. A management or GitOps cluster can help keep workload configuration consistent and reduce drift.
A simpler platform is often easier to operate at scale. In some environments, virtualized nodes with infrastructure-as-code, a straightforward CNI, and externally managed storage are more reliable than combining several networking, ingress, mesh, and storage systems. Thanos can be a reasonable choice for long-term, multi-cluster metrics, but validate object-storage durability, query cost, retention, cardinality, and failure behavior with representative volumes before committing to it.
Cilium is a reasonable foundation, but egress gateways add routing and troubleshooting complexity. Use them when stable source addresses or a security requirement justifies the cost, and test failover, conntrack behavior, asymmetric routing, return paths, and upgrades. For BGP, measure convergence and verify what the upstream routers do during withdrawal and restart events rather than assuming the control plane will fail over cleanly.
For ingress, compare dedicated nodes with L2 announcements or an external load balancer. A lease-based announcement can avoid hard-coding a particular node, while a dedicated pool can simplify firewall rules but reduces the number of possible failover targets. The right choice depends on traffic volume, router behavior, and operational ownership.
The separate ingress tier was mainly intended to simplify firewall policy and isolate internet-facing traffic from internal management traffic. I’ll evaluate whether Gateway API, ambient mesh features, and load-balancer announcements provide that isolation without requiring a small dedicated node pool.