I'm a junior engineer designing a production-oriented reference architecture for on-premises Kubernetes and implementing it as reproducible Ansible code. The goal is not just to automate a cluster installation, but to begin with failure domains and operational requirements, encode those decisions as infrastructure, and make unsafe topologies fail preflight checks before deployment.
The current design distributes etcd across independent racks, separates control-plane, etcd, ingress, egress, and worker roles with taints, and uses dedicated ingress nodes for north-south traffic. Cilium provides eBPF networking, BGP, network policy, and Egress Gateway. Traefik handles operational and internal applications, while Istio is reserved for workloads that need features such as canary releases, weighted routing, blue-green deployments, or mTLS. Stateful systems such as PostgreSQL, Redis, and object storage are treated as separate failure domains rather than assuming that pod distribution alone provides high availability.
I also want to validate assumptions through measurements and failure tests, including rack and node failures, BGP convergence, egress gateway failures, MTU behavior, storage and network latency, and multi-cluster observability. Thanos is one option I'm considering for long-term metrics storage and cross-cluster querying.
What would you change before using this in a real enterprise environment? In particular, are rack-aware etcd placement and dedicated ingress or egress nodes worthwhile, is the Traefik/Istio split justified, are there hidden failure domains or single points of failure in the Cilium BGP and Egress Gateway design, and which operational tests should be mandatory? I'm especially interested in practical constraints such as IPAM, router state, storage, node lifecycle, disaster recovery, legacy systems, procurement, and ownership. The intended workflow is architecture, Ansible, preflight validation, deployment, failure testing, review, and iteration rather than simply installing Kubernetes and checking that the nodes are ready.
5 Answers
Cilium BGP and Egress Gateway can work well, but test the behavior rather than assuming it. Measure route advertisement and withdrawal time, failover during node and link loss, return-path symmetry, connection tracking, NAT and port exhaustion, overlapping address ranges, and what happens when the gateway is drained or isolated. Check how external routers handle route churn and whether the network team can operate the design.
Dedicated ingress nodes can simplify firewall rules and capacity planning, but they also reduce the number of possible failover targets. Depending on the network, L2 announcements or a proper Gateway API implementation may provide the same exposure without permanently reserving a small node pool. The choice should follow a tested security and traffic-isolation requirement, not just role-based symmetry.
The foundation is thoughtful, but a universal on-prem reference architecture is probably too broad. Rack layout, dedicated ingress, service mesh, storage, and GitOps should be presented as selectable patterns for different use cases rather than mandatory components. Start with the smallest stable design that meets a specific requirement.
Important pieces still need to be defined: how LoadBalancer services obtain addresses, how IPAM and router state are managed, where persistent storage comes from, whether storage follows workloads, and how nodes and the underlying hardware or hypervisor are replaced. External Ceph, a clear network integration model, and infrastructure automation for VM or node lifecycle can reduce complexity. A management cluster running a GitOps controller can also help prevent configuration drift.
That makes sense. I need to separate the core architecture from optional patterns and document the infrastructure integration, storage, and lifecycle assumptions instead of treating them as implementation details.
Running Traefik and Istio together can be justified if their boundaries are very clear, but it is easy to create duplicate routing, policy, certificate, and upgrade workflows. If the main requirements are Gateway API, TLS, and ordinary ingress, one gateway implementation may be enough. Use a service mesh only where its traffic-management or identity features have a concrete business requirement.
Gateway API is generally a better direction than building new behavior around the older Ingress model. If using Istio Ambient, verify how its HBONE traffic interacts with Cilium network policies and authorization policies; policies written only for ordinary pod traffic may not behave as expected once traffic is encapsulated or redirected.
Rack-aware etcd placement is valid when the racks really are independent failure domains and the network between them has predictable latency and bandwidth. It does not create high availability by itself; power, top-of-rack switches, upstream routing, shared storage, management systems, and the facility must also be examined. Three members across three genuinely independent domains is usually more useful than placing extra members in one failure domain.
Also plan for the disaster that takes out the whole site. A second location, tested etcd backups, restore procedures, application-level replication, and recovery objectives matter more than adding roles to the cluster. Stateful platforms such as Ceph or database clusters need their own placement and failure-domain policies.
For a multi-cluster environment with high metric volume, Thanos can be a reasonable choice, especially when paired with Prometheus for local collection and object storage for long retention. Plan for cardinality control, labels that identify cluster and failure domain, compaction and deduplication behavior, query fan-out, object-store outages, and the cost of retaining high-resolution data. It is not a substitute for defining useful recording rules and alert ownership.
Before production, run controlled tests for rack and site loss, etcd quorum and restore, control-plane recovery, CNI and MTU problems, BGP convergence, gateway failure, DNS failure, certificate expiry, storage degradation, noisy neighbors, and observability loss. Record recovery time, data loss, alerting delay, and the manual steps required; those results should drive the architecture more than the component list.
The dedicated nodes were mainly intended to limit exposed firewall addresses and separate public traffic from internal management traffic. I'll compare that against L2 announcements and Gateway API while measuring failover capacity and the security boundary explicitly.