I'm looking to retire our current Ingress controller and am exploring options with respect to the Gateway API for multi-tenant Kubernetes clusters, especially in the context of Azure Kubernetes Service (AKS). We've evaluated a few solutions, including the Envoy Gateway API, which offers multi-tenancy options according to their documentation, but it seems the versions available aren't stable yet.
We've also considered using App Gateway for Containers, but that requires substantial architectural changes which complicate our existing design—specifically, we already have App Gateways in front of our AKS clusters. Furthermore, AGC lacks private IP frontends, and managing several AGC instances for different AKS clusters would be expensive and problematic to configure. I still have some temporary measures in place for the current setup, but I'm looking for a long-term solution. I'd appreciate any thoughts or suggestions regarding this challenge!
5 Answers
If you’re considering Envoy Gateway, the implementation can be stable for multi-tenant setups if you set it up correctly. I've done this for large tenant clusters with their own issuers. However, certificate management remains a headache! Listener sets can help manage complexity, but you should weigh that against your current architecture's requirements.
Yeah, that's true. Until the documentation improves, it feels risky to base enterprise solutions on that.
You might want to check out Traefik. It's worked well for us and is pretty straightforward for most use cases. I found it easier to implement compared to some other options, especially if you're already using a Kubernetes distribution that supports it.
Have you faced any significant issues migrating from ingress-nginx to Traefik? I didn't find their documentation super helpful myself.
Are you using the OSS or Plus version of Nginx? Just curious!
I'm currently transitioning from ingress-nginx to Envoy Gateway on AKS. Initially, I tried Nginx Gateway Fabric, but it lacked features I needed, like client certificates support. I like that Envoy provides nice Grafana dashboards and I've managed to implement various policies successfully.
Did you implement this setup for multi-tenant clusters? I'm curious about how you managed the design for gateways and controllers.
For a large enterprise, careful planning is essential with multi-tenancy. What approach are you taking for certificate management?
Your concerns about the Envoy Gateway are valid. It does seem early for strict multi-tenant production use, and switching to AGC looks like a heavy lift considering your current architecture. I’d suggest sticking with your current Ingress setup for now while testing the Gateway API on the side to see how it matures.
What specific challenges have you faced with the Gateway API that make it feel unready for enterprise use?
I think Cilium might be a little less mature compared to Envoy. It could be a risk.
HAProxy is a solid choice. The open-source version is very robust, and in my experience, it outperforms many of the other solutions. I even built a custom authentication system on top of it. It's surprising how few folks recommend HAProxy considering its capabilities!
We evaluated HAProxy, but it doesn't support all the annotations we use. It seems a bit limiting for our needs.
I've heard it doesn't allow multiple replicas without going enterprise, which is a big drawback for some.
But the Envoy Gateway API documentation suggests the multi-tenancy features aren’t stable yet—how can you deploy that for enterprise environments?