Hey folks! I'm currently managing a Kubernetes cluster powered by Talos, and I'm interested in setting up Istio in Ambient mode, which is a sidecar-less service mesh option. Before I dive in, I'm curious if any of you have successfully deployed Istio Ambient on a Talos cluster. I'm especially interested in hearing about any challenges related to Talos's immutable and minimal host setup (like the lack of `nsenter` and SSH). Did you have to adjust the CNI configuration (like Flannel, Cilium, or Istio's own CNI)? Which version of Istio did you use, and were you able to get `ztunnel` and the ambient data plane working without issues? I've read that Istio 1.15 and above have made strides in being compatible with minimal host operating systems, but I haven't found much input from Talos users yet. Any insights, configurations, or tips would be greatly appreciated! Thanks!
4 Answers
So, I’ve been using Istio in Ambient mode with Cilium as my CNI on a Talos cluster, and I can confirm that it works out of the box as long as you’re using Istio 1.27 or newer. Older versions can cause socket exhaustion issues. Just make sure you set the right annotations on your namespaces or deployments. If you’re going with Cilium as your CNI, it’s important to use Istio as the mesh because Cilium’s mesh capabilities are still developing and can lead to frustrating TCP connection resets with little feedback on what’s wrong. Overall, though, once set up, it runs smoothly!
Yes, I’ve successfully run Talos with Cilium and Istio Ambient together. The key is to ensure exclusivity is turned off for Cilium. I didn’t face many challenges; everything just worked as expected, both in production and in my dev setup.
I attempted to deploy Istio with Cilium but ran into a wall; after installation, my entire cluster became unresponsive, which was a total disaster. I couldn’t even access it through Kubernetes or Talos. After several unsuccessful attempts with different settings, I gave up due to the lack of visibility on what went wrong. It was really frustrating!
I faced something similar! There was a bug where Istio applied iptables rules incorrectly for host networks, especially when running services with `hostNetwork: true`. It caused nodes to become unreachable immediately. That could’ve been your issue as well.
I’ve been playing around with this setup myself in my home lab. Actually, I have Istio with Cilium set up, similar to what we have at work with GKE. I had to do several installations and removals to get it right. When I upgraded to Istio 1.27.2, everything clicked! Just a heads up, if you’re using Cilium, set up your CNI chart properly (cni.exclusive=false) and double check these security contexts and ipam settings. It should eliminate a lot of potential headaches when integrating with Istio.
Agreed! When I initially set up Cilium, I ran into issues after replacing kube-proxy. It caused a lot of hiccups before I even got Istio running. Best to keep kube-proxy in your config if you can.