Hey folks! I'm exploring the possibility of deploying Istio in Ambient mode, which is a sidecar-less service mesh, on my Talos-based Kubernetes cluster. Before I dive into the installation, I'm curious if anyone here has successfully done this. Specifically, I'd love to know if there are any challenges with Talos's minimal host environment, which lacks features like `nsenter` or SSH. Also, did you have to adjust the CNI setup, whether it was Flannel, Cilium, or Istio's own CNI? Lastly, which version of Istio did you use, and was `ztunnel` or the ambient data plane working smoothly? I've heard that Istio 1.15+ improved compatibility for minimal host OSes, but concrete feedback from Talos users on Ambient mode seems scarce. Any insights, configurations, or tips would be greatly appreciated! Thanks!
5 Answers
I managed to get Talos working with Cilium and Istio Ambient just fine. The key was turning off exclusivity for Cilium. Besides that, the setup was pretty straightforward. It's even running well in production, so it seems stable!
I've had decent luck with Ambient mode using Istio 1.27+. It usually works out of the box once the right annotations are set on your namespace or deployment. However, if you're using Cilium as your CNI, it's best to stick to using Istio for the service mesh instead. Cilium's current integration isn’t very mature and can lead to unclear errors when things go wrong. Just a heads-up!
I set up Istio with Cilium in my home lab and it has been a bit of a rollercoaster. Make sure your Cilium configuration allows CNI chaining by setting `cni.exclusive=false`. I also advise against replacing kube-proxy in your config to avoid issues during setup. I recently upgraded to Istio 1.27.2 and it has been stable. If everything works well, consider contributing your setup to the documentation - that would be super helpful!
Will do! I'm gathering everything I learn for a potential PR to the docs.
I attempted to set this up with Cilium, but the entire cluster became unresponsive right after the Istio installation. Unfortunately, I couldn’t troubleshoot it since I lost connectivity entirely. I think there might've been an issue with Istio creating iptables rules when a pod has `hostNetwork: true`. That's what ended my attempts prematurely. Just something to watch out for!
That sounds rough! I had a similar experience but figured out that a misconfiguration in the iptables rules was to blame. I hope you can get it running next time!
If you're successful with your setup, sharing that experience would be amazing for other users facing the same issues. Community contributions like that really help everyone out!
Absolutely! I've found Istio's status reporting much more reliable than Cilium's. If you go with Istio, you'll get clearer error messages, which is a huge plus for debugging.