Hey everyone! I'm in the process of improving our microservices' deployment strategy on our Kubernetes (K8s) clusters, and I'm currently dealing with the challenge of updating images that use floating tags like 'sit-latest'. I know this isn't ideal, especially as we transition to GitOps with ArgoCD, but our Dev environment still relies on these 'latest' tags and we can't get the developers to switch to versioned tags just yet.
We run EKS (Amazon's Elastic Kubernetes Service) in a private setup, which complicates things since only admins can access ArgoCD through a VPN. In UAT and Prod environments, image updates happen through commits to our config repos, but in Dev, there's no automatic way for the pods to pull newer images tagged as 'sit-latest'. While setting `imagePullPolicy: Always` helps, we still have to manually restart the deployments using `kubectl rollout restart`, which isn't feasible for our CI setup in GitLab, especially since it can't access the private cluster.
I also contemplated using the `argocd` CLI to manage restarts, but similar access issues hinder that, and I worry ArgoCD Image Updater adds unnecessary complexity. I found a couple of alternatives, such as Keel and Diun, but I'm unsure if they would be too much for our current needs. Any suggestions or personal experiences on better managing these 'latest' tags in Dev environments would be greatly appreciated!
5 Answers
You're hitting the nail on the head with the issues surrounding floating tags like 'latest'. It's not so much an ArgoCD problem but a Kubernetes limitation. One potential fix is to allow developers to deploy directly to their own namespaces using tools like Devspace, Skaffold, or Tilt. That way, they can deploy without needing to commit code every time. Moreover, adopting the git SHA as an image tag upon commits could streamline the process since ArgoCD would then see updates automatically. It’s also worth considering semantic versioning to create tagged images basing on your release cadence!
I recently had to pull off something similar due to an emergency situation. What worked for me was setting a CronJob that restarts the container at scheduled intervals. Just ensure your image pull policy is set to 'Always', and it should regularly update your pods. Here’s a quick snippet of how I set it up! It’s not ideal, but it gets the job done for the time being.
Absolutely! Just make sure your application can handle restarts gracefully. It’s critical to test this setup thoroughly!
Both ArgoCD and FluxCD have this 'image updater' feature designed specifically for situations like yours. It automates monitoring of image updates and ensures your deployments get the latest images without manual actions. If you want to stick with the best practices in the long run, consider visualizing a plan for a gradual shift towards proper tagging. That said, take advantage of these built-in tools for now!
If you're really tied to using 'latest', the ArgoCD Image Updater could help here. It's in beta, so it's not foolproof, but quite a few teams are successfully using it. Still, you should look into switching to versioned tags where possible—it's definitely the best practice. If you're stuck, Image Updater might just rescue you for the time being!
Real talk: your dev process should definitely include proper git tagging. It not only helps with tracking but also ensures you know exactly what version of the image you're using instead of relying on something vague like 'latest'. It may take some convincing, but pushing for versioned tags in your Dev environment is essential!
That’s a clever workaround! Just be cautious with this approach. Restarting containers frequently can lead to disruptions, especially if your applications aren't resilient to restarts. It's a good temporary measure, though!