How should DNS be handled during blue/green EKS cluster upgrades with external-dns?

0
4
Asked By MellowCedar47 On

I'm planning blue/green upgrades where myService runs in clusterBlue and is brought up ahead of time in clusterGreen. At the moment, external-dnsBlue manages records in the mydomain.com hosted zone, and the service's Ingress creates or updates myservice.mydomain.com through annotations. During the upgrade, both clusters may be running simultaneously. What is the safest way to transfer DNS ownership to external-dnsGreen without the controllers fighting, especially when there may be hundreds of records with different TTLs? Our Ingress resources are included in Helm charts, so deploying the same chart to both clusters would normally make both controllers target the same hostname. Is scaling external-dnsGreen down until the cutover a reasonable approach, or is it better to keep DNS ownership outside the clusters and switch traffic through a load balancer or another layer?

4 Answers

Answered By BriskWillow64 On

If both controllers must coexist temporarily, do not let them independently discover and manage the same hostname. Use distinct hostnames or carefully scoped domain filters and ownership identifiers, and make the handoff an explicit automated operation: stop or restrict the old controller, verify the desired records, then enable the new controller. Test the sequence first, because TXT ownership records and stale state can make a supposedly simple transfer surprisingly fragile.

Answered By QuietOrbit8 On

The cleanest approach is usually to avoid making DNS the blue/green switch. Give each cluster its own hostname and load balancer, such as servicea.blue.mydomain.com and servicea.green.mydomain.com. Keep servicea.mydomain.com under a separate, stable owner—often infrastructure automation—and have it route to the blue and green endpoints using weighted or health-checked records. Then each external-dns instance only manages its cluster-specific names, so there is no ownership handoff or controller race.

SilverPine26 -

That also means the common service name can remain stable while the underlying destinations change. The cluster-specific records can use their own load balancers, listeners, target groups, and certificates as needed.

Answered By UrbanMaple19 On

Scaling the green controller down until cutover can work as an operational guardrail, but it is more of a workaround than a design. It still leaves you coordinating every record and dealing with propagation and TTL behavior. Keeping the main record owned by Terraform or another central DNS layer, with blue and green endpoints behind it, avoids coupling routine cluster replacement to external-dns ownership migration.

Answered By CopperLynx31 On

For a real cutover, a load balancer is generally safer than changing a public DNS record. You can expose both clusters as separate target groups and shift traffic gradually, while health checks determine whether the new cluster is eligible. DNS TTLs and clients that ignore TTLs create a long tail, so even a very low TTL does not guarantee an immediate or complete switch.

AmberNook52 -

If the existing service already terminates at a load balancer, this can often be introduced without changing the application hostname. The load balancer becomes the stable front door and the clusters become replaceable backends.

Related Questions

LEAVE A REPLY

Please enter your comment!
Please enter your name here

This site uses Akismet to reduce spam. Learn how your comment data is processed.