We need to upgrade an EKS cluster with minimal downtime, but both the current and replacement clusters would mount the same EFS filesystem. Several workloads are stateful, and data integrity matters more than making the cutover completely seamless. What risks or migration patterns should we consider?
Can blue-green EKS upgrades safely share one EFS filesystem?
Answers
The main danger is not the cluster upgrade itself. It is having two clusters run the same stateful application against the same files at once. Many databases and streaming systems expect exclusive ownership of their data directory. Sharing a filesystem does not make those applications safe for concurrent access, and a live copy or second instance can corrupt data.
For a blue-green cutover, treat each stateful workload separately. Stop writes on Blue, confirm the application is shut down cleanly, then start it on Green and verify recovery before sending traffic. If an application supports replication or a documented migration procedure, use that instead of mounting its live data directory in both clusters. A short maintenance window may be safer than promising zero downtime.
Reply to this answer
I would separate the storage question from the cluster upgrade question. EFS can be mounted from more than one cluster, but that does not mean every workload can safely share its files. For databases and systems such as Kafka or OpenSearch, check the vendor's supported storage and migration guidance before assuming an EFS-backed data directory is valid.
Also, the access point and directory settings deserve a careful test. Reusing access points and disabling unique directory naming can make two claims resolve to the same location, while automatically assigned POSIX IDs may not give you the ownership layout you expect. Verify the actual paths, access point IDs, UID/GID behavior, and permissions with disposable claims before production. Keep backups and a tested restore path; a retained volume setting is not a backup.
Reply to this answer
For the GID issue, I would avoid depending on sequential allocation as a stable identity scheme. If workloads need predictable ownership, explicitly define and test the UID/GID expectations, and make sure separate PVCs cannot accidentally map to the same directory. The exact CSI behavior depends on its version and configuration, so validate it against the driver documentation and a small test filesystem.
Do not let both clusters provision claims with identical namespace and claim names unless sharing that directory is intentional. A green cluster can otherwise appear healthy while quietly attaching to Blue's data. Make the cutover checklist include a path and access-point check, not just a pod-ready check.
b4!Clapper:
That makes sense. We should test whether matching claim names actually resolve to the same access point and directory before trying this outside a test environment.
Reply to this answer
One filesystem for everything is a pretty sharp constraint when the list includes databases, caches, monitoring data, and streaming systems. Their persistence and failover assumptions differ a lot. Some data may be disposable or rebuildable, while other services need replication, snapshots, or application-level recovery. I would inventory those requirements first rather than treating all PVCs alike.
If the restriction rules out block storage, consider whether the application can use its own supported replication or external managed storage pattern. If not, plan a controlled stop-and-start migration and accept a brief outage. Traffic routing can be manual, but it cannot make two writers to one data directory safe.
wicket-0Q:
We cannot change the storage architecture quickly, so a planned outage for the most sensitive services may be the realistic option while we review the longer-term design.
Reply to this answer
I'd make the upgrade rehearsal boring and reversible. Restore representative backups into a nonproduction environment, bring up Green, verify application-level data, then test the exact shutdown, mount, and startup sequence you intend to use. Keep Blue available but prevent it from writing once the handoff begins. Have a clear rollback point, because rolling back after Green has accepted writes may require data reconciliation.
Also check whether each workload actually needs its persistent files moved this way. For some services, rebuilding from a backup or using a supported replica promotion may be cleaner than having both clusters share the same EFS paths.
Reply to this answer
Add your answer
Answers are reviewed before publication.
E6__Pumice:
So the shared EFS mount is not itself the safety mechanism. We would need to ensure only one cluster runs each stateful service at a time?