We are running an EKS cluster in private subnets and use one AWS EFS filesystem in Elastic Throughput mode as the shared storage layer. Due to architectural and compliance constraints, we cannot move to EBS or gp3. We want to perform a zero-downtime blue-green cluster upgrade: the existing Blue cluster would continue serving production traffic while a Green cluster runs the target EKS version. Traffic cutover through the load balancer and DNS would be handled manually, so routing is not the main concern.
Both clusters would mount the same EFS filesystem through the AWS EFS CSI driver. We use dynamic provisioning with a StorageClass similar to this:
provisioningMode: efs-ap
fileSystemId: fs-xxxxxxxxxxxxxxxxx
reclaimPolicy: Retain
basePath: /dynamic_provisioning
subPathPattern: ${.PVC.namespace}/${.PVC.name}
ensureUniqueDirectory: false
reuseAccessPoint: true
gidRangeStart: 50000
gidRangeEnd: 1000000
deleteAccessPointRootDir: true
The environment contains many stateful systems, including MySQL, PostgreSQL, MariaDB, OpenSearch, MongoDB, Redis, Memcached, DuckDB, Kafka, Airflow, Flink, DataHub, Prometheus, and Grafana.
There are two major concerns. First, EFS CSI dynamically assigns POSIX GIDs sequentially. If namespaces and PVCs are created over time, the assigned GIDs become fragmented rather than staying within predictable per-application or per-namespace ranges. We need deterministic access boundaries without hardcoding every individual GID or forcing all containers to use a common 1000:1000 identity that conflicts with application security contexts.
Second, during validation of the Green cluster, both clusters could mount the same EFS access point and data directory at the same time. For databases such as MySQL using InnoDB, having two instances access the same files could cause startup failures, lock contention, split-brain writes, or transaction-log corruption. We also cannot simply generate separate access points and copy data manually for every upgrade because that would be error-prone and would prevent the Green StatefulSets from using the existing data directly.
Is there a safe way to use EFS for this upgrade pattern, or is the architecture fundamentally unsuitable for shared stateful workloads? What would be the recommended approach for preserving data integrity while keeping the upgrade downtime as low as possible?
4 Answers
The GID issue is separate from the blue-green database problem. The EFS CSI driver allocates access-point identities from its configured range; it does not natively reserve a clean, deterministic GID block per namespace while also automatically managing arbitrary future PVCs. Reusing access points and disabling unique directory suffixes can make paths predictable, but they do not create transactional allocation or namespace-level GID partitioning. If deterministic identities are mandatory, you generally need an explicit provisioning convention, separately managed access points or StorageClasses, and carefully coordinated POSIX ownership and security contexts. Avoid changing ownership or permissions on a live shared directory as part of a cluster upgrade.
The safest answer is that both clusters must not run two writable instances of the same database against the same EFS directory. EFS provides shared file storage, not database-level replication, fencing, or split-brain protection. A second MySQL, PostgreSQL, MongoDB, or similar process can fail to start, but relying on file locks to protect the data is not a safe high-availability design. For the cutover, keep the Blue database as the sole writer, stop or fence it cleanly, and only then start the Green instance on that data. That introduces at least a controlled maintenance window unless the database is replicated through its own clustering or replication mechanism.
For genuinely low-downtime database upgrades, use database-native replication or a supported managed database architecture instead of having two independent database processes share one filesystem. Replicate Blue to Green, verify the replica, briefly quiesce writes, promote Green, and then redirect clients. Kafka, OpenSearch, MongoDB, and other stateful platforms have similar cluster-specific migration or replication procedures. Each system needs its own failover plan; a single shared EFS mount does not make all of them safely portable between clusters.
That may require a substantial redesign, especially with the existing cross-zone and compliance constraints, but it is safer than treating a shared filesystem as a database cluster.
EFS can work well for shared files, uploads, backups, and some application data, but it is a poor substitute for block storage or database replication. Even where a database appears to start successfully from EFS, latency, locking behavior, fsync semantics, and concurrent access can create operational risks. A safer migration is to keep storage ownership in one cluster at a time, take a consistent backup or snapshot-equivalent export, restore or replicate into Green, and perform a controlled final cutover. Stateless services can be tested in parallel, but stateful writers should remain single-owner until the handoff is complete.
Understood. The planned upgrade window is roughly six hours, but the goal was to avoid taking the applications offline during validation. It sounds like shared EFS cannot provide that safety by itself.