We have one roughly 3 GB EFS filesystem containing Hugging Face sentence-transformer snapshots in an / directory structure. It is mounted by four workloads across three EKS clusters, including two GPU inference services with about 107 production pods, an embedding API, and batch jobs. The models range from about 130 MB to 2.1 GB.
All services run with TRANSFORMERS_OFFLINE=1 and HF_DATASETS_OFFLINE=1, so models must already exist on the shared volume. Runtime downloads are intentionally disabled to avoid startup failures caused by external availability or changed model contents.
The old process was to upload a model directory to S3, find an EC2 instance with EFS mounted, and copy the files from S3 onto EFS. After moving from ECS to EKS, that instance-based process disappeared. The current workaround requires a data scientist to upload to S3 and submit an operations ticket for a manual copy, which is too slow.
We are considering a few approaches: using an EKS writer pod to copy S3 objects onto EFS, AWS DataSync from S3 to EFS, packaging models into container images, or skipping EFS and using the S3 CSI mount. A versioned-directory layout with an atomic symlink switch also seems useful, since readers would only see a fully copied model revision. The inference workloads currently mount EFS read-write even though they only read models, and the filesystem has no backups.
What architecture would you recommend for safely and automatically publishing models from S3 to EFS while keeping readers from seeing partial uploads?
3 Answers
I would not use an S3 mount as the default replacement for this workload without benchmarking it carefully. A large number of pods repeatedly reading model files can create cache and request behavior that is quite different from a shared EFS filesystem, and startup performance may become less predictable. It could be useful for isolated workloads, but the current multi-cluster inference setup sounds better suited to a controlled publish-to-EFS process.
A small Kubernetes writer job or CronJob is probably the simplest fit here. Give it a dedicated service account with permission to read the approved S3 location and mount EFS read-write. Have it copy each model into a new revision directory, verify checksums, and only then update a pointer such as current. The inference pods should mount EFS read-only.
I would avoid copying directly into a directory that active pods are reading. Publish to something like /, validate all expected files, and switch the active reference only after the copy succeeds. Keep the previous revision around so a failed deployment can be rolled back. Since you already have a working shared-volume refresh job, this also gives you a familiar operational pattern without adding a new AWS service.
AWS DataSync is worth testing for this exact S3-to-EFS transfer. It can handle the transfer and integrity checks without requiring you to maintain a long-lived EC2 host. I would still put the destination under a new versioned directory rather than syncing over the directory currently used by inference pods.
After DataSync completes, a small publish step can validate the expected model files and update the active revision. Treat the S3 model location as the source of truth, enable versioning or immutability there, and add alerts for failed or incomplete transfers. Also make the EFS consumer mounts read-only and establish backups before relying on the filesystem as the only serving copy.
Related Questions
Can't Load PhpMyadmin On After Server Update
Redirect www to non-www in Apache Conf
How To Check If Your SSL Cert Is SHA 1
Windows TrackPad Gestures