We have a shared EFS filesystem of roughly 3 GB containing Hugging Face sentence-transformer snapshots in an / directory structure. It is mounted by four workloads across three EKS clusters: two GPU inference services with about 107 production pods running on HPA-managed spot GPU nodes, a small embedding API, and several batch jobs.
The models range from about 130 MB to 2.1 GB. All services use TRANSFORMERS_OFFLINE=1 and HF_DATASETS_OFFLINE=1 intentionally, so they cannot download models at runtime. The required files must already exist on EFS or the pod fails to start.
Before moving from ECS to EKS, the process was manual: a data scientist uploaded the model directory to S3, someone found an EC2 instance with EFS mounted, and an operator copied the files from S3 to EFS. Since the ECS instances were removed, there is no working automated delivery path. The current workaround is an S3 upload followed by a ticket for an operations engineer, which is too slow.
A manual S3-to-EFS copy has been tested successfully, including SHA-256 verification. We also already run a production CronJob that writes reference data to another shared EFS volume, so a single writer pod refreshing a shared volume is a familiar operational pattern.
The options under consideration are:
- A Kubernetes Job or CronJob that mounts EFS and copies approved model files from S3.
- AWS DataSync from S3 to EFS, potentially writing each model revision into a staging directory before making it active.
- Lambda with an EFS mount, although the execution-time limit appears unsuitable for the largest model.
- Mountpoint for Amazon S3 instead of EFS, though the expected number of pods and per-instance caching/API traffic seem like a poor fit.
- Versioned model directories with an atomic pointer or symlink switch, so readers never observe a partially copied model.
The EFS filesystem currently has no backups, and inference services mount it read-write even though they only read models. What architecture would you recommend for reliable, automated model delivery to EFS on EKS? Are there important concerns around synchronization, permissions, integrity checks, backups, or serving the models directly from container images or object storage?
5 Answers
A Kubernetes Job or CronJob that mounts EFS and copies approved objects from S3 is probably the simplest option, especially since you already have this pattern working for other shared data. Give the job its own IAM role through IRSA, mount the EFS filesystem read-write only in that job, and keep inference pods read-only. The job can download into a new versioned directory, verify checksums and expected files, and then update an active pointer only after the copy succeeds. That avoids readers seeing a half-written model and gives you a clear audit trail for each deployment.
AWS DataSync is worth testing if you want a managed S3-to-EFS transfer without maintaining copy logic. It can handle integrity verification and recurring transfers, but you would still need to think about how a model becomes active. Have it target a staging or revision directory, validate the result, and then perform the small activation step separately. I would not let DataSync overwrite the directory currently used by inference pods.
I would avoid putting the models into the application image unless the models are tightly coupled to a specific release. Large model layers make builds, pushes, pulls, and rollbacks heavier, and every new model version forces an image rollout. It can be reasonable for a small, rarely changing model, but a separate model artifact in S3 with controlled promotion is usually cleaner here.
Serving directly from S3 through an S3 mount is not automatically equivalent to local model storage. Model loading often involves many metadata and weight-file reads, and dozens or hundreds of pods can create substantial request and cache pressure. A shared EFS cache or preloaded local storage is more predictable for this scale. If startup time becomes an issue, consider copying the active model from EFS to node-local storage or an attached volume during initialization, but keep the immutable artifact in S3 and the shared release process separate.
Before choosing the transfer mechanism, I would fix the storage controls. Create an EFS access point with a dedicated directory and POSIX identity, make the model path read-only for all inference workloads, and give only the publishing job write access. Turn on EFS backups or another recoverable source of truth, because a bad sync or accidental deletion should not be able to destroy the only copy. Also define a manifest containing the model revision, file hashes, and metadata so the publisher can verify completeness before activation.
That was one of my concerns too. The larger model and the number of pods make image size and rollout time seem like a bad trade, even if it would be straightforward operationally.