I'm trying to access an S3 Bucket from a Windows Fargate container, primarily for reading data with very few write operations. While I understand that using FSx would be the ideal solution, I'm wondering if I can set up an S3 to Storage Gateway (using S3 File Gateway) and then mount it with SMB in the Fargate container during startup. Are there any other recommendations for achieving this?
5 Answers
It's interesting to see that Windows Server can run on Fargate. Just a heads up, though: I wouldn't recommend going that route if you can avoid it. Building Docker images can be complicated, and startup times can be a hassle unless you frequently update your base images since Fargate only caches a limited number of them.
How much data are you looking to transfer? I think understanding the volume might help in figuring out the best approach.
I've had good results with Backblaze and rclone, although these aren't native AWS solutions. You might want to check if rclone supports S3 specifically.
Have you considered just using S3's Get and Put operations? It might actually simplify things for you.
That's what I initially suggested. However, we're looking to lift and shift from on-premises where we're currently using network-attached storage. Management wants to move to the cloud with minimal effort and cost, and the migration team isn't keen on rewriting the app for simple S3 GET operations, but they want to avoid the high costs associated with FSx.
Using Storage Gateway is one way to achieve this, but it's quite costly. I wouldn't recommend using a Windows File System directly linked to Object Storage like S3. If you really need to go down this route, make sure Storage Gateway meets your needs. Alternatively, consider FSx and a backend process for syncing files to S3, or even better, if your Windows application can directly manage reads and writes using S3 APIs, that would be significant progress.
We're dealing with about 4.5 TB of data, and we have new files coming in daily.