I'm currently dealing with storage limitations at my workplace, where we have a lot of data stored on on-premise devices. Given the rising costs, expanding our storage isn't an option right now. What I'm looking for is a way to manage these file shares by identifying files that haven't been accessed or modified in a certain number of days, and then archiving that data to the cloud. I want something similar to Amazon S3's lifecycle policies that can progressively move less frequently accessed data into colder storage while ensuring that our on-prem SAN remains fast and responsive. Ideally, the reads and writes should be seamless. My budget is tight, but I'm open to exploring open-source software to meet these needs. If a commercial product is necessary, I would prefer one that can be configured with Terraform. Any recommendations would be greatly appreciated!
4 Answers
Have you considered using something like an AWS Storage Gateway? It facilitates the integration of on-premise environments with cloud storage efficiently. Just make sure to analyze the cost of S3 compared to your current on-prem expenses.
How about using an Nginx cache? You could configure it so that writes go directly to S3, while content is fetched through an Nginx S3 gateway with local caching. This could provide a good balance—especially if you’re okay with HTTP/HTTPS access instead of needing a POSIX-compliant filesystem.
Sure thing! I have a robust setup for caching with Nginx that I can outline for you if you're interested.
There are definitely some tailored solutions for the life sciences and bioinformatics sector. You might want to check out companies like Starfish Storage. They outline specific problems and solutions that could be relevant to your situation. Even if you don't choose them, their concepts might guide you to the right technology for your needs.
It sounds like you're after a hierarchical storage management (HSM) setup or some tiering solution. Have you checked out tools like MinIO? It offers lifecycle policies that could help you achieve some of your goals. Alternatively, using rclone with scheduled jobs might also give you part of the functionality you’re looking for. Just keep in mind that fully transparent reads and writes can be a bit tricky and might require a dedicated filesystem layer or a commercial product to manage that seamlessly.
That makes sense! Do you think the MinIO solution would handle large datasets well?
Definitely! MinIO is optimized for handling large object storage. Just ensure you have the right setup for performance.
This sounds interesting! Do you have a full solution you can share? I’d love to learn more.