I'm looking to modernize the workflow for a bunch of shell scripts that my small team uses. We have around 10-15 scripts that typically handle tasks like pulling data from a database and uploading it to an SFTP server, along with a few ETL scripts that merge files and perform actions on datasets. I'm curious about what modern approaches there are for streamlining this process. I tried asking ChatGPT for advice, and it suggested containerizing the scripts, setting up a logging server, and using an orchestrator for scheduling. I'm comfortable with that, but I worry about creating a single point of failure, and I want to ensure it's not overly complex for anyone who takes over. I'm also considering using systemd to run containers and timers for scheduling them instead. If anyone has experience or articles they can share on similar transitions, I'd appreciate it!
4 Answers
If it’s working fine, maybe don’t change too much. But GitHub Actions could be a good place to run it. Just make sure the processing time fits within their limits, and it should offer enough flexibility for your needs.
Where are you primarily running these scripts? If they execute on on-prem Linux or Windows servers, you might find some options easier to implement based on the environment.
Have you thought about using GitHub Actions or Azure DevOps pipelines? You could pull your script files directly from version control and run them on any machine as a runner or agent. It streamlines things quite a bit!
That's certainly an option I could look into.
You're essentially looking at reverse ETL here, right? Tools like Airbyte or MapForce could help. Airbyte is well-known, while MapForce offers a more visual approach, which could reduce your bus factor. Airflow is another great option for scheduling and executing jobs if you're comfortable with Python. Plus, if you're considering AWS, Fargate can simplify tasks over Kubernetes. Lots of options out there!
I'll check that out.
They're mostly running on on-prem Linux or Windows servers, but they could get reimaged with a new OS.