I'm struggling to set up an ELT pipeline to sync data from Postgres RDS to BigQuery using Airbyte. I didn't realize how resource-heavy Airbyte would be for my needs, especially with tables containing thousands of rows. While I got it running on our RKE2 Cluster, it kept failing due to insufficient resources. I've now tried deploying it on a K3S cluster with 16GB RAM and 8 CPUs, but it won't even deploy. The Temporal deployment keeps failing, and the bootloader is complaining about a missing environment variable in a secrets file that I've never even set. I've tried both v1 and v2 Helm charts, but the v2 chart is particularly frustrating, throwing an error about a missing ingressClass config that isn't even in the official Helm chart. I'm at my wit's end! Are there any simpler open-source ELT pipeline tools out there that can help sync data between Postgres and Google BigQuery? Thanks for any recommendations!
4 Answers
Why not give Apache Airflow a shot? It's open-source, comes with a nice UI, and you can easily configure DAGs using Python. It might be a bit complex to set up initially, but it's quite powerful once you get the hang of it!
Have you checked out Meltano? It's an open-source tool that's designed to be more lightweight than Airbyte. You can run it locally, and it supports multiple destinations including BigQuery. It could be a good alternative if you're looking to dodge the heavy resource usage of Airbyte.
I'm working on setting up Debezium to sync data from Planetscale to MySQL. I wouldn't say it's a simpler solution though, just different. It may suit your needs depending on your setup,
I had to give up on Airbyte and switched to their SaaS model. It wasn't that pricey, and honestly, it saved me a lot of headaches since Airbyte was demanding a high-performance setup to work right, which wasn’t worth it for what we needed. Sometimes it's just easier to pay a bit and avoid the complex setup and maintenance.
Related Questions
How To: Running Codex CLI on Windows with Azure OpenAI
Set Wordpress Featured Image Using Javascript
How To Fix PHP Random Being The Same
Why no WebP Support with Wordpress
Replace Wordpress Cron With Linux Cron
Customize Yoast Canonical URL Programmatically