Frustrations with Airbyte: Need Alternatives for ELT Pipelines
Asked by
TechyNinja42
I'm struggling to set up an ELT pipeline to sync data from Postgres RDS to BigQuery using Airbyte. I didn't realize how resource-heavy Airbyte would be for my needs, especially with tables containing thousands of rows. While I got it running on our RKE2 Cluster, it kept failing due to insufficient resources. I've now tried deploying it on a K3S cluster with 16GB RAM and 8 CPUs, but it won't even deploy. The Temporal deployment keeps failing, and the bootloader is complaining about a missing environment variable in a secrets file that I've never even set. I've tried both v1 and v2 Helm charts, but the v2 chart is particularly frustrating, throwing an error about a missing ingressClass config that isn't even in the official Helm chart. I'm at my wit's end! Are there any simpler open-source ELT pipeline tools out there that can help sync data between Postgres and Google BigQuery? Thanks for any recommendations!
Answers
I had to give up on Airbyte and switched to their SaaS model. It wasn't that pricey, and honestly, it saved me a lot of headaches since Airbyte was demanding a high-performance setup to work right, which wasn’t worth it for what we needed. Sometimes it's just easier to pay a bit and avoid the complex setup and maintenance.
Reply to this answer
I'm working on setting up Debezium to sync data from Planetscale to MySQL. I wouldn't say it's a simpler solution though, just different. It may suit your needs depending on your setup,
Reply to this answer
Have you checked out Meltano? It's an open-source tool that's designed to be more lightweight than Airbyte. You can run it locally, and it supports multiple destinations including BigQuery. It could be a good alternative if you're looking to dodge the heavy resource usage of Airbyte.
Reply to this answer
Why not give Apache Airflow a shot? It's open-source, comes with a nice UI, and you can easily configure DAGs using Python. It might be a bit complex to set up initially, but it's quite powerful once you get the hang of it!
Reply to this answer
Add your answer
Answers are reviewed before publication.
Related Discussions