What does a production-ready Python pipeline to Snowflake need?
I'm a recent computer science graduate with no professional data engineering experience, and I've been asked to build a small data pipeline at a credit union. Our data…
ETL
What does a production-ready Python-to-Snowflake data pipeline need?
I'm a recent computer science graduate with no professional data-development experience, and I've been asked to start building a data pipeline at a credit union. Our data is…
What no-code tools can sync data between different applications?
I'm looking for a point-and-click system that can move or synchronize records between different applications and databases without requiring me to write custom scripts. Ideally, I want to…
Looking for a Tool to Extract Data from .docx Files
I'm working at a small manufacturing company where all our production floor documentation is stuck in Word .docx files. The problem is, the bill of materials data in…
How do you usually clean up messy CSV and JSON data before processing?
I often find myself dealing with messy data, especially when working with third-party exports and API responses. Issues I frequently encounter include CSV files that have inconsistent column…
How to Modernize Shell Scripts and Crontab Workflows?
I'm looking to modernize the workflow for a bunch of shell scripts that my small team uses. We have around 10-15 scripts that typically handle tasks like pulling…
Should I Use Beautiful Soup or xml.etree.ElementTree for ETL with ENML?
I'm working on an ETL process to extract notes from Evernote's ENML format and I'm trying to figure out if I should use Beautiful Soup (BS4) or stick…
Is GitHub Actions the Best Choice for Automation Orchestration?
I'm exploring the idea of using GitHub Actions (GHA) for automation orchestration within our organization. It feels like many teams leverage GHA along with CI/CD platforms like Azure…
How can I move my ETL logs from a VM to Azure Storage and use KQL for querying?
Hey everyone! We're making the switch from Azure Function Apps to Azure VMs to handle our ETL processes. Right now, we're just saving our logs locally on the…
Best Way to Transfer Data Between Two JDBC Connections?
Hey everyone! I often find myself needing to transfer data between two different databases, each accessed through its own JDBC connection. I usually just maintain two connections, retrieve…
What are some of the latest tools and methods for ETL pipelines?
Hey everyone! I'm curious about the latest technologies and libraries that you've found useful for ETL and ELT pipelines. I've recently been experimenting with ConnectorX and DuckDB, and…
Is HSQLDB a Good Choice for In-Memory Use in Production with Oracle Queries?
Hey everyone! I've been exploring the idea of using HSQLDB as an in-memory database for production, especially because it has an Oracle compatibility mode. This could be a…