What does a production-ready Python pipeline to Snowflake need?
I'm a recent computer science graduate with no professional data engineering experience, and I've been asked to build a small data pipeline at a credit union. Our data…
data pipelines
How are you detecting unauthorized AI models and data flows across Azure, GCP, and AWS?
We recently reviewed our cloud accounts for unauthorized AI deployments and found several Azure OpenAI instances that had never been approved, a self-hosted model with overly broad access…
How should I build an analytics pipeline for billions of tiny, flat S3 files?
I have an S3 bucket containing billions of very small JSON objects, ranging from a few bytes to about 10 KB. Objects are continuously added or overwritten, and…
RDBMS, Synapse, or Databricks: What's the Best Choice for Our Data Architecture?
Hey everyone! We're currently weighing our options between sticking with our EDBPS (PostgreSQL) setup or moving to a lakehouse architecture. Our goal is to calculate stock replenishment based…
What are some of the latest tools and methods for ETL pipelines?
Hey everyone! I'm curious about the latest technologies and libraries that you've found useful for ETL and ELT pipelines. I've recently been experimenting with ConnectorX and DuckDB, and…