I'm reaching out to the community for assistance because I'm stuck trying to set up a Data/BI stack. My goal is to have Metabase as the BI visualizer with PostgreSQL as the database, and I'm using Airbyte for ETL to get data into PostgreSQL. I initially started with a CSV for testing but plan to integrate Salesforce API later.
Here's my situation: I managed to get all the main services running, but I'm having trouble configuring the PostgreSQL destination in Airbyte. The local SQLite isn't even functioning correctly either.
The specific issue is that Airbyte requires explicit setup for how credentials are stored, and I can't seem to navigate this setup. Also, I'm deploying everything via Docker and using Docker Compose for about 3-6 months now. If anyone has insights or tips, I would greatly appreciate your help!
2 Answers
From what I read, you need to use the internal ports when referring to your databases inside the Docker network. The external port mapping is for accessing it from outside the Docker network.
It sounds like your issue is more related to the Airbyte configuration rather than Docker itself. Make sure you're properly setting the environment variables for your PostgreSQL instance in your Docker Compose file. The secrets management setup can be tricky, but you might want to look at how Airbyte is handling those credentials. It’s worth checking their documentation for more info!
Good point! I suggest also reviewing the GitHub repository for Airbyte since they have mentioned some deprecated features with Docker Compose, which could impact your setup.
Exactly, I guess I might have overlooked that. Thanks for clarifying!