I'm currently setting up CloudNativePG (CNPG) on Kubernetes and trying to figure out how to populate it with data. I have an Airflow instance that runs a scheduled DAG to transfer data daily. Now, I need to get that data to Postgres hosted by CNPG. The issue is figuring out how to send that data since CNPG allows cluster-only connections by default. Also, exposing the read-write service via HTTP isn't suitable because I believe it requires TCP instead. I'm not very well-versed in Kubernetes administration, just a developer with limited experience on the platform, so I'd really appreciate any guidance on this!
4 Answers
You can also expose the CNPG service using a NodePort. It could make things easier if Airflow is set up properly. Just be aware of the communication challenges since Airflow is on a VM, not in the cluster.
To send your data, you might want to set up a LoadBalancer service. This will give you an external endpoint to send your data to. Keep in mind that CNPG generally uses TCP, not HTTP, for connections.
For what you're trying to do, a TCP connection to your CNPG cluster is key. If you're unsure about Kubernetes, tools like ClawCloud can help simplify tasks. They even have straightforward guides to get you started!
An alternative approach could be to create a small HTTP service that reads from Airflow and writes directly to CNPG. However, since you're handling around 5GB of data daily, consider the feasibility of this setup.
Related Questions
How To: Running Codex CLI on Windows with Azure OpenAI
Set Wordpress Featured Image Using Javascript
How To Fix PHP Random Being The Same
Why no WebP Support with Wordpress
Replace Wordpress Cron With Linux Cron
Customize Yoast Canonical URL Programmatically