I'm trying to figure out how to send data from Airflow to a CloudNativePG (CNPG) database hosted in Kubernetes. I have an Airflow setup that runs a scheduled DAG to transfer data daily, but I need to find a way to connect to the CNPG database. The tricky part is that CNPG only allows cluster-only connections by default, and exposing its read-write service via HTTP doesn't seem like an option since I think I need to use TCP. I'm not very experienced with Kubernetes as I'm more of a developer than an admin, so any guidance would be greatly appreciated!
1 Answer
To connect to your CNPG, you may want to set up a LoadBalancer service or a NodePort service. This way, you can create an external endpoint that your Airflow instance can use to send data over TCP. CNPG typically uses TCP connections, not HTTP, so that’s the route you need to go.
Does creating a LoadBalancer service mean I need to assign a specific IP address to the pod?