AWS and Google Cloud both provide established managed Apache Airflow services, but Azure's options seem limited and inconsistent. The earlier Azure Data Factory managed Airflow offering was discontinued, and Fabric Apache Airflow jobs now come with a major restriction: private networks and virtual networks aren't supported.
I previously ran Airflow on AKS, but maintaining the deployment, upgrades, scaling, and support requires significant effort. For a new organization, a managed service also makes it easier to establish ownership and support expectations with leadership. Third-party marketplace products exist, but many are VM-based and still require operational work; the more fully managed options may not support private networking.
Why hasn't Microsoft provided a straightforward, managed Apache Airflow service that preserves Airflow's normal functionality and supports private connectivity, instead of steering users toward Fabric-specific connectors?
3 Answers
Running Airflow on AKS is possible with Helm, but it remains a self-managed deployment. Your request is specifically for a Microsoft-supported PaaS version, with Microsoft handling scaling, upgrades, availability, and operational support. The Fabric implementation may cover some of those responsibilities, but the lack of private and virtual network support makes it unsuitable for many enterprise environments.
The likely business reason is strategic: Microsoft wants customers using Fabric and its surrounding services. A fully independent Airflow service could reduce the incentive to adopt Fabric, so the current offering appears designed around Fabric connectors rather than providing Airflow as a neutral orchestration platform.
Azure Marketplace has several third-party Airflow offerings, so one option is to use a managed vendor rather than running Airflow yourself. The main thing to evaluate is the vendor’s support model, upgrade process, scaling capabilities, and responsibility for the underlying infrastructure.
That’s the challenge: many marketplace products are VM-based, so upgrades and scaling still become our responsibility. A managed service with clear enterprise support would make adoption much easier, especially when private networking is required.
We’re already committed to Azure, so the goal isn’t to avoid the platform. We just need Airflow to connect directly to private Azure resources instead of requiring Fabric-specific connectors for basic workflows.