I'm trying to understand how Azure Data Factory's main artifacts fit together in a real project. For example, if I'm moving data from Azure SQL to Azure Data Lake Storage, my current understanding is that a linked service handles the connection, a dataset describes the data being accessed, and a pipeline controls the workflow. Is that accurate? How should I decide when to create a new linked service or dataset versus reusing an existing one?
2 Answers
That understanding is basically right. A linked service defines how ADF connects to a system, such as a particular database or storage account. You’ll usually have one linked service per source system and environment, unless different credentials or permissions are required. Avoid creating multiple copies that point to the same system, since credential changes would then need to be updated everywhere.
A dataset represents the data an activity works with, such as a SQL table, a folder, or a CSV file. You can create a dataset for a specific entity and import its schema, or make a parameterized dataset that accepts values like schema, table, container, folder, and filename. Parameterized datasets are useful when many sources share the same format.
The pipeline handles orchestration: scheduling, dependencies, looping, and calling activities. Transformations are generally done with Mapping Data Flows, stored procedures, notebooks, or another processing service rather than by the pipeline itself.
A helpful way to picture it is: the linked service is the key to a system, the dataset identifies the drawer or folder inside it, and the pipeline describes what to do with what you found.
For linked services, a common rule is one per system per environment rather than one per table. For datasets, create separate ones when the file format or connection behavior differs, such as Parquet versus delimited text or JSON. If the only difference is the table or file name, a single parameterized dataset can often be reused.
In larger projects, it’s common to use generic SQL and ADLS datasets with parameters and drive those values from a metadata table in a ForEach activity. Also keep environment-specific settings outside the core definitions by using parameters, global parameters, or Key Vault references, so deployments don’t require manually editing production connections.
That clears it up. So the pipeline coordinates the work, while a data flow or stored procedure would handle transformations such as joins and filtering.