How do Azure Data Factory Pipelines, Datasets, and Linked Services Work Together?

For environment-specific deployments, avoid hardcoding development or production values inside every artifact. Use parameters, global parameters, and a secrets store where appropriate. This lets the same pipeline and datasets move between environments without manually editing connection details, and it reduces the risk of accidentally pointing a production workflow at a development system.

A useful way to think about it is: the linked service is the key to a system, the dataset identifies the drawer or folder you want, and the pipeline describes what to do with it. Avoid creating separate linked services for every table. Reusing one per system makes credential rotation and environment changes much easier. For datasets, create separate ones when the format or connection behavior is different, such as Parquet versus CSV or JSON. Otherwise, parameterized datasets can represent different tables, folders, or filenames without creating dozens of nearly identical artifacts.

That’s the right general model. A linked service represents a connection to a system, such as an Azure SQL server or a storage account. In most cases, use one linked service per system and environment, unless different credentials or permissions are required. A dataset describes the data you want to work with, such as a table, file, or folder, and can include schema information. A pipeline orchestrates activities such as copying data, scheduling runs, and controlling dependencies. Transformations are usually handled with Mapping Data Flows, stored procedures, notebooks, or another processing service.