We can now pass dynamic values to linked services at run time in Data Factory. This enables us to do things like connecting to different databases on the same server using one linked service. Some linked services in Azure Data Factory can parameterized through the UI. Others require that you modify the JSON to achieve your goal.
Recently, I needed to parameterize a Data Factory linked service pointing to a REST API. At this time, REST APIs require you to modify the JSON yourself.
In order to pass dynamic values to a linked service, we need to parameterize the linked service, the dataset, and the activity.
I have a pipeline where I log the pipeline start to a database with a stored procedure, lookup a username in Key Vault, copy data from a REST API to data lake storage, and log the end of the pipeline with a stored procedure. My username and password are stored in separate secrets in Key Vault, so I had to do a lookup with a web activity to get the username. The password is retrieved using Key Vault inside the linked service. Data Factory doesn’t currently support retrieving the username from Key Vault so I had to roll my own Key Vault lookup there.
I have parameterized my linked service that points to the source of the data I am copying. My linked service has 3 parameters: BaseUrl, Username, and SecretName. The JSON for my linked service is below. You can see that I need to reference the parameter as the value for the appropriate property and also define the parameter at the bottom.
{
"name": "LS_RESTSourceParam",
"properties": {
"annotations": [],
"type": "RestService",
"typeProperties": {
"url": "@{linkedService().BaseUrl}",
"enableServerCertificateValidation": true,
"authenticationType": "Basic",
"userName": "@{linkedService().Username}",
"password": {
"type": "AzureKeyVaultSecret",
"store": {
"referenceName": "MyKeyVault",
"type": "LinkedServiceReference"
},
"secretName": "@{linkedService().SecretName}"
}
},
"parameters": {
"Username": {
"type": "String"
},
"SecretName": {
"type": "String"
},
"BaseUrl": {
"type": "String"
}
}
}
}
I have defined these three parameters in my dataset, along with one more parameter that is specific to the dataset (that doesn’t get passed to the linked service). I don’t need to set the default value on the Parameters tab of the dataset.
On the Connection tab of the dataset, I set the value as shown below. We can see that Data Factory recognizes that I have 3 parameters on the linked service being used. The relativeURL is only used in the dataset and is not used in the linked service. The value of each of these properties must match the parameter name on the Parameters tab of the dataset.
In my copy activity, I can see my 4 dataset parameters on the Source tab. There, I can write expressions to provide the values that should be passed through to the dataset, 3 of which are passed through to the linked service. In my case, this is a child pipeline that is called from a parent pipeline that passes in some values through pipeline parameters which are used in the expressions in the copy activity source.
And that’s it. I can run my pipeline and have it call different REST APIs using one linked service and one dataset.
