---
title: "ADF as FMC for Databricks"
canonical: "https://docs.vaultspeed.com/space/VPD/3164635181/ADF%20as%20FMC%20for%20Databricks"
format: markdown
---
This document explains how to set up ADF as the FMC for Databricks.

> Macro (toc)

> ⚠️ This requires a Databricks Runtime version >= 10.3

### Creating ADF linked service

Creating linked service on existing data bricks cluster (Databricks access token method)

#### 1. Generate a Databricks access token for ADF user:

Within Databricks workspace:

User settings --> Generate new token

![image](media://dcd28c29-5c9e-4527-bdc7-cacd054fe8bb)

#### 2. Add a new linked service in ADF (Existing interactive cluster method)

 

![image](media://1802ab3a-b349-458a-8fce-34d3d4a77423)

 

How to obtain Databricks cluster id and any other environment variables:  
[https://docs.microsoft.com/en-us/azure/databricks/workspace/workspace-details](https://docs.microsoft.com/en-us/azure/databricks/workspace/workspace-details)

 

### Setting the FMC type in VaultSpeed:

Before you can start generating code for running Databricks ETL with ADF, go to the Data Vault parameters and set the FMC type parameter to ADF. After creating and locking a new DV release, the new ETL code can be generated, this will now contain some extra notebooks required for the FMC logic.

### Create FMC flows in VaultSpeed:

Create the FMC flows in VaultSpeed as described in [https://vaultspeed.atlassian.net/wiki/spaces/VPD/pages/3012788295](https://vaultspeed.atlassian.net/wiki/spaces/VPD/pages/3012788295)  . For Databricks, you also have to fill in an `etl_connection_name`. This should be set to the Directory in Databricks where the code has been deployed (the same as the `path` variable of the connection in the agent).

Note that only grouped flows are supported. Mappings in each group will be executed in parallel from a Notebook. The concurrency defined in VaultSpeed defines how many groups there are.  
To control the concurrency within Databricks, create an environment variable in the cluster named `FMC_CONCURRENCY` and set that equal to the desired number of parallel execution per group. The default value is 4.

Another environment variable that can be set is the `FMC_TIMEOUT`, this determines the maximum runtime of a mapping before it will fail with a timeout error, the default value is one hour.

### Deploying the FMC flows:

The generated code consists of JSON files with ADF code and SQL files with Databricks SQL code. To deploy this code with the Agent, you will have to define a connection with ADF and Databricks connection properties. The connection definition should look like this:

```
# ADF conection part
adf.secret= shown when creating a secret for the application in Azure
adf.client_id= Application (client) ID can be found in application definition in Azure
adf.tenant= Directory (tenant) ID can be found in application definition in Azure
adf.subscription_id= subscription ID can be found on the subscriptions page in Azure
adf.resource_group_name= name of the resource group the data factory was created in
adf.factory_name= name of the data factory

# Databricks conection part
adf.url= jdbc url to databricks cluster
adf.token= personal access token: https://docs.databricks.com/dev-tools/api/latest/authentication.html
adf.host= url of databricks environment
adf.path= directory where the databricks code should be deployed (ending with /), this should be the same directory as where the ETL code as been deployed to!
```