--- title: "Azure Data Factory" canonical: "https://docs.vaultspeed.com/space/VPD/3012788295/Azure%20Data%20Factory" format: markdown --- **Table of Contents** > Macro (toc) ## Architecture The VaultSpeed FMC uses only Azure PAAS components, Azure Data Factory, and the Data Warehouse Database (SQL server or Synapse). The Database contains procedures and loads metadata tables. The ADF FMC will use stored procedure activities to execute those procedures. ADF has built-in monitoring and you can also create Azure Dashboards based on ADF metrics. It is also possible to export the metrics into an external reporting tool (i.e. Power BI, Grafana …). ![image](media://c1dcb9b5-3205-4ac9-86dd-29a2c81d22ec) ## Cost In ADF you have to pay for each run and for every activity execution. The more loads you do, the more you have to pay. In contrast with Airflow, where you have a constantly running environment. If you are loading every 5min it will probably be cheaper to use Airflow instead of ADF. ## Limitations - The initial load scheduling is currently unavailable. Manual triggering is required for the initial load until a workaround is implemented. - Source loading functionality is not yet implemented, but it is planned for addition in a future release. - Auto-deployment to ADF is incompatible with data factories using Git integration. In such cases, Git deploy alongside Azure DevOps would be necessary. - Azure Data Factory (ADF) has a strict limit of 80 activities per pipeline. This constraint means non-grouped flows are not feasible for sources with more than 10-12 objects. Additionally, for grouped flows, the concurrency is capped at 18 for CDC sources and 14 for sources with delete detection (with additional prep delete mappings). ### Limitation Update **Increased Activity Limit:** > ℹ️ **Note:** Azure Data Factory (ADF) previously had a strict limit of 40 activities per pipeline. > ℹ️ > ℹ️ **However, with a recent update, the activity limit per pipeline in ADF has been increased to 80**, providing more flexibility in pipeline design. For further details, you can check the official announcement. "[Data Factory Increases Maximum Activities Per Pipeline to 80](https://techcommunity.microsoft.com/t5/azure-data-factory-blog/data-factory-increases-maximum-activities-per-pipeline-to-80/ba-p/4096418)". ## Setup - Create a Data Factory (ideally without Git integration) [https://docs.microsoft.com/en-us/azure/data-factory/quickstart-create-data-factory-portal#create-a-data-factory](https://docs.microsoft.com/en-us/azure/data-factory/quickstart-create-data-factory-portal#create-a-data-factory) - Create a linked service to your data warehouse [https://docs.microsoft.com/en-us/azure/data-factory/author-management-hub#manage-connections](https://docs.microsoft.com/en-us/azure/data-factory/author-management-hub#manage-connections) - for auto deploy: - create an application in Active Directory with a secret (authentication option 2): [https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal#register-an-application-with-azure-ad-and-create-a-service-principal](https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal#register-an-application-with-azure-ad-and-create-a-service-principal) - Add a new connection definition in the agent with the following properties: ## VaultSpeed Parameter Settings In the System parameters, enable USE_FMC (FMC parameters) In the Data Vault parameters, change the FMC type to ADF: ![image](media://b3eb71aa-8cbc-4090-b2a4-41c5b12d3ada) Note that almost all the other FMC parameters are only applicable for Airflow (source loading will be enabled in a future release). The only other parameter which can be used is SOURCE_PL_DEP, this determines whether the Business Vault load should wait for this specific source or not (Presentation Layer is not yet supported in ADF). ## Generation Create a new workflow on the Flow Management Control menu. Each workflow Loads a specific part of the Data Vault and will generate an Azure Data Factory pipeline. There are 2 types of workflows: - FL: This workflow loads the Raw Vault (a.k.a Foundation Layer) for a specific source. - BV: This workflow loads the Business Vault (for all sources). Each workflow has a specific **load type**: - INIT: Initial load, this workflow will only be executed once (has to be triggered manually for now). - INCR: Incremental load, this workflow will be executed with a certain frequency. An ADF trigger will be generated to run the resulting pipeline. The workflow **name **will become the name of the pipeline so it should uniquely represent a certain load example: `__` for the FL and `_bv_` for the BV. The description is only for adding extra information, and will not show up in the generated pipeline. The **Start Date** defines the Initial load date. This date can be in the past if you want to do historical loads using CDC data. It does not define when your initial load actually runs but instead defines the start of your loading window for the incremental loads. So this might be the date of the database dump for the initial data. The **concurrency **determines how many activities there are for each layer, the workflow executes the procedures layer per layer (e.g. EXT → STG → FL). Each of the layers has a concurrency number of activities (see limitations). Each of these activities will execute a set of loading procedures in sequence in the same transaction. This only has an impact if group tasks are enabled. **Group Tasks** determines whether each procedure has its own activity (not grouped) or whether multiple procedures are grouped into a single activity (see limitations). The size of which is determined by the concurrency attribute and the number of objects per layer. The **Schedule Interval** determines when and how often the incremental load is executed. This attribute is not available for INIT loads. This field should contain a `recurrence` object for ADF triggers. A simple example that results in an execution every 15 minutes starting from the Start Date would be: `{"frequency": "Minute", "interval": 15}` More examples can be found in the ADF Documentation [https://docs.microsoft.com/en-us/azure/data-factory/concepts-pipeline-execution-triggers#examples-of-trigger-recurrence-schedules](https://docs.microsoft.com/en-us/azure/data-factory/concepts-pipeline-execution-triggers#examples-of-trigger-recurrence-schedules) Of the **Connection **properties, only the Data Vault Connection Name is used for ADF at the moment. The connection name is the name of the linked service created in the Setup steps. ![image](media://74f9909f-fb7a-4be9-8ef9-f3122b4e021c) To generate code for the workflows, select one of them and use the options in the Actions menu to either Generate code based on the latest ETL generation, of just use the Generate option to be able to choose based on which ETL generation you want to create the code. ![image](media://ed2408d4-6bb2-4e3d-85a3-23f18c08219c)   When you select ‘Generate’, you get a screen with all ETL generations related to the source that is attached to this flow. Select the ETL generation where you want your flow to be based on and click the ‘Start Generation’ button. ![image](media://2bd2c21e-3ab7-47c5-a051-61a3a3b96885)   This will launch new tasks, one to generate the code and one for the agent to download. > ℹ️ If you clicked the ‘Generate the Latest' button, it will automatically start the tasks based on your latest ETL generation. Selecting a workflow and clicking "View generations" will show you all your workflow generations, with their settings at that time, for which data vault and source release it was generated, when they were generated, and the name of the zip file containing the generated code. ![image](media://d97cf8f7-f331-4cf0-8875-070b2edf7a24) A new screen will open containing a list of all ETL generations for the specific configuration. ![image](media://25f4b7fa-f2f2-4992-9b09-c5d5e11a6c24)   The generated zip file for FL flows contains the following files: - _AZURE_FMC_BATCH_PROCS_.sql procedures which group mappings (only if **Group Tasks** is enabled). procedure names: `.__grp_` - _AZURE_FMC_FMC_MTD_.sql FMC metadata configuration procedure. name: `SET_FMC_FL_MTD_` - _AZURE_FMC_INFO_.txt text file containing generation info (dv & source releases and Vaultspeed version and generation date) - _AZURE_FMC_PIPELINE_.json ADF pipeline definition json - _AZURE_FMC_SET_FAILURE_.sql procedure which marks a run as failed - _AZURE_FMC_SET_SUCCESS_.sql procedure which marks a run as successful - _AZURE_FMC_TRIGGER_.json ADF trigger definition json The generated zip file for BV flows contains the following files: - _AZURE_FMC_BATCH_PROCS_.sql procedures which group mappings (only if **Group Tasks** is enabled). procedure names: `.__bv_grp_` - _AZURE_FMC_BV_MTD_.sql FMC metadata configuration procedure. name: `set_fmc_bv_mtd` - _AZURE_FMC_BV_CHECK_RUNNING_SRC_.sql procedure which checks whether any source loads are still active, the BV load wont start as long as this procedure fails. - _AZURE_FMC_BV_CHECK_SRC_.sql procedure which checks whether the latest load for one of the sources failed, if so will stop BV execution (should be restarted after source load is fixed). - _AZURE_FMC_INFO_.txt text file containing generation info (dv & source releases and Vaultspeed version and generation date) - _AZURE_FMC_PIPELINE_.json ADF pipeline definition json - _AZURE_FMC_SET_FAILURE_.sql procedure which marks a run as failed - _AZURE_FMC_SET_SUCCESS_.sql procedure which marks a run as successful - _AZURE_FMC_TRIGGER_.json ADF trigger definition json ## Deployment Open the Automatic Deployment page, select the FMC generation you want to deploy, click Deploy and select a link, or use Git. The automatic deployment uses the ADF API to deploy the pipelines and triggers and a JDBC link to deploy the procedures. See **setup **for the required link definition. ## Usage After deploying all the pipelines, go to ADF. You should see all your pipelines in the Author tab: ![image](media://c4adec8a-3952-4783-bc6a-9814a11ea8a8) Select one of the INIT FL pipelines: ![image](media://204b8b27-9a4e-42c9-8f36-98dd1dbf4a12) Click `Add Trigger` and select `Trigger now`. Click ok or modify the load date if needed, by default the **Start Date** of the workflow as defined in VaultSpeed is used, see discussion above for its meaning/use. Do this for all your source initial loads and then for the business vault. ![image](media://bb8c48c0-978f-48b7-a073-b8666ae5a1ef)   The incremental load triggers are paused by default (this seems to be a limitation of creating triggers with the API). To start the incremental loads, open the Manage tab in ADF, go to Triggers and click Activate for all of them. ![image](media://a3653136-48ad-4d35-851b-b28a779f21a6) ![image](media://579e24c8-a303-4e73-9c42-7ac325054f78) After all the triggers are activated, click Publish all. The pipelines will start running. ![image](media://31f299d4-708f-4117-b8d7-dac24cf42305)