--- title: "Flow Management Control (FMC)" canonical: "https://docs.vaultspeed.com/space/VPD/3013640233/Flow%20Management%20Control%20(FMC)" format: markdown --- VaultSpeed's Flow Management Control (FMC) is a powerful tool designed to streamline and automate the management of your data warehouse loading processes. FMC revolutionizes the way you schedule and execute data warehouse loads, bringing efficiency and improved data quality to the forefront. Explore the following topics to unravel the depth and capabilities of FMC: **Table of Content** > Macro (toc) ### How FMC Works FMC works by generating code that defines a workflow for either [Apache Airflow](https://airflow.apache.org/) or [Azure Data Factory](https://azure.microsoft.com/en-us/products/data-factory/) (ADF). This workflow specifies the steps involved in loading data into the Data Vault, including the data sources to be used, the transformation steps to be applied, and the target Data Vault tables. ### FMC for Airflow For Airflow, VaultSpeed has developed a custom plugin specifically tailored for Data Vault loads. This plugin provides extra features that enhance the Data Vault loading experience, including: - Optimized scheduling and execution of Data Vault loads - Enhanced monitoring and alerting capabilities - Seamless integration with Airflow's existing infrastructure The VaultSpeed Airflow plugin can be effortlessly downloaded through the VaultSpeed cloud application, ensuring a straightforward installation process. ### FMC for Azure Data Factory (ADF) For ADF, FMC leverages the native functionalities of ADF, eliminating the need for additional plugins. This approach ensures seamless integration with ADF's existing infrastructure and capabilities, allowing users to utilize ADF's robust features for managing Data Vault loads. ### FMC Features FMC offers a range of features for Data Vault load management, including: - **Automated Scheduling**: Automatically schedule Data Vault loads based on factors like data availability or time of day. - **Dependency Management**: Handle complex data dependencies to ensure correct data loading order. - **Platform Support**: Supports multiple platforms, including Google BigQuery and Microsoft Azure Synapse Analytics. - **Monitoring and Alerting**: Provides capabilities to ensure smooth Data Vault loads. ### Handling FMC During Delta Generation **Overview ** During Delta generation, FMC plays a crucial role in ensuring that you only load those entities that were created based on the changes (deltas) triggered by the latest release. When you choose to generate an initial load setup of an FMC flow based on a Delta Generation ETL, a specific workflow will be generated that will execute an initial load of only the newly created objects in your data model. The incremental load version of a workflow setup that is done on the Delta Generation ETL will contain the complete incremental workflow for the entire target data model. > ⚠️ ### **Attention** > ⚠️ > ⚠️ **When generating an FMC (Flow Management Configuration) setup for an INIT load targeting a specific source and based on a particular delta generation, it’s important to understand how ETL generations are filtered.** > ⚠️ > ⚠️ **A generation marked as "COMP" (compare) from a delta generation will not always appear in the list of available ETL generations. This is because delta generations do not necessarily include a loading job for the selected source. As a result, an actual run of that generation for the INIT load of that source is required for it to be considered valid in this context.** > ⚠️ > ⚠️ **Additionally, selecting the “Latest” option from the ETL generations list does not simply refer to the most recently created generation. Instead, it refers to the most recent generation that includes execution steps for the selected source. This ensures that the INIT load configuration aligns with a generation where actual execution is needed, avoiding unnecessary or invalid setups.** **Key Benefits** - **Up-to-Date Configurations**: Automatically keeps all configurations and components current, reducing the risk of using outdated settings. - **Efficient Data Processing**: Focuses only on the changes (deltas), minimizing unnecessary processing and improving overall efficiency. - **Simplified Workflow**: Automates deployment and configuration updates, so you can focus on other tasks without manual updates. > ℹ️ **Important Considerations** > ℹ️ > ℹ️ - **Workflow Adjustments**: Make sure your FMC workflows are set up to handle only the incremental data changes to avoid reprocessing entire datasets. > ℹ️ - **Dependency Management**: Properly manage data dependencies to ensure all required data is available before processing the deltas. > ℹ️ - **Performance Optimization**: Fine-tune FMC settings to address any performance challenges, particularly with large datasets. > ℹ️ - **Error Handling**: Implement mechanisms to handle any issues that might arise during Delta generation, such as incomplete data loads. > 📝 **Common Issues and Solutions** > 📝 > 📝 - **Incomplete Delta Processing**: Verify that FMC workflows correctly identify and process the delta data. > 📝 - **Performance Bottlenecks**: If you experience slowdowns, adjust the scheduling to spread out the load or run delta processing more frequently. > 📝 - **Data Consistency**: Always check that each delta is fully processed before moving on to the next one to avoid data inconsistencies. ### Getting Started with FMC To get started with FMC, install Apache Airflow on your server. Once Airflow is installed, download the FMC plugin from the VaultSpeed cloud application. This plugin integrates with Airflow, providing a streamlined approach to managing data warehouse loads. With FMC, efficiently configure and manage Data Vault loads directly within Airflow. ### Additional Resources [Airflow](https://vaultspeed.atlassian.net/wiki/spaces/VPD/pages/3012624432) - [Airflow installation instructions](https://vaultspeed.atlassian.net/wiki/pages/createpage.action?spaceKey=vpd&title=Airflow%20Installation%20Instructions&linkCreation=true&fromPageId=3013640233) - [Airflow FMC walkthrough](https://vaultspeed.atlassian.net/wiki/spaces/VPD/pages/3038412824) - [Astronomer](https://vaultspeed.atlassian.net/wiki/spaces/VPD/pages/3038937120) [Azure Data Factory](https://vaultspeed.atlassian.net/wiki/spaces/VPD/pages/3012788295) - [ADF as FMC for Databricks](https://vaultspeed.atlassian.net/wiki/spaces/VPD/pages/3164635181) - [Matillion Native Scheduling](https://vaultspeed.atlassian.net/wiki/spaces/VPD/pages/3012984865) - [FMC loading windows & source data management](https://vaultspeed.atlassian.net/wiki/spaces/VPD/pages/3081633856) - [Generic FMC](https://vaultspeed.atlassian.net/wiki/spaces/VPD/pages/3364388956)