--- title: "DataOps.live Integration" canonical: "https://docs.vaultspeed.com/space/VPD/3496673330/DataOps.live%20Integration" format: markdown --- **Table of Contents** > Macro (toc) > ⚠️ **Public Preview in Development** > ⚠️ > ⚠️ This integration is currently in Public Preview and still in development. ## Integration Purpose [DataOps.live](https://www.dataops.live/) provides an orchestrator for integrating with VaultSpeed. This integration facilitates the deployment of a DataVault defined in VaultSpeed to Snowflake, enabling scheduled loading of the Vault. ## Workflow Overview 1. Setup VaultSpeed Agent. 2. Create a Project, sources, a Data Vault, and a Business Vault in VaultSpeed. 3. Create a project in DataOps: For detailed instructions on creating a project in DataOps, refer to the documentation [here](https://docs.dataops.live/docs/get-started/). 4. Set up a DataOps Runner: Learn how to set up a DataOps Runner by following the instructions outlined [here](https://docs.dataops.live/docs/administration/docker-runner/installation/). 5. Integrate the Vaultspeed Orchestrator into your DataOps project with step-by-step guidance available [here](https://docs.dataops.live/docs/orchestration/vaultspeed-orchestrator/). 6. Add any other prerequisites to the DataOps project (Snowflake setup, landing zone configuration, etc.). 7. Schedule your DataOps pipeline. ## How It works The VaultSpeed integration will add the following jobs to your DataOps pipeline: > Macro (drawio) ### VaultSpeed Manager The VaultSpeed Manager manages code generation, retrieval, and deployment: 1. Authenticate to the VaultSpeed API `DATAOPS_VAULTSPEED_URL` using credentials `DATAOPS_VAULTSPEED_USER` and `DATAOPS_VAULTSPEED_PASSWORD`. 2. If `DATAOPS_VAULTSPEED_DATA_VAULT_RELEASE` is empty, then select the latest locked Release of Data Vault `DATAOPS_VAULTSPEED_DATA_VAULT_NAME` in project `DATAOPS_VAULTSPEED_PROJECT`. > 📝 *A "locked release" refers to a stable version of the Data Vault or Business Vault in VaultSpeed. It ensures reliability by preventing unintended changes during integration.* 3. If `DATAOPS_VAULTSPEED_BUSINESS_VAULT_RELEASE` is empty, then select the latest locked Business Vault Release of the selected Data Vault Release from 2. 4. If there is a Production Release before the selected release, it will generate Deltas from the last production release before the selected release to the selected release. Otherwise, a full DDL and ETL generation will be done. > 📝 *"Production Release" signifies a version of the Data Vault or Business Vault in VaultSpeed intended for stable, production-ready use. This version ensures reliability during integration by providing a finalized, operational iteration for deployment.* 5. If `DATAOPS_VAULTSPEED_FORCE_GENERATION` is false, get the last DELTA or DDL+ETL Generation for the selected Release. 6. If no Generations are found in 3, or `DATAOPS_VAULTSPEED_FORCE_GENERATION` is true, then generate new DELTA or DDL+ETL code 7. If `DATAOPS_VAULTSPEED_FORCE_GENERATION` is false, then for all FMC Flows of the Data Vault, get the last FMC Generation for the selected Release. 8. If no Generations are found in 7, or `DATAOPS_VAULTSPEED_FORCE_GENERATION` is true, then generate new FMC code. 9. Download and store the code from all the selected generations. 10. If the selected Release is different from the one stored in the persisted cache or `DATAOPS_VAULTSPEED_TYPE_OF_RUN` is set to `INIT`, then Deploy the Generated DDL & MIGR code to Snowflake and set the State of the Orchestrator to INIT. Otherwise, the state will be INCR. > 📝 *"Persisted cache" refers to a stored memory or database that retains previously accessed or computed data for quicker access in subsequent operations. In this context, it stores information about selected releases or states. The Orchestrator checks this cache to determine if a selected release differs from the stored one or if a certain state has been set, optimizing deployment and execution decisions based on prior information.* 11. If the state is INIT, then trigger child pipeline execution for All FL and BV Initial load Flows. 12. If `DATAOPS_VAULTSPEED_SKIP_SCHEDULE_CONTROLLER` is true and the state is INCR, then trigger child pipeline execution for All FL and BV Incremental load Flows. > 📝 *"Child pipeline execution" refers to the automated triggering of specific sub-pipelines within the main integration process. When the state is INIT, it initiates the execution of sub-pipelines responsible for the initial loading of FL (Data Vault) and BV (Business Vault) flows.* > 📝 > 📝 *Similarly, if DATAOPS_VAULTSPEED_SKIP_SCHEDULE_CONTROLLER is set to true and the state is INCR, it activates the execution of sub-pipelines handling the incremental loading of FL and BV flows.* 13. If the state is INCR and `DATAOPS_VAULTSPEED_SKIP_SCHEDULE_CONTROLLER` is false, then for each FMC Flow, check if the pipeline execution time matches the schedule_interval for that Flow defined in VaultSpeed. If it matches, trigger the child pipeline execution for that Incremental load Flow. > Macro (drawio) ### Trigger FL Flow This task oversees individual pipelines responsible for executing the FL (Loading) Flows. The VaultSpeed Manager determines which specific loading processes are executed, enabling flexibility in the number of child pipelines generated during each run. For instance, in an incremental run scenario with two sources—source A set to load every 15 minutes and source B set to load every 60 minutes—running a pipeline at 15-minute intervals would execute source A four times and source B once across four consecutive runs. This setup demonstrates how the scheduling frequency influences the execution count of specific loading processes within the FL Flow. ### Trigger BV Flow Similarly, this task manages a dedicated pipeline for executing the BV (Business Vault) Flow, regulated by the VaultSpeed Manager. There may be instances where no child pipeline is generated for certain executions. For example, considering the previous scenario with two sources, if the Business Vault operates at 1-hour intervals, across three runs, only source A would be loaded. However, during the fourth run, sources A, B, and the BV will be loaded, illustrating how the BV Flow's execution aligns with specified schedule intervals. These examples highlight the connection between scheduling intervals and the execution frequency of associated processes within FL and BV Flows. ## Prerequisites for Using VaultSpeed with the Orchestrator To use a Data Vault from VaultSpeed with the Orchestrator, ensure the following requirements are met: ### Target Technology Requirements - The target technology should be Snowflake. ### Source System Requirements - The system parameter `USE_FMC` should be set to `Y`. - The source parameter `FMC_GENERATE_SRC_LOADING_SCRIPTS` should be set to `N` when `CDC_BASED_LOADING_WINDOW` is `Y`. - Note that no code will be generated to do the source loading, only metadata will be generated. This metadata can then be used to create your own loading logic in the pipeline. ### Data Vault Requirements - The DV parameter `ETL_GENERATION_TYPE` should be set to `DBT`. - The DV parameter `FMC_TYPE` should be set to `generic`. - The Data Vault parameter `DELTA_GENERATE_INIT_MAPPINGS` should be set to `new`. - The Data Vault parameter `DELTA_GENERATE_INCR_MAPPINGS` should be set to `all`. - The Data Vault Release names, numbers, and the Business Vault release names should be unique since they are used to select a certain release in the pipeline (see DataOps Configuration section) Note that the Business Vault release names should be unique within a Data Vault release, not across them. ### FMC Flow Requirements - All FMC flows should have the group flag enabled. - There should only be one FMC flow for each Data Vault, Source, and Load type combination. - The connection names for the FMC flows are not used and can be set to any value. - The FMC flow schedule interval should be set to a Cron expression like `*/15 * * * *` or a time expression such as `15 minutes` or `1 hour 20 minutes`. ### Example Schedule Interval For example, to set a flow to run every 15 minutes, you would use the following Cron expression: `*/15 * * * *` This expression specifies that the flow should run every 15 minutes, every hour, every day of the week, every month, and every year. ## DataOps Configuration Add the VaultSpeed Orchestrator jobs to your pipeline as described [here](https://docs.dataops.live/docs/orchestration/vaultspeed-orchestrator/). Take extra care when setting the following Parameters in the pipeline: - The `DATAOPS_VAULTSPEED_URL` should be set to something similar to `"https://app.vaultspeed.com/api"`, depending on the target VaultSpeed environment, make sure the `/api` part is present. - The `DATAOPS_VAULTSPEED_USER` should be set to your VaultSpeed username (not your email address). - The `DATAOPS_VAULTSPEED_PROJECT` should be set equal to the Name of your project in VaultSpeed. - The `DATAOPS_VAULTSPEED_DATA_VAULT_NAME` should equal the Name of your Data Vault in VaultSpeed. - The `DATAOPS_VAULTSPEED_DATA_VAULT_RELEASE` (optional) should be set to the Name of your Data Vault release. - The `DATAOPS_VAULTSPEED_BUSINESS_VAULT_RELEASE` (optional) should be set to the Name of your Business Vault release. ## Importance of Setting Parameters Correctly It is important to set the VaultSpeed Orchestrator job parameters correctly to ensure that the pipeline can successfully connect to your VaultSpeed environment and load data from your source systems to Snowflake. ### Using Optional Parameters The `DATAOPS_VAULTSPEED_DATA_VAULT_RELEASE` and `DATAOPS_VAULTSPEED_BUSINESS_VAULT_RELEASE` parameters are optional. You would only use these parameters if you want to load data from a specific release of your Data Vault or Business Vault. If you do not set these parameters, the pipeline will load data from the latest released version of your Data Vault and Business Vault.