---
title: "2021"
canonical: "https://docs.vaultspeed.com/space/VPD/3164700720/2021"
format: markdown
---
> Macro (excerpt)
> 
> > Macro (toc)

### Version 4.2.6

Release date: 19/12/2021

> ⚠️ The Agent client.properties file needs to be updated to use a new URL: `vs.url=https://app.vaultspeed.com/api/agent/` or `vs.url=https://app-au.vaultspeed.com/api/agent/` for the Australian instance  
> ⚠️ The Agent itself also has to be updated due to major API changes.

- Features:
  - The API behind the application has been completely redesigned to make it publicly accessible and consumable. All data shown in the application and every action done in the app can also be done by calling the corresponding API endpoints. This allows for additional automation and integration with other tools.  
This is only a closed beta release, during which we will be collecting feedback. Documentation and access will only be provided for select customers and partners.  
API access is a separate subscription package and is only available for gold and platinum subscriptions. Some endpoints will be available for all customers, such as downloading the Agent and the Airflow plugin.
  - Added** ETL settings**, this allows the user to add extra code to the generated mappings. Depending on the ETL generation language, different settings can be added.  
The following is an example for snowflake procedures. Everywhere you see “@” followed by a name is a possible location for adding extra code.
    The possibilities are endless, from changing execution rights by adding `Execute as owner` to adding custom logging statements with row counts after every DML statement.  
The ETL settings work similarly to the DDL settings. The full documentation can be found at [https://vaultspeed.atlassian.net/wiki/spaces/VPD/pages/3012755484](https://vaultspeed.atlassian.net/wiki/spaces/VPD/pages/3012755484) .
  - Added support for **Matillion Synapse**, Matillion is now also available as an ETL tool for Synapse (Azure DWH) Data Vaults.  
This is only supported for <u>INSERT ONLY</u> Data Vaults at the moment due to a limitation in transaction handling between Matillion and Synapse. We are working on a workaround such that we can remove this limitation in a future release.
  - Added support for **Spark Structured Streaming** in alpha version. This is only available to selected customers and partners at this moment.
- Important changes:
  - Added the possibility to define DDL settings for all the standard BV objects (so no studio templates yet, this will be added later on).
  - The initial load STG mappings now also use the SATs for looking up BKs instead of only using the extraction table. This is mainly useful during delta generations when loading the initial data for a new object which references an existing object.
  - Added two extra DV parameters: CAST_TO_NVARCHAR_IN_HASH and CAST_TO_VARCHAR_IN_HASH. These can be used to control the hashing behavior and determine if and to which type the business keys are cast before hashing them. These parameters are beneficial for SQL Server and Synapse and are mutually exclusive.
  - Added extra logic to the BV release creation to catch cases where bridges become invalid. When an object gets deleted from a Data Vault while being used in a bridge, the initial BV release created when locking that DV release will be unlocked. This means that no code can be generated for this new DV release. Only after resolving the issue in the bridge and locking the business vault can code be generated. While there is still an invalid bridge in a BV, hovering over the (grayed out) lock button will display which bridge is faulty.
  - Updated the template language to allow repeating templates. This means that in the VaultSpeed Studio templates, you can now generate a query for every SAT of a HUB or all DV objects in a bridge. Example template code:  
`Template $ DVO_TEMPL`  
`templaterepeatedbycomponent DVO`
  - A list of logged-in users can be shown on the dashboard page(“show” button next to the active users counter) and on the screen displayed when the active user limit is reached.
- quality of life changes:
  - All tasks can now be restarted, also the agent tasks.
  - Suppose a delta generation does not detect any changes. In that case, the file download task will fail with a message indicating this instead of failing with a technical error.
  - Improved performance for ETL generation.
  - Improved performance for delta DDL generation.
  - Added a default sorting on the PIT tables screen.
  - After locking a source, the user is now redirected back to the releases screen, similar to what happens after saving the source model.
  - In the source editor, moved the selection of which object names to use to a dropdown in the toolbar instead of hiding it in the extra’s menu.
  - The Auto deploy screen will now also show generations for which the "get files” task is not yet completed. These generations cannot be deployed, but it is possible to download the generated code through the browser. This means that you don't need to have an agent running to download the generated code directly.
  - On the Generation pages, the Data Vault release selection dropdowns now also show the release comment instead of only the date and name.
  - Added an extra check to enforce that all objects in a source with CDC_UPDATE_RECORD_ALL_ATTRIBUTES = N have the CDC type of “CDC” since mixed CDC-type sources are not supported in this case.
  - Added extra checks to prevent the creation of pattern removals which would cause objects names to become empty strings.
  - Added a new SCHEMA_INI_R parameter. This determines the source schema used by the FMC source loading. Previously it always used the source’s physical schema.
- Bug fixes:
  - Fix auto-deploy to Databricks failing for agents running on Windows
  - Fixed an issue with the Snapshot PIT view for Snowflake, Bigquery, and Spark.
  - Fixed CSV file name for metadata export allowing illegal characters.
  - Fixed duplicate records for business views appearing in the metadata export.
  - Fixed an issue in the FMC when source loading is enabled for objects with CDC type “no CDC incremental.”
  - Created a fix for Spark JDBC connections in the FMC source loading operator. It did not work due to a bug in the Airflow JDBC Hook.
  - Fixed an issue in the FMC source loading operator when loading date fields into Spark.
  - Fixed Databricks timestamp fields in the generated DDL using a scale definition.
  - Fixed an issue with PIT tables on multi-active Sattelites with a subsequence attribute.
  - Fixed the REF table logic for insert only. It now contains a delete and an insert when a record gets updated.
  - If Merges are enabled and insert only is used, then the hub and link mappings will use an insert statement instead of a merge. This is to align them with the satellite mappings and comply with the strict definition of insert only.
  - Many-to-Many links that do not have matching driving key definitions will no longer be automatically grouped since this could cause issues in the generation.
  - When using early arriving facts and an object is altered. The hub mapping of the referenced object will be properly regenerated in a delta generation since it will have to include a new staging object to load from.
  - The Data Vault release creation screen will no longer list hidden source releases.
  - Fixed an issue where searching in the HUB grouping screen crashed the browser in some cases for large Data Vaults.
  - Fixed an issue where if a task failed due to a misconfiguration error by the user, the error message was not correctly shown in the task info and instead offered a generic message intended for technical errors.
  - Fixed the Agent metadata harvesting from Kafka schema registry crashing when there are objects created by a KSQL statement containing aggregations.
  - Fixed the search bar in the source editor being case-sensitive. It is now case insensitive again.
  - Fixed the save time in the DDL settings not being in the proper format and not getting converted to local time.
  - Fixed a rare issue where an ETL generation was not visible when trying to generate FMC code when a source task failed for a source with the same id as the ETL generation.
  - Fixed the “Upgrade subscription” button on the dashboard redirecting to the wrong URL.
  - Fixed missing description for the Agent ping task.
  - Fixed Business views for Databricks being generated when selecting load type INIT, instead of only when selecting load type DDL.
  - Fixed records getting added to transaction timestamp based snapshot PITs where the load date > snapshot timestamp during the initial load.
  - Fixed performance issues in the HUB grouping screen which could cause the page to crash for large Data Vaults.Vaul

### Version 4.2.5

Release date: 12/09/2021

- Features:
  - Added Support for **Matillion ETL for Snowflake: **For Snowflake Data Vaults, there is now a new ETL Generation Type, “Matillion Snowflake”. This will generate JSON files that can be imported into Matillion to create the ETL mapping, either by using the VaultSpeed auto-deploy or your own deployment pipeline. The VaultSpeed FMC for Airflow can then execute these Matillion Jobs. More info can be found in our documentation: [https://vaultspeed.atlassian.net/wiki/spaces/VPD/pages/3012952079](https://vaultspeed.atlassian.net/wiki/spaces/VPD/pages/3012952079) . Matillion ETL for Synapse will be coming soon, and support for other FMC technologies such as ADF and Matillion native scheduling.  
Note that this is still a beta release and there could still be issues in the generated code for some configurations.
  - We have added the ability to generate ETL code for only a specific object. If a source object is selected, then all the code to load the resulting Data Vault objects will be generated.  
If a Data Vault object is selected, then all the code to load that object will be generated. This can be further limited by also specifying a source (see example below). If a Business object is selected, then only the code for that object will be generated.
  - Added support for Multi-active Sattelites without a subsequence attribute. This is only available for objects with no CDC or no CDC incremental since we need all the records per key to be delivered each load.
- Important changes:
  - We have renamed the mapping counter to a more generic name: VaultSpeed Automation Units. This should avoid some confusion since not only the generated mappings are counted. More information can be found in the documentation [https://vaultspeed.atlassian.net/wiki/pages/createpage.action?spaceKey=vpd&title=VaultSpeed%20Automation%20Unit%20%28VAU%29&linkCreation=true&fromPageId=3164700720](https://vaultspeed.atlassian.net/wiki/pages/createpage.action?spaceKey=vpd&title=VaultSpeed%20Automation%20Unit%20%28VAU%29&linkCreation=true&fromPageId=3164700720) .
  - Source copies will now also consume VAUs. The number of units consumed is equal to the number of objects copied.
  - The format mask for source attributes is now a free text field instead of a selection menu. Note that this format must be valid for the target database, meaning that ids must be entered when the source is used in a SQL Server or Azure Synapse Data Vault.
  - Added the record source attribute to Links and reference tables instead of only to HUBs.
  - Added the load cycle id attribute to the PIT and Bridge tables. This should help with the performant loading of business objects on top of them.
  - Enabled Non Insert only logic for Azure Synapse since it now supports the required Update functionality.
- quality of life changes:
  - In the source editor, you can now change which names(full, abbreviated, or short) are shown for objects in the selection pane and on the canvas. This should help for cases where the source object names are not intelligible, and the abbreviated names are used to give meaningful names to the objects for use in the Data Vault.
  - Improved performance for ETL metadata generation.
  - Renamed the source graphical editor to just the source editor, because the other editor has been deprecated for a while now.
  - Improved performance of the mass update screen in the source editor.
  - Increased the width of the flow name column in the FMC screen to improve readability when using long names.
- Bug fixes:
  - Fixed the timestamp attribute name for a no CDC SAT in a PIT that uses the Transaction timestamp. It will now more appropriately be named based on the load date instead of the transaction date.
  - Fixed an issue with the ODI connection test in the agent which caused it to always fail. <span style="color: #ff5630">This fix requires an update of the agent.</span>
  - Fixed an issue where if cast_source_attributes is enabled and there is an attribute with a non-default format mask, the exception records would not be added with the correct format, causing the extraction mapping to fail.
  - Fixed an issue in the source editor, where a multi-active object would disappear from the selection pane when the “show only completed objects” toggle was used until a browser refresh was done.
  - Fixed the BK management page not refreshing when using the button to update the concatenation flag for all selected objects. This caused the errors to still show even when they were fixed by the update.
  - Fixed an issue where Talend jobs could fail when a “\” was used in an expression.

### Version 4.2.4

Release date: 16/06/21

- Features:
  - When creating a new Data Vault release, you can now select which sources should be included in the release. This selection will also be remembered for the following releases. This allows for the creation of multiple Data Vaults in 1 project which shares only a subset of sources. Before this would require changing the source build flags before creating the DV releases.
  - It is now possible to mark objects in the source editor as completed, the completed objects will appear with a checkmark. This status can be toggled by right-clicking on an object. The selection page can filter out completed objects, and there is also a button to remove all completed objects from the canvas.
  - Added a button to the source editor toolbar which widens all objects to fit the entire object and attribute names.
  - Added support for Airflow 2.0, the plugin and generated code have been reworked such that it can also work with Airflow versions > 2.0.   
<span style="color: #ff5630">The old FMC code will not work with the updated plugin.</span>
  - Added support for deploying spark code to Databricks and executing it with Airflow. The deployment will create Spark SQL notebooks in Databricks for all the mappings. Airflow will launch jobs for running those Notebooks.  
In order to generate and be able to deploy the correct code, the target Database type is still “Spark”, but the ETL generation type has to be set to Databricks SQL. <span style="color: #ff5630">This requires an update of the agent.</span>
  - Added the ability to Copy source configurations. When creating a new source there is now an extra option to copy an existing one, this copy can work within a project or between them. In order to be able to copy a source, its latest release must be locked, all the data of that locked release will be copied to the new source.
  - Added the ability to change the ordering and naming of business keys in the DV. A new screen has been added, where for each hub group the business keys of the grouped object can be renamed and reordered, and the business keys of the hubs in the group can be reordered to match. So the keys in the different sources can now have different orders and names and still result in the same hash key calculation.
  - Added the ability to reorder the linked hubs in many-to-many links ( and non historized links). A new screen has been added, where for each many to many grouped objects, the order of the HUBs can be matched for all objects in the group.
- Important changes:
  - Renamed the “build flag” property to “ignored” everywhere in the application.
  - Added extra template variables for the custom deploy scripts in the agent, instead of only the zip name, you can now also get the generation id, the generation info, and the generation type, similar to the git commit message functionality. example:  
`deploy.cmd = sh C:\\Users\\name\\Documents\\agent\\deploy.sh {zipname} {code_type} ”{info}”`
  - The compare functionality in the source graphical overview will now skip ignored releases. This means that it will compare with the last non ignored locked release before the current one.
  - Added support for overlapping loading windows to the Azure Data Factory FMC, this can be configured by using the following parameters: FMC_OVERLAPPING_LOADING_WINDOWS, FMC_WINDOW_OVERLAP_SIZE, FMC_WINDOW_OVERLAP_TYPE.
  - The metadata-export has been converted to a task, this is done to support exporting data for very large Data Vaults. Before the export would time out and not return a file if it takes too long.
- quality of life changes:
  - Airflow code for Snowflake no longer requires JDBC, which means that it can be run on the AWS Airflow service.
  - Added a toggle to the Hub BK management screen which allows the BK concatenation flag to be updated for all shown objects at once.
  - Added an extra option to the deployment screen for using a custom script, previously this was always used when the parameter was enabled, now you can still choose to use a DBlink after enabling the parameter.  
For FMC generations there is instead an option to use the custom FMC deploy script.
  - Added pagination to the generation table on the auto-deploy screen.
  - Improved rendering performance of the source graphical editor.
  - Renamed the source attribute “Name in BK” to “'source system business key” to match the source creation screen.
  - Renamed “table” to “object” and “column” to “attributes” in places where this was not done yet, mainly in toast messages and some of the business vault screens.
  - Updated the logo in the generated files to the VaultSpeed logo.
  - Improved the performance of the source graphical overview.
  - The BV workflow in Airflow no longer requires a PL mapping JSON file when it is not used.
- Bug fixes:
  - Fixed sources with only ref tables causing empty DDL files and invalid FMC JSON files to be generated.
  - Fixed Link across sources missing from the metadata export.
  - Fixed filtering in the Data Vault overview not applying across pages.
  - Fixed the source editor not refreshing after uploading short and abbreviated names.
  - Fixed Airflow SQL code not containing identifier quotes, which means that it could not be used with a different casing than the target default.
  - Fixed expressions in generated SQL ETL code being split across lines over identifiers or strings which contain special characters.
  - Fixed Satellite tables having a load cycle id with data type numeric instead of integer (when available). New objects will have the new type, existing objects will retain the type and will have to be manually migrated when desired.

### Version 4.2.3.20

Release date: 13/04/21

This is an intermediate release focussed on delta generation fixes and test coverage.

- Features:
  - Added a **record source attribute** to the HUBs, if the parameter `add_record_source_source_name` is enabled an extra attribute is added to all hubs, the values which are stored in it are determined by the `RECORD_SOURCE_VALUE` parameter, this is a source object level parameter that can be templated, its default value is `<record_src_name>.<table_abbreviated_name>`.  
This allows tracking from which source object a hub record was loaded.
- Important changes:
  - Greatly increased our test coverage for delta generations, the document [https://vaultspeed.atlassian.net/wiki/spaces/VPD/pages/3012460559](https://vaultspeed.atlassian.net/wiki/spaces/VPD/pages/3012460559)  contains all currently supported delta changes.
  - PIT tables with snapshots can now be generated for Azure Synapse. We implemented a workaround for the recursive snapshot views using cross joins.
- quality of life changes:
  - INIT mappings will now only be generated in the delta when they are actually needed for loading new objects.
  - Added an extra check to prevent defining a many-to-many link where all relationships are driving keys.
  - Improved performance for the SGE, both in loading and rendering time. This should prevent the page from becoming unresponsive and returning to the sources page when working with very large sources.
- Bug fixes:
  - Fixed issues with default values in the delta generation for not null attributes when adding or dropping certain attributes.
  - Fixed SAT split migration when the split also contains new attributes.
  - Fixed some issues with delta generation for data type changes.
  - Fixed some mappings being regenerated for objects which did not change in a delta generation.
  - Added a workaround for Azure Synapse not supporting constraint renames.
  - Fixed some issues with constraint dropping and recreation in delta generation.
  - Fixed an issue where the deploy of a delta crashes because some of the delta scripts are empty. Empty files will no longer be generated.
  - Fixed migration scripts for Snowflake incorrectly containing JavaScript syntax from procedures.
  - Fixed logic for handling removed sources in the delta generation.
  - Fixed SAT split migration for multi active satellites with a subsequence attribute.
  - Fixed mapping drops not being generated for some deltas.
  - Fixed deleted links across sources not being properly shown in the source editor.
  - Fixed the metadata export for NHL objects.
  - Fixed an issue with CDC reliable and Insert only SAT loading logic which could cause only 1 record to be loaded in the LKS when an insert and update arrive in the same batch.
  - Fixed some issues in the single to multi master hub migration scripts.

### Version 4.2.3

Release date: 07/02/21

- Features:
  - Added the ability to define the** INI and CDC** tables as **external/foreign tables**:** **DDL options can be applied to INI or CDC tables and the template variables `{table_name}` and `{schema_name}` can be added to the options text.
  - **VaultSpeed Studio code preview**:** **When viewing a template you can now generate code for a specific object using that template in order to verify its correctness.
  - Added **Object mass update** to the source graphical editor, it is now possible to set the object type, CDC type, … for all objects matching a certain pattern.
- Important changes:
  - Greatly improved delta generation performance.
  - Greatly improved ETL metadata generation performance.
  - Improved the layout of objects in the source graphical editor, and added the ability to switch between a vertical and horizontal orientation.
  - Added the ability to show the shortest path between 2 objects on the source graphical editor.
  - The agent will now only harvest metadata for objects that are included in the release, instead of everything in the schema, this can greatly improve metadata retrieval performance for large sources with only a limited number of objects selected.   
<span style="color: #ff5630">**An agent update is required to activate this behaviour!**</span>
  - Added the ability to define a PK for Many-to-Many Links, this key will be used in the logic for CDC_UPDATE_RECORD_ALL_ATTRIBUTES = N, to find the previous version of a record. Before, it was only able to use the combination of foreign keys (and subsequence attribute).
  - The source tabular editor has been deprecated and is no longer available.
- quality of life changes:
  - The source graphical overview is now a tab inside the graphical editor, next to the source data vault visualiser.
  - Added a limit to the number of objects that can be loaded into the source graphical editors in order to maintain good performance.
  - Improved error messages for ETL generation to help with VSS template debugging.
  - The delta generation will now only generate the changed business views instead of all of them.
  - The source and intermediate layers DDL scripts are now only regenerated for changed sources.
  - The DV temp tables are now only regenerated in the delta if they actually changed.
- Bug fixes:
  - Fixed a bug where you could only delete individual relations in the source editor and not entire relationships.
  - Fixed the STG BK lookup to account for business key updates and primary key reuse.
  - Fixed unauthenticated errors showing up just after logging in.
  - Changes in parameters now properly trigger delta generations.
  - Fixed changing the CDC type of an object not triggering a delta.

### Version 4.2.2

Release date: 05/01/21

- Features:
  - Added support for **Azure Data Factory**(ADF) as FMC, it is now possible to generate & deploy ADF pipelines for executing the VaultSpeed code (only for SQL server and Synapse Data Vaults). [https://vaultspeed.atlassian.net/wiki/spaces/VPD/pages/3012788295](https://vaultspeed.atlassian.net/wiki/spaces/VPD/pages/3012788295)
- quality of life changes:
  - Attributes with errors are now shown in red on the source graphical editor.
  - Every page in VaultSpeed now contains a link to the relevant Documentation (book icon).
  - Added confirmation dialogs to delete actions such as deleting a source or project.
  - Searching in the grouped hubs now also filters on the source name.
  - Added All the subscription info to the dashboard, such as extra modules and support tier.
  - After locking a Data Vault you will now be send back to the releases screen while the task is running.
  - Double clicking on an object in the source Data Vault visualiser will now switch it between table and icon mode.
  - On the FMC screen, when generating, the latest generation is now selected automatically and after starting the generation task the page will automatically close.
  - Improved loading performance of PIT screens.
- Bug fixes:
  - Fixed an issue where grouping 2 hubs referenced by the same Many to Many link would crash the Data Vault generation.
  - Fixed the graph of the Data Vault overview being cleared when switching between show all and shortest path or single object.
  - Fixed the release ordering in the source graphical editor when selecting another source for adding cross source links.
  - Fixed the removal of a sat object not triggering a regeneration of the related PIT(s).