---
title: "Confluent Kafka"
canonical: "https://docs.vaultspeed.com/space/VPD/3081666598/Confluent%20Kafka"
format: markdown
---
VaultSpeed supports [Confluent Kafka](https://www.confluent.io/) - an event streaming platform for Apache Kafka.

The current integration allows reading the metadata, collecting and storing information of Kafka topics and their messages. Reading the messages and storing them in the target database can be done by the [Flow Management Control (FMC) ](https://vaultspeed.atlassian.net/wiki/spaces/VPD/pages/3013640233)application. At the start of a load, all messages in a configured window of time are collected and stored in the target database. Streaming of the real-time data is supported in our streaming solution: [https://vaultspeed.atlassian.net/wiki/spaces/VPD/pages/3080028173](https://vaultspeed.atlassian.net/wiki/spaces/VPD/pages/3080028173).

### Getting Metadata from Kafka Schema Registry

Schema Registry is used to get the metadata about Kafka topics which are used to stream messages. It is a RESTful interface for storing and retrieving Apache Avro® schemas. Since it contains the metadata of registered Kafka topics, it is currently used to configure the target database prior to reading and storing the data in the target database. Therefore, only the topics which have been registered in Schema Registry are read and collected with the current version of Kafka integration. More information about schema management on Confluent can be found [here](https://docs.confluent.io/current/schema-registry/index.html).

![image](media://e785213e-744a-49f7-ab2d-10306ae0b491)

### Collecting Data from Kafka Topics

A list of available Kafka topics is received from the Schema Registry. Source and target metadata files are used to define which topics and respective tables and columns should be imported. On the first data load, all available messages as consumed from the Kafka log. Received messages are then sent to the target database in batches. How big the batch is, source and target metadata files can be defined within the [FMC](https://vaultspeed.atlassian.net/wiki/spaces/VPD/pages/3038412824) DAG configuration. After the first load is completed, the consecutive data loads are then performed to collect any new data from the Kafka topic. The period at which the data is being collected is defined within and executed by FMC. If the collection or storage of the Kafka messages fails, the consumed number of messages is not acknowledged which means the messages are going to be collected with the next DAG run.

![image](media://17e4945d-9b6f-4183-9e1c-168917d5d820)

### Custom Feature Requests

Custom Kafka features or modifications are considered upon request. Contact us to get more information.