Kafka Migration Synchronization to Kafka

NineData data replication supports incremental data replication between Kafka instances.

Feature Introduction

NineData data replication supports high-performance replication of structure, full data, and incremental data between data sources. For MySQL data sources, it also provides bidirectional replication, enabling the quick construction of geo-distributed active-active business architectures.

• Structure: Supports the replication of object structures between homogeneous and heterogeneous data sources, greatly reducing the barrier to data replication between two sources.
• Full Data: Achieves row-level concurrent batch replication through intelligent data sharding, effectively ensuring replication performance. The independently developed novel breakpoint resume technology ensures the accuracy of data in tables without primary keys.
• Incremental Data: Supports incremental data replication of full object types DML|DDL, combined with transaction-level concurrency, hotspot merging, and other technologies to provide robust replication performance while ensuring transaction consistency.
• Bidirectional Real-time Data Replication (only between MySQL instances): Direct bidirectional replication of data between multiple nodes ensures that all node data remains up-to-date.

With the above features, it is easy and efficient to achieve scenarios such as full|incremental data replication, full|incremental data migration, full|incremental data synchronization, data integration, and seamless migration without downtime, providing enterprises with flexible and reliable data replication solutions.

Prerequisites

• The source and target data sources have been added to NineData. For instructions on how to add them, see Create Data Source.
• The source and target data sources are Kafka, with versions 2.0.X, 1.0.X, 0.10.X.
• Based on the names of the fields in the Kafka JSON object, corresponding databases, tables, and columns have been created in the MySQL on the target side.
• If the source and target data sources selected different regions when added to NineData, you also need to add the IP addresses of the respective NineData servers to each other's data source whitelist. You can view the IP addresses of both when creating a replication task, on the node page.

Operation Steps

Commercialization Notice

NineData’s data replication product has been commercialized. You can still use 10 replication tasks for free, with the following considerations:

• Among the 10 replication tasks, you can include 1 task, with a specification of Micro.

• Tasks with a status of do not count towards the 10-task limit. If you have already created 10 replication tasks and want to create more, you can terminate previous replication tasks and then create new ones.

• When creating replication tasks, you can only select the you have purchased. Specifications that have not been purchased will be grayed out and cannot be selected. If you need to purchase additional specifications, please contact us through the customer service icon at the bottom right of the page.

1. Log in to the NineData Console.

2. Click on in the left navigation bar.

3. On the page, click on .

4. On the tab, configure according to the table below and click on .

   Parameter	Description
   	Enter the name of the data synchronization task. To facilitate subsequent search and management, please use a meaningful name. Up to 64 characters are supported.
   	The data source where the synchronization object is located, select the Kafka data source where the data to be replicated is located.
   	The data source that receives the synchronization object, select the target Kafka data source.
   	: Find the earliest message in the Kafka Topic and replicate it to MySQL in order. : Ignore the existing data in the Kafka Topic and only replicate the messages generated after the task starts. : Customize the message position in the Kafka Topic, i.e., the Offset number. The system will start replicating messages from this Offset number in order.
   	: Stop the task if data is detected in the target table during the pre-inspection stage. : Ignore the data if it exists in the target table during the pre-inspection stage and append the other data. : Delete the data if it exists in the target table during the pre-inspection stage and rewrite it.

5. On the tab, configure the following parameters, then click on .

   Parameter	Description
   	Select the object to be replicated. You can choose to include all Kafka Topics, or you can choose , select the Topics to be replicated in the list, and then click > to add to the right list.

6. On the tab, you can configure the mapping relationship between the source and target Kafka Topic names individually.

7. On the tab, wait for the system to complete the pre-inspection. After the pre-inspection is passed, click on .

   tip

   • If the pre-inspection is not passed, you need to click on the in the column on the right side of the target inspection item to investigate the cause of the failure, manually fix it, and then click on to re-execute the pre-inspection until it passes.

   • The with can be fixed or ignored according to the specific situation.

8. On the page, it prompts , and the synchronization task starts running. At this time, you can perform the following operations:

   • Click on to view the execution of each stage of the synchronization task.
   • Click on to return to the task list page.

View Synchronization Results

1. Log in to the NineData Console.

2. Click on in the left navigation bar.

3. Click on the ID of the target synchronization task on the page to open the page, the page description is as follows.

   

   No.	Function	Description
   1	Synchronization Delay	The data synchronization delay between the source and target data sources, 0 seconds means there is no delay between the two ends.
   2	Configure Alerts	After configuring the alert, the system will notify you through the method you choose when the task fails. For more information, see Introduction to Operations Monitoring.
   3	More Operations	: Pause the task, only tasks with the status are selectable. : End the unfinished or listening (i.e., incremental synchronization) tasks, the task cannot be restarted after termination, please operate with caution. : Delete the task, the task cannot be restored after deletion, please operate with caution.
   4	Incremental Replication	Display the monitoring indicators of incremental replication. Click on the on the right side of the page: View the execution log of incremental replication. Click on the icon on the right side of the page: View the latest information.
   5	Modify Object	Display the modification records of the synchronization object. Click on the on the right side of the page, you can configure the synchronization object. Click on the icon on the right side of the page: View the latest information.
   6	More	Display the detailed information of the current replication task, including the Kafka Topic being synchronized, replication object, , etc.

Appendix: Pre-inspection Item List

Inspection Item	Inspection Content
Source Data Source Connection Check	Check the gateway status of the source data source, whether the instance is reachable, and the accuracy of the username and password
Target Data Source Connection Check	Check the gateway status of the target data source, whether the instance is reachable, and the accuracy of the username and password
Target Database Data Existence Check	Check if the data to be replicated already exists in the target database
Target Database Permission Check	Check if the Kafka account has access permission to the Topic
Target Database Data Existence Check	Check if there is already data in the Topic

Related Documents

Introduction to Data Replication