Kafka Migration Synchronization to Kafka
NineData data replication supports incremental data replication between Kafka instances.
Feature Introduction
NineData data replication supports high-performance replication of structure, full data, and incremental data between data sources. For certain data sources, it also provides bidirectional replication, enabling the quick construction of geo-distributed active-active business architectures.
- Structure: Supports the replication of object structures between homogeneous and heterogeneous data sources, greatly reducing the barrier to data replication between two sources.
- Full Data: Achieves row-level concurrent batch replication through intelligent data sharding, effectively ensuring replication performance. The independently developed novel breakpoint resume technology ensures the accuracy of data in tables without primary keys.
- Incremental Data: Supports DML and DDL incremental data replication for all object types. By leveraging technologies such as row-level concurrency and hotspot merging, it delivers robust replication performance.
- Bidirectional Real-time Data Replication (only between MySQL instances): Direct bidirectional replication of data between multiple nodes ensures that all node data remains up-to-date.
With the above features, it is easy and efficient to achieve scenarios such as full|incremental data replication, full|incremental data migration, full|incremental data synchronization, data integration, and seamless migration without downtime, providing enterprises with flexible and reliable data replication solutions.
Prerequisites
- The source and target data sources have been added to NineData. For instructions on how to add them, see Create Data Source.
- The source and target data sources are Kafka, with versions 2.0.X, 1.0.X, 0.10.X.
- Based on the names of the fields in the Kafka JSON object, corresponding databases, tables, and columns have been created in the MySQL on the target side.
- If the source and target data sources selected different regions when added to NineData, you also need to add the IP addresses of the respective NineData servers to each other's data source whitelist. You can view the IP addresses of both when creating a replication task, on the Source & Target node page.
Operation Steps
NineData’s data replication product has been commercialized. You can still use 10 replication tasks for free, with the following considerations:
Among the 10 replication tasks, you can include 1 Incremental task, with a specification of Micro.
Tasks with a status of Terminated do not count towards the 10-task limit. If you have already created 10 replication tasks and want to create more, you can terminate previous replication tasks and then create new ones.
When creating replication tasks, you can only select the Spec you have purchased. Specifications that have not been purchased will be grayed out and cannot be selected. If you need to purchase additional specifications, please contact us through the customer service icon at the bottom right of the page.
Log in to the NineData Console.
Click on Replication in the left navigation bar.
On the Replication page, click on Create Replication.
On the Source & Target tab, configure according to the table below and click on Next.
Parameter Description Name Enter the name of the data synchronization task. To facilitate subsequent search and management, please use a meaningful name. Up to 64 characters are supported. Source The data source where the synchronization object is located, select the Kafka data source where the data to be replicated is located. Target The data source that receives the synchronization object, select the target Kafka data source. Replication Started - Earliest (Copy Earliest Message from Topic): Find the earliest message in the Kafka Topic and replicate it to MySQL in order.
- Latest (Copy Latest Message from Topic after Starting): Ignore the existing data in the Kafka Topic and only replicate the messages generated after the task starts.
- Customer Offset (Start From Customer Offset): Customize the message position in the Kafka Topic, i.e., the Offset number. The system will start replicating messages from this Offset number in order.
Target Table Exists Data - Pre-Check Error and Stop Task: Stop the task if data is detected in the target table during the pre-inspection stage.
- Ignore the existing data and append : Ignore the data if it exists in the target table during the pre-inspection stage and append the other data.
- Clear the existing data before write: Delete the data if it exists in the target table during the pre-inspection stage and rewrite it.
On the Objects tab, configure the following parameters, then click on Next.
Parameter Description Replicate Objects Select the object to be replicated. You can choose All Objects to include all Kafka Topics, or you can choose Customized Object, select the Topics to be replicated in the Source Object list, and then click > to add to the right Target Object list. On the Mapping tab, you can configure the mapping relationship between the source and target Kafka Topic names individually.
On the Pre-check tab, wait for the system to complete the pre-inspection. After the pre-inspection is passed, click on Launch.
tipIf the pre-inspection is not passed, you need to click on the Details in the Actions column on the right side of the target inspection item to investigate the cause of the failure, manually fix it, and then click on Check Again to re-execute the pre-inspection until it passes.
The Result with Warning can be fixed or ignored according to the specific situation.
On the Launch page, it prompts Launch Successfully, and the synchronization task starts running. At this time, you can perform the following operations:
- Click on View Details to view the execution of each stage of the synchronization task.
- Click on Back to list to return to the Replication task list page.
View Synchronization Results
Log in to the NineData Console.
Click on Replication in the left navigation bar.
Click on the ID of the target synchronization task on the Replication page to open the Details page, the page description is as follows.
No. Function Description 1 Synchronization Delay The data synchronization delay between the source and target data sources, 0 seconds means there is no delay between the two ends. 2 Configure Alerts After configuring the alert, the system will notify you through the method you choose when the task fails. For more information, see Introduction to Operations Monitoring. 3 More Operations - Pause: Pause the task, only tasks with the status Running are selectable.
- Terminate: End the unfinished or listening (i.e., incremental synchronization) tasks, the task cannot be restarted after termination, please operate with caution.
- Delete: Delete the task, the task cannot be restored after deletion, please operate with caution.
4 Incremental Replication Display the monitoring indicators of incremental replication. - Click on the Log on the right side of the page: View the execution log of incremental replication.
- Click on the
icon on the right side of the page: View the latest information.
5 Modify Object Display the modification records of the synchronization object. - Click on the Modify Objects on the right side of the page, you can configure the synchronization object.
- Click on the icon on the right side of the page: View the latest information.
6 More Display the detailed information of the current replication task, including the Kafka Topic being synchronized, replication object, Started, etc.
Appendix: Pre-inspection Item List
| Inspection Item | Inspection Content |
|---|---|
| Source Data Source Connection Check | Check the gateway status of the source data source, whether the instance is reachable, and the accuracy of the username and password |
| Target Data Source Connection Check | Check the gateway status of the target data source, whether the instance is reachable, and the accuracy of the username and password |
| Target Database Data Existence Check | Check if the data to be replicated already exists in the target database |
| Target Database Permission Check | Check if the Kafka account has access permission to the Topic |
| Target Database Data Existence Check | Check if there is already data in the Topic |