Kafka to Kafka Replication

Use this page to replicate topic data between Kafka data sources in NineData. Kafka to Kafka replication supports incremental data replication.

Overview

NineData data replication supports schema, full data, and incremental data replication between data sources. For supported data sources, it also supports bidirectional replication for geo-distributed active-active architectures.

• Schema replication: Replicates object structures between homogeneous and heterogeneous data sources.
• Full data replication: Uses data sharding and row-level concurrent batch replication to improve throughput. Breakpoint resume helps preserve data accuracy, including for tables without primary keys.
• Incremental data replication: Replicates DML and DDL changes for supported object types. Row-level concurrency and hotspot merge processing help maintain replication throughput.
• Bidirectional real-time data replication (only between MySQL instances): Replicates changes in both directions between nodes so data can stay current across participating nodes.

Use these capabilities for full or incremental data replication, migration, synchronization, data integration, and low-downtime migration workflows.

Before you begin

• Add the source and target data sources to NineData. For details, see Create Data Source.
• The source and target data sources are Kafka 2.0.X, 1.0.X, or 0.10.X.
• Prepare the source and target Kafka topics, and plan the topic-name mapping before you create the task.
• If the source and target data sources use different regions in NineData, add the IP addresses of the corresponding NineData servers to each other's data source whitelist. View both IP addresses when creating the replication task on the Source & Target step.

Procedure

1. Sign in to the NineData Console.

2. Go to Replication.

3. On the Replication page, click Create Replication.

4. On the Source & Target tab, set the following fields, and then click Next.

   Parameter	Description
   Name	Enter a replication task name that is easy to recognize later. Up to 64 characters are supported.
   Source	Select the Kafka data source that contains the data to replicate.
   Target	Select the target Kafka data source.
   Replication Started	Earliest (Copy Earliest Message from Topic): Start from the earliest message in the Kafka topic and replicate messages to the target Kafka topic in order. Latest (Copy Latest Message from Topic after Starting): Ignore existing data in the Kafka topic and replicate only messages generated after the task starts. Customer Offset (Start From Customer Offset): Specify the message position in the Kafka topic, that is, the offset. NineData starts replicating messages from this offset in order.
   Target Table Exists Data	Pre-Check Error and Stop Task: Stop the task if the precheck detects data in the target. Ignore existing target data and append to it.: Ignore the target data detected during precheck and append the rest. Clear target existing data before write: Delete the target data detected during precheck and write it again.

5. On the Objects tab, set the following fields, and then click Next.

   Parameter	Description
   Replicate Objects	Select the topics to replicate. Choose All Objects to include all Kafka topics, or choose Customized Object, select topics from the Source Object list, and click > to add them to the Target Object list.

6. On the Mapping tab, map the source and target Kafka topic names.

7. On the Pre-check tab, wait for NineData to finish the precheck. After the precheck passes, click Launch.

   tip

   • If the precheck fails, click Details in the Actions column of the failed check item, review the cause, fix it, and then click Check Again to run the precheck again.

   • For items whose Result is Warning, fix the issue or ignore the warning when appropriate.

8. On the Launch page, after Launch Successfully appears, the replication task starts running. Use these actions to review the task:

• Click View Details to view each synchronization stage.
• Click Back to list to return to the Replication task list.

Result

1. Sign in to the NineData Console.

2. Go to Replication.

3. On the Replication page, click the ID of the target replication task to open the Details page. The task details page shows the following areas.

   

   No.	Function	Description
   1	Synchronization Delay	Shows the replication lag between the source and target data sources. 0 seconds means the target is caught up with the source.
   2	Configure Alerts	Configure alerts to notify selected contacts when the task fails. For more information, see Introduction to operational monitoring.
   3	More Operations	Pause: Pause a task whose status is Running. Terminate: Terminate a task that is unfinished or still listening for incremental changes. A terminated task cannot be restarted. Proceed with caution. Delete: Delete the task. Deleted tasks cannot be restored. Proceed with caution.
   4	Incremental Replication	Shows incremental replication metrics. Click Log to view incremental replication logs. Click the icon to view the latest information.
   5	Modify Object	Shows modification history for the synchronization object. Click Modify Objects to configure the synchronization object. Click the icon to view the latest information.
   6	More	Shows details of the current replication task, including the Kafka Topic being synchronized, replication object, Started, and other settings.

Appendix: Precheck items

Check Item	Check Content
Source Data Source Connection Check	Checks the source data source gateway status, instance reachability, and username/password validity.
Target Data Source Connection Check	Checks the target data source gateway status, instance reachability, and username/password validity.
Target Database Data Existence Check	Checks whether the data to be replicated already exists in the target database.
Target Database Permission Check	Checks whether the Kafka account has permission to access the Topic.
Target Database Data Existence Check	Checks whether data already exists in the Topic.

Next steps

Introduction to Data Replication