Data Replication between MongoDB Instances

NineData supports data synchronization from MongoDB to MongoDB.

Prerequisites

• The source and target data sources have been added to NineData. For instructions on adding data sources, please refer to Adding Data Sources.

• The deployment architecture of MongoDB is either standalone or replica set.

• The versions of the source and target data sources are as listed in the table below. Note that the version of the source data source must be less than or equal to the version of the target data source.

  Source Datasource	Target Datasource
  MongoDB 6.0	MongoDB 6.0
  MongoDB 5.0	MongoDB 6.0、5.0
  MongoDB 4.4	MongoDB 6.0、5.0、4.4
  MongoDB 4.2	MongoDB 6.0、5.0、4.4、4.2
  MongoDB 4.0	MongoDB 6.0、5.0、4.4、4.2、4.0
  MongoDB 3.6	MongoDB 6.0、5.0、4.4、4.2、4.0、3.6

• You must have read access to the source database and read-write access to the target data source.

• If you need to use incremental replication, the oplog must be enabled for the source data source. If your deployment architecture is standalone, you need to specify the --replSet <replicaSetName> option when starting the mongod process to start the database in replica set mode. For example: mongod --replSet rs0. After starting, execute the rs.initiate() command in the MongoShell (MongoDB client) to initialize the replica set. MongoDB will automatically create the oplog.

Usage Limitations

Before performing data synchronization, evaluate the performance of the source data source and target data source, and it is recommended to perform data synchronization during low traffic periods. Otherwise, during full data initialization, certain read-write resources of the source and target data sources will be occupied, causing an increase in database load.

Procedure

1. Log in to the NineData console.

2. Click on > in the left navigation bar.

3. On the page, click on in the upper right corner.

4. On the tab, configure the parameters as per the table below, and click .

   Parameter	Description
   	Enter the name of the data synchronization task. For easy identification and management, please use a meaningful name. Supports up to 64 characters.
   	The MongoDB data source where the synchronization object is located.
   	The MongoDB data source that will receive the synchronized objects.
   	: Only synchronize the database and table structure of the source data source. : Synchronize all objects and data of the source data source, i.e., full data replication. : After full synchronization, perform incremental synchronization based on the logs of the source data source. Note: You can also click to expand and choose the handling strategy for cases where objects or data with the same name exist.
   	(select ): : Stop the task if a table with the same name is detected during the pre-check phase. : Send a prompt and continue the task if a table with the same name is detected during the pre-check phase. When copying the structure, ignore the table with the same name. If you are also copying data, the data will be appended to the table with the same name without overwriting the existing data. : Send a prompt and continue the task if a table with the same name is detected during the pre-check phase. When copying the structure, delete the table with the same name in the target database and copy the table structure based on the source database. If you are also copying data, the data will be written after the table structure has been copied. (optional when copying both structure and data): Send a prompt and continue the task if a table with the same name is detected during the pre-check phase. When copying the structure, retain the table structure in the target database and clear the data in the table with the same name when data copying begins, then re-copy from the original table. (required when is not selected): : Stop the task if data exists in the target table is detected during the pre-check phase. : Ignore the detected data in the target table during the pre-check phase and append other data. : Delete the detected data in the target table during the pre-check phase and rewrite.

5. On the tab, configure the parameters as per the table below, and then click .

   Parameter	Description
   	Select the content to be copied. You can choose to copy all content from the source database, or to select the content to be copied from the left list and click > to add it to the right list.

6. On the tab, configure the names of the collections in the target data source after synchronization from the source library, and click .

   tip

   In general, you do not need to change the configuration in the tab and can keep it in its default state.

7. On the tab, perform different operations based on the selected replication type.

   • Including : Configure the names of the database and collections (Collections) in the target data source after synchronization from the source data source, and click .
   • Excluding : The target database with the same name is automatically selected by the system. If it does not exist, you need to manually select the target database. The collection names in the target database need to match the synchronization object. If they do not match, you can manually map the collection names.

8. On the tab, wait for the system to complete the pre-check. After the pre-check passes, click .

   tip

   • If the pre-check fails, you need to click on the right side of the target check item to identify the reason for the failure, manually fix it, and click to re-run the pre-check until it passes.
   • For check items with as , you can repair or ignore them based on the specific situation.

9. On the page, when is displayed, the synchronization task starts running. At this point, you can:

   • Click to view the execution status of each stage of the synchronization task.
   • Click to return to the list.

Viewing Synchronization Results

1. Log in to the NineData console.

2. Click on > in the left navigation bar.

3. On the page, click on the target synchronization task's , and the page is as follows.

   

   Number	Function	Description
   1	Synchronization Delay	The data synchronization delay between the source data source and the target data source. 0 seconds means there is no delay between the two ends, and you can choose to switch the business to the target data source for smooth migration.
   2	Configure Alerts	After configuring alerts, the system will notify you in the way you choose when the task fails. For more information, please see Operations and Monitoring Introduction.
   3	More	Pause: Pause the task, only tasks with the status Running can be selected. Terminate: End tasks that are incomplete or listening (i.e., incremental synchronization is in progress). After terminating the task, it cannot be restarted, so please proceed with caution. If the synchronization objects include triggers, a trigger copy option will pop up, please choose as needed. Delete: Delete the task. After the task is deleted, it cannot be recovered, so please proceed with caution.
   4	Structure Replication (displayed in scenarios involving structure replication)	Display the progress and detailed information of structure replication. Click Logs on the right side of the page: View the execution logs of structure replication. Click on the right side of the page: View the latest information. Click View DDL in the Operation column on the right side of the target object in the list: View SQL playback.
   5	Full Data Replication (displayed in scenarios involving full data replication)	Display the progress and detailed information of full data replication. Click Monitoring on the right side of the page: View various monitoring indicators during full data replication. During full data replication, you can also click Flow Control Settings on the right side of the monitoring indicators page to limit the rate of writing to the target data source per second. Click Logs on the right side of the page: View the execution logs of full data replication. Click on the right side of the page: View the latest information.
   6	Incremental Replication (displayed in scenarios involving incremental replication)	Display various monitoring indicators for incremental replication. Click on the right side of the page: View the operations currently being executed by the current replication task, including: : The replication task is executed in multiple threads, showing the current thread number that is in progress. : Details of the statement currently being executed by the current thread. : The response time of the current thread. If this value increases, it may indicate that the current thread is stuck for some reason. : The timestamp when the current thread was started. : The status of the current thread. Click Flow Control Settings on the right side of the page: Limit the rate of writing to the target data source per second. The unit is rows/second. Click Logs on the right side of the page: View the execution logs of incremental replication. Click on the right side of the page: View the latest information.
   7	Modify Object	Display the modification records of the synchronization object. Click Modify Synchronization Object on the right side of the page to configure the synchronization object. Click on the right side of the page: View the latest information.
   8	Comparison	Display the results of the data comparison between the source data source and the target data source. If you haven't enabled data comparison, click Enable Data Comparison on the page. Click Recompare on the right side of the page: Initiate a new comparison of the data between the current source and target. Click Logs on the right side of the page: View the execution logs of the consistency comparison. Click Monitoring (visible only during data comparison): View the trend chart of RPS (records compared per second). Click Details to view earlier records. Click in the Operation column on the right side of the comparison list (visible in case of inconsistencies): View the comparison details between the source and target. Click in the Operation column on the right side of the comparison list (visible in case of inconsistencies): Generate change SQL. You can copy this SQL directly to the target data source for execution and modify the inconsistent content.
   9	Expand	Display detailed information about the current replication task, including Replication Type, Replication Object, Start Time, etc.

Appendix: Pre-Check Item Checklist

Check Item	Check Content
Target Database Same Name Object Existence Check	Check if the objects to be replicated already exist in the target database
Source Data Source Connection Check	Check the status of the source data source gateway, instance accessibility, and accuracy of username and password
Target Data Source Connection Check	Check the status of the target data source gateway, instance accessibility, and accuracy of username and password
Target Database Permission Check	Check if the account permissions in the target database meet the requirements
Source Database Permission Check	Check if the account permissions in the source database meet the requirements
Target Database Data Existence Check	Check if data exists for the objects to be replicated in the target database
Check if Source Database is Enabled with oplog	Check if the source database is enabled with oplog

Related Documents

Data Replication Introduction