Skip to main content

MongoDB to MongoDB Replication

Use this workflow to replicate data between MongoDB instances in NineData. It supports schema replication, full data replication, and incremental replication.

Before you begin​

  • Add both data sources to NineData. For details, see Add Data Source.

  • Use a MongoDB standalone deployment or replica set deployment.

  • Confirm the supported source and target versions. The source MongoDB version must be earlier than or equal to the target MongoDB version.

    Source DatasourceTarget Datasource
    MongoDB 6.0MongoDB 6.0
    MongoDB 5.0MongoDB 6.0, 5.0
    MongoDB 4.4MongoDB 6.0, 5.0, 4.4
    MongoDB 4.2MongoDB 6.0, 5.0, 4.4, 4.2
    MongoDB 4.0MongoDB 6.0, 5.0, 4.4, 4.2, 4.0
    MongoDB 3.6MongoDB 6.0, 5.0, 4.4, 4.2, 4.0, 3.6

  • Make sure the source account has read access and the target account has read and write access.

  • For incremental replication, enable oplog on the source data source. If the source is a standalone deployment, specify --replSet <replicaSetName> when you start the mongod process so the database starts in replica set mode. For example: mongod --replSet rs0. After startup, run rs.initiate() in MongoShell to initialize the replica set. MongoDB then creates the oplog.

Limitations​

Check source and target performance before you start. Run replication during off-peak hours when possible because full initialization reads and writes both data sources and can increase load.

Procedure​

  1. Sign in to the NineData Console.

  2. Click Replication > Data Replication in the left navigation.

  3. On the Replication page, click Create Replication.

  4. On the Source & Target tab, set the following fields, then click Next.

    Parameter
    		Description
    NameEnter a task name that is easy to recognize later. Up to 64 characters are supported.
    SourceSelect the MongoDB source data source.
    TargetSelect the MongoDB target data source.
    Type
    • Schema: Copy only the database and collection structure from the source data source.
    • Full: Copy all objects and data from the source. This is full data replication.
    • Incremental: After full synchronization finishes, sync changes from the source logs.
    Note: Expand Target Database Preparation to choose how to handle objects or data with the same name.
    Target Database Preparation
    • If target table already exists (select Schema):
      • Pre-Check Error and Stop Task: Stop the task if the pre-check finds a collection with the same name.
      • Skip and Continue Task: Show a message and continue the task if the pre-check finds a collection with the same name. During schema replication, skip the collection with the same name. If data replication is also enabled, append data to the existing collection without overwriting it.
      • Delete Objects and Rewrite: Show a message and continue the task if the pre-check finds a collection with the same name. During schema replication, delete the collection with the same name in the target database and copy the structure from the source again. If data replication is also enabled, write data after the structure is copied.
      • Keep Schema and Clear Data, then Rewrite (optional when both schema and data replication are enabled): Show a message and continue the task if the pre-check finds a collection with the same name. Keep the target collection structure, clear its data when data replication starts, and then copy data from the source again.
    • Target Table Exists Data (required when Schema is not selected):
      • Pre-Check Error and Stop Task: Stop the task if the pre-check finds existing data in the target collection.
      • Ignore existing target data and append to it.: Ignore the existing data in the target collection and append other data.
      • Clear target existing data before write: Delete the existing data in the target collection and rewrite it.

  5. On the Objects tab, set the following fields, then click Next.

    Parameter
    		Description
    Replication ObjectsSelect the objects to replicate. Choose All Objects to replicate everything, or choose Customized Object, select objects from the left list, and click > to move them to the right list.

  6. On the Mapping tab, review the target collection names, then click Save and Pre-Check.

    tip

    In most cases, keep the default configuration on the Mapping tab.

  7. On the Mapping tab, choose the operation based on the selected replication type.

    • Includes Schema: Configure the target database and collection names, then click Save and Pre-Check.
    • Does not include Schema: NineData selects the same-named database in the target data source by default. If it does not exist, select the target database manually. The target collection names must match the replication objects. If they do not match, map the collection names manually.

  8. On the Pre-check tab, wait for NineData to complete the pre-check. After the pre-check passes, click Launch.

    tip
    • If the pre-check fails, click Details for the failed check item, fix the issue, and click Check Again until the pre-check passes.
    • For items whose Result is Warning, fix the issue or ignore the warning when appropriate.

  9. On the Launch page, after Launch Successfully appears, the replication task starts running. Use these actions to review the task:

    • Click View Details to view the status of each replication phase.
    • Click Back to list to return to the Replication list.

Results​

  1. Sign in to the NineData Console.

  2. Click Replication > Data Replication in the left navigation.

  3. On the Replication page, click the target replication task's Task ID. Use the following areas to monitor the task.

    mongo_result

    Number
    		Function
    		Description
    1Synchronization DelayThe delay between the source and target data sources. A value of 0 seconds means there is no delay. At that point, switch business traffic to the target data source for a smooth cutover.
    2Configure AlertsConfigure alerts to get task-failure notifications by the method you choose. For more information, see Introduction to operational monitoring.
    3More
    • Pause: Pause a task whose status is Running.
    • Terminate: Terminate a task that is unfinished or still monitoring incremental changes. A terminated task cannot be restarted. Proceed with caution. If the synchronization objects include triggers, trigger copy options appear. Choose the option that matches your task.
    • Delete: Delete the task. Deleted tasks cannot be recovered. Proceed with caution.
    4Structure Replication (shown when structure replication is included)Shows the progress and details of structure replication.
    • Click Logs to view structure replication logs.
    • Click refresh to view the latest information.
    • Click View DDL in the Operation column for the target object to view the SQL replay.
    5Full Data Replication (shown when full data replication is included)Shows the progress and details of full data replication.
    • Click Monitoring to view full data replication metrics. During full data replication, click Flow Control Settings to limit the write rate to the target data source. The unit is rows per second.
    • Click Logs to view full data replication logs.
    • Click refresh to view the latest information.
    6Incremental Replication (shown when incremental replication is included)Shows incremental replication metrics.
    • Click View Threads to view the operations currently running in the replication task, including:
      • Thread ID: The replication task runs in multiple threads. This shows the current thread number.
      • Execute SQL: Details of the statement the thread is currently running.
      • Response Time: The current thread response time. If this value increases, the thread may be stuck.
      • Event Time: The timestamp when the current thread started.
      • Status: The current thread status.
    • Click Flow Control Settings to limit the write rate to the target data source. The unit is rows per second.
    • Click Logs to view incremental replication logs.
    • Click refresh to view the latest information.
    7Modify ObjectShows change records for synchronization objects.
    • Click Modify Synchronization Object to update the synchronization object.
    • Click refresh to view the latest information.
    8ComparisonShows the comparison results between the source and target data sources. If data comparison is not enabled, click Enable Data Comparison on the page.
    • Click Recompare to compare the current source and target data again.
    • Click Logs to view consistency comparison logs.
    • Click Monitoring (visible only during data comparison) to view the RPS (records compared per second) trend chart. Click Details to view earlier records.
    • Click details in the Operation column of the comparison list (shown when there are inconsistencies) to view the detailed comparison between source and target.
    • Click sql in the Operation column of the comparison list (shown when there are inconsistencies) to generate a change SQL statement. Copy it to the target data source and run it to fix the inconsistency.
    9ExpandShows detailed information about the current replication task, including Replication Type, Replication Object, Start Time, and more.

Appendix: Pre-check items​

Check ItemCheck Content
Target Database Same Name Object Existence CheckCheck whether objects with the same names already exist in the target database
Source Data Source Connection CheckCheck the source gateway status, instance accessibility, and username and password accuracy
Target Data Source Connection CheckCheck the target gateway status, instance accessibility, and username and password accuracy
Target Database Permission CheckCheck whether the target database account permissions meet the requirements
Source Database Permission CheckCheck whether the source database account permissions meet the requirements
Target Database Data Existence CheckCheck whether data already exists for the objects to be replicated in the target database
Source Database oplog Enabled CheckCheck whether oplog is enabled for the source database

Introduction to Data Replication