Periodic Full Replication

NineData's data replication feature supports periodic full replication between two data sources, automatically replicating all data from the source end to the target end according to a predefined schedule.

Background Information

To effectively protect the core data of an enterprise, it is common for businesses to implement robust backup and synchronization solutions, ensuring that data from core business databases is replicated entirely to target systems. As business databases are continuously updated, businesses need to execute replication tasks on a regular cycle based on configured intervals, such as daily, hourly, or even every ten minutes, to keep the data in the target system up to date.

Prerequisites

• The data sources participating in periodic full replication have been added to NineData. For instructions on how to add data sources, please refer to Adding Data Sources.
• The target replication link supports full replication. For a list of supported replication links, please refer to Data Replication.

Procedure

This article uses MySQL to MySQL replication link as an example to illustrate the configuration method for periodic full replication.

1. Log in to the NineData Console.

2. In the left navigation pane, click Replication > Data Replication.

3. On the Replication page, click Create Replication.

4. On the Source & Target tab, configure as per the table below, and click Next.

   Parameter	Description
   Name	Enter a name for the data synchronization task. For ease of subsequent retrieval and management, please use a meaningful name. Maximum 64 characters supported.
   Source	The data source where the synchronization object resides.
   Target	The data source receiving the synchronized object.
   Replication Mode	Select Uni-directional.
   Type	Check Schema and Full, then click the slider switch on the right of Full to enable periodic full replication. Note: Since periodic full replication cannot run simultaneously with Incremental, do not check Incremental.
   Frequency	The execution frequency of periodic full replication, supporting Once a Day, Every Hour, and Every 10 minutes. Please select according to actual business needs.
   Launch Time	Select the start time point of periodic full replication. You can choose the time zone according to the region of your business needs.
   Data Processing Strategy	Choose the data processing strategy in the target data source before each full replication starts. Drop Table, Recreate, and Rewrite Data (optional when Schema is selected): Deletes (DROP TABLE) the target table objects in the target data source, then rebuilds and writes data based on the structure of the source table. Clear Existing Data and Rewrite (optional when Schema is not selected): Clears (TRUNCATE TABLE) all data in the target table in the target data source, then writes based on the data of the source table. Ignore Existing Data and Overwrite: Directly overwrites (REPLACE) the data of the target table based on the data of the source table.

5. On the Objects tab, configure the following parameters, then click Next.

   Parameter	Description
   Replication Objects	Select the content to be copied. You can choose All Objects to copy all content of the source database, or choose Customized Object, select the content to be copied in the Source Object list, and click > to add it to the right Target Object list.
   Blacklist (optional)	Click Add to add a blacklist record, select the database or object to be added to the blacklist, and these contents will not be copied. Used to exclude certain databases or objects from full replication of Customized Object or replication of Full Instance. Left dropdown: Select the name of the database to be added to the blacklist. Right dropdown: Select the objects in the corresponding database. You can click on multiple objects for multiple selections, leave it blank to add the entire database to the blacklist. If you want to add multiple databases to the blacklist, you can click the Add button below to add a row.

6. On the Mapping tab, select different operations according to the selected replication type.

   • Including Schema: Configure the table name after synchronization to the target data source, and click Save and Pre-Check.

     tip

     You can click on the Mapping & Filtering on the right side of the page to customize the column names after synchronization to the target data source. In addition, you can set Data Filter to synchronize only the data that meets the filtering conditions to the target data source. Taking the test data table employees as an example, if you set the filter condition to emp_no>=10005, then data with emp_no less than 10005 will not be synchronized to the target data source.

   • Excluding Schema: The system defaults to selecting the same database in the target data source. If it does not exist, you need to manually select the target database. The table names and column names in the target database need to be consistent with the synchronized objects. If they are inconsistent, you can manually map the table names and column names.

7. On the Pre-check tab, wait for the system to complete the pre-check. After the pre-check passes, click Launch.

   tip

   • You can check Enable data consistency comparison. After the synchronization task is completed, automatically start data consistency comparison based on the source data source to ensure data consistency on both ends. Depending on the selected Type, the timing of the startup of Enable data consistency comparison is as follows:
     • Schema: Starts after the structure replication is completed.
     • Schema + Full, Full: Starts after full replication is completed.
   • If the pre-check fails, click the Details on the right side of the target check item, troubleshoot the cause of the failure, manually fix it, and click Check Again to perform the pre-check again until it passes.
   • For check items with Result as Warning, you can fix or ignore them according to the specific situation.

8. On the Launch page, when prompted with Launch Successfully, the synchronization task starts running. At this point, you can:

   • Click View Details to view the execution status of each stage of the synchronization task.
   • Click Back to list to return to the Replication task list page.