Data Replication between Redis Instances

NineData Data Replication supports data replication between Redis data sources.

Background Information

Redis database has become a very popular NOSQL database due to its ease of use and high performance. In certain business scenarios, you may need to replicate and synchronize data between multiple Redis instances. In practical use, Redis data replication is usually used in the following scenarios:

• Data backup: The Redis instance running on the master node contains important data. If this data is lost or damaged in certain situations, it may have a serious impact on the business. To protect this data, Redis data replication can be used to replicate the data on the master node to the backup node, so that it can be recovered in case of failure or data loss.
• Load balancing: Using multiple Redis instances for load balancing can help improve system performance and reliability. Using Redis data replication can ensure data consistency between multiple instances, thereby avoiding errors caused by inconsistent data.
• Fault recovery: When the Redis master node fails, one of the slave nodes can be upgraded to the master node, and data consistency can be ensured through Redis data replication. This method can quickly restore normal operation of the system and reduce the impact on the business.

Prerequisites

• Redis version is one of the following: 7.0, 6.0, 5.0, 4.0, 3.2, 3.0
• Redis architecture is as follows:
  • Source data source: standalone, master-slave, read-write separation
  • Target data source: standalone, master-slave, cluster

Operation Steps

Commercialization Notice

NineData’s data replication product has been commercialized. You can still use 10 replication tasks for free, with the following considerations:

• Among the 10 replication tasks, you can include 1 task, with a specification of Micro.

• Tasks with a status of do not count towards the 10-task limit. If you have already created 10 replication tasks and want to create more, you can terminate previous replication tasks and then create new ones.

• When creating replication tasks, you can only select the you have purchased. Specifications that have not been purchased will be grayed out and cannot be selected. If you need to purchase additional specifications, please contact us through the customer service icon at the bottom right of the page.

1. Log in to the NineData console.

2. Click > on the left navigation bar.

3. On the page, click in the upper right corner.

4. On the tab, configure according to the following table, and click .

   Parameter	Description
   	Enter the name of the data synchronization task. To facilitate subsequent searching and management, please use a meaningful name as much as possible. Up to 64 characters are supported.
   	The Redis data source where the synchronization object is located.
   	The Redis data source that receives the synchronization object.
   	: Synchronize all objects and data of the source data source, that is, full data replication. : After the full synchronization is completed, perform incremental synchronization based on the log of the source data source. Note: In the data replication scenario from Redis to Redis, and are selected by default and cannot be canceled.
   	Select the conflict resolution policy when is selected: : Stop the task when the same named library is detected during the pre-check stage. : Send a prompt when the same named library is detected during the pre-check stage and continue the task. Append new data based on the original data during the data replication process without overwriting the original data. : Send a prompt when the same named library is detected during the pre-check stage and continue the task. Clear the data in the target same named library during the data replication process and write it again based on it.

5. On the tab, configure the following parameters, and then click .

   Parameter	Description
   	Select the content that needs to be copied. You can choose to copy all content in the source library, or choose , select the content to be copied in the left list, and click > to add it to the right list.

6. On the tab, configure the name of the library after the source library is synchronized to the target data source, and click .

   tip

   In general, you do not need to change the configuration on the tab and keep the default status.

7. On the tab, wait for the system to complete the pre-check. After the pre-check is passed, click .

   tip

   • You can select . After the synchronization task is completed, automatic data consistency comparison based on the source data source is started to ensure that the data on both sides is consistent. It is started when the incremental data is consistent with the source data source for the first time and is 0. You can view on the tab on the task details page.

     

   • If the pre-check fails, you need to click the in the column on the right side of the target check item to troubleshoot the reason for the failure, manually fix it, and then click to execute the pre-check again until it passes.

   • Check items with in can be repaired or ignored depending on the specific situation.

8. On the page, when prompted with , the synchronization task starts running. At this point, you can perform the following operations:

   • Click to view the execution of each stage of the synchronization task.
   • Click to return to the list.

View Synchronization Results

1. Log in to the NineData console.

2. Click > on the left navigation bar.

3. On the page, click the ID of the target synchronization task to open the page. The page is explained as follows.

   

   Number	Function	Explanation
   1	Configure alert	After configuring the alert, the system will notify you through the method you choose when a task fails. For more information, please see Operation and maintenance monitoring introduction.
   2	More	: Pause the task, only tasks with status are available. : End the unfinished or monitored (incrementally synchronized) tasks. After terminating the task, it cannot be restarted, please proceed with caution. If the synchronization object contains a trigger, a trigger copy option will pop up, please choose as needed. : Delete the task. After the task is deleted, it cannot be restored, please proceed with caution.
   3	Full replication	Displays the progress and detailed information of full replication. Click on the right side of the page: View various monitoring indicators during full replication. During full replication, you can also clickon the right side of the monitoring indicator page to limit the rate at which data is written to the target data source. The unit is MB/s. Clickon the right side of the page: View the execution log of full replication. Clickicon on the right side of the page: View the latest information.
   4	Incremental replication	Displays various monitoring indicators of incremental replication. Clickon the right side of the page: Limit the rate at which data is written to the target data source. The unit is rows/second. Clickon the right side of the page: View the execution log of incremental replication. Clickicon on the right side of the page: View the latest information.
   5	Data comparison	Display the results of data comparison between the source and target data sources. If you have not enabled data comparison, click on the page. Click on the right side of the page to initiate a new comparison between the current source and target data. Click on the right side of the page to view the execution logs of consistency comparison. Click Monitoring on the right side of the page to view the trend chart of the comparison RPS (records per second). Click Details to view earlier records. Click in the column on the right side of the comparison list (displayed when there is inconsistency) to view the comparison details of the source and target sides. On this page, you can also: Click the icon on the right side of specific inconsistent results to view more detailed comparison details. Click the icon on the right side of specific inconsistent results to generate change commands based on Key. You can directly copy this command to the target data source for execution and modify the inconsistent content. Click to export the correct source data in JSON format for easy data correction. Click the icon in the upper right corner of the list to customize the display of columns. Click in the column on the right side of the comparison list (displayed when there is inconsistency) to generate change SQL. You can directly copy this SQL to the target data source for execution and modify the inconsistent content.
   6	Expand	Display detailed information about the current replication task, including , , , etc.

Appendix: Commands Supported by Incremental Replication

During incremental replication, only Redis data change commands supported by NineData Data Replication will be synchronized to the target end after being executed on the source end in real time. This section introduces the Redis data types and data change commands supported by NineData Data Replication.

Supported Data Types	Supported Commands
Strings	APPEND、DECR、DECRBY、GETDEL（6.2.0 版本以上）、GETEX（6.2.0 版本以上）、GETSET、INCR、INCRBY、INCRBYFLOAT、MSET、MSETNX、PSETEX、SET、SETEX、SETNX、SETRANGE
Lists	BLMOVE、BLMPOP、BLPOP、BRPOP、BRPOPLPUSH、LINSERT、LMOVE、LMPOP、LPOP、LPUSH、LPUSHX、LREM、LSET、RPOP、RPOPLPUSH、RPUSH、RPUSHX
Sets	SADD、SDIFFSTORE、SINTERSTORE、SMOVE、SPOP、SREM、SUNIONSTORE
Hashes	HDEL、HINCRBY、HINCRBYFLOAT、HMSET、HSET、HSETNX、
Sorted Sets	BZMPOP、BZPOPMAX、BZPOPMIN、ZADD、ZDIFFSTORE、ZINCRBY、ZINTERSTORE、ZMPOP、ZPOPMAX、ZPOPMIN、ZRANGESTORE、ZREM、ZREMRANGEBYLEX、ZREMRANGEBYSCORE、ZUNIONSTORE
Streams	XADD、XTRIM、XGROUP、 XAUTOCLAIM、XDELXREADGROUP
Geospacial	GEOADD、GEOSEARCHSTORE
HyperLogLog	PFADD
Bitmaps	SETBIT、BITFIELD、BITOP

Appendix 2: Checklist of Pre-Check Items

Check Item	Check Content
Target Database Data Existence Check	Check if data exists for the objects to be replicated in the target database
Source Instance Log Buffer Size Check	Check the size configuration of the source instance parameter repl-backlog-size
Check Source Instance Client Output Buffer Size	Check the size configuration of the source instance parameter client-output-buffer-limit replica
Source Data Source Connection Check	Check the status of the source data source gateway, instance accessibility, and accuracy of username and password
Target Data Source Connection Check	Check the status of the target data source gateway, instance accessibility, and accuracy of username and password
Source Database Permission Check	Check if the account permissions in the source database meet the requirements