Skip to main content

Master Node Replacement

This document describes how to replace a failed Master node in a 3-Master high-availability cluster when the failed node cannot be recovered.

Run this procedure during a maintenance window, and always follow the order of removing the old node first and adding the new node afterward.

Step 1: Confirm Preconditions for Replacement​

  1. Connect to a healthy Master node and confirm the current node status:

    kubectl get nodes -o wide

  2. Confirm the NineData service container name and status:

    docker ps --format "table {{.Names}}\t{{.Status}}"

  3. Record the following information:

    • The IP address of the failed Master node.
    • The IP address of any healthy Master node.
    • The IP address of the new server.
    Note

    This document applies to clusters that are already deployed in a 3-Master architecture. If the current environment has only 1 Master node, do not perform this replacement procedure directly.

Step 2: Get the Token Required for the New Node to Join​

  1. Run the following command on a healthy Master node:

    docker exec -i ninedata-server cat /var/lib/rancher/k3s/server/token

  2. Save the returned token. You will use it when editing config.properties.

    Notes

    The default container name is ninedata-server. If your environment uses a different container name, replace it with the actual one.

Step 3: Prepare the install_source Directory​

  1. On the healthy Master node, confirm the location of the installation directory:

    find /root -maxdepth 3 -type d -name install_source

  2. Copy the installation directory to the new server. Example:

    scp -r <install_source_directory> root@<NEW_SERVER_IP>:/root/

  3. Connect to the new server and enter the directory. Example:

    cd /root/install_source
    Notes

    The actual install_source path depends on where the installation package was extracted, for example /root/niup_ins_xxx/install_source. If the directory name is different after copying, use the real path.

Step 4: Update the New Node Configuration​

  1. In the install_source directory on the new server, run:

    grep -nE 'k3s_token|first_master_ip|hostname' config.properties

  2. Update config.properties based on the actual environment. Pay special attention to:

    • k3s_token: set it to the token obtained in step 2.
    • first_master_ip: set it to the IP address of any healthy Master node.
    • hostname: set it to the hostname or IP address of the new node if needed.

  3. If you want to edit the file directly, run:

    vi config.properties
    Notes

    If hostname is left empty, the installation script uses the local IP of the current host by default.

Step 5: Stop the Failed Node and Remove It from the Cluster​

  1. Connect to the failed Master node and run:

    docker stop ninedata-server

  2. Return to a healthy Master node and confirm that the failed node becomes NotReady:

    kubectl get nodes

  3. Delete the failed node:

    kubectl delete node <FAILED_NODE_IP>
    Note

    The old node must be removed from the cluster before the new node is added. Otherwise, the etcd member count and quorum increase first, and the new node may fail to join.

Step 6: Add the New Master Node​

  1. Connect to the new server and enter the install_source directory.

  2. Run the following command:

    sh add.sh

  3. Keep the terminal connected until the installation finishes.

    Notes

    To specify a custom data directory or hostname, pass script parameters, for example sh add.sh --data-dir=/opt/ninedata --hostname=<NEW_HOSTNAME>.

Step 7: Verify the Replacement Result​

  1. Return to a healthy Master node and run:

    kubectl get nodes -o wide

  2. If you also want to confirm the service status, run:

    kubectl get pods -A

  3. Confirm that the new Master node has joined the cluster and that key services are back in the Running state.

Common Error and Root Cause​

If you add the new node before deleting the old one, you may see the following error:

Waiting for other members to finish joining etcd cluster: etcdserver: unhealthy cluster

Root cause:

  • In the original 3-node etcd cluster, the quorum is 2.
  • If old node A has failed but its etcd member record still exists, only nodes B and C are actually alive.
  • If new node D is added at this time, the cluster size becomes 4, and the quorum increases to 3.
  • Because only B and C are healthy, 2 < 3, so the cluster is considered unhealthy and the new node is rejected.

Therefore, you must run kubectl delete node <FAILED_NODE_IP> first to remove the failed node, restore the cluster to a healthy 2/2 state, and then add the new Master node.