Skip to main content

Manage datasources

NineData supports adding data sources of various types in multiple environments with unified management console, providing Database DevOps, backup and recovery, data replication, database comparison and other functions for the data sources.

Prerequisite

  • Supported cloud vendors: Alibaba Cloud, Tencent Cloud, Huawei Cloud, AWS, China Mobile Cloud, Baidu Cloud

  • Supported data source: MySQL, SQL Server, PostgreSQL, KingBase, Oracle, Dameng, OceanBase Oracle, OceanBase MySQL, DB2, Redis, ClickHouse, MongoDB, Kafka, Elasticsearch, Doris, SelectDB, Greenplum, Redshift (AWS instance), GaussDB (Huawei Cloud Instance), DWS (Huawei Cloud Instance), openGauss, StarRocks, SingleStore, Sybase, TiDB, Hive

  • To ensure smooth connectivity of the service, please first select the option on the page. Then, add the NineData service IP address located at the position in the diagram below to the allowlist of your server. For instructions on how to add a allowlist in a self-built database in the IDC, refer to Appendix 2. For adding a allowlist in cloud provider servers or databases, please refer to the official documentation of each respective cloud provider.

    server_ip_address

  • Please make sure you still have available data source quota, otherwise, you won't be able to input a new data source. You can quickly check the remaining quota on the NineData console page in the upper right corner.

    check_quota

Adding Self-Built Data Source

  1. Log in to the NineData Console.

  2. On the left navigation pane, click > .

  3. Click  tab, and click  on the page. In the popup window for selecting the data source type, choose > (the type of data source to be added), and configure the parameters based on the table below on the page.
    tip

    If you make a mistake during the operation, you can click the arrow_down icon at the top of the page to make a new selection.

Relational Databases
Parameter
Description
Enter a name for the data source to facilitate future searching and management.
Choose the connection method for the data source. Options include , , and SSH Tunnel.
  • : Connect using a public network address.
  • : Use NineData's secure and convenient intranet access method. You need to establish a connection to the host where the data source is located using the method described in adding a gateway.
  • SSH Tunnel: Connect through an SSH tunnel.
Configuration under for : Enter the public network connection address and port of the data source.
Configuration under for
  • : Choose the NineData gateway installed on the host where the data source is located.
  • : You can use localhost (if the data source is on the same machine) or the internal IP of the data source's host.
Configuration under for SSH Tunnel
  • : Enter the public IP or domain name of the target server where the data source is located, along with the corresponding port number (default SSH port is 22).
  • : Choose the authentication method for SSH.
    • : Connect using (server's login name) and (server's login password).
      • : Enter the login username for the target server where the data source is located.
      • : Enter the login password for the target server where the data source is located.
    • (recommended): Connect using and .
      • : Enter the login username for the target server where the data source is located.
      • : Click on to upload the private key file, which is the key file without an extension. If you haven't created it yet, see Generate SSH Tunnel Key File.
      • : Enter the passphrase you set when generating the key file. If you didn't set a passphrase during key generation, leave this field blank.
  • After configuring SSH, click on the right side of . You might get either of the following results:
    • Success message : Indicates a successful SSH tunnel connection.
    • Error message: Indicates a connection failure. Troubleshoot the issue based on the error message and retry.
  • : You can use localhost (if the data source is on the same machine) or the internal IP of the data source's host.
Choose the deployment architecture of MySQL.
  • : MySQL database deployed on a standalone server, only require filling in one connection address and port.
  • : MySQL database deployed with disaster recovery architecture, typically consists of one primary node and one or more disaster recovery nodes. You can click the button below to input addresses and ports of multiple nodes.
  • : MySQL database deployed with read-write splitting architecture, typically consists of one write node (master node) and multiple read nodes (slave nodes). You can click the button below to input addresses and ports of multiple nodes.
Username for the data source.
Password for the data source.
Choose the region closest to the location of your data source to reduce network latency.
Select the actual business purpose of this data source as the environmental identifier. Default options include and , and you can also create custom environments.
Note: In organization mode, database environments can also be applied to permission policy management. For example, the default role only supports accessing data sources in the environment and cannot access data sources in other environments. For more information, see manage roles.
Whether to access the data source using SSL encryption (default enabled). If the data source enforces SSL encrypted connection, this switch must be turned on, otherwise the connection will fail.
Click on the switch on the right to enable or disable encrypted transmission. Click on the > on the left of to expand detailed configurations.
  • : Supports the following two options.
    • : Automatically detects the SSL status of the server. If SSL is enabled on the server, it will prioritize an SSL connection. If SSL is not enabled on the server, it will use a non-SSL connection.
    • : Always use SSL to connect to the data source. If the server does not support this method or cannot establish an SSL connection for other reasons, the connection will fail.
  • : Specifies the SSL-Cipher encryption algorithm.
  • : If the MySQL server uses a self-signed CA-issued certificate, upload the root certificate of that CA here.
  • : Check to verify the server's name and IP address, etc., to ensure that you are connecting to the genuine MySQL server and to avoid Man-in-the-Middle Attacks.
  • : If the MySQL server requires the client to provide a certificate, upload the client's certificate and key here. The MySQL server will verify the information you uploaded to ensure the security of the connection.
Note: In most cases, if the MySQL server supports SSL encrypted connections, you only need to select , and no further configuration is needed. The system will automatically generate keys for you to establish a connection. For more SSL configuration methods, please refer to the official documentation: Configuring MySQL Encrypted Connections.
NoSQL Databases
ParameterDescription
Enter the name of the input data source. Please use a meaningful name for easy future lookup and management.
Select the deployment type of your Redis database.
  • : Standalone deployment.
  • : High availability solution based on master-slave replication. In sentinel mode, there are multiple Redis instances and several sentinel processes in the system. If the master node fails, a sentinel will automatically promote one of the slave nodes to be the new master, ensuring high availability of the system.
  • : Data is distributed across multiple nodes in the cluster, with automatic data sharding and migration between nodes. Each node is responsible for processing a portion of the data, providing high performance and availability.
Choose the access method for the data source. Supports three methods: , , and SSH Tunnel.
  • : Access the data source via public IP address.
  • (not supported for and ): Secure and efficient intranet access method provided by NineData. The host of the data source needs to be connected. Refer to add gateway for the connection method.
  • SSH Tunnel (not supported for and ): Access the data source via SSH tunnel.
Configuration options for
    • : Public connection address and port of the data source.
    • : Specify the name of the master node to be monitored in the sentinel configuration using the master-name parameter.
    • : In sentinel mode, addresses and ports of the sentinel processes, which monitor the status of master and slave nodes. Sentinels take corresponding measures in case of failures or the need for failover. Click to add multiple sentinel nodes.
  • : In cluster mode, addresses and ports of all Redis nodes. Click to add multiple Redis nodes. Add all nodes here, as missing nodes may result in the inability to perform certain cluster-related operations.
Configuration options for
  • : Choose the NineData gateway installed on the host where the data source resides.
  • : Can be set as localhost (data source is on the local machine) or the intranet IP of the data source's host.
Configuration options for SSH Tunnel
  • : Enter the public IP or domain name of the target data source's server, along with the corresponding port number (default port for SSH service is 22).
  • : Choose the SSH authentication method.
    • : Connect using (server login name) and (server login password).
    • (recommended): Connect using and .
      • : Enter the login username of the target data source's server.
      • : Upload the private key file by clicking . This is the key file without an extension. If you haven't created it yet, refer to generate SSH Tunnel key file.
      • : Enter the passphrase set during key file generation. Leave it blank if no passphrase was set during key generation.
  • Note: After configuring SSH, you need to click the on the right. Two outcomes are possible:
    • prompt: Indicates the successful establishment of the SSH Tunnel.
    • Error message prompt: Indicates connection failure. Troubleshoot the issue based on the error message and retry.
  • : Can be set as localhost (data source is on the local machine) or the intranet IP of the data source's host.
Username of the data source. For some older versions of instances from certain cloud providers, there may be authentication issues. If you have confirmed that the account password is correct but still receive authentication failures, please leave this field blank. Instead, try entering the username and password in the <username>@<password> or <username>:<password> format (formats may vary for different cloud providers) in the field. For example: example_user@abcdefg or example_user:abcdefg.
Password of the data source.
Choose the region closest to the location of your data source to effectively reduce network latency.
Select the actual business purpose of this data source as the environmental identifier. Default options include and , and you can also create custom environments.
Note: In organization mode, database environments can also be applied to permission policy management. For example, the default role only supports accessing data sources in the environment and cannot access data sources in other environments. For more information, see manage roles.
Whether to use SSL encryption when accessing the data source (default is off). If the data source requires a mandatory SSL encrypted connection, this switch must be turned on; otherwise, the connection will fail.
Click the switch on the right to enable or disable encrypted transmission. Click the > on the left of to expand detailed configurations.
  • : Supports the following two options.
    • : Automatically detects the SSL status of the server. If the server has SSL enabled, it will prioritize an SSL connection. If the server does not have SSL enabled, it will use a non-SSL connection.
    • : Always use SSL to connect to the data source. If the server does not support this method or cannot establish an SSL connection for other reasons, the connection will fail.
  • : If the server uses a self-signed CA-issued certificate, upload the root certificate of the CA here.
  • : If the server requires the client to provide a certificate, upload the client's certificate and key here. The MySQL server will verify the information you upload to ensure the security of the connection.
For more SSL configuration methods, please refer to the official documentation: Configuring Redis Encrypted Connections.
Data Warehouse
ParameterDescription
Enter the name of the input data source. Please use a meaningful name for easy future lookup and management.
Choose the access method for the data source. Supports three methods: , , and SSH Tunnel.
  • : Access the data source via public IP address.
  • : Secure and efficient intranet access method provided by NineData. The host of the data source needs to be connected. Refer to add gateway for the connection method.
  • SSH Tunnel : Access the data source via SSH tunnel.
Configuration options for : Public connection address and port of the data source.
Configuration options for
  • : Choose the NineData gateway installed on the host where the data source resides.
  • : Can be set as localhost (data source is on the local machine) or the intranet IP of the data source's host.
Configuration options for SSH Tunnel
  • : Enter the public IP or domain name of the target data source's server, along with the corresponding port number (default port for SSH service is 22).
  • : Choose the SSH authentication method.
    • : Connect using (server login name) and (server login password).
    • (recommended): Connect using and .
      • : Enter the login username of the target data source's server.
      • : Upload the private key file by clicking . This is the key file without an extension. If you haven't created it yet, refer to generate SSH Tunnel key file.
      • : Enter the passphrase set during key file generation. Leave it blank if no passphrase was set during key generation.
  • Note: After configuring SSH, you need to click the on the right. Two outcomes are possible:
    • prompt: Indicates the successful establishment of the SSH Tunnel.
    • Error message prompt: Indicates connection failure. Troubleshoot the issue based on the error message and retry.
  • : Can be set as localhost (data source is on the local machine) or the intranet IP of the data source's host.
Username of the data source.
Password of the data source.
Choose the region closest to the location of your data source to effectively reduce network latency.
Select the actual business purpose of this data source as the environmental identifier. Default options include and , and you can also create custom environments.
Note: In organization mode, database environments can also be applied to permission policy management. For example, the default role only supports accessing data sources in the environment and cannot access data sources in other environments. For more information, see manage roles.
Whether to use SSL encryption to access the data source. If the data source enforces SSL encryption, you must enable this switch, otherwise the connection will fail. Refer to the official documentation for SSL configuration: Configure ClickHouse SSL Connection.
Other Databases
ParameterDescription
Enter the name of the input data source. Please use a meaningful name for easy future lookup and management.
Choose the access method for the data source. Supports three methods: , , and SSH Tunnel.
  • : Access the data source via public IP address.
  • : Secure and efficient intranet access method provided by NineData. The host of the data source needs to be connected. Refer to add gateway for the connection method.
  • SSH Tunnel : Access the data source via SSH tunnel.
Configuration options for Broker List: Kafka cluster composed of multiple broker connection addresses. Each broker is an independent Kafka server instance. Enter the public connection address and port number of the Kafka data source below. If there are multiple brokers, click to continue filling in.
Configuration options for
  • : Choose the NineData gateway installed on the host where the data source resides.
  • Broker List: Enter the intranet connection address and port number of the Kafka data source. Broker can be set as localhost (data source is on the local machine) or the intranet IP of the data source's host.
Configuration options for SSH Tunnel
  • : Enter the public IP or domain name of the target data source's server, along with the corresponding port number (default port for SSH service is 22).
  • : Choose the SSH authentication method.
    • : Connect using (server login name) and (server login password).
    • (recommended): Connect using and .
      • : Enter the login username of the target data source's server.
      • : Upload the private key file by clicking . This is the key file without an extension. If you haven't created it yet, refer to generate SSH Tunnel key file.
      • : Enter the passphrase set during key file generation. Leave it blank if no passphrase was set during key generation.
  • Note: After configuring SSH, you need to click the on the right. Two outcomes are possible:
    • prompt: Indicates the successful establishment of the SSH Tunnel.
    • Error message prompt: Indicates connection failure. Troubleshoot the issue based on the error message and retry.
  • Broker List: Enter the intranet connection address and port number of the Kafka data source. Broker can be set as localhost (data source is on the local machine) or the intranet IP of the data source's host.
Choose the authentication mode for Kafka.
  • PLAINTEXT: No authentication required, data is transmitted in plain text. Typically used in development and testing environments.
  • SASL/PLAIN (default): SASL-based authentication method that requires a username and password. Data is transmitted in plain text. Recommended for use on encrypted networks.
  • SASL/SCRAM-SHA-256: SASL-based authentication method that requires a username and password. Passwords are hashed using the SHA-256 algorithm for added security.
  • SASL/SCRAM-SHA-512: SASL-based authentication method that requires a username and password. Passwords are hashed using the SHA-512 algorithm for higher security.
Kafka username.
Kafka password.
Choose the region closest to the location of your data source to effectively reduce network latency.
Select the actual business purpose of this data source as the environmental identifier. Default options include and , and you can also create custom environments.
Note: In organization mode, database environments can also be applied to permission policy management. For example, the default role only supports accessing data sources in the environment and cannot access data sources in other environments. For more information, see manage roles.
Whether to access the data source using SSL encryption (default off). If the data source enforces SSL encrypted connections, this switch must be turned on, otherwise the connection will fail. Click on the > on the left of to expand the detailed configuration:
  • Trust Store: Trusted certificates issued by the CA organization, i.e., the server certificate. This certificate is used to verify the identity of the server, ensuring that the Kafka server you are connecting to is trusted.
  • Trust Store Password: Password protecting the Trust Store certificate.
  • Key Store: Certificate used to verify the user's identity, i.e., the client certificate. This certificate ensures that the user connecting to Kafka is a trusted user.
  • Key Store Password: Password protecting the Key Store certificate.
For SSL configuration methods, please refer to the official documentation: Configuring Kafka Encrypted Connections.
  1. Once all configurations are complete, click the button on the right side of to test whether the data source can be accessed properly. If the prompt shows , click to finish adding the data source. If not, please review the connection settings until the connection test succeeds.

Add cloud vendor data source

  1. Log in to the NineData Console.

  2. On the left navigation pane, click > .

  3. Click  tab, and click  on the page. In the popup window for selecting the data source type, choose (Cloud provider name) > (the type of data source to be added), and configure the parameters based on the table below on the page.
    tip

    If you make a mistake during the operation, you can click the arrow_down icon at the top of the page to make a new selection.

Supported database types include:

ParameterDescription
Enter the name of the input data source. Please use a meaningful name for easy future lookup and management.
Choose the access mode for the data source.
  • : Access via cloud provider's private network address. Suitable for scenarios with high transmission rate requirements. You need to connect this private network address to NineData, follow the add private network address guide and then select the connected option below.
  • : Access via public IP address of the instance. Enter the public connection address and port in .
  • (Self-built DB only): Secure and efficient intranet access method provided by NineData. The host of the data source needs to be connected. Refer to add gateway. Then select the connected and below. The can be set as localhost or the intranet IP.
  • SSH Tunnel: Access via SSH tunnel. You need to provide SSH-related information including , , and . Detailed configuration steps can be found in:
Choose the access method for Alibaba Cloud instances. Supports access via or .
  • : Access via ECS or Cloud Database instance ID. This option also requires selecting , , and parameters. needs to be configured in advance. Refer to configure cloud vendor access credentials for configuration details.
  • : Access via instance connection address. The format of depends on for different modes. Refer to the description above for details.
When the data source is MongoDB, it is necessary to input and specify the authentication database for the current user.
Data source username.
Data source password.
Only required when accessing via connection address. Choose the region closest to the location of your data source to effectively reduce network latency.
Select the actual business purpose of this data source as the environmental identifier. Default options include and , and you can also create custom environments.
Note: In organization mode, database environments can also be applied to permission policy management. For example, the default role only supports accessing data sources in the environment and cannot access data sources in other environments. For more information, see manage roles.
Choose whether to use SSL encryption for accessing the data source. If the data source requires SSL encryption, you must enable this option, otherwise the connection will fail.
Click the switch on the right to toggle encryption on or off. For some data source types, click on the left of to expand detailed configuration.
Note: Different data source types support different options, please refer to the console for accurate options.
  1. After completing all the configurations, you can click on the right side of and select to test if the data source can be accessed successfully. If you see indicating a successful connection test, you can then proceed to click on to finalize the addition of the data source. If the connection test does not succeed, please review your connection settings and keep testing until the connection is successfully established.

Appendix 1: Creating an Environment

To meet the needs of different enterprises, in addition to providing default and environments, NineData also supports manually creating custom environments.

  1. Log in to the NineData Console.

  2. In the left navigation bar, click >.

  3. Click on the tab. This shows all the environments currently under the user (personal mode) or organization (organization mode). Here you can or custom created environments.

  4. Click on in the upper right corner of the page.

  5. Complete the configuration according to the following table and click .

    ParameterDescription
    Enter the name of the environment, which can contain 1 to 16 characters. This name serves as the identifier of the environment and needs to have some distinguishing characteristics. Please try to use meaningful names.
    Choose the color of the environment label displayed in the console.

Appendix 2: Add NineData's IP address to the database allowlist (self-built database environment)

When adding a data source located at , configuring an allowlist is only required when adding it through public network addresses. Users who add the data source using internal network addresses (such as gateway or SSH tunnel) do not need to perform this step.

This section takes MySQL 8.0 as an example to introduce how to add IP allowlist.

  • New account: Log in to the database using the Root account, and create an account for NineData to access and authorize the corresponding permissions with the following command.

    CREATE USER '<account name>'@'<NineData IP address>' IDENTIFIED BY '<access password>';
    GRANT <privilege name> ON <database name>.* TO '<account name>'@'<NineData IP address>';
  • Existing account: Log in to the database using the Root account, and add NineData IP allowlist and grant corresponding permissions to the existing account with the following command.

GRANT <privilege name> ON <database name>.* TO '<account name>'@'<NineData IP address>' IDENTIFIED BY '<access password>';
tip

When adding a MySQL data source, you need to use the account and corresponding password that have added NineData's IP address as mentioned above.

Appendix 3: Add multi-active tag to data sources

If the data source serves for multi-node replication tasks (data replication between three or more data sources), in order to prevent circular replication, multi-active flags are required for all data sources participating in the replication, The multi-active tag of each data source must be globally unique.

  1. Log in to NineData Console.

  2. In the navigation pane on the left, click Datasource>Datasource.

  3. Click the target data source ID to open the Data Source Details page.

    1. In the data source details area (including data source name, ID, creator, creation time, etc.), click Show Details.
  4. Find Multi-Active Tag, click the edit icon on the right side of it.

  5. Enter the Multi-Active Tag, and click OK.

    tip
    • Multi-active tag can contain 1~64 characters.
    • The multi-active tag must be globally unique and cannot be duplicated with other multi-active tag.

Appendix 4: Create Tags

Tags are used for custom grouping in the slow query analysis feature of the Database DevOps module. By configuring tags for specific data sources, you can flexibly group and display slow queries on the dashboard based on business logic, departments, or other requirements. This helps you more precisely identify and address performance bottlenecks.

  1. Log in to the NineData console.

  2. In the left navigation bar, click > .

  3. Click the tab and then click on the page.

  4. In the pop-up dialog, enter the and click .

    tip
    • The tag name must be globally unique.
    • After creating the tag, you need to bind it to the target data source in the slow query analysis module. For more information, see Slow Query Analysis.

Common problem

IssuePossible CauseSolution
Error when adding a data source: http connections are not allowed from xxx.xxx.xxx.xxx:xxxxx.NineData service's IP address is not whitelisted on your server.Before adding a data source, make sure to add NineData's service address to your data source whitelist. For more information, please refer to the Precautions section in this article.