Create a Kafka Data Source

Add a Kafka data source to NineData for data replication and other supported features. This guide covers broker connection settings, connection mode, authentication, environment settings, SSL settings, and the connection test.

Before you begin

• The server IP address of NineData has been added to the data source allowlist. The server IP address can be viewed by selecting the target region in Access Region on the Create Datasource page.

  

• Make sure you have available data source quota; otherwise, the data source cannot be added. You can check your remaining quota in the NineData Console header.

Procedure

1. Sign in to the NineData Console.

2. On the left navigation pane, click Datasource > Datasource.

3. Click Datasource tab, and click Create Datasource on the page. In the popup window for selecting the data source type, choose Database > (the type of data source to be added), and configure the parameters based on the table below on the Create Datasource page.
   tip

   If you make a mistake during the operation, you can click the icon at the top of the Create Datasource page to make a new selection.

4. Configure the data source parameters:

   Parameter	Description
   Name	Enter a data source name. Use a meaningful name to find and manage it later.
   Connection	Select how NineData connects to the data source. Supported methods are IP Address, Gateway, and SSH Tunnel. IP Address: Connect through a public network address. Gateway: Use a NineData gateway for private-network access. Connect the host that runs the data source first. For instructions, see Add Gateway. SSH Tunnel: Connect through an SSH tunnel. Select how NineData connects to the data source. Supported methods are IP Address and SSH Tunnel. IP Address: Connect through a public network address. SSH Tunnel: Connect through an SSH tunnel.
   Configuration items when Connection is IP Address	Broker List: Enter the public network address and port of each Kafka broker. A Kafka cluster can include multiple brokers, and each broker is an independent Kafka server instance. If there are multiple brokers, click Add to add more broker addresses.
   Configuration items when Connection is Gateway	Gateway: Select the NineData gateway installed on the host where the data source is located. Broker List: Enter the internal connection address and port of the Kafka data source. For Broker, enter localhost if the data source is on the same host as the gateway, or enter the internal IP address of the host where the data source is located.
   Configuration items when Connection is SSH Tunnel	SSH Host: Enter the public IP address or domain name and SSH port of the server where the target data source is located. The default SSH port is 22. SSH Authentication Method: Select the SSH authentication method. Password: Connect with SSH Username and Password. SSH Username: Enter the login username of the server where the target data source is located. Password: Enter the login password of the server where the target data source is located. Key (recommended): Connect with SSH Username and Key File. SSH Username: Enter the login username of the server where the target data source is located. Key File: Click Upload to upload the private key file, which is a key file without a suffix. If you have not created one yet, see Generate SSH Tunnel Key File. Password: Enter the password set when the key file was generated. If no password was set during key generation, leave this field blank. Note: After configuring SSH, click Connection Test to test the tunnel. Possible results: Connection Successfully: The SSH tunnel is established. Error message: The connection failed. Troubleshoot the cause based on the error message and try again. Broker List: Enter the internal connection address and port of the Kafka data source. For Broker, enter localhost if the data source is on the same host as the SSH server, or enter the internal IP address of the host where the data source is located.
   Authentication Method	Select the Kafka authentication method. PLAINTEXT: Does not require authentication. Data is transmitted in plaintext. This method is usually suitable for development and testing environments. SASL/PLAIN (default): A SASL-based authentication method that requires a username and password. Data is transmitted in plaintext. Use this method on encrypted transmission networks. SASL/SCRAM-SHA-256: A SASL-based authentication method that requires a username and password and uses the SHA-256 algorithm to hash the password for higher security. SASL/SCRAM-SHA-512: A SASL-based authentication method that requires a username and password and uses the SHA-512 algorithm to hash the password for higher security.
   Username	Kafka username.
   Password	Kafka password.
   Access Region	Select the region closest to your data source to effectively reduce network latency.
   Environment	Choose an environment based on the business purpose of the data source. This environment identifies the data source. The default environments are PROD and DEV. Custom environments are supported. For details, see create a custom environment. Note: Under organization mode, the database environment can also be applied to permission policy management. For example, the default Prod Admin role can access only data sources in the PROD environment and cannot access data sources in other environments. For more information, see Manage Roles.
   SSL	Configure SSL encryption for the data source (default: off). If the data source requires SSL-encrypted connections, enable this option; otherwise, the connection fails. Click the > to the left of Encryption to expand detailed configuration: Trust Store: The trusted CA-issued certificate, which is the server-side certificate. This certificate verifies the server identity to make sure you are connecting to a trusted Kafka service. Trust Store Password: The password used to protect the Trust Store certificate. Key Store: The client-side certificate used to verify user identity. This certificate verifies that the user connecting to Kafka is trusted. Key Store Password: The password used to protect the Key Store certificate. For SSL configuration methods, see the official documentation: Configure Kafka Encrypted Connections.

5. After you configure all parameters, click Connection Test next to Create Datasource to test connectivity. When Connection Successfully is displayed, click Create Datasource to create the data source. Otherwise, review the connection settings and run the test again.

Result

After the connection test succeeds and the data source is created, it appears in the data source list and is available to NineData features that support Kafka.

Appendix: Add NineData IP addresses to the Kafka whitelist

When you add a data source located in On-Premise/Other Cloud, add the NineData service IP address to the Kafka whitelist so NineData can access Kafka.

The following example uses Kafka 3.3.2.

1. Open Kafka's configuration file server.properties, which is usually located at: <Kafka installation directory>/config/server.properties.

2. Find the listeners parameter and set its value to the IP address and port number that allow access to Kafka. For example, to allow NineData to access Kafka, set it to:

   listeners=PLAINTEXT://121.199.39.25:9092

3. Save the changes and restart the Kafka service.