Accessing Kafka Data Source through the Gateway

NineData supports accessing Kafka data sources through a gateway, allowing you to connect to Kafka without providing an external address, even if Kafka is in a private or dedicated network environment.

Prerequisites

• The host where the gateway is deployed must be within the intranet environment of the Kafka data source and must be able to access the Kafka data source smoothly through the intranet.
• The host where the gateway is deployed must have internet access. If your host cannot access the internet and is within a local area network, you can deploy a proxy gateway. For more information, see Using a Proxy Gateway.

Step 1: Deploy the Gateway

This step is applicable to hosts that can access the public network. If your host cannot access the public network and is in the local area network, see Using Proxy Gateway to deploy the gateway.

1. Log in to the NineData console on the host where the gateway needs to be deployed.

2. In the left navigation pane, click Datasource>Gateway.

3. Click Create Gateway in the upper right corner of the page.

4. Configure the parameters based on the following table.

   Parameter	Description
   Name	The name of the gateway. For easy searching and management, manual specification is supported.
   Environment	Select the operating system of the host where the gateway needs to be deployed, and the corresponding gateway installation method is displayed. The following operating systems are supported: Windows(x86_64) Linux(x86_64) macOS
   Access Region	Select the region closest to the host where the gateway is deployed to obtain relatively fast access speed.
   Advanced	No configuration required.

5. Install and configure the gateway on the host of the deployment end based on the installation method displayed below. After the configuration is completed, the connection diagram on the right side of the page changes from Waiting for the local gateway to launch... to The local gateway is connected successfully.

6. Click on Create Datasource, select the desired data source in the pop-up window, and proceed to Step 2.

Step 2: Access the Kafka Data Source

1. On the Create Datasource page, configure the parameters based on the table below.

   Parameter	Description
   Name	Enter the name of the data source. For ease of later retrieval and management, please use a meaningful name.
   Connection	Select Gateway.
   Gateway	Select the gateway ID deployed in Step 1.
   Broker List	Enter the connection address and port of the Kafka cluster. If Kafka is installed on the local machine: Enter localhost or 127.0.0.1 on the left and the actual access port of Kafka on the right. If Kafka is installed on another host within the intranet: Enter the intranet IP of that host on the left and the actual access port of Kafka on the right. Note: If there are multiple connection addresses, click Add to add a new record and continue filling in.
   Authentication Method	Select the authentication mode for Kafka. PLAINTEXT: No authentication information is required, and data is transmitted in plain text. This mode is usually suitable for development and testing environments. SASL/PLAIN (default): Authentication is based on SASL and requires a username and password. Data is transmitted in plain text. It is recommended to use this mode over encrypted networks. SASL/SCRAM-SHA-256: Authentication is based on SASL and requires a username and password. The password is hashed using the SHA-256 algorithm, providing higher security. SASL/SCRAM-SHA-512: Authentication is based on SASL and requires a username and password. The password is hashed using the SHA-512 algorithm, providing higher security.
   Username	Enter the login username for Kafka.
   Password	Enter the login password for Kafka.
   Access Region	Select the region closest to your Kafka host to effectively reduce network latency.
   Environment	Select according to the actual business purpose of the data source as an environment identifier for the data source. The default provides the PROD and DEV environments, and also supports you to create a custom environment. Note: In organization mode, the database environment can also be applied to permission policy management. For example, the default Prod Admin role only supports access to data sources in the PROD environment and cannot access data sources in other environments. For more information, see Managing Roles.

2. After configuring all parameters, click Create Datasource on the right of Connection Test to test whether the data source can be accessed normally. If Connection Successfully is displayed, click Create Datasource to complete adding the data source. If not, please recheck the connection settings until the connection test succeeds.

Appendix: Using a Proxy Gateway

If the host where the gateway needs to be deployed cannot access the public network, a proxy gateway is needed. You need to prepare two hosts in the same LAN that can access each other:

• Host A: A host that can access the public network.
• Host B: The host where the gateway needs to be deployed.

Create a gateway in Host A. The role of this gateway is to act as a proxy so that Host B can connect to the NineData server through the proxy.

Procedure

1. Log in to the NineData console on Host A.

2. In the left navigation bar, click Datasource > Gateway.

3. Click Create Gateway in the upper right corner of the page.

4. Configure the parameters according to the following table.

   Parameter	Description
   Name	The name of the gateway, which can be manually specified for easy search and management.
   Environment	Select the operating system of the host where the gateway needs to be deployed, and the corresponding gateway installation method will be displayed. The following operating systems are supported: Windows (x86_64) Linux (x86_64) macOS
   Access Region	Select the region closest to the host where the gateway is deployed to obtain relatively fast access speed.
   Advanced	No need to configure.

5. Install and configure the gateway in Host A according to the installation method displayed below. After the configuration is completed, the connection diagram on the right side of the page will change from Waiting for the local gateway to launch... to The local gateway is connected successfully.

6. Click Completed and back to List, and then click Create Gateway in the upper right corner of the page again.

7. In Environment, select the operating system of Host B.

8. Click Advanced, select Use Proxy, and click the drop-down box below Proxy. Select the gateway created in Host A from the drop-down menu.

   tip

   At this point, the gateway process will automatically open a port on Host A for Host B's access. When the page prompts "xxx has enabled proxy function, port xxx", it means that the proxy gateway has been successfully opened.

9. Install the gateway on Host B according to the installation steps displayed below.

10. Wait for the connection diagram on the right side of the page to change from Waiting for the local gateway to launch... to The local gateway is connected successfully.

11. Click Create Datasource to enter Step 2.