Configure an Elasticsearch External Collection Source

When slow logs are already written to Elasticsearch, you can create an Elasticsearch external collection source. NineData then reads slow logs from the target Elasticsearch data source and writes them into slow query analysis.

Prerequisites

• You have created or joined an organization, and this organization has subscribed to either DevOps Pro or DevOps Enterprise. Please ensure that your annual or monthly subscription is still active. For more information, please refer to Manage Organizations.
• Your current account has switched to the target organization. For more information, please refer to Switching to an Organization.

• The Elasticsearch data source has already been added to NineData.
• The target Elasticsearch cluster already contains slow log data that matches the NineData external collection script requirements.
• The Elasticsearch data source account has read access to the target slow log indices or data streams.
• You have obtained the Elasticsearch data source ID. You can find it in the data source detail page URL. For example, in /configuration/datasource/detail/ds-xxxxxxxxxxxx, the last path segment is the data source ID.

Permission Requirements

If security is enabled on the target Elasticsearch cluster, grant the following index privileges on the target slow log indices or data streams to the account used by the NineData data source.

Privilege	Purpose
read	Read slow log documents and run searches.
view_index_metadata	Read index, mapping, field capability, and other metadata.

If you access slow log indices through aliases or wildcards, make sure all matched indices are included in the permission scope. NineData does not need write, delete, or index management privileges to read slow logs.

Step 1: Open the External Collection Sources Page

1. Log in to the NineData Console.

2. In the left navigation pane, click DevOps > Slow Query.

3. On the top tab bar, click External Collection Source.

Step 2: Create an Elasticsearch External Collection Source

1. On the External Collection Source tab, click Create External Collection Source.

2. Configure the drawer as follows.

   Parameter	Description
   Source Type	Select Elasticsearch.
   Datasource	Select the target Elasticsearch data source.
   Collection Source Name	Enter a custom source name.
   Description	Optional. Enter a description for this source.
   Collection Definition Configuration	Keep the default MySQL tab.
   Enable Configuration	Keep it enabled.

3. Click OK to create the source.

Step 3: Test the Collection Definition

1. Return to the External Collection Source list, find the target source, and click Edit.

2. In the Collection Definition Configuration section, make sure the current tab is MySQL and Enable Configuration is enabled.

3. In the fetch row, click Test.

4. In the Input Information box of the test dialog, replace the default example with the following JSON, then click Test.

   {
     "DATASOURCE_ID": "ds-xxxxxxxxxxxx",
     "DB_ID": "",
     "DB_TYPE": "",
     "POSITION": {
       "index": "",
       "lastTaskEndTime": "",
       "sort": ""
     }
   }

   Item	Required	Description
   DATASOURCE_ID	Yes	Enter the target Elasticsearch data source ID.
   DB_ID	No	If you need to filter by business database and the Elasticsearch document contains dbId, enter the value. Leave it empty for the first test.
   DB_TYPE	No	If you need to filter by database type and the Elasticsearch document contains dbType, enter the value. Leave it empty for the first test.
   POSITION	No	Checkpoint information. Keep empty strings for the first test. For checkpoint testing, reuse the position value returned by the previous test.

5. After the test succeeds, Return Information displays the returned slow log data. The step status also changes to Successful or Success.

6. Click Close to exit the test dialog.

Step 4: Confirm the Configuration Status

After you return to the edit drawer, confirm the following:

• The MySQL tab shows Success.
• The fetch step shows Successful.

Then click OK to save or leave the page.

FAQ

Symptom	Recommended Check
The slow log storage data source is not found	Check whether DATASOURCE_ID is the correct data source ID.
The test succeeds but returns no data	Retry with DB_ID and DB_TYPE left empty. If it is still empty, check whether the external log format matches the script requirements.
You need to test checkpoint continuation	Copy position from the previous result and use it as POSITION in the next test.