Configure an Elasticsearch External Collection Source

When slow logs are already written to Elasticsearch, create an Elasticsearch external collection source. NineData then reads slow logs from the target Elasticsearch data source and writes them into slow query analysis.

Before you begin

• You have created or joined an organization, and this organization has subscribed to either DevOps Pro or DevOps Enterprise. Please ensure that your annual or monthly subscription is still active. For more information, please refer to Manage Organizations.
• Your current account has switched to the target organization. For more information, please refer to Switching to an Organization.

• The Elasticsearch data source has already been added to NineData.
• The target Elasticsearch cluster already contains slow log data that matches the NineData external collection script requirements.
• The Elasticsearch data source account has read access to the target slow log indices or data streams.
• You have obtained the Elasticsearch data source ID. Find it in the data source detail page URL. For example, in /configuration/datasource/detail/ds-xxxxxxxxxxxx, the last path segment is the data source ID.

Permission Requirements

If security is enabled on the target Elasticsearch cluster, grant the following index privileges on the target slow log indices or data streams to the account used by the NineData data source.

Privilege	Purpose
read	Read slow log documents and run searches.
view_index_metadata	Read index, mapping, field capability, and other metadata.

If you access slow log indices through aliases or wildcards, make sure all matched indices are included in the permission scope. NineData does not need write, delete, or index management privileges to read slow logs.

Step 1: Open the External Collection Sources Page

1. Sign in to the NineData Console.

2. Go to DevOps > Slow Query.

3. On the top tab bar, click External Collection Source.

Step 2: Create an Elasticsearch External Collection Source

1. On the External Collection Source tab, click Create External Collection Source.

2. Configure the drawer as follows.

   Parameter	Description
   Source Type	Select Elasticsearch.
   Datasource	Select the target Elasticsearch data source.
   Collection Source Name	Enter a custom source name.
   Description	Optional. Enter a description for this source.
   Collection Definition Configuration	Keep the default MySQL tab.
   Enable Configuration	Keep it enabled.

3. Click OK to create the source.

Step 3: Test the Collection Definition

1. Return to the External Collection Source list, find the target source, and click Edit.

2. In the Collection Definition Configuration section, make sure the current tab is MySQL and Enable Configuration is enabled.

3. In the fetch row, click Test.

4. In the Input Information box of the test dialog, replace the default example with the following JSON, then click Test.

   {
     "DATASOURCE_ID": "ds-xxxxxxxxxxxx",
     "DB_ID": "",
     "DB_TYPE": "",
     "POSITION": {
       "index": "",
       "lastTaskEndTime": "",
       "sort": ""
     }
   }

   Item	Required	Description
   DATASOURCE_ID	Yes	Enter the target Elasticsearch data source ID.
   DB_ID	No	To filter by business database, enter the value when the Elasticsearch document contains dbId. Leave it empty for the first test.
   DB_TYPE	No	To filter by database type, enter the value when the Elasticsearch document contains dbType. Leave it empty for the first test.
   POSITION	No	Checkpoint information. Keep empty strings for the first test. For checkpoint testing, reuse the position value returned by the previous test.

5. After the test succeeds, Return Information displays the returned slow log data. The step status also changes to Successful or Success.

6. Click Close to exit the test dialog.

Step 4: Confirm the Configuration Status

After you return to the edit drawer, confirm the following:

• The MySQL tab shows Success.
• The fetch step shows Successful.

Then click OK to save or leave the page.

Result

The Elasticsearch external collection source is saved with a tested MySQL collection definition. When you bind this source during slow log collection, NineData can read matching slow log documents from Elasticsearch and write them into slow query analysis.

Next Steps

Bind the external collection source to the target database data source, wait for a collection cycle, and then check the slow query detail page for collected slow log records.

FAQ

Symptom	Recommended Check
The slow log storage data source is not found	Check whether DATASOURCE_ID is the correct data source ID.
The test succeeds but returns no data	Retry with DB_ID and DB_TYPE left empty. If it is still empty, check whether the external log format matches the script requirements.
Test checkpoint continuation	Copy position from the previous result and use it as POSITION in the next test.