Elasticsearch

Elasticsearch is a distributed, open-source search and analytics engine built on top of the Apache Lucene library. It is commonly used for full-text search, structured data querying, and real-time data analytics. Elasticsearch is a core component of the Elastic Stack (ELK Stack), which includes Logstash, Kibana, and Beats, allowing users to ingest, visualize, and analyze data.

Arcanna - Elasticsearch integration

The Arcanna integration with Elasticsearch serves multiple roles: it acts as storage for processed alerts, an input for gathering data, as context enrichment to enhance alerts with information from an external Elasticsearch, and a post-decision mechanism to update the original alert based on Arcanna's predictions.

Steps to configure Elasticsearch integration:

Prerequisites

• A valid Arcanna instance - for setup, follow this user guide.
• Elasticsearch instance - for setup, follow this user guide.
• Kibana, free under basic license (guide).

How to connect

Go to Arcanna instance:

Create the integration

1. Go to Integration tab and search for Elasticsearch:

2. Click on it and complete it with your Elasticsearch instance information. You can find a short description of each of its roles here:

Use as storage integration

Arcanna uses an internal Elasticsearch to store data by default. You can skip this step if you prefer to use Arcanna's internal storage.

1. Create an Arcanna job using Elasticsearch integration:
   • Go to AI Jobs tab and click Create job
   • Expand the storage tab:
   
   • Select the Elasticsearch just created as the new storage
   

Use as input integration

• Click add input integration (Add +) and select the Elasticsearch integration
• Complete the fields with the desired configuration. The only required field is the Index pattern. Enable Preserve original id to ensure that documents in Arcanna retain the same ID as in Elasticsearch. Here's an example:

• Click Save and run to save and start the job.

Use as context enrichment integration

1. Go to Flows page, click Add integration + and select Context Enrichment:

2. Select filters for the data to be used for context enrichment, or leave it as-is to apply context enrichment for all data:

3. Select the Elasticsearch integration, complete the URL with the index used for context enrichment, and provide the desired query in the body:

4. Then, select the fields or field patterns from the document to be added as enrichment to the alert:

5. In the final step, complete the Title for the context enrichment and click Save:

6. Navigate to the Event Explorer, expand a new alert, and observe how the context enrichment fields have been applied to the alert:

Use as post decision integration

Use Elasticsearch post-decision capabilities to update SIEM offenses, incidents, investigation notes, or events based on the Arcanna decision.

1. Go to Flows page, click Add integration + and select Post decision:

2. Configure the post-decision settings.
   • The Custom alert identifier field and Custom alert index field settings are used to identify the alert for applying the post-decision. By default, Elasticsearch internal, _id and _index, are used.
   • Select the decision labels to apply the post-decision to.
   • Enable Add comments to alerts to include a custom note in the alert. Here, the default message is used, which specifies the Arcanna decision and the job.
   • Enable Update field in index to modify a field with a specified value. In this example, we use event.severity=3 for the Escalate label, event.severity=2 for the Investigate label, event.severity=1 for the Drop label:

For the post-decision to be applied, the job must have a trained model. Navigate to Decision Points, provide feedback on buckets or events, and initiate a training session

6. Navigate to the Event Explorer, expand a new alert, and observe how the post-decision have been executed:

7. Go to Kibana and search for the alert to observe the newly added fields by Arcanna. The index, in this case, is the same as the input index unless a different one is configured in Custom alert index in Step 2.

• We can observed that the event.severity field was updated to the value 1 based on the Drop decision.