Context Enrichment
Overview
Context Enrichment is a functionality that Arcanna.AI provides which empowers security analysts to enhance the information contained within their AI job events. By adding additional data from external sources, users can gain deeper insights, improve threat detection accuracy and make more informed decisions. The enrichment process can be applied selectively to specific events based on user-defined filters or uniformly to all events within a job. This documentation outlines the four steps of the Context Enrichment process.
Setting up Context Enrichment
In order to add a Context Enrichment on a job, first, the users need to go to the Flows page by clicking the Flows icon on the side menu:
Then, they need to go to the Context tab and click Add+:
Now, the users have to follow these 4 steps:
Step 1: Select Phase
In the Select Phase, users can see the events processed by the AI job displayed inside a table, similar to the event explorer table. On this page, they can apply filters based on event properties to target specific events for enrichment. The enrichment will be applied only on the newly ingested events which match the filters that the user set. If no filters are selected, the enrichment will be applied on all incoming events.
Step 2: Gather Phase
The Gather Phase involves retrieving additional information from external sources. The external sources are Context Enrichment integrations which the users previously created (i.e. Splunk, External REST API). Examples of relevant External REST API data sources can be: threat intelligence platforms, IP reputation databases etc.
After choosing an integration, analysts can perform searches and queries on the fields of the events filtered in the previous step. They can use Jinja templates to dynamically retrieve values from the selected events. Jinja templating can be used either in the Headers, as Query Parameters or in the Request Body. The response from the query is the actual context and will be used in the next step.
Here is an example of a REST API request using a dynamic query parameter. We want to enrich the events with extra information about the country of destination. This is why we call an API which provides us with the necessary context. On the right you can see the returned response after the user hit the Play button:
Arcanna.AI also provides an event sample to help users identify which fields to query by, without returning to the previous phase. Here, the users can see the available fields which they can build queries around. These fields are called pivots.
Step 3: Build Phase
In the Build Phase, users select which new fields from the gathered data to add to their events. The data retrieved in the Gather Phase is displayed in a structured JSON format. Users can choose specific fields from the JSON response to enrich their original events with.
In the example below, we chose to enrich our incoming events with information (context) about the calling code of the country, the population and its capital.
Step 4: Save Phase
The final phase is the Save Phase, where users finalize the enrichment process. A summary of the fields that will be added to the events is displayed. Users can assign a name to the enrichment process for future reference.
Then, the enrichment process is saved and can be edited later if needed. This allows for adjustments and refinements to the enriched data as new information becomes available or requirements change.
Running Context Enrichment
After the Context Enrichment is saved and enabled on the AI job, it is applied automatically to incoming events processed by the job.
As new events are ingested, they are evaluated against the filters defined in the Select Phase of process of setting up the enrichment. Only events that match the specified filters will undergo context enrichment. Context Enrichment is not applied to the events ingested before it was set up and enabled.
For each matching event, a custom request/query, which likely involves one or more properties of the event, is sent to the external data source (integration) defined in the Gather Phase. The query is dynamically generated using Jinja templates to incorporate specific event properties. The query is executed, and the external source returns a response. Arcanna.AI caches the requests and responses to not repeat expensive requests for every event.
From the response gathered, the fields previously chosen during the Build Phase are extracted from the response. These extracted fields are added to the newly processed event, enhancing it with additional context and information, such as IP reputation, DNS information, or risk assessments. This means that the event has been enriched with context.
In order to see the extra information on events, the user needs to select the new fields to be displayed in event explorer:
Events enriched with context:
Users can monitor the status of the enrichment process and review how enriched events appear compared to non-enriched ones. If necessary, analysts can disable the enrichment on a job. This action stops the application of the enrichment process to any new events processed by the job.
When an enrichment is disabled, it will not be applied to any incoming events. However, previously enriched events remain unchanged. Disabled enrichments can be re-enabled at any time, resuming their application to future events that match the set filters.
Conclusion
Arcanna.AI's Context Enrichment functionality enhances AI job events with valuable external data, enabling more informed threat analysis and decision-making. By following the four-step process(Select, Gather, Build, and Save) and understanding the dynamics of running context enrichment, users can maximize the utility and accuracy of their AI-driven insights. The flexibility to manage, edit, and disable enrichments ensures that the process remains aligned with evolving needs and conditions.