Arcanna.ai and IBM Security QRadar SIEM

Introduction

This is a step-by-step guide on integrating Arcanna.ai with IBM Security QRadar SIEM as part of your Cybersecurity toolkit.

The out-of-the-box REST API integration with IBM Security QRadar SIEM allows the read of offenses, and after the AI decides either to Escalate or Drop the offense, it is also capable of updating the offense accordingly by:

• Creating an investigation note with the result of the inference
• Turning on a flag such as follow-up flag
• Assigning the offense to a preconfigured user
• Updating the offense status (soon)

Additionally, Arcanna's decision can be sent to a 3rd party tool, or open a ticket in an Incident Management platform.

Prerequisites

• Arcanna.ai - Arcanna.ai can be installed on-premise or in your Kubernetes cluster. For setup, you can follow this user guide. For flexibility reasons, Arcanna.ai is exporting the AI processed alerts to an internal Elasticsearch/Opensearch data warehouse, that needs be installed at the same time with Arcanna.ai. You can use an Elasticsearch/Opensearch instance you already have, or a new one dedicated just for Arcanna.ai backend.

• IBM Security QRadar SIEM

  This should be already installed and configured for detection and triggering offenses.

  info

  If you want to create a lab environment, IBM Security QRadar SIEM offers a community version, just enough for test purposes: https://www.ibm.com/community/qradar/ce/

IBM Security QRadar SIEM - how to connect

1. Active API token

   • It can be generated directly from IBM Security QRadar SIEM interface.

   • Go to the Admin page, and in the User Management section, click on Authorized Services.

   

   • Create a new Service and give it a name, Admin user role and Admin security profile. Click Save and Check if the token was created. Example of token: c9be4b41-xxxx-xxxx-xxxx-xxxxxxxxxxxx

   • Make sure you deploy the configuration changes since the API token will be active only after the changes are deployed. This will restart the services associated with this Change.

   

2. Configure Arcanna.ai with a new Integration

   • Go to Integrations and select QRadar as part of the Detection System category.

   • Connectivity on port 443 is required.

   • Arcanna.ai does a precheck and if the integration is successful, you are having everything in place to create an AI job.

   

Create an AI Job

To showcase this process, we will create a simple AI job that will train an AI model to perform alert triage on QRadar alerts.

The job will read the alerts (Input), it will process the alerts with a universal AI model (Process), and will save the results in the data warehouse (Output).

1. Go on the AI Jobs page and click on Create new job

2. Give a name to the Job and select Decision Inteligence as Category

   

3. Tune your input parameter based on previously defined integration

   Arcanna.ai lets you constrain the offense collection based on the Start Time of the offense, or by allowing you to use IBM Security QRadar SIEM specific API, filtering capabilities as described in IBM Security QRadar SIEM API Documentation: https://www.ibm.com/docs/en/qradar-common?topic=overview-filter-syntax

   In the below example, we collect only offenses part of the default domain. Many managed security service providers are using domains for multi-tenancy and in this way, you can create a job per client, for example.

   Arcanna.ai is also capable of collecting the events associated with the offense to be used as features in decision-making. It uses AQL searches in Qradar database where Qradar stores them in an uniform way described here. The raw events will be seen in the fields list as events.<qradar_field_name> or in a deduplicated form as events_<qradar_field_name>.

   
   

   At this point, you can define Automations such as:

   • returning the results in IBM Security QRadar SIEM under the form of a Follow Up flag
   • investigation note
   • assigning the offense to a user (will be available soon)

   

   info

   All the above parameters can be edited anytime (besides job name)

   When the job is started, Arcanna.ai begins to read offenses using GET /Siem/offenses API calls.

   Please note that only offenses with OPEN status are collected.

   You should check if the processed count increases while reading the offenses (the hecking interval is 10s by default).

   

Train an AI model

Each AI job has an AI model behind it. To allow user feedback, first, you need to select Machine Learning features - these are fields used as decision points.

Open the Job’s Feedback page and click on Feature Selection.

All below fields are available to be used as part of the decision. Usually, the ones depicted in the below image are enough for a quick start.

As part of the integration, Arcanna.ai also parses the offense description based on containing and preceding keywords.

IBM Security QRadar SIEM has a strict offense structure. An Offense object contains the following fields:

Field name	Description
id	Number - The ID of the offense. (Filterable. Sortable)
description	String - The description of the offense
assigned_to	String - The user the offense is assigned to. (Filterable. Sortable)
categories	Number - The number of event categories that are associated with the offense. (Filterable. Sortable)
category_count	Number - The number of event categories that are associated with the offense. (Filterable. Sortable)
policy_category_count	Number - The number of policy event categories that are associated with the offense. (Filterable. Sortable)
security_category_count	Number - The number of security event categories that are associated with the offense. (Filterable. Sortable)
close_time	Number - The number of milliseconds since epoch when the offense was closed. (Filterable. Sortable)
closing_user	String - The user that closed the offense. (Filterable. Sortable)
closing_reason_id	String - The user the offense is assigned to. (Filterable. Sortable)
credibility	Number - The credibility of the offense. (Filterable. Sortable)
relevance	Number - The relevance of the offense. (Filterable. Sortable)
severity	Number - The severity of the offense. (Filterable. Sortable)
magnitude	Number - The magnitude of the offense. (Filterable. Sortable)
destination_networks	Array of strings - The destination networks that are associated with the offense. (Filterable)
source_network	String - The source network that is associated with the offense.
device_count	Number - The number of devices that are associated with the offense. (Filterable. Sortable)
event_count	Number - The number of events that are associated with the offense. (Filterable. Sortable)
flow_count	Number - The number of flows that are associated with the offense. (Filterable. Sortable)
inactive	Boolean - True if the offense is inactive. (Filterable. Sortable)
destination_networks	Array of strings - The destination networks that are associated with the offense. (Filterable)
last_updated_time	Number - The number of milliseconds since epoch when the offense was last updated. (Filterable. Sortable)
local_destination_count	Number - The number of local destinations that are associated with the offense. (Filterable.)
offense_source	String - The source of the offense. (Sortable)
offense_type	Number - A number that represents the offense type. Use GET /siem/offense_types to retrieve the list. (Filterable. Sortable)
protected	Boolean - True if the offense is protected. (Filterable. Sortable)
follow_up	Boolean - True if the offense is marked for follow-up. (Filterable. Sortable)
remote_destination_count	Number - The number of remote destinations that are associated with the offense. (Filterable. Sortable)
source_count	Number - The number of sources that are associated with the offense. (Filterable)
start_time	Number - The number of milliseconds since epoch when the offense was started. (Filterable. Sortable)
status	String - The status of the offense. One of "OPEN", "HIDDEN", or "CLOSED". (Filterable, but the following operators are not supported: <, >, <=, >=, BETWEEN. Sortable)
username_count	Number - The number of usernames that are associated with the offense. (Filterable. Sortable)
source_address_ids	Array of numbers -The source address IDs that are associated with the offense. (Filterable)
local_destination_address_ids	Array of numbers - The local destination address IDs that are associated with the offense. (Filterable)
domain_id	Number - Optional. ID of associated domain if the offense is associated with a single domain. (Filterable)
rules	Array - An array of rules that contributed to the offense (Filterable):
log_sources	Array - An array of log sources contributed to the offense (Filterable)

info

Numerical fields can generate AI overfitting problems so they are not taken into consideration. However, these fields can be transformed into meaningful strings using the Create custom field button in the top right corner.

info

By integrating Arcanna.ai with Virus Total, IP, hash and domain fields can be cross validated with Virus Total and the result can be used as feature

Newly generated fields can be selected as Machine Learning features as well.

Buckets will be generated and offenses with identical Machine Learning features will be aggregated in the same buckets. You can provide feedback on buckets by selecting the bucket and clicking on Escalate or Drop and Save. This action aggregates the feedback into a knowledge base used for training the model.

info

Make sure you provide both Escalate and Drop feedback to avoid biases.

Also, take into consideration that until a first Train, every bucket (and inherently every offense) will receive a pre-populated Escalate label.

After finishing the feedback session, by going to the Job Retrain page you’ll be able to click on retrain.

Based on the amount of feedback you provided, this step can take seconds or several minutes to complete.

Results can be seen directly in Qradar, in Arcanna.ai's Job Event Explorer page, or directly in Qradar Offenses Dashboard.

Based on the inference result, the offense is updated with the follow-up flag and a suggestive investigation note.