Skip to main content

Arcanna.ai and IBM Security QRadar SIEM

Introduction

This is a step-by-step guide on integrating Arcanna.ai with IBM Security QRadar SIEM as part of your Cybersecurity toolkit.

The out of the box REST API integration with IBM Security QRadar SIEM allows the read of offenses, and after the AI decides either to Escalate or Drop the offense, it is also capable of updating the offense accordingly by:

  • Creating an investigation note with the result of the inference
  • Turning on a flag such as follow-up flag
  • Assigning the offense to a preconfigured user
  • Updating the offense status (soon)

Additionally, Arcanna's decision can be sent to a 3rd party tool, or open a ticket in an Incident Management platform.

Prerequisites

  • Arcanna.ai - Arcanna.ai can be installed on-premise or in your Kubernetes cluster. For setup, you can follow this user guide. For flexibility reasons, Arcanna.ai is exporting the AI processed alerts to an internal Elasticsearch/Opensearch data warehouse, that needs be installed at the same time with Arcanna.ai. You can use an Elasticsearch/Opensearch instance you already have, or a new one dedicated just for Arcanna.ai backend.

  • IBM Security QRadar SIEM

    This should be already installed and configured for detection and triggering offenses.

    info

    If you want to create a lab environment, IBM Security QRadar SIEM offers a community version, just enough for test purposes: https://www.ibm.com/community/qradar/ce/

IBM Security QRadar SIEM - how to connect

  1. Active API token

    • It can be generated directly from IBM Security QRadar SIEM interface.

    • Go to the Admin page, and in the User Management section, click on Authorized Services.

    • Create a new Service and give it a name, Admin user role and Admin security profile. Click Save and Check if the token was created. Example of token: c9be4b41-xxxx-xxxx-xxxx-xxxxxxxxxxxx

    • Make sure you deploy the configuration changes since the API token will be active only after the changes are deployed. This will restart the services associated with this Change.

  2. Configure Arcanna.ai with a new Integration

    • Go to Integrations and select QRadar as part of the Detection System category.

    • Connectivity on port 443 is required.

    • Arcanna.ai does a precheck and if the integration is successful, you are having everything in place to create an AI job.

Create an AI Job

To showcase this process, we will create a simple AI job that will train an AI model to perform alert triage on QRadar alerts.

The job will read the alerts (Input), it will process the alerts with a universal AI model (Process), and will save the results in the data warehouse (Output).

  1. Go on the AI Jobs page and click on Create new job

  2. Give a name to the Job and select Decision Inteligence as Category

  3. Tune your input parameter based on previously defined integration

    Arcanna.ai lets you constrain the offense collection based on the Start Time of the offense, or by allowing you to use IBM Security QRadar SIEM specific API, filtering capabilities as described in IBM Security QRadar SIEM API Documentation: https://www.ibm.com/docs/en/qradar-common?topic=overview-filter-syntax

    In the below example, we collect only offenses part of the default domain. Many managed security service providers are using domains for multi-tenancy and in this way, you can create a job per client, for example.

    Arcanna.ai is also capable of collecting the events associated with the offense to be used as features in decision-making. It uses AQL searches in Qradar database where Qradar stores them in an uniform way described here. The raw events will be seen in the fields list as events.<qradarfield_name> or in a deduplicated form as events<qradar_field_name>.


    At this point, you can define Automations such as:

    • returning the results in IBM Security QRadar SIEM under the form of a Follow Up flag
    • investigation note
    • assigning the offense to a user (will be available soon)

    info

    All the above parameters can be edited anytime (besides job name)

    When the job is started, Arcanna.ai begins to read offenses using GET /Siem/offenses API calls.

    Please note that only offenses with OPEN status are collected.

    You should check if the processed count increases while reading the offenses (the hecking interval is 10s by default).

Train an AI model

Each AI job has an AI model behind it. To allow user feedback, first, you need to select Machine Learning features - these are fields used as decision points.

Open the Job’s Feedback page and click on Feature Selection.

All below fields are available to be used as part of the decision. Usually, the ones depicted in the below image are enough for a quick start.

As part of the integration, Arcanna.ai also parses the offense description based on containing and preceding keywords.

IBM Security QRadar SIEM has a strict offense structure. An Offense object contains the following fields:

Field nameDescription
idNumber - The ID of the offense. (Filterable. Sortable)
description String - The description of the offense
assigned_to String - The user the offense is assigned to. (Filterable. Sortable)
categories Number - The number of event categories that are associated with the offense. (Filterable. Sortable)
category_count Number - The number of event categories that are associated with the offense. (Filterable. Sortable)
policy_category_count Number - The number of policy event categories that are associated with the offense. (Filterable. Sortable)
security_category_count Number - The number of security event categories that are associated with the offense. (Filterable. Sortable)
close_time Number - The number of milliseconds since epoch when the offense was closed. (Filterable. Sortable)
closing_user String - The user that closed the offense. (Filterable. Sortable)
closing_reason_id String - The user the offense is assigned to. (Filterable. Sortable)
credibility Number - The credibility of the offense. (Filterable. Sortable)
relevanceNumber - The relevance of the offense. (Filterable. Sortable)
severityNumber - The severity of the offense. (Filterable. Sortable)
magnitude Number - The magnitude of the offense. (Filterable. Sortable)
destination_networksArray of strings - The destination networks that are associated with the offense. (Filterable)
source_networkString - The source network that is associated with the offense.
device_countNumber - The number of devices that are associated with the offense. (Filterable. Sortable)
event_countNumber - The number of events that are associated with the offense. (Filterable. Sortable)
flow_countNumber - The number of flows that are associated with the offense. (Filterable. Sortable)
inactiveBoolean - True if the offense is inactive. (Filterable. Sortable)
destination_networksArray of strings - The destination networks that are associated with the offense. (Filterable)
last_updated_timeNumber - The number of milliseconds since epoch when the offense was last updated. (Filterable. Sortable)
local_destination_countNumber - The number of local destinations that are associated with the offense. (Filterable.)
offense_sourceString - The source of the offense. (Sortable)
offense_typeNumber - A number that represents the offense type. Use GET /siem/offense_types to retrieve the list. (Filterable. Sortable)
protectedBoolean - True if the offense is protected. (Filterable. Sortable)
follow_upBoolean - True if the offense is marked for follow-up. (Filterable. Sortable)
remote_destination_countNumber - The number of remote destinations that are associated with the offense. (Filterable. Sortable)
source_countNumber - The number of sources that are associated with the offense. (Filterable)
start_timeNumber - The number of milliseconds since epoch when the offense was started. (Filterable. Sortable)
statusString - The status of the offense. One of "OPEN", "HIDDEN", or "CLOSED". (Filterable, but the following operators are not supported: <, >, <=, >=, BETWEEN. Sortable)
username_countNumber - The number of usernames that are associated with the offense. (Filterable. Sortable)
source_address_idsArray of numbers -The source address IDs that are associated with the offense. (Filterable)
local_destination_address_idsArray of numbers - The local destination address IDs that are associated with the offense. (Filterable)
domain_idNumber - Optional. ID of associated domain if the offense is associated with a single domain. (Filterable)
rulesArray - An array of rules that contributed to the offense (Filterable):
log_sourcesArray - An array of log sources contributed to the offense (Filterable)
info

Numerical fields can generate AI overfitting problems so they are not taken into consideration. However, these fields can be transformed into meaningful strings using the Create custom field button in the top right corner.

info

By integrating Arcanna.ai with Virus Total, IP, hash and domain fields can be cross validated with Virus Total and the result can be used as feature

Newly generated fields can be selected as Machine Learning features as well.

Buckets will be generated and offenses with identical Machine Learning features will be aggregated in the same buckets. You can provide feedback on buckets by selecting the bucket and clicking on Escalate or Drop and Save. This action aggregates the feedback into a knowledge base used for training the model.

info

Make sure you provide both Escalate and Drop feedback to avoid biases.

Also, take into consideration that until a first Train, every bucket (and inherently every offense) will receive a pre-populated Escalate label.

After finishing the feedback session, by going to the Job Retrain page you’ll be able to click on retrain.

Based on the amount of feedback you provided, this step can take seconds or several minutes to complete.

Results can be seen directly in Qradar, in Arcanna.ai's Job Event Explorer page, or directly in Qradar Offenses Dashboard.

Based on the inference result, the offense is updated with the follow-up flag and a suggestive investigation note.