Arcanna.ai and AWS GuardDuty

Introduction

This is a step-by-step guide on deploying an Arcanna.ai instance in your AWS environment and how to use AWS Guard Duty as a source of alerts for Arcanna.ai

Prerequisites

• Arcanna.ai - Arcanna.ai can be deployed in the cloud (AWS), or on-premise. For setup, you can follow this user guide. For flexibility reasons, Arcanna.ai is exporting the AI processed alerts to an internal Elasticsearch/Opensearch data warehouse, that needs be installed at the same time with Arcanna.ai. You can use an Elasticsearch/Opensearch instance you already have, or a new one dedicated just for Arcanna.ai backend.

• AWS OpenSearch

• GuardDuty

GuardDuty and OpenSearch/Elastic - how to connect

1. Ensure you have GuardDuty activated

2. Set up the Lambda function to send findings from S3 to OpenSearch

3. Go to the github repository and follow the steps in the README file: https://github.com/siscale/aws-lambda-guardduty-to-elastic

4. Setup eventBridge to execute the Lambda function deployed above

Processing alerts using AI

To showcase this process, we will create a simple AI job that will train an AI model to perform alert triage on GuardDuty alerts.

The job will read the alerts (Input), it will process the alerts with a universal AI model (Process), and will save the results in the data warehouse (Output).

1. Go on the AI Jobs page and click on Create new job

2. Give a name to the Job and select Alert Triage as a Category

   

3. Pick the data warehouse and the index with GuardDuty alerts.

   You can filter based on timestamp (start - end time) or by any other field using an Elasticsearch query filter.

   

4. Select Generic Processor

   This will help us to train an AI model from scratch

   

5. Select an Output data warehouse

   It can be the same or a different one as the Input data warehouse

   

6. Choose the Automation Integration

   We don’t plan to use any automation for this example, so you can click Create job. For more Automations info go to guide.

   

7. Start the AI job

   Start the job by selecting it and clicking on Play

   Check if the status of the job turns into STARTED

   You will notice that the Processed count increases, meaning that the alerts are collected by Arcanna.ai

   You can always EDIT the job to tune the above parameters

   

8. Select AI features

   You need to pick some relevant alert fields that the AI will use for training and decision-making. These are called features. and are also used to create buckets.

   Buckets are grouped alerts with the same values for the selected features.

   Go to the Job feedback page and click on Feature Selection

   

   Since the selection of features defines the fields used to make a decision, in this step we should either select only the fields from the list that are relevant for the investigation, or decide first-hand if an alert is false-positive.

   The job created in this tutorial will perform alert triage on all alerts and look at high-level information. Thus, to achieve this target, we can pick the following fields:

   

   Alert information:

   Field name	Description
   resource.resourceType	Describes Resource Type
   service.action.actionType	Describes the Finding Action Type
   service.resource	Describes what role had the resource for the Finding: Target or Actor

   After saving the Feature Selection, the buckets will be created and will appear on the Feedback page.

   Process time will vary depending on the number of alerts.

9. Feedback and Retrain

   With buckets being generated, you can provide feedback. By default, the Arcanna.ai labels with "ESCALATE" all alerts before training a model.

   Based on the selected features, you should be able to make a decision of ESCALATE or DROP.

   Make sure you express your feedback for both cases because unbalanced feedback might result in an unwanted bias.

   With the help of automation, REST API, or just by reading the results in the Output data warehouse, ESCALATE and DROP results can mean opening a ticket or just investigating the alert later (or never based on how much you are sure about your decision and on Arcanna.ai’ inference results).

   

   After you finish, go to the Retrain Page and click Retrain.

   

   Once it finishes, you have a trained AI model, ready to make decisions accordingly to your feedback.

   For checking the results, you can import the sample set one more time and go to the Events page to check the inference for each alert.

   

   Now all newly generated alerts will be processed by the AI model. If you want to test you can do it by importing the samples again.