Gather Phase Jinja Templating

Overview

In the Gather Phase of Arcanna.ai's Context Enrichment functionality, security analysts can use Jinja templating to create dynamic and context-aware queries for REST API and Splunk integrations. This page provides detailed instructions and examples on how to effectively use Jinja templates for both types of integrations, including the use of Jinja filters, conditionals (IFs) and loops ( FORs).

Before we start, let's suppose our event structure looks like this:

    {
        "alert_id": 1234,
        "source": {
            "ip_address": "10.128.0.30",
            "port": 9000
        },
        "description": "this is an important alert",
        "severity": "high",
        "cloud": {
            "provider": "gcp",
            "project_id": "arcanna_project"
        }
        "user": {
            "sp_id": "0579935b-b2f5-446a-8af0-d4e3d395c572",
            "role": "admin",
            "username": "arcanna_user"
        },
        "destination": {
            "country": "Mexico",
            "ip_address": "90.95.30.142",
            "port": 8001,
            "city": "Ciudad de Mexico"
        },
        "ip_list": "['10.128.0.30', '90.95.30.142']",
        "@timestamp": "2024-06-05T16:02:22"
    }

REST API Integrations

For REST API integrations, Jinja templates can be used to dynamically construct query parameters, headers, and request bodies. This allows the enrichment process to adapt based on the properties of the security alerts being processed.

Using Jinja Templating for Query Parameters

Jinja Templates can be used to populate endpoint query parameters based on event properties.

Example:

Suppose you want to get information about the Cloud from which the alert came from.

    GET /gcp/api/cloud/{{ cloud.project_id }}

Explanation: In this example, the Jinja template {{ cloud.project_id }} dynamically inserts the project id from the event into the query parameter ip. This allows the API call to be specific to that project id.

Using Jinja Templating for Headers

You can also use Jinja templates to set dynamic headers in your API requests.

Example:

If an API requires a custom header with a dynamic username:

    GET /api/user-info
    {
        "headers": {
            "user": "{{ user.username }}"
        }
    }

Using Jinja Templating in Request Body

Jinja Templates can be used to create request bodies.

Example:

Suppose you want to query an IP reputation service using the IP address from a security alert.

    GET /api/ip-reputation
    {
        "body": {
            "ip": "{{ source.ip_address }}"
        }
    }

Explanation: In this example, the Jinja template {{ source.ip_address }} dynamically inserts the IP address from the security alert into the body. This allows the API call to be specific to the source IP address related to the alert.

Advanced Jinja Usage

Using the values dictionary

The Jinja Templating system uses a values dictionary to access properties of an event. Normally, the values methods and the simple shorthand notation are equivalent:

{{ values['property_name'] }} = {{ property_name }}

However, there are certain scenarios when they are not:

The property name contains special characters. Because of its name, a field from an event cannot be used directly in a Jinja Template. For example, such a value is @timestamp. Using a shorthand notation by simply enclosing the property name in double curly braces * will not work*:
```
{
  "body": {
      "ts": "{{ @timestamp }}"
  }
}
```
To solve the issue, we can use the values dictionary like this:
```
{
  "body": {
      "ts": "{{values['@timestamp']}}"
  }
}
```

We want to retrieve a value based on a reference from our document. Assuming the following input document

{
  "reference_to_field": "my_field_of_interest",
  "my_field_of_interest": "my value of interest"
}

We can retrieve the value my value of interest as such:

{
  "body": {
    "my_value": "{{ values[reference_to_field] }}" // to obtain "my_value": "my value of interest"
  }
}

Using the shorthand notation will not work in this case.

Using Filters

Filters can transform values within your templates.

Here is the list of the Jinja builtin filters: https://jinja.palletsprojects.com/en/3.1.x/templates/#list-of-builtin-filters

Example:

Using the lower filter to convert a country name to lowercase:

https://countryinfoapi.com/api/countries/name/{{ destination.geo.country | lower }}

Explanation: The {{ destination.geo.country | upper }} template converts the country name value from the security alert to lowercase because this specific API is case-sensitive and only accepts lowercase values.

Chaining Multiple Filters

You can apply multiple filters to a single value to achieve more complex transformations. The users can achieve this using a pipe '|' between filters.

    GET https://countryinfoapi.com/api/cities/name/
    {
        "body": {
            "city": "{{ destination.geo.city | upper | replace(' ', '_') }}"
        }
    }

Explanation: The {{ destination.geo.city | upper | replace(' ', '_') }} template first converts the city value to uppercase using the upper filter, and then replaces any spaces with underscores using the replace filter. In our case, "Ciudad de Mexico" will become "CIUDAD_DE_MEXICO".

We also provide a few Custom Filters for handling date and time values:

to_date - turns a timestamp value to a datetime value
timedelta - allows the user to modify a date by adding time units to it
isoformat - formats a date value to the ISO8601_FORMAT

Example:

{
  "body": {
      "ts": "{{ values['@timestamp'] | isoformat }}"
  }
}

Using IF Statements

Conditionals allow for more complex logic within your templates.

Example:

Including additional fields based on the severity of an alert:

    GET /api/fetch-alert
    {
        "body": {
            "alert_id": "{{ alert_id }}",
            "severity": "{{ values.severity }}",
            {% if values.severity == 'high' %}
            "priority": "high",
            "urgent": true
            {% endif %}
        }
    }

Explanation: The {% if values.severity == 'high' %} condition checks if the severity of the alert is high. If true, additional fields (priority and urgent) are included in the request body. This allows the request to be tailored based on the alert's severity.

Using FOR Loops

Loops can iterate over lists or dictionaries.

Example:

Submitting multiple IP addresses in a single request:

    GET /api/fetch-ips
    {
        "body": {
            "ip_addresses": [
              {% for ip in values['ip_list'] %}
              "{{ ip }}"{% if not loop.last %}, {% endif %}
              {% endfor %}
           ]
        }
    }

Explanation: The {% for ip in values.ip_list %} loop iterates over the list of IP addresses in the ip_list property of the alert. Each IP address is included in the ip_addresses array, with commas separating them. This allows multiple IPs to be submitted in one request.

Splunk Integrations

For Splunk integrations, Jinja templates can be used to construct full Splunk queries dynamically. This allows for the enrichment process to pull in relevant log data or perform complex searches based on the event properties.

Example:

Creating a search query to find events related to a specific IP address from a security alert:

    search index=security_logs sourcetype=firewall_logs src_ip={{ source_ip }}

Explanation: The {{ source_ip }} template dynamically inserts the source IP address from the security alert into the Splunk query. This allows the search to be specific to the IP address related to the alert.

Advanced Jinja Usage in Splunk Queries

Using Filters

Transform values within your Splunk queries.

Example:

Converting a timestamp to a specific format:

    search index=security_logs sourcetype=firewall_logs earliest={{ values['@timestamp'] | dateformat("%Y-%m-%dT%H:%M:%S") }}

Explanation: The {{ values['@timestamp'] | dateformat("%Y-%m-%dT%H:%M:%S") }} template formats the @timestamp property of the event to the specified date format before including it in the Splunk query.

Using IF Statements

Apply conditional logic within your Splunk queries.

Example:

Adjusting the search query based on the severity of an event:

    {% if severity == 'high' %}
    search index=security_logs severity=high
    {% else %}
    search index=security_logs
    {% endif %}

Explanation: The {% if severity == 'high' %} condition checks the severity of the event. If the severity is 'high', the Splunk query includes a severity filter. Otherwise, it performs a general search. This allows the query to be adjusted based on the event's severity.

Using FOR Loops

Iterate over lists to construct complex queries.

Example:

Searching for multiple IP addresses within a single query:

    search index=security_logs sourcetype=firewall_logs
    (
      {% for ip in values['ip_list'] %}
      src_ip={{ ip }}{% if not loop.last %} OR {% endif %}
      {% endfor %}
    )

Explanation: The {% for ip in values.ip_list %} loop iterates over the list of IP addresses in the ip_list property of the alert. Each IP address is included in the Splunk query, separated by OR. This allows the query to search for multiple IPs at once.

Conclusion

Using Jinja Templating in the Gather Phase can be a powerful tool. Users can create dynamic and context aware queries in Splunk or REST API Integrations. By using Jinja filters (built-in and custom), conditionals and loops, you can enrich security alerts with relevant information.

Overview​

REST API Integrations​

Using Jinja Templating for Query Parameters​

Using Jinja Templating for Headers​

Using Jinja Templating in Request Body​

Advanced Jinja Usage​

Using the values dictionary​

Using Filters​

Chaining Multiple Filters​

Using IF Statements​

Using FOR Loops​

Splunk Integrations​

Advanced Jinja Usage in Splunk Queries​

Using Filters​

Using IF Statements​

Using FOR Loops​

Conclusion​

Overview

REST API Integrations

Using Jinja Templating for Query Parameters

Using Jinja Templating for Headers

Using Jinja Templating in Request Body

Advanced Jinja Usage

Using the values dictionary

Using Filters

Chaining Multiple Filters

Using IF Statements

Using FOR Loops

Splunk Integrations

Advanced Jinja Usage in Splunk Queries

Using Filters

Using IF Statements

Using FOR Loops

Conclusion