Skip to main content

Confidence in decisions

In order to build trust in the model, Arcanna provides both a confidence score and an outlier flag for each decision. These concepts and their usages are explained below.

Confidence score​

Definition​

The Confidence Score is a percentage (from 0% to 100%) that shows how certain Arcanna is about its decision for a specific event. It’s calculated by comparing the event to known past events that received the same decision. If most similar past events had the same decision, the confidence score is high. If only a few did, the score is low.

Interpretation​

  • 0-59%: Low Confidence - The event is significantly different from known events with the same decision, suggesting a potentially wrong decision.
  • 60-100%: High Confidence - The event is very similar to known events with the same decision, indicating a high likelihood of a correct decision.

Usage​

The Confidence Score helps users to:

  • Assess the reliability of the decision.
  • Prioritize events for review, focusing on those with lower confidence scores.
  • Improve the model by examining and refining the decision on low-confidence events.

Outliers​

Definition​

Outliers are events that are significantly different from the known events in the training dataset, and may indicate anomalies or novel scenarios not previously encountered by the model.

Handling Outliers​

  • Manual Review: Outliers are flagged for human review to determine their nature and appropriate decision.
  • Model Update: If the outliers represent a new valid pattern, they can be included in the training data through feedback, in order to improve the model's performance.