Skip to main content

Confidence in decisions

In order to build trust in the model, Arcanna provides both a confidence score and an outlier flag for each decision. These concepts and their usages are explained below.

Confidence score​

Definition​

The Confidence Score is a statistical measure that represents the likelihood (expressed as a percentage between 0% and 100%) that Arcanna's decision for a given event is correct. This score is based on the similarity between the current event and known events that have received the same decision.

Interpretation​

  • 0-50%: Low Confidence - The event is significantly different from known events with the same decision, suggesting a potentially wrong decision.
  • 51-75%: Medium Confidence - The event shares some similarities with known events but has notable differences, or there are not many similar events with the same decision, given the total size of the knowledge base.
  • 76-100%: High Confidence - The event is very similar to known events with the same decision, indicating a high likelihood of a correct decision.

Usage​

The Confidence Score helps users to:

  • Assess the reliability of the decision.
  • Prioritize events for review, focusing on those with lower confidence scores.
  • Improve the model by examining and refining the decision on low-confidence events.

Outliers​

Definition​

Outliers are events that are significantly different from the known events in the training dataset, and may indicate anomalies or novel scenarios not previously encountered by the model.

Handling Outliers​

  • Manual Review: Outliers are flagged for human review to determine their nature and appropriate decision.
  • Model Update: If the outliers represent a new valid pattern, they can be included in the training data through feedback, in order to improve the model's performance.