Consensus

Definition

By textbook, consensus means general agreement in opinion reached by a group.

Due to the fact that individuals have their own experience and expertise, different conclusions might be reached on the same decision points, by different analysts. Having different analysts validate events with the same decision points, and reaching different conclusions, leads to data inconsistencies that affect the performance and the models' ability to learn.

Therefore, having a single analyst validate events and their decisions leads to having a model trained to decide like said analyst.

To account for multiple analysts giving feedback and ensure that the learning process is not affected by conflicts of opinion, we have integrated a consensus approach in Arcanna. Multiple analysts can give feedback on the same events, and only when consensus is reached, the events will be used to train the AI models.

There are various techniques for applying consensus; Arcanna uses majority voting for computing the final decisions, with the ability for admin users to override the final decision. Events on which consensus is not reached are marked as "Undecided" and flagged in Arcanna for further treatment.

Each bucket displays the consensus result, who contributed to it, and for administrative users the ability to override the decision, directly on the Feedback Page.