Skip to main content

Model Metrics

POST /api/v2/metrics/model​

Description​

The model endpoint retrieves detailed performance metrics and statistics for a specific Arcanna model. This endpoint provides comprehensive evaluation data including accuracy metrics, confusion matrix, knowledge base statistics, and model-specific performance indicators.

Quick Start Examples​

Get metrics for a specific model​

curl -X GET "https://your.arcanna.ai/api/v2/metrics/model?job_id=1234&model_id=model_v2.1.0" \
  -H "X-Arcanna-Api-Key: your-api-key-here"

Get metrics using job name instead of ID​

curl -X GET "https://your.arcanna.ai/api/v2/metrics/model?job_id=security-alerts&model_id=model_v2.1.0" \
  -H "X-Arcanna-Api-Key: your-api-key-here"

Request Parameters​

Query Parameters​

ParameterTypeRequiredDescription
job_idinteger or stringYesThe ID or name of the job associated with the model
model_idstringYesThe unique identifier of the model to retrieve metrics for

Response​

Success Response (200 OK)​

{
  "confusion_matrix": [
    [4490, 0, 0],
    [0, 749, 0],
    [0, 0, 37]
  ],
  "overall_accuracy": 1.0,
  "overall_f1_score": 1.0,
  "overall_recall": 1.0,
  "overall_precision": 1.0,
  "metrics_per_decision": {
    "Drop": {
      "precision": 1.0,
      "recall": 1.0,
      "f1_score": 1.0,
      "true_positives": 4490,
      "true_negatives": 786,
      "false_positives": 0,
      "false_negatives": 0
    },
    "Investigate": {
      "precision": 1.0,
      "recall": 1.0,
      "f1_score": 1.0,
      "true_positives": 749,
      "true_negatives": 4527,
      "false_positives": 0,
      "false_negatives": 0
    },
    "Escalate": {
      "precision": 1.0,
      "recall": 1.0,
      "f1_score": 1.0,
      "true_positives": 37,
      "true_negatives": 5239,
      "false_positives": 0,
      "false_negatives": 0
    }
  },
  "model_id": "model_v2.1.0",
  "is_recomputing_metrics": false,
  "last_recomputed_timestamp": "2025-01-15T10:30:00Z",
  "kb_count_per_decision": {
    "Drop": {
      "alerts_count": 2066,
      "buckets_count": 10
    },
    "Investigate": {
      "alerts_count": 429,
      "buckets_count": 5
    },
    "Escalate": {
      "alerts_count": 21,
      "buckets_count": 4
    }
  },
  "buckets_in_kb": 19,
  "events_in_kb": 2516
}

Error Response (404 Not Found)​

{
  "error": "Model not found"
}

Error Response (422 Validation Error)​

{
  "detail": [
    {
      "loc": ["query", "model_id"],
      "msg": "Model ID is required",
      "type": "value_error"
    }
  ]
}

Response Fields​

Core Performance Metrics​

  • confusion_matrix: 2D array representing the confusion matrix of model decisions
  • overall_accuracy: Mean accuracy across all decisions (0-1)
  • overall_f1_score: Mean F1 score across all decisions (0-1)
  • overall_recall: Mean recall across all decisions (0-1)
  • overall_precision: Mean precision across all decisions (0-1)

Decision-Level Metrics​

  • metrics_per_decision: Object containing detailed metrics for each decision type:
    • precision: Precision score for this decision type (0-1)
    • recall: Recall score for this decision type (0-1)
    • f1_score: F1 score for this decision type (0-1)
    • true_positives: Number of true positive predictions
    • true_negatives: Number of true negative predictions
    • false_positives: Number of false positive predictions
    • false_negatives: Number of false negative predictions

Model Information​

  • model_id: The unique identifier of the model
  • is_recomputing_metrics: Boolean indicating if metrics are currently being recomputed
  • last_recomputed_timestamp: Timestamp of the last metrics recomputation

Knowledge Base Statistics​

  • kb_count_per_decision: Object containing knowledge base counts for each decision type:
    • alerts_count: Number of alerts in the knowledge base for this decision
    • buckets_count: Number of buckets in the knowledge base for this decision
  • buckets_in_kb: Total number of buckets in the knowledge base
  • events_in_kb: Total number of events in the knowledge base

Use Cases​

  • Evaluate specific model performance before deployment
  • Compare performance between different model versions
  • Monitor model degradation over time
  • Analyze knowledge base composition and balance
  • Generate model-specific performance reports
  • Validate model training effectiveness