Monitoring Guide¶
Last Updated: 2026-07-11 Version: v0.2.0
**Last Updated: 2026-06-22
Grafana Dashboards¶
Setup¶
# Start Grafana
docker run -d -p 3000:3000 grafana/grafana
# Import dashboard
-H "Content-Type: application/json" \
-d @configs/grafana/dashboards/ml_operations.json
Available Dashboards¶
- ML Operations - Model performance, requests, errors
- Drift Detection - Data/config/model drift trends
- Resource Utilization - GPU, CPU, memory
- Deployment History - Version timeline, rollbacks
AlertManager¶
Configuration¶
Alert routing is defined in configs/alertmanager/alertmanager.yml.
Severity Routing¶
- Critical PagerDuty (immediate page)
- High Slack #ml-ops-critical
- Medium Slack #ml-ops-warnings
- Low Email
Custom Alerts¶
Add to prometheus_rules.yml:
groups:
- name: ml_alerts
rules:
- alert: ModelAccuracyDrop
expr: model_accuracy < 0.90
for: 5m
labels:
severity: high
annotations:
description: "Model accuracy below 90%"
Prometheus Metrics¶
Exposed at /metrics endpoint:
model_accuracy- Current model accuracyhttp_requests_total- Request counterhttp_errors_total- Error counteractive_models- Number of active models
See src/codex_ml/monitoring/metrics.py for full list.