Alerting
In one line
Alerting is the practice of automatically notifying the team when metrics cross thresholds or errors occur.
In simple words
Alerting is how you find out about problems before your users do. You define rules — e.g., "if error rate > 5% for 5 minutes, alert the team" — and when the condition is met, the system sends a notification.
Good alerting is a balance. Too few alerts and you miss critical issues. Too many alerts and you get alert fatigue — the team ignores them all. The best alerts are actionable: when an alert fires, there is a clear action to take.
Alerts are routed through an Alertmanager (like Prometheus Alertmanager) to the right destination — Slack for warnings, PagerDuty for critical issues at 3 AM, email for informational alerts. Alerts should include enough context (links to dashboards, logs) to help diagnose the problem.
Example
# Alertmanager routing
route:
receiver: 'slack'
routes:
- match:
severity: critical
receiver: 'pagerduty'
receivers:
- name: 'slack'
slack_configs:
- api_url: 'https://hooks.slack.com/...'
- name: 'pagerduty'
pagerduty_configs:
- service_key: 'your-key'This routes critical alerts to PagerDuty and all others to Slack.
How it works
- You define alert rules (e.g., error rate > 5% for 5 min).
- The monitoring system evaluates rules continuously.
- When a rule fires, it sends an alert to the Alertmanager.
- The Alertmanager routes the alert to the right channel (Slack, PagerDuty, email).
Common confusions
Confused with: Monitoring
The difference: Monitoring collects and displays metrics; alerting is the action taken when metrics indicate a problem. Alerting is a part of monitoring.
Related terms
Decode error messages
Plain-English explanations of common errors — what they mean, why they happen, and how to fix them.