Alerting
Configure meaningful alerts, escalation policies, and on-call rotations for incident response.
What you'll learn
- What makes a good alert vs alert fatigue
- Setting thresholds and SLO-based alerts
- Escalation policies and on-call rotations
- Tools: PagerDuty, Opsgenie, Alertmanager
Concept
Good Alerts vs Alert Fatigue
A good alert is actionable — there is something you can do to fix it. It is specific — it tells you what is wrong and where. It is urgent — it matters right now, not tomorrow. A bad alert fires for transient blips, cannot be acted on, or duplicates information. When engineers receive too many bad alerts, they develop alert fatigue — they start ignoring alerts, including the important ones. This is dangerous. Every alert should be reviewed regularly and silenced or removed if it is not useful.
SLO-Based Alerting
Service Level Objectives (SLOs) define your reliability targets — for example, "99.9% of requests succeed within 200ms over 30 days." SLO-based alerts fire when you are at risk of violating the SLO, not when a threshold is crossed. This approach is more principled than threshold-based alerting because it directly measures user impact. Calculate the error budget — the allowed failures within the SLO — and alert when the budget is being consumed too quickly.
Escalation and On-Call
When an alert fires, someone needs to respond. On-call rotations ensure 24/7 coverage by cycling through team members. Escalation policies define what happens if the primary on-call doesn't respond — the alert goes to a secondary, then a manager. Tools like PagerDuty, Opsgenie, and Alertmanager manage these workflows. A good escalation policy gives the on-call engineer enough time to investigate before escalating, but not so much that users suffer.
Alert Routing and Deduplication
In a complex system, many alerts may fire from a single root cause. Deduplication groups related alerts so the on-call engineer gets one notification, not fifty. Routing sends alerts to the right team — database alerts to the DBA team, frontend alerts to the web team. Inhibition suppresses lower-priority alerts when a higher-priority one is active. These features reduce noise and help engineers focus on the actual problem.
Example
# Prometheus Alertmanager configuration
route:
group_by: ['alertname', 'cluster']
group_wait: 30s
group_interval: 5m
repeat_interval: 4h
receiver: 'team-default'
routes:
- matchers: ['severity="critical"']
receiver: 'pagerduty-critical'
- matchers: ['team="database"']
receiver: 'dba-team'
receivers:
- name: 'pagerduty-critical'
pagerduty_configs:
- service_key: 'YOUR_KEY'
- name: 'team-default'
slack_configs:
- channel: '#alerts'
api_url: 'https://hooks.slack.com/services/...'
Alertmanager routes critical alerts to PagerDuty and database alerts to the DBA team. Grouping deduplicates related alerts, and repeat_interval prevents notification spam.
Try it
- Cron Parser
Parse alert evaluation schedules.
- Crontab Generator
Generate on-call rotation schedules.
- HTTP Status Codes
Reference for HTTP error-based alert rules.
Common mistakes
The mistake
Alerting on CPU or memory thresholds
The fix
Alert on user-facing symptoms like error rate and latency instead. CPU at 85% might be fine if the system is handling load well. Symptom-based alerts reduce false positives.
The mistake
No deduplication for related alerts
The fix
Configure grouping in Alertmanager so related alerts from a single incident are combined into one notification. Without this, a single outage can generate hundreds of pages.
Practice Alerting
1 exercise
Look up unfamiliar terms
A beginner-friendly glossary explains jargon in plain English — variables, functions, DOM, Promise, and more.
Build real projects
Apply what you learned by building guided projects with starter code and solutions.
Decode error messages
Plain-English explanations of common errors — what they mean, why they happen, and how to fix them.