Alerting
配置 meaningful alerts, escalation policies, and on-call rotations for incident 响应.
你将学到
- 什么制作好 alert 对比 alert fatigue
- 设置 thresholds and SLO-based alerts
- Escalation policies and on-call rotations
- Tools: PagerDuty, Opsgenie, Alertmanager
概念讲解
好的告警 vs 告警疲劳
A good alert is actionable — there is something you can do to fix it. It is specific — it tells you what is wrong and where. It is urgent — it matters right now, not tomorrow. A bad alert fires for transient blips, cannot be acted on, or duplicates information. When engineers receive too many bad alerts, they develop alert fatigue — they start ignoring alerts, including the important ones. This is dangerous. Every alert should be reviewed regularly and silenced or removed if it is not useful.
SLO-Based Alerting
Service Level Objectives (SLOs) define your reliability targets — for example, "99.9% of requests succeed within 200ms over 30 days." SLO-based alerts fire when you are at risk of violating the SLO, not when a threshold is crossed. This approach is more principled than threshold-based alerting because it directly measures user impact. Calculate the error budget — the allowed failures within the SLO — and alert when the budget is being consumed too quickly.
Escalation and On-Call
When an alert fires, someone needs to respond. On-call rotations ensure 24/7 coverage by cycling through team members. Escalation policies define what happens if the primary on-call doesn't respond — the alert goes to a secondary, then a manager. Tools like PagerDuty, Opsgenie, and Alertmanager manage these workflows. A good escalation policy gives the on-call engineer enough time to investigate before escalating, but not so much that users suffer.
Alert Routing and Deduplication
In a complex system, many alerts may fire from a single root cause. Deduplication groups related alerts so the on-call engineer gets one notification, not fifty. Routing sends alerts to the right team — database alerts to the DBA team, frontend alerts to the web team. Inhibition suppresses lower-priority alerts when a higher-priority one is active. These features reduce noise and help engineers focus on the actual problem.
示例代码
# Prometheus Alertmanager configuration
route:
group_by: ['alertname', 'cluster']
group_wait: 30s
group_interval: 5m
repeat_interval: 4h
receiver: 'team-default'
routes:
- matchers: ['severity="critical"']
receiver: 'pagerduty-critical'
- matchers: ['team="database"']
receiver: 'dba-team'
receivers:
- name: 'pagerduty-critical'
pagerduty_configs:
- service_key: 'YOUR_KEY'
- name: 'team-default'
slack_configs:
- channel: '#alerts'
api_url: 'https://hooks.slack.com/services/...'
Alertmanager routes 关键 alerts 到 PagerDuty and 数据库 alerts 到 DBA team. 分组 deduplicates related alerts, and repeat_interval prevents notification spam.
动手试一试
- Cron Parser
Parse alert evaluation schedules.
- Crontab Generator
Generate on-call rotation schedules.
- HTTP 状态 Codes
引用 for HTTP 错误-based alert 规则.
常见错误
错误写法
Alerting on CPU or memory thresholds
正确做法
Alert on 用户-facing symptoms 如错误速率 and latency 相反. CPU at 85% 可能是 fine 如果系统是 handling 加载 well. Symptom-based alerts reduce 假 positives.
错误写法
否 deduplication for related alerts
正确做法
配置分组 in Alertmanager 所以 related alerts 从单个 incident 是组合到 one notification. 不使用这个, 单个 outage 能 generate hundreds 的 pages.
练习 Alerting
1 个练习
查阅不认识的术语
新手友好的术语表用大白话解释编程术语——变量、函数、DOM、Promise 等等。
动手做项目
通过引导式项目应用所学,含起始代码与解决方案。
解读错误信息
用大白话解释常见错误——什么意思、为什么发生、如何修复。