Skip to content

Alerting

配置 meaningful alerts, escalation policies, and on-call rotations for incident 响应.

你将学到

  • 什么制作好 alert 对比 alert fatigue
  • 设置 thresholds and SLO-based alerts
  • Escalation policies and on-call rotations
  • Tools: PagerDuty, Opsgenie, Alertmanager

概念讲解

好的告警 vs 告警疲劳

A good alert is actionable — there is something you can do to fix it. It is specific — it tells you what is wrong and where. It is urgent — it matters right now, not tomorrow. A bad alert fires for transient blips, cannot be acted on, or duplicates information. When engineers receive too many bad alerts, they develop alert fatigue — they start ignoring alerts, including the important ones. This is dangerous. Every alert should be reviewed regularly and silenced or removed if it is not useful.

SLO-Based Alerting

Service Level Objectives (SLOs) define your reliability targets — for example, "99.9% of requests succeed within 200ms over 30 days." SLO-based alerts fire when you are at risk of violating the SLO, not when a threshold is crossed. This approach is more principled than threshold-based alerting because it directly measures user impact. Calculate the error budget — the allowed failures within the SLO — and alert when the budget is being consumed too quickly.

Escalation and On-Call

When an alert fires, someone needs to respond. On-call rotations ensure 24/7 coverage by cycling through team members. Escalation policies define what happens if the primary on-call doesn't respond — the alert goes to a secondary, then a manager. Tools like PagerDuty, Opsgenie, and Alertmanager manage these workflows. A good escalation policy gives the on-call engineer enough time to investigate before escalating, but not so much that users suffer.

Alert Routing and Deduplication

In a complex system, many alerts may fire from a single root cause. Deduplication groups related alerts so the on-call engineer gets one notification, not fifty. Routing sends alerts to the right team — database alerts to the DBA team, frontend alerts to the web team. Inhibition suppresses lower-priority alerts when a higher-priority one is active. These features reduce noise and help engineers focus on the actual problem.

示例代码

              # Prometheus Alertmanager configuration
route:
  group_by: ['alertname', 'cluster']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: 'team-default'
  routes:
    - matchers: ['severity="critical"']
      receiver: 'pagerduty-critical'
    - matchers: ['team="database"']
      receiver: 'dba-team'

receivers:
  - name: 'pagerduty-critical'
    pagerduty_configs:
      - service_key: 'YOUR_KEY'
  - name: 'team-default'
    slack_configs:
      - channel: '#alerts'
        api_url: 'https://hooks.slack.com/services/...'
            

Alertmanager routes 关键 alerts 到 PagerDuty and 数据库 alerts 到 DBA team. 分组 deduplicates related alerts, and repeat_interval prevents notification spam.

动手试一试

常见错误

错误写法

Alerting on CPU or memory thresholds

正确做法

Alert on 用户-facing symptoms 如错误速率 and latency 相反. CPU at 85% 可能是 fine 如果系统是 handling 加载 well. Symptom-based alerts reduce 假 positives.

错误写法

否 deduplication for related alerts

正确做法

配置分组 in Alertmanager 所以 related alerts 从单个 incident 是组合到 one notification. 不使用这个, 单个 outage 能 generate hundreds 的 pages.

练习 Alerting

1 个练习

练习 Alerting

查阅不认识的术语

新手友好的术语表用大白话解释编程术语——变量、函数、DOM、Promise 等等。

查看术语表

动手做项目

通过引导式项目应用所学,含起始代码与解决方案。

查看项目

解读错误信息

用大白话解释常见错误——什么意思、为什么发生、如何修复。

查看错误