Incident 响应
Handle production incidents 使用 structured 处理: detect, respond, resolve, and 学习.
你将学到
- incident 响应 lifecycle
- Roles: incident commander, scribe, responder
- Communication 期间 incidents
- Post-mortems and blameless culture
概念讲解
The Incident Lifecycle
A production incident is an event that degrades service — an outage, a slowdown, a data loss. Incident response follows a lifecycle: detect (monitoring alerts you), respond (acknowledge and assess severity), mitigate (stop the bleeding), resolve (fix the root cause), and learn (post-mortem). The goal of the response phase is not to find the root cause — it is to restore service. Root cause analysis happens after the system is healthy.
Roles
Effective incident response has clear roles. The incident commander (IC) coordinates the response and makes decisions — they don't fix things, they direct others. The scribe records a timeline of actions and discoveries. Responders investigate and apply fixes. The communications lead updates stakeholders and users. One person can fill multiple roles for small incidents, but for major outages, these roles should be separate. The IC has authority to pull in anyone needed.
Communication
Communication during an incident is critical. Use a dedicated incident channel in Slack or Teams for real-time coordination. Post regular updates — at least every 30 minutes for major incidents. Update the status page for user-facing issues. Be honest: "我们是 investigating elevated 错误速率" is better than silence. After resolution, notify stakeholders that the incident is over. Keep all communication factual — no speculation about causes during the incident.
Post-Mortems
After every incident, write a post-mortem — a document that describes what happened, why, and how to prevent it. Post-mortems must be blameless: focus on systems and processes, not individuals. Ask "什么启用这个失败?" not "谁 caused 它?" Create action items with owners and deadlines. Track action items to completion. Share post-mortems widely so the whole organization learns. A culture that punishes individuals for incidents leads to hidden problems and worse reliability.
Incident Severity
Define clear severity levels: SEV1 is a complete outage affecting all users — page immediately, assemble the team. SEV2 is a major degradation affecting many users — page the on-call. SEV3 is a minor issue or workaround available — notify during business hours. SEV4 is a cosmetic or non-urgent issue — create a ticket. Clear severity definitions prevent over-paging for minor issues and under-paging for major ones. Document the criteria so everyone can classify incidents consistently under pressure.
示例代码
# Incident response quick-reference
# 1. ACKNOWLEDGE the alert
# - PagerDuty/Opsgenie: acknowledge the incident
# - Create an incident channel: #incident-2024-01-15
# 2. ASSESS severity and impact
# - Who is affected? How many users?
# - Is data being lost? Is it revenue-impacting?
# 3. MITIGATE — restore service first
kubectl rollout undo deployment/myapp # rollback recent deploy
docker service scale myapp=10 # scale up to handle load
# 4. INVESTIGATE — find root cause
kubectl logs --since=30m deployment/myapp | grep ERROR
docker logs --since=30m myapp 2>&1 | tail -100
# 5. RESOLVE and write a post-mortem
# - What happened? Timeline? Root cause? Action items?
期间 incident: acknowledge, assess, mitigate (rollback or scale), investigate (logs), 然后 resolve and 编写 blameless post-mortem 使用操作项.
动手试一试
- HTTP 状态 Codes
Identify 错误 codes 期间 incidents.
- Timestamp Converter
Convert incident timeline timestamps.
- Cron Parser
Parse alert schedules for on-call rotations.
常见错误
错误写法
Trying 到查找根 cause 之前 restoring service
正确做法
Mitigate 首先, investigate 其次. Roll 后面, scale 上, or redirect traffic 到 restore service. 然后 investigate 根 cause 使用系统 stable.
错误写法
Blame-focused post-mortems
正确做法
Post-mortems 必须是 blameless. People don't cause incidents — 系统 and 处理做. Focus on 什么启用失败 and 如何 prevent 它, 不谁 点击 button.
练习 Incident 响应
1 个练习
查阅不认识的术语
新手友好的术语表用大白话解释编程术语——变量、函数、DOM、Promise 等等。
动手做项目
通过引导式项目应用所学,含起始代码与解决方案。
解读错误信息
用大白话解释常见错误——什么意思、为什么发生、如何修复。