Incident Response
Handle production incidents with a structured process: detect, respond, resolve, and learn.
What you'll learn
- The incident response lifecycle
- Roles: incident commander, scribe, responder
- Communication during incidents
- Post-mortems and blameless culture
Concept
The Incident Lifecycle
A production incident is an event that degrades service — an outage, a slowdown, a data loss. Incident response follows a lifecycle: detect (monitoring alerts you), respond (acknowledge and assess severity), mitigate (stop the bleeding), resolve (fix the root cause), and learn (post-mortem). The goal of the response phase is not to find the root cause — it is to restore service. Root cause analysis happens after the system is healthy.
Roles
Effective incident response has clear roles. The incident commander (IC) coordinates the response and makes decisions — they don't fix things, they direct others. The scribe records a timeline of actions and discoveries. Responders investigate and apply fixes. The communications lead updates stakeholders and users. One person can fill multiple roles for small incidents, but for major outages, these roles should be separate. The IC has authority to pull in anyone needed.
Communication
Communication during an incident is critical. Use a dedicated incident channel in Slack or Teams for real-time coordination. Post regular updates — at least every 30 minutes for major incidents. Update the status page for user-facing issues. Be honest: "We are investigating elevated error rates" is better than silence. After resolution, notify stakeholders that the incident is over. Keep all communication factual — no speculation about causes during the incident.
Post-Mortems
After every incident, write a post-mortem — a document that describes what happened, why, and how to prevent it. Post-mortems must be blameless: focus on systems and processes, not individuals. Ask "what enabled this failure?" not "who caused it?" Create action items with owners and deadlines. Track action items to completion. Share post-mortems widely so the whole organization learns. A culture that punishes individuals for incidents leads to hidden problems and worse reliability.
Incident Severity
Define clear severity levels: SEV1 is a complete outage affecting all users — page immediately, assemble the team. SEV2 is a major degradation affecting many users — page the on-call. SEV3 is a minor issue or workaround available — notify during business hours. SEV4 is a cosmetic or non-urgent issue — create a ticket. Clear severity definitions prevent over-paging for minor issues and under-paging for major ones. Document the criteria so everyone can classify incidents consistently under pressure.
Example
# Incident response quick-reference
# 1. ACKNOWLEDGE the alert
# - PagerDuty/Opsgenie: acknowledge the incident
# - Create an incident channel: #incident-2024-01-15
# 2. ASSESS severity and impact
# - Who is affected? How many users?
# - Is data being lost? Is it revenue-impacting?
# 3. MITIGATE — restore service first
kubectl rollout undo deployment/myapp # rollback recent deploy
docker service scale myapp=10 # scale up to handle load
# 4. INVESTIGATE — find root cause
kubectl logs --since=30m deployment/myapp | grep ERROR
docker logs --since=30m myapp 2>&1 | tail -100
# 5. RESOLVE and write a post-mortem
# - What happened? Timeline? Root cause? Action items?
During an incident: acknowledge, assess, mitigate (rollback or scale), investigate (logs), then resolve and write a blameless post-mortem with action items.
Try it
- HTTP Status Codes
Identify error codes during incidents.
- Timestamp Converter
Convert incident timeline timestamps.
- Cron Parser
Parse alert schedules for on-call rotations.
Common mistakes
The mistake
Trying to find root cause before restoring service
The fix
Mitigate first, investigate second. Roll back, scale up, or redirect traffic to restore service. Then investigate the root cause with the system stable.
The mistake
Blame-focused post-mortems
The fix
Post-mortems must be blameless. People don't cause incidents — systems and processes do. Focus on what enabled the failure and how to prevent it, not who clicked the button.
Practice Incident Response
1 exercise
Look up unfamiliar terms
A beginner-friendly glossary explains jargon in plain English — variables, functions, DOM, Promise, and more.
Build real projects
Apply what you learned by building guided projects with starter code and solutions.
Decode error messages
Plain-English explanations of common errors — what they mean, why they happen, and how to fix them.