Incident Management That Builds Knowledge
Track · Post-Mortem · Learn · Improve

1. Define Clear Severity Levels

Why it matters: Without severity definitions, everything becomes "urgent" and nothing is.

Standard severity framework (used by Google, Amazon):

• SEV-1 (Critical): Complete service outage or data loss
  • Example: Production database down, all users affected
  • Response: All hands on deck, exec notification, immediate action
  • Typical frequency: 0-2 per quarter
• SEV-2 (High): Major feature broken or significant user impact
  • Example: Payment processing failing, login issues for 20% of users
  • Response: On-call + backup respond within 15 min
  • Typical frequency: 2-5 per quarter
• SEV-3 (Medium): Minor feature degraded, workaround available
  • Example: Search slower than normal, email notifications delayed
  • Response: On-call investigates during business hours
• SEV-4 (Low): Cosmetic issue, no user impact
  • Example: UI button misaligned, typo in error message
  • Response: Create ticket, fix in normal sprint

2. Establish Incident Roles (ICS Framework)

Why it matters: Chaos happens when everyone is both firefighting and coordinating.

Incident Command System (from Google SRE book):

• Incident Commander (IC):
  • Owns the incident end-to-end
  • Makes final decisions, delegates work
  • Focuses on coordination, NOT fixing
  • "Jake, can you check database? Sarah, keep customers updated."
• Tech Lead:
  • Investigates root cause
  • Implements fixes
  • Updates IC on technical progress
• Communications Lead:
  • Updates status page, customers, stakeholders
  • Shields responders from interruptions
  • Keeps internal team (PM, CEO) informed
• Scribe:
  • Captures timeline in real-time
  • "2:34pm - Deploy started", "2:47pm - Error rate spiked"
  • Makes post-mortem easy later

Key rule: IC should never be typing in terminal. Delegate technical work.

3. Communicate Early & Often

Why it matters: Silence breeds panic. Over-communicate during incidents.

The communication cadence (PagerDuty playbook):

• T+0 (immediately): "We're aware of login issues. Investigating."
• T+15 min: "Still investigating. Auth service showing errors. Team working on it."
• T+30 min: "Identified root cause: database connection pool exhausted. Implementing fix."
• T+45 min: "Fix deployed. Monitoring. Service recovering."
• T+60 min: "Incident resolved. All systems normal. Post-mortem will follow."

Where to communicate:

• External: Status page (for customers)
• Internal: Dedicated Slack channel (#incident-2024-02-18)
• Stakeholders: Direct updates to VP Eng, CEO, PM

Template to use: What's happening + Impact + What we're doing + ETA (if known)

4. Write Blameless Post-Mortems

Why it matters: Blame = people hide problems. Blamelessness = learning culture.

The blameless principle (from Etsy, Google):

• Never: "Jake broke production by deploying without testing"
• Instead: "Deploy process allowed untested code to reach prod - need automated gates"

Post-mortem structure:

• Summary: One-line description
• Impact: Users affected, duration, revenue lost
• Timeline: Minute-by-minute of what happened
• Root cause: 5 Whys analysis (go deep, not surface)
• What went well: Celebrate good response
• What went wrong: Gaps in systems, processes
• Action items: Concrete improvements with owners

Timing: Draft within 24 hours while fresh. Finalize within 72 hours.

5. Track Action Items to Completion

Why it matters: 73% of incident action items never get done. This guarantees repeat incidents.

The accountability system:

• Create action items during post-mortem:
  • "Add database connection pool monitoring" - Jake (DevOps) - Due: 2 weeks
  • "Create pre-deploy checklist" - Sarah (Eng Lead) - Due: 1 week
  • "Conduct load testing training" - Marcus - Due: 1 month
• Track in central place: Not in Google Doc that gets forgotten
• Weekly review: Team lead checks status in weekly meeting
• Mark completed with evidence: "Monitoring added - dashboard link here"

The metric: Track "% of action items completed within 30 days". Aim for 90%+.

6. Analyze Trends & Patterns

Why it matters: Individual incidents are events. Patterns reveal systemic issues.

Metrics to track (from Google SRE):

• Incident volume: "23 incidents this quarter (was 28 last quarter)" = improving
• MTTR (Mean Time To Resolve): "Average 34 min (was 56 min)" = faster response
• SEV-1 frequency: "2 SEV-1s this quarter (should be 0-1)" = need focus
• Repeat incidents: "Database timeout happened 3 times" = underlying problem
• Root cause categories: "60% deployment issues, 25% infrastructure, 15% external"

Pattern detection questions:

• Same component failing repeatedly?
• Incidents cluster around deployments?
• Certain engineers always involved (knowledge silo)?
• Incidents spike at certain times (traffic patterns)?

Use quarterly: Review trends, identify top 3 systemic improvements