When things break in production — and they will — the difference between a 5-minute recovery and a 5-hour outage comes down to preparation. This guide covers incident response frameworks, on-call practices, and postmortem culture.
Incident Severity Levels
Define severity levels so everyone knows how to respond:
| Level | Name | Description | Response Time | Example |
|---|---|---|---|---|
| SEV1 | Critical | Complete service outage | < 15 min | Site is down, all users affected |
| SEV2 | Major | Significant degradation | < 30 min | Payment processing failing |
| SEV3 | Minor | Partial impact, workaround exists | < 2 hours | Search feature slow |
| SEV4 | Low | Cosmetic or minor issue | Next business day | Dashboard chart broken |
The Incident Response Process
1. Detection
Incidents are detected through:
- Monitoring alerts (Prometheus/Alertmanager, PagerDuty, Datadog)
- Customer reports (support tickets, social media)
- Automated health checks (synthetic monitoring, canary deploys)
The goal is to detect before customers do. Invest in:
yaml
# Alertmanager rule example
groups:
- name: slo-alerts
rules:
- alert: HighErrorRate
expr: |
sum(rate(http_requests_total{status=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m]))
> 0.01
for: 5m
labels:
severity: critical
annotations:
summary: "Error rate exceeds 1% SLO"
runbook: "https://wiki.internal/runbooks/high-error-rate"
2. Triage
When an alert fires:
- Acknowledge the alert (stops escalation)
- Assess severity using the table above
- Open an incident channel (e.g., Slack
#incident-2026-04-12) - Assign roles: Incident Commander (IC), Communications Lead, Technical Lead
3. Mitigation
Focus on restoring service first, not finding root cause:
bash
# Common rapid mitigations
# Rollback to last known good deployment
helm rollback my-app 42
# Scale up to handle load
kubectl scale deployment api --replicas=10
# Toggle feature flag
curl -X POST https://api.internal/flags/new-feature -d '{"enabled": false}'
# Restart problematic pods
kubectl rollout restart deployment/api
# Failover to secondary database
# (follow your runbook)
The best mitigations are pre-documented in runbooks.
4. Resolution
Once the service is stable:
- Verify recovery with metrics and customer reports
- Stand down the incident team
- Schedule a postmortem within 48 hours
Runbooks
Every alert should link to a runbook. A good runbook contains:
markdown
# High Error Rate Runbook
## Alert
HighErrorRate: Error rate exceeds 1% SLO
## Impact
Users experiencing 5xx errors on API requests.
## Diagnosis Steps
1. Check which endpoints are failing:
\`kubectl logs -l app=api --tail=100 | grep "500"\`
2. Check database connectivity:
\`kubectl exec -it api-pod -- pg_isready -h db-host\`
3. Check resource usage:
\`kubectl top pods -l app=api\`
## Mitigation Steps
1. If recent deploy: \`helm rollback my-app <previous-revision>\`
2. If database issue: failover to replica (see DB runbook)
3. If traffic spike: scale up \`kubectl scale deployment api --replicas=10\`
## Escalation
- Database team: @db-oncall in #db-support
- Platform team: @platform-oncall in #platform
Alerting Best Practices
- Alert on symptoms, not causes — alert on high error rate, not high CPU
- Every alert must be actionable — if you can't act on it, it's noise
- Use multiple severity levels — page for SEV1/2, Slack for SEV3/4
- Set appropriate thresholds — avoid alert fatigue from noisy alerts
- Include runbook links in every alert annotation
Alerting Anti-Patterns
| Anti-Pattern | Problem | Fix |
|---|---|---|
| Too many alerts | On-call fatigue, alerts get ignored | Reduce to actionable alerts only |
| No runbooks | Slow response, knowledge silos | Write runbook for every alert |
| Alerting on causes | False positives (high CPU is OK during deploys) | Alert on user-facing symptoms |
| Same threshold everywhere | Dev alerts wake you at 3 AM | Per-environment alert routing |
Postmortems
Postmortems are blameless reviews of what happened and how to prevent recurrence.
Template
markdown
# Incident Postmortem: [Title]
**Date:** 2026-04-12
**Duration:** 47 minutes
**Severity:** SEV2
**Author:** [Name]
## Summary
Brief description of what happened and impact.
## Timeline
- 14:32 — Alert fires: HighErrorRate
- 14:35 — On-call acknowledges, opens #incident channel
- 14:42 — Root cause identified: bad database migration
- 14:48 — Migration rolled back
- 15:02 — Error rate returns to normal
- 15:19 — Incident closed
## Root Cause
A database migration added a NOT NULL column without a default value,
causing INSERT failures for the API.
## What Went Well
- Alert fired within 3 minutes of the issue
- Runbook was up to date and helped with diagnosis
## What Went Poorly
- Migration was not tested against production-like data
- Rollback took 6 minutes because the process was manual
## Action Items
- [ ] Add migration testing to CI pipeline (@alice, due 2026-04-19)
- [ ] Automate migration rollback in deploy script (@bob, due 2026-04-26)
- [ ] Add pre-deploy database backup step (@alice, due 2026-04-19)
Postmortem Culture
- Blameless — focus on systems and processes, not individuals
- Mandatory for SEV1 and SEV2 incidents
- Shared widely — publish to the team or organization
- Track action items — unfinished items mean the same incident will repeat
On-Call Rotation
A healthy on-call setup includes:
- Rotation schedule — weekly or biweekly, with handoff meetings
- Primary and secondary — backup in case primary is unreachable
- Escalation policy — auto-escalate if not acknowledged within 15 minutes
- Compensation — on-call pay or time off in lieu
- Quiet hours tooling — route low-severity alerts to Slack, not pager
- On-call handoff document — current known issues, recent deploys, things to watch
Tools
| Category | Tools |
|---|---|
| Alerting | PagerDuty, Opsgenie, Grafana OnCall |
| Monitoring | Prometheus, Grafana, Datadog |
| Incident Management | incident.io, Rootly, FireHydrant |
| Communication | Slack, Microsoft Teams |
| Status Pages | Statuspage.io, Cachet, Instatus |
Good incident response is a skill you practice before you need it. Run game days, rehearse your runbooks, and continuously improve your process with every postmortem.
Tagged with
Enjoyed this article?
Get more DevOps insights delivered to your inbox.
Get new posts by email
Subscribe to get an email when a new blog post is published. Skip anytime.
No spam, unsubscribe anytime.
N
Related Posts
Discussion
0 comments
Sign in to join the conversation.
Be the first to comment
Start a conversation about this post