Your First Week On Call: An SRE Incident Response Guide

Your phone goes off at 02:14. The alert says HighErrorRate — checkout-api. You have been on the rotation for three days, you did not write this service, and you do not know what it talks to.

This is the situation nobody prepares you for, and the advice you usually get — "stay calm, follow the runbook" — assumes a runbook exists and that you can find it at two in the morning. This page is about what to actually do, in order, when you have no idea what is broken.

Who this is for

You are on a rotation for a service you did not write, and you are not sure what you are allowed to do at 02:14. You do not need to know the system. You need a way to find out safely and tell people what you found.

Before your first shift

Do these while you are awake and nothing is on fire. Every one of them is something you will otherwise try to do for the first time during an incident.

Prove your access works. Not "I think I have access" — actually run the commands. Can you reach production logs? Can you get a shell? Can you see the dashboards? An expired token discovered at 02:14 costs fifteen minutes you do not have.

Find out how to escalate, and to whom. Write the name and the number somewhere you can reach from your phone. "Who do I wake up" is not a question to research during an incident.

Ask what broke most recently. Whoever held the pager before you knows. Three sentences from them is worth more than an hour of reading.

Find out how to roll back. One command, written down. In a large fraction of incidents this is the entire fix, and knowing it is a different skill from knowing the system.

The first five minutes

The instinct when a page arrives is to start fixing. That instinct is wrong and it is worth actively resisting, because a fix applied to a misunderstood problem usually makes the next twenty minutes harder.

Do these instead, in this order.

1. Establish whether it is real

Some alerts fire on a symptom users would notice. Others fire on a cause that may or may not matter. A CPU alert at 90% is not an incident if every request is still being served in time.

The question is: is a person having a bad time right now? If you cannot answer that from the alert, find the one dashboard or endpoint that tells you. If nobody is affected, you have a problem to fix in the morning, not an incident.

This distinction is worth understanding properly before you are on call — symptom vs cause alerting covers why most pages are not worth waking for, and what to do about the ones that are.

2. Ask what changed

Systems that ran fine for six months rarely break spontaneously at 02:14. Something changed. In rough order of likelihood:

A deploy went out
A config or feature flag was toggled
A dependency you do not own changed or degraded
Something crossed a threshold that had been climbing for weeks — a disk, a certificate, a connection pool

Check the deploy timeline first. If the error rate turns upward within minutes of a release, you have your answer, and the fix is a rollback rather than a diagnosis.

3. Decide: is it getting worse?

This changes everything about your next move.

Getting worse means stop investigating and mitigate. Roll back, scale up, shed load, fail over — whatever stops the bleeding. You can find out why afterwards. A correct diagnosis delivered after the outage ended is worth much less than a crude fix delivered during it.

Stable means you have time to understand it before acting, and you should use that time. Acting under time pressure you do not actually have is how a single failure becomes two.

Then: read the symptom

Once you know it is real and roughly how urgent it is, the job becomes ordinary debugging. The useful move is to name the symptom precisely, because named failures have finite, documented causes.

Here is where to look, by what you are seeing. Each of these has a page with the causes ranked and, for each one, the command that confirms or rules it out.

A container or pod will not start or stay up

CrashLoopBackOff — it starts, dies, and Kubernetes keeps retrying more slowly
OOMKilled — the container hit a memory limit and was killed outright
ImagePullBackOff — the image could not be fetched; usually the name, the credentials or the network, in that order
Pod stuck in Pending — not yet placed on a node, or still fetching its image (also Pending), so there are no logs to read
Container exits immediately — the process finished, and a container lives exactly as long as its main process

Something is returning errors or timing out

502 Bad Gateway — a proxy reached for something behind it and got an answer it could not use, or none
504 Gateway Timeout — a proxy gave up waiting; a statement about a deadline, not about a fault
502 vs 504 — if you are unsure which you have, this is the distinction

If you have a status code and want to know who most likely produced it before you go reading logs, the HTTP status explainer answers that specifically — a 502 from a load balancer and a 502 from your application call for completely different next moves.

Networking and DNS

Containers cannot reach each other — service names, networks, and why localhost is not it
DNS stops resolving — what actually happens in a cluster when it does

Infrastructure and state

Error acquiring the state lock — and why the obvious fix corrupts state if the other run is still alive
Terraform state drift — when the world stopped matching the file
No space left on device — where the disk actually went
Port is already allocated — what is holding it

Pipelines and identity

Not authorized to perform sts:AssumeRoleWithWebIdentity — the error names a permission and the fault is a string comparison
Error: no matching signatures — image signature verification, and why the message is ambiguous
Push declined due to repository rule violations — a secret in a commit, and why deleting the line does not clear it

Whole-system failures

The habit that matters most

Narrate what you are doing, in writing, as you do it.

Post it in the incident channel. Not polished — just a running log:

02:17 error rate on checkout-api started 02:11, deploy went out 02:09
02:19 rolling back to previous revision
02:24 errors dropping, 12% -> 3%
02:31 back to baseline, staying on for 15 min

This costs almost nothing and pays three times over. It tells whoever joins what you already know, so they do not repeat it. It stops you re-checking the same thing twice at 3am, which you will otherwise do. And it means the write-up afterwards is mostly assembled already, by the person who actually knew what was happening.

The version of this that goes wrong is silence for forty minutes followed by "fixed it". Nobody learns anything, including you.

Escalating is not failure

This is the part people get wrong for the longest, and it is worth being blunt about.

You are not on call because you know everything. You are on call because somebody needs to look at it within minutes rather than hours. Deciding that somebody else should look at it is doing the job.

Escalate when any of these is true:

Fifteen or twenty minutes have passed and you are no better oriented than when you started
It is getting worse and you do not have a mitigation
It touches something you have no access to, or no authority over
It involves data loss, a security concern, or anything you cannot undo

That last one deserves emphasis. If the fix you are considering is irreversible — dropping data, force-unlocking shared state, deleting a resource you cannot recreate — get a second person on the call before you run it, even if you are fairly confident. Being fairly confident at 3am has a worse track record than it feels like it does.

When you genuinely cannot fix it

Sometimes the answer is that this will not be resolved tonight. That is an acceptable outcome, and handling it well is a skill.

Reduce the impact to whatever degree you can. Turn off the feature. Serve a degraded version. Fail over to the other region. Put up a status message. A partly-working system with an honest status page is a much better outcome than an engineer grinding at a root cause until dawn.

Then hand over properly: what you know, what you tried, what you ruled out, and what you would try next.

Common mistakes

Fixing before understanding. Restarting things until the symptom goes away works often enough to be a habit, and it destroys the evidence you would have needed when it recurs next week.

Trusting the alert's framing. The alert tells you what a threshold noticed, not what is wrong. "HighMemoryUsage" may be a memory leak, a cache behaving correctly, or a limit set too low a year ago.

Going quiet. Covered above, and it is the most common one.

Reading logs before establishing scope. Logs are enormous and seductive. Find out what is affected and when it started first; then you know which logs, and which minutes, are worth reading.

Treating every page as urgent. If everything is urgent you will burn out, and the thing that actually matters will arrive when you have nothing left. Part of the job is deciding what can wait.

Afterwards

Write it up while it is fresh — the next day, not the next week.

The useful shape is: what happened, when, what the impact was, what you did, and what would have made it shorter. That last question is the one that improves things. "What would have told us sooner" and "what would have made the fix obvious" produce better actions than "whose fault was it", which produces none.

Keep it blameless, and mean it. An engineer who fears the write-up will be slower to escalate next time, which costs you far more than any individual mistake.

A note on the rotation itself

If you are being paged more than a couple of times a night, the problem is not your technique. Alert volume is a system property and it is fixable: alerts that fire on causes rather than symptoms, thresholds nobody has revisited, and pages for things that could wait until morning.

If you are designing or arguing about a rotation, the on-call load calculator will tell you whether the one you are proposing is survivable, and the error budget calculator will tell you how much room a reliability target actually leaves before any of this becomes urgent.