SRE
symptom-based alerts vs cause-based alerts
Short answer
A symptom alert fires when users are affected. A cause alert fires when a thing you can measure looks unusual. The second set is far larger, mostly harmless, and is where alert fatigue comes from.
In simple terms
Cause-based alerting is the natural thing to build, because causes are what you can see. CPU is at 90%, the queue has 5,000 items, disk is 80% full, a pod restarted. All easy to measure and easy to threshold.
The trouble is that none of those statements says anything about whether anything is wrong. CPU at 90% might be a service working efficiently. A deep queue might be a batch job doing its job. Most of the time, the unusual thing is fine — so most of the pages are noise.
Symptom-based alerting starts from the other end: is anybody actually experiencing a problem? Error rate up, latency past what users tolerate, a queue whose age means work will miss its deadline. Far fewer of these fire, and when one does, something genuinely needs attention.
The reason this is the highest-leverage change available is arithmetic rather than philosophy. Causes vastly outnumber symptoms, because any given symptom has many possible causes — so alerting on causes means alerting on a much bigger set, and most of that set is harmless.
What actually happens
The practical rule: alert on the things in your SLI, and let everything else be a dashboard. If a metric is not part of how you measure whether the service is meeting its objective, it is diagnostic data rather than a paging condition.
This does not mean discarding cause metrics — they are exactly what you need once you are already investigating. It means changing what wakes somebody up. The cause data should be one click from the symptom alert, not the thing that fired it.
The counter-argument is real and worth stating: symptom alerts fire later. A disk filling at a steady rate is a cause you can act on hours before any user notices, and waiting for the symptom means waiting for an outage. So the honest position is that a small number of cause alerts are justified — specifically the ones where the lead time is long and the trajectory is predictable, like disk capacity or certificate expiry.
That is a narrow exception, not a general licence. The test is whether you can name the user-visible failure it prevents and whether the trajectory genuinely gives you useful warning.
The same incident, alerted both ways
# Cause-based — fires often, mostly harmlessly ALERT HighCPU cpu > 90% for 5m ALERT QueueDepth queue_length > 5000 ALERT PodRestarted increase(restarts[10m]) > 0 ALERT DiskAt80 disk_used > 80% # Symptom-based — fires rarely, and means something ALERT ErrorRateHigh rate(5xx) / rate(all) > 0.02 for 5m ALERT LatencyBad p95_latency > 1s for 10m ALERT QueueStale oldest_message_age > 300s # The justified cause exception: long lead time, predictable trajectory ALERT DiskFullSoon predict_linear(disk_free[6h], 4*3600) < 0
Note what changed in the disk alert. `disk > 80%` fires on a state that may be entirely stable; the prediction fires only when the trajectory says you will actually run out — and it tells you roughly when.
◑ The mistake this causes
Adding an alert after every incident.
Why people do it It feels like learning, and it is the standard post-incident action item. Something was missed, so we add an alert for the thing that was missed — and each addition is individually defensible.
What you see Alert volume climbing steadily while the actionable rate falls, and nobody able to point at when it became unmanageable because no single addition was wrong. Eventually the rational response to a page is to assume it is noise, and people adopt that response whether or not they say so — which means the next real page is treated the same way.
How it shows up in production
A shared dependency slows down. Because most alerting is cause-based, every service that touches it fires: CPU alerts from retry loops, queue-depth alerts from backpressure, restart alerts from failing health checks, latency alerts at every layer.
Forty pages arrive in ten minutes describing one problem, and the responder's first job is not diagnosis — it is working out how many distinct incidents they are looking at. That triage is pure cost, created by the alerting rather than by the fault.
With symptom alerting the same incident produces a handful of pages naming user-visible impact, and the cause metrics are all still there to investigate with. Same information, radically different demand on the person holding the pager at 3am.
How to tell which one you are hitting
- Does this alert name something a user would notice?
- If you cannot describe the user-visible impact in one sentence, it is a cause alert. That is not automatically wrong, but it should be a dashboard unless it passes the lead-time test.
- What proportion of these pages led to an action?
- Below roughly 30% actionable, the alert is training people to ignore it. That is worth measuring per rule rather than in aggregate — volume is almost always concentrated in a handful of rules.
- Would waiting have been worse?
- The question that justifies a cause alert. Disk filling and certificates expiring both fail this in the other direction — waiting for the symptom means waiting for an outage, so the cause alert earns its place.
How this gets asked in an interviewpreparation
Usually phrased as “How would you reduce alert fatigue?”
What a strong answer contains Lead with the shift from cause to symptom and explain why it works arithmetically — causes vastly outnumber symptoms, so alerting on causes means alerting on a much larger and mostly harmless set. Then give the practical rule: alert on your SLIs, keep the rest as dashboards. Naming the justified exception for long-lead-time causes shows you are reasoning rather than reciting.
The follow-up “What about disk filling up?” is the standard test of whether you hold the position dogmatically. The good answer is that predictable trajectories with long lead times are exactly where a cause alert is right, and that the alert should fire on the prediction rather than on a static threshold.
What comes next
Did this get you to an answer?
No text box on purpose — please do not paste production logs anywhere