Monitoring Your Infrastructure with Prometheus and Grafana

Who this is for

You run something in production and currently find out it is broken when somebody tells you. You do not need to know Go or PromQL yet.

How it fits together

Prometheus pulls. Your service exposes a plain-text endpoint, usually /metrics, and Prometheus fetches it on a fixed interval. Nothing is pushed, which means a service that has fallen over simply stops being scraped — and up == 0 is itself a metric you can alert on.

your app  ──/metrics──▶  Prometheus  ──queries──▶  Grafana
node_exporter ──────────▶     │
                              └──▶ Alertmanager ──▶ email / Slack / pager

Grafana stores no metrics. It asks Prometheus on every panel refresh, which is why a dashboard with forty panels on a thirty-second refresh is a load source of its own.

A stack you can run locally

services:
  prometheus:
    image: prom/prometheus:latest
    ports: ["9090:9090"]
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prom-data:/prometheus

  grafana:
    image: grafana/grafana:latest
    ports: ["3000:3000"]
    environment:
      GF_SECURITY_ADMIN_PASSWORD: ${GRAFANA_PASSWORD:?set this}
    volumes:
      - grafana-data:/var/lib/grafana

volumes:
  prom-data:
  grafana-data:

# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: app
    static_configs:
      - targets: ["app:8080"]

Both volumes matter. Without them the data lives inside the container, so removing it — a docker compose down is enough — throws away every metric you have collected, and the first time that happens is usually while you are reading a graph.

The four metric types

Prefer a histogram to a summary unless you have a specific reason. Summary quantiles are calculated per instance and cannot be aggregated — averaging the 99th percentile of three replicas does not give you the 99th percentile of the service, it gives you a number with no meaning.

PromQL, the parts you need

# A counter is cumulative. Always wrap it in rate().
rate(http_requests_total[5m])

# Error ratio
sum(rate(http_requests_total{status=~"5.."}[5m]))
  / sum(rate(http_requests_total[5m]))

# 95th percentile latency from a histogram
histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))

# Is it being scraped at all?
up{job="app"}

Two rules that remove most confusion. Never graph a counter directly — it only ever rises, so the line tells you about uptime, not traffic; rate() is what turns it into per-second change. And make the range at least four times the scrape interval: rate(...[5m]) at a 15-second scrape has twenty samples to work with, while [20s] has one or two and produces gaps the moment a scrape is missed.

Stop reporting averages for latency. An average response time of 200 ms is consistent with every request taking 200 ms and with 95% taking 50 ms while 5% take three seconds. Only one of those is an incident, and the average cannot tell you which you have.

Cardinality, which is how people break Prometheus

Every unique combination of label values is a separate time series held in memory. Labels with bounded values — method, status, endpoint — are fine. Labels with unbounded values are not:

# Fine: a handful of series
http_requests_total{method="GET", status="200", route="/api/users"}

# Catastrophic: one series per user, forever
http_requests_total{user_id="8f21c...", request_id="a93f..."}

User IDs, request IDs, email addresses, full URLs with query strings and raw error messages are the usual culprits. The failure is not gradual: memory climbs, queries slow, and eventually the process is OOM-killed — taking your monitoring down at the moment you need it. Before adding a label, ask how many distinct values it can ever hold. If the answer is "as many as we have users", it belongs in a log, not a metric.

Alerts that are worth waking up for

groups:
  - name: app
    rules:
      - alert: HighErrorRate
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m]))
            / sum(rate(http_requests_total[5m])) > 0.05
        for: 10m
        labels:
          severity: page
        annotations:
          summary: "5xx rate above 5% for 10 minutes"
          runbook: "https://example.internal/runbooks/high-error-rate"

      - alert: TargetDown
        expr: up == 0
        for: 5m
        labels:
          severity: page

for: is the difference between an alert and a nuisance — it requires the condition to hold, so a single bad scrape does not page anyone. The runbook annotation is the one people skip and then regret at 3am.

Alert on symptoms your users feel, not on causes. "Error rate is above 5%" is worth a page. "CPU is at 80%" is not — it may be entirely healthy, and if it is causing harm the symptom alert fires anyway. This site has a page on exactly that trade: Symptom vs cause alerting.

How much failure you are allowed before any of this matters is an error budget question — the error budget calculator turns an availability target into minutes.

An alert that can no longer fire

Added 28 October 2026. An alert you never hear from is either good news or broken, and nothing tells you which. Two ways it breaks without an error:

• The metric is renamed. Kubernetes 1.36 renamed kube-controller-manager's volume_operation_total_errors to volume_operation_errors_total. A query on the old name does not fail; it returns nothing, and an alert on nothing never fires.
• The series disappears. A job is renamed or a target stops exporting, and every rule built on it goes quiet.

For metrics that should always exist, alert on their absence. Prometheus's absent() returns 1 when no series match, which its documentation describes as useful for exactly this:

      - alert: WebMetricsMissing
        expr: absent(up{job="web"})
        for: 10m
        labels:
          severity: ticket

Do not point absent() at a counter that only appears after its first event, such as an error counter: it would fire on every healthy day. And test the rules themselves: promtool test rules feeds series into your rule files and checks which alerts fire, so a rename shows up in CI rather than in an incident.

Common mistakes

1. Unbounded labels. See above. It is the one that takes the system down.
2. Alerting without for:. Every transient blip becomes a page, people start ignoring pages, and the monitoring is now worse than none.
3. Dashboards nobody opens. A panel is only useful if somebody can say what action a change in it would cause. Build the alert first and the dashboard to explain it.
4. No retention plan. Prometheus stores locally and is not a long-term archive. Decide the retention period deliberately; send what you need to keep somewhere built for it.
5. Scraping every ten seconds because it feels responsive. It multiplies storage and query cost for resolution nobody reads.

If this did not work

• The target shows as down. Curl the endpoint from inside the Prometheus container, not from your laptop — the usual cause is a hostname that resolves on one network and not the other.
• A query returns nothing. Check the metric name in /metrics directly. Client libraries add suffixes: a histogram called http_request_duration exposes _bucket, _sum and _count, and histogram_quantile needs _bucket.
• Grafana shows "No data" but Prometheus has the series. Nearly always the dashboard's time range, or a variable that resolved to nothing.
• Prometheus restarts and memory climbs immediately. Cardinality. Read /api/v1/status/tsdb (the TSDB status page in the UI): its seriesCountByMetricName shows which metric is carrying the series count without a query that has to touch every series.

Next steps

Instrument one thing properly rather than everything shallowly: request count, error count and duration on your busiest endpoint. That is enough to build an error-rate alert and a latency panel, and it is the set of three almost every useful dashboard is built from.

Related on this site

• Symptom vs cause alerting — Which of your alerts should wake somebody, and which should not.
• SLO vs SLA — Two words used interchangeably that commit you to different things.
• Error budget calculator — How much downtime a target actually allows.