Monitoring Your Infrastructure with Prometheus and Grafana
You can't fix what you can't see. Prometheus collects metrics, and Grafana visualizes them — together they form the most popular open-source monitoring stack.
Architecture Overview
- Prometheus — a time-series database that scrapes metrics from targets at regular intervals
- Grafana — a visualization platform that queries Prometheus and renders dashboards
- Exporters — agents that expose metrics in Prometheus format (Node Exporter, cAdvisor, etc.)
Setting Up Prometheus
Create a prometheus.yml configuration:
yaml
global:
scrape_interval: 15s
evaluation_interval: 15s
rule_files:
- "alert_rules.yml"
scrape_configs:
- job_name: "prometheus"
static_configs:
- targets: ["localhost:9090"]
- job_name: "node-exporter"
static_configs:
- targets: ["node-exporter:9100"]
- job_name: "app"
metrics_path: "/metrics"
static_configs:
- targets: ["app:3000"]
Docker Compose Stack
yaml
version: "3.9"
services:
prometheus:
image: prom/prometheus:v2.51.0
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml
- prometheus_data:/prometheus
ports:
- "9090:9090"
grafana:
image: grafana/grafana:10.4.0
environment:
GF_SECURITY_ADMIN_PASSWORD: admin
volumes:
- grafana_data:/var/lib/grafana
ports:
- "3001:3000"
depends_on:
- prometheus
node-exporter:
image: prom/node-exporter:v1.7.0
ports:
- "9100:9100"
volumes:
prometheus_data:
grafana_data:
PromQL Basics
PromQL is Prometheus's query language:
promql
# Current CPU usage across all cores
100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)
# Memory usage percentage
(1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100
# HTTP request rate
rate(http_requests_total[5m])
# 95th percentile latency
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))
# Error rate
rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m])
Alert Rules
Define alerts in alert_rules.yml:
yaml
groups:
- name: infrastructure
rules:
- alert: HighCPUUsage
expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
for: 5m
labels:
severity: warning
annotations:
summary: "High CPU usage on {{ $labels.instance }}"
- alert: DiskSpaceLow
expr: (node_filesystem_avail_bytes / node_filesystem_size_bytes) * 100 < 10
for: 5m
labels:
severity: critical
annotations:
summary: "Disk space below 10% on {{ $labels.instance }}"
Instrumenting Your Application
For a Node.js app, use the prom-client library:
javascript
const client = require("prom-client");
const httpDuration = new client.Histogram({
name: "http_request_duration_seconds",
help: "Duration of HTTP requests in seconds",
labelNames: ["method", "route", "status"],
buckets: [0.01, 0.05, 0.1, 0.5, 1, 5],
});
app.use((req, res, next) => {
const end = httpDuration.startTimer();
res.on("finish", () => {
end({ method: req.method, route: req.path, status: res.statusCode });
});
next();
});
app.get("/metrics", async (req, res) => {
res.set("Content-Type", client.register.contentType);
res.end(await client.register.metrics());
});
Grafana Dashboard Tips
- Import community dashboards from grafana.com/dashboards (e.g., Node Exporter Full: ID 1860)
- Use variables for dynamic dashboards (e.g., dropdown to select instance)
- Set up notification channels (Slack, PagerDuty, email) for alerts
- Create a golden signals dashboard: latency, traffic, errors, saturation
Tagged with
Enjoyed this article?
Get more DevOps insights delivered to your inbox.
Get new posts by email
Subscribe to get an email when a new blog post is published. Skip anytime.
No spam, unsubscribe anytime.
N
Related Posts
Discussion
0 comments
Sign in to join the conversation.
Be the first to comment
Start a conversation about this post