Who this is for
You can SSH into a box and read a prompt. You are not trying to become a sysadmin; you are trying to find out why something is broken and get back to what you were doing.
"The box is slow"
uptime # load average: 1, 5 and 15 minute
top # or htop, if it is installed
free -h # memory
Load average on Linux is not CPU utilisation. It counts processes that are
running and processes stuck in uninterruptible sleep — almost always waiting
on disk or network storage. A load of 8 on an 8-core box can mean fully busy,
or it can mean eight processes stuck waiting on a slow disk while every core idles. top
tells you which: look at %wa (IO wait) against %us (user).
In free -h, read the available column, not free. Linux spends spare
memory on page cache deliberately and gives it back on demand — a box showing
almost no "free" memory and plenty "available" is working correctly. This is
a classic false alarm on a Linux server.
"Is it really short of memory?"
Added 26 October 2026. The available column says how much memory is left. It does not say whether the box is struggling; that shows as activity:
vmstat 5 # si and so: pages swapped in and out per second
cat /proc/pressure/memory # share of time tasks waited for memory (on kernels with PSI)
Two readings from this site's own 2 GB server, measured on 8 October 2026.
On a normal day, free showed 94 MB free but 1,363 MB available, 54 MB of
swap in use, no swapping in progress and memory pressure at zero: healthy,
however alarming the free column looks. During a Next.js build, available
fell to 269 MB, swap in use passed 1.1 GB and pages moved in and out of swap at
tens of megabytes a second: that is pressure. Swap in use on its own is
history, not a problem.
Do not "fix" low free memory with echo 3 > /proc/sys/vm/drop_caches. The
kernel documentation says those caches are reclaimed automatically when
memory is needed, and that dropping them costs I/O and CPU to rebuild.
The comparison in full:
free vs available memory.
"The disk is full"
df -h # which filesystem, and how full
du -sh /var/* | sort -rh | head # where it went, under a path
When df says the disk is full and du cannot account for it, the space is
almost always held by files that have been deleted but are still open. A
process holding a deleted log file keeps its blocks allocated until the process
exits — the directory entry is gone, so du cannot see it, but the space is
still spent.
sudo lsof +L1 # open files whose link count is 0
Restart whatever shows up, or truncate it. Deleting "the big log file" again achieves nothing, which is why this one wastes so much time when you have not seen it before.
"The service will not start"
systemctl status nginx # state, and the last few log lines
journalctl -xeu nginx # the actual reason, with explanations
journalctl -u nginx --since "10 min ago" --no-pager
systemctl status shows a handful of recent lines and people stop there. The
cause is usually further back, which is what journalctl -u is for. -x adds
explanatory text for systemd's own messages, and -e jumps to the end.
"What is listening on that port?"
sudo ss -tulnp # every listening socket, with the process
sudo lsof -i :8080 # who holds one specific port
Run both with sudo. Without it they still list the sockets but omit the
process names, which is the part you wanted — a surprisingly common ten minutes
lost.
"Can this box reach that one?"
curl -v https://example.com # the whole exchange: DNS, TLS, headers
curl -sS -o /dev/null -w '%{http_code} %{time_total}s\n' https://example.com
dig example.com +short # what DNS actually returns
nc -zv hostname 443 # is the port open at all
ip route get 10.0.1.5 # which interface and gateway it would use
Work down the stack. dig first — if the name resolves to the wrong address,
nothing above it matters. Then nc for reachability, then curl -v for TLS
and HTTP. Skipping to curl and reading the error message is how a DNS problem
gets diagnosed as a certificate problem.
"Find it in the logs"
grep -c "ERROR" app.log # how many, before you read any
grep -n "timeout" app.log | tail -20 # with line numbers
awk '$9 >= 500 {print $7}' access.log | sort | uniq -c | sort -rn | head
sed -n '2000,2050p' huge.log # a slice, without loading it all
tail -f /var/log/syslog # follow (journalctl -f where there is no syslog file)
The sort | uniq -c | sort -rn pipeline is the one to internalise: count
occurrences, then rank them. It turns "a lot of errors" into "97% of them are
one error", which is a different problem.
For compressed and rotated logs, zgrep reads them without unpacking first.
Permissions, briefly
chmod +x deploy.sh # make it runnable
chmod 644 config.yml # rw for owner, r for everyone else
chown -R www-data:www-data /var/www
id # who am I, and in which groups
sudo -u postgres psql # run as another user
If a script fails with "Permission denied" and the file looks executable, check the directory too — you need execute permission on every directory in the path to reach the file.
Common mistakes
- Reaching for
kill -9first.-9is SIGKILL: the process cannot catch it, so it never flushes a buffer, closes a connection or removes a lock file. Sendkill(SIGTERM) first and give it a few seconds; keep-9for something that has genuinely stopped responding. base64without-n.echo "secret" | base64encodes a trailing newline, and the decoded value will not match.echo -nmatters, and this is the usual reason a hand-made Kubernetes secret authenticates nowhere.chmod 777to make it work. It does make it work, and it makes it world-writable. The problem is nearly always ownership, whichchownfixes without the side effect.find / -name ...on a production box under load. It walks every filesystem, including network mounts. Give it a starting path and-xdevto stay on one filesystem.
If this did not work
- A command is not found. Minimal images ship very little —
htop,dig,lsofandnetstatare frequently absent.ssandipare usually there;cat /proc/net/tcpalways is. - You are inside a container and the numbers look wrong.
topandfreeread the host's/proc, so they report the host's memory and CPU, not the container's limits. Read/sys/fs/cgroup/for what the container is actually allowed. journalctlreturns nothing. The service may not be running under systemd, or logs may be going to a file. Check the unit'sStandardOutput, or look in/var/log/.
Next steps
Practise the disk-full and the port-in-use ones on a throwaway box before you need them. Both come up in real incidents, and both have a counter-intuitive answer that is hard to reason out under pressure.
Related on this site
- Container exit codes — What 137 and 143 mean, and the rule that decodes any of them.
- Cron expression explainer — Read a crontab line without guessing.
- No space left on device — Where the disk actually went.
Enjoyed this article?
Get more DevOps insights delivered to your inbox.
Get new posts by email
Subscribe to get an email when a new blog post is published. Skip anytime.
No spam, unsubscribe anytime.
Terms used here (2)plain definitions
Short definitions for the vocabulary on this page, in case any of it is new. Each links to the full entry.
- Ansible
- An open-source automation tool for configuration management, application deployment, and task automation. Ansible uses YAML-based playbooks and operates agentlessly over SSH, making it simple to get started without installing software on target machines.
- SSH
- Secure Shell — a cryptographic network protocol for operating network services securely over an unsecured network. SSH is the standard method for remotely accessing Linux servers, transferring files, and tunneling other protocols.
Related Posts
Discussion
0 comments
Sign in to join the conversation.
Be the first to comment
Start a conversation about this post



