Docker · why it works that way
Why are containers not just lightweight virtual machines?
Short answer
First published
On this page · 7 sections
The constraint behind it
A virtual machine needs a hypervisor presenting virtual hardware, on which a complete guest operating system boots. That is why a traditional VM costs far more to start and to hold than a container: you are booting an operating system. Purpose-built microVMs cut both sharply by booting a minimal kernel on minimal virtual hardware, but there is still a kernel to start.
A container needs none of that. It is a process the kernel has been asked to lie to: namespaces restrict what it can see (its own process tree, network interfaces, filesystem mounts) and cgroups restrict what it can use.
So a container starts quickly because nothing boots. There is no kernel to initialise, no hardware to enumerate — the kernel is already running.
Words used on this page
- kernel
- The core of the operating system: it runs processes, manages memory and devices, and answers system calls. Every container on a host uses the same one.
- hypervisor
- Software that presents virtual hardware to virtual machines. Each VM boots its own kernel on that hardware.
- namespace
- A Linux kernel feature that gives a group of processes their own view of one kind of resource, such as process IDs, network interfaces, mount points or the hostname.
- PID namespace
- A namespace with its own process ID numbering. Processes inside see only the processes in it, and the first one gets PID 1.
- cgroup
- Control group: a Linux kernel feature that groups processes so their use of memory, CPU and other resources can be limited and monitored.
- system call
- A request from a program to the kernel, such as opening a file or starting a process. With a shared kernel, this interface is the isolation boundary.
- seccomp
- A kernel feature that filters which system calls a process may make. Docker applies a default profile; Kubernetes applies one only when the Pod asks for it or the kubelet's seccomp defaulting is on.
- capability
- One slice of root's privileges, such as
SYS_PTRACEorNET_BIND_SERVICE, that can be granted to or removed from a process on its own. - pod sandbox
- What the container runtime sets up for a Pod for its containers to share. In the example its process is
/pause, PID 1 of the shared PID namespace.
If it were the other way
If containers really were lightweight VMs, they would have their own kernels — and the two properties people value most would disappear.
Start-up would mean booting a kernel. MicroVMs show that can take well under a second, but it is still work a container never does. Density would fall, because each container would carry its own kernel and the memory to run it. And you could run a Linux container on a Windows kernel directly, which you cannot.
The security story would improve, though. A VM's isolation boundary is the hypervisor rather than a kernel shared with every other workload on the host, which is why container runtimes that use hardware virtualization exist for workloads that need more isolation than a container gives.
What it buys and costs you
What it buys you
- Fast start-up, with no kernel to boot, which is what makes autoscaling and per-request scheduling practical.
- Density: more applications on the same hardware, because they share one kernel and one page cache.
- Layer sharing. Ten images on the same base store that base once, so pulls are fast and disk use is sublinear.
What it costs you
- Isolation is weaker, and the boundary is the system call interface — a very large surface. A kernel vulnerability can become a container escape, which is why running as root inside a container matters so much more than it intuitively should.
- The kernel is shared, so it is not yours. Kernel modules, the kernel version and every sysctl that is not namespaced are the host's, and a workload that needs a specific one needs a specific host.
- You cannot cross kernels. Linux containers on macOS or Windows run inside a hidden Linux VM, which is the source of a whole family of confusing filesystem and networking behaviours.
Where it bites in practice
In how resource limits are observed. A container is not a machine, so tools that read /proc see the host's CPU count and memory unless they are cgroup-aware. A runtime that sizes its thread pool or heap from that gets it badly wrong — and then gets OOM-killed for exceeding a limit it never knew about.
Try it yourself
Before you start
- A Kubernetes cluster with Linux nodes that you may experiment on, and
kubectlconfigured for it. The task page the example comes from suggests minikube or an online playground if you have none. - A namespace whose Pod Security level allows the example. Its
shellcontainer adds theSYS_PTRACEcapability, which the Baseline and Restricted profiles forbid, so try it where neither is enforced. - Network access for the cluster to pull the
nginxandbusybox:1.28images. - Somewhere to read the Linux man pages this page relies on: namespaces(7), pid_namespaces(7) and cgroups(7), with
manon a Linux machine or on manpages.debian.org. On macOS or Windows, Docker Desktop runs containers inside a Linux virtual machine, so the kernel your containers share there is that VM's.
Two containers, one process list
Quoted from the Kubernetes task page Share Process Namespace between Containers in a Pod; not reproduced here, because there is no cluster and no Docker on this machine. The Pod nginx has two containers, nginx and shell (busybox 1.28 with the SYS_PTRACE capability), and sets shareProcessNamespace: true. It was created with kubectl apply -f https://k8s.io/examples/pods/share-process-namespace.yaml, and both commands run inside the shell container, opened with kubectl exec -it nginx -c shell -- /bin/sh. The page shows the output as "similar to this".
Command
ps ax head /proc/8/root/etc/nginx/nginx.conf
Output · as shown in Kubernetes — Share Process Namespace between Containers in a Pod
$ ps ax
PID USER TIME COMMAND
1 root 0:00 /pause
8 root 0:00 nginx: master process nginx -g daemon off;
14 101 0:00 nginx: worker process
15 root 0:00 sh
21 root 0:00 ps ax
$ head /proc/8/root/etc/nginx/nginx.conf
user nginx;
worker_processes 1;
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;
events {
worker_connections 1024;How to read it
ps axran in theshellcontainer, yet it lists nginx, which runs in the other container. Both are ordinary processes on the node's one kernel, andshareProcessNamespace: trueput them in the same PID namespace. Without that field each container sees only its own processes.- PID 1 is
/pause, the pod sandbox, not nginx. The numbers start again at 1 because a PID namespace has its own numbering; on the node, the same processes have other PIDs, since a process has one ID in each PID namespace that can see it. - Nothing booted to make this happen. The
shellcontainer's view of the system changed because of one field in the Pod spec: that is what it means for a container to be a restricted view of a shared kernel rather than a machine. head /proc/8/root/etc/nginx/nginx.confread a file from the nginx container's filesystem./proc/<pid>/rootgives the same view of the filesystem that process has, and reading it is governed by a ptrace access check, which is why the example grantsSYS_PTRACE. The page warns that, with a shared process namespace, such files are protected only by filesystem permissions.
If it goes wrong
- Creating the Pod fails with
non-default capabilitiesandcontainer "shell" must not include "SYS_PTRACE" in securityContext.capabilities.add - The namespace enforces the Baseline or Restricted Pod Security profile, which does not allow adding
SYS_PTRACE. Use a throwaway namespace without that enforcement for the exercise; do not relax the policy on a shared namespace to make it work. ps axin theshellcontainer shows only its own processes, not nginx- The Pod does not set
shareProcessNamespace: true, so each container has its own PID namespace. Add the field and create the Pod again: a running Pod's spec cannot be changed to add it. - A Java or other runtime in a container sizes its heap or thread pool for the whole host, then is OOM-killed
- Files such as
/proc/meminforeport the whole system, not the container's limit. Use a runtime that reads its cgroup limits (in current JVMs,-XX:+UseContainerSupportis on by default on Linux), or set the heap size explicitly below the container's memory limit. - A sysctl you need cannot be set from the Pod's
securityContext - Only namespaced sysctls, such as
kernel.shm*,kernel.msg*,kernel.sem,fs.mqueue.*and manynet.*, can be set per Pod. The rest are node-level: they belong to the shared kernel and have to be set on each node, which affects every Pod there.
Security considerations
- The shared kernel is the isolation boundary, so its bugs are your bugs. Docker's security documentation names incomplete isolation "when used in combination with kernel vulnerabilities" as a primary risk. Patching a container image does not patch the kernel underneath it; keep the hosts' kernels patched too.
- Run container processes as a non-root user. Docker's documentation calls this the best way to prevent privilege-escalation attacks from within a container; where root is unavoidable, user namespace remapping maps it to an unprivileged user on the host.
- Keep a seccomp profile on. Docker's default profile blocks around 44 of the 300+ system calls, and
--security-opt seccomp=unconfinedremoves it. In Kubernetes the default isUnconfinedunless the kubelet's seccomp defaulting is enabled, so setseccompProfile.type: RuntimeDefaultin the Pod'ssecurityContext. - Sharing a process namespace widens what each container can see. In the example the
shellcontainer read nginx's configuration through/proc/8/root, and the documentation warns that command-line arguments and environment variables, passwords included, become visible to the Pod's other containers. - For code you do not trust, a container runtime that uses hardware virtualization, selected with a RuntimeClass, gives extra isolation at the cost of some overhead, because the boundary is then a virtual machine rather than the shared kernel.
Sources
The behaviour this page describes, as documented by the projects themselves.
- Linux man-pages — pid_namespaces(7)
- Linux man-pages — network_namespaces(7)
- Docker — Docker Engine security
- Docker — Seccomp security profiles for Docker
- Docker Desktop — Networking
- Kubernetes — Runtime Class
- Kubernetes — Using sysctls in a Kubernetes Cluster
- Kubernetes — Share Process Namespace between Containers in a Pod
- Firecracker v1.17.0 — Specification
- Docker — What is a container?
- Linux man-pages — namespaces(7)
- Linux man-pages — cgroups(7)
Terms used here (1)plain definitions
Short definitions for the vocabulary on this page, in case any of it is new. Each links to the full entry.
- Container
- A running instance of an image: an isolated process with its own view of the filesystem, network and other system resources, plus a thin writable layer on top of the image's read-only layers that is discarded when the container is deleted. Containers share the host OS kernel, making them far more efficient than traditional virtual machines.
Did this get you to an answer?
No text box on purpose — please do not paste production logs anywhere
Something wrong or out of date? Report it