Skip to main content
DevOps Insightsguides & answers

Docker · why it works that way

Why are containers not just lightweight virtual machines?

Short answer

Because they share the host's kernel. A VM virtualises hardware and runs its own kernel; a container is an ordinary process with a restricted view of the same kernel everything else is using. Almost every practical difference follows from that one fact.

First published

On this page · 7 sections

The constraint behind it

A virtual machine needs a hypervisor presenting virtual hardware, on which a complete guest operating system boots. That is why a traditional VM costs far more to start and to hold than a container: you are booting an operating system. Purpose-built microVMs cut both sharply by booting a minimal kernel on minimal virtual hardware, but there is still a kernel to start.

A container needs none of that. It is a process the kernel has been asked to lie to: namespaces restrict what it can see (its own process tree, network interfaces, filesystem mounts) and cgroups restrict what it can use.

So a container starts quickly because nothing boots. There is no kernel to initialise, no hardware to enumerate — the kernel is already running.

Words used on this page

kernel
The core of the operating system: it runs processes, manages memory and devices, and answers system calls. Every container on a host uses the same one.
hypervisor
Software that presents virtual hardware to virtual machines. Each VM boots its own kernel on that hardware.
namespace
A Linux kernel feature that gives a group of processes their own view of one kind of resource, such as process IDs, network interfaces, mount points or the hostname.
PID namespace
A namespace with its own process ID numbering. Processes inside see only the processes in it, and the first one gets PID 1.
cgroup
Control group: a Linux kernel feature that groups processes so their use of memory, CPU and other resources can be limited and monitored.
system call
A request from a program to the kernel, such as opening a file or starting a process. With a shared kernel, this interface is the isolation boundary.
seccomp
A kernel feature that filters which system calls a process may make. Docker applies a default profile; Kubernetes applies one only when the Pod asks for it or the kubelet's seccomp defaulting is on.
capability
One slice of root's privileges, such as SYS_PTRACE or NET_BIND_SERVICE, that can be granted to or removed from a process on its own.
pod sandbox
What the container runtime sets up for a Pod for its containers to share. In the example its process is /pause, PID 1 of the shared PID namespace.

If it were the other way

If containers really were lightweight VMs, they would have their own kernels — and the two properties people value most would disappear.

Start-up would mean booting a kernel. MicroVMs show that can take well under a second, but it is still work a container never does. Density would fall, because each container would carry its own kernel and the memory to run it. And you could run a Linux container on a Windows kernel directly, which you cannot.

The security story would improve, though. A VM's isolation boundary is the hypervisor rather than a kernel shared with every other workload on the host, which is why container runtimes that use hardware virtualization exist for workloads that need more isolation than a container gives.

What it buys and costs you

What it buys you

  • Fast start-up, with no kernel to boot, which is what makes autoscaling and per-request scheduling practical.
  • Density: more applications on the same hardware, because they share one kernel and one page cache.
  • Layer sharing. Ten images on the same base store that base once, so pulls are fast and disk use is sublinear.

What it costs you

  • Isolation is weaker, and the boundary is the system call interface — a very large surface. A kernel vulnerability can become a container escape, which is why running as root inside a container matters so much more than it intuitively should.
  • The kernel is shared, so it is not yours. Kernel modules, the kernel version and every sysctl that is not namespaced are the host's, and a workload that needs a specific one needs a specific host.
  • You cannot cross kernels. Linux containers on macOS or Windows run inside a hidden Linux VM, which is the source of a whole family of confusing filesystem and networking behaviours.

Where it bites in practice

In how resource limits are observed. A container is not a machine, so tools that read /proc see the host's CPU count and memory unless they are cgroup-aware. A runtime that sizes its thread pool or heap from that gets it badly wrong — and then gets OOM-killed for exceeding a limit it never knew about.

Try it yourself

Before you start

  • A Kubernetes cluster with Linux nodes that you may experiment on, and kubectl configured for it. The task page the example comes from suggests minikube or an online playground if you have none.
  • A namespace whose Pod Security level allows the example. Its shell container adds the SYS_PTRACE capability, which the Baseline and Restricted profiles forbid, so try it where neither is enforced.
  • Network access for the cluster to pull the nginx and busybox:1.28 images.
  • Somewhere to read the Linux man pages this page relies on: namespaces(7), pid_namespaces(7) and cgroups(7), with man on a Linux machine or on manpages.debian.org. On macOS or Windows, Docker Desktop runs containers inside a Linux virtual machine, so the kernel your containers share there is that VM's.

Two containers, one process list

Quoted from the Kubernetes task page Share Process Namespace between Containers in a Pod; not reproduced here, because there is no cluster and no Docker on this machine. The Pod nginx has two containers, nginx and shell (busybox 1.28 with the SYS_PTRACE capability), and sets shareProcessNamespace: true. It was created with kubectl apply -f https://k8s.io/examples/pods/share-process-namespace.yaml, and both commands run inside the shell container, opened with kubectl exec -it nginx -c shell -- /bin/sh. The page shows the output as "similar to this".

Command

ps ax
head /proc/8/root/etc/nginx/nginx.conf

Output · as shown in Kubernetes — Share Process Namespace between Containers in a Pod

$ ps ax
PID   USER     TIME  COMMAND
    1 root      0:00 /pause
    8 root      0:00 nginx: master process nginx -g daemon off;
   14 101       0:00 nginx: worker process
   15 root      0:00 sh
   21 root      0:00 ps ax

$ head /proc/8/root/etc/nginx/nginx.conf
user  nginx;
worker_processes  1;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;


events {
    worker_connections  1024;

How to read it

  • ps ax ran in the shell container, yet it lists nginx, which runs in the other container. Both are ordinary processes on the node's one kernel, and shareProcessNamespace: true put them in the same PID namespace. Without that field each container sees only its own processes.
  • PID 1 is /pause, the pod sandbox, not nginx. The numbers start again at 1 because a PID namespace has its own numbering; on the node, the same processes have other PIDs, since a process has one ID in each PID namespace that can see it.
  • Nothing booted to make this happen. The shell container's view of the system changed because of one field in the Pod spec: that is what it means for a container to be a restricted view of a shared kernel rather than a machine.
  • head /proc/8/root/etc/nginx/nginx.conf read a file from the nginx container's filesystem. /proc/<pid>/root gives the same view of the filesystem that process has, and reading it is governed by a ptrace access check, which is why the example grants SYS_PTRACE. The page warns that, with a shared process namespace, such files are protected only by filesystem permissions.

If it goes wrong

Creating the Pod fails with non-default capabilities and container "shell" must not include "SYS_PTRACE" in securityContext.capabilities.add
The namespace enforces the Baseline or Restricted Pod Security profile, which does not allow adding SYS_PTRACE. Use a throwaway namespace without that enforcement for the exercise; do not relax the policy on a shared namespace to make it work.
ps ax in the shell container shows only its own processes, not nginx
The Pod does not set shareProcessNamespace: true, so each container has its own PID namespace. Add the field and create the Pod again: a running Pod's spec cannot be changed to add it.
A Java or other runtime in a container sizes its heap or thread pool for the whole host, then is OOM-killed
Files such as /proc/meminfo report the whole system, not the container's limit. Use a runtime that reads its cgroup limits (in current JVMs, -XX:+UseContainerSupport is on by default on Linux), or set the heap size explicitly below the container's memory limit.
A sysctl you need cannot be set from the Pod's securityContext
Only namespaced sysctls, such as kernel.shm*, kernel.msg*, kernel.sem, fs.mqueue.* and many net.*, can be set per Pod. The rest are node-level: they belong to the shared kernel and have to be set on each node, which affects every Pod there.

Security considerations

  • The shared kernel is the isolation boundary, so its bugs are your bugs. Docker's security documentation names incomplete isolation "when used in combination with kernel vulnerabilities" as a primary risk. Patching a container image does not patch the kernel underneath it; keep the hosts' kernels patched too.
  • Run container processes as a non-root user. Docker's documentation calls this the best way to prevent privilege-escalation attacks from within a container; where root is unavoidable, user namespace remapping maps it to an unprivileged user on the host.
  • Keep a seccomp profile on. Docker's default profile blocks around 44 of the 300+ system calls, and --security-opt seccomp=unconfined removes it. In Kubernetes the default is Unconfined unless the kubelet's seccomp defaulting is enabled, so set seccompProfile.type: RuntimeDefault in the Pod's securityContext.
  • Sharing a process namespace widens what each container can see. In the example the shell container read nginx's configuration through /proc/8/root, and the documentation warns that command-line arguments and environment variables, passwords included, become visible to the Pod's other containers.
  • For code you do not trust, a container runtime that uses hardware virtualization, selected with a RuntimeClass, gives extra isolation at the cost of some overhead, because the boundary is then a virtual machine rather than the shared kernel.

Sources

The behaviour this page describes, as documented by the projects themselves.

Terms used here (1)

Short definitions for the vocabulary on this page, in case any of it is new. Each links to the full entry.

Container
A running instance of an image: an isolated process with its own view of the filesystem, network and other system resources, plus a thin writable layer on top of the image's read-only layers that is discarded when the container is deleted. Containers share the host OS kernel, making them far more efficient than traditional virtual machines.

All 59 terms in the glossary

Did this get you to an answer?

No text box on purpose — please do not paste production logs anywhere

Something wrong or out of date? Report it