---
title: "Virtualization and Containers: Two Ways to Share One Machine"
domain: cloud-computing
level: beginner
duration: "50 min"
objectives:
  - "Explain how hardware virtualization works and compare type 1 and type 2 hypervisors."
  - "Compare virtual machines and containers in isolation, startup time, density and overhead."
  - "Explain how Linux namespaces and control groups (cgroups) isolate and limit a container."
  - "Build a container image and explain its layers and the Open Container Initiative (OCI) formats."
  - "Configure resource limits for a container and troubleshoot a container stopped for exceeding its memory."
---

# Virtualization and Containers: Two Ways to Share One Machine

## Why this matters
You launch a "small" server in the cloud and it is ready in under a minute. Behind that button, the provider has not wired a new computer for you. It has carved a slice out of a much bigger machine, and dozens of other customers share the same hardware. Two techniques make that sharing possible: **virtualization**, which gives each customer a complete pretend computer, and **containers**, which give each application its own private view of one shared operating system.

The choice matters every day. A virtual machine (VM) is a strong wall but a heavy one. A container is light and fast but shares its host's kernel, so the wall is thinner. A container that eats all the memory can be killed in the middle of a request, and a container started as root with the wrong options can put the whole host at risk. To pick the right tool, size it correctly and run it safely, you need to know what each one really is and what actually separates a container from the machine it runs on. This lesson explains both.

## Learning objectives
- Explain how hardware virtualization works and compare type 1 and type 2 hypervisors.
- Compare virtual machines and containers in isolation, startup time, density and overhead.
- Explain how Linux namespaces and control groups (cgroups) isolate and limit a container.
- Build a container image and explain its layers and the Open Container Initiative (OCI) formats.
- Configure resource limits for a container and troubleshoot a container stopped for exceeding its memory.

## Key concepts

### Hardware virtualization and hypervisors
Think of a large office building. One option is to rent each company a floor with its own walls, doors and plumbing. Another is to rent desks in one shared open-plan floor, separated by partitions. The first is virtualization: every tenant gets a whole "building" of its own. The second, which comes later in this lesson, is containers.

**Virtualization** means running several isolated virtual computers on one physical computer, the **host**. Each virtual computer is a **virtual machine (VM)**, also called a **guest**. A guest has its own virtual CPUs, memory, disk and network cards, and it runs its own complete operating system (OS) with its own **kernel**, the core of the OS that talks to the hardware. The guest believes it owns real hardware.

The software that creates and runs VMs is the **hypervisor**, or virtual machine monitor (VMM). It has three jobs:
- **Share the hardware.** It schedules the virtual CPUs of all guests on the real CPU cores and gives each guest its own slice of real memory.
- **Isolate the guests.** A guest can't read another guest's memory or take over the host, because the hypervisor controls what each one may touch.
- **Present virtual devices.** The guest sees a virtual disk and a virtual network card, which the hypervisor maps onto real files, storage and network ports.

Early virtualization was slow because a guest kernel expects to run in the CPU's most privileged mode, and only one kernel can hold it. Software tricks (translating or trapping privileged instructions) made it work but cost performance. Modern CPUs solved this with **hardware-assisted virtualization**: **Intel VT-x** and **AMD-V** (flags `vmx` and `svm` in `/proc/cpuinfo` on Linux). They add a new CPU mode for guests: the guest kernel runs at full privilege, but in a mode where the CPU hands control back to the hypervisor, called a **VM exit**, whenever the guest does something that must be controlled, such as touching a device. Other extensions speed up memory (second-level address translation, called EPT on Intel and NPT on AMD) and let a hypervisor give a guest direct, safe access to a device (an input-output memory management unit, IOMMU: Intel VT-d, AMD-Vi). On most computers these features can be switched off in the firmware settings, which is why a hypervisor sometimes reports that virtualization isn't available.

A guest can also run faster when it knows it is virtual. **Paravirtualized** drivers, such as the `virtio` family used by KVM, replace a pretend "real" disk or network card with a simple interface made for virtual machines.

Hypervisors come in two designs:

| | Type 1 (bare metal) | Type 2 (hosted) |
|---|---|---|
| **Runs** | Directly on the hardware | As an application on a normal host OS |
| **Examples** | KVM, Xen, VMware ESXi, Microsoft Hyper-V | Oracle VirtualBox, VMware Workstation, Parallels Desktop |
| **Layers** between a guest and the hardware | Hypervisor | Hypervisor application, then host OS |
| **Overhead and control** | Lower overhead, direct hardware control | More overhead, the host OS stays in charge of the hardware |
| **Typical use** | Data centres and public clouds | Laptops, developer machines, testing |

![Two stacks drawn from the hardware upwards. Type 1, bare metal, with examples KVM, Xen, ESXi and Hyper-V: the hardware (CPU with VT-x or AMD-V, RAM, disks, network cards) carries the hypervisor directly, and on top of it three virtual machines, each with its own guest operating system and kernel and an application. A note says it has fewer layers and less overhead and is the usual choice in data centres and clouds. Type 2, hosted, with examples VirtualBox, VMware Workstation and Parallels: the hardware carries a host operating system such as Windows, macOS or Linux; on the host run a hypervisor application, which asks the host OS for CPU, memory and disk, next to other apps; the hypervisor application carries three virtual machines with guest OS and application. A note says it has one more layer but is easy to install on a laptop.](../../../assets/diagrams/virt-hypervisor-types.svg)

*Figure 1. A type 1 hypervisor sits directly on the hardware; a type 2 hypervisor is one more application on top of a host operating system.*

> [!NOTE]
> The line between the two types is blurry, and the examples above are only examples. KVM is a module inside the Linux kernel: it turns Linux itself into a hypervisor, with a program such as QEMU providing the virtual devices, so it is usually counted as type 1 even though Linux is also a general-purpose OS. Hyper-V is installed on top of Windows, yet it then runs underneath it, and Windows becomes a special VM. What matters is how much sits between the guest and the hardware, not the label.

> [!CHECK]
> Why do modern CPUs have extensions such as Intel VT-x and AMD-V?
>
> - [ ] They let a guest VM bypass the hypervisor and use the physical devices without any control
> - [x] They give guest kernels a safe mode in which most instructions run directly on the CPU and the hypervisor regains control only when needed
> - [ ] They let several VMs share one kernel, which makes them as light as containers
> - [ ] They encrypt each VM's disk so the host can't read it
>
> With hardware assistance the guest runs its own kernel at full speed, and the CPU switches to the hypervisor only on a VM exit. This removes most of the slow software translation older hypervisors needed. It does not remove the hypervisor's control.

### Virtual machines versus containers
A VM virtualizes the **hardware**: each guest brings its own kernel. A **container** virtualizes the **operating system**: it is a normal process (or group of processes) on the host, started with a restricted view of the system. All containers on a host share the host's single kernel. There is no guest OS to boot, no virtual hardware and no hypervisor in the path.

![Left, virtual machines: the hardware carries a hypervisor, which carries three virtual machines, each with its own guest kernel, libraries and app. Tags: own kernel per VM, hardware-level boundary. Facts: starts in seconds to minutes because it boots an operating system, typically gigabytes per image, fewer per host. Right, containers: the hardware carries a host operating system with one shared kernel, whose namespaces and cgroups isolate each container; on it a container runtime such as Podman or containerd with runc or crun runs three containers, each with only its libraries and app. Tags: one shared kernel, kernel-level boundary. Facts: starts in milliseconds to seconds, typically megabytes per image, many more per host.](../../../assets/diagrams/virt-vm-vs-container.svg)

*Figure 2. Each VM carries its own guest kernel; all containers share the host kernel, so a container holds only the application and its libraries.*

This single difference explains every row of the comparison:

| | Virtual machine | Container |
|---|---|---|
| **What is virtualized** | Hardware | The OS view (processes, network, files) |
| **Kernel** | One per VM | Shared with the host and other containers |
| **Isolation boundary** | Hypervisor and CPU virtualization extensions | Kernel features: namespaces, cgroups, capabilities, seccomp |
| **Strength of isolation** | Strong: the attack surface is the hypervisor | Weaker: the attack surface is the whole shared kernel |
| **Startup time** | Seconds to minutes (boots an OS) | Milliseconds to a few seconds (starts a process) |
| **Typical image size** | Gigabytes (a full OS) | Megabytes (app and libraries) |
| **Overhead** | A guest OS per VM: memory, disk, CPU for background services | Close to that of a normal process |
| **Density** | Fewer per host (tens) | Many more per host (hundreds) |
| **Guest OS** | Anything the hardware can run: Linux, Windows, BSD | Same kernel type as the host: Linux containers need a Linux kernel |
| **Typical use** | Different operating systems, strong tenant separation, legacy software | Microservices, fast scaling, repeatable deployment |

The numbers in the table are typical orders of magnitude, not guarantees; they depend on the workload and the platform.

Two practical consequences are worth remembering:
- **Containers on Windows or macOS** use a hidden Linux VM, because those systems have no Linux kernel to share. Your laptop's container tool is a VM plus containers.
- **Real systems mix both.** Public clouds run your containers inside VMs, so a kernel bug in one customer's container doesn't reach another customer. Lightweight options sit between the two: **microVMs** (such as Firecracker) give each workload a tiny VM that boots quickly, and sandboxed runtimes (such as gVisor and Kata Containers) add an extra isolation layer around containers.

> [!CHECK]
> You must run a Windows application and a Linux application on the same server, with strong separation between them. Which is the best fit?
>
> - [ ] Two containers, because containers start faster than VMs
> - [ ] One Linux container for both, because containers can run any application
> - [x] Two VMs, each with its own guest OS
> - [ ] A type 2 hypervisor, because it needs no host OS
>
> Containers share the host kernel, so a Linux host can only run Linux containers. VMs bring their own kernel, so each can run a different OS. A type 2 hypervisor does need a host OS.

### Linux namespaces: what a container can see
How can a normal process believe it is alone on a machine? The Linux kernel can give a process its own private copy of a system resource. Each kind of private copy is a **namespace**. A process lives in one namespace of each type, and processes in the same namespace see the same thing. The namespaces behind a container are:

| Namespace | Isolates | Effect inside a container |
|---|---|---|
| **pid** | Process IDs | The first process is PID 1 and the container sees only its own processes. |
| **net** | Network interfaces, routes, firewall rules, ports | Its own network interface and IP address, and its own port numbers. |
| **mnt** | Mount points | Its own root file system (the image) instead of the host's. |
| **uts** | Hostname and domain name | Its own hostname. |
| **ipc** | Shared memory and message queues | Processes can't use another container's shared memory. |
| **user** | User and group IDs | Root (UID 0) inside can map to an ordinary, unprivileged user on the host. |
| **cgroup** | The view of the cgroup tree | The container sees its own cgroup as the root of the tree. |

Newer kernels also have a **time** namespace, which containers don't use by default. A process normally starts in the host's namespaces; a container runtime creates new ones with the `clone()` and `unshare()` system calls, as described in `namespaces(7)`. You can see which namespaces a process is in as links in `/proc/<pid>/ns/`: two processes share a namespace when the number in the link is the same.

Namespaces give **isolation of view**, not **limits**. A process in its own pid namespace can't see other processes, but it can still use all the CPU and all the memory. That is the job of the next mechanism.

### Control groups (cgroups): what a container can use
A **control group (cgroup)** is a kernel feature that groups processes and controls how much of the machine they can use. Modern Linux uses **cgroups version 2 (v2)**, with one **unified hierarchy** mounted at `/sys/fs/cgroup`. Each group is a directory, and you read and set its limits by reading and writing files in it. Each file belongs to a **controller**:

| File | Controller | Meaning |
|---|---|---|
| `memory.max` | memory | Hard limit on memory in bytes (or `max` for none). Above it, the kernel first reclaims memory and then starts the OOM killer inside the group. |
| `memory.high` | memory | Soft limit: above it the group is slowed down by reclaim but not killed. |
| `memory.current`, `memory.events` | memory | Current use, and counters such as how many times `oom_kill` happened. |
| `cpu.max` | cpu | A quota and a period in microseconds: `50000 100000` means "50 ms of CPU every 100 ms", or half a CPU. `max 100000` means no limit. |
| `cpu.weight` | cpu | Relative share when the CPUs are busy (default 100). |
| `pids.max` | pids | Most processes and threads in the group: a guard against fork bombs. |
| `io.max` | io | Limits on disk bandwidth and operations per second per device. |

![A container process, sleep 3600, is PID 1 inside its container and a normal host PID outside. Two arrows lead from it. Left, namespaces decide what it can see: pid, its own process tree; net, its own interfaces, routes and ports; mnt, its own mount table and root file system; uts, its own hostname; ipc, its own shared memory and queues; user, root inside maps to an unprivileged user; cgroup, its own view of the cgroup tree. Right, cgroups version 2 decide what it can use: memory.max, a RAM limit above which the OOM killer acts; cpu.max, a CPU quota such as 50000 100000, half a CPU; pids.max, the most processes; io.max, disk limits. A note says these are kernel features, not a hypervisor: the process runs directly on the host kernel.](../../../assets/diagrams/virt-namespaces-cgroups.svg)

*Figure 3. Namespaces limit what a container can see; cgroups limit what it can use. Together with the kernel's other security features, they are what a "container" is.*

So a container is not a thing the kernel knows about. It is an ordinary process, started inside a set of namespaces and a cgroup, with a different root file system. That is why it starts in milliseconds and has almost no overhead. It is also why the isolation is only as strong as the kernel.

Going over the limits behaves differently for each resource:
- **CPU** is *compressible*: a group that uses its quota is **throttled** (paused until the next period), not killed.
- **Memory** is *not*: if a group needs more than `memory.max` and the kernel can't reclaim enough (for example by dropping file cache), the **out-of-memory (OOM) killer** picks a process in that group and sends it `SIGKILL`.

> [!CHECK]
> A container runs with a CPU limit of 0.5 CPU and a memory limit of 256 MB. It keeps trying to use a whole CPU and 300 MB. What happens?
>
> - [ ] The kernel stops the container at once, because it tried to go over a limit
> - [ ] Both limits only slow the container down; it is never killed
> - [x] The CPU use is throttled to about half a CPU, and the memory use triggers the OOM killer
> - [ ] The limits are only hints for the scheduler; the container gets what it needs
>
> CPU is compressible, so the cgroup throttles it. Memory can't be taken back from a running process, so past `memory.max` the OOM killer terminates a process in the group.

### Container images, layers and the OCI formats
A container starts from an **image**: a read-only package holding a root file system plus metadata (which command to run, environment variables, the user). The image is a template; the running **container** is an instance of it. Think of a class and an object, or of a recipe and a meal.

To make images small and quick to share, an image is built from **layers**. Each instruction in a build file (a `Containerfile`, or `Dockerfile` in Docker) that changes the file system, such as `RUN` and `COPY`, adds one layer: a compressed archive holding only the files that changed. Instructions that only change metadata, such as `CMD` and `LABEL`, add no layer. Layers are **content-addressed**: each is named by the **digest** of its content, `sha256:` followed by 64 hexadecimal digits. Identical layers are stored once and shared, so ten images based on the same base image keep that base on disk only once, and pulling an updated image downloads only the layers that changed.

When you start a container, the runtime stacks the image layers with a union file system (usually **overlayfs**) and adds one thin **writable layer** on top, so the container sees a single file system. This works by **copy-on-write**: a lower layer is never modified. If the container changes a file that lives in a lower layer, the file is first copied up into the writable layer and modified there; if it deletes one, a marker (a "whiteout") hides it. When the container is removed, its writable layer goes with it. Anything that must outlive the container belongs in a **volume**, a directory stored outside the container.

![On the left, the image lab-virt-hello:1.0 as three stacked read-only layers shared by every container: layer 1, the Alpine root file system from alpine:3.20.10, 8.09 MB; layer 2, RUN apk add procps-ng, 1.27 MB; layer 3, COPY app.sh, 3.58 kB. A note says CMD and LABEL only change the image configuration and add no layer. On top of the image sit two containers, A and B, each with its own writable copy-on-write layer, deleted with the container. A note explains that writing to a file from a lower layer copies it up first and deleting it leaves a whiteout marker. On the right, the image manifest lists the config (the image ID) and the three layers by sha256 digest, and a card contrasts a tag, a name that can move, with a digest, this exact content forever.](../../../assets/diagrams/virt-image-layers.svg)

*Figure 4. Layers are read-only and shared; each container adds only a thin writable layer on top. The sizes come from the image you build in the lab.*

The **Open Container Initiative (OCI)**, a vendor-neutral project under the Linux Foundation, defines the standards that let tools from different vendors work together. That is why an image built with Podman runs on Docker, containerd or a Kubernetes cluster, and the other way round. There are three specifications:

| Specification | Defines |
|---|---|
| **Image specification** | The image format: a **manifest** (lists the config and the layers by digest), a **configuration** (command, environment, history) and the layer archives. |
| **Runtime specification** | How to run an unpacked image: a root file system and a `config.json` that lists the namespaces, cgroup limits, mounts and capabilities. **runc** is the reference runtime; **crun** is another. |
| **Distribution specification** | The HTTP API a **registry** uses to push and pull images. |

A **registry** stores and serves images (examples: Docker Hub, Quay.io, GitHub Container Registry, and the registries of the public clouds). An image reference has the form `registry/repository:tag`, for example `docker.io/library/alpine:3.20.10`. Two ways to name an image behave very differently:
- A **tag** (`3.20.10`, `latest`) is a human-friendly label that the publisher can **move** to a different image at any time. `latest` is only a default tag name; it doesn't mean "newest".
- A **digest** (`alpine@sha256:<64 hex digits>`) identifies exact content. It can never point to anything else.

For repeatable builds, pin the base image to an exact version tag (`alpine:3.20.10`, not `alpine` or `alpine:latest`), and pin by digest where you need a guarantee that the content hasn't changed.

> [!CHECK]
> A `Containerfile` has `FROM alpine:3.20.10`, then `RUN apk add --no-cache curl`, then `COPY app.sh /app.sh` and `CMD ["/app.sh"]`. How many new layers does the build add on top of the base image?
>
> - [ ] Four, one per instruction
> - [x] Two: the `RUN` and the `COPY`
> - [ ] One, because the build merges everything into a single layer
> - [ ] Three: `RUN`, `COPY` and `CMD`
>
> Only instructions that change the file system add a layer. `CMD` (like `LABEL`) only changes the image configuration, and `FROM` selects the base layers rather than adding one.

### Running containers safely: rootless mode, limits and the OOM killer
By default, many container engines run as root. If something escapes the container, it lands on the host as root. Two features reduce that risk.

**Rootless containers** run the engine and the container as an ordinary user. With a **user namespace**, UID 0 *inside* the container is mapped to your unprivileged UID *outside*, and the other container UIDs map to a range of subordinate UIDs reserved for you in `/etc/subuid`. The container thinks it is root, but on the host it has only your rights. Rootless Podman works like this, and Docker also supports a rootless mode. Podman has no central daemon; each container is started by the user's own process. The trade-offs: a rootless container can't do things that need real root, such as binding to ports below 1024 by default, loading kernel modules or changing the host's network, and its networking goes through a user-space helper (such as `pasta` or `slirp4netns`), which is slower than a root network setup.

**Resource limits** protect the host and the neighbours from a runaway container. In Podman and Docker they are options of `run`:

| Option | Sets |
|---|---|
| `--memory 64m` | `memory.max` (Podman also sets `memory.swap.max` to the same value) |
| `--cpus 0.5` | `cpu.max` (quota 50000 per period 100000) |
| `--pids-limit 100` | `pids.max` |

A container stopped by the OOM killer ends with **exit code 137**, which is 128 plus signal 9 (`SIGKILL`). Exit code 137 alone does not prove an out-of-memory event: any `SIGKILL` gives it, for example when `stop` times out and the engine kills the container. To diagnose it:
1. Check the engine: `podman inspect <name> --format '{{.State.OOMKilled}}'` is `true` when the cgroup's OOM killer acted.
2. Check the kernel log (`journalctl -k` or `dmesg`) for the line `Memory cgroup out of memory: Killed process ...`.
3. Check the cgroup's `memory.events`: the `oom_kill` counter.
4. Fix it: find out why the process needs that much memory (a leak, a big cache or a bad size estimate), then fix the code or raise the limit. A limit that is too low for the real workload only turns slowness into crashes.

> [!CHECK]
> A container exited with code 137. `podman inspect` shows `OOMKilled=false`. What is the most likely explanation?
>
> - [ ] The container used too much CPU and was throttled to death
> - [ ] The kernel always uses 137 when a container finishes normally
> - [x] Some process sent `SIGKILL` to it for another reason, for example the engine after a stop timeout
> - [ ] The image has too many layers
>
> Exit code 137 means "killed by signal 9". The OOM killer is only one possible sender. `OOMKilled=false` tells you that the memory cgroup did not do it. CPU limits throttle, they never kill.

## How it works
Here is what happens when you type `podman run --memory 64m alpine:3.20.10 sleep 3600` as an ordinary user:

1. **Resolve the image.** The engine looks for `alpine:3.20.10` in the local store. If it isn't there, it asks the registry (distribution specification) for the manifest, then downloads the config and each layer by digest, skipping those it already has, and checks every digest.
2. **Unpack the layers.** The runtime stacks the layers with overlayfs and adds a new empty writable layer.
3. **Write the runtime configuration.** The engine produces an OCI runtime `config.json`: the command, the namespaces to create, the cgroup limits (`memory.max` = 67108864), the mounts and the capabilities to drop.
4. **Create the isolated process.** A low-level runtime (`crun` or `runc`) creates the namespaces, places the process in a new cgroup directory below `/sys/fs/cgroup`, applies the limits, switches the root file system, drops privileges and `exec`s `sleep 3600`. In rootless mode a user namespace comes first, so the later steps are allowed without real root.
5. **Run.** From now on, `sleep` is a normal process, scheduled by the host kernel. It is PID 1 in its own pid namespace, and has a normal PID on the host.
6. **Stop.** When the process exits, the runtime removes the namespaces and the cgroup. When the container is removed, its writable layer is deleted; the image stays in the store.

A VM start-up, by contrast, has to start the hypervisor's virtual hardware, run the firmware, boot a whole kernel and start system services before your application even begins. That is the difference in startup time you saw in the table.

## Worked example
First, check whether your CPU supports hardware virtualization and whether Linux exposes the KVM device (output from an AMD machine; an Intel machine shows `vmx` and `VT-x`):

```bash
grep -o -m1 -w -E 'vmx|svm' /proc/cpuinfo
lscpu | grep -i virtualization
ls -l /dev/kvm
```

```text
svm
Virtualization:                          AMD-V
crw-rw-rw-. 1 root kvm 10, 232 Nov  7 19:11 /dev/kvm
```

If `grep` prints nothing, either the CPU lacks the extensions or they are disabled in the firmware; inside a VM, nested virtualization must be enabled by the host.

Now, how do the limits you pass to `podman run` become cgroup values?

| You ask for | The kernel stores | Calculation |
|---|---|---|
| `--memory 64m` | `memory.max` = `67108864` | 64 × 1024 × 1024 bytes |
| `--cpus 0.5` | `cpu.max` = `50000 100000` | 0.5 × 100,000 µs period = 50,000 µs of CPU per period |
| `--pids-limit 20` | `pids.max` = `20` | the number as given |

Finally, a diagnosis. A team runs a container with `--memory 64m` that builds a 200 MB string in memory. It dies after about a second:

```text
$ podman inspect report-job --format 'OOMKilled={{.State.OOMKilled}} ExitCode={{.State.ExitCode}}'
OOMKilled=true ExitCode=137
```

```text
kernel: Memory cgroup out of memory: Killed process 18629 (sh) total-vm:136808kB, anon-rss:60812kB ...
```

`OOMKilled=true` plus the kernel log line prove that the memory limit was the cause: the process had about 60 MB resident when it reached the 64 MB limit (which also counts some kernel memory). The fix is to process the data in a stream instead of holding all of it, or, if 200 MB is really needed, to raise the limit (for example `--memory 512m`) *and* check that the host has that memory to give. You reproduce this exact case in the lab.

## Common pitfalls
- **Treating a container as a small VM.** A container is a process with a restricted view. It has no kernel of its own: `uname -r` inside shows the host's kernel, and a kernel vulnerability affects every container on the host.
- **Using `latest` or a short tag.** `alpine` and `latest` change under you, and a rebuild next month may give a different image. Pin an exact version tag, or a digest.
- **Storing data in the writable layer.** It disappears with the container. Use a volume for data that must survive.
- **Forgetting that exit code 137 isn't a diagnosis.** It means "SIGKILL"; confirm out-of-memory with `OOMKilled` and the kernel log before you change anything.
- **Setting no limits.** A container without `--memory` can use all the host's memory, and the OOM killer may then pick a process that belongs to someone else.
- **Reading CPU "limits" as reservations.** `--cpus 0.5` caps the container; it doesn't guarantee it half a CPU when the host is busy. Use weights (or scheduler requests in an orchestrator) for guarantees.
- **Running as root inside the container "because it works".** Add a non-root `USER` in the image and drop what the app doesn't need.

## Security considerations
Virtualization and containers are isolation tools, and each has a different attack surface. **NIST SP 800-125** covers the security of full virtualization (hypervisors, guests and management), and **NIST SP 800-190**, the Application Container Security Guide, covers containers. This section follows the structure of SP 800-190: images, registries, orchestrators, containers and the host OS.

**Isolation strength.** A VM escape (a guest breaking out through a hypervisor bug) is rare and serious, because the hypervisor is a small, well-tested interface. A container escape has a larger target: the whole kernel interface that all containers share. Kernel and runtime bugs, such as the `runc` flaw CVE-2019-5736, have let a container overwrite the host's runtime. For workloads from different tenants that don't trust each other, use VMs or microVMs, or put sandboxed runtimes around the containers.

**Images.** The risks in SP 800-190 start here:
- *Vulnerabilities in the image.* An image bundles an OS layer and libraries that age. Rebuild regularly, scan images with a vulnerability scanner and keep them small (a minimal base image has fewer packages to patch).
- *Embedded secrets.* Passwords, tokens and keys copied into a layer stay in it forever, even if a later layer deletes the file. Pass secrets at run time through a secrets mechanism.
- *Untrusted or tampered images.* Use images from sources you trust, pin versions or digests, and verify signatures (for example with Sigstore's `cosign`) where your registry supports it.

**Registries.** Use TLS for every connection, require authentication, restrict who can push, and remove stale images that keep old vulnerabilities available.

**Containers and the host.** The defaults of the runtime already help: a restricted set of Linux capabilities, a default seccomp profile (a filter on system calls) and, on Fedora-family and Debian-family systems, SELinux or AppArmor profiles. Don't turn them off without a strong reason:
- Avoid `--privileged`: it removes nearly all the isolation. Avoid `--cap-add` beyond the minimum.
- Never mount the engine's socket (`/var/run/docker.sock`) into a container: whoever controls it controls the host.
- Don't mount the host's sensitive paths (`/`, `/etc`, `/proc`) into a container, and mount volumes read-only when you can (`:ro`). Use `--read-only` for the root file system when the app allows it.
- Prefer **rootless** engines and a non-root `USER` in the image: then even a successful escape lands as an ordinary user.
- Set memory, CPU and PID limits so one container can't starve the others.
- Run a minimal, regularly patched host OS dedicated to containers, because every container shares its kernel.

**Network.** Containers can talk to each other and to the network by default. Publish only the ports you need, and bind to `127.0.0.1` when a service is for local use.

> [!IMPORTANT]
> The lab runs only containers you start yourself, on your own machine, with names starting `lab-virt-`, without root and without `--privileged`. Don't practise escape techniques on shared hosts or on any system you don't own.

## Summary
- Hardware virtualization runs complete guest operating systems on one host; the hypervisor shares and isolates the hardware, and CPU extensions (Intel VT-x, AMD-V) make it fast. Type 1 hypervisors run on the hardware, type 2 hypervisors on a host OS.
- A VM virtualizes hardware and has its own kernel: strong isolation, slower startup, more overhead. A container virtualizes the OS view and shares the host kernel: fast and dense, but only as isolated as that kernel.
- Namespaces (pid, net, mnt, uts, ipc, user, cgroup) decide what a container can see; cgroups v2 (`memory.max`, `cpu.max`, `pids.max`) decide what it can use. A container is a normal process inside both.
- An image is a stack of read-only, content-addressed layers plus a configuration; a container adds a writable copy-on-write layer. OCI standardizes the image, runtime and distribution formats. Tags can move, digests can't.
- Memory over the limit triggers the OOM killer (exit code 137, `OOMKilled=true`); CPU over the limit is only throttled. Run rootless, set limits, and follow NIST SP 800-190.

## Glossary
| Term | Definition |
|---|---|
| Virtualization | Running several isolated virtual computers on one physical computer. |
| Host | The physical machine (or the OS on it) that runs guests or containers. |
| Guest | A virtual machine and the operating system running inside it. |
| Virtual machine (VM) | A software-defined computer with virtual CPUs, memory, disks and network cards, running its own OS and kernel. |
| Hypervisor | Software that creates and runs VMs and shares and isolates the hardware between them; also called virtual machine monitor (VMM). |
| Type 1 hypervisor | A hypervisor that runs directly on the hardware (bare metal), for example KVM, Xen, ESXi, Hyper-V. |
| Type 2 hypervisor | A hypervisor that runs as an application on a host OS, for example VirtualBox, VMware Workstation. |
| Intel VT-x, AMD-V | CPU extensions for hardware-assisted virtualization; flags `vmx` and `svm`. |
| VM exit | The moment the CPU stops running a guest and gives control back to the hypervisor. |
| EPT, NPT | Intel and AMD hardware support for translating guest memory addresses to real ones (second-level address translation). |
| IOMMU | Hardware that controls and translates device access to memory, used to give a guest direct access to a device safely (Intel VT-d, AMD-Vi). |
| Paravirtualization | Guest drivers made for virtual hardware (for example virtio) that are faster than pretending to be real devices. |
| Kernel | The core of an operating system, which manages hardware, processes and memory. |
| Container | A process, or group of processes, that runs on the host kernel with its own namespaces, cgroup and root file system. |
| Namespace | A kernel feature that gives processes a private view of a resource (pid, net, mnt, uts, ipc, user, cgroup, time). |
| Control group (cgroup) | A kernel feature that groups processes and limits or accounts for the resources they use. |
| cgroups v2 | The current cgroup design, with one unified hierarchy at `/sys/fs/cgroup` and one control file per setting. |
| Controller | A cgroup component that manages one resource: memory, cpu, io, pids. |
| Throttling | Pausing a cgroup that has used its CPU quota until the next period. |
| OOM killer | The kernel function that kills a process when memory runs out, or when a memory cgroup reaches `memory.max` and can't reclaim memory. |
| Exit code 137 | 128 + 9: the process was killed by `SIGKILL`; with `OOMKilled=true` it was the OOM killer. |
| Image | A read-only package with a root file system and metadata, used as a template to start containers. |
| Layer | A content-addressed archive with the files one build step changed; layers are stacked to form an image. |
| Containerfile (Dockerfile) | A text file with the build steps of an image. |
| Digest | The SHA-256 hash of a manifest or layer (`sha256:…`); it identifies exact content. |
| Tag | A movable, human-friendly label for an image version, such as `3.20.10` or `latest`. |
| Copy-on-write | Changing a file from a lower, read-only layer by first copying it into the writable layer. |
| overlayfs | The union file system that stacks image layers and a writable layer into a single view. |
| Volume | Storage outside the container's writable layer that survives the container. |
| Registry | A server that stores and distributes images (for example Docker Hub, Quay.io). |
| Open Container Initiative (OCI) | The vendor-neutral project that defines the image, runtime and distribution specifications. |
| runc, crun | Low-level OCI runtimes that create the namespaces and cgroups and start the container process. |
| Rootless container | A container run by an ordinary user, with a user namespace that maps container root to an unprivileged host user. |
| Capability | One of the fine-grained pieces of root's power in Linux, which a container can keep or drop. |
| seccomp | A kernel filter that restricts which system calls a process may use. |
| microVM | A very small, fast-booting VM used to isolate a single workload (for example Firecracker). |

## Check your understanding
1. Explain in your own words why a VM starts in seconds to minutes and a container in milliseconds. What does each have to start?
2. A colleague says "containers are just lightweight VMs". Give two things that are different, one about the kernel and one about security.
3. Two containers run the same image. Container A writes a 50 MB file inside its own file system. Does container B see it? Where does the file live, and when is it deleted?
4. Your `Containerfile` starts with `FROM python`. Give two ways in which this can break a build that worked last month, and rewrite the line to prevent both.
5. A container is started with `--memory 128m --cpus 1`. Which `memory.max` and `cpu.max` values do you expect in the cgroup files? What happens when it needs 200 MB of memory? And when it needs two CPUs?
6. A database container exits with code 137. List the steps you take to find out whether it ran out of memory, and name two possible fixes.
7. A team wants to run code from different customers on the same server. Compare VMs, containers and sandboxed containers for this, and say which controls from NIST SP 800-190 you would require.

## References
- [NIST SP 800-125](https://csrc.nist.gov/pubs/sp/800/125/final) — Guide to Security for Full Virtualization Technologies.
- [NIST SP 800-190](https://csrc.nist.gov/pubs/sp/800/190/final) — Application Container Security Guide.
- [OCI Image Format Specification](https://github.com/opencontainers/image-spec).
- [OCI Runtime Specification](https://github.com/opencontainers/runtime-spec).
- [OCI Distribution Specification](https://github.com/opencontainers/distribution-spec).
- [namespaces(7)](https://man7.org/linux/man-pages/man7/namespaces.7.html) — Linux manual page.
- [cgroups(7)](https://man7.org/linux/man-pages/man7/cgroups.7.html) — Linux manual page.
- [Control Group v2](https://docs.kernel.org/admin-guide/cgroup-v2.html) — Linux kernel documentation.
- [Podman documentation](https://docs.podman.io/).
- [CVE-2019-5736](https://nvd.nist.gov/vuln/detail/CVE-2019-5736) — runc container breakout vulnerability.
