Showing raw Markdown: the renderer could not load.
## Goal
You start a small web server, flood it from two "bot" containers on an isolated network and watch a normal user get stuck. You find the attackers in the access log. Then you turn on rate limiting, repeat the flood, and see the excess requests refused with HTTP status 429 while the normal user is served at full speed.
## Objectives
- Measure a web server's normal latency (baseline) and its latency under a flood.
- Detect the flood by counting requests per client address and per status code in the access log.
- Enable nginx rate limiting (`limit_req`) and connection limiting (`limit_conn`) and reload it without downtime.
- Show that the limit refuses the flood (status 429) but still serves a normal client (status 200).
- Read the SYN-cookie setting of the host and of a container and explain what it does.
## Prerequisites
- A Linux machine with **Podman 4 or later** running **rootless** (as a normal user; you never need `sudo`) and the netavark network backend. Tested with Podman 5.8.7 on Fedora with kernel 7.2. Check with `podman info --format '{{.Host.NetworkBackend}}'`: it should print `netavark`.
- Internet access once, to download the nginx and Alpine Linux images (a few tens of MB in total) and two small packages (about 8 MB). After that, nothing in the lab needs the Internet.
- About 400 MB of free disk space and at least 2 CPU cores.
- The [lesson](lesson.md) of this module. The Podman techniques are the ones of the [Network Security lab](../06-network-security/lab.md): `--internal` networks, pinned images, rootless containers.
> [!WARNING]
> **Authorization and ethics.** Flooding a system that you don't own or aren't explicitly authorized in writing to test is illegal in most countries, and it harms other people even if you call it a test. In this lab the "attack" runs only between four containers that you create yourself, on a network created with `--internal` (nothing leaves your machine). The load is modest and capped by time (15 seconds) and by the number of workers. Never point the tools of this lab at anything else, not even your own server at a hosting provider, without reading the provider's rules on load testing first.
## Architecture
<figure class="diagram" style="--dg-w: 1180px"><svg xmlns="http://www.w3.org/2000/svg" class="dg" viewBox="0 0 1180 560" width="1180" height="560" role="img" aria-labelledby="ddos-lab-architecture-title ddos-lab-architecture-desc" font-family="Inter, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" xml:lang="en">
<title id="ddos-lab-architecture-title">Lab architecture: one web server, two bots and one normal user on an isolated network</title>
<desc id="ddos-lab-architecture-desc">One internal Podman network, lab-ddos-net, 10.60.0.0/24. The web server container lab-ddos-web is at 10.60.0.10 on tcp 80, runs nginx 1.31.6-alpine with a CPU limit of 0.5 and a memory limit of 128 MB, and serves /report.html, a page that nginx compresses with gzip level 9 on every request. Its limits.conf starts empty and is later filled with limit_req 5 requests per second with a burst of 10, limit_conn 10 per address, and status 429 when exceeded. Two bot containers, lab-ddos-bot1 at 10.60.0.20 and lab-ddos-bot2 at 10.60.0.21, each run hey for 15 seconds with 25 concurrent workers and send a flood of requests to the web server on thick red lines. A normal user container, lab-ddos-user at 10.60.0.30, sends one normal request with curl on a thin green line. On the left, a card for the folder ~/ddos-lab on your host lists nginx.conf, conf/limits.conf and www/report.html, mounted read-only into the web container. A note says the network is internal: nothing leaves your machine.</desc>
<rect class="dg-bg" x="0.5" y="0.5" width="1179" height="559" rx="16" fill="#ffffff" stroke="#e3e3dd"/>
<g class="dg-l1"><rect class="dg-tint" x="300" y="20" width="860" height="470" rx="16" fill="#f0f6f9" stroke="#9ac3d9" stroke-width="1.2" stroke-dasharray="6 5"/>
<g class="dg-icon" transform="translate(316 33) scale(0.7500)" fill="none" stroke="#0369a1" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M17.5 19H9a7 7 0 1 1 6.71-9h1.79a4.5 4.5 0 1 1 0 9Z"/></g>
<text x="342" y="47" font-size="13" class="dg-ink" fill="#0369a1" font-weight="700">lab-ddos-net · 10.60.0.0/24 · --internal</text></g>
<rect class="dg-card" x="20" y="130" width="240" height="150" rx="12" fill="#ffffff" stroke="#cfcfc8"/>
<g class="dg-l3"><g class="dg-icon" transform="translate(36 146) scale(1.2500)" fill="none" stroke="#b45309" stroke-width="1.8" stroke-linecap="round" stroke-linejoin="round"><path d="M15 2H6a2 2 0 0 0-2 2v16a2 2 0 0 0 2 2h12a2 2 0 0 0 2-2V7Z"/><path d="M14 2v4a2 2 0 0 0 2 2h4"/><path d="M10 9H8"/><path d="M16 13H8"/><path d="M16 17H8"/></g></g>
<g class="dg-l3"><text x="78" y="167" font-size="14" class="dg-ink" fill="#b45309" font-weight="700">~/ddos-lab on your host</text></g>
<text x="38" y="198" font-size="12.5" class="dg-text" fill="#17171c" font-weight="500" font-family="JetBrains Mono, ui-monospace, Menlo, Consolas, monospace">nginx.conf</text>
<text x="38" y="218" font-size="12.5" class="dg-text" fill="#17171c" font-weight="500" font-family="JetBrains Mono, ui-monospace, Menlo, Consolas, monospace">conf/limits.conf</text>
<text x="38" y="238" font-size="12.5" class="dg-text" fill="#17171c" font-weight="500" font-family="JetBrains Mono, ui-monospace, Menlo, Consolas, monospace">www/report.html</text>
<g class="dg-l3"><path class="dg-stroke" d="M260 205H460" stroke="#b45309" stroke-width="2.5" fill="none" stroke-linecap="round" stroke-linejoin="round" stroke-dasharray="7 6"/></g>
<g class="dg-l3"><path class="dg-stroke" d="M471 199l8 6-8 6" stroke="#b45309" stroke-width="2.5" fill="none" stroke-linecap="round" stroke-linejoin="round"/></g>
<text x="20" y="308" font-size="11.5" class="dg-muted" fill="#5c5c68" font-weight="500">mounted read-only in the web container</text>
<rect class="dg-card" x="480" y="80" width="320" height="280" rx="12" fill="#ffffff" stroke="#cfcfc8"/>
<g class="dg-l4"><g class="dg-icon" transform="translate(496 96) scale(1.2500)" fill="none" stroke="#c2410c" stroke-width="1.8" stroke-linecap="round" stroke-linejoin="round"><rect width="20" height="8" x="2" y="2" rx="2"/><rect width="20" height="8" x="2" y="14" rx="2"/><path d="M6 6h.01"/><path d="M6 18h.01"/></g></g>
<g class="dg-l4"><text x="538" y="117" font-size="15" class="dg-ink" fill="#c2410c" font-weight="700">lab-ddos-web</text></g>
<text x="498" y="148" font-size="12.5" class="dg-text" fill="#17171c" font-weight="500" font-family="JetBrains Mono, ui-monospace, Menlo, Consolas, monospace">eth0 10.60.0.10/24 · tcp 80</text>
<text x="498" y="168" font-size="12.5" class="dg-text" fill="#17171c" font-weight="500" font-family="JetBrains Mono, ui-monospace, Menlo, Consolas, monospace">nginx 1.31.6-alpine</text>
<text x="498" y="188" font-size="12.5" class="dg-text" fill="#17171c" font-weight="500" font-family="JetBrains Mono, ui-monospace, Menlo, Consolas, monospace">--cpus 0.5 · --memory 128m</text>
<text x="498" y="208" font-size="13" class="dg-muted" fill="#5c5c68" font-weight="500">/report.html: gzip level 9 on every request</text>
<text x="498" y="250" font-size="12" class="dg-muted" fill="#5c5c68" font-weight="700">limits.conf: off, then on</text>
<rect class="dg-tint" x="494" y="262" width="292" height="24" rx="8" fill="#ecf5ef" stroke="#a1ccb1"/>
<text x="504" y="278.5" font-size="11.5" class="dg-text" fill="#17171c" font-weight="600" font-family="JetBrains Mono, ui-monospace, Menlo, Consolas, monospace">limit_req 5r/s, burst 10</text>
<rect class="dg-tint" x="494" y="292" width="292" height="24" rx="8" fill="#ecf5ef" stroke="#a1ccb1"/>
<text x="504" y="308.5" font-size="11.5" class="dg-text" fill="#17171c" font-weight="600" font-family="JetBrains Mono, ui-monospace, Menlo, Consolas, monospace">limit_conn 10 per address</text>
<rect class="dg-tint" x="494" y="322" width="292" height="24" rx="8" fill="#ecf5ef" stroke="#a1ccb1"/>
<text x="504" y="338.5" font-size="11.5" class="dg-text" fill="#17171c" font-weight="600" font-family="JetBrains Mono, ui-monospace, Menlo, Consolas, monospace">status 429 when exceeded</text>
<rect class="dg-card" x="900" y="60" width="240" height="94" rx="12" fill="#ffffff" stroke="#cfcfc8"/>
<g class="dg-warn"><g class="dg-icon" transform="translate(916 76) scale(1.2500)" fill="none" stroke="#b91c1c" stroke-width="1.8" stroke-linecap="round" stroke-linejoin="round"><path d="M20 16V7a2 2 0 0 0-2-2H6a2 2 0 0 0-2 2v9m16 0H4m16 0 1.28 2.55a1 1 0 0 1-.9 1.45H3.62a1 1 0 0 1-.9-1.45L4 16"/></g></g>
<g class="dg-warn"><text x="958" y="97" font-size="13.5" class="dg-ink" fill="#b91c1c" font-weight="700">lab-ddos-bot1</text></g>
<text x="918" y="128" font-size="12.5" class="dg-text" fill="#17171c" font-weight="500" font-family="JetBrains Mono, ui-monospace, Menlo, Consolas, monospace">eth0 10.60.0.20/24</text>
<text x="918" y="148" font-size="12.5" class="dg-muted" fill="#5c5c68" font-weight="500" font-family="JetBrains Mono, ui-monospace, Menlo, Consolas, monospace">hey -z 15s -c 25 -q 20</text>
<rect class="dg-card" x="900" y="170" width="240" height="94" rx="12" fill="#ffffff" stroke="#cfcfc8"/>
<g class="dg-warn"><g class="dg-icon" transform="translate(916 186) scale(1.2500)" fill="none" stroke="#b91c1c" stroke-width="1.8" stroke-linecap="round" stroke-linejoin="round"><path d="M20 16V7a2 2 0 0 0-2-2H6a2 2 0 0 0-2 2v9m16 0H4m16 0 1.28 2.55a1 1 0 0 1-.9 1.45H3.62a1 1 0 0 1-.9-1.45L4 16"/></g></g>
<g class="dg-warn"><text x="958" y="207" font-size="13.5" class="dg-ink" fill="#b91c1c" font-weight="700">lab-ddos-bot2</text></g>
<text x="918" y="238" font-size="12.5" class="dg-text" fill="#17171c" font-weight="500" font-family="JetBrains Mono, ui-monospace, Menlo, Consolas, monospace">eth0 10.60.0.21/24</text>
<text x="918" y="258" font-size="12.5" class="dg-muted" fill="#5c5c68" font-weight="500" font-family="JetBrains Mono, ui-monospace, Menlo, Consolas, monospace">hey -z 15s -c 25 -q 20</text>
<g class="dg-warn"><path class="dg-stroke" d="M900 107H850V135H810" stroke="#b91c1c" stroke-width="6" fill="none" stroke-linecap="round" stroke-linejoin="round"/></g>
<g class="dg-warn"><path class="dg-stroke" d="M900 217H850V165H810" stroke="#b91c1c" stroke-width="6" fill="none" stroke-linecap="round" stroke-linejoin="round"/></g>
<g class="dg-warn"><path class="dg-stroke" d="M811 129l-8 6 8 6" stroke="#b91c1c" stroke-width="2.5" fill="none" stroke-linecap="round" stroke-linejoin="round"/></g>
<g class="dg-warn"><path class="dg-stroke" d="M811 159l-8 6 8 6" stroke="#b91c1c" stroke-width="2.5" fill="none" stroke-linecap="round" stroke-linejoin="round"/></g>
<g class="dg-warn"><rect class="dg-bg-plain" x="824.2" y="138" width="51.6" height="24" rx="12" fill="#ffffff"/><text x="850" y="154.5" font-size="12" class="dg-ink" fill="#b91c1c" font-weight="600" text-anchor="middle">flood</text></g>
<rect class="dg-card" x="900" y="330" width="240" height="94" rx="12" fill="#ffffff" stroke="#cfcfc8"/>
<g class="dg-l2"><g class="dg-icon" transform="translate(916 346) scale(1.2500)" fill="none" stroke="#15803d" stroke-width="1.8" stroke-linecap="round" stroke-linejoin="round"><path d="M20 16V7a2 2 0 0 0-2-2H6a2 2 0 0 0-2 2v9m16 0H4m16 0 1.28 2.55a1 1 0 0 1-.9 1.45H3.62a1 1 0 0 1-.9-1.45L4 16"/></g></g>
<g class="dg-l2"><text x="958" y="367" font-size="13.5" class="dg-ink" fill="#15803d" font-weight="700">lab-ddos-user</text></g>
<text x="918" y="398" font-size="12.5" class="dg-text" fill="#17171c" font-weight="500" font-family="JetBrains Mono, ui-monospace, Menlo, Consolas, monospace">eth0 10.60.0.30/24</text>
<text x="918" y="418" font-size="12.5" class="dg-muted" fill="#5c5c68" font-weight="500" font-family="JetBrains Mono, ui-monospace, Menlo, Consolas, monospace">curl: one normal request</text>
<g class="dg-l2"><path class="dg-stroke" d="M900 377H850V300H810" stroke="#15803d" stroke-width="2.5" fill="none" stroke-linecap="round" stroke-linejoin="round"/></g>
<g class="dg-l2"><path class="dg-stroke" d="M811 294l-8 6 8 6" stroke="#15803d" stroke-width="2.5" fill="none" stroke-linecap="round" stroke-linejoin="round"/></g>
<g class="dg-l2"><rect class="dg-bg-plain" x="810.76" y="328" width="78.48" height="24" rx="12" fill="#ffffff"/><text x="850" y="344.5" font-size="12" class="dg-ink" fill="#15803d" font-weight="600" text-anchor="middle">1 request</text></g>
<g class="dg-l1"><rect class="dg-tint" x="416" y="516" width="348" height="26" rx="13" fill="#e6f0f6" stroke="#8ebcd5" stroke-width="1.2"/><text x="590" y="533.5" font-size="12" class="dg-ink" fill="#0369a1" font-weight="700" text-anchor="middle">Internal network: nothing leaves your machine</text></g>
</svg></figure>
*Figure 1. Two bots and one normal user share one small web server. At first nothing limits the bots; later `limits.conf` does.*
| Container | Address | Role |
|---|---|---|
| `lab-ddos-web` | `10.60.0.10`, TCP 80 | nginx, limited to half a CPU and 128 MB, serving `/report.html` |
| `lab-ddos-bot1` | `10.60.0.20` | A "bot": floods the web server with `hey` |
| `lab-ddos-bot2` | `10.60.0.21` | A second bot (so the attack comes from more than one address) |
| `lab-ddos-user` | `10.60.0.30` | A normal user: a few `curl` requests |
**Why this page is "expensive".** nginx serves a small static file almost for free, and no tiny lab could flood it. So `/report.html` is a large text file (about 900 KB) that nginx must **compress with gzip on every request**, which uses real processor time. The web container may use only half a CPU, and nginx runs one worker process. That gives the server a capacity of roughly 20 requests per second, which is easy to exceed in a lab, and it stands for any page that is costly to build (a search, a report, a database query).
> [!NOTE]
> One address flooding a server is strictly a DoS attack. The lab uses two bot containers to show that limits are applied per address, and to make you think about what happens with thousands. The conclusions are the same.
## Steps
### Step 1 — Check your tools and create a working folder (3 min)
Make sure Podman is rootless and uses netavark, and that nothing is left over from an earlier attempt.
```bash
podman --version
podman info --format '{{.Host.NetworkBackend}} rootless={{.Host.Security.Rootless}}'
podman ps -a --filter name=lab-ddos --format '{{.Names}}'
podman network ls --filter name=lab-ddos --format '{{.Name}}'
mkdir -p ~/ddos-lab && cd ~/ddos-lab
```
Expected output (your version may differ):
```text
podman version 5.8.7
netavark rootless=true
```
The two `ls` commands print nothing. If they list `lab-ddos-` names, run the [Cleanup](#cleanup) first. All commands that follow assume that you are in `~/ddos-lab`. If you open a new terminal, run `cd ~/ddos-lab` again.
### Step 2 — Build the tools image (4 min)
The bots and the user need two tools: `curl` (one request, with timing) and `hey` (an HTTP load generator: many requests at once, with a summary). Alpine Linux packages both. The tag is pinned to Alpine 3.20.10.
```bash
cat > Containerfile <<'EOT'
FROM docker.io/library/alpine:3.20.10
RUN apk add --no-cache hey curl
EOT
podman build -t lab-ddos-tools:1 . 2>&1 | tail -4
podman run --rm lab-ddos-tools:1 sh -c 'cat /etc/alpine-release; apk list -I 2>/dev/null | grep -E "^(hey|curl)-"'
```
Expected output (the image ID and package versions may differ slightly):
```text
COMMIT lab-ddos-tools:1
--> 5b5d9cc480f1
Successfully tagged localhost/lab-ddos-tools:1
5b5d9cc480f1dd3f292ce2e615536cf0b39fcbd2c72460b0be3b8fc2e8c51b88
3.20.10
curl-8.14.1-r2 x86_64 {curl} (curl) [installed]
hey-0.1.4-r26 x86_64 {hey} (Apache-2.0) [installed]
```
The build uses your normal network once, to download the packages. The lab containers will not have any outside network.
### Step 3 — Write the web server's files (5 min)
nginx needs three files, kept on your host and mounted into the container. Here is what each one does:
- `www/report.html` is the expensive page: 150,000 lines of numbers.
- `nginx.conf` is the whole server configuration. It turns on gzip level 9 (the expensive part), sends the access log to standard output so you can read it with `podman logs`, and **defines** two counters per client address (`limit_req_zone`, `limit_conn_zone`). Defining them doesn't limit anything yet.
- `conf/limits.conf` is included by the server block. It's empty for now. In Step 8 you put the limits in it.
```bash
mkdir -p conf www
seq 1 150000 > www/report.html
echo '# no limits yet' > conf/limits.conf
cat > nginx.conf <<'EOT'
worker_processes 1;
error_log /dev/stderr notice;
pid /tmp/nginx.pid;
events {
worker_connections 1024;
}
http {
include /etc/nginx/mime.types;
access_log /dev/stdout;
# An "expensive" page: nginx compresses it on every request, which costs CPU.
gzip on;
gzip_comp_level 9;
# Two counters, one per client address. They do nothing until limits.conf uses them.
limit_req_zone $binary_remote_addr zone=perip:1m rate=5r/s;
limit_conn_zone $binary_remote_addr zone=connperip:1m;
server {
listen 80;
root /usr/share/nginx/html;
location /report.html {
alias /srv/www/report.html;
}
include /etc/nginx/lab/limits.conf;
}
}
EOT
wc -lc www/report.html
```
Expected output:
```text
150000 938895 www/report.html
```
The file has 150,000 lines and 938,895 bytes (about 0.9 MB). `rate=5r/s` is the limit you will turn on later: five requests per second per client address.
### Step 4 — Create the isolated network and the four containers (5 min)
One Podman network is the whole lab. `--internal` means that Podman adds no route to the outside world and no NAT to your host, so nothing can leave your machine. `10.60.0.0/24` is a private range ([RFC 1918](https://www.rfc-editor.org/rfc/rfc1918)).
The web container gets `--cpus 0.5` and `--memory 128m`: it can use at most half a CPU, so it is easy to overload and can never harm your machine. `:ro,Z` mounts the files read-only (the `Z` label makes them readable on systems with SELinux, such as Fedora, and does nothing elsewhere). The image tag is pinned to an exact version. The other three containers just `sleep infinity`; you run commands in them with `podman exec`.
```bash
podman network create --internal --subnet 10.60.0.0/24 lab-ddos-net
podman network inspect lab-ddos-net --format '{{.Name}} subnet={{(index .Subnets 0).Subnet}} internal={{.Internal}}'
podman run -d --name lab-ddos-web --network lab-ddos-net:ip=10.60.0.10 --cpus 0.5 --memory 128m \
-v ~/ddos-lab/nginx.conf:/etc/nginx/nginx.conf:ro,Z \
-v ~/ddos-lab/conf:/etc/nginx/lab:ro,Z \
-v ~/ddos-lab/www:/srv/www:ro,Z \
docker.io/library/nginx:1.31.6-alpine
podman run -d --name lab-ddos-bot1 --network lab-ddos-net:ip=10.60.0.20 lab-ddos-tools:1 sleep infinity
podman run -d --name lab-ddos-bot2 --network lab-ddos-net:ip=10.60.0.21 lab-ddos-tools:1 sleep infinity
podman run -d --name lab-ddos-user --network lab-ddos-net:ip=10.60.0.30 lab-ddos-tools:1 sleep infinity
podman ps --filter name=lab-ddos --format 'table {{.Names}}\t{{.Status}}'
podman exec lab-ddos-web nginx -v
```
Expected output (the four long container IDs vary; the first run downloads the nginx image):
```text
lab-ddos-net
lab-ddos-net subnet=10.60.0.0/24 internal=true
e296e3434133eb65348ff7d7faddaf2d1ef80540dde41d434314abaab37b7fb4
8dbf7b4179c98cbd8c67e41f33c93215f78b912f4e397d48016d7a14a584be7d
1c262b3347113f409267c40260d01089054015daad18efbe960669ee88773df1
24f0c870b1f753edee63a94317a8c675d407a974608211a5982aca0caed2af03
NAMES STATUS
lab-ddos-web Up Less than a second
lab-ddos-bot1 Up Less than a second
lab-ddos-bot2 Up Less than a second
lab-ddos-user Up Less than a second
nginx version: nginx/1.31.6
```
Now check the isolation. The user container has one route, to the lab network only, and a connection to an outside address (`192.0.2.1` is a documentation address, [RFC 5737](https://www.rfc-editor.org/rfc/rfc5737)) fails at once because there is no route:
```bash
podman exec lab-ddos-user ip route
podman exec lab-ddos-user curl -sS -m 3 -o /dev/null http://192.0.2.1/ ; echo "curl exit code: $?"
```
Expected output:
```text
10.60.0.0/24 dev eth0 scope link src 10.60.0.30
curl: (7) Failed to connect to 192.0.2.1 port 80 after 0 ms: Could not connect to server
curl exit code: 7
```
Exit code 7 means "failed to connect". The attack tools can't reach anything except the web server.
### Step 5 — Baseline: how fast is the server when nobody attacks? (5 min)
Before an attack you need to know what normal looks like. This is your **baseline**. First one request from the normal user, with its timing (`--compressed` asks for gzip, like every browser does, so the server does the costly work). Then 30 requests, one after the other (`-c 1` means one at a time), with `hey`.
```bash
podman exec lab-ddos-user curl -sS --compressed -m 5 -o /dev/null -w 'status=%{http_code} time=%{time_total}s\n' http://10.60.0.10/report.html
podman exec lab-ddos-user hey -n 30 -c 1 http://10.60.0.10/report.html 2>&1 | grep -E 'Total:|Average:|Slowest:|Requests/sec|^ \[[0-9]+\]'
```
Expected output (the times vary; the order of magnitude is what matters):
```text
status=200 time=0.030266s
Total: 1.6079 secs
Slowest: 0.1015 secs
Average: 0.0536 secs
Requests/sec: 18.6580
[200] 30 responses
```
How to read it: `status=200` is "OK". One request takes about 30 to 50 milliseconds (ms), because of the compression. All 30 got `[200]`. The server handles about **18 requests per second** when one client asks one at a time, which is its rough capacity. Note the baseline: **about 0.03 seconds for a normal user**.
### Step 6 — Flood the server and see what a normal user sees (8 min)
Now the "attack". Each bot runs `hey` for 15 seconds (`-z 15s`) with 25 workers (`-c 25`), and each worker tries 20 requests per second (`-q 20`): about 500 requests per second per bot, 1,000 in total, against a server that can answer about 18. The two `hey` commands run in the background (`&`). Three seconds later, the normal user asks for the same page once. `wait` blocks until the bots finish. Your shell may print job lines like `[1] 12345`; they are harmless.
```bash
for b in bot1 bot2; do
podman exec lab-ddos-$b hey -z 15s -c 25 -q 20 http://10.60.0.10/report.html > flood1-$b.txt 2>&1 &
done
sleep 3
podman exec lab-ddos-user curl -sS --compressed -m 60 -o /dev/null -w 'status=%{http_code} time=%{time_total}s\n' http://10.60.0.10/report.html
wait
for b in bot1 bot2; do echo "== $b"; grep -E 'Total:|Average:|Requests/sec|^ \[[0-9]+\]' flood1-$b.txt; done
```
Expected output (it takes about 20 seconds; **the numbers vary, the picture doesn't**):
```text
status=200 time=14.640884s
== bot1
Total: 17.6109 secs
Average: 16.4463 secs
Requests/sec: 1.4764
[200] 26 responses
== bot2
Total: 16.5475 secs
Average: 1.3776 secs
Requests/sec: 17.2232
[200] 285 responses
```
Read it carefully:
- The normal user asked for the page once and **waited about 15 seconds** (in your run: many seconds, not milliseconds) for an answer that took 0.03 seconds before. Nothing was hacked and no data was lost, yet the service is effectively down. This is a denial of service.
- Each bot attempted thousands of requests, but the server completed only about 300 in total in 15 seconds, its capacity. The rest queued up or timed out.
- One bot got most of the answers (here `bot2`: 285 against 26). Which one varies; nginx doesn't share fairly. That is normal.
### Step 7 — Detect it: read the access log (5 min)
The access log is your first detection tool. nginx writes one line per **completed** request to its standard output, and `podman logs` shows it. This command keeps the lines of real requests (`GET`) in a file and counts them: the first column is the client address, the ninth is the HTTP status.
```bash
podman logs lab-ddos-web 2>&1 | awk '$6=="\"GET"' > access-all.txt
wc -l < access-all.txt
awk '{print $1, $9}' access-all.txt | sort | uniq -c | sort -rn
```
Expected output (counts vary slightly):
```text
343
285 10.60.0.21 200
32 10.60.0.30 200
26 10.60.0.20 200
```
The `awk` program prints address and status for each line, `sort | uniq -c` counts equal pairs, and `sort -rn` puts the biggest on top. Reading the table:
- The two addresses `10.60.0.20` and `10.60.0.21` are the **top talkers** among the clients that got through, and in reality they tried far more. The normal user `10.60.0.30` appears with the 32 requests of the baseline and the single request that waited.
- Notice a limit of this method: **the log lists only requests the server completed**. The overload made the server answer fewer, not more, so the log under-reports the attack. A real detection also needs network-level counters (packets and connections per second), load, latency and error rate from monitoring. In the log, the signal is that a few addresses account for most of the traffic.
### Step 8 — Turn on rate limiting and reload (5 min)
Now the defence. The file `conf/limits.conf` gets four lines:
- `limit_req zone=perip burst=10 nodelay;` applies the `perip` counter: each client address may make **5 requests per second** (defined in `nginx.conf`), and may exceed it by a **burst** of 10 extra requests. `nodelay` serves the burst at once instead of slowing it. Requests above that are refused.
- `limit_req_status 429;` makes the refusal HTTP status `429 Too Many Requests` (the default is 503).
- `limit_conn connperip 10;` allows each address at most **10 connections with a request in progress** at the same time. It helps when each request takes a long time to serve (a slow database, a big upload). nginx starts to count a request only after its headers have fully arrived.
- `limit_conn_status 429;` gives it the same status.
```bash
cat > conf/limits.conf <<'EOT'
# At most 5 requests per second per client address, with a burst of 10 extra requests
limit_req zone=perip burst=10 nodelay;
limit_req_status 429;
# At most 10 simultaneous connections per client address
limit_conn connperip 10;
limit_conn_status 429;
EOT
podman exec lab-ddos-web nginx -t
podman exec lab-ddos-web nginx -s reload
```
Expected output (the `notice` line shows the time of your reload):
```text
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
2026/11/07 20:08:04 [notice] 28#28: signal process started
```
`nginx -t` tests the configuration first (always do), and `nginx -s reload` makes nginx start new workers with the new rules and retire the old ones without dropping connections. If the test fails, nothing is reloaded and the old rules stay active.
### Step 9 — Flood again, with the limit on (8 min)
The same attack, the same commands. This time the normal user asks three times, one second apart, like a person reloading a page.
```bash
for b in bot1 bot2; do
podman exec lab-ddos-$b hey -z 15s -c 25 -q 20 http://10.60.0.10/report.html > flood2-$b.txt 2>&1 &
done
sleep 3
for i in 1 2 3; do
podman exec lab-ddos-user curl -sS --compressed -m 60 -o /dev/null -w 'status=%{http_code} time=%{time_total}s\n' http://10.60.0.10/report.html
sleep 1
done
wait
for b in bot1 bot2; do echo "== $b"; grep -E 'Total:|Average:|Requests/sec|^ \[[0-9]+\]' flood2-$b.txt; done
```
Expected output (counts vary by a few; the picture doesn't):
```text
status=200 time=0.030948s
status=200 time=0.029585s
status=200 time=0.029022s
== bot1
Total: 15.0048 secs
Average: 0.0193 secs
Requests/sec: 431.1286
[200] 81 responses
[429] 6388 responses
== bot2
Total: 15.0118 secs
Average: 0.0201 secs
Requests/sec: 442.9185
[200] 85 responses
[429] 6564 responses
```
Compare with Step 6:
| | Step 6: no limit | Step 9: with limit |
|---|---|---|
| Normal user's wait | about 15 seconds | about 0.03 seconds, as in the baseline |
| Answers each bot received | one bot a few hundred, the other almost none | about 85 each |
| Refusals (status 429) | none | about 6,400 per bot |
| Average time per bot request | seconds | about 0.02 seconds |
Two things to notice. First, the **normal user is served at normal speed**: the limit is per address, and the user's address is far below it. Second, each bot got only about 85 answers. That is no accident: `10 + 5 × 15 = 85`, the burst plus 5 requests per second for 15 seconds. Everything else was refused with a tiny `429` reply that costs almost nothing to produce, which is why the server is free again.
### Step 10 — Detect and prove it in the log (4 min)
The same log analysis as in Step 7, now only on the lines written since the snapshot of Step 7 (`tail -n +N` prints from line N on). Then look at one refused request and at nginx's own error log.
```bash
podman logs lab-ddos-web 2>&1 | awk '$6=="\"GET"' | tail -n +$(( $(wc -l < access-all.txt) + 1 )) > access-flood2.txt
wc -l < access-flood2.txt
awk '{print $1, $9}' access-flood2.txt | sort | uniq -c | sort -rn
grep ' 429 ' access-flood2.txt | head -1
podman logs lab-ddos-web 2>&1 | grep 'limiting requests' | head -1
podman logs lab-ddos-web 2>&1 | grep -c 'limiting requests'
```
Expected output (counts vary):
```text
13121
6564 10.60.0.21 429
6388 10.60.0.20 429
85 10.60.0.21 200
81 10.60.0.20 200
3 10.60.0.30 200
10.60.0.21 - - [07/Nov/2026:20:08:05 +0000] "GET /report.html HTTP/1.1" 429 169 "-" "hey/0.0.1"
2026/11/07 20:08:05 [error] 29#29: *54 limiting requests, excess: 10.815 by zone "perip", client: 10.60.0.21, server: , request: "GET /report.html HTTP/1.1", host: "10.60.0.10"
12952
```
Now the log tells the story clearly: 13,121 requests in 15 seconds, almost all of them `429` from two addresses, and the normal user with three `200`. This is the signal a detection system would use. Two more observations:
- The number of lines `limiting requests` in the error log (12,952) equals the number of 429s in the access log (6,388 + 6,564 = 12,952): every refusal is recorded. In a real system, you would alert on a high count of 429 from the same addresses, and could feed those addresses to a firewall or to the provider.
- `limit_conn` never fired. nginx serves each request here in one quick burst, so an address never has more than one request in progress, and the refused ones are answered at once. `limit_conn` pays off when requests stay in progress for long, and it is not a defence against slow attacks that never finish their headers (see the challenge).
### Step 11 — Read the SYN-cookie setting (4 min)
The limits so far are for HTTP floods (layer 7). Against a SYN flood (layer 4, see the lesson) the Linux kernel has its own defence, SYN cookies. You only **read** the setting, both on your machine and inside the web container, and the kernel's counters. You don't need to change anything, and a SYN flood needs raw packets, which this lab deliberately doesn't use.
```bash
cat /proc/sys/net/ipv4/tcp_syncookies
podman exec lab-ddos-web cat /proc/sys/net/ipv4/tcp_syncookies
podman exec lab-ddos-web awk '/^TcpExt:/ { if (!h) { split($0, n); h = 1 } else { split($0, v); for (i = 2; i <= NF; i++) if (n[i] ~ /^(SyncookiesSent|SyncookiesRecv|SyncookiesFailed|ListenOverflows|ListenDrops)$/) print n[i], v[i] } }' /proc/net/netstat
```
Expected output:
```text
1
1
SyncookiesSent 0
SyncookiesRecv 0
SyncookiesFailed 0
ListenOverflows 0
ListenDrops 0
```
How to read it:
- The values of `net.ipv4.tcp_syncookies` mean: `0` cookies off, `1` cookies **only when the half-open queue overflows** (the usual default), `2` cookies always. You see `1` in both places: your system is protected from SYN floods out of the box. The container has its own copy of the setting, inside its own network namespace, and it starts with the same value.
- The counters in the last command are the kernel's own record for the container's network: `SyncookiesSent` counts the cookies sent, and `ListenOverflows` and `ListenDrops` count connection attempts lost because the queue was full. They are all `0`: the HTTP flood opened full connections, it didn't fill the half-open queue, so SYN cookies never had to act. That is the point of classification: the right defence depends on the class of attack, and a SYN-cookie setting would not have saved the server from this flood, but the rate limit did.
> [!NOTE]
> Your counters may not be exactly zero if something else on your machine used the container's network. The meaning is the same: they only grow when a half-open queue overflows.
## Verification
Check these results; each must hold in your run:
| Check | Expected result |
|---|---|
| Baseline (Step 5) | Normal request answers `200` in tens of milliseconds |
| Flood without limit (Step 6) | The normal user waited several seconds, not milliseconds |
| Access log (Step 7) | The two bot addresses account for most of the lines |
| Flood with limit (Step 9) | The normal user got `200` in about 0.03 s; the bots received mostly `429` and about 85 `200` each |
| Access log (Step 10) | `429` lines for the two bot addresses only; `200` for `10.60.0.30` |
| SYN cookies (Step 11) | `net.ipv4.tcp_syncookies` is `1` on the host and in the container |
Run this last test to prove again that the normal user is served while limits are on:
```bash
for i in 1 2 3; do podman exec lab-ddos-user curl -sS --compressed -m 5 -o /dev/null -w '%{http_code}\n' http://10.60.0.10/report.html; done
```
Expected output:
```text
200
200
200
```
## Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| `Error: ... cpu ... controller` or `--cpus` is not supported when starting `lab-ddos-web` | The CPU controller isn't delegated to your user (older systems or cgroup v1) | Remove `--cpus 0.5` and make the page heavier: `seq 1 600000 > www/report.html`. The flood effect in Step 6 may be smaller; use `-c 50` in the bots' command |
| `podman run` fails with `permission denied` on the mounted files | SELinux label or wrong path | Check that you are in `~/ddos-lab` and the files exist; keep the `:Z` flags; don't mount files that are outside your home |
| Step 6 shows no slowdown for the normal user (about 0.03 s) | The server is faster than expected, or the flood didn't start | Check that `flood1-bot1.txt` has `[200]` lines and no error; use a bigger page (`seq 1 600000`) or `-c 50` |
| `hey: not found` or `curl: not found` | You ran the command on your host or in the nginx container | Prefix with `podman exec lab-ddos-user` (or `-bot1`, `-bot2`), which have the tools image |
| After the reload all responses are still `200` (Step 9) | `limits.conf` wasn't updated or the reload failed | `podman exec lab-ddos-web cat /etc/nginx/lab/limits.conf` shows the file nginx sees; run `nginx -t` and read the line it complains about |
| `nginx -t` says `unknown directive` or `invalid parameter` | A typo in `limits.conf` | Compare it with the Step 8 text; nginx keeps the old rules until the test passes |
| The normal user also gets `429` | The user asked more than 5 times per second plus the burst | Wait a few seconds and ask again, slowly. That is the rule working: a limit can also hit real users who are very fast, or share an address |
| `Error: ... name "lab-ddos-web" is already in use` | A container from an earlier attempt exists | Run the [Cleanup](#cleanup), then start again |
| The counts in your tables differ from this page | Timing, CPU speed and scheduling differ on every machine | Compare the *picture*: seconds against milliseconds, and thousands of `429` against about 85 `200` per bot |
## Challenge (optional)
Without step-by-step help, try one or more of these, then restore the files from Step 8:
1. **A slow attack.** Slowloris-style attacks open a connection and send a request header very slowly, never finishing it. From `lab-ddos-bot1`, use `nc` to open a connection to `10.60.0.10` port 80 and send only the first line of a request, then wait. Which nginx setting finally closes such a connection (look up `client_header_timeout` in the nginx documentation)? Why doesn't `limit_conn` see it?
2. **A global limit.** Add a second counter keyed on `$server_name` with a rate of 30 requests per second, and apply it to `/report.html`. What happens to the normal user during a flood now? What does this teach you about per-address limits against a botnet of thousands of addresses?
3. **Cheaper pages.** Cache the compressed page: look up `gzip_static` or `proxy_cache` in the nginx documentation, and measure how the server's capacity in the baseline changes. Which layer of the lesson's defence figure does this correspond to?
## Cleanup
Remove every resource that the lab created. `-t 0` skips the 10 second wait for containers that ignore the stop signal.
```bash
podman rm -f -t 0 lab-ddos-web lab-ddos-bot1 lab-ddos-bot2 lab-ddos-user
podman network rm lab-ddos-net
podman rmi lab-ddos-tools:1
cd ~ && rm -rf ~/ddos-lab
podman ps -a --filter name=lab-ddos --format '{{.Names}}'
podman network ls --filter name=lab-ddos --format '{{.Name}}'
```
The last two commands print nothing when everything is gone. The pulled `docker.io/library/nginx:1.31.6-alpine` and `alpine:3.20.10` images remain in your image store (they are shared with other work); remove them with `podman rmi` if you want the space back.
## What you learned
- A flood doesn't need to be huge to hurt: a server with a limited capacity turned a harmless-looking set of requests into a 15-second wait for a normal user.
- A baseline of normal latency and traffic tells you at once that something is wrong, and the access log, counted per address and per status, shows who is responsible.
- A per-address rate limit (`limit_req`, status 429) refuses the excess cheaply, keeps the server responsive and lets normal users through; the refused requests cost almost nothing to answer.
- A limit's arithmetic is predictable: burst plus rate times time, here `10 + 5 × 15 = 85` answers per bot. Against thousands of addresses, the sum is large, so you also need upstream layers.
- SYN cookies protect against a different class (state exhaustion): the setting is `1` by default on Linux, and it didn't matter here because the flood was an HTTP flood.