CloudAcademia
EN

Cybersecurity · Module 06 · Network Security: Firewalls and Segmentation

Lab: Segmenting a Network with a Default-Deny Firewall

Lab75 minIntermediateMarkdown

Showing raw Markdown: the renderer could not load.

## Goal
You build three network zones (an "Internet", a DMZ and an internal zone) joined by one firewall container. You scan from the outside while everything is open, apply a default-deny nftables policy that allows only two flows, scan again, read the firewall's counters and drop log, and prove with tests that every unwanted flow is blocked.

## Objectives
- Build three isolated zones and route them through one firewall.
- Scan the network from the outside with `nmap` and read `open`, `closed` and `filtered` results.
- Write a stateful default-deny rule set that allows only Internet to web (TCP 80) and web to database (TCP 5432).
- Read the firewall's rule counters and its drop log to confirm that blocked traffic was blocked by your rules.
- Verify the policy with a table of positive and negative tests, and show why rule order matters.

## Prerequisites
- A Linux machine with **Podman 4 or later** running **rootless** (as a normal user; you never need `sudo`) and the netavark network backend. Tested with Podman 5.8.7 on Fedora with kernel 7.2. Check with `podman info --format '{{.Host.NetworkBackend}}'`: it should print `netavark`.
- Internet access once, to download the Alpine Linux image (about 8 MB) and a few packages (about 40 MB in total). After that, nothing in the lab needs the Internet.
- About 400 MB of free disk space.
- The [lesson](lesson.md) of this module, and [Networking 08 — Transport Layer: TCP, UDP and Ports](../../networking/08-tcp-udp/README.md). The container techniques are the ones of the [Cloud Networking lab](../../cloud-computing/06-cloud-networking/lab.md); here you focus on zones, default deny, logging and verification.

> [!WARNING]
> **Authorization and ethics.** Port scanning is an attack technique when it is aimed at systems you don't control, and it can be illegal. In this lab you scan only the three containers that you create yourself, on networks created with `--internal` (nothing leaves your machine). Never point `nmap` at a system you don't own or aren't explicitly authorized in writing to test. Scanning your own isolated lab to check your own firewall is the defensive use of the tool.

## Architecture
<figure class="diagram" style="--dg-w: 1180px"><svg xmlns="http://www.w3.org/2000/svg" class="dg" viewBox="0 0 1180 560" width="1180" height="560" role="img" aria-labelledby="netsec-lab-architecture-title netsec-lab-architecture-desc" font-family="Inter, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" xml:lang="en">
<title id="netsec-lab-architecture-title">Lab architecture: three zones and one default-deny firewall</title>
<desc id="netsec-lab-architecture-desc">Three internal Podman networks meet in one firewall container, lab-netsec-fw, which has the interfaces net0 203.0.113.254, dmz0 10.40.2.254 and int0 10.40.3.254. The internet network 203.0.113.0/24 holds lab-netsec-client at 203.0.113.10, the outside client with nmap, nc and curl and a route to 10.40.0.0/16 via 203.0.113.254. The dmz network 10.40.2.0/24 holds lab-netsec-web at 10.40.2.10, a web server listening on tcp 80, 8080 and 22, with a default route via 10.40.2.254. The internal network 10.40.3.0/24 holds lab-netsec-db at 10.40.3.10, a database-like listener on tcp 5432 and 22, with a default route via 10.40.3.254. The firewall applies the nftables table inet lab_fw on the forward hook with policy drop: rule 1 accepts established and related traffic, rule 2 accepts new connections from net0 to dmz0 on tcp 80, rule 3 accepts new connections from dmz0 to int0 on tcp 5432, and rule 4 logs to nflog group 1 and then drops everything else.</desc>
<rect class="dg-bg" x="0.5" y="0.5" width="1179" height="559" rx="16" fill="#ffffff" stroke="#e3e3dd"/>
<g class="dg-l1"><rect class="dg-tint" x="20" y="150" width="300" height="230" rx="16" fill="#f0f6f9" stroke="#9ac3d9" stroke-width="1.2" stroke-dasharray="6 5"/>
<g class="dg-icon" transform="translate(36 163) scale(0.7500)" fill="none" stroke="#0369a1" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M17.5 19H9a7 7 0 1 1 6.71-9h1.79a4.5 4.5 0 1 1 0 9Z"/></g>
<text x="62" y="177" font-size="13" class="dg-ink" fill="#0369a1" font-weight="700">lab-netsec-internet · 203.0.113.0/24</text></g>
<g class="dg-l4"><rect class="dg-tint" x="820" y="20" width="340" height="230" rx="16" fill="#fbf4f0" stroke="#e7b39e" stroke-width="1.2" stroke-dasharray="6 5"/>
<g class="dg-icon" transform="translate(836 33) scale(0.7500)" fill="none" stroke="#c2410c" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="m21.73 18-8-14a2 2 0 0 0-3.48 0l-8 14A2 2 0 0 0 4 21h16a2 2 0 0 0 1.73-3"/><path d="M12 9v4"/><path d="M12 17h.01"/></g>
<text x="862" y="47" font-size="13" class="dg-ink" fill="#c2410c" font-weight="700">lab-netsec-dmz · 10.40.2.0/24</text></g>
<g class="dg-l2"><rect class="dg-tint" x="820" y="290" width="340" height="230" rx="16" fill="#f1f7f3" stroke="#a1ccb1" stroke-width="1.2" stroke-dasharray="6 5"/>
<g class="dg-icon" transform="translate(836 303) scale(0.7500)" fill="none" stroke="#15803d" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><rect x="4" y="11" width="16" height="10" rx="2"/><path d="M8 11V7a4 4 0 0 1 8 0v4"/></g>
<text x="862" y="317" font-size="13" class="dg-ink" fill="#15803d" font-weight="700">lab-netsec-internal · 10.40.3.0/24</text></g>
<g class="dg-l1"><path class="dg-stroke" d="M300 265H430" stroke="#0369a1" stroke-width="3.5" fill="none" stroke-linecap="round" stroke-linejoin="round"/></g>
<g class="dg-l4"><path class="dg-stroke" d="M740 140H840" stroke="#c2410c" stroke-width="3.5" fill="none" stroke-linecap="round" stroke-linejoin="round"/></g>
<g class="dg-l2"><path class="dg-stroke" d="M740 410H840" stroke="#15803d" stroke-width="3.5" fill="none" stroke-linecap="round" stroke-linejoin="round"/></g>
<rect class="dg-card" x="40" y="200" width="260" height="150" rx="12" fill="#ffffff" stroke="#cfcfc8"/>
<g class="dg-l1"><g class="dg-icon" transform="translate(56 216) scale(1.2500)" fill="none" stroke="#0369a1" stroke-width="1.8" stroke-linecap="round" stroke-linejoin="round"><path d="M20 16V7a2 2 0 0 0-2-2H6a2 2 0 0 0-2 2v9m16 0H4m16 0 1.28 2.55a1 1 0 0 1-.9 1.45H3.62a1 1 0 0 1-.9-1.45L4 16"/></g></g>
<g class="dg-l1"><text x="98" y="237" font-size="14" class="dg-ink" fill="#0369a1" font-weight="700">lab-netsec-client</text></g>
<text x="58" y="268" font-size="12.5" class="dg-text" fill="#17171c" font-weight="500" font-family="JetBrains Mono, ui-monospace, Menlo, Consolas, monospace">eth0 203.0.113.10/24</text>
<text x="58" y="288" font-size="12.5" class="dg-muted" fill="#5c5c68" font-weight="500" font-family="JetBrains Mono, ui-monospace, Menlo, Consolas, monospace">route 10.40.0.0/16 via .254</text>
<text x="58" y="308" font-size="13" class="dg-muted" fill="#5c5c68" font-weight="500">Outside client: nmap, nc, curl</text>
<rect class="dg-card" x="430" y="80" width="310" height="400" rx="12" fill="#ffffff" stroke="#cfcfc8"/>
<g class="dg-l3"><g class="dg-icon" transform="translate(446 96) scale(1.2500)" fill="none" stroke="#b45309" stroke-width="1.8" stroke-linecap="round" stroke-linejoin="round"><path d="M20 13c0 5-3.5 7.5-7.66 8.95a1 1 0 0 1-.67-.01C7.5 20.5 4 18 4 13V6a1 1 0 0 1 1-1c2 0 4.5-1.2 6.24-2.72a1.17 1.17 0 0 1 1.52 0C14.51 3.81 17 5 19 5a1 1 0 0 1 1 1z"/></g></g>
<g class="dg-l3"><text x="488" y="117" font-size="15" class="dg-ink" fill="#b45309" font-weight="700">lab-netsec-fw</text></g>
<text x="448" y="148" font-size="12.5" class="dg-text" fill="#17171c" font-weight="500" font-family="JetBrains Mono, ui-monospace, Menlo, Consolas, monospace">net0  203.0.113.254</text>
<text x="448" y="168" font-size="12.5" class="dg-text" fill="#17171c" font-weight="500" font-family="JetBrains Mono, ui-monospace, Menlo, Consolas, monospace">dmz0  10.40.2.254</text>
<text x="448" y="188" font-size="12.5" class="dg-text" fill="#17171c" font-weight="500" font-family="JetBrains Mono, ui-monospace, Menlo, Consolas, monospace">int0  10.40.3.254</text>
<text x="448" y="248" font-size="12" class="dg-muted" fill="#5c5c68" font-weight="700">table inet lab_fw · hook forward</text>
<text x="448" y="268" font-size="12.5" class="dg-ink" fill="#b91c1c" font-weight="700" font-family="JetBrains Mono, ui-monospace, Menlo, Consolas, monospace">policy drop</text>
<rect class="dg-tint" x="444" y="294" width="282" height="32" rx="8" fill="#ecf5ef" stroke="#a1ccb1"/>
<text x="454" y="315" font-size="12" class="dg-text" fill="#17171c" font-weight="600" font-family="JetBrains Mono, ui-monospace, Menlo, Consolas, monospace">1 established,related  accept</text>
<rect class="dg-tint" x="444" y="334" width="282" height="32" rx="8" fill="#ecf5ef" stroke="#a1ccb1"/>
<text x="454" y="355" font-size="12" class="dg-text" fill="#17171c" font-weight="600" font-family="JetBrains Mono, ui-monospace, Menlo, Consolas, monospace">2 net0 → dmz0  tcp 80  accept</text>
<rect class="dg-tint" x="444" y="374" width="282" height="32" rx="8" fill="#ecf5ef" stroke="#a1ccb1"/>
<text x="454" y="395" font-size="12" class="dg-text" fill="#17171c" font-weight="600" font-family="JetBrains Mono, ui-monospace, Menlo, Consolas, monospace">3 dmz0 → int0  tcp 5432  accept</text>
<rect class="dg-tint" x="444" y="414" width="282" height="32" rx="8" fill="#f9eded" stroke="#e3a4a4"/>
<text x="454" y="435" font-size="12" class="dg-text" fill="#17171c" font-weight="600" font-family="JetBrains Mono, ui-monospace, Menlo, Consolas, monospace">4 log group 1, then drop</text>
<rect class="dg-card" x="840" y="66" width="300" height="150" rx="12" fill="#ffffff" stroke="#cfcfc8"/>
<g class="dg-l4"><g class="dg-icon" transform="translate(856 82) scale(1.2500)" fill="none" stroke="#c2410c" stroke-width="1.8" stroke-linecap="round" stroke-linejoin="round"><rect width="20" height="8" x="2" y="2" rx="2"/><rect width="20" height="8" x="2" y="14" rx="2"/><path d="M6 6h.01"/><path d="M6 18h.01"/></g></g>
<g class="dg-l4"><text x="898" y="103" font-size="14" class="dg-ink" fill="#c2410c" font-weight="700">lab-netsec-web</text></g>
<text x="858" y="134" font-size="12.5" class="dg-text" fill="#17171c" font-weight="500" font-family="JetBrains Mono, ui-monospace, Menlo, Consolas, monospace">eth0 10.40.2.10/24 · tcp 80 8080 22</text>
<text x="858" y="154" font-size="12.5" class="dg-muted" fill="#5c5c68" font-weight="500" font-family="JetBrains Mono, ui-monospace, Menlo, Consolas, monospace">default via 10.40.2.254</text>
<text x="858" y="174" font-size="13" class="dg-muted" fill="#5c5c68" font-weight="500">DMZ web server</text>
<rect class="dg-card" x="840" y="336" width="300" height="150" rx="12" fill="#ffffff" stroke="#cfcfc8"/>
<g class="dg-l2"><g class="dg-icon" transform="translate(856 352) scale(1.2500)" fill="none" stroke="#15803d" stroke-width="1.8" stroke-linecap="round" stroke-linejoin="round"><ellipse cx="12" cy="5" rx="9" ry="3"/><path d="M3 5V19A9 3 0 0 0 21 19V5"/><path d="M3 12A9 3 0 0 0 21 12"/></g></g>
<g class="dg-l2"><text x="898" y="373" font-size="14" class="dg-ink" fill="#15803d" font-weight="700">lab-netsec-db</text></g>
<text x="858" y="404" font-size="12.5" class="dg-text" fill="#17171c" font-weight="500" font-family="JetBrains Mono, ui-monospace, Menlo, Consolas, monospace">eth0 10.40.3.10/24 · tcp 5432 22</text>
<text x="858" y="424" font-size="12.5" class="dg-muted" fill="#5c5c68" font-weight="500" font-family="JetBrains Mono, ui-monospace, Menlo, Consolas, monospace">default via 10.40.3.254</text>
<text x="858" y="444" font-size="13" class="dg-muted" fill="#5c5c68" font-weight="500">Database-like listener</text>
<g class="dg-l1"><rect class="dg-bg-plain" x="342.58" y="253" width="44.84" height="24" rx="12" fill="#ffffff"/><text x="365" y="269.5" font-size="11" class="dg-ink" fill="#0369a1" font-weight="600" text-anchor="middle" font-family="JetBrains Mono, ui-monospace, Menlo, Consolas, monospace">net0</text></g>
<g class="dg-l4"><rect class="dg-bg-plain" x="767.58" y="128" width="44.84" height="24" rx="12" fill="#ffffff"/><text x="790" y="144.5" font-size="11" class="dg-ink" fill="#c2410c" font-weight="600" text-anchor="middle" font-family="JetBrains Mono, ui-monospace, Menlo, Consolas, monospace">dmz0</text></g>
<g class="dg-l2"><rect class="dg-bg-plain" x="767.58" y="398" width="44.84" height="24" rx="12" fill="#ffffff"/><text x="790" y="414.5" font-size="11" class="dg-ink" fill="#15803d" font-weight="600" text-anchor="middle" font-family="JetBrains Mono, ui-monospace, Menlo, Consolas, monospace">int0</text></g>
<g class="dg-l1"><rect class="dg-tint" x="329.5" y="526" width="521" height="26" rx="13" fill="#e6f0f6" stroke="#8ebcd5" stroke-width="1.2"/><text x="590" y="543.5" font-size="12" class="dg-ink" fill="#0369a1" font-weight="700" text-anchor="middle">All three Podman networks are --internal: nothing leaves your machine</text></g>
</svg></figure>

*Figure 1. One firewall container joins three zones. At the end only the two flows on its rule list can cross it.*

| Container | Network and address | Role |
|---|---|---|
| `lab-netsec-client` | `lab-netsec-internet`, `203.0.113.10` | The outside: a visitor, or an attacker, with `nmap`, `nc` and `curl` |
| `lab-netsec-web` | `lab-netsec-dmz`, `10.40.2.10` | The public web server (TCP 80), plus two services that should **not** be public: TCP 8080 and a fake SSH banner on TCP 22 |
| `lab-netsec-db` | `lab-netsec-internal`, `10.40.3.10` | A database-like listener on TCP 5432 (it answers `lab-db ready`) and a fake SSH banner on TCP 22 |
| `lab-netsec-fw` | `203.0.113.254`, `10.40.2.254`, `10.40.3.254` | The router and firewall between the zones |

> [!NOTE]
> The "SSH" and "database" services are tiny `nc` listeners that print one line. They are enough to be found by a scan and to accept or refuse connections, which is all this lab needs. In a real network the web server would also need a public (or translated) address; here the client simply routes to it through the firewall.

Podman runs a command in a container with `podman exec lab-netsec-<name> <command>`. The lab uses that all the time, so each command is spelled out in full.

## Steps

### Step 1 — Check your tools and create a working folder (3 min)
Make sure Podman is rootless and uses netavark, and that nothing is left over from an earlier attempt.

```bash
podman --version
podman info --format '{{.Host.NetworkBackend}} rootless={{.Host.Security.Rootless}}'
podman ps -a --filter name=lab-netsec --format '{{.Names}}'
podman network ls --filter name=lab-netsec --format '{{.Name}}'
mkdir -p ~/netsec-lab && cd ~/netsec-lab
```

Expected output (your version may differ):
```text
podman version 5.8.7
netavark rootless=true
```

The two `ls` commands print nothing. If they list `lab-netsec-` names, run the [Cleanup](#cleanup) first. All files and commands that follow assume you are in `~/netsec-lab`.

### Step 2 — Build the tools image (5 min)
Alpine Linux is tiny but has none of the tools you need. You build one image with `nftables` (the Linux firewall), `nmap` (the scanner), `tcpdump` (to read the firewall's log), `curl`, the full `ip` command and `busybox-extras` (a small web server). The tag is pinned to Alpine 3.20, so you get the same results as the outputs below.

```bash
cat > Containerfile <<'EOT'
FROM docker.io/library/alpine:3.20.10
RUN apk add --no-cache nftables nmap curl iproute2 busybox-extras tcpdump
EOT
podman build -t lab-netsec-tools:1 .
podman run --rm lab-netsec-tools:1 sh -c 'nft --version; nmap --version | head -1; tcpdump --version 2>&1 | head -1; cat /etc/alpine-release'
```

Expected output (the end of the build, then the versions; package versions may differ slightly):
```text
OK: 47 MiB in 49 packages
COMMIT lab-netsec-tools:1
Successfully tagged localhost/lab-netsec-tools:1
nftables v1.0.9 (Old Doc Yak #3)
Nmap version 7.95 ( https://nmap.org )
tcpdump version 4.99.4
3.20.10
```

### Step 3 — Create the three zones (3 min)
Each Podman network is one zone: a bridge with its own address range. `--internal` means that Podman adds no route to the outside world and no NAT to your host, so nothing in the lab can leave your machine, and nothing can cross between zones except through the firewall you build. The "Internet" uses a documentation range ([RFC 5737](https://www.rfc-editor.org/rfc/rfc5737)); the DMZ and the internal zone use private ranges ([RFC 1918](https://www.rfc-editor.org/rfc/rfc1918)).

```bash
podman network create --internal --subnet 203.0.113.0/24 lab-netsec-internet
podman network create --internal --subnet 10.40.2.0/24 lab-netsec-dmz
podman network create --internal --subnet 10.40.3.0/24 lab-netsec-internal
podman network inspect lab-netsec-internet lab-netsec-dmz lab-netsec-internal \
  --format '{{.Name}} subnet={{(index .Subnets 0).Subnet}} internal={{.Internal}}'
```

Expected output:
```text
lab-netsec-internet
lab-netsec-dmz
lab-netsec-internal
lab-netsec-internet subnet=203.0.113.0/24 internal=true
lab-netsec-dmz subnet=10.40.2.0/24 internal=true
lab-netsec-internal subnet=10.40.3.0/24 internal=true
```

### Step 4 — Start the firewall (4 min)
The firewall container has one interface in each zone, with the last usable address (`.254`) of each range. `interface_name` gives the interfaces readable names: `net0`, `dmz0` and `int0`. `--cap-add NET_ADMIN` lets it edit its own firewall rules, and `--sysctl net.ipv4.ip_forward=1` turns it into a router. `sleep infinity` keeps it running.

```bash
podman run -d --name lab-netsec-fw \
  --network lab-netsec-internet:ip=203.0.113.254,interface_name=net0 \
  --network lab-netsec-dmz:ip=10.40.2.254,interface_name=dmz0 \
  --network lab-netsec-internal:ip=10.40.3.254,interface_name=int0 \
  --cap-add NET_ADMIN --sysctl net.ipv4.ip_forward=1 \
  lab-netsec-tools:1 sleep infinity
podman exec lab-netsec-fw ip -br addr
podman exec lab-netsec-fw sysctl net.ipv4.ip_forward
podman exec lab-netsec-fw nft list ruleset
```

Expected output (the container ID, the order of the interfaces, the `@if` numbers and the `fe80::` addresses vary):
```text
0864d79130acc3d47b16a11b13019e9c56a6e87d68de7908d9aed6cf13f1fd69
lo               UNKNOWN        127.0.0.1/8 ::1/128 
int0@if4         UP             10.40.3.254/24 fe80::3813:93ff:fe04:a0c8/64 
dmz0@if6         UP             10.40.2.254/24 fe80::58f6:56ff:fe7b:1dba/64 
net0@if8         UP             203.0.113.254/24 fe80::6c81:b4ff:feb7:80d7/64 
net.ipv4.ip_forward = 1
```

The last command, `nft list ruleset`, prints **nothing**: the firewall has no rules yet, so it forwards everything. Remember that: it is your starting point.

### Step 5 — Start the three hosts (6 min)
Each host joins one zone. An internal network gives a container no default route, so each host adds a route towards the firewall: the web server and the database use it as their default gateway, and the client sends the `10.40.0.0/16` range to it. `NET_ADMIN` is needed only for `ip route add`.

The web server runs the BusyBox web server `httpd` on TCP 80 and 8080, and a `nc` listener on TCP 22 that prints a fake SSH banner. The database runs two `nc` listeners: TCP 22 (the same banner) and TCP 5432, which answers `lab-db ready`. They stand for services that exist on real machines and shouldn't be reachable from everywhere.

```bash
I=lab-netsec-tools:1
podman run -d --name lab-netsec-web --network lab-netsec-dmz:ip=10.40.2.10 --cap-add NET_ADMIN $I \
  sh -c 'ip route add default via 10.40.2.254; mkdir -p /www; echo "hello from the DMZ web server" > /www/index.html; httpd -p 80 -h /www; httpd -p 8080 -h /www; exec nc -lk -p 22 -e echo SSH-2.0-lab-fake'
podman run -d --name lab-netsec-db --network lab-netsec-internal:ip=10.40.3.10 --cap-add NET_ADMIN $I \
  sh -c 'ip route add default via 10.40.3.254; nc -lk -p 22 -e echo SSH-2.0-lab-fake & exec nc -lk -p 5432 -e echo lab-db ready'
podman run -d --name lab-netsec-client --network lab-netsec-internet:ip=203.0.113.10 --cap-add NET_ADMIN $I \
  sh -c 'ip route add 10.40.0.0/16 via 203.0.113.254; exec sleep infinity'
podman ps --filter name=lab-netsec --format 'table {{.Names}}\t{{.Status}}'
```

Expected output (three container IDs first; the times vary):
```text
NAMES              STATUS
lab-netsec-fw      Up 3 seconds
lab-netsec-web     Up 2 seconds
lab-netsec-db      Up 2 seconds
lab-netsec-client  Up 2 seconds
```

Check the routes, and which ports each server listens on:

```bash
podman exec lab-netsec-client ip route
podman exec lab-netsec-web netstat -tln
podman exec lab-netsec-db netstat -tln
```

Expected output:
```text
10.40.0.0/16 via 203.0.113.254 dev eth0 
203.0.113.0/24 dev eth0 proto kernel scope link src 203.0.113.10 
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       
tcp        0      0 :::80                   :::*                    LISTEN      
tcp        0      0 :::22                   :::*                    LISTEN      
tcp        0      0 :::8080                 :::*                    LISTEN      
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       
tcp        0      0 :::22                   :::*                    LISTEN      
tcp        0      0 :::5432                 :::*                    LISTEN      
```

You now know the **truth**: the web server listens on 80, 8080 and 22; the database on 5432 and 22. In the next step you find out what a stranger sees.

### Step 6 — Baseline: everything is open, and a scan shows it (10 min)
With forwarding on and no rules, the firewall passes everything. Test it from the outside client, then scan. First, two ordinary requests:

```bash
podman exec lab-netsec-client curl -sS -m 3 http://10.40.2.10
podman exec lab-netsec-client nc -w 2 10.40.3.10 5432
podman exec lab-netsec-client nc -w 2 -z -v 10.40.3.10 22
```

Expected output:
```text
hello from the DMZ web server
lab-db ready
10.40.3.10 (10.40.3.10:22) open
```

The client on the "Internet" talked **directly to the database**. That is the flat network of the lesson's opening story. Now scan both servers from the client:

```bash
podman exec lab-netsec-client nmap -n -Pn -sT --reason --max-retries 1 -T4 -p 1-1024,5432,8080 10.40.2.10 10.40.3.10
```

What the options mean:
- `-n` skips name resolution; `-p 1-1024,5432,8080` scans the well-known ports plus the two extra ones.
- `-sT` is a TCP **connect scan**: it completes the TCP handshake with the normal `connect()` call. It needs no special privileges, which is why it works in a rootless container (the faster SYN scan, `-sS`, needs raw sockets and fails here).
- `-Pn` skips "is the host up?" discovery, which would also need raw sockets in this container. Every probe is treated as a port probe.
- `--reason` shows why each state was decided; `--max-retries 1 -T4` makes the scan fast enough for a lab.

Expected output (the latency values and the date vary):
```text
Starting Nmap 7.95 ( https://nmap.org ) at 2026-11-07 19:38 UTC
Nmap scan report for 10.40.2.10
Host is up, received user-set (0.00028s latency).
Not shown: 1023 closed tcp ports (conn-refused)
PORT     STATE SERVICE    REASON
22/tcp   open  ssh        syn-ack
80/tcp   open  http       syn-ack
8080/tcp open  http-proxy syn-ack

Nmap scan report for 10.40.3.10
Host is up, received user-set (0.00030s latency).
Not shown: 1024 closed tcp ports (conn-refused)
PORT     STATE SERVICE    REASON
22/tcp   open  ssh        syn-ack
5432/tcp open  postgresql syn-ack

Nmap done: 2 IP addresses (2 hosts up) scanned in 0.08 seconds
```

Read it. The outside client sees **five open ports on two hosts**, including the database that should never be reachable, and the "SSH" and 8080 services nobody meant to publish. The other ports are `closed`: the host answered each probe with a refusal (`conn-refused`), so the scan took 0.08 seconds. The scan has just built an inventory of the attack surface. Write the open ports down: you compare them with the next scan.

> [!NOTE]
> The service names (`ssh`, `postgresql`, `http-proxy`) are guesses from the port number, not from what answers. `nmap` maps ports to names from a table. The fake services here would not fool a version scan, which needs `nmap`'s script files that this small image doesn't include.

### Step 7 — Apply the default-deny policy (10 min)
You now write the policy of the lesson's worked example. Put it in a file, then load it into the firewall. Read it before you load it.

```bash
cat > fw.nft <<'EOT'
table inet lab_fw
delete table inet lab_fw
table inet lab_fw {
  chain forward {
    type filter hook forward priority 0; policy drop;
    ct state invalid drop
    ct state established,related counter accept comment "return traffic"
    iifname "net0" oifname "dmz0" ip daddr 10.40.2.10 tcp dport 80 ct state new counter accept comment "internet to web: 80"
    iifname "dmz0" oifname "int0" ip saddr 10.40.2.10 ip daddr 10.40.3.10 tcp dport 5432 ct state new counter accept comment "web to db: 5432"
    limit rate 5/second burst 10 packets log group 1 prefix "lab-netsec DROP: "
    counter drop comment "default deny"
  }
}
EOT
podman exec -i lab-netsec-fw nft -f - < fw.nft
podman exec lab-netsec-fw nft list chain inet lab_fw forward
```

What each part does:
- The first two lines make the file safe to load again: they create the table if it doesn't exist, then delete it, so every load starts from a clean table.
- `table inet` applies to IPv4 and IPv6. `chain forward` with `hook forward` filters traffic that **passes through** the firewall; it is the path between the zones.
- `policy drop` is the default deny: a packet that matches no `accept` is dropped.
- `ct state invalid drop` drops packets that belong to no valid connection.
- `ct state established,related ... accept` is the stateful part: **replies** to allowed connections need no rule of their own. It comes first because it matches most packets.
- The two `ct state new ... accept` rules are the **only** allowed new flows. Each one names the interfaces, the exact destination (and source for the second rule) and the port. The `counter` keyword counts the packets of each rule.
- `limit ... log group 1` writes a record of the dropped packets to the kernel's **nflog** channel number 1, at most 5 per second (after a burst of 10), so a scan can't flood the log. Rules without a verdict, such as `log`, let the packet continue to the next rule.
- The last rule counts and drops everything else (the policy would also drop, but an explicit rule gives you a counter).

`podman exec -i ... nft -f -` reads the file from standard input, so it stays on your machine. Loading prints nothing. Expected output of the `list` command (all counters are zero at this point):
```text
table inet lab_fw {
	chain forward {
		type filter hook forward priority filter; policy drop;
		ct state invalid drop
		ct state established,related counter packets 0 bytes 0 accept comment "return traffic"
		iifname "net0" oifname "dmz0" ip daddr 10.40.2.10 tcp dport 80 ct state new counter packets 0 bytes 0 accept comment "internet to web: 80"
		iifname "dmz0" oifname "int0" ip saddr 10.40.2.10 ip daddr 10.40.3.10 tcp dport 5432 ct state new counter packets 0 bytes 0 accept comment "web to db: 5432"
		limit rate 5/second burst 10 packets log prefix "lab-netsec DROP: " group 1
		counter packets 0 bytes 0 drop comment "default deny"
	}
}
```

> [!NOTE]
> A plain `log` rule (without `group`) writes to the kernel log, which you read with `dmesg` or `journalctl -k`. In a container that log isn't available: in this test `dmesg` inside the container failed with "Operation not permitted", and by default the kernel only logs from the host's own network namespace. The `nflog` channel works without any special permission and `tcpdump` can read it, so the lab uses that. If your setup differs, the counters in Step 9 still prove what was dropped.

### Step 8 — Scan again and compare (8 min)
Repeat **exactly** the same scan, and then try the two flows you allowed:

```bash
podman exec lab-netsec-client nmap -n -Pn -sT --reason --max-retries 1 -T4 -p 1-1024,5432,8080 10.40.2.10 10.40.3.10
podman exec lab-netsec-client curl -sS -m 3 http://10.40.2.10
podman exec lab-netsec-web nc -w 2 10.40.3.10 5432
```

Expected output (the scan takes several seconds now: the dropped probes get no answer, so `nmap` waits for them. The times vary):
```text
Starting Nmap 7.95 ( https://nmap.org ) at 2026-11-07 19:38 UTC
Nmap scan report for 10.40.2.10
Host is up, received user-set (0.00020s latency).
Not shown: 1025 filtered tcp ports (no-response)
PORT   STATE SERVICE REASON
80/tcp open  http    syn-ack

Nmap scan report for 10.40.3.10
Host is up, received user-set.
All 1026 scanned ports on 10.40.3.10 are in ignored states.
Not shown: 1026 filtered tcp ports (no-response)

Nmap done: 2 IP addresses (2 hosts up) scanned in 7.14 seconds

hello from the DMZ web server
lab-db ready
```

Compare it with Step 6:

| | Before (everything open) | After (default deny) |
|---|---|---|
| Web server, port 80 | `open` | `open` (allowed) |
| Web server, ports 22 and 8080 | `open` | `filtered` |
| Database, ports 22 and 5432 | `open` | `filtered` |
| Other ports | `closed` (refused) | `filtered` (no answer) |
| Scan time | 0.08 s | several seconds |

The attack surface seen from outside shrank from five open ports to one. Note also the **change of the word**: `closed` means "the host refused", `filtered` means "nothing came back", which is what a **drop** looks like. The database is no longer even a visible target: all 1026 ports are filtered. The web server still serves port 80, and the allowed second flow works: the web server reached the database.

> [!NOTE]
> The "Host is up, received user-set" line is `nmap` repeating that you told it (`-Pn`) to assume the host is up. It doesn't prove the host answered. Without `-Pn`, `nmap` would try host discovery first: in this container that fails with "Couldn't open a raw socket", and against a real firewall that drops pings it would wrongly report "Host seems down".

Now look at the policy from the **DMZ**, where an attacker who took over the web server would stand. Scan the database and the "Internet" client from the web server:

```bash
podman exec lab-netsec-web nmap -n -Pn -sT --max-retries 1 -T4 -p 22,80,443,5432,8080 10.40.3.10 203.0.113.10
```

Expected output:
```text
Starting Nmap 7.95 ( https://nmap.org ) at 2026-11-07 19:38 UTC
Nmap scan report for 10.40.3.10
Host is up (0.000095s latency).

PORT     STATE    SERVICE
22/tcp   filtered ssh
80/tcp   filtered http
443/tcp  filtered https
5432/tcp open     postgresql
8080/tcp filtered http-proxy

Nmap scan report for 203.0.113.10
Host is up.

PORT     STATE    SERVICE
22/tcp   filtered ssh
80/tcp   filtered http
443/tcp  filtered https
5432/tcp filtered postgresql
8080/tcp filtered http-proxy

Nmap done: 2 IP addresses (2 hosts up) scanned in 1.23 seconds
```

From the DMZ, the database shows **one** open port (5432, the allowed flow), and the "Internet" client shows none: the web server can't open connections to the outside (**egress filtering**). If the web server is taken over, the attacker can talk to the database port and nothing else. Scanning from each zone is how you test the internal rules, which a perimeter scan can't see.

### Step 9 — Read the log and the counters (12 min)
Scans and tests don't prove the packets were blocked **by your rules**. The counters and the log do. First reset the counters, so that the numbers below are only from this step. Then start a capture on the firewall's nflog channel, in the background, writing to a file inside the firewall container. It stops by itself after 60 seconds.

```bash
podman exec lab-netsec-fw nft reset rules chain inet lab_fw forward > /dev/null
podman exec -d lab-netsec-fw sh -c 'timeout 60 tcpdump -n -l -q -i nflog:1 > /tmp/drops.log 2>&1'
sleep 1
```

`nflog:1` is the capture interface for log group 1, the `group 1` of the rule. `-n` skips name lookups and `-q` prints a short line per packet. Now try five flows that must **not** work. Each `nc -z` only tests whether the connection opens; `-w 2` gives up after 2 seconds.

```bash
podman exec lab-netsec-client nc -w 2 -z -v 10.40.3.10 5432; echo rc=$?
podman exec lab-netsec-client nc -w 2 -z -v 10.40.2.10 22; echo rc=$?
podman exec lab-netsec-web nc -w 2 -z -v 10.40.3.10 22; echo rc=$?
podman exec lab-netsec-db nc -w 2 -z -v 10.40.2.10 80; echo rc=$?
podman exec lab-netsec-web nc -w 2 -z -v 203.0.113.10 80; echo rc=$?
```

The five tests are: Internet to the **database** (5432), Internet to the web server's **SSH** port, web server to the database's **SSH** port, database to the web server (a connection started from inside), and web server to the "Internet" (egress). Expected output (the timeouts take about 2 seconds each, five times):
```text
nc: 10.40.3.10 (10.40.3.10:5432): Operation timed out
rc=1
nc: 10.40.2.10 (10.40.2.10:22): Operation timed out
rc=1
nc: 10.40.3.10 (10.40.3.10:22): Operation timed out
rc=1
nc: 10.40.2.10 (10.40.2.10:80): Operation timed out
rc=1
nc: 203.0.113.10 (203.0.113.10:80): Operation timed out
rc=1
```

All five **time out** (there is no refusal): the firewall drops the packet and says nothing. Now read the drop log and the counters:

```bash
sleep 3
podman exec lab-netsec-fw cat /tmp/drops.log
podman exec lab-netsec-fw nft list chain inet lab_fw forward | tail -6
```

Expected output (the timestamps and the source ports vary):
```text
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on nflog:1, link-type NFLOG (Linux netfilter log messages), snapshot length 262144 bytes
19:38:34.206451 IP 203.0.113.10.45915 > 10.40.3.10.5432: tcp 0
19:38:35.231427 IP 203.0.113.10.45915 > 10.40.3.10.5432: tcp 0
19:38:36.254469 IP 203.0.113.10.34173 > 10.40.2.10.22: tcp 0
19:38:37.278435 IP 203.0.113.10.34173 > 10.40.2.10.22: tcp 0
19:38:38.366492 IP 10.40.2.10.35221 > 10.40.3.10.22: tcp 0
19:38:39.391417 IP 10.40.2.10.35221 > 10.40.3.10.22: tcp 0
19:38:40.414413 IP 10.40.3.10.41825 > 10.40.2.10.80: tcp 0
19:38:41.438422 IP 10.40.3.10.41825 > 10.40.2.10.80: tcp 0
19:38:42.526436 IP 10.40.2.10.33531 > 203.0.113.10.80: tcp 0
19:38:43.550426 IP 10.40.2.10.33531 > 203.0.113.10.80: tcp 0

		iifname "net0" oifname "dmz0" ip daddr 10.40.2.10 tcp dport 80 ct state new counter packets 0 bytes 0 accept comment "internet to web: 80"
		iifname "dmz0" oifname "int0" ip saddr 10.40.2.10 ip daddr 10.40.3.10 tcp dport 5432 ct state new counter packets 0 bytes 0 accept comment "web to db: 5432"
		limit rate 5/second burst 10 packets log prefix "lab-netsec DROP: " group 1
		counter packets 10 bytes 600 drop comment "default deny"
	}
}
```

Read the evidence:
- The log has **ten packet lines, in five pairs** (`tcpdump` prints its "packets captured" summary only when the capture ends, after the 60 seconds). Each pair is one blocked connection attempt: a TCP SYN (the first packet of a connection, `tcp 0` means no data) sent twice, because the sender retried after about one second. The pairs match your five tests, from source to destination and port.
- The `default deny` counter shows **10 packets, 600 bytes**: the same ten packets (60 bytes each). The two `accept` rules for new connections show **0**, so none of the blocked flows slipped through an allow rule by mistake.
- Because the log and the counter agree, you know every block came from the default deny and not from a typo or a missing route.

Your counts may differ by a few packets if your timeouts or retries differ, but the log and the counter must match each other.

> [!NOTE]
> `tcpdump` doesn't print the `prefix` text of the log rule here. In a production setup a log daemon such as `ulogd` stores the nflog records, with the prefix, to a file or a log server. Reading it with `tcpdump` is enough for the lab.

### Step 10 — Rule order: where a new rule goes (8 min)
Suppose the web team asks you to publish the port 8080 service too. You add the rule the obvious way, with `nft add rule`, which **appends** the rule to the end of the chain.

```bash
podman exec lab-netsec-fw nft add rule inet lab_fw forward iifname net0 oifname dmz0 ip daddr 10.40.2.10 tcp dport 8080 ct state new counter accept comment '"internet to web: 8080 (added)"'
podman exec lab-netsec-client nc -w 2 -z -v 10.40.2.10 8080; echo rc=$?
podman exec lab-netsec-fw nft -a list chain inet lab_fw forward | tail -5
```

Expected output:
```text
nc: 10.40.2.10 (10.40.2.10:8080): Operation timed out
rc=1
		limit rate 5/second burst 10 packets log prefix "lab-netsec DROP: " group 1 # handle 6
		counter packets 12 bytes 720 drop comment "default deny" # handle 7
		iifname "net0" oifname "dmz0" ip daddr 10.40.2.10 tcp dport 8080 ct state new counter packets 0 bytes 0 accept comment "internet to web: 8080 (added)" # handle 8
	}
}
```

The rule exists, and the port **still doesn't open**. The new rule (handle 8) sits **after** the `drop` (handle 7), so the packet is dropped before it can match, and the counter of the new rule stays at 0. The `-a` option shows each rule's **handle**, its permanent number. Delete the useless rule and insert a new one **before** the log rule, at the position of handle 6:

```bash
podman exec lab-netsec-fw nft delete rule inet lab_fw forward handle 8
podman exec lab-netsec-fw nft insert rule inet lab_fw forward position 6 iifname net0 oifname dmz0 ip daddr 10.40.2.10 tcp dport 8080 ct state new counter accept comment '"internet to web: 8080 (inserted)"'
podman exec lab-netsec-client nc -w 2 -z -v 10.40.2.10 8080; echo rc=$?
podman exec lab-netsec-fw nft list chain inet lab_fw forward | grep 8080
```

Expected output (the handle numbers you typed must match your own `nft -a` listing):
```text
10.40.2.10 (10.40.2.10:8080) open
rc=0
		iifname "net0" oifname "dmz0" ip daddr 10.40.2.10 tcp dport 8080 ct state new counter packets 1 bytes 60 accept comment "internet to web: 8080 (inserted)"
```

Now the port is open and the rule's counter shows the one connection. That is exactly the exposure the default-deny policy was removing, so this is a good moment to undo it: reload the original policy, which deletes and recreates the whole table.

```bash
podman exec -i lab-netsec-fw nft -f - < fw.nft
```

The load prints nothing, and the table is back to the five rules of Step 7.

> [!TIP]
> In real life, keep the rule file in version control and **reload the file** instead of typing single rules. Then the file is always the truth, a reviewer can read the change, and you can't leave a forgotten test rule in the firewall.

## Verification
The policy is correct when every positive test passes and every negative test is blocked. Write the checks into a script. `t` runs one test: it uses `nc` from a container to a destination and port and compares the result with what you expect.

```bash
cat > check.sh <<'EOT'
#!/bin/sh
# usage: ./check.sh (run from ~/netsec-lab); prints PASS or FAIL for every flow in the policy
pass=0; fail=0
t() {  # t <expected: open|blocked> <from-container> <dst-ip> <port>
  if podman exec "lab-netsec-$2" nc -w 2 -z "$3" "$4" >/dev/null 2>&1; then got=open; else got=blocked; fi
  if [ "$got" = "$1" ]; then r=PASS; pass=$((pass+1)); else r=FAIL; fail=$((fail+1)); fi
  printf '%s  %-7s -> %-12s :%-5s expected %-7s got %s\n' "$r" "$2" "$3" "$4" "$1" "$got"
}
t open    client 10.40.2.10 80
t blocked client 10.40.2.10 22
t blocked client 10.40.2.10 8080
t blocked client 10.40.3.10 5432
t blocked client 10.40.3.10 22
t open    web    10.40.3.10 5432
t blocked web    10.40.3.10 22
t blocked web    203.0.113.10 80
t blocked db     10.40.2.10 80
echo "passed=$pass failed=$fail"
[ "$fail" -eq 0 ]
EOT
chmod +x check.sh
./check.sh; echo rc=$?
```

Expected output (the run takes about 15 seconds, because each blocked test waits 2 seconds):
```text
PASS  client  -> 10.40.2.10   :80    expected open    got open
PASS  client  -> 10.40.2.10   :22    expected blocked got blocked
PASS  client  -> 10.40.2.10   :8080  expected blocked got blocked
PASS  client  -> 10.40.3.10   :5432  expected blocked got blocked
PASS  client  -> 10.40.3.10   :22    expected blocked got blocked
PASS  web     -> 10.40.3.10   :5432  expected open    got open
PASS  web     -> 10.40.3.10   :22    expected blocked got blocked
PASS  web     -> 203.0.113.10 :80    expected blocked got blocked
PASS  db      -> 10.40.2.10   :80    expected blocked got blocked
passed=9 failed=0
rc=0
```

Pass criteria:
- All nine lines say `PASS` and the script exits with `rc=0`. The two `open` lines are your two allowed flows; the seven `blocked` lines are the negative tests.
- `podman exec lab-netsec-fw nft list chain inet lab_fw forward` shows only the five rules of Step 7 (no 8080 rule), with non-zero counters on the two accept rules and on the drop.
- A scan from the client (Step 8) shows only port 80 open on `10.40.2.10` and nothing open on `10.40.3.10`.

> [!NOTE]
> This script treats "the connection failed" as "blocked". A refusal and a timeout both count. In this lab that is fine, because the services you test are listening, so a failure means the firewall dropped it. In a real network, remember the difference: a refusal means the packet arrived, a timeout means something dropped it.

## Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| `nmap` prints "Couldn't open a raw socket" | You used `-sS` or left out `-Pn`: both need raw sockets, which the rootless container doesn't have | Use `-sT -Pn` as in the lab |
| Everything times out from the client, even port 80, before you load any rule | A route is missing: the client needs `10.40.0.0/16 via 203.0.113.254`, the servers need their default route | Compare `podman exec lab-netsec-client ip route` with Step 5. Recreate the container that is missing the route |
| Port 80 is blocked after you loaded the policy | The interface names or addresses in the rule don't match, or the file was edited | `nft list chain inet lab_fw forward`: if the `internet to web` counter is 0 while the drop counter grows, the rule doesn't match. Check `net0`, `dmz0` and `10.40.2.10` |
| The drop log (`/tmp/drops.log`) is empty but the drop counter grows | The capture wasn't running when you tested, or the log rule is missing, or the log group differs | Start the capture (Step 9) first and test within its 60 seconds. Check the rule says `group 1` |
| `Error: No such file or directory` when you load the rules | The `nft` binary can't create the table: the container lacks `NET_ADMIN`, or a typo in the file | Check the `--cap-add NET_ADMIN` of the `podman run` in Step 4. Read the line number printed with the error |
| `the container name "lab-netsec-fw" is already in use` | A container from a previous attempt is left | Run the [Cleanup](#cleanup), then start again |
| A new rule doesn't open the port | The rule was appended after the final drop | List with `nft -a`, delete by handle and `insert ... position <handle>` before the log rule (Step 10) |

## Challenge (optional)
Extend the policy without step-by-step help:
1. The web server needs to download updates. Start a small web server in the client container on TCP 8000 (`httpd -p 8000` with a folder of your choice), and write the **narrowest** rule that lets `lab-netsec-web` (and only it) open connections to `203.0.113.10` on TCP 8000. Prove with `./check.sh` (add the test) and a scan that nothing else was opened.
2. Add a fourth host in the internal zone, `lab-netsec-admin` at `10.40.3.20`, that may open connections to the web server on TCP 22, and nobody else may. Where do you put the rule, and which test proves that the database can't do the same?
3. Make the Internet-facing rule a **reject** instead of a drop (`reject with tcp reset`) for a single port and compare how a scan reports it with `filtered`.

## Cleanup
Remove every container, network and the image created by this lab, then the folder. `-t 0` skips the 10-second wait for containers that ignore the stop signal.

```bash
podman rm -f -t 0 lab-netsec-client lab-netsec-web lab-netsec-db lab-netsec-fw
podman network rm lab-netsec-internet lab-netsec-dmz lab-netsec-internal
podman rmi lab-netsec-tools:1
cd ~ && rm -rf ~/netsec-lab
podman ps -a --filter name=lab-netsec --format '{{.Names}}'
podman network ls --filter name=lab-netsec --format '{{.Name}}'
```

The two last commands print nothing when the cleanup is complete. The Alpine base image stays in your Podman storage (about 8 MB); remove it with `podman rmi docker.io/library/alpine:3.20.10` if no other lab needs it.

## What you learned
- Scanning from the outside shows the real attack surface; scanning from each zone tests the internal rules too.
- A stateful, default-deny rule set needs rules only for the flows you want: here two new-connection rules and one `established` rule were enough.
- `closed` means refused and `filtered` means dropped: a firewall's drops look like silence.
- Egress filtering and DMZ containment mean that a taken-over web server can reach one database port and nothing else.
- The rule counters and the drop log must agree; together they prove that a flow was blocked by your policy.
- Rule order matters: a rule after the final drop never matches. Keep rules in a file and reload the file.