---
title: "Lab: Build a Virtual Network on Your Machine"
domain: cloud-computing
level: intermediate
duration: "90 min"
cost: "free"
environment: "containers (rootless Podman on a local Linux machine)"
---

# Lab: Build a Virtual Network on Your Machine

## Goal
You build a miniature VPC with Podman: a "public" subnet, a "private" subnet and a gateway container that routes between them, does NAT for the private subnet and enforces security-group-like (stateful) and network-ACL-like (stateless) rules. You test it, break it in three realistic ways and troubleshoot it with `curl`, `ping` and the packet counters of the firewall.

## Objectives
- Build subnets with non-overlapping private ranges and give them routes through a gateway.
- Show that a NAT gateway makes outbound connections work and keeps the private subnet unreachable from outside.
- Write a stateful security-group-style rule set and a stateless ACL-style rule set with nftables, and show the difference with return traffic and ephemeral ports.
- Show that the first matching ACL rule wins.
- Diagnose a missing route, a missing NAT rule and a missing ACL rule from the symptoms and the counters.

## Prerequisites
- A Linux machine with **Podman 4 or later** running **rootless** (as a normal user; you never need `sudo`) and the netavark network backend. Tested with Podman 5.8.7 on Fedora with kernel 7.2. Check with `podman info --format '{{.Host.NetworkBackend}}'`: it should print `netavark`.
- Internet access once, to download the Alpine Linux image (about 8 MB) and a few packages (about 18 MB in total). After that, nothing in the lab needs the Internet.
- About 300 MB of free disk space.
- The [lesson](lesson.md) of this module, [Networking 06 — Subnetting](../../networking/06-subnetting/README.md) and [Networking 10 — Routing Fundamentals](../../networking/10-routing-fundamentals/README.md). The routing lab there built routers from network namespaces with `sudo`; here you use containers and no `sudo`.

> [!WARNING]
> Everything runs on your own machine, in containers and networks whose names start with `lab-vnet-`. All three networks are created with `--internal`, so nothing in the lab can reach your real network or the Internet, and nothing can reach it from outside. The addresses are private ranges (`10.30.0.0/16`, RFC 1918) and the documentation range `198.51.100.0/24` (RFC 5737). If your own network already uses `10.30.1.0/24` or `10.30.2.0/24`, change the second-to-last number in every address consistently, or pick a free range. The gateway container gets the `NET_ADMIN` capability, which lets it change its own network settings; in rootless Podman that power applies only inside your user's namespace, not to the host. Don't reuse these options for containers that run untrusted software. Run the [Cleanup](#cleanup) section at the end.

## Architecture
![Three internal Podman networks meet in the gateway container lab-vnet-gw, which has interfaces pub0 10.30.1.254, priv0 10.30.2.254 and out0 198.51.100.254 plus 198.51.100.50. The public subnet 10.30.1.0/24 holds lab-vnet-bastion 10.30.1.10 with default route via 10.30.1.254. The private subnet 10.30.2.0/24 holds lab-vnet-web 10.30.2.10 on tcp 8080 with default route via 10.30.2.254. The outside network 198.51.100.0/24 holds lab-vnet-internet 198.51.100.10 on tcp 8000, which stands for the Internet. The gateway applies three nftables tables: lab_nat, lab_sg and lab_acl.](../../../assets/diagrams/vnet-lab-architecture.svg)

*Figure 1. One gateway container plays the roles of router, Internet gateway, NAT gateway and firewall. The "Internet" is a web server on a third, isolated network.*

| Container | Network and address | Role in a real VPC |
|---|---|---|
| `lab-vnet-bastion` | `lab-vnet-public`, `10.30.1.10` | An instance in a public subnet, with an elastic IP |
| `lab-vnet-web` | `lab-vnet-private`, `10.30.2.10` | An instance in a private subnet, serving HTTP on TCP 8080 |
| `lab-vnet-internet` | `lab-vnet-outside`, `198.51.100.10` | The Internet: a web server on TCP 8000 |
| `lab-vnet-gw` | `10.30.1.254`, `10.30.2.254`, `198.51.100.254` and `198.51.100.50` | Route tables' next hop, Internet gateway, NAT gateway and the firewall |

> [!NOTE]
> A real cloud splits the gateway's work into several managed pieces, and applies filters at every interface. Here one container does it all, and the security group and the ACL are rules on that container, written for one destination or subnet. The behaviour you observe (routing, NAT, stateful versus stateless filtering) is the real one, the packaging is a simplification. Also note that a "public" and a "private" subnet are both `--internal` networks: what makes them public or private is the gateway's rules, as in a real VPC where it is the route table and the gateway.

Podman runs a command in a container with `podman exec lab-vnet-<name> <command>`. The lab uses that all the time, so each command is spelled out in full.

## Steps

### Step 1 — Check your tools and create a working folder (3 min)
Make sure Podman is rootless and uses netavark, and that nothing is left over from an earlier attempt.

```bash
podman --version
podman info --format '{{.Host.NetworkBackend}} rootless={{.Host.Security.Rootless}}'
podman ps -a --filter name=lab-vnet --format '{{.Names}}'
podman network ls --filter name=lab-vnet --format '{{.Name}}'
mkdir -p ~/vnet-lab && cd ~/vnet-lab
```

Expected output (your version may differ):
```text
podman version 5.8.7
netavark rootless=true
```

The two `ls` commands print nothing. If they list `lab-vnet-` names, run the [Cleanup](#cleanup) first. All the files and commands that follow assume you are in `~/vnet-lab`.

### Step 2 — Build the tools image (5 min)
Alpine Linux is tiny, but it doesn't include the tools you need. You build one image with `nftables` (the Linux firewall), `curl`, the full `ip` command and a small web server (`httpd`, from `busybox-extras`). The image tag is pinned to Alpine 3.20, so you get the same result as the outputs below.

```bash
cat > Containerfile <<'EOF'
FROM docker.io/library/alpine:3.20.10
RUN apk add --no-cache nftables curl iproute2 busybox-extras
EOF
podman build -t lab-vnet-tools:1 .
podman run --rm lab-vnet-tools:1 sh -c 'nft --version; curl --version | head -1; ip -V; cat /etc/alpine-release'
```

Expected output (the end of the build and the versions; package versions may differ slightly):
```text
OK: 18 MiB in 40 packages
COMMIT lab-vnet-tools:1
Successfully tagged localhost/lab-vnet-tools:1
nftables v1.0.9 (Old Doc Yak #3)
curl 8.14.1 (x86_64-alpine-linux-musl) libcurl/8.14.1 OpenSSL/3.3.7 zlib/1.3.2 brotli/1.1.0 zstd/1.5.6 c-ares/1.33.1 libidn2/2.3.7 libpsl/0.21.5 nghttp2/1.62.1
ip utility, iproute2-6.9.0
3.20.10
```

### Step 3 — Create the three networks (4 min)
Each Podman network is a subnet: a bridge with its own CIDR block. `--internal` means that Podman adds no route to the outside world and no NAT to your host, so nothing inside can get out except through the gateway you build. Two subnets are inside the VPC range `10.30.0.0/16` and don't overlap. The third, from the documentation range, stands for the Internet.

```bash
podman network create --internal --subnet 10.30.1.0/24 lab-vnet-public
podman network create --internal --subnet 10.30.2.0/24 lab-vnet-private
podman network create --internal --subnet 198.51.100.0/24 lab-vnet-outside
podman network inspect lab-vnet-public lab-vnet-private lab-vnet-outside \
  --format '{{.Name}} subnet={{(index .Subnets 0).Subnet}} internal={{.Internal}}'
```

Expected output:
```text
lab-vnet-public
lab-vnet-private
lab-vnet-outside
lab-vnet-public subnet=10.30.1.0/24 internal=true
lab-vnet-private subnet=10.30.2.0/24 internal=true
lab-vnet-outside subnet=198.51.100.0/24 internal=true
```

### Step 4 — Start the gateway (6 min)
The gateway has one interface in each network, with a fixed address: the last usable one (`.254`) in each subnet, like a cloud's "router" address. `interface_name` gives the interfaces readable names. `--cap-add NET_ADMIN` lets it edit its own firewall and addresses, and `--sysctl net.ipv4.ip_forward=1` turns it into a router, as you did with namespaces in the routing lab. `sleep infinity` keeps it running.

```bash
podman run -d --name lab-vnet-gw \
  --network lab-vnet-public:ip=10.30.1.254,interface_name=pub0 \
  --network lab-vnet-private:ip=10.30.2.254,interface_name=priv0 \
  --network lab-vnet-outside:ip=198.51.100.254,interface_name=out0 \
  --cap-add NET_ADMIN --sysctl net.ipv4.ip_forward=1 \
  lab-vnet-tools:1 sleep infinity
podman exec lab-vnet-gw ip -br addr
podman exec lab-vnet-gw sysctl net.ipv4.ip_forward
podman exec lab-vnet-gw ip route
```

Expected output (the container ID, the order of the interfaces, the `@if` numbers and the `fe80::` addresses vary):
```text
aefbd0e9520a9e36dd5b2e54f5d4799ecc592e6ee97c80d9ff1663484a52dda8
lo               UNKNOWN        127.0.0.1/8 ::1/128 
priv0@if4        UP             10.30.2.254/24 fe80::e8e5:faff:fe8d:8e43/64 
out0@if6         UP             198.51.100.254/24 fe80::fc55:f5ff:fef6:ab65/64 
pub0@if8         UP             10.30.1.254/24 fe80::647e:b8ff:fe3d:9f28/64 
net.ipv4.ip_forward = 1
10.30.1.0/24 dev pub0 proto kernel scope link src 10.30.1.254 
10.30.2.0/24 dev priv0 proto kernel scope link src 10.30.2.254 
198.51.100.0/24 dev out0 proto kernel scope link src 198.51.100.254 
```

The gateway's route table has only the three **connected routes**, one per subnet: the equivalent of the implicit local route. It has no default route, and it needs none: it is directly connected to all three networks.

The gateway also needs a second public address, the "elastic IP" of the bastion. You add it to the outside interface:

```bash
podman exec lab-vnet-gw ip addr add 198.51.100.50/24 dev out0
podman exec lab-vnet-gw ip -br addr show out0
```

Expected output:
```text
out0@if6         UP             198.51.100.254/24 198.51.100.50/24 fe80::fc55:f5ff:fef6:ab65/64 
```

### Step 5 — Launch the instances and give them routes (6 min)
Each instance joins one subnet. An internal network gives a container no default route, like a subnet whose route table is empty. Each instance adds one default route through the gateway (`ip route add default via ...`): that is the route table of its subnet. `NET_ADMIN` is needed in the containers that run the `ip route add` command. The web server and the "Internet" each publish a one-line page with the built-in `httpd`; `-v` makes it log every connection to the container's log.

```bash
I=lab-vnet-tools:1
podman run -d --name lab-vnet-web --network lab-vnet-private:ip=10.30.2.10 --cap-add NET_ADMIN $I \
  sh -c 'ip route add default via 10.30.2.254; mkdir -p /www; echo "hello from the private web tier" > /www/index.html; exec httpd -f -v -p 8080 -h /www'
podman run -d --name lab-vnet-bastion --network lab-vnet-public:ip=10.30.1.10 --cap-add NET_ADMIN $I \
  sh -c 'ip route add default via 10.30.1.254; exec sleep infinity'
podman run -d --name lab-vnet-internet --network lab-vnet-outside:ip=198.51.100.10 $I \
  sh -c 'mkdir -p /www; echo "hello from the Internet" > /www/index.html; exec httpd -f -v -p 8000 -h /www'
podman ps --filter name=lab-vnet --format 'table {{.Names}}\t{{.Status}}'
```

Expected output (three container IDs first; the times vary):
```text
NAMES              STATUS
lab-vnet-gw        Up 8 seconds
lab-vnet-web       Up 1 second
lab-vnet-bastion   Up 1 second
lab-vnet-internet  Up 1 second
```

Now read the "route tables":

```bash
podman exec lab-vnet-web ip route
podman exec lab-vnet-bastion ip route
podman exec lab-vnet-internet ip route
```

Expected output:
```text
default via 10.30.2.254 dev eth0 
10.30.2.0/24 dev eth0 proto kernel scope link src 10.30.2.10 
default via 10.30.1.254 dev eth0 
10.30.1.0/24 dev eth0 proto kernel scope link src 10.30.1.10 
198.51.100.0/24 dev eth0 proto kernel scope link src 198.51.100.10 
```

The "Internet" host has no default route, and no route to your private ranges: like the real Internet, it knows nothing about `10.30.0.0/16`, because RFC 1918 addresses aren't routed there.

### Step 6 — Routing alone: everything is open (8 min)
With forwarding on and no filter rules yet, the gateway passes everything between the two private subnets. Test from the bastion:

```bash
podman exec lab-vnet-bastion curl -sS -m 3 http://10.30.2.10:8080
podman exec lab-vnet-bastion ping -c 2 10.30.2.10 | tail -3
podman exec lab-vnet-bastion curl -sS -m 3 http://10.30.2.10:22; echo "rc=$?"
```

Expected output (the ping times vary):
```text
hello from the private web tier
--- 10.30.2.10 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.050/0.055/0.061 ms
curl: (7) Failed to connect to 10.30.2.10 port 22 after 0 ms: Could not connect to server
rc=7
```

Two lessons. The path works because the bastion's route sends `10.30.2.10` to the gateway, which forwards it. And nothing stops port 22: it fails only because **nothing listens** there, and the host answers at once with a refusal (`rc=7`, "refused"). Remember this: it is the opposite of a timeout.

Next, the private web server tries to reach the Internet, and so does the bastion:

```bash
podman exec lab-vnet-web curl -sS -m 3 http://198.51.100.10:8000; echo "rc=$?"
podman exec lab-vnet-bastion curl -sS -m 3 http://198.51.100.10:8000; echo "rc=$?"
```

Expected output (the timings in milliseconds may vary):
```text
curl: (28) Connection timed out after 3002 milliseconds
rc=28
curl: (28) Connection timed out after 3002 milliseconds
rc=28
```

Both time out. The requests reach the "Internet" host, but it can't answer: the source is `10.30.2.10` or `10.30.1.10`, and the Internet host has no route to those private ranges. The reply is dropped silently. That is why private subnets need NAT.

### Step 7 — The NAT gateway and the elastic IP (10 min)
You add a `nat` table to the gateway with the Linux firewall, **nftables**. A source NAT rule in the `postrouting` hook rewrites the source address of packets leaving towards the "Internet":
- The whole private subnet is **masqueraded** to the gateway's own address `198.51.100.254`: many-to-one NAT, the way a NAT gateway works. Replies are translated back automatically.
- The bastion gets a fixed public address, `198.51.100.50`: a one-to-one mapping, the way a public or elastic IP works for an instance.

Write the rules to a file. The first two lines make the file safe to load again: they create the table if it doesn't exist and then delete it, so each load starts clean.

```bash
cat > nat.nft <<'EOF'
table ip lab_nat
delete table ip lab_nat
table ip lab_nat {
  chain postrouting {
    type nat hook postrouting priority srcnat; policy accept;
    ip saddr 10.30.1.10 ip daddr 198.51.100.0/24 counter snat to 198.51.100.50 comment "elastic IP of the bastion"
    ip saddr 10.30.2.0/24 ip daddr 198.51.100.0/24 counter snat to 198.51.100.254 comment "NAT gateway for the private subnet"
  }
}
EOF
podman exec -i lab-vnet-gw nft -f - < nat.nft
podman exec lab-vnet-web curl -sS -m 3 http://198.51.100.10:8000
podman exec lab-vnet-bastion curl -sS -m 3 http://198.51.100.10:8000
```

`podman exec -i ... nft -f -` reads the rules from standard input, so the file stays on your machine. The load prints nothing. Expected output of the two requests:
```text
hello from the Internet
hello from the Internet
```

Both work now. Look at the "Internet" server's log, which shows who it thinks is connecting, and at the NAT table and the gateway's connection tracking:

```bash
podman logs lab-vnet-internet
podman exec lab-vnet-gw nft list table ip lab_nat
podman exec lab-vnet-gw sh -c "grep 198.51.100.10 /proc/net/nf_conntrack | sed 's/ \[ASSURED\].*//; s/^ipv4 *2 //'"
```

Expected output (the port numbers vary):
```text
[::ffff:198.51.100.254]:52550: response:200
[::ffff:198.51.100.50]:43524: response:200
table ip lab_nat {
	chain postrouting {
		type nat hook postrouting priority srcnat; policy accept;
		ip saddr 10.30.1.10 ip daddr 198.51.100.0/24 counter packets 1 bytes 60 snat to 198.51.100.50 comment "elastic IP of the bastion"
		ip saddr 10.30.2.0/24 ip daddr 198.51.100.0/24 counter packets 1 bytes 60 snat to 198.51.100.254 comment "NAT gateway for the private subnet"
	}
}
tcp      6 119 TIME_WAIT src=10.30.2.10 dst=198.51.100.10 sport=52550 dport=8000 src=198.51.100.10 dst=198.51.100.254 sport=8000 dport=52550
tcp      6 119 TIME_WAIT src=10.30.1.10 dst=198.51.100.10 sport=43524 dport=8000 src=198.51.100.10 dst=198.51.100.50 sport=8000 dport=43524
```

The web server logged `198.51.100.254` (the NAT gateway) for the private instance and `198.51.100.50` (the elastic IP) for the bastion. Neither private address ever left. Each tracking line shows both directions: the original (`src=10.30.2.10`) and the reply the gateway expects (`dst=198.51.100.254`). That table is how the NAT gateway knows where each reply goes, and why it is outbound only. Now try the opposite direction, from the "Internet" to the private web server:

```bash
podman exec lab-vnet-internet curl -sS -m 3 http://10.30.2.10:8080; echo "rc=$?"
```

Expected output:
```text
curl: (7) Failed to connect to 10.30.2.10 port 8080 after 0 ms: Could not connect to server
rc=7
```

It fails at once: the Internet host has no route to a private address. Even with a route, the NAT gateway has no connection entry for a new inbound flow.

### Step 8 — The security group: stateful rules (12 min)
A security group is stateful and allow-only. In nftables you get the same behaviour with connection tracking (`ct state`) and a **default drop** policy on the `forward` hook, the path traffic takes through the gateway. The rules are the ones of the lesson's worked example, reduced to what the lab needs:

- The first rule accepts `established,related` packets: the replies of any connection that was allowed. That one line is the "stateful" part.
- `sg-web` accepts new connections to `10.30.2.10` only from the public subnet, only on TCP 8080, plus ping.
- Both subnets may start connections to the "Internet".
- Anything else hits the last rule, which counts and drops.

```bash
cat > sg.nft <<'EOF'
table ip lab_sg
delete table ip lab_sg
table ip lab_sg {
  chain forward {
    type filter hook forward priority 0; policy drop;
    ct state invalid drop
    ct state established,related counter accept comment "replies are automatic (stateful)"
    # sg-web, inbound: only the public subnet, only TCP 8080 and ping
    ip saddr 10.30.1.0/24 ip daddr 10.30.2.10 tcp dport 8080 counter accept comment "sg-web in: TCP 8080 from public"
    ip saddr 10.30.1.0/24 ip daddr 10.30.2.10 icmp type echo-request counter accept comment "sg-web in: ping from public"
    # outbound to the Internet
    ip saddr 10.30.1.0/24 ip daddr 198.51.100.0/24 counter accept comment "public out: Internet"
    ip saddr 10.30.2.0/24 ip daddr 198.51.100.0/24 counter accept comment "private out: Internet"
    counter drop comment "default deny"
  }
}
EOF
podman exec -i lab-vnet-gw nft -f - < sg.nft
```

Test it. The first three are allowed, the last two must fail:

```bash
podman exec lab-vnet-bastion curl -sS -m 3 http://10.30.2.10:8080
podman exec lab-vnet-bastion ping -c 2 10.30.2.10 | tail -3
podman exec lab-vnet-web curl -sS -m 3 http://198.51.100.10:8000
podman exec lab-vnet-bastion curl -sS -m 3 http://10.30.2.10:22; echo "rc=$?"
podman exec lab-vnet-web ping -c 1 -W 2 10.30.1.10 | tail -2
```

Expected output (the ping times vary):
```text
hello from the private web tier
--- 10.30.2.10 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.082/0.088/0.094 ms
hello from the Internet
curl: (28) Connection timed out after 3002 milliseconds
rc=28
--- 10.30.1.10 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
```

Compare port 22 with Step 6. Before the firewall it was **refused** at once (`rc=7`): the packet arrived. Now it **times out** (`rc=28`): the gateway dropped the packet without any reply. And the private web server, which could answer the bastion, cannot start a connection to it: nothing allows that direction.

Look at the counters:

```bash
podman exec lab-vnet-gw nft list chain ip lab_sg forward
```

Expected output (the packet counts differ on your machine):
```text
table ip lab_sg {
	chain forward {
		type filter hook forward priority filter; policy drop;
		ct state invalid drop
		ct state established,related counter packets 25 bytes 2047 accept comment "replies are automatic (stateful)"
		ip saddr 10.30.1.0/24 ip daddr 10.30.2.10 tcp dport 8080 counter packets 1 bytes 60 accept comment "sg-web in: TCP 8080 from public"
		ip saddr 10.30.1.0/24 ip daddr 10.30.2.10 icmp type echo-request counter packets 1 bytes 84 accept comment "sg-web in: ping from public"
		ip saddr 10.30.1.0/24 ip daddr 198.51.100.0/24 counter packets 0 bytes 0 accept comment "public out: Internet"
		ip saddr 10.30.2.0/24 ip daddr 198.51.100.0/24 counter packets 1 bytes 60 accept comment "private out: Internet"
		counter packets 4 bytes 264 drop comment "default deny"
	}
}
```

Each accepted connection used exactly one "new" rule, for its first packet (the TCP SYN or the ping request). Every other packet, in both directions, matched the `established,related` line: the replies needed no rule. The `drop` line counted the blocked SYN packets (several, because `curl` retried), and the ping to the bastion.

> [!NOTE]
> The curl to port 22 is retried while it waits, so the drop counter rises by more than one. The exact numbers depend on timing.

### Step 9 — The network ACL: stateless, ordered rules (12 min)
Now a second filter for the **private subnet** `10.30.2.0/24`, in its own table at an earlier priority (`-10`), so it runs before the security group. A packet must pass both. This ACL follows the rules of a cloud network ACL:

- It never uses `ct state`: each packet is judged alone, so it is **stateless**.
- Rules are in order and the first match wins. The comments show the rule numbers "in 100", "out 100" and so on.
- It has both allow and deny, with a final `*` deny in each direction.
- Return traffic needs explicit rules, with the **ephemeral port range** `1024-65535`.

```bash
cat > acl.nft <<'EOF'
table ip lab_acl
delete table ip lab_acl
table ip lab_acl {
  chain forward {
    type filter hook forward priority -10; policy accept;
    # Network ACL of the private subnet 10.30.2.0/24: ordered, first match wins, no "ct state" anywhere
    # Inbound (packets entering the subnet)
    ip daddr 10.30.2.0/24 ip saddr 10.30.1.0/24 tcp dport 8080 counter accept comment "in 100: TCP 8080 from public"
    ip daddr 10.30.2.0/24 ip saddr 10.30.1.0/24 icmp type echo-request counter accept comment "in 110: ping from public"
    ip daddr 10.30.2.0/24 ip saddr 198.51.100.0/24 tcp dport 1024-65535 counter accept comment "in 120: replies from the Internet"
    ip daddr 10.30.2.0/24 counter drop comment "in *: deny"
    # Outbound (packets leaving the subnet)
    ip saddr 10.30.2.0/24 ip daddr 10.30.1.0/24 tcp sport 8080 tcp dport 1024-65535 counter accept comment "out 100: replies to public"
    ip saddr 10.30.2.0/24 ip daddr 10.30.1.0/24 icmp type echo-reply counter accept comment "out 110: ping replies"
    ip saddr 10.30.2.0/24 ip daddr 198.51.100.0/24 tcp dport 8000 counter accept comment "out 120: HTTP to the Internet"
    ip saddr 10.30.2.0/24 counter drop comment "out *: deny"
  }
}
EOF
podman exec -i lab-vnet-gw nft -f - < acl.nft
```

Mind the order of events inside the gateway. Outbound packets still have their private source address when the `forward` hook runs, because source NAT happens later, in `postrouting`. Replies from the Internet have already been translated back to the private address when they reach `forward`. So the ACL always sees the private addresses, which is why its rules mention `10.30.2.0/24`.

Repeat the tests:

```bash
podman exec lab-vnet-bastion curl -sS -m 3 http://10.30.2.10:8080
podman exec lab-vnet-web curl -sS -m 3 http://198.51.100.10:8000
podman exec lab-vnet-web curl -sS -m 3 http://198.51.100.10:80; echo "rc=$?"
```

Expected output:
```text
hello from the private web tier
hello from the Internet
curl: (28) Connection timed out after 3002 milliseconds
rc=28
```

The first two pass both filters. The third goes to TCP 80 on the Internet host: the security group allows any new connection to the Internet, but the ACL's outbound rule 120 allows only TCP 8000, so the ACL drops it. Two layers, and a packet needs both.

### Step 10 — Stateless versus stateful: remove the return rule (10 min)
This is the experiment of the lesson. Remove rule "out 100", the one that lets the web server's replies go back to the public subnet. The security group still has its stateful `established,related` rule and the ACL's inbound rule still lets the request in. First find the rule's handle (its number inside nftables):

```bash
podman exec lab-vnet-gw nft -a list chain ip lab_acl forward | grep -E "out 100|out \*"
```

Expected output (the handle numbers may differ):
```text
		ip saddr 10.30.2.0/24 ip daddr 10.30.1.0/24 tcp sport 8080 tcp dport 1024-65535 counter packets 6 bytes 561 accept comment "out 100: replies to public" # handle 6
		ip saddr 10.30.2.0/24 counter packets 0 bytes 0 drop comment "out *: deny" # handle 9
```

Note that "out 100" already counted 6 packets: the replies of the HTTP request from Step 9. The deny at the end counted 0. Now delete the rule, using its handle (`6` here; use yours) and repeat the request:

```bash
podman exec lab-vnet-gw nft delete rule ip lab_acl forward handle 6
podman exec lab-vnet-bastion curl -sS -m 3 http://10.30.2.10:8080; echo "rc=$?"
podman exec lab-vnet-gw nft list chain ip lab_acl forward | grep "out \*"
podman exec lab-vnet-gw nft list chain ip lab_sg forward | grep "TCP 8080"
```

Expected output (the counts vary):
```text
curl: (28) Connection timed out after 3002 milliseconds
rc=28
		ip saddr 10.30.2.0/24 counter packets 4 bytes 240 drop comment "out *: deny"
		ip saddr 10.30.1.0/24 ip daddr 10.30.2.10 tcp dport 8080 counter packets 2 bytes 120 accept comment "sg-web in: TCP 8080 from public"
```

The web server received the request and answered, but the answer is a new packet for the stateless ACL, and `out *: deny` dropped it. The deny counter went from 0 to 4: the first reply and its retransmissions. The security group, which is stateful, isn't the problem: the same request was fine before. The rule you removed was the missing return path. Restore the ACL by loading the file again (it recreates the table with all rules) and test:

```bash
podman exec -i lab-vnet-gw nft -f - < acl.nft
podman exec lab-vnet-bastion curl -sS -m 3 http://10.30.2.10:8080
```

Expected output:
```text
hello from the private web tier
```

### Step 11 — Rule order: the first match wins (5 min)
An ACL is evaluated in order. Insert a deny rule for the bastion **before** rule "in 100". The `insert` command puts the rule at the top of the chain:

```bash
podman exec lab-vnet-gw nft insert rule ip lab_acl forward ip saddr 10.30.1.10 ip daddr 10.30.2.0/24 counter drop comment \"in 90: deny bastion\"
podman exec lab-vnet-gw nft -a list chain ip lab_acl forward | sed -n 3,5p
podman exec lab-vnet-bastion curl -sS -m 3 http://10.30.2.10:8080; echo "rc=$?"
```

Expected output (the handle numbers and counters may differ):
```text
		type filter hook forward priority filter - 10; policy accept;
		ip saddr 10.30.1.10 ip daddr 10.30.2.0/24 counter packets 0 bytes 0 drop comment "in 90: deny bastion" # handle 10
		ip daddr 10.30.2.0/24 ip saddr 10.30.1.0/24 tcp dport 8080 counter packets 6 bytes 399 accept comment "in 100: TCP 8080 from public" # handle 2
curl: (28) Connection timed out after 3002 milliseconds
rc=28
```

Rule "in 100" still allows TCP 8080 from the public subnet, but the deny rule comes first, and the first match wins. Remove it with its handle (`10` here; use yours):

```bash
podman exec lab-vnet-gw nft delete rule ip lab_acl forward handle 10
podman exec lab-vnet-bastion curl -sS -m 3 http://10.30.2.10:8080
```

Expected output:
```text
hello from the private web tier
```

> [!NOTE]
> A security group has no such order: any allow rule that matches is enough, and there are no deny rules. That is one reason to use security groups for normal access control and ACLs only for coarse blocks.

### Step 12 — Break and fix a route and a NAT rule (8 min)
Troubleshooting means reading symptoms. You make two faults and compare them with the timeouts you have seen. First, delete the default route of the private web server, as if its route table had lost the route to the gateway:

```bash
podman exec lab-vnet-web ip route del default
podman exec lab-vnet-web ip route
podman exec lab-vnet-web curl -sS -m 3 http://198.51.100.10:8000; echo "rc=$?"
podman exec lab-vnet-web ip route get 198.51.100.10; echo "rc=$?"
```

Expected output:
```text
10.30.2.0/24 dev eth0 proto kernel scope link src 10.30.2.10 
curl: (7) Failed to connect to 198.51.100.10 port 8000 after 0 ms: Could not connect to server
rc=7
RTNETLINK answers: Network unreachable
rc=2
```

The failure is **immediate** and `ip route get` says "Network unreachable": no route matches, so the packet never leaves the instance. That is the signature of a routing problem, not of a filter. Fix it, and test again:

```bash
podman exec lab-vnet-web ip route add default via 10.30.2.254
podman exec lab-vnet-web curl -sS -m 3 http://198.51.100.10:8000
```

Expected output:
```text
hello from the Internet
```

Now the second fault: empty the NAT table, as if the NAT gateway were gone or its rule had been removed. Then look at the counters of the filters, which show the packets did go out:

```bash
podman exec lab-vnet-gw nft flush chain ip lab_nat postrouting
podman exec lab-vnet-web curl -sS -m 3 http://198.51.100.10:8000; echo "rc=$?"
podman exec lab-vnet-gw nft list chain ip lab_sg forward | grep "private out"
```

Expected output (the counter varies):
```text
curl: (28) Connection timed out after 3002 milliseconds
rc=28
		ip saddr 10.30.2.0/24 ip daddr 198.51.100.0/24 counter packets 6 bytes 360 accept comment "private out: Internet"
```

This time the symptom is a **timeout**, like a filter drop. But the filters accepted the packets (the "private out" counter went up), and neither the SG nor the ACL dropped anything new. The packets left with the private source address, and the reply could not come back: the same failure as in Step 6. Here is the method of the lesson: a timeout means a silent drop; check the filters' counters, and if they accept the packet, check the return path and NAT. Fix it by loading the NAT file again:

```bash
podman exec -i lab-vnet-gw nft -f - < nat.nft
podman exec lab-vnet-web curl -sS -m 3 http://198.51.100.10:8000
```

Expected output:
```text
hello from the Internet
```

### Step 13 — Read the drop counters (4 min)
Every filter rule has a counter. When a flow doesn't work, the quickest way to find the layer is to ask which drop rule is counting. List all the drop rules with a comment:

```bash
podman exec lab-vnet-gw nft list ruleset | grep "drop comment" | sed 's/^\t*//'
```

Expected output (your counts depend on what you did, and are higher if you ran the verification script first):
```text
counter packets 0 bytes 0 drop comment "default deny"
ip daddr 10.30.2.0/24 counter packets 9 bytes 540 drop comment "in *: deny"
ip saddr 10.30.2.0/24 counter packets 2 bytes 168 drop comment "out *: deny"
```

The ACL drops (`in *` and `out *`) have counted packets, but the security group's `default deny` shows 0 here. That is because the ACL runs first (priority `-10`) and dropped those packets before the security group saw them. A rule that counts packets matched. A drop rule that counts is the one blocking you. A rule that stays at 0 is not the one that applies.

## Verification
Save this script and run it. It tests six behaviours and prints PASS or FAIL for each:

```bash
cat > verify.sh <<'EOF'
#!/bin/sh
# Prints PASS or FAIL for each expected behaviour of the virtual network.
check() { # check <description> <expected: ok|fail> <command...>
  desc=$1; want=$2; shift 2
  if "$@" >/dev/null 2>&1; then got=ok; else got=fail; fi
  if [ "$got" = "$want" ]; then echo "PASS  $desc"; else echo "FAIL  $desc (expected $want, got $got)"; fi
}
check "bastion reaches the web tier on TCP 8080"        ok   podman exec lab-vnet-bastion curl -s -m 3 http://10.30.2.10:8080
check "bastion cannot reach the web tier on TCP 22"     fail podman exec lab-vnet-bastion curl -s -m 3 http://10.30.2.10:22
check "bastion can ping the web tier"                   ok   podman exec lab-vnet-bastion ping -c 1 -W 2 10.30.2.10
check "web tier reaches the Internet through NAT"       ok   podman exec lab-vnet-web curl -s -m 3 http://198.51.100.10:8000
check "web tier cannot start a connection to the bastion" fail podman exec lab-vnet-web ping -c 1 -W 2 10.30.1.10
check "the Internet cannot reach the web tier"          fail podman exec lab-vnet-internet curl -s -m 3 http://10.30.2.10:8080
EOF
sh verify.sh
```

Expected output:
```text
PASS  bastion reaches the web tier on TCP 8080
PASS  bastion cannot reach the web tier on TCP 22
PASS  bastion can ping the web tier
PASS  web tier reaches the Internet through NAT
PASS  web tier cannot start a connection to the bastion
PASS  the Internet cannot reach the web tier
```

All six must pass. The script takes about 10 seconds, because the three "fail" checks wait for a timeout.

## Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| `podman network create` fails with "subnet already used" or similar | Another Podman network or an earlier attempt uses the same range | Run `podman network ls`, remove old `lab-vnet-` networks with the [Cleanup](#cleanup), or pick another free range |
| `unknown flag` or `interface_name` is rejected in Step 4 | Podman older than 4.x, or the CNI backend instead of netavark | Check `podman --version` and `podman info --format '{{.Host.NetworkBackend}}'`; use Podman 4+ with netavark |
| `nft` prints `Operation not permitted` | The gateway was started without `--cap-add NET_ADMIN` | Remove it with `podman rm -f -t 0 lab-vnet-gw` and run Step 4 again |
| Everything times out, even the first test of Step 6 | IP forwarding is off in the gateway, or an instance has no default route | `podman exec lab-vnet-gw sysctl net.ipv4.ip_forward` must print `1`; check `ip route` in the instance |
| `podman build` can't download packages | No Internet access or a proxy | The build needs the Internet once; configure your proxy for Podman, or build on a connected machine |
| `curl` in a container says `Could not resolve host` | You used a name instead of an address | The lab has no DNS: always use the IP addresses from the table |
| A step works but the counters stay at 0 | The packet matched an earlier rule, or it goes through another table | Read the chain from the top: rules apply in order; the first `accept` or `drop` ends it for that table |
| `Error: No such file or directory` when loading a file with `nft -f -` | You ran the command outside `~/vnet-lab`, so the shell can't find the `.nft` file | `cd ~/vnet-lab` and run it again |
| Containers are slow to stop | `sleep infinity` ignores the polite stop signal | Use `podman rm -f -t 0 ...` as in the Cleanup |

## Challenge (optional)
Make the bastion reachable **from the Internet** on its elastic IP, the way a public instance is. Start a second web server in the bastion (`podman exec -d lab-vnet-bastion sh -c 'mkdir -p /srv && echo "hello from the bastion" > /srv/index.html && exec httpd -f -p 8081 -h /srv'`) and make `podman exec lab-vnet-internet curl -sS -m 3 http://198.51.100.50:8081` print its page. You need three things, and each one is a concept from the lesson:
1. A **destination NAT** rule that translates `198.51.100.50` port 8081 to `10.30.1.10`. Use a new chain in `lab_nat` with `type nat hook prerouting priority dstnat; policy accept;`.
2. A rule in `lab_sg` that allows the new connection to `10.30.1.10` on TCP 8081. It must come before the final `drop`: use `nft insert rule` to put it at the top.
3. A look at the packet counters, to see whether the translation and the filter both matched.

Then check that the private web server is still unreachable from the Internet. Why is it, even though the same gateway is used?

## Cleanup
This removes every container, network and image of the lab, and the working folder. The `-t 0` option skips the stop delay.

```bash
podman rm -f -t 0 lab-vnet-gw lab-vnet-web lab-vnet-bastion lab-vnet-internet
podman network rm lab-vnet-public lab-vnet-private lab-vnet-outside
podman rmi lab-vnet-tools:1
cd ~ && rm -rf ~/vnet-lab
```

Check that nothing is left. The three commands should print nothing:

```bash
podman ps -a --filter name=lab-vnet --format '{{.Names}}'
podman network ls --filter name=lab-vnet --format '{{.Name}}'
podman images --filter reference='lab-vnet*' --format '{{.Repository}}'
```

`podman rmi` removes only the image you built. The downloaded Alpine base image stays in your cache, and other containers may use it. If you want it gone, run `podman rmi docker.io/library/alpine:3.20.10`.

## What you learned
- A VPC is a set of subnets joined by a router; a subnet is public or private because of its routes and its gateway rules, not its address range.
- A NAT gateway rewrites the source of outbound connections and tracks them, so replies return and nothing new can come in; a public or elastic IP is a one-to-one mapping.
- A stateful security group needs only the rule that starts the connection; a stateless ACL needs rules for both directions, including ephemeral ports.
- ACL rules are ordered and the first match wins; security-group rules are not ordered.
- "Network unreachable" means routing, "refused" means the host answered, and a "timeout" is a silent drop that can come from a filter, a missing return path or missing NAT.
- Counters tell you which rule saw your packet.
