CloudAcademia
EN

Networking · Module 10 · Routing Fundamentals

Lab: Three Routers with Static Routes, then OSPF

Lab75 minIntermediateMarkdown

Showing raw Markdown: the renderer could not load.

## Goal
You build the network from the lesson's [worked example](lesson.md#worked-example) on one Linux machine: two hosts and three routers in a line, each one a network namespace. You make it work with static and default routes, break it in realistic ways and troubleshoot it with `ping`, `ip route get` and `traceroute`. In an optional part, you replace the static routes with OSPF running in FRRouting containers.

## Objectives
- Turn a Linux network namespace into a router by enabling IP forwarding.
- Configure static routes and default routes so that traffic works in both directions.
- Diagnose a missing return route, a missing forward route and an unanswered traceroute hop with `ip route get` and `traceroute`.
- Watch longest prefix match choose between a default route, a `/24` and a `/32`.
- (Optional) Run OSPF between three FRRouting routers, read the neighbours and routes it builds, and watch it react to a link failure.

## Prerequisites
- A Linux machine or VM where you can use `sudo`. The lab was tested on Fedora with kernel 7.2.
- iproute2 (the `ip` command), 5.0 or later. Check with `ip -V` (tested with 6.17).
- `ping` (iputils) and `traceroute` (tested with "Modern traceroute for Linux" 2.1.6). If `traceroute` is missing, install it with `sudo dnf install traceroute` or `sudo apt install traceroute`.
- The [lesson](lesson.md) of this module, and [06 — Subnetting](../06-subnetting/README.md) for the prefixes.
- Optional part only: Podman 4 or later (tested with 5.8) or Docker 24 or later, Internet access to download the FRRouting image (about 95 MB). Part A takes about 50 minutes and Part B about 25.

> [!WARNING]
> Run this lab only on your own machine or a VM you control, not on a shared or production server. Everything happens inside network namespaces and containers that you create: the lab doesn't change your real interfaces or routes, and it doesn't send traffic to any other system. The addresses are private RFC 1918 ranges (`10.0.0.0/8`) plus the documentation range `198.51.100.0/24` (RFC 5737). If your own network already uses `10.1.1.0/24`, `10.3.3.0/24`, `10.0.12.0/24` or `10.0.23.0/24`, the lab still works, because namespaces are isolated, but the optional container networks may clash: see [Troubleshooting](#troubleshooting). Always run the [Cleanup](#cleanup) section at the end.

## Architecture
Five **network namespaces** stand in for five devices. Each namespace has its own interfaces, addresses, routing table and settings, just like a separate machine. Virtual Ethernet (veth) pairs act as cables between them.

<figure class="diagram" style="--dg-w: 1200px"><svg xmlns="http://www.w3.org/2000/svg" class="dg" viewBox="0 0 1200 300" width="1200" height="300" role="img" aria-labelledby="routing-lab-architecture-title routing-lab-architecture-desc" font-family="Inter, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" xml:lang="en">
<title id="routing-lab-architecture-title">Lab architecture: five network namespaces, three of them routers</title>
<desc id="routing-lab-architecture-desc">Five network namespaces in a line, joined by veth pairs. lab-h1 has eth0 10.1.1.10/24 and default route via 10.1.1.1. lab-r1 has to-h1 10.1.1.1/24 and to-r2 10.0.12.1/30. lab-r2 has to-r1 10.0.12.2/30 and to-r3 10.0.23.1/30. lab-r3 has to-r2 10.0.23.2/30 and to-h2 10.3.3.1/24. lab-h2 has eth0 10.3.3.10/24 and default route via 10.3.3.1. The three routers have IP forwarding enabled. Networks: 10.1.1.0/24, 10.0.12.0/30, 10.0.23.0/30 and 10.3.3.0/24.</desc>
<rect class="dg-bg" x="0.5" y="0.5" width="1199" height="299" rx="16" fill="#ffffff" stroke="#e3e3dd"/>
<g class="dg-l1"><path class="dg-stroke" d="M190.0 185H250.0" stroke="#0369a1" stroke-width="4" fill="none" stroke-linecap="round" stroke-linejoin="round"/></g>
<g class="dg-l1"><path class="dg-stroke" d="M220.0 179V60" stroke="#0369a1" stroke-width="1.5" fill="none" stroke-linecap="round" stroke-linejoin="round" stroke-dasharray="7 6"/></g>
<g class="dg-l1"><rect class="dg-tint" x="168.5" y="34" width="103" height="26" rx="13" fill="#e6f0f6" stroke="#8ebcd5" stroke-width="1.2"/><text x="220" y="51.5" font-size="12" class="dg-ink" fill="#0369a1" font-weight="700" text-anchor="middle">10.1.1.0/24</text></g>
<g class="dg-l1"><path class="dg-stroke" d="M450.0 185H500.0" stroke="#0369a1" stroke-width="4" fill="none" stroke-linecap="round" stroke-linejoin="round"/></g>
<g class="dg-l1"><path class="dg-stroke" d="M475.0 179V60" stroke="#0369a1" stroke-width="1.5" fill="none" stroke-linecap="round" stroke-linejoin="round" stroke-dasharray="7 6"/></g>
<g class="dg-l1"><rect class="dg-tint" x="420" y="34" width="110" height="26" rx="13" fill="#e6f0f6" stroke="#8ebcd5" stroke-width="1.2"/><text x="475" y="51.5" font-size="12" class="dg-ink" fill="#0369a1" font-weight="700" text-anchor="middle">10.0.12.0/30</text></g>
<g class="dg-l1"><path class="dg-stroke" d="M700.0 185H750.0" stroke="#0369a1" stroke-width="4" fill="none" stroke-linecap="round" stroke-linejoin="round"/></g>
<g class="dg-l1"><path class="dg-stroke" d="M725.0 179V60" stroke="#0369a1" stroke-width="1.5" fill="none" stroke-linecap="round" stroke-linejoin="round" stroke-dasharray="7 6"/></g>
<g class="dg-l1"><rect class="dg-tint" x="670" y="34" width="110" height="26" rx="13" fill="#e6f0f6" stroke="#8ebcd5" stroke-width="1.2"/><text x="725" y="51.5" font-size="12" class="dg-ink" fill="#0369a1" font-weight="700" text-anchor="middle">10.0.23.0/30</text></g>
<g class="dg-l1"><path class="dg-stroke" d="M950.0 185H1010.0" stroke="#0369a1" stroke-width="4" fill="none" stroke-linecap="round" stroke-linejoin="round"/></g>
<g class="dg-l1"><path class="dg-stroke" d="M980.0 179V60" stroke="#0369a1" stroke-width="1.5" fill="none" stroke-linecap="round" stroke-linejoin="round" stroke-dasharray="7 6"/></g>
<g class="dg-l1"><rect class="dg-tint" x="928.5" y="34" width="103" height="26" rx="13" fill="#e6f0f6" stroke="#8ebcd5" stroke-width="1.2"/><text x="980" y="51.5" font-size="12" class="dg-ink" fill="#0369a1" font-weight="700" text-anchor="middle">10.3.3.0/24</text></g>
<rect class="dg-card" x="20" y="110" width="170" height="150" rx="12" fill="#ffffff" stroke="#cfcfc8"/>
<g class="dg-l7"><g class="dg-icon" transform="translate(36 126) scale(1.2500)" fill="none" stroke="#6d28d9" stroke-width="1.8" stroke-linecap="round" stroke-linejoin="round"><path d="M20 16V7a2 2 0 0 0-2-2H6a2 2 0 0 0-2 2v9m16 0H4m16 0 1.28 2.55a1 1 0 0 1-.9 1.45H3.62a1 1 0 0 1-.9-1.45L4 16"/></g></g>
<g class="dg-l7"><text x="78" y="147" font-size="14" class="dg-ink" fill="#6d28d9" font-weight="700">lab-h1</text></g>
<text x="38" y="178" font-size="12.5" class="dg-muted" fill="#5c5c68" font-weight="500" font-family="JetBrains Mono, ui-monospace, Menlo, Consolas, monospace">eth0</text>
<text x="38" y="198" font-size="12.5" class="dg-text" fill="#17171c" font-weight="500" font-family="JetBrains Mono, ui-monospace, Menlo, Consolas, monospace">10.1.1.10/24</text>
<text x="38" y="218" font-size="13" class="dg-muted" fill="#5c5c68" font-weight="500">host</text>
<rect class="dg-card" x="250" y="110" width="200" height="150" rx="12" fill="#ffffff" stroke="#cfcfc8"/>
<g class="dg-l3"><g class="dg-icon" transform="translate(266 126) scale(1.2500)" fill="none" stroke="#b45309" stroke-width="1.8" stroke-linecap="round" stroke-linejoin="round"><rect width="20" height="8" x="2" y="14" rx="2"/><path d="M6.01 18H6"/><path d="M10.01 18H10"/><path d="M15 10v4"/><path d="M17.84 7.17a4 4 0 0 0-5.66 0"/><path d="M20.66 4.34a8 8 0 0 0-11.31 0"/></g></g>
<g class="dg-l3"><text x="308" y="147" font-size="14" class="dg-ink" fill="#b45309" font-weight="700">lab-r1</text></g>
<text x="268" y="178" font-size="12.5" class="dg-text" fill="#17171c" font-weight="500" font-family="JetBrains Mono, ui-monospace, Menlo, Consolas, monospace">to-h1 10.1.1.1/24</text>
<text x="268" y="198" font-size="12.5" class="dg-text" fill="#17171c" font-weight="500" font-family="JetBrains Mono, ui-monospace, Menlo, Consolas, monospace">to-r2 10.0.12.1/30</text>
<text x="268" y="218" font-size="13" class="dg-muted" fill="#5c5c68" font-weight="500">router · ip_forward=1</text>
<rect class="dg-card" x="500" y="110" width="200" height="150" rx="12" fill="#ffffff" stroke="#cfcfc8"/>
<g class="dg-l3"><g class="dg-icon" transform="translate(516 126) scale(1.2500)" fill="none" stroke="#b45309" stroke-width="1.8" stroke-linecap="round" stroke-linejoin="round"><rect width="20" height="8" x="2" y="14" rx="2"/><path d="M6.01 18H6"/><path d="M10.01 18H10"/><path d="M15 10v4"/><path d="M17.84 7.17a4 4 0 0 0-5.66 0"/><path d="M20.66 4.34a8 8 0 0 0-11.31 0"/></g></g>
<g class="dg-l3"><text x="558" y="147" font-size="14" class="dg-ink" fill="#b45309" font-weight="700">lab-r2</text></g>
<text x="518" y="178" font-size="12.5" class="dg-text" fill="#17171c" font-weight="500" font-family="JetBrains Mono, ui-monospace, Menlo, Consolas, monospace">to-r1 10.0.12.2/30</text>
<text x="518" y="198" font-size="12.5" class="dg-text" fill="#17171c" font-weight="500" font-family="JetBrains Mono, ui-monospace, Menlo, Consolas, monospace">to-r3 10.0.23.1/30</text>
<text x="518" y="218" font-size="13" class="dg-muted" fill="#5c5c68" font-weight="500">router · ip_forward=1</text>
<rect class="dg-card" x="750" y="110" width="200" height="150" rx="12" fill="#ffffff" stroke="#cfcfc8"/>
<g class="dg-l3"><g class="dg-icon" transform="translate(766 126) scale(1.2500)" fill="none" stroke="#b45309" stroke-width="1.8" stroke-linecap="round" stroke-linejoin="round"><rect width="20" height="8" x="2" y="14" rx="2"/><path d="M6.01 18H6"/><path d="M10.01 18H10"/><path d="M15 10v4"/><path d="M17.84 7.17a4 4 0 0 0-5.66 0"/><path d="M20.66 4.34a8 8 0 0 0-11.31 0"/></g></g>
<g class="dg-l3"><text x="808" y="147" font-size="14" class="dg-ink" fill="#b45309" font-weight="700">lab-r3</text></g>
<text x="768" y="178" font-size="12.5" class="dg-text" fill="#17171c" font-weight="500" font-family="JetBrains Mono, ui-monospace, Menlo, Consolas, monospace">to-r2 10.0.23.2/30</text>
<text x="768" y="198" font-size="12.5" class="dg-text" fill="#17171c" font-weight="500" font-family="JetBrains Mono, ui-monospace, Menlo, Consolas, monospace">to-h2 10.3.3.1/24</text>
<text x="768" y="218" font-size="13" class="dg-muted" fill="#5c5c68" font-weight="500">router · ip_forward=1</text>
<rect class="dg-card" x="1010" y="110" width="170" height="150" rx="12" fill="#ffffff" stroke="#cfcfc8"/>
<g class="dg-l7"><g class="dg-icon" transform="translate(1026 126) scale(1.2500)" fill="none" stroke="#6d28d9" stroke-width="1.8" stroke-linecap="round" stroke-linejoin="round"><rect width="20" height="8" x="2" y="2" rx="2"/><rect width="20" height="8" x="2" y="14" rx="2"/><path d="M6 6h.01"/><path d="M6 18h.01"/></g></g>
<g class="dg-l7"><text x="1068" y="147" font-size="14" class="dg-ink" fill="#6d28d9" font-weight="700">lab-h2</text></g>
<text x="1028" y="178" font-size="12.5" class="dg-muted" fill="#5c5c68" font-weight="500" font-family="JetBrains Mono, ui-monospace, Menlo, Consolas, monospace">eth0</text>
<text x="1028" y="198" font-size="12.5" class="dg-text" fill="#17171c" font-weight="500" font-family="JetBrains Mono, ui-monospace, Menlo, Consolas, monospace">10.3.3.10/24</text>
<text x="1028" y="218" font-size="13" class="dg-muted" fill="#5c5c68" font-weight="500">host</text>
<text x="105" y="284" font-size="11.5" class="dg-muted" fill="#5c5c68" font-weight="600" text-anchor="middle" font-family="JetBrains Mono, ui-monospace, Menlo, Consolas, monospace">default via 10.1.1.1</text>
<text x="1095" y="284" font-size="11.5" class="dg-muted" fill="#5c5c68" font-weight="600" text-anchor="middle" font-family="JetBrains Mono, ui-monospace, Menlo, Consolas, monospace">default via 10.3.3.1</text>
</svg></figure>

*Figure 1. The lab topology. Interface names say where each cable goes: `to-r2` on `lab-r1` leads to `lab-r2`.*

| Namespace | Role | Interface → address |
|---|---|---|
| `lab-h1` | Host | `eth0` → `10.1.1.10/24` |
| `lab-r1` | Router | `to-h1` → `10.1.1.1/24`, `to-r2` → `10.0.12.1/30` |
| `lab-r2` | Router | `to-r1` → `10.0.12.2/30`, `to-r3` → `10.0.23.1/30` |
| `lab-r3` | Router | `to-r2` → `10.0.23.2/30`, `to-h2` → `10.3.3.1/24` |
| `lab-h2` | Host | `eth0` → `10.3.3.10/24` |

Two commands run things inside a namespace:
- `sudo ip -n <namespace> <ip command>` runs an `ip` command in it, for example `sudo ip -n lab-r1 route`.
- `sudo ip netns exec <namespace> <command>` runs any other command in it, for example `ping` or `sysctl`.

`<namespace>` is one of the five names in the table.

## Steps

### Part A — Static routing
You build the network, make it work with static and default routes, then break and fix it. This part takes about 50 minutes.

### Step 1 — Check your tools (2 min)
Make sure the commands exist before you build anything, and that no namespaces with these names are left over from an earlier attempt.

```bash
ip -V
traceroute --version
ip netns list
```

Expected output (versions may differ):
```text
ip utility, iproute2-6.17.0, libbpf 1.6.3
Modern traceroute for Linux, version 2.1.6
Copyright (c) 2016  Dmitry Butskoy,   License: GPL v2 or any later
```

`ip netns list` should print nothing. If it lists `lab-h1` or any other `lab-` namespace, run the [Cleanup](#cleanup) first.

> [!TIP]
> If your system language isn't English, some messages appear translated. Their meaning is the same as in the outputs shown here.

### Step 2 — Create the five namespaces (2 min)
Each namespace starts empty, with only a loopback interface that is down. You bring loopback up so that local tools behave normally.

```bash
for ns in lab-h1 lab-r1 lab-r2 lab-r3 lab-h2; do
  sudo ip netns add $ns
  sudo ip -n $ns link set lo up
done
ip netns list
```

Expected output (the order may differ):
```text
lab-h2
lab-r3
lab-r2
lab-r1
lab-h1
```

### Step 3 — Cable the devices and assign addresses (6 min)
Each veth pair is created with one end in each namespace. Then every interface gets its address from the table and is brought up.

```bash
# Cables: one veth pair per link
sudo ip link add eth0  netns lab-h1 type veth peer name to-h1 netns lab-r1
sudo ip link add to-r2 netns lab-r1 type veth peer name to-r1 netns lab-r2
sudo ip link add to-r3 netns lab-r2 type veth peer name to-r2 netns lab-r3
sudo ip link add to-h2 netns lab-r3 type veth peer name eth0  netns lab-h2

# Addresses
sudo ip -n lab-h1 addr add 10.1.1.10/24  dev eth0
sudo ip -n lab-r1 addr add 10.1.1.1/24   dev to-h1
sudo ip -n lab-r1 addr add 10.0.12.1/30  dev to-r2
sudo ip -n lab-r2 addr add 10.0.12.2/30  dev to-r1
sudo ip -n lab-r2 addr add 10.0.23.1/30  dev to-r3
sudo ip -n lab-r3 addr add 10.0.23.2/30  dev to-r2
sudo ip -n lab-r3 addr add 10.3.3.1/24   dev to-h2
sudo ip -n lab-h2 addr add 10.3.3.10/24  dev eth0

# Bring every interface up
sudo ip -n lab-h1 link set eth0 up
sudo ip -n lab-r1 link set to-h1 up
sudo ip -n lab-r1 link set to-r2 up
sudo ip -n lab-r2 link set to-r1 up
sudo ip -n lab-r2 link set to-r3 up
sudo ip -n lab-r3 link set to-r2 up
sudo ip -n lab-r3 link set to-h2 up
sudo ip -n lab-h2 link set eth0 up

sudo ip -n lab-r2 -br addr
```

Expected output (your `@if` numbers and `fe80::` link-local addresses will differ):
```text
lo               UNKNOWN        127.0.0.1/8 ::1/128
to-r1@if3        UP             10.0.12.2/30 fe80::9c1d:9ff:fee0:e853/64
to-r3@if2        UP             10.0.23.1/30 fe80::508d:83ff:fecf:55ce/64
```

Both of `lab-r2`'s interfaces are `UP` with the right addresses.

### Step 4 — Read the connected routes (4 min)
Adding an address creates a connected route for its subnet. That's all each device knows so far.

```bash
sudo ip -n lab-h1 route
sudo ip -n lab-r2 route
```

Expected output:
```text
10.1.1.0/24 dev eth0 proto kernel scope link src 10.1.1.10
10.0.12.0/30 dev to-r1 proto kernel scope link src 10.0.12.2
10.0.23.0/30 dev to-r3 proto kernel scope link src 10.0.23.1
```

`h1` knows only its own LAN, and `r2` knows only its two links. Test a neighbour on the same subnet, then the far host:

```bash
sudo ip netns exec lab-h1 ping -c 2 10.1.1.1
sudo ip netns exec lab-h1 ping -c 2 10.3.3.10
```

Expected output (timings vary):
```text
PING 10.1.1.1 (10.1.1.1) 56(84) bytes of data.
64 bytes from 10.1.1.1: icmp_seq=1 ttl=64 time=0.047 ms
64 bytes from 10.1.1.1: icmp_seq=2 ttl=64 time=0.028 ms

--- 10.1.1.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1035ms
rtt min/avg/max/mdev = 0.028/0.037/0.047/0.009 ms
ping: connect: Network is unreachable
```

The neighbour works: it is on a connected network. For `10.3.3.10`, `h1` has no matching route and no default route, so the kernel refuses to send the packet at all.

### Step 5 — Turn three namespaces into routers (2 min)
A Linux machine only forwards packets for other hosts when IP forwarding is enabled. It is off by default, and each namespace has its own setting.

```bash
sudo ip netns exec lab-r1 sysctl net.ipv4.ip_forward
for r in lab-r1 lab-r2 lab-r3; do
  sudo ip netns exec $r sysctl -w net.ipv4.ip_forward=1
done
```

Expected output:
```text
net.ipv4.ip_forward = 0
net.ipv4.ip_forward = 1
net.ipv4.ip_forward = 1
net.ipv4.ip_forward = 1
```

The first line shows the default, 0. The setting lives inside each namespace, so your host's own forwarding setting doesn't change.

### Step 6 — Give the hosts a default gateway (4 min)
Hosts send everything outside their subnet to their default gateway. Point each host at the router on its LAN.

```bash
sudo ip -n lab-h1 route add default via 10.1.1.1
sudo ip -n lab-h2 route add default via 10.3.3.1
sudo ip netns exec lab-h1 ping -c 2 10.3.3.10
```

Expected output:
```text
PING 10.3.3.10 (10.3.3.10) 56(84) bytes of data.
From 10.1.1.1 icmp_seq=1 Destination Net Unreachable
From 10.1.1.1 icmp_seq=2 Destination Net Unreachable

--- 10.3.3.10 ping statistics ---
2 packets transmitted, 0 received, +2 errors, 100% packet loss, time 1006ms
```

Progress: the packet now leaves `h1` and reaches `r1`. `r1` has no route to `10.3.3.0/24`, so it drops the packet and returns ICMP *Destination Unreachable*. Confirm it from `r1`'s point of view:

```bash
sudo ip -n lab-r1 route get 10.3.3.10
```

Expected output:
```text
RTNETLINK answers: Network is unreachable
```

`ip route get` asks the kernel which route it would use. "Network is unreachable" means no route matches, not even a default.

### Step 7 — Add the forward path (4 min)
Give each router on the way to `10.3.3.0/24` a route to it. `r3` doesn't need one, because `10.3.3.0/24` is connected to it.

```bash
sudo ip -n lab-r1 route add 10.3.3.0/24 via 10.0.12.2
sudo ip -n lab-r2 route add 10.3.3.0/24 via 10.0.23.2
sudo ip netns exec lab-h1 ping -c 2 -W 1 10.3.3.10
```

Expected output:
```text
PING 10.3.3.10 (10.3.3.10) 56(84) bytes of data.

--- 10.3.3.10 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1062ms
```

Still 100% loss, but this time *without* any error message. Something is silently dropping packets. `-W 1` makes `ping` wait only one second for each reply.

### Step 8 — Find and fix the missing return route (8 min)
Follow the path hop by hop with `ip route get`, first toward the destination, then back toward the source.

```bash
sudo ip -n lab-r2 route get 10.3.3.10
sudo ip -n lab-r3 route get 10.3.3.10
sudo ip -n lab-r3 route get 10.1.1.10
```

Expected output:
```text
10.3.3.10 via 10.0.23.2 dev to-r3 src 10.0.23.1 uid 0
    cache
10.3.3.10 dev to-h2 src 10.3.3.1 uid 0
    cache
RTNETLINK answers: Network is unreachable
```

The forward path is complete: `r2` sends to `r3`, and `r3` delivers directly on `to-h2`. The echo requests reach `h2`, and `h2` replies to its gateway `r3`. But `r3` has no route to `10.1.1.10`, so it drops every reply.

<figure class="diagram" style="--dg-w: 1100px"><svg xmlns="http://www.w3.org/2000/svg" class="dg" viewBox="0 0 1100 300" width="1100" height="300" role="img" aria-labelledby="routing-lab-asymmetric-title routing-lab-asymmetric-desc" font-family="Inter, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" xml:lang="en">
<title id="routing-lab-asymmetric-title">A missing return route: requests arrive, replies are dropped</title>
<desc id="routing-lab-asymmetric-desc">The Echo Request from h1 10.1.1.10 to h2 10.3.3.10 passes R1, R2 and R3 because each has a route to 10.3.3.0/24, and it reaches h2. h2 sends the Echo Reply to its gateway R3, but R3 has no route to 10.1.1.0/24, so it drops the reply. Ping reports 100% packet loss even though the request arrived.</desc>
<rect class="dg-bg" x="0.5" y="0.5" width="1099" height="299" rx="16" fill="#ffffff" stroke="#e3e3dd"/>
<path class="dg-line" d="M174 142H266" stroke="#cfcfc8" stroke-width="2" fill="none" stroke-linecap="round" stroke-linejoin="round"/>
<path class="dg-line" d="M394 142H486" stroke="#cfcfc8" stroke-width="2" fill="none" stroke-linecap="round" stroke-linejoin="round"/>
<path class="dg-line" d="M614 142H706" stroke="#cfcfc8" stroke-width="2" fill="none" stroke-linecap="round" stroke-linejoin="round"/>
<path class="dg-line" d="M834 142H926" stroke="#cfcfc8" stroke-width="2" fill="none" stroke-linecap="round" stroke-linejoin="round"/>
<rect class="dg-card" x="46" y="96" width="128" height="92" rx="12" fill="#ffffff" stroke="#cfcfc8"/>
<g class="dg-l7"><g class="dg-icon" transform="translate(95 106) scale(1.2500)" fill="none" stroke="#6d28d9" stroke-width="1.8" stroke-linecap="round" stroke-linejoin="round"><path d="M20 16V7a2 2 0 0 0-2-2H6a2 2 0 0 0-2 2v9m16 0H4m16 0 1.28 2.55a1 1 0 0 1-.9 1.45H3.62a1 1 0 0 1-.9-1.45L4 16"/></g></g>
<text x="110" y="156" font-size="14" class="dg-text" fill="#17171c" font-weight="700" text-anchor="middle">h1</text>
<text x="110" y="175" font-size="11.5" class="dg-muted" fill="#5c5c68" font-weight="500" text-anchor="middle" font-family="JetBrains Mono, ui-monospace, Menlo, Consolas, monospace">10.1.1.10</text>
<rect class="dg-card" x="266" y="96" width="128" height="92" rx="12" fill="#ffffff" stroke="#cfcfc8"/>
<g class="dg-l3"><g class="dg-icon" transform="translate(315 106) scale(1.2500)" fill="none" stroke="#b45309" stroke-width="1.8" stroke-linecap="round" stroke-linejoin="round"><rect width="20" height="8" x="2" y="14" rx="2"/><path d="M6.01 18H6"/><path d="M10.01 18H10"/><path d="M15 10v4"/><path d="M17.84 7.17a4 4 0 0 0-5.66 0"/><path d="M20.66 4.34a8 8 0 0 0-11.31 0"/></g></g>
<text x="330" y="156" font-size="14" class="dg-text" fill="#17171c" font-weight="700" text-anchor="middle">R1</text>
<text x="330" y="175" font-size="11.5" class="dg-muted" fill="#5c5c68" font-weight="500" text-anchor="middle" font-family="JetBrains Mono, ui-monospace, Menlo, Consolas, monospace">10.1.1.1</text>
<rect class="dg-card" x="486" y="96" width="128" height="92" rx="12" fill="#ffffff" stroke="#cfcfc8"/>
<g class="dg-l3"><g class="dg-icon" transform="translate(535 106) scale(1.2500)" fill="none" stroke="#b45309" stroke-width="1.8" stroke-linecap="round" stroke-linejoin="round"><rect width="20" height="8" x="2" y="14" rx="2"/><path d="M6.01 18H6"/><path d="M10.01 18H10"/><path d="M15 10v4"/><path d="M17.84 7.17a4 4 0 0 0-5.66 0"/><path d="M20.66 4.34a8 8 0 0 0-11.31 0"/></g></g>
<text x="550" y="156" font-size="14" class="dg-text" fill="#17171c" font-weight="700" text-anchor="middle">R2</text>
<text x="550" y="175" font-size="11.5" class="dg-muted" fill="#5c5c68" font-weight="500" text-anchor="middle" font-family="JetBrains Mono, ui-monospace, Menlo, Consolas, monospace">10.0.12.2</text>
<rect class="dg-card" x="706" y="96" width="128" height="92" rx="12" fill="#ffffff" stroke="#cfcfc8"/>
<g class="dg-l3"><g class="dg-icon" transform="translate(755 106) scale(1.2500)" fill="none" stroke="#b45309" stroke-width="1.8" stroke-linecap="round" stroke-linejoin="round"><rect width="20" height="8" x="2" y="14" rx="2"/><path d="M6.01 18H6"/><path d="M10.01 18H10"/><path d="M15 10v4"/><path d="M17.84 7.17a4 4 0 0 0-5.66 0"/><path d="M20.66 4.34a8 8 0 0 0-11.31 0"/></g></g>
<text x="770" y="156" font-size="14" class="dg-text" fill="#17171c" font-weight="700" text-anchor="middle">R3</text>
<text x="770" y="175" font-size="11.5" class="dg-muted" fill="#5c5c68" font-weight="500" text-anchor="middle" font-family="JetBrains Mono, ui-monospace, Menlo, Consolas, monospace">10.0.23.2</text>
<rect class="dg-card" x="926" y="96" width="128" height="92" rx="12" fill="#ffffff" stroke="#cfcfc8"/>
<g class="dg-l7"><g class="dg-icon" transform="translate(975 106) scale(1.2500)" fill="none" stroke="#6d28d9" stroke-width="1.8" stroke-linecap="round" stroke-linejoin="round"><rect width="20" height="8" x="2" y="2" rx="2"/><rect width="20" height="8" x="2" y="14" rx="2"/><path d="M6 6h.01"/><path d="M6 18h.01"/></g></g>
<text x="990" y="156" font-size="14" class="dg-text" fill="#17171c" font-weight="700" text-anchor="middle">h2</text>
<text x="990" y="175" font-size="11.5" class="dg-muted" fill="#5c5c68" font-weight="500" text-anchor="middle" font-family="JetBrains Mono, ui-monospace, Menlo, Consolas, monospace">10.3.3.10</text>
<g class="dg-ok"><path class="dg-stroke" d="M110 92V66H990V88" stroke="#15803d" stroke-width="2.5" fill="none" stroke-linecap="round" stroke-linejoin="round"/></g>
<g class="dg-ok"><path class="dg-stroke" d="M984 81l6 8 6-8" stroke="#15803d" stroke-width="2.5" fill="none" stroke-linecap="round" stroke-linejoin="round"/></g>
<g class="dg-ok"><rect class="dg-bg-plain" x="373" y="54" width="354" height="24" rx="12" fill="#ffffff"/><text x="550" y="70.5" font-size="12" class="dg-ink" fill="#15803d" font-weight="600" text-anchor="middle">Echo Request: every hop has a route to 10.3.3.0/24</text></g>
<g class="dg-warn"><path class="dg-stroke" d="M990 192V222H784" stroke="#b91c1c" stroke-width="2.5" fill="none" stroke-linecap="round" stroke-linejoin="round" stroke-dasharray="7 6"/></g>
<g class="dg-warn"><path class="dg-stroke" d="M791 216l-8 6 8 6" stroke="#b91c1c" stroke-width="2.5" fill="none" stroke-linecap="round" stroke-linejoin="round"/></g>
<g class="dg-warn"><rect class="dg-bg-plain" x="796.94" y="210" width="166.12" height="24" rx="12" fill="#ffffff"/><text x="880" y="226.5" font-size="11.5" class="dg-ink" fill="#b91c1c" font-weight="600" text-anchor="middle">Echo Reply to 10.1.1.10</text></g>
<g class="dg-warn"><circle class="dg-solid" cx="770" cy="222" r="11" fill="#b91c1c"/><text x="770" y="226.5" font-size="13" class="dg-on" fill="#ffffff" font-weight="700" text-anchor="middle">✕</text></g>
<g class="dg-warn"><rect class="dg-tint" x="605" y="244" width="290" height="26" rx="13" fill="#f8e8e8" stroke="#e09999" stroke-width="1.2"/><text x="750" y="261.5" font-size="12" class="dg-ink" fill="#b91c1c" font-weight="700" text-anchor="middle">R3: no route to 10.1.1.0/24 → dropped</text></g>
</svg></figure>

*Figure 2. Requests arrive, replies die at the first router without a route back. From `h1`, it just looks like silence.*

`r3` is a stub router with only one way out, so a default route toward `r2` is the right fix:

```bash
sudo ip -n lab-r3 route add default via 10.0.23.1
sudo ip netns exec lab-h1 ping -c 2 -W 1 10.3.3.10
```

Expected output:
```text
PING 10.3.3.10 (10.3.3.10) 56(84) bytes of data.

--- 10.3.3.10 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1039ms
```

Still failing. Keep following the return path: the reply now reaches `r2`.

```bash
sudo ip -n lab-r2 route get 10.1.1.10
```

Expected output:
```text
RTNETLINK answers: Network is unreachable
```

`r2` sits in the middle, so it needs a specific route back to `h1`'s LAN:

```bash
sudo ip -n lab-r2 route add 10.1.1.0/24 via 10.0.12.1
sudo ip netns exec lab-h1 ping -c 3 10.3.3.10
```

Expected output (timings vary):
```text
PING 10.3.3.10 (10.3.3.10) 56(84) bytes of data.
64 bytes from 10.3.3.10: icmp_seq=1 ttl=61 time=0.068 ms
64 bytes from 10.3.3.10: icmp_seq=2 ttl=61 time=0.052 ms
64 bytes from 10.3.3.10: icmp_seq=3 ttl=61 time=0.054 ms

--- 10.3.3.10 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2064ms
rtt min/avg/max/mdev = 0.052/0.058/0.068/0.007 ms
```

It works. `ttl=61` confirms the path: `h2` sent the reply with TTL 64, and three routers each decremented it once. `r1` already knows `10.1.1.0/24`, because it's connected.

### Step 9 — Trace the path and explain a silent hop (7 min)
Traceroute shows each router on the forward path.

```bash
sudo ip netns exec lab-h1 traceroute -n 10.3.3.10
```

Expected output:
```text
traceroute to 10.3.3.10 (10.3.3.10), 30 hops max, 60 byte packets
 1  10.1.1.1  0.022 ms  0.005 ms  0.005 ms
 2  10.0.12.2  0.017 ms  0.006 ms  0.007 ms
 3  * * *
 4  10.3.3.10  0.013 ms  0.009 ms  0.009 ms
```

`-n` shows addresses without trying to resolve names. Hop 4, the destination, answers, so hop 3 (`r3`) does forward traffic. Why doesn't `r3` answer for itself? Its *Time Exceeded* messages come from its address on the link to `r2`, `10.0.23.2`. Check whether `r1` knows the way back to that address:

```bash
sudo ip -n lab-r1 route get 10.0.23.2
sudo ip netns exec lab-r1 sysctl net.ipv4.conf.to-r2.rp_filter
```

Expected output:
```text
RTNETLINK answers: Network is unreachable
net.ipv4.conf.to-r2.rp_filter = 2
```

`r1` has routes for `10.1.1.0/24`, `10.0.12.0/30` and `10.3.3.0/24`, but not for the link `10.0.23.0/30`. Its **reverse path filter** (`rp_filter`, the Linux form of uRPF from the lesson) checks the source address of every incoming packet. In loose mode (`2`), it drops packets from a source the routing table can't reach at all. So `r3`'s messages die at `r1`.

> [!NOTE]
> Most distributions set `rp_filter` to 2 (loose) by default, and new namespaces inherit it. If yours shows `0` (off), `r1` forwards `r3`'s messages anyway. They then reach `h1` and hop 3 already shows `10.0.23.2`. Read the explanation anyway and do the fix: a router without a route back to part of the network is a real problem, even if a filter doesn't reveal it.

Like `r3`, `r1` is a stub router. Give it a default route toward `r2`, as in the lesson's design:

```bash
sudo ip -n lab-r1 route add default via 10.0.12.2
sudo ip netns exec lab-h1 traceroute -n 10.3.3.10
```

Expected output:
```text
traceroute to 10.3.3.10 (10.3.3.10), 30 hops max, 60 byte packets
 1  10.1.1.1  0.040 ms  0.006 ms  0.004 ms
 2  10.0.12.2  0.015 ms  0.008 ms  0.006 ms
 3  10.0.23.2  0.018 ms  0.008 ms  0.007 ms
 4  10.3.3.10  0.019 ms  0.010 ms *
```

All four hops answer now. A single `*` among the answers (like the last one here) is normal: Linux rate-limits the ICMP errors it sends (`net.ipv4.icmp_ratelimit`), so if you run traceroute several times in quick succession, some probes go unanswered. Wait a couple of seconds between runs.

### Step 10 — Watch longest prefix match decide (6 min)
`r1` now has both a `/24` route and a default route that point the same way. Add a third, more specific route to see that the longest match wins, whatever order the routes were added in.

```bash
sudo ip -n lab-r1 route
sudo ip -n lab-r1 route get 10.3.3.10
sudo ip -n lab-r1 route get 198.51.100.7
```

Expected output:
```text
default via 10.0.12.2 dev to-r2
10.0.12.0/30 dev to-r2 proto kernel scope link src 10.0.12.1
10.1.1.0/24 dev to-h1 proto kernel scope link src 10.1.1.1
10.3.3.0/24 via 10.0.12.2 dev to-r2
10.3.3.10 via 10.0.12.2 dev to-r2 src 10.0.12.1 uid 0
    cache
198.51.100.7 via 10.0.12.2 dev to-r2 src 10.0.12.1 uid 0
    cache
```

`10.3.3.10` matches the `/24` and `198.51.100.7` (a documentation address) matches only the default route. Both go to `10.0.12.2`, so you can't tell them apart yet. Now add a **prohibit** route for exactly one address. It's a route that rejects packets instead of forwarding them, which makes the winner visible:

```bash
sudo ip -n lab-r1 route add prohibit 10.3.3.10/32
sudo ip -n lab-r1 route get 10.3.3.10
sudo ip -n lab-r1 route get 10.3.3.20
sudo ip netns exec lab-h1 ping -c 2 10.3.3.10
```

Expected output:
```text
RTNETLINK answers: Permission denied
10.3.3.20 via 10.0.12.2 dev to-r2 src 10.0.12.1 uid 0
    cache
PING 10.3.3.10 (10.3.3.10) 56(84) bytes of data.
From 10.1.1.1 icmp_seq=1 Packet filtered
From 10.1.1.1 icmp_seq=2 Packet filtered

--- 10.3.3.10 ping statistics ---
2 packets transmitted, 0 received, +2 errors, 100% packet loss, time 1061ms
```

For `10.3.3.10`, the `/32` beats the `/24` and the default. The kernel rejects the packet and `r1` sends ICMP *administratively prohibited*, which `ping` shows as `Packet filtered`. Its neighbour `10.3.3.20` only matches the `/24`, so it still goes to `r2`. Remove the test route:

```bash
sudo ip -n lab-r1 route del prohibit 10.3.3.10/32
```

There is no output when it succeeds.

### Step 11 — Break a forward route and locate the fault (5 min)
A colleague "cleans up" `r2` and deletes its route to `10.3.3.0/24`. Play the colleague, then troubleshoot as if you didn't know.

```bash
sudo ip -n lab-r2 route del 10.3.3.0/24
sudo ip netns exec lab-h1 ping -c 2 10.3.3.10
```

Expected output:
```text
PING 10.3.3.10 (10.3.3.10) 56(84) bytes of data.
From 10.0.12.2 icmp_seq=1 Destination Net Unreachable
From 10.0.12.2 icmp_seq=2 Destination Net Unreachable

--- 10.3.3.10 ping statistics ---
2 packets transmitted, 0 received, +2 errors, 100% packet loss, time 1064ms
```

The error names the culprit: `10.0.12.2` is `r2`. Traceroute shows the same thing:

```bash
sudo ip netns exec lab-h1 traceroute -n 10.3.3.10
```

Expected output:
```text
traceroute to 10.3.3.10 (10.3.3.10), 30 hops max, 60 byte packets
 1  10.1.1.1  0.026 ms  0.005 ms  0.004 ms
 2  10.0.12.2  0.010 ms !N  0.006 ms !N *
```

`!N` means *network unreachable*: hop 2 answered that it has no route. Compare this with Step 7: a missing *forward* route usually produces an error from the router that lacks it, while a missing *return* route produces silence. Confirm on `r2` and fix it:

```bash
sudo ip -n lab-r2 route get 10.3.3.10
sudo ip -n lab-r2 route add 10.3.3.0/24 via 10.0.23.2
sudo ip netns exec lab-h1 ping -c 2 10.3.3.10
```

Expected output:
```text
RTNETLINK answers: Network is unreachable
PING 10.3.3.10 (10.3.3.10) 56(84) bytes of data.
64 bytes from 10.3.3.10: icmp_seq=1 ttl=61 time=0.073 ms
64 bytes from 10.3.3.10: icmp_seq=2 ttl=61 time=0.045 ms

--- 10.3.3.10 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1056ms
rtt min/avg/max/mdev = 0.045/0.059/0.073/0.014 ms
```

> [!NOTE]
> The outputs in Part A come from a real run with iproute2 6.17, iputils 20250605 and traceroute 2.1.6 on Linux 7.2. It was run in an unprivileged user and network namespace (`unshare -rn`) instead of with `sudo`, which doesn't change the routing behaviour. Other versions may word messages slightly differently.

### Part B (optional) — Replace static routes with OSPF
In Part A you typed seven routes and tracked down three faults by hand. Here, three routers learn the same networks by themselves with OSPF. You run **FRRouting** (FRR), an open-source routing suite, in three containers. This part is optional and takes about 25 minutes.

<figure class="diagram" style="--dg-w: 1160px"><svg xmlns="http://www.w3.org/2000/svg" class="dg" viewBox="0 0 1160 340" width="1160" height="340" role="img" aria-labelledby="routing-lab-ospf-title routing-lab-ospf-desc" font-family="Inter, system-ui, -apple-system, Segoe UI, Roboto, sans-serif" xml:lang="en">
<title id="routing-lab-ospf-title">Optional part: three FRRouting containers running OSPF</title>
<desc id="routing-lab-ospf-desc">Three containers, lab-ospf-r1, lab-ospf-r2 and lab-ospf-r3, run FRRouting with OSPF in area 0 and router IDs 10.255.0.1, 10.255.0.2 and 10.255.0.3. Container networks: ospf-lan1 10.1.1.0/24 with r1 at .11, ospf-r1r2 10.0.12.0/24 with r1 .11 and r2 .12, ospf-r2r3 10.0.23.0/24 with r2 .12 and r3 .13, and ospf-lan3 10.3.3.0/24 with r3 at .13. OSPF hellos are exchanged on the two router-to-router networks.</desc>
<rect class="dg-bg" x="0.5" y="0.5" width="1159" height="339" rx="16" fill="#ffffff" stroke="#e3e3dd"/>
<g class="dg-l2"><rect class="dg-tint" x="20" y="20" width="1120" height="300" rx="16" fill="#f1f7f3" stroke="#a1ccb1" stroke-width="1.2" stroke-dasharray="6 5"/>
<g class="dg-icon" transform="translate(36 33) scale(0.7500)" fill="none" stroke="#15803d" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M21 8a2 2 0 0 0-1-1.73l-7-4a2 2 0 0 0-2 0l-7 4A2 2 0 0 0 3 8v8a2 2 0 0 0 1 1.73l7 4a2 2 0 0 0 2 0l7-4A2 2 0 0 0 21 16Z"/><path d="m3.3 7 8.7 5 8.7-5"/><path d="M12 22V12"/></g>
<text x="62" y="47" font-size="13" class="dg-ink" fill="#15803d" font-weight="700">FRRouting containers, OSPF area 0</text></g>
<g class="dg-l1"><path class="dg-stroke" d="M420 186H490" stroke="#0369a1" stroke-width="4" fill="none" stroke-linecap="round" stroke-linejoin="round"/></g>
<g class="dg-l1"><path class="dg-stroke" d="M670 186H740" stroke="#0369a1" stroke-width="4" fill="none" stroke-linecap="round" stroke-linejoin="round"/></g>
<g class="dg-l1"><path class="dg-stroke" d="M60 186H240" stroke="#0369a1" stroke-width="4" fill="none" stroke-linecap="round" stroke-linejoin="round"/></g>
<g class="dg-l1"><path class="dg-stroke" d="M920 186H1100" stroke="#0369a1" stroke-width="4" fill="none" stroke-linecap="round" stroke-linejoin="round"/></g>
<rect class="dg-card" x="240" y="120" width="180" height="140" rx="12" fill="#ffffff" stroke="#cfcfc8"/>
<g class="dg-l2"><g class="dg-icon" transform="translate(256 136) scale(1.2500)" fill="none" stroke="#15803d" stroke-width="1.8" stroke-linecap="round" stroke-linejoin="round"><rect width="20" height="8" x="2" y="14" rx="2"/><path d="M6.01 18H6"/><path d="M10.01 18H10"/><path d="M15 10v4"/><path d="M17.84 7.17a4 4 0 0 0-5.66 0"/><path d="M20.66 4.34a8 8 0 0 0-11.31 0"/></g></g>
<g class="dg-l2"><text x="298" y="157" font-size="13" class="dg-ink" fill="#15803d" font-weight="700">lab-ospf-r1</text></g>
<text x="258" y="188" font-size="12.5" class="dg-text" fill="#17171c" font-weight="500" font-family="JetBrains Mono, ui-monospace, Menlo, Consolas, monospace">RID 10.255.0.1</text>
<text x="258" y="208" font-size="12.5" class="dg-text" fill="#17171c" font-weight="500" font-family="JetBrains Mono, ui-monospace, Menlo, Consolas, monospace">eth0 10.1.1.11</text>
<text x="258" y="228" font-size="12.5" class="dg-text" fill="#17171c" font-weight="500" font-family="JetBrains Mono, ui-monospace, Menlo, Consolas, monospace">eth1 10.0.12.11</text>
<rect class="dg-card" x="490" y="120" width="180" height="140" rx="12" fill="#ffffff" stroke="#cfcfc8"/>
<g class="dg-l2"><g class="dg-icon" transform="translate(506 136) scale(1.2500)" fill="none" stroke="#15803d" stroke-width="1.8" stroke-linecap="round" stroke-linejoin="round"><rect width="20" height="8" x="2" y="14" rx="2"/><path d="M6.01 18H6"/><path d="M10.01 18H10"/><path d="M15 10v4"/><path d="M17.84 7.17a4 4 0 0 0-5.66 0"/><path d="M20.66 4.34a8 8 0 0 0-11.31 0"/></g></g>
<g class="dg-l2"><text x="548" y="157" font-size="13" class="dg-ink" fill="#15803d" font-weight="700">lab-ospf-r2</text></g>
<text x="508" y="188" font-size="12.5" class="dg-text" fill="#17171c" font-weight="500" font-family="JetBrains Mono, ui-monospace, Menlo, Consolas, monospace">RID 10.255.0.2</text>
<text x="508" y="208" font-size="12.5" class="dg-text" fill="#17171c" font-weight="500" font-family="JetBrains Mono, ui-monospace, Menlo, Consolas, monospace">eth0 10.0.12.12</text>
<text x="508" y="228" font-size="12.5" class="dg-text" fill="#17171c" font-weight="500" font-family="JetBrains Mono, ui-monospace, Menlo, Consolas, monospace">eth1 10.0.23.12</text>
<rect class="dg-card" x="740" y="120" width="180" height="140" rx="12" fill="#ffffff" stroke="#cfcfc8"/>
<g class="dg-l2"><g class="dg-icon" transform="translate(756 136) scale(1.2500)" fill="none" stroke="#15803d" stroke-width="1.8" stroke-linecap="round" stroke-linejoin="round"><rect width="20" height="8" x="2" y="14" rx="2"/><path d="M6.01 18H6"/><path d="M10.01 18H10"/><path d="M15 10v4"/><path d="M17.84 7.17a4 4 0 0 0-5.66 0"/><path d="M20.66 4.34a8 8 0 0 0-11.31 0"/></g></g>
<g class="dg-l2"><text x="798" y="157" font-size="13" class="dg-ink" fill="#15803d" font-weight="700">lab-ospf-r3</text></g>
<text x="758" y="188" font-size="12.5" class="dg-text" fill="#17171c" font-weight="500" font-family="JetBrains Mono, ui-monospace, Menlo, Consolas, monospace">RID 10.255.0.3</text>
<text x="758" y="208" font-size="12.5" class="dg-text" fill="#17171c" font-weight="500" font-family="JetBrains Mono, ui-monospace, Menlo, Consolas, monospace">eth0 10.3.3.13</text>
<text x="758" y="228" font-size="12.5" class="dg-text" fill="#17171c" font-weight="500" font-family="JetBrains Mono, ui-monospace, Menlo, Consolas, monospace">eth1 10.0.23.13</text>
<g class="dg-l1"><rect class="dg-tint" x="95.5" y="60" width="89" height="26" rx="13" fill="#e6f0f6" stroke="#8ebcd5" stroke-width="1.2"/><text x="140" y="77.5" font-size="12" class="dg-ink" fill="#0369a1" font-weight="700" text-anchor="middle">ospf-lan1</text></g>
<text x="140" y="106" font-size="12" class="dg-ink" fill="#0369a1" font-weight="700" text-anchor="middle" font-family="JetBrains Mono, ui-monospace, Menlo, Consolas, monospace">10.1.1.0/24</text>
<g class="dg-l1"><rect class="dg-tint" x="410.5" y="60" width="89" height="26" rx="13" fill="#e6f0f6" stroke="#8ebcd5" stroke-width="1.2"/><text x="455" y="77.5" font-size="12" class="dg-ink" fill="#0369a1" font-weight="700" text-anchor="middle">ospf-r1r2</text></g>
<text x="455" y="106" font-size="12" class="dg-ink" fill="#0369a1" font-weight="700" text-anchor="middle" font-family="JetBrains Mono, ui-monospace, Menlo, Consolas, monospace">10.0.12.0/24</text>
<g class="dg-l1"><rect class="dg-tint" x="660.5" y="60" width="89" height="26" rx="13" fill="#e6f0f6" stroke="#8ebcd5" stroke-width="1.2"/><text x="705" y="77.5" font-size="12" class="dg-ink" fill="#0369a1" font-weight="700" text-anchor="middle">ospf-r2r3</text></g>
<text x="705" y="106" font-size="12" class="dg-ink" fill="#0369a1" font-weight="700" text-anchor="middle" font-family="JetBrains Mono, ui-monospace, Menlo, Consolas, monospace">10.0.23.0/24</text>
<g class="dg-l1"><rect class="dg-tint" x="975.5" y="60" width="89" height="26" rx="13" fill="#e6f0f6" stroke="#8ebcd5" stroke-width="1.2"/><text x="1020" y="77.5" font-size="12" class="dg-ink" fill="#0369a1" font-weight="700" text-anchor="middle">ospf-lan3</text></g>
<text x="1020" y="106" font-size="12" class="dg-ink" fill="#0369a1" font-weight="700" text-anchor="middle" font-family="JetBrains Mono, ui-monospace, Menlo, Consolas, monospace">10.3.3.0/24</text>
<g class="dg-l2"><rect class="dg-tint" x="359" y="278" width="442" height="26" rx="13" fill="#e8f2ec" stroke="#96c6a8" stroke-width="1.2"/><text x="580" y="295.5" font-size="12" class="dg-ink" fill="#15803d" font-weight="700" text-anchor="middle">OSPF hellos every 10 s to 224.0.0.5 on each OSPF interface</text></g>
</svg></figure>

*Figure 3. The same line of three routers, now as containers. The container networks reuse the Part A prefixes, but the links are /24s and the routers use host numbers 11, 12 and 13.*

> [!NOTE]
> Container engines keep the first address of each network (`.1`) for their own bridge, so the routers use `.11`, `.12` and `.13`. The commands were tested with rootless Podman 5.8 and the image `quay.io/frrouting/frr:10.7.1`. Docker uses the same commands, but they weren't tested with Docker for this lab. With Docker, you may need `sudo` or membership in the `docker` group.

### Step 12 — Prepare the FRR configuration (5 min)
Each router needs three files in its own folder: `daemons` (which FRR daemons to start), `vtysh.conf` (settings of FRR's command shell) and `frr.conf` (the routing configuration). First choose your container engine and fetch the default `daemons` file from the image:

```bash
CT=podman          # or: CT=docker
IMG=quay.io/frrouting/frr:10.7.1
mkdir -p ~/ospf-lab && cd ~/ospf-lab
$CT run --rm --entrypoint cat $IMG /etc/frr/daemons > daemons.orig
for n in 1 2 3; do
  mkdir -p r$n
  sed 's/^ospfd=no/ospfd=yes/' daemons.orig > r$n/daemons
  echo 'service integrated-vtysh-config' > r$n/vtysh.conf
done
grep '^ospfd' r1/daemons
```

Expected output (after the image download messages):
```text
ospfd=yes
ospfd_options="  -A 127.0.0.1"
```

`CT` holds the container engine and `IMG` the exact image. The `sed` command enables the OSPF daemon, which the image ships disabled. Now write each router's OSPF configuration. A `network` statement enables OSPF in area 0 on every interface whose address falls in that prefix, and advertises the prefix.

```bash
cat > r1/frr.conf <<'EOF'
frr defaults traditional
hostname r1
!
router ospf
 ospf router-id 10.255.0.1
 network 10.1.1.0/24 area 0
 network 10.0.12.0/24 area 0
!
EOF
cat > r2/frr.conf <<'EOF'
frr defaults traditional
hostname r2
!
router ospf
 ospf router-id 10.255.0.2
 network 10.0.12.0/24 area 0
 network 10.0.23.0/24 area 0
!
EOF
cat > r3/frr.conf <<'EOF'
frr defaults traditional
hostname r3
!
router ospf
 ospf router-id 10.255.0.3
 network 10.0.23.0/24 area 0
 network 10.3.3.0/24 area 0
!
EOF
```

There is no output. The **router ID** is a 32-bit number, written like an IPv4 address, that names each router in the LSDB. It must be unique.

### Step 13 — Create the networks and start the routers (4 min)
Each link becomes an internal container network (no Internet access). Each router starts on one network and is then connected to its second one.

```bash
$CT network create --internal --subnet 10.1.1.0/24  ospf-lan1
$CT network create --internal --subnet 10.0.12.0/24 ospf-r1r2
$CT network create --internal --subnet 10.0.23.0/24 ospf-r2r3
$CT network create --internal --subnet 10.3.3.0/24  ospf-lan3

for spec in "r1 ospf-lan1 10.1.1.11" "r2 ospf-r1r2 10.0.12.12" "r3 ospf-lan3 10.3.3.13"; do
  set -- $spec
  $CT run -d --name lab-ospf-$1 --hostname $1 \
    --network $2 --ip $3 \
    --cap-add NET_ADMIN --cap-add NET_RAW --cap-add SYS_ADMIN \
    --sysctl net.ipv4.ip_forward=1 \
    -v "$PWD/$1:/etc/frr:Z" $IMG
done

$CT network connect --ip 10.0.12.11 ospf-r1r2 lab-ospf-r1
$CT network connect --ip 10.0.23.12 ospf-r2r3 lab-ospf-r2
$CT network connect --ip 10.0.23.13 ospf-r2r3 lab-ospf-r3
$CT ps --filter name=lab-ospf --format '{{.Names}} {{.Status}}'
```

Expected output (the network names, then three long container IDs, then):
```text
lab-ospf-r1 Up Less than a second
lab-ospf-r2 Up Less than a second
lab-ospf-r3 Up Less than a second
```

What the options do:
- `--cap-add NET_ADMIN` lets FRR install routes in the container's routing table, `NET_RAW` lets it send OSPF packets (IP protocol 89). `SYS_ADMIN` is also required: FRR's daemons ask for it when they start (it is used for network namespace support), and without it they stop with `privs_init: initial cap_set_proc failed: Operation not permitted`.

> [!WARNING]
> `SYS_ADMIN` is the broadest Linux capability. Give it only to these lab containers, which sit on internal networks with no Internet access, and prefer rootless Podman: there the capability applies inside your user namespace, not to the host. Never reuse these options for containers that run untrusted software.
- `--sysctl net.ipv4.ip_forward=1` turns each container into a router, as in Step 5.
- `-v "$PWD/$1:/etc/frr:Z"` mounts the router's configuration folder. `:Z` relabels it for SELinux, where that is enabled.

### Step 14 — Watch the neighbours come up (4 min)
OSPF routers find each other with hellos every 10 seconds. On an Ethernet-like network, they also wait up to 40 seconds to elect a designated router before they synchronize. Wait about a minute, then look at `r2`, which has two neighbours:

```bash
sleep 60
$CT exec lab-ospf-r2 vtysh -c 'show ip ospf neighbor'
```

Expected output (times vary):
```text
Neighbor ID     Pri State           Up Time         Dead Time Address         Interface                        RXmtL RqstL DBsmL
10.255.0.1        1 Full/Backup     10.206s           39.735s 10.0.12.11      eth0:10.0.12.12                      0     0     0
10.255.0.3        1 Full/DR         10.108s           39.894s 10.0.23.13      eth1:10.0.23.12                      0     0     0
```

Both neighbours are **Full**: their databases are synchronized. `DR` and `Backup` show which router was elected designated router and backup on each link. If you look too early, you'll see states such as `Init` or `2-Way`.

### Step 15 — Read the routes OSPF built (5 min)
Look at `r1`'s routing table in FRR, then in the Linux kernel:

```bash
$CT exec lab-ospf-r1 vtysh -c 'show ip route'
$CT exec lab-ospf-r1 ip route
```

Expected output (after a legend of route codes; times vary):
```text
IPv4 unicast VRF default:
C>* 10.0.12.0/24 is directly connected, eth1, weight 1, 00:00:50
L>* 10.0.12.11/32 is directly connected, eth1, weight 1, 00:00:50
O>* 10.0.23.0/24 [110/20] via 10.0.12.12, eth1, weight 1, 00:00:05
C>* 10.1.1.0/24 is directly connected, eth0, weight 1, 00:00:50
L>* 10.1.1.11/32 is directly connected, eth0, weight 1, 00:00:50
O>* 10.3.3.0/24 [110/30] via 10.0.12.12, eth1, weight 1, 00:00:05
10.0.12.0/24 dev eth1 proto kernel scope link src 10.0.12.11
10.0.23.0/24 nhid 10 via 10.0.12.12 dev eth1 proto ospf metric 20
10.1.1.0/24 dev eth0 proto kernel scope link src 10.1.1.11
10.3.3.0/24 nhid 10 via 10.0.12.12 dev eth1 proto ospf metric 20
```

How to read it:
- `O` marks routes learned by OSPF, `C` connected routes and `L` the router's own addresses. `>*` means "selected and installed".
- `[110/30]` is *[administrative distance/metric]*. FRR gives OSPF a distance of 110, like Cisco IOS. The cost to `10.3.3.0/24` is 30: three interfaces with a cost of 10 each (`r1`→`r2`, `r2`→`r3` and `r3`'s LAN).
- In the kernel table, the same routes appear with `proto ospf`. The Linux kernel has no administrative distance, so FRR chooses the best route itself and installs only that one.

You never typed a route, but `r1` knows `10.3.3.0/24` and the link `10.0.23.0/24`, so the return-path and silent-hop problems of Part A can't happen here.

### Step 16 — Fail a link and watch OSPF react (4 min)
Disconnect `r2` from the `r2`–`r3` link, as if someone unplugged the cable:

```bash
$CT network disconnect ospf-r2r3 lab-ospf-r2
sleep 3
$CT exec lab-ospf-r1 vtysh -c 'show ip route ospf'
$CT exec lab-ospf-r1 ping -c 2 -W 1 10.3.3.13
```

Expected output:
```text
PING 10.3.3.13 (10.3.3.13): 56 data bytes
ping: sendto: Network unreachable
```

`show ip route ospf` prints nothing: within seconds, `r2` flooded the change and `r1` withdrew both routes behind the broken link. The `ping` inside the container is BusyBox's, which words its error differently from Part A. Reconnect the link and wait for the adjacency:

```bash
$CT network connect --ip 10.0.23.12 ospf-r2r3 lab-ospf-r2
sleep 60
$CT exec lab-ospf-r1 vtysh -c 'show ip route ospf'
```

Expected output (after the legend; times vary):
```text
IPv4 unicast VRF default:
O>* 10.0.23.0/24 [110/20] via 10.0.12.12, eth1, weight 1, 00:00:50
O>* 10.3.3.0/24 [110/30] via 10.0.12.12, eth1, weight 1, 00:00:39
```

The routes are back without anyone typing them. With only one path, OSPF can't route around the failure, but it removes dead routes and restores them automatically. With a redundant link, it would switch to the other path. That's the challenge below.

> [!NOTE]
> Part B outputs come from a real run with rootless Podman 5.8.7 and FRR 10.7.1 (`quay.io/frrouting/frr:10.7.1`, about 94 MB compressed). It was not tested with Docker.

## Verification
You have completed the lab when:

1. Both hosts reach each other through three routers:
   ```bash
   sudo ip netns exec lab-h1 ping -c 2 10.3.3.10
   sudo ip netns exec lab-h2 ping -c 2 10.1.1.10
   ```
   Both show `0% packet loss` and `ttl=61`.
2. The reverse trace lists every router:
   ```bash
   sudo ip netns exec lab-h2 traceroute -n 10.1.1.10
   ```
   Expected output (timings vary):
   ```text
   traceroute to 10.1.1.10 (10.1.1.10), 30 hops max, 60 byte packets
    1  10.3.3.1  0.042 ms  0.006 ms  0.005 ms
    2  10.0.23.1  0.018 ms  0.008 ms  0.006 ms
    3  10.0.12.1  0.017 ms  0.008 ms  0.007 ms
    4  10.1.1.10  0.017 ms  0.010 ms  0.009 ms
   ```
3. You can explain, for each of these symptoms, which route was missing and on which router: `Destination Net Unreachable` from `10.1.1.1` (Step 6), silent 100% loss (Steps 7–8), `* * *` on hop 3 (Step 9) and `!N` on hop 2 (Step 11).
4. (Optional Part B) `show ip ospf neighbor` on `lab-ospf-r2` shows two neighbours in state `Full`, and `r1` has `O` routes to `10.0.23.0/24` and `10.3.3.0/24`.

## Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| `RTNETLINK answers: Operation not permitted` | You ran a command that changes the network without `sudo`. | Prefix the command with `sudo`. |
| `Cannot open network namespace "lab-r1": No such file or directory` | The namespace doesn't exist: a step was skipped, there's a typo, or cleanup already ran. | Run `ip netns list` and create what's missing (Step 2). |
| `RTNETLINK answers: File exists` | The route, address or interface already exists, for example from running a step twice. | Check with `ip -n <namespace> route`. Use `ip route replace` to change a route, or delete it first. |
| `Error: Nexthop has invalid gateway.` | The next hop isn't on a connected subnet of that router, often a typo or the wrong router. | Use the neighbour's address on the shared link, such as `10.0.12.2` from `r1`. |
| Ping to a router's neighbour works, but beyond it there's 100% loss with no error, and traceroute shows `* * *` from the next hop on | IP forwarding is off on that router (`sysctl net.ipv4.ip_forward` shows 0). It drops the packets without any ICMP message. | Run Step 5 again for that router. |
| 100% loss with no error, forwarding is on | A return route is missing somewhere on the way back. | Run `ip route get <source address>` on each router from the destination back (Step 8). |
| `From <router> Destination Net Unreachable` or `!N` in traceroute | That router has no route to the destination. | Add the forward route on that router (Steps 7 and 11). |
| `traceroute: command not found` | The `traceroute` package isn't installed. | Install it (see Prerequisites), or use `tracepath -n`, which is part of iputils. |
| (Part B) `Error: ... subnet ... is already used` or `Pool overlaps with other one on this address space` | Another container network or a host route already uses one of the subnets. | Remove the conflicting network, or change the prefixes consistently in the network commands and `frr.conf` files. |
| (Part B) Neighbours stuck in `Init` or `2-Way`, or no `O` routes | You looked before the 40-second wait ended, or the configuration has an error. | Wait a minute and check again. Read the log with `$CT logs lab-ospf-r2`, and check `network` statements and router IDs. |

## Challenge (optional)
1. **Floating static route.** On `lab-h1`, add a second default route with a higher metric (`metric 200`) through a router address that doesn't exist yet. Predict which route `ip route get 198.51.100.7` uses, check, and then delete the main default route to see the backup take over.
2. **Redundant OSPF path.** In Part B, create a fifth network `ospf-r1r3` (`10.0.13.0/24`), connect `r1` and `r3` to it and add it to their OSPF configuration (`network 10.0.13.0/24 area 0`; reload with `$CT restart`). Which path does `r1` use to `10.3.3.0/24` now, and with what cost? Disconnect the new link and measure how long it takes for `r1` to switch back.
3. **Protect OSPF.** Enable OSPF cryptographic authentication on one link, using FRR's documentation for `ip ospf authentication message-digest` or key chains. Configure it on only one side first and watch the adjacency drop.

## Cleanup
Delete the five namespaces. Deleting a namespace also deletes the veth interfaces in it, and deleting one end of a veth pair deletes the other end.

```bash
for ns in lab-h1 lab-r1 lab-r2 lab-r3 lab-h2; do
  sudo ip netns del $ns
done
ip netns list
```

Expected output: `ip netns list` prints nothing (or at least no `lab-` namespaces).

If you did Part B, remove the containers, the networks, the configuration folder and the image. Set `CT` again if you opened a new terminal.

```bash
CT=podman          # or: CT=docker
$CT rm -f lab-ospf-r1 lab-ospf-r2 lab-ospf-r3
$CT network rm ospf-lan1 ospf-r1r2 ospf-r2r3 ospf-lan3
$CT rmi quay.io/frrouting/frr:10.7.1
cd ~ && rm -r ~/ospf-lab
$CT ps -a --filter name=lab-ospf
$CT network ls --filter name=ospf-
```

Expected output: the last two commands list only their header lines, with no `lab-ospf` containers and no `ospf-` networks.

## What you learned
- A Linux network namespace with `net.ipv4.ip_forward=1` is a router. Its routing table starts with connected routes only.
- Routing works in each direction separately: a missing forward route produces an ICMP error from the router that lacks it, while a missing return route often produces silence.
- `ip route get` on each hop, toward the destination and back toward the source, finds the router with the missing route quickly.
- A `* * *` hop can still forward traffic. Here its replies were dropped by a reverse path filter, because one router had no route back to part of the network.
- Longest prefix match picks the most specific route, whatever the order the routes were added in.
- (Optional) OSPF builds the same routes automatically from hellos and a shared database, shows them with distance 110 and a cost metric, and withdraws and restores them when a link fails.