**Tailscale IP:** `100.127.64.74`

# OPNsense

Network firewall and router. Runs FreeBSD. Member of the Tailscale mesh.

**Current version:** 26.7.1_1 "Xenial Xenops" (FreeBSD 15.1) — upgraded 2026-07-30
**Pre-upgrade ZFS snapshot:** `zroot/ROOT/default@pre-upgrade-26.7-20260730` (kept as safety net)
**Full upgrade procedure + rollback plan:** `/home/kaburu/wiki/wiki/procedures/opnsense-backup/README.md`

## 26.7 upgrade — what changed (2026-07-30)

- Upgraded from 26.1.11_10 → 26.7.1_1 via web GUI (System → Firmware → Updates)
- FreeBSD 14.x → 15.1-RELEASE (major OS jump, OpenSSL 3.0 → 3.5)
- Firewall rules migrated to MVC/API (UI changed; rules preserved)
- Interface assignments migrated to MVC/API
- Outbound NAT → source NAT migration available (we have no outbound NAT rules, no action needed)
- Custom files **survived intact**: `local-kaburu.conf`, `caddy-hosts.conf`, all DHCP leases
- DNS still working post-upgrade (no re-application of `start.sh` wrapper needed)
- Reboot was required; lasted ~3 minutes

## Version history

- **2026-07-30:** Upgraded 26.1.11_10 → 26.7.1_1 (major release, FreeBSD 15.1, OpenSSL 3.5, MVC/API migration). ZFS snapshot taken pre-upgrade, rollback plan in `/home/kaburu/wiki/wiki/procedures/opnsense-backup/README.md`. Custom files (`local-kaburu.conf`, `caddy-hosts.conf`) survived.
- **2026-07-26:** dnsmasq + Unbound outage (heredoc bug, fixed)
- **2026-06-05:** Previous version

## Roles

- DHCP server for `192.168.0.0/24`
- Local DNS via dnsmasq (hostname resolution on LAN)
- Firewall — outbound SSH (port 22) from LAN blocked except via Tailscale subnet
- Tailscale node — has mesh access to all other nodes

## Static DHCP assignments

| Host | IP | MAC(s) |
|------|----|--------|
| kaburuaibox | 192.168.0.253 | `b4:96:91:11:44:dc` (ens1f0), `b4:96:91:11:44:de` (ens1f1) |
| ubuntu-svr | 192.168.0.252 | — |

Both 10Gb NIC MACs registered for kaburuaibox so either port gets the same IP regardless of which is active in the bond.

## Security policy

- Outbound SSH from LAN to internet is **blocked** — intentional
- SSH to Hetzner must go via Tailscale (`100.112.54.2`)
- This prevents accidental direct SSH exposure and enforces Tailscale as the access layer

## OPNsense monitoring — CRITICAL NOTE

- **OPNsense (192.168.0.1) NEVER responds to ICMP/ping** — this is intentional, NOT a fault
- **Never report OPNsense as down based on ping failure alone**
- Use `opnsense:diagnostics_manage` API or check routing to verify it is up
- SSH from kaburuaibox uses 192.168.0.1 (LAN), not Tailscale IP
- SSH to OPNsense from LAN is blocked by firewall policy
- SSH to OPNsense from kaburuaibox must use Tailscale IP: `ssh root@100.127.64.74`
- SSH config Host entry: `opnsense` → 100.127.64.74

## DNS architecture

- **Unbound** (recursive resolver, upstream queries) — listens on 0.0.0.0:53 / :::53, serves external + cached DNS
- **dnsmasq** (DHCP + local overrides) — listens on `igc0` only, **`port=0` (DHCP-only, no DNS)**
- **OPNsense forwards**: Unbound handles `kaburu.lan` local-zone via `private-address: 192.168.0.0/16` and root hints for external
- **Files of record**:
  - `/usr/local/etc/dnsmasq.conf` — auto-generated by OPNsense, do NOT edit
  - `/usr/local/etc/dnsmasq.conf.d/*.conf` — manually-maintained drop-ins (read in alpha order)
  - `/var/unbound/unbound.conf` — Unbound active config
  - `/var/etc/dnsmasq-hosts` — DHCP-served host entries (auto)
  - `/var/etc/dnsmasq-leases` — current DHCP leases (auto)

## Service management — gotchas

- **Unbound start requires `onestart`**, not `start`. The `start` command fails with "Set unbound_enable to YES in /etc/rc.conf or use 'onestart'". OPNsense manages service enable flags via the web UI; the rc.d script's plain `start` is gated. Always use `onestart` when manually recovering Unbound from SSH.
  - `ssh root@opnsense '/usr/local/etc/rc.d/unbound onestart'`
- **dnsmasq uses `start`/`restart`/`status` normally.** OPNsense's dnsmasq rc.d script is happy to take those.
- **After fixing dnsmasq config, always `dnsmasq --test` first** to verify syntax before restarting. `dnsmasq --test` returns `dnsmasq: syntax check OK.` on success and exit code 0.

## caddy-hosts.conf (DHCP-only — dead weight)

The file `/usr/local/etc/dnsmasq.conf.d/caddy-hosts.conf` contains `address=/X.kaburu.lan/192.168.0.1` entries (openwebui, portainer, ollama, frigate, omada, ha). **These do NOT work** because dnsmasq has `port=0` (DHCP-only) — Unbound handles DNS, not dnsmasq. The file is read silently and ignored. The caddy reverse-proxy services on 192.168.0.1 are reached via direct IP+port, not via these hostnames.

**Do not "fix" this by removing `port=0` from dnsmasq.conf** — that would make dnsmasq and Unbound both try to bind port 53, causing port conflicts and intermittent DNS failures. The current setup is correct.

If you want `openwebui.kaburu.lan` style hostnames to work, add them to Unbound's `private-domain` / `local-data` config via the OPNsense web UI (Services → Unbound DNS → Overrides), not via dnsmasq.

## Caddy plugin — NOT installed (2026-08-22 audit)

The `caddy-reverse-proxy.md` wiki page describes a Caddy-on-OPNsense reverse proxy with auto-TLS. **The Caddy plugin was never actually installed on OPNsense.** Evidence (kaburuaibox LAN probes, all 2026-08-22):

| Check | Result | Meaning |
|-------|--------|---------|
| `nc -zv 192.168.0.1 443` | timed out | Caddy not listening |
| `nc -zv 192.168.0.1 4443` | timed out | Caddy UI not listening |
| `curl https://192.168.0.1/api/caddy/diagnostics/general` | HTTP 404 | Plugin not installed |
| `curl https://192.168.0.1/api/caddy/settings/get` | HTTP 404 | Plugin not installed |
| `dig openwebui.kaburu.lan @192.168.0.1` | NXDOMAIN | Unbound doesn't know kaburu.lan either |

Only ports 22 (SSH) and 80 (HTTP redirect to dead HTTPS) respond on OPNsense from the LAN. The wiki `caddy-reverse-proxy.md` documents a **wishlist, not reality**.

**To actually deploy Caddy reverse proxy:** install `os-caddy` plugin via System → Firmware → Plugins in OPNsense UI, configure domain mappings, then add the corresponding Unbound overrides so `*.kaburu.lan` resolves to 192.168.0.1. Or **just use direct IP+port access** (`http://192.168.0.253:3000` for Open WebUI, `https://192.168.0.252:8123` for HA, etc.) — which is what's currently working.

## 2026-07-26 dnsmasq + Unbound outage — root cause + fix

- **Symptom:** ShieldTV (and other LAN clients) lost internet. OPNsense update server couldn't resolve. Both ShieldTV and OPNsense UI sluggish. Web UI shows Unbound "stopped" (red).
- **Initial investigation:** Rebooted OPNsense. dnsmasq failed to start: log showed `bad option at line 8 of /usr/local/etc/dnsmasq.conf.d/caddy-hosts.conf` and `FAILED to start up`.
- **First fix (dnsmasq):** rewrote caddy-hosts.conf with 7 valid lines, restarted dnsmasq. Verified DHCP leases resumed.
- **Second issue found:** Unbound ALSO down. Tried `onestart` — process started but bound to 127.0.0.1 only (loopback), not 192.168.0.1. DNS still broken for LAN clients.
- **Real root cause (Unbound):** The hand-written file `/usr/local/etc/unbound.opnsense.d/local-kaburu.conf` (which provides local-data entries for caddy services like `openwebui.local.kaburu.cc`) had THREE bugs:
  1. **EOF + echo garbage on lines 8-9** — same heredoc bug as caddy-hosts.conf. Someone (past session) tried to use a `cat <<EOF > file` block but the heredoc wasn't processed, so the literal text was written to the file. unbound rejects `EOF` as a syntax error.
  2. **Bad `local-data:` format** — used RFC-style `name. 3600 IN A value` instead of unbound's space-separated `name TYPE value`.
  3. **Missing `server:` section wrapper** — included files need a section header, otherwise unbound rejects with "syntax error, is there no section start after an include-toplevel directive perhaps".
- **Proper fix:**
  1. Backed up broken source to `/usr/local/etc/unbound.opnsense.d/local-kaburu.conf.broken-2026-07-26`
  2. Rewrote with correct content: `server:` section + space-separated `local-data:` lines, no garbage
  3. Restarted via `/usr/local/opnsense/scripts/unbound/start.sh` (the proper OPNsense wrapper, NOT `configctl unbound restart` which has its own quirks)
- **Why the wrapper, not configctl:** `configctl unbound restart` was returning "OK" but the unbound process didn't actually come up. The OPNsense wrapper script (`/usr/local/opnsense/scripts/unbound/start.sh`) is what configd uses during boot, and it works correctly: it copies fresh configs from `.opnsense.d/` to `/var/unbound/etc/`, fixes permissions, then exec's unbound. This is the canonical way to start unbound.
- **Verification:**
  - Unbound pid 85596, bound to `*:53` (all interfaces, IPv4+IPv6)
  - `configctl unbound status` → "running as pid 85596"
  - `dig @192.168.0.1 google.com` → returns real IPs (no longer cached-only)
  - sockstat shows unbound listening on tcp4/udp4/tcp6/udp6 port 53
- **Durability of the fix:**
  - `local-kaburu.conf` is NOT regenerated by configd templates (configd regenerates `access_lists.conf`, `safesearch.conf`, `dot.conf`, `domainoverrides.conf` — but no template exists for `local-kaburu.conf`). Will survive reboots, OPNsense updates, Web UI config changes.
  - Only way to break it again: someone hand-edits `local-kaburu.conf` with the same heredoc bug. The wiki note on caddy-hosts.conf already covers this pattern.
- **Side notes:**
  - Also added `unbound_enable=YES` and `dnsmasq_enable=YES` to `/etc/rc.conf` — these are technically redundant (configd handles starting services via its own mechanism, not rc.d) but harmless. May get overwritten on next Web UI config save.
  - The caddy LAN entries (e.g. `openwebui.local.kaburu.cc`) currently don't resolve — that's because they need a `local-zone:` declaration in addition to `local-data:`. The caddy reverse proxy itself is on a separate machine, not OPNsense. If you want those to resolve, add `local-zone: "local.kaburu.cc" static` to the config.

## HOW TO START UNBOUND MANUALLY (the right way)

**Wrong ways that don't actually work:**
- `configctl unbound restart` — returns "OK" but doesn't always start the process (state desync bug in 26.1.11_10)
- `/usr/local/etc/rc.d/unbound start` — uses the FreeBSD package's default config (`/usr/local/etc/unbound/unbound.conf` — sample file, doesn't bind to LAN interfaces), so DNS still doesn't work
- `/usr/local/etc/rc.d/unbound onestart` — same problem as above

**Right way:**
```
/usr/local/opnsense/scripts/unbound/start.sh
```
This is the wrapper that OPNsense's configd uses during boot. It copies the proper config files, fixes perms, and starts unbound.

**Even more right way (also re-generates the OPNsense template configs):**
```
configctl filter reload   # forces configd to re-render templates
/usr/local/opnsense/scripts/unbound/start.sh
```

## HOW TO START DNSMASQ MANUALLY

```
/usr/local/etc/rc.d/dnsmasq restart
```
This one works normally — `start`, `restart`, `status` all behave as expected. (Unlike unbound.)

## Monitoring — daily-report.sh checks OPNsense DNS

Added a check to `/home/kaburu/.hermes/scripts/opnsense-dns-check.sh` (~20 lines). Runs from `daily-report.sh` (cron `c2698066941c`) and:

1. SSHes into OPNsense via the API tunnel
2. Checks if Unbound is bound to `*:53` (not 127.0.0.1 only — that was the silent failure mode today)
3. Checks if Unbound process is actually running (sockstat check)
4. Runs `dig +short @192.168.0.1 opnsense.org` from Z840 to verify real LAN-side resolution
5. Alerts if any of the above fail

**Why this matters:** the OPNsense Web UI shows "stopped" when Unbound fails to start, but the failure mode today was sneakier — Unbound "started" (process existed) but was bound only to loopback. The UI said stopped (because of `unbound_enable` flag missing in rc.conf) but a fresh dig would silently fail. This check catches both modes.

**Where it lives:**
- Script: `/home/kaburu/.hermes/scripts/opnsense-dns-check.sh`
- Backup: `/home/kaburu/wiki/scripts/opnsense-dns-check.sh`
- Wired into: `/home/kaburu/.hermes/scripts/daily-report.sh`

**Output format:**
- Silent on healthy: no line added
- Alert on failure: `🚨 opnsense-dns: <reason>` with detail

## Heredoc bug — DO NOT REPEAT

**Symptom pattern (seen twice today):**
- File written with `cat <<EOF > /path/to/file` block
- Shell context didn't process the heredoc
- File ends up with `EOF` and `echo ...` literal text appended
- The target service (dnsmasq, unbound) fails to parse the file with "bad option" or "syntax error"

**Prevention:**
- For OPNsense files, use `ssh` + `scp` from a local file, not inline heredoc
- For OPNsense files, use `/usr/local/opnsense/scripts/unbound/start.sh` after editing to apply
- If you must use a heredoc, use `cat <<'EOF'` (quoted) so `$` and backticks aren't interpreted

**Files affected today (fixed):**
- `/usr/local/etc/dnsmasq.conf.d/caddy-hosts.conf` (dnsmasq config)
- `/usr/local/etc/unbound.opnsense.d/local-kaburu.conf` (unbound local-data)

## Caddy entries — current state

The 6 entries in `local-kaburu.conf` (openwebui, portainer, ollama, frigate, omada, ha, all pointing to `192.168.0.1`) are **defined but not active**. They need a `local-zone` declaration to be discoverable. To make them work, add to `/usr/local/etc/unbound.opnsense.d/local-kaburu.conf`:
```
server:
    local-zone: "local.kaburu.cc" static
    local-data: "openwebui.local.kaburu.cc A 192.168.0.1"
    ...
```

## Network layers

OPNsense handles routing/firewall. The TP-Link switching layer underneath is managed by the Omada Controller on [[ubuntu-svr]].

## SSH

```
ssh opnsense
```

**SSH config Host entry 'opnsense' maps to 100.127.64.74 with user root and key id_ed256.**

## 2026-07-30 — Upgrade 26.1.11_10 → 26.7.1_1

- **Reason:** OPNsense 26.1 reached EOL, no more security updates
- **Pre-flight:**
  - ZFS snapshot `zroot/ROOT/default@pre-upgrade-26.7-20260730` taken (free ZFS rollback option)
  - Full config.xml exported to `/home/kaburu/wiki/wiki/procedures/opnsense-backup/config-26.1.11_10-pre-upgrade.xml` (290KB)
  - Custom files backed up: `local-kaburu.conf`, `caddy-hosts.conf` in `/home/kaburu/wiki/wiki/procedures/opnsense-backup/`
  - TC-McQwen loaded locally to maintain local inference during upgrade (internet would drop)
- **Upgrade:** Performed via web UI (System → Firmware → Updates → click OK on "26.1 has reached EOL" prompt)
- **Post-upgrade state:**
  - Version: `opnsense-26.7.1_1` (FreeBSD 15.1, PHP 8.5, OpenVPN 2.7.5, lighttpd 1.4.85, dnsmasq 2.93)
  - **Two unbound processes ran after upgrade** — same 2026-07-26 bug: one bound to 127.0.0.1:53 (rogue), one to *:53 (proper). Killed the rogue pid 52050, DNS restored.
  - Custom files survived intact (`local-kaburu.conf` still 340 bytes, `caddy-hosts.conf` still 289 bytes)
  - Tailscale ✅ connected (100.127.64.74)
  - WireGuard (Mullvad) ✅ wg0 interface present
  - snmpd ✅ running
  - pf firewall ✅ enabled, 601 state entries
- **Watch points for next 24h:**
  - Firewall rules — MVC/API migration, rules should be unchanged but verify in UI
  - Interface assignments — also migrated to MVC/API
  - If DNS breaks: see "HOW TO START UNBOUND MANUALLY" section above
- **Files:** `/home/kaburu/wiki/wiki/procedures/opnsense-backup/` contains full rollback documentation
- **Rollback still available:** `zfs rollback zroot/ROOT/default@pre-upgrade-26.7-20260730` + reboot, if needed

## Recurring unbound duplicate process issue — FIXED 2026-07-30

Same bug appeared twice (2026-07-26 and 2026-07-30 post-upgrade):

### Root cause

`/etc/rc.conf` contained `unbound_enable=YES` from the FreeBSD pkg install.
That causes `/usr/local/etc/rc.d/unbound` to run at boot using the pkg sample config at `/usr/local/etc/unbound/unbound.conf` (binds loopback only).
Then OPNsense's configd template engine starts its own unbound with `/var/unbound/unbound.conf` (binds `*:53`).
Result: two processes, one bound to 127.0.0.1:53 (rogue), one bound to *:53 (proper).

### Triggers observed

- Fresh boot (every reboot since 26.7.1_1 install)
- Any UI/configd action that re-starts unbound

### Symptoms

- `sockstat -l -P udp -P tcp | grep unbound` shows two PIDs for port 53
- One binds `127.0.0.1:53`, other binds `*:53`
- LAN clients work because the `*:53` one answers, but the loopback one races for cache

### Fix applied

```bash
sed -i '' 's|^unbound_enable=YES|unbound_config="/var/unbound/unbound.conf"; unbound_enable=YES|' /etc/rc.conf
```

This makes the FreeBSD RC startup point at OPNsense's config instead of the pkg sample. After reboot only ONE process binds *:53, no rogue sample instance.

### Related fix: SO_SNDBUF buffer warnings

FreeBSD 15 changed `net.inet.udp.maxdgram` default from 1MB down to 57344 bytes. Unbound's default `so-sndbuf: 4m` request gets capped at 56KB and logs "No buffer space available" warnings on every connection.

**Fix applied:**

```bash
echo 'net.inet.udp.maxdgram=1048576' >> /boot/loader.conf.local
```

This persists the 1MB maxdgram across reboots (it would reset to 57344 otherwise).

### Verification after reboot

```bash
sockstat -l -P udp -P tcp | grep unbound
# Now shows ONE pid bound to *:53 with multiple threads, plus 127.0.0.1:953 for control
```

### Detection (for daily check)

Added duplicate unbound detection in `/home/kaburu/.hermes/scripts/opnsense-dns-check.sh` — counts unique PIDs in sockstat output and alerts if >1.