Kaburu Wiki

Tailscale IP: `100.127.64.74`

# OPNsense

Network firewall and router. Runs FreeBSD. Member of the Tailscale mesh.

Current version: 26.7.1_1 “Xenial Xenops” (FreeBSD 15.1) — upgraded 2026-07-30 Pre-upgrade ZFS snapshot: `zroot/ROOT/[email protected]` (kept as safety net) Full upgrade procedure + rollback plan: `/home/kaburu/wiki/wiki/procedures/opnsense-backup/README.md`

## 26.7 upgrade — what changed (2026-07-30)

- Upgraded from 26.1.11_10 → 26.7.1_1 via web GUI (System → Firmware → Updates) - FreeBSD 14.x → 15.1-RELEASE (major OS jump, OpenSSL 3.0 → 3.5) - Firewall rules migrated to MVC/API (UI changed; rules preserved) - Interface assignments migrated to MVC/API - Outbound NAT → source NAT migration available (we have no outbound NAT rules, no action needed) - Custom files survived intact: `local-kaburu.conf`, `caddy-hosts.conf`, all DHCP leases - DNS still working post-upgrade (no re-application of `start.sh` wrapper needed) - Reboot was required; lasted ~3 minutes

## Version history

- 2026-07-30: Upgraded 26.1.11_10 → 26.7.1_1 (major release, FreeBSD 15.1, OpenSSL 3.5, MVC/API migration). ZFS snapshot taken pre-upgrade, rollback plan in `/home/kaburu/wiki/wiki/procedures/opnsense-backup/README.md`. Custom files (`local-kaburu.conf`, `caddy-hosts.conf`) survived. - 2026-07-26: dnsmasq + Unbound outage (heredoc bug, fixed) - 2026-06-05: Previous version

## Roles

- DHCP server for `192.168.0.0/24` - Local DNS via dnsmasq (hostname resolution on LAN) - Firewall — outbound SSH (port 22) from LAN blocked except via Tailscale subnet - Tailscale node — has mesh access to all other nodes

## Static DHCP assignments

Both 10Gb NIC MACs registered for kaburuaibox so either port gets the same IP regardless of which is active in the bond.

## Security policy

- Outbound SSH from LAN to internet is blocked — intentional - SSH to Hetzner must go via Tailscale (`100.112.54.2`) - This prevents accidental direct SSH exposure and enforces Tailscale as the access layer

## OPNsense monitoring — CRITICAL NOTE

- OPNsense (192.168.0.1) NEVER responds to ICMP/ping — this is intentional, NOT a fault - Never report OPNsense as down based on ping failure alone - Use `opnsense:diagnostics_manage` API or check routing to verify it is up - SSH from kaburuaibox uses 192.168.0.1 (LAN), not Tailscale IP - SSH to OPNsense from LAN is blocked by firewall policy - SSH to OPNsense from kaburuaibox must use Tailscale IP: `ssh [email protected]` - SSH config Host entry: `opnsense` → 100.127.64.74

## DNS architecture

- Unbound (recursive resolver, upstream queries) — listens on 0.0.0.0:53 / :::53, serves external + cached DNS - dnsmasq (DHCP + local overrides) — listens on `igc0` only, `port=0` (DHCP-only, no DNS) - OPNsense forwards: Unbound handles `kaburu.lan` local-zone via `private-address: 192.168.0.0/16` and root hints for external - Files of record:

1. `/usr/local/etc/dnsmasq.conf` — auto-generated by OPNsense, do NOT edit
2. `/usr/local/etc/dnsmasq.conf.d/*.conf` — manually-maintained drop-ins (read in alpha order)
3. `/var/unbound/unbound.conf` — Unbound active config
4. `/var/etc/dnsmasq-hosts` — DHCP-served host entries (auto)
5. `/var/etc/dnsmasq-leases` — current DHCP leases (auto)

## Service management — gotchas

- Unbound start requires `onestart`, not `start`. The `start` command fails with “Set unbound_enable to YES in /etc/rc.conf or use 'onestart'”. OPNsense manages service enable flags via the web UI; the rc.d script's plain `start` is gated. Always use `onestart` when manually recovering Unbound from SSH.

1. `ssh root@opnsense '/usr/local/etc/rc.d/unbound onestart'`

- dnsmasq uses `start`/`restart`/`status` normally. OPNsense's dnsmasq rc.d script is happy to take those. - After fixing dnsmasq config, always `dnsmasq –test` first to verify syntax before restarting. `dnsmasq –test` returns `dnsmasq: syntax check OK.` on success and exit code 0.

## caddy-hosts.conf (DHCP-only — dead weight)

The file `/usr/local/etc/dnsmasq.conf.d/caddy-hosts.conf` contains `address=/X.kaburu.lan/192.168.0.1` entries (openwebui, portainer, ollama, frigate, omada, ha). These do NOT work because dnsmasq has `port=0` (DHCP-only) — Unbound handles DNS, not dnsmasq. The file is read silently and ignored. The caddy reverse-proxy services on 192.168.0.1 are reached via direct IP+port, not via these hostnames.

Do not “fix” this by removing `port=0` from dnsmasq.conf — that would make dnsmasq and Unbound both try to bind port 53, causing port conflicts and intermittent DNS failures. The current setup is correct.

If you want `openwebui.kaburu.lan` style hostnames to work, add them to Unbound's `private-domain` / `local-data` config via the OPNsense web UI (Services → Unbound DNS → Overrides), not via dnsmasq.

## Caddy plugin — NOT installed (2026-08-22 audit)

The `caddy-reverse-proxy.md` wiki page describes a Caddy-on-OPNsense reverse proxy with auto-TLS. The Caddy plugin was never actually installed on OPNsense. Evidence (kaburuaibox LAN probes, all 2026-08-22):

Only ports 22 (SSH) and 80 (HTTP redirect to dead HTTPS) respond on OPNsense from the LAN. The wiki `caddy-reverse-proxy.md` documents a wishlist, not reality.

To actually deploy Caddy reverse proxy: install `os-caddy` plugin via System → Firmware → Plugins in OPNsense UI, configure domain mappings, then add the corresponding Unbound overrides so `*.kaburu.lan` resolves to 192.168.0.1. Or just use direct IP+port access (`http://192.168.0.253:3000` for Open WebUI, `https://192.168.0.252:8123` for HA, etc.) — which is what's currently working.

## 2026-07-26 dnsmasq + Unbound outage — root cause + fix

- Symptom: ShieldTV (and other LAN clients) lost internet. OPNsense update server couldn't resolve. Both ShieldTV and OPNsense UI sluggish. Web UI shows Unbound “stopped” (red). - Initial investigation: Rebooted OPNsense. dnsmasq failed to start: log showed `bad option at line 8 of /usr/local/etc/dnsmasq.conf.d/caddy-hosts.conf` and `FAILED to start up`. - First fix (dnsmasq): rewrote caddy-hosts.conf with 7 valid lines, restarted dnsmasq. Verified DHCP leases resumed. - Second issue found: Unbound ALSO down. Tried `onestart` — process started but bound to 127.0.0.1 only (loopback), not 192.168.0.1. DNS still broken for LAN clients. - Real root cause (Unbound): The hand-written file `/usr/local/etc/unbound.opnsense.d/local-kaburu.conf` (which provides local-data entries for caddy services like `openwebui.local.kaburu.cc`) had THREE bugs:

1. **EOF + echo garbage on lines 8-9** — same heredoc bug as caddy-hosts.conf. Someone (past session) tried to use a `cat <<EOF > file` block but the heredoc wasn't processed, so the literal text was written to the file. unbound rejects `EOF` as a syntax error.
2. **Bad `local-data:` format** — used RFC-style `name. 3600 IN A value` instead of unbound's space-separated `name TYPE value`.
3. **Missing `server:` section wrapper** — included files need a section header, otherwise unbound rejects with "syntax error, is there no section start after an include-toplevel directive perhaps".

- Proper fix:

1. Backed up broken source to `/usr/local/etc/unbound.opnsense.d/local-kaburu.conf.broken-2026-07-26`
2. Rewrote with correct content: `server:` section + space-separated `local-data:` lines, no garbage
3. Restarted via `/usr/local/opnsense/scripts/unbound/start.sh` (the proper OPNsense wrapper, NOT `configctl unbound restart` which has its own quirks)

- Why the wrapper, not configctl: `configctl unbound restart` was returning “OK” but the unbound process didn't actually come up. The OPNsense wrapper script (`/usr/local/opnsense/scripts/unbound/start.sh`) is what configd uses during boot, and it works correctly: it copies fresh configs from `.opnsense.d/` to `/var/unbound/etc/`, fixes permissions, then exec's unbound. This is the canonical way to start unbound. - Verification:

1. Unbound pid 85596, bound to `*:53` (all interfaces, IPv4+IPv6)
2. `configctl unbound status` → “running as pid 85596”
3. `dig @192.168.0.1 google.com` → returns real IPs (no longer cached-only)
4. sockstat shows unbound listening on tcp4/udp4/tcp6/udp6 port 53

- Durability of the fix:

1. `local-kaburu.conf` is NOT regenerated by configd templates (configd regenerates `access_lists.conf`, `safesearch.conf`, `dot.conf`, `domainoverrides.conf` — but no template exists for `local-kaburu.conf`). Will survive reboots, OPNsense updates, Web UI config changes.
2. Only way to break it again: someone hand-edits `local-kaburu.conf` with the same heredoc bug. The wiki note on caddy-hosts.conf already covers this pattern.

- Side notes:

1. Also added `unbound_enable=YES` and `dnsmasq_enable=YES` to `/etc/rc.conf` — these are technically redundant (configd handles starting services via its own mechanism, not rc.d) but harmless. May get overwritten on next Web UI config save.
2. The caddy LAN entries (e.g. `openwebui.local.kaburu.cc`) currently don't resolve — that's because they need a `local-zone:` declaration in addition to `local-data:`. The caddy reverse proxy itself is on a separate machine, not OPNsense. If you want those to resolve, add `local-zone: “local.kaburu.cc” static` to the config.

## HOW TO START UNBOUND MANUALLY (the right way)

Wrong ways that don't actually work: - `configctl unbound restart` — returns “OK” but doesn't always start the process (state desync bug in 26.1.11_10) - `/usr/local/etc/rc.d/unbound start` — uses the FreeBSD package's default config (`/usr/local/etc/unbound/unbound.conf` — sample file, doesn't bind to LAN interfaces), so DNS still doesn't work - `/usr/local/etc/rc.d/unbound onestart` — same problem as above

Right way: ``` /usr/local/opnsense/scripts/unbound/start.sh ``` This is the wrapper that OPNsense's configd uses during boot. It copies the proper config files, fixes perms, and starts unbound.

Even more right way (also re-generates the OPNsense template configs): ``` configctl filter reload # forces configd to re-render templates /usr/local/opnsense/scripts/unbound/start.sh ```

## HOW TO START DNSMASQ MANUALLY

``` /usr/local/etc/rc.d/dnsmasq restart ``` This one works normally — `start`, `restart`, `status` all behave as expected. (Unlike unbound.)

## Monitoring — daily-report.sh checks OPNsense DNS

Added a check to `/home/kaburu/.hermes/scripts/opnsense-dns-check.sh` (~20 lines). Runs from `daily-report.sh` (cron `c2698066941c`) and:

1. SSHes into OPNsense via the API tunnel 2. Checks if Unbound is bound to `*:53` (not 127.0.0.1 only — that was the silent failure mode today) 3. Checks if Unbound process is actually running (sockstat check) 4. Runs `dig +short @192.168.0.1 opnsense.org` from Z840 to verify real LAN-side resolution 5. Alerts if any of the above fail

Why this matters: the OPNsense Web UI shows “stopped” when Unbound fails to start, but the failure mode today was sneakier — Unbound “started” (process existed) but was bound only to loopback. The UI said stopped (because of `unbound_enable` flag missing in rc.conf) but a fresh dig would silently fail. This check catches both modes.

Where it lives: - Script: `/home/kaburu/.hermes/scripts/opnsense-dns-check.sh` - Backup: `/home/kaburu/wiki/scripts/opnsense-dns-check.sh` - Wired into: `/home/kaburu/.hermes/scripts/daily-report.sh`

Output format: - Silent on healthy: no line added - Alert on failure: `🚨 opnsense-dns: <reason>` with detail

## Heredoc bug — DO NOT REPEAT

Symptom pattern (seen twice today): - File written with `cat «EOF > /path/to/file` block - Shell context didn't process the heredoc - File ends up with `EOF` and `echo …` literal text appended - The target service (dnsmasq, unbound) fails to parse the file with “bad option” or “syntax error”

Prevention: - For OPNsense files, use `ssh` + `scp` from a local file, not inline heredoc - For OPNsense files, use `/usr/local/opnsense/scripts/unbound/start.sh` after editing to apply - If you must use a heredoc, use `cat «'EOF'` (quoted) so `$` and backticks aren't interpreted

Files affected today (fixed): - `/usr/local/etc/dnsmasq.conf.d/caddy-hosts.conf` (dnsmasq config) - `/usr/local/etc/unbound.opnsense.d/local-kaburu.conf` (unbound local-data)

## Caddy entries — current state

The 6 entries in `local-kaburu.conf` (openwebui, portainer, ollama, frigate, omada, ha, all pointing to `192.168.0.1`) are defined but not active. They need a `local-zone` declaration to be discoverable. To make them work, add to `/usr/local/etc/unbound.opnsense.d/local-kaburu.conf`: ``` server:

  local-zone: "local.kaburu.cc" static
  local-data: "openwebui.local.kaburu.cc A 192.168.0.1"
  ...

```

## Network layers

OPNsense handles routing/firewall. The TP-Link switching layer underneath is managed by the Omada Controller on ubuntu-svr.

## SSH

``` ssh opnsense ```

SSH config Host entry 'opnsense' maps to 100.127.64.74 with user root and key id_ed256.

## 2026-07-30 — Upgrade 26.1.11_10 → 26.7.1_1

- Reason: OPNsense 26.1 reached EOL, no more security updates - Pre-flight:

1. ZFS snapshot `zroot/ROOT/[email protected]` taken (free ZFS rollback option)
2. Full config.xml exported to `/home/kaburu/wiki/wiki/procedures/opnsense-backup/config-26.1.11_10-pre-upgrade.xml` (290KB)
3. Custom files backed up: `local-kaburu.conf`, `caddy-hosts.conf` in `/home/kaburu/wiki/wiki/procedures/opnsense-backup/`
4. TC-McQwen loaded locally to maintain local inference during upgrade (internet would drop)

- Upgrade: Performed via web UI (System → Firmware → Updates → click OK on “26.1 has reached EOL” prompt) - Post-upgrade state:

1. Version: `opnsense-26.7.1_1` (FreeBSD 15.1, PHP 8.5, OpenVPN 2.7.5, lighttpd 1.4.85, dnsmasq 2.93)
2. Two unbound processes ran after upgrade — same 2026-07-26 bug: one bound to 127.0.0.1:53 (rogue), one to *:53 (proper). Killed the rogue pid 52050, DNS restored.
3. Custom files survived intact (`local-kaburu.conf` still 340 bytes, `caddy-hosts.conf` still 289 bytes)
4. Tailscale ✅ connected (100.127.64.74)
5. WireGuard (Mullvad) ✅ wg0 interface present
6. snmpd ✅ running
7. pf firewall ✅ enabled, 601 state entries

- Watch points for next 24h:

1. Firewall rules — MVC/API migration, rules should be unchanged but verify in UI
2. Interface assignments — also migrated to MVC/API
3. If DNS breaks: see “HOW TO START UNBOUND MANUALLY” section above

- Files: `/home/kaburu/wiki/wiki/procedures/opnsense-backup/` contains full rollback documentation - Rollback still available: `zfs rollback zroot/ROOT/[email protected]` + reboot, if needed

## Recurring unbound duplicate process issue — FIXED 2026-07-30

Same bug appeared twice (2026-07-26 and 2026-07-30 post-upgrade):

### Root cause

`/etc/rc.conf` contained `unbound_enable=YES` from the FreeBSD pkg install. That causes `/usr/local/etc/rc.d/unbound` to run at boot using the pkg sample config at `/usr/local/etc/unbound/unbound.conf` (binds loopback only). Then OPNsense's configd template engine starts its own unbound with `/var/unbound/unbound.conf` (binds `*:53`). Result: two processes, one bound to 127.0.0.1:53 (rogue), one bound to *:53 (proper).

### Triggers observed

- Fresh boot (every reboot since 26.7.1_1 install) - Any UI/configd action that re-starts unbound

### Symptoms

- `sockstat -l -P udp -P tcp | grep unbound` shows two PIDs for port 53 - One binds `127.0.0.1:53`, other binds `*:53` - LAN clients work because the `*:53` one answers, but the loopback one races for cache

### Fix applied

```bash sed -i '' 's|^unbound_enable=YES|unbound_config=“/var/unbound/unbound.conf”; unbound_enable=YES|' /etc/rc.conf ```

This makes the FreeBSD RC startup point at OPNsense's config instead of the pkg sample. After reboot only ONE process binds *:53, no rogue sample instance.

### Related fix: SO_SNDBUF buffer warnings

FreeBSD 15 changed `net.inet.udp.maxdgram` default from 1MB down to 57344 bytes. Unbound's default `so-sndbuf: 4m` request gets capped at 56KB and logs “No buffer space available” warnings on every connection.

Fix applied:

```bash echo 'net.inet.udp.maxdgram=1048576' » /boot/loader.conf.local ```

This persists the 1MB maxdgram across reboots (it would reset to 57344 otherwise).

### Verification after reboot

```bash sockstat -l -P udp -P tcp | grep unbound # Now shows ONE pid bound to *:53 with multiple threads, plus 127.0.0.1:953 for control ```

### Detection (for daily check)

Added duplicate unbound detection in `/home/kaburu/.hermes/scripts/opnsense-dns-check.sh` — counts unique PIDs in sockstat output and alerts if >1.