# SNMP Infrastructure

SNMP trap receiver hosted on kaburuaibox. All 6 nodes send traps. Hermes polls the trap log every 5 minutes and alerts to Telegram on significant events.

## Decision rationale

- kaburuaibox chosen as receiver: 99GB RAM free, Owl Alpha nerve centre, always on - TrueNAS was historically most reliable box pre-kaburuaibox, but resources favour Z840 - All Linux nodes use net-snmp community v2c with community string `kaburu`

## Trap receiver — kaburuaibox

Item Value
————-
Package NET-SNMP 5.9.4 (snmptrapd)
Listen UDP 162, all interfaces
Trap log `/var/log/snmptrapd-events.log`
Handler script `/usr/local/bin/snmp-trap-handler.sh`
Redis key `hermes:snmp_traps`
Hermes module `/opt/hermes/agent/snmp_monitor.py`
Poll interval Every 5 minutes

## Node inventory

Node IP used Community trap2sink Persistent
————————–———–————
kaburusvr (Hetzner) 100.112.54.2 (Tailscale) kaburu 100.120.18.44 ✅ systemd
ubuntu-svr 100.101.7.86 (Tailscale) kaburu 100.120.18.44 ✅ systemd
kaburuaibox (self) 127.0.0.1 kaburu 127.0.0.1 ✅ systemd
truenas-scale 100.74.178.14 (Tailscale) kaburu 100.120.18.44 ✅ systemd + midclt
KaburuLINK (Unraid) 192.168.0.250 (LAN) kaburu 192.168.0.253 ✅ /boot/config/go
OPNsense 192.168.0.1 (LAN) kaburu 100.120.18.44 ✅ /etc/rc.conf.local

## OPNsense note

OPNsense snmpd responds on UDP 161 but requires a LAN firewall rule to allow polling from kaburuaibox (192.168.0.0/24 → This Firewall → UDP 161).

To add: Firewall > Rules > LAN > Add - Action: Pass, Protocol: UDP, Source: LAN net, Destination: This Firewall, Port: 161 - Description: SNMP from LAN

SNMP polling of OPNsense from kaburuaibox times out due to OPNsense routing replies via WireGuard tunnel rather than back on igc0 — a known “traffic to self” quirk. This does not matter — OPNsense is trap-driven only. Traps flow outbound to kaburuaibox correctly (confirmed coldStart/warmStart traps on every snmpd restart). Polling not needed — Hermes uses the SSH/API connector for OPNsense health checks.

## Unraid persistence

net-snmp 5.9.5.2 installed from Slackware64-current. Package saved to `/boot/extra/net-snmp.txz` — reinstalled on each boot via `/boot/config/go`. Config at `/etc/snmp/snmpd.conf` — rebuilt on boot from go script.

## Alert mapping

OID / keyword Emoji Meaning Action
———————-—————–
linkDown 🔴 Interface went down Check network, investigate host
linkUp 🟢 Interface came back up Info only
coldStart 🔄 Device rebooted cold Investigate if unexpected
warmStart 🔄 Device restarted service Info only
authenticationFailure 🚨 Wrong community string used Investigate immediately

## Management commands

```bash # Check trap log live tail -f /var/log/snmptrapd-events.log

# Test trap from any node (run on that node) snmptrap -v2c -c kaburu 100.120.18.44 '' .1.3.6.1.6.3.1.1.5.1

# Poll a node manually snmpwalk -v2c -c kaburu <node-ip> .1.3.6.1.2.1.1.1.0

# Check snmptrapd status systemctl status snmptrapd

# Check snmpd status on ubuntu-svr / hetzner systemctl status snmpd ```

## History

Date Event
————-
2026-05-25 Full SNMP stack deployed across all 6 nodes
2026-05-25 Hermes snmp_monitor.py + check_snmp() integrated
2026-05-25 Trap flow confirmed from all nodes

## ⚠️ IMPORTANT — OPNsense monitoring rules (READ BEFORE CHECKING)

1. OPNsense NEVER responds to ping — this is by design, not a fault. Do NOT report it as down based on ping failure. 2. SSH to OPNsense from kaburuaibox — use LAN IP `192.168.0.1`, NOT Tailscale IP. OPNsense SSH is restricted to LAN. 3. Verify OPNsense is up via its API (MCP tool `opnsense:diagnostics_manage`) or by checking if it's routing traffic (default gateway responds = firewall is up). 4. Tailscale relay via lhr from kaburuaibox to OPNsense is normal — direct connection is not available between these two nodes.