# SNMP Infrastructure
SNMP trap receiver hosted on kaburuaibox. All 6 nodes send traps. Hermes polls the trap log every 5 minutes and alerts to Telegram on significant events.
## Decision rationale
- kaburuaibox chosen as receiver: 99GB RAM free, Owl Alpha nerve centre, always on - TrueNAS was historically most reliable box pre-kaburuaibox, but resources favour Z840 - All Linux nodes use net-snmp community v2c with community string `kaburu`
## Trap receiver — kaburuaibox
| Item | Value |
| —— | ——- |
| Package | NET-SNMP 5.9.4 (snmptrapd) |
| Listen | UDP 162, all interfaces |
| Trap log | `/var/log/snmptrapd-events.log` |
| Handler script | `/usr/local/bin/snmp-trap-handler.sh` |
| Redis key | `hermes:snmp_traps` |
| Hermes module | `/opt/hermes/agent/snmp_monitor.py` |
| Poll interval | Every 5 minutes |
## Node inventory
| Node | IP used | Community | trap2sink | Persistent |
| —— | ——— | ———– | ———– | ———— |
| kaburusvr (Hetzner) | 100.112.54.2 (Tailscale) | kaburu | 100.120.18.44 | ✅ systemd |
| ubuntu-svr | 100.101.7.86 (Tailscale) | kaburu | 100.120.18.44 | ✅ systemd |
| kaburuaibox (self) | 127.0.0.1 | kaburu | 127.0.0.1 | ✅ systemd |
| truenas-scale | 100.74.178.14 (Tailscale) | kaburu | 100.120.18.44 | ✅ systemd + midclt |
| KaburuLINK (Unraid) | 192.168.0.250 (LAN) | kaburu | 192.168.0.253 | ✅ /boot/config/go |
| OPNsense | 192.168.0.1 (LAN) | kaburu | 100.120.18.44 | ✅ /etc/rc.conf.local |
## OPNsense note
OPNsense snmpd responds on UDP 161 but requires a LAN firewall rule to allow polling from kaburuaibox (192.168.0.0/24 → This Firewall → UDP 161).
To add: Firewall > Rules > LAN > Add - Action: Pass, Protocol: UDP, Source: LAN net, Destination: This Firewall, Port: 161 - Description: SNMP from LAN
SNMP polling of OPNsense from kaburuaibox times out due to OPNsense routing replies via WireGuard tunnel rather than back on igc0 — a known “traffic to self” quirk. This does not matter — OPNsense is trap-driven only. Traps flow outbound to kaburuaibox correctly (confirmed coldStart/warmStart traps on every snmpd restart). Polling not needed — Hermes uses the SSH/API connector for OPNsense health checks.
## Unraid persistence
net-snmp 5.9.5.2 installed from Slackware64-current. Package saved to `/boot/extra/net-snmp.txz` — reinstalled on each boot via `/boot/config/go`. Config at `/etc/snmp/snmpd.conf` — rebuilt on boot from go script.
## Alert mapping
| OID / keyword | Emoji | Meaning | Action |
| ————— | ——- | ——— | ——– |
| linkDown | 🔴 | Interface went down | Check network, investigate host |
| linkUp | 🟢 | Interface came back up | Info only |
| coldStart | 🔄 | Device rebooted cold | Investigate if unexpected |
| warmStart | 🔄 | Device restarted service | Info only |
| authenticationFailure | 🚨 | Wrong community string used | Investigate immediately |
## Management commands
```bash # Check trap log live tail -f /var/log/snmptrapd-events.log
# Test trap from any node (run on that node) snmptrap -v2c -c kaburu 100.120.18.44 '' .1.3.6.1.6.3.1.1.5.1
# Poll a node manually snmpwalk -v2c -c kaburu <node-ip> .1.3.6.1.2.1.1.1.0
# Check snmptrapd status systemctl status snmptrapd
# Check snmpd status on ubuntu-svr / hetzner systemctl status snmpd ```
## History
| Date | Event |
| —— | ——- |
| 2026-05-25 | Full SNMP stack deployed across all 6 nodes |
| 2026-05-25 | Hermes snmp_monitor.py + check_snmp() integrated |
| 2026-05-25 | Trap flow confirmed from all nodes |
## ⚠️ IMPORTANT — OPNsense monitoring rules (READ BEFORE CHECKING)
1. OPNsense NEVER responds to ping — this is by design, not a fault. Do NOT report it as down based on ping failure. 2. SSH to OPNsense from kaburuaibox — use LAN IP `192.168.0.1`, NOT Tailscale IP. OPNsense SSH is restricted to LAN. 3. Verify OPNsense is up via its API (MCP tool `opnsense:diagnostics_manage`) or by checking if it's routing traffic (default gateway responds = firewall is up). 4. Tailscale relay via lhr from kaburuaibox to OPNsense is normal — direct connection is not available between these two nodes.