# SNMP Infrastructure

SNMP trap receiver hosted on kaburuaibox. All 6 nodes send traps. Hermes polls the
trap log every 5 minutes and alerts to Telegram on significant events.

## Decision rationale

- kaburuaibox chosen as receiver: 99GB RAM free, Owl Alpha nerve centre, always on
- TrueNAS was historically most reliable box pre-kaburuaibox, but resources favour Z840
- All Linux nodes use net-snmp community v2c with community string `kaburu`

## Trap receiver — kaburuaibox

| Item | Value |
|------|-------|
| Package | NET-SNMP 5.9.4 (snmptrapd) |
| Listen | UDP 162, all interfaces |
| Trap log | `/var/log/snmptrapd-events.log` |
| Handler script | `/usr/local/bin/snmp-trap-handler.sh` |
| Redis key | `hermes:snmp_traps` |
| Hermes module | `/opt/hermes/agent/snmp_monitor.py` |
| Poll interval | Every 5 minutes |

## Node inventory

| Node | IP used | Community | trap2sink | Persistent |
|------|---------|-----------|-----------|------------|
| kaburusvr (Hetzner) | 100.112.54.2 (Tailscale) | kaburu | 100.120.18.44 | ✅ systemd |
| ubuntu-svr | 100.101.7.86 (Tailscale) | kaburu | 100.120.18.44 | ✅ systemd |
| kaburuaibox (self) | 127.0.0.1 | kaburu | 127.0.0.1 | ✅ systemd |
| truenas-scale | 100.74.178.14 (Tailscale) | kaburu | 100.120.18.44 | ✅ systemd + midclt |
| KaburuLINK (Unraid) | 192.168.0.250 (LAN) | kaburu | 192.168.0.253 | ✅ /boot/config/go |
| OPNsense | 192.168.0.1 (LAN) | kaburu | 100.120.18.44 | ✅ /etc/rc.conf.local |

## OPNsense note

OPNsense snmpd responds on UDP 161 but requires a LAN firewall rule to allow polling
from kaburuaibox (192.168.0.0/24 → This Firewall → UDP 161).

To add: Firewall > Rules > LAN > Add
- Action: Pass, Protocol: UDP, Source: LAN net, Destination: This Firewall, Port: 161
- Description: SNMP from LAN

SNMP polling of OPNsense from kaburuaibox times out due to OPNsense routing replies via WireGuard tunnel rather than back on igc0 — a known "traffic to self" quirk.
This does not matter — OPNsense is trap-driven only. Traps flow outbound to kaburuaibox correctly (confirmed coldStart/warmStart traps on every snmpd restart).
Polling not needed — Hermes uses the SSH/API connector for OPNsense health checks.

## Unraid persistence

net-snmp 5.9.5.2 installed from Slackware64-current.
Package saved to `/boot/extra/net-snmp.txz` — reinstalled on each boot via `/boot/config/go`.
Config at `/etc/snmp/snmpd.conf` — rebuilt on boot from go script.

## Alert mapping

| OID / keyword | Emoji | Meaning | Action |
|---------------|-------|---------|--------|
| linkDown | 🔴 | Interface went down | Check network, investigate host |
| linkUp | 🟢 | Interface came back up | Info only |
| coldStart | 🔄 | Device rebooted cold | Investigate if unexpected |
| warmStart | 🔄 | Device restarted service | Info only |
| authenticationFailure | 🚨 | Wrong community string used | Investigate immediately |

## Management commands

```bash
# Check trap log live
tail -f /var/log/snmptrapd-events.log

# Test trap from any node (run on that node)
snmptrap -v2c -c kaburu 100.120.18.44 '' .1.3.6.1.6.3.1.1.5.1

# Poll a node manually
snmpwalk -v2c -c kaburu <node-ip> .1.3.6.1.2.1.1.1.0

# Check snmptrapd status
systemctl status snmptrapd

# Check snmpd status on ubuntu-svr / hetzner
systemctl status snmpd
```

## History

| Date | Event |
|------|-------|
| 2026-05-25 | Full SNMP stack deployed across all 6 nodes |
| 2026-05-25 | Hermes snmp_monitor.py + check_snmp() integrated |
| 2026-05-25 | Trap flow confirmed from all nodes |

## ⚠️ IMPORTANT — OPNsense monitoring rules (READ BEFORE CHECKING)

1. **OPNsense NEVER responds to ping** — this is by design, not a fault. Do NOT report it as down based on ping failure.
2. **SSH to OPNsense from kaburuaibox** — use LAN IP `192.168.0.1`, NOT Tailscale IP. OPNsense SSH is restricted to LAN.
3. **Verify OPNsense is up** via its API (MCP tool `opnsense:diagnostics_manage`) or by checking if it's routing traffic (default gateway responds = firewall is up).
4. **Tailscale relay via lhr** from kaburuaibox to OPNsense is normal — direct connection is not available between these two nodes.