# Hermes — Infrastructure Agent

**Search engine:** SearXNG at search.kaburu.cc (Docker on kaburusvr, LiteSpeed reverse proxy). Use for web searches.

Two separate systems share the name "Hermes" on this infrastructure. Do not confuse them.

---

## 1. Hermes Docker Agent (custom monitoring stack)

Automated monitoring, alerting, and management agent for the Kaburu infrastructure.
Runs on kaburuaibox as a Docker Compose stack. Communicates via Telegram.

### Architecture

```
hermes_agent    — Python scheduler + AI checks + Telegram bot
hermes_redis    — State store
```

Stack location: `/opt/hermes/` on kaburuaibox (192.168.0.253)

> Note: The old `hermes_dashboard` FastAPI container was torn down 2026-06-01 (Hermes Workspace).
> The NousResearch dashboard (port 9119) now serves as the web UI — see Section 2.

### Model routing (router.py v3 — last updated 2026-07-02)

All tasks route to llama-server (qwopus-mccoder) as primary. DeepSeek API as fallback if llama-server down.
Claude/Anthropic removed entirely — `ANTHROPIC_API_KEY` is blank.
`ollama.service` stopped and disabled 2026-07-14 to prevent phantom scheduled burns. Start manually when needed for interactive use. See log.md entry 2026-07-14 (Gwen) for full incident.

**CRITICAL: No reasoning models for automation.** `qwen36-reap:28b` and `glm47-reap:23b` perform "prompt processing" (internal thought tokens) at ~800 tok/s with zero output, consuming 75k+ tokens and 10+ GB VRAM per request. Gateway timeout 1800s + 3 retries = 90 min GPU burn per failed task. All recurring crons must use `no_agent: true` + bash scripts. Never route auxiliary tasks, cron, or fallback defaults to reasoning models.

OpenWebUI stopped and disabled 2026-07-14 (user declares emergency-only).

| Tier | Model | When |
|------|-------|------|
| Primary | qwopus-mccoder via llama-server (192.168.0.253:11440/v1) | Always |
| Fallback | deepseek-chat via DeepSeek API | llama-server unreachable |

Task routing:
- `local_capable` (qwopus-mccoder): security, anomaly, architecture, strategic, log analysis, code gen
- `local_fast` (qwopus-mccoder): health checks, backups, reports, WordPress, maintenance, routine

Headroom Proxy at localhost:8787 provides context compression for all llama-server requests.

`_ClaudeStub` class in router.py redirects any `router.claude.messages.create()` calls (used in Telegram chat path in hermes.py ~line 564) to qwopus-mccoder with an Anthropic-compatible response object.

### Scheduled jobs

| Job | Schedule | Alert threshold |
|-----|----------|-----------------|
| Security check (Fail2Ban + SSH) | Every 60 min | 🚨 alert-only — no Telegram on clean |
| SNMP trap monitor | Every 5 min | Any new trap |
| Site uptime (16 sites) | Every 5 min | Any site returning 5xx or unreachable |
| Server health — all hosts | Every 2 hours | 🚨 alert-only — no Telegram on clean |
| SSL certificate expiry | 06:00 daily | <30 days warn, <7 days alert |
| Update check — all servers | 06:30 daily | Security updates = immediate alert |
| Backup log check | 07:00 daily (`BACKUP_CHECK_HOUR`) | Always notifies — success summary or failure alert |
| WordPress check | 07:30 daily | Updates available or errors |
| Daily summary report | 08:00 daily | Always sent |

### Monitored hosts

| Host | Method | What's checked |
|------|--------|----------------|
| kaburusvr (Hetzner) | SSH via Tailscale 100.112.54.2 | Disk, memory, load, Docker, Fail2Ban, SSL, WordPress, backups |
| kaburuaibox (Z840) | subprocess (local, inside container) | Disk, memory, load, Docker, ZFS, GPU, snapshots |
| ubuntu-svr (UM790) | SSH via Tailscale 100.101.7.86 | Docker (Frigate, Omada, Portainer, Open-WebUI), Coral TPU, disk, memory |
| truenas-scale | SSH via Tailscale 100.74.178.14 | ZFS pools, scrub, datasets, disk, memory |
| ~~KaburuLINK (Unraid)~~ | Decommissioned 2026-05-29 | `check_unraid_health()` is a no-op |

### Update checker (updates.py — fixed 2026-06-03)

Uses `apt-get upgrade --dry-run | grep '^Inst'` to determine actually-installable updates.
Previous version used `apt list --upgradable` which included phased packages (Ubuntu gradual rollout)
that cannot actually be applied — this caused false "feature updates available" reports.

Phased packages only appear in the dry-run output if the specific machine is within the phase window — correct behaviour.

Categories:
- 🔴 security — package source URL contains "security"
- 🔵 feature — everything else actually installable

### Backup check (hermes.py — fixed 2026-06-03)

`check_backups()` now always sends a Telegram notification:
- ✅ Success: `notify()` with LLM summary of the backup log
- ⚠️ No log: `notify()` warning if `/var/log/hetzner-backup.log` not found on Hetzner
- 🚨 Failed: `alert()` with LLM detail if ERROR in log and no completion marker

Previous version was silent on success — only alerted on failure.

### SNMP monitoring

Hermes receives SNMP traps from all nodes via snmptrapd on kaburuaibox (UDP 162).
Trap log at `/var/log/snmptrapd-events.log`. `check_snmp()` polls every 5 minutes.

| Trap type | Emoji | Alert level |
|-----------|-------|-------------|
| linkDown | 🔴 | Immediate alert |
| linkUp | 🟢 | Info notification |
| coldStart / warmStart | 🔄 | Info (unexpected reboots escalated) |
| authenticationFailure | 🚨 | Immediate urgent alert |

See [[snmp]] for full SNMP infrastructure documentation.

### Alert levels

| Level | When | Telegram format |
|-------|------|-----------------|
| 🚨 URGENT | Security breach, site down, pool degraded, backup failed | Bold alert, immediate |
| ⚠️ WARNING | Resources >80%, updates available, container restart | Standard notification |
| ✅ INFO | Routine status, all clear | Silent store only (redis) |

### Telegram commands

| Command | Action |
|---------|--------|
| /status | All panel statuses with timestamps |
| /health | Latest health check across all hosts |
| /security | Latest security report |
| /wordpress | WP sites status + pending updates |
| /wpupdate | Apply all WordPress updates now |
| /backups | Latest backup status |
| /ssl | SSL certificate status |
| /uptime | Current site uptime status |
| /updates | Latest update check report |
| /updatesec | Apply security updates on all in-scope servers |
| /updateall \<server\> | Apply all updates on named server |
| /report | Generate daily report now |
| /banned | List Fail2Ban banned IPs |
| /unban \<ip\> | Unban an IP |
| /restart \<container\> | Restart a Docker container on Hetzner |
| /ask \<question\> | Ask Hermes anything about the infrastructure |

### File structure

```
/opt/hermes/
├── docker-compose.yml
├── .env                          ← API keys + credentials
├── config/
│   ├── hetzner_id_ed25519        ← SSH key for Hetzner
│   ├── truenas_id_ed25519        ← SSH key for TrueNAS + ubuntu-svr
│   ├── hermes_shell_id_ed25519   ← SSH key for OPNsense shell connector
│   └── domains.txt               ← domains to monitor SSL for
└── agent/
    ├── hermes.py                 ← main scheduler + checks
    ├── router.py                 ← model routing (v3, llama-server primary)
    ├── hetzner.py                ← SSH connector to kaburusvr.uk
    ├── truenas.py                ← SSH connector to TrueNAS
    ├── ubuntu_svr.py             ← SSH connector to ubuntu-svr
    ├── unraid.py                 ← stub (KaburuLINK decommissioned)
    ├── opnsense.py               ← HTTP API connector (hermes user, read-only)
    ├── shell.py                  ← generic SSH shell connector
    ├── snmp_monitor.py           ← SNMP trap log watcher
    ├── updates.py                ← cross-server update checker
    ├── telegram_bot.py           ← send alerts/notifications
    ├── telegram_commands.py      ← receive /commands
    ├── wordpress.py              ← WP-CLI operations
    ├── memory.py                 ← Redis-backed memory/state
    └── requirements.txt
```

### Environment variables (`/opt/hermes/.env`)

| Variable | Purpose |
|----------|---------|
| `ANTHROPIC_API_KEY` | Blank — Claude removed from routing |
| `DEEPSEEK_API_KEY` | DeepSeek API fallback |
| `OPENAI_API_KEY` | Retained but unused in routing |
| `HETZNER_HOST/USER/SSH_KEY_PATH/PORT` | SSH to kaburusvr.uk via Tailscale |
| `TELEGRAM_TOKEN / TELEGRAM_CHAT_ID` | Telegram bot credentials |
| `BACKUP_CHECK_HOUR` | Hour for daily backup check (default 3) |
| `SECURITY_CHECK_INTERVAL` | Minutes between security checks (default 60) |
| `CHECK_INTERVAL_MINUTES` | Minutes between health checks (default 15) |
| `TRUENAS_HOST/USER/SSH_KEY_PATH` | TrueNAS SSH |
| `UBUNTU_SVR_HOST/USER/SSH_KEY_PATH` | ubuntu-svr SSH |
| `OPNSENSE_HOST/KEY/SECRET` | OPNsense API credentials |

### Management

```bash
cd /opt/hermes

# Status
docker ps | grep hermes

# Logs
docker logs hermes_agent --tail 50 -f
docker logs hermes_redis --tail 20

# Restart
docker restart hermes_agent

# Full rebuild (after code changes)
docker-compose down && docker-compose build --no-cache && docker-compose up -d

# Check redis state
docker exec hermes_redis redis-cli keys "hermes:*"

# Check SNMP trap log
tail -f /var/log/snmptrapd-events.log
```

---

## 2. Hermes Agent (NousResearch) — Production

Open-source autonomous agent from Nous Research. Installed on kaburuaibox.
Distinct from the Docker monitoring stack above.

**Install location:** `/usr/local/lib/hermes-agent/` (system-wide)
**User data:** `/home/kaburu/.hermes/`
**Binary:** `/usr/local/bin/hermes`
**Current version:** v0.15.1 (current HEAD as of 2026-06-03, 180 commits pulled)

### Services

Three systemd units, all boot-persistent:

| Service | User | Port | Purpose |
|---------|------|------|---------|
| `hermes-gateway.service` | kaburu | 8642 | Gateway + OpenAI-compatible API server |
| `hermes-dashboard.service` | kaburu | 9119 | Web dashboard + TUI chat |
| `hermes-mcpo.service` | root | 11435 | MCP-to-HTTP bridge for Open WebUI |

### Gateway (hermes-gateway.service)

Runs `hermes gateway run --replace` as kaburu.

Previous unit (`hermes-agent.service`) ran `hermes gateway start` which is a management
command, not a runner. It crash-looped 5711+ times. Fixed 2026-06-03.

Unit at `/etc/systemd/system/hermes-gateway.service`:
- `TimeoutStopSec=210` — matches `restart_drain_timeout=180s` + margin
- `StartLimitIntervalSec=120 / Burst=5` — in `[Unit]` section (not `[Service]`)
- `Wants=llama-server.service`

### Dashboard (hermes-dashboard.service)

Runs `hermes dashboard --host 0.0.0.0 --port 9119 --no-open --insecure --tui --skip-build`.

`--insecure` flag disables same-origin auth guard — required for LAN/Tailscale access.
Do NOT expose port 9119 to the internet.

Access: `http://192.168.0.253:9119` from any LAN or Tailscale device.
Install as PWA on phone/tablet for app-like experience.

### API server (port 8642)

Enabled via `display.platforms.api_server.enabled: true` in config.yaml.
Bound to `0.0.0.0:8642`. OpenAI-compatible API.

Used by:
- Nous Desktop app (remote mode)
- Any OpenAI-compatible frontend

Connection details for desktop app:
- URL: `http://192.168.0.253:8642`
- API key: stored in `/home/kaburu/.hermes/.env` as `API_SERVER_KEY`

For Tailscale access: substitute kaburuaibox Tailscale IP (100.120.18.44).

### Model configuration (config.yaml)

**Current (2026-08-03):** Primary is `deepseek-v4-flash` via DeepSeek API (`provider: deepseek`, base_url `https://api.deepseek.com/v1`, key `DEEPSEEK_API_KEY` in `.env`). Configured at Steve's request. Key verified against `GET /v1/models` — DeepSeek's API serves ONLY `deepseek-v4-flash` / `deepseek-v4-pro`; the dated `-0731` ID would 404 there. NOTE: `deepseek-v4-flash-0731` IS DeepSeek's production V4-Flash release (Jul 31, supersedes April preview; huge agentic gains — AAII 50 vs 40) — it's served under the STABLE ID `deepseek-v4-flash` per DeepSeek API docs ("The `deepseek-v4-flash` model has been updated to DeepSeek-V4-Flash-0731"). So the stable ID already runs the 0731 weights. Dated IDs exist only on HF/OpenRouter/Nous catalog. Verified end-to-end with one-shot `hermes chat -q` (PONG) through the config default. Cron model snapshots cleared (no pins — jobs inherit global model).

```yaml
model:
  default: deepseek-v4-flash
  provider: deepseek
  base_url: https://api.deepseek.com/v1
```

**Housekeeping (2026-08-03):** `updates.pre_update_backup: quick` (was legacy `true` = full zip of HERMES_HOME pre-update → 20GB zip pile-up, hung updates). ~/.hermes cleaned 32G → 4.2G: removed 3 old pre-update zips, orphaned 8.8GB GGUF, malformed state.db backup. Keep the newest pre-update zip + state-snapshots until update confirmed stable (~1.2GB floor after: state.db 0.75GB is legit session history). ⚠️ state.db index corruption — FIXED (dup rowid purged, unique index added). ✅ state.db ghost NULL cell — **PURGED 2026-08-06** (was "residual ghost NULL artifact benign"): ghost was NOT an index phantom — a real corrupt row (rowid 274877908374, NULL id + NULL started_at + 4050B blob source, dup-rowid class from Jul 30 torn write) invisible to index-backed queries (`WHERE started_at IS NULL` uses idx_sessions_started which lacked the entry) but visible to `WHERE id IS NULL` / `.dump`. Deleted the single cell via `DELETE WHERE rowid=274877908374` — integrity_check now **ok** (first clean since Jul 30), sessions 2962→2961, FTS in sync. No downtime. Pre-delete rollback: /home/kaburu/state.db.pre-ghost-delete-20260806.db. ✅ Nightly z840 backup permission errors — FIXED (midclt chown + `-rlptz` flags + symlink purge).

**History:** 2026-07-26 `z-ai/glm-5.1` via Nous Portal replaced unstable `qwen/qwen3.7-plus` (404'd twice, Nous catalog rotation). 2026-08-03 `deepseek-v4-flash` direct via DeepSeek API per Steve (backup: `~/.hermes/config.yaml.bak.deepseek.20260803171528`).

**Fallback chain** (in `~/.hermes/config.yaml`, must be added manually with proper YAML list syntax — `hermes config set` stringifies it):

```yaml
fallback_providers:
  - provider: kat
    model: KAT-Coder-V2.5-Dev-APEX-I-Mini
    base_url: http://127.0.0.1:11440/v1
  - provider: custom
    model: qwopus-mccoder:latest
```

**Why these specific fallbacks:**
- `KAT-Coder-V2.5-Dev-APEX-I-Mini` — **first local fallback** (2026-08-04). Qwen3.5 MoE coder, 13.5GB via llama-server on port 11440. **ENABLED persistently 2026-08-08** (`systemd enable --now llama-server-kat.service`) — survived reboot check, auto-loads at boot. Native 262K ctx, evaluated 46/50 on the ops eval (same as TC-McQwen) at 6.4× the speed. Replaces TC-McQwen as the preferred local ops/code model.
- `qwopus-mccoder:latest` — 27B, code/Divi capable, second-tier fallback
- GLM Reap (`glm47-reap:23b`) **REMOVED 2026-08-08** — Steve: "thick as shit, no context". Deleted from Ollama + GGUF on disk, removed from fallback chain, zero references in config.yaml. First fallback when the Nous/deepseek primary errors is now KAT directly.

Local Ollama custom providers still configured (`local-ollama`, `ollama-vision`) for when Ollama is manually started. Local models available: `qwen36-reap:28b`, `qwopus-mccoder:latest`, `mistral-nemo:12b`, `minicpm-v:8b`. (glm47-reap:23b removed 2026-08-08.)

**Removed 2026-07-26:** `qwen/qwen3.7-plus` from Nous Portal — closed-source, Nous rotates the ID without notice, mid-session 404s. Replaced with `z-ai/glm-5.1` (Z.ai, 200K context, stable).

**Cleaned 2026-07-08:** Removed OpenRouter (API key + config block), and earlier Nous-primary setup. Dead `local-llama` custom provider removed. `ollama-vision` → `minicpm-v:8b`.

`compression.enabled: true`. `model_catalog.url: ""` (disabled phoning home).

### .env (`/home/kaburu/.hermes/.env`)

| Variable | Value |
|----------|-------|
| `OLLAMA_BASE_URL` | http://localhost:11434 |
| `DEEPSEEK_API_KEY` | set |
| `XAI_API_KEY` | set (primary provider as of 2026-07-14) |
| `API_SERVER_KEY` | set — bearer token for port 8642 |
| `API_SERVER_HOST` | 0.0.0.0 |
| `API_SERVER_PORT` | 8642 |
| `GATEWAY_ALLOW_ALL_USERS` | true — open to all on LAN/Tailscale |
| `OPENROUTER_API_KEY` | REMOVED 2026-07-08 — OpenRouter provider deleted |

### MCP bridge (hermes-mcpo.service)

Wraps `hermes mcp serve` via mcpo, exposed on port 11435.
Consumed by Open WebUI as a tool server.

### Desktop app access (cross-device)

| Device | Method | URL |
|--------|--------|-----|
| Any browser (LAN/Tailscale) | Direct | http://192.168.0.253:9119 |
| Phone/tablet | PWA install | http://192.168.0.253:9119 |
| Nous Desktop app | Remote mode | http://192.168.0.253:8642 + API key |
| Tailscale (off-LAN) | Substitute Tailscale IP | http://100.120.18.44:9119 or :8642 |

---

## History

| Date | Event |
|------|-------|
| 2026-05-11 | Initial Docker agent build on ubuntu-svr |
| 2026-05-16 | Fail2Ban alert tuning (bantime 600→86400) |
| 2026-05-17 | Docker agent migrated to Z840 (kaburuaibox) |
| 2026-05-18 | Added TrueNAS, ubuntu-svr, Unraid, OPNsense monitoring |
| 2026-05-18 | SSH reconnect retry logic added to all connectors |
| 2026-05-19 | Update checker added (updates.py) |
| 2026-05-25 | SNMP trap receiver added. All nodes sending traps |
| 2026-05-25 | OPNsense SSH → HTTP API (hermes user, read-only) |
| 2026-05-29 | OPNsense connector rewritten |
| 2026-05-30 | Health check intervals reduced; alert-only mode |
| 2026-05-30 | KaburuLINK (Unraid) decommissioned |
| 2026-06-01 | NousResearch Hermes gateway crash-loop disabled (hermes-agent.service) |
| 2026-06-01 | Open WebUI connected to llama-server via OPENAI_API_BASE_URL |
| 2026-06-01 | Docker agent router.py rewritten (v3) — Claude removed, all tasks → qwen36-reap |
| 2026-06-01 | GLM VRAM OOM documented — cannot coexist with qwen36 in 24GB |
| 2026-06-03 | GLM removed from models.ini — qwen36-reap only model in llama-server |
| 2026-06-03 | NousResearch Hermes updated to current HEAD (180 commits) |
| 2026-06-03 | hermes-gateway.service rewritten — correct exec, TimeoutStopSec=210 |
| 2026-06-03 | hermes-dashboard.service created — port 9119, LAN/Tailscale access |
| 2026-06-03 | API server enabled on port 8642 — desktop app + cross-device access confirmed |
| 2026-06-03 | provider: openai → provider: custom in config.yaml (fixes "Unknown provider" error) |
| 2026-07-08 | Model cleanup: removed OpenRouter (API key + config), removed Nous Research refs (MoA, portal, catalog) |
| 2026-07-08 | Removed dead custom_provider local-llama (llama-server disabled). Fixed ollama-vision -> minicpm-v |
| 2026-07-08 | Ollama model purge: removed glm4:9b, moondream, mistral-nemo:128k. Kept glm47-reap, qwen36-reap |
| 2026-07-08 | qwopus-mccoder imported from GGUF into Ollama (Qwopus3.6-27B-v2-MTP-Q4_K_M, 16GB) |
| 2026-08-08 | KAT-Coder enabled persistently (llama-server-kat.service was disabled → didn't survive reboot). GLM Reap removed from Ollama + fallback chain ("thick as shit, no context"). kaburu.uk DB registered in CyberPanel, PHP limits fixed |
| 2026-06-03 | updates.py — switched to apt-get dry-run to filter phased packages |
| 2026-06-03 | check_backups() — now notifies on success, not just on failure |

## Notes

- Security check is conservative — only alerts on genuine issues, not SSH brute-force noise
- `PermitRootLogin=prohibit-root-only` on all servers — password brute-force always fails
- Tailscale 100.64.0.0/10 always whitelisted in Fail2Ban
- Docker agent logs persist on ZFS at `/kaburudata/hermes/logs/`
- Port 9119 (dashboard) must not be exposed to internet — no auth guard when `--insecure`
- Port 8642 (API) requires bearer token — safer to expose if needed, but Tailscale preferred

### MCP servers (config.yaml — command/stdio transport)

| Name | Script |
|------|--------|
| hetzner | /usr/local/bin/kaburu-shell-mcp-hetzner.py |
| kaburuaibox | /usr/local/bin/kaburu-shell-mcp.py |
| opnsense | /usr/local/bin/kaburu-shell-mcp-opnsense.py |
| truenas | /usr/local/bin/kaburu-shell-mcp-truenas.py |

ubuntu-svr, cyberpanel, cloudflare accessible via hetzner tool (SSH proxy).
approvals.cron_mode = auto — tools work unattended in cron jobs.

## Cron model assignments (2026-07-08)

| Model | Jobs | Schedule |
|-------|------|----------|
| **qwopus-mccoder** | ssl-check, update-check, wordpress-check | 06:00, 06:30, 07:30 daily |
| | docker-watch, lynis-review | Mon 09:00, 10:00 weekly |
| **glm47-reap:23b** | backup-check | 07:00 daily |
| | observer, reflector, pruner (memory pipeline) | every 4h, 03:00 daily, Sun 04:00 |
| **DeepSeek** | health-check, security-audit | hourly, every 6h |
| | daily-report, pagespeed-report | 08:00, 08:30 daily |
| **mistral-nemo:12b** | security-check (no_agent script) | every 30min |
| **minicpm-v:8b** | Always loaded (vision) | KEEP_ALIVE=-1 |
| **none (no_agent script)** | state-db-integrity (`0267b2efda86`) — PRAGMA integrity_check on ~/.hermes/state.db, silent when clean, 🚨 Telegram on failure. Detection ≤24h for torn-write corruption. Deployed 2026-08-06 (born from the ghost NULL cell incident) | 08:45 daily |

### VRAM strategy
minicpm-v:8b stays loaded 24/7 (4.9GB, partially on CPU when coexisting with large models).
qwopus (18GB) and glm47 (12GB) swap on-demand — ~2 min load from NVMe.
Only ONE large model loaded at a time — the scheduler handles unloading automatically.