Kaburu Wiki

# Hermes — Infrastructure Agent

Search engine: SearXNG at search.kaburu.cc (Docker on kaburusvr, LiteSpeed reverse proxy). Use for web searches.

Two separate systems share the name “Hermes” on this infrastructure. Do not confuse them.

—

## 1. Hermes Docker Agent (custom monitoring stack)

Automated monitoring, alerting, and management agent for the Kaburu infrastructure. Runs on kaburuaibox as a Docker Compose stack. Communicates via Telegram.

### Architecture

``` hermes_agent — Python scheduler + AI checks + Telegram bot hermes_redis — State store ```

Stack location: `/opt/hermes/` on kaburuaibox (192.168.0.253)

Note: The old `hermes_dashboard` FastAPI container was torn down 2026-06-01 (Hermes Workspace).
The NousResearch dashboard (port 9119) now serves as the web UI — see Section 2.

### Model routing (router.py v3 — last updated 2026-07-02)

All tasks route to llama-server (qwopus-mccoder) as primary. DeepSeek API as fallback if llama-server down. Claude/Anthropic removed entirely — `ANTHROPIC_API_KEY` is blank. `ollama.service` stopped and disabled 2026-07-14 to prevent phantom scheduled burns. Start manually when needed for interactive use. See log.md entry 2026-07-14 (Gwen) for full incident.

CRITICAL: No reasoning models for automation. `qwen36-reap:28b` and `glm47-reap:23b` perform “prompt processing” (internal thought tokens) at ~800 tok/s with zero output, consuming 75k+ tokens and 10+ GB VRAM per request. Gateway timeout 1800s + 3 retries = 90 min GPU burn per failed task. All recurring crons must use `no_agent: true` + bash scripts. Never route auxiliary tasks, cron, or fallback defaults to reasoning models.

OpenWebUI stopped and disabled 2026-07-14 (user declares emergency-only).

Task routing: - `local_capable` (qwopus-mccoder): security, anomaly, architecture, strategic, log analysis, code gen - `local_fast` (qwopus-mccoder): health checks, backups, reports, WordPress, maintenance, routine

Headroom Proxy at localhost:8787 provides context compression for all llama-server requests.

`_ClaudeStub` class in router.py redirects any `router.claude.messages.create()` calls (used in Telegram chat path in hermes.py ~line 564) to qwopus-mccoder with an Anthropic-compatible response object.

### Scheduled jobs

### Monitored hosts

### Update checker (updates.py — fixed 2026-06-03)

Uses `apt-get upgrade –dry-run | grep '^Inst'` to determine actually-installable updates. Previous version used `apt list –upgradable` which included phased packages (Ubuntu gradual rollout) that cannot actually be applied — this caused false “feature updates available” reports.

Phased packages only appear in the dry-run output if the specific machine is within the phase window — correct behaviour.

Categories: - 🔴 security — package source URL contains “security” - 🔵 feature — everything else actually installable

### Backup check (hermes.py — fixed 2026-06-03)

`check_backups()` now always sends a Telegram notification: - ✅ Success: `notify()` with LLM summary of the backup log - ⚠️ No log: `notify()` warning if `/var/log/hetzner-backup.log` not found on Hetzner - 🚨 Failed: `alert()` with LLM detail if ERROR in log and no completion marker

Previous version was silent on success — only alerted on failure.

### SNMP monitoring

Hermes receives SNMP traps from all nodes via snmptrapd on kaburuaibox (UDP 162). Trap log at `/var/log/snmptrapd-events.log`. `check_snmp()` polls every 5 minutes.

See snmp for full SNMP infrastructure documentation.

### Alert levels

### Telegram commands

### File structure

``` /opt/hermes/ ├── docker-compose.yml ├── .env ← API keys + credentials ├── config/ │ ├── hetzner_id_ed25519 ← SSH key for Hetzner │ ├── truenas_id_ed25519 ← SSH key for TrueNAS + ubuntu-svr │ ├── hermes_shell_id_ed25519 ← SSH key for OPNsense shell connector │ └── domains.txt ← domains to monitor SSL for └── agent/

  ├── hermes.py                 ← main scheduler + checks
  ├── router.py                 ← model routing (v3, llama-server primary)
  ├── hetzner.py                ← SSH connector to kaburusvr.uk
  ├── truenas.py                ← SSH connector to TrueNAS
  ├── ubuntu_svr.py             ← SSH connector to ubuntu-svr
  ├── unraid.py                 ← stub (KaburuLINK decommissioned)
  ├── opnsense.py               ← HTTP API connector (hermes user, read-only)
  ├── shell.py                  ← generic SSH shell connector
  ├── snmp_monitor.py           ← SNMP trap log watcher
  ├── updates.py                ← cross-server update checker
  ├── telegram_bot.py           ← send alerts/notifications
  ├── telegram_commands.py      ← receive /commands
  ├── wordpress.py              ← WP-CLI operations
  ├── memory.py                 ← Redis-backed memory/state
  └── requirements.txt

```

### Environment variables (`/opt/hermes/.env`)

### Management

```bash cd /opt/hermes

# Status docker ps | grep hermes

# Logs docker logs hermes_agent –tail 50 -f docker logs hermes_redis –tail 20

# Restart docker restart hermes_agent

# Full rebuild (after code changes) docker-compose down && docker-compose build –no-cache && docker-compose up -d

# Check redis state docker exec hermes_redis redis-cli keys “hermes:*”

# Check SNMP trap log tail -f /var/log/snmptrapd-events.log ```

—

## 2. Hermes Agent (NousResearch) — Production

Open-source autonomous agent from Nous Research. Installed on kaburuaibox. Distinct from the Docker monitoring stack above.

Install location: `/usr/local/lib/hermes-agent/` (system-wide) User data: `/home/kaburu/.hermes/` Binary: `/usr/local/bin/hermes` Current version: v0.15.1 (current HEAD as of 2026-06-03, 180 commits pulled)

### Services

Three systemd units, all boot-persistent:

### Gateway (hermes-gateway.service)

Runs `hermes gateway run –replace` as kaburu.

Previous unit (`hermes-agent.service`) ran `hermes gateway start` which is a management command, not a runner. It crash-looped 5711+ times. Fixed 2026-06-03.

Unit at `/etc/systemd/system/hermes-gateway.service`: - `TimeoutStopSec=210` — matches `restart_drain_timeout=180s` + margin - `StartLimitIntervalSec=120 / Burst=5` — in `[Unit]` section (not `[Service]`) - `Wants=llama-server.service`

### Dashboard (hermes-dashboard.service)

Runs `hermes dashboard –host 0.0.0.0 –port 9119 –no-open –insecure –tui –skip-build`.

`–insecure` flag disables same-origin auth guard — required for LAN/Tailscale access. Do NOT expose port 9119 to the internet.

Access: `http://192.168.0.253:9119` from any LAN or Tailscale device. Install as PWA on phone/tablet for app-like experience.

### API server (port 8642)

Enabled via `display.platforms.api_server.enabled: true` in config.yaml. Bound to `0.0.0.0:8642`. OpenAI-compatible API.

Used by: - Nous Desktop app (remote mode) - Any OpenAI-compatible frontend

Connection details for desktop app: - URL: `http://192.168.0.253:8642` - API key: stored in `/home/kaburu/.hermes/.env` as `API_SERVER_KEY`

For Tailscale access: substitute kaburuaibox Tailscale IP (100.120.18.44).

### Model configuration (config.yaml)

Current (2026-08-03): Primary is `deepseek-v4-flash` via DeepSeek API (`provider: deepseek`, base_url `https://api.deepseek.com/v1`, key `DEEPSEEK_API_KEY` in `.env`). Configured at Steve's request. Key verified against `GET /v1/models` — DeepSeek's API serves ONLY `deepseek-v4-flash` / `deepseek-v4-pro`; the dated `-0731` ID would 404 there. NOTE: `deepseek-v4-flash-0731` IS DeepSeek's production V4-Flash release (Jul 31, supersedes April preview; huge agentic gains — AAII 50 vs 40) — it's served under the STABLE ID `deepseek-v4-flash` per DeepSeek API docs (“The `deepseek-v4-flash` model has been updated to DeepSeek-V4-Flash-0731”). So the stable ID already runs the 0731 weights. Dated IDs exist only on HF/OpenRouter/Nous catalog. Verified end-to-end with one-shot `hermes chat -q` (PONG) through the config default. Cron model snapshots cleared (no pins — jobs inherit global model).

```yaml model:

default: deepseek-v4-flash
provider: deepseek
base_url: https://api.deepseek.com/v1

```

Housekeeping (2026-08-03): `updates.pre_update_backup: quick` (was legacy `true` = full zip of HERMES_HOME pre-update → 20GB zip pile-up, hung updates). ~/.hermes cleaned 32G → 4.2G: removed 3 old pre-update zips, orphaned 8.8GB GGUF, malformed state.db backup. Keep the newest pre-update zip + state-snapshots until update confirmed stable (~1.2GB floor after: state.db 0.75GB is legit session history). ⚠️ state.db index corruption — FIXED (dup rowid purged, unique index added). ✅ state.db ghost NULL cell — PURGED 2026-08-06 (was “residual ghost NULL artifact benign”): ghost was NOT an index phantom — a real corrupt row (rowid 274877908374, NULL id + NULL started_at + 4050B blob source, dup-rowid class from Jul 30 torn write) invisible to index-backed queries (`WHERE started_at IS NULL` uses idx_sessions_started which lacked the entry) but visible to `WHERE id IS NULL` / `.dump`. Deleted the single cell via `DELETE WHERE rowid=274877908374` — integrity_check now ok (first clean since Jul 30), sessions 2962→2961, FTS in sync. No downtime. Pre-delete rollback: /home/kaburu/state.db.pre-ghost-delete-20260806.db. ✅ Nightly z840 backup permission errors — FIXED (midclt chown + `-rlptz` flags + symlink purge).

History: 2026-07-26 `z-ai/glm-5.1` via Nous Portal replaced unstable `qwen/qwen3.7-plus` (404'd twice, Nous catalog rotation). 2026-08-03 `deepseek-v4-flash` direct via DeepSeek API per Steve (backup: `~/.hermes/config.yaml.bak.deepseek.20260803171528`).

Fallback chain (in `~/.hermes/config.yaml`, must be added manually with proper YAML list syntax — `hermes config set` stringifies it):

```yaml fallback_providers:

1. provider: kat

model: KAT-Coder-V2.5-Dev-APEX-I-Mini

  base_url: http://127.0.0.1:11440/v1
- provider: custom
  model: qwopus-mccoder:latest

```

Why these specific fallbacks: - `KAT-Coder-V2.5-Dev-APEX-I-Mini` — first local fallback (2026-08-04). Qwen3.5 MoE coder, 13.5GB via llama-server on port 11440. ENABLED persistently 2026-08-08 (`systemd enable –now llama-server-kat.service`) — survived reboot check, auto-loads at boot. Native 262K ctx, evaluated 46/50 on the ops eval (same as TC-McQwen) at 6.4× the speed. Replaces TC-McQwen as the preferred local ops/code model. - `qwopus-mccoder:latest` — 27B, code/Divi capable, second-tier fallback - GLM Reap (`glm47-reap:23b`) REMOVED 2026-08-08 — Steve: “thick as shit, no context”. Deleted from Ollama + GGUF on disk, removed from fallback chain, zero references in config.yaml. First fallback when the Nous/deepseek primary errors is now KAT directly.

Local Ollama custom providers still configured (`local-ollama`, `ollama-vision`) for when Ollama is manually started. Local models available: `qwen36-reap:28b`, `qwopus-mccoder:latest`, `mistral-nemo:12b`, `minicpm-v:8b`. (glm47-reap:23b removed 2026-08-08.)

Removed 2026-07-26: `qwen/qwen3.7-plus` from Nous Portal — closed-source, Nous rotates the ID without notice, mid-session 404s. Replaced with `z-ai/glm-5.1` (Z.ai, 200K context, stable).

Cleaned 2026-07-08: Removed OpenRouter (API key + config block), and earlier Nous-primary setup. Dead `local-llama` custom provider removed. `ollama-vision` → `minicpm-v:8b`.

`compression.enabled: true`. `model_catalog.url: ““` (disabled phoning home).

### .env (`/home/kaburu/.hermes/.env`)

### MCP bridge (hermes-mcpo.service)

Wraps `hermes mcp serve` via mcpo, exposed on port 11435. Consumed by Open WebUI as a tool server.

### Desktop app access (cross-device)

—

## History

## Notes

- Security check is conservative — only alerts on genuine issues, not SSH brute-force noise - `PermitRootLogin=prohibit-root-only` on all servers — password brute-force always fails - Tailscale 100.64.0.0/10 always whitelisted in Fail2Ban - Docker agent logs persist on ZFS at `/kaburudata/hermes/logs/` - Port 9119 (dashboard) must not be exposed to internet — no auth guard when `–insecure` - Port 8642 (API) requires bearer token — safer to expose if needed, but Tailscale preferred

### MCP servers (config.yaml — command/stdio transport)

ubuntu-svr, cyberpanel, cloudflare accessible via hetzner tool (SSH proxy). approvals.cron_mode = auto — tools work unattended in cron jobs.

## Cron model assignments (2026-07-08)

### VRAM strategy minicpm-v:8b stays loaded 24/7 (4.9GB, partially on CPU when coexisting with large models). qwopus (18GB) and glm47 (12GB) swap on-demand — ~2 min load from NVMe. Only ONE large model loaded at a time — the scheduler handles unloading automatically.