# Hermes — Memory and Persistence

Hermes is designed to learn and accumulate knowledge over time, not just react to point-in-time checks. This page documents how memory works, what persists, and how the agents use it.

## Memory tiers

## What Hermes learns

### Baselines (learned normal values) Hermes uses an exponential moving average (EMA, α=0.1) to build a learned sense of normal for each host metric. Each check cycle updates the baseline:

``` new_baseline = 0.1 × current_value + 0.9 × old_baseline ```

This means baselines shift slowly — a single spike doesn't change what “normal” means. After ~20 check cycles a baseline is well-established. After ~50 it's very stable.

Baselines are stored as `hermes:baseline:<host>:<metric>` in Redis (90 day TTL). They are injected into every health check prompt so the AI can say “load is 2.1 which is above the learned baseline of 0.4” rather than alerting on absolute values.

### Memory categories

## Persistence architecture

``` kaburuaibox ZFS pool: kaburudata ├── kaburudata/hermes/redis/ ← Redis data dir (AOF + RDB) │ ├── dump.rdb ← RDB snapshot (every 60s if 1000+ changes) │ └── appendonlydir/ ← AOF files (fsync every second) ├── kaburudata/hermes/memory/ ← Long-term memory markdown files │ ├── incidents.md │ ├── anomalies.md │ ├── resolutions.md │ ├── baselines.md │ ├── patterns.md │ ├── actions.md │ └── observations.md └── kaburudata/hermes/logs/ ← Agent logs (hermes.log) ```

All three datasets are on ZFS with automatic snapshots (hourly x24, daily x7, weekly x4). In a disaster recovery scenario, Redis state, memory files, and logs all survive independently of the Docker containers and volumes.

## Redis configuration

``` appendonly yes ← Write-ahead log — max 1 second of data loss on crash appendfsync everysec ← fsync every second (balance between safety and performance) save 900 1 ← RDB snapshot if 1 change in 15 min save 300 10 ← RDB snapshot if 10 changes in 5 min save 60 1000 ← RDB snapshot if 1000 changes in 1 min ```

Previously Redis data lived in a Docker-managed volume on `/dev/sda2` (local disk, no ZFS). Migrated to ZFS on 2026-05-25.

## How memory is used in AI prompts

Every AI check call receives a memory context block:

``` MEMORY CONTEXT:

RECENT INCIDENTS:
  [2026-05-25 11:51] ubuntu-svr: Hard lockup — required physical reboot
  [2026-05-25 12:10] network: SNMP trap received — coldStart from OPNsense

RECENT ANOMALIES:
  (none currently)

LEARNED BASELINES (normal values):
  kaburusvr load_1min: 0.51
  kaburuaibox gpu_temp: 38.0
  ubuntu-svr mem_percent: 42.0

```

This allows the AI to: - Avoid false alerts on values that are historically normal for this infrastructure - Recognise when something is genuinely elevated vs background noise - Reference previous incidents when diagnosing recurring issues

## Chat workspace memory

The Hermes chat interface (dashboard Workspace tab) loads: 1. Current status snapshot — all 12 Redis check keys (health, security, uptime, etc.) 2. Wiki index — fetched live from kaburusvr, so Hermes knows what documentation exists 3. Memory context — recent incidents, anomalies, resolutions, and learned baselines 4. Infrastructure overview — hardcoded in system prompt with all 6 nodes, IPs, roles

Hermes can read any wiki page on demand during a chat session using the Hetzner connector.

## Weekly memory sync to wiki

Every Sunday at 08:00 (during daily report), Hermes syncs the long-term memory files to the wiki on Hetzner as `wiki/infrastructure/hermes-memory-<category>.md`. These are git-committed automatically and serve as a permanent searchable incident log.

## Backup

Long-term memory is included in the Z840 nightly backup to TrueNAS: ``` /kaburudata/hermes/ → rsync → /mnt/tank/backups/z840/ ```

Redis AOF/RDB files are also included. Full recovery possible from TrueNAS backup alone.

## History