# Backup Routine

## Architecture

All boxes back up to TrueNAS (192.168.0.251) via Tailscale, then TrueNAS ZFS snapshots provide versioning.

## Box-by-Box

### Hetzner (kaburusvr) — Main Backup ✅ Running
- **Script:** `/root/backup.sh` on kaburusvr
- **Driver:** Hermes cron `kaburusvr-daily-backup` (job `b35c8dd3eb55`)
- **Hermes script:** `/home/kaburu/.hermes/scripts/kaburusvr-backup.sh` (on Z840) — SSHes in and runs `/root/backup.sh`
- **Schedule:** Daily 02:00 UTC via Hermes (staggered: main 02:00 → extended 02:45 → z840 03:30 → ubuntu-svr 04:15 → OPNsense 05:00)
- **What:**
  - MySQL all-databases dump (gzip, ~836MB)
  - rsync /home (all 23+ WordPress sites) → `/mnt/tank/backups/hetzner/home/`
  - rsync SSH keys, secrets, acme.sh, scripts, LiteSpeed config, Fail2Ban, Postfix
  - Monthly full on 1st of month → `/mnt/tank/backups/hetzner-monthly/`
- **Retention:** Daily DB x7, local DB x3
- **Log:** `/var/log/hetzner-backup.log` on kaburusvr

### Hetzner (kaburusvr) — Extended/DR Backup ✅ Running
- **Script:** `/root/extended-backup.sh` on kaburusvr
- **Driver:** Hermes cron `het-server-extended-backup` (job `aeb9f353feae`)
- **Hermes script:** `/home/kaburu/.hermes/scripts/hetzner-extended-backup.sh` (on Z840)
- **Schedule:** Daily 02:45 UTC — 45min after main backup, waits up to 30 min for main to finish
- **What (DR-critical configs NOT in main backup):**
  - `/var/lib/docker/` — Docker volumes, image layers, container metadata
  - `/usr/local/CyberCP/` — CyberPanel config
  - `/usr/local/lsws/conf/` — LiteSpeed web server config
  - `/var/vmail/` — Dovecot mailboxes (skipped if absent)
  - `/etc/postfix/`, `/etc/dovecot/`, `/etc/opendkim/` — mail config
  - `/etc/fail2ban/`, `/etc/ssh/` — security
  - `/etc/netplan/` — network config
  - `/etc/crontab` + `/var/spool/cron/crontabs/` — cron configs
  - Docker inspect dump for all containers → `containers-inspect-YYYY-MM-DD.json`
- **Bandwidth:** `--bwlimit=30m` to avoid stomping live services
- **Space gate:** Skips if TrueNAS free < 5GB (current: 8.7TB)
- **SSH target:** TrueNAS via Tailscale `100.74.178.14` (LAN 192.168.0.251 unreachable from Hetzner public network)
- **Auth:** `/root/.ssh/truenas_backup` key, `root@100.74.178.14` (Hetzner-side config)
- **Log:** `/var/log/hetzner-extended-backup.log` on kaburusvr
- **Total size:** ~4.1GB (CyberPanel 1.7G + Docker 668M + configs ~5M)

### ⚠️ Hetzner cron daemon (2026-07-26 — duplicate cron fixed)
- The Hetzner system crontab entry `0 2 * * * /root/backup.sh` was removed on 2026-07-26.
- It had been duplicating the Hermes cron since Jul 3 (when the Hermes cron was reinstated) — every day, both fired ~15s apart, causing rsync race conditions and partial DB sync failures.
- The Hetzner system crontab now only has the acme.sh cron and WP-cron wget calls — no backup.sh.
- Hetzner's `cron.service` has been observed to die silently (Jun 29 incident) — never rely on it for critical backups.

### TrueNAS (truenas) — ✅ Running
- **ZFS snapshots:** `/data/scripts/zfs-snapshot.sh`
  - Daily at 01:30
  - Every 6 hours
  - Weekly at 02:00 Sundays
- **Retention:** hourly x24, daily x7, weekly x4
- **Note:** `/usr/local/bin` is read-only on TrueNAS — scripts go to `/data/scripts/`

### Kaburuaibox (z840) — ✅ Now Fixed
- **Script:** `/usr/local/bin/z840-backup.sh`
- **Cron:** `0 3 * * *` (daily at 03:00)
- **What:** rsync /opt/hermes, docker volumes (excl. Ollama), /etc, /home/kaburu, /root, /kaburudata → `/mnt/tank/backups/z840/`

### Ubuntu-svr — ✅ Now Running
- **Script:** `/usr/local/bin/ubuntu-svr-backup.sh`
- **Cron:** `15 4 * * *` (daily at 04:15 UTC — 45min after z840)
- **What:** rsync frigate, omada, coral, /etc, /home/kaburu → `/mnt/tank/backups/ubuntu-svr/`
- **Retention:** 14 days (auto-cleaned by script)

### OPNsense — ✅ Running (NEW 2026-06-19)
- **Script:** `/home/kaburu/.hermes/scripts/opnsense-backup.sh` (runs on kaburuaibox)
- **Cron:** `0 5 * * *` (daily at 05:00 via Hermes cron — staggered 45min after ubuntu-svr)
- **What:** SSH to OPNsense, pull `/conf/config.xml`, package with version info, rsync to TrueNAS
- **Retention:** ZFS snapshots on TrueNAS handle versioning (daily@0130, 6hourly, weekly)
- **Note:** Config is ~142KB. No port forwards exist — pure outbound NAT setup. SSH via key auth to 192.168.0.1

## Backup Locations on TrueNAS

| Source | Path | Size | Last Updated |
|--------|------|------|--------------|
| hetzner (main) | `/mnt/tank/backups/hetzner/{home,databases,ssh-keys,secrets,acme,scripts,litespeed-conf,etc-fail2ban,etc-postfix}` | ~7.6GB | daily 02:00 |
| hetzner (extended/DR) | `/mnt/tank/backups/hetzner/{docker,cyberpanel,etc-dovecot,etc-opendkim,etc-ssh,etc-netplan,crontabs}` | ~4.1GB | daily 02:45 |
| hetzner-monthly | `/mnt/tank/backups/hetzner-monthly/` | 19G | 1st of month |
| opnsense | `/mnt/tank/backups/opnsense/` | 43K | daily 05:00 |
| ubuntu-svr | `/mnt/tank/backups/ubuntu-svr/` | 25G | daily 04:15 |
| z840 | `/mnt/tank/backups/z840/` | varies | daily 03:30 |

## DR Gaps
- ubuntu-svr has no automated daily backup yet (needs SSH key)
- No offsite backup beyond MEGA sync for hetzner config files

## ubuntu-svr → TrueNAS → kaburuaibox Replication (NEW 2026-06-08)
- **Backup script:** `/usr/local/bin/ubuntu-svr-backup.sh` (runs on ubuntu-svr)
- **Replication script:** `/data/scripts/truenas-replicate-to-z840.sh` (runs on TrueNAS)
- **Schedule:** Daily at 03:30 (backup) + automatic replication after backup completes
- **Destination on kaburuaibox:** `/mnt/tank/backups/remote/ubuntu-svr/`
- **Retention:** 14 days (auto-cleaned)

## SSH Keys
- **kaburusvr → TrueNAS:** `/root/.ssh/truenas_backup` (ed25519) — used by main + extended backup via Tailscale `100.74.178.14`
- **TrueNAS → kaburuaibox:** `/root/.ssh/truenas-replication` (RSA 4096-bit)
- **ubuntu-svr → TrueNAS:** `/home/kaburu/.ssh/hermes_id_ed25519` (ed25519)

## Cron Job Inventory (Hermes)
| Job ID | Name | Schedule | Script |
|--------|------|----------|--------|
| `b35c8dd3eb55` | kaburusvr-daily-backup | `0 2 * * *` | `kaburusvr-backup.sh` |
| `aeb9f353feae` | het-server-extended-backup | `45 2 * * *` | `hetzner-extended-backup.sh` |
| `7ab4e94e5c6a` | OPNsense config backup | `0 5 * * *` | `opnsense-backup.sh` |

## Health Monitoring
- **`health-check` cron (`c8933ec3090e`)** — runs every hour, checks both DB backup timestamp and log completion markers
- **`backup-check` cron (`ef277ee9495c`)** — paused 2026-07-18 (Ollama model); should be re-pointed to local bash script for true daily coverage