# Backup Routine

## Architecture

All boxes back up to TrueNAS (192.168.0.251) via Tailscale, then TrueNAS ZFS snapshots provide versioning.

## Box-by-Box

### Hetzner (kaburusvr) — Main Backup ✅ Running - Script: `/root/backup.sh` on kaburusvr - Driver: Hermes cron `kaburusvr-daily-backup` (job `b35c8dd3eb55`) - Hermes script: `/home/kaburu/.hermes/scripts/kaburusvr-backup.sh` (on Z840) — SSHes in and runs `/root/backup.sh` - Schedule: Daily 02:00 UTC via Hermes (staggered: main 02:00 → extended 02:45 → z840 03:30 → ubuntu-svr 04:15 → OPNsense 05:00) - What:

1. MySQL all-databases dump (gzip, ~836MB)
2. rsync /home (all 23+ WordPress sites) → `/mnt/tank/backups/hetzner/home/`
3. rsync SSH keys, secrets, acme.sh, scripts, LiteSpeed config, Fail2Ban, Postfix
4. Monthly full on 1st of month → `/mnt/tank/backups/hetzner-monthly/`

- Retention: Daily DB x7, local DB x3 - Log: `/var/log/hetzner-backup.log` on kaburusvr

### Hetzner (kaburusvr) — Extended/DR Backup ✅ Running - Script: `/root/extended-backup.sh` on kaburusvr - Driver: Hermes cron `het-server-extended-backup` (job `aeb9f353feae`) - Hermes script: `/home/kaburu/.hermes/scripts/hetzner-extended-backup.sh` (on Z840) - Schedule: Daily 02:45 UTC — 45min after main backup, waits up to 30 min for main to finish - What (DR-critical configs NOT in main backup):

1. `/var/lib/docker/` — Docker volumes, image layers, container metadata
2. `/usr/local/CyberCP/` — CyberPanel config
3. `/usr/local/lsws/conf/` — LiteSpeed web server config
4. `/var/vmail/` — Dovecot mailboxes (skipped if absent)
5. `/etc/postfix/`, `/etc/dovecot/`, `/etc/opendkim/` — mail config
6. `/etc/fail2ban/`, `/etc/ssh/` — security
7. `/etc/netplan/` — network config
8. `/etc/crontab` + `/var/spool/cron/crontabs/` — cron configs
9. Docker inspect dump for all containers → `containers-inspect-YYYY-MM-DD.json`

- Bandwidth: `–bwlimit=30m` to avoid stomping live services - Space gate: Skips if TrueNAS free < 5GB (current: 8.7TB) - SSH target: TrueNAS via Tailscale `100.74.178.14` (LAN 192.168.0.251 unreachable from Hetzner public network) - Auth: `/root/.ssh/truenas_backup` key, `[email protected]` (Hetzner-side config) - Log: `/var/log/hetzner-extended-backup.log` on kaburusvr - Total size: ~4.1GB (CyberPanel 1.7G + Docker 668M + configs ~5M)

### ⚠️ Hetzner cron daemon (2026-07-26 — duplicate cron fixed) - The Hetzner system crontab entry `0 2 * * * /root/backup.sh` was removed on 2026-07-26. - It had been duplicating the Hermes cron since Jul 3 (when the Hermes cron was reinstated) — every day, both fired ~15s apart, causing rsync race conditions and partial DB sync failures. - The Hetzner system crontab now only has the acme.sh cron and WP-cron wget calls — no backup.sh. - Hetzner's `cron.service` has been observed to die silently (Jun 29 incident) — never rely on it for critical backups.

### TrueNAS (truenas) — ✅ Running - ZFS snapshots: `/data/scripts/zfs-snapshot.sh`

1. Daily at 01:30
2. Every 6 hours
3. Weekly at 02:00 Sundays

- Retention: hourly x24, daily x7, weekly x4 - Note: `/usr/local/bin` is read-only on TrueNAS — scripts go to `/data/scripts/`

### Kaburuaibox (z840) — ✅ Now Fixed - Script: `/usr/local/bin/z840-backup.sh` - Cron: `0 3 * * *` (daily at 03:00) - What: rsync /opt/hermes, docker volumes (excl. Ollama), /etc, /home/kaburu, /root, /kaburudata → `/mnt/tank/backups/z840/`

### Ubuntu-svr — ✅ Now Running - Script: `/usr/local/bin/ubuntu-svr-backup.sh` - Cron: `15 4 * * *` (daily at 04:15 UTC — 45min after z840) - What: rsync frigate, omada, coral, /etc, /home/kaburu → `/mnt/tank/backups/ubuntu-svr/` - Retention: 14 days (auto-cleaned by script)

### OPNsense — ✅ Running (NEW 2026-06-19) - Script: `/home/kaburu/.hermes/scripts/opnsense-backup.sh` (runs on kaburuaibox) - Cron: `0 5 * * *` (daily at 05:00 via Hermes cron — staggered 45min after ubuntu-svr) - What: SSH to OPNsense, pull `/conf/config.xml`, package with version info, rsync to TrueNAS - Retention: ZFS snapshots on TrueNAS handle versioning (daily@0130, 6hourly, weekly) - Note: Config is ~142KB. No port forwards exist — pure outbound NAT setup. SSH via key auth to 192.168.0.1

## Backup Locations on TrueNAS

## DR Gaps - ubuntu-svr has no automated daily backup yet (needs SSH key) - No offsite backup beyond MEGA sync for hetzner config files

## ubuntu-svr → TrueNAS → kaburuaibox Replication (NEW 2026-06-08) - Backup script: `/usr/local/bin/ubuntu-svr-backup.sh` (runs on ubuntu-svr) - Replication script: `/data/scripts/truenas-replicate-to-z840.sh` (runs on TrueNAS) - Schedule: Daily at 03:30 (backup) + automatic replication after backup completes - Destination on kaburuaibox: `/mnt/tank/backups/remote/ubuntu-svr/` - Retention: 14 days (auto-cleaned)

## SSH Keys - kaburusvr → TrueNAS: `/root/.ssh/truenas_backup` (ed25519) — used by main + extended backup via Tailscale `100.74.178.14` - TrueNAS → kaburuaibox: `/root/.ssh/truenas-replication` (RSA 4096-bit) - ubuntu-svr → TrueNAS: `/home/kaburu/.ssh/hermes_id_ed25519` (ed25519)

## Cron Job Inventory (Hermes)

## Health Monitoring - `health-check` cron (`c8933ec3090e`) — runs every hour, checks both DB backup timestamp and log completion markers - `backup-check` cron (`ef277ee9495c`) — paused 2026-07-18 (Ollama model); should be re-pointed to local bash script for true daily coverage