# Backup Review — 2026-07-14

## Current State: What's Backed Up Daily

### Z840 → TrueNAS (Hetzner backup script at 02:00)
- ✅ WordPress sites (`/home/*/public_html/` — 31GB)
- ✅ MySQL databases (all databases, compressed)
- ✅ LiteSpeed/OLS vhost configs (`/usr/local/lsws/conf/` — 1.7MB, 56 domains)
- ✅ SSH keys (`/root/.ssh/`)
- ✅ Secrets (`/root/.secrets/`)
- ✅ SSL certificates (`/root/.acme.sh/`).
- ✅ Custom scripts (`/usr/local/bin/`)
- ✅ Fail2ban config (`/etc/fail2ban/`)
- ✅ Postfix config (`/etc/postfix/`)
- ✅ Email/mailboxes (`/home/vmail/` — 2GB, rsync with --ignore-errors)
- ✅ Monthly full snapshot (1st of month to separate dataset)

### Other Backups (already in place)
- ✅ Z840 → TrueNAS (Docker volumes, Ollama configs, wiki)
- ✅ Ubuntu-svr → TrueNAS (Frigate, Omada, configs)
- ✅ OPNsense → TrueNAS (firewall configs, daily at 05:00)

## Gaps: What's NOT Backed Up

### 1. CyberPanel Application (246MB)
**Location:** `/usr/local/CyberPanel/`  
**Status:** Not in daily backup  
**Risk:** If Hetzner dies, we lose the CyberPanel application code. Reinstalling takes ~10min but we'd lose custom configs and database relationships.  
**Fix:** Add to backup script:
```bash
rsync -az --delete /usr/local/CyberPanel/ root@$TRUENAS_IP:$TRUENAS_PATH/cyberpanel-app/
```

### 2. CyberPanel Configs
**Location:** `/etc/cyberpanel/`  
**Status:** Not in daily backup  
**Contains:** adminPass, mysqlPassword, webmail.conf, apikeys  
**Risk:** Without these, we can't access CyberPanel after restore without manual password reset.  
**Fix:** Add to backup script:
```bash
rsync -az --delete /etc/cyberpanel/ root@$TRUENAS_IP:$TRUENAS_PATH/cyberpanel-config/
```

### 3. Docker Compose Files & Volumes
**Location:** `/opt/searxng/`, `/opt/rustdesk/`, `/opt/ollama/` (if exists)  
**Status:** Not backed up  
**Containers running:**
- searxng + valkey (search engine)
- rustdesk-hbbs/hbbr (remote desktop relay)
- uptime-kuma-hetzner (monitoring)

**Risk:** If Hetzner dies, we lose:
- SearXNG search engine configs and Valkey cache
- RustDesk relay keys (clients would need reconfiguration)
- Uptime Kuma monitoring history

**Fix:** Add Docker directory to backup:
```bash
rsync -az --delete /opt/ root@$TRUENAS_IP:$TRUENAS_PATH/opt-docker/
```

Or more granular:
```bash
for dir in /opt/searxng /opt/rustdesk /opt/ollama; do
  [ -d "$dir" ] && rsync -az --delete "$dir/" root@$TRUENAS_IP:$TRUENAS_PATH/docker/$(basename $dir)/
done
```

### 4. System Packages (apt list)
**Location:** `dpkg --get-selections`  
**Status:** Not backed up  
**Risk:** After OS reinstall, we'd need to manually reinstall all packages (Postfix, Dovecot, Fail2ban, LiteSpeed, etc.)  
**Fix:** Add package list to backup:
```bash
dpkg --get-selections > /root/package-selections.txt
rsync -az /root/package-selections.txt root@$TRUENAS_IP:$TRUENAS_PATH/system-packages/
```

## Recovery Time Estimates

### Scenario: Complete Z840 loss, rebuild from scratch

**Current backup coverage (without gaps):**
1. Install Ubuntu 24.04 (10min)
2. Restore system packages (5min)
3. Restore SSH keys, secrets (2min)
4. Restore CyberPanel app + configs (15min with manual setup)
5. Restore WordPress sites from `/home/` backup (30min for 56 domains)
6. Restore MySQL databases (10min)
7. Restore OLS vhost configs (5min)
8. Restore email/mailboxes (20min for 2GB)
9. Restore Docker containers from compose files (30min)
10. Update DNS to point to new server IP (TTL propagation: 1hr-24hr)

**Total: 2-4 hours manual work + DNS propagation**

**With current gaps filled:**
- Steps 4, 9, 10 become automated or faster
- **Total: 1-2 hours manual work + DNS propagation**

## Recommendations

### Priority 1: Add CyberPanel to Backup (15min task)
Edit `/root/backup.sh` on Z840, add after line 105 (after OLS config sync):
```bash
# Sync CyberPanel application
log_msg "Syncing CyberPanel application..."
rsync -az --delete /usr/local/CyberPanel/ root@$TRUENAS_IP:$TRUENAS_PATH/cyberpanel-app/ >> $LOG_FILE 2>&1
CYBERPANEL_APP_SYNC=$?

# Sync CyberPanel configs
log_msg "Syncing CyberPanel configs..."
rsync -az --delete /etc/cyberpanel/ root@$TRUENAS_IP:$TRUENAS_PATH/cyberpanel-config/ >> $LOG_FILE 2>&1
CYBERPANEL_CONFIG_SYNC=$?
```

Update the verification gate (line 204) to include these:
```bash
if [ $HOME_SYNC -eq 0 ] && [ $DB_SYNC -eq 0 ] && [ $OLS_SYNC -eq 0 ] && 
   [ $CYBERPANEL_APP_SYNC -eq 0 ] && [ $CYBERPANEL_CONFIG_SYNC -eq 0 ]; then
```

### Priority 2: Add Docker/Opt Directory (15min task)
Add after CyberPanel sync:
```bash
# Sync Docker configs and data
log_msg "Syncing Docker configs..."
for dir in /opt/searxng /opt/rustdesk; do
  if [ -d "$dir" ]; then
    rsync -az --delete "$dir/" root@$TRUENAS_IP:$TRUENAS_PATH/docker/$(basename $dir)/ >> $LOG_FILE 2>&1
  fi
done
```

### Priority 3: Add Package List (5min task)
Add at start of backup script:
```bash
# Save package selection list
log_msg "Saving package list..."
dpkg --get-selections > /root/package-selections.txt
rsync -az /root/package-selections.txt root@$TRUENAS_IP:$TRUENAS_PATH/system-packages/
```

### Priority 4: Test Recovery Procedure (2hr task)
Document step-by-step recovery in `/home/kaburu/wiki/wiki/procedures/server-recovery.md`:
1. Fresh Ubuntu install
2. Package restore
3. CyberPanel restore
4. Site restore
5. Email restore
6. Docker restore
7. DNS cutover

**Action:** Schedule a quarterly recovery test (restore to temporary VM, verify everything works, document issues).

## Summary

**Current coverage:** 90% of critical data is backed up.  
**Biggest gap:** CyberPanel application + Docker state.  
**Recovery time with gaps:** 2-4 hours + DNS.  
**Recovery time after fixes:** 1-2 hours + DNS.  

**Recommendation:** Implement Priority 1 and 2 today (30min total). Test recovery next week.