# Hetzner Upgrade — Recovery KB
Purpose: Step-by-step recovery when kaburusvr (Hetzner) fails to boot after a kernel or system update.
## Known Failure Modes
### 1. systemd Cyclic Dependency (Most Common)
Symptom: Server hangs at ~3 seconds into boot. Unreachable on both public IP (49.13.202.144) and Tailscale (100.112.54.2).
Root cause: `sysinit.target` enters infinite loop: ``` sysinit.target → cloud-init → networkd-wait-online → firewalld → dbus → basic.target → sockets.target → sysinit.target ```
Prevention (applied 2026-07-29): - Drop-in overrides in `/etc/systemd/system/cloud-init.service.d/fix-cycle.conf` and `cloud-init-local.service.d/fix-cycle.conf` — clear `Before=sysinit.target` - `snmptrapd.socket` masked via `systemctl mask` - `GRUB_DEFAULT=0` in `/etc/default/grub` — boots newest kernel
⚠️ These drop-ins survive apt upgrades (they live in `/etc/systemd/system/` which dpkg doesn't touch), but verify after major cloud-init package updates.
### 2. Bad Kernel Selection
Symptom: GRUB menu shows, selected kernel panics.
Historical example: - Bad: 6.8.0-135-generic (panic-booting) - Good: 6.8.0-134-generic
Prevention: `GRUB_DEFAULT=0` boots newest kernel automatically. Never pin to a hardcoded kernel string.
### 3. UFW Boot Failure — DOCKER-USER Chain Missing
Symptom: Server boots normally but `ufw.service` shows `failed` in `systemctl status`. UFW rules partially applied — some ports open, some blocked.
Root cause (identified 2026-08-11): The DOCKER-USER iptables chain doesn't exist when UFW starts its `after.rules`. Docker creates this chain when its daemon starts, which is *after* UFW initializes. The rule was in `/etc/ufw/after.rules`: ``` -A DOCKER-USER -p tcp –dport 3001 ! -s 100.64.0.0/10 -j DROP ``` This caused: `iptables-restore: line 34 failed` → `Problem running '/etc/ufw/after.rules'` → UFW fails to start.
Fix (applied 2026-08-11): - Removed the rule from `/etc/ufw/after.rules` - Created systemd service `/etc/systemd/system/ufw-docker-3001.service`: ```ini [Unit] Description=UFW DOCKER-USER rule for Uptime Kuma After=docker.service Wants=docker.service
[Service] Type=oneshot ExecStart=/usr/sbin/iptables -I DOCKER-USER 1 -p tcp –dport 3001 ! -s 100.64.0.0/10 -j DROP
[Install] WantedBy=multi-user.target ``` - Enabled: `systemctl enable ufw-docker-3001.service` - UFW now starts cleanly; DOCKER-USER rule is applied post-Docker
Prevention: Never put Docker-specific iptables rules in `/etc/ufw/after.rules`. Use a dedicated systemd service with `After=docker.service` instead.
## Diagnosis Flow
``` kaburusvr unreachable? ├── Check public IP (49.13.202.144) AND Tailscale (100.112.54.2) ├── Both dead? → Hetzner Robot console ├── GRUB menu visible? → Select known-good kernel ├── Kernel panic? → Rescue console └── Rescue console → Mount root, check journal ```
## Rescue Console Procedure
### Step 1: Activate Rescue
1. Hetzner Robot → Rescue tab 2. Click “Activate rescue system” 3. Note the root password (changes each activation)
### Step 2: SSH Into Rescue
```bash # Clear host key (changes each activation) ssh-keygen -R 49.13.202.144
# Connect sshpass -p '<password>' ssh -o StrictHostKeyChecking=no [email protected] ```
⚠️ Rescue keyboard is German QWERTZ: - `/` = Shift+7 - Or just SSH in — no keyboard needed
### Step 3: Mount Root & Inspect
```bash # Mount root filesystem mount /dev/sda1 /mnt
# Check journal from last boot journalctl -D /mnt/var/log/journal –no-pager -b -1 | tail -80
# Look for these patterns: # - “Unable to break cycle starting with sysinit.target” # - “Failed to start Network Configuration” # - kernel panic traces ```
### Step 4: Chroot & Fix
```bash # Bind mount virtual filesystems mount –bind /dev /mnt/dev mount –bind /proc /mnt/proc mount –bind /sys /mnt/sys
# Enter chroot chroot /mnt
# Inside chroot — apply fixes:
# Option A: Fix cyclic dependency cat > /etc/systemd/system/cloud-init.service.d/fix-cycle.conf « 'EOF' [Unit] Before= EOF
cat > /etc/systemd/system/cloud-init-local.service.d/fix-cycle.conf « 'EOF' [Unit] Before= EOF
systemctl daemon-reload
# Option B: Mask problematic services systemctl mask snmptrapd.socket
# Option C: Fix GRUB if needed cat /etc/default/grub | grep GRUB_DEFAULT # Should show: GRUB_DEFAULT=0 # If wrong, edit and run: update-grub
# Exit chroot exit
# Unmount umount /mnt/dev /mnt/proc /mnt/sys umount /mnt ```
### Step 5: Reboot
Hetzner Robot → Control tab → Reboot
## Post-Recovery Verification
```bash # 1. Check basic connectivity ping 49.13.202.144 ping 100.112.54.2
# 2. Verify CyberPanel curl -k https://localhost:8090
# 3. Check a client site curl -I https://kaburu.co.uk
# 4. Verify backup crontab crontab -l | grep backup
# 5. Check systemd journal for errors journalctl -p 0..3 –since “10 min ago”
# 6. Verify UFW rules intact ufw status numbered
# 6b. Verify DOCKER-USER chain (Uptime Kuma :3001 Tailscale-only) iptables -L DOCKER-USER -n –line-numbers # Should show: DROP tcp dpt:3001 !100.64.0.0/10
# 7. Verify Tailscale tailscale status ```
## Prevention Checklist
Before running `apt upgrade` on kaburusvr:
- [ ] Verify drop-ins exist: `ls /etc/systemd/system/cloud-init.service.d/fix-cycle.conf` - [ ] Check GRUB default: `grep GRUB_DEFAULT /etc/default/grub` - [ ] Confirm Tailscale is healthy (out-of-band access) - [ ] Ensure recent backup exists on TrueNAS - [ ] Verify UFW doesn't fail on boot: `systemctl is-active ufw` - [ ] Verify DOCKER-USER rule present: `iptables -L DOCKER-USER -n` - [ ] Note: Rescue password is generated fresh each activation
## Related
- kaburu-core-sop — Full SOP with all recovery scenarios - kaburusvr — Server details and service inventory - fail2ban — SSH brute-force protection (Tailscale-only SSH) - Uptime Kuma `docker ps` shows `uptime-kuma-hetzner` container on port 3001