# Hetzner Upgrade — Recovery KB

Purpose: Step-by-step recovery when kaburusvr (Hetzner) fails to boot after a kernel or system update.

## Known Failure Modes

### 1. systemd Cyclic Dependency (Most Common)

Symptom: Server hangs at ~3 seconds into boot. Unreachable on both public IP (49.13.202.144) and Tailscale (100.112.54.2).

Root cause: `sysinit.target` enters infinite loop: ``` sysinit.target → cloud-init → networkd-wait-online → firewalld → dbus → basic.target → sockets.target → sysinit.target ```

Prevention (applied 2026-07-29): - Drop-in overrides in `/etc/systemd/system/cloud-init.service.d/fix-cycle.conf` and `cloud-init-local.service.d/fix-cycle.conf` — clear `Before=sysinit.target` - `snmptrapd.socket` masked via `systemctl mask` - `GRUB_DEFAULT=0` in `/etc/default/grub` — boots newest kernel

⚠️ These drop-ins survive apt upgrades (they live in `/etc/systemd/system/` which dpkg doesn't touch), but verify after major cloud-init package updates.

### 2. Bad Kernel Selection

Symptom: GRUB menu shows, selected kernel panics.

Historical example: - Bad: 6.8.0-135-generic (panic-booting) - Good: 6.8.0-134-generic

Prevention: `GRUB_DEFAULT=0` boots newest kernel automatically. Never pin to a hardcoded kernel string.

### 3. UFW Boot Failure — DOCKER-USER Chain Missing

Symptom: Server boots normally but `ufw.service` shows `failed` in `systemctl status`. UFW rules partially applied — some ports open, some blocked.

Root cause (identified 2026-08-11): The DOCKER-USER iptables chain doesn't exist when UFW starts its `after.rules`. Docker creates this chain when its daemon starts, which is *after* UFW initializes. The rule was in `/etc/ufw/after.rules`: ``` -A DOCKER-USER -p tcp –dport 3001 ! -s 100.64.0.0/10 -j DROP ``` This caused: `iptables-restore: line 34 failed` → `Problem running '/etc/ufw/after.rules'` → UFW fails to start.

Fix (applied 2026-08-11): - Removed the rule from `/etc/ufw/after.rules` - Created systemd service `/etc/systemd/system/ufw-docker-3001.service`: ```ini [Unit] Description=UFW DOCKER-USER rule for Uptime Kuma After=docker.service Wants=docker.service

[Service] Type=oneshot ExecStart=/usr/sbin/iptables -I DOCKER-USER 1 -p tcp –dport 3001 ! -s 100.64.0.0/10 -j DROP

[Install] WantedBy=multi-user.target ``` - Enabled: `systemctl enable ufw-docker-3001.service` - UFW now starts cleanly; DOCKER-USER rule is applied post-Docker

Prevention: Never put Docker-specific iptables rules in `/etc/ufw/after.rules`. Use a dedicated systemd service with `After=docker.service` instead.

## Diagnosis Flow

``` kaburusvr unreachable? ├── Check public IP (49.13.202.144) AND Tailscale (100.112.54.2) ├── Both dead? → Hetzner Robot console ├── GRUB menu visible? → Select known-good kernel ├── Kernel panic? → Rescue console └── Rescue console → Mount root, check journal ```

## Rescue Console Procedure

### Step 1: Activate Rescue

1. Hetzner Robot → Rescue tab 2. Click “Activate rescue system” 3. Note the root password (changes each activation)

### Step 2: SSH Into Rescue

```bash # Clear host key (changes each activation) ssh-keygen -R 49.13.202.144

# Connect sshpass -p '<password>' ssh -o StrictHostKeyChecking=no [email protected] ```

⚠️ Rescue keyboard is German QWERTZ: - `/` = Shift+7 - Or just SSH in — no keyboard needed

### Step 3: Mount Root & Inspect

```bash # Mount root filesystem mount /dev/sda1 /mnt

# Check journal from last boot journalctl -D /mnt/var/log/journal –no-pager -b -1 | tail -80

# Look for these patterns: # - “Unable to break cycle starting with sysinit.target” # - “Failed to start Network Configuration” # - kernel panic traces ```

### Step 4: Chroot & Fix

```bash # Bind mount virtual filesystems mount –bind /dev /mnt/dev mount –bind /proc /mnt/proc mount –bind /sys /mnt/sys

# Enter chroot chroot /mnt

# Inside chroot — apply fixes:

# Option A: Fix cyclic dependency cat > /etc/systemd/system/cloud-init.service.d/fix-cycle.conf « 'EOF' [Unit] Before= EOF

cat > /etc/systemd/system/cloud-init-local.service.d/fix-cycle.conf « 'EOF' [Unit] Before= EOF

systemctl daemon-reload

# Option B: Mask problematic services systemctl mask snmptrapd.socket

# Option C: Fix GRUB if needed cat /etc/default/grub | grep GRUB_DEFAULT # Should show: GRUB_DEFAULT=0 # If wrong, edit and run: update-grub

# Exit chroot exit

# Unmount umount /mnt/dev /mnt/proc /mnt/sys umount /mnt ```

### Step 5: Reboot

Hetzner Robot → Control tab → Reboot

## Post-Recovery Verification

```bash # 1. Check basic connectivity ping 49.13.202.144 ping 100.112.54.2

# 2. Verify CyberPanel curl -k https://localhost:8090

# 3. Check a client site curl -I https://kaburu.co.uk

# 4. Verify backup crontab crontab -l | grep backup

# 5. Check systemd journal for errors journalctl -p 0..3 –since “10 min ago”

# 6. Verify UFW rules intact ufw status numbered

# 6b. Verify DOCKER-USER chain (Uptime Kuma :3001 Tailscale-only) iptables -L DOCKER-USER -n –line-numbers # Should show: DROP tcp dpt:3001 !100.64.0.0/10

# 7. Verify Tailscale tailscale status ```

## Prevention Checklist

Before running `apt upgrade` on kaburusvr:

- [ ] Verify drop-ins exist: `ls /etc/systemd/system/cloud-init.service.d/fix-cycle.conf` - [ ] Check GRUB default: `grep GRUB_DEFAULT /etc/default/grub` - [ ] Confirm Tailscale is healthy (out-of-band access) - [ ] Ensure recent backup exists on TrueNAS - [ ] Verify UFW doesn't fail on boot: `systemctl is-active ufw` - [ ] Verify DOCKER-USER rule present: `iptables -L DOCKER-USER -n` - [ ] Note: Rescue password is generated fresh each activation

## Related

- kaburu-core-sop — Full SOP with all recovery scenarios - kaburusvr — Server details and service inventory - fail2ban — SSH brute-force protection (Tailscale-only SSH) - Uptime Kuma `docker ps` shows `uptime-kuma-hetzner` container on port 3001