# Hetzner Upgrade — Recovery KB

**Purpose:** Step-by-step recovery when kaburusvr (Hetzner) fails to boot after a kernel or system update.

## Known Failure Modes

### 1. systemd Cyclic Dependency (Most Common)

**Symptom:** Server hangs at ~3 seconds into boot. Unreachable on both public IP (49.13.202.144) and Tailscale (100.112.54.2).

**Root cause:** `sysinit.target` enters infinite loop:
```
sysinit.target → cloud-init → networkd-wait-online → firewalld → dbus → basic.target → sockets.target → sysinit.target
```

**Prevention (applied 2026-07-29):**
- Drop-in overrides in `/etc/systemd/system/cloud-init.service.d/fix-cycle.conf` and `cloud-init-local.service.d/fix-cycle.conf` — clear `Before=sysinit.target`
- `snmptrapd.socket` masked via `systemctl mask`
- `GRUB_DEFAULT=0` in `/etc/default/grub` — boots newest kernel

**⚠️ These drop-ins survive apt upgrades** (they live in `/etc/systemd/system/` which dpkg doesn't touch), but verify after major cloud-init package updates.

### 2. Bad Kernel Selection

**Symptom:** GRUB menu shows, selected kernel panics.

**Historical example:**
- **Bad:** 6.8.0-135-generic (panic-booting)
- **Good:** 6.8.0-134-generic

**Prevention:** `GRUB_DEFAULT=0` boots newest kernel automatically. Never pin to a hardcoded kernel string.

### 3. UFW Boot Failure — DOCKER-USER Chain Missing

**Symptom:** Server boots normally but `ufw.service` shows `failed` in `systemctl status`. UFW rules partially applied — some ports open, some blocked.

**Root cause (identified 2026-08-11):** The DOCKER-USER iptables chain doesn't exist when UFW starts its `after.rules`. Docker creates this chain when its daemon starts, which is *after* UFW initializes. The rule was in `/etc/ufw/after.rules`:
```
-A DOCKER-USER -p tcp --dport 3001 ! -s 100.64.0.0/10 -j DROP
```
This caused: `iptables-restore: line 34 failed` → `Problem running '/etc/ufw/after.rules'` → UFW fails to start.

**Fix (applied 2026-08-11):**
- Removed the rule from `/etc/ufw/after.rules`
- Created systemd service `/etc/systemd/system/ufw-docker-3001.service`:
```ini
[Unit]
Description=UFW DOCKER-USER rule for Uptime Kuma
After=docker.service
Wants=docker.service

[Service]
Type=oneshot
ExecStart=/usr/sbin/iptables -I DOCKER-USER 1 -p tcp --dport 3001 ! -s 100.64.0.0/10 -j DROP

[Install]
WantedBy=multi-user.target
```
- Enabled: `systemctl enable ufw-docker-3001.service`
- UFW now starts cleanly; DOCKER-USER rule is applied post-Docker

**Prevention:** Never put Docker-specific iptables rules in `/etc/ufw/after.rules`. Use a dedicated systemd service with `After=docker.service` instead.

## Diagnosis Flow

```
kaburusvr unreachable?
├── Check public IP (49.13.202.144) AND Tailscale (100.112.54.2)
├── Both dead? → Hetzner Robot console
├── GRUB menu visible? → Select known-good kernel
├── Kernel panic? → Rescue console
└── Rescue console → Mount root, check journal
```

## Rescue Console Procedure

### Step 1: Activate Rescue

1. Hetzner Robot → **Rescue** tab
2. Click "Activate rescue system"
3. Note the root password (changes each activation)

### Step 2: SSH Into Rescue

```bash
# Clear host key (changes each activation)
ssh-keygen -R 49.13.202.144

# Connect
sshpass -p '<password>' ssh -o StrictHostKeyChecking=no root@49.13.202.144
```

**⚠️ Rescue keyboard is German QWERTZ:**
- `/` = Shift+7
- Or just SSH in — no keyboard needed

### Step 3: Mount Root & Inspect

```bash
# Mount root filesystem
mount /dev/sda1 /mnt

# Check journal from last boot
journalctl -D /mnt/var/log/journal --no-pager -b -1 | tail -80

# Look for these patterns:
# - "Unable to break cycle starting with sysinit.target"
# - "Failed to start Network Configuration"
# - kernel panic traces
```

### Step 4: Chroot & Fix

```bash
# Bind mount virtual filesystems
mount --bind /dev /mnt/dev
mount --bind /proc /mnt/proc
mount --bind /sys /mnt/sys

# Enter chroot
chroot /mnt

# Inside chroot — apply fixes:

# Option A: Fix cyclic dependency
cat > /etc/systemd/system/cloud-init.service.d/fix-cycle.conf << 'EOF'
[Unit]
Before=
EOF

cat > /etc/systemd/system/cloud-init-local.service.d/fix-cycle.conf << 'EOF'
[Unit]
Before=
EOF

systemctl daemon-reload

# Option B: Mask problematic services
systemctl mask snmptrapd.socket

# Option C: Fix GRUB if needed
cat /etc/default/grub | grep GRUB_DEFAULT
# Should show: GRUB_DEFAULT=0
# If wrong, edit and run: update-grub

# Exit chroot
exit

# Unmount
umount /mnt/dev /mnt/proc /mnt/sys
umount /mnt
```

### Step 5: Reboot

Hetzner Robot → **Control** tab → **Reboot**

## Post-Recovery Verification

```bash
# 1. Check basic connectivity
ping 49.13.202.144
ping 100.112.54.2

# 2. Verify CyberPanel
curl -k https://localhost:8090

# 3. Check a client site
curl -I https://kaburu.co.uk

# 4. Verify backup crontab
crontab -l | grep backup

# 5. Check systemd journal for errors
journalctl -p 0..3 --since "10 min ago"

# 6. Verify UFW rules intact
ufw status numbered

# 6b. Verify DOCKER-USER chain (Uptime Kuma :3001 Tailscale-only)
iptables -L DOCKER-USER -n --line-numbers
# Should show: DROP tcp dpt:3001 !100.64.0.0/10

# 7. Verify Tailscale
tailscale status
```

## Prevention Checklist

Before running `apt upgrade` on kaburusvr:

- [ ] Verify drop-ins exist: `ls /etc/systemd/system/cloud-init.service.d/fix-cycle.conf`
- [ ] Check GRUB default: `grep GRUB_DEFAULT /etc/default/grub`
- [ ] Confirm Tailscale is healthy (out-of-band access)
- [ ] Ensure recent backup exists on TrueNAS
- [ ] Verify UFW doesn't fail on boot: `systemctl is-active ufw`
- [ ] Verify DOCKER-USER rule present: `iptables -L DOCKER-USER -n`
- [ ] Note: Rescue password is generated fresh each activation

## Related

- [[kaburu-core-sop]] — Full SOP with all recovery scenarios
- [[kaburusvr]] — Server details and service inventory
- [[fail2ban]] — SSH brute-force protection (Tailscale-only SSH)
- Uptime Kuma `docker ps` shows `uptime-kuma-hetzner` container on port 3001