# CPU Usage Investigation — 2026-08-20

## Alert

kaburusvr CPU load 3.81 on 4 cores (Hetzner CCX23, 16GB RAM).

Top processes at alert time: - 2x lsphp rainbowvapes.co.uk/index.php: 100% + 91% CPU - 2x lsphp stats.kaburu.co: 35% + 32% CPU - 1x lsphp dm-ph.com/index.php: 75% CPU - ps (root): 200% CPU - 192 active HTTPS connections

Load recovered to 0.95 within minutes — burst, not sustained.

## Root Cause (3-agent consensus: Gwen + Qwen3.8 + Nemotron)

IP blocking fails because botnets rotate every 2-4 days. The CPU spikes are a combination of:

1. Bot traffic hitting PHP directly — not stopped at Cloudflare edge 2. Wordfence scans with unlimited duration — 3 sites had maxDuration=0, maxExecutionTime=0 (could run for hours during traffic peaks) 3. wp-cron contention — jafricasafari + rainbowvapes at */5, spawning PHP workers every 5 min 4. No per-site resource isolation — 25 sites sharing 4 cores, each allowed 10 concurrent workers + 400MB per process 5. Recurring brute-force attacks — botnets cycle every 2-4 days, 40-60 new IPs per burst

### What was NOT the cause

- LiteSpeed internal stats — the `lsphp stats` processes are serving `stats.kaburu.co` (a regular vhost / Matomo), NOT internal LiteSpeed statistics - Memory pressure — 6.5GB used, 10.5GB cached, 905MB swap — memory was fine - Redis — healthy at 63MB, not the bottleneck

## Analysis Agents

## Changes Applied

### Fix 1: Wordfence Scan Tuning ✅

Applied: 2026-08-20 06:51 UTC

Expected CPU savings: 20-40% SEO impact: None (scans are background processes) Risk: Low — too aggressive limits may cause “Scan incomplete” warnings

Rollback: ```sql – rainbowflavours UPDATE rainbowflavours.l9rc_options SET option_value='custom' WHERE option_name='wordfence_scanType'; UPDATE rainbowflavours.l9rc_options SET option_value='0' WHERE option_name='wordfence_scanMaxDuration'; UPDATE rainbowflavours.l9rc_options SET option_value='0' WHERE option_name='wordfence_maxExecutionTime';

– rainbowvapes UPDATE rainbowvapes_co_uk_wp.l9rc_options SET option_value='custom' WHERE option_name='wordfence_scanType'; UPDATE rainbowvapes_co_uk_wp.l9rc_options SET option_value='0' WHERE option_name='wordfence_scanMaxDuration'; UPDATE rainbowvapes_co_uk_wp.l9rc_options SET option_value='0' WHERE option_name='wordfence_maxExecutionTime';

– rf.kaburu.co UPDATE rainbowvapes.l9rc_options SET option_value='custom' WHERE option_name='wordfence_scanType'; UPDATE rainbowvapes.l9rc_options SET option_value='0' WHERE option_name='wordfence_scanMaxDuration'; UPDATE rainbowvapes.l9rc_options SET option_value='0' WHERE option_name='wordfence_maxExecutionTime'; ```

### Fix 2: Cloudflare Bot Fight Mode ⏳ PENDING — MANUAL

Status: Not yet applied — requires Cloudflare dashboard access

Steps: 1. Log in to Cloudflare dashboard 2. For each domain: Security → Settings → Bot Fight Mode → ON 3. Verify: Security → WAF → Tools → IP Access Rules — ensure “Search Engine Crawlers” is set to Allow 4. After enabling, check Security → Analytics for challenge rates

Expected CPU savings: 5-15% SEO impact: Low — Googlebot/Bingbot exempted by default (Cloudflare's “Known Bots” list) Risk: Low — JS challenge adds <50ms for bad bots, zero for real users

Rollback: Security → Settings → Bot Fight Mode → OFF

### Fix 3: Cloudflare Rate Limiting ⏳ PENDING — MANUAL

Status: Not yet applied — requires Cloudflare dashboard access

Steps (per domain): 1. Security → WAF → Rate limiting rules → Create rule 2. Rule 1: `wp-login.php` — 5 requests per 10 seconds → Challenge 3. Rule 2: `xmlrpc.php` — 2 requests per 10 seconds → Block 4. Rule 3: `wp-admin/` (exclude `/wp-admin/admin-ajax.php`) — 10 requests per 30 seconds → Challenge 5. Add Firewall Rule: `if (cf.client.bot) { allow }` (whitelists search engines)

Expected CPU savings: 10-25% SEO impact: Very low — admin URLs only, Googlebot doesn't hit them Risk: Low — mis-set thresholds could lock out legit admin users (monitor for 48h)

Rollback: Delete the rate limiting rules

### Fix 4: WP-Cron Frequency ✅

Applied: 2026-08-20 06:52 UTC

Note: Most other sites were already at */15 in root crontab. rainbowvapes had its own user-level crontab (rainb1286) which was separate.

Expected CPU savings: 5-10% SEO impact: None (wp-cron is internal) Risk: Low — WooCommerce subscriptions/stock updates tolerate 15-min delays

Rollback: ```bash # jafricasafari (root crontab) crontab -l | sed 's|*/15 * * * * wget -q -O - https://jafricasafari.com/|*/5 * * * * wget -q -O - https://jafricasafari.com/|' | crontab -

# rainbowvapes (user crontab) sudo -u rainb1286 crontab -l | sed 's|*/15|*/5|' | sudo -u rainb1286 crontab - ```

Or restore from backup: `crontab /tmp/crontab-backup-*`

### Fix 5: LiteSpeed Worker Caps ✅ (PARTIAL REVERT)

Applied: 2026-08-20 06:53 UTC + LiteSpeed restart Partially reverted: 2026-08-20 07:29 UTC — maxConns restored to 10 (was causing queue bottlenecks, 3 sites hitting 5-worker cap). procSoftLimit/procHardLimit changes kept.

Why partial revert: maxConns=5 caused queue bottlenecks — 3 sites (rainbowvapes, myretonmarquees, friendscic) hit the 5-worker cap with 202 active connections, spiking load to 6.34. The per-site vhosts already had maxConns=10 (the fix only changed the global extProcessor). The real CPU control is procSoftLimit (memory per process), not maxConns.

Expected CPU savings: 10-30% SEO impact: None (internal resource limits) Risk: MEDIUM — may cause 503 errors during genuine traffic spikes. Monitor for 48h.

Rollback: ```bash # Restore global config cp /tmp/litespeed-backup-20260820/httpd_config.conf /usr/local/lsws/conf/httpd_config.conf

# Restore per-site vhosts sed -i 's/procSoftLimit 200/procSoftLimit 400/' /usr/local/lsws/conf/vhosts/*/vhost.conf sed -i 's/procHardLimit 300/procHardLimit 500/' /usr/local/lsws/conf/vhosts/*/vhost.conf

# Restart LiteSpeed /usr/local/lsws/bin/lswsctrl restart ```

## Combined Expected Impact

## Monitoring Plan

- 48h watch: Check for 503 errors (LiteSpeed worker caps too tight) - 48h watch: Check Wordfence scan logs for “Scan incomplete” warnings - Weekly: Review Cloudflare Security Analytics for challenge rates - Ongoing: CPU alert script already runs every 2 min (`/usr/local/bin/cpu-alert.sh`)

## Key Learnings

1. IP blocking is whack-a-mole — botnets rotate every 2-4 days. Behavior-based blocking at Cloudflare edge is the real fix. 2. “Disable LiteSpeed stats” wouldn't help — the `lsphp stats` processes are serving stats.kaburu.co (Matomo), not internal stats. 3. Unlimited Wordfence scans are dangerous on shared hosting — always cap maxDuration and maxExecutionTime. 4. Per-site resource isolation matters — 25 sites × 10 workers = 250 potential PHP processes on 4 cores. 5. Three-agent analysis works — Gwen (infrastructure), Qwen3.8 (deep reasoning), Nemotron (SEO/security) each caught things the others missed.

## References

- Wordfence brute-force pattern: `wordpress-vulnerability-scanning/references/wfbrute-force-response-pattern.md` - LiteSpeed config: `/usr/local/lsws/conf/httpd_config.conf` - Per-site vhosts: `/usr/local/lsws/conf/vhosts/*/vhost.conf` - Config backups: `/tmp/litespeed-backup-20260820/` - Crontab backups: `/tmp/crontab-backup-*`, `/tmp/rainb1286-crontab-backup`