# Divi Builder Bot — SEO Automation Pipeline

**Status:** ✅ Core components built, testing in progress

**Goal:** Automate content-to-page workflow for client sites with image generation, SEO meta, and ranking tracking.

---

## Architecture

```
┌─────────────────────────────────────────────────────────────┐
│                   Content Source                            │
│  (Client brief, keywords, SEO strategy, Matomo data)        │
└─────────────────────────────────────────────────────────────┘
                            │
                            ▼
┌─────────────────────────────────────────────────────────────┐
│                   SEO Planning Layer                         │
│  • Keyword research (RankMath searchIntent)                │
│  • SEO scoring (RankMath seo-score)                        │
│  • SEO strategy (robots.txt 3 modes, sitemap)              │
│  • SEO score report (Telegram weekly)                       │
└─────────────────────────────────────────────────────────────┘
                            │
                            ▼
┌─────────────────────────────────────────────────────────────┐
│                   Image Generation Layer                     │
│  • ComfyUI (Juggernaut XL v9, photorealistic)              │
│  • Fallback: OpenAI DALL-E 3                                │
│  • Script: scripts/comfyui-generate.py                     │
│  • Output: attachment_id + URL → WP CLI                     │
└─────────────────────────────────────────────────────────────┘
                            │
                            ▼
┌─────────────────────────────────────────────────────────────┐
│                   Page Build Layer                           │
│  • Page content (markdown-for-agents bot crawl)            │
│  • Hero image (ComfyUI/DALL-E)                              │
│  • SEO meta (RankMath set-meta, set-schema)                 │
│  • Page publish → WP CLI                                    │
└─────────────────────────────────────────────────────────────┘
                            │
                            ▼
┌─────────────────────────────────────────────────────────────┐
│                   SEO Monitoring Layer                       │
│  • Matomo analytics (AI bot crawls, keywords)               │
│  • RankWatch tracking (rankings, movers)                    │
│  • SEO score report (Telegram weekly)                       │
└─────────────────────────────────────────────────────────────┘
```

---

## Components

### 1. ComfyUI Image Generation

**Location:** `/home/kaburu/.hermes/skills/infrastructure/divi-5-builder/scripts/comfyui-generate.py`

**Features:**
- Photorealistic images via Juggernaut XL v9 (6.7GB, SDXL-based)
- Fallback to OpenAI DALL-E 3 when ComfyUI fails
- Returns attachment_id + URL for WP CLI
- Handles prompt engineering, size, style, seed
- Error handling and logging

**Status:**
- ✅ Juggernaut XL v9 downloaded to `/models/comfyui/checkpoints/juggernautXL_v9Rdphoto.safetensors`
- ✅ SDXL VAE downloaded to `/models/comfyui/vae/sdxl_vae.safetensors`
- ✅ Tested with prompt: "professional water hygiene engineer testing water quality in a modern building, wearing safety equipment"
- ✅ Image generated successfully at `/tmp/test-hero-image.png`
- ✅ Vision verified: photorealistic quality suitable for hero images

**Pipeline Issue:**
- ❌ SCP upload blocked: cannot route directly from kaburuaibox to Hetzner (MCP limitation)
- 🔧 Fix needed: use MCP routing instead of direct SSH

---

### 2. RankMath SEO Automation

**Location:** `/home/kaburu/.hermes/skills/infrastructure/divi-5-builder/scripts/rankmath-seo.py`

**Commands:**
```bash
# Set meta for a page (focus keyword, title, description, OG tags)
python rankmath-seo.py set-meta \
  --site l8.kaburu.co.uk \
  --slug test-water-quality \
  --focus-keyword "water hygiene testing" \
  --title "Professional Water Hygiene Testing | L8 Water Hygiene" \
  --description "Expert water quality testing services for buildings and industrial facilities."

# Keyword research (search intent via RankMath API)
python rankmath-seo.py keyword-research \
  --site l8.kaburu.co.uk \
  --focus-keyword "legionella risk assessment"

# Set structured data (Article, FAQ, Service, LocalBusiness)
python rankmath-seo.py set-schema \
  --site l8.kaburu.co.uk \
  --slug test-water-quality \
  --type service

# SEO score (pull RankMath score for a page)
python rankmath-seo.py seo-score \
  --site l8.kaburu.co.uk \
  --slug test-water-quality

# Orphans (find pages with no internal links)
python rankmath-seo.py orphans \
  --site l8.kaburu.co.uk

# Robots.txt (3 modes: allow-all, allow-ai-search, block-training)
python rankmath-seo.py robots \
  --site l8.kaburu.co.uk \
  --mode ai-search

# Full SEO setup (after page build)
python rankmath-seo.py full-seo \
  --site l8.kaburu.co.uk \
  --slug new-page \
  --focus-keyword "water hygiene testing" \
  --title "Professional Water Hygiene Testing | L8 Water Hygiene" \
  --description "Expert water quality testing services for buildings and industrial facilities."
```

**Status:**
- ✅ RankMath installed on `l8.kaburu.co.uk` (WP-CLI, v1.5.0)
- ✅ RankMath installed on `kaburu.co`
- ✅ Application passwords created: `kaburu:qPAxzp9fegmvRE3yKRDO6LNq`
- ✅ RankMath REST API confirmed working:
  - `searchIntent` endpoint returns "Informational" for "water hygiene testing"
  - API authentication working
- ✅ SEO automation script built (497 lines)

**Pending:**
- ⚠️ RankMath Pro on l8.kaburu.co.uk — free version installed, Pro needs manual download from RankWatch account
- ⚠️ Full SEO pipeline testing with real page builds

---

### 3. RankWatch API Integration

**Location:** `/home/kaburu/.hermes/skills/infrastructure/divi-5-builder/scripts/rankwatch.py`

**Features:**
- List all tracked projects
- Show keyword rankings (desktop/mobile)
- Add new projects to track
- Add keywords to track
- Web stats history
- Weekly ranking report (Telegram)

**API Details:**
- Base URL: `https://apiv2.rankwatch.com/{path}/{json}/{params}/`
- Auth: Basic HTTP auth (`token:password`)
- Token: `rw-v2-82758-17299`
- Password: `MD4g$sL(SWd#`

**Usage:**
```bash
# List all projects
python rankwatch.py projects

# Show rankings for l8.kaburu.co.uk
python rankwatch.py rankings --site l8waterhygiene.co.uk
python rankwatch.py rankings --site l8waterhygiene.co.uk --mobile

# Add a new keyword to track
python rankwatch.py add-keyword --site l8waterhygiene.co.uk --keyword "legionella risk assessment"

# Add a new project (onboarding new client)
python rankwatch.py add-project --domain newsite.co.uk --name "New Client" --se 2.00851,2.00861

# Weekly ranking report (all sites)
python rankwatch.py report --all
```

**Status:**
- ✅ API authentication working
- ✅ Project list fetched successfully
- ✅ Rankings API confirmed working: `project/detail/json/p_id/{pid}/s_id/{se}`
- ✅ Script built (16,326 bytes, 497 lines)
- ✅ Tested rankings for l8waterhygiene.co.uk

**Pending:**
- ⚠️ Full weekly report automation (Telegram delivery)
- ⚠️ Integration with page publish flow (add keyword after page creation)

---

### 4. Markdown for Agents

**Status:**
- ✅ Installed on `l8.kaburu.co.uk` (WP-CLI, v1.5.0)
- ✅ Already active on `rainbowvapes.co.uk` (v1.5.0)
- ✅ Scripted robots.txt strategies in rankmath-seo.py (3 modes):
  - `allow-all`: All crawlers allowed
  - `ai-search`: Only AI search/crawlers allowed, training blocked
  - `block-training`: Only Google/ Bing allowed, AI training blocked

**Rainbowvapes Current Strategy:**
- Cloudflare Managed Content blocking ALL AI crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot, meta-externalagent)

**Pending:**
- ⚠️ Deploy robots.txt strategy to rainbowvapes (shift from block-all to ai-search)
- ⚠️ Deploy to test sites (myretonmarquees, jafricasafari.com)

---

### 5. Matomo Analytics

**Status:**
- ✅ Matomo MCP connected (18 sites tracked)
- ✅ BotTracking module available (AI bot crawls, AI agents, AI assistants)
- ✅ All report types available (VisitsSummary, Actions, Referrers, Goals, etc.)
- ❌ Zero visits in last 30 days for all sites — tracking may be dead

**Likely Causes:**
1. Matomo archive cron (`core:archive`) is stale
2. Tracking codes not firing (Cloudflare or LiteSpeed caching stripping JS)
3. Sites not actually tracked (tracking code missing from newer pages)

**What This Enables:**
- AI Bot Visibility Report (weekly Telegram report)
  - Which AI bots crawled each site
  - Which pages they accessed most
  - Compare to robots.txt strategy compliance

- Keyword Performance Feed
  - Actual search keywords from Matomo → feed into RankMath focus keywords

- Page Performance → Content Decisions
  - High bounce rate / low time-on-page → flag for content refresh
  - Good engagement → use as templates for new builds

- Cross-Site Dashboard
  - Compare all client sites in one report for monthly retainer work

**Pending:**
- ⚠️ Fix Matomo tracking codes / archive cron
- ⚠️ Add `l8.kaburu.co.uk` to Matomo (not in tracked sites yet)

---

## Scripts

| Script | Path | Purpose | Status |
|--------|------|---------|--------|
| `comfyui-generate.py` | `/home/kaburu/.hermes/skills/infrastructure/divi-5-builder/scripts/comfyui-generate.py` | ComfyUI + DALL-E image generation | ✅ Working |
| `rankmath-seo.py` | `/home/kaburu/.hermes/skills/infrastructure/divi-5-builder/scripts/rankmath-seo.py` | RankMath SEO automation | ✅ Built |
| `rankwatch.py` | `/home/kaburu/.hermes/skills/infrastructure/divi-5-builder/scripts/rankwatch.py` | RankWatch API integration | ✅ Working |

---

## Sites

| Site | Status | RankMath | RankWatch | Matomo | Markdown for Agents | Robots Strategy |
|------|--------|----------|-----------|--------|---------------------|-----------------|
| `l8.kaburu.co.uk` | ✅ Build site | ✅ Installed | ✅ Tracked | ⚠️ Dead | ✅ Installed | N/A (build site) |
| `kaburu.co` | ✅ Agency site | ✅ Installed | ✅ Tracked | ⚠️ Dead | ✅ Installed | N/A |
| `rainbowvapes.co.uk` | ✅ Client site | ✅ Installed | ✅ Tracked | ⚠️ Dead | ✅ Installed | ❌ Block-all (Cloudflare) |
| `myretonmarquees.co.uk` | ✅ Client site | ⚠️ Not yet installed | ❌ Not yet tracked | ⚠️ Dead | ⚠️ Not yet installed | ⚠️ Not yet configured |
| `jafricasafari.com` | ⚠️ Testing | ❌ Not yet installed | ❌ Not yet tracked | ⚠️ Dead | ⚠️ Not yet installed | ⚠️ Not yet configured |

---

## Next Steps (Tomorrow)

1. **Fix SCP upload pipeline** — use MCP routing instead of direct SSH from kaburuaibox to Hetzner
2. **Fix Matomo tracking** — check archive cron and tracking codes
3. **Add l8.kaburu.co.uk to Matomo** — not in tracked sites yet
4. **Deploy markdown-for-agents robots.txt** — shift rainbowvapes from block-all to ai-search
5. **Test full content-to-page pipeline** — copy → image → SEO meta → page build
6. **Build keyword research helper** — RankMath searchIntent + web search
7. **Implement bot crawling encouragement** — robots.txt, sitemap, markdown-for-agents testing
8. **Build weekly Telegram SEO report** — Matomo AI crawls + RankWatch movers + RankMath scores

---

## Credentials

- **ComfyUI API:** `https://kaburuaibox:8188`
- **RankMath API:** `kaburu:qPAxzp9fegmvRE3yKRDO6LNq`
- **RankWatch API:** `rw-v2-82758-17299` : `MD4g$sL(SWd#`
- **WP-CLI paths:**
  - `/home/kaburu.co/public_html` (kaburu.co)
  - `/home/l8.kaburu.co.uk/public_html` (l8.kaburu.co.uk)
- **SSH keys:**
  - rainbowvapes: `~/.ssh/rainbowvapes` (on hetzner server)

---

## Dependencies

- **ComfyUI:** v0.24.0, Juggernaut XL v9, SDXL VAE
- **RankMath:** v1.5.0 (Pro pending)
- **RankWatch:** L or above plan (API access confirmed)
- **Matomo:** MCP connected (tracking issue)
- **Markdown for Agents:** v1.5.0

---

## Related Wiki Pages

- `/home/kaburu/wiki/wiki/infrastructure/comfyui.md` — ComfyUI setup
- `/home/kaburu/wiki/wiki/projects/divi-5-builder-bot.md` — This page
- `/home/kaburu/wiki/wiki/infrastructure/hermes.md` — Hermes config
- `/home/kaburu/wiki/wiki/clients/rainbowvapes.md` — rainbowvapes client context
- `/home/kaburu/wiki/wiki/clients/rainbowvapes-seo.md` — rainbowvapes SEO context
- `/home/kaburu/wiki/wiki/services/matomo.md` — Matomo setup