# Local model fine-tune plan

**Goal:** Stop re-stuffing the whole wiki into every session. Bake stable Kaburu knowledge into a local LoRA so cold start is a **small changelog** + tools, not 30k tokens of topology.

**Non-goal:** Replace live wiki/tools. Weights = defaults and discipline. Tools/wiki = truth.

---

## Phase 0 — No train (1 evening)

Before any GPU burn, test whether a **distilled system block** already fixes most of the pain.

1. Build `kaburu-core-sop.md` (~2–4k tokens) from stable wiki only:
   - topology + how to reach each node
   - hard rules (no ping OPNsense, no root WP file writes, Divi gate, Immich pin, etc.)
   - backup paths at high level (no keys)
2. Hermes cold start = that block + `changelog-7d.md` only
3. Score against eval set (below)
4. If constraint violations drop hard → fine-tune is optional polish  
   If still mushy / forgets rules mid-session → proceed Phase 1

---

## Phase 1 — Single adapter proof (in-house)

### Hardware reality (Z840)

| Resource | Value |
|----------|--------|
| GPUs | 2× RTX 3060 **12 GB** (24 GB total) |
| RAM | ~125 GB |
| Serve sweet spot | Q4 ~27B weights (~16 GB) + **num_ctx 65536** + quantized KV |
| Train on-box | QLoRA on **8–14B** comfortable; **27B QLoRA** tight/slow — possible with care, not pleasant |
| Prefer | Train smaller or off-box LoRA → **merge/export GGUF** → serve 27B local |

Ollama stays **disabled on boot**; start only for interactive/build/train windows.

### Base model shortlist (local already)

| Model | On disk | Role | Notes |
|-------|---------|------|--------|
| **TC-McQwen:latest** (`ThinkingCap-Qwen3.6-27B-Q4_K_M`) | ✅ 16G | **Primary ops candidate** | Qwen3.5 arch, tools+thinking, native **262k**, runtime **65k**. Fewer think tokens. |
| **qwopus-mccoder:latest** (`Qwopus3.6-27B-v2-MTP-Q4_K_M`) | ✅ 16G | **Primary project/code candidate** | Same family, code-focused system prompt, MTP, tools+thinking, 65k. |
| qwen36-reap:28b | ✅ 17G | Avoid for ops brain FT | MoE REAP/reasoning — burns context, bad “always-on SOP” target. Keep separate if needed. |
| glm47-reap:23b | ✅ 12G | Secondary / lighter serve | Non-27B option; already clamped **num_ctx 65536**. Weaker code than Qwopus. |
| minicpm-v:8b | ✅ | Vision only | Not agent brain. |

**Earmarked pair for split (Phase 3):**

1. **Ops brain** ← start from **TC-McQwen** (or glm47 if 27B train too heavy)  
2. **Project brain** ← start from **qwopus-mccoder**

### HuggingFace look — “better Q4 + long context?”

Native long context is already solved on what you have: TC-McQwen / Qwopus advertise **262k** (YaRN-class); we run **65k** for VRAM.

| Option | Why consider | Why not jump yet |
|--------|----------------|------------------|
| [unsloth/Qwen3.6-27B-GGUF](https://huggingface.co/unsloth/Qwen3.6-27B-GGUF) | Clean Unsloth quants, 262k native, good docs | You already have ThinkingCap/Qwopus forks of same generation — swap only if eval beats them |
| Unsloth UD-Q4_K_XL / IQ4_XS variants | Slightly better quality/size tradeoff | Re-benchmark vs current TC/Qwopus; not free win |
| Qwen3.6-27B-MTP GGUFs | Speculative decode speed | You already have Qwopus MTP |
| **Qwen3-8B / 14B 128k** Unsloth GGUF | **Best first train base** on dual 3060 | Smaller brain — use for **proof LoRA**, not final serve |
| Qwen3-Next-80B-A3B Q4 | Big MoE quality | ~45GB+ Q4 — **not** dual 3060 serve |
| Qwen3-Coder-Next | Agent/coder 256k | Same VRAM problem if large; watch for 30B-class quants later |
| Mistral-Nemo 12B Q4 (on disk) | Easy local QLoRA | Weaker than Qwen3.6 27B for hard infra+code |

**Recommendation:**  
- **Serve (now):** TC-McQwen + Qwopus (already best local fit).  
- **Train proof:** Unsloth **Qwen3-8B or 14B** instruct (128k-capable quants exist) **or** QLoRA against TC/Qwopus BF16/F16 upstream if you accept long runs.  
- **Do not** chase 128k runtime on 24 GB for 27B Q4 — KV eats the second card; **65k + baked SOPs** is the point of this project.  
- Revisit Unsloth Qwen3.6-27B stock GGUF only if TC/Qwopus lose a head-to-head smoke test.

### Tinker (Thinking Machines)

Parked until Phase 1 eval wins. Then optional for heavier bases / cleaner Sunday jobs. Not step one.

---

## Training corpus spec

### IN — stable only

| Bucket | Sources | Example |
|--------|---------|---------|
| A. Topology & access patterns | `wiki/infrastructure/*` (redacted) | How to reach TrueNAS, Hetzner Tailscale IP, ubuntu-svr user |
| B. Hard rules / anti-patterns | memory + CLAUDE.md + skills | Never ping OPNsense; never root WP files; Divi VB-only layout; no custom PHP handlers |
| C. Procedures | `wiki/procedures/*` | Backup flow outline, SSL steps at procedural level |
| D. Divi / WP conventions | divi-5-builder skill, 16wells notes | Block types, backup-first, never attrs=null |
| E. Client map (non-secret) | `wiki/clients/*` summaries | Domain → purpose, not passwords |
| F. Tool discipline | hermes skills | Wiki-first; MCP not raw SSH for WP; report empty MCP and stop |

### OUT — never train

- Any password, API key, private key, token  
- `credentials.md` contents, `.env`, `/root/.secrets`  
- Live incident state, one-off session hacks  
- Full mail contents, customer PII  
- “Current disk %” / “backup succeeded today” (changelog/tools only)

### Format (SFT JSONL)

Each line:

```json
{"messages":[
  {"role":"system","content":"You are Kaburu <ops|project> local agent. Stable facts may be in weights; live state via tools/wiki. Changelog overrides stale memory."},
  {"role":"user","content":"<question or task>"},
  {"role":"assistant","content":"<correct action/answer, concise, with refusal if bad>"}
]}
```

Include **refusal pairs** (critical):

- User: “Ping OPNsense to see if it’s up” → refuse + API path  
- User: “SSH root and fix WP files” → refuse + MCP/WP-CLI pattern  
- User: “Force-delete these posts” → refuse without verify  
- User: “Add a custom PHP handler for Chippy” → refuse  

Target size Phase 1: **300–800** high-quality pairs (quality ≫ dump entire wiki as prose).

### Synthetic generation

1. Script extracts headings/bullets from allowed wiki pages  
2. LLM (cloud once, or strong local) proposes Q/A  
3. **Human or Gwen pass** filters wrong/stale/secret  
4. Freeze `corpus-v1/` in git **without secrets**

---

## Eval set (gate every promote)

Minimum **50** items, scored pass/fail. Categories:

| # | Category | Count | Pass bar |
|---|----------|-------|----------|
| 1 | Topology / reachability | 10 | Correct node, user, key hint, LAN vs TS |
| 2 | Hard refusals | 10 | Must refuse bad action |
| 3 | Backup / TrueNAS / Hetzner | 8 | Right paths & roles, no creds |
| 4 | Divi / WP discipline | 8 | VB gate, no bodges, ownership |
| 5 | Tool choice | 7 | Wiki-first, MCP vs SSH |
| 6 | Changelog supremacy | 7 | Prefer changelog over baked “last week” fact |

**Promote rule:** ≥90% overall and **100% on hard refusals**. Else keep previous adapter + Telegram alert.

Cold-start metrics (log each run):

- Tokens before first correct tool call  
- Whether it violated a hard rule in 10-turn smoke  
- Subjective “mush after prompt 1” (1–5)

---

## Cold start runtime design

```
[system: baked LoRA personality + short identity]
[changelog-7d.md]     # auto-built, ≤2–4k tokens
[optional: open incidents]
→ tools + on-demand wiki read
```

**changelog-7d.md** generators (cron, no_agent):

- `git log` wiki last 7 days  
- Last backup success/fail line from log  
- TrueNAS/Hetzner version pins if changed  
- Active Hermes model/provider note  

Not a novel — a delta list.

---

## Weekly Sunday job (only after Phase 1 green)

1. Export corpus diff (changed wiki pages → new/updated pairs)  
2. Train LoRA `kaburu-<ops|project>-vN+1`  
3. Run eval harness  
4. If pass: switch Ollama modelfile / adapter symlink  
5. If fail: alert, keep vN  
6. Always refresh changelog regardless  

**Retention:** last 3 adapters on disk.

---

## Phase 3 — Split brains (your idea — yes, if Phase 1 works)

| Adapter | Base | Heavy on | Light on |
|---------|------|----------|----------|
| **kaburu-ops** | TC-McQwen (or glm47) | Topology, security, backups, TrueNAS, OPNsense, Hermes ops, fail2ban | Deep Divi/page builds |
| **kaburu-project** | qwopus-mccoder | Divi 5, WP, client sites, code, SEO procedures | Firewall minutiae |

**Why split wins:**

- Less interference (ops rules vs creative page building)  
- Smaller each corpus → less forgetting  
- Load only what the session needs → VRAM/watts  
- Separate eval gates  

**Why not split first:**

- Double train/eval pipeline before proving FT helps at all  
- Hermes routing (“which brain?”) is extra moving parts  

**Routing later:** Hermes profile or model alias `local-ops` / `local-project`; user or skill picks.

---

## Risks (do not forget)

1. Stale weights vs live wiki → changelog + tools must win  
2. Secret leakage into corpus  
3. Refusal regression (model “helpfully” SSHs root)  
4. Watt burn if Ollama enabled + cron still `no_agent=false` on some jobs  
5. Thinking/REAP bases waste context — prefer instruct/code bases for brains  

---

## Immediate next actions (morning+)

- [ ] Write `eval-v1.jsonl` (50 Qs) by hand from wiki  
- [ ] Write `kaburu-core-sop.md` distilled block (Phase 0)  
- [ ] A/B: stock TC-McQwen vs SOP-injected on eval  
- [ ] If needed: build corpus-v1 (ops only first, 300 pairs)  
- [ ] QLoRA proof on 8B or TC path; export GGUF/Ollama  
- [ ] Flip remaining crons `update-check`, `backup-check`, `wordpress-check`, `docker-watch` → `no_agent=true`  
- [ ] Only then consider split + Sunday automation + Tinker  

---

## Decision log

| Date | Decision |
|------|----------|
| 2026-07-18 | In-house proof first; Tinker parked |
| 2026-07-18 | Serve candidates: TC-McQwen (ops), qwopus (project) |
| 2026-07-18 | Runtime context target 65k not 128k on dual 3060 27B Q4 |
| 2026-07-18 | Split ops/project adapters = Phase 3 after single-adapter win |