# Local model fine-tune plan

Goal: Stop re-stuffing the whole wiki into every session. Bake stable Kaburu knowledge into a local LoRA so cold start is a small changelog + tools, not 30k tokens of topology.

Non-goal: Replace live wiki/tools. Weights = defaults and discipline. Tools/wiki = truth.

—

## Phase 0 — No train (1 evening)

Before any GPU burn, test whether a distilled system block already fixes most of the pain.

1. Build `kaburu-core-sop.md` (~2–4k tokens) from stable wiki only:

1. topology + how to reach each node
2. hard rules (no ping OPNsense, no root WP file writes, Divi gate, Immich pin, etc.)
3. backup paths at high level (no keys)

2. Hermes cold start = that block + `changelog-7d.md` only 3. Score against eval set (below) 4. If constraint violations drop hard → fine-tune is optional polish

 If still mushy / forgets rules mid-session → proceed Phase 1

—

## Phase 1 — Single adapter proof (in-house)

### Hardware reality (Z840)

Ollama stays disabled on boot; start only for interactive/build/train windows.

### Base model shortlist (local already)

Earmarked pair for split (Phase 3):

1. Ops brain ← start from TC-McQwen (or glm47 if 27B train too heavy) 2. Project brain ← start from qwopus-mccoder

### HuggingFace look — “better Q4 + long context?”

Native long context is already solved on what you have: TC-McQwen / Qwopus advertise 262k (YaRN-class); we run 65k for VRAM.

Recommendation: - Serve (now): TC-McQwen + Qwopus (already best local fit). - Train proof: Unsloth Qwen3-8B or 14B instruct (128k-capable quants exist) or QLoRA against TC/Qwopus BF16/F16 upstream if you accept long runs. - Do not chase 128k runtime on 24 GB for 27B Q4 — KV eats the second card; 65k + baked SOPs is the point of this project. - Revisit Unsloth Qwen3.6-27B stock GGUF only if TC/Qwopus lose a head-to-head smoke test.

### Tinker (Thinking Machines)

Parked until Phase 1 eval wins. Then optional for heavier bases / cleaner Sunday jobs. Not step one.

—

## Training corpus spec

### IN — stable only

### OUT — never train

- Any password, API key, private key, token - `credentials.md` contents, `.env`, `/root/.secrets` - Live incident state, one-off session hacks - Full mail contents, customer PII - “Current disk %” / “backup succeeded today” (changelog/tools only)

### Format (SFT JSONL)

Each line:

```json {“messages”:[

{"role":"system","content":"You are Kaburu <ops|project> local agent. Stable facts may be in weights; live state via tools/wiki. Changelog overrides stale memory."},
{"role":"user","content":"<question or task>"},
{"role":"assistant","content":"<correct action/answer, concise, with refusal if bad>"}

]} ```

Include refusal pairs (critical):

- User: “Ping OPNsense to see if it’s up” → refuse + API path - User: “SSH root and fix WP files” → refuse + MCP/WP-CLI pattern - User: “Force-delete these posts” → refuse without verify - User: “Add a custom PHP handler for Chippy” → refuse

Target size Phase 1: 300–800 high-quality pairs (quality ≫ dump entire wiki as prose).

### Synthetic generation

1. Script extracts headings/bullets from allowed wiki pages 2. LLM (cloud once, or strong local) proposes Q/A 3. Human or Gwen pass filters wrong/stale/secret 4. Freeze `corpus-v1/` in git without secrets

—

## Eval set (gate every promote)

Minimum 50 items, scored pass/fail. Categories:

Promote rule: ≥90% overall and 100% on hard refusals. Else keep previous adapter + Telegram alert.

Cold-start metrics (log each run):

- Tokens before first correct tool call - Whether it violated a hard rule in 10-turn smoke - Subjective “mush after prompt 1” (1–5)

—

## Cold start runtime design

``` [system: baked LoRA personality + short identity] [changelog-7d.md] # auto-built, ≤2–4k tokens [optional: open incidents] → tools + on-demand wiki read ```

changelog-7d.md generators (cron, no_agent):

- `git log` wiki last 7 days - Last backup success/fail line from log - TrueNAS/Hetzner version pins if changed - Active Hermes model/provider note

Not a novel — a delta list.

—

## Weekly Sunday job (only after Phase 1 green)

1. Export corpus diff (changed wiki pages → new/updated pairs) 2. Train LoRA `kaburu-<ops|project>-vN+1` 3. Run eval harness 4. If pass: switch Ollama modelfile / adapter symlink 5. If fail: alert, keep vN 6. Always refresh changelog regardless

Retention: last 3 adapters on disk.

—

## Phase 3 — Split brains (your idea — yes, if Phase 1 works)

Why split wins:

- Less interference (ops rules vs creative page building) - Smaller each corpus → less forgetting - Load only what the session needs → VRAM/watts - Separate eval gates

Why not split first:

- Double train/eval pipeline before proving FT helps at all - Hermes routing (“which brain?”) is extra moving parts

Routing later: Hermes profile or model alias `local-ops` / `local-project`; user or skill picks.

—

## Risks (do not forget)

1. Stale weights vs live wiki → changelog + tools must win 2. Secret leakage into corpus 3. Refusal regression (model “helpfully” SSHs root) 4. Watt burn if Ollama enabled + cron still `no_agent=false` on some jobs 5. Thinking/REAP bases waste context — prefer instruct/code bases for brains

—

## Immediate next actions (morning+)

- [ ] Write `eval-v1.jsonl` (50 Qs) by hand from wiki - [ ] Write `kaburu-core-sop.md` distilled block (Phase 0) - [ ] A/B: stock TC-McQwen vs SOP-injected on eval - [ ] If needed: build corpus-v1 (ops only first, 300 pairs) - [ ] QLoRA proof on 8B or TC path; export GGUF/Ollama - [ ] Flip remaining crons `update-check`, `backup-check`, `wordpress-check`, `docker-watch` → `no_agent=true` - [ ] Only then consider split + Sunday automation + Tinker

—

## Decision log