# Model Test SOP — pull → verify → load → benchmark → compare

**Purpose:** Standardised workflow for pulling a new GGUF, verifying its true identity, loading it within GPU constraints, benchmarking it, and comparing against the incumbent (KAT-Coder). Used 2026-08-15 for Qwen3.8-27B ("Gwen McQwen").

## 0. Prerequisites

- HF Hub reachable + `hf_xet` installed (see `gguf-import` skill, "xet-backed HF files")
- `/models/` free space — check `df -h /models`
- llama.cpp build (`/home/kaburu/llama.cpp/build/bin/llama-server --version`)
- Know the target GPU layout:
  - **kaburuaibox now:** 2× RTX 3060 12GB = **24GB total** (~23GB usable)
  - **P40s arriving ~week of 2026-08-18:** 24GB each, Pascal (no tensor cores — same quant runs, slower, `--flash-attn off`)

## 1. Select the quant (fit maths first, don't guess)

Rules of thumb for the 24GB pair:

| Quant | ~size | 24GB + long context |
|---|---|---|
| Q8_0 | 29GB | ❌ never |
| Q6_K | 23GB | ❌ no KV room |
| Q5_K_M | 20GB | ⚠️ tight |
| Q4_K_M | 17GB | ⚠️ tight-ish |
| **IQ4_XS** | **15.7GB** | ✅ sweet spot |
| Q3_K_M | 14GB | ✅ lower quality |

- **Dense vs MoE matters more than raw size.** A dense 27B (all params active every token) needs big compute buffers + full KV cache. A 35B MoE (KAT, ~3B active) is *lighter to serve* despite larger file. Don't assume "bigger model = harder to fit".
- **KV cache is the real ceiling for long context.** 262K context in FP16 KV cache overflows 24GB. Use `--cache-type-k q8_0 --cache-type-v q8_0` to halve it.
- Context ceiling on 24GB (dense 27B): **128K loads, 192K fails, 262K fails** — even with q8_0 KV cache. On P40s (48GB) 262K will fit.

### Vision (VLM) — Qwen3.8 is a native VLM, KAT is not

Qwen3.8-27B ships an `mmproj` vision projector. Pull it and pass `--mmproj` to llama-server to enable image/video input:

```bash
hf download <user>/<repo>-GGUF mmproj-F16.gguf --local-dir /models/<dir>
# load with --mmproj /models/<dir>/mmproj-F16.gguf
```

- **Thinking mode splits output** — the model emits `reasoning_content` first and the actual answer lands in `content`. A naive client that only reads `content` sees an empty string on the first pass. Read **both fields** (or `finish_reason` will be `stop` with content filled after reasoning ends).
- **Vision doesn't cost generation speed** — 20 t/s text == 20 t/s vision. Image prompt processing is fast (~349 t/s, 2.9ms/token).
- This is the one capability **KAT lacks** (text-only MoE, no mmproj). Division of labour: KAT = primary agent (83.8 t/s), Gwen McQwen = on-demand vision (20 t/s).

## 2. Pull (background + notify)

```bash
sudo mkdir -p /models/<model-dir> && sudo chown kaburu:kaburu /models/<model-dir>
# /models is root-owned — must sudo-mkdir + chown, else "Permission denied"
cd /models/<model-dir>
hf download <user>/<repo>-GGUF <file.gguf> --local-dir /models/<model-dir>
```

Run via `terminal(background=true, notify_on_complete=true)`. 15.7GB ≈ 3–4 min on the LAN link.

## 3. Verify identity BEFORE trusting the filename

Filenames/aliases are cosmetic. Read the GGUF header metadata:

```bash
strings -n 4 <file.gguf> | grep -iE 'architecture|base_model|general\.' | head -20
```

Confirm:
- `general.architecture` — must match a family the local llama.cpp build supports (qwen35, qwen35moe, qwen3next)
- `general.base_model.0.name` — the real base model
- `general.name = Safetensors` → straight conversion, no fine-tune (distrust custom names)

**Real example (2026-08-15):** Qwen3.8-27B-IQ4_XS.gguf → `architecture = qwen35`, `base_model.0.name = "Qwen3.8 27B!"`. Genuine unsloth quant of Qwen3.8, dense, not a rename.

## 4. Load (free VRAM first — shuffle pattern)

The incumbent (KAT) holds the GPU. Stop it, load the test model on a **separate port**, benchmark, then restore.

```bash
# 1. Free VRAM
sudo systemctl stop llama-server-kat.service
nvidia-smi   # confirm ~1 MiB used per card

# 2. Load test model on :11441
llama-server --model /models/<dir>/<file.gguf> --host 127.0.0.1 --port 11441 \
  --n-gpu-layers 99 --ctx-size 131072 \
  --cache-type-k q8_0 --cache-type-v q8_0 --flash-attn on \
  --threads 16 --no-webui --alias "<FriendlyName>"

# 3. Health-check until ready
curl -s http://127.0.0.1:11441/health   # {"status":"ok"} when loaded
```

**Loading failure signatures** (from `process(action=log)`):
- `cudaMalloc failed: out of memory` on a KV/compute buffer → context too large for this card layout. Step ctx-size down (262144→131072) or drop to q8_0/q4_0 KV cache.
- `unknown model architecture` → llama.cpp build predates the arch. Upgrade llama.cpp.
- `failed to fit params ... n_gpu_layers already set by user` → warning only, often followed by the real OOM line.

## 5. Benchmark (matched prompt, both models)

```bash
# Smoke test
curl -s http://127.0.0.1:11441/v1/chat/completions -H "Content-Type: application/json" \
  -d '{"model":"<FriendlyName>","messages":[{"role":"user","content":"Who are you? One sentence."}],"max_tokens":100,"temperature":0.7}'

# Timed generation test — identical prompt, max_tokens 256, temp 0.7
time curl -s http://127.0.0.1:11441/v1/chat/completions ... > /tmp/out.json
```

**Get authoritative t/s from the server log, not wall-clock** — `journalctl -u <svc>` (systemd) or `process(action=log)` shows:
```
print_timing: ... eval time = 3055.52 ms / 256 tokens (11.94 ms per token, 83.78 tokens per second)
```
`tg = NN.NN t/s` lines are the live generation rate.

## 6. Compare

| Metric | KAT (35B MoE, 3B active) | Qwen3.8-27B (dense) |
|---|---|---|
| Generation t/s | **83.8** | 19.9 |
| Prompt eval t/s | 99.4 | 80.2 |
| Weights (IQ4_XS) | 18.8GB | 15.7GB |
| Max context @24GB | 192K | 128K |

**Takeaway:** MoE is ~4× faster per token than dense at similar weight sizes — active-param count, not file size, drives throughput. Dense 27B only wins if the MoE's ~3B active params is a quality ceiling (it isn't for KAT's use case).

## Pitfalls

- **Never trust the filename/alias** — always verify `general.architecture` + `base_model` first (KAT fake-rename lesson).
- **`/models` is root-owned** — sudo-mkdir + chown or the `hf download` dies instantly with "Permission denied".
- **Don't benchmark by wall-clock alone** — prompt caching and graph reuse skew it. Read `print_timing`/`tg` from the server log.
- **Dense ≠ easier than MoE.** Compute buffers + KV cache can make a smaller dense model harder to fit than a bigger MoE.
- **Restore the incumbent when done** — `sudo systemctl start llama-server-kat.service`, verify `:11440/health` before moving on.

## Related

- `gguf-import` skill — xet pull, Ollama import, GPU troubleshooting, identity verification reference
- `mlops-models`, `serving-llms-vllm` — serving alternatives
- `z600-vulkan-gpu-farm` — P40 fleet project (72GB Vulkan)