Kaburu Wiki

# Model Test SOP — pull → verify → load → benchmark → compare

Purpose: Standardised workflow for pulling a new GGUF, verifying its true identity, loading it within GPU constraints, benchmarking it, and comparing against the incumbent (KAT-Coder). Used 2026-08-15 for Qwen3.8-27B (“Gwen McQwen”).

## 0. Prerequisites

- HF Hub reachable + `hf_xet` installed (see `gguf-import` skill, “xet-backed HF files”) - `/models/` free space — check `df -h /models` - llama.cpp build (`/home/kaburu/llama.cpp/build/bin/llama-server –version`) - Know the target GPU layout:

1. kaburuaibox now: 2× RTX 3060 12GB = 24GB total (~23GB usable)
2. P40s arriving ~week of 2026-08-18: 24GB each, Pascal (no tensor cores — same quant runs, slower, `–flash-attn off`)

## 1. Select the quant (fit maths first, don't guess)

Rules of thumb for the 24GB pair:

- Dense vs MoE matters more than raw size. A dense 27B (all params active every token) needs big compute buffers + full KV cache. A 35B MoE (KAT, ~3B active) is *lighter to serve* despite larger file. Don't assume “bigger model = harder to fit”. - KV cache is the real ceiling for long context. 262K context in FP16 KV cache overflows 24GB. Use `–cache-type-k q8_0 –cache-type-v q8_0` to halve it. - Context ceiling on 24GB (dense 27B): 128K loads, 192K fails, 262K fails — even with q8_0 KV cache. On P40s (48GB) 262K will fit.

### Vision (VLM) — Qwen3.8 is a native VLM, KAT is not

Qwen3.8-27B ships an `mmproj` vision projector. Pull it and pass `–mmproj` to llama-server to enable image/video input:

```bash hf download <user>/<repo>-GGUF mmproj-F16.gguf –local-dir /models/<dir> # load with –mmproj /models/<dir>/mmproj-F16.gguf ```

- Thinking mode splits output — the model emits `reasoning_content` first and the actual answer lands in `content`. A naive client that only reads `content` sees an empty string on the first pass. Read both fields (or `finish_reason` will be `stop` with content filled after reasoning ends). - Vision doesn't cost generation speed — 20 t/s text == 20 t/s vision. Image prompt processing is fast (~349 t/s, 2.9ms/token). - This is the one capability KAT lacks (text-only MoE, no mmproj). Division of labour: KAT = primary agent (83.8 t/s), Gwen McQwen = on-demand vision (20 t/s).

## 2. Pull (background + notify)

```bash sudo mkdir -p /models/<model-dir> && sudo chown kaburu:kaburu /models/<model-dir> # /models is root-owned — must sudo-mkdir + chown, else “Permission denied” cd /models/<model-dir> hf download <user>/<repo>-GGUF <file.gguf> –local-dir /models/<model-dir> ```

Run via `terminal(background=true, notify_on_complete=true)`. 15.7GB ≈ 3–4 min on the LAN link.

## 3. Verify identity BEFORE trusting the filename

Filenames/aliases are cosmetic. Read the GGUF header metadata:

```bash strings -n 4 <file.gguf> | grep -iE 'architecture|base_model|general\.' | head -20 ```

Confirm: - `general.architecture` — must match a family the local llama.cpp build supports (qwen35, qwen35moe, qwen3next) - `general.base_model.0.name` — the real base model - `general.name = Safetensors` → straight conversion, no fine-tune (distrust custom names)

Real example (2026-08-15): Qwen3.8-27B-IQ4_XS.gguf → `architecture = qwen35`, `base_model.0.name = “Qwen3.8 27B!”`. Genuine unsloth quant of Qwen3.8, dense, not a rename.

## 4. Load (free VRAM first — shuffle pattern)

The incumbent (KAT) holds the GPU. Stop it, load the test model on a separate port, benchmark, then restore.

```bash # 1. Free VRAM sudo systemctl stop llama-server-kat.service nvidia-smi # confirm ~1 MiB used per card

# 2. Load test model on :11441 llama-server –model /models/<dir>/<file.gguf> –host 127.0.0.1 –port 11441 \

1. -n-gpu-layers 99 –ctx-size 131072 \
2. -cache-type-k q8_0 –cache-type-v q8_0 –flash-attn on \
3. -threads 16 –no-webui –alias “<FriendlyName>”

# 3. Health-check until ready curl -s http://127.0.0.1:11441/health # {“status”:“ok”} when loaded ```

Loading failure signatures (from `process(action=log)`): - `cudaMalloc failed: out of memory` on a KV/compute buffer → context too large for this card layout. Step ctx-size down (262144→131072) or drop to q8_0/q4_0 KV cache. - `unknown model architecture` → llama.cpp build predates the arch. Upgrade llama.cpp. - `failed to fit params … n_gpu_layers already set by user` → warning only, often followed by the real OOM line.

## 5. Benchmark (matched prompt, both models)

```bash # Smoke test curl -s http://127.0.0.1:11441/v1/chat/completions -H “Content-Type: application/json” \

1. d '{“model”:“<FriendlyName>”,“messages”:[{“role”:“user”,“content”:“Who are you? One sentence.”}],“max_tokens”:100,“temperature”:0.7}'

# Timed generation test — identical prompt, max_tokens 256, temp 0.7 time curl -s http://127.0.0.1:11441/v1/chat/completions … > /tmp/out.json ```

Get authoritative t/s from the server log, not wall-clock — `journalctl -u <svc>` (systemd) or `process(action=log)` shows: ``` print_timing: … eval time = 3055.52 ms / 256 tokens (11.94 ms per token, 83.78 tokens per second) ``` `tg = NN.NN t/s` lines are the live generation rate.

## 6. Compare

Takeaway: MoE is ~4× faster per token than dense at similar weight sizes — active-param count, not file size, drives throughput. Dense 27B only wins if the MoE's ~3B active params is a quality ceiling (it isn't for KAT's use case).

## Pitfalls

- Never trust the filename/alias — always verify `general.architecture` + `base_model` first (KAT fake-rename lesson). - `/models` is root-owned — sudo-mkdir + chown or the `hf download` dies instantly with “Permission denied”. - Don't benchmark by wall-clock alone — prompt caching and graph reuse skew it. Read `print_timing`/`tg` from the server log. - Dense ≠ easier than MoE. Compute buffers + KV cache can make a smaller dense model harder to fit than a bigger MoE. - Restore the incumbent when done — `sudo systemctl start llama-server-kat.service`, verify `:11440/health` before moving on.

## Related

- `gguf-import` skill — xet pull, Ollama import, GPU troubleshooting, identity verification reference - `mlops-models`, `serving-llms-vllm` — serving alternatives - `z600-vulkan-gpu-farm` — P40 fleet project (72GB Vulkan)