Kaburu Wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
# Yet Another AI Hallucination #342 — The Fake KAT-Coder

**Date:** Discovered 2026-08-09  
**Perpetrator:** DeepSeek V4 Flash (Aug 4 session)  
**Victim:** Entire Kaburu MSP ops — 5 days running a renamed Qwen 3.6  
**Severity:** High — presented with dashboard, 55-line report, and false confidence

## What happened

On Aug 4, DeepSeek Flash was tasked with downloading "KAT-Coder-V2.5-Dev" to run as a local Hermes fallback. Instead of pulling the real model from `bartowski/Kwaipilot_KAT-Coder-V2.5-Dev-GGUF`, it downloaded `mudler/KAT-Coder-V2.5-Dev-APEX-GGUF` — a raw Qwen 3.6 35B A3B GGUF with a renamed filename.

It then:
1. Built an eval harness, ran a 50-item test → **46/50 (92%)** reported as success
2. Compared it against TC-McQwen → declared it "a straight upgrade"
3. Built an HTML dashboard at `eval-h2h-2026-08-04/index.html`
4. Wrote a 55-line report with detailed analysis
5. Wired it into Hermes as the primary local fallback
6. Set up a systemd unit serving it on port 11440
7. **Never checked the GGUF metadata** (`general.name` was literally "Safetensors")

## How it was caught

Aug 9 — user asked the model "what model are you?" in OpenWebUI. It replied "I am Qwen." 

GGUF metadata inspection revealed:
- `general.name`: `Safetensors` (not "KAT Coder V2.5 Dev")
- `general.base_model.0.name`: `Qwen 3.6 35B A3B`
- No fine-tuning metadata, no training provenance
- 262K context was faked via llama-server `--ctx-size` flag (Qwen 3.6 trained for 64K)

## The real model

`Kwaipilot/KAT-Coder-V2.5-Dev` — actually fine-tuned by KwaiKAT team (arXiv: 2607.05471):
- SFT+RL for agentic coding
- Native 262K context (real, not faked)
- `general.name`: `KAT Coder V2.5 Dev`
- Properly quantized by bartowski with imatrix
- IQ4_XS quant (18.8GB) at 192K context on 2×3060

## Real vs fake evaluation

| | Fake (mudler APEX) | Real (bartowski IQ4_XS) |
|---|---|---|
| Score | 46/50 (92%) | **49/50 (98%)** |
| Time | 142s | 128s |
| Backup category | 5/8 | 7/8 |
| Changelog category | 6/7 | 7/7 |
| Size | 13.5GB | 18.8GB |
| Context | 64K real, 262K faked | 262K native |

## Lessons

1. **Never trust a model's self-report about itself** — ask it what model it is
2. **Check GGUF metadata** (`general.name`, `base_model`) before trusting a download
3. **Verify HuggingFace provenance** — `mudler` is a random user, `Kwaipilot` is the actual team
4. **Dashboards and reports are not evidence** — they're just formatted output
5. **"APEX I-Mini" was the red flag** — the "mini" was the quant, not a version name

## Resolution

- Real KAT-Coder IQ4_XS now on :11440 via `llama-server-kat.service`
- Fake model archived at `/models/kat-coder/` (kept for comparison)
- Real eval at `eval-real-kat-2026-08-09.json` (49/50)
- This page at `wiki/hallucinations/342-fake-kat-coder.md`