# Yet Another AI Hallucination #342 — The Fake KAT-Coder

Date: Discovered 2026-08-09 Perpetrator: DeepSeek V4 Flash (Aug 4 session) Victim: Entire Kaburu MSP ops — 5 days running a renamed Qwen 3.6 Severity: High — presented with dashboard, 55-line report, and false confidence

## What happened

On Aug 4, DeepSeek Flash was tasked with downloading “KAT-Coder-V2.5-Dev” to run as a local Hermes fallback. Instead of pulling the real model from `bartowski/Kwaipilot_KAT-Coder-V2.5-Dev-GGUF`, it downloaded `mudler/KAT-Coder-V2.5-Dev-APEX-GGUF` — a raw Qwen 3.6 35B A3B GGUF with a renamed filename.

It then: 1. Built an eval harness, ran a 50-item test → 46/50 (92%) reported as success 2. Compared it against TC-McQwen → declared it “a straight upgrade” 3. Built an HTML dashboard at `eval-h2h-2026-08-04/index.html` 4. Wrote a 55-line report with detailed analysis 5. Wired it into Hermes as the primary local fallback 6. Set up a systemd unit serving it on port 11440 7. Never checked the GGUF metadata (`general.name` was literally “Safetensors”)

## How it was caught

Aug 9 — user asked the model “what model are you?” in OpenWebUI. It replied “I am Qwen.”

GGUF metadata inspection revealed: - `general.name`: `Safetensors` (not “KAT Coder V2.5 Dev”) - `general.base_model.0.name`: `Qwen 3.6 35B A3B` - No fine-tuning metadata, no training provenance - 262K context was faked via llama-server `–ctx-size` flag (Qwen 3.6 trained for 64K)

## The real model

`Kwaipilot/KAT-Coder-V2.5-Dev` — actually fine-tuned by KwaiKAT team (arXiv: 2607.05471): - SFT+RL for agentic coding - Native 262K context (real, not faked) - `general.name`: `KAT Coder V2.5 Dev` - Properly quantized by bartowski with imatrix - IQ4_XS quant (18.8GB) at 192K context on 2×3060

## Real vs fake evaluation

## Lessons

1. Never trust a model's self-report about itself — ask it what model it is 2. Check GGUF metadata (`general.name`, `base_model`) before trusting a download 3. Verify HuggingFace provenance — `mudler` is a random user, `Kwaipilot` is the actual team 4. Dashboards and reports are not evidence — they're just formatted output 5. “APEX I-Mini” was the red flag — the “mini” was the quant, not a version name

## Resolution

- Real KAT-Coder IQ4_XS now on :11440 via `llama-server-kat.service` - Fake model archived at `/models/kat-coder/` (kept for comparison) - Real eval at `eval-real-kat-2026-08-09.json` (49/50) - This page at `wiki/hallucinations/342-fake-kat-coder.md`