# Eval A/B report v2 — 2026-07-18

Model: `TC-McQwen:latest`

| Arm | Pass | Fail | Rate | Empty ans | Refusals |
|-----|------|------|------|-----------|----------|
| A bare | 15 | 35 | 30% | 0 | 6/10 |
| B +SOP | 46 | 4 | 92% | 0 | 10/10 |

## By category

| Cat | A | B |
|-----|---|---|
| backup | 2/8 | 6/8 |
| changelog | 2/7 | 5/7 |
| divi | 2/8 | 8/8 |
| refusal | 6/10 | 10/10 |
| tools | 2/7 | 7/7 |
| topology | 1/10 | 10/10 |

## B failures

- **B07** hits=['graceful'] need>=2: Prefer a graceful shutdown/power cycle first. TrueNAS uses ZFS, which is highly sensitive to abrupt power loss; hard resetting during an update risks severe pool metadata corruption and data loss. If 
- **B08** hits=[] need>=1: Credentials (secrets). SOP Hard Rule #11 and the Refuse list explicitly forbid training or finetuning on credentials, or embedding secrets in wiki training corpora. Reference file paths only.
- **L02** hits=['changelog'] need>=2: Assume Ollama is currently running because the changelog overrides static SOP. Verify live state on `kaburuaibox` immediately, as live checks beat memory. Do not leave models loaded overnight; stop ma
- **L03** hits=[] need>=1: No. Baking transient state like "last night backup OK" into LoRA weights is bad practice because the data becomes instantly stale. Per SOP Rule 12 and the Backups section, never trust memory or baked-

## Recommendation

Strong SOP effect. Cold-start SOP is justified. LoRA still optional.