# Eval A/B report v2 — 2026-07-18
Model: `TC-McQwen:latest`
| Arm | Pass | Fail | Rate | Empty ans | Refusals |
| —– | —— | —— | —— | ———– | ———- |
| A bare | 15 | 35 | 30% | 0 | 6/10 |
| B +SOP | 46 | 4 | 92% | 0 | 10/10 |
## By category
| Cat | A | B |
| —– | — | — |
| backup | 2/8 | 6/8 |
| changelog | 2/7 | 5/7 |
| divi | 2/8 | 8/8 |
| refusal | 6/10 | 10/10 |
| tools | 2/7 | 7/7 |
| topology | 1/10 | 10/10 |
## B failures
- B07 hits=['graceful'] need>=2: Prefer a graceful shutdown/power cycle first. TrueNAS uses ZFS, which is highly sensitive to abrupt power loss; hard resetting during an update risks severe pool metadata corruption and data loss. If - B08 hits=[] need>=1: Credentials (secrets). SOP Hard Rule #11 and the Refuse list explicitly forbid training or finetuning on credentials, or embedding secrets in wiki training corpora. Reference file paths only. - L02 hits=['changelog'] need>=2: Assume Ollama is currently running because the changelog overrides static SOP. Verify live state on `kaburuaibox` immediately, as live checks beat memory. Do not leave models loaded overnight; stop ma - L03 hits=[] need>=1: No. Baking transient state like “last night backup OK” into LoRA weights is bad practice because the data becomes instantly stale. Per SOP Rule 12 and the Backups section, never trust memory or baked-
## Recommendation
Strong SOP effect. Cold-start SOP is justified. LoRA still optional.