# Z840 VRAM Math — KV Cache Reality

**Source:** Cloud Codes "Get 4× More Context From the Same Card (VRAM Calculators Are Wrong)" — Aug 22 2026 — https://www.youtube.com/watch?v=629J4G3aek4

## TL;DR

Online VRAM calculators (HF NyxKrage, etc.) all use the same flawed formula:

```
cache = 2 × layers × kv_heads × head_dim × bytes × tokens
```

That formula assumes **every layer keeps a KV cache**. Modern hybrid architectures break that assumption in 3 patterns. The fix is to read `config.json` directly and only count layers whose `layer_types` says `full_attention`.

## The 3 architectural escapes

### 1. Gated DeltaNet / linear attention (Qwen3.8-27B)
- 64 total layers, but only **16 are full attention**, the other 48 are linear attention (gated delta net blocks)
- Linear attention holds a **fixed-size state that gets overwritten**, not a growing list
- **62.5 MiB per 1,000 tokens** vs calculator's 250 MiB — **4× error**

### 2. Sliding window attention (Gemma 4 12B)
- 48 layers, 40 sliding (window=1024) + 8 full
- Sliding layers only remember the last 1024 tokens; oldest evicted
- At 8K context: real = 832 MB, formula says 3 GB — **3.7× error**
- At 128K context: real = 8.5 GB, formula says 48 GB — **5.6× error**
- **The error grows with context length** — backwards from intuition

### 3. Layer alternation (gpt-oss-120B)
- 36 layers alternating sliding/full, half windowed
- Window = 128 tokens (not 1024)
- Head dimension = 64 (vs Gemma 256), 8 kv heads → 2 KB/layer/token
- **Full 131K context costs 4.5 GiB of cache** on a 120B parameter model
- For reference: a 7B Llama (2023) burned 2 GB cache to hold 4K tokens

## What actually fits

| VRAM | Model | Quant | Weights | Practical context |
|------|-------|-------|---------|-------------------|
| 8 GB | Qwen3.5-4B | 8-bit | 4.5 GB | full |
| 16 GB | Gemma 4 12B | 8-bit | 12.67 GB | full |
| 24 GB | Qwen3.8-27B | 4-bit | 16.46 GB | ~7.5 GB cache ≈ 120K tokens |
| 64 GB | gpt-oss-120B | 4-bit | 63.39 GB | 4.5 GB cache = full 131K (tight) |

*Note: Z840 has 2× RTX 3060 (12GB each) = 24GB total split across GPUs. The 24GB row above matches your effective budget, but compute buffer allocation is per-GPU, so practical ceiling may be lower than a single 24GB card.*

## Z840 implications

**Current setup:** 2× RTX 3060 (12GB each, 24GB total), Qwen3.8-27B IQ4_XS.

- Reached **96K context** (works)
- OOM at **131K context** (fails)

The video confirms the **KV cache math is fine** at 96K — Qwen3.8-27B's real cache is ~6 GB at 96K (96K × 62.5 MiB/1000), well within budget. **The OOM at 131K is NOT cache**, it's **llama.cpp compute buffers** (working memory for attention computation, much larger than the cache itself).

### Verified against actual config.json (2026-08-22)

Pulled `https://huggingface.co/Qwen/Qwen3.8-27B/raw/main/config.json` and confirmed every number in the video:

```json
"num_hidden_layers": 64,
"layer_types": [/* 64 entries, pattern is [linear,linear,linear,full] × 16 */],
"full_attention_interval": 4,
"num_key_value_heads": 4,
"head_dim": 256,
"max_position_embeddings": 262144
```

- 64 layers total, 16 full_attention, 48 linear_attention ✓
- 4 kv heads × 256 head_dim × 2 bytes × 2 (k+v) = 4 KB/layer/token ✓
- × 16 full layers = 64 KB/token → 62.5 MiB per 1K tokens ✓
- **Bonus: native context is 262K, not 131K** — model is built for it, llama.cpp just can't allocate compute buffers that big on 24GB

### To get past 96K on this Z840

| Option | What it needs |
|--------|--------------|
| 3rd RTX 3060 | More total VRAM → bigger compute buffer headroom |
| Reduce `n_gpu_layers` / force CPU offload for some layers | Trades speed for memory |
| Tune `--ctx-size` with explicit compute buffer alloc | llama.cpp internal, fragile |
| Swap model for gpt-oss-120B (alternation) | Full 131K in 4.5 GB cache, would fit trivially |

## Other lessons from the video

- **Quantizing an MXFP4 model is a rounding error.** gpt-oss-120B's "quant ladder" 2bit→8bit moves only 1.3% in size — weights are already 4-bit before you touch them.
- **KV cache Q8 quantization is safe.** Tested on 4 models, ≤4/500 answer changes vs full precision. **Q4 on keys is catastrophic** — one Qwen 2.5 changed 375/500 (worse than random). Always Q8, not Q4.
- **Speed cost at Q4 cache is significant**: 3% at 8K tokens, **35% at 64K tokens**. The setting that saves the most memory takes the most speed away exactly where you needed it.
- **AutoGPTQ is archived** (last commit April 2025). **AutoAWQ is archived** (one month later). Use **GPTQModel** instead.
- **ExLlamaV2 dormant**, **ExLlamaV3 active** but **no ROCm support** — matters if AMD GPUs ever enter the picture.
- **NVFP4 needs Blackwell** (5090, B200). 4090 gets memory savings only.
- **IQ4_XS doesn't need an imatrix file** despite tutorial claims. Same for IQ4, IQ3S. Only ≤3-bit quants require it.

## Action items

- [ ] Confirm Qwen3.8-27B config.json layer_types shows the 16/48 split (sanity check the thesis on our actual model)
- [ ] If you go 3rd GPU: revisit 131K context target
- [ ] If you swap model: gpt-oss-120B at IQ4_XS would replace both Qwen3.8-27B and the context-cap concern
- [ ] Update memory note: KV cache Q8 quantization is free, Q4 is dangerous