# Z840 VRAM Math — KV Cache Reality
Source: Cloud Codes “Get 4× More Context From the Same Card (VRAM Calculators Are Wrong)” — Aug 22 2026 — https://www.youtube.com/watch?v=629J4G3aek4
## TL;DR
Online VRAM calculators (HF NyxKrage, etc.) all use the same flawed formula:
``` cache = 2 × layers × kv_heads × head_dim × bytes × tokens ```
That formula assumes every layer keeps a KV cache. Modern hybrid architectures break that assumption in 3 patterns. The fix is to read `config.json` directly and only count layers whose `layer_types` says `full_attention`.
## The 3 architectural escapes
### 1. Gated DeltaNet / linear attention (Qwen3.8-27B) - 64 total layers, but only 16 are full attention, the other 48 are linear attention (gated delta net blocks) - Linear attention holds a fixed-size state that gets overwritten, not a growing list - 62.5 MiB per 1,000 tokens vs calculator's 250 MiB — 4× error
### 2. Sliding window attention (Gemma 4 12B) - 48 layers, 40 sliding (window=1024) + 8 full - Sliding layers only remember the last 1024 tokens; oldest evicted - At 8K context: real = 832 MB, formula says 3 GB — 3.7× error - At 128K context: real = 8.5 GB, formula says 48 GB — 5.6× error - The error grows with context length — backwards from intuition
### 3. Layer alternation (gpt-oss-120B) - 36 layers alternating sliding/full, half windowed - Window = 128 tokens (not 1024) - Head dimension = 64 (vs Gemma 256), 8 kv heads → 2 KB/layer/token - Full 131K context costs 4.5 GiB of cache on a 120B parameter model - For reference: a 7B Llama (2023) burned 2 GB cache to hold 4K tokens
## What actually fits
| VRAM | Model | Quant | Weights | Practical context |
| —— | ——- | ——- | ——— | ——————- |
| 8 GB | Qwen3.5-4B | 8-bit | 4.5 GB | full |
| 16 GB | Gemma 4 12B | 8-bit | 12.67 GB | full |
| 24 GB | Qwen3.8-27B | 4-bit | 16.46 GB | ~7.5 GB cache ≈ 120K tokens |
| 64 GB | gpt-oss-120B | 4-bit | 63.39 GB | 4.5 GB cache = full 131K (tight) |
*Note: Z840 has 2× RTX 3060 (12GB each) = 24GB total split across GPUs. The 24GB row above matches your effective budget, but compute buffer allocation is per-GPU, so practical ceiling may be lower than a single 24GB card.*
## Z840 implications
Current setup: 2× RTX 3060 (12GB each, 24GB total), Qwen3.8-27B IQ4_XS.
- Reached 96K context (works) - OOM at 131K context (fails)
The video confirms the KV cache math is fine at 96K — Qwen3.8-27B's real cache is ~6 GB at 96K (96K × 62.5 MiB/1000), well within budget. The OOM at 131K is NOT cache, it's llama.cpp compute buffers (working memory for attention computation, much larger than the cache itself).
### Verified against actual config.json (2026-08-22)
Pulled `https://huggingface.co/Qwen/Qwen3.8-27B/raw/main/config.json` and confirmed every number in the video:
```json “num_hidden_layers”: 64, “layer_types”: [/* 64 entries, pattern is [linear,linear,linear,full] × 16 */], “full_attention_interval”: 4, “num_key_value_heads”: 4, “head_dim”: 256, “max_position_embeddings”: 262144 ```
- 64 layers total, 16 full_attention, 48 linear_attention ✓ - 4 kv heads × 256 head_dim × 2 bytes × 2 (k+v) = 4 KB/layer/token ✓ - × 16 full layers = 64 KB/token → 62.5 MiB per 1K tokens ✓ - Bonus: native context is 262K, not 131K — model is built for it, llama.cpp just can't allocate compute buffers that big on 24GB
### To get past 96K on this Z840
| Option | What it needs |
| ——– | ————– |
| 3rd RTX 3060 | More total VRAM → bigger compute buffer headroom |
| Reduce `n_gpu_layers` / force CPU offload for some layers | Trades speed for memory |
| Tune `–ctx-size` with explicit compute buffer alloc | llama.cpp internal, fragile |
| Swap model for gpt-oss-120B (alternation) | Full 131K in 4.5 GB cache, would fit trivially |
## Other lessons from the video
- Quantizing an MXFP4 model is a rounding error. gpt-oss-120B's “quant ladder” 2bit→8bit moves only 1.3% in size — weights are already 4-bit before you touch them. - KV cache Q8 quantization is safe. Tested on 4 models, ≤4/500 answer changes vs full precision. Q4 on keys is catastrophic — one Qwen 2.5 changed 375/500 (worse than random). Always Q8, not Q4. - Speed cost at Q4 cache is significant: 3% at 8K tokens, 35% at 64K tokens. The setting that saves the most memory takes the most speed away exactly where you needed it. - AutoGPTQ is archived (last commit April 2025). AutoAWQ is archived (one month later). Use GPTQModel instead. - ExLlamaV2 dormant, ExLlamaV3 active but no ROCm support — matters if AMD GPUs ever enter the picture. - NVFP4 needs Blackwell (5090, B200). 4090 gets memory savings only. - IQ4_XS doesn't need an imatrix file despite tutorial claims. Same for IQ4, IQ3S. Only ≤3-bit quants require it.
## Action items
- [ ] Confirm Qwen3.8-27B config.json layer_types shows the 16/48 split (sanity check the thesis on our actual model) - [ ] If you go 3rd GPU: revisit 131K context target - [ ] If you swap model: gpt-oss-120B at IQ4_XS would replace both Qwen3.8-27B and the context-cap concern - [ ] Update memory note: KV cache Q8 quantization is free, Q4 is dangerous