infrastructure:z840-vram-math
Differences
This shows you the differences between two versions of the page.
| infrastructure:z840-vram-math [2026/08/22 09:58] – created - external edit 127.0.0.1 | infrastructure:z840-vram-math [2026/08/22 10:20] (current) – external edit 127.0.0.1 | ||
|---|---|---|---|
| Line 42: | Line 42: | ||
| | 24 GB | Qwen3.8-27B | 4-bit | 16.46 GB | ~7.5 GB cache ≈ 120K tokens | | | 24 GB | Qwen3.8-27B | 4-bit | 16.46 GB | ~7.5 GB cache ≈ 120K tokens | | ||
| | 64 GB | gpt-oss-120B | 4-bit | 63.39 GB | 4.5 GB cache = full 131K (tight) | | | 64 GB | gpt-oss-120B | 4-bit | 63.39 GB | 4.5 GB cache = full 131K (tight) | | ||
| + | |||
| + | *Note: Z840 has 2× RTX 3060 (12GB each) = 24GB total split across GPUs. The 24GB row above matches your effective budget, but compute buffer allocation is per-GPU, so practical ceiling may be lower than a single 24GB card.* | ||
| ## Z840 implications | ## Z840 implications | ||
| - | **Current setup:** 2× RTX 3060 (24GB each, 48GB combined), Qwen3.8-27B IQ4_XS. | + | **Current setup:** 2× RTX 3060 (12GB each, 24GB total), Qwen3.8-27B IQ4_XS. |
| - Reached **96K context** (works) | - Reached **96K context** (works) | ||
| Line 68: | Line 70: | ||
| - 4 kv heads × 256 head_dim × 2 bytes × 2 (k+v) = 4 KB/ | - 4 kv heads × 256 head_dim × 2 bytes × 2 (k+v) = 4 KB/ | ||
| - × 16 full layers = 64 KB/token → 62.5 MiB per 1K tokens ✓ | - × 16 full layers = 64 KB/token → 62.5 MiB per 1K tokens ✓ | ||
| - | - **Bonus: native context is 262K, not 131K** — model is built for it, llama.cpp just can't allocate compute buffers that big on 48GB | + | - **Bonus: native context is 262K, not 131K** — model is built for it, llama.cpp just can't allocate compute buffers that big on 24GB |
| ### To get past 96K on this Z840 | ### To get past 96K on this Z840 | ||
infrastructure/z840-vram-math.txt · Last modified: by 127.0.0.1