Kaburu Wiki

# Z840 — kaburuaibox

Primary AI and automation server. Runs Hermes agent, Ollama, Open WebUI, and supporting services.

## Specs

## Network

Netplan config: `/etc/netplan/01-netcfg.yaml` - bond0 metric 50 (preferred over eno1) - Both 10Gb MACs registered in OPNsense dnsmasq → 192.168.0.253

1. ens1f0: `b4:96:91:11:44:dc`
2. ens1f1: `b4:96:91:11:44:de` ← active port (has cable)

## Access

- SSH: `ssh [email protected]` (LAN) or `ssh [email protected]` (Tailscale) - Open WebUI: `http://100.120.18.44:3000` (Tailscale only) - Hermes Dashboard: `http://100.120.18.44:8765` (Tailscale only)

## Services

## DR / Remote Access (installed 2026-08-19)

Connect via RDP: `<kaburuaibox IP>:3389` → login as `kaburu` Connect via RustDesk: ID `339562074` → password `kaburuDR2026`

Services: ``` xvfb.service — virtual display :1 (enabled, auto-start) xrdp.service — RDP server (enabled, auto-start) rustdesk.service — remote desktop via kaburu.cc relay (enabled, auto-start) ```

Config files: - `/root/.config/rustdesk/RustDesk2.toml` — RustDesk config (password set) - `/etc/systemd/system/rustdesk.service.d/override.conf` — relay server args - `/home/kaburu/.xsession` — XFCE session for xrdp

## Inference stack (Ollama — on-demand, single-model-at-a-time)

llama-server permanently disabled 2026-07-04. All models served via Ollama on port 11434. Models loaded on-demand — only one large model fits in 24GB VRAM at a time. `OLLAMA_KEEP_ALIVE=-1` on always-on models; others load/unload as needed.

### Ollama Setup

Version: 0.31.1 Systemd service: `ollama.service` (enabled, auto-start) Models directory: `/models/ollama`

Systemd Configuration (`/etc/systemd/system/ollama.service.d/override.conf`): ```ini [Service] Environment=“OLLAMA_MODELS=/models/ollama” Environment=“OLLAMA_HOST=0.0.0.0:11434” Environment=“CUDA_VISIBLE_DEVICES=0,1” Environment=“OLLAMA_GPU_OVERHEAD=536870912” Environment=“OLLAMA_KEEP_ALIVE=-1” Environment=“OLLAMA_NUM_GPU=2” Environment=“OLLAMA_SCHED_SPREAD=1” ```

Installed Models (updated 2026-07-17):

Removed: `mistral-nemo:12b` (fired for hallucinating 2026-07-08); `glm47-reap:23b` (fired 2026-08-08 — Steve: “thick as shit, no context”, deleted from Ollama + GGUF, removed from Hermes fallback chain)

GPU Memory Constraints: - Each RTX 3060: 12 GB VRAM (24 GB total across both GPUs) - `KAT-Coder-V2.5-Dev-APEX-I-Mini` via llama-server (11440): ~20GB VRAM, 262K ctx — always-on - `qwopus-mccoder:latest`: ~16GB VRAM - `qwen36-reap:28b`: ~17GB VRAM - `minicpm-v:8b`: ~5GB VRAM (can coexist with a large model) - Only ONE large model (23b+) fits at a time (except minicpm-v)

Optimal Operation:

KAT-Coder-V2.5-Dev-APEX-I-Mini (llama-server, :11440) is the default local model — 262K context, native MoE architecture.

Switching Ollama models: ```bash ollama run qwopus-mccoder # Loads when first called ollama run qwen36-reap:28b # Loads when first called ollama ps # Check what's loaded ollama show <model> –modelfile # Check model config ```

The `OLLAMA_KEEP_ALIVE=-1` setting means models stay loaded until explicitly replaced or Ollama restarts.

## Hermes

Infrastructure monitoring and automation agent. Full documentation: hermes

- Stack: `/opt/hermes/` - Logs: `/mnt/data/hermes/logs/` (ZFS) - Config/SSH key: `/opt/hermes/config/hetzner_id_ed25519` - Manages: Hetzner server health, WordPress updates, backups, SSL, security - Migrated from ubuntu-svr → Z840 on 2026-05-17 - Note: full rebuild planned as part of Owl Alpha Phase 4

## Key paths

## Docker management

```bash # Hermes cd /opt/hermes && docker-compose ps docker logs hermes_agent –tail 50 docker-compose restart

# Open WebUI docker ps | grep open-webui docker logs open-webui –tail 50 ```

## Open WebUI

Chat interface backed by Ollama (11434).

⚠️ Volume: Named volume `open-webui` — NOT a bind mount. Data survives container removal. Do NOT use `-v /opt/open-webui/data:…` — that path is a stale empty copy and must stay deleted.

Docker run command (for reference/rebuild): ```bash docker run -d \

1. -name open-webui \
2. -restart unless-stopped \
3. p 3000:8080 \
4. v open-webui:/app/backend/data \
5. e WEBUI_AUTH=true \
6. e DO_NOT_TRACK=true \
7. e OPENAI_API_BASE_URLS='[“http://192.168.0.253:11440/v1”]' \
8. e OPENAI_API_KEYS='[“dummy”]' \
9. e DEFAULT_MODELS=qwopus-mccoder \
10. e ANONYMIZED_TELEMETRY=false \
11. e SCARF_NO_ANALYTICS=true \
12. e RAG_EMBEDDING_MODEL=sentence-transformers/all-MiniLM-L6-v2 \
13. e AUXILIARY_EMBEDDING_MODEL=TaylorAI/bge-micro-v2 \
14. e WHISPER_MODEL=base \
15. e ENABLE_OLLAMA_API=false \

ghcr.io/open-webui/open-webui:v0.9.6 ```

## Notes

- docker-compose version is 1.29.2 (legacy) — use `docker-compose` not `docker compose` - Open WebUI backed by llama-server (11440) only — Ollama removed 2026-07-02 - Default model: `qwopus-mccoder` (Qwopus3.6-27B-v2-MTP, 16GB GGUF) - Headroom proxy at localhost:8787 providing context compression - Open WebUI data in named Docker volume `open-webui` — persists across container restarts and removals - 2× RTX 3060 installed and fully powered (6-pin→8-pin adapters fitted 2026-05-26) - Box hardware complete as of 2026-05-26 - 6-pin PSU rails confirmed sufficient — 1450W on UK 240V, ~700W expected load under full GPU load - NVIDIA driver upgraded 535 → 580.159.03 on 2026-05-26 (CUDA 13.0 support)

—

## Hermes Agent (NousResearch) — Production

*Installed 2026-05-31. Production as of 2026-06-03.*

Not to be confused with the Docker monitoring stack at `/opt/hermes/`. Full documentation: hermes

GitHub: https://github.com/NousResearch/hermes-agent Version: v0.15.1 (current HEAD, updated 2026-06-03)

- Model: qwen36-reap via llama-server (localhost:11440/v1), provider: custom - Dashboard: http://192.168.0.253:9119 — accessible from any LAN/Tailscale device, PWA-installable - Desktop app: connect to http://192.168.0.253:8642 with API key from ~/.hermes/.env

## PCIe Slot Map + P40 Plan (verified 2026-08-18)

Three x16 slots confirmed via root-port max_link_width (skill claim “3× PCIe Gen3 x16” verified):

⚠️ Physical layout constraints (verified 2026-08-18): - 10Gb NIC (X540-AT2) is SOLDERED to the board at root port 00:01.1 — NOT a removable card - Onboard “VGA” is actually a serial port — NO onboard video output at all - P40 is a dual-width card — installing one in slot 1 (00:02.0) physically blocks/overlaps the soldered 10Gb NIC - Once 3060s removed + P40s installed → zero video output (P40s are compute-only, no display)

Other slots:

Plan (updated 2026-08-18): 3× Tesla P40 → the three x16 slots (00:02.0/00:03.0/80:02.0). Move /models NVMe to 80:00.0 with a proper PCIe3 x4 M.2 adapter (current £8.99 adapter is x1-traced — that's the 890 MB/s ceiling). Second NVMe → 80:01.0 (or 00:1c.4). 3060s → Z600 (turnkey AI box). Power: 3× 250W P40 + ~200W system ≈ 950W — within 1125W PSU (1450W UK).

NIC resolution options (choose one): 1. USB 10Gb adapter (Aquantia AQC107-based, ~£30-50) — good throughput, loses a USB port 2. Live with onboard 1Gb I210 — fine for SSH/Tailscale/model serving on LAN 3. 10Gb NIC in a spare x4/x8 slot — NO, no spare after P40s occupy all x16 + NVMe takes x4 slots

Video output resolution: - Lenovo ThinkPad Pro USB 3.0 Dock 40A7 (DisplayLink DL-3900) — purchased £12.94 (eBay 358912340509) - Provides VGA/DP output once Linux loads evdi driver — NO BIOS/POST display - For BIOS access: swap a spare GPU into a slot temporarily (hassle but rare need — zero local video needed in ~2 years of headless operation) - Intel AMT: NOT available (Z840 has HECI but AMT firmware not configured) — AND KVM anyway needs a GPU output to capture, so moot with no video card

P40 power connector (verified 2026-08-06): card has ONE female 8-pin, EPS-keyed with Tesla-specific pinout — standard PCIe 8-pin does NOT fit/function. NVIDIA ships dongle 030-0571-000 (CPU→PCIe 8-pin) in the box; if missing, buy “Dual 6-pin → EPS 8-pin for Tesla P40” (~£10). Z840 PSU feeds 6-pin rails (3060s run on 6-pin→8-pin adapters, fitted 2026-05-26) → chain: 6-pin rail → adapter → dongle → card. Steve ordered 1 spare 6-pin→8-pin adapter in case cards ship without dongle. P40 TDP 250W × 3 + system ≈ 950W total.

NVMe adapter to buy (bought 2026-08-06): JEYI M.2 NVMe → PCIe x4, aluminium heatsink, £7.09 ea ×2 (Fast Sam Shop, eBay 226923563359, PCIe x4 variant — NOT the x16). ECM20 discontinued on UK eBay; JEYI is the replacement. Links at PCIe 3.0 x4 (~3.5 GB/s) in Z840 slots.

NVMe adapter spec to buy: M.2 NVMe → PCIe 3.0 x4 (NOT x1!), full-height bracket, M.2 2280. e.g. SilverStone ECM20 (~£12) or generic “PCIe 3.0 x4 M.2 NVMe adapter” — check edge connector has x4 traces (most cheap £8.99 ones are x1). Heatsink recommended for sustained model loads.

### Phase 2 — Model Curation (pull → test → drop)

Once 3× P40 are live on GRID/Vulkan, the job is curating a model set that works as a unit within Hermes — served via llama-server (Vulkan multi-GPU), fitting the shared 72GB (KV caches included), and passing a real quality bar.

Method per candidate: pull trusted GGUF → load `-ngl 99 -ts <split>` → measure tok/s + VRAM + max context → keep (log serving config) or drop (log why).

Candidate queue:

Nemotron 3.5 Lightning notes (verified 2026-08-13): - Q4_K_M (25.48GB) needs 2 cards; IQ4_XS (18.92GB) / Q4_0 (19.06GB) fit 1 card. - Vulkan backend has `ssm_conv`/`ssm_scan` shaders → Mamba layers run on GPU (not CPU-offload). - Skip NVFP4 (22.46GB) — Blackwell-only, won't run on Pascal. - Pascal caveats: no tensor cores / bf16; 3B-active MoE is compute-light so tok/s should be usable, but serial Mamba scan is the throughput ceiling. - Source: `bartowski/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF`

## Model Storage (NVMe)

Model load times (approximate): - llama4:scout (67GB): ~75 seconds - qwen2.5:32b / deepseek-r1:32b (19GB): ~22 seconds - mistral-small3.1:24b (15GB): ~17 seconds

—

## Immich

Photo management and backup system.

Known issue — wal_compression (2026-05-30): After apt upgrade, immich_postgres crashed with `FATAL: parameter “wal_compression” requires a Boolean value`. Root cause: docker-compose.yml had `wal_compression=on`; pgvecto-rs pg14 image does not accept `on`. Fix: removed `wal_compression` parameter entirely from both `docker-compose.yml` and `/kaburudata/immich/postgres/postmaster.opts` (data volume persists startup opts — must patch both). Also removed `ports: 2283:2283` from immich-server (incompatible with `network_mode: host`).

## Portainer CE

Docker management UI. Installed 2026-05-30.

Note: Fresh Portainer installs time out after 5 minutes if admin account not created. If you see the timeout page, `docker restart portainer` and set the password immediately.

## ComfyUI

Image generation UI running on Docker with GPU access.

Setup notes: - Requires driver 580+ (CUDA 13.0) — upgraded 2026-05-26 - Use `ubuntu24_cuda12.6.3-latest` tag — the `cuda12.5.1` image ships a broken `comfy_kitchen` extension that requires CUDA 13 PyTorch despite the label - venv lives at `/kaburudata/docker/comfyui/run/venv-ubuntu24_cuda12.6.3` — persists across container rebuilds - Only GPU 0 assigned (`device=0`) — GPU 1 available for Ollama

Docker run command (for reference): ```bash docker run -d \

1. -name comfyui-image \
2. -runtime nvidia \
3. -gpus '“device=0”' \
4. p 8188:8188 \
5. v /kaburudata/docker/comfyui/run:/comfy/mnt \
6. e SECURITY_LEVEL=normal \
7. -restart unless-stopped \

mmartial/comfyui-nvidia-docker:ubuntu24_cuda12.6.3-latest ```

## VibeVoice (community TTS)

Long-form conversational TTS (Microsoft VibeVoice community fork). Installed 2026-08-07.

Note: txt scripts MUST use `Speaker N:` prefix format (see `demo/text_examples/1p_abs.txt`).

⚠️ NEVER use `–disable_prefill` for quality work (verified 2026-08-07): it skips voice conditioning entirely — model generates uncued, hallucinated speech (“horror movie” effect). Same text: no-prefill = 300 gen tokens / 37s audio; with prefill = 87 gen tokens / 11.3s of clean speech conditioned on the voice sample. Prefill costs ~same RTF (12-13x CPU) but produces far less junk audio.

P40 constraint (verified 2026-08-07): VibeVoice is PyTorch-CUDA, NOT llama.cpp/Vulkan. Torch dropped Pascal (sm_61) support in 2.8+/cu128 builds — installed torch 2.13 supports sm_75+ only (`torch.cuda.get_arch_list()`). P40s need torch ≤ 2.7.x (last with sm_61, e.g. 2.7.1+cu126) or a source build with `TORCH_CUDA_ARCH_LIST=“6.1”`. ComfyUI (also torch) has the same constraint. Plan: separate venv with torch 2.7.1+cu126 for VibeVoice/ComfyUI on P40s.

Future RLM trials (pending P40 install): See recursive-language-model (future RLM project). When 3× P40s are live, test Prime Agent / RLM harness for recursive context extension — could enable much longer context without context rot.

## Recent Changes