From e07fddf0ada85757a0d30d6054d651ba4a777c7f Mon Sep 17 00:00:00 2001 From: "K. Isom" Date: Thu, 17 Sep 2026 00:38:32 -0700 Subject: [PATCH] Brief P8: describe the shared router as deployed Approved 2026-09-17. Evidence is in docs/decisions.md. Co-Authored-By: Claude Fable 5.1 --- docs/decisions.md | 2 +- docs/design.md | 26 +++++++++++++++----------- 2 files changed, 16 insertions(+), 12 deletions(-) diff --git a/docs/decisions.md b/docs/decisions.md index e8d9414..e35a1c0 100644 --- a/docs/decisions.md +++ b/docs/decisions.md @@ -32,6 +32,7 @@ Approved by the owner on 2026-09-17. One commit each. | P5 | Settle the open question: chat-completions with server-side tool parsing. | M0 (b), (c). | | P6 | Code constraints: `Decision` lives in `brokerd`, has a private field and does not implement `Deserialize`. `proto` carries a plain `DecisionRecord` for the audit log. | Rust privacy is per crate, and a deserializable type can be built by anyone. | | P7 | Inference contract 1: the baseline budget test needs the server's tokenizer, so `make gate` has an offline part and an on-device part (`make verify-device`). | `/tokenize` is a server endpoint. | +| P8 | Target environment: describe the shared router as deployed (router mode, `--models-max 2`, other clients on the same Ornith instance, q8_0 KV, six-hour idle unload). Drop "memory is abundant". | `docs/inference-contract.md`, "What is running". | ## Proposed changes to the design brief @@ -39,7 +40,6 @@ From M0 and the kickoff review. None is applied yet. Each lands as its own commi | # | Change | Evidence | |---|---|---| -| P8 | Target environment: describe the shared router as deployed (router mode, `--models-max 2`, other clients on the same Ornith instance, q8_0 KV, six-hour idle unload). Drop "memory is abundant". | `docs/inference-contract.md`, "What is running". | | P9 | Inference contract 5: slots are pinned with `id_slot` but not reserved. Another client can evict a harness slot, and the router can unload Ornith. Cache loss is an expected event: `loopd` detects it (`cache_n` far below the previous request's total), logs it in the session log, and carries on. It is never an error. A session always uses the same slot; threads may share the main slot. | Shared router decision. M0 (d): evicted sessions came back from the host-RAM prompt cache in under a second on the same slot, and cost a full re-read on another slot. | | P10 | Inference contract 7: a pinned request can queue behind another client's request on the same slot, and the router can spend tens of seconds reloading the model. The liveness timer starts at the first byte received, and a separate, longer "waiting for slot or model" limit covers the time before it. | Shared router decision. Needs a measurement of what the stream sends while queued. | | P11 | Inference contract 10: the serving flags live in `~/src/nixos`, not here. This repo records the expected values (template hash, per-slot `n_ctx`, slot count, KV type, sampling) and the startup self-test compares them with `/props` and `/slots`. KV type is recorded as q8_0. | Shared router decision. | diff --git a/docs/design.md b/docs/design.md index 604d565..aa48986 100644 --- a/docs/design.md +++ b/docs/design.md @@ -16,17 +16,21 @@ authority, state in plain files the owner can read, a codebase small enough to a ## Target environment -- Inference host: `straylight`, AMD Strix Halo, 128 GB unified memory, llama.cpp `llama-server`, - ~128k context configured. -- Model: Ornith-1.5-35B-A3B (Qwen3.5-family MoE, hybrid linear/full attention, ~20 KB/token KV - at f16, Qwen XML tool calls with thinking blocks). -- Memory is abundant. Generation is fast. **Prompt processing is slow and cache invalidation is - the dominant cost.** Hybrid-attention models cannot partially rewind their cache. -- Reference numbers for the same model family on Strix Halo (Qwen3.6-35B-A3B Q4, llama.cpp, - community benchmark grid, May 2026): prompt processing ~1,100 tokens/s at depth 0 and ~700 at - 32k depth; generation ~60 tokens/s falling to ~49. M0 replaces these with measurements from - straylight. At these rates an uncached 18k-token prompt costs 16–25 s and a 64k re-read costs - well over a minute. +- Inference host: `straylight`, AMD Strix Halo, 128 GB unified memory, of which the GPU may + address 104 GiB. The harness runs on the same host. +- Serving: llama.cpp `llama-server` in router mode. One endpoint, one child server per model, at + most two models loaded. The owner's coding agents and Open WebUI use the same router and the + same Ornith instance. Details and measurements are in `docs/inference-contract.md`. +- Model: Ornith-1.5-35B-A3B (Qwen3.5-family MoE, hybrid linear/full attention, Qwen XML tool calls + with thinking blocks). As deployed: Q4_K_M weights, q8_0 KV cache, two slots of 131,072 tokens. + Weights and cache are unloaded after six idle hours, or when a third model is loaded. +- Memory is not abundant: Laguna S 2.1 (69 GB) and Ornith (22 GB) are normally both loaded. + Generation is fast. **Prompt processing is slow and cache invalidation is the dominant cost.** + Hybrid-attention models cannot partially rewind their cache; M0 confirmed it. +- Measured in M0 with the GPU otherwise idle: prompt processing 1,170 tokens/s at depth 0 and 669 + at 32k depth; generation 69 tokens/s falling to 60. A second active session on Ornith roughly + halves generation speed. An uncached 32k-token prompt took 36 s, so a 64k re-read costs well + over a minute. - Deployment: rootless containers on a host that is already a tailnet node. Containers do not get their own tailnet identities, so tailnet ACLs cannot tell one role from another. Isolation between roles is done with container networking on the host (see "Network isolation"). Tailnet