Brief P8: describe the shared router as deployed

Approved 2026-09-17. Evidence is in docs/decisions.md.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

docs/decisions.md

@@ -32,6 +32,7 @@ Approved by the owner on 2026-09-17. One commit each.
 | P5 | Settle the open question: chat-completions with server-side tool parsing. | M0 (b), (c). |
 | P6 | Code constraints: `Decision` lives in `brokerd`, has a private field and does not implement `Deserialize`. `proto` carries a plain `DecisionRecord` for the audit log. | Rust privacy is per crate, and a deserializable type can be built by anyone. |
 | P7 | Inference contract 1: the baseline budget test needs the server's tokenizer, so `make gate` has an offline part and an on-device part (`make verify-device`). | `/tokenize` is a server endpoint. |
+| P8 | Target environment: describe the shared router as deployed (router mode, `--models-max 2`, other clients on the same Ornith instance, q8_0 KV, six-hour idle unload). Drop "memory is abundant". | `docs/inference-contract.md`, "What is running". |
 
 ## Proposed changes to the design brief
 
@@ -39,7 +40,6 @@ From M0 and the kickoff review. None is applied yet. Each lands as its own commi
 
 | # | Change | Evidence |
 |---|---|---|
-| P8 | Target environment: describe the shared router as deployed (router mode, `--models-max 2`, other clients on the same Ornith instance, q8_0 KV, six-hour idle unload). Drop "memory is abundant". | `docs/inference-contract.md`, "What is running". |
 | P9 | Inference contract 5: slots are pinned with `id_slot` but not reserved. Another client can evict a harness slot, and the router can unload Ornith. Cache loss is an expected event: `loopd` detects it (`cache_n` far below the previous request's total), logs it in the session log, and carries on. It is never an error. A session always uses the same slot; threads may share the main slot. | Shared router decision. M0 (d): evicted sessions came back from the host-RAM prompt cache in under a second on the same slot, and cost a full re-read on another slot. |
 | P10 | Inference contract 7: a pinned request can queue behind another client's request on the same slot, and the router can spend tens of seconds reloading the model. The liveness timer starts at the first byte received, and a separate, longer "waiting for slot or model" limit covers the time before it. | Shared router decision. Needs a measurement of what the stream sends while queued. |
 | P11 | Inference contract 10: the serving flags live in `~/src/nixos`, not here. This repo records the expected values (template hash, per-slot `n_ctx`, slot count, KV type, sampling) and the startup self-test compares them with `/props` and `/slots`. KV type is recorded as q8_0. | Shared router decision. |

docs/design.md

@@ -16,17 +16,21 @@ authority, state in plain files the owner can read, a codebase small enough to a
 
 ## Target environment
 
-- Inference host: `straylight`, AMD Strix Halo, 128 GB unified memory, llama.cpp `llama-server`,
+- Inference host: `straylight`, AMD Strix Halo, 128 GB unified memory, of which the GPU may
-  ~128k context configured.
+  address 104 GiB. The harness runs on the same host.
-- Model: Ornith-1.5-35B-A3B (Qwen3.5-family MoE, hybrid linear/full attention, ~20 KB/token KV
+- Serving: llama.cpp `llama-server` in router mode. One endpoint, one child server per model, at
-  at f16, Qwen XML tool calls with thinking blocks).
+  most two models loaded. The owner's coding agents and Open WebUI use the same router and the
-- Memory is abundant. Generation is fast. **Prompt processing is slow and cache invalidation is
+  same Ornith instance. Details and measurements are in `docs/inference-contract.md`.
-  the dominant cost.** Hybrid-attention models cannot partially rewind their cache.
+- Model: Ornith-1.5-35B-A3B (Qwen3.5-family MoE, hybrid linear/full attention, Qwen XML tool calls
-- Reference numbers for the same model family on Strix Halo (Qwen3.6-35B-A3B Q4, llama.cpp,
+  with thinking blocks). As deployed: Q4_K_M weights, q8_0 KV cache, two slots of 131,072 tokens.
-  community benchmark grid, May 2026): prompt processing ~1,100 tokens/s at depth 0 and ~700 at
+  Weights and cache are unloaded after six idle hours, or when a third model is loaded.
-  32k depth; generation ~60 tokens/s falling to ~49. M0 replaces these with measurements from
+- Memory is not abundant: Laguna S 2.1 (69 GB) and Ornith (22 GB) are normally both loaded.
-  straylight. At these rates an uncached 18k-token prompt costs 16–25 s and a 64k re-read costs
+  Generation is fast. **Prompt processing is slow and cache invalidation is the dominant cost.**
-  well over a minute.
+  Hybrid-attention models cannot partially rewind their cache; M0 confirmed it.
+- Measured in M0 with the GPU otherwise idle: prompt processing 1,170 tokens/s at depth 0 and 669
+  at 32k depth; generation 69 tokens/s falling to 60. A second active session on Ornith roughly
+  halves generation speed. An uncached 32k-token prompt took 36 s, so a 64k re-read costs well
+  over a minute.
 - Deployment: rootless containers on a host that is already a tailnet node. Containers do not get
   their own tailnet identities, so tailnet ACLs cannot tell one role from another. Isolation
   between roles is done with container networking on the host (see "Network isolation"). Tailnet