From 4b6ea143c763b6b82e19f7f26e75551a8fe5d4f6 Mon Sep 17 00:00:00 2001 From: "K. Isom" Date: Thu, 17 Sep 2026 00:30:50 -0700 Subject: [PATCH] Replace dedicated-instance recommendation with shared-router consequences Co-Authored-By: Claude Fable 5.1 --- docs/inference-contract.md | 35 ++++++++++++++++------------------- 1 file changed, 16 insertions(+), 19 deletions(-) diff --git a/docs/inference-contract.md b/docs/inference-contract.md index 7009477..db42886 100644 --- a/docs/inference-contract.md +++ b/docs/inference-contract.md @@ -96,8 +96,7 @@ whole prefill. The liveness timeout in the brief needs this field. `--host` accepts a path ending in `.sock` (README, build b10809). The harness and `llama-server` are on the same host. In router mode the router sets each child's host and port itself, so only the router's public listener could move to a socket, and Open WebUI, OpenCode and Tailscale Serve need -it on TCP. `loopd` runs with `--network=none` and cannot reach host loopback. So with the shared -router `inferproxy` stays. It goes away only if the harness gets its own `llama-server` on a socket. +it on TCP. `loopd` runs with `--network=none` and cannot reach host loopback. So `inferproxy` stays. ### Runaway control @@ -143,23 +142,21 @@ are: 4. The startup self-test compares the template hash and `n_ctx` from `/props?model=...` with the values recorded here. -## Recommended serving setup +## Serving setup -The contract cannot be met on a shared instance: any unpinned request from Open WebUI or OpenCode -can take a harness slot, loading a third model can unload Ornith, and the idle timer drops the -cache. Recommended: a dedicated `llama-server` systemd unit for Boxmaker, defined as a NixOS module -kept in this repo under `deploy/` and imported by `~/src/nixos`. +Decided 2026-09-17: Boxmaker uses the shared router and the shared Ornith instance +(`docs/decisions.md`). A dedicated `llama-server` on a Unix socket was considered. It would have +given reserved slots and removed `inferproxy`, but a second resident copy of Ornith (about 22 GB +plus KV cache) does not fit beside Laguna's 69 GB under the 104 GiB GPU memory cap. -| Flag | Value | Reason | -|---|---|---| -| `--host` | `/run/boxmaker/llama.sock` | No TCP listener, no strangers, and `inferproxy` is not needed | -| `--parallel` | 3 | Main, subagent and scheduled slots | -| `--ctx-size` | 393216 | 131072 per slot | -| `--cache-type-k/v` | f16 | As the brief specifies. Not compared with q8_0 in M0. | -| `--jinja --flash-attn on --no-mmap --n-gpu-layers 999` | as now | | -| `--sleep-idle-seconds` | unset | Keep the cache across idle periods | -| no draft model | | The brief rules out speculative decoding | +What the shared instance means for the harness: -Cost: a second resident copy of Ornith, about 22 GB of weights plus KV cache, beside Laguna's 69 GB -under the 104 GiB GPU memory cap. It will not fit if the shared router also keeps its own Ornith -loaded. This is the owner's call and is listed in `docs/decisions.md`. +- `id_slot` pins the harness's own requests. It does not reserve the slot. An unpinned request + from another client can take it, loading a third model can unload Ornith (`--models-max 2`), and + six idle hours drop weights and cache. `loopd` must treat a cold cache as a normal event. +- A pinned request can wait behind another client's generation on the same slot. +- Another active session on Ornith roughly halves generation speed. +- Serving flags are owned by `~/src/nixos`. This repo records expected values and checks them at + startup. +- Suggested change on the NixOS side: Ornith `parallel = 4`, `ctx-size = 524288`, so the harness + can pin three slots and one is left for other clients. Extra KV memory not yet measured.