kyle/boxmaker
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Replace dedicated-instance recommendation with shared-router consequences

Browse Source 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
kyle
2026-09-17 00:30:50 -07:00
co-authored by Claude Fable 5.1
parent e104b77289
commit 4b6ea143c7
1 changed files with 16 additions and 19 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/inference-contract.md
+16 -19
View File
@@ -96,8 +96,7 @@ whole prefill. The liveness timeout in the brief needs this field.
`--host` accepts a path ending in `.sock` (README, build b10809). The harness and `llama-server`
are on the same host. In router mode the router sets each child's host and port itself, so only the
router's public listener could move to a socket, and Open WebUI, OpenCode and Tailscale Serve need
it on TCP. `loopd` runs with `--network=none` and cannot reach host loopback. So with the shared
router `inferproxy` stays. It goes away only if the harness gets its own `llama-server` on a socket.
it on TCP. `loopd` runs with `--network=none` and cannot reach host loopback. So `inferproxy` stays.
### Runaway control
@@ -143,23 +142,21 @@ are:
4. The startup self-test compares the template hash and `n_ctx` from `/props?model=...` with the
values recorded here.
## Recommended serving setup
## Serving setup
The contract cannot be met on a shared instance: any unpinned request from Open WebUI or OpenCode
can take a harness slot, loading a third model can unload Ornith, and the idle timer drops the
cache. Recommended: a dedicated `llama-server` systemd unit for Boxmaker, defined as a NixOS module
kept in this repo under `deploy/` and imported by `~/src/nixos`.
Decided 2026-09-17: Boxmaker uses the shared router and the shared Ornith instance
(`docs/decisions.md`). A dedicated `llama-server` on a Unix socket was considered. It would have
given reserved slots and removed `inferproxy`, but a second resident copy of Ornith (about 22 GB
plus KV cache) does not fit beside Laguna's 69 GB under the 104 GiB GPU memory cap.
| Flag | Value | Reason |
|---|---|---|
| `--host` | `/run/boxmaker/llama.sock` | No TCP listener, no strangers, and `inferproxy` is not needed |
| `--parallel` | 3 | Main, subagent and scheduled slots |
| `--ctx-size` | 393216 | 131072 per slot |
| `--cache-type-k/v` | f16 | As the brief specifies. Not compared with q8_0 in M0. |
| `--jinja --flash-attn on --no-mmap --n-gpu-layers 999` | as now | |
| `--sleep-idle-seconds` | unset | Keep the cache across idle periods |
| no draft model | | The brief rules out speculative decoding |
What the shared instance means for the harness:
Cost: a second resident copy of Ornith, about 22 GB of weights plus KV cache, beside Laguna's 69 GB
under the 104 GiB GPU memory cap. It will not fit if the shared router also keeps its own Ornith
loaded. This is the owner's call and is listed in `docs/decisions.md`.
- `id_slot` pins the harness's own requests. It does not reserve the slot. An unpinned request
from another client can take it, loading a third model can unload Ornith (`--models-max 2`), and
six idle hours drop weights and cache. `loopd` must treat a cold cache as a normal event.
- A pinned request can wait behind another client's generation on the same slot.
- Another active session on Ornith roughly halves generation speed.
- Serving flags are owned by `~/src/nixos`. This repo records expected values and checks them at
startup.
- Suggested change on the NixOS side: Ornith `parallel = 4`, `ctx-size = 524288`, so the harness
can pin three slots and one is left for other clients. Extra KV memory not yet measured.