boxmaker/docs/inference-contract.md

Inference contract: M0 measurements

Measured 2026-09-17 against straylight by spike/m0.py (throwaway, Python stdlib). Requests went to https://straylight.scylla-hammerhead.ts.net:10000, model id ornith-1.5-35b-a3b, through /v1/chat/completions with temperature 0.6, top_p 0.95, top_k 20. Request fields were taken from the server README at tag b10809, the build that is running.

Conditions. Every number below was taken while another session was generating on Ornith slot 1 (an 85k to 105k-token conversation at about 50 tokens/s). Token counts (cache_n, prompt_n) are not affected by that. Throughput is, so section (a) reports a separate run taken after the GPU went idle. One check is still open: slot pinning (d).

What is running

Item	Value
Build	llama.cpp b10809-5266f24 (nixpkgs-unstable llama-cpp-0.4.0, Vulkan backend)
Mode	Router: one public endpoint, one child llama-server per model, --models-max 2
Public listener	0.0.0.0:11434, firewalled to the tailnet; Tailscale Serve adds HTTPS on :10000
Other clients	Open WebUI and OpenCode use the same endpoint and the same Ornith instance
Ornith flags	--jinja --no-mmap --ctx-size 262144 --parallel 2 --cache-type-k q8_0 --cache-type-v q8_0 --flash-attn on --n-gpu-layers 999 --sleep-idle-seconds 21600 --hf-repo ornith-ai/Ornith-1.5-35B-A3B-GGUF:Q4_K_M
Slots	2, each n_ctx 131072 (the 262144 is split, not shared)
Server default sampling	temperature 1.0, top_k 20, top_p 0.95, min_p 0.05. The harness must send its own.
Chat template	7,828 bytes, sha256 f55f52930aa8bf44ab5cb85f99370fcc3c56e9a85640b812086d5330bce5d86b
Source of truth for flags	~/src/nixos/hw/straylight/default.nix on straylight, not this repo

Differences from the design brief: KV cache is q8_0, not f16. There are two slots, not three. The server is shared, so slots are not reserved for the harness. Weights and KV cache are dropped after six idle hours. Host memory was 110 of 125 GB in use with Laguna S 2.1 and Ornith both loaded.

Findings

(b) Cache reuse over a 3-turn conversation: passes

Four tool schemas, thinking on, reasoning_content and tool_calls echoed back exactly as received.

Request	cache_n	prompt_n
turn 1, request 1	0	539
turn 1, request 2 (after tool result)	591	29
turn 2, request 1	680	27
turn 2, request 2	759	40
turn 3, request 1	840	26

Each request processes only its new tokens. Raw timing fields for one request: {"cache_n": 591, "prompt_n": 29, "prompt_ms": 227.163, "prompt_per_second": 127.66, "predicted_n": 61, "predicted_ms": 1762.327, "predicted_per_second": 34.05}.

The whole baseline here (system line, four tool schemas, first user message) was 539 tokens, so the 3,000-token baseline budget is realistic.

(c) Tool-call parsing through chat-completions: 0 failures in 20

Five prompts for each of four tools. Every response had finish_reason: "tool_calls", the expected tool, valid JSON arguments and all required arguments. Types survived: an integer timeout_s, a nested headers object, and strings containing quotes, &, <> and embedded JSON.

(e) Thinking blocks and the cache

• Ornith's template renders the <think> block of every assistant turn, not only the last one. It does not strip earlier reasoning. The brief's "known risk" does not apply as long as the harness sends reasoning_content back unchanged.
• If the harness drops reasoning_content, the prompt diverges at the latest assistant turn. The cost was small (prompt_n 65 to 80 instead of 27 to 40), because the server keeps a checkpoint near the end of the previous request.
• A change anywhere earlier costs a full re-read. Editing turn 3 or turn 2 of a 5-turn, 7.6k-token conversation gave cache_n 34. Changing text 10k tokens into a 24k-token prompt gave cache_n 0 and 26 s of prompt processing. This confirms the brief: no partial rewind in practice.
• The server sometimes restored an older prompt from its host-RAM prompt cache (--cache-ram, default 8 GiB): resending the original 24k prompt after the edited one gave cache_n 23758. It did not do so every time. Do not design around it.

(h) Changing the tool list mid-session: full invalidation

The template renders tool schemas at the very top of the prompt, before the system text. Adding a fifth tool at turn 3 gave cache_n 23, prompt_n 920. The tools array must be fixed for the whole epoch.

Progressive disclosure still works if the schema arrives as a tool result:

Variant	Result
find_tool returns a schema, model calls it through a fixed call_tool(name, arguments) meta-tool	4 of 5 correct. The one miss called call_tool without find_tool first, which brokerd can reject.
find_tool returns a schema, model calls the new tool directly by name	0 of 5. The server's grammar only allows declared names, so the model was forced into a wrong declared tool: three times it emitted write_file with placeholder content.

The second row is a safety finding, not only a cache one. Never tell the model to call a tool that is not in the tools array.

(i) Liveness during prompt processing

With stream: true and return_progress: true, a 16k-token prefill produced 11 prompt_progress events and the longest silence was 2.2 s. Without return_progress the stream is silent for the whole prefill. The liveness timeout in the brief needs this field.

(g) Unix socket, co-location

--host accepts a path ending in .sock (README, build b10809). The harness and llama-server are on the same host. In router mode the router sets each child's host and port itself, so only the router's public listener could move to a socket, and Open WebUI, OpenCode and Tailscale Serve need it on TCP. loopd runs with --network=none and cannot reach host loopback. So inferproxy stays.

Runaway control

--reasoning-budget is a server flag, not a request field. Per request there is max_tokens, and reasoning_control: true plus POST /v1/chat/completions/control with action: "reasoning_end", which ends the thinking block of a running completion. The second one fits a per-turn thinking cap enforced by loopd while it counts streamed reasoning tokens. Not yet exercised.

(a) Throughput

Method: timed requests, server-reported timings, thinking off. "At depth 32k" means a 2k-token suffix appended to a cached 32k-token prefix, then generation from there.

Measurement	GPU otherwise idle	Other Ornith slot generating
Prompt processing at depth 0 (2k tokens)	1,170 tokens/s	875 to 1,040 tokens/s
Prompt processing, average over 0 to 32k	898 tokens/s (32k in 36 s)	929 to 936 tokens/s over 0 to 24k
Prompt processing at depth 32k (2k suffix)	669 tokens/s	not measured
Generation at depth 0	69 tokens/s	28 to 34 tokens/s
Generation at depth 32k	60 tokens/s	not measured

The idle numbers match the brief's reference figures (about 1,100 and 700 tokens/s, 60 falling to 49). A second active session on the same model roughly halves generation speed.

(d) Slot pinning: not yet measured

id_slot is a documented request field and the pinned requests above all landed on slot 0. The two-session test (m0.py pin) needs slot 1 and sends unpinned requests, which would evict the 100k-token cache of the session that is using slot 1. It runs when the owner says slot 1 is free.

Recommendation on the open question

Use the server's chat-completions endpoint with server-side tool parsing. Do not render the template in-process. The cache measurements pass and tool parsing had no failures. The conditions are:

1. The session log stores each assistant message exactly as returned (content, reasoning_content, tool_calls) and replays it unchanged.
2. The tools array is fixed per epoch. Tools outside the core set are reached through find_tool and a call_tool meta-tool.
3. Every request carries id_slot, cache_prompt: true, the sampling settings, stream: true and return_progress: true.
4. The startup self-test compares the template hash and n_ctx from /props?model=... with the values recorded here.

Serving setup

Decided 2026-09-17: Boxmaker uses the shared router and the shared Ornith instance (docs/decisions.md). A dedicated llama-server on a Unix socket was considered. It would have given reserved slots and removed inferproxy, but a second resident copy of Ornith (about 22 GB plus KV cache) does not fit beside Laguna's 69 GB under the 104 GiB GPU memory cap.

What the shared instance means for the harness:

• id_slot pins the harness's own requests. It does not reserve the slot. An unpinned request from another client can take it, loading a third model can unload Ornith (--models-max 2), and six idle hours drop weights and cache. loopd must treat a cold cache as a normal event.
• A pinned request can wait behind another client's generation on the same slot.
• Another active session on Ornith roughly halves generation speed.
• Serving flags are owned by ~/src/nixos. This repo records expected values and checks them at startup.
• Suggested change on the NixOS side: Ornith parallel = 4, ctx-size = 524288, so the harness can pin three slots and one is left for other clients. Extra KV memory not yet measured.