boxmaker/docs/inference-contract.md

Inference contract: M0 measurements

Measured 2026-09-17 against straylight by spike/m0.py (throwaway, Python stdlib). Requests went to https://straylight.scylla-hammerhead.ts.net:10000, model id ornith-1.5-35b-a3b, through /v1/chat/completions with temperature 0.6, top_p 0.95, top_k 20. Request fields were taken from the server README at tag b10809, the build that is running.

Conditions. Every number below was taken while another session was generating on Ornith slot 1 (an 85k to 105k-token conversation at about 50 tokens/s). Token counts (cache_n, prompt_n) are not affected by that. Throughput is, so treat the rates as lower bounds. Two checks are still open for the same reason: clean throughput (a) and slot pinning (d).

What is running

Item	Value
Build	llama.cpp b10809-5266f24 (nixpkgs-unstable llama-cpp-0.4.0, Vulkan backend)
Mode	Router: one public endpoint, one child llama-server per model, --models-max 2
Public listener	0.0.0.0:11434, firewalled to the tailnet; Tailscale Serve adds HTTPS on :10000
Other clients	Open WebUI and OpenCode use the same endpoint and the same Ornith instance
Ornith flags	--jinja --no-mmap --ctx-size 262144 --parallel 2 --cache-type-k q8_0 --cache-type-v q8_0 --flash-attn on --n-gpu-layers 999 --sleep-idle-seconds 21600 --hf-repo ornith-ai/Ornith-1.5-35B-A3B-GGUF:Q4_K_M
Slots	2, each n_ctx 131072 (the 262144 is split, not shared)
Server default sampling	temperature 1.0, top_k 20, top_p 0.95, min_p 0.05. The harness must send its own.
Chat template	7,828 bytes, sha256 f55f52930aa8bf44ab5cb85f99370fcc3c56e9a85640b812086d5330bce5d86b
Source of truth for flags	~/src/nixos/hw/straylight/default.nix on straylight, not this repo

Differences from the design brief: KV cache is q8_0, not f16. There are two slots, not three. The server is shared, so slots are not reserved for the harness. Weights and KV cache are dropped after six idle hours. Host memory was 110 of 125 GB in use with Laguna S 2.1 and Ornith both loaded.

Findings

(b) Cache reuse over a 3-turn conversation: passes

Four tool schemas, thinking on, reasoning_content and tool_calls echoed back exactly as received.

Request	cache_n	prompt_n
turn 1, request 1	0	539
turn 1, request 2 (after tool result)	591	29
turn 2, request 1	680	27
turn 2, request 2	759	40
turn 3, request 1	840	26

Each request processes only its new tokens. Raw timing fields for one request: {"cache_n": 591, "prompt_n": 29, "prompt_ms": 227.163, "prompt_per_second": 127.66, "predicted_n": 61, "predicted_ms": 1762.327, "predicted_per_second": 34.05}.

The whole baseline here (system line, four tool schemas, first user message) was 539 tokens, so the 3,000-token baseline budget is realistic.

(c) Tool-call parsing through chat-completions: 0 failures in 20

Five prompts for each of four tools. Every response had finish_reason: "tool_calls", the expected tool, valid JSON arguments and all required arguments. Types survived: an integer timeout_s, a nested headers object, and strings containing quotes, &, <> and embedded JSON.

(e) Thinking blocks and the cache

• Ornith's template renders the <think> block of every assistant turn, not only the last one. It does not strip earlier reasoning. The brief's "known risk" does not apply as long as the harness sends reasoning_content back unchanged.
• If the harness drops reasoning_content, the prompt diverges at the latest assistant turn. The cost was small (prompt_n 65 to 80 instead of 27 to 40), because the server keeps a checkpoint near the end of the previous request.
• A change anywhere earlier costs a full re-read. Editing turn 3 or turn 2 of a 5-turn, 7.6k-token conversation gave cache_n 34. Changing text 10k tokens into a 24k-token prompt gave cache_n 0 and 26 s of prompt processing. This confirms the brief: no partial rewind in practice.
• The server sometimes restored an older prompt from its host-RAM prompt cache (--cache-ram, default 8 GiB): resending the original 24k prompt after the edited one gave cache_n 23758. It did not do so every time. Do not design around it.

(h) Changing the tool list mid-session: full invalidation

The template renders tool schemas at the very top of the prompt, before the system text. Adding a fifth tool at turn 3 gave cache_n 23, prompt_n 920. The tools array must be fixed for the whole epoch.

Progressive disclosure still works if the schema arrives as a tool result:

Variant	Result
find_tool returns a schema, model calls it through a fixed call_tool(name, arguments) meta-tool	4 of 5 correct. The one miss called call_tool without find_tool first, which brokerd can reject.
find_tool returns a schema, model calls the new tool directly by name	0 of 5. The server's grammar only allows declared names, so the model was forced into a wrong declared tool: three times it emitted write_file with placeholder content.

The second row is a safety finding, not only a cache one. Never tell the model to call a tool that is not in the tools array.

(i) Liveness during prompt processing

With stream: true and return_progress: true, a 16k-token prefill produced 11 prompt_progress events and the longest silence was 2.2 s. Without return_progress the stream is silent for the whole prefill. The liveness timeout in the brief needs this field.

(g) Unix socket, co-location

--host accepts a path ending in .sock (README, build b10809). The harness and llama-server are on the same host. In router mode the router sets each child's host and port itself, so only the router's public listener could move to a socket, and Open WebUI, OpenCode and Tailscale Serve need it on TCP. loopd runs with --network=none and cannot reach host loopback. So with the shared router inferproxy stays. It goes away only if the harness gets its own llama-server on a socket.

Runaway control

--reasoning-budget is a server flag, not a request field. Per request there is max_tokens, and reasoning_control: true plus POST /v1/chat/completions/control with action: "reasoning_end", which ends the thinking block of a running completion. The second one fits a per-turn thinking cap enforced by loopd while it counts streamed reasoning tokens. Not yet exercised.

(a) Throughput: contended numbers only

Measurement	Value (other slot busy)
Prompt processing, average over 0 to 24k	929 to 936 tokens/s
Prompt processing, 1.5k chunks at depth 0 to 7.6k	875 to 1,040 tokens/s
Generation	28 to 34 tokens/s

Method: timed requests, server-reported timings. The clean run (depth 0 and a 2k suffix at depth 32k) is scripted as m0.py throughput and waits for the GPU to be idle.

(d) Slot pinning: not yet measured

id_slot is a documented request field and the pinned requests above all landed on slot 0. The two-session test (m0.py pin) needs slot 1 and sends unpinned requests, which would evict the 100k-token cache of the session that is using slot 1. It runs when the owner says slot 1 is free.

Recommendation on the open question

Use the server's chat-completions endpoint with server-side tool parsing. Do not render the template in-process. The cache measurements pass and tool parsing had no failures. The conditions are:

1. The session log stores each assistant message exactly as returned (content, reasoning_content, tool_calls) and replays it unchanged.
2. The tools array is fixed per epoch. Tools outside the core set are reached through find_tool and a call_tool meta-tool.
3. Every request carries id_slot, cache_prompt: true, the sampling settings, stream: true and return_progress: true.
4. The startup self-test compares the template hash and n_ctx from /props?model=... with the values recorded here.

Recommended serving setup

The contract cannot be met on a shared instance: any unpinned request from Open WebUI or OpenCode can take a harness slot, loading a third model can unload Ornith, and the idle timer drops the cache. Recommended: a dedicated llama-server systemd unit for Boxmaker, defined as a NixOS module kept in this repo under deploy/ and imported by ~/src/nixos.

Flag	Value	Reason
--host	/run/boxmaker/llama.sock	No TCP listener, no strangers, and inferproxy is not needed
--parallel	3	Main, subagent and scheduled slots
--ctx-size	393216	131072 per slot
--cache-type-k/v	f16	As the brief specifies. Not compared with q8_0 in M0.
--jinja --flash-attn on --no-mmap --n-gpu-layers 999	as now
--sleep-idle-seconds	unset	Keep the cache across idle periods
no draft model		The brief rules out speculative decoding

Cost: a second resident copy of Ornith, about 22 GB of weights plus KV cache, beside Laguna's 69 GB under the 104 GiB GPU memory cap. It will not fit if the shared router also keeps its own Ornith loaded. This is the owner's call and is listed in docs/decisions.md.