kyle/boxmaker
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add M0 measurement spike and findings

Browse Source 
spike/m0.py is throwaway. Findings are in docs/inference-contract.md:
cache reuse and tool parsing pass through chat-completions, the tools
array must stay fixed per epoch, and the shared router cannot meet the
slot contract. Clean throughput and slot pinning are still open because
another session was using the GPU.

docs/decisions.md lists the brief changes this implies as proposals.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
kyle
2026-09-17 00:22:31 -07:00
co-authored by Claude Fable 5.1
parent 96bf688b20
commit 06c05e0023
4 changed files with 491 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/inference-contract.md
+160
View File
@@ -0,0 +1,160 @@
# Inference contract: M0 measurements
Measured 2026-09-17 against straylight by `spike/m0.py` (throwaway, Python stdlib). Requests went to
`https://straylight.scylla-hammerhead.ts.net:10000`, model id `ornith-1.5-35b-a3b`, through
`/v1/chat/completions` with `temperature 0.6, top_p 0.95, top_k 20`. Request fields were taken from
the server README at tag `b10809`, the build that is running.
**Conditions.** Every number below was taken while another session was generating on Ornith slot 1
(an 85k to 105k-token conversation at about 50 tokens/s). Token counts (`cache_n`, `prompt_n`) are
not affected by that. Throughput is, so treat the rates as lower bounds. Two checks are still
open for the same reason: clean throughput (a) and slot pinning (d).
## What is running
| Item | Value |
|---|---|
| Build | llama.cpp `b10809-5266f24` (nixpkgs-unstable `llama-cpp-0.4.0`, Vulkan backend) |
| Mode | Router: one public endpoint, one child `llama-server` per model, `--models-max 2` |
| Public listener | `0.0.0.0:11434`, firewalled to the tailnet; Tailscale Serve adds HTTPS on `:10000` |
| Other clients | Open WebUI and OpenCode use the same endpoint and the same Ornith instance |
| Ornith flags | `--jinja --no-mmap --ctx-size 262144 --parallel 2 --cache-type-k q8_0 --cache-type-v q8_0 --flash-attn on --n-gpu-layers 999 --sleep-idle-seconds 21600 --hf-repo ornith-ai/Ornith-1.5-35B-A3B-GGUF:Q4_K_M` |
| Slots | 2, each `n_ctx` 131072 (the 262144 is split, not shared) |
| Server default sampling | temperature 1.0, top_k 20, top_p 0.95, min_p 0.05. The harness must send its own. |
| Chat template | 7,828 bytes, sha256 `f55f52930aa8bf44ab5cb85f99370fcc3c56e9a85640b812086d5330bce5d86b` |
| Source of truth for flags | `~/src/nixos/hw/straylight/default.nix` on straylight, not this repo |
Differences from the design brief: KV cache is q8_0, not f16. There are two slots, not three. The
server is shared, so slots are not reserved for the harness. Weights and KV cache are dropped after
six idle hours. Host memory was 110 of 125 GB in use with Laguna S 2.1 and Ornith both loaded.
## Findings
### (b) Cache reuse over a 3-turn conversation: passes
Four tool schemas, thinking on, `reasoning_content` and `tool_calls` echoed back exactly as received.
| Request | `cache_n` | `prompt_n` |
|---|---|---|
| turn 1, request 1 | 0 | 539 |
| turn 1, request 2 (after tool result) | 591 | 29 |
| turn 2, request 1 | 680 | 27 |
| turn 2, request 2 | 759 | 40 |
| turn 3, request 1 | 840 | 26 |
Each request processes only its new tokens. Raw timing fields for one request:
`{"cache_n": 591, "prompt_n": 29, "prompt_ms": 227.163, "prompt_per_second": 127.66, "predicted_n": 61, "predicted_ms": 1762.327, "predicted_per_second": 34.05}`.
The whole baseline here (system line, four tool schemas, first user message) was 539 tokens, so the
3,000-token baseline budget is realistic.
### (c) Tool-call parsing through chat-completions: 0 failures in 20
Five prompts for each of four tools. Every response had `finish_reason: "tool_calls"`, the expected
tool, valid JSON arguments and all required arguments. Types survived: an integer `timeout_s`, a
nested `headers` object, and strings containing quotes, `&`, `<>` and embedded JSON.
### (e) Thinking blocks and the cache
- Ornith's template renders the `<think>` block of **every** assistant turn, not only the last one.
It does not strip earlier reasoning. The brief's "known risk" does not apply as long as the
harness sends `reasoning_content` back unchanged.
- If the harness drops `reasoning_content`, the prompt diverges at the latest assistant turn. The
cost was small (`prompt_n` 65 to 80 instead of 27 to 40), because the server keeps a checkpoint
near the end of the previous request.
- A change anywhere earlier costs a full re-read. Editing turn 3 or turn 2 of a 5-turn, 7.6k-token
conversation gave `cache_n` 34. Changing text 10k tokens into a 24k-token prompt gave `cache_n` 0
and 26 s of prompt processing. This confirms the brief: no partial rewind in practice.
- The server sometimes restored an older prompt from its host-RAM prompt cache (`--cache-ram`,
default 8 GiB): resending the original 24k prompt after the edited one gave `cache_n` 23758. It
did not do so every time. Do not design around it.
### (h) Changing the tool list mid-session: full invalidation
The template renders tool schemas at the very top of the prompt, before the system text. Adding a
fifth tool at turn 3 gave `cache_n` 23, `prompt_n` 920. The `tools` array must be fixed for the
whole epoch.
Progressive disclosure still works if the schema arrives as a tool result:
| Variant | Result |
|---|---|
| `find_tool` returns a schema, model calls it through a fixed `call_tool(name, arguments)` meta-tool | 4 of 5 correct. The one miss called `call_tool` without `find_tool` first, which `brokerd` can reject. |
| `find_tool` returns a schema, model calls the new tool directly by name | 0 of 5. The server's grammar only allows declared names, so the model was forced into a **wrong declared tool**: three times it emitted `write_file` with placeholder content. |
The second row is a safety finding, not only a cache one. Never tell the model to call a tool that
is not in the `tools` array.
### (i) Liveness during prompt processing
With `stream: true` and `return_progress: true`, a 16k-token prefill produced 11 `prompt_progress`
events and the longest silence was 2.2 s. Without `return_progress` the stream is silent for the
whole prefill. The liveness timeout in the brief needs this field.
### (g) Unix socket, co-location
`--host` accepts a path ending in `.sock` (README, build b10809). The harness and `llama-server`
are on the same host. In router mode the router sets each child's host and port itself, so only the
router's public listener could move to a socket, and Open WebUI, OpenCode and Tailscale Serve need
it on TCP. `loopd` runs with `--network=none` and cannot reach host loopback. So with the shared
router `inferproxy` stays. It goes away only if the harness gets its own `llama-server` on a socket.
### Runaway control
`--reasoning-budget` is a server flag, not a request field. Per request there is `max_tokens`, and
`reasoning_control: true` plus `POST /v1/chat/completions/control` with `action: "reasoning_end"`,
which ends the thinking block of a running completion. The second one fits a per-turn thinking cap
enforced by `loopd` while it counts streamed reasoning tokens. Not yet exercised.
### (a) Throughput: contended numbers only
| Measurement | Value (other slot busy) |
|---|---|
| Prompt processing, average over 0 to 24k | 929 to 936 tokens/s |
| Prompt processing, 1.5k chunks at depth 0 to 7.6k | 875 to 1,040 tokens/s |
| Generation | 28 to 34 tokens/s |
Method: timed requests, server-reported `timings`. The clean run (depth 0 and a 2k suffix at depth
32k) is scripted as `m0.py throughput` and waits for the GPU to be idle.
### (d) Slot pinning: not yet measured
`id_slot` is a documented request field and the pinned requests above all landed on slot 0. The
two-session test (`m0.py pin`) needs slot 1 and sends unpinned requests, which would evict the
100k-token cache of the session that is using slot 1. It runs when the owner says slot 1 is free.
## Recommendation on the open question
Use the server's chat-completions endpoint with server-side tool parsing. Do not render the
template in-process. The cache measurements pass and tool parsing had no failures. The conditions
are:
1. The session log stores each assistant message exactly as returned (`content`,
`reasoning_content`, `tool_calls`) and replays it unchanged.
2. The `tools` array is fixed per epoch. Tools outside the core set are reached through `find_tool`
and a `call_tool` meta-tool.
3. Every request carries `id_slot`, `cache_prompt: true`, the sampling settings, `stream: true` and
`return_progress: true`.
4. The startup self-test compares the template hash and `n_ctx` from `/props?model=...` with the
values recorded here.
## Recommended serving setup
The contract cannot be met on a shared instance: any unpinned request from Open WebUI or OpenCode
can take a harness slot, loading a third model can unload Ornith, and the idle timer drops the
cache. Recommended: a dedicated `llama-server` systemd unit for Boxmaker, defined as a NixOS module
kept in this repo under `deploy/` and imported by `~/src/nixos`.
| Flag | Value | Reason |
|---|---|---|
| `--host` | `/run/boxmaker/llama.sock` | No TCP listener, no strangers, and `inferproxy` is not needed |
| `--parallel` | 3 | Main, subagent and scheduled slots |
| `--ctx-size` | 393216 | 131072 per slot |
| `--cache-type-k/v` | f16 | As the brief specifies. Not compared with q8_0 in M0. |
| `--jinja --flash-attn on --no-mmap --n-gpu-layers 999` | as now | |
| `--sleep-idle-seconds` | unset | Keep the cache across idle periods |
| no draft model | | The brief rules out speculative decoding |
Cost: a second resident copy of Ornith, about 22 GB of weights plus KV cache, beside Laguna's 69 GB
under the 104 GiB GPU memory cap. It will not fit if the shared router also keeps its own Ornith
loaded. This is the owner's call and is listed in `docs/decisions.md`.