166 lines
9.4 KiB
Markdown
166 lines
9.4 KiB
Markdown
# Inference contract: M0 measurements
|
|
|
|
Measured 2026-09-17 against straylight by `spike/m0.py` (throwaway, Python stdlib). Requests went to
|
|
`https://straylight.scylla-hammerhead.ts.net:10000`, model id `ornith-1.5-35b-a3b`, through
|
|
`/v1/chat/completions` with `temperature 0.6, top_p 0.95, top_k 20`. Request fields were taken from
|
|
the server README at tag `b10809`, the build that is running.
|
|
|
|
**Conditions.** Every number below was taken while another session was generating on Ornith slot 1
|
|
(an 85k to 105k-token conversation at about 50 tokens/s). Token counts (`cache_n`, `prompt_n`) are
|
|
not affected by that. Throughput is, so section (a) reports a separate run taken after the GPU went
|
|
idle. One check is still open: slot pinning (d).
|
|
|
|
## What is running
|
|
|
|
| Item | Value |
|
|
|---|---|
|
|
| Build | llama.cpp `b10809-5266f24` (nixpkgs-unstable `llama-cpp-0.4.0`, Vulkan backend) |
|
|
| Mode | Router: one public endpoint, one child `llama-server` per model, `--models-max 2` |
|
|
| Public listener | `0.0.0.0:11434`, firewalled to the tailnet; Tailscale Serve adds HTTPS on `:10000` |
|
|
| Other clients | Open WebUI and OpenCode use the same endpoint and the same Ornith instance |
|
|
| Ornith flags | `--jinja --no-mmap --ctx-size 262144 --parallel 2 --cache-type-k q8_0 --cache-type-v q8_0 --flash-attn on --n-gpu-layers 999 --sleep-idle-seconds 21600 --hf-repo ornith-ai/Ornith-1.5-35B-A3B-GGUF:Q4_K_M` |
|
|
| Slots | 2, each `n_ctx` 131072 (the 262144 is split, not shared) |
|
|
| Server default sampling | temperature 1.0, top_k 20, top_p 0.95, min_p 0.05. The harness must send its own. |
|
|
| Chat template | 7,828 bytes, sha256 `f55f52930aa8bf44ab5cb85f99370fcc3c56e9a85640b812086d5330bce5d86b` |
|
|
| Source of truth for flags | `~/src/nixos/hw/straylight/default.nix` on straylight, not this repo |
|
|
|
|
Differences from the design brief: KV cache is q8_0, not f16. There are two slots, not three. The
|
|
server is shared, so slots are not reserved for the harness. Weights and KV cache are dropped after
|
|
six idle hours. Host memory was 110 of 125 GB in use with Laguna S 2.1 and Ornith both loaded.
|
|
|
|
## Findings
|
|
|
|
### (b) Cache reuse over a 3-turn conversation: passes
|
|
|
|
Four tool schemas, thinking on, `reasoning_content` and `tool_calls` echoed back exactly as received.
|
|
|
|
| Request | `cache_n` | `prompt_n` |
|
|
|---|---|---|
|
|
| turn 1, request 1 | 0 | 539 |
|
|
| turn 1, request 2 (after tool result) | 591 | 29 |
|
|
| turn 2, request 1 | 680 | 27 |
|
|
| turn 2, request 2 | 759 | 40 |
|
|
| turn 3, request 1 | 840 | 26 |
|
|
|
|
Each request processes only its new tokens. Raw timing fields for one request:
|
|
`{"cache_n": 591, "prompt_n": 29, "prompt_ms": 227.163, "prompt_per_second": 127.66, "predicted_n": 61, "predicted_ms": 1762.327, "predicted_per_second": 34.05}`.
|
|
|
|
The whole baseline here (system line, four tool schemas, first user message) was 539 tokens, so the
|
|
3,000-token baseline budget is realistic.
|
|
|
|
### (c) Tool-call parsing through chat-completions: 0 failures in 20
|
|
|
|
Five prompts for each of four tools. Every response had `finish_reason: "tool_calls"`, the expected
|
|
tool, valid JSON arguments and all required arguments. Types survived: an integer `timeout_s`, a
|
|
nested `headers` object, and strings containing quotes, `&`, `<>` and embedded JSON.
|
|
|
|
### (e) Thinking blocks and the cache
|
|
|
|
- Ornith's template renders the `<think>` block of **every** assistant turn, not only the last one.
|
|
It does not strip earlier reasoning. The brief's "known risk" does not apply as long as the
|
|
harness sends `reasoning_content` back unchanged.
|
|
- If the harness drops `reasoning_content`, the prompt diverges at the latest assistant turn. The
|
|
cost was small (`prompt_n` 65 to 80 instead of 27 to 40), because the server keeps a checkpoint
|
|
near the end of the previous request.
|
|
- A change anywhere earlier costs a full re-read. Editing turn 3 or turn 2 of a 5-turn, 7.6k-token
|
|
conversation gave `cache_n` 34. Changing text 10k tokens into a 24k-token prompt gave `cache_n` 0
|
|
and 26 s of prompt processing. This confirms the brief: no partial rewind in practice.
|
|
- The server sometimes restored an older prompt from its host-RAM prompt cache (`--cache-ram`,
|
|
default 8 GiB): resending the original 24k prompt after the edited one gave `cache_n` 23758. It
|
|
did not do so every time. Do not design around it.
|
|
|
|
### (h) Changing the tool list mid-session: full invalidation
|
|
|
|
The template renders tool schemas at the very top of the prompt, before the system text. Adding a
|
|
fifth tool at turn 3 gave `cache_n` 23, `prompt_n` 920. The `tools` array must be fixed for the
|
|
whole epoch.
|
|
|
|
Progressive disclosure still works if the schema arrives as a tool result:
|
|
|
|
| Variant | Result |
|
|
|---|---|
|
|
| `find_tool` returns a schema, model calls it through a fixed `call_tool(name, arguments)` meta-tool | 4 of 5 correct. The one miss called `call_tool` without `find_tool` first, which `brokerd` can reject. |
|
|
| `find_tool` returns a schema, model calls the new tool directly by name | 0 of 5. The server's grammar only allows declared names, so the model was forced into a **wrong declared tool**: three times it emitted `write_file` with placeholder content. |
|
|
|
|
The second row is a safety finding, not only a cache one. Never tell the model to call a tool that
|
|
is not in the `tools` array.
|
|
|
|
### (i) Liveness during prompt processing
|
|
|
|
With `stream: true` and `return_progress: true`, a 16k-token prefill produced 11 `prompt_progress`
|
|
events and the longest silence was 2.2 s. Without `return_progress` the stream is silent for the
|
|
whole prefill. The liveness timeout in the brief needs this field.
|
|
|
|
### (g) Unix socket, co-location
|
|
|
|
`--host` accepts a path ending in `.sock` (README, build b10809). The harness and `llama-server`
|
|
are on the same host. In router mode the router sets each child's host and port itself, so only the
|
|
router's public listener could move to a socket, and Open WebUI, OpenCode and Tailscale Serve need
|
|
it on TCP. `loopd` runs with `--network=none` and cannot reach host loopback. So with the shared
|
|
router `inferproxy` stays. It goes away only if the harness gets its own `llama-server` on a socket.
|
|
|
|
### Runaway control
|
|
|
|
`--reasoning-budget` is a server flag, not a request field. Per request there is `max_tokens`, and
|
|
`reasoning_control: true` plus `POST /v1/chat/completions/control` with `action: "reasoning_end"`,
|
|
which ends the thinking block of a running completion. The second one fits a per-turn thinking cap
|
|
enforced by `loopd` while it counts streamed reasoning tokens. Not yet exercised.
|
|
|
|
### (a) Throughput
|
|
|
|
Method: timed requests, server-reported `timings`, thinking off. "At depth 32k" means a 2k-token
|
|
suffix appended to a cached 32k-token prefix, then generation from there.
|
|
|
|
| Measurement | GPU otherwise idle | Other Ornith slot generating |
|
|
|---|---|---|
|
|
| Prompt processing at depth 0 (2k tokens) | 1,170 tokens/s | 875 to 1,040 tokens/s |
|
|
| Prompt processing, average over 0 to 32k | 898 tokens/s (32k in 36 s) | 929 to 936 tokens/s over 0 to 24k |
|
|
| Prompt processing at depth 32k (2k suffix) | 669 tokens/s | not measured |
|
|
| Generation at depth 0 | 69 tokens/s | 28 to 34 tokens/s |
|
|
| Generation at depth 32k | 60 tokens/s | not measured |
|
|
|
|
The idle numbers match the brief's reference figures (about 1,100 and 700 tokens/s, 60 falling to
|
|
49). A second active session on the same model roughly halves generation speed.
|
|
|
|
### (d) Slot pinning: not yet measured
|
|
|
|
`id_slot` is a documented request field and the pinned requests above all landed on slot 0. The
|
|
two-session test (`m0.py pin`) needs slot 1 and sends unpinned requests, which would evict the
|
|
100k-token cache of the session that is using slot 1. It runs when the owner says slot 1 is free.
|
|
|
|
## Recommendation on the open question
|
|
|
|
Use the server's chat-completions endpoint with server-side tool parsing. Do not render the
|
|
template in-process. The cache measurements pass and tool parsing had no failures. The conditions
|
|
are:
|
|
|
|
1. The session log stores each assistant message exactly as returned (`content`,
|
|
`reasoning_content`, `tool_calls`) and replays it unchanged.
|
|
2. The `tools` array is fixed per epoch. Tools outside the core set are reached through `find_tool`
|
|
and a `call_tool` meta-tool.
|
|
3. Every request carries `id_slot`, `cache_prompt: true`, the sampling settings, `stream: true` and
|
|
`return_progress: true`.
|
|
4. The startup self-test compares the template hash and `n_ctx` from `/props?model=...` with the
|
|
values recorded here.
|
|
|
|
## Recommended serving setup
|
|
|
|
The contract cannot be met on a shared instance: any unpinned request from Open WebUI or OpenCode
|
|
can take a harness slot, loading a third model can unload Ornith, and the idle timer drops the
|
|
cache. Recommended: a dedicated `llama-server` systemd unit for Boxmaker, defined as a NixOS module
|
|
kept in this repo under `deploy/` and imported by `~/src/nixos`.
|
|
|
|
| Flag | Value | Reason |
|
|
|---|---|---|
|
|
| `--host` | `/run/boxmaker/llama.sock` | No TCP listener, no strangers, and `inferproxy` is not needed |
|
|
| `--parallel` | 3 | Main, subagent and scheduled slots |
|
|
| `--ctx-size` | 393216 | 131072 per slot |
|
|
| `--cache-type-k/v` | f16 | As the brief specifies. Not compared with q8_0 in M0. |
|
|
| `--jinja --flash-attn on --no-mmap --n-gpu-layers 999` | as now | |
|
|
| `--sleep-idle-seconds` | unset | Keep the cache across idle periods |
|
|
| no draft model | | The brief rules out speculative decoding |
|
|
|
|
Cost: a second resident copy of Ornith, about 22 GB of weights plus KV cache, beside Laguna's 69 GB
|
|
under the 104 GiB GPU memory cap. It will not fit if the shared router also keeps its own Ornith
|
|
loaded. This is the owner's call and is listed in `docs/decisions.md`.
|