From e104b77289910b86781f32b314dc4e3709cb4824 Mon Sep 17 00:00:00 2001 From: "K. Isom" Date: Thu, 17 Sep 2026 00:30:38 -0700 Subject: [PATCH] Add idle-GPU throughput numbers to M0 findings Co-Authored-By: Claude Fable 5.1 --- docs/inference-contract.md | 25 +++++++++++++++---------- 1 file changed, 15 insertions(+), 10 deletions(-) diff --git a/docs/inference-contract.md b/docs/inference-contract.md index f412b19..7009477 100644 --- a/docs/inference-contract.md +++ b/docs/inference-contract.md @@ -7,8 +7,8 @@ the server README at tag `b10809`, the build that is running. **Conditions.** Every number below was taken while another session was generating on Ornith slot 1 (an 85k to 105k-token conversation at about 50 tokens/s). Token counts (`cache_n`, `prompt_n`) are -not affected by that. Throughput is, so treat the rates as lower bounds. Two checks are still -open for the same reason: clean throughput (a) and slot pinning (d). +not affected by that. Throughput is, so section (a) reports a separate run taken after the GPU went +idle. One check is still open: slot pinning (d). ## What is running @@ -106,16 +106,21 @@ router `inferproxy` stays. It goes away only if the harness gets its own `llama- which ends the thinking block of a running completion. The second one fits a per-turn thinking cap enforced by `loopd` while it counts streamed reasoning tokens. Not yet exercised. -### (a) Throughput: contended numbers only +### (a) Throughput -| Measurement | Value (other slot busy) | -|---|---| -| Prompt processing, average over 0 to 24k | 929 to 936 tokens/s | -| Prompt processing, 1.5k chunks at depth 0 to 7.6k | 875 to 1,040 tokens/s | -| Generation | 28 to 34 tokens/s | +Method: timed requests, server-reported `timings`, thinking off. "At depth 32k" means a 2k-token +suffix appended to a cached 32k-token prefix, then generation from there. -Method: timed requests, server-reported `timings`. The clean run (depth 0 and a 2k suffix at depth -32k) is scripted as `m0.py throughput` and waits for the GPU to be idle. +| Measurement | GPU otherwise idle | Other Ornith slot generating | +|---|---|---| +| Prompt processing at depth 0 (2k tokens) | 1,170 tokens/s | 875 to 1,040 tokens/s | +| Prompt processing, average over 0 to 32k | 898 tokens/s (32k in 36 s) | 929 to 936 tokens/s over 0 to 24k | +| Prompt processing at depth 32k (2k suffix) | 669 tokens/s | not measured | +| Generation at depth 0 | 69 tokens/s | 28 to 34 tokens/s | +| Generation at depth 32k | 60 tokens/s | not measured | + +The idle numbers match the brief's reference figures (about 1,100 and 700 tokens/s, 60 falling to +49). A second active session on the same model roughly halves generation speed. ### (d) Slot pinning: not yet measured