kyle/boxmaker
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add idle-GPU throughput numbers to M0 findings

Browse Source 
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
kyle
2026-09-17 00:30:38 -07:00
co-authored by Claude Fable 5.1
parent 5bd0444aeb
commit e104b77289
1 changed files with 15 additions and 10 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/inference-contract.md
+15 -10
View File
@@ -7,8 +7,8 @@ the server README at tag `b10809`, the build that is running.
**Conditions.** Every number below was taken while another session was generating on Ornith slot 1 **Conditions.** Every number below was taken while another session was generating on Ornith slot 1
(an 85k to 105k-token conversation at about 50 tokens/s). Token counts (`cache_n`, `prompt_n`) are (an 85k to 105k-token conversation at about 50 tokens/s). Token counts (`cache_n`, `prompt_n`) are
not affected by that. Throughput is, so treat the rates as lower bounds. Two checks are still not affected by that. Throughput is, so section (a) reports a separate run taken after the GPU went
open for the same reason: clean throughput (a) and slot pinning (d). idle. One check is still open: slot pinning (d).
## What is running ## What is running
@@ -106,16 +106,21 @@ router `inferproxy` stays. It goes away only if the harness gets its own `llama-
which ends the thinking block of a running completion. The second one fits a per-turn thinking cap which ends the thinking block of a running completion. The second one fits a per-turn thinking cap
enforced by `loopd` while it counts streamed reasoning tokens. Not yet exercised. enforced by `loopd` while it counts streamed reasoning tokens. Not yet exercised.
### (a) Throughput: contended numbers only ### (a) Throughput
| Measurement | Value (other slot busy) | Method: timed requests, server-reported `timings`, thinking off. "At depth 32k" means a 2k-token
|---|---| suffix appended to a cached 32k-token prefix, then generation from there.
| Prompt processing, average over 0 to 24k | 929 to 936 tokens/s |
| Prompt processing, 1.5k chunks at depth 0 to 7.6k | 875 to 1,040 tokens/s |
| Generation | 28 to 34 tokens/s |
Method: timed requests, server-reported `timings`. The clean run (depth 0 and a 2k suffix at depth | Measurement | GPU otherwise idle | Other Ornith slot generating |
32k) is scripted as `m0.py throughput` and waits for the GPU to be idle. |---|---|---|
| Prompt processing at depth 0 (2k tokens) | 1,170 tokens/s | 875 to 1,040 tokens/s |
| Prompt processing, average over 0 to 32k | 898 tokens/s (32k in 36 s) | 929 to 936 tokens/s over 0 to 24k |
| Prompt processing at depth 32k (2k suffix) | 669 tokens/s | not measured |
| Generation at depth 0 | 69 tokens/s | 28 to 34 tokens/s |
| Generation at depth 32k | 60 tokens/s | not measured |
The idle numbers match the brief's reference figures (about 1,100 and 700 tokens/s, 60 falling to
49). A second active session on the same model roughly halves generation speed.
### (d) Slot pinning: not yet measured ### (d) Slot pinning: not yet measured