Record M2a limits, the accepted availability risk and three more measurements

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
2026-09-17 12:53:41 -07:00
co-authored by Claude Fable 5.1
parent eb1b7d2da2
commit 62ec663a0e
2 changed files with 20 additions and 0 deletions
+17
View File
@@ -125,6 +125,23 @@ checkpoint at the end of the previous prompt. The cost is bounded by the thinkin
answer, and it is paid once. The control run also shows that streamed deltas, concatenated, replay
byte for byte.
### (l) Closing the connection cancels generation
Measured 2026-09-17: a streaming request was closed by the client while the model was thinking.
`/slots` showed the slot idle 0.1 s later. A client that aborts frees the GPU.
### (m) Exact token counts while streaming
With `timings_per_token: true` every streamed chunk carries a `timings` object, including
`predicted_n`, the number of tokens generated so far. A thinking cap can count real tokens.
### (n) Tool-call arguments stream token by token
Measured 2026-09-17: the model wrote a 1,500-word essay into a `write_file` call. 1,999 tokens over
30 s arrived as 1,948 tool-call deltas, the longest gap between events was 0.33 s, and the
concatenated `arguments` fragments were valid JSON. The server does not hold a tool call back
until it parses, so a silence-based liveness limit is safe during long tool calls.
### (a) Throughput
Method: timed requests, server-reported `timings`, thinking off. "At depth 32k" means a 2k-token