Record M2a limits, the accepted availability risk and three more measurements
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
@@ -125,6 +125,23 @@ checkpoint at the end of the previous prompt. The cost is bounded by the thinkin
|
||||
answer, and it is paid once. The control run also shows that streamed deltas, concatenated, replay
|
||||
byte for byte.
|
||||
|
||||
### (l) Closing the connection cancels generation
|
||||
|
||||
Measured 2026-09-17: a streaming request was closed by the client while the model was thinking.
|
||||
`/slots` showed the slot idle 0.1 s later. A client that aborts frees the GPU.
|
||||
|
||||
### (m) Exact token counts while streaming
|
||||
|
||||
With `timings_per_token: true` every streamed chunk carries a `timings` object, including
|
||||
`predicted_n`, the number of tokens generated so far. A thinking cap can count real tokens.
|
||||
|
||||
### (n) Tool-call arguments stream token by token
|
||||
|
||||
Measured 2026-09-17: the model wrote a 1,500-word essay into a `write_file` call. 1,999 tokens over
|
||||
30 s arrived as 1,948 tool-call deltas, the longest gap between events was 0.33 s, and the
|
||||
concatenated `arguments` fragments were valid JSON. The server does not hold a tool call back
|
||||
until it parses, so a silence-based liveness limit is safe during long tool calls.
|
||||
|
||||
### (a) Throughput
|
||||
|
||||
Method: timed requests, server-reported `timings`, thinking off. "At depth 32k" means a 2k-token
|
||||
|
||||
Reference in New Issue
Block a user