kyle/boxmaker
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Measure queued-request behaviour and reasoning_control before the M2 design

Browse Source 
A request pinned to a busy slot receives no bytes until the slot frees.
reasoning_control ends a thinking block on demand; the capped turn is
re-read once on the next request.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
kyle
2026-09-17 11:03:55 -07:00
co-authored by Claude Fable 5.1
parent 35a7a4e7ea
commit 362f962803
3 changed files with 113 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/inference-contract.md
+24 -4
View File
@@ -98,12 +98,32 @@ are on the same host. In router mode the router sets each child's host and port
router's public listener could move to a socket, and Open WebUI, OpenCode and Tailscale Serve need
it on TCP. `loopd` runs with `--network=none` and cannot reach host loopback. So `inferproxy` stays.
### Runaway control
### (j) What a queued request receives: nothing
Measured 2026-09-17 before the M2 design. Request A generated on slot 0 for 10 s. Request B, pinned
to the same slot with `stream` and `return_progress`, was sent while A was running. B received no
bytes at all, not even response headers, until A finished: first byte at 10.02 s, then progress
events and tokens as usual. The stream cannot tell "queued" from "dead". `GET /slots?model=…` can:
it shows `is_processing` for the slot. So the wait before the first byte needs its own, longer
limit, and `loopd` can poll `/slots` during it to tell a busy slot from a dead server.
### (k) Runaway control: `reasoning_control` works, and costs one turn of cache
`--reasoning-budget` is a server flag, not a request field. Per request there is `max_tokens`, and
`reasoning_control: true` plus `POST /v1/chat/completions/control` with `action: "reasoning_end"`,
which ends the thinking block of a running completion. The second one fits a per-turn thinking cap
enforced by `loopd` while it counts streamed reasoning tokens. Not yet exercised.
`reasoning_control: true` plus `POST /v1/chat/completions/control` with the completion's `id`,
`action: "reasoning_end"` and `model`.
Measured 2026-09-17: the control call was sent after 150 streamed reasoning chunks and returned
`{"success": true}`. Three more reasoning chunks arrived, then the model wrote its answer and
finished with `finish_reason: "stop"`.
The next turn, with the capped turn replayed exactly as streamed, had `cache_n` 85 and `prompt_n`
743: the server re-read the whole capped assistant turn. The same two turns with a natural end to
thinking gave `cache_n` 167, `prompt_n` 22. So what the server generated at a forced end is not
what the template renders from the replayed message, and the hybrid cache falls back to the
checkpoint at the end of the previous prompt. The cost is bounded by the thinking cap plus one
answer, and it is paid once. The control run also shows that streamed deltas, concatenated, replay
byte for byte.
### (a) Throughput