kyle/boxmaker
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add M0 measurement spike and findings

Browse Source 
spike/m0.py is throwaway. Findings are in docs/inference-contract.md:
cache reuse and tool parsing pass through chat-completions, the tools
array must stay fixed per epoch, and the shared router cannot meet the
slot contract. Clean throughput and slot pinning are still open because
another session was using the GPU.

docs/decisions.md lists the brief changes this implies as proposals.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
kyle
2026-09-17 00:22:31 -07:00
co-authored by Claude Fable 5.1
parent 96bf688b20
commit 06c05e0023
4 changed files with 491 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/decisions.md
+16
View File
@@ -15,5 +15,21 @@ Newest first. A decision that changes `docs/design.md` lands in the same commit
| Decision | Needed by |
|---|---|
| Serving setup: a dedicated `llama-server` unit for Boxmaker on a Unix socket (recommended in `docs/inference-contract.md`), or the shared router plus `inferproxy`. Decides whether the `inferproxy` crate exists and whether three pinned slots are available. | M1 |
| Secret store backend, and where the v0 file's key lives. straylight has no secrets manager today. | M3 |
| Whether cloud-led sessions are ever allowed, and for which data classes. | M6 |
## Proposed changes to the design brief
From M0 and the kickoff review. None is applied yet. Each lands as its own commit once the owner agrees.
| # | Change | Evidence |
|---|---|---|
| P1 | Inference contract 6: the `tools` array is fixed per epoch. `find_tool` returns schemas as a tool result and the model calls them through a `call_tool(name, arguments)` meta-tool. Never instruct the model to call an undeclared tool. | M0 (h): adding a tool re-read the whole prompt; undeclared calls were coerced into `write_file`. |
| P2 | Inference contract 2: the session log stores assistant messages exactly as returned, including `reasoning_content`, and replays them unchanged. Remove the "known risk" about dropped thinking blocks. | M0 (e): Ornith's template keeps every think block. |
| P3 | Inference contract 7: requests set `return_progress: true`; progress events count as liveness. | M0 (i): otherwise the stream is silent during prefill. |
| P4 | Inference contract 8: the thinking cap uses `reasoning_control` and the control endpoint. | README b10809. Not yet exercised. |
| P5 | Settle the open question: chat-completions with server-side tool parsing. | M0 (b), (c). |
| P6 | Code constraints: `Decision` lives in `brokerd`, has a private field and does not implement `Deserialize`. `proto` carries a plain `DecisionRecord` for the audit log. | Rust privacy is per crate, and a deserializable type can be built by anyone. |
| P7 | Inference contract 1: the baseline budget test needs the server's tokenizer, so `make gate` has an offline part and an on-device part (`make verify-device`). | `/tokenize` is a server endpoint. |
| P8 | Target environment: replace "memory is abundant" and the f16 and slot assumptions with the measured setup, once the serving decision is made. | `docs/inference-contract.md`, "What is running". |