kyle/boxmaker
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Brief P7: split the gate into offline and on-device parts

Browse Source 
Approved 2026-09-17. Evidence is in docs/decisions.md.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
kyle
2026-09-17 00:38:32 -07:00
co-authored by Claude Fable 5.1
parent 4fc7ad3deb
commit 46918e5208
2 changed files with 4 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/decisions.md
+1 -1
View File
@@ -31,6 +31,7 @@ Approved by the owner on 2026-09-17. One commit each.
| P4 | Inference contract 8: the thinking cap uses `reasoning_control` and the control endpoint. | README b10809. Not yet exercised. | | P4 | Inference contract 8: the thinking cap uses `reasoning_control` and the control endpoint. | README b10809. Not yet exercised. |
| P5 | Settle the open question: chat-completions with server-side tool parsing. | M0 (b), (c). | | P5 | Settle the open question: chat-completions with server-side tool parsing. | M0 (b), (c). |
| P6 | Code constraints: `Decision` lives in `brokerd`, has a private field and does not implement `Deserialize`. `proto` carries a plain `DecisionRecord` for the audit log. | Rust privacy is per crate, and a deserializable type can be built by anyone. | | P6 | Code constraints: `Decision` lives in `brokerd`, has a private field and does not implement `Deserialize`. `proto` carries a plain `DecisionRecord` for the audit log. | Rust privacy is per crate, and a deserializable type can be built by anyone. |
| P7 | Inference contract 1: the baseline budget test needs the server's tokenizer, so `make gate` has an offline part and an on-device part (`make verify-device`). | `/tokenize` is a server endpoint. |
## Proposed changes to the design brief ## Proposed changes to the design brief
@@ -38,7 +39,6 @@ From M0 and the kickoff review. None is applied yet. Each lands as its own commi
| # | Change | Evidence | | # | Change | Evidence |
|---|---|---| |---|---|---|
| P7 | Inference contract 1: the baseline budget test needs the server's tokenizer, so `make gate` has an offline part and an on-device part (`make verify-device`). | `/tokenize` is a server endpoint. |
| P8 | Target environment: describe the shared router as deployed (router mode, `--models-max 2`, other clients on the same Ornith instance, q8_0 KV, six-hour idle unload). Drop "memory is abundant". | `docs/inference-contract.md`, "What is running". | | P8 | Target environment: describe the shared router as deployed (router mode, `--models-max 2`, other clients on the same Ornith instance, q8_0 KV, six-hour idle unload). Drop "memory is abundant". | `docs/inference-contract.md`, "What is running". |
| P9 | Inference contract 5: slots are pinned with `id_slot` but not reserved. Another client can evict a harness slot, and the router can unload Ornith. Cache loss is an expected event: `loopd` detects it (`cache_n` far below the previous request's total), logs it in the session log, and carries on. It is never an error. A session always uses the same slot; threads may share the main slot. | Shared router decision. M0 (d): evicted sessions came back from the host-RAM prompt cache in under a second on the same slot, and cost a full re-read on another slot. | | P9 | Inference contract 5: slots are pinned with `id_slot` but not reserved. Another client can evict a harness slot, and the router can unload Ornith. Cache loss is an expected event: `loopd` detects it (`cache_n` far below the previous request's total), logs it in the session log, and carries on. It is never an error. A session always uses the same slot; threads may share the main slot. | Shared router decision. M0 (d): evicted sessions came back from the host-RAM prompt cache in under a second on the same slot, and cost a full re-read on another slot. |
| P10 | Inference contract 7: a pinned request can queue behind another client's request on the same slot, and the router can spend tens of seconds reloading the model. The liveness timer starts at the first byte received, and a separate, longer "waiting for slot or model" limit covers the time before it. | Shared router decision. Needs a measurement of what the stream sends while queued. | | P10 | Inference contract 7: a pinned request can queue behind another client's request on the same slot, and the router can spend tens of seconds reloading the model. The liveness timer starts at the first byte received, and a separate, longer "waiting for slot or model" limit covers the time before it. | Shared router decision. Needs a measurement of what the stream sends while queued. |
docs/design.md
+3 -1
View File
@@ -81,7 +81,8 @@ ordinary outbound networking can reach the whole tailnet and the internet as the
## Inference contract (the part existing harnesses get wrong) ## Inference contract (the part existing harnesses get wrong)
1. **Baseline budget.** System prompt plus tool schemas at session start: 3,000 tokens or less, 1. **Baseline budget.** System prompt plus tool schemas at session start: 3,000 tokens or less,
measured with the server's tokenizer, enforced by a test. measured with the server's tokenizer (`/tokenize`), enforced by a test. That test needs
straylight, so it runs in `make verify-device`, not in the offline `make gate`.
2. **Append-only.** The request for turn N+1 is a strict extension of the request for turn N. 2. **Append-only.** The request for turn N+1 is a strict extension of the request for turn N.
Nothing volatile (time, heartbeat notes, memory refreshes) goes anywhere but the newest message. Nothing volatile (time, heartbeat notes, memory refreshes) goes anywhere but the newest message.
Enforced by a property test on the serialized message array. The session log stores each Enforced by a property test on the serialized message array. The session log stores each
@@ -201,4 +202,5 @@ Files are the source of truth. SQLite is allowed only for rebuildable indexes an
field and does not implement `Deserialize`, so no other crate or wire message can produce one. field and does not implement `Deserialize`, so no other crate or wire message can produce one.
`proto` carries a plain `DecisionRecord` for the audit log and the wire. `proto` carries a plain `DecisionRecord` for the audit log and the wire.
- Dependencies are few and justified in `docs/dependencies.md`. `cargo-deny` runs in the gate. - Dependencies are few and justified in `docs/dependencies.md`. `cargo-deny` runs in the gate.
- `make gate` runs offline. Checks that need straylight run in `make verify-device`.
- No telemetry, no update checks, no outbound call not listed in `docs/egress.md`. - No telemetry, no update checks, no outbound call not listed in `docs/egress.md`.