Brief P9: slots are pinned but not reserved; cache loss is a normal event

Approved 2026-09-17. Evidence is in docs/decisions.md.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
2026-09-17 00:38:32 -07:00
co-authored by Claude Fable 5.1
parent e07fddf0ad
commit 27b7ff45ee
2 changed files with 9 additions and 3 deletions
+1 -1
View File
@@ -33,6 +33,7 @@ Approved by the owner on 2026-09-17. One commit each.
| P6 | Code constraints: `Decision` lives in `brokerd`, has a private field and does not implement `Deserialize`. `proto` carries a plain `DecisionRecord` for the audit log. | Rust privacy is per crate, and a deserializable type can be built by anyone. |
| P7 | Inference contract 1: the baseline budget test needs the server's tokenizer, so `make gate` has an offline part and an on-device part (`make verify-device`). | `/tokenize` is a server endpoint. |
| P8 | Target environment: describe the shared router as deployed (router mode, `--models-max 2`, other clients on the same Ornith instance, q8_0 KV, six-hour idle unload). Drop "memory is abundant". | `docs/inference-contract.md`, "What is running". |
| P9 | Inference contract 5: slots are pinned with `id_slot` but not reserved. Another client can evict a harness slot, and the router can unload Ornith. Cache loss is an expected event: `loopd` detects it (`cache_n` far below the previous request's total), logs it in the session log, and carries on. It is never an error. A session always uses the same slot; threads may share the main slot. | Shared router decision. M0 (d): evicted sessions came back from the host-RAM prompt cache in under a second on the same slot, and cost a full re-read on another slot. |
## Proposed changes to the design brief
@@ -40,7 +41,6 @@ From M0 and the kickoff review. None is applied yet. Each lands as its own commi
| # | Change | Evidence |
|---|---|---|
| P9 | Inference contract 5: slots are pinned with `id_slot` but not reserved. Another client can evict a harness slot, and the router can unload Ornith. Cache loss is an expected event: `loopd` detects it (`cache_n` far below the previous request's total), logs it in the session log, and carries on. It is never an error. A session always uses the same slot; threads may share the main slot. | Shared router decision. M0 (d): evicted sessions came back from the host-RAM prompt cache in under a second on the same slot, and cost a full re-read on another slot. |
| P10 | Inference contract 7: a pinned request can queue behind another client's request on the same slot, and the router can spend tens of seconds reloading the model. The liveness timer starts at the first byte received, and a separate, longer "waiting for slot or model" limit covers the time before it. | Shared router decision. Needs a measurement of what the stream sends while queued. |
| P11 | Inference contract 10: the serving flags live in `~/src/nixos`, not here. This repo records the expected values (template hash, per-slot `n_ctx`, slot count, KV type, sampling) and the startup self-test compares them with `/props` and `/slots`. KV type is recorded as q8_0. | Shared router decision. |
| P12 | Network isolation 2: `inferproxy` is kept. Remove the conditional about dropping it. Its upstream is the router's TCP listener on the host. | Shared router decision; M0 (g). |
+8 -2
View File
@@ -96,8 +96,14 @@ ordinary outbound networking can reach the whole tailnet and the internet as the
3. **No in-place pruning.** Tool results are size-capped when first appended, never trimmed later.
4. **Compaction is an epoch change.** It happens only when the session is idle, writes a summary,
and starts a new epoch whose prefix is the baseline plus the summary. The old log is kept.
5. **Pinned slots.** Main session, subagents, and scheduled runs each use their own server slot so
they do not evict each other's cache. Verify the pinning mechanism on the deployed build.
5. **Pinned slots, not reserved.** Every request carries `id_slot`, and a session always uses the
same slot: moving it costs a full re-read. Slot assignment is configuration. With the two slots
deployed today, main sessions (one per Mattermost thread) share one slot, and subagents and
scheduled runs share the other. The server is shared, so another client can take a harness
slot and the router can unload Ornith. An evicted session is normally restored from the
server's host-RAM prompt cache when it returns on its slot; an unloaded model is not. Cache
loss is therefore an expected event: `loopd` detects it (`cache_n` far below the previous
request's total), records it in the session log and carries on. It is never an error.
6. **Progressive disclosure.** The `tools` array is fixed for the whole epoch. The template renders
it at the top of the prompt, so any change re-reads everything. Only a small core tool set has
schemas in it. Everything else is found through `find_tool`, which returns schemas as an