Brief P11: serving settings are recorded here and owned by the NixOS config

Approved 2026-09-17. Evidence is in docs/decisions.md.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
2026-09-17 00:38:32 -07:00
co-authored by Claude Fable 5.1
parent adab43add1
commit a4ebdc11af
2 changed files with 7 additions and 3 deletions
+1 -1
View File
@@ -35,6 +35,7 @@ Approved by the owner on 2026-09-17. One commit each.
| P8 | Target environment: describe the shared router as deployed (router mode, `--models-max 2`, other clients on the same Ornith instance, q8_0 KV, six-hour idle unload). Drop "memory is abundant". | `docs/inference-contract.md`, "What is running". |
| P9 | Inference contract 5: slots are pinned with `id_slot` but not reserved. Another client can evict a harness slot, and the router can unload Ornith. Cache loss is an expected event: `loopd` detects it (`cache_n` far below the previous request's total), logs it in the session log, and carries on. It is never an error. A session always uses the same slot; threads may share the main slot. | Shared router decision. M0 (d): evicted sessions came back from the host-RAM prompt cache in under a second on the same slot, and cost a full re-read on another slot. |
| P10 | Inference contract 7: a pinned request can queue behind another client's request on the same slot, and the router can spend tens of seconds reloading the model. The liveness timer starts at the first byte received, and a separate, longer "waiting for slot or model" limit covers the time before it. | Shared router decision. Needs a measurement of what the stream sends while queued. |
| P11 | Inference contract 10: the serving flags live in `~/src/nixos`, not here. This repo records the expected values (template hash, per-slot `n_ctx`, slot count, KV type, sampling) and the startup self-test compares them with `/props` and `/slots`. KV type is recorded as q8_0. | Shared router decision. |
## Proposed changes to the design brief
@@ -42,5 +43,4 @@ From M0 and the kickoff review. None is applied yet. Each lands as its own commi
| # | Change | Evidence |
|---|---|---|
| P11 | Inference contract 10: the serving flags live in `~/src/nixos`, not here. This repo records the expected values (template hash, per-slot `n_ctx`, slot count, KV type, sampling) and the startup self-test compares them with `/props` and `/slots`. KV type is recorded as q8_0. | Shared router decision. |
| P12 | Network isolation 2: `inferproxy` is kept. Remove the conditional about dropping it. Its upstream is the router's TCP listener on the host. | Shared router decision; M0 (g). |
+6 -2
View File
@@ -125,8 +125,12 @@ ordinary outbound networking can reach the whole tailnet and the internet as the
9. **Startup self-test.** On boot `loopd` checks: tool-call round trip parses, turn-2 prompt
processing count shows a cache hit, configured context matches what the server reports.
It refuses to start if any check fails.
10. **Serving settings are part of the repo.** Launch flags, chat template hash, sampling settings
(temp 0.6, top-p 0.95, top-k 20), f16 KV cache, no speculative decoding.
10. **Serving settings are recorded, not owned.** The launch flags live in the owner's NixOS
configuration (`~/src/nixos` on straylight). This repo records the expected values: chat
template hash, per-slot context, slot count, q8_0 KV cache, no speculative decoding. The
startup self-test compares them with `/props` and `/slots`. Sampling settings (temp 0.6,
top-p 0.95, top-k 20) are sent with every request, because the server's default temperature
is 1.0.
Settled by M0 (`docs/inference-contract.md`): `loopd` uses the server's chat-completions endpoint
with server-side tool parsing. It does not render the template in-process. Cache reuse held across