Add slot pinning and eviction measurements to M0 findings

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

docs/decisions.md

@@ -33,7 +33,7 @@ From M0 and the kickoff review. None is applied yet. Each lands as its own commi
 | P6 | Code constraints: `Decision` lives in `brokerd`, has a private field and does not implement `Deserialize`. `proto` carries a plain `DecisionRecord` for the audit log. | Rust privacy is per crate, and a deserializable type can be built by anyone. |
 | P7 | Inference contract 1: the baseline budget test needs the server's tokenizer, so `make gate` has an offline part and an on-device part (`make verify-device`). | `/tokenize` is a server endpoint. |
 | P8 | Target environment: describe the shared router as deployed (router mode, `--models-max 2`, other clients on the same Ornith instance, q8_0 KV, six-hour idle unload). Drop "memory is abundant". | `docs/inference-contract.md`, "What is running". |
-| P9 | Inference contract 5: slots are pinned with `id_slot` but not reserved. Another client can evict a harness slot, and the router can unload Ornith. Cache loss is an expected event: `loopd` detects it (`cache_n` far below the previous request's total), logs it in the session log, and carries on. It is never an error. | Shared router decision. |
+| P9 | Inference contract 5: slots are pinned with `id_slot` but not reserved. Another client can evict a harness slot, and the router can unload Ornith. Cache loss is an expected event: `loopd` detects it (`cache_n` far below the previous request's total), logs it in the session log, and carries on. It is never an error. A session always uses the same slot; threads may share the main slot. | Shared router decision. M0 (d): evicted sessions came back from the host-RAM prompt cache in under a second on the same slot, and cost a full re-read on another slot. |
 | P10 | Inference contract 7: a pinned request can queue behind another client's request on the same slot, and the router can spend tens of seconds reloading the model. The liveness timer starts at the first byte received, and a separate, longer "waiting for slot or model" limit covers the time before it. | Shared router decision. Needs a measurement of what the stream sends while queued. |
 | P11 | Inference contract 10: the serving flags live in `~/src/nixos`, not here. This repo records the expected values (template hash, per-slot `n_ctx`, slot count, KV type, sampling) and the startup self-test compares them with `/props` and `/slots`. KV type is recorded as q8_0. | Shared router decision. |
 | P12 | Network isolation 2: `inferproxy` is kept. Remove the conditional about dropping it. Its upstream is the router's TCP listener on the host. | Shared router decision; M0 (g). |

docs/inference-contract.md

@@ -8,7 +8,7 @@ the server README at tag `b10809`, the build that is running.
 **Conditions.** Every number below was taken while another session was generating on Ornith slot 1
 (an 85k to 105k-token conversation at about 50 tokens/s). Token counts (`cache_n`, `prompt_n`) are
 not affected by that. Throughput is, so section (a) reports a separate run taken after the GPU went
-idle. One check is still open: slot pinning (d).
+idle. Section (d) was also run with the GPU idle.
 
 ## What is running
 
@@ -121,11 +121,34 @@ suffix appended to a cached 32k-token prefix, then generation from there.
 The idle numbers match the brief's reference figures (about 1,100 and 700 tokens/s, 60 falling to
 49). A second active session on the same model roughly halves generation speed.
 
-### (d) Slot pinning: not yet measured
+### (d) Slot pinning and eviction
 
-`id_slot` is a documented request field and the pinned requests above all landed on slot 0. The
-two-session test (`m0.py pin`) needs slot 1 and sends unpinned requests, which would evict the
-100k-token cache of the session that is using slot 1. It runs when the owner says slot 1 is free.
+Run with the GPU otherwise idle and both slots free.
+
+**Pinning works.** Session A pinned to slot 0 and session B pinned to slot 1 (6k tokens each) did not
+disturb each other: A's second turn had `cache_n` 6028, `prompt_n` 17.
+
+**Eviction is recovered from the server's host-RAM prompt cache** (`--cache-ram`, default 8 GiB,
+with `--cache-idle-slots` on by default), as long as the session comes back on the same slot.
+
+| Scenario | Result for the returning session |
+|---|---|
+| Two unpinned 3k-token requests from "other clients" between turns of A and B | A: `cache_n` 6088, `prompt_n` 18. B: `cache_n` 6107, `prompt_n` 17. |
+| A (30k tokens) on slot 0, then three unrelated 8k prompts pinned to slot 0, then A again on slot 0 | `cache_n` 30695, `prompt_n` 16, 0.43 s |
+| The same session A then sent to slot 1 instead | `cache_n` 0, `prompt_n` 30728, 34.5 s |
+| Two sessions sharing a 3k baseline (same system text and tools), 10k tokens each, taking turns on slot 0 for three rounds | Every switch after the first: `cache_n` about 13k, `prompt_n` about 320, 1 s. The second session's first request also reused the 3,031-token shared baseline. |
+
+Consequences:
+
+- A session must always use the same slot. Moving it to another slot costs a full re-read.
+- Several Mattermost threads can share the main slot. Switching between them costs about a second,
+  not a re-read, while their saved states fit in the prompt cache.
+- Another client taking a harness slot is usually harmless for the same reason.
+- This does not survive a model unload (`--models-max 2`, or six idle hours), and the 8 GiB limit
+  was not probed. The size of one saved session was not measured. `loopd` still has to treat a cold
+  cache as a normal event.
+- One exception was seen in (e): after a prompt that shared a long prefix with the saved one but
+  differed in the middle, the saved prompt was restored on one attempt and not on another.
 
 ## Recommendation on the open question
 
@@ -152,8 +175,10 @@ plus KV cache) does not fit beside Laguna's 69 GB under the 104 GiB GPU memory c
 What the shared instance means for the harness:
 
 - `id_slot` pins the harness's own requests. It does not reserve the slot. An unpinned request
-  from another client can take it, loading a third model can unload Ornith (`--models-max 2`), and
-  six idle hours drop weights and cache. `loopd` must treat a cold cache as a normal event.
+  from another client can take it, but the evicted session is normally restored from the host-RAM
+  prompt cache (see (d)). Loading a third model can unload Ornith (`--models-max 2`), and six idle
+  hours drop weights and cache; neither is recoverable. `loopd` must treat a cold cache as a
+  normal event.
 - A pinned request can wait behind another client's generation on the same slot.
 - Another active session on Ornith roughly halves generation speed.
 - Serving flags are owned by `~/src/nixos`. This repo records expected values and checks them at

spike/m0.py

@@ -1,7 +1,7 @@
 #!/usr/bin/env python3
 """THROWAWAY M0 measurement spike for Boxmaker. Not harness code. Stdlib only.
 
-Usage: python3 m0.py <check> [...]   (checks: slots throughput convo tools pin progress rewind findtool rewind2)
+Usage: python3 m0.py <check> [...]   (checks: slots throughput convo tools pin progress rewind findtool rewind2 evict alternate)
 Raw results are appended to out/<check>.jsonl; findings go in docs/inference-contract.md.
 """
 import json, os, random, sys, time, urllib.request
@@ -307,6 +307,34 @@ def check_rewind2():
     log("rewind2", {"requests": recs})
 
 
+def check_evict():
+    """(d2) certain eviction: a different prompt is pinned onto the session's own slot. Is the session restored?"""
+    recs, kw = [], {"chat_template_kwargs": {"enable_thinking": False}}
+    a = [{"role": "system", "content": SYSTEM}, {"role": "user", "content": "Session A. " + filler(30000, "EA") + "\nReply: alpha"}]
+    r = chat(a, None, 0, 16, **kw); recs.append(tline("A turn1 slot0 (30k)", r)); a.append(assistant_msg(r))
+    for i in (1, 2, 3):
+        c = [{"role": "user", "content": "Intruder %d. %s\nReply: gamma" % (i, filler(8000, "EI%d" % i))}]
+        r = chat(c, None, 0, 16, **kw); recs.append(tline("intruder %d pinned to slot0 (8k)" % i, r))
+    a.append({"role": "user", "content": "Again."})
+    r = chat(a, None, 0, 16, **kw); recs.append(tline("A turn2 slot0 (after 3 intruders)", r)); a.append(assistant_msg(r))
+    a.append({"role": "user", "content": "Again."})
+    r = chat(a, None, 1, 16, **kw); recs.append(tline("A turn3 moved to slot1", r))
+    log("evict", {"requests": recs})
+
+
+def check_alternate():
+    """(d3) two harness sessions sharing one baseline prefix take turns on one slot, as Mattermost threads would."""
+    recs, kw = [], {"chat_template_kwargs": {"enable_thinking": False}}
+    base = SYSTEM + " Shared baseline follows. " + filler(2500, "BASE")
+    sess = {n: [{"role": "system", "content": base}, {"role": "user", "content": "Thread %s. %s\nReply: ok" % (n, filler(10000, "AL" + n))}] for n in "AB"}
+    for rnd in (1, 2, 3):
+        for n in "AB":
+            if rnd > 1:
+                sess[n].append({"role": "user", "content": "Round %d. %s\nReply: ok" % (rnd, filler(300, "ALr%d%s" % (rnd, n)))})
+            r = chat(sess[n], TOOLS, 0, 16, **kw); recs.append(tline("thread %s round %d slot0" % (n, rnd), r)); sess[n].append(assistant_msg(r))
+    log("alternate", {"requests": recs})
+
+
 if __name__ == "__main__":
     os.makedirs(OUT, exist_ok=True)
     for name in sys.argv[1:] or ["slots"]: