diff --git a/.gitignore b/.gitignore index b83d222..42e21a3 100644 --- a/.gitignore +++ b/.gitignore @@ -1 +1,2 @@ /target/ +/spike/out/ diff --git a/docs/decisions.md b/docs/decisions.md index ba67bca..28ac241 100644 --- a/docs/decisions.md +++ b/docs/decisions.md @@ -15,5 +15,21 @@ Newest first. A decision that changes `docs/design.md` lands in the same commit | Decision | Needed by | |---|---| +| Serving setup: a dedicated `llama-server` unit for Boxmaker on a Unix socket (recommended in `docs/inference-contract.md`), or the shared router plus `inferproxy`. Decides whether the `inferproxy` crate exists and whether three pinned slots are available. | M1 | | Secret store backend, and where the v0 file's key lives. straylight has no secrets manager today. | M3 | | Whether cloud-led sessions are ever allowed, and for which data classes. | M6 | + +## Proposed changes to the design brief + +From M0 and the kickoff review. None is applied yet. Each lands as its own commit once the owner agrees. + +| # | Change | Evidence | +|---|---|---| +| P1 | Inference contract 6: the `tools` array is fixed per epoch. `find_tool` returns schemas as a tool result and the model calls them through a `call_tool(name, arguments)` meta-tool. Never instruct the model to call an undeclared tool. | M0 (h): adding a tool re-read the whole prompt; undeclared calls were coerced into `write_file`. | +| P2 | Inference contract 2: the session log stores assistant messages exactly as returned, including `reasoning_content`, and replays them unchanged. Remove the "known risk" about dropped thinking blocks. | M0 (e): Ornith's template keeps every think block. | +| P3 | Inference contract 7: requests set `return_progress: true`; progress events count as liveness. | M0 (i): otherwise the stream is silent during prefill. | +| P4 | Inference contract 8: the thinking cap uses `reasoning_control` and the control endpoint. | README b10809. Not yet exercised. | +| P5 | Settle the open question: chat-completions with server-side tool parsing. | M0 (b), (c). | +| P6 | Code constraints: `Decision` lives in `brokerd`, has a private field and does not implement `Deserialize`. `proto` carries a plain `DecisionRecord` for the audit log. | Rust privacy is per crate, and a deserializable type can be built by anyone. | +| P7 | Inference contract 1: the baseline budget test needs the server's tokenizer, so `make gate` has an offline part and an on-device part (`make verify-device`). | `/tokenize` is a server endpoint. | +| P8 | Target environment: replace "memory is abundant" and the f16 and slot assumptions with the measured setup, once the serving decision is made. | `docs/inference-contract.md`, "What is running". | diff --git a/docs/inference-contract.md b/docs/inference-contract.md new file mode 100644 index 0000000..f412b19 --- /dev/null +++ b/docs/inference-contract.md @@ -0,0 +1,160 @@ +# Inference contract: M0 measurements + +Measured 2026-09-17 against straylight by `spike/m0.py` (throwaway, Python stdlib). Requests went to +`https://straylight.scylla-hammerhead.ts.net:10000`, model id `ornith-1.5-35b-a3b`, through +`/v1/chat/completions` with `temperature 0.6, top_p 0.95, top_k 20`. Request fields were taken from +the server README at tag `b10809`, the build that is running. + +**Conditions.** Every number below was taken while another session was generating on Ornith slot 1 +(an 85k to 105k-token conversation at about 50 tokens/s). Token counts (`cache_n`, `prompt_n`) are +not affected by that. Throughput is, so treat the rates as lower bounds. Two checks are still +open for the same reason: clean throughput (a) and slot pinning (d). + +## What is running + +| Item | Value | +|---|---| +| Build | llama.cpp `b10809-5266f24` (nixpkgs-unstable `llama-cpp-0.4.0`, Vulkan backend) | +| Mode | Router: one public endpoint, one child `llama-server` per model, `--models-max 2` | +| Public listener | `0.0.0.0:11434`, firewalled to the tailnet; Tailscale Serve adds HTTPS on `:10000` | +| Other clients | Open WebUI and OpenCode use the same endpoint and the same Ornith instance | +| Ornith flags | `--jinja --no-mmap --ctx-size 262144 --parallel 2 --cache-type-k q8_0 --cache-type-v q8_0 --flash-attn on --n-gpu-layers 999 --sleep-idle-seconds 21600 --hf-repo ornith-ai/Ornith-1.5-35B-A3B-GGUF:Q4_K_M` | +| Slots | 2, each `n_ctx` 131072 (the 262144 is split, not shared) | +| Server default sampling | temperature 1.0, top_k 20, top_p 0.95, min_p 0.05. The harness must send its own. | +| Chat template | 7,828 bytes, sha256 `f55f52930aa8bf44ab5cb85f99370fcc3c56e9a85640b812086d5330bce5d86b` | +| Source of truth for flags | `~/src/nixos/hw/straylight/default.nix` on straylight, not this repo | + +Differences from the design brief: KV cache is q8_0, not f16. There are two slots, not three. The +server is shared, so slots are not reserved for the harness. Weights and KV cache are dropped after +six idle hours. Host memory was 110 of 125 GB in use with Laguna S 2.1 and Ornith both loaded. + +## Findings + +### (b) Cache reuse over a 3-turn conversation: passes + +Four tool schemas, thinking on, `reasoning_content` and `tool_calls` echoed back exactly as received. + +| Request | `cache_n` | `prompt_n` | +|---|---|---| +| turn 1, request 1 | 0 | 539 | +| turn 1, request 2 (after tool result) | 591 | 29 | +| turn 2, request 1 | 680 | 27 | +| turn 2, request 2 | 759 | 40 | +| turn 3, request 1 | 840 | 26 | + +Each request processes only its new tokens. Raw timing fields for one request: +`{"cache_n": 591, "prompt_n": 29, "prompt_ms": 227.163, "prompt_per_second": 127.66, "predicted_n": 61, "predicted_ms": 1762.327, "predicted_per_second": 34.05}`. + +The whole baseline here (system line, four tool schemas, first user message) was 539 tokens, so the +3,000-token baseline budget is realistic. + +### (c) Tool-call parsing through chat-completions: 0 failures in 20 + +Five prompts for each of four tools. Every response had `finish_reason: "tool_calls"`, the expected +tool, valid JSON arguments and all required arguments. Types survived: an integer `timeout_s`, a +nested `headers` object, and strings containing quotes, `&`, `<>` and embedded JSON. + +### (e) Thinking blocks and the cache + +- Ornith's template renders the `` block of **every** assistant turn, not only the last one. + It does not strip earlier reasoning. The brief's "known risk" does not apply as long as the + harness sends `reasoning_content` back unchanged. +- If the harness drops `reasoning_content`, the prompt diverges at the latest assistant turn. The + cost was small (`prompt_n` 65 to 80 instead of 27 to 40), because the server keeps a checkpoint + near the end of the previous request. +- A change anywhere earlier costs a full re-read. Editing turn 3 or turn 2 of a 5-turn, 7.6k-token + conversation gave `cache_n` 34. Changing text 10k tokens into a 24k-token prompt gave `cache_n` 0 + and 26 s of prompt processing. This confirms the brief: no partial rewind in practice. +- The server sometimes restored an older prompt from its host-RAM prompt cache (`--cache-ram`, + default 8 GiB): resending the original 24k prompt after the edited one gave `cache_n` 23758. It + did not do so every time. Do not design around it. + +### (h) Changing the tool list mid-session: full invalidation + +The template renders tool schemas at the very top of the prompt, before the system text. Adding a +fifth tool at turn 3 gave `cache_n` 23, `prompt_n` 920. The `tools` array must be fixed for the +whole epoch. + +Progressive disclosure still works if the schema arrives as a tool result: + +| Variant | Result | +|---|---| +| `find_tool` returns a schema, model calls it through a fixed `call_tool(name, arguments)` meta-tool | 4 of 5 correct. The one miss called `call_tool` without `find_tool` first, which `brokerd` can reject. | +| `find_tool` returns a schema, model calls the new tool directly by name | 0 of 5. The server's grammar only allows declared names, so the model was forced into a **wrong declared tool**: three times it emitted `write_file` with placeholder content. | + +The second row is a safety finding, not only a cache one. Never tell the model to call a tool that +is not in the `tools` array. + +### (i) Liveness during prompt processing + +With `stream: true` and `return_progress: true`, a 16k-token prefill produced 11 `prompt_progress` +events and the longest silence was 2.2 s. Without `return_progress` the stream is silent for the +whole prefill. The liveness timeout in the brief needs this field. + +### (g) Unix socket, co-location + +`--host` accepts a path ending in `.sock` (README, build b10809). The harness and `llama-server` +are on the same host. In router mode the router sets each child's host and port itself, so only the +router's public listener could move to a socket, and Open WebUI, OpenCode and Tailscale Serve need +it on TCP. `loopd` runs with `--network=none` and cannot reach host loopback. So with the shared +router `inferproxy` stays. It goes away only if the harness gets its own `llama-server` on a socket. + +### Runaway control + +`--reasoning-budget` is a server flag, not a request field. Per request there is `max_tokens`, and +`reasoning_control: true` plus `POST /v1/chat/completions/control` with `action: "reasoning_end"`, +which ends the thinking block of a running completion. The second one fits a per-turn thinking cap +enforced by `loopd` while it counts streamed reasoning tokens. Not yet exercised. + +### (a) Throughput: contended numbers only + +| Measurement | Value (other slot busy) | +|---|---| +| Prompt processing, average over 0 to 24k | 929 to 936 tokens/s | +| Prompt processing, 1.5k chunks at depth 0 to 7.6k | 875 to 1,040 tokens/s | +| Generation | 28 to 34 tokens/s | + +Method: timed requests, server-reported `timings`. The clean run (depth 0 and a 2k suffix at depth +32k) is scripted as `m0.py throughput` and waits for the GPU to be idle. + +### (d) Slot pinning: not yet measured + +`id_slot` is a documented request field and the pinned requests above all landed on slot 0. The +two-session test (`m0.py pin`) needs slot 1 and sends unpinned requests, which would evict the +100k-token cache of the session that is using slot 1. It runs when the owner says slot 1 is free. + +## Recommendation on the open question + +Use the server's chat-completions endpoint with server-side tool parsing. Do not render the +template in-process. The cache measurements pass and tool parsing had no failures. The conditions +are: + +1. The session log stores each assistant message exactly as returned (`content`, + `reasoning_content`, `tool_calls`) and replays it unchanged. +2. The `tools` array is fixed per epoch. Tools outside the core set are reached through `find_tool` + and a `call_tool` meta-tool. +3. Every request carries `id_slot`, `cache_prompt: true`, the sampling settings, `stream: true` and + `return_progress: true`. +4. The startup self-test compares the template hash and `n_ctx` from `/props?model=...` with the + values recorded here. + +## Recommended serving setup + +The contract cannot be met on a shared instance: any unpinned request from Open WebUI or OpenCode +can take a harness slot, loading a third model can unload Ornith, and the idle timer drops the +cache. Recommended: a dedicated `llama-server` systemd unit for Boxmaker, defined as a NixOS module +kept in this repo under `deploy/` and imported by `~/src/nixos`. + +| Flag | Value | Reason | +|---|---|---| +| `--host` | `/run/boxmaker/llama.sock` | No TCP listener, no strangers, and `inferproxy` is not needed | +| `--parallel` | 3 | Main, subagent and scheduled slots | +| `--ctx-size` | 393216 | 131072 per slot | +| `--cache-type-k/v` | f16 | As the brief specifies. Not compared with q8_0 in M0. | +| `--jinja --flash-attn on --no-mmap --n-gpu-layers 999` | as now | | +| `--sleep-idle-seconds` | unset | Keep the cache across idle periods | +| no draft model | | The brief rules out speculative decoding | + +Cost: a second resident copy of Ornith, about 22 GB of weights plus KV cache, beside Laguna's 69 GB +under the 104 GiB GPU memory cap. It will not fit if the shared router also keeps its own Ornith +loaded. This is the owner's call and is listed in `docs/decisions.md`. diff --git a/spike/m0.py b/spike/m0.py new file mode 100644 index 0000000..f0eeb4c --- /dev/null +++ b/spike/m0.py @@ -0,0 +1,314 @@ +#!/usr/bin/env python3 +"""THROWAWAY M0 measurement spike for Boxmaker. Not harness code. Stdlib only. + +Usage: python3 m0.py [...] (checks: slots throughput convo tools pin progress rewind findtool rewind2) +Raw results are appended to out/.jsonl; findings go in docs/inference-contract.md. +""" +import json, os, random, sys, time, urllib.request + +BASE = os.environ.get("LLAMA_URL", "https://straylight.scylla-hammerhead.ts.net:10000") +MODEL = os.environ.get("LLAMA_MODEL", "ornith-1.5-35b-a3b") +OUT = os.path.join(os.path.dirname(os.path.abspath(__file__)), "out") +SAMPLING = {"temperature": 0.6, "top_p": 0.95, "top_k": 20} +NONCE = "%08x" % random.getrandbits(32) + + +def http(path, body=None, timeout=1800): + data = None if body is None else json.dumps(body).encode() + req = urllib.request.Request(BASE + path, data=data, headers={"Content-Type": "application/json"}) + with urllib.request.urlopen(req, timeout=timeout) as r: + return json.load(r) + + +def log(check, rec): + rec = {"check": check, "nonce": NONCE, "t": time.strftime("%FT%T"), **rec} + with open(os.path.join(OUT, check + ".jsonl"), "a") as f: + f.write(json.dumps(rec) + "\n") + + +def chat(messages, tools=None, slot=-1, max_tokens=2048, **extra): + body = {"model": MODEL, "messages": messages, "id_slot": slot, "cache_prompt": True, + "max_tokens": max_tokens, "stream": False, **SAMPLING, **extra} + if tools: + body["tools"] = tools + t0 = time.time() + r = http("/v1/chat/completions", body) + r["_wall_s"] = round(time.time() - t0, 2) + return r + + +def tline(label, r): + t = r["timings"] + print("%-34s cache_n=%-6d prompt_n=%-6d pp=%7.1f t/s predicted_n=%-5d tg=%5.1f t/s wall=%ss" % ( + label, t["cache_n"], t["prompt_n"], t.get("prompt_per_second", 0), t["predicted_n"], + t.get("predicted_per_second", 0), r["_wall_s"])) + return {"label": label, "timings": t, "wall_s": r["_wall_s"], + "finish": r["choices"][0]["finish_reason"]} + + +def ntokens(text): + return len(http("/tokenize", {"model": MODEL, "content": text})["tokens"]) + + +def filler(target_tokens, seed): + """Pseudo-random English-ish text of about target_tokens tokens, unique per run and seed.""" + rng = random.Random(NONCE + str(seed)) + words = ("box fragment shell cork glass bone thread map brass feather shard coil lattice drift " + "orbit cable vault ledger salt ash lens hinge spool wire amber slate quill tin reed").split() + chunk = " ".join(rng.choice(words) + ("." if rng.random() < .1 else "") for _ in range(2000)) + per = ntokens(chunk) / 2000 + n = int(target_tokens / per) + return " ".join(rng.choice(words) + ("." if rng.random() < .1 else "") for _ in range(n)) + + +def tool(name, desc, props, required): + return {"type": "function", "function": {"name": name, "description": desc, "parameters": { + "type": "object", "properties": props, "required": required}}} + + +TOOLS = [ + tool("read_file", "Read a text file and return its contents.", + {"path": {"type": "string", "description": "Absolute path"}}, ["path"]), + tool("write_file", "Write text to a file, replacing it.", + {"path": {"type": "string"}, "content": {"type": "string"}}, ["path", "content"]), + tool("shell", "Run a shell command in a sandbox and return stdout and stderr.", + {"command": {"type": "string"}, "timeout_s": {"type": "integer"}}, ["command"]), + tool("http_fetch", "Fetch a URL with GET and return the body.", + {"url": {"type": "string"}, "headers": {"type": "object"}}, ["url"]), +] +EXTRA_TOOL = tool("recall", "Search long-term memory notes.", + {"query": {"type": "string"}, "k": {"type": "integer"}}, ["query"]) +SYSTEM = "You are Boxmaker, a careful personal agent. Use tools when they are needed. Run id %s." % NONCE +FAKE_RESULTS = {"read_file": "hostname = straylight\nport = 11434\n", "write_file": "ok, 24 bytes written", + "shell": "total 4\n-rw-r--r-- 1 kyle users 31 Sep 17 notes.txt\n", "http_fetch": "ok", + "recall": "notes/2026-09-01.md:3 owner prefers metric units"} + + +def assistant_msg(r, keep_reasoning=True): + m = r["choices"][0]["message"] + out = {"role": "assistant", "content": m.get("content") or ""} + if keep_reasoning and m.get("reasoning_content"): + out["reasoning_content"] = m["reasoning_content"] + if m.get("tool_calls"): + out["tool_calls"] = m["tool_calls"] + return out + + +def run_turn(label, messages, tools, slot, keep_reasoning, recs, max_iters=4): + """One user turn: loop model -> tool results until the model answers in text.""" + for i in range(max_iters): + r = chat(messages, tools, slot) + recs.append(tline("%s req%d" % (label, i + 1), r)) + messages.append(assistant_msg(r, keep_reasoning)) + calls = r["choices"][0]["message"].get("tool_calls") or [] + if not calls: + return + for c in calls: + messages.append({"role": "tool", "tool_call_id": c["id"], + "content": FAKE_RESULTS.get(c["function"]["name"], "ok")}) + + +USER_TURNS = ["Read /etc/boxmaker/config.toml and tell me which port is configured.", + "Now list the files in /home/kyle/notes using the shell.", + "Thanks. In one sentence, what did you learn from both steps?"] + + +def check_convo(): + """(b) cache reuse over 3 turns, (e) effect of dropping reasoning, (h) adding a tool mid-session.""" + for variant in ("keep_reasoning", "drop_reasoning", "add_tool_turn3"): + print("\n== variant:", variant) + msgs, recs = [{"role": "system", "content": SYSTEM + " Variant " + variant}], [] + for n, u in enumerate(USER_TURNS, 1): + tools = TOOLS + [EXTRA_TOOL] if (variant == "add_tool_turn3" and n == 3) else TOOLS + msgs.append({"role": "user", "content": u}) + run_turn("turn%d" % n, msgs, tools, 0, variant != "drop_reasoning", recs) + log("convo", {"variant": variant, "requests": recs}) + + +TRIALS = { + "read_file": ["Show me what is in /etc/hosts.", "Open /home/kyle/todo.md and summarise it.", + "What does /etc/os-release say?", "Read the file /var/log/boot.log.", + "I need the contents of /home/kyle/.gitconfig."], + "write_file": ["Save the text 'buy milk' to /home/kyle/shopping.txt.", + "Create /tmp/hello.txt containing the single word hello.", + "Write a two-line haiku about rain into /home/kyle/haiku.txt.", + "Put the JSON {\"a\": 1, \"b\": [2, 3]} into /tmp/data.json exactly.", + "Replace /home/kyle/motd with: Stay \"curious\" & ."], + "shell": ["How much disk space is free? Use the shell.", "Run uname -a for me.", + "Count the lines in /etc/passwd with a shell command.", + "Use the shell to find files larger than 1GB under /var, with a 30 second timeout.", + "Run: echo \"a && b\" | tr a-z A-Z"], + "http_fetch": ["Fetch https://example.com and tell me the title.", + "GET https://api.github.com/zen please.", + "Download https://example.org/robots.txt.", + "Fetch https://example.com/api with the header Accept: application/json.", + "What does http://neverssl.com return?"], +} + + +def check_tools(): + """(c) 20 trials across 4 tools: is the tool call parsed server-side, with valid arguments?""" + fails = 0 + for want, prompts in TRIALS.items(): + required = [t for t in TOOLS if t["function"]["name"] == want][0]["function"]["parameters"]["required"] + for p in prompts: + r = chat([{"role": "system", "content": SYSTEM}, {"role": "user", "content": p}], TOOLS, 0, 1536) + m, why = r["choices"][0]["message"], [] + calls = m.get("tool_calls") or [] + if not calls: + why.append("no tool_calls") + for c in calls: + try: + args = json.loads(c["function"]["arguments"]) + why += ["missing arg " + a for a in required if a not in args] + except ValueError: + why.append("arguments not JSON") + if c["function"]["name"] != want: + why.append("called " + c["function"]["name"]) + if "" in (m.get("content") or "") or "" in (m.get("reasoning_content") or ""): + why.append("raw text leaked") + if r["choices"][0]["finish_reason"] != "tool_calls": + why.append("finish_reason=" + r["choices"][0]["finish_reason"]) + fails += bool(why) + print("%-10s %-4s %s" % (want, "FAIL" if why else "ok", "; ".join(why) or p[:50])) + log("tools", {"want": want, "prompt": p, "problems": why, "message": m, + "predicted_n": r["timings"]["predicted_n"]}) + print("failures: %d / 20" % fails) + + +def slots(): + return http("/slots?model=" + MODEL) + + +def check_slots(): + for s in slots(): + print({k: s.get(k) for k in ("id", "n_ctx", "is_processing", "id_task")}, + "n_past/prompt tokens:", s.get("n_past", s.get("next_token"))) + + +def check_pin(): + """(d) slot pinning, and what unpinned traffic does to a pinned session's cache.""" + recs = [] + a = [{"role": "system", "content": SYSTEM}, {"role": "user", "content": "Session A. " + filler(6000, "A") + "\nReply with the single word: alpha"}] + b = [{"role": "system", "content": SYSTEM}, {"role": "user", "content": "Session B. " + filler(6000, "B") + "\nReply with the single word: beta"}] + for label, msgs, slot in (("A turn1 slot0", a, 0), ("B turn1 slot1", b, 1)): + r = chat(msgs, None, slot, 512) + recs.append(tline(label, r)); msgs.append(assistant_msg(r)) + a.append({"role": "user", "content": "Again, one word."}) + r = chat(a, None, 0, 512); recs.append(tline("A turn2 slot0 (after B)", r)); a.append(assistant_msg(r)) + # Two unpinned strangers, as Open WebUI or OpenCode would send. + for i in (1, 2): + c = [{"role": "user", "content": "Stranger %d. %s\nReply: gamma" % (i, filler(3000, "C%d" % i))}] + r = chat(c, None, -1, 512); recs.append(tline("stranger %d unpinned" % i, r)) + a.append({"role": "user", "content": "Once more, one word."}) + r = chat(a, None, 0, 512); recs.append(tline("A turn3 slot0 (after strangers)", r)) + b.append({"role": "user", "content": "Again, one word."}) + r = chat(b, None, 1, 512); recs.append(tline("B turn2 slot1 (after strangers)", r)) + log("pin", {"requests": recs}) + + +def check_throughput(): + """(a) timed requests. pp at depth D = rate for a 2k suffix appended to a cached D-token prefix.""" + recs = [] + for depth in (0, 32000): + msgs = [{"role": "user", "content": "Depth %d. " % depth + (filler(depth, depth) if depth else "")}] + if depth: + msgs[0]["content"] += "\nReply with: ok" + r = chat(msgs, None, 0, 64, chat_template_kwargs={"enable_thinking": False}) + recs.append(tline("prefill 0..%d (average)" % depth, r)); msgs.append(assistant_msg(r)) + msgs.append({"role": "user", "content": ""}) + msgs[-1]["content"] += filler(2000, "s%d" % depth) + "\nNow write about 250 words on tide pools." + r = chat(msgs, None, 0, 400, chat_template_kwargs={"enable_thinking": False}) + recs.append(tline("pp+tg at depth %d" % depth, r)) + log("throughput", {"requests": recs}) + + +def check_progress(): + """(i) does return_progress give bytes during prefill? Prints gaps between stream events.""" + body = {"model": MODEL, "stream": True, "return_progress": True, "id_slot": 0, "max_tokens": 32, **SAMPLING, + "chat_template_kwargs": {"enable_thinking": False}, + "messages": [{"role": "user", "content": filler(16000, "p") + "\nReply: ok"}]} + req = urllib.request.Request(BASE + "/v1/chat/completions", json.dumps(body).encode(), {"Content-Type": "application/json"}) + t0 = last = time.time(); gaps, events = [], [] + with urllib.request.urlopen(req, timeout=1800) as r: + for line in r: + if not line.startswith(b"data: {"): + continue + now = time.time(); gaps.append(round(now - last, 2)); last = now + d = json.loads(line[6:]) + if "prompt_progress" in d: + events.append(d["prompt_progress"]) + print("progress events: %d, first: %s, last: %s" % (len(events), events[:1], events[-1:])) + print("max gap between events: %.2fs, total %.1fs" % (max(gaps), time.time() - t0)) + log("progress", {"n_events": len(events), "events": events[:3] + events[-2:], "max_gap_s": max(gaps)}) + + +def check_rewind(): + """(e2) hybrid-attention rewind: change text 10k tokens into a 24k prompt. How much cache survives?""" + f1, f2, f2b, f3 = filler(10000, "r1"), filler(10000, "r2"), filler(10000, "r2b"), filler(4000, "r3") + recs, kw = [], {"chat_template_kwargs": {"enable_thinking": False}} + for label, mid in (("original 24k", f2), ("same again", f2), ("diverge at ~10k", f2b), ("back to original", f2)): + r = chat([{"role": "user", "content": f1 + "\n" + mid + "\n" + f3 + "\nReply: ok"}], None, 0, 16, **kw) + recs.append(tline(label, r)) + log("rewind", {"requests": recs}) + + +FIND_TOOL = tool("find_tool", "Search for additional tools by keyword. Returns tool schemas.", + {"query": {"type": "string"}}, ["query"]) +CALL_TOOL = tool("call_tool", "Call a tool that was returned by find_tool. Pass its name and an arguments object.", + {"name": {"type": "string"}, "arguments": {"type": "object"}}, ["name", "arguments"]) +ASKS = ["What do my memory notes say about units of measurement?", "Search my notes for anything about Mattermost.", + "Do I have a note about my dentist? Check memory, top 3 results.", "Look in long-term memory for 'tailnet ACL'.", + "Recall what I wrote about coffee."] + + +def check_findtool(): + """(h2) progressive disclosure without touching the tools array: schema arrives as a tool result.""" + for variant, tools in (("meta call_tool", TOOLS[:2] + [FIND_TOOL, CALL_TOOL]), ("undeclared direct", TOOLS[:2] + [FIND_TOOL])): + ok = 0 + for ask in ASKS: + msgs = [{"role": "system", "content": SYSTEM + " If no listed tool fits, use find_tool first."}, + {"role": "user", "content": ask}] + r1 = chat(msgs, tools, 0, 1536); m1 = r1["choices"][0]["message"] + calls = m1.get("tool_calls") or [] + if not calls or calls[0]["function"]["name"] != "find_tool": + print(variant, "| step1 did not call find_tool:", (calls[0]["function"]["name"] if calls else m1.get("content", "")[:60])); continue + msgs.append(assistant_msg(r1)) + hint = "Call it with call_tool." if "meta" in variant else "Call it directly by name." + msgs.append({"role": "tool", "tool_call_id": calls[0]["id"], + "content": "1 tool found. " + hint + "\n" + json.dumps(EXTRA_TOOL["function"])}) + r2 = chat(msgs, tools, 0, 1536); m2 = r2["choices"][0]["message"] + c2 = (m2.get("tool_calls") or [None])[0] + desc = "no tool call; content=" + repr((m2.get("content") or "")[:120]) + if c2: + a = json.loads(c2["function"]["arguments"]); desc = c2["function"]["name"] + " " + json.dumps(a) + good = (c2["function"]["name"] == "call_tool" and a.get("name") == "recall" and isinstance(a.get("arguments"), dict) + and "query" in a["arguments"]) if "meta" in variant else (c2["function"]["name"] == "recall" and "query" in a) + ok += good + print("%-18s cache_n=%-5d %s" % (variant, r2["timings"]["cache_n"], desc)) + log("findtool", {"variant": variant, "ask": ask, "step2": m2, "timings": r2["timings"]}) + print("==", variant, "ok %d/5" % ok) + + +def check_rewind2(): + """(e3) divergence several turns back in a multi-request conversation: are request-boundary checkpoints reused?""" + kw, recs = {"chat_template_kwargs": {"enable_thinking": False}}, [] + msgs = [{"role": "system", "content": SYSTEM}] + for i in range(1, 6): + msgs.append({"role": "user", "content": "Part %d. %s\nReply: ok %d" % (i, filler(1500, "t%d" % i), i)}) + r = chat(msgs, None, 0, 16, **kw); recs.append(tline("grow turn %d" % i, r)); msgs.append(assistant_msg(r)) + for back in (5, 3, 2): # edit the user message of turn `back`, keep everything else + m2 = json.loads(json.dumps(msgs)); idx = 1 + 2 * (back - 1) + m2[idx]["content"] = m2[idx]["content"].replace("Reply: ok", "Reply now: ok") + r = chat(m2[:-1] if False else m2 + [{"role": "user", "content": "Final. Reply: done"}], None, 0, 16, **kw) + recs.append(tline("edit turn %d of 5, then ask" % back, r)) + r = chat(msgs + [{"role": "user", "content": "Final. Reply: done"}], None, 0, 16, **kw) + recs.append(tline(" restore original, then ask", r)) + log("rewind2", {"requests": recs}) + + +if __name__ == "__main__": + os.makedirs(OUT, exist_ok=True) + for name in sys.argv[1:] or ["slots"]: + print("\n#### %s (nonce %s)" % (name, NONCE)) + globals()["check_" + name]()