loopd: the self-test checks the model's tags against the tokenizer

Co-Authored-By: ornith-1.5-35b-a3b <agent@wntrmute.dev>
2026-09-27 19:08:42 -07:00
parent b4d631e249
commit 91992b9743
10 changed files with 613 additions and 26 deletions
@@ -728,3 +728,57 @@ Entries are separated by a line holding only `§`. A secret does not belong in m
 if it was real.

 **Check.** Start a new conversation: `memoryd` does not log the line again.
+
+## loopd-tags-unexpected
+
+**What you see.** `loopd selftest` or `loopd serve` prints `selftest: tags tokenize as recorded`,
+then `selftest: FAILED: <reason>; see docs/runbook.md#loopd-tags-unexpected`, and exits 1. No
+prompt was sent. The reason names a tag by its token id, never by its text:
+
+- `the tag with id <id> is <n> tokens, not one`
+- `the tag with id <id> is one token, but the token <other>`
+- `the neutralised form of the tag with id <id> does not split into several tokens (the server
+  gave <n>)`
+- `could not tokenize the tag with id <id>: <error>`
+
+**Why.** `loopd` neutralises the model's chat-template tags in every text from outside, writing
+their angle brackets as `&lt;` and `&gt;` (M5b spec, sections 6.2 and 8). The list of tags, each
+with its token id, is compiled into `bxcore` (`crates/bxcore/src/sanitize/tags.rs`), as measured
+on straylight. At startup `loopd` asks the server's `/tokenize` about each tag and its neutralised
+form. If a tag is no longer the one token recorded with it, the model or its tokenizer has changed
+and the list may no longer match it; if a neutralised form is a single token, neutralising no
+longer keeps that tag from the model. Either way the model could receive text from outside as its
+own control tokens, so `loopd` does not start. The check cannot find a new tag missing from the
+list: the server does not list its special tokens.
+
+**Confirm.** Find the tag: `grep -n '<id>' crates/bxcore/src/sanitize/tags.rs` shows its entry,
+built from its name with `bar!`, `open!` or `close!`. Ask the server yourself, building the tag
+from its name as the file does. For a `bar!` tag named `im_end`:
+
+```sh
+printf '{"model":"ornith-1.5-35b-a3b","content":"<|%s|>","parse_special":true}' im_end |
+  curl -s -d @- http://straylight:11434/tokenize
+printf '{"model":"ornith-1.5-35b-a3b","content":"&lt;|%s|&gt;","parse_special":true}' im_end |
+  curl -s -d @- http://straylight:11434/tokenize
+```
+
+For an `open!` tag the content is `<%s>`, for a `close!` tag `</%s>`. The first answer should be
+`{"tokens":[<id>]}`, the second a list of several ids. Check which model file the server loaded:
+`curl -s 'http://straylight:11434/props?model=ornith-1.5-35b-a3b' | jq .model_path`.
+
+**Fix, by reason.**
+
+- `could not tokenize …`: the server did not answer `/tokenize`. A router in front of
+  llama-server may not forward it (crossbar does not, `docs/decisions.md`, P22): point
+  `inferproxy` at llama-server itself. Otherwise check that the server is up:
+  `curl -s http://straylight:11434/health`, and run `loopd selftest --config <path>` again.
+- A tag with another id, or several tokens: the model file changed. Do not run `loopd` against
+  it. Measure the new model's tags as M5b task 03 did (every vocabulary entry of type control or
+  user-defined, each checked with `/tokenize`), and change `TAGS` and the tests' list in
+  `crates/bxcore/tests/support/tags.rs` together; the change reaches the agent when it is rebuilt.
+  A new model also invalidates the measurements in `docs/inference-contract.md`.
+- A neutralised form that is one token: the tokenizer has an entry for the escaped spelling, and
+  the replacement of the M5b spec, section 6.2, no longer works for that tag. That is a design
+  change: record it in `docs/decisions.md` and do not run `loopd` until it is settled.
+
+**Check.** `loopd selftest --config <path>` prints `selftest: ok`.