loopd: the tag check asks /tokenize twice, not once per tag

Found in the M5b device check on straylight: the self-test asked about
each of the 33 tags and each neutralised form, 66 requests, one
connection each, and inferproxy's token bucket (burst 10) refused the
ninth, so loopd would not start. Now it asks about all the tags at once,
one per line, and then all their neutralised forms: the answer to the
first must hold each tag's recorded token in order, the second none of
them. Each tag is a special token, split out before anything else, so
the lines between cannot change it; and a neutralised form is only
dangerous as a tag's own token, so that is what the second answer must
not hold. The errors name a tag by its id, as before.

The fake tokenizer reads text with several tags in it; the tests cover
each tag as ordinary tokens, as another listed or unlisted token, a
duplicated token in the middle and at the end, a form holding each tag,
and both requests failing. Spec section 8 and the runbook entry follow.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

crates/loopd/tests/support/mod.rs

@@ -180,11 +180,36 @@ pub fn tokenizer(request: &Recorded) -> Reply {
     let Some(content) = body.get("content").and_then(|c| c.as_str()) else {
         return Reply::json(400, r#"{"error":"no content"}"#);
     };
-    let ids: Vec<u32> = match bxcore::sanitize::TAGS.iter().find(|t| t.text == content) {
-        Some(tag) => vec![tag.id],
-        None => content.bytes().map(u32::from).collect(),
-    };
-    tokens(&ids)
+    tokens(&tokenize(content, &[]))
+}
+
+/// The fake tokenizer's ids for `text`: the id of each listed tag in it, the longest where two
+/// start at the same place, and one id per byte for everything else. A tag whose text is a key of
+/// `overrides` gets that entry's ids instead.
+pub fn tokenize(text: &str, overrides: &[(&str, Vec<u32>)]) -> Vec<u32> {
+    let mut ids = Vec::new();
+    let mut rest = text;
+    while let Some(c) = rest.chars().next() {
+        let tag = bxcore::sanitize::TAGS
+            .iter()
+            .filter(|t| rest.starts_with(t.text))
+            .max_by_key(|t| t.text.len());
+        let len = match tag {
+            Some(tag) => {
+                match overrides.iter().find(|(text, _)| *text == tag.text) {
+                    Some((_, instead)) => ids.extend(instead),
+                    None => ids.push(tag.id),
+                }
+                tag.text.len()
+            }
+            None => {
+                ids.extend(c.to_string().bytes().map(u32::from));
+                c.len_utf8()
+            }
+        };
+        rest = &rest[len..];
+    }
+    ids
 }
 
 /// A `/tokenize` answer as llama-server gives one without `with_pieces`: `{"tokens": [ids]}`.