kyle/boxmaker
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

loopd: the tag check asks /tokenize twice, not once per tag

Browse Source 
Found in the M5b device check on straylight: the self-test asked about
each of the 33 tags and each neutralised form, 66 requests, one
connection each, and inferproxy's token bucket (burst 10) refused the
ninth, so loopd would not start. Now it asks about all the tags at once,
one per line, and then all their neutralised forms: the answer to the
first must hold each tag's recorded token in order, the second none of
them. Each tag is a special token, split out before anything else, so
the lines between cannot change it; and a neutralised form is only
dangerous as a tag's own token, so that is what the second answer must
not hold. The errors name a tag by its id, as before.

The fake tokenizer reads text with several tags in it; the tests cover
each tag as ordinary tokens, as another listed or unlisted token, a
duplicated token in the middle and at the end, a form holding each tag,
and both requests failing. Spec section 8 and the runbook entry follow.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
kyle
2026-09-27 19:08:42 -07:00
co-authored by Claude Opus 5.5
parent 1fb0ec8a06
commit 5a2464d81d
6 changed files with 277 additions and 176 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/loopd/src/selftest.rs
+59 -61
View File
@@ -3,7 +3,7 @@
//! parsed, and that a second turn reuses the first turn's cache. Any check failing means `loopd`
//! does not start.
use bxcore::sanitize::{TAGS, Tag, neutralise};
use bxcore::sanitize::{TAGS, neutralise};
use crate::llama::{
ChatMessage, ChatRequest, Client, FinishReason, InferError, ToolSchema,
@@ -24,25 +24,18 @@ pub enum SelfTestError {
},
Infer(InferError),
Hash,
/// The tag with the recorded id `id` is `count` tokens, not one.
TagNotOneToken {
/// The first tag, by its recorded id, whose token is not in its place in the server's answer
/// about the tags (or, past the last tag, an extra tag token the server gave).
TagNotRecorded {
id: u32,
count: usize,
},
/// The tag with the recorded id `id` is one token, but the token `got`.
TagOtherId {
/// The server's answer about the neutralised forms holds the tag token `id`.
NeutralisedIsTag {
id: u32,
got: u32,
},
/// The neutralised form of the tag with the recorded id `id` is `count` tokens, not more
/// than one.
NeutralisedTooFew {
id: u32,
count: usize,
},
/// `/tokenize` failed for the tag with the recorded id `id`, or for its neutralised form.
/// `/tokenize` failed; `what` is "the tags" or "the neutralised tags".
Tokenize {
id: u32,
what: &'static str,
error: InferError,
},
}
@@ -52,9 +45,8 @@ impl SelfTestError {
/// its own entry; every other failure shares the self-test's.
pub fn pointer(&self) -> &'static str {
match self {
SelfTestError::TagNotOneToken { .. }
| SelfTestError::TagOtherId { .. }
| SelfTestError::NeutralisedTooFew { .. }
SelfTestError::TagNotRecorded { .. }
| SelfTestError::NeutralisedIsTag { .. }
| SelfTestError::Tokenize { .. } => "see docs/runbook.md#loopd-tags-unexpected",
SelfTestError::Mismatch { .. }
| SelfTestError::ToolCall(_)
@@ -82,18 +74,17 @@ impl std::fmt::Display for SelfTestError {
),
SelfTestError::Infer(e) => write!(f, "{e}"),
SelfTestError::Hash => write!(f, "the chat template could not be hashed"),
SelfTestError::TagNotOneToken { id, count } => {
write!(f, "the tag with id {id} is {count} tokens, not one")
SelfTestError::TagNotRecorded { id } => {
write!(
f,
"the tag with id {id} does not tokenize as its recorded token"
)
}
SelfTestError::TagOtherId { id, got } => {
write!(f, "the tag with id {id} is one token, but the token {got}")
SelfTestError::NeutralisedIsTag { id } => {
write!(f, "a neutralised tag tokenizes to the tag with id {id}")
}
SelfTestError::NeutralisedTooFew { id, count } => write!(
f,
"the neutralised form of the tag with id {id} does not split into several tokens (the server gave {count})"
),
SelfTestError::Tokenize { id, error } => {
write!(f, "could not tokenize the tag with id {id}: {error}")
SelfTestError::Tokenize { what, error } => {
write!(f, "could not tokenize {what}: {error}")
}
}
}
@@ -161,42 +152,49 @@ fn server_matches(client: &Client) -> Result<(), SelfTestError> {
Ok(())
}
/// Check 2: each of the model's tags (`bxcore::sanitize::TAGS`) is one token, the one recorded
/// with it, and its neutralised form is more than one (M5b spec, section 8). Like check 1 it sends
/// no prompt and uses no slot, so a server whose tokenizer is not the recorded one is sent no
/// prompt either. A tag missing from the list cannot be found: the server does not list them.
/// Check 2 (M5b spec, section 8): asked about every tag of `bxcore::sanitize::TAGS` at once, one
/// per line, the server gives each tag's recorded token, in the list's order; asked about every
/// neutralised form, it gives none of those tokens. Each tag is a special token, which the
/// tokenizer splits out before anything else, so the lines between cannot change it. Two requests,
/// so that the check fits in `inferproxy`'s burst of connections. Like check 1 it sends no prompt
/// and uses no slot. A tag missing from the list cannot be found: the server does not list them.
/// An error names a tag by its id, never by its text, which the operator's terminal and log would
/// show whole.
fn tags_tokenize(client: &Client) -> Result<(), SelfTestError> {
for tag in TAGS {
check_tag(client, tag)?;
let recorded: Vec<u32> = TAGS.iter().map(|tag| tag.id).collect();
let tags: Vec<&str> = TAGS.iter().map(|tag| tag.text).collect();
let got: Vec<u32> = client
.token_ids(&tags.join("\n"))
.map_err(|error| SelfTestError::Tokenize {
what: "the tags",
error,
})?
.into_iter()
.filter(|id| recorded.contains(id))
.collect();
let places = recorded.len().max(got.len());
if let Some(at) = (0..places).find(|&at| got.get(at) != recorded.get(at)) {
let id = recorded
.get(at)
.or(got.get(at))
.copied()
.unwrap_or_default();
return Err(SelfTestError::TagNotRecorded { id });
}
Ok(())
}
/// Two `/tokenize` requests: `tag.text`, then `neutralise(tag.text)`. An error names the tag by
/// its id, never by its text, which the operator's terminal and log would show whole.
fn check_tag(client: &Client, tag: &Tag) -> Result<(), SelfTestError> {
let id = tag.id;
let tokenize = |text: &str| {
client
.token_ids(text)
.map_err(|error| SelfTestError::Tokenize { id, error })
};
let ids = tokenize(tag.text)?;
match ids.as_slice() {
[got] if *got == id => {}
[got] => return Err(SelfTestError::TagOtherId { id, got: *got }),
_ => {
return Err(SelfTestError::TagNotOneToken {
id,
count: ids.len(),
});
}
let forms: Vec<String> = TAGS
.iter()
.map(|tag| neutralise(tag.text).into_owned())
.collect();
let got = client
.token_ids(&forms.join("\n"))
.map_err(|error| SelfTestError::Tokenize {
what: "the neutralised tags",
error,
})?;
match got.into_iter().find(|id| recorded.contains(id)) {
Some(id) => Err(SelfTestError::NeutralisedIsTag { id }),
None => Ok(()),
}
let count = tokenize(&neutralise(tag.text))?.len();
if count <= 1 {
return Err(SelfTestError::NeutralisedTooFew { id, count });
}
Ok(())
}
/// Check 3: a tool call comes back parsed, with a `read_file` tool and a string `path`.