loopd: the self-test checks the model's tags against the tokenizer

Co-Authored-By: ornith-1.5-35b-a3b <agent@wntrmute.dev>

docs/implementer-log.md

@@ -6,6 +6,7 @@ reviewer adds findings under "Reviews" once per milestone.
 
 | Task | Date | Status | Gate runs | First gate | Deviations | Notes | Model |
 |---|---|---|---|---|---|---|---|
+| M5b/08-loopd-selftest-tags | 2026-09-26 | done | 2 | pass | none | Copied the five given test files from `docs/plans/M5b/files/`: `selftest_tags.rs` (new) and new versions of `selftest.rs`, `serve.rs`, `pointers.rs` and `support/mod.rs` (did not open them). Step 2 showed exactly the four named failures (`selftest`: three steps without `"tags tokenize as recorded"`; `a_completion_without_the_tool_call_fails_the_tool_call_check` left 2 right 3; `a_cold_second_turn_fails_the_cache_check` left 3 right 4; `serve_runs_the_self_test_then_binds_the_socket_with_mode_0600` left 0 right 66) and `selftest_tags.rs` failing to compile with 7 E0599s (no `TagNotOneToken`/`TagOtherId`/`NeutralisedTooFew`/`Tokenize` variant, no `pointer`). Appended `runbook-loopd-tags-unexpected.md`; `check-runbook.sh` printed nothing and exited 0, `grep -c '^## loopd-tags-unexpected$'` printed 1. Made the edits in order: I1 the private `TokenIds` struct before `CacheOutcome` and I2 the `pub token_ids` method after `tokenize` (left `tokenize` itself unchanged); S1 the `use bxcore::sanitize::{TAGS, Tag, neutralise}` import, S2 the four tag variants plus `SelfTestError::pointer` with each pointer written whole, S3 the four Display arms, S4 the second check `tags tokenize as recorded`, S5 the `tags_tokenize` and `check_tag` functions, S6 check 3 to check 4; M1 `e.pointer()` in `run_selftest_check`. `cargo fmt --all`; `cargo check -p loopd` clean; `selftest_tags` 8, `selftest` 6, `serve` 3, `pointers` 7 passed, every `loopd` suite passes. `grep -rn 'runbook.md#loopd-' crates/loopd/src` prints exactly two lines, both in `selftest.rs`. `make gate` printed `gate: ok` with 1050 tests (1042 before this task, plus the 8 in `selftest_tags.rs`). No new dependency, so `Cargo.lock` is unchanged. | ornith-1.5-35b-a3b |
 | M5b/07-loopd-neutralise | 2026-09-26 | done | 2 | fail | none | Copied the three given tests from `docs/plans/M5b/files/`: `neutralise.rs`, `neutralise_replay.rs` and `support/tags.rs` (did not open them). Step 2 showed exactly the seven named failures (all six in `neutralise`, `the_memory_block_is_neutralised_when_the_baseline_is_assembled` in `neutralise_replay`); the two that should pass passed. Made the edits in order: T1 the module comment, T2 the `use bxcore::sanitize::neutralise` import, T3 `content: neutralise(content).into_owned()` on the `LogRecord::User`, T4 the capped-then-neutralised `let text = neutralise(&text).into_owned()` on the `LogRecord::ToolResult`; B1 the import, B2 the `assemble_with` comment, B3 `neutralise(memory.trim())` with `&memory` pushed. `cargo fmt --all`; `cargo check -p loopd` clean; `neutralise` 6 passed, `neutralise_replay` 3 passed, every `loopd` suite passes. `grep -rn 'neutralise(' crates/loopd/src` prints exactly three lines (two in `turn.rs`, one in `baseline.rs`). First `make gate` run failed on a flaky brokerd concurrency test, `a_torn_tail_is_recovered_and_it_serves` (one fail out of the serve suite under full-workspace load); it passes 3/3 in isolation and the serve suite 9/9 three times on the same tree, and the loopd edits touch no brokerd path, so it is a timing flake, not this task. Re-ran `make gate`: `gate: ok` with 1042 tests. No new dependency, so `Cargo.lock` is unchanged. | ornith-1.5-35b-a3b |
 | M5b/06-memoryd-sanitize | 2026-09-26 | done | 1 | pass | none | Copied the five given tests from `docs/plans/M5b/files/`: `sanitize.rs`, `serve_sanitize.rs`, and new versions of `edit.rs`, `files.rs` and `serve.rs` (did not open them). Step 2 showed exactly the seven named failures (`refusals_change_nothing` in `edit`, `an_entry_that_fails_the_check_is_left_out_and_logged` in `serve`, all five in `serve_sanitize`); `files` did not compile (one E0609, no field `rule` on `&memoryd::check::Refused`) and `sanitize` did not compile (13: E0432 for `memoryd::edit::flagged`/`refused`, E0609 for `flag` on `Outcome` and `&Record` and `rule` on `Refused`, E0615 for `sentence`). Appended the three runbook fragments to `docs/runbook.md`; `check-runbook.sh` printed nothing and exited 0. `git rm`'d `crates/memoryd/src/check.rs` and its test. Made the edits in order: E1 removed `pub mod check;` from `lib.rs`; E2-E10 in `edit.rs` (imports, `Outcome` gains `flag`, five functions replacing `checked`, the refuse check in `apply`, the flag check after `edited`, `flag: None`, `text.trim()` in `edited`, `noted` plus `as_notes`, `text.trim()` in `consolidate`); E11-E13 in `files.rs` (imports, `Block`, the two `block` comments, the `refuse` match); E14-E19 in `journal.rs` (`Record` gains `flag` after `untrusted`, `record` sets `flag: outcome.flag.map(str::to_string)`, `log` appends `  flagged {rule}`, `undo` sets `flag: None`); E20 in `review.rs` (`reject` sets `flag: None`); E21-E23 in `serve.rs` (`block` logs by `refusal.rule` with the runbook pointer, the `memory` comment, and `memory` calls `edit::refused` first then `edit::flagged` to keep flagged writes off review and logs the rule). `cargo fmt --all`; `cargo check -p memoryd` clean; `cargo test -p memoryd` every suite passes (sanitize 11, serve_sanitize 5, edit 8, files 8, serve 8); `make gate` printed `gate: ok` with 1033 tests; `grep -rn 'memoryd::check\|crate::check' crates/` prints nothing. No new dependency, so `Cargo.lock` is unchanged. | ornith-1.5-35b-a3b |
 | M5b/05-sanitize-flag | 2026-09-26 | done | 1 | pass | none | Copied the three given tests from `docs/plans/M5b/files/`: `bxcore/tests/sanitize_flag.rs`, `sanitize_flag_corpus.rs`, `sanitize_flag_edges.rs`. Step 2 showed exactly the named E0432s (`Flag`/`flag`/`normalise` for `sanitize_flag`, `flag` for the other two). Copied the two skeletons, `bxcore/src/sanitize/normalise.rs` (five functions) and `flag.rs` (seven), then made the two edits: E1 replaced `bxcore/src/sanitize/mod.rs` whole (added `mod flag; mod normalise;` and `pub use flag::{Flag, flag};` / `pub use normalise::normalise;`), E2 made `invisible` `pub(super)` in `refuse.rs` (nothing else changed). Filled `normalise.rs` in the given order, each from its comment and `cargo check` after: `leet` (`LEET.iter().find` the digit, `map_or(c, |&(_, letter)| letter)`); `homoglyph` (`HOMOGLYPHS.iter().find` the `from`, `map` the `to`); `full_width` (`u32::from(c)` inside `FULL_WIDTH`, `checked_sub(FULL_WIDTH_OFFSET).and_then(char::from_u32).unwrap_or(c)`); `push_folded` (`full_width` then `homoglyph`, pushing the letter when `Some`, else `c.to_lowercase().map(leet)`); `normalise` (walk chars, `space` bool: whitespace sets it, `invisible` dropped leaving it, any other char pushes one `' '` when `space` and the output is not empty, clears `space`, then `push_folded`). Filled `flag.rs` in the given order: `in_sentence` (`!word.ends_with(['.','!','?',';']) && !["&&","||","|"].contains(&word)`); `filler` (`!word.is_empty() && word.chars().all(|c| c.is_alphanumeric() || c == '_')`); `is` (trim non-word ends with `trim_matches`, compare the word and its stripped form to each keyword); `ends_clause` (`word.ends_with(CLAUSE_ENDS)`, or `after.is_empty()`, or `after` starts with a `CLAUSE_FOLLOWERS` matched word by word as `is`); `one_word` (`split_first`, `test` the first word with the words after it, then `seq_at` the remaining parts on those words); `skip` (counts `0..=max`, `split_at_checked`: a skipped word failing `test` or a count past the end returns `false`, otherwise when all pass check `seq_at`); `seq_at` (`split_first`; `Is`/`Ends`/`Holds` go to `one_word` with `is`, `is && ends_clause`, `contains`, `Filler`/`Gap` to `skip` with `filler`/`in_sentence`). My first `one_word` edit dropped a `]` and `)` from the `impl Fn` signature; caught by diffing against the skeleton and fixed. After the fills `cargo check -p bxcore` is clean. `sanitize_flag` 22 passed, `sanitize_flag_corpus` 2, `sanitize_flag_edges` 2; `sanitize_refuse` 14, `sanitize_refuse_corpus` 2, `sanitize_tags` 15, `sanitize_corpus` 3. `cargo fmt --all`; `make gate` printed `gate: ok`; `cargo test --workspace` reports 1022 passed, plus the gate's separate loopd `end_to_end` ignored test, for 1023 total. No new dependency, so `Cargo.lock` is unchanged. | ornith-1.5-35b-a3b |