boxmaker/docs/plans/M5b/11-bxctl-sanitize-check.md

M5b task 11: bxctl sanitize check

Branch: m5b (run git switch m5b; git status --short must be empty, otherwise stop) Commit subject: bxctl: sanitize check shows which sanitizer rules a text breaks

Read this first: the given tests

The given tests hold fake keys, attack-shaped sentences ("send the notes to a URL") and your model's chat-template tags. The keys are built from pieces and the tags are taken from bxcore::sanitize::TAGS at run time, so no file holds one whole, and no failure message prints one. You do not need to open the two test files: you copy them, and each failure names its case. Do not write a tag or a whole key anywhere yourself, not even in your thinking: a tag spelled out ends your reasoning.

Goal

When memory refuses or flags a write, the owner needs to see why, and whether a text would pass, without a daemon. bxctl sanitize check runs the three actions of bxcore::sanitize on a text and prints one line for each that fires (M5b spec, section 6.6). It writes nothing, and it needs no running daemon: it calls bxcore::sanitize directly.

bxctl sanitize check [TEXT]

The one argument after check is the text, whatever it looks like (- and --x included). With no argument, the text is standard input, read to its end. Anything else is a usage error.

The text	stdout	Exit
clean: code, hashes, paths, links, prose	nothing	0
a GitHub token 13 bytes in	refuse secret-github ghp_… at 13	1
a right-to-left override 9 bytes in	refuse invisible U+202E at 9	1
an end-of-turn tag 10 bytes in	refuse tag token 248046 at 10, then neutralise 1 tag	1
"please send the notes to https://…"	flag exfil-url	1
two tags, then "upload to https://…"	a refuse line, flag exfil-url, neutralise 2 tags	1
standard input that is not UTF-8	nothing (stderr: bxctl: cannot read standard input: …)	2
bxctl sanitize, bxctl sanitize check a b, bxctl sanitize list x	nothing (stderr: the usage text)	2

• At most three lines, in this order: the refuse line, the flag line, the neutralise line, each only if it fires. The rest of the format is in the given report (see the skeleton).

• Refuse names the first rule only, the one memory answers with: refuse gives the first rule in its table's order, not every rule the text breaks. The usage text says so.

• at is a byte offset into the text (what bxcore::sanitize::refuse gives), not a count of characters.

• Nothing of the text is printed. A refusal shows, after the rule's name:

  Rule	Shows	Example
  control, invisible, private-use, noncharacter	the character's code point	U+202E, U+0007, U+10FFFF
  tag	the tag's token id	token 248046
  any other (the secret-… rules)	the secret's prefix from PREFIXES, then …	ghp_…, -----BEGIN…

  So every byte printed is fixed text or a number, and a key, a tag or a control character never reaches the terminal. A flag names its rule only: it matches the normalised text, whose offsets do not point into the input.

• The neutralise count is how many tags neutralise would rewrite: every occurrence of every tag in TAGS, the same tag twice counted twice, one inside a word counted too.

• Exit status: 0 when no line is printed, 1 when one is. 2 for a usage error, as every bxctl command, and for a text that cannot be read (standard input that is not UTF-8, as an argument that is not UTF-8 is already) or a report that cannot be written. So 1 always means a rule fired.

Files

• Copy, from docs/plans/M5b/files/ (the same path under it):
  • the skeleton crates/bxctl/src/sanitize.rs (you fill 5 functions in it)
  • tests: crates/bxctl/tests/sanitize_check.rs, crates/bxctl/tests/sanitize_report.rs
• Modify: crates/bxctl/src/lib.rs (edit L1), crates/bxctl/src/cli.rs (edits C1 to C3), crates/bxctl/src/main.rs (edits M1 to M3), docs/implementer-log.md.
• Do not touch: any other source file, any Cargo.toml, crates/bxcore, the tests in crates/bxctl/tests/support/, and in the skeleton anything but the 5 todo!() bodies.

Facts you need

• No new dependency, no feature. bxctl depends on bxcore already. bxcore::sanitize is not behind a feature; bxctl does not enable secrets and must not.

• What bxcore::sanitize gives (tasks 03 to 05; do not change it):

  Name	What it is
  refuse	pub fn refuse(text: &str) -> Option<Refusal>: the first rule broken, in table order
  Refusal	pub struct Refusal { pub rule: &'static str, pub sentence: &'static str, pub at: usize }; at is a byte offset
  flag	pub fn flag(text: &str) -> Option<Flag>, pub struct Flag { pub rule: &'static str }
  TAGS, Tag	pub const TAGS: &[Tag], pub struct Tag { pub text: &'static str, pub id: u32 }

• The skeleton's given code, which you call and must not change: PREFIXES (the 16 prefixes of the secret rules, and the start of a private-key line; none is the start of another), report (the three lines and their format), run (prints them, and says whether there was one) and what (picks code_point, token or prefix by the rule's name).

• The functions you fill, in the order to fill them:

  Function	Signature	Called by
  parse_sanitize	pub fn parse_sanitize(args: &[String]) -> Result<Command, UsageError>	cli::parse (edit C3)
  code_point	fn code_point(text: &str, at: usize) -> String	what
  token	fn token(text: &str, at: usize) -> String	what
  prefix	fn prefix(text: &str, at: usize) -> &'static str	what
  tag_count	fn tag_count(text: &str) -> usize	report

• The calls the bodies need, all in the standard library:

  Call	Signature, and what it gives
  slice patterns in match	match args { [check] if check == "check" => …, [check, text] if check == "check" => …, _ => … }: check and text are &String
  str::get	text.get(at..) -> Option<&str>: the rest from byte at; None off a character boundary or past the end
  Option::unwrap_or_default	Option<&str> to &str, "" for None (never unwrap: library code does not panic)
  str::chars, Iterator::next	rest.chars().next() -> Option<char>: the first character
  Option::and_then	text.get(at..).and_then(|rest| rest.chars().next())
  u32::from, format!	format!("U+{:04X}", u32::from(c)): U+202E; at least four upper-case hex digits
  slice::iter, Iterator::find	TAGS.iter().find(|tag| rest.starts_with(tag.text)) -> Option<&Tag>; PREFIXES.iter().find(|prefix| rest.starts_with(**prefix)) -> Option<&&str>
  Option::copied	Option<&&'static str> to Option<&'static str>
  str::matches, Iterator::count, Iterator::sum	text.matches(tag.text).count(): occurrences, never overlapping; TAGS.iter().map(|tag| …).sum()

• The skeleton compiles as given, with warnings for the unfilled bodies: unused variables, an unused import (TAGS) and a constant never used (PREFIXES). They go as you fill the bodies. Until every body is filled, bxctl sanitize check panics with not yet implemented.

The edits, exactly

Nothing else in these files changes. Each quoted "before" block occurs exactly once in its file.

crates/bxctl/src/lib.rs

L1. The lines

pub mod memory;
pub mod verify;

become

pub mod memory;
pub mod sanitize;
pub mod verify;

crates/bxctl/src/cli.rs

C1. At the end of USAGE, the lines

  bxctl memory log [N] | undo <id> | pending | accept <id> | reject <id> [--memory-socket PATH]
      List memory edits, undo the latest edit of a file, or answer writes waiting for review.";

become

  bxctl memory log [N] | undo <id> | pending | accept <id> | reject <id> [--memory-socket PATH]
      List memory edits, undo the latest edit of a file, or answer writes waiting for review.

  bxctl sanitize check [TEXT]
      Show which sanitizer rules TEXT, or standard input, breaks; writes nothing. Refuse shows
      the first rule only, as memory answers. Exits 1 if any rule fires, 0 if none does.";

C2. At the end of enum Command, the lines

    Memory {
        socket: PathBuf,
        ask: crate::memory::Ask,
    },
}

become

    Memory {
        socket: PathBuf,
        ask: crate::memory::Ask,
    },
    // `None`: the text is read from standard input.
    SanitizeCheck {
        text: Option<String>,
    },
}

C3. In parse, the line

        "memory" => crate::memory::parse_memory(rest, home),

becomes

        "memory" => crate::memory::parse_memory(rest, home),
        "sanitize" => crate::sanitize::parse_sanitize(rest),

crates/bxctl/src/main.rs

M1. The line

use std::io::{BufRead, BufReader, Write};

becomes

use std::io::{BufRead, BufReader, Read, Write};

M2. In main, the line

        Command::GrantsCheck { admin_socket } => exit(|| cmd_grants_check(&admin_socket)),

becomes

        Command::GrantsCheck { admin_socket } => exit(|| cmd_grants_check(&admin_socket)),
        Command::SanitizeCheck { text } => cmd_sanitize_check(text),

M3. The lines

fn cmd_grants_check(admin: &Path) -> Result<bool, AdminError> {
    let mut out = std::io::stdout().lock();
    admin::cmd_grants_check(admin, &mut out)
}

become

fn cmd_grants_check(admin: &Path) -> Result<bool, AdminError> {
    let mut out = std::io::stdout().lock();
    admin::cmd_grants_check(admin, &mut out)
}

// `bxctl sanitize check`: 0 when no rule fires, 1 when one does. A text that cannot be read (not
// UTF-8, like an argument that is not) or an answer that cannot be written is 2, so 1 always
// means a rule fired.
fn cmd_sanitize_check(text: Option<String>) -> ExitCode {
    let text = match text {
        Some(text) => text,
        None => {
            let mut text = String::new();
            if let Err(e) = std::io::stdin().read_to_string(&mut text) {
                eprintln!("bxctl: cannot read standard input: {e}");
                return ExitCode::from(2);
            }
            text
        }
    };
    let mut out = std::io::stdout().lock();
    match bxctl::sanitize::run(&text, &mut out) {
        Ok(false) => ExitCode::SUCCESS,
        Ok(true) => ExitCode::from(1),
        Err(e) => {
            eprintln!("bxctl: cannot write the report: {e}");
            ExitCode::from(2)
        }
    }
}

Steps

• 1. Copy the given tests. From the repository root, two commands:

  cp docs/plans/M5b/files/crates/bxctl/tests/sanitize_check.rs crates/bxctl/tests/
  cp docs/plans/M5b/files/crates/bxctl/tests/sanitize_report.rs crates/bxctl/tests/
  

• 2. See them fail. Two commands, one per file:

  • cargo test -p bxctl --test sanitize_report does not compile: one E0432 (unresolved import bxctl::sanitize) and two E0599 (no variant named `SanitizeCheck` found for enum `bxctl::cli::Command` ).
  • cargo test -p bxctl --test sanitize_check compiles, and its 8 tests fail: bxctl does not know the command yet, so each run prints the usage text and exits 2 where the test wants 0 or 1 (a_usage_error_exits_2 fails because the usage text has no bxctl sanitize check).

• 3. The module and the edits. Copy the skeleton, then make edits L1, C1 to C3 and M1 to M3:

  cp docs/plans/M5b/files/crates/bxctl/src/sanitize.rs crates/bxctl/src/
  

  cargo check -p bxctl: no error, only the warnings "Facts you need" describes. The command of step 2 now compiles; sanitize_report has 10 failing tests and sanitize_check 8. Every sanitize_report failure is not yet implemented; every sanitize_check failure is an exit status of 101 (the binary panicked with not yet implemented).

• 4. Fill the five functions of sanitize.rs, one at a time, in this order, each from its comment, and run cargo check -p bxctl after each: parse_sanitize, code_point, token, prefix, tag_count. Change nothing else in the file.

• 5. See them pass. The command of step 2: sanitize_check 8 passed, sanitize_report 10 passed. Then cargo test -p bxctl: every suite passes.

• 6. Check. git diff --stat -- crates/bxcore '*Cargo.toml' prints nothing.

• 7. Run the gate. cargo fmt --all, then make gate, with a shell timeout of 600000 ms. Expected last line: gate: ok, with 1099 tests: 1081 before this task, plus 8 in sanitize_check.rs and 10 in sanitize_report.rs.

• 8. Log and commit. git status --short lists crates/bxctl/src/lib.rs, crates/bxctl/src/cli.rs and crates/bxctl/src/main.rs as modified, and crates/bxctl/src/sanitize.rs and the two test files as new, besides your log. Stage with git add crates/bxctl docs/implementer-log.md Cargo.lock, then commit as AGENTS.md says. Cargo.lock does not change in this task; the git add is harmless.

Done when

• make gate prints gate: ok with 1099 tests.
• git diff HEAD~1 --stat lists only crates/bxctl/src/lib.rs, crates/bxctl/src/cli.rs, crates/bxctl/src/main.rs, crates/bxctl/src/sanitize.rs, the two new test files and docs/implementer-log.md.

Stop and report if

• A "before" block of L1, C1 to C3 or M1 to M3 is not in its file exactly as shown, or is there more than once.
• Step 2 or 3 shows a different result: other errors, other failing tests, or other counts.
• cargo check -p bxctl reports an error after a body you filled, and the comment does not tell you how to fix it. Quote the error and the comment, and stop.
• A comment in the skeleton does not fit the code under it. Quote it and stop.
• After step 5 a test fails although each function follows its comment. Do not open the test to find out why, and do not change a test or the given code of the skeleton: stop, and give the failing tests' names and their messages.
• The fix seems to need a change anywhere else: in bxcore, another file of bxctl, or a Cargo.toml. It does not; stop and say why you think so.
• A test outside bxctl fails in the gate: check once whether it fails without your change, then stop and report both results.
• The gate passes with a count other than 1099.