boxmaker/docs/plans/M5b/04-sanitize-refuse.md

M5b task 04: sanitize::refuse, what a memory write may never hold

Branch: m5b (run git switch m5b; git status --short must be empty, otherwise stop) Commit subject: bxcore: sanitize::refuse, what a memory write may never hold

Read this first: the markup strings and the fake keys

One rule of this task is about your own model's chat-template tags, which task 03 listed. If you spell one out, even in your thinking, it ends your reasoning or your turn. So:

• Nothing in this task needs a tag's text. The tag rule calls task 03's next_tag, which already knows them. Call them "the listed tags" if you must refer to them.
• Do not open the given test files (sanitize_refuse.rs, sanitize_refuse_corpus.rs, support/keys.rs), and do not work through their cases in your reasoning. You copy them; you never need to read them. Their failure messages name a case and a rule, never a text.
• The test files build every fake key from pieces, so that no file in the repository holds one whole. Do not write a whole key-shaped string anywhere either.

Goal

memoryd checks every memory write with M5a's check.rs: control characters, invisible characters, chat-template markup and six instruction phrases. M5b moves that check into bxcore as sanitize::refuse and adds to it (M5b spec, section 6.1): private-use characters, noncharacters, and secrets known by their prefix (GitHub, GitLab, sk- keys, Slack, AWS, private keys). The phrases are not refused any more; a later task makes them a flag.

refuse(text) gives the first rule in the table below that text breaks, or None. "First" is the table's order, not the position in the text: a text holding a tag and, later, a control character is refused by control.

#	Rule name	Refuses	Sentence (after memory refused this entry: it contains )
1	control	a control character other than \n and \t	a control character
2	invisible	M5a's default-ignorable table and its three extra characters	an invisible or direction-changing character
3	private-use	U+E000 to U+F8FF, U+F0000 to U+FFFFD, U+100000 to U+10FFFD	a private-use character
4	noncharacter	U+FDD0 to U+FDEF, and the last two code points of every plane	a Unicode noncharacter
5	secret-github	ghp_, gho_, ghu_, ghs_, ghr_, github_pat_ tokens	what looks like a GitHub token
6	secret-gitlab	glpat- tokens	what looks like a GitLab token
7	secret-sk	sk- keys	what looks like an API key
8	secret-slack	xoxb-, xoxa-, xoxp-, xoxr-, xoxs- tokens	what looks like a Slack token
9	secret-aws	AKIA or ASIA access key ids	what looks like an AWS access key
10	secret-private-key	a line that starts a private-key block: PEM, OpenSSH or OpenPGP	a private key
11	tag	any listed tag, matched exactly as neutralise matches it	chat-template markup

The table, the sentences and every list of characters and prefixes are given, filled in, in the skeleton. You write eight small functions.

memoryd does not change in this task. A later task switches it to refuse and deletes check.rs. Until then the character tables exist twice, in crates/memoryd/src/check.rs and in the new file. That is expected: do not touch crates/memoryd.

Name	What it is
bxcore::sanitize::Refusal	pub struct Refusal { pub rule: &'static str, pub sentence: &'static str, pub at: usize } (given)
bxcore::sanitize::refuse	pub fn refuse(text: &str) -> Option<Refusal> (given)
first_char (private)	fn first_char(text: &str, rule: fn(char) -> bool) -> Option<usize>: you write it
control (private)	fn control(c: char) -> bool: you write it
private_use (private)	fn private_use(c: char) -> bool: you write it
noncharacter (private)	fn noncharacter(c: char) -> bool: you write it
starts_word (private)	fn starts_word(before: &str) -> bool: you write it
key_run (private)	fn key_run(after: &str, secret: &Secret) -> bool: you write it
private_key (private)	fn private_key(text: &str) -> Option<usize>: you write it
listed_tag (private)	fn listed_tag(text: &str) -> Option<usize>: you write it

Files

• Copy, from docs/plans/M5b/files/ (the same path under it):
  • the skeleton crates/bxcore/src/sanitize/refuse.rs (you fill eight functions in it)
  • tests: crates/bxcore/tests/sanitize_refuse.rs, crates/bxcore/tests/sanitize_refuse_corpus.rs, crates/bxcore/tests/support/keys.rs
• Modify: crates/bxcore/src/sanitize/mod.rs (edit E1), crates/bxcore/src/sanitize/tags.rs (edit E2, one line), docs/implementer-log.md
• Do not touch: crates/memoryd, task 03's test files, anything else in tags.rs.

Facts you need

• at in a Refusal is a byte offset into the text (what char_indices and match_indices give), not a count of characters: where the character, the prefix, the header line's dashes or the tag starts.
• Characters. char::is_control is true for U+0000 to U+001F and U+007F to U+009F only. u32::from(c) gives a char's code point. code & 0xFFFE == 0xFFFE is true for exactly the last two code points of each plane (U+FFFE, U+FFFF, U+1FFFE, …, U+10FFFF): every char is at most U+10FFFF. The private-use ranges of planes 15 and 16 stop two short of the plane's end, so those last two are noncharacters, not private use (a test checks which rule names them).
• A secret is one of its Secret's prefixes at the start of a word, followed by a run of key characters (secret.key) whose length is between secret.min and secret.max, inclusive. "The start of a word": the character before the prefix is not an ASCII letter or digit, _ or -, or there is none. So disk-… and my_sk-… are not sk- keys. For sk-, github_pat_ and one ghs_ form, secret.mixed is true: the run must also hold an ASCII upper-case letter and an ASCII digit, because names like sk-folding-cube-spinner start the same way. find_secret (given) walks the prefixes and calls your starts_word and key_run.
• A private key: a line that, white space aside, starts -----BEGIN and ends with one of PRIVATE_KEY_ENDS (given): PRIVATE KEY----- (PEM and OpenSSH) or PRIVATE KEY BLOCK----- (OpenPGP's armor header, RFC 9580 section 6.2.1). A public key, a PGP public key block, a signature or a certificate ends differently and passes; so does a header in the middle of a sentence.
• The tag rule is task 03's matching, reused: next_tag in tags.rs finds the first listed tag exactly as neutralise does. Edit E2 lets refuse.rs call it. This is narrower than M5a's markup rule, which matched pieces of tags in any case: an upper-case tag is not the model's token (measured in task 03), and a tag split by a zero-width character is refused by invisible.
• Given code you must not change: the RULES table (its closures turn into fn pointers), the Secret tables, PRIVATE_KEY_ENDS, find_secret (its if let … && chain is Rust 2024; it compiles on this toolchain), invisible, refuse and the key-character functions.
• The calls you need, all in the standard library:

  Call	Signature, and what it gives
  str::char_indices	text.char_indices(): an iterator of (usize, char), each character with its byte offset
  Iterator::find, Option::map	.find(|&(_, c)| rule(c)) gives Option<(usize, char)>; .map(|(at, _)| at)
  char::is_control	c.is_control() -> bool
  u32::from	u32::from(c): the code point of a char
  slice::iter, Iterator::any	PRIVATE_USE.iter().any(|&(low, high)| (low..=high).contains(&code))
  RangeInclusive::contains	(0xFDD0..=0xFDEF).contains(&code) -> bool
  str::chars, DoubleEndedIterator::next_back	before.chars().next_back() -> Option<char>: the last character
  Option::is_some_and	opt.is_some_and(|c| …) -> bool: false for None
  char::is_ascii_alphanumeric, is_ascii_uppercase, is_ascii_digit	c.is_ascii_…() -> bool
  Iterator::take_while	after.chars().take_while(|&c| (secret.key)(c)): the run; secret.key is a fn(char) -> bool field, called with parentheses around it
  str::match_indices	text.match_indices("-----BEGIN"): an iterator of (usize, &str)
  str::split_at_checked	fn split_at_checked(&self, mid: usize) -> Option<(&str, &str)>
  str::rsplit, str::split, Iterator::next	before.rsplit('\n').next(): the last line piece; from.split('\n').next(): the first
  Option::unwrap_or_default	Option<&str> to &str, "" for None (not unwrap: library code never panics)
  str::trim, str::trim_end, str::is_empty, str::ends_with	white space off both ends, off the end; line.ends_with(end)
  next_tag (in tags.rs)	fn next_tag(text: &str) -> Option<(&str, &'static Tag, &str)>: before, tag, after

The edits, exactly

E1. crates/bxcore/src/sanitize/mod.rs today is exactly:

//! The sanitizer (M5b spec, section 6): what is done to text from outside before it reaches the
//! model or memory. Neutralising the model's chat-template tags (6.2) is in `tags`.

mod tags;

pub use tags::{TAGS, Tag, neutralise};

Replace the whole file with exactly this text (one newline at the end):

//! The sanitizer (M5b spec, section 6): what is done to text from outside before it reaches the
//! model or memory. Neutralising the model's chat-template tags (6.2) is in `tags`; refusing what a
//! memory write may never hold (6.1) is in `refuse`.

mod refuse;
mod tags;

pub use refuse::{Refusal, refuse};
pub use tags::{TAGS, Tag, neutralise};

E2. In crates/bxcore/src/sanitize/tags.rs, the line

fn next_tag(text: &str) -> Option<(&str, &'static Tag, &str)> {

becomes

pub(super) fn next_tag(text: &str) -> Option<(&str, &'static Tag, &str)> {

Nothing else in tags.rs changes: not its comment, not its body.

Steps

• 1. Copy the given tests. From the repository root, four commands:

  mkdir -p crates/bxcore/tests/support
  cp docs/plans/M5b/files/crates/bxcore/tests/sanitize_refuse.rs crates/bxcore/tests/
  cp docs/plans/M5b/files/crates/bxcore/tests/sanitize_refuse_corpus.rs crates/bxcore/tests/
  cp docs/plans/M5b/files/crates/bxcore/tests/support/keys.rs crates/bxcore/tests/support/
  

• 2. See them fail. cargo test -p bxcore --test sanitize_refuse --test sanitize_refuse_corpus --no-fail-fast: neither compiles. Every error is E0432, unresolved import bxcore::sanitize::refuse (two for sanitize_refuse, one for sanitize_refuse_corpus).

• 3. The module. Copy the skeleton, then make edits E1 and E2:

  cp docs/plans/M5b/files/crates/bxcore/src/sanitize/refuse.rs crates/bxcore/src/sanitize/
  

  The command of step 2 now compiles, with warnings about unused variables, an unused import, PRIVATE_USE, PRIVATE_KEY_ENDS and the fields of Secret: they are the unfilled functions, and go as you fill them. In sanitize_refuse 14 tests fail, in sanitize_refuse_corpus 2; every failure is not yet implemented.

• 4. Fill the eight functions, one at a time, in this order, each from its comment, and run cargo check -p bxcore after each: control, first_char, private_use, noncharacter, listed_tag, starts_word, key_run, private_key. Change nothing else in refuse.rs.

• 5. See them pass. The command of step 2: sanitize_refuse 14 passed, sanitize_refuse_corpus 2 passed. Also cargo test -p bxcore --test sanitize_tags --test sanitize_corpus: 15 and 3 passed, as after task 03.

• 6. Run the gate. cargo fmt --all, then make gate, with a shell timeout of 600000 ms. Expected last line: gate: ok, with 997 tests: 981 before this task, plus 14 in sanitize_refuse.rs and 2 in sanitize_refuse_corpus.rs.

• 7. Log and commit. git status --short lists crates/bxcore/src/sanitize/mod.rs and tags.rs as modified and four new files (refuse.rs, the two test files, tests/support/keys.rs) besides your log. Stage with git add crates/bxcore docs/implementer-log.md Cargo.lock, then commit as AGENTS.md says. Cargo.lock does not change in this task; the git add is harmless.

Done when

• make gate prints gate: ok with 997 tests.
• git diff HEAD~1 --stat lists only crates/bxcore/src/sanitize/mod.rs, crates/bxcore/src/sanitize/tags.rs, the four new files under crates/bxcore/, and docs/implementer-log.md.

Stop and report if

• The "before" text of E1 is not exactly the file's text, or the line of E2 is not in tags.rs exactly as shown, or is there more than once.
• Step 2 or 3 shows a different result: another error, or other counts.
• After step 5 a test fails although the eight functions follow their comments. Do not open the test to find out why, and do not change a test, a table or a sentence: stop, and give the failing tests' names and their messages.
• A test outside bxcore fails in the gate: check once whether it fails without your change, then stop and report both results.
• The gate passes with a count other than 997.
• A comment in the skeleton does not fit the code under it. Quote it and stop.