# M5b task 04: `sanitize::refuse`, what a memory write may never hold

**Branch:** `m5b` (run `git switch m5b`; `git status --short` must be empty, otherwise stop)
**Commit subject:** `bxcore: sanitize::refuse, what a memory write may never hold`

## Read this first: the markup strings and the fake keys

One rule of this task is about your own model's chat-template tags, which task 03 listed. **If you
spell one out, even in your thinking, it ends your reasoning or your turn.** So:

- Nothing in this task needs a tag's text. The tag rule calls task 03's `next_tag`, which already
  knows them. Call them "the listed tags" if you must refer to them.
- **Do not open** the given test files (`sanitize_refuse.rs`, `sanitize_refuse_corpus.rs`,
  `support/keys.rs`), and do not work through their cases in your reasoning. You copy them; you
  never need to read them. Their failure messages name a case and a rule, never a text.
- The test files build every fake key from pieces, so that no file in the repository holds one
  whole. Do not write a whole key-shaped string anywhere either.

## Goal

`memoryd` checks every memory write with M5a's `check.rs`: control characters, invisible
characters, chat-template markup and six instruction phrases. M5b moves that check into `bxcore`
as `sanitize::refuse` and adds to it (M5b spec, section 6.1): private-use characters,
noncharacters, and secrets known by their prefix (GitHub, GitLab, `sk-` keys, Slack, AWS, private
keys). The phrases are not refused any more; a later task makes them a flag.

`refuse(text)` gives the **first rule in the table below** that `text` breaks, or `None`. "First"
is the table's order, not the position in the text: a text holding a tag and, later, a control
character is refused by `control`.

| # | Rule name | Refuses | Sentence (after `memory refused this entry: it contains `) |
|---|---|---|---|
| 1 | `control` | a control character other than `\n` and `\t` | `a control character` |
| 2 | `invisible` | M5a's default-ignorable table and its three extra characters | `an invisible or direction-changing character` |
| 3 | `private-use` | U+E000 to U+F8FF, U+F0000 to U+FFFFD, U+100000 to U+10FFFD | `a private-use character` |
| 4 | `noncharacter` | U+FDD0 to U+FDEF, and the last two code points of every plane | `a Unicode noncharacter` |
| 5 | `secret-github` | `ghp_`, `gho_`, `ghu_`, `ghs_`, `ghr_`, `github_pat_` tokens | `what looks like a GitHub token` |
| 6 | `secret-gitlab` | `glpat-` tokens | `what looks like a GitLab token` |
| 7 | `secret-sk` | `sk-` keys | `what looks like an API key` |
| 8 | `secret-slack` | `xoxb-`, `xoxa-`, `xoxp-`, `xoxr-`, `xoxs-` tokens | `what looks like a Slack token` |
| 9 | `secret-aws` | `AKIA` or `ASIA` access key ids | `what looks like an AWS access key` |
| 10 | `secret-private-key` | a line that starts a private-key block | `a private key` |
| 11 | `tag` | any listed tag, matched exactly as `neutralise` matches it | `chat-template markup` |

The table, the sentences and every list of characters and prefixes are **given, filled in**, in the
skeleton. You write eight small functions.

**`memoryd` does not change in this task.** A later task switches it to `refuse` and deletes
`check.rs`. Until then the character tables exist twice, in `crates/memoryd/src/check.rs` and in
the new file. That is expected: do not touch `crates/memoryd`.

| Name | What it is |
|---|---|
| `bxcore::sanitize::Refusal` | `pub struct Refusal { pub rule: &'static str, pub sentence: &'static str, pub at: usize }` (given) |
| `bxcore::sanitize::refuse` | `pub fn refuse(text: &str) -> Option<Refusal>` (given) |
| `first_char` (private) | `fn first_char(text: &str, rule: fn(char) -> bool) -> Option<usize>`: **you write it** |
| `control` (private) | `fn control(c: char) -> bool`: **you write it** |
| `private_use` (private) | `fn private_use(c: char) -> bool`: **you write it** |
| `noncharacter` (private) | `fn noncharacter(c: char) -> bool`: **you write it** |
| `starts_word` (private) | `fn starts_word(before: &str) -> bool`: **you write it** |
| `key_run` (private) | `fn key_run(after: &str, secret: &Secret) -> bool`: **you write it** |
| `private_key` (private) | `fn private_key(text: &str) -> Option<usize>`: **you write it** |
| `listed_tag` (private) | `fn listed_tag(text: &str) -> Option<usize>`: **you write it** |

## Files

- Copy, from `docs/plans/M5b/files/` (the same path under it):
  - the skeleton `crates/bxcore/src/sanitize/refuse.rs` (you fill eight functions in it)
  - tests: `crates/bxcore/tests/sanitize_refuse.rs`,
    `crates/bxcore/tests/sanitize_refuse_corpus.rs`, `crates/bxcore/tests/support/keys.rs`
- Modify: `crates/bxcore/src/sanitize/mod.rs` (edit E1), `crates/bxcore/src/sanitize/tags.rs`
  (edit E2, one line), `docs/implementer-log.md`
- Do not touch: `crates/memoryd`, task 03's test files, anything else in `tags.rs`.

## Facts you need

- **`at`** in a `Refusal` is a **byte** offset into the text (what `char_indices` and
  `match_indices` give), not a count of characters: where the character, the prefix, the header
  line's dashes or the tag starts.
- **Characters.** `char::is_control` is true for U+0000 to U+001F and U+007F to U+009F only.
  `u32::from(c)` gives a `char`'s code point. `code & 0xFFFE == 0xFFFE` is true for exactly the
  last two code points of each plane (U+FFFE, U+FFFF, U+1FFFE, …, U+10FFFF): every `char` is at
  most U+10FFFF. The private-use ranges of planes 15 and 16 stop two short of the plane's end, so
  those last two are noncharacters, not private use (a test checks which rule names them).
- **A secret** is one of its `Secret`'s prefixes at the start of a word, followed by a run of key
  characters (`secret.key`) whose length is between `secret.min` and `secret.max`, inclusive.
  "The start of a word": the character before the prefix is not an ASCII letter or digit, `_` or
  `-`, or there is none. So `disk-…` and `my_sk-…` are not `sk-` keys. For `sk-`, `github_pat_`
  and one `ghs_` form, `secret.mixed` is true: the run must also hold an ASCII upper-case letter
  and an ASCII digit, because names like `sk-folding-cube-spinner` start the same way. `find_secret`
  (given) walks the prefixes and calls your `starts_word` and `key_run`.
- **A private key**: a line that, white space aside, starts `-----BEGIN` and ends
  `PRIVATE KEY-----`. A public key or a certificate ends differently and passes; so does the header
  in the middle of a sentence.
- **The tag rule** is task 03's matching, reused: `next_tag` in `tags.rs` finds the first listed
  tag exactly as `neutralise` does. Edit E2 lets `refuse.rs` call it. This is narrower than M5a's
  markup rule, which matched pieces of tags in any case: an upper-case tag is not the model's token
  (measured in task 03), and a tag split by a zero-width character is refused by `invisible`.
- **Given code you must not change**: the `RULES` table (its closures turn into `fn` pointers), the
  `Secret` tables, `find_secret` (its `if let … &&` chain is Rust 2024; it compiles on this
  toolchain), `invisible`, `refuse` and the key-character functions.
- The calls you need, all in the standard library:

  | Call | Signature, and what it gives |
  |---|---|
  | `str::char_indices` | `text.char_indices()`: an iterator of `(usize, char)`, each character with its byte offset |
  | `Iterator::find`, `Option::map` | `.find(\|&(_, c)\| rule(c))` gives `Option<(usize, char)>`; `.map(\|(at, _)\| at)` |
  | `char::is_control` | `c.is_control() -> bool` |
  | `u32::from` | `u32::from(c)`: the code point of a `char` |
  | `slice::iter`, `Iterator::any` | `PRIVATE_USE.iter().any(\|&(low, high)\| (low..=high).contains(&code))` |
  | `RangeInclusive::contains` | `(0xFDD0..=0xFDEF).contains(&code) -> bool` |
  | `str::chars`, `DoubleEndedIterator::next_back` | `before.chars().next_back() -> Option<char>`: the last character |
  | `Option::is_some_and` | `opt.is_some_and(\|c\| …) -> bool`: false for `None` |
  | `char::is_ascii_alphanumeric`, `is_ascii_uppercase`, `is_ascii_digit` | `c.is_ascii_…() -> bool` |
  | `Iterator::take_while` | `after.chars().take_while(\|&c\| (secret.key)(c))`: the run; `secret.key` is a `fn(char) -> bool` field, called with parentheses around it |
  | `str::match_indices` | `text.match_indices("-----BEGIN")`: an iterator of `(usize, &str)` |
  | `str::split_at_checked` | `fn split_at_checked(&self, mid: usize) -> Option<(&str, &str)>` |
  | `str::rsplit`, `str::split`, `Iterator::next` | `before.rsplit('\n').next()`: the last line piece; `from.split('\n').next()`: the first |
  | `Option::unwrap_or_default` | `Option<&str>` to `&str`, `""` for `None` (not `unwrap`: library code never panics) |
  | `str::trim`, `str::trim_end`, `str::is_empty`, `str::ends_with` | white space off both ends, off the end; `s.ends_with("PRIVATE KEY-----")` |
  | `next_tag` (in `tags.rs`) | `fn next_tag(text: &str) -> Option<(&str, &'static Tag, &str)>`: before, tag, after |

## The edits, exactly

**E1.** `crates/bxcore/src/sanitize/mod.rs` today is exactly:

```rust
//! The sanitizer (M5b spec, section 6): what is done to text from outside before it reaches the
//! model or memory. Neutralising the model's chat-template tags (6.2) is in `tags`.

mod tags;

pub use tags::{TAGS, Tag, neutralise};
```

Replace the whole file with exactly this text (one newline at the end):

```rust
//! The sanitizer (M5b spec, section 6): what is done to text from outside before it reaches the
//! model or memory. Neutralising the model's chat-template tags (6.2) is in `tags`; refusing what a
//! memory write may never hold (6.1) is in `refuse`.

mod refuse;
mod tags;

pub use refuse::{Refusal, refuse};
pub use tags::{TAGS, Tag, neutralise};
```

**E2.** In `crates/bxcore/src/sanitize/tags.rs`, the line

```rust
fn next_tag(text: &str) -> Option<(&str, &'static Tag, &str)> {
```

becomes

```rust
pub(super) fn next_tag(text: &str) -> Option<(&str, &'static Tag, &str)> {
```

Nothing else in `tags.rs` changes: not its comment, not its body.

## Steps

- [ ] **1. Copy the given tests.** From the repository root, four commands:

  ```sh
  mkdir -p crates/bxcore/tests/support
  cp docs/plans/M5b/files/crates/bxcore/tests/sanitize_refuse.rs crates/bxcore/tests/
  cp docs/plans/M5b/files/crates/bxcore/tests/sanitize_refuse_corpus.rs crates/bxcore/tests/
  cp docs/plans/M5b/files/crates/bxcore/tests/support/keys.rs crates/bxcore/tests/support/
  ```

- [ ] **2. See them fail.** `cargo test -p bxcore --test sanitize_refuse --test
  sanitize_refuse_corpus --no-fail-fast`: neither compiles. Every error is `E0432`, unresolved
  import `bxcore::sanitize::refuse` (two for `sanitize_refuse`, one for `sanitize_refuse_corpus`).
- [ ] **3. The module.** Copy the skeleton, then make edits E1 and E2:

  ```sh
  cp docs/plans/M5b/files/crates/bxcore/src/sanitize/refuse.rs crates/bxcore/src/sanitize/
  ```

  The command of step 2 now compiles, with warnings about unused variables, an unused import,
  `PRIVATE_USE` and the fields of `Secret`: they are the unfilled functions, and go as you fill
  them. In `sanitize_refuse` 14 tests fail, in `sanitize_refuse_corpus` 2; every failure is
  `not yet implemented`.
- [ ] **4. Fill the eight functions, one at a time, in this order**, each from its comment, and
  run `cargo check -p bxcore` after each: `control`, `first_char`, `private_use`, `noncharacter`,
  `listed_tag`, `starts_word`, `key_run`, `private_key`. Change nothing else in `refuse.rs`.
- [ ] **5. See them pass.** The command of step 2: `sanitize_refuse` 14 passed,
  `sanitize_refuse_corpus` 2 passed. Also `cargo test -p bxcore --test sanitize_tags --test
  sanitize_corpus`: 15 and 3 passed, as after task 03.
- [ ] **6. Run the gate.** `cargo fmt --all`, then `make gate`, with a shell timeout of 600000 ms.
  Expected last line: `gate: ok`, with 997 tests: 981 before this task, plus 14 in
  `sanitize_refuse.rs` and 2 in `sanitize_refuse_corpus.rs`.
- [ ] **7. Log and commit.** `git status --short` lists `crates/bxcore/src/sanitize/mod.rs` and
  `tags.rs` as modified and four new files (`refuse.rs`, the two test files,
  `tests/support/keys.rs`) besides your log. Stage with
  `git add crates/bxcore docs/implementer-log.md Cargo.lock`, then commit as `AGENTS.md` says.
  `Cargo.lock` does not change in this task; the `git add` is harmless.

## Done when

- `make gate` prints `gate: ok` with 997 tests.
- `git diff HEAD~1 --stat` lists only `crates/bxcore/src/sanitize/mod.rs`,
  `crates/bxcore/src/sanitize/tags.rs`, the four new files under `crates/bxcore/`, and
  `docs/implementer-log.md`.

## Stop and report if

- The "before" text of E1 is not exactly the file's text, or the line of E2 is not in `tags.rs`
  exactly as shown, or is there more than once.
- Step 2 or 3 shows a different result: another error, or other counts.
- After step 5 a test fails although the eight functions follow their comments. Do not open the
  test to find out why, and do not change a test, a table or a sentence: stop, and give the failing
  tests' names and their messages.
- A test outside `bxcore` fails in the gate: check once whether it fails without your change, then
  stop and report both results.
- The gate passes with a count other than 997.
- A comment in the skeleton does not fit the code under it. Quote it and stop.