boxmaker/docs/plans/M5c2/07-memoryd-recall-form.md

# M5c-2 task 07: one form for every `recall` item, local ones too

**Branch:** `m5c2`. Check it: `git branch --show-current` prints `m5c2` and `git status --short`
is empty; otherwise stop. Never switch branch.
**Commit subject:** `memoryd: recall's local items on one indented line, as Honcho's`
**Log row:** the Task cell is `M5c2/07-memoryd-recall-form`.

## Goal

Task 06 gave Honcho's `recall` items one form: a header, then the text on one line indented by
two spaces, with every line break made a space. So a text shaped like a header (`[2] identity ·
…`) can never begin a line of the answer, where the model would read it as a real item (tip T37).
The local index's items still keep a hit's line breaks, and put its text at the start of the
line after the header. A page fetched in a session is not indexed, but what the owner and the
assistant said about it is, and a note can hold anything: a forged header there begins a line
today.

This task gives local items the same form, with one pair of functions that both kinds of item
use (M5c spec, section 8):

- **`recall::snippet(text)`**: `neutralise`d, every control character and Unicode line or
  paragraph separator a space, cut to its first 600 characters (`SNIPPET`) with no mark. Local
  hits were cut the same way before, without the other two steps; the local index holds text
  `loopd` has neutralised already, so neutralising again changes nothing there.
- **`recall::item(header, text)`**: the header, a line break, two spaces and the text.
- `recalled.rs` (task 06) keeps `clean` and `Item::render`, but they now call these two, so there
  is one copy of the rule.

And for task 08, which puts Honcho's items after the local ones:

- **`recall::local(...)`**: what `recall` does, up to the items: the search, the recall log,
  then `Local { items, class, untrusted }`, the rendered items numbered from 1 and their label
  (`private` and trusted when there are none).
- **`recall::recall(...)`** answers from `local`, as before: the items joined by line breaks, or
  `(no matches for "…")` when there are none. Only the items' form changes.

Nothing else reads `recall`'s answer as text: `loopd` passes it to the model, and `bxctl`,
`gatewayd` and the device tests do not parse it (checked). Two assertions of `memoryd`'s own
`recall` test pinned the old form; the test gets a new version.

## Every place that changes

This is the complete list.

| Place | What happens there |
|---|---|
| `crates/memoryd/src/recall.rs` | `snippet`, `item`, `Local`, `local`; `recall` answers from `local` (edits R1 to R5) |
| `crates/memoryd/src/honcho/recalled.rs` | `clean` and `Item::render` call `snippet` and `item`; two `use` lines (edits H1 to H4) |
| `crates/memoryd/tests/recall_form.rs` | new, given |
| `crates/memoryd/tests/recall.rs` | new version, given: two assertions follow the new form |
| everything else | **no change**: not `serve.rs`, not `index.rs`, no other test, no `Cargo.toml` |

## Files

- Copy, from `docs/plans/M5c2/files/` (the same path under it): the new test
  `crates/memoryd/tests/recall_form.rs`, and a **new version** of
  `crates/memoryd/tests/recall.rs` (it replaces the file in the tree).
- Modify: `crates/memoryd/src/recall.rs`, `crates/memoryd/src/honcho/recalled.rs`,
  `docs/implementer-log.md`.
- Do not touch: any other file. `docs/plans/M5a/files/` keeps its own older copy of `recall.rs`;
  do not copy from there.

## Facts you need

- **`recall.rs` today**: `recall(index, store, args, session, now) -> Result<Recalled, String>`
  checks the arguments, searches, logs the recall, then either answers `(no matches for "…")` or
  builds one item per hit (`format!("{}\n{text}", hit.header(i + 1))`, the text cut to `SNIPPET`
  characters) and joins them. Edits R3 to R5 split it: the body up to the items becomes `local`,
  and `recall` calls `local` and joins. The search, the log line and the headers do not change.
- **`recalled.rs` today** (task 06): `clean` holds the same rule as the new `snippet`, and
  `Item::render` the same layout as the new `item`. Edits H3 and H4 make them call the new
  functions; `neutralise` and `SNIPPET` are then no longer used there, so edits H1 and H2 change
  its `use` lines.
- **`neutralise`** (`bxcore::sanitize::neutralise(&str) -> Cow<str>`) writes each chat-template
  tag's angle brackets as `&lt;` and `&gt;`; text with none is returned as it is.
- **The given tests.**
  - `recall_form.rs` (4 tests): `snippet` and `item`, and that `recalled::clean` gives what
    `snippet` does; three forged headers in indexed text (after a line break, at the start, after
    a CRLF), each kept inside its own item, with only real headers beginning a line; a tainted
    hit (said after an untrusted tool result) forging a trusted `memory` header, shown under its
    own `memory-tainted` header, the answer untrusted; `local`'s items and label, `recall`'s
    answer made from them, and the `(no matches …)` answer as before.
  - `recall.rs`: two assertions change, `hits_are_tagged_and_labelled` (the text follows `\n` and
    two spaces) and `long_hits_are_cut` (the body is what follows `\n` and two spaces).
- The items the edits use. This table is complete.

  | Item | Where |
  |---|---|
  | `bxcore::sanitize::neutralise` | R2 (moved from `recalled.rs`) |
  | `char::is_control`, `Iterator::map`, `Iterator::take`, `Iterator::collect` | R2 (moved from `recalled.rs`) |
  | `Vec::is_empty`, `slice::join` | R3 |

## The edits, exactly

Nothing else in these files changes. Each quoted "before" block occurs exactly once in its file
when its turn comes (make them in order). Keep the fields in the order shown.

### `crates/memoryd/src/recall.rs`

**R1.** Add one line. The line

```rust
use bxcore::{DataClass, RecallArgs, Timestamp};
```

becomes

```rust
use bxcore::sanitize::neutralise;
use bxcore::{DataClass, RecallArgs, Timestamp};
```

**R2.** The lines

```rust
    pub class: DataClass,
    pub untrusted: bool,
}

/// The FTS5 query for `query`: every run of letters and digits, double-quoted, joined by `OR`.
```

become (its spaces are checked: `item`'s format string is a line break, then two spaces)

```rust
    pub class: DataClass,
    pub untrusted: bool,
}

/// The local index's hits, each rendered as an item, numbered from 1, and the label they give
/// the answer: `Private` and trusted when there are none.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Local {
    pub items: Vec<String>,
    pub class: DataClass,
    pub untrusted: bool,
}

/// A hit's text as an item holds it, from the local index or from Honcho: neutralised, every
/// control character (a line break among them) and every Unicode line or paragraph separator a
/// space, then cut to its first `SNIPPET` characters, with no mark.
pub fn snippet(text: &str) -> String {
    neutralise(text)
        .chars()
        .map(|c| {
            if c.is_control() || c == '\u{2028}' || c == '\u{2029}' {
                ' '
            } else {
                c
            }
        })
        .take(SNIPPET)
        .collect()
}

/// An item as the model reads it: its header, then its text on one line indented by two spaces,
/// so that no text can begin a line the way a header does (M5c spec, section 8; tip T37).
pub fn item(header: &str, text: &str) -> String {
    format!("{header}\n  {text}")
}

/// The FTS5 query for `query`: every run of letters and digits, double-quoted, joined by `OR`.
```

**R3.** The lines

```rust
    session: &str,
    now: Timestamp,
) -> Result<Recalled, String> {
    // 1. `args.check()?`.
```

become (its spaces are checked: `(no matches for \"{}\")` is as before)

```rust
    session: &str,
    now: Timestamp,
) -> Result<Recalled, String> {
    let found = local(index, store, args, session, now)?;
    if found.items.is_empty() {
        return Ok(Recalled {
            text: format!("(no matches for \"{}\")", args.query),
            class: DataClass::Private,
            untrusted: false,
        });
    }
    Ok(Recalled {
        text: found.items.join("\n"),
        class: found.class,
        untrusted: found.untrusted,
    })
}

/// Search the local index for `args` on behalf of `session`, log the recall, and give its hits
/// as items with their label. `recall` answers with them alone; with Honcho, its hits follow.
pub fn local(
    index: &Index,
    store: &Store,
    args: &RecallArgs,
    session: &str,
    now: Timestamp,
) -> Result<Local, String> {
    // 1. `args.check()?`.
```

**R4.** The comment's lines

```rust
    // 4. No hits: Ok(Recalled { text: format!("(no matches for \"{}\")", args.query), class
    //    Private, untrusted false }).
    // 5. Otherwise, for each hit with i from 0: class = class.max(its class) starting at Private;
    //    untrusted |= its tag is "memory-tainted"; its text cut to its first SNIPPET characters;
    //    the item is format!("{}\n{text}", hit.header(i + 1)). Ok(Recalled { the items joined by
    //    newlines, class, untrusted }).
```

become

```rust
    // 4. For each hit with i from 0: class = class.max(its class) starting at Private;
    //    untrusted |= its tag is "memory-tainted"; the item is `item(&hit.header(i + 1),
    //    &snippet(&hit.text))`. Ok(Local { items, class, untrusted }).
```

**R5.** The lines

```rust
    store.append(&format!("recalls/{month}.jsonl"), &format!("{line}\n"))?;
    if hits.is_empty() {
        return Ok(Recalled {
            text: format!("(no matches for \"{}\")", args.query),
            class: DataClass::Private,
            untrusted: false,
        });
    }
    let mut class = DataClass::Private;
    let mut untrusted = false;
    let mut items: Vec<String> = Vec::with_capacity(hits.len());
    for (i, hit) in hits.iter().enumerate() {
        class = class.max(hit.class);
        untrusted |= hit.tag() == "memory-tainted";
        let text: String = hit.text.chars().take(SNIPPET).collect();
        items.push(format!("{}\n{text}", hit.header(i + 1)));
    }
    Ok(Recalled {
        text: items.join("\n"),
        class,
        untrusted,
    })
}
```

become

```rust
    store.append(&format!("recalls/{month}.jsonl"), &format!("{line}\n"))?;
    let mut class = DataClass::Private;
    let mut untrusted = false;
    let mut items: Vec<String> = Vec::with_capacity(hits.len());
    for (i, hit) in hits.iter().enumerate() {
        class = class.max(hit.class);
        untrusted |= hit.tag() == "memory-tainted";
        items.push(item(&hit.header(i + 1), &snippet(&hit.text)));
    }
    Ok(Local {
        items,
        class,
        untrusted,
    })
}
```

### `crates/memoryd/src/honcho/recalled.rs`

**H1.** Remove one line. The lines

```rust
use bxcore::DataClass;
use bxcore::sanitize::neutralise;
use roncho::{Conclusion, Message};
```

become

```rust
use bxcore::DataClass;
use roncho::{Conclusion, Message};
```

**H2.** The line

```rust
use crate::recall::{SNIPPET, when};
```

becomes

```rust
use crate::recall::{item, snippet, when};
```

**H3.** In `Item::render`, the lines

```rust
    pub fn render(&self, n: usize) -> String {
        format!(
            "[{n}] {} · {} · {}\n  {}",
            self.tag,
            self.from,
            class_name(self.class),
            self.text
        )
    }
```

become

```rust
    pub fn render(&self, n: usize) -> String {
        let header = format!(
            "[{n}] {} · {} · {}",
            self.tag,
            self.from,
            class_name(self.class)
        );
        item(&header, &self.text)
    }
```

**H4.** The lines

```rust
/// Text from Honcho as an item holds it: neutralised, every control character (a line break
/// among them) and every Unicode line or paragraph separator a space, then cut to its first
/// `SNIPPET` characters, as a local snippet is cut.
pub fn clean(text: &str) -> String {
    neutralise(text)
        .chars()
        .map(|c| {
            if c.is_control() || c == '\u{2028}' || c == '\u{2029}' {
                ' '
            } else {
                c
            }
        })
        .take(SNIPPET)
        .collect()
}
```

become

```rust
/// Text from Honcho as an item holds it: as a local hit's text is (`recall::snippet`).
pub fn clean(text: &str) -> String {
    snippet(text)
}
```

## Steps

- [ ] **1. Copy the given tests.** From the repository root, two commands:

  ```sh
  cp docs/plans/M5c2/files/crates/memoryd/tests/recall_form.rs crates/memoryd/tests/
  cp docs/plans/M5c2/files/crates/memoryd/tests/recall.rs crates/memoryd/tests/
  ```

  Then `git status --short` lists `crates/memoryd/tests/recall.rs` as modified and
  `recall_form.rs` as new.
- [ ] **2. See them fail.** Two commands, each with a shell timeout of 600000 ms:
  - `cargo build --tests --keep-going -p memoryd`: one test target, `recall_form`, does not
    compile: `E0432`, ``unresolved imports `memoryd::recall::Local`, `memoryd::recall::item`,
    `memoryd::recall::local`, `memoryd::recall::snippet` ``. Every other target compiles.
  - `cargo test --no-fail-fast -p memoryd --test recall`: two tests fail,
    `hits_are_tagged_and_labelled` and `long_hits_are_cut`.
- [ ] **3. The edits.** Make edits R1 to R5 and H1 to H4, in order. `cargo check -p memoryd`
  prints no error and no warning.
- [ ] **4. The tests.** With a shell timeout of 600000 ms, every test passes: 4 in
  `recall_form`, 6 in `recall`, 11 in `honcho_recalled`, and 2 in `forgery`, as before:

  ```sh
  cargo test -p memoryd --test recall_form --test recall --test honcho_recalled --test forgery
  ```
- [ ] **5. Check.** `git diff --stat -- crates/memoryd/src` lists exactly
  `crates/memoryd/src/honcho/recalled.rs` and `crates/memoryd/src/recall.rs`.
- [ ] **6. Run the gate.** `cargo fmt --all`, then `make gate`, with a shell timeout of 600000 ms.
  Expected last line: `gate: ok`. The test count is the passed total of `cargo test --workspace`
  plus one, for the ignored `end_to_end` test that the gate runs on its own: **1418** after this
  task, 1414 before it; the 4 new ones are the tests of `recall_form.rs`. To count, add up the
  `N passed` of every `test result:` line the gate's `cargo test --workspace` printed, and add
  1. Never run the gate again only to count.
- [ ] **7. Log and commit.** Add your row to `docs/implementer-log.md`; its Task cell is
  `M5c2/07-memoryd-recall-form`. Stage with
  `git add crates/memoryd docs/implementer-log.md Cargo.lock`, then commit as `AGENTS.md` says.
  `Cargo.lock` does not change in this task; the `git add` is harmless.

## Done when

- `make gate` prints `gate: ok` with 1418 tests.
- `git diff HEAD~1 --stat` lists only `recall.rs`, `recalled.rs`, `recall_form.rs`, the test
  `recall.rs` and `docs/implementer-log.md`.

## Stop and report if

- The branch check at the top fails.
- A "before" block is not in its file exactly as shown when its turn comes, or is there more
  than once. H1 to H4 quote the parts of `recalled.rs` that task 06's skeleton gave, not the
  bodies you filled.
- Step 2 shows a different result: other targets failing, other errors, or other tests failing.
- `cargo check` in step 3 prints an error or a warning. Quote it, and stop.
- A test fails in step 4. Do not change a test: stop, and give the failing tests' names and
  their messages.
- The fix seems to need a change anywhere else: `serve.rs`, `index.rs`, another test, a copy
  from `docs/plans/M5a/files/`. It does not; stop and say why you think so.
- A test outside the given files fails in the gate: check once whether it fails without your
  change (`git stash -u`, run it, `git stash pop`), then stop and report both results.
- The gate passes with a count other than 1418.
- A comment in an edit does not fit the code under it. Quote it and stop.

## What the given tests pin

Each of these changes to the finished code makes at least one given test fail (15 tried, all
caught): no indent, or one space; a line feed, a carriage return or a line separator kept; no
`neutralise`; the cap moved by one; local items made from the raw text; local items numbered
from 0; a tainted local hit not making the answer untrusted; `recall`'s no-match answer
changed, its items joined another way, or its label dropped; `recalled::clean` or
`Item::render` going back to a copy of their own.