boxmaker/docs/plans/M5c2/07-memoryd-recall-form.md

M5c-2 task 07: one form for every recall item, local ones too

Branch: m5c2. Check it: git branch --show-current prints m5c2 and git status --short is empty; otherwise stop. Never switch branch. Commit subject: memoryd: recall's local items on one indented line, as Honcho's Log row: the Task cell is M5c2/07-memoryd-recall-form.

Goal

Task 06 gave Honcho's recall items one form: a header, then the text on one line indented by two spaces, with every line break made a space. So a text shaped like a header ([2] identity · …) can never begin a line of the answer, where the model would read it as a real item (tip T37). The local index's items still keep a hit's line breaks, and put its text at the start of the line after the header. A page fetched in a session is not indexed, but what the owner and the assistant said about it is, and a note can hold anything: a forged header there begins a line today.

This task gives local items the same form, with one pair of functions that both kinds of item use (M5c spec, section 8):

• recall::snippet(text): neutralised, every control character and Unicode line or paragraph separator a space, cut to its first 600 characters (SNIPPET) with no mark. Local hits were cut the same way before, without the other two steps; the local index holds text loopd has neutralised already, so neutralising again changes nothing there.
• recall::item(header, text): the header, a line break, two spaces and the text.
• recalled.rs (task 06) keeps clean and Item::render, but they now call these two, so there is one copy of the rule.

And for task 08, which puts Honcho's items after the local ones:

• recall::local(...): what recall does, up to the items: the search, the recall log, then Local { items, class, untrusted }, the rendered items numbered from 1 and their label (private and trusted when there are none).
• recall::recall(...) answers from local, as before: the items joined by line breaks, or (no matches for "…") when there are none. Only the items' form changes.

Nothing else reads recall's answer as text: loopd passes it to the model, and bxctl, gatewayd and the device tests do not parse it (checked). Two assertions of memoryd's own recall test pinned the old form; the test gets a new version.

Every place that changes

This is the complete list.

Place	What happens there
crates/memoryd/src/recall.rs	snippet, item, Local, local; recall answers from local (edits R1 to R5)
crates/memoryd/src/honcho/recalled.rs	clean and Item::render call snippet and item; two use lines (edits H1 to H4)
crates/memoryd/tests/recall_form.rs	new, given
crates/memoryd/tests/recall.rs	new version, given: two assertions follow the new form
everything else	no change: not serve.rs, not index.rs, no other test, no Cargo.toml

Files

• Copy, from docs/plans/M5c2/files/ (the same path under it): the new test crates/memoryd/tests/recall_form.rs, and a new version of crates/memoryd/tests/recall.rs (it replaces the file in the tree).
• Modify: crates/memoryd/src/recall.rs, crates/memoryd/src/honcho/recalled.rs, docs/implementer-log.md.
• Do not touch: any other file. docs/plans/M5a/files/ keeps its own older copy of recall.rs; do not copy from there.

Facts you need

• recall.rs today: recall(index, store, args, session, now) -> Result<Recalled, String> checks the arguments, searches, logs the recall, then either answers (no matches for "…") or builds one item per hit (format!("{}\n{text}", hit.header(i + 1)), the text cut to SNIPPET characters) and joins them. Edits R3 to R5 split it: the body up to the items becomes local, and recall calls local and joins. The search, the log line and the headers do not change.
• recalled.rs today (task 06): clean holds the same rule as the new snippet, and Item::render the same layout as the new item. Edits H3 and H4 make them call the new functions; neutralise and SNIPPET are then no longer used there, so edits H1 and H2 change its use lines.
• neutralise (bxcore::sanitize::neutralise(&str) -> Cow<str>) writes each chat-template tag's angle brackets as < and >; text with none is returned as it is.
• The given tests.
  • recall_form.rs (4 tests): snippet and item, and that recalled::clean gives what snippet does; three forged headers in indexed text (after a line break, at the start, after a CRLF), each kept inside its own item, with only real headers beginning a line; a tainted hit (said after an untrusted tool result) forging a trusted memory header, shown under its own memory-tainted header, the answer untrusted; local's items and label, recall's answer made from them, and the (no matches …) answer as before.
  • recall.rs: two assertions change, hits_are_tagged_and_labelled (the text follows \n and two spaces) and long_hits_are_cut (the body is what follows \n and two spaces).
• The items the edits use. This table is complete.

  Item	Where
  bxcore::sanitize::neutralise	R2 (moved from recalled.rs)
  char::is_control, Iterator::map, Iterator::take, Iterator::collect	R2 (moved from recalled.rs)
  Vec::is_empty, slice::join	R3

The edits, exactly

Nothing else in these files changes. Each quoted "before" block occurs exactly once in its file when its turn comes (make them in order). Keep the fields in the order shown.

crates/memoryd/src/recall.rs

R1. Add one line. The line

use bxcore::{DataClass, RecallArgs, Timestamp};

becomes

use bxcore::sanitize::neutralise;
use bxcore::{DataClass, RecallArgs, Timestamp};

R2. The lines

    pub class: DataClass,
    pub untrusted: bool,
}

/// The FTS5 query for `query`: every run of letters and digits, double-quoted, joined by `OR`.

become (its spaces are checked: item's format string is a line break, then two spaces)

    pub class: DataClass,
    pub untrusted: bool,
}

/// The local index's hits, each rendered as an item, numbered from 1, and the label they give
/// the answer: `Private` and trusted when there are none.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Local {
    pub items: Vec<String>,
    pub class: DataClass,
    pub untrusted: bool,
}

/// A hit's text as an item holds it, from the local index or from Honcho: neutralised, every
/// control character (a line break among them) and every Unicode line or paragraph separator a
/// space, then cut to its first `SNIPPET` characters, with no mark.
pub fn snippet(text: &str) -> String {
    neutralise(text)
        .chars()
        .map(|c| {
            if c.is_control() || c == '\u{2028}' || c == '\u{2029}' {
                ' '
            } else {
                c
            }
        })
        .take(SNIPPET)
        .collect()
}

/// An item as the model reads it: its header, then its text on one line indented by two spaces,
/// so that no text can begin a line the way a header does (M5c spec, section 8; tip T37).
pub fn item(header: &str, text: &str) -> String {
    format!("{header}\n  {text}")
}

/// The FTS5 query for `query`: every run of letters and digits, double-quoted, joined by `OR`.

R3. The lines

    session: &str,
    now: Timestamp,
) -> Result<Recalled, String> {
    // 1. `args.check()?`.

become (its spaces are checked: (no matches for \"{}\") is as before)

    session: &str,
    now: Timestamp,
) -> Result<Recalled, String> {
    let found = local(index, store, args, session, now)?;
    if found.items.is_empty() {
        return Ok(Recalled {
            text: format!("(no matches for \"{}\")", args.query),
            class: DataClass::Private,
            untrusted: false,
        });
    }
    Ok(Recalled {
        text: found.items.join("\n"),
        class: found.class,
        untrusted: found.untrusted,
    })
}

/// Search the local index for `args` on behalf of `session`, log the recall, and give its hits
/// as items with their label. `recall` answers with them alone; with Honcho, its hits follow.
pub fn local(
    index: &Index,
    store: &Store,
    args: &RecallArgs,
    session: &str,
    now: Timestamp,
) -> Result<Local, String> {
    // 1. `args.check()?`.

R4. The comment's lines

    // 4. No hits: Ok(Recalled { text: format!("(no matches for \"{}\")", args.query), class
    //    Private, untrusted false }).
    // 5. Otherwise, for each hit with i from 0: class = class.max(its class) starting at Private;
    //    untrusted |= its tag is "memory-tainted"; its text cut to its first SNIPPET characters;
    //    the item is format!("{}\n{text}", hit.header(i + 1)). Ok(Recalled { the items joined by
    //    newlines, class, untrusted }).

become

    // 4. For each hit with i from 0: class = class.max(its class) starting at Private;
    //    untrusted |= its tag is "memory-tainted"; the item is `item(&hit.header(i + 1),
    //    &snippet(&hit.text))`. Ok(Local { items, class, untrusted }).

R5. The lines

    store.append(&format!("recalls/{month}.jsonl"), &format!("{line}\n"))?;
    if hits.is_empty() {
        return Ok(Recalled {
            text: format!("(no matches for \"{}\")", args.query),
            class: DataClass::Private,
            untrusted: false,
        });
    }
    let mut class = DataClass::Private;
    let mut untrusted = false;
    let mut items: Vec<String> = Vec::with_capacity(hits.len());
    for (i, hit) in hits.iter().enumerate() {
        class = class.max(hit.class);
        untrusted |= hit.tag() == "memory-tainted";
        let text: String = hit.text.chars().take(SNIPPET).collect();
        items.push(format!("{}\n{text}", hit.header(i + 1)));
    }
    Ok(Recalled {
        text: items.join("\n"),
        class,
        untrusted,
    })
}

become

    store.append(&format!("recalls/{month}.jsonl"), &format!("{line}\n"))?;
    let mut class = DataClass::Private;
    let mut untrusted = false;
    let mut items: Vec<String> = Vec::with_capacity(hits.len());
    for (i, hit) in hits.iter().enumerate() {
        class = class.max(hit.class);
        untrusted |= hit.tag() == "memory-tainted";
        items.push(item(&hit.header(i + 1), &snippet(&hit.text)));
    }
    Ok(Local {
        items,
        class,
        untrusted,
    })
}

crates/memoryd/src/honcho/recalled.rs

H1. Remove one line. The lines

use bxcore::DataClass;
use bxcore::sanitize::neutralise;
use roncho::{Conclusion, Message};

become

use bxcore::DataClass;
use roncho::{Conclusion, Message};

H2. The line

use crate::recall::{SNIPPET, when};

becomes

use crate::recall::{item, snippet, when};

H3. In Item::render, the lines

    pub fn render(&self, n: usize) -> String {
        format!(
            "[{n}] {} · {} · {}\n  {}",
            self.tag,
            self.from,
            class_name(self.class),
            self.text
        )
    }

become

    pub fn render(&self, n: usize) -> String {
        let header = format!(
            "[{n}] {} · {} · {}",
            self.tag,
            self.from,
            class_name(self.class)
        );
        item(&header, &self.text)
    }

H4. The lines

/// Text from Honcho as an item holds it: neutralised, every control character (a line break
/// among them) and every Unicode line or paragraph separator a space, then cut to its first
/// `SNIPPET` characters, as a local snippet is cut.
pub fn clean(text: &str) -> String {
    neutralise(text)
        .chars()
        .map(|c| {
            if c.is_control() || c == '\u{2028}' || c == '\u{2029}' {
                ' '
            } else {
                c
            }
        })
        .take(SNIPPET)
        .collect()
}

become

/// Text from Honcho as an item holds it: as a local hit's text is (`recall::snippet`).
pub fn clean(text: &str) -> String {
    snippet(text)
}

Steps

• 1. Copy the given tests. From the repository root, two commands:

  cp docs/plans/M5c2/files/crates/memoryd/tests/recall_form.rs crates/memoryd/tests/
  cp docs/plans/M5c2/files/crates/memoryd/tests/recall.rs crates/memoryd/tests/
  

  Then git status --short lists crates/memoryd/tests/recall.rs as modified and recall_form.rs as new.

• 2. See them fail. Two commands, each with a shell timeout of 600000 ms:

  • cargo build --tests --keep-going -p memoryd: one test target, recall_form, does not compile: E0432, unresolved imports `memoryd::recall::Local`, `memoryd::recall::item`, `memoryd::recall::local`, `memoryd::recall::snippet` . Every other target compiles.
  • cargo test --no-fail-fast -p memoryd --test recall: two tests fail, hits_are_tagged_and_labelled and long_hits_are_cut.

• 3. The edits. Make edits R1 to R5 and H1 to H4, in order. cargo check -p memoryd prints no error and no warning.

• 4. The tests. With a shell timeout of 600000 ms, every test passes: 4 in recall_form, 6 in recall, 11 in honcho_recalled, and 2 in forgery, as before:

  cargo test -p memoryd --test recall_form --test recall --test honcho_recalled --test forgery
  

• 5. Check. git diff --stat -- crates/memoryd/src lists exactly crates/memoryd/src/honcho/recalled.rs and crates/memoryd/src/recall.rs.

• 6. Run the gate. cargo fmt --all, then make gate, with a shell timeout of 600000 ms. Expected last line: gate: ok. The test count is the passed total of cargo test --workspace plus one, for the ignored end_to_end test that the gate runs on its own: 1418 after this task, 1414 before it; the 4 new ones are the tests of recall_form.rs. To count, add up the N passed of every test result: line the gate's cargo test --workspace printed, and add

  1. Never run the gate again only to count.

• 7. Log and commit. Add your row to docs/implementer-log.md; its Task cell is M5c2/07-memoryd-recall-form. Stage with git add crates/memoryd docs/implementer-log.md Cargo.lock, then commit as AGENTS.md says. Cargo.lock does not change in this task; the git add is harmless.

Done when

• make gate prints gate: ok with 1418 tests.
• git diff HEAD~1 --stat lists only recall.rs, recalled.rs, recall_form.rs, the test recall.rs and docs/implementer-log.md.

Stop and report if

• The branch check at the top fails.
• A "before" block is not in its file exactly as shown when its turn comes, or is there more than once. H1 to H4 quote the parts of recalled.rs that task 06's skeleton gave, not the bodies you filled.
• Step 2 shows a different result: other targets failing, other errors, or other tests failing.
• cargo check in step 3 prints an error or a warning. Quote it, and stop.
• A test fails in step 4. Do not change a test: stop, and give the failing tests' names and their messages.
• The fix seems to need a change anywhere else: serve.rs, index.rs, another test, a copy from docs/plans/M5a/files/. It does not; stop and say why you think so.
• A test outside the given files fails in the gate: check once whether it fails without your change (git stash -u, run it, git stash pop), then stop and report both results.
• The gate passes with a count other than 1418.
• A comment in an edit does not fit the code under it. Quote it and stop.

What the given tests pin

Each of these changes to the finished code makes at least one given test fail (15 tried, all caught): no indent, or one space; a line feed, a carriage return or a line separator kept; no neutralise; the cap moved by one; local items made from the raw text; local items numbered from 0; a tainted local hit not making the answer untrusted; recall's no-match answer changed, its items joined another way, or its label dropped; recalled::clean or Item::render going back to a copy of their own.