boxmaker/docs/plans/M5c1/12-memoryd-honcho-mirror.md

M5c-1 task 12: memoryd decides memory writes for Honcho as they are journalled

Branch: m5c1 (run git switch m5c1; git status --short must be empty, otherwise stop) Commit subject: memoryd: memory writes decided into the Honcho ledger as they are journalled Log row: the Task cell is M5c1/12-memoryd-honcho-mirror.

Goal

Every memory write is mirrored in Honcho the moment it is journalled (M5c spec, section 4), so that "remember this" reaches Honcho at once, through the memory tool and its checks. This task decides each journalled edit into the ledger (task 07); the worker sends them in task 13. With [honcho] only: without it nothing changes.

• When. Right after journal::record (the memory tool, and a reviewed write when the owner accepts it) and after journal::undo (bxctl memory undo), in the same call, under the index lock those calls already hold. So edits are decided in the journal's order.

• Items. Item::Edit { id: <the journal id>, part }, one part per Honcho action, numbered from 0 in the order they must be sent. The outbox's order is then the send order.

• The mapping (spec section 4's table, as journal records show it):

  The journal record	The actions, in order
  an identity edit (file is memory.md or user.md): add, replace, remove, consolidate, and an undo	each entry dropped: Unconclude; then each entry added: Conclude about the owner (user.md) or the agent (memory.md)
  entries consolidate moved to notes (§ moved notes of the record)	nothing: not counted as dropped
  a write saved as notes (file is none) with a flag rule (flag set)	one Skip { why: "flagged: <rule>" }
  the same, untrusted (untrusted), not flagged	each noted note: Fleet
  the same, trusted, not flagged	each noted note: Conclude about the agent
  anything else (a rejection)	nothing

  "Dropped" and "added" are the multiset difference of the entries before and after, by text: a replace is one of each (part 0 the Unconclude of the old, part 1 the Conclude of the new), a reorder is none, and one of two equal entries removed is one dropped. An undo's before is the file as it was, its after what the undo restored, so the same difference is its inverse: the conclusions its edit made are deleted, and any it deleted are made again.

• The session. A Conclude or Fleet carries the Boxmaker session id of the write, as task 07's schema has it (which Honcho session that is sent into is task 13's). The owner's undo is journalled with the session owner: its conclusions carry the undone edit's session instead, following undoes back (through an undo of an undo). With no session at all, a part that needs one is a Skip { why: "the edit has no Boxmaker session" }; a deletion needs none.

• Text. Kept as journalled; the worker neutralises it when it sends (task 13).

• Nothing to send: an edit whose mapping is empty gets no line. A flagged write gets its Skip line, so that bxctl memory sync (task 14) can show that it stayed local.

• Retractions. An undo never touches a note (notes are append-only: journal::undo refuses them), so nothing here ever decides Action::Retract; it stays in the ledger's schema unused.

The locks. A walk (task 09) holds TurnSync's lock while it asks brokerd, up to 5 s. A memory write must not wait for that: it is the model's tool call. So this task gives the ledger a lock of its own, held only to read or append (edits T1 to T8): the walk takes it briefly for each check and append, the worker for each read and append, and a memory write for its appends. The order is always index → ledger, or walk → ledger, and the ledger's lock is never held while another is taken. A given test holds a walk on a silent brokerd and makes a memory call.

Interfaces

In crates/memoryd/src/honcho/mirror.rs (the module memoryd::honcho::mirror):

pub const OWNER: &str = "owner";
pub const NO_SESSION: &str = "the edit has no Boxmaker session";
pub fn diff(before: &[String], after: &[String]) -> (Vec<String>, Vec<String>); // (dropped, added)
pub fn session_of(record: &journal::Record, records: &[journal::Record]) -> Option<SessionId>;
pub fn actions(record: &journal::Record, session: Option<&SessionId>, agent: &str, owner: &str)
    -> Vec<Action>;
pub struct Mirror { /* private */ }
impl Mirror {
    pub fn new(ledger: Arc<Mutex<Ledger>>, agent: &str, owner: &str) -> Mirror;
    pub fn journalled(&self, store: &Store, id: Option<&str>, now: Timestamp, log: &dyn Fn(&str));
}

TurnSync (task 09): its ledger is an Arc<Mutex<Ledger>>; ledger() returns the locked ledger (a MutexGuard, which the existing calls use as before), and shared_ledger() the lock itself. worker::Opened gains mirror(), and Memoryd a mirror: Option<Mirror> and a private mirror(&self, id).

Every place that changes

This is the complete list.

Place	What happens there
crates/memoryd/src/honcho/mod.rs	pub mod mirror; (edit H1)
crates/memoryd/src/honcho/mirror.rs	new: a skeleton you copy, then fill 7 bodies
crates/memoryd/src/honcho/turns.rs	the ledger's own lock (edits T1 to T8)
crates/memoryd/src/honcho/worker.rs	the worker reads and appends under the ledger's lock; Opened::mirror (edits W1 to W9)
crates/memoryd/src/serve.rs	Memoryd.mirror, and the two calls (edits S1 to S7)
everything else (journal.rs, edit.rs, review.rs, ledger.rs, tests of earlier tasks)	no change

Files

(A copy command names <file>-taskNN where a later task gives a newer version of that file: this task's version is the one named.)

• Copy, from docs/plans/M5c1/files/ (the same path under it): the new tests crates/memoryd/tests/honcho_mirror.rs and crates/memoryd/tests/honcho_mirror_serve.rs, and the skeleton crates/memoryd/src/honcho/mirror.rs.
• Modify: the files of edits H1 to S7, docs/implementer-log.md.
• Do not touch: any other source file, any other test or support file, any Cargo.toml, and in the skeleton anything but the 7 todo!() bodies.

Facts you need

• A journal record (memoryd::journal::Record, M5a): id, session, action, file (the identity file's name, or None when the write was saved as notes), before and after (the file's entries), notes (each note appended, rendered: a header line and its text), untrusted, flag (the flag rule a write matched), undoes (for an undo, the edit it reverted). An undo is journalled with the session owner.
• Notes. memoryd::notes::parse(text: &str) -> Vec<Note> reads rendered notes back; Note { kind, time, session, text }, and kind is Kind::Noted { class, untrusted }, Kind::Moved(file) (consolidate took it out of the prompt: still held true) or Kind::Dropped(file) (no longer held true; it is also missing from after).
• Which records come here. Memoryd::memory journals a write (the id is returned); with review on, an identity write from a clean session is staged instead and journalled only when the owner accepts it (review::accept); a rejection is journalled too, and asks nothing. The given mirror in serve.rs passes the id for the memory tool, and None (the latest record, the one just journalled: the index lock is held) for an undo and an accept.
• What the skeleton gives whole: the constants, actions (which picks identity_actions or note_actions), Mirror with new and journalled (reads the journal, finds the record and its session, appends each part not yet decided, and logs a failure with the ledger's pointer: the write itself stands).
• The given tests.
  • honcho_mirror.rs (8 tests): the multiset diff (add, replace, remove, reorder only, duplicates, order); each table row from journal records; the undo of an add, a replace and a remove; the session of an owner's undo, an undo of an undo, and none.
  • honcho_mirror_serve.rs (6 tests): memoryd serving: the memory tool's writes decided in the journal's order (ids and parts); a flagged write's Skip; the owner's undo with the undone edit's session; a reviewed write decided at accept, a rejection not at all; nothing without [honcho]; a memory call answered within 1 s, and decided at once, while a walk waits for a silent brokerd (the fake from task 09).
• The calls the bodies use:

  Call	Signature, and what it gives
  slice::iter().position	left.iter().position(|t| t == text) -> Option<usize>
  Vec::swap_remove	left.swap_remove(at): removes one element, order not kept (fine: a multiset)
  Iterator::filter, cloned, collect	a.iter().filter(..).cloned().collect::<Vec<String>>()
  Iterator::flat_map	record.notes.iter().flat_map(|n| notes::parse(n))
  Option::as_deref, ?	current.undoes.as_deref()?: the &str, or return None
  SessionId::new	SessionId::new(&s).ok() -> Option<SessionId>
  matches!	matches!(n.kind, Kind::Moved(_))

The edits, exactly

Nothing else in these files changes. Each quoted "before" block occurs exactly once in its file. Edits W5 and W7 change code you wrote in task 11 from its comments: if yours differs from the "before" block, stop and report.

crates/memoryd/src/honcho/mod.rs

H1. The line

pub mod ledger;

becomes

pub mod ledger;
pub mod mirror;

crates/memoryd/src/honcho/turns.rs

T1. The line

use std::path::{Path, PathBuf};

becomes

use std::path::{Path, PathBuf};
use std::sync::{Arc, Mutex, MutexGuard};

T2. In turns.rs, the lines

/// The turns reported and not yet decided, and the ledger their decisions go to.
pub struct TurnSync {
    ledger: Ledger,

become

/// The turns reported and not yet decided, and the ledger their decisions go to. The ledger has a
/// lock of its own, held only to read or append: a walk holds `TurnSync`'s lock while it asks
/// `brokerd`, and a memory write, decided under the index lock, takes only the ledger's.
pub struct TurnSync {
    ledger: Arc<Mutex<Ledger>>,

T3. In TurnSync::new, the lines

        TurnSync {
            ledger,

become

        TurnSync {
            ledger: Arc::new(Mutex::new(ledger)),

T4. The lines

    pub fn ledger(&self) -> &Ledger {
        &self.ledger
    }

become

    /// The ledger, locked. Never hold it while taking another lock.
    pub fn ledger(&self) -> MutexGuard<'_, Ledger> {
        self.ledger.lock().unwrap_or_else(|p| p.into_inner())
    }

    /// The ledger's lock itself, for the worker and for memory writes.
    pub fn shared_ledger(&self) -> Arc<Mutex<Ledger>> {
        Arc::clone(&self.ledger)
    }

T5. In resume, the line

        for (item, _) in self.ledger.decisions() {

becomes

        for (item, _) in self.ledger().decisions() {

T6. In append, the line

        self.ledger.append(record)

becomes

        self.ledger().append(record)

T7. In walk, the line

            .filter(|turn| !self.ledger.is_decided(&item(*turn)))

becomes

            .filter(|turn| !self.ledger().is_decided(&item(*turn)))

T8. In walk, the line

            if let Err(e) = self.ledger.append(record) {

becomes

            let appended = self.ledger().append(record);
            if let Err(e) = appended {

crates/memoryd/src/honcho/worker.rs

W1. The line

use std::path::Path;

becomes

use std::path::{Path, PathBuf};

W2. The line

use crate::honcho::naming::{HonchoSession, honcho_session};

becomes

use crate::honcho::mirror::Mirror;
use crate::honcho::naming::{HonchoSession, honcho_session};

W3. The lines

pub struct Worker {
    sync: Arc<Mutex<TurnSync>>,

become

pub struct Worker {
    sync: Arc<Mutex<TurnSync>>,
    /// The ledger's own lock (task 12): taken alone, briefly, never with `sync`'s.
    ledger: Arc<Mutex<Ledger>>,
    sessions: PathBuf,

W4. In Worker::new, the lines

    ) -> Worker {
        Worker {
            sync,

become

    ) -> Worker {
        let (ledger, sessions) = {
            let sync = sync.lock().unwrap_or_else(|p| p.into_inner());
            (sync.shared_ledger(), sync.sessions().to_path_buf())
        };
        Worker {
            sync,
            ledger,
            sessions,

W5. In next, the lines

        let sync = self.sync.lock().unwrap_or_else(|p| p.into_inner());
        let ledger = sync.ledger();

become

        let ledger = self.ledger.lock().unwrap_or_else(|p| p.into_inner());

W6. In send, the lines

        let sessions = self
            .sync
            .lock()
            .unwrap_or_else(|p| p.into_inner())
            .sessions()
            .to_path_buf();
        let (text, reply) = read_turn(&log_path(&sessions, &next.session), next.turn)

become

        let (text, reply) = read_turn(&log_path(&self.sessions, &next.session), next.turn)

W7. In record, the lines

        let result = self
            .sync
            .lock()

become

        let result = self
            .ledger
            .lock()

W8. In pub struct Opened, the lines

    clients: Clients,
    agent: String,
}

become

    clients: Clients,
    agent: String,
    owner: String,
}

impl Opened {
    /// The memory writes' side of the ledger (task 12).
    pub fn mirror(&self) -> Mirror {
        let ledger = self
            .sync
            .lock()
            .unwrap_or_else(|p| p.into_inner())
            .shared_ledger();
        Mirror::new(ledger, &self.agent, &self.owner)
    }
}

W9. In open, the lines

        agent: honcho.agent.clone(),
    })

become

        agent: honcho.agent.clone(),
        owner: honcho.owner.clone(),
    })

crates/memoryd/src/serve.rs

S1. The line

use crate::honcho::ledger::{self, LedgerError};

becomes

use crate::honcho::ledger::{self, LedgerError};
use crate::honcho::mirror::Mirror;

S2. In pub struct Memoryd, the lines

    wake: Arc<Wake>,
    log: Log,
}

become

    wake: Arc<Wake>,
    /// With `[honcho]`: decides memory writes into the ledger, under the ledger's own lock.
    mirror: Option<Mirror>,
    log: Log,
}

S3. In start, the lines

    let honcho = match opened {
        Some(opened) => {
            let sync = Arc::clone(&opened.sync);
            worker::spawn(opened, &config, Arc::clone(&wake), Arc::clone(&log))?;
            Some(sync)
        }
        None => None,
    };

become

    let (honcho, mirror) = match opened {
        Some(opened) => {
            let sync = Arc::clone(&opened.sync);
            let mirror = opened.mirror();
            worker::spawn(opened, &config, Arc::clone(&wake), Arc::clone(&log))?;
            (Some(sync), Some(mirror))
        }
        None => (None, None),
    };

S4. In start, where Memoryd is built, the lines

            honcho,
            wake,
            log,

become

            honcho,
            wake,
            mirror,
            log,

S5. The line

    /// With `[honcho]`: decide the reported turn, and every other that waits (`TurnSync`).

becomes

    /// With `[honcho]`: decide journalled edit `id` (the latest when `None`) into the ledger. Called
    /// under the index lock, so edits are decided in the journal's order (M5c spec, section 4).
    fn mirror(&self, id: Option<&str>) {
        if let Some(mirror) = &self.mirror {
            mirror.journalled(&self.store, id, Timestamp::now(), &*self.log);
        }
    }

    /// With `[honcho]`: decide the reported turn, and every other that waits (`TurnSync`).

S6. In memory, the line

            let id = journal::record(&self.store, &ctx, &outcome)?;

becomes

            let id = journal::record(&self.store, &ctx, &outcome)?;
            self.mirror(Some(&id));

S7. In admin, the lines

            MemoryCommand::Reject => review::reject(&self.store, arg, now),
        };

become

            MemoryCommand::Reject => review::reject(&self.store, arg, now),
        };
        if result.is_ok() && matches!(command, MemoryCommand::Undo | MemoryCommand::Accept) {
            self.mirror(None);
        }

Steps

• 1. Copy the given tests. From the repository root, two commands:

  cp docs/plans/M5c1/files/crates/memoryd/tests/honcho_mirror.rs-task12 crates/memoryd/tests/honcho_mirror.rs
  cp docs/plans/M5c1/files/crates/memoryd/tests/honcho_mirror_serve.rs-task12 crates/memoryd/tests/honcho_mirror_serve.rs
  

• 2. See them fail. With a shell timeout of 600000 ms:

  • cargo build --tests --keep-going -p memoryd: one test target does not compile, honcho_mirror, with one error, E0432 (unresolved import `memoryd::honcho::mirror` ). Every other target compiles.
  • cargo test -p memoryd --test honcho_mirror_serve: 1 test passes (without_honcho_nothing_is_decided) and 5 fail: nothing is decided yet.

• 3. The module. Make edit H1, then copy the skeleton:

  cp docs/plans/M5c1/files/crates/memoryd/src/honcho/mirror.rs crates/memoryd/src/honcho/
  

  cargo check -p memoryd: no error, and 24 warnings, all from the unfilled bodies: function `conclusion` is never used, function `minus` is never used, unused import: `Kind` , unused import: `Workspace` , and twenty unused variable (record four times, session three times, agent and notes twice each, and a, after, b, before, file, owner, peer, records, text once each).

• 4. Fill the seven bodies of mirror.rs, one at a time, in the order of the file, each from its comment, and run cargo check -p memoryd after each: diff, minus, notes_of, session_of, conclusion, identity_actions, note_actions. When a body is filled, delete its instruction comment (the // lines inside the body) and keep the function's doc comment (///) as it is. Change nothing else. Then cargo check -p memoryd prints no warning, and cargo test -p memoryd --test honcho_mirror passes its 8 tests.

• 5. The locks and the calls. Make edits T1 to T8, W1 to W9 and S1 to S7. cargo check -p memoryd: no error and no warning.

• 6. The tests. With a shell timeout of 600000 ms, and every test passes (8, 6, and the earlier ones unchanged):

  cargo test -p memoryd --test honcho_mirror --test honcho_mirror_serve --test honcho_turns \
    --test honcho_worker --test honcho_worker_more --test honcho_serve --test serve
  

• 7. Check. Three commands:

  • grep -n '\.ledger()' crates/memoryd/src/honcho/worker.rs prints nothing: the worker takes the ledger's own lock, never through TurnSync::ledger.
  • grep -n 'self.mirror(' crates/memoryd/src/serve.rs prints two lines: after journal::record in memory, and in admin.
  • git diff --stat -- crates/bxcore crates/loopd crates/brokerd crates/gatewayd crates/bxctl Cargo.toml Cargo.lock crates/memoryd/Cargo.toml crates/memoryd/src/journal.rs crates/memoryd/src/edit.rs crates/memoryd/src/honcho/ledger.rs prints nothing.

• 8. Run the gate. cargo fmt --all, then make gate, with a shell timeout of 600000 ms. Expected last line: gate: ok, with 1314 tests: 1300 before this task, plus 14 new ones (8 in honcho_mirror.rs, 6 in honcho_mirror_serve.rs).

• 9. Log and commit. Add your row to docs/implementer-log.md; its Task cell is M5c1/12-memoryd-honcho-mirror. git status --short lists honcho/mod.rs, turns.rs, worker.rs, serve.rs and your log as modified, and mirror.rs and the two test files as new. Stage with git add crates/memoryd docs/implementer-log.md Cargo.lock, then commit as AGENTS.md says. Cargo.lock does not change in this task; the git add is harmless.

Done when

• make gate prints gate: ok with 1314 tests.
• git diff HEAD~1 --stat lists only the files of step 9.

Stop and report if

• A "before" block of H1, T1 to T8, W1 to W9 or S1 to S7 is not in its file exactly as shown, or is there more than once (W5 and W7 are code you wrote in task 11).
• Step 2 or 3 shows a different result: other test targets, errors of another kind, other warnings, or other tests passing or failing.
• cargo check reports an error after a body you filled or an edit you made, and the comment or the edit does not tell you how to fix it. Quote the error, and stop.
• A comment in the skeleton does not fit the code under it, or disagrees with this task's table. Quote both and stop.
• After step 4 or 6 a test fails although each body follows its comment and each edit is as shown. Do not change a test or the skeleton's given code: stop, and give the failing tests' names and their messages.
• The fix seems to need a change anywhere else: journal.rs, edit.rs, review.rs, ledger.rs, another crate, or a lock taken in another order. It does not; stop and say why you think so.
• A test outside the new files fails in the gate: check once whether it fails without your change, then stop and report both results.
• The gate passes with a count other than 1314.

What the given tests pin

Each of these changes to the finished code makes at least one given test fail (21 tried; four cannot change behaviour, see below): the diff taken as sets, or its sides swapped; moved entries counted as dropped, or dropped notes left out; the wrong peer for either file; the new before the old; a flag ignored; an untrusted note concluded, or a trusted one sent to the fleet; the owner's session kept, or the undo chain cut short; a conclusion without a session sent anyway; the parts numbered from 1; the memory tool's write, or the undo and accept, not decided; and a walk that holds the ledger's lock while it asks brokerd. Not caught, since they cannot change what is decided: moved notes treated as noted ones (only identity edits have them), the latest record taken instead of the id (under the index lock they are the same), and the check against deciding an edit twice (nothing calls it twice).