boxmaker/docs/plans/M5c1/07-memoryd-honcho-ledger.md

M5c-1 task 07: memoryd's Honcho ledger, memory/honcho.jsonl

Branch: m5c1 (run git switch m5c1; git status --short must be empty, otherwise stop) Commit subject: memoryd: the Honcho ledger, decided, sent and failed, read through at start Log row: the Task cell is M5c1/07-memoryd-honcho-ledger.

Goal

From M5c on, memoryd sends finished turns and memory writes to Honcho (M5c spec, section 5). Each one is first decided (where it goes, or that it goes nowhere), then sent or failed (Honcho refused it). This task adds the record of that: the ledger, an append-only file of JSON lines, <home>/memory/honcho.jsonl. What is decided and neither sent nor failed is the outbox, the work still to do; there is no other queue. memoryd reads the ledger through when it starts, so nothing is decided twice or sent twice.

This task is the module memoryd::honcho::ledger alone. Nothing calls it yet: task 09 opens it at start and decides turns into it, later tasks send from its outbox. Its path is passed in.

• Items. A turn of a session (Item::Turn { session, turn }), or one part of a journalled memory edit (Item::Edit { id, part }; the id is the journal's, 2026-09-28-0001).
• Actions (what a Decided record says): the turn's messages to a workspace; a conclusion to create or delete; a message, or its retraction, in the fleet workspace; or Skip, nothing to send, and why. A turn's text is never stored here: the sender reads it from the session log.
• The rules. An item has exactly one Decided record. It has at most one Sent or Failed record, and only while it is waiting: decided, not Skip, and neither sent nor failed yet. append refuses a record that breaks a rule, and writes nothing for it.
• Durable before believed. append writes the line, syncs it to disk, and only then takes it into the ledger's state in memory, as loopd does with its session log. A failed write leaves the state as it was, so the ledger never says an item is decided or sent when the file does not.
• Reading through. open reads the whole file and replays it. A missing file is an empty ledger. A last line with no newline was cut off by a crash in the middle of an append: it is dropped and cut from the file, and repaired() says so. Any other line that does not parse, or that breaks a rule when replayed in order, stops the ledger: LedgerError::Damaged with its line number and the runbook pointer honcho-ledger-damaged, and the file is left as it is.

Interfaces

Later tasks use these exactly as written. They are in the skeleton you copy (step 3); only 12 private or small function bodies are yours to fill.

In crates/memoryd/src/honcho/ledger.rs (the module memoryd::honcho::ledger):

pub const RUNBOOK: &str = "see docs/runbook.md#honcho-ledger-damaged";

pub enum Workspace { Trusted, Untrusted, Fleet }
pub enum Item { Turn { session: SessionId, turn: u64 }, Edit { id: String, part: u32 } }
pub enum Action {
    Turn { workspace: Workspace, speaker: String },
    Conclude { workspace: Workspace, peer: String, session: SessionId, text: String },
    Unconclude { workspace: Workspace, text: String },
    Fleet { session: SessionId, text: String },
    Retract { session: SessionId, text: String },
    Skip { why: String },
}
pub enum Record {
    Decided { time: Timestamp, item: Item, action: Action },
    Sent { time: Timestamp, item: Item, ids: Vec<String> },
    Failed { time: Timestamp, item: Item, status: Option<u16>, kind: String },
}
pub struct Counts { pub decided: usize, pub sent: usize, pub waiting: usize,
                    pub failed: usize, pub skipped: usize }
pub enum LedgerError {
    Io(PathBuf, std::io::Error),
    Damaged { path: PathBuf, line: usize, why: String },
    Encode(serde_json::Error),
    AlreadyDecided(Item),
    NotWaiting(Item),
}
pub struct Ledger { /* private */ }

impl Ledger {
    pub fn open(path: &Path) -> Result<Ledger, LedgerError>;
    pub fn repaired(&self) -> bool;
    pub fn append(&mut self, record: Record) -> Result<(), LedgerError>;
    pub fn is_decided(&self, item: &Item) -> bool;
    pub fn outbox(&self) -> Vec<(Item, Action)>;
    pub fn sent_ids(&self, item: &Item) -> Option<&[String]>;
    pub fn conclusion(&self, workspace: Workspace, text: &str) -> Option<(Item, String)>;
    pub fn counts(&self) -> Counts;
}

SessionId and Timestamp are bxcore's. The derives and serde attributes are in the skeleton; do not change them (see "The lines").

What each public function gives

Function	Gives
open	the ledger read through; see "Reading through" above
repaired	true when open dropped a cut-off last line
append	the record checked, written with its newline in one write, synced, then taken into the state; a refused or failed append changes nothing
is_decided	whether the item has its Decided record (a Skip counts)
outbox	every waiting item with its action, in the order of the Decided records
sent_ids	the ids of the item's Sent record; None when it has none (waiting, failed, skipped, unknown)
conclusion	the conclusion to delete for an undo: see below
counts	how many items are decided (every one, Skip included), sent, waiting, failed and skipped; decided is always the sum of the other four

conclusion(workspace, text). An undo must delete the conclusion that the undone write created. Honcho gave its id in the Sent record of the Conclude (a Conclude's Sent has exactly one id; one with no ids is ignored). Replay the Sent records in the order they were appended: a sent Conclude of this workspace and text puts (item, its id) on top of a stack, and a sent Unconclude of the same workspace and text takes the top one off. The answer is the top of the stack, or None. So it is the latest one sent that is not undone yet; another workspace's, or one decided but not sent, is never the answer.

The errors' text (Display, given in the skeleton):

Error	Text
Io(path, e)	<path>: <e>
Damaged { path, line, why }	<path>:<line>: <why>; see docs/runbook.md#honcho-ledger-damaged (the line counted from 1)
Encode(e)	a ledger record could not be encoded: <e>
AlreadyDecided(item)	turn 3 of session chat-1: decided already, or part 0 of edit 2026-09-28-0001: decided already
NotWaiting(item)	turn 3 of session chat-1: not waiting to be sent (the same naming)

When a replayed line breaks a rule, why is the text of AlreadyDecided or NotWaiting; when it does not parse, why is serde's error text.

The lines

One record per line, as serde_json::to_string writes it, then \n. Every enum except Workspace is internally tagged: its variant is the field "type", in snake case, first; and every one refuses unknown fields. Workspace is a plain string. A Timestamp is RFC 3339 with milliseconds; a SessionId a string. For example (each is one line in the file):

{"type":"decided","time":"2026-09-28T10:00:00.000Z","item":{"type":"turn","session":"chat-1","turn":3},"action":{"type":"turn","workspace":"trusted","speaker":"kyle"}}
{"type":"decided","time":"2026-09-28T10:00:00.000Z","item":{"type":"edit","id":"2026-09-28-0001","part":0},"action":{"type":"conclude","workspace":"trusted","peer":"kyle","session":"chat-1","text":"Kyle takes his coffee black."}}
{"type":"sent","time":"2026-09-28T10:00:00.000Z","item":{"type":"turn","session":"chat-1","turn":3},"ids":["msg-a1","msg-a2"]}
{"type":"failed","time":"2026-09-28T10:00:00.000Z","item":{"type":"turn","session":"chat-2","turn":1},"status":null,"kind":"refused"}

This is our own format, so unknown fields are errors: #[serde(tag = "type", rename_all = "snake_case", deny_unknown_fields)] on Item, Action and Record. serde honours deny_unknown_fields on internally tagged enums, at every level (a given test adds a field to the record, to its item and to its action, and expects an error each time). The skeleton has all of this already.

Every place that changes

This is the complete list.

Place	What happens there
crates/memoryd/src/honcho/mod.rs	pub mod ledger; (edit H1)
crates/memoryd/src/honcho/ledger.rs	new: a skeleton you copy, then fill 12 bodies (step 4)
docs/runbook.md	the entry honcho-ledger-damaged, appended (step 6)
everything else in memoryd (config.rs, serve.rs, main.rs, honcho/config.rs, …)	no change

Files

(A copy command names <file>-taskNN where a later task gives a newer version of that file: this task's version is the one named.)

• Copy, from docs/plans/M5c1/files/ (the same path under it):
  • new tests: crates/memoryd/tests/honcho_ledger.rs, crates/memoryd/tests/honcho_ledger_file.rs
  • new test support: crates/memoryd/tests/support/ledger.rs
  • the skeleton crates/memoryd/src/honcho/ledger.rs (you fill 12 bodies in it)
• Append, from docs/plans/M5c1/files/, to the end of docs/runbook.md, with one blank line before it: runbook-honcho-ledger-damaged.md (step 6 gives the command).
• Modify: crates/memoryd/src/honcho/mod.rs, docs/implementer-log.md.
• Do not touch: any other source file, any file of another crate, any Cargo.toml, any other test, crates/memoryd/tests/support/tmp.rs, and in the skeleton anything but the 12 todo!() bodies.

Facts you need

• The state in memory. The skeleton gives it: order (every decided item, in the order of its Decided record), entries (a HashMap from each item to its Entry { action, state }), sends (every sent item, in the order of its Sent record), and repaired. An entry's State is Skipped (decided Skip), Waiting, Sent(ids) or Failed. Only apply changes them, and only for a record check accepted.
• open and append are given, as glue over your functions. open checks every complete line before it cuts a cut-off tail, so a damaged file is never changed. append calls check, encodes, calls write_line, and only then apply. Read both before you start.
• One write per line. write_line builds the line and its newline as one string and writes it with one write_all, then sync_data: a crash leaves either the whole line or a cut-off line without its newline, which open drops. Two writes, or no sync, would pass every test and still be wrong; step 7 checks for them.
• A missing file creates nothing. open only reads. write_line creates the file on the first append, at mode 0o600, and never creates the directory: a missing memory/ is an Io error. The file's mode is given only when the file is created (OpenOptionsExt::mode, and the process's umask can only take bits away).
• Bytes, not text. open reads the file as bytes (std::fs::read), because a cut-off last line can end inside a multi-byte character. A complete line is parsed with serde_json::from_slice, which accepts the trailing newline.
• The given tests.
  • honcho_ledger.rs (12 tests): each kind of line exactly; append writes the line and a newline; unknown fields and values refused; the outbox (what is in it, and its order); a second decision refused and nothing written; Sent or Failed refused before the decision, after an outcome and for a Skip; sent_ids; conclusion (the latest not undone, only its workspace and text, only once sent); the counts add up.
  • honcho_ledger_file.rs (12 tests): a missing and an empty file; reopening gives the same state; a cut-off last line (in the middle, one byte short, inside a character, the only line) dropped and cut; a damaged line and a line that breaks a rule, with their numbers, the text and the file unchanged; mode 0600; a missing directory; a failed write changes nothing (the test turns the ledger's path into a directory); an unreadable ledger.
  • support/ledger.rs: record builders (decided, sent, failed, turn, edit, …) and Home, a ledger path in a directory removed on drop.
• The calls the bodies use:

  Call	Signature, and what it gives
  HashMap::contains_key	self.entries.contains_key(item) -> bool
  HashMap::get, get_mut	self.entries.get(item) -> Option<&Entry>; get_mut(&item) -> Option<&mut Entry>
  HashMap::insert	self.entries.insert(item, Entry { action, state }) (the old value, ignored)
  HashMap::values	self.entries.values(): every &Entry
  Iterator::filter_map	.iter().filter_map(|item| { ..; Some(..) }).collect()
  bool::then	(cond).then(|| value) -> Option<T>: Some(value) when cond
  slice::first	ids.first() -> Option<&String>
  Vec::push, Vec::pop	live.push(x); live.pop() -> Option<T>: takes the last one off
  Vec::as_slice	ids.as_slice() -> &[String]
  OpenOptions	OpenOptions::new().append(true).create(true).mode(0o600).open(&path) -> io::Result<File>; .mode is std::os::unix::fs::OpenOptionsExt (imported)
  Write::write_all	file.write_all(bytes: &[u8]) -> io::Result<()> (std::io::Write, imported)
  File::sync_data	file.sync_data() -> io::Result<()>: the data on disk, not necessarily the metadata
  File::set_len	file.set_len(size: u64) -> io::Result<()>: shrinks the file to size bytes; the file needs write(true)
  std::fs::read	std::fs::read(path) -> io::Result<Vec<u8>>
  io::Error::kind	e.kind() == std::io::ErrorKind::NotFound
  io::Error::other	std::io::Error::other(e): wraps any error as an io::Error
  u64::try_from	u64::try_from(keep: usize) -> Result<u64, TryFromIntError>
  Iterator::rposition	bytes.iter().rposition(|b| *b == b'\n') -> Option<usize>: the index of the last newline
  serde_json::from_slice	serde_json::from_slice::<Record>(line: &[u8]) -> serde_json::Result<Record>

The edit, exactly

crates/memoryd/src/honcho/mod.rs

H1. The line

pub mod config;

becomes

pub mod config;
pub mod ledger;

Steps

• 1. Copy the given tests. From the repository root, three commands:

  cp docs/plans/M5c1/files/crates/memoryd/tests/honcho_ledger.rs-task07 crates/memoryd/tests/honcho_ledger.rs
  cp docs/plans/M5c1/files/crates/memoryd/tests/honcho_ledger_file.rs-task07 crates/memoryd/tests/honcho_ledger_file.rs
  cp docs/plans/M5c1/files/crates/memoryd/tests/support/ledger.rs-task07 crates/memoryd/tests/support/ledger.rs
  

• 2. See them fail. cargo build --tests --keep-going -p memoryd, with a shell timeout of 600000 ms: two test targets do not compile, honcho_ledger and honcho_ledger_file, with 2 errors each, both E0432 (unresolved import `memoryd::honcho::ledger` ): one in the test file, one in support/ledger.rs. Every other target compiles.

• 3. The module. Make edit H1, then copy the skeleton:

  cp docs/plans/M5c1/files/crates/memoryd/src/honcho/ledger.rs crates/memoryd/src/honcho/
  

  cargo check -p memoryd: no error, and 20 warnings, all from the unfilled bodies: three unused import (std::fs::OpenOptions, std::io::Write, std::os::unix::fs::OpenOptionsExt); fourteen unused variable (path three times, item, line and record twice each, workspace, text, bytes, number, keep); fields `action` and `state` are never read; fields `path`, `order`, `entries`, and `sends` are never read; and variants `Skipped`, `Waiting`, `Sent`, and `Failed` are never constructed. They go as you fill the bodies.

  Then cargo test -p memoryd --test honcho_ledger: 2 tests pass (each_record_is_one_line_exactly_as_given and an_unknown_field_or_value_is_refused: the types are given) and 10 fail with not yet implemented.

• 4. Fill the twelve bodies of honcho/ledger.rs, one at a time, in the order of the file, each from its comment, and run cargo check -p memoryd after each: is_decided, outbox, sent_ids, conclusion, counts, check, apply, write_line, then the free functions read, complete, parse, cut. When a body is filled, delete its instruction comment (the // lines inside the body) and keep the function's doc comment (///) as it is. Change nothing else in the file. After the twelfth, cargo check -p memoryd prints no warning.

• 5. The tests. This command, and every test passes (12 and 12):

  cargo test -p memoryd --test honcho_ledger --test honcho_ledger_file
  

• 6. The runbook. From the repository root:

  { echo; cat docs/plans/M5c1/files/runbook-honcho-ledger-damaged.md; } >> docs/runbook.md
  

  Then grep -c '^## honcho-ledger-damaged$' docs/runbook.md prints 1.

• 7. Check. Four commands:

  • grep -c 'sync_data' crates/memoryd/src/honcho/ledger.rs prints 2 (in write_line and in cut), and grep -c 'write_all' crates/memoryd/src/honcho/ledger.rs prints 1.
  • grep -rn 'runbook.md#honcho-ledger-damaged' crates/memoryd/src prints exactly one line, the RUNBOOK constant in honcho/ledger.rs.
  • grep -n 'create_dir' crates/memoryd/src/honcho/ledger.rs prints nothing: the ledger never makes its directory.
  • git diff --stat -- crates/bxcore crates/loopd crates/brokerd crates/gatewayd crates/bxctl Cargo.toml crates/memoryd/Cargo.toml prints nothing.

• 8. Run the gate. cargo fmt --all, then make gate, with a shell timeout of 600000 ms. Expected last line: gate: ok, with 1212 tests: 1188 before this task, plus 24 new ones (12 in honcho_ledger.rs, 12 in honcho_ledger_file.rs).

• 9. Log and commit. Add your row to docs/implementer-log.md; its Task cell is M5c1/07-memoryd-honcho-ledger. git status --short lists crates/memoryd/src/honcho/mod.rs, docs/runbook.md and your log as modified, and crates/memoryd/src/honcho/ledger.rs, the two test files and crates/memoryd/tests/support/ledger.rs as new. Stage with git add crates/memoryd docs/runbook.md docs/implementer-log.md Cargo.lock, then commit as AGENTS.md says. Cargo.lock does not change in this task; the git add is harmless.

Done when

• make gate prints gate: ok with 1212 tests.
• git diff HEAD~1 --stat lists only honcho/mod.rs, honcho/ledger.rs, the two new test files, tests/support/ledger.rs, docs/runbook.md and docs/implementer-log.md.

Stop and report if

• The line of edit H1 is not in crates/memoryd/src/honcho/mod.rs exactly as shown, or is there more than once, or crates/memoryd/tests/support/ledger.rs exists before step 1.
• Step 2 or 3 shows a different result: other test targets, errors of another kind, other warnings, or other tests passing or failing.
• cargo check reports an error after a body you filled, and the comment does not tell you how to fix it. Quote the error, and stop.
• A comment in the skeleton does not fit the code under it, or disagrees with this task's text. Quote both and stop.
• After step 4 a test fails although each body follows its comment. Do not change a test, the support file or the skeleton's given code (open, append, the types, Display): stop, and give the failing tests' names and their messages.
• The fix seems to need a change anywhere else: another memoryd file, another crate, a dependency, or a change to a derive or a serde attribute. It does not; stop and say why you think so.
• A test outside these two files fails in the gate: check once whether it fails without your change, then stop and report both results.
• The gate passes with a count other than 1212.

What the given tests pin

Each of these changes to a finished ledger.rs makes at least one given test fail: the outbox taking in a skipped, sent or failed item, or losing the decision order; conclusion ignoring an Unconclude, the workspace, the order of the sends, or a Sent with no ids, or answering the oldest; a second decision, or a Sent or Failed before the decision, after an outcome or for a Skip, accepted; the state taken in before the write; the directory made; the mode; a cut-off line kept, not cut, cut before the other lines are checked, or not reported by repaired; line numbers from 0; replay without the rules; a missing file an error or an unreadable one empty; any error text; the counts; a serde attribute removed. Two writes for one line, and a missing sync_data, pass the tests; step 7 checks those.