boxmaker/docs/plans/M5c1/07-memoryd-honcho-ledger.md

# M5c-1 task 07: `memoryd`'s Honcho ledger, `memory/honcho.jsonl`

**Branch:** `m5c1` (run `git switch m5c1`; `git status --short` must be empty, otherwise stop)
**Commit subject:** `memoryd: the Honcho ledger, decided, sent and failed, read through at start`
**Log row:** the Task cell is `M5c1/07-memoryd-honcho-ledger`.

## Goal

From M5c on, `memoryd` sends finished turns and memory writes to Honcho (M5c spec, section 5).
Each one is first **decided** (where it goes, or that it goes nowhere), then **sent** or
**failed** (Honcho refused it). This task adds the record of that: the ledger, an append-only file
of JSON lines, `<home>/memory/honcho.jsonl`. What is decided and neither sent nor failed is the
**outbox**, the work still to do; there is no other queue. `memoryd` reads the ledger through when
it starts, so nothing is decided twice or sent twice.

This task is the module `memoryd::honcho::ledger` alone. Nothing calls it yet: task 09 opens it
at start and decides turns into it, later tasks send from its outbox. Its path is passed in.

- **Items.** A turn of a session (`Item::Turn { session, turn }`), or one part of a journalled
  memory edit (`Item::Edit { id, part }`; the id is the journal's, `2026-09-28-0001`).
- **Actions** (what a `Decided` record says): the turn's messages to a workspace; a conclusion
  to create or delete; a message, or its retraction, in the fleet workspace; or `Skip`, nothing
  to send, and why. A turn's text is never stored here: the sender reads it from the session log.
- **The rules.** An item has exactly one `Decided` record. It has at most one `Sent` or `Failed`
  record, and only while it is **waiting**: decided, not `Skip`, and neither sent nor failed yet.
  `append` refuses a record that breaks a rule, and writes nothing for it.
- **Durable before believed.** `append` writes the line, syncs it to disk, and only then takes it
  into the ledger's state in memory, as `loopd` does with its session log. A failed write leaves
  the state as it was, so the ledger never says an item is decided or sent when the file does not.
- **Reading through.** `open` reads the whole file and replays it. A missing file is an empty
  ledger. A last line with no newline was cut off by a crash in the middle of an append: it is
  dropped and cut from the file, and `repaired()` says so. Any other line that does not parse, or
  that breaks a rule when replayed in order, stops the ledger: `LedgerError::Damaged` with its
  line number and the runbook pointer `honcho-ledger-damaged`, and the file is left as it is.

## Interfaces

Later tasks use these exactly as written. They are in the skeleton you copy (step 3); only 12
private or small function bodies are yours to fill.

In `crates/memoryd/src/honcho/ledger.rs` (the module `memoryd::honcho::ledger`):

```rust
pub const RUNBOOK: &str = "see docs/runbook.md#honcho-ledger-damaged";

pub enum Workspace { Trusted, Untrusted, Fleet }
pub enum Item { Turn { session: SessionId, turn: u64 }, Edit { id: String, part: u32 } }
pub enum Action {
    Turn { workspace: Workspace, speaker: String },
    Conclude { workspace: Workspace, peer: String, session: SessionId, text: String },
    Unconclude { workspace: Workspace, text: String },
    Fleet { session: SessionId, text: String },
    Retract { session: SessionId, text: String },
    Skip { why: String },
}
pub enum Record {
    Decided { time: Timestamp, item: Item, action: Action },
    Sent { time: Timestamp, item: Item, ids: Vec<String> },
    Failed { time: Timestamp, item: Item, status: Option<u16>, kind: String },
}
pub struct Counts { pub decided: usize, pub sent: usize, pub waiting: usize,
                    pub failed: usize, pub skipped: usize }
pub enum LedgerError {
    Io(PathBuf, std::io::Error),
    Damaged { path: PathBuf, line: usize, why: String },
    Encode(serde_json::Error),
    AlreadyDecided(Item),
    NotWaiting(Item),
}
pub struct Ledger { /* private */ }

impl Ledger {
    pub fn open(path: &Path) -> Result<Ledger, LedgerError>;
    pub fn repaired(&self) -> bool;
    pub fn append(&mut self, record: Record) -> Result<(), LedgerError>;
    pub fn is_decided(&self, item: &Item) -> bool;
    pub fn outbox(&self) -> Vec<(Item, Action)>;
    pub fn sent_ids(&self, item: &Item) -> Option<&[String]>;
    pub fn conclusion(&self, workspace: Workspace, text: &str) -> Option<(Item, String)>;
    pub fn counts(&self) -> Counts;
}
```

`SessionId` and `Timestamp` are `bxcore`'s. The derives and serde attributes are in the skeleton;
do not change them (see "The lines").

## What each public function gives

| Function | Gives |
|---|---|
| `open` | the ledger read through; see "Reading through" above |
| `repaired` | `true` when `open` dropped a cut-off last line |
| `append` | the record checked, written with its newline in one write, synced, then taken into the state; a refused or failed append changes nothing |
| `is_decided` | whether the item has its `Decided` record (a `Skip` counts) |
| `outbox` | every waiting item with its action, in the order of the `Decided` records |
| `sent_ids` | the ids of the item's `Sent` record; `None` when it has none (waiting, failed, skipped, unknown) |
| `conclusion` | the conclusion to delete for an undo: see below |
| `counts` | how many items are decided (every one, `Skip` included), sent, waiting, failed and skipped; `decided` is always the sum of the other four |

**`conclusion(workspace, text)`.** An undo must delete the conclusion that the undone write
created. Honcho gave its id in the `Sent` record of the `Conclude` (a `Conclude`'s `Sent` has
exactly one id; one with no ids is ignored). Replay the `Sent` records in the order they were
appended: a sent `Conclude` of this workspace and text puts `(item, its id)` on top of a stack,
and a sent `Unconclude` of the same workspace and text takes the top one off. The answer is the
top of the stack, or `None`. So it is the latest one sent that is not undone yet; another
workspace's, or one decided but not sent, is never the answer.

**The errors' text** (`Display`, given in the skeleton):

| Error | Text |
|---|---|
| `Io(path, e)` | `<path>: <e>` |
| `Damaged { path, line, why }` | `<path>:<line>: <why>; see docs/runbook.md#honcho-ledger-damaged` (the line counted from 1) |
| `Encode(e)` | `a ledger record could not be encoded: <e>` |
| `AlreadyDecided(item)` | `turn 3 of session chat-1: decided already`, or `part 0 of edit 2026-09-28-0001: decided already` |
| `NotWaiting(item)` | `turn 3 of session chat-1: not waiting to be sent` (the same naming) |

When a replayed line breaks a rule, `why` is the text of `AlreadyDecided` or `NotWaiting`; when it
does not parse, `why` is serde's error text.

## The lines

One record per line, as `serde_json::to_string` writes it, then `\n`. Every enum except
`Workspace` is internally tagged: its variant is the field `"type"`, in snake case, first; and
every one refuses unknown fields. `Workspace` is a plain string. A `Timestamp` is RFC 3339 with
milliseconds; a `SessionId` a string. For example (each is one line in the file):

```text
{"type":"decided","time":"2026-09-28T10:00:00.000Z","item":{"type":"turn","session":"chat-1","turn":3},"action":{"type":"turn","workspace":"trusted","speaker":"kyle"}}
{"type":"decided","time":"2026-09-28T10:00:00.000Z","item":{"type":"edit","id":"2026-09-28-0001","part":0},"action":{"type":"conclude","workspace":"trusted","peer":"kyle","session":"chat-1","text":"Kyle takes his coffee black."}}
{"type":"sent","time":"2026-09-28T10:00:00.000Z","item":{"type":"turn","session":"chat-1","turn":3},"ids":["msg-a1","msg-a2"]}
{"type":"failed","time":"2026-09-28T10:00:00.000Z","item":{"type":"turn","session":"chat-2","turn":1},"status":null,"kind":"refused"}
```

This is our own format, so unknown fields are errors: `#[serde(tag = "type", rename_all =
"snake_case", deny_unknown_fields)]` on `Item`, `Action` and `Record`. serde honours
`deny_unknown_fields` on internally tagged enums, at every level (a given test adds a field to the
record, to its item and to its action, and expects an error each time). The skeleton has all of
this already.

## Every place that changes

This is the complete list.

| Place | What happens there |
|---|---|
| `crates/memoryd/src/honcho/mod.rs` | `pub mod ledger;` (edit H1) |
| `crates/memoryd/src/honcho/ledger.rs` | new: a skeleton you copy, then fill 12 bodies (step 4) |
| `docs/runbook.md` | the entry `honcho-ledger-damaged`, appended (step 6) |
| everything else in `memoryd` (`config.rs`, `serve.rs`, `main.rs`, `honcho/config.rs`, …) | **no change** |

## Files

- Copy, from `docs/plans/M5c1/files/` (the same path under it):
  - new tests: `crates/memoryd/tests/honcho_ledger.rs`, `crates/memoryd/tests/honcho_ledger_file.rs`
  - new test support: `crates/memoryd/tests/support/ledger.rs`
  - the skeleton `crates/memoryd/src/honcho/ledger.rs` (you fill 12 bodies in it)
- Append, from `docs/plans/M5c1/files/`, to the end of `docs/runbook.md`, with one blank line
  before it: `runbook-honcho-ledger-damaged.md` (step 6 gives the command).
- Modify: `crates/memoryd/src/honcho/mod.rs`, `docs/implementer-log.md`.
- Do not touch: any other source file, any file of another crate, any `Cargo.toml`, any other
  test, `crates/memoryd/tests/support/tmp.rs`, and in the skeleton anything but the 12 `todo!()`
  bodies.

## Facts you need

- **The state in memory.** The skeleton gives it: `order` (every decided item, in the order of
  its `Decided` record), `entries` (a `HashMap` from each item to its `Entry { action, state }`),
  `sends` (every sent item, in the order of its `Sent` record), and `repaired`. An entry's
  `State` is `Skipped` (decided `Skip`), `Waiting`, `Sent(ids)` or `Failed`. Only `apply` changes
  them, and only for a record `check` accepted.
- **`open` and `append` are given**, as glue over your functions. `open` checks every complete
  line before it cuts a cut-off tail, so a damaged file is never changed. `append` calls `check`,
  encodes, calls `write_line`, and only then `apply`. Read both before you start.
- **One write per line.** `write_line` builds the line and its newline as one string and writes
  it with one `write_all`, then `sync_data`: a crash leaves either the whole line or a cut-off
  line without its newline, which `open` drops. Two writes, or no sync, would pass every test and
  still be wrong; step 7 checks for them.
- **A missing file creates nothing.** `open` only reads. `write_line` creates the file on the
  first append, at mode `0o600`, and never creates the directory: a missing `memory/` is an
  `Io` error. The file's mode is given only when the file is created (`OpenOptionsExt::mode`,
  and the process's umask can only take bits away).
- **Bytes, not text.** `open` reads the file as bytes (`std::fs::read`), because a cut-off last
  line can end inside a multi-byte character. A complete line is parsed with
  `serde_json::from_slice`, which accepts the trailing newline.
- **The given tests.**
  - `honcho_ledger.rs` (12 tests): each kind of line exactly; `append` writes the line and a
    newline; unknown fields and values refused; the outbox (what is in it, and its order); a
    second decision refused and nothing written; `Sent` or `Failed` refused before the decision,
    after an outcome and for a `Skip`; `sent_ids`; `conclusion` (the latest not undone, only its
    workspace and text, only once sent); the counts add up.
  - `honcho_ledger_file.rs` (12 tests): a missing and an empty file; reopening gives the same
    state; a cut-off last line (in the middle, one byte short, inside a character, the only line)
    dropped and cut; a damaged line and a line that breaks a rule, with their numbers, the text
    and the file unchanged; mode 0600; a missing directory; a failed write changes nothing (the
    test turns the ledger's path into a directory); an unreadable ledger.
  - `support/ledger.rs`: record builders (`decided`, `sent`, `failed`, `turn`, `edit`, …) and
    `Home`, a ledger path in a directory removed on drop.
- The calls the bodies use:

  | Call | Signature, and what it gives |
  |---|---|
  | `HashMap::contains_key` | `self.entries.contains_key(item) -> bool` |
  | `HashMap::get`, `get_mut` | `self.entries.get(item) -> Option<&Entry>`; `get_mut(&item) -> Option<&mut Entry>` |
  | `HashMap::insert` | `self.entries.insert(item, Entry { action, state })` (the old value, ignored) |
  | `HashMap::values` | `self.entries.values()`: every `&Entry` |
  | `Iterator::filter_map` | `.iter().filter_map(\|item\| { ..; Some(..) }).collect()` |
  | `bool::then` | `(cond).then(\|\| value) -> Option<T>`: `Some(value)` when `cond` |
  | `slice::first` | `ids.first() -> Option<&String>` |
  | `Vec::push`, `Vec::pop` | `live.push(x)`; `live.pop() -> Option<T>`: takes the last one off |
  | `Vec::as_slice` | `ids.as_slice() -> &[String]` |
  | `OpenOptions` | `OpenOptions::new().append(true).create(true).mode(0o600).open(&path) -> io::Result<File>`; `.mode` is `std::os::unix::fs::OpenOptionsExt` (imported) |
  | `Write::write_all` | `file.write_all(bytes: &[u8]) -> io::Result<()>` (`std::io::Write`, imported) |
  | `File::sync_data` | `file.sync_data() -> io::Result<()>`: the data on disk, not necessarily the metadata |
  | `File::set_len` | `file.set_len(size: u64) -> io::Result<()>`: shrinks the file to `size` bytes; the file needs `write(true)` |
  | `std::fs::read` | `std::fs::read(path) -> io::Result<Vec<u8>>` |
  | `io::Error::kind` | `e.kind() == std::io::ErrorKind::NotFound` |
  | `io::Error::other` | `std::io::Error::other(e)`: wraps any error as an `io::Error` |
  | `u64::try_from` | `u64::try_from(keep: usize) -> Result<u64, TryFromIntError>` |
  | `Iterator::rposition` | `bytes.iter().rposition(\|b\| *b == b'\n') -> Option<usize>`: the index of the last newline |
  | `serde_json::from_slice` | `serde_json::from_slice::<Record>(line: &[u8]) -> serde_json::Result<Record>` |

## The edit, exactly

### `crates/memoryd/src/honcho/mod.rs`

**H1.** The line

```rust
pub mod config;
```

becomes

```rust
pub mod config;
pub mod ledger;
```

## Steps

- [ ] **1. Copy the given tests.** From the repository root, three commands:

  ```sh
  cp docs/plans/M5c1/files/crates/memoryd/tests/honcho_ledger.rs crates/memoryd/tests/
  cp docs/plans/M5c1/files/crates/memoryd/tests/honcho_ledger_file.rs crates/memoryd/tests/
  cp docs/plans/M5c1/files/crates/memoryd/tests/support/ledger.rs crates/memoryd/tests/support/
  ```

- [ ] **2. See them fail.** `cargo build --tests --keep-going -p memoryd`, with a shell timeout
  of 600000 ms: two test targets do not compile, `honcho_ledger` and `honcho_ledger_file`, with 2
  errors each, both `E0432` (``unresolved import `memoryd::honcho::ledger` ``): one in the test
  file, one in `support/ledger.rs`. Every other target compiles.
- [ ] **3. The module.** Make edit H1, then copy the skeleton:

  ```sh
  cp docs/plans/M5c1/files/crates/memoryd/src/honcho/ledger.rs crates/memoryd/src/honcho/
  ```

  `cargo check -p memoryd`: no error, and 20 warnings, all from the unfilled bodies: three
  `unused import` (`std::fs::OpenOptions`, `std::io::Write`, `std::os::unix::fs::OpenOptionsExt`);
  fourteen `unused variable` (`path` three times, `item`, `line` and `record` twice each,
  `workspace`, `text`, `bytes`, `number`, `keep`); ``fields `action` and `state` are never
  read``; ``fields `path`, `order`, `entries`, and `sends` are never read``; and ``variants
  `Skipped`, `Waiting`, `Sent`, and `Failed` are never constructed``. They go as you fill the
  bodies.

  Then `cargo test -p memoryd --test honcho_ledger`: 2 tests pass
  (`each_record_is_one_line_exactly_as_given` and `an_unknown_field_or_value_is_refused`: the
  types are given) and 10 fail with `not yet implemented`.
- [ ] **4. Fill the twelve bodies of `honcho/ledger.rs`, one at a time, in the order of the
  file**, each from its comment, and run `cargo check -p memoryd` after each: `is_decided`,
  `outbox`, `sent_ids`, `conclusion`, `counts`, `check`, `apply`, `write_line`, then the free
  functions `read`, `complete`, `parse`, `cut`. When a body is filled, delete its instruction
  comment (the `//` lines inside the body) and keep the function's doc comment (`///`) as it is.
  Change nothing else in the file. After the twelfth, `cargo check -p memoryd` prints no warning.
- [ ] **5. The tests.** This command, and every test passes (12 and 12):

  ```sh
  cargo test -p memoryd --test honcho_ledger --test honcho_ledger_file
  ```
- [ ] **6. The runbook.** From the repository root:

  ```sh
  { echo; cat docs/plans/M5c1/files/runbook-honcho-ledger-damaged.md; } >> docs/runbook.md
  ```

  Then `grep -c '^## honcho-ledger-damaged$' docs/runbook.md` prints `1`.
- [ ] **7. Check.** Four commands:
  - `grep -c 'sync_data' crates/memoryd/src/honcho/ledger.rs` prints `2` (in `write_line` and
    in `cut`), and `grep -c 'write_all' crates/memoryd/src/honcho/ledger.rs` prints `1`.
  - `grep -rn 'runbook.md#honcho-ledger-damaged' crates/memoryd/src` prints exactly one line,
    the `RUNBOOK` constant in `honcho/ledger.rs`.
  - `grep -n 'create_dir' crates/memoryd/src/honcho/ledger.rs` prints nothing: the ledger never
    makes its directory.
  - `git diff --stat -- crates/bxcore crates/loopd crates/brokerd crates/gatewayd crates/bxctl
    Cargo.toml crates/memoryd/Cargo.toml` prints nothing.
- [ ] **8. Run the gate.** `cargo fmt --all`, then `make gate`, with a shell timeout of 600000 ms.
  Expected last line: `gate: ok`, with 1212 tests: 1188 before this task, plus 24 new ones (12
  in `honcho_ledger.rs`, 12 in `honcho_ledger_file.rs`).
- [ ] **9. Log and commit.** Add your row to `docs/implementer-log.md`; its Task cell is
  `M5c1/07-memoryd-honcho-ledger`. `git status --short` lists `crates/memoryd/src/honcho/mod.rs`,
  `docs/runbook.md` and your log as modified, and `crates/memoryd/src/honcho/ledger.rs`, the two
  test files and `crates/memoryd/tests/support/ledger.rs` as new. Stage with
  `git add crates/memoryd docs/runbook.md docs/implementer-log.md Cargo.lock`, then commit as
  `AGENTS.md` says. `Cargo.lock` does not change in this task; the `git add` is harmless.

## Done when

- `make gate` prints `gate: ok` with 1212 tests.
- `git diff HEAD~1 --stat` lists only `honcho/mod.rs`, `honcho/ledger.rs`, the two new test
  files, `tests/support/ledger.rs`, `docs/runbook.md` and `docs/implementer-log.md`.

## Stop and report if

- The line of edit H1 is not in `crates/memoryd/src/honcho/mod.rs` exactly as shown, or is there
  more than once, or `crates/memoryd/tests/support/ledger.rs` exists before step 1.
- Step 2 or 3 shows a different result: other test targets, errors of another kind, other
  warnings, or other tests passing or failing.
- `cargo check` reports an error after a body you filled, and the comment does not tell you how
  to fix it. Quote the error, and stop.
- A comment in the skeleton does not fit the code under it, or disagrees with this task's text.
  Quote both and stop.
- After step 4 a test fails although each body follows its comment. Do not change a test, the
  support file or the skeleton's given code (`open`, `append`, the types, `Display`): stop, and
  give the failing tests' names and their messages.
- The fix seems to need a change anywhere else: another `memoryd` file, another crate, a
  dependency, or a change to a derive or a serde attribute. It does not; stop and say why you
  think so.
- A test outside these two files fails in the gate: check once whether it fails without your
  change, then stop and report both results.
- The gate passes with a count other than 1212.

## What the given tests pin

Each of these changes to a finished `ledger.rs` makes at least one given test fail: the outbox
taking in a skipped, sent or failed item, or losing the decision order; `conclusion` ignoring an
`Unconclude`, the workspace, the order of the sends, or a `Sent` with no ids, or answering the
oldest; a second decision, or a `Sent` or `Failed` before the decision, after an outcome or for
a `Skip`, accepted; the state taken in before the write; the directory made; the mode; a cut-off
line kept, not cut, cut before the other lines are checked, or not reported by `repaired`; line
numbers from 0; replay without the rules; a missing file an error or an unreadable one empty; any
error text; the counts; a serde attribute removed. Two writes for one line, and a missing
`sync_data`, pass the tests; step 7 checks those.