boxmaker/docs/plans/M5d1/04-memoryd-honcho-epochs.md

M5d-1 task 04: Honcho reads every epoch file

Branch: m5d1. Check it: git branch --show-current prints m5d1 and git status --short is empty; otherwise stop. Never switch branch. Commit subject: memoryd: Honcho concatenates every epoch of a session Log row: the Task cell is M5d1/04-memoryd-honcho-epochs.

Goal

memoryd keys a turn as Item::Turn { session, turn } with no epoch (spec 3.2). After compaction, turn 3 may live in 1.jsonl. log_path still names 0.jsonl (existing tests pin that path). A new read_session concatenates every <n>.jsonl in epoch order. TurnSync walks that concatenation, and the worker's send_turn does too, so a turn in epoch 1 is decided and sent.

turn_records / turns do not change: they already count User records on a slice.

Every place that changes

Place	What happens there
crates/memoryd/src/honcho/turns.rs	epoch_logs; read_session; records and resume use it
crates/memoryd/src/honcho/worker.rs	send_turn reads the concatenated log (W1–W3)
crates/memoryd/tests/honcho_epochs.rs	new, given
everything else	no change: not log_path's return value, not decide.rs

Files

• Copy, from docs/plans/M5d1/files/ (the same path under it): the new test crates/memoryd/tests/honcho_epochs.rs.
• Modify: crates/memoryd/src/honcho/turns.rs, crates/memoryd/src/honcho/worker.rs, docs/implementer-log.md.
• Do not touch: any other file.

Facts you need

• read_records / LogError already exist in turns.rs. read_session returns Result<Vec<LogRecord>, LogError> so a damaged line still names its file and LOG_RUNBOOK.
• Listing. Under sessions/<id>/, every file named <n>.jsonl where n parses as u32, sorted by n. A missing session directory is empty, not an error (same as a missing 0.jsonl today).
• log_path stays .../0.jsonl. Do not change its body. readable in the worker still names that path when the concatenated read succeeds, so existing recovery log lines stay stable.
• send_turn today calls read_turn(&log_path(...), turn), which only sees epoch 0. After this task it loads read_session, then turn_records / user_text / final_reply from crate::honcho::decide (already pub). read_turn maps a damaged line to Ok(None) (the turn is marked failed, kind log). Keep that: LogError::Damaged is Unsent::Local("log"); an I/O error stays Unsent::Unreadable and is retried.
• turns.rs after this task stays under 500 lines (it is 332 now).
• The items the edits use. This table is complete.

  Item	What it does
  read_records(&Path) -> Result<Vec<LogRecord>, LogError>	one file, already in turns.rs
  turn_records(&[LogRecord], u64) -> Option<&[LogRecord]>	decide.rs
  user_text(&[LogRecord]) -> (&str, Option<&str>)	decide.rs
  final_reply(&[LogRecord]) -> Option<&str>	decide.rs
  str::strip_suffix(".jsonl"), str::parse::<u32>	epoch numbers
  Unsent::Unreadable(String)	worker, e.to_string()

The edits, exactly

crates/memoryd/src/honcho/turns.rs

H1. The doc comment on log_path, the lines

/// The path of `session`'s log under `sessions`: epoch 0, the only one `loopd` writes.
pub fn log_path(sessions: &Path, session: &SessionId) -> PathBuf {

become

/// The path of `session`'s epoch-0 log under `sessions`. Later epochs are sibling files;
/// `read_session` concatenates them. Existing tests pin this path.
pub fn log_path(sessions: &Path, session: &SessionId) -> PathBuf {

H1b. After read_records (the function that ends Ok(records) / }), insert these two functions. They sit below LogError and read_records, so both names resolve.

/// Every `<n>.jsonl` of `session`, sorted by `n`. A missing directory is empty.
pub fn epoch_logs(sessions: &Path, session: &SessionId) -> Result<Vec<PathBuf>, LogError> {
    let dir = sessions.join(session.as_str());
    let entries = match std::fs::read_dir(&dir) {
        Ok(e) => e,
        Err(e) if e.kind() == std::io::ErrorKind::NotFound => return Ok(Vec::new()),
        Err(e) => return Err(LogError::Io(format!("{}: {e}", dir.display()))),
    };
    let mut found: Vec<(u32, PathBuf)> = Vec::new();
    for entry in entries {
        let entry = match entry {
            Ok(e) => e,
            Err(e) => return Err(LogError::Io(format!("{}: {e}", dir.display()))),
        };
        let name = entry.file_name();
        let Some(name) = name.to_str() else { continue };
        let Some(stem) = name.strip_suffix(".jsonl") else { continue };
        let Ok(n) = stem.parse::<u32>() else { continue };
        found.push((n, entry.path()));
    }
    found.sort_by_key(|(n, _)| *n);
    Ok(found.into_iter().map(|(_, p)| p).collect())
}

/// Every epoch's records, in epoch order. A missing session has none.
pub fn read_session(
    sessions: &Path,
    session: &SessionId,
) -> Result<Vec<LogRecord>, LogError> {
    let mut out = Vec::new();
    for path in epoch_logs(sessions, session)? {
        out.extend(read_records(&path)?);
    }
    Ok(out)
}

The next function after this insert is finished. Do not change finished.

H2. In resume, the line

            if let Ok(records) = read_log(&log_path(&self.sessions, &session)) {

becomes

            if let Ok(records) = read_session(&self.sessions, &session) {

H3. In records, the line

        match read_log(&log_path(&self.sessions, session)) {

becomes

        match read_session(&self.sessions, session).map_err(|e| e.to_string()) {

read_log already mapped LogError through Display; this keeps the logged line the same shape (memoryd: {why}; {LOG_RUNBOOK}).

crates/memoryd/src/honcho/worker.rs

W1. The use line

use crate::honcho::turns::{TurnSync, log_path};

becomes

use crate::honcho::decide::{final_reply, turn_records, user_text};
use crate::honcho::turns::{LogError, TurnSync, log_path, read_session};

W2. The body of send_turn, the lines

        let path = log_path(&self.sessions, session);
        let read = read_turn(&path, turn).map_err(Unsent::Unreadable)?;
        self.readable(&path);
        let (text, reply) = read.ok_or(Unsent::Local("log"))?;

become

        let path = log_path(&self.sessions, session);
        let records = match read_session(&self.sessions, session) {
            Ok(records) => {
                self.readable(&path);
                records
            }
            Err(LogError::Damaged(_)) => return Err(Unsent::Local("log")),
            Err(e) => return Err(Unsent::Unreadable(e.to_string())),
        };
        let slice = turn_records(&records, turn).ok_or(Unsent::Local("log"))?;
        let (text, _) = user_text(slice);
        let reply = final_reply(slice).ok_or(Unsent::Local("log"))?;

W3. The batch call, the line

        let messages = batch(&text, &reply, speaker, &self.agent, session, turn);

becomes

        let messages = batch(text, reply, speaker, &self.agent, session, turn);

text and reply are already &str; clippy denies the extra borrows (needless_borrow). The rest of send_turn is unchanged (honcho.ok_or, MAX_BATCH, add_messages).

Steps

• 1. Copy the given test.

  cp docs/plans/M5d1/files/crates/memoryd/tests/honcho_epochs.rs crates/memoryd/tests/
  

• 2. See it fail. cargo build --tests --keep-going -p memoryd, timeout 600000 ms: test target honcho_epochs does not compile (E0432 unresolved import memoryd::honcho::turns::read_session). Every other target compiles.

• 3. The edits. H1, H1b, H2, H3, W1, W2, W3. cargo check -p memoryd prints no error and no warning. wc -l crates/memoryd/src/honcho/turns.rs and wc -l crates/memoryd/src/honcho/worker.rs are each ≤ 500.

• 4. The tests. cargo test -p memoryd --test honcho_epochs --test honcho_turns --test honcho_worker: every test passes, 6 in honcho_epochs and the others as before.

• 5. Check. git diff --stat lists exactly crates/memoryd/src/honcho/turns.rs and crates/memoryd/src/honcho/worker.rs.

• 6. Run the gate. cargo fmt --all, then make gate, timeout 600000 ms. Last line gate: ok. Test count 1484 (1478 before, plus 6).

• 7. Log and commit. Task cell M5d1/04-memoryd-honcho-epochs. Stage with git add crates/memoryd docs/implementer-log.md Cargo.lock, then commit as AGENTS.md says.

Done when

• make gate prints gate: ok with 1484 tests.
• git diff HEAD~1 --stat lists only the two honcho sources, honcho_epochs.rs, and the log.

Stop and report if

• The branch check fails; a "before" block is not found exactly once; step 2 differs; a file exceeds 500 lines; a test fails; the gate count is not 1484.
• You think log_path should return every file. It must still return 0.jsonl.

What the given tests pin

Two users in epoch 0 and one in epoch 1 is turns() == 3 and turn_records(..., 3) is the epoch-1 user; epoch 0 alone still reads; a damaged 1.jsonl names that file; TurnSync decides turn 3 from epoch 1; send_turn posts turn 3 from epoch 1 (three worker steps, oldest first); a missing session directory is empty.