diff --git a/docs/plans/M5d1/04-memoryd-honcho-epochs.md b/docs/plans/M5d1/04-memoryd-honcho-epochs.md index be45103..00745c6 100644 --- a/docs/plans/M5d1/04-memoryd-honcho-epochs.md +++ b/docs/plans/M5d1/04-memoryd-honcho-epochs.md @@ -20,7 +20,7 @@ decided and sent. | Place | What happens there | |---|---| | `crates/memoryd/src/honcho/turns.rs` | `epoch_logs`; `read_session`; `records` and `resume` use it | -| `crates/memoryd/src/honcho/worker.rs` | `send_turn` reads the concatenated log | +| `crates/memoryd/src/honcho/worker.rs` | `send_turn` reads the concatenated log (W1–W3) | | `crates/memoryd/tests/honcho_epochs.rs` | new, given | | everything else | **no change**: not `log_path`'s return value, not `decide.rs` | @@ -45,7 +45,9 @@ decided and sent. stable. - **`send_turn` today** calls `read_turn(&log_path(...), turn)`, which only sees epoch 0. After this task it loads `read_session`, then `turn_records` / `user_text` / `final_reply` from - `crate::honcho::decide` (already `pub`). + `crate::honcho::decide` (already `pub`). `read_turn` maps a damaged line to `Ok(None)` (the + turn is marked failed, kind `log`). Keep that: `LogError::Damaged` is `Unsent::Local("log")`; + an I/O error stays `Unsent::Unreadable` and is retried. - `turns.rs` after this task stays under 500 lines (it is 332 now). - The items the edits use. This table is complete. @@ -159,7 +161,7 @@ becomes ```rust use crate::honcho::decide::{final_reply, turn_records, user_text}; -use crate::honcho::turns::{TurnSync, log_path, read_session}; +use crate::honcho::turns::{LogError, TurnSync, log_path, read_session}; ``` **W2.** The body of `send_turn`, the lines @@ -180,6 +182,7 @@ become self.readable(&path); records } + Err(LogError::Damaged(_)) => return Err(Unsent::Local("log")), Err(e) => return Err(Unsent::Unreadable(e.to_string())), }; let slice = turn_records(&records, turn).ok_or(Unsent::Local("log"))?; @@ -187,9 +190,20 @@ become let reply = final_reply(slice).ok_or(Unsent::Local("log"))?; ``` -The rest of `send_turn` is unchanged (`honcho.ok_or`, `batch`, `MAX_BATCH`, `add_messages`). -`text` is `&str` from `user_text`; `reply` is `&str` from `final_reply`. `batch` already takes -`&str`. +**W3.** The `batch` call, the line + +```rust + let messages = batch(&text, &reply, speaker, &self.agent, session, turn); +``` + +becomes + +```rust + let messages = batch(text, reply, speaker, &self.agent, session, turn); +``` + +`text` and `reply` are already `&str`; clippy denies the extra borrows (`needless_borrow`). +The rest of `send_turn` is unchanged (`honcho.ok_or`, `MAX_BATCH`, `add_messages`). ## Steps @@ -202,31 +216,32 @@ The rest of `send_turn` is unchanged (`honcho.ok_or`, `batch`, `MAX_BATCH`, `add - [ ] **2. See it fail.** `cargo build --tests --keep-going -p memoryd`, timeout 600000 ms: test target `honcho_epochs` does not compile (`E0432` unresolved import `memoryd::honcho::turns::read_session`). Every other target compiles. -- [ ] **3. The edits.** H1, H1b, H2, H3, W1, W2. `cargo check -p memoryd` +- [ ] **3. The edits.** H1, H1b, H2, H3, W1, W2, W3. `cargo check -p memoryd` prints no error and no warning. `wc -l crates/memoryd/src/honcho/turns.rs` and `wc -l crates/memoryd/src/honcho/worker.rs` are each ≤ 500. - [ ] **4. The tests.** `cargo test -p memoryd --test honcho_epochs --test honcho_turns --test - honcho_worker`: every test passes, 5 in `honcho_epochs` and the others as before. + honcho_worker`: every test passes, 6 in `honcho_epochs` and the others as before. - [ ] **5. Check.** `git diff --stat` lists exactly `crates/memoryd/src/honcho/turns.rs` and `crates/memoryd/src/honcho/worker.rs`. - [ ] **6. Run the gate.** `cargo fmt --all`, then `make gate`, timeout 600000 ms. Last line - `gate: ok`. Test count **1483** (1478 before, plus 5). + `gate: ok`. Test count **1484** (1478 before, plus 6). - [ ] **7. Log and commit.** Task cell `M5d1/04-memoryd-honcho-epochs`. Stage with `git add crates/memoryd docs/implementer-log.md Cargo.lock`, then commit as `AGENTS.md` says. ## Done when -- `make gate` prints `gate: ok` with 1483 tests. +- `make gate` prints `gate: ok` with 1484 tests. - `git diff HEAD~1 --stat` lists only the two `honcho` sources, `honcho_epochs.rs`, and the log. ## Stop and report if - The branch check fails; a "before" block is not found exactly once; step 2 differs; a file - exceeds 500 lines; a test fails; the gate count is not 1483. + exceeds 500 lines; a test fails; the gate count is not 1484. - You think `log_path` should return every file. It must still return `0.jsonl`. ## What the given tests pin Two users in epoch 0 and one in epoch 1 is `turns() == 3` and `turn_records(..., 3)` is the epoch-1 user; epoch 0 alone still reads; a damaged `1.jsonl` names that file; `TurnSync` decides -turn 3 from epoch 1; a missing session directory is empty. +turn 3 from epoch 1; `send_turn` posts turn 3 from epoch 1 (three worker steps, oldest first); +a missing session directory is empty. diff --git a/docs/plans/M5d1/05-loopd-compact-due.md b/docs/plans/M5d1/05-loopd-compact-due.md index 50f25cb..ab8b2d1 100644 --- a/docs/plans/M5d1/05-loopd-compact-due.md +++ b/docs/plans/M5d1/05-loopd-compact-due.md @@ -107,21 +107,21 @@ moves the line, accept that). `crates/loopd/src/compact.rs`, `config/compact.md`. `git status --short` shows `compact_due.rs` as new. - [ ] **6. Run the gate.** `cargo fmt --all`, then `make gate`, timeout 600000 ms. Last line - `gate: ok`. Test count **1489** (1483 before, plus 6). + `gate: ok`. Test count **1490** (1484 before, plus 6). - [ ] **7. Log and commit.** Task cell `M5d1/05-loopd-compact-due`. Stage with `git add crates/loopd config/compact.md docs/implementer-log.md Cargo.lock`, then commit as `AGENTS.md` says. ## Done when -- `make gate` prints `gate: ok` with 1489 tests. +- `make gate` prints `gate: ok` with 1490 tests. - `git diff HEAD~1 --stat` lists only `crates/loopd/src/lib.rs`, `crates/loopd/src/compact.rs`, `crates/loopd/tests/compact_due.rs`, `config/compact.md` and the log. ## Stop and report if - The branch check fails; step 2 fails to compile for a reason other than the missing module - or `todo!()`; a test fails; the gate count is not 1489; you add an idle timer. There is none. + or `todo!()`; a test fails; the gate count is not 1490; you add an idle timer. There is none. ## What the given tests pin diff --git a/docs/plans/M5d1/06-loopd-compact-summary.md b/docs/plans/M5d1/06-loopd-compact-summary.md index 40cb51b..bd01985 100644 --- a/docs/plans/M5d1/06-loopd-compact-summary.md +++ b/docs/plans/M5d1/06-loopd-compact-summary.md @@ -186,21 +186,21 @@ pub fn summarise(cfg: &Config, client: &Client, session: &Session) -> Result 0` is the new system message, then one `ChatMessage::User` (fixed heading plus carry, not a log record), then the new epoch's records; `run_turn` uses `session.messages()` | loopd | 1478 | 5 | -| 04 | Honcho concatenates every `.jsonl` in epoch order; `turn_records` of turn 3 is the epoch-1 user; `send_turn` reads the concatenated log | memoryd | 1483 | 5 | -| 05 | `loopd::compact`: `due`, `fallback_summary`; starter `config/compact.md` | loopd | 1489 | 6 | -| 06 | Summary call on the background slot, no tools, thinking on; tag-neutralised and capped; context-full uses the fallback; `session-compact-failed` pointer and runbook entry | loopd | 1495 | 6 | -| 07 | Writing the next epoch: `EpochEnd`, fresh baseline from `memoryd`, `SessionStart` on the live slot; resume if `EpochEnd` is already the last record | loopd | 1501 | 6 | -| 08 | After a turn: compact when `due`; on `SessionFull`, compact and retry the user message once; compaction holds the session busy | loopd | 1507 | 6 | +| # | Delivers | Crates | Gate after | Tests added | Mutations tried, caught | +|---|---|---|---|---|---| +| 01 | `[compact]` on `loopd.toml`: `threshold` 0.85, `summary_chars` 4000, `summarise` default `["http_fetch"]` (stored, unused); `Config::compact_prompt` is `compact.md` beside `system.md` | loopd | 1464 | 8 | 5, 5 | +| 02 | `Session::open` loads the highest epoch that has both a baseline and a log, the carry from the previous `EpochEnd`, `turns` / `untrusted` / `next_call` over every epoch; persist writes the current epoch | loopd | 1473 | 9 | 5, 4 (`log_path` after `open` does not move the already-open `File`) | +| 03 | `messages_with_carry`: epoch `n > 0` is the new system message, then one `ChatMessage::User` (fixed heading plus carry, not a log record), then the new epoch's records; `run_turn` uses `session.messages()` | loopd | 1478 | 5 | 2, 2 | +| 04 | Honcho concatenates every `.jsonl` in epoch order; `turn_records` of turn 3 is the epoch-1 user; `send_turn` reads the concatenated log | memoryd | 1484 | 6 | 3, 3 | +| 05 | `loopd::compact`: `due`, `fallback_summary`; starter `config/compact.md` | loopd | 1490 | 6 | 3, 3 | +| 06 | Summary call on the background slot, no tools, thinking on; tag-neutralised and capped; context-full uses the fallback; `session-compact-failed` pointer and runbook entry | loopd | 1496 | 6 | 3, 3 | +| 07 | Writing the next epoch: `EpochEnd`, fresh baseline from `memoryd`, `SessionStart` on the live slot; resume if `EpochEnd` is already the last record | loopd | 1502 | 6 | 2, 2 | +| 08 | After a turn: compact when `due`; on `SessionFull`, compact and retry the user message once; compaction holds the session busy | loopd | 1508 | 6 | 2, 2 (`finish_turn` after `drop(held)` is wiring in `channel.rs`; the given tests call `finish_turn` directly) | After task 08, the design model runs the M5d-1 device check (spec 7) on straylight: a `bxctl` session writes a distinctive fact into `memory.md`, continues until compaction with @@ -116,4 +116,4 @@ OPENCODE_MODEL=straylight/ornith-1.5-35b-a3b RESUME=1 \ tools/run-plan.sh docs/plans/M5d1 01 ``` -After task 08, the gate count is 1507. Then the device check above. +After task 08, the gate count is 1508. Then the device check above. diff --git a/docs/plans/M5d1/files/crates/loopd/tests/compact_summary.rs b/docs/plans/M5d1/files/crates/loopd/tests/compact_summary.rs index a61c535..9442d16 100644 --- a/docs/plans/M5d1/files/crates/loopd/tests/compact_summary.rs +++ b/docs/plans/M5d1/files/crates/loopd/tests/compact_summary.rs @@ -106,7 +106,7 @@ fn tags_in_the_completion_are_neutralised() { let client = Client::new(cfg.clone()); let s = session(&home, &cfg); let text = compact::summarise(&cfg, &client, &s).unwrap(); - assert!(!text.contains(&tag), "{text}"); + assert!(!text.contains(tag), "{text}"); assert!(text.contains("hello"), "{text}"); } diff --git a/docs/plans/M5d1/files/crates/loopd/tests/compact_turn.rs b/docs/plans/M5d1/files/crates/loopd/tests/compact_turn.rs index 856e1cb..1612189 100644 --- a/docs/plans/M5d1/files/crates/loopd/tests/compact_turn.rs +++ b/docs/plans/M5d1/files/crates/loopd/tests/compact_turn.rs @@ -3,7 +3,7 @@ mod support; use bxcore::{Epoch, LogRecord, SessionId, TurnEvent}; -use loopd::baseline::Baseline; +use loopd::baseline::{Baseline, CARRY_HEADING}; use loopd::compact; use loopd::llama::Client; use loopd::memory::NoMemory; @@ -125,11 +125,31 @@ fn usage_at_the_threshold_compacts_before_the_turn_returns() { w.finish(&mut s, "hi", outcome).unwrap(); assert_eq!(s.epoch(), Epoch(1)); assert_eq!(s.carry(), Some("the sum")); + w.server.route(CHAT, vec![sse("next", 0, 20)]); + let mut events = Vec::new(); + run_turn(&mut s, &w.runtime(), "again", None, &mut |e| { + events.push(e.clone()) + }) + .unwrap(); let sent = w.server.requests_to(CHAT); - assert_eq!(sent.len(), 2); + assert_eq!(sent.len(), 3); assert_eq!(sent[0].json()["id_slot"], 0); assert_eq!(sent[1].json()["id_slot"], 1); assert!(sent[1].json().get("tools").is_none()); + let live = sent[2].json(); + assert_eq!(live["id_slot"], 0); + let contents: Vec<&str> = live["messages"] + .as_array() + .unwrap() + .iter() + .map(|m| m["content"].as_str().unwrap_or("")) + .collect(); + assert!( + contents + .iter() + .any(|c| c.contains(CARRY_HEADING) && c.contains("the sum")), + "{contents:?}" + ); let old = w.home.records("a"); assert!(matches!( old.last(), diff --git a/docs/plans/M5d1/files/crates/loopd/tests/session_epochs.rs b/docs/plans/M5d1/files/crates/loopd/tests/session_epochs.rs index 7716b80..565ead3 100644 --- a/docs/plans/M5d1/files/crates/loopd/tests/session_epochs.rs +++ b/docs/plans/M5d1/files/crates/loopd/tests/session_epochs.rs @@ -236,6 +236,10 @@ fn an_incomplete_next_epoch_is_ignored() { assert_eq!(s.epoch(), Epoch(0)); assert_eq!(s.carry(), None); assert_eq!(s.turns(), 1); + drop(s); + write_jsonl(&home, "a", 2, &[start("a", 2, 0, &later())]); + let s = Session::open(&home.dir, id("a")).unwrap(); + assert_eq!(s.epoch(), Epoch(0), "a log without its baseline is ignored"); } #[test] diff --git a/docs/plans/M5d1/files/crates/memoryd/tests/honcho_epochs.rs b/docs/plans/M5d1/files/crates/memoryd/tests/honcho_epochs.rs index 5be528e..0b00eb8 100644 --- a/docs/plans/M5d1/files/crates/memoryd/tests/honcho_epochs.rs +++ b/docs/plans/M5d1/files/crates/memoryd/tests/honcho_epochs.rs @@ -1,21 +1,28 @@ //! Honcho reads every epoch file of a session, in order. Do not edit. +#[path = "support/honcho.rs"] +mod honcho; #[path = "support/logs.rs"] mod logs; #[path = "support/tmp.rs"] mod tmp; #[path = "support/turns.rs"] mod turns; +#[path = "support/worker.rs"] +mod worker; use std::cell::RefCell; use bxcore::{Epoch, LogRecord, SessionId, Timestamp}; +use honcho::FakeHoncho; use logs::{assistant, write_log}; use memoryd::honcho::decide::{turn_records, turns}; use memoryd::honcho::ledger::{Action, Item, Ledger, Record, Workspace}; use memoryd::honcho::turns::{LOG_RUNBOOK, TurnSync, read_session}; +use memoryd::honcho::worker::Step; use tmp::TempDir; use turns::{clean, sid, spoken}; +use worker::{FAST, Home as WorkerHome}; const LEDGER: &str = "honcho.jsonl"; @@ -85,10 +92,10 @@ impl World { } } -fn to(workspace: Workspace, peer: &str) -> Action { - Action::To { +fn to(workspace: Workspace, speaker: &str) -> Action { + Action::Turn { workspace, - peer: peer.to_string(), + speaker: speaker.to_string(), } } @@ -108,10 +115,7 @@ fn concatenated_logs_count_three_turns() { home, "mm-a", 1, - &[ - spoken("three", Some("kyle")), - assistant(4, Some("ok"), &[]), - ], + &[spoken("three", Some("kyle")), assistant(4, Some("ok"), &[])], ); let records = read_session(&home.join("sessions"), &sid("mm-a")).unwrap(); assert_eq!(turns(&records), 3); @@ -164,10 +168,7 @@ fn turn_sync_decides_a_turn_that_lives_in_epoch_1() { w.dir.path(), "mm-a", 1, - &[ - spoken("three", Some("kyle")), - assistant(4, Some("ok"), &[]), - ], + &[spoken("three", Some("kyle")), assistant(4, Some("ok"), &[])], ); w.updated("mm-a", 3); assert_eq!(w.action("mm-a", 3), Some(to(Workspace::Trusted, "kyle"))); @@ -179,3 +180,46 @@ fn a_missing_session_directory_is_empty() { let records = read_session(&dir.path().join("sessions"), &sid("mm-none")).unwrap(); assert!(records.is_empty()); } + +#[test] +fn send_turn_reads_a_turn_that_lives_in_epoch_1() { + let fake = FakeHoncho::start(); + let home = WorkerHome::new("honcho-epochs-send"); + write_log( + home.dir.path(), + "mm-a", + &[ + spoken("one", Some("kyle")), + assistant(2, Some("ok"), &[]), + spoken("two", Some("kyle")), + assistant(3, Some("ok"), &[]), + epoch_end("sum"), + ], + ); + write_epoch( + home.dir.path(), + "mm-a", + 1, + &[spoken("three", Some("kyle")), assistant(4, Some("ok"), &[])], + ); + home.report("mm-a", 3); + let mut w = home.worker(&fake, FAST); + assert_eq!(w.step(), Step::Done); + assert_eq!(w.step(), Step::Done); + assert_eq!(w.step(), Step::Done); + let posts: Vec<_> = fake + .requests() + .into_iter() + .filter(|r| r.path.ends_with("/messages")) + .collect(); + assert_eq!(posts.len(), 3, "{posts:?}"); + let body = posts[2].body.as_ref().expect("message batch"); + let contents: Vec<&str> = body["messages"] + .as_array() + .unwrap() + .iter() + .map(|m| m["content"].as_str().unwrap()) + .collect(); + assert_eq!(contents, ["three", "ok"]); + assert_eq!(body["messages"][0]["metadata"]["turn"], 3); +}