diff --git a/docs/plans/M5d1/01-loopd-compact-config.md b/docs/plans/M5d1/01-loopd-compact-config.md new file mode 100644 index 0000000..0fd7036 --- /dev/null +++ b/docs/plans/M5d1/01-loopd-compact-config.md @@ -0,0 +1,209 @@ +# M5d-1 task 01: `[compact]` on `loopd.toml` + +**Branch:** `m5d1`. Check it: `git branch --show-current` prints `m5d1` and `git status --short` +is empty; otherwise stop. Never switch branch. +**Commit subject:** `loopd: [compact] threshold, summary_chars and summarise` +**Log row:** the Task cell is `M5d1/01-loopd-compact-config`. + +## Goal + +M5d-1 compact after a turn whose last `Usage` is at or above a fraction of `expect.n_ctx` (spec +3.3, 5). This task adds only the `[compact]` table so later tasks can read it: + +- `threshold` (float, default `0.85`) +- `summary_chars` (default `4000`) +- `summarise` (list of tool names, default `["http_fetch"]`) + +Nothing in this task uses the values. `summarise` is stored so a spec-complete `loopd.toml` +loads; M5d-2 reads it. Unknown keys in `[compact]` are errors, as in every table we define. + +`Config::compact_prompt` is `compact.md` beside `[baseline] system` (the same directory join +`load` already does for `system.md`). The starter file is copied in task 05. + +## Every place that changes + +This is the complete list. + +| Place | What happens there | +|---|---| +| `crates/loopd/src/config.rs` | `Compact`; `Config.compact`; `Config::compact_prompt` (edits C1 to C3) | +| `crates/loopd/tests/config_compact.rs` | new, given | +| everything else | **no change**: not `session.rs`, not the existing `config.rs` tests or fixtures | + +## Files + +- Copy, from `docs/plans/M5d1/files/` (the same path under it): the new test + `crates/loopd/tests/config_compact.rs`. +- Modify: `crates/loopd/src/config.rs`, `docs/implementer-log.md`. +- Do not touch: any other file. + +## Facts you need + +- **`Config` already has `#[serde(default)]` on optional tables** (`sampling`, `limits`, `paths`, + `channel`, `loop`, `baseline`, `broker`, `memory`). `[compact]` is the same: missing means + `Compact::default()`. +- **`threshold` is `f64`.** `Sampling.temperature` is already `f64` on `Config`, so `PartialEq` + on `Config` still compiles. Do not add a range check: the spec does not name one. +- **`summarise` default.** `vec!["http_fetch".to_string()]`. An explicit `summarise = []` is + allowed and is empty. Unknown names are not checked here (M5d-2). +- **`compact_prompt`.** `self.baseline.system.with_file_name("compact.md")`. After `load`, + `baseline.system` is already joined to the config file's directory, so the prompt follows it + into a subdirectory when `system` does (`prompts/agent.md` → `prompts/compact.md`). +- The items the edits use. This table is complete; `serde` is already a dependency of `loopd`. + + | Item | What it does | + |---|---| + | `#[serde(deny_unknown_fields, default)]` | on `Compact`: missing table uses `Default`; unknown keys error | + | `Path::with_file_name` | `compact.md` beside `system.md` | + | `Vec` | `summarise` | + +## The edits, exactly + +Nothing else in this file changes. Each quoted "before" block occurs exactly once in its file +when its turn comes (make them in order). Keep the fields in the order shown. + +### `crates/loopd/src/config.rs` + +**C1.** After the `Memory` struct (the lines) + +```rust +pub struct Memory { + pub socket: std::path::PathBuf, +} +``` + +become + +```rust +pub struct Memory { + pub socket: std::path::PathBuf, +} + +/// `[compact]`: when a session starts a new epoch, and how long a summary may be (M5d spec, +/// section 5). `summarise` is stored here and unused until M5d-2. +#[derive(Debug, Clone, PartialEq, Deserialize)] +#[serde(deny_unknown_fields, default)] +pub struct Compact { + pub threshold: f64, + pub summary_chars: usize, + pub summarise: Vec, +} + +impl Default for Compact { + fn default() -> Self { + Self { + threshold: 0.85, + summary_chars: 4_000, + summarise: vec!["http_fetch".to_string()], + } + } +} +``` + +**C2.** The lines + +```rust + #[serde(default)] + pub memory: Memory, +} +``` + +become + +```rust + #[serde(default)] + pub memory: Memory, + #[serde(default)] + pub compact: Compact, +} +``` + +**C3.** After `channel_socket` (the lines) + +```rust + pub fn channel_socket(&self) -> std::path::PathBuf { + if self.channel.socket.as_os_str().is_empty() { + self.paths.home.join("run/loop/loop.sock") + } else { + self.channel.socket.clone() + } + } +} +``` + +become + +```rust + pub fn channel_socket(&self) -> std::path::PathBuf { + if self.channel.socket.as_os_str().is_empty() { + self.paths.home.join("run/loop/loop.sock") + } else { + self.channel.socket.clone() + } + } + + /// `compact.md` beside the system prompt: the summary call's last user message. + pub fn compact_prompt(&self) -> std::path::PathBuf { + self.baseline.system.with_file_name("compact.md") + } +} +``` + +## Steps + +- [ ] **1. Copy the given test.** From the repository root, one command: + + ```sh + cp docs/plans/M5d1/files/crates/loopd/tests/config_compact.rs crates/loopd/tests/ + ``` + + Then `git status --short` lists one new file, `crates/loopd/tests/config_compact.rs`. +- [ ] **2. See it fail.** `cargo build --tests --keep-going -p loopd`, with a shell timeout of + 600000 ms: one test target, `config_compact`, does not compile, with `E0609` (`no field + \`compact\` on type \`Config\``) and `E0599` (`no method named \`compact_prompt\`` / + `no variant or associated item named \`default\` for \`Compact\``). Every other target + compiles. +- [ ] **3. The edits.** Make edits C1 to C3, in order. `cargo check -p loopd` prints no error and + no warning. +- [ ] **4. The tests.** `cargo test -p loopd --test config_compact --test config`: every test + passes, 8 in `config_compact` and the others as before. +- [ ] **5. Check.** `git diff --stat` lists exactly `crates/loopd/src/config.rs`. +- [ ] **6. Run the gate.** `cargo fmt --all`, then `make gate`, with a shell timeout of 600000 ms. + Expected last line: `gate: ok`. The test count is **1464** after this task, 1456 before it + (1455 passed in `cargo test --workspace`, plus 1), and the 8 new ones are the tests of + `config_compact.rs`. To count, add up the `N passed` of every `test result:` line the gate's + `cargo test --workspace` printed, and add 1. Never run the gate again only to count. +- [ ] **7. Log and commit.** Add your row to `docs/implementer-log.md`; its Task cell is + `M5d1/01-loopd-compact-config`. Stage with + `git add crates/loopd docs/implementer-log.md Cargo.lock`, then commit as `AGENTS.md` says. + `Cargo.lock` does not change in this task; the `git add` is harmless. + +## Done when + +- `make gate` prints `gate: ok` with 1464 tests. +- `git diff HEAD~1 --stat` lists only `crates/loopd/src/config.rs`, + `crates/loopd/tests/config_compact.rs` and `docs/implementer-log.md`. + +## Stop and report if + +- The branch check at the top fails. +- A "before" block is not in its file exactly as shown when its turn comes, or is there more + than once. +- Step 2 shows a different result: another target failing, or other errors. +- `cargo check` in step 3 prints an error or a warning. Quote it, and stop. +- A test fails in step 4. Do not change the test: stop, and give the failing tests' names and + their messages. +- The fix seems to need a change anywhere else: a fixture, `config.rs` tests, a range check on + `threshold`. It does not; stop and say why you think so. +- A test outside `config_compact.rs` fails in the gate: check once whether it fails without your + change (`git stash -u`, run it, `git stash pop`), then stop and report both results. +- The gate passes with a count other than 1464. +- A comment in an edit does not fit the code under it. Quote it and stop. + +## What the given tests pin + +Each of these changes to the finished code makes at least one given test fail: the default +threshold moved; `summary_chars` default moved; `summarise` default emptied or given another +name; unknown keys accepted; a misspelt `summary_char` accepted; a partial table wiping the +other defaults; `compact_prompt` using `home` or the config file's name instead of `system.md`'s +directory; `Compact::default` disagreeing with a missing table. diff --git a/docs/plans/M5d1/02-loopd-session-epochs.md b/docs/plans/M5d1/02-loopd-session-epochs.md new file mode 100644 index 0000000..ae38750 --- /dev/null +++ b/docs/plans/M5d1/02-loopd-session-epochs.md @@ -0,0 +1,513 @@ +# M5d-1 task 02: `Session::open` loads the current epoch + +**Branch:** `m5d1`. Check it: `git branch --show-current` prints `m5d1` and `git status --short` +is empty; otherwise stop. Never switch branch. +**Commit subject:** `loopd: open the highest epoch and carry its summary` +**Log row:** the Task cell is `M5d1/02-loopd-session-epochs`. + +## Goal + +A session directory may have `0.jsonl` and, after compaction, `1.baseline.json` / `1.jsonl` and +so on (spec 3.1). `Session::open` loads the highest epoch that has **both** a baseline file and a +log. For epoch `n > 0` the previous log must end with `EpochEnd { next: n, summary }`; that +summary is the carry. `turns()` and `untrusted()` span every epoch. `next_call` is one more than +the highest `ToolResult` call in any epoch. `append` writes the current epoch's log. + +`records()` stays the current epoch only: later tasks replay that file (plus the carry) as the +request. + +## Every place that changes + +This is the complete list. + +| Place | What happens there | +|---|---| +| `crates/loopd/src/session.rs` | `NoCarry`; fields `epoch`, `slot`, `carry`, `prior_users`, `prior_untrusted`; `open`; `turns`; `untrusted`; `log_path`; accessors; helpers | +| `crates/loopd/tests/session_epochs.rs` | new, given | +| everything else | **no change**: not `session.rs`'s existing tests, not `turn.rs`, not Honcho | + +## Files + +- Copy, from `docs/plans/M5d1/files/` (the same path under it): the new test + `crates/loopd/tests/session_epochs.rs`. +- Modify: `crates/loopd/src/session.rs`, `docs/implementer-log.md`. +- Do not touch: any other file. + +## Facts you need + +- **`Epoch` is `pub struct Epoch(pub u32)`** in `bxcore`. Format a file name with + `format!("{}.jsonl", epoch.0)` and `format!("{}.baseline.json", epoch.0)`. +- **A complete epoch** is both `.baseline.json` and `.jsonl` in the session directory. + A baseline without a log, or a log without a baseline, does not count. The highest complete + `n` is the current epoch. If none is complete, `open` is `NotFound`, as today when `0.jsonl` + is missing. +- **Carry.** For current epoch `n > 0`, read `(n-1).jsonl`. Among its records, the last + `EpochEnd` whose `next` is `Epoch(n)` gives `summary`. If that file is missing, or has no such + record, `SessionError::NoCarry { path }` naming `(n-1).jsonl`. Epoch 0 has `carry: None`. +- **`records` on the live `Session`** is only the current epoch's log, so `messages` in task 03 + can replay it. `prior_users` / `prior_untrusted` are counted from every **earlier** complete + epoch while `open` walks them. `turns()` is `prior_users` plus the `User` records in + `records`. `untrusted()` is `prior_untrusted` or any untrusted `ToolResult` in `records`. +- **`next_call`** is one more than the highest `ToolResult.call.0` in **every** complete epoch + (current included). A call id of 4 in epoch 0 and 2 in epoch 1 continues at 5. +- **`slot`** is the `slot` on the current epoch's `SessionStart`. `create` stores the `slot` + argument. If the current log has no `SessionStart`, use `0`. +- **`append` / `persist`** already write `self.log_path()`. Change `log_path` to the current + epoch's file; `create` still starts at `0.jsonl`. +- **A last `EpochEnd` on epoch 0 with no epoch-1 files** means compact stopped halfway: `open` + stays on epoch 0, `carry` is `None`, and the `EpochEnd` is in `records`. Task 07 resumes that. +- The torn-log checks stay as they are, per file, with the line number of that file. +- `session.rs` must stay at or under 500 lines (`wc -l`). After these edits it is about 430. +- The items the edits use. This table is complete. + + | Item | What it does | + |---|---| + | `std::fs::read_dir` | list `.jsonl` names | + | `str::strip_suffix(".jsonl")`, `str::parse::` | the epoch number | + | `Path::exists` | the matching baseline | + | `u64::try_from(count).unwrap_or(u64::MAX)` | `turns`, as today | + | `CallId(highest + 1)` | `next_call`, as today, over every epoch | + | `OpenOptions::new().append(true).open` | the current log | + +## The edits, exactly + +Nothing else in this file changes. Each quoted "before" block occurs exactly once. Keep struct +fields in the order shown. + +### `crates/loopd/src/session.rs` + +**S1.** The module doc comment, the lines + +```rust +//! A session on disk: `sessions//0.baseline.json` and `0.jsonl`. The log is the state — +//! create, open, append and resume all live here, and nothing is durable until `append` syncs it. +``` + +become + +```rust +//! A session on disk: `sessions//.baseline.json` and `.jsonl`. The log is the +//! state — create, open, append and resume all live here, and nothing is durable until `append` +//! syncs it. Epoch 0 is the first; later epochs appear after compaction. +``` + +**S2.** After `Encode(serde_json::Error),` in `SessionError`, add a variant (the lines) + +```rust + Encode(serde_json::Error), +} +``` + +become + +```rust + Encode(serde_json::Error), + NoCarry { + path: PathBuf, + }, +} +``` + +**S3.** In `Display`, the lines + +```rust + SessionError::Encode(err) => write!(f, "{err}"), + } +``` + +become + +```rust + SessionError::Encode(err) => write!(f, "{err}"), + SessionError::NoCarry { path } => write!( + f, + "{}: compacted epoch is missing its EpochEnd; see docs/runbook.md#session-log-damaged", + path.display() + ), + } +``` + +**S4.** The struct, the lines + +```rust +pub struct Session { + id: SessionId, + dir: PathBuf, + baseline: Baseline, + records: Vec, + log: File, + next_call: CallId, +} +``` + +become + +```rust +pub struct Session { + id: SessionId, + dir: PathBuf, + baseline: Baseline, + records: Vec, + log: File, + next_call: CallId, + epoch: Epoch, + slot: u32, + carry: Option, + prior_users: u64, + prior_untrusted: bool, +} +``` + +**S5.** In `create`, the lines + +```rust + let mut session = Session { + id, + dir, + baseline, + records: Vec::new(), + log, + next_call: CallId(1), + }; +``` + +become + +```rust + let mut session = Session { + id, + dir, + baseline, + records: Vec::new(), + log, + next_call: CallId(1), + epoch: Epoch(0), + slot, + carry: None, + prior_users: 0, + prior_untrusted: false, + }; +``` + +**S6.** Replace the whole `open` function. The lines from `pub fn open(home: &Path, id: SessionId)` +through the `Ok(Session {` block that ends `next_call: CallId(highest + 1),` and `})` `}` become +the function below. Fill any `todo!()` from its comment, then delete the comment. Each comment +is at most six lines. + +```rust + pub fn open(home: &Path, id: SessionId) -> Result { + let dir = Session::dir_for(home, &id); + let epochs = complete_epochs(&dir)?; + let Some(&n) = epochs.last() else { + return Err(SessionError::NotFound(id)); + }; + let epoch = Epoch(n); + let baseline_path = dir.join(format!("{n}.baseline.json")); + let baseline = Baseline::load(&baseline_path).map_err(SessionError::Baseline)?; + let log_path = dir.join(format!("{n}.jsonl")); + let records = load_records(&log_path)?; + let (carry, prior_users, prior_untrusted) = priors(&dir, n)?; + let mut highest = max_call(&records); + for earlier in &epochs[..epochs.len() - 1] { + let path = dir.join(format!("{earlier}.jsonl")); + highest = highest.max(max_call(&load_records(&path)?)); + } + let slot = live_slot(&records, 0); + let log = OpenOptions::new() + .append(true) + .open(&log_path) + .map_err(|e| SessionError::Io(log_path.clone(), e))?; + Ok(Session { + id, + dir, + baseline, + records, + log, + next_call: CallId(highest + 1), + epoch, + slot, + carry, + prior_users, + prior_untrusted, + }) + } +``` + +**S7.** After `pub fn records(&self)`, add accessors (the lines) + +```rust + pub fn records(&self) -> &[LogRecord] { + &self.records + } +``` + +become + +```rust + pub fn records(&self) -> &[LogRecord] { + &self.records + } + + pub fn epoch(&self) -> Epoch { + self.epoch + } + + pub fn slot(&self) -> u32 { + self.slot + } + + pub fn carry(&self) -> Option<&str> { + self.carry.as_deref() + } +``` + +**S8.** `untrusted` and `turns`, the lines + +```rust + pub fn untrusted(&self) -> bool { + self.records.iter().any(|record| { + matches!( + record, + LogRecord::ToolResult { + untrusted: true, + .. + } + ) + }) + } + + /// How many turns the session has had: its `User` records, one per turn that reached the + /// log, a failed turn's too. Counted from `records`, which `open` reads back whole, so the + /// count goes on after a reopen or a restart; right after a turn it is that turn's number, + /// the first being 1. Epoch 0 is a session's only log until compaction is built. + pub fn turns(&self) -> u64 { + let users = self + .records + .iter() + .filter(|record| matches!(record, LogRecord::User { .. })) + .count(); + u64::try_from(users).unwrap_or(u64::MAX) + } +``` + +become + +```rust + pub fn untrusted(&self) -> bool { + self.prior_untrusted || any_untrusted(&self.records) + } + + /// How many turns the session has had: its `User` records across every epoch, a failed + /// turn's too. The first `User` of the session is turn 1 for its life. + pub fn turns(&self) -> u64 { + self.prior_users + .saturating_add(count_users(&self.records)) + } +``` + +**S9.** `log_path`, the lines + +```rust + fn log_path(&self) -> PathBuf { + self.dir.join("0.jsonl") + } +} +``` + +become + +```rust + fn log_path(&self) -> PathBuf { + self.dir.join(format!("{}.jsonl", self.epoch.0)) + } +} + +/// Epoch numbers that have both `.jsonl` and `.baseline.json`, sorted. +fn complete_epochs(dir: &Path) -> Result, SessionError> { + let entries = match std::fs::read_dir(dir) { + Ok(e) => e, + Err(e) if e.kind() == std::io::ErrorKind::NotFound => return Ok(Vec::new()), + Err(e) => return Err(SessionError::Io(dir.to_path_buf(), e)), + }; + let mut out = Vec::new(); + for entry in entries { + let entry = entry.map_err(|e| SessionError::Io(dir.to_path_buf(), e))?; + let name = entry.file_name(); + let Some(name) = name.to_str() else { continue }; + let Some(stem) = name.strip_suffix(".jsonl") else { continue }; + let Ok(n) = stem.parse::() else { continue }; + if dir.join(format!("{n}.baseline.json")).exists() { + out.push(n); + } + } + out.sort_unstable(); + Ok(out) +} + +fn load_records(path: &Path) -> Result, SessionError> { + let text = match std::fs::read_to_string(path) { + Ok(t) => t, + Err(e) if e.kind() == std::io::ErrorKind::NotFound => { + return Err(SessionError::Io(path.to_path_buf(), e)); + } + Err(e) => return Err(SessionError::Io(path.to_path_buf(), e)), + }; + let mut records = Vec::new(); + if text.is_empty() { + return Ok(records); + } + for (idx, line) in text.split_inclusive('\n').enumerate() { + let line_no = idx + 1; + if !line.ends_with('\n') { + return Err(SessionError::Torn { + path: path.to_path_buf(), + line: line_no, + why: "last line does not end in a newline".to_string(), + }); + } + match serde_json::from_str::(line) { + Ok(record) => records.push(record), + Err(why) => { + return Err(SessionError::Torn { + path: path.to_path_buf(), + line: line_no, + why: why.to_string(), + }); + } + } + } + Ok(records) +} + +/// Carry from epoch `n-1`, and user / untrusted counts from every epoch before `n`. +fn priors(dir: &Path, n: u32) -> Result<(Option, u64, bool), SessionError> { + let mut prior_users = 0u64; + let mut prior_untrusted = false; + let mut carry = None; + if n == 0 { + return Ok((None, 0, false)); + } + for earlier in 0..n { + let path = dir.join(format!("{earlier}.jsonl")); + let records = match load_records(&path) { + Ok(r) => r, + Err(SessionError::Io(path, e)) if e.kind() == std::io::ErrorKind::NotFound => { + return Err(SessionError::NoCarry { path }); + } + Err(e) => return Err(e), + }; + prior_users = prior_users.saturating_add(count_users(&records)); + prior_untrusted = prior_untrusted || any_untrusted(&records); + if earlier + 1 == n { + carry = carry_from(&records, Epoch(n)).map(str::to_string); + if carry.is_none() { + return Err(SessionError::NoCarry { path }); + } + } + } + Ok((carry, prior_users, prior_untrusted)) +} + +fn carry_from(records: &[LogRecord], next: Epoch) -> Option<&str> { + records.iter().rev().find_map(|record| match record { + LogRecord::EpochEnd { + next: n, + summary, + .. + } if *n == next => Some(summary.as_str()), + _ => None, + }) +} + +fn count_users(records: &[LogRecord]) -> u64 { + let users = records + .iter() + .filter(|record| matches!(record, LogRecord::User { .. })) + .count(); + u64::try_from(users).unwrap_or(u64::MAX) +} + +fn any_untrusted(records: &[LogRecord]) -> bool { + records.iter().any(|record| { + matches!( + record, + LogRecord::ToolResult { + untrusted: true, + .. + } + ) + }) +} + +fn max_call(records: &[LogRecord]) -> u64 { + let mut highest = 0u64; + for record in records { + if let LogRecord::ToolResult { call, .. } = record { + highest = highest.max(call.0); + } + } + highest +} + +fn live_slot(records: &[LogRecord], fallback: u32) -> u32 { + records + .iter() + .find_map(|record| match record { + LogRecord::SessionStart { slot, .. } => Some(*slot), + _ => None, + }) + .unwrap_or(fallback) +} +``` + +The `open` body in S6 is given whole: copy it, do not leave a `todo!()` in `open`. The helpers +in S9 are given whole too. + +## Steps + +- [ ] **1. Copy the given test.** From the repository root: + + ```sh + cp docs/plans/M5d1/files/crates/loopd/tests/session_epochs.rs crates/loopd/tests/ + ``` + +- [ ] **2. See it fail.** `cargo build --tests --keep-going -p loopd`, timeout 600000 ms: test + target `session_epochs` does not compile (`E0599` no method `epoch` / `carry` / `slot`, and/or + `E0433` `NoCarry`). Every other target compiles. Existing `session` tests still compile. +- [ ] **3. The edits.** Make S1 to S9 in order. `cargo check -p loopd` prints no error and no + warning. `wc -l crates/loopd/src/session.rs` prints a number ≤ 500. +- [ ] **4. The tests.** `cargo test -p loopd --test session_epochs --test session`: every test + passes, 9 in `session_epochs` and the others as before. +- [ ] **5. Check.** `git diff --stat` lists exactly `crates/loopd/src/session.rs`. +- [ ] **6. Run the gate.** `cargo fmt --all`, then `make gate`, timeout 600000 ms. Last line + `gate: ok`. Test count **1473** (1464 before, plus 9). +- [ ] **7. Log and commit.** Task cell `M5d1/02-loopd-session-epochs`. Stage with + `git add crates/loopd docs/implementer-log.md Cargo.lock`, then commit as `AGENTS.md` says. + +## Done when + +- `make gate` prints `gate: ok` with 1473 tests. +- `git diff HEAD~1 --stat` lists only `crates/loopd/src/session.rs`, + `crates/loopd/tests/session_epochs.rs` and `docs/implementer-log.md`. + +## Stop and report if + +- The branch check fails. +- A "before" block is not found exactly once. +- Step 2 shows a different result. +- `cargo check` warns or errors, or `session.rs` exceeds 500 lines. +- A test fails in step 4. Do not change the test. +- An existing `session` test fails: stop and report. Those tests still create and open epoch 0 + only. +- The gate count is not 1473. +- A comment does not fit the code under it. + +## What the given tests pin + +Opening a directory with epochs 0 and 1 loads 1 and the carry from 0's `EpochEnd`; a missing +`EpochEnd` on epoch 0 when epoch 1 files exist is `NoCarry` naming `0.jsonl` with the damaged-log +runbook pointer; epoch 0 has no carry and keeps its `SessionStart` slot; two `User`s in epoch 0 +and one in epoch 1 is `turns() == 3` while `records()` has one `User`; an untrusted `ToolResult` +in epoch 0 taints after compact; `next_call` is one more than the highest call in any epoch; +`append` after opening epoch 1 writes `1.jsonl`; a `1.baseline.json` without `1.jsonl` is ignored; +an `EpochEnd` on epoch 0 without the next files stays on epoch 0. diff --git a/docs/plans/M5d1/03-loopd-carry-messages.md b/docs/plans/M5d1/03-loopd-carry-messages.md new file mode 100644 index 0000000..517881c --- /dev/null +++ b/docs/plans/M5d1/03-loopd-carry-messages.md @@ -0,0 +1,185 @@ +# M5d-1 task 03: the carried summary in the next epoch's prefix + +**Branch:** `m5d1`. Check it: `git branch --show-current` prints `m5d1` and `git status --short` +is empty; otherwise stop. Never switch branch. +**Commit subject:** `loopd: replay a carried summary as an unlogged user message` +**Log row:** the Task cell is `M5d1/03-loopd-carry-messages`. + +## Goal + +For epoch `n > 0`, the request prefix is the new system message, then one `ChatMessage::User` +whose content is a fixed heading plus the carry, then the new epoch's records (spec 3.5). That +user message is **not** a `LogRecord::User`: it is not a turn, not reported to Honcho, and not +counted by `turns()`. Epoch 0 is unchanged (`carry` is `None`). + +`run_turn` must send this prefix, or a compacted session's next live turn would omit the +summary. + +## Every place that changes + +| Place | What happens there | +|---|---| +| `crates/loopd/src/baseline.rs` | `CARRY_HEADING`; `messages_with_carry`; `messages` calls it with `None` | +| `crates/loopd/src/turn.rs` | the request uses `messages_with_carry` and `session.carry()` | +| `crates/loopd/tests/carry_messages.rs` | new, given | +| everything else | **no change**: not the existing `baseline` tests (they still call `messages`) | + +## Files + +- Copy, from `docs/plans/M5d1/files/` (the same path under it): the new test + `crates/loopd/tests/carry_messages.rs`. +- Modify: `crates/loopd/src/baseline.rs`, `crates/loopd/src/turn.rs`, + `docs/implementer-log.md`. +- Do not touch: any other file. + +## Facts you need + +- **The heading** is built from pieces so no file holds a chat-template tag whole (T33): + + ```rust + pub const CARRY_HEADING: &str = concat!("Earlier ", "turns, summarised:"); + ``` + + The user content is `format!("{CARRY_HEADING}\n\n{carry}")` when `carry` is `Some`. `None` + inserts nothing. +- **`messages` stays.** Existing tests and `limits.rs` call `messages(baseline, records)`. Make + it `messages_with_carry(baseline, records, None)` so those tests do not change. +- **`run_turn` today** (one place): + + ```rust + messages: messages(baseline, session.records()), + ``` + + It must pass `session.carry()`. `Session::carry` is `Option<&str>` from task 02. +- Insert the carry user **after** the system message and **before** walking `records`. +- The items the edits use. This table is complete. + + | Item | What it does | + |---|---| + | `concat!` | heading from pieces | + | `Session::carry(&self) -> Option<&str>` | task 02 | + | `ChatMessage::User { content }` | the carry, and log users | + +## The edits, exactly + +### `crates/loopd/src/baseline.rs` + +**B1.** After the `Baseline` impl, replace `messages`. The lines + +```rust +/// The system message, then each log record replayed in order. Every record variant is named, so a +/// new one is a compile error rather than a silent skip. +pub fn messages(baseline: &Baseline, records: &[LogRecord]) -> Vec { + let mut out = vec![ChatMessage::System { + content: baseline.system.clone(), + }]; + for record in records { +``` + +become + +```rust +/// The heading in front of a carried summary. Built from pieces (T33). +pub const CARRY_HEADING: &str = concat!("Earlier ", "turns, summarised:"); + +/// The system message, then each log record replayed in order. Every record variant is named, so a +/// new one is a compile error rather than a silent skip. +pub fn messages(baseline: &Baseline, records: &[LogRecord]) -> Vec { + messages_with_carry(baseline, records, None) +} + +/// Like `messages`, and when `carry` is `Some` inserts one user message after the system +/// prompt: the heading, a blank line, the summary. That user is not a `LogRecord::User`. +pub fn messages_with_carry( + baseline: &Baseline, + records: &[LogRecord], + carry: Option<&str>, +) -> Vec { + let mut out = vec![ChatMessage::System { + content: baseline.system.clone(), + }]; + if let Some(summary) = carry { + out.push(ChatMessage::User { + content: format!("{CARRY_HEADING}\n\n{summary}"), + }); + } + for record in records { +``` + +The rest of the `match` is unchanged; do not touch it. + +### `crates/loopd/src/turn.rs` + +**T1.** The `use` line + +```rust +use crate::baseline::messages; +``` + +becomes + +```rust +use crate::baseline::messages_with_carry; +``` + +**T2.** The lines + +```rust + let request = ChatRequest { + slot: rt.cfg.slots.main, + messages: messages(baseline, session.records()), + tools: baseline.tools.clone(), + thinking: true, + }; +``` + +become + +```rust + let request = ChatRequest { + slot: rt.cfg.slots.main, + messages: messages_with_carry(baseline, session.records(), session.carry()), + tools: baseline.tools.clone(), + thinking: true, + }; +``` + +## Steps + +- [ ] **1. Copy the given test.** + + ```sh + cp docs/plans/M5d1/files/crates/loopd/tests/carry_messages.rs crates/loopd/tests/ + ``` + +- [ ] **2. See it fail.** `cargo build --tests --keep-going -p loopd`, timeout 600000 ms: test + target `carry_messages` does not compile (`E0432` unresolved import + `loopd::baseline::{CARRY_HEADING, messages_with_carry}`). Every other target compiles. +- [ ] **3. The edits.** B1, T1, T2. `cargo check -p loopd` prints no error and no warning. +- [ ] **4. The tests.** `cargo test -p loopd --test carry_messages --test baseline --test turn`: + every test passes, 5 in `carry_messages` and the others as before. +- [ ] **5. Check.** `git diff --stat` lists exactly `crates/loopd/src/baseline.rs` and + `crates/loopd/src/turn.rs`. +- [ ] **6. Run the gate.** `cargo fmt --all`, then `make gate`, timeout 600000 ms. Last line + `gate: ok`. Test count **1478** (1473 before, plus 5). +- [ ] **7. Log and commit.** Task cell `M5d1/03-loopd-carry-messages`. Stage with + `git add crates/loopd docs/implementer-log.md Cargo.lock`, then commit as `AGENTS.md` says. + +## Done when + +- `make gate` prints `gate: ok` with 1478 tests. +- `git diff HEAD~1 --stat` lists only `crates/loopd/src/baseline.rs`, `crates/loopd/src/turn.rs`, + `crates/loopd/tests/carry_messages.rs` and `docs/implementer-log.md`. + +## Stop and report if + +- The branch check fails; a "before" block is not found exactly once; step 2 differs; `cargo + check` warns; a test fails; the gate count is not 1478; a comment does not fit its code. +- You think `messages` should insert the carry. It must not: existing tests call `messages` and + would see an extra user. `run_turn` is the one call that must pass `session.carry()`. + +## What the given tests pin + +Epoch 0 with `None` matches `messages`; a carry is a user after the system prompt and before the +epoch's users; the carry is not in the log records; the heading is the `concat!` pieces; the +first request of epoch 1 is not a prefix-extension of epoch 0's messages. diff --git a/docs/plans/M5d1/04-memoryd-honcho-epochs.md b/docs/plans/M5d1/04-memoryd-honcho-epochs.md new file mode 100644 index 0000000..be45103 --- /dev/null +++ b/docs/plans/M5d1/04-memoryd-honcho-epochs.md @@ -0,0 +1,232 @@ +# M5d-1 task 04: Honcho reads every epoch file + +**Branch:** `m5d1`. Check it: `git branch --show-current` prints `m5d1` and `git status --short` +is empty; otherwise stop. Never switch branch. +**Commit subject:** `memoryd: Honcho concatenates every epoch of a session` +**Log row:** the Task cell is `M5d1/04-memoryd-honcho-epochs`. + +## Goal + +`memoryd` keys a turn as `Item::Turn { session, turn }` with no epoch (spec 3.2). After +compaction, turn 3 may live in `1.jsonl`. `log_path` still names `0.jsonl` (existing tests pin +that path). A new `read_session` concatenates every `.jsonl` in epoch order. `TurnSync` +walks that concatenation, and the worker's `send_turn` does too, so a turn in epoch 1 is +decided and sent. + +`turn_records` / `turns` do not change: they already count `User` records on a slice. + +## Every place that changes + +| Place | What happens there | +|---|---| +| `crates/memoryd/src/honcho/turns.rs` | `epoch_logs`; `read_session`; `records` and `resume` use it | +| `crates/memoryd/src/honcho/worker.rs` | `send_turn` reads the concatenated log | +| `crates/memoryd/tests/honcho_epochs.rs` | new, given | +| everything else | **no change**: not `log_path`'s return value, not `decide.rs` | + +## Files + +- Copy, from `docs/plans/M5d1/files/` (the same path under it): the new test + `crates/memoryd/tests/honcho_epochs.rs`. +- Modify: `crates/memoryd/src/honcho/turns.rs`, `crates/memoryd/src/honcho/worker.rs`, + `docs/implementer-log.md`. +- Do not touch: any other file. + +## Facts you need + +- **`read_records` / `LogError`** already exist in `turns.rs`. `read_session` returns + `Result, LogError>` so a damaged line still names its file and + `LOG_RUNBOOK`. +- **Listing.** Under `sessions//`, every file named `.jsonl` where `n` parses as `u32`, + sorted by `n`. A missing session directory is empty, not an error (same as a missing + `0.jsonl` today). +- **`log_path` stays** `.../0.jsonl`. Do not change its body. `readable` in the worker still + names that path when the concatenated read succeeds, so existing recovery log lines stay + stable. +- **`send_turn` today** calls `read_turn(&log_path(...), turn)`, which only sees epoch 0. After + this task it loads `read_session`, then `turn_records` / `user_text` / `final_reply` from + `crate::honcho::decide` (already `pub`). +- `turns.rs` after this task stays under 500 lines (it is 332 now). +- The items the edits use. This table is complete. + + | Item | What it does | + |---|---| + | `read_records(&Path) -> Result, LogError>` | one file, already in `turns.rs` | + | `turn_records(&[LogRecord], u64) -> Option<&[LogRecord]>` | `decide.rs` | + | `user_text(&[LogRecord]) -> (&str, Option<&str>)` | `decide.rs` | + | `final_reply(&[LogRecord]) -> Option<&str>` | `decide.rs` | + | `str::strip_suffix(".jsonl")`, `str::parse::` | epoch numbers | + | `Unsent::Unreadable(String)` | worker, `e.to_string()` | + +## The edits, exactly + +### `crates/memoryd/src/honcho/turns.rs` + +**H1.** The doc comment on `log_path`, the lines + +```rust +/// The path of `session`'s log under `sessions`: epoch 0, the only one `loopd` writes. +pub fn log_path(sessions: &Path, session: &SessionId) -> PathBuf { +``` + +become + +```rust +/// The path of `session`'s epoch-0 log under `sessions`. Later epochs are sibling files; +/// `read_session` concatenates them. Existing tests pin this path. +pub fn log_path(sessions: &Path, session: &SessionId) -> PathBuf { +``` + +**H1b.** After `read_records` (the function that ends `Ok(records)` / `}`), insert these two +functions. They sit below `LogError` and `read_records`, so both names resolve. + +```rust +/// Every `.jsonl` of `session`, sorted by `n`. A missing directory is empty. +pub fn epoch_logs(sessions: &Path, session: &SessionId) -> Result, LogError> { + let dir = sessions.join(session.as_str()); + let entries = match std::fs::read_dir(&dir) { + Ok(e) => e, + Err(e) if e.kind() == std::io::ErrorKind::NotFound => return Ok(Vec::new()), + Err(e) => return Err(LogError::Io(format!("{}: {e}", dir.display()))), + }; + let mut found: Vec<(u32, PathBuf)> = Vec::new(); + for entry in entries { + let entry = match entry { + Ok(e) => e, + Err(e) => return Err(LogError::Io(format!("{}: {e}", dir.display()))), + }; + let name = entry.file_name(); + let Some(name) = name.to_str() else { continue }; + let Some(stem) = name.strip_suffix(".jsonl") else { continue }; + let Ok(n) = stem.parse::() else { continue }; + found.push((n, entry.path())); + } + found.sort_by_key(|(n, _)| *n); + Ok(found.into_iter().map(|(_, p)| p).collect()) +} + +/// Every epoch's records, in epoch order. A missing session has none. +pub fn read_session( + sessions: &Path, + session: &SessionId, +) -> Result, LogError> { + let mut out = Vec::new(); + for path in epoch_logs(sessions, session)? { + out.extend(read_records(&path)?); + } + Ok(out) +} +``` + +The next function after this insert is `finished`. Do not change `finished`. + +**H2.** In `resume`, the line + +```rust + if let Ok(records) = read_log(&log_path(&self.sessions, &session)) { +``` + +becomes + +```rust + if let Ok(records) = read_session(&self.sessions, &session) { +``` + +**H3.** In `records`, the line + +```rust + match read_log(&log_path(&self.sessions, session)) { +``` + +becomes + +```rust + match read_session(&self.sessions, session).map_err(|e| e.to_string()) { +``` + +`read_log` already mapped `LogError` through `Display`; this keeps the logged line the same +shape (`memoryd: {why}; {LOG_RUNBOOK}`). + +### `crates/memoryd/src/honcho/worker.rs` + +**W1.** The `use` line + +```rust +use crate::honcho::turns::{TurnSync, log_path}; +``` + +becomes + +```rust +use crate::honcho::decide::{final_reply, turn_records, user_text}; +use crate::honcho::turns::{TurnSync, log_path, read_session}; +``` + +**W2.** The body of `send_turn`, the lines + +```rust + let path = log_path(&self.sessions, session); + let read = read_turn(&path, turn).map_err(Unsent::Unreadable)?; + self.readable(&path); + let (text, reply) = read.ok_or(Unsent::Local("log"))?; +``` + +become + +```rust + let path = log_path(&self.sessions, session); + let records = match read_session(&self.sessions, session) { + Ok(records) => { + self.readable(&path); + records + } + Err(e) => return Err(Unsent::Unreadable(e.to_string())), + }; + let slice = turn_records(&records, turn).ok_or(Unsent::Local("log"))?; + let (text, _) = user_text(slice); + let reply = final_reply(slice).ok_or(Unsent::Local("log"))?; +``` + +The rest of `send_turn` is unchanged (`honcho.ok_or`, `batch`, `MAX_BATCH`, `add_messages`). +`text` is `&str` from `user_text`; `reply` is `&str` from `final_reply`. `batch` already takes +`&str`. + +## Steps + +- [ ] **1. Copy the given test.** + + ```sh + cp docs/plans/M5d1/files/crates/memoryd/tests/honcho_epochs.rs crates/memoryd/tests/ + ``` + +- [ ] **2. See it fail.** `cargo build --tests --keep-going -p memoryd`, timeout 600000 ms: test + target `honcho_epochs` does not compile (`E0432` unresolved import + `memoryd::honcho::turns::read_session`). Every other target compiles. +- [ ] **3. The edits.** H1, H1b, H2, H3, W1, W2. `cargo check -p memoryd` + prints no error and no warning. `wc -l crates/memoryd/src/honcho/turns.rs` and + `wc -l crates/memoryd/src/honcho/worker.rs` are each ≤ 500. +- [ ] **4. The tests.** `cargo test -p memoryd --test honcho_epochs --test honcho_turns --test + honcho_worker`: every test passes, 5 in `honcho_epochs` and the others as before. +- [ ] **5. Check.** `git diff --stat` lists exactly `crates/memoryd/src/honcho/turns.rs` and + `crates/memoryd/src/honcho/worker.rs`. +- [ ] **6. Run the gate.** `cargo fmt --all`, then `make gate`, timeout 600000 ms. Last line + `gate: ok`. Test count **1483** (1478 before, plus 5). +- [ ] **7. Log and commit.** Task cell `M5d1/04-memoryd-honcho-epochs`. Stage with + `git add crates/memoryd docs/implementer-log.md Cargo.lock`, then commit as `AGENTS.md` says. + +## Done when + +- `make gate` prints `gate: ok` with 1483 tests. +- `git diff HEAD~1 --stat` lists only the two `honcho` sources, `honcho_epochs.rs`, and the log. + +## Stop and report if + +- The branch check fails; a "before" block is not found exactly once; step 2 differs; a file + exceeds 500 lines; a test fails; the gate count is not 1483. +- You think `log_path` should return every file. It must still return `0.jsonl`. + +## What the given tests pin + +Two users in epoch 0 and one in epoch 1 is `turns() == 3` and `turn_records(..., 3)` is the +epoch-1 user; epoch 0 alone still reads; a damaged `1.jsonl` names that file; `TurnSync` decides +turn 3 from epoch 1; a missing session directory is empty. diff --git a/docs/plans/M5d1/05-loopd-compact-due.md b/docs/plans/M5d1/05-loopd-compact-due.md new file mode 100644 index 0000000..50f25cb --- /dev/null +++ b/docs/plans/M5d1/05-loopd-compact-due.md @@ -0,0 +1,130 @@ +# M5d-1 task 05: when to compact, and the fallback summary + +**Branch:** `m5d1`. Check it: `git branch --show-current` prints `m5d1` and `git status --short` +is empty; otherwise stop. Never switch branch. +**Commit subject:** `loopd: compaction threshold and the fallback summary` +**Log row:** the Task cell is `M5d1/05-loopd-compact-due`. + +## Goal + +A session is due to compact when the current epoch's last `Usage` has +`cache_n + prompt_n` at or above `threshold × expect.n_ctx` (spec 3.3). There is no idle timer. +If the summary call is context-full (task 06), the fallback is the epoch's `User` texts, newest +last, cut to `summary_chars` on a character boundary (spec 3.4). + +This task is those two pure functions, the `compact` module, and the starter `config/compact.md`. +No inference, no disk besides copying the starter. + +## Every place that changes + +| Place | What happens there | +|---|---| +| `crates/loopd/src/compact.rs` | new: `due`, `fallback_summary`, `cap_chars` | +| `crates/loopd/src/lib.rs` | `pub mod compact;` | +| `config/compact.md` | new, given starter | +| `crates/loopd/tests/compact_due.rs` | new, given | +| everything else | **no change** | + +## Files + +- Copy, from `docs/plans/M5d1/files/` (the same path under it): the new test + `crates/loopd/tests/compact_due.rs` and the starter `config/compact.md`. +- Create: `crates/loopd/src/compact.rs` (skeleton below). +- Modify: `crates/loopd/src/lib.rs`, `docs/implementer-log.md`. +- Do not touch: any other file. + +## Facts you need + +- **Depth is saturating.** `cache_n.saturating_add(prompt_n) as f64 >= threshold * (n_ctx as f64)`. + 0.85 × 262144 is 222822.4, so 222822 is not due and 222823 is. 0.85 × 1000 is 850 exactly. +- **Cast.** `u64 as f64` is exact for these magnitudes (n_ctx is 262144). Do not use `as` on + untrusted indexes; these are our own config and the server's counts. +- **Fallback join.** `"\n\n"` between texts. Newest last: reverse the `User` contents, then join, + then cap. +- **Cap.** `s.chars().take(n).collect::()`. `n == 0` is empty. A multi-byte character + is not split. +- **`config/compact.md`** is copied like `config/system.md`. Task 06 reads it from + `Config::compact_prompt`. +- The items the edits use. This table is complete. + + | Item | What it does | + |---|---| + | `bxcore::Usage` | `cache_n`, `prompt_n` | + | `LogRecord::User { content, .. }` | fallback texts | + | `str::chars`, `Iterator::take` | the cap | + +## Skeleton: `crates/loopd/src/compact.rs` + +Write this file, then fill the two `todo!()` bodies from their comments and delete the +comments. Each comment is at most six lines. + +```rust +//! Compaction: when a session starts a new epoch, the summary call, and writing the next files +//! (M5d spec, section 3). Later tasks fill the call and the write. + +use bxcore::{LogRecord, Usage}; + +/// True when `cache_n + prompt_n` is at or above `threshold * n_ctx`. +pub fn due(usage: &Usage, threshold: f64, n_ctx: u64) -> bool { + // saturating_add the two counts; compare as f64 to `threshold * (n_ctx as f64)`. + todo!() +} + +/// The epoch's `User` texts, newest last, joined with `"\n\n"`, then `cap_chars`. +pub fn fallback_summary(records: &[LogRecord], summary_chars: usize) -> String { + // Collect `&str` contents of every `User`, reverse, join `"\n\n"`, then `cap_chars`. + todo!() +} + +pub(crate) fn cap_chars(s: &str, n: usize) -> String { + s.chars().take(n).collect() +} +``` + +### `crates/loopd/src/lib.rs` + +**L1.** After `pub mod channel;` add `pub mod compact;` (keep rustfmt's order; if rustfmt +moves the line, accept that). + +## Steps + +- [ ] **1. Copy the given files.** + + ```sh + cp docs/plans/M5d1/files/crates/loopd/tests/compact_due.rs crates/loopd/tests/ + cp docs/plans/M5d1/files/config/compact.md config/compact.md + ``` + +- [ ] **2. Write the skeleton** at `crates/loopd/src/compact.rs` and make L1. See it fail: + `cargo test -p loopd --test compact_due`, timeout 600000 ms: the two `due` tests and the + three `fallback` tests panic on `todo!()` (or fail their asserts if you filled the bodies + early). If the test target does not compile (`E0432` unresolved `loopd::compact`), the + `mod` line is missing. +- [ ] **3. Fill `due` and `fallback_summary`.** Delete each instruction comment. `cargo check + -p loopd` prints no error and no warning. +- [ ] **4. The tests.** `cargo test -p loopd --test compact_due`: 6 passed. +- [ ] **5. Check.** `git diff --stat` lists `crates/loopd/src/lib.rs`, + `crates/loopd/src/compact.rs`, `config/compact.md`. `git status --short` shows + `compact_due.rs` as new. +- [ ] **6. Run the gate.** `cargo fmt --all`, then `make gate`, timeout 600000 ms. Last line + `gate: ok`. Test count **1489** (1483 before, plus 6). +- [ ] **7. Log and commit.** Task cell `M5d1/05-loopd-compact-due`. Stage with + `git add crates/loopd config/compact.md docs/implementer-log.md Cargo.lock`, then commit + as `AGENTS.md` says. + +## Done when + +- `make gate` prints `gate: ok` with 1489 tests. +- `git diff HEAD~1 --stat` lists only `crates/loopd/src/lib.rs`, `crates/loopd/src/compact.rs`, + `crates/loopd/tests/compact_due.rs`, `config/compact.md` and the log. + +## Stop and report if + +- The branch check fails; step 2 fails to compile for a reason other than the missing module + or `todo!()`; a test fails; the gate count is not 1489; you add an idle timer. There is none. + +## What the given tests pin + +849/1000 is not due, 850 is; 222822/262144 is not due, 222823 is; threshold 1.0 is the full +window; fallback is newest-last with `"\n\n"`; the cap is in characters on a boundary; no +users is empty. diff --git a/docs/plans/M5d1/06-loopd-compact-summary.md b/docs/plans/M5d1/06-loopd-compact-summary.md new file mode 100644 index 0000000..40cb51b --- /dev/null +++ b/docs/plans/M5d1/06-loopd-compact-summary.md @@ -0,0 +1,213 @@ +# M5d-1 task 06: the summary call, and `session-compact-failed` + +**Branch:** `m5d1`. Check it: `git branch --show-current` prints `m5d1` and `git status --short` +is empty; otherwise stop. Never switch branch. +**Commit subject:** `loopd: compact summary on the background slot` +**Log row:** the Task cell is `M5d1/06-loopd-compact-summary`. + +## Goal + +Compaction asks for a summary of the current epoch on the **background** slot, with **no tools**, +thinking on (spec 3.4). The last user message is `compact.md` (task 01's `compact_prompt`), +trimmed at the end. The completion is tag-neutralised and cut to `summary_chars`. + +If that call is context-full (`exceed_context_size_error`), use task 05's `fallback_summary`. +If `compact.md` cannot be read, or the call fails for any other reason, return +`CompactError::Failed` whose `Display` ends with the runbook pointer +`see docs/runbook.md#session-compact-failed`. There is no fallback for a missing prompt. + +This task does not write the next epoch. Task 07 does. + +## Every place that changes + +| Place | What happens there | +|---|---| +| `crates/loopd/src/compact.rs` | `CompactError`; `summarise` (edits E1, E2) | +| `docs/runbook.md` | append the given `session-compact-failed` entry | +| `crates/loopd/tests/compact_summary.rs` | new, given | +| everything else | **no change**: not `turn.rs`, not `channel.rs`, not `session.rs` | + +## Files + +- Copy, from `docs/plans/M5d1/files/` (the same path under it): the new test + `crates/loopd/tests/compact_summary.rs`. +- Append, from `docs/plans/M5d1/files/runbook-session-compact-failed.md`, to `docs/runbook.md`. +- Modify: `crates/loopd/src/compact.rs`, `docs/runbook.md`, `docs/implementer-log.md`. +- Do not touch: any other file. `config/compact.md` is already in the tree from task 05. + +## Facts you need + +- **The request.** `ChatRequest { slot: cfg.slots.background, messages, tools: Vec::new(), + thinking: true }`. Empty `tools` is omitted from the JSON (`skip_serializing_if` on the body). + The given test asserts `body.get("tools").is_none()` and `id_slot == 1` (the test config's + background slot). +- **The messages.** `messages_with_carry(session.baseline(), session.records(), session.carry())`, + then one `ChatMessage::User` whose content is `compact.md` after `trim_end`. Epoch 0 has no + carry; later epochs keep the carried summary in the prefix so the model summarises what the + session actually sends. +- **Read the prompt** with `std::fs::read_to_string(&cfg.compact_prompt())`. An I/O error is + `Failed { detail: format!("{}: {e}", path.display()) }` so the path (and `compact.md`) appears + in the message. +- **The call.** `client.chat_with_retry(&request, &mut |_| {})`. A 400 that is not context-full + is not retried (`is_retryable` is already false for it). +- **Context-full.** `crate::turn::is_context_full(&e)` then `fallback_summary(session.records(), + cfg.compact.summary_chars)`. Do not fall back for a missing file or any other `InferError`. +- **The text.** `completion.content.unwrap_or_default()`, then `bxcore::sanitize::neutralise`, + then `cap_chars(..., cfg.compact.summary_chars)`. Neutralise first so a tag is not split by + the cap. +- **`Display`.** `"{detail}; see docs/runbook.md#session-compact-failed"`. The given tests box + the error as `dyn Error` and check `to_string()`. +- **`compact.rs` stays ≤ 500 lines.** After this task it is about 120. +- The items the edits use. This table is complete. + + | Item | What it does | + |---|---| + | `Config::compact_prompt` | task 01, `compact.md` beside `system.md` | + | `messages_with_carry` | task 03 | + | `Client::chat_with_retry` | same as a live turn | + | `turn::is_context_full` | the one 400 `exceed_context_size_error` | + | `bxcore::sanitize::neutralise` | tags to a visible look-alike | + | `fallback_summary`, `cap_chars` | task 05 | + | `ChatRequest.tools: Vec::new()` | omits `tools` on the wire | + +## The edits, exactly + +Nothing else in `compact.rs` changes. Each quoted "before" block occurs exactly once after +task 05. Keep the fields in the order shown. + +### `crates/loopd/src/compact.rs` + +**E1.** The module doc and the `use` line. The lines + +```rust +//! Compaction: when a session starts a new epoch, the summary call, and writing the next files +//! (M5d spec, section 3). Later tasks fill the call and the write. + +use bxcore::{LogRecord, Usage}; +``` + +become + +```rust +//! Compaction: when a session starts a new epoch, the summary call, and writing the next files +//! (M5d spec, section 3). Task 07 writes the next files; task 08 calls this after a turn. + +use bxcore::sanitize::neutralise; +use bxcore::{LogRecord, Usage}; + +use crate::baseline::messages_with_carry; +use crate::config::Config; +use crate::llama::{ChatMessage, ChatRequest, Client}; +use crate::session::Session; +use crate::turn::is_context_full; + +/// Why a summary call or a next-epoch write failed. Every variant's `Display` ends with the +/// runbook pointer `session-compact-failed`. +#[derive(Debug)] +pub enum CompactError { + Failed { detail: String }, +} + +impl std::fmt::Display for CompactError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + CompactError::Failed { detail } => write!( + f, + "{detail}; see docs/runbook.md#session-compact-failed" + ), + } + } +} + +impl std::error::Error for CompactError {} +``` + +**E2.** After `cap_chars` (the function that ends `s.chars().take(n).collect()` / `}`), add +`summarise`. Copy this function whole. There is no `todo!()`. + +```rust +/// One completion on the background slot: no tools, thinking on, last user `compact.md`. +/// Context-full uses `fallback_summary`. A missing prompt, or any other infer error, is +/// `CompactError::Failed` with the runbook pointer. No fallback for a missing prompt. +pub fn summarise(cfg: &Config, client: &Client, session: &Session) -> Result { + let path = cfg.compact_prompt(); + let prompt = std::fs::read_to_string(&path).map_err(|e| CompactError::Failed { + detail: format!("{}: {e}", path.display()), + })?; + let mut messages = messages_with_carry(session.baseline(), session.records(), session.carry()); + messages.push(ChatMessage::User { + content: prompt.trim_end().to_string(), + }); + let request = ChatRequest { + slot: cfg.slots.background, + messages, + tools: Vec::new(), + thinking: true, + }; + match client.chat_with_retry(&request, &mut |_| {}) { + Ok(completion) => { + let text = completion.content.unwrap_or_default(); + let text = neutralise(&text); + Ok(cap_chars(text.as_ref(), cfg.compact.summary_chars)) + } + Err(e) if is_context_full(&e) => { + Ok(fallback_summary(session.records(), cfg.compact.summary_chars)) + } + Err(e) => Err(CompactError::Failed { + detail: e.to_string(), + }), + } +} +``` + +## Steps + +- [ ] **1. Copy the given test.** + + ```sh + cp docs/plans/M5d1/files/crates/loopd/tests/compact_summary.rs crates/loopd/tests/ + ``` + +- [ ] **2. See it fail.** `cargo build --tests --keep-going -p loopd`, timeout 600000 ms: test + target `compact_summary` does not compile (`E0432` unresolved import `loopd::compact::CompactError` + and/or `E0599` no function `summarise`). Every other target compiles. `compact_due` still + compiles. +- [ ] **3. The edits.** E1, E2. `cargo check -p loopd` prints no error and no warning. + `wc -l crates/loopd/src/compact.rs` prints a number ≤ 500. +- [ ] **4. The tests.** `cargo test -p loopd --test compact_summary --test compact_due`: every + test passes, 6 in `compact_summary` and 6 in `compact_due`. +- [ ] **5. The runbook.** From the repository root: + + ```sh + { echo; cat docs/plans/M5d1/files/runbook-session-compact-failed.md; } >> docs/runbook.md + ``` + + Then `grep -c '^## session-compact-failed$' docs/runbook.md` prints `1`. +- [ ] **6. Check.** `git diff --stat` lists `crates/loopd/src/compact.rs` and `docs/runbook.md`. + `git status --short` shows `compact_summary.rs` as new. +- [ ] **7. Run the gate.** `cargo fmt --all`, then `make gate`, timeout 600000 ms. Last line + `gate: ok`. Test count **1495** (1489 before, plus 6). +- [ ] **8. Log and commit.** Task cell `M5d1/06-loopd-compact-summary`. Stage with + `git add crates/loopd docs/runbook.md docs/implementer-log.md Cargo.lock`, then commit as + `AGENTS.md` says. + +## Done when + +- `make gate` prints `gate: ok` with 1495 tests. +- `git diff HEAD~1 --stat` lists only `crates/loopd/src/compact.rs`, + `crates/loopd/tests/compact_summary.rs`, `docs/runbook.md` and the log. + +## Stop and report if + +- The branch check fails; a "before" block is not found exactly once; step 2 differs; `cargo + check` warns; a test fails; the gate count is not 1495; a comment does not fit its code; + `compact.rs` exceeds 500 lines; you add a fallback for a missing `compact.md`; you send tools + on the summary request; you use `slots.main` for the summary. + +## What the given tests pin + +The summary request is `id_slot` of the background slot, omits `tools`, has thinking on, and its +last user is `compact.md` after `trim_end`; the completion is capped to `summary_chars`; a +context-full summary is the epoch's `User` texts; tags in the completion are neutralised; a +missing `compact.md` names the file and the runbook; a non-context-full infer error is +`CompactError::Failed` with the same pointer. diff --git a/docs/plans/M5d1/07-loopd-compact-advance.md b/docs/plans/M5d1/07-loopd-compact-advance.md new file mode 100644 index 0000000..6c57607 --- /dev/null +++ b/docs/plans/M5d1/07-loopd-compact-advance.md @@ -0,0 +1,268 @@ +# M5d-1 task 07: writing the next epoch + +**Branch:** `m5d1`. Check it: `git branch --show-current` prints `m5d1` and `git status --short` +is empty; otherwise stop. Never switch branch. +**Commit subject:** `loopd: write the next epoch from a compaction summary` +**Log row:** the Task cell is `M5d1/07-loopd-compact-advance`. + +## Goal + +Given a summary string, `commit` starts the next epoch (spec 3.5): + +1. Append `EpochEnd { next, summary }` to the current log, unless that log already ends with an + `EpochEnd` (a compact that stopped after the end record and before the next files). +2. Ask `memoryd` for a fresh block; write `.baseline.json` from `assemble_with` (so identity + edits applied in the old epoch land here, and `registry.core_schemas()` is the new tools + array). +3. Create `.jsonl` with `SessionStart { epoch: n, slot: the session's live slot, baseline: + hash of the new file }`. +4. The in-memory `Session` switches: new baseline, records that are only that `SessionStart`, + `carry = Some(summary)`, `prior_users` / `prior_untrusted` accumulated from the old records. + +A crash after step 1 and before the next files exist reopens epoch 0 with `EpochEnd` last (task +02). The next `commit` resumes: it uses the summary already in that `EpochEnd` as the carry and +does not write a second `EpochEnd`. The summary argument is ignored in that case. + +## Every place that changes + +| Place | What happens there | +|---|---| +| `crates/loopd/src/session.rs` | `begin_epoch` (edit S1) | +| `crates/loopd/src/compact.rs` | `commit` (edit C1) | +| `crates/loopd/tests/compact_advance.rs` | new, given | +| everything else | **no change**: not `turn.rs`, not `channel.rs` | + +## Files + +- Copy, from `docs/plans/M5d1/files/` (the same path under it): the new test + `crates/loopd/tests/compact_advance.rs`. +- Modify: `crates/loopd/src/session.rs`, `crates/loopd/src/compact.rs`, + `docs/implementer-log.md`. +- Do not touch: any other file. + +## Facts you need + +- **`begin_epoch` is `pub(crate)`** on `Session` so `compact.rs` can switch the private fields. + Tests call `compact::commit`, never `begin_epoch`. +- **Order inside `begin_epoch`.** Write `.baseline.json` (overwrite is fine on resume), + `create_new` the log file, then set `epoch` / `baseline` / `carry` / empty `records` / the new + `File`, **then** `persist` `SessionStart`. `log_path` uses `self.epoch`, so `epoch` must be + `next` before that persist. Accumulate `prior_users` and `prior_untrusted` from the **old** + `records` before clearing them. Do not reset `next_call` or `slot`. +- **Resume.** If `session.records().last()` is `EpochEnd { next, summary }`, skip the append; + `next` and `summary` (cloned) are what `begin_epoch` gets. Otherwise append `EpochEnd` with + `next = Epoch(session.epoch().0.saturating_add(1))` and the summary argument, then + `begin_epoch`. +- **Memory.** `memory.block().map_err(|e| CompactError::Failed { detail: e })?`. Then + `Baseline::assemble_with(cfg, registry, &block)`. `assemble_with` already calls + `registry.core_schemas()`, so a new core tool lands at this boundary. Map assemble and session + errors to `Failed { detail: e.to_string() }`. +- **`NoMemory.block()`** is `Ok("")`. The given test `the_new_baseline_takes_a_fresh_memory_block` + uses a `MemorySource` that returns a distinctive sentence. +- **`session.rs` stays ≤ 500 lines.** After task 02 it is about 430; `begin_epoch` is about 40. +- **`compact.rs` stays ≤ 500 lines.** After this task it is about 180. +- The items the edits use. This table is complete. + + | Item | What it does | + |---|---| + | `Baseline::assemble_with` | new system + memory block + `registry.core_schemas()` | + | `Baseline::to_json` / `hash` | the file and `SessionStart.baseline` | + | `OpenOptions::create_new(true).append(true)` | the new log, as `create` does | + | `count_users` / `any_untrusted` | task 02, module-private in `session.rs` | + | `Session::persist` | private; `begin_epoch` is in the same impl | + | `MemorySource::block` | fresh text; `Err` is `Failed` | + | `Timestamp::now` | `EpochEnd` and `SessionStart` | + | `Epoch(n.saturating_add(1))` | the next number | + +## The edits, exactly + +Each quoted "before" block occurs exactly once after task 06 (session.rs is as task 02 left it). + +### `crates/loopd/src/session.rs` + +**S1.** `log_path`, the lines + +```rust + fn log_path(&self) -> PathBuf { + self.dir.join(format!("{}.jsonl", self.epoch.0)) + } +} +``` + +become + +```rust + fn log_path(&self) -> PathBuf { + self.dir.join(format!("{}.jsonl", self.epoch.0)) + } + + /// Write `.baseline.json` and `.jsonl` with `SessionStart`, and switch this session + /// onto that epoch. The live slot and `next_call` stay. Callers pass the carry that + /// `messages_with_carry` will insert. + pub(crate) fn begin_epoch( + &mut self, + next: Epoch, + baseline: Baseline, + carry: String, + ) -> Result<(), SessionError> { + let n = next.0; + let baseline_path = self.dir.join(format!("{n}.baseline.json")); + let json = baseline.to_json().map_err(SessionError::Encode)?; + std::fs::write(&baseline_path, json) + .map_err(|e| SessionError::Io(baseline_path.clone(), e))?; + let log_path = self.dir.join(format!("{n}.jsonl")); + let log = OpenOptions::new() + .create_new(true) + .append(true) + .open(&log_path) + .map_err(|e| SessionError::Io(log_path.clone(), e))?; + self.prior_users = self.prior_users.saturating_add(count_users(&self.records)); + self.prior_untrusted = self.prior_untrusted || any_untrusted(&self.records); + self.epoch = next; + self.baseline = baseline; + self.carry = Some(carry); + self.records = Vec::new(); + self.log = log; + let hash = self.baseline.hash().map_err(SessionError::Baseline)?; + self.persist(LogRecord::SessionStart { + time: Timestamp::now(), + session: self.id.clone(), + epoch: next, + slot: self.slot, + baseline: hash, + }) + } +} +``` + +The helpers after the `impl` (`complete_epochs` and the rest) stay where they are. Do not move +them. + +### `crates/loopd/src/compact.rs` + +**C1.** After `summarise` (the function that ends the `match client.chat_with_retry` / `}`), add +the `use` items this function needs by extending the existing `use crate::` group, then add +`commit`. The current crate uses (after task 06) are: + +```rust +use crate::baseline::messages_with_carry; +use crate::config::Config; +use crate::llama::{ChatMessage, ChatRequest, Client}; +use crate::session::Session; +use crate::turn::is_context_full; +``` + +They become + +```rust +use crate::baseline::{messages_with_carry, Baseline}; +use crate::config::Config; +use crate::llama::{ChatMessage, ChatRequest, Client}; +use crate::memory::MemorySource; +use crate::session::Session; +use crate::tools::Registry; +use crate::turn::is_context_full; +``` + +And the bxcore use + +```rust +use bxcore::{LogRecord, Usage}; +``` + +becomes + +```rust +use bxcore::{Epoch, LogRecord, Timestamp, Usage}; +``` + +Then this function, whole, after `summarise`: + +```rust +/// Append `EpochEnd` (unless it is already last), rebuild the baseline from `memory`, and +/// switch `session` onto the next epoch. When the current log already ends with `EpochEnd`, +/// that record's summary is the carry and the argument is ignored. +pub fn commit( + session: &mut Session, + cfg: &Config, + registry: &Registry, + memory: &dyn MemorySource, + summary: String, +) -> Result<(), CompactError> { + let (next, summary) = match session.records().last() { + Some(LogRecord::EpochEnd { next, summary, .. }) => (*next, summary.clone()), + _ => { + let next = Epoch(session.epoch().0.saturating_add(1)); + session + .append(LogRecord::EpochEnd { + time: Timestamp::now(), + next, + summary: summary.clone(), + }) + .map_err(|e| CompactError::Failed { + detail: e.to_string(), + })?; + (next, summary) + } + }; + let block = memory.block().map_err(|e| CompactError::Failed { detail: e })?; + let baseline = Baseline::assemble_with(cfg, registry, &block).map_err(|e| { + CompactError::Failed { + detail: e.to_string(), + } + })?; + session + .begin_epoch(next, baseline, summary) + .map_err(|e| CompactError::Failed { + detail: e.to_string(), + })?; + Ok(()) +} +``` + +If rustfmt splits the `map_err` closures, accept that. + +## Steps + +- [ ] **1. Copy the given test.** + + ```sh + cp docs/plans/M5d1/files/crates/loopd/tests/compact_advance.rs crates/loopd/tests/ + ``` + +- [ ] **2. See it fail.** `cargo build --tests --keep-going -p loopd`, timeout 600000 ms: test + target `compact_advance` does not compile (`E0599` no function `commit`). Every other target + compiles. +- [ ] **3. The edits.** S1, then C1. `cargo check -p loopd` prints no error and no warning. + `wc -l crates/loopd/src/session.rs` and `wc -l crates/loopd/src/compact.rs` each print a + number ≤ 500. +- [ ] **4. The tests.** `cargo test -p loopd --test compact_advance --test session_epochs --test + compact_summary`: every test passes, 6 in `compact_advance`, 9 in `session_epochs`, 6 in + `compact_summary`. +- [ ] **5. Check.** `git diff --stat` lists exactly `crates/loopd/src/session.rs` and + `crates/loopd/src/compact.rs`. `git status --short` shows `compact_advance.rs` as new. +- [ ] **6. Run the gate.** `cargo fmt --all`, then `make gate`, timeout 600000 ms. Last line + `gate: ok`. Test count **1501** (1495 before, plus 6). +- [ ] **7. Log and commit.** Task cell `M5d1/07-loopd-compact-advance`. Stage with + `git add crates/loopd docs/implementer-log.md Cargo.lock`, then commit as `AGENTS.md` says. + +## Done when + +- `make gate` prints `gate: ok` with 1501 tests. +- `git diff HEAD~1 --stat` lists only `crates/loopd/src/session.rs`, + `crates/loopd/src/compact.rs`, `crates/loopd/tests/compact_advance.rs` and the log. + +## Stop and report if + +- The branch check fails; a "before" block is not found exactly once; step 2 differs; `cargo + check` warns; a test fails; the gate count is not 1501; a file exceeds 500 lines; you write a + second `EpochEnd` on resume; you reset `slot` or `next_call`; you log the summary as a + `LogRecord::User`. + +## What the given tests pin + +`commit` writes `EpochEnd` on the old log, a new baseline file, and a new log whose +`SessionStart` has the next epoch; the new baseline contains a fresh memory block; reopen loads +the new epoch and its carry; a second commit leaves epoch 0 with one `EpochEnd` and writes the +next on epoch 1; resume after `EpochEnd` without next files uses that summary as carry and does +not write a second end; the new `SessionStart` keeps the live slot (including a non-default 7). diff --git a/docs/plans/M5d1/08-loopd-compact-turn.md b/docs/plans/M5d1/08-loopd-compact-turn.md new file mode 100644 index 0000000..ad749da --- /dev/null +++ b/docs/plans/M5d1/08-loopd-compact-turn.md @@ -0,0 +1,261 @@ +# M5d-1 task 08: compact after a turn, and retry once on `SessionFull` + +**Branch:** `m5d1`. Check it: `git branch --show-current` prints `m5d1` and `git status --short` +is empty; otherwise stop. Never switch branch. +**Commit subject:** `loopd: compact after a full turn and retry SessionFull once` +**Log row:** the Task cell is `M5d1/08-loopd-compact-turn`. + +## Goal + +After `run_turn` returns, `finish_turn` decides whether to compact (spec 3.3): + +- A successful turn whose last `Usage` is `due` is compacted before the caller returns. If + compact fails, the turn stays successful and the session stays on its epoch; `loopd` logs the + compact error with the runbook pointer. +- `TurnError::SessionFull` is compacted, then that user message is `run_turn`'d once in the new + epoch. If compact fails, the error stays `SessionFull`. The retry is not compacted again. +- Any other `TurnError` is returned as it is. No compact. + +`Runtime` does not gain `MemorySource` (given `turn` tests build a `Runtime` without one). +`finish_turn` takes `memory` as its own argument. + +The channel holds the session busy for this whole call: `handle` runs `finish_turn` **before** +`drop(held)`. A compact is not an idle timer and does not start after the busy set releases. + +Live turns still send `slots.main` (task 03). Only the summary call uses the background slot. + +## Every place that changes + +| Place | What happens there | +|---|---| +| `crates/loopd/src/compact.rs` | `finish_turn` and its two helpers (edit F1) | +| `crates/loopd/src/channel.rs` | `handle` calls `finish_turn` before `drop(held)` (edits H1, H2) | +| `crates/loopd/tests/compact_turn.rs` | new, given | +| everything else | **no change**: not `turn.rs`, not `Runtime`, not existing `turn` / `channel` tests | + +## Files + +- Copy, from `docs/plans/M5d1/files/` (the same path under it): the new test + `crates/loopd/tests/compact_turn.rs`. +- Modify: `crates/loopd/src/compact.rs`, `crates/loopd/src/channel.rs`, + `docs/implementer-log.md`. +- Do not touch: any other file. + +## Facts you need + +- **`due` looks at `session.last_usage()`**, which is the current epoch's last `Usage`. After a + successful `run_turn` that record is there. After `SessionFull`, `run_turn` has already + appended the `User` and returned before any `Usage`; compact runs because of the error, not + because of `due`. +- **Compact helper.** `summarise` then `commit`. Log failures with + + ```text + loopd: could not compact session : + ``` + + `Display` already includes the runbook pointer, matching the runbook's example line. +- **Retry.** `run_turn` appends the user message again in the new epoch. The old epoch keeps its + `User` (Honcho skips it: no reply). `turns()` is 2. That is accepted (README, known limits). +- **Existing channel tests stay green.** Their `n_ctx` is 131072 and fixture `prompt_n` is tens, + so `due` is false. `SessionFull` without `compact.md` in `Home` stays `SessionFull` (the + successful-turn-when-compact-fails behaviour, on the error path). +- **`compact.rs` stays ≤ 500 lines.** After this task it is about 250. +- **`channel.rs` stays ≤ 500 lines.** After this task it is about 290. +- The items the edits use. This table is complete. + + | Item | What it does | + |---|---| + | `due` | task 05 | + | `summarise` / `commit` | tasks 06, 07 | + | `run_turn` | retry the same `text` and `speaker` | + | `Session::last_usage` | current epoch | + | `Config.compact.threshold`, `expect.n_ctx` | the `due` arguments | + | `MemorySource` | `finish_turn`'s extra argument; channel passes `&*ctx.memory` | + | `Held` | already in `channel.rs`; drop it after `finish_turn` | + +## The edits, exactly + +Each quoted "before" block occurs exactly once after task 07 (`channel.rs` is still as on +master, except rustfmt). + +### `crates/loopd/src/compact.rs` + +**F1.** Extend the crate uses. After task 07 they include `MemorySource`, `Session`, `Config`, +`Registry`. Add: + +```rust +use crate::turn::{Runtime, TurnError, TurnOutcome, run_turn}; +use bxcore::TurnEvent; +``` + +Keep rustfmt's order of `use` lines; if it moves them, accept that. + +Then after `commit`, add these three functions whole. There is no `todo!()`. + +```rust +/// After `run_turn`: compact when the last `Usage` is due, or on `SessionFull` (then retry the +/// user message once). Compact failure on a successful turn keeps the success; on `SessionFull` +/// keeps `SessionFull`. Other errors skip compact. +pub fn finish_turn( + session: &mut Session, + rt: &Runtime<'_>, + memory: &dyn MemorySource, + text: &str, + speaker: Option<&str>, + on_event: &mut dyn FnMut(&TurnEvent), + outcome: Result, +) -> Result { + match outcome { + Err(TurnError::SessionFull) => match compact_session(session, rt, memory) { + Ok(()) => run_turn(session, rt, text, speaker, on_event), + Err(e) => { + log_compact(session, &e); + Err(TurnError::SessionFull) + } + }, + Ok(done) => { + if let Some(usage) = session.last_usage() + && due(&usage, rt.cfg.compact.threshold, rt.cfg.expect.n_ctx) + && let Err(e) = compact_session(session, rt, memory) + { + log_compact(session, &e); + } + Ok(done) + } + Err(e) => Err(e), + } +} + +fn compact_session( + session: &mut Session, + rt: &Runtime<'_>, + memory: &dyn MemorySource, +) -> Result<(), CompactError> { + let summary = summarise(rt.cfg, rt.client, session)?; + commit(session, rt.cfg, rt.registry, memory, summary) +} + +fn log_compact(session: &Session, e: &CompactError) { + eprintln!( + "loopd: could not compact session {}: {e}", + session.id().as_str() + ); +} +``` + +The `Ok(done)` arm uses `&& let Err(e) = ...` so a compact failure is logged and the original +`done` is still returned. Do not `?` that error. + +### `crates/loopd/src/channel.rs` + +**H1.** The `run_turn` call and the `drop(held)` that follows it. The lines + +```rust + let outcome = run_turn( + &mut session, + &runtime, + &turn.content, + turn.speaker.as_deref(), + &mut |event| { + let _ = write_frame( + &mut stream, + &Envelope { + v: PROTOCOL_VERSION, + id: request.id, + r#final: false, + msg: Message::TurnEvent(event.clone()), + }, + ); + }, + ); + + // 5. Release the session before the final frame, so the next turn does not find it busy. + drop(held); +``` + +become + +```rust + let mut emit = |event: &bxcore::TurnEvent| { + let _ = write_frame( + &mut stream, + &Envelope { + v: PROTOCOL_VERSION, + id: request.id, + r#final: false, + msg: Message::TurnEvent(event.clone()), + }, + ); + }; + + let outcome = run_turn( + &mut session, + &runtime, + &turn.content, + turn.speaker.as_deref(), + &mut emit, + ); + let outcome = crate::compact::finish_turn( + &mut session, + &runtime, + &*ctx.memory, + &turn.content, + turn.speaker.as_deref(), + &mut emit, + outcome, + ); + + // 5. Release the session before the final frame, so the next turn does not find it busy. + // Compaction ran inside the hold: a second turn on this session waits until the new + // epoch is on disk (or compact failed and the epoch is unchanged). + drop(held); +``` + +**H2.** Do not add `MemorySource` to `Runtime`. Do not change `turn_code`. The `memory.updated` +block after `drop(held)` stays: a successful compact-and-retry still reports `session.turns()`, +which now spans epochs. + +## Steps + +- [ ] **1. Copy the given test.** + + ```sh + cp docs/plans/M5d1/files/crates/loopd/tests/compact_turn.rs crates/loopd/tests/ + ``` + +- [ ] **2. See it fail.** `cargo build --tests --keep-going -p loopd`, timeout 600000 ms: test + target `compact_turn` does not compile (`E0599` no function `finish_turn`). Every other + target compiles, including `turn` and `channel`. +- [ ] **3. The edits.** F1, H1. `cargo check -p loopd` prints no error and no warning. + `wc -l crates/loopd/src/compact.rs` and `wc -l crates/loopd/src/channel.rs` each print a + number ≤ 500. +- [ ] **4. The tests.** `cargo test -p loopd --test compact_turn --test turn --test channel + --test compact_advance`: every test passes, 6 in `compact_turn` and the others as before. +- [ ] **5. Check.** `git diff --stat` lists exactly `crates/loopd/src/compact.rs` and + `crates/loopd/src/channel.rs`. `git status --short` shows `compact_turn.rs` as new. +- [ ] **6. Run the gate.** `cargo fmt --all`, then `make gate`, timeout 600000 ms. Last line + `gate: ok`. Test count **1507** (1501 before, plus 6). +- [ ] **7. Log and commit.** Task cell `M5d1/08-loopd-compact-turn`. Stage with + `git add crates/loopd docs/implementer-log.md Cargo.lock`, then commit as `AGENTS.md` says. + +## Done when + +- `make gate` prints `gate: ok` with 1507 tests. +- `git diff HEAD~1 --stat` lists only `crates/loopd/src/compact.rs`, + `crates/loopd/src/channel.rs`, `crates/loopd/tests/compact_turn.rs` and the log. + +## Stop and report if + +- The branch check fails; a "before" block is not found exactly once; step 2 differs; `cargo + check` warns; a test fails; the gate count is not 1507; a file exceeds 500 lines; you add + `MemorySource` to `Runtime`; you compact inside `run_turn`; you call `finish_turn` after + `drop(held)`; you add an idle timer; you compact a `TurnError` other than `SessionFull`; you + retry `SessionFull` more than once. + +## What the given tests pin + +Usage 849/1000 leaves epoch 0 (one chat); 850/1000 compact before return (live slot then +background, no tools on the summary); `SessionFull` compact-and-retry (three chats, user logged +in both epochs, `turns() == 2`); `SessionFull` stays `SessionFull` when compact fails; a +successful due turn stays successful when `compact.md` is missing (epoch stays 0); another +infer error is not compacted. diff --git a/docs/plans/M5d1/README.md b/docs/plans/M5d1/README.md new file mode 100644 index 0000000..7d1642b --- /dev/null +++ b/docs/plans/M5d1/README.md @@ -0,0 +1,119 @@ +# M5d-1 implementation plan: epochs and compaction + +> For the implementer: each task file is one closed task for one fresh session. Read `AGENTS.md`, +> then the task file, and do exactly that task. The driver is `tools/run-plan.sh`. + +**Goal:** a live session can start a new epoch. `loopd` writes `N.jsonl` / `N.baseline.json` for +`N > 0`, turn numbers and `untrusted()` span every epoch, compaction runs after a turn whose last +`Usage` is at or above a configured fraction of `expect.n_ctx`, the summary call uses the +background slot, and `memoryd`'s Honcho walker reads every epoch file of a session. + +**Architecture:** `loopd` gains a `[compact]` table and a `compact` module (when to compact, the +summary call, writing the next epoch). `Session::open` loads the highest complete epoch and the +carried summary. `messages` for epoch `n > 0` inserts that carry as a `ChatMessage::User` that is +never a `LogRecord::User`. Honcho concatenates epoch logs in order so `Item::Turn` numbers stay +monotonic. Heartbeat, cron, `ToolSummary`, and tiering are M5d-2. + +**Tech stack:** Rust 1.98.1, edition 2024; no new crate. + +**Spec:** `docs/specs/2026-10-03-m5d-scheduler.md`, sections 3, 5 (`[compact]` without using +`summarise`), 7 (the M5d-1 tests), and 8 (`session-compact-failed`). + +## The owner's decisions (2026-10-03) + +1. Compaction runs after a turn, when that turn's `cache_n + prompt_n` is at or above a configured + fraction of `expect.n_ctx` (default 0.85). A quiet session is not a reason to compact. +2. If a live request is `exceed_context_size_error`, compact and retry that user message once in + the new epoch. +3. The summary call is on the background slot, with no tools, thinking on. A context-full summary + falls back to the epoch's `User` texts, newest last, cut to `summary_chars`. +4. The carried summary is not a `LogRecord::User`, so it is not a Honcho turn. + +## What the code already has, and what it lacks + +- `LogRecord::EpochEnd { time, next, summary }` is in `bxcore` and unused. `messages` already + ignores it. This plan writes it. +- `Session::open` reads only `0.baseline.json` and `0.jsonl`. `turns()` and `untrusted()` look at + that file only. `next_call` is recovered from that file only. +- `run_turn` always sends `slots.main`. Live turns stay on the session's live slot (main, from + `SessionStart`). Only the summary call uses `slots.background`. Changing live turns to the + session's slot is M5d-2. +- `memoryd`'s indexer already walks every `.jsonl`. Honcho's `log_path` is still + `sessions//0.jsonl`. +- `Runtime` has no `MemorySource`. Compaction asks `memoryd` for a fresh block from `channel.rs` + (and `compact::finish_turn`), so given `turn` tests that build a `Runtime` do not change. +- `[compact] summarise` is stored from this plan so a spec-complete `loopd.toml` loads; nothing + in M5d-1 reads the list. M5d-2 opts in `http_fetch`. + +## Global constraints + +- Roles depend on `bxcore` only. No `unsafe`, no async runtime, no new crate. +- No file the implementer reads holds a chat-template tag whole (T33). The carry heading is built + from pieces and named in words. +- Every path a given test creates is removed by a guard (T44); every process a test starts has a + deadline. +- The fail-closed state `session-compact-failed` has a runbook entry, added by the task that adds + its pointer (T49). +- Each task gives its baseline and its gate count with what the count includes (T50, T53); the + branch line is a check, never `git switch` (T51). +- `Cargo.lock` is in the `git add` line (T24). Gate timeout 600000 ms (T40). +- Commit trailer from the driver. + +## Tasks + +Every task below has been written to be replayed from its text. The gate count is the passed +total of `cargo test --workspace` plus one, for the ignored `end_to_end` test the gate runs on +its own. Before task 01 it is **1456** (talos, rustc 1.98.1, clippy of the same release; master +`069cd16`, docs-only commits after the M5c-2 review). + +| # | Delivers | Crates | Gate after | Tests added | +|---|---|---|---|---| +| 01 | `[compact]` on `loopd.toml`: `threshold` 0.85, `summary_chars` 4000, `summarise` default `["http_fetch"]` (stored, unused); `Config::compact_prompt` is `compact.md` beside `system.md` | loopd | 1464 | 8 | +| 02 | `Session::open` loads the highest epoch that has both a baseline and a log, the carry from the previous `EpochEnd`, `turns` / `untrusted` / `next_call` over every epoch; persist writes the current epoch | loopd | 1473 | 9 | +| 03 | `messages_with_carry`: epoch `n > 0` is the new system message, then one `ChatMessage::User` (fixed heading plus carry, not a log record), then the new epoch's records; `run_turn` uses `session.messages()` | loopd | 1478 | 5 | +| 04 | Honcho concatenates every `.jsonl` in epoch order; `turn_records` of turn 3 is the epoch-1 user; `send_turn` reads the concatenated log | memoryd | 1483 | 5 | +| 05 | `loopd::compact`: `due`, `fallback_summary`; starter `config/compact.md` | loopd | 1489 | 6 | +| 06 | Summary call on the background slot, no tools, thinking on; tag-neutralised and capped; context-full uses the fallback; `session-compact-failed` pointer and runbook entry | loopd | 1495 | 6 | +| 07 | Writing the next epoch: `EpochEnd`, fresh baseline from `memoryd`, `SessionStart` on the live slot; resume if `EpochEnd` is already the last record | loopd | 1501 | 6 | +| 08 | After a turn: compact when `due`; on `SessionFull`, compact and retry the user message once; compaction holds the session busy | loopd | 1507 | 6 | + +After task 08, the design model runs the M5d-1 device check (spec 7) on straylight: a `bxctl` +session writes a distinctive fact into `memory.md`, continues until compaction with +`[compact] threshold` lowered, and the next turn's baseline contains the fact; `recall` of a +phrase from epoch 0 still hits; `TurnDone.untrusted` stays true across the epoch change after a +tainted recall. + +## Known limits + +These are known, accepted for M5d-1, and not bugs to fix in a task: + +- **Live turns still use `slots.main`.** `SessionStart.slot` is recorded and copied onto the new + epoch; `run_turn` does not read it yet. Isolated sessions on the background slot are M5d-2. +- **`[compact] summarise` is unused.** Unknown tool names are not a config error until M5d-2. +- **No idle path exists.** Compaction is only called after a turn or on `SessionFull`. There is + no timer to test. +- **A crash after `EpochEnd` and before the next files exist** reopens epoch 0 with `EpochEnd` at + the end of the log. The next compact resumes without writing a second `EpochEnd` (task 07). +- **A `SessionFull` retry logs the user message twice**, once in the old epoch (Honcho skips it: + no reply) and once in the new epoch. Turn numbers stay monotonic. + +## Before hand-over + +Work through `docs/plans/CHECKLIST.md` with a reference in a scratch worktree with its own build +directory, replayed from its task text; mutations; `/tmp` compared around a gate run; scratch +files in the worktree's `.state/`; the reference moved to straylight as a bundle before the run +(T18). Task files name their log row (T47). + +## Running it + +The reference (`m5d1-ref`, tasks 01 to 08) is moved to straylight as a bundle and deleted here +before the run (T18). Then, from the repository root, with the OpenCode TUI closed: + +```sh +git switch -c m5d1 master +OPENCODE_MODEL=straylight/ornith-1.5-35b-a3b RESUME=1 \ + SLOTS_URL='http://straylight:11434/slots?model=ornith-1.5-35b-a3b' \ + tools/run-plan.sh docs/plans/M5d1 01 +``` + +After task 08, the gate count is 1507. Then the device check above. diff --git a/docs/plans/M5d1/files/config/compact.md b/docs/plans/M5d1/files/config/compact.md new file mode 100644 index 0000000..b468535 --- /dev/null +++ b/docs/plans/M5d1/files/config/compact.md @@ -0,0 +1,8 @@ +Summarise this conversation for the next epoch of the same session. + +Write a short account of: +- what the owner asked for and decided +- facts that were established +- open threads and anything still waiting + +Do not use tools. Do not address the owner. Do not invent facts that are not in the messages. diff --git a/docs/plans/M5d1/files/crates/loopd/tests/carry_messages.rs b/docs/plans/M5d1/files/crates/loopd/tests/carry_messages.rs new file mode 100644 index 0000000..2d5bfb4 --- /dev/null +++ b/docs/plans/M5d1/files/crates/loopd/tests/carry_messages.rs @@ -0,0 +1,120 @@ +//! Tests for the carried summary in the next epoch's prefix. Do not edit. + +use bxcore::{Epoch, Hash32, LogRecord, SessionId, Timestamp}; +use loopd::baseline::{Baseline, CARRY_HEADING, messages, messages_with_carry}; +use loopd::llama::ChatMessage; + +fn ts() -> Timestamp { + Timestamp::parse("2026-09-18T08:00:00.000Z").unwrap() +} + +fn user(content: &str) -> LogRecord { + LogRecord::User { + time: ts(), + content: content.to_string(), + speaker: None, + } +} + +fn start(epoch: u32) -> LogRecord { + LogRecord::SessionStart { + time: ts(), + session: SessionId::new("a").unwrap(), + epoch: Epoch(epoch), + slot: 0, + baseline: Hash32::ZERO, + } +} + +#[test] +fn epoch_0_matches_messages_without_a_carry() { + let b = Baseline { + system: "sys".to_string(), + tools: vec![], + }; + let records = vec![start(0), user("hi")]; + assert_eq!( + messages_with_carry(&b, &records, None), + messages(&b, &records) + ); +} + +#[test] +fn a_carry_is_a_user_message_after_the_system_prompt() { + let b = Baseline { + system: "new-sys".to_string(), + tools: vec![], + }; + let records = vec![start(1), user("next")]; + let got = messages_with_carry(&b, &records, Some("the summary")); + assert_eq!( + got, + vec![ + ChatMessage::System { + content: "new-sys".to_string(), + }, + ChatMessage::User { + content: format!("{CARRY_HEADING}\n\nthe summary"), + }, + ChatMessage::User { + content: "next".to_string(), + }, + ] + ); +} + +#[test] +fn the_carry_is_not_taken_from_a_log_user_record() { + let b = Baseline { + system: "sys".to_string(), + tools: vec![], + }; + let records = vec![start(1)]; + let got = messages_with_carry(&b, &records, Some("carried")); + assert_eq!(got.len(), 2); + assert!( + !records.iter().any(|r| matches!(r, LogRecord::User { .. })), + "the carry is not in the log" + ); +} + +#[test] +fn the_heading_is_built_from_pieces() { + assert_eq!(CARRY_HEADING, concat!("Earlier ", "turns, summarised:")); + assert!( + !CARRY_HEADING.contains('<'), + "the heading is words, not a tag" + ); +} + +#[test] +fn epoch_1_is_not_an_extension_of_epoch_0() { + let old = Baseline { + system: "old-sys".to_string(), + tools: vec![], + }; + let new = Baseline { + system: "new-sys".to_string(), + tools: vec![], + }; + let epoch0 = vec![start(0), user("first")]; + let epoch1 = vec![start(1), user("second")]; + let prev = messages(&old, &epoch0); + let next = messages_with_carry(&new, &epoch1, Some("sum of first")); + assert!( + next.len() >= 2, + "system plus carry, then the new epoch's user" + ); + assert_ne!(next[0], prev[0], "the system message changed"); + assert!( + !next.starts_with(&prev[..]), + "the new prefix is not an extension of the old messages" + ); + match &next[1] { + ChatMessage::User { content } => { + assert!(content.contains("sum of first"), "{content}"); + assert!(!content.contains("second"), "{content}"); + } + other => panic!("{other:?}"), + } +} diff --git a/docs/plans/M5d1/files/crates/loopd/tests/compact_advance.rs b/docs/plans/M5d1/files/crates/loopd/tests/compact_advance.rs new file mode 100644 index 0000000..dc7bd7a --- /dev/null +++ b/docs/plans/M5d1/files/crates/loopd/tests/compact_advance.rs @@ -0,0 +1,201 @@ +//! Tests for writing the next epoch. Do not edit. + +mod support; + +use bxcore::{Epoch, LogRecord, SessionId}; +use loopd::baseline::Baseline; +use loopd::compact; +use loopd::memory::{MemorySource, NoMemory}; +use loopd::session::Session; +use loopd::tools::Registry; +use support::Home; + +struct Block(&'static str); + +impl MemorySource for Block { + fn block(&self) -> Result { + Ok(self.0.to_string()) + } + fn updated(&self, _session: &SessionId, _turn: u64) -> Result<(), String> { + Ok(()) + } +} + +fn id(s: &str) -> SessionId { + SessionId::new(s).unwrap() +} + +fn session(home: &Home) -> (loopd::config::Config, Session) { + let cfg = home.config(std::path::Path::new("/tmp/unused.sock")); + let baseline = Baseline::assemble(&cfg, &Registry::m2b()).unwrap(); + let s = Session::create(&home.dir, id("a"), baseline, cfg.slots.main).unwrap(); + (cfg, s) +} + +#[test] +fn commit_writes_epoch_end_a_new_baseline_and_a_new_log() { + let home = Home::new(); + let (cfg, mut s) = session(&home); + s.append(LogRecord::User { + time: bxcore::Timestamp::parse("2026-09-18T08:00:00.000Z").unwrap(), + content: "hi".to_string(), + speaker: None, + }) + .unwrap(); + compact::commit( + &mut s, + &cfg, + &Registry::m2b(), + &NoMemory, + "the sum".to_string(), + ) + .unwrap(); + + assert_eq!(s.epoch(), Epoch(1)); + assert_eq!(s.carry(), Some("the sum")); + assert_eq!(s.slot(), cfg.slots.main); + assert_eq!(s.turns(), 1, "the old user still counts"); + assert_eq!( + s.records() + .iter() + .filter(|r| matches!(r, LogRecord::User { .. })) + .count(), + 0 + ); + + let old = home.records("a"); + assert!( + matches!(old.last(), Some(LogRecord::EpochEnd { next, summary, .. }) if *next == Epoch(1) && summary == "the sum") + ); + assert!(home.dir.join("sessions/a/1.baseline.json").is_file()); + let new_log = home.read("sessions/a/1.jsonl"); + assert!(new_log.contains("\"epoch\":1"), "{new_log}"); +} + +#[test] +fn the_new_baseline_takes_a_fresh_memory_block() { + let home = Home::new(); + let (cfg, mut s) = session(&home); + compact::commit( + &mut s, + &cfg, + &Registry::m2b(), + &Block("I take my tea strong."), + "sum".to_string(), + ) + .unwrap(); + assert!( + s.baseline().system.contains("I take my tea strong."), + "{}", + s.baseline().system + ); + let on_disk = home.read("sessions/a/1.baseline.json"); + assert!(on_disk.contains("I take my tea strong."), "{on_disk}"); +} + +#[test] +fn reopen_loads_the_new_epoch() { + let home = Home::new(); + let (cfg, mut s) = session(&home); + compact::commit( + &mut s, + &cfg, + &Registry::m2b(), + &NoMemory, + "carry".to_string(), + ) + .unwrap(); + drop(s); + let s = Session::open(&home.dir, id("a")).unwrap(); + assert_eq!(s.epoch(), Epoch(1)); + assert_eq!(s.carry(), Some("carry")); +} + +#[test] +fn a_second_commit_does_not_write_another_epoch_end_on_the_old_log() { + let home = Home::new(); + let (cfg, mut s) = session(&home); + compact::commit( + &mut s, + &cfg, + &Registry::m2b(), + &NoMemory, + "first".to_string(), + ) + .unwrap(); + let ends = home + .records("a") + .iter() + .filter(|r| matches!(r, LogRecord::EpochEnd { .. })) + .count(); + assert_eq!(ends, 1); + compact::commit( + &mut s, + &cfg, + &Registry::m2b(), + &NoMemory, + "second".to_string(), + ) + .unwrap(); + assert_eq!(s.epoch(), Epoch(2)); + let ends = home + .records("a") + .iter() + .filter(|r| matches!(r, LogRecord::EpochEnd { .. })) + .count(); + assert_eq!(ends, 1, "epoch 0 still has one EpochEnd"); + let mid = home.read("sessions/a/1.jsonl"); + assert!(mid.contains("\"type\":\"epoch_end\""), "{mid}"); +} + +#[test] +fn resume_after_epoch_end_without_next_files_does_not_write_a_second_end() { + let home = Home::new(); + let (cfg, mut s) = session(&home); + s.append(LogRecord::EpochEnd { + time: bxcore::Timestamp::now(), + next: Epoch(1), + summary: "already".to_string(), + }) + .unwrap(); + drop(s); + let mut s = Session::open(&home.dir, id("a")).unwrap(); + assert_eq!(s.epoch(), Epoch(0)); + compact::commit( + &mut s, + &cfg, + &Registry::m2b(), + &NoMemory, + "ignored".to_string(), + ) + .unwrap(); + let ends = home + .records("a") + .iter() + .filter(|r| matches!(r, LogRecord::EpochEnd { .. })) + .count(); + assert_eq!(ends, 1); + assert_eq!(s.epoch(), Epoch(1)); + assert_eq!( + s.carry(), + Some("already"), + "the summary already in EpochEnd is the carry" + ); +} + +#[test] +fn the_new_session_start_keeps_the_live_slot() { + let home = Home::new(); + let cfg = home.config(std::path::Path::new("/tmp/unused.sock")); + let baseline = Baseline::assemble(&cfg, &Registry::m2b()).unwrap(); + let mut s = Session::create(&home.dir, id("a"), baseline, 7).unwrap(); + compact::commit(&mut s, &cfg, &Registry::m2b(), &NoMemory, "s".to_string()).unwrap(); + assert_eq!(s.slot(), 7); + match &s.records()[0] { + LogRecord::SessionStart { slot, epoch, .. } => { + assert_eq!(*slot, 7); + assert_eq!(*epoch, Epoch(1)); + } + other => panic!("{other:?}"), + } +} diff --git a/docs/plans/M5d1/files/crates/loopd/tests/compact_due.rs b/docs/plans/M5d1/files/crates/loopd/tests/compact_due.rs new file mode 100644 index 0000000..53b6edd --- /dev/null +++ b/docs/plans/M5d1/files/crates/loopd/tests/compact_due.rs @@ -0,0 +1,79 @@ +//! Tests for when compaction is due and for the fallback summary. Do not edit. + +use bxcore::{Epoch, Hash32, LogRecord, SessionId, Timestamp, Usage}; +use loopd::compact::{due, fallback_summary}; + +fn usage(cache_n: u64, prompt_n: u64) -> Usage { + Usage { + cache_n, + prompt_n, + predicted_n: 0, + reasoning_tokens: 0, + thinking_capped: false, + } +} + +fn user(content: &str) -> LogRecord { + LogRecord::User { + time: Timestamp::parse("2026-09-18T08:00:00.000Z").unwrap(), + content: content.to_string(), + speaker: None, + } +} + +#[test] +fn due_at_the_threshold_and_not_below() { + assert!(!due(&usage(400, 449), 0.85, 1_000), "849 < 850"); + assert!(due(&usage(400, 450), 0.85, 1_000), "850 == 0.85 * 1000"); + assert!(due(&usage(0, 851), 0.85, 1_000)); + assert!(!due(&usage(0, 0), 0.85, 1_000)); +} + +#[test] +fn due_uses_this_agents_n_ctx() { + // 0.85 * 262144 = 222822.4, so 222823 is the first integer at or above. + assert!(!due(&usage(0, 222_822), 0.85, 262_144)); + assert!(due(&usage(0, 222_823), 0.85, 262_144)); + assert!(due(&usage(200_000, 22_823), 0.85, 262_144)); +} + +#[test] +fn a_threshold_of_one_is_the_full_window() { + assert!(!due(&usage(99, 0), 1.0, 100)); + assert!(due(&usage(100, 0), 1.0, 100)); +} + +#[test] +fn fallback_is_user_texts_newest_last() { + let records = vec![ + LogRecord::SessionStart { + time: Timestamp::parse("2026-09-18T08:00:00.000Z").unwrap(), + session: SessionId::new("a").unwrap(), + epoch: Epoch(0), + slot: 0, + baseline: Hash32::ZERO, + }, + user("first"), + user("second"), + user("third"), + ]; + assert_eq!( + fallback_summary(&records, 4_000), + "third\n\nsecond\n\nfirst" + ); +} + +#[test] +fn fallback_is_cut_to_summary_chars_on_a_character_boundary() { + let records = vec![user("café"), user("ab")]; + assert_eq!(fallback_summary(&records, 3), "ab\n"); + assert_eq!(fallback_summary(&records, 1), "a"); + assert_eq!(fallback_summary(&records, 0), ""); + let long = user("ååå"); + assert_eq!(fallback_summary(&[long], 2), "åå"); +} + +#[test] +fn fallback_with_no_users_is_empty() { + assert_eq!(fallback_summary(&[], 100), ""); +} diff --git a/docs/plans/M5d1/files/crates/loopd/tests/compact_summary.rs b/docs/plans/M5d1/files/crates/loopd/tests/compact_summary.rs new file mode 100644 index 0000000..a61c535 --- /dev/null +++ b/docs/plans/M5d1/files/crates/loopd/tests/compact_summary.rs @@ -0,0 +1,147 @@ +//! Tests for the compaction summary call. Do not edit. + +mod support; + +use bxcore::SessionId; +use loopd::baseline::Baseline; +use loopd::compact::{self, CompactError}; +use loopd::llama::Client; +use loopd::session::Session; +use loopd::tools::Registry; +use support::{FakeServer, Home, Reply}; + +const CHAT: &str = "/v1/chat/completions"; +const PROMPT: &str = "Summarise this conversation for the next epoch.\n"; + +fn sse(content: &str, cache_n: u64, prompt_n: u64) -> Reply { + let content_json = serde_json::to_string(content).unwrap(); + let data1 = format!( + r#"data: {{"choices":[{{"finish_reason":null,"index":0,"delta":{{"content":{content_json}}}}}],"id":"sum","timings":{{"cache_n":{cache_n},"prompt_n":{prompt_n},"predicted_n":1}}}}"# + ); + let data2 = format!( + r#"data: {{"choices":[{{"finish_reason":"stop","index":0,"delta":{{}}}}],"id":"sum","timings":{{"cache_n":{cache_n},"prompt_n":{prompt_n},"predicted_n":1}}}}"# + ); + let body = format!("{data1}\n\n{data2}\n\ndata: [DONE]\n\n"); + Reply::raw(format!( + "HTTP/1.1 200 OK\r\nContent-Type: text/event-stream\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}", + body.len() + )) +} + +fn session(home: &Home, cfg: &loopd::config::Config) -> Session { + let baseline = Baseline::assemble(cfg, &Registry::m2b()).unwrap(); + Session::create( + &home.dir, + SessionId::new("a").unwrap(), + baseline, + cfg.slots.main, + ) + .unwrap() +} + +#[test] +fn the_summary_call_is_on_the_background_slot_with_no_tools_and_thinking() { + let home = Home::new(); + home.write("compact.md", PROMPT); + let server = FakeServer::start(); + let cfg = home.config(&server.socket); + server.route(CHAT, vec![sse("the sum", 0, 10)]); + let client = Client::new(cfg.clone()); + let s = session(&home, &cfg); + let text = compact::summarise(&cfg, &client, &s).unwrap(); + assert_eq!(text, "the sum"); + let sent = server.requests_to(CHAT); + assert_eq!(sent.len(), 1); + let body = sent[0].json(); + assert_eq!(body["id_slot"], 1); + assert!(body.get("tools").is_none(), "{body}"); + assert_eq!(body["chat_template_kwargs"]["enable_thinking"], true); + let messages = body["messages"].as_array().unwrap(); + let last = messages.last().unwrap(); + assert_eq!(last["role"], "user"); + assert_eq!(last["content"], PROMPT.trim_end()); +} + +#[test] +fn the_summary_is_neutralised_and_capped() { + let home = Home::new(); + home.write("compact.md", PROMPT); + let server = FakeServer::start(); + let mut cfg = home.config(&server.socket); + cfg.compact.summary_chars = 4; + server.route(CHAT, vec![sse("abcdefghij", 0, 10)]); + let client = Client::new(cfg.clone()); + let s = session(&home, &cfg); + let text = compact::summarise(&cfg, &client, &s).unwrap(); + assert_eq!(text, "abcd"); +} + +#[test] +fn a_context_full_summary_uses_the_fallback() { + let home = Home::new(); + home.write("compact.md", PROMPT); + let server = FakeServer::start(); + let cfg = home.config(&server.socket); + server.route(CHAT, vec![Reply::fixture("context_full")]); + let client = Client::new(cfg.clone()); + let mut s = session(&home, &cfg); + s.append(bxcore::LogRecord::User { + time: bxcore::Timestamp::parse("2026-09-18T08:00:00.000Z").unwrap(), + content: "remember the tea".to_string(), + speaker: None, + }) + .unwrap(); + let text = compact::summarise(&cfg, &client, &s).unwrap(); + assert_eq!(text, "remember the tea"); +} + +#[test] +fn tags_in_the_completion_are_neutralised() { + let home = Home::new(); + home.write("compact.md", PROMPT); + let server = FakeServer::start(); + let cfg = home.config(&server.socket); + let tag = concat!("<", "think", ">"); + server.route(CHAT, vec![sse(&format!("hello {tag} world"), 0, 10)]); + let client = Client::new(cfg.clone()); + let s = session(&home, &cfg); + let text = compact::summarise(&cfg, &client, &s).unwrap(); + assert!(!text.contains(&tag), "{text}"); + assert!(text.contains("hello"), "{text}"); +} + +#[test] +fn a_missing_prompt_file_names_the_runbook() { + let home = Home::new(); + let server = FakeServer::start(); + let cfg = home.config(&server.socket); + let client = Client::new(cfg.clone()); + let s = session(&home, &cfg); + let err = compact::summarise(&cfg, &client, &s).unwrap_err(); + let e: Box = Box::new(err); + assert!(e.to_string().contains("compact.md"), "{e}"); + assert!( + e.to_string() + .ends_with("see docs/runbook.md#session-compact-failed"), + "{e}" + ); +} + +#[test] +fn an_inference_error_that_is_not_context_full_is_a_compact_failure() { + let home = Home::new(); + home.write("compact.md", PROMPT); + let server = FakeServer::start(); + let cfg = home.config(&server.socket); + server.route(CHAT, vec![Reply::fixture("bad_request")]); + let client = Client::new(cfg.clone()); + let s = session(&home, &cfg); + let err = compact::summarise(&cfg, &client, &s).unwrap_err(); + assert!(matches!(err, CompactError::Failed { .. }), "{err:?}"); + let e: Box = Box::new(err); + assert!( + e.to_string() + .ends_with("see docs/runbook.md#session-compact-failed"), + "{e}" + ); +} diff --git a/docs/plans/M5d1/files/crates/loopd/tests/compact_turn.rs b/docs/plans/M5d1/files/crates/loopd/tests/compact_turn.rs new file mode 100644 index 0000000..856e1cb --- /dev/null +++ b/docs/plans/M5d1/files/crates/loopd/tests/compact_turn.rs @@ -0,0 +1,257 @@ +//! Tests for compacting after a turn and retrying a full context. Do not edit. + +mod support; + +use bxcore::{Epoch, LogRecord, SessionId, TurnEvent}; +use loopd::baseline::Baseline; +use loopd::compact; +use loopd::llama::Client; +use loopd::memory::NoMemory; +use loopd::session::Session; +use loopd::tools::Registry; +use loopd::turn::{Runtime, TurnError, run_turn}; +use support::{FakeServer, Home, Reply}; + +const CHAT: &str = "/v1/chat/completions"; +const PROMPT: &str = "Summarise this conversation for the next epoch.\n"; + +fn sse(content: &str, cache_n: u64, prompt_n: u64) -> Reply { + let content_json = serde_json::to_string(content).unwrap(); + let data1 = format!( + r#"data: {{"choices":[{{"finish_reason":null,"index":0,"delta":{{"content":{content_json}}}}}],"id":"t","timings":{{"cache_n":{cache_n},"prompt_n":{prompt_n},"predicted_n":1}}}}"# + ); + let data2 = format!( + r#"data: {{"choices":[{{"finish_reason":"stop","index":0,"delta":{{}}}}],"id":"t","timings":{{"cache_n":{cache_n},"prompt_n":{prompt_n},"predicted_n":1}}}}"# + ); + let body = format!("{data1}\n\n{data2}\n\ndata: [DONE]\n\n"); + Reply::raw(format!( + "HTTP/1.1 200 OK\r\nContent-Type: text/event-stream\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}", + body.len() + )) +} + +struct World { + home: Home, + server: FakeServer, + cfg: loopd::config::Config, + client: Client, + port: support::ScriptedPort, + registry: Registry, +} + +impl World { + fn new() -> World { + let home = Home::new(); + home.write("compact.md", PROMPT); + let server = FakeServer::start(); + let mut cfg = home.config(&server.socket); + cfg.expect.n_ctx = 1_000; + cfg.compact.threshold = 0.85; + let client = Client::new(cfg.clone()); + World { + home, + server, + cfg, + client, + port: support::ScriptedPort::new(vec![]), + registry: Registry::m2b(), + } + } + + fn session(&self) -> Session { + let baseline = Baseline::assemble(&self.cfg, &self.registry).unwrap(); + Session::create( + &self.home.dir, + SessionId::new("a").unwrap(), + baseline, + self.cfg.slots.main, + ) + .unwrap() + } + + fn runtime(&self) -> Runtime<'_> { + Runtime { + cfg: &self.cfg, + client: &self.client, + port: &self.port, + registry: &self.registry, + } + } + + fn finish( + &self, + session: &mut Session, + text: &str, + outcome: Result, + ) -> Result { + let mut events = Vec::new(); + compact::finish_turn( + session, + &self.runtime(), + &NoMemory, + text, + None, + &mut |e: &TurnEvent| events.push(e.clone()), + outcome, + ) + } +} + +#[test] +fn usage_under_the_threshold_leaves_the_epoch() { + let w = World::new(); + w.server.route(CHAT, vec![sse("ok", 400, 449)]); + let mut s = w.session(); + let mut events = Vec::new(); + let outcome = run_turn(&mut s, &w.runtime(), "hi", None, &mut |e| { + events.push(e.clone()) + }); + let outcome = w.finish(&mut s, "hi", outcome).unwrap(); + assert_eq!(outcome.content, "ok"); + assert_eq!(s.epoch(), Epoch(0)); + assert_eq!(w.server.requests_to(CHAT).len(), 1); +} + +#[test] +fn usage_at_the_threshold_compacts_before_the_turn_returns() { + let w = World::new(); + w.server + .route(CHAT, vec![sse("ok", 400, 450), sse("the sum", 0, 10)]); + let mut s = w.session(); + let mut events = Vec::new(); + let outcome = run_turn(&mut s, &w.runtime(), "hi", None, &mut |e| { + events.push(e.clone()) + }); + w.finish(&mut s, "hi", outcome).unwrap(); + assert_eq!(s.epoch(), Epoch(1)); + assert_eq!(s.carry(), Some("the sum")); + let sent = w.server.requests_to(CHAT); + assert_eq!(sent.len(), 2); + assert_eq!(sent[0].json()["id_slot"], 0); + assert_eq!(sent[1].json()["id_slot"], 1); + assert!(sent[1].json().get("tools").is_none()); + let old = w.home.records("a"); + assert!(matches!( + old.last(), + Some(LogRecord::EpochEnd { next, .. }) if *next == Epoch(1) + )); +} + +#[test] +fn session_full_compacts_and_retries_the_user_message_once() { + let w = World::new(); + w.server.route( + CHAT, + vec![ + Reply::fixture("context_full"), + sse("the sum", 0, 10), + sse("retried", 0, 20), + ], + ); + let mut s = w.session(); + let mut events = Vec::new(); + let outcome = run_turn(&mut s, &w.runtime(), "please fit", None, &mut |e| { + events.push(e.clone()) + }); + assert!(matches!(outcome, Err(TurnError::SessionFull))); + let done = w.finish(&mut s, "please fit", outcome).unwrap(); + assert_eq!(done.content, "retried"); + assert_eq!(s.epoch(), Epoch(1)); + let users: Vec<_> = s + .records() + .iter() + .filter_map(|r| match r { + LogRecord::User { content, .. } => Some(content.as_str()), + _ => None, + }) + .collect(); + assert_eq!(users, ["please fit"]); + let old_users = w + .home + .records("a") + .iter() + .filter(|r| matches!(r, LogRecord::User { .. })) + .count(); + assert_eq!(old_users, 1); + assert_eq!(s.turns(), 2); + let sent = w.server.requests_to(CHAT); + assert_eq!(sent.len(), 3); + assert_eq!(sent[1].json()["id_slot"], 1); + assert_eq!(sent[2].json()["id_slot"], 0); +} + +#[test] +fn session_full_stays_session_full_when_compact_fails() { + let w = World::new(); + w.server.route( + CHAT, + vec![ + Reply::fixture("context_full"), + Reply::fixture("bad_request"), + ], + ); + let mut s = w.session(); + let mut events = Vec::new(); + let outcome = run_turn(&mut s, &w.runtime(), "x", None, &mut |e| { + events.push(e.clone()) + }); + let err = w.finish(&mut s, "x", outcome).unwrap_err(); + assert!(matches!(err, TurnError::SessionFull), "{err:?}"); + assert_eq!(s.epoch(), Epoch(0)); +} + +#[test] +fn a_successful_turn_stays_successful_when_compact_fails() { + let home = Home::new(); + let server = FakeServer::start(); + let mut cfg = home.config(&server.socket); + cfg.expect.n_ctx = 1_000; + cfg.compact.threshold = 0.85; + let client = Client::new(cfg.clone()); + server.route(CHAT, vec![sse("ok", 400, 450)]); + let baseline = Baseline::assemble(&cfg, &Registry::m2b()).unwrap(); + let mut s = Session::create( + &home.dir, + SessionId::new("a").unwrap(), + baseline, + cfg.slots.main, + ) + .unwrap(); + let port = support::ScriptedPort::new(vec![]); + let registry = Registry::m2b(); + let rt = Runtime { + cfg: &cfg, + client: &client, + port: &port, + registry: ®istry, + }; + let mut events = Vec::new(); + let outcome = run_turn(&mut s, &rt, "hi", None, &mut |e| events.push(e.clone())); + let done = compact::finish_turn( + &mut s, + &rt, + &NoMemory, + "hi", + None, + &mut |e| events.push(e.clone()), + outcome, + ) + .unwrap(); + assert_eq!(done.content, "ok"); + assert_eq!(s.epoch(), Epoch(0), "compact.md is missing, so no epoch"); +} + +#[test] +fn finish_turn_does_not_compact_an_error_other_than_session_full() { + let w = World::new(); + w.server.route(CHAT, vec![Reply::fixture("bad_request")]); + let mut s = w.session(); + let mut events = Vec::new(); + let outcome = run_turn(&mut s, &w.runtime(), "x", None, &mut |e| { + events.push(e.clone()) + }); + let err = w.finish(&mut s, "x", outcome).unwrap_err(); + assert!(matches!(err, TurnError::Infer(_)), "{err:?}"); + assert_eq!(s.epoch(), Epoch(0)); + assert_eq!(w.server.requests_to(CHAT).len(), 1); +} diff --git a/docs/plans/M5d1/files/crates/loopd/tests/config_compact.rs b/docs/plans/M5d1/files/crates/loopd/tests/config_compact.rs new file mode 100644 index 0000000..b07dd40 --- /dev/null +++ b/docs/plans/M5d1/files/crates/loopd/tests/config_compact.rs @@ -0,0 +1,119 @@ +//! Tests for `[compact]` on `loopd.toml`. Do not edit. + +use loopd::config::Config; +use std::path::{Path, PathBuf}; + +fn fixture(name: &str) -> PathBuf { + Path::new(env!("CARGO_MANIFEST_DIR")) + .join("tests/fixtures/config") + .join(name) +} + +fn minimal() -> String { + std::fs::read_to_string(fixture("minimal.toml")).unwrap() +} + +#[test] +fn missing_table_gets_the_documented_defaults() { + let c = Config::load(&fixture("minimal.toml")).unwrap(); + assert_eq!(c.compact.threshold, 0.85); + assert_eq!(c.compact.summary_chars, 4_000); + assert_eq!(c.compact.summarise, vec!["http_fetch".to_string()]); +} + +#[test] +fn the_table_overrides_every_default() { + let text = format!( + "{}\n[compact]\nthreshold = 0.5\nsummary_chars = 100\nsummarise = [\"clock\"]\n", + minimal() + ); + let c = Config::parse(&text).unwrap(); + assert_eq!(c.compact.threshold, 0.5); + assert_eq!(c.compact.summary_chars, 100); + assert_eq!(c.compact.summarise, vec!["clock".to_string()]); +} + +#[test] +fn a_partial_table_keeps_the_other_defaults() { + let text = format!("{}\n[compact]\nthreshold = 0.9\n", minimal()); + let c = Config::parse(&text).unwrap(); + assert_eq!(c.compact.threshold, 0.9); + assert_eq!(c.compact.summary_chars, 4_000); + assert_eq!(c.compact.summarise, vec!["http_fetch".to_string()]); +} + +#[test] +fn unknown_keys_are_errors() { + let text = minimal(); + let bad = format!("{text}\n[compact]\nzz_unknown = 1\n"); + assert!( + Config::parse(&bad).is_err(), + "[compact] accepted an unknown key" + ); + let misspelt = format!("{text}\n[compact]\nsummary_char = 10\n"); + assert!( + Config::parse(&misspelt).is_err(), + "[compact] accepted a misspelt key" + ); +} + +#[test] +fn empty_summarise_is_allowed() { + let text = format!("{}\n[compact]\nsummarise = []\n", minimal()); + let c = Config::parse(&text).unwrap(); + assert!(c.compact.summarise.is_empty()); +} + +#[test] +fn compact_prompt_is_compact_md_beside_system_md() { + let c = Config::load(&fixture("minimal.toml")).unwrap(); + assert_eq!(c.compact_prompt(), fixture("compact.md")); + let home = Scratch::new(); + let cfg_path = home.dir.join("loopd.toml"); + std::fs::write(&cfg_path, minimal()).unwrap(); + std::fs::write(home.dir.join("system.md"), "sys\n").unwrap(); + let c = Config::load(&cfg_path).unwrap(); + assert_eq!(c.compact_prompt(), home.dir.join("compact.md")); +} + +struct Scratch { + dir: PathBuf, +} + +impl Scratch { + fn new() -> Scratch { + let dir = std::env::temp_dir().join(format!( + "loopd-compact-cfg-{}-{}", + std::process::id(), + std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .unwrap() + .as_nanos() + )); + std::fs::create_dir_all(&dir).unwrap(); + Scratch { dir } + } +} + +impl Drop for Scratch { + fn drop(&mut self) { + let _ = std::fs::remove_dir_all(&self.dir); + } +} + +#[test] +fn compact_prompt_follows_an_overridden_system_path() { + let text = std::fs::read_to_string(fixture("m2b.toml")).unwrap(); + let c = Config::load(&fixture("m2b.toml")).unwrap(); + assert!( + text.contains("prompts/agent.md"), + "m2b fixture names a prompt in a subdirectory" + ); + assert_eq!(c.compact_prompt(), fixture("prompts/compact.md")); +} + +#[test] +fn defaults_match_the_struct_default() { + let c = Config::load(&fixture("minimal.toml")).unwrap(); + assert_eq!(c.compact, loopd::config::Compact::default()); +} diff --git a/docs/plans/M5d1/files/crates/loopd/tests/session_epochs.rs b/docs/plans/M5d1/files/crates/loopd/tests/session_epochs.rs new file mode 100644 index 0000000..7716b80 --- /dev/null +++ b/docs/plans/M5d1/files/crates/loopd/tests/session_epochs.rs @@ -0,0 +1,255 @@ +//! Tests for multi-epoch session open, turns, trust and call ids. Do not edit. + +mod support; + +use bxcore::{CallId, DataClass, Epoch, LogRecord, SessionId, Timestamp}; +use loopd::baseline::Baseline; +use loopd::session::{Session, SessionError}; +use support::Home; + +fn baseline() -> Baseline { + Baseline { + system: "sys".to_string(), + tools: vec![], + } +} + +fn later() -> Baseline { + Baseline { + system: "sys-epoch-1".to_string(), + tools: vec![], + } +} + +fn id(s: &str) -> SessionId { + SessionId::new(s).unwrap() +} + +fn ts() -> Timestamp { + Timestamp::parse("2026-09-18T08:00:00.000Z").unwrap() +} + +fn user(content: &str) -> LogRecord { + LogRecord::User { + time: ts(), + content: content.to_string(), + speaker: None, + } +} + +fn write_jsonl(home: &Home, session: &str, epoch: u32, records: &[LogRecord]) { + let text: String = records + .iter() + .map(|r| format!("{}\n", serde_json::to_string(r).unwrap())) + .collect(); + home.write(&format!("sessions/{session}/{epoch}.jsonl"), &text); +} + +fn write_baseline(home: &Home, session: &str, epoch: u32, b: &Baseline) { + home.write( + &format!("sessions/{session}/{epoch}.baseline.json"), + &b.to_json().unwrap(), + ); +} + +fn start(session: &str, epoch: u32, slot: u32, b: &Baseline) -> LogRecord { + LogRecord::SessionStart { + time: ts(), + session: id(session), + epoch: Epoch(epoch), + slot, + baseline: b.hash().unwrap(), + } +} + +fn epoch_end(next: u32, summary: &str) -> LogRecord { + LogRecord::EpochEnd { + time: ts(), + next: Epoch(next), + summary: summary.to_string(), + } +} + +fn tool_result(call: u64, untrusted: bool) -> LogRecord { + LogRecord::ToolResult { + time: ts(), + call: CallId(call), + tool_call_id: format!("c{call}"), + content: "out".to_string(), + class: DataClass::Public, + untrusted, + truncated: false, + } +} + +#[test] +fn open_loads_the_highest_complete_epoch_and_its_carry() { + let home = Home::new(); + let mut s = Session::create(&home.dir, id("a"), baseline(), 3).unwrap(); + s.append(user("one")).unwrap(); + s.append(user("two")).unwrap(); + s.append(epoch_end(1, "carry-text")).unwrap(); + drop(s); + write_baseline(&home, "a", 1, &later()); + write_jsonl(&home, "a", 1, &[start("a", 1, 3, &later()), user("three")]); + + let s = Session::open(&home.dir, id("a")).unwrap(); + assert_eq!(s.epoch(), Epoch(1)); + assert_eq!(s.slot(), 3); + assert_eq!(s.carry(), Some("carry-text")); + assert_eq!(s.baseline(), &later()); + assert_eq!(s.records().len(), 2); + assert!(matches!(s.records()[1], LogRecord::User { ref content, .. } if content == "three")); +} + +#[test] +fn a_missing_carry_on_epoch_1_is_an_error() { + let home = Home::new(); + let s = Session::create(&home.dir, id("a"), baseline(), 0).unwrap(); + drop(s); + write_baseline(&home, "a", 1, &later()); + write_jsonl(&home, "a", 1, &[start("a", 1, 0, &later())]); + + match Session::open(&home.dir, id("a")) { + Err(SessionError::NoCarry { ref path, .. }) => { + assert!(path.ends_with("0.jsonl"), "{path:?}"); + } + other => panic!("{other:?}"), + } + let e: Box = + Box::new(Session::open(&home.dir, id("a")).map(|_| ()).unwrap_err()); + assert!(e.to_string().contains("0.jsonl"), "{e}"); + assert!( + e.to_string() + .contains("see docs/runbook.md#session-log-damaged"), + "{e}" + ); +} + +#[test] +fn epoch_0_has_no_carry() { + let home = Home::new(); + let s = Session::create(&home.dir, id("a"), baseline(), 7).unwrap(); + drop(s); + let s = Session::open(&home.dir, id("a")).unwrap(); + assert_eq!(s.epoch(), Epoch(0)); + assert_eq!(s.carry(), None); + assert_eq!(s.slot(), 7); +} + +#[test] +fn turns_count_users_across_every_epoch() { + let home = Home::new(); + let mut s = Session::create(&home.dir, id("a"), baseline(), 0).unwrap(); + s.append(user("one")).unwrap(); + s.append(user("two")).unwrap(); + s.append(epoch_end(1, "sum")).unwrap(); + drop(s); + write_baseline(&home, "a", 1, &later()); + write_jsonl(&home, "a", 1, &[start("a", 1, 0, &later()), user("three")]); + let s = Session::open(&home.dir, id("a")).unwrap(); + assert_eq!(s.turns(), 3); + assert_eq!( + s.records() + .iter() + .filter(|r| matches!(r, LogRecord::User { .. })) + .count(), + 1, + "records() is the current epoch only" + ); +} + +#[test] +fn untrusted_reads_every_epoch() { + let home = Home::new(); + let mut s = Session::create(&home.dir, id("a"), baseline(), 0).unwrap(); + s.append(tool_result(1, true)).unwrap(); + s.append(epoch_end(1, "sum")).unwrap(); + drop(s); + write_baseline(&home, "a", 1, &later()); + write_jsonl( + &home, + "a", + 1, + &[start("a", 1, 0, &later()), tool_result(2, false)], + ); + let s = Session::open(&home.dir, id("a")).unwrap(); + assert!(s.untrusted(), "an untrusted result in epoch 0 still taints"); + assert!( + !s.records().iter().any(|r| matches!( + r, + LogRecord::ToolResult { + untrusted: true, + .. + } + )), + "the current epoch's records look clean" + ); +} + +#[test] +fn call_ids_continue_across_epochs() { + let home = Home::new(); + let mut s = Session::create(&home.dir, id("a"), baseline(), 0).unwrap(); + s.append(tool_result(4, false)).unwrap(); + s.append(epoch_end(1, "sum")).unwrap(); + drop(s); + write_baseline(&home, "a", 1, &later()); + write_jsonl( + &home, + "a", + 1, + &[start("a", 1, 0, &later()), tool_result(2, false)], + ); + let mut s = Session::open(&home.dir, id("a")).unwrap(); + assert_eq!( + s.next_call(), + CallId(5), + "one more than the highest call in any epoch" + ); +} + +#[test] +fn append_writes_the_current_epoch_file() { + let home = Home::new(); + let mut s = Session::create(&home.dir, id("a"), baseline(), 0).unwrap(); + s.append(epoch_end(1, "sum")).unwrap(); + drop(s); + write_baseline(&home, "a", 1, &later()); + write_jsonl(&home, "a", 1, &[start("a", 1, 0, &later())]); + let mut s = Session::open(&home.dir, id("a")).unwrap(); + s.append(user("after")).unwrap(); + let text = home.read("sessions/a/1.jsonl"); + assert!(text.contains("after"), "{text}"); + let old = home.read("sessions/a/0.jsonl"); + assert!(!old.contains("after"), "{old}"); +} + +#[test] +fn an_incomplete_next_epoch_is_ignored() { + let home = Home::new(); + let mut s = Session::create(&home.dir, id("a"), baseline(), 0).unwrap(); + s.append(user("still-here")).unwrap(); + drop(s); + write_baseline(&home, "a", 1, &later()); + let s = Session::open(&home.dir, id("a")).unwrap(); + assert_eq!(s.epoch(), Epoch(0)); + assert_eq!(s.carry(), None); + assert_eq!(s.turns(), 1); +} + +#[test] +fn epoch_end_without_the_next_files_stays_on_the_old_epoch() { + let home = Home::new(); + let mut s = Session::create(&home.dir, id("a"), baseline(), 0).unwrap(); + s.append(user("x")).unwrap(); + s.append(epoch_end(1, "sum")).unwrap(); + drop(s); + let s = Session::open(&home.dir, id("a")).unwrap(); + assert_eq!(s.epoch(), Epoch(0)); + assert_eq!(s.carry(), None); + assert_eq!(s.turns(), 1); + assert!( + matches!(s.records().last(), Some(LogRecord::EpochEnd { next, .. }) if *next == Epoch(1)) + ); +} diff --git a/docs/plans/M5d1/files/crates/memoryd/tests/honcho_epochs.rs b/docs/plans/M5d1/files/crates/memoryd/tests/honcho_epochs.rs new file mode 100644 index 0000000..5be528e --- /dev/null +++ b/docs/plans/M5d1/files/crates/memoryd/tests/honcho_epochs.rs @@ -0,0 +1,181 @@ +//! Honcho reads every epoch file of a session, in order. Do not edit. + +#[path = "support/logs.rs"] +mod logs; +#[path = "support/tmp.rs"] +mod tmp; +#[path = "support/turns.rs"] +mod turns; + +use std::cell::RefCell; + +use bxcore::{Epoch, LogRecord, SessionId, Timestamp}; +use logs::{assistant, write_log}; +use memoryd::honcho::decide::{turn_records, turns}; +use memoryd::honcho::ledger::{Action, Item, Ledger, Record, Workspace}; +use memoryd::honcho::turns::{LOG_RUNBOOK, TurnSync, read_session}; +use tmp::TempDir; +use turns::{clean, sid, spoken}; + +const LEDGER: &str = "honcho.jsonl"; + +fn epoch_end(summary: &str) -> LogRecord { + LogRecord::EpochEnd { + time: Timestamp::parse("2026-09-28T10:00:00.000Z").unwrap(), + next: Epoch(1), + summary: summary.to_string(), + } +} + +fn write_epoch(home: &std::path::Path, session: &str, epoch: u32, records: &[LogRecord]) { + let dir = home.join("sessions").join(session); + std::fs::create_dir_all(&dir).unwrap(); + let path = dir.join(format!("{epoch}.jsonl")); + let text: String = records + .iter() + .map(|r| format!("{}\n", serde_json::to_string(r).unwrap())) + .collect(); + std::fs::write(path, text).unwrap(); +} + +struct World { + sync: TurnSync, + answer: RefCell>, + dir: TempDir, +} + +impl World { + fn new(tag: &str) -> World { + let dir = TempDir::new(tag); + std::fs::create_dir_all(dir.path().join("memory")).unwrap(); + let mut ledger = Ledger::open(&dir.path().join("memory").join(LEDGER)).unwrap(); + ledger + .append(Record::Began { + time: Timestamp::parse("2026-09-28T09:00:00.000Z").unwrap(), + }) + .unwrap(); + World { + sync: TurnSync::new(ledger, "boxmaker0", &dir.path().join("sessions")), + answer: RefCell::new(Ok(clean())), + dir, + } + } + + fn updated(&mut self, session: &str, turn: u64) { + let ask = |s: &SessionId| { + let _ = s; + self.answer.borrow().clone() + }; + let log = |_line: &str| {}; + let now = Timestamp::parse("2026-09-28T10:00:00.000Z").unwrap(); + self.sync.updated(&sid(session), turn, &ask, now, &log); + } + + fn action(&self, session: &str, turn: u64) -> Option { + let item = Item::Turn { + session: sid(session), + turn, + }; + let path = self.dir.path().join("memory").join(LEDGER); + let text = std::fs::read_to_string(path).unwrap_or_default(); + let key = serde_json::to_string(&item).unwrap(); + let line = text.lines().find(|l| l.contains(&key))?; + let value: serde_json::Value = serde_json::from_str(line).unwrap(); + Some(serde_json::from_value(value["action"].clone()).unwrap()) + } +} + +fn to(workspace: Workspace, peer: &str) -> Action { + Action::To { + workspace, + peer: peer.to_string(), + } +} + +#[test] +fn concatenated_logs_count_three_turns() { + let dir = TempDir::new("honcho-epochs-count"); + let home = dir.path(); + let epoch0 = vec![ + spoken("one", Some("kyle")), + assistant(2, Some("ok"), &[]), + spoken("two", Some("kyle")), + assistant(3, Some("ok"), &[]), + epoch_end("sum"), + ]; + write_log(home, "mm-a", &epoch0); + write_epoch( + home, + "mm-a", + 1, + &[ + spoken("three", Some("kyle")), + assistant(4, Some("ok"), &[]), + ], + ); + let records = read_session(&home.join("sessions"), &sid("mm-a")).unwrap(); + assert_eq!(turns(&records), 3); + let turn3 = turn_records(&records, 3).unwrap(); + assert!( + matches!(turn3[0], LogRecord::User { ref content, .. } if content == "three"), + "{turn3:?}" + ); +} + +#[test] +fn epoch_0_alone_still_reads() { + let dir = TempDir::new("honcho-epochs-zero"); + let home = dir.path(); + write_log(home, "mm-a", &[spoken("only", Some("kyle"))]); + let records = read_session(&home.join("sessions"), &sid("mm-a")).unwrap(); + assert_eq!(turns(&records), 1); +} + +#[test] +fn a_damaged_later_epoch_is_an_error_naming_that_file() { + let dir = TempDir::new("honcho-epochs-damaged"); + let home = dir.path(); + write_log(home, "mm-a", &[spoken("one", Some("kyle"))]); + let path = home.join("sessions/mm-a/1.jsonl"); + std::fs::write(&path, "{\"type\":\"zz\"}\n").unwrap(); + let err = read_session(&home.join("sessions"), &sid("mm-a")).unwrap_err(); + assert!(err.to_string().contains("1.jsonl"), "{err}"); + assert!( + err.to_string().contains(LOG_RUNBOOK) || err.to_string().contains("session-log-damaged"), + "{err}" + ); +} + +#[test] +fn turn_sync_decides_a_turn_that_lives_in_epoch_1() { + let mut w = World::new("honcho-epochs-sync"); + write_log( + w.dir.path(), + "mm-a", + &[ + spoken("one", Some("kyle")), + assistant(2, Some("ok"), &[]), + spoken("two", Some("kyle")), + assistant(3, Some("ok"), &[]), + epoch_end("sum"), + ], + ); + write_epoch( + w.dir.path(), + "mm-a", + 1, + &[ + spoken("three", Some("kyle")), + assistant(4, Some("ok"), &[]), + ], + ); + w.updated("mm-a", 3); + assert_eq!(w.action("mm-a", 3), Some(to(Workspace::Trusted, "kyle"))); +} + +#[test] +fn a_missing_session_directory_is_empty() { + let dir = TempDir::new("honcho-epochs-missing"); + let records = read_session(&dir.path().join("sessions"), &sid("mm-none")).unwrap(); + assert!(records.is_empty()); +} diff --git a/docs/plans/M5d1/files/runbook-session-compact-failed.md b/docs/plans/M5d1/files/runbook-session-compact-failed.md new file mode 100644 index 0000000..ec184eb --- /dev/null +++ b/docs/plans/M5d1/files/runbook-session-compact-failed.md @@ -0,0 +1,35 @@ +## session-compact-failed + +**What you see.** `loopd` logs that it could not compact a session, then this entry. The session +stays on its current epoch. A turn that was already too large to fit (`session_full`) stays +`session_full`. A turn that finished stays finished; the next turn on that session may be +`session_full` if it would not fit. + +```text +loopd: could not compact session chat-0-0: compact.md: No such file or directory (os error 2); see docs/runbook.md#session-compact-failed +``` + +**Why.** Compaction needs a summary of the current epoch (a completion on the background slot +with `compact.md` as the last user message, or the fallback of that epoch's owner texts). If the +prompt file cannot be read, the summary call fails for a reason other than a full context, or +writing the next epoch's files fails, `loopd` refuses to guess a prefix. It does not start a new +epoch from an empty summary when the model call failed. + +**Confirm.** + +```sh +ls -l "$(dirname "$BOXMAKER_HOME/../config")/compact.md" 2>/dev/null +ls -l "$BOXMAKER_HOME/sessions/"/ +tail -n 5 "$BOXMAKER_HOME/sessions//0.jsonl" +``` + +`compact.md` lives next to `system.md` (the path in `loopd.toml`'s `[baseline] system`). A last +line of type `epoch_end` without `.baseline.json` and `.jsonl` is a compact that stopped +halfway; the next compact resumes it. + +**Fix.** Restore `compact.md` beside `system.md` (the starter is `config/compact.md` in the +repository). Check that `inferproxy` still answers. If the session directory is mid-write, do not +delete `EpochEnd`; the next compact continues from it. + +**Check.** The next turn on that session either runs on the current epoch, or compact succeeds +and the next turn's log is `.jsonl` with `epoch` n. diff --git a/docs/plans/M5d2/01-loopd-session-slot.md b/docs/plans/M5d2/01-loopd-session-slot.md new file mode 100644 index 0000000..5386730 --- /dev/null +++ b/docs/plans/M5d2/01-loopd-session-slot.md @@ -0,0 +1,124 @@ +# M5d-2 task 01: `run_turn` sends `session.slot()` + +**Branch:** `m5d2`. Check it: `git branch --show-current` prints `m5d2` and `git status --short` +is empty; otherwise stop. Never switch branch. +**Commit subject:** `loopd: live turns use the session's slot` +**Log row:** the Task cell is `M5d2/01-loopd-session-slot`. + +## Goal + +Heartbeat and cron run on the background slot (spec 4). Live Mattermost and `bxctl chat` +sessions keep `cfg.slots.main`. Today `run_turn` always sends `slots.main`. This task makes it +send `session.slot()`: the slot stored on that session's `SessionStart` (M5d-1 task 02). + +Nothing else changes. Compaction's summary call already uses `slots.background`. The scheduler +does not exist yet. + +## Every place that changes + +| Place | What happens there | +|---|---| +| `crates/loopd/src/turn.rs` | `ChatRequest.slot` is `session.slot()` (edit T1) | +| `crates/loopd/tests/session_slot.rs` | new, given | +| everything else | **no change**: not `session.rs`, not `compact.rs`, not channel | + +## Files + +- Copy, from `docs/plans/M5d2/files/` (the same path under it): the new test + `crates/loopd/tests/session_slot.rs`. +- Modify: `crates/loopd/src/turn.rs`, `docs/implementer-log.md`. +- Do not touch: any other file. + +## Facts you need + +- **`Session::slot()`** returns the `u32` stored on create / on the current epoch's + `SessionStart`. It is already `pub` (M5d-1 task 02). +- **`messages_with_carry`** stays as it is. Only the `slot:` line changes. +- **Existing turn tests** create sessions with `cfg.slots.main` (0). They still send 0. +- The items the edits use. This table is complete. + + | Item | What it does | + |---|---| + | `Session::slot` | the live slot for this session | + | `ChatRequest.slot` | becomes `id_slot` on the wire | + +## The edits, exactly + +Each quoted "before" block occurs exactly once after M5d-1 task 08. + +### `crates/loopd/src/turn.rs` + +**T1.** The lines + +```rust + let request = ChatRequest { + slot: rt.cfg.slots.main, + messages: messages_with_carry(baseline, session.records(), session.carry()), + tools: baseline.tools.clone(), + thinking: true, + }; +``` + +become + +```rust + let request = ChatRequest { + slot: session.slot(), + messages: messages_with_carry(baseline, session.records(), session.carry()), + tools: baseline.tools.clone(), + thinking: true, + }; +``` + +## Steps + +- [ ] **1. Copy the given test.** From the repository root, one command: + + ```sh + cp docs/plans/M5d2/files/crates/loopd/tests/session_slot.rs crates/loopd/tests/ + ``` + + Then `git status --short` lists one new file, `crates/loopd/tests/session_slot.rs`. +- [ ] **2. See it fail.** `cargo test -p loopd --test session_slot`, with a shell timeout of + 600000 ms: the four tests compile (they call `session.slot()`, which exists) and + `a_session_created_on_the_background_slot_sends_that_slot` fails: `id_slot` is `0`, not `1`. + `a_main_slot_session_still_sends_main` may pass. `the_request_slot_follows_session_start_not_the_config_main` + fails: `id_slot` is `0`, not `7`. +- [ ] **3. The edit.** Make edit T1. `cargo check -p loopd` prints no error and no warning. +- [ ] **4. The tests.** `cargo test -p loopd --test session_slot --test turn --test channel + --test compact_turn`: every test passes, 4 in `session_slot` and the others as before. +- [ ] **5. Check.** `git diff --stat` lists exactly `crates/loopd/src/turn.rs`. +- [ ] **6. Run the gate.** `cargo fmt --all`, then `make gate`, with a shell timeout of 600000 ms. + Expected last line: `gate: ok`. The test count is **1511** after this task, 1507 before it, + and the 4 new ones are the tests of `session_slot.rs`. To count, add up the `N passed` of + every `test result:` line the gate's `cargo test --workspace` printed, and add 1. Never run + the gate again only to count. +- [ ] **7. Log and commit.** Add your row to `docs/implementer-log.md`; its Task cell is + `M5d2/01-loopd-session-slot`. Stage with + `git add crates/loopd docs/implementer-log.md Cargo.lock`, then commit as `AGENTS.md` says. + `Cargo.lock` does not change in this task; the `git add` is harmless. + +## Done when + +- `make gate` prints `gate: ok` with 1511 tests. +- `git diff HEAD~1 --stat` lists only `crates/loopd/src/turn.rs`, + `crates/loopd/tests/session_slot.rs` and `docs/implementer-log.md`. + +## Stop and report if + +- The branch check at the top fails. +- A "before" block is not in its file exactly as shown, or is there more than once. +- Step 2 shows a different result: the tests do not compile, or a different assertion fails. +- `cargo check` in step 3 prints an error or a warning. Quote it, and stop. +- A test fails in step 4. Do not change the test: stop, and give the failing tests' names and + their messages. +- The fix seems to need a change anywhere else. It does not; stop and say why you think so. +- A test outside `session_slot.rs` fails in the gate: check once whether it fails without your + change (`git stash -u`, run it, `git stash pop`), then stop and report both results. +- The gate passes with a count other than 1511. + +## What the given tests pin + +A session created with `cfg.slots.background` sends that slot; one created with `cfg.slots.main` +still sends main; a session created with slot `7` sends `7` even when `cfg.slots.main` is `0`; a +heartbeat turn between two main turns does not change the main session's message prefix. diff --git a/docs/plans/M5d2/02-bxcore-tool-summary.md b/docs/plans/M5d2/02-bxcore-tool-summary.md new file mode 100644 index 0000000..f91895a --- /dev/null +++ b/docs/plans/M5d2/02-bxcore-tool-summary.md @@ -0,0 +1,165 @@ +# M5d-2 task 02: `LogRecord::ToolSummary` + +**Branch:** `m5d2`. Check it: `git branch --show-current` prints `m5d2` and `git status --short` +is empty; otherwise stop. Never switch branch. +**Commit subject:** `bxcore: ToolSummary log record; messages ignore it` +**Log row:** the Task cell is `M5d2/02-bxcore-tool-summary`. + +## Goal + +A tool that opts in gets a second record, for the index only (spec 4.3): + +```text +LogRecord::ToolSummary { time, call, tool_call_id, content, class, untrusted } +``` + +JSON type `tool_summary`. `messages()` / `messages_with_carry` ignore it, so the epoch's prefix +does not change. `support/turn.rs` `types` names it (T19: this task ships its own copy). + +Nothing summarises yet. That is task 08. + +## Every place that changes + +| Place | What happens there | +|---|---| +| `crates/bxcore/src/log.rs` | the variant, after `ToolResult` (edit L1) | +| `crates/loopd/src/baseline.rs` | ignore arm (edit B1) | +| `crates/loopd/tests/support/turn.rs` | replaced, given | +| `crates/bxcore/tests/tool_summary.rs` | new, given | +| `crates/loopd/tests/tool_summary_messages.rs` | new, given | +| everything else | **no change**: not `index.rs` (task 09), not `turn.rs` | + +`memoryd/src/index.rs` `index_record` already has `_ => {}`. It compiles. Summaries are skipped +until task 09. + +## Files + +- Copy, from `docs/plans/M5d2/files/` (the same path under it): the new tests + `crates/bxcore/tests/tool_summary.rs` and + `crates/loopd/tests/tool_summary_messages.rs`, and the replacement + `crates/loopd/tests/support/turn.rs`. +- Modify: `crates/bxcore/src/log.rs`, `crates/loopd/src/baseline.rs`, + `docs/implementer-log.md`. +- Do not touch: any other file. + +## Facts you need + +- **Field order is the wire format.** `time`, `call`, `tool_call_id`, `content`, `class`, + `untrusted`. No `truncated`: a summary is already capped. +- **Unknown fields are errors**, as on every `LogRecord` variant (`deny_unknown_fields` is on + the enum). +- **`types` is exhaustive.** Adding the variant without the replacement `support/turn.rs` fails + to compile `turn` / `limits` tests. Copy the given file over the existing one. The only + intended change is the `ToolSummary => "tool_summary"` arm. `Setup::turn` still returns + `(Result, Vec)`. +- **`messages` names every variant.** The new arm is empty, like `Usage` / `CacheLoss` / + `EpochEnd`. +- The items the edits use. This table is complete. + + | Item | What it does | + |---|---| + | `CallId` | already imported in `log.rs` | + | `DataClass` | already imported in `log.rs` | + +## The edits, exactly + +Each quoted "before" block occurs exactly once after task 01. + +### `crates/bxcore/src/log.rs` + +**L1.** After the `ToolResult` variant (the lines) + +```rust + truncated: bool, + }, + Usage { +``` + +become + +```rust + truncated: bool, + }, + ToolSummary { + time: Timestamp, + call: CallId, + tool_call_id: String, + content: String, + class: DataClass, + untrusted: bool, + }, + Usage { +``` + +Keep the fields in that order. + +### `crates/loopd/src/baseline.rs` + +**B1.** After the `ToolResult` arm (the lines) + +```rust + } => out.push(ChatMessage::Tool { + tool_call_id: tool_call_id.clone(), + content: content.clone(), + }), + LogRecord::Usage { .. } => {} +``` + +become + +```rust + } => out.push(ChatMessage::Tool { + tool_call_id: tool_call_id.clone(), + content: content.clone(), + }), + LogRecord::ToolSummary { .. } => {} + LogRecord::Usage { .. } => {} +``` + +## Steps + +- [ ] **1. Copy the given files.** From the repository root: + + ```sh + cp docs/plans/M5d2/files/crates/bxcore/tests/tool_summary.rs crates/bxcore/tests/ + cp docs/plans/M5d2/files/crates/loopd/tests/tool_summary_messages.rs crates/loopd/tests/ + cp docs/plans/M5d2/files/crates/loopd/tests/support/turn.rs crates/loopd/tests/support/turn.rs + ``` + +- [ ] **2. See them fail.** `cargo build --tests --keep-going -p bxcore -p loopd`, timeout + 600000 ms: `tool_summary` and `tool_summary_messages` do not compile (`E0599` no variant + `ToolSummary`). `support/turn.rs` does not compile for the same reason until L1. After L1 + and before B1, `baseline.rs` fails (`non-exhaustive patterns`). Do L1 then B1 in step 3; + this step is the first `cargo build` against the given tests only, which is E0599. +- [ ] **3. The edits.** Make L1 then B1. `cargo check -p bxcore -p loopd` prints no error and + no warning. +- [ ] **4. The tests.** `cargo test -p bxcore --test tool_summary --test records` and + `cargo test -p loopd --test tool_summary_messages --test turn --test baseline --test + session_slot`: every test passes, 3 in `tool_summary`, 3 in `tool_summary_messages`. +- [ ] **5. Check.** `git diff --stat` lists `crates/bxcore/src/log.rs`, + `crates/loopd/src/baseline.rs`, `crates/loopd/tests/support/turn.rs`. +- [ ] **6. Run the gate.** `cargo fmt --all`, then `make gate`, timeout 600000 ms. Last line + `gate: ok`. Test count **1517** (1511 before, plus 6). +- [ ] **7. Log and commit.** Task cell `M5d2/02-bxcore-tool-summary`. Stage with + `git add crates/bxcore crates/loopd docs/implementer-log.md Cargo.lock`, then commit as + `AGENTS.md` says. + +## Done when + +- `make gate` prints `gate: ok` with 1517 tests. +- `git diff HEAD~1 --stat` lists only `crates/bxcore/src/log.rs`, + `crates/bxcore/tests/tool_summary.rs`, `crates/loopd/src/baseline.rs`, + `crates/loopd/tests/tool_summary_messages.rs`, `crates/loopd/tests/support/turn.rs` and the + log. + +## Stop and report if + +- The branch check fails; a "before" block is not found exactly once; step 2 differs; `cargo + check` warns; a test fails; the gate count is not 1517; you replay a summary as a `User` or + `Tool` message; you add `truncated`; you change `Setup::turn`'s return type. + +## What the given tests pin + +JSON type is `tool_summary`; unknown fields are refused; field order is the struct order; +`messages()` with and without a summary are equal; a summary is not a tool message; inserting a +summary between a result and a later user does not break prefix extension. diff --git a/docs/plans/M5d2/03-loopd-heartbeat-config.md b/docs/plans/M5d2/03-loopd-heartbeat-config.md new file mode 100644 index 0000000..cf29a12 --- /dev/null +++ b/docs/plans/M5d2/03-loopd-heartbeat-config.md @@ -0,0 +1,218 @@ +# M5d-2 task 03: `[heartbeat]` and unknown `[compact] summarise` names + +**Branch:** `m5d2`. Check it: `git branch --show-current` prints `m5d2` and `git status --short` +is empty; otherwise stop. Never switch branch. +**Commit subject:** `loopd: [heartbeat] table and summarise name check` +**Log row:** the Task cell is `M5d2/03-loopd-heartbeat-config`. + +## Goal + +`[heartbeat]` on `loopd.toml` (spec 4.1, 5): `every_s` (default `0`, disables) and `prompt` +(default `heartbeat.md`, next to the config file). Unknown keys are errors. + +Unknown names in `[compact] summarise` are a config error (spec 4.3). `Config::check_summarise` +takes `&Registry`. `loopd` calls it after `load` with `Registry::m5c()`. `parse` still succeeds +with an unknown name: the given tests call `check_summarise` themselves. + +The starter `config/heartbeat.md` is copied like `config/system.md`. Nothing ticks yet. + +## Every place that changes + +| Place | What happens there | +|---|---| +| `crates/loopd/src/config.rs` | `Heartbeat`; field; `load` joins the prompt; `heartbeat_prompt`; `check_summarise` | +| `crates/loopd/src/main.rs` | after each `Config::load`, `check_summarise(&Registry::m5c())` | +| `config/heartbeat.md` | new, given starter | +| `crates/loopd/tests/config_heartbeat.rs` | new, given | +| everything else | **no change** | + +## Files + +- Copy, from `docs/plans/M5d2/files/` (the same path under it): the new test + `crates/loopd/tests/config_heartbeat.rs` and the starter `config/heartbeat.md`. +- Modify: `crates/loopd/src/config.rs`, `crates/loopd/src/main.rs`, + `docs/implementer-log.md`. +- Do not touch: any other file. + +## Facts you need + +- **`every_s` is `u64`.** `0` means off. No range check. +- **`prompt` after `load`.** Join it to the config file's parent, the same way `load` already + joins `baseline.system`. `heartbeat_prompt()` returns `self.heartbeat.prompt.clone()`. After + `parse` (no file), the path stays relative. +- **`check_summarise`.** For each name in `self.compact.summarise`, `registry.get(name)` must be + `Some`. Empty list is allowed. The error `String` contains both `summarise` and the unknown + name. `Registry::get` is already `pub`. `http_fetch` and `clock` are in `m5c()`. +- **`main.rs`.** Both `run_selftest` and `run_serve` load a config. After a successful `load`, + if `check_summarise` fails, print `loopd: {e}` and return exit 1, as for a load error. +- **`#[serde(deny_unknown_fields, default)]`** on `Heartbeat`, like `Compact`. +- The items the edits use. This table is complete. + + | Item | What it does | + |---|---| + | `Registry::get` | known tool names | + | `Registry::m5c` | what `loopd serve` uses | + | `Path::join` | prompt beside the config file | + +## The edits, exactly + +Each quoted "before" block occurs exactly once after M5d-1 (compact is already on `Config`). + +### `crates/loopd/src/config.rs` + +**H1.** After the `Compact` `Default` impl (the closing `}` of that impl, then the `Config` +struct's `memory` / `compact` fields). After + +```rust + #[serde(default)] + pub compact: Compact, +} +``` + +become + +```rust + #[serde(default)] + pub compact: Compact, + #[serde(default)] + pub heartbeat: Heartbeat, +} +``` + +**H2.** After the `Compact` `Default` impl, add `Heartbeat` (before the `Config` struct if that +is where `Compact` sits; keep `Heartbeat` next to `Compact`). The new type: + +```rust +/// `[heartbeat]`: how often the standing `heartbeat` session runs (M5d spec, section 4.1). +/// `every_s` 0 disables it. +#[derive(Debug, Clone, PartialEq, Eq, Deserialize)] +#[serde(deny_unknown_fields, default)] +pub struct Heartbeat { + pub every_s: u64, + pub prompt: std::path::PathBuf, +} + +impl Default for Heartbeat { + fn default() -> Self { + Self { + every_s: 0, + prompt: std::path::PathBuf::from("heartbeat.md"), + } + } +} +``` + +If `Compact` sits just above `Config`, put `Heartbeat` immediately after `Compact`'s `Default` +impl and **then** do H1. If a "before" block is not found exactly once, stop. + +**H3.** In `load`, the lines + +```rust + if let Some(parent) = path.parent() { + config.baseline.system = parent.join(&config.baseline.system); + } + Ok(config) +``` + +become + +```rust + if let Some(parent) = path.parent() { + config.baseline.system = parent.join(&config.baseline.system); + config.heartbeat.prompt = parent.join(&config.heartbeat.prompt); + } + Ok(config) +``` + +**H4.** After `compact_prompt` (the lines) + +```rust + pub fn compact_prompt(&self) -> std::path::PathBuf { + self.baseline.system.with_file_name("compact.md") + } +} +``` + +become + +```rust + pub fn compact_prompt(&self) -> std::path::PathBuf { + self.baseline.system.with_file_name("compact.md") + } + + /// The heartbeat prompt, already joined to the config file's directory by `load`. + pub fn heartbeat_prompt(&self) -> std::path::PathBuf { + self.heartbeat.prompt.clone() + } + + /// Unknown `[compact] summarise` names are a config error (M5d spec, section 4.3). + pub fn check_summarise(&self, registry: &crate::tools::Registry) -> Result<(), String> { + for name in &self.compact.summarise { + if registry.get(name).is_none() { + return Err(format!("[compact] summarise: unknown tool {name}")); + } + } + Ok(()) + } +} +``` + +### `crates/loopd/src/main.rs` + +**M1.** In `run_selftest`, after a successful `Config::load` and before `Client::new`, add: + +```rust + if let Err(e) = cfg.check_summarise(&loopd::tools::Registry::m5c()) { + eprintln!("loopd: {e}"); + return ExitCode::from(1); + } +``` + +`Registry` is already imported in `main.rs`. Use `Registry::m5c()`. + +**M2.** The same check in `run_serve`, after a successful `load` and before the instance lock. + +If `main.rs` already has `use loopd::tools::{Registry, ToolPort};`, write `Registry::m5c()`. + +## Steps + +- [ ] **1. Copy the given files.** + + ```sh + cp docs/plans/M5d2/files/crates/loopd/tests/config_heartbeat.rs crates/loopd/tests/ + cp docs/plans/M5d2/files/config/heartbeat.md config/heartbeat.md + ``` + +- [ ] **2. See it fail.** `cargo build --tests --keep-going -p loopd`, timeout 600000 ms: test + target `config_heartbeat` does not compile (`E0609` no field `heartbeat`, `E0599` no method + `heartbeat_prompt` / `check_summarise`). Every other target compiles. +- [ ] **3. The edits.** H1 to H4, M1, M2, in order. `cargo check -p loopd` prints no error and + no warning. +- [ ] **4. The tests.** `cargo test -p loopd --test config_heartbeat --test config --test + config_compact`: every test passes, 8 in `config_heartbeat`. +- [ ] **5. Check.** `git diff --stat` lists `crates/loopd/src/config.rs` and + `crates/loopd/src/main.rs`. `git status --short` shows the new test and `config/heartbeat.md`. +- [ ] **6. Run the gate.** `cargo fmt --all`, then `make gate`, timeout 600000 ms. Last line + `gate: ok`. Test count **1525** (1517 before, plus 8). +- [ ] **7. Log and commit.** Task cell `M5d2/03-loopd-heartbeat-config`. Stage with + `git add crates/loopd config/heartbeat.md docs/implementer-log.md Cargo.lock`, then commit + as `AGENTS.md` says. + +## Done when + +- `make gate` prints `gate: ok` with 1525 tests. +- `git diff HEAD~1 --stat` lists only `crates/loopd/src/config.rs`, `crates/loopd/src/main.rs`, + `crates/loopd/tests/config_heartbeat.rs`, `config/heartbeat.md` and the log. + +## Stop and report if + +- The branch check fails; a "before" block is not found exactly once; step 2 differs; `cargo + check` warns; a test fails; the gate count is not 1525; you join `heartbeat.prompt` onto + `system.md`'s directory instead of the config file's parent; you reject an empty `summarise` + list; you check names inside `parse`. + +## What the given tests pin + +A missing table is disabled and `heartbeat_prompt` is beside the config file; overrides replace +defaults; unknown keys error; default `http_fetch` is known to `m5c`; an unknown name errors +with `summarise` and that name; empty is allowed; `http_fetch` and `clock` together are known. diff --git a/docs/plans/M5d2/04-loopd-cron.md b/docs/plans/M5d2/04-loopd-cron.md new file mode 100644 index 0000000..bb6e786 --- /dev/null +++ b/docs/plans/M5d2/04-loopd-cron.md @@ -0,0 +1,192 @@ +# M5d-2 task 04: five-field UTC cron matcher + +**Branch:** `m5d2`. Check it: `git branch --show-current` prints `m5d2` and `git status --short` +is empty; otherwise stop. Never switch branch. +**Commit subject:** `loopd: five-field UTC cron matcher` +**Log row:** the Task cell is `M5d2/04-loopd-cron`. + +## Goal + +`loopd::cron::due(schedule, Timestamp) -> Result` (spec 4.2). Five fields: +minute hour day-of-month month day-of-week. Each field is `*`, a single number, `n-m`, or +`*/n`. Day-of-week `0` is Sunday. No cron crate. DOM and DOW are AND (README known limits). + +Nothing loads job files yet. + +## Every place that changes + +| Place | What happens there | +|---|---| +| `crates/loopd/src/cron.rs` | new: `due`, `CronError` | +| `crates/loopd/src/lib.rs` | `pub mod cron;` | +| `crates/loopd/tests/cron.rs` | new, given | +| everything else | **no change** | + +## Files + +- Copy, from `docs/plans/M5d2/files/` (the same path under it): the new test + `crates/loopd/tests/cron.rs`. +- Create: `crates/loopd/src/cron.rs` (skeleton below). +- Modify: `crates/loopd/src/lib.rs`, `docs/implementer-log.md`. +- Do not touch: any other file. + +## Facts you need + +- **Civil fields from RFC 3339.** `Timestamp::to_rfc3339` is `YYYY-MM-DDTHH:MM:SS.mmmZ`. + Minute = `rfc[14..16]`, hour = `rfc[11..13]`, day = `rfc[8..10]`, month = `rfc[5..7]`, + parsed as `u32`. If a slice is missing, `CronError::Fields` naming the schedule. +- **Weekday.** `0` is Sunday. `(unix_millis() / 1000 / 86400 + 4) % 7` as `u32` (`1970-01-01` + was Thursday, so `+ 4`). `unix_millis` is `pub` on `Timestamp`. +- **Ranges.** Inclusive. `*/n` matches when `value % n == 0` (multiples from 0). `n == 0` in + a step is `Fields`. +- **Bounds.** Minute 0–59, hour 0–23, DOM 1–31, month 1–12, DOW 0–6. A number outside its + field's range is `Fields`. Four or six fields, `@daily`, and any other token are `Fields`. +- **`Display`.** Names the schedule, e.g. contains `x x x x x` when that was the input. Do not + add a runbook pointer; this error is used by job loading in task 05. +- **Split.** `split_whitespace`. Five pieces or `Fields`. +- rustfmt decides `mod` order in `lib.rs`. After `pub mod compact;` alphabetically `cron` + comes after `config` and before `http`. Insert where rustfmt will put it; run `cargo fmt` + before the gate. +- The items the edits use. This table is complete. + + | Item | What it does | + |---|---| + | `Timestamp::to_rfc3339` | civil fields | + | `Timestamp::unix_millis` | weekday | + | `str::parse::` | numbers | + +## Skeleton + +`crates/loopd/src/cron.rs`: + +```rust +//! Five-field UTC crontab matcher (M5d spec, section 4.2). No cron crate. + +use bxcore::Timestamp; + +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum CronError { + Fields(String), +} + +impl std::fmt::Display for CronError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + CronError::Fields(schedule) => { + write!(f, "cron schedule {schedule:?} is not five UTC fields") + } + } + } +} + +impl std::error::Error for CronError {} + +/// Whether `schedule` matches `now` in UTC. +pub fn due(schedule: &str, now: Timestamp) -> Result { + let fields: Vec<&str> = schedule.split_whitespace().collect(); + if fields.len() != 5 { + return Err(CronError::Fields(schedule.to_string())); + } + let rfc = now.to_rfc3339(); + let minute = number(&rfc, 14, 16, schedule)?; + let hour = number(&rfc, 11, 13, schedule)?; + let day = number(&rfc, 8, 10, schedule)?; + let month = number(&rfc, 5, 7, schedule)?; + let dow = { + let days = now.unix_millis() / 1000 / 86_400; + ((days + 4) % 7) as u32 + }; + Ok(match_field(fields[0], minute, 0, 59, schedule)? + && match_field(fields[1], hour, 0, 23, schedule)? + && match_field(fields[2], day, 1, 31, schedule)? + && match_field(fields[3], month, 1, 12, schedule)? + && match_field(fields[4], dow, 0, 6, schedule)?) +} + +fn number(rfc: &str, start: usize, end: usize, schedule: &str) -> Result { + let piece = rfc + .get(start..end) + .ok_or_else(|| CronError::Fields(schedule.to_string()))?; + piece + .parse() + .map_err(|_| CronError::Fields(schedule.to_string())) +} + +fn match_field( + field: &str, + value: u32, + min: u32, + max: u32, + schedule: &str, +) -> Result { + let err = || CronError::Fields(schedule.to_string()); + if field == "*" { + return Ok(true); + } + if let Some(step) = field.strip_prefix("*/") { + let n: u32 = step.parse().map_err(|_| err())?; + if n == 0 { + return Err(err()); + } + return Ok(value.is_multiple_of(n)); + } + if let Some((a, b)) = field.split_once('-') { + let lo: u32 = a.parse().map_err(|_| err())?; + let hi: u32 = b.parse().map_err(|_| err())?; + if lo < min || hi > max || lo > hi { + return Err(err()); + } + return Ok(value >= lo && value <= hi); + } + let n: u32 = field.parse().map_err(|_| err())?; + if n < min || n > max { + return Err(err()); + } + Ok(n == value) +} +``` + +`u32::is_multiple_of` is stable in 1.98. Do not use `%` with a comment that `n` was checked; +the skeleton already returns `Fields` for `n == 0`. + +If `is_multiple_of` is not in scope on `u32` in this toolchain, use `value % n == 0` after the +`n == 0` check, and stop only if both fail to compile. + +## Steps + +- [ ] **1. Copy the given test.** + + ```sh + cp docs/plans/M5d2/files/crates/loopd/tests/cron.rs crates/loopd/tests/ + ``` + +- [ ] **2. See it fail.** `cargo build --tests --keep-going -p loopd`, timeout 600000 ms: test + target `cron` does not compile (`E0432` unresolved import `loopd::cron`). Every other + target compiles. +- [ ] **3. The module.** Write `cron.rs` from the skeleton. Add `pub mod cron;` to `lib.rs` + where rustfmt will put it. `cargo check -p loopd` prints no error and no warning. +- [ ] **4. The tests.** `cargo test -p loopd --test cron`: every test passes, 10 tests. +- [ ] **5. Check.** `git diff --stat` lists `crates/loopd/src/cron.rs` and + `crates/loopd/src/lib.rs`. +- [ ] **6. Run the gate.** `cargo fmt --all`, then `make gate`, timeout 600000 ms. Last line + `gate: ok`. Test count **1535** (1525 before, plus 10). +- [ ] **7. Log and commit.** Task cell `M5d2/04-loopd-cron`. Stage with + `git add crates/loopd docs/implementer-log.md Cargo.lock`, then commit as `AGENTS.md` says. + +## Done when + +- `make gate` prints `gate: ok` with 1535 tests. +- `git diff HEAD~1 --stat` lists only `crates/loopd/src/cron.rs`, `crates/loopd/src/lib.rs`, + `crates/loopd/tests/cron.rs` and the log. + +## Stop and report if + +- The branch check fails; step 2 differs; `cargo check` warns; a test fails; the gate count is + not 1535; you add a dependency; you treat DOM and DOW as OR; you accept `@daily` or a sixth + field; `cron.rs` exceeds 500 lines. + +## What the given tests pin + +`0 7 * * *` matches that minute only; `* * * * *` matches; `*/5` is multiples from 0; `0-10` +is inclusive; Sunday is 0 on 2026-10-04; the 1st of the month; four or six fields error; +minute 60, hour 24, DOW 7 error; `@daily` errors; `Display` names the schedule. diff --git a/docs/plans/M5d2/05-loopd-jobs.md b/docs/plans/M5d2/05-loopd-jobs.md new file mode 100644 index 0000000..e6ca485 --- /dev/null +++ b/docs/plans/M5d2/05-loopd-jobs.md @@ -0,0 +1,219 @@ +# M5d-2 task 05: job files and the run ledger + +**Branch:** `m5d2`. Check it: `git branch --show-current` prints `m5d2` and `git status --short` +is empty; otherwise stop. Never switch branch. +**Commit subject:** `loopd: job files, run ledger, jobs-invalid` +**Log row:** the Task cell is `M5d2/05-loopd-jobs`. + +## Goal + +Owner-written `{home}/jobs/.toml` and a run ledger `{home}/jobs/runs.jsonl` (spec 4.2). +`loopd` reads job files and cannot write them. Unknown keys, a name that is not `[a-z0-9-]+`, +or a name so long that `job--t` is not a `SessionId`, are +`jobs-invalid`. A missing jobs directory is an empty list. + +Nothing ticks yet. `loopd serve` refuses to start when `load` errors, so a misspelt job cannot +sit silently. + +## Every place that changes + +| Place | What happens there | +|---|---| +| `crates/loopd/src/jobs.rs` | new: `Job`, `Run`, `RunStatus`, `load`, `session_id`, `append`, `close_leftovers`, `live_session` | +| `crates/loopd/src/lib.rs` | `pub mod jobs;` | +| `crates/loopd/src/main.rs` | `run_serve` calls `jobs::load` after `check_summarise` | +| `docs/runbook.md` | append the given `jobs-invalid` entry | +| `crates/loopd/tests/jobs.rs` | new, given | +| everything else | **no change** | + +## Files + +- Copy, from `docs/plans/M5d2/files/` (the same path under it): the new test + `crates/loopd/tests/jobs.rs`. +- Append, from `docs/plans/M5d2/files/runbook-jobs-invalid.md`, to `docs/runbook.md`. +- Create: `crates/loopd/src/jobs.rs` (skeleton below). +- Modify: `crates/loopd/src/lib.rs`, `crates/loopd/src/main.rs`, `docs/implementer-log.md`. +- Do not touch: any other file. + +## Facts you need + +- **`name` from the file stem.** `mail.toml` → `mail`. The stem must match `^[a-z0-9-]+$` + (bytes, nonempty) and `session_id(name, now)` must succeed. `Mail.toml` is `JobsError::Name`. +- **`session_id`.** `format!("job-{name}-{}t{}", ymd, hm)` with `ymd` = `rfc[0..4]+rfc[5..7]+rfc[8..10]` + and `hm` = `rfc[11..13]+rfc[14..16]`, all from `now.to_rfc3339()`, then + `SessionId::new`. `SessionId` is `[a-z0-9-]`, max 64. `"n".repeat(50)` is too long (`job-` + + 50 + `-` + 8 + `t` + 4 = 68). +- **`load`.** `read_dir`. `ErrorKind::NotFound` → `Ok(vec![])`. Any other IO error is + `JobsError::Invalid` naming the directory. Skip names that do not end in `.toml`. Sort by + `name` so the order is stable. Parse with `#[serde(deny_unknown_fields)]` on a private + `File { schedule, prompt }`. +- **`RunStatus`.** `started` / `finished` / `failed` (`rename_all = "snake_case"`). +- **`append`.** Create the parent directory (0700 is fine; do not weaken a tighter mode). One + JSON object, then a newline. Open append, write, `sync_all`. +- **`live_session`.** Walk every line of the ledger (missing file → `Ok(None)`). The last + `Run` whose `name` matches: `Started` → `Some(session)`, anything else → `None`. +- **`close_leftovers`.** Names whose last run is `Started` get one more `Run` with + `status: Failed` and the same `session`, `time: now`. Return how many names were closed. + A missing file is `Ok(0)`. +- **`Display` for `Invalid`.** `"{path}: {detail}; see docs/runbook.md#jobs-invalid"`. The + given unknown-keys test checks `ends_with("see docs/runbook.md#jobs-invalid")` and that + the path's file name is in the string. +- **`run_serve`.** After `check_summarise`, `jobs::load(&cfg.paths.home.join("jobs"))`. On + `Err`, print `loopd: {e}` and return exit 1. Do not create the directory here. +- A file that exists but cannot be read is an error. Only a missing directory is empty. +- The items the edits use. This table is complete. + + | Item | What it does | + |---|---| + | `toml::from_str` | already a `loopd` dependency | + | `SessionId::new` | the 64-byte alphabet check | + | `serde` `deny_unknown_fields` | unknown keys | + +## Skeleton + +`crates/loopd/src/jobs.rs` — fill every function; no `todo!()`. Keep the types in this order. + +```rust +//! Owner-written job files and the run ledger (M5d spec, section 4.2). + +use std::path::{Path, PathBuf}; + +use bxcore::{SessionId, Timestamp}; +use serde::{Deserialize, Serialize}; + +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct Job { + pub name: String, + pub schedule: String, + pub prompt: String, +} + +#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum RunStatus { + Started, + Finished, + Failed, +} + +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct Run { + pub time: Timestamp, + pub name: String, + pub session: SessionId, + pub status: RunStatus, +} + +#[derive(Debug)] +pub enum JobsError { + Name(String), + Invalid { path: PathBuf, detail: String }, +} + +impl std::fmt::Display for JobsError { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + match self { + JobsError::Name(name) => write!( + f, + "job name {name:?} is not a valid job; see docs/runbook.md#jobs-invalid" + ), + JobsError::Invalid { path, detail } => write!( + f, + "{}: {detail}; see docs/runbook.md#jobs-invalid", + path.display() + ), + } + } +} + +impl std::error::Error for JobsError {} + +#[derive(Deserialize)] +#[serde(deny_unknown_fields)] +struct File { + schedule: String, + prompt: String, +} + +pub fn session_id(name: &str, now: Timestamp) -> Result { + // Build job--t from now.to_rfc3339() as in Facts. SessionId::new + // maps a failure to JobsError::Name(name.to_string()). + let _ = (name, now); + Err(JobsError::Name(name.to_string())) +} + +pub fn load(dir: &Path) -> Result, JobsError> { + let _ = dir; + Ok(Vec::new()) +} + +pub fn append(path: &Path, run: &Run) -> Result<(), JobsError> { + let _ = (path, run); + Ok(()) +} + +pub fn live_session(path: &Path, name: &str) -> Result, JobsError> { + let _ = (path, name); + Ok(None) +} + +pub fn close_leftovers(path: &Path, now: Timestamp) -> Result { + let _ = (path, now); + Ok(0) +} +``` + +Replace the four stub bodies in the same task. Do not leave a stub that returns `Ok(None)` / +`Ok(0)` / `Ok(vec![])` once you have written the real body. `session_id` must not stay on the +`Err` path. + +## Steps + +- [ ] **1. Copy the given test.** + + ```sh + cp docs/plans/M5d2/files/crates/loopd/tests/jobs.rs crates/loopd/tests/ + ``` + +- [ ] **2. See it fail.** `cargo build --tests --keep-going -p loopd`, timeout 600000 ms: test + target `jobs` does not compile (`E0432` unresolved `loopd::jobs`). Every other target + compiles. +- [ ] **3. The module.** Write `jobs.rs` from the skeleton, then fill the four bodies one at a + time (`session_id`, `load`, `append`, `live_session` / `close_leftovers`). Add `pub mod jobs;` + to `lib.rs`. Wire `run_serve` as in Facts. `cargo check -p loopd` prints no error and no + warning. `wc -l crates/loopd/src/jobs.rs` prints a number ≤ 500. +- [ ] **4. The tests.** `cargo test -p loopd --test jobs`: every test passes, 8 tests. +- [ ] **5. The runbook.** From the repository root: + + ```sh + { echo; cat docs/plans/M5d2/files/runbook-jobs-invalid.md; } >> docs/runbook.md + ``` + + Then `grep -c '^## jobs-invalid$' docs/runbook.md` prints `1`. +- [ ] **6. Check.** `git diff --stat` lists `crates/loopd/src/jobs.rs`, + `crates/loopd/src/lib.rs`, `crates/loopd/src/main.rs`, `docs/runbook.md`. +- [ ] **7. Run the gate.** `cargo fmt --all`, then `make gate`, timeout 600000 ms. Last line + `gate: ok`. Test count **1543** (1535 before, plus 8). +- [ ] **8. Log and commit.** Task cell `M5d2/05-loopd-jobs`. Stage with + `git add crates/loopd docs/runbook.md docs/implementer-log.md Cargo.lock`, then commit as + `AGENTS.md` says. + +## Done when + +- `make gate` prints `gate: ok` with 1543 tests. +- `git diff HEAD~1 --stat` lists only `crates/loopd/src/jobs.rs`, `crates/loopd/src/lib.rs`, + `crates/loopd/src/main.rs`, `crates/loopd/tests/jobs.rs`, `docs/runbook.md` and the log. + +## Stop and report if + +- The branch check fails; step 2 differs; `cargo check` warns; a test fails; the gate count is + not 1543; you add a cron crate; a missing jobs directory is an error; `Display` for `Invalid` + does not end with the runbook pointer; `jobs.rs` exceeds 500 lines; you write a job file. + +## What the given tests pin + +A valid `mail.toml` loads; unknown keys name the file and the runbook; `Mail.toml` is `Name`; +`session_id` is `job-mail-20261003t0700`; a 50-character name is `Name`; a missing directory is +empty; leftover `started` becomes `failed` and is no longer live; a `started` without a close +is live until `finished`. diff --git a/docs/plans/M5d2/06-loopd-scheduler-heartbeat.md b/docs/plans/M5d2/06-loopd-scheduler-heartbeat.md new file mode 100644 index 0000000..5b0fedc --- /dev/null +++ b/docs/plans/M5d2/06-loopd-scheduler-heartbeat.md @@ -0,0 +1,216 @@ +# M5d-2 task 06: heartbeat tick + +**Branch:** `m5d2`. Check it: `git branch --show-current` prints `m5d2` and `git status --short` +is empty; otherwise stop. Never switch branch. +**Commit subject:** `loopd: heartbeat tick on the background slot` +**Log row:** the Task cell is `M5d2/06-loopd-scheduler-heartbeat`. + +## Goal + +A `Scheduler` whose `tick(&Context, now)` runs the standing `heartbeat` session when +`every_s` has elapsed since the last heartbeat turn, or since `Scheduler::new(started)` if +none (spec 4.1). The user text is `It is .\n\n` plus the prompt file. Slot is +background. Speaker is `None`. Skip if the session is busy. Skip and log once if the prompt +cannot be read (`heartbeat-prompt-unreadable`). + +Tests call `tick(ctx, now)` with injected time. No sleeps (T27). + +## Every place that changes + +| Place | What happens there | +|---|---| +| `crates/loopd/src/channel.rs` | `Held` is `pub`; `Context::hold` | +| `crates/loopd/src/scheduler.rs` | new: `Scheduler`, `TickWork`, `tick` | +| `crates/loopd/src/lib.rs` | `pub mod scheduler;` | +| `crates/loopd/src/main.rs` | a thread that `tick`s once a second, before `channel::serve` | +| `docs/runbook.md` | append the given `heartbeat-prompt-unreadable` entry | +| `crates/loopd/tests/scheduler_heartbeat.rs` | new, given | +| everything else | **no change**: not cron job runs (task 07) | + +## Files + +- Copy, from `docs/plans/M5d2/files/` (the same path under it): the new test + `crates/loopd/tests/scheduler_heartbeat.rs`. +- Append, from `docs/plans/M5d2/files/runbook-heartbeat-prompt-unreadable.md`, to + `docs/runbook.md`. +- Create: `crates/loopd/src/scheduler.rs` (skeleton below). +- Modify: `crates/loopd/src/channel.rs`, `crates/loopd/src/lib.rs`, + `crates/loopd/src/main.rs`, `docs/implementer-log.md`. +- Do not touch: any other file. + +## Facts you need + +- **`Context::hold`.** Same insert as `handle`: if the id is already in `busy`, `Err(())`; + else insert and return `Held`. `Held` must be `pub` so the given test can keep it. `Drop` + already removes the id. +- **Due.** `every_s == 0` → no heartbeat work. Else `now.unix_millis().saturating_sub(last.unix_millis()) >= every_s.saturating_mul(1000)` + where `last` is `last_heartbeat.unwrap_or(started)`. `07:00` to `07:01` with `every_s = 60` + is due. `07:01` then `07:01:30` is not. +- **Prompt first, then hold.** If `read_to_string(cfg.heartbeat_prompt())` fails, do not hold, + push `HeartbeatUnreadable`, log + `loopd: {path}: {e}; see docs/runbook.md#heartbeat-prompt-unreadable` only while + `heartbeat_unreadable_logged` is false, then set that flag. Do not update `last_heartbeat`. + A directory at the prompt path is the given-test stand-in for an unreadable file (T27: no + `chmod 000`). +- **Session id `heartbeat`.** `Session::open`; `NotFound` → `Session::create` with + `Baseline::assemble(&ctx.cfg, &ctx.registry)` and `ctx.cfg.slots.background`. Other open + errors: log and skip (do not panic). +- **The turn.** User text `format!("It is {}.\n\n{file}", now.to_rfc3339())` where `file` is + the prompt contents as read (no extra trim beyond what the file has). `run_turn` then + `compact::finish_turn` with `&*ctx.memory`, speaker `None`, `on_event` a no-op. Then + `last_heartbeat = Some(now)`, push `Heartbeat`. +- **`TickWork`.** Include `Job` / `SkippedJob` now so task 07 does not change the type. + Task 06 never pushes them. +- **`close_leftovers`.** On the first `tick` only, call + `jobs::close_leftovers(&ctx.cfg.paths.home.join("jobs").join("runs.jsonl"), now)` and ignore + a missing-file `Ok(0)`. A leftover that is currently `hold`ing its session is **not** closed: + skip names whose `live_session` session is in `busy` (or: close only when `hold` of that + session would succeed, then immediately `drop` that hold). Task 07's live-skip test depends + on this. If `close_leftovers` has no busy-set filter, add a loop here: for each live name, + if `hold(session)` is `Ok`, drop it and treat as leftover (append Failed yourself or call a + helper). Simplest: **do not call `close_leftovers` blindly**. Walk: if `live_session` is + `Some(id)` and `hold(id)` is `Ok`, drop the hold and append a `Failed` for that name (or + call `close_leftovers` only when no job session is held). The given leftover test does not + hold; the live-skip test does. Closing only leftovers that are not busy is required. +- **Main thread.** After `Context::new` and before `channel::serve`: + + ```rust + let ctx_sched = Arc::clone(&ctx); + std::thread::spawn(move || { + let mut sched = loopd::scheduler::Scheduler::new(bxcore::Timestamp::now()); + loop { + sched.tick(&ctx_sched, bxcore::Timestamp::now()); + std::thread::sleep(std::time::Duration::from_secs(1)); + } + }); + ``` + + That sleep is production only. Tests never sleep. +- **`channel.rs` stays ≤ 500 lines.** `hold` is about 15 lines. +- The items the edits use. This table is complete. + + | Item | What it does | + |---|---| + | `Session::slot` / `create` / `open` | standing session | + | `run_turn` / `finish_turn` | the pulse | + | `Baseline::assemble` | new session | + | `jobs::close_leftovers` / `live_session` / `append` | leftovers | + | `Timestamp::unix_millis` / `to_rfc3339` | due and the time line | + +## The edits, exactly + +### `crates/loopd/src/channel.rs` + +**C1.** `struct Held` becomes `pub struct Held`. + +**C2.** Inside `impl Context`, after `new`, add: + +```rust + /// Mark `id` busy. `Err(())` if it already is. Tests and the scheduler use this so a skip + /// is not a sleep (T27). + pub fn hold(&self, id: SessionId) -> Result, ()> { + let mut busy = self.busy.lock().unwrap_or_else(|p| p.into_inner()); + if busy.contains(&id) { + return Err(()); + } + busy.insert(id.clone()); + drop(busy); + Ok(Held { ctx: self, id }) + } +``` + +### `crates/loopd/src/scheduler.rs` + +Write this module. Fill `tick`. Cron jobs stay a comment until task 07: after heartbeat, do +not iterate job files yet. + +```rust +//! Isolated heartbeat (and, from task 07, cron) turns (M5d spec, section 4). + +use crate::channel::Context; +use bxcore::{SessionId, Timestamp}; + +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum TickWork { + Heartbeat, + SkippedBusy { session: SessionId }, + HeartbeatUnreadable, + Job { name: String, session: SessionId }, + SkippedJob { name: String }, +} + +pub struct Scheduler { + started: Timestamp, + last_heartbeat: Option, + leftovers_closed: bool, + heartbeat_unreadable_logged: bool, +} + +impl Scheduler { + pub fn new(started: Timestamp) -> Scheduler { + Scheduler { + started, + last_heartbeat: None, + leftovers_closed: false, + heartbeat_unreadable_logged: false, + } + } + + pub fn tick(&mut self, ctx: &Context, now: Timestamp) -> Vec { + let _ = (ctx, now); + Vec::new() + } +} +``` + +Replace the `tick` body in this task. `leftovers_closed` is set true after the first leftover +pass so it runs once. + +## Steps + +- [ ] **1. Copy the given test.** + + ```sh + cp docs/plans/M5d2/files/crates/loopd/tests/scheduler_heartbeat.rs crates/loopd/tests/ + ``` + +- [ ] **2. See it fail.** `cargo build --tests --keep-going -p loopd`, timeout 600000 ms: test + target `scheduler_heartbeat` does not compile (`E0432` unresolved `loopd::scheduler`, + `E0599` no method `hold`). Every other target compiles. +- [ ] **3. The edits.** C1, C2, `scheduler.rs`, `pub mod scheduler;`, main thread. Fill + `tick`. `cargo check -p loopd` prints no error and no warning. + `wc -l crates/loopd/src/scheduler.rs` and `wc -l crates/loopd/src/channel.rs` each ≤ 500. +- [ ] **4. The tests.** `cargo test -p loopd --test scheduler_heartbeat --test channel --test + session_slot`: every test passes, 6 in `scheduler_heartbeat`. +- [ ] **5. The runbook.** + + ```sh + { echo; cat docs/plans/M5d2/files/runbook-heartbeat-prompt-unreadable.md; } >> docs/runbook.md + ``` + + `grep -c '^## heartbeat-prompt-unreadable$' docs/runbook.md` prints `1`. +- [ ] **6. Run the gate.** `cargo fmt --all`, then `make gate`, timeout 600000 ms. Last line + `gate: ok`. Test count **1549** (1543 before, plus 6). +- [ ] **7. Log and commit.** Task cell `M5d2/06-loopd-scheduler-heartbeat`. Stage with + `git add crates/loopd docs/runbook.md docs/implementer-log.md Cargo.lock`, then commit as + `AGENTS.md` says. + +## Done when + +- `make gate` prints `gate: ok` with 1549 tests. +- `git diff HEAD~1 --stat` lists only `crates/loopd/src/channel.rs`, + `crates/loopd/src/scheduler.rs`, `crates/loopd/src/lib.rs`, `crates/loopd/src/main.rs`, + `crates/loopd/tests/scheduler_heartbeat.rs`, `docs/runbook.md` and the log. + +## Stop and report if + +- The branch check fails; a "before" block is not found exactly once; step 2 differs; `cargo + check` warns; a test fails; the gate count is not 1549; you sleep in a test; you post to + Mattermost; you send `slots.main`; you set a speaker; you `chmod 000`; `Held` stays private. + +## What the given tests pin + +`every_s` 0 does nothing; a due heartbeat sends `id_slot` 1 with a time line and the file +text, session `heartbeat` has `slot() == 1`; a second tick before `every_s` does not run; a +held session is `SkippedBusy` with no request; a directory at the prompt path is +`HeartbeatUnreadable` with no request; the user record has `speaker: None`. diff --git a/docs/plans/M5d2/07-loopd-scheduler-cron.md b/docs/plans/M5d2/07-loopd-scheduler-cron.md new file mode 100644 index 0000000..5e9d16b --- /dev/null +++ b/docs/plans/M5d2/07-loopd-scheduler-cron.md @@ -0,0 +1,107 @@ +# M5d-2 task 07: cron tick + +**Branch:** `m5d2`. Check it: `git branch --show-current` prints `m5d2` and `git status --short` +is empty; otherwise stop. Never switch branch. +**Commit subject:** `loopd: cron tick, new session per run` +**Log row:** the Task cell is `M5d2/07-loopd-scheduler-cron`. + +## Goal + +After the heartbeat work, `tick` loads `{home}/jobs/*.toml` and runs each job whose schedule +matches `now` (spec 4.2). A new session `job--t` on the background slot. +Skip the tick when a previous run of that name is still live. Leftover `started` from a dead +process becomes `failed` on the first tick (already started in task 06: only when that session +is not busy). Speaker `None`. Same time-line prefix as heartbeat. + +## Every place that changes + +| Place | What happens there | +|---|---| +| `crates/loopd/src/scheduler.rs` | after heartbeat, iterate jobs (edit T1) | +| `crates/loopd/tests/scheduler_cron.rs` | new, given | +| everything else | **no change**: not `TickWork`, not `hold`, not heartbeat | + +## Files + +- Copy, from `docs/plans/M5d2/files/` (the same path under it): the new test + `crates/loopd/tests/scheduler_cron.rs`. +- Modify: `crates/loopd/src/scheduler.rs`, `docs/implementer-log.md`. +- Do not touch: any other file. + +## Facts you need + +- **Load.** `jobs::load(&ctx.cfg.paths.home.join("jobs"))`. On `Err`, log `loopd: {e}` and + skip cron this tick (the process stays up: README known limits). On `Ok`, for each `Job` in + order: + 1. `cron::due(&job.schedule, now)`. `Err` → log and skip that job. `Ok(false)` → skip. + 2. Ledger path `{home}/jobs/runs.jsonl`. If `live_session` is `Some(_)`, push + `SkippedJob { name }` and continue. (Task 06 has already closed leftovers that were not + busy.) + 3. `session_id(&job.name, now)`. `Err` → log and skip. + 4. `ctx.hold(session.clone())`. `Err` → `SkippedJob`. `Ok(held)` → create the session + (`Session::create` with `assemble` and `slots.background`; if `Exists`, `open` instead), + `jobs::append` `Started`, run `run_turn` then `finish_turn` with user text + `format!("It is {}.\n\n{}", now.to_rfc3339(), job.prompt)`, speaker `None`, then + `append` `Finished` or `Failed` (turn `Err` → `Failed`). `drop(held)`. Push + `TickWork::Job { name, session }`. +- **Two due minutes.** Two ticks at 06:59 and 07:00 with `* * * * *` make two session ids. +- **Not due.** `0 7 * * *` at 06:59 pushes nothing. +- **`scheduler.rs` stays ≤ 500 lines.** After this task it is about 200. +- The items the edits use. This table is complete. + + | Item | What it does | + |---|---| + | `jobs::load` / `session_id` / `append` / `live_session` | task 05 | + | `cron::due` | task 04 | + | `Context::hold` | task 06 | + | `run_turn` / `finish_turn` | same as heartbeat | + +## The edits, exactly + +There is no quoted "before" for a unique heartbeat-only block: you add a loop after the +heartbeat section of `tick`, before `work` is returned. If `tick` currently ends with +`work` after heartbeat, insert the job loop there. If you cannot find a single place to +insert without touching heartbeat, stop. + +Do not change heartbeat behaviour. Do not add a sleep. + +## Steps + +- [ ] **1. Copy the given test.** + + ```sh + cp docs/plans/M5d2/files/crates/loopd/tests/scheduler_cron.rs crates/loopd/tests/ + ``` + +- [ ] **2. See it fail.** `cargo test -p loopd --test scheduler_cron`, timeout 600000 ms: the + tests compile (`TickWork::Job` already exists) and `a_due_job_starts_a_new_session_on_the_background_slot` + fails: `work` is empty, not a `Job`. `a_job_not_due_does_not_run` may pass. +- [ ] **3. The edit.** Add the job loop. `cargo check -p loopd` prints no error and no + warning. `wc -l crates/loopd/src/scheduler.rs` prints a number ≤ 500. +- [ ] **4. The tests.** `cargo test -p loopd --test scheduler_cron --test scheduler_heartbeat + --test jobs --test cron`: every test passes, 6 in `scheduler_cron`. +- [ ] **5. Check.** `git diff --stat` lists exactly `crates/loopd/src/scheduler.rs`. +- [ ] **6. Run the gate.** `cargo fmt --all`, then `make gate`, timeout 600000 ms. Last line + `gate: ok`. Test count **1555** (1549 before, plus 6). +- [ ] **7. Log and commit.** Task cell `M5d2/07-loopd-scheduler-cron`. Stage with + `git add crates/loopd docs/implementer-log.md Cargo.lock`, then commit as `AGENTS.md` says. + +## Done when + +- `make gate` prints `gate: ok` with 1555 tests. +- `git diff HEAD~1 --stat` lists only `crates/loopd/src/scheduler.rs`, + `crates/loopd/tests/scheduler_cron.rs` and the log. + +## Stop and report if + +- The branch check fails; step 2 differs (tests do not compile, or a different assertion + fails); `cargo check` warns; a test fails; the gate count is not 1555; you reuse a session + across runs; you send `slots.main`; you set a speaker; you skip leftovers that are not busy; + a heartbeat test fails. + +## What the given tests pin + +A due job creates `job-mail-…` on slot 1 with the time line and the prompt; a job not due +does nothing; a live previous run (Started + `hold`) is `SkippedJob` with no request; leftover +`started` is `failed` on the first tick (the due job may then run); speaker is `None`; two due +minutes make two sessions. diff --git a/docs/plans/M5d2/08-loopd-tool-summary.md b/docs/plans/M5d2/08-loopd-tool-summary.md new file mode 100644 index 0000000..4870862 --- /dev/null +++ b/docs/plans/M5d2/08-loopd-tool-summary.md @@ -0,0 +1,257 @@ +# M5d-2 task 08: opted-in tool-output summaries + +**Branch:** `m5d2`. Check it: `git branch --show-current` prints `m5d2` and `git status --short` +is empty; otherwise stop. Never switch branch. +**Commit subject:** `loopd: background ToolSummary for opted-in results` +**Log row:** the Task cell is `M5d2/08-loopd-tool-summary`. + +## Goal + +After a turn that logged an opted-in `ToolResult` with no matching `ToolSummary`, run a +no-tools completion on the background slot and append `ToolSummary` with the result's labels +(spec 4.3). `messages()` already ignores it (task 02). `http_fetch` opts in via +`[compact] summarise`. + +`summarise::missing` is the function. `finish_turn` calls it on `Ok` and on `SessionFull`, +**before** compaction, so the current epoch still holds the `ToolResult` (README known +limits: v1 runs this inside the turn's hold). + +The starter `config/summarise.md` is copied like `config/heartbeat.md`. + +## Every place that changes + +| Place | What happens there | +|---|---| +| `crates/loopd/src/summarise.rs` | new: `missing` | +| `crates/loopd/src/lib.rs` | `pub mod summarise;` | +| `crates/loopd/src/compact.rs` | `finish_turn` calls `missing` before compact (edit F1) | +| `config/summarise.md` | new, given starter | +| `crates/loopd/tests/tool_summary_call.rs` | new, given | +| everything else | **no change** | + +## Files + +- Copy, from `docs/plans/M5d2/files/` (the same path under it): the new test + `crates/loopd/tests/tool_summary_call.rs` and the starter `config/summarise.md`. +- Create: `crates/loopd/src/summarise.rs` (function below). +- Modify: `crates/loopd/src/lib.rs`, `crates/loopd/src/compact.rs`, + `docs/implementer-log.md`. +- Do not touch: any other file. + +## Facts you need + +- **Which results.** Walk `session.records()` once. From each `Assistant`, remember + `tool_calls` by `id → name`. From each `ToolSummary`, remember `call: CallId`. Then for + each `ToolResult` whose `tool_call_id` maps to a name in `cfg.compact.summarise` and whose + `call` is not already summarised, queue it. Do not append while iterating `records()`. +- **The call.** `messages_with_carry(session.baseline(), session.records(), session.carry())`, + then one `ChatMessage::User` whose content is + `format!("{}\n\n{}", prompt.trim_end(), crate::compact::cap_chars(result_content, cfg.r#loop.tool_result_cap))` + where `prompt` is `read_to_string` of + `cfg.baseline.system.with_file_name("summarise.md")`. Missing file: return `Err` naming + the path (do not guess). `ChatRequest { slot: cfg.slots.background, tools: Vec::new(), + thinking: true, messages }`. `client.chat_with_retry(&request, &mut |_| {})`. +- **The record.** Completion `content.unwrap_or_default()`, `neutralise`, `cap_chars` to + `cfg.compact.summary_chars`. Append `LogRecord::ToolSummary { time: Timestamp::now(), + call, tool_call_id, content, class, untrusted }` copying `call`, `tool_call_id`, `class`, + `untrusted` from that `ToolResult`. +- **Context-full / other infer errors.** Skip that result (log `loopd: tool summary: {e}`), + continue the queue. Do not fail the turn. +- **`finish_turn`.** At the start of the `Ok(done)` arm and the `Err(SessionFull)` arm, before + `compact_session`: + + ```rust + if let Err(e) = crate::summarise::missing(rt.cfg, rt.client, session) { + eprintln!("loopd: tool summary: {e}"); + } + ``` + + A compact-due turn summarises first, then compacts. The last given test goes through + `finish_turn` (not `missing` alone) and expects a `ToolSummary`. +- **`cap_chars` is `pub(crate)`** in `compact.rs` (M5d-1 task 05). +- **`tool_call.http` calls `read_file`.** The last given test sets + `summarise = ["read_file"]`. Default `["http_fetch"]` would leave it alone. +- rustfmt `mod` order: `summarise` after `sse` / `session` as rustfmt decides. +- The items the edits use. This table is complete. + + | Item | What it does | + |---|---| + | `messages_with_carry` | prefix plus carry | + | `ChatRequest.tools: Vec::new()` | omits `tools` on the wire | + | `neutralise` | `bxcore::sanitize` | + | `cap_chars` | character cap | + | `Session::append` | the new record | + +## The function + +`crates/loopd/src/summarise.rs`: + +```rust +//! Opted-in tool-output summaries (M5d spec, section 4.3). + +use crate::baseline::messages_with_carry; +use crate::compact::cap_chars; +use crate::config::Config; +use crate::llama::{ChatMessage, ChatRequest, Client}; +use crate::session::Session; +use bxcore::sanitize::neutralise; +use bxcore::{CallId, DataClass, LogRecord, Timestamp, ToolCall}; +use std::collections::{HashMap, HashSet}; + +/// Append a `ToolSummary` for each opted-in `ToolResult` in this epoch that does not yet have +/// one. Inference uses the background slot and no tools. +pub fn missing( + cfg: &Config, + client: &Client, + session: &mut Session, +) -> Result<(), String> { + let path = cfg.baseline.system.with_file_name("summarise.md"); + let prompt = std::fs::read_to_string(&path) + .map_err(|e| format!("{}: {e}", path.display()))?; + let mut names: HashMap = HashMap::new(); + let mut done: HashSet = HashSet::new(); + let mut pending: Vec<(CallId, String, String, DataClass, bool)> = Vec::new(); + for record in session.records() { + match record { + LogRecord::Assistant { tool_calls, .. } => { + for ToolCall { id, name, .. } in tool_calls { + names.insert(id.clone(), name.clone()); + } + } + LogRecord::ToolSummary { call, .. } => { + done.insert(call.0); + } + LogRecord::ToolResult { + call, + tool_call_id, + content, + class, + untrusted, + .. + } => { + if done.contains(&call.0) { + continue; + } + let Some(name) = names.get(tool_call_id) else { + continue; + }; + if !cfg.compact.summarise.iter().any(|n| n == name) { + continue; + } + pending.push(( + *call, + tool_call_id.clone(), + content.clone(), + *class, + *untrusted, + )); + } + _ => {} + } + } + for (call, tool_call_id, content, class, untrusted) in pending { + let mut messages = messages_with_carry(session.baseline(), session.records(), session.carry()); + messages.push(ChatMessage::User { + content: format!( + "{}\n\n{}", + prompt.trim_end(), + cap_chars(&content, cfg.r#loop.tool_result_cap) + ), + }); + let request = ChatRequest { + slot: cfg.slots.background, + messages, + tools: Vec::new(), + thinking: true, + }; + let text = match client.chat_with_retry(&request, &mut |_| {}) { + Ok(completion) => completion.content.unwrap_or_default(), + Err(e) => { + eprintln!("loopd: tool summary: {e}"); + continue; + } + }; + let text = cap_chars(neutralise(&text).as_ref(), cfg.compact.summary_chars); + session + .append(LogRecord::ToolSummary { + time: Timestamp::now(), + call, + tool_call_id, + content: text, + class, + untrusted, + }) + .map_err(|e| e.to_string())?; + } + Ok(()) +} +``` + +`CallId` is a tuple struct `CallId(pub u64)` (already used as `CallId(1)` in tests). If +`call.0` does not compile, stop and report. + +### `crates/loopd/src/compact.rs` + +**F1.** The `Ok(done)` arm. The lines + +```rust + Ok(done) => { + if let Some(usage) = session.last_usage() + && due(&usage, rt.cfg.compact.threshold, rt.cfg.expect.n_ctx) + && let Err(e) = compact_session(session, rt, memory) + { + log_compact(session, &e); + } + Ok(done) + } +``` + +become the same, with the `missing` call inserted as the first statement of the arm (see +Facts). The `Err(TurnError::SessionFull)` arm gets the same `missing` call before +`compact_session`. + +## Steps + +- [ ] **1. Copy the given files.** + + ```sh + cp docs/plans/M5d2/files/crates/loopd/tests/tool_summary_call.rs crates/loopd/tests/ + cp docs/plans/M5d2/files/config/summarise.md config/summarise.md + ``` + +- [ ] **2. See it fail.** `cargo build --tests --keep-going -p loopd`, timeout 600000 ms: test + target `tool_summary_call` does not compile (`E0432` unresolved `loopd::summarise`). Every + other target compiles. +- [ ] **3. The edits.** `summarise.rs`, `pub mod summarise;`, F1. `cargo check -p loopd` + prints no error and no warning. `wc -l crates/loopd/src/summarise.rs` and + `wc -l crates/loopd/src/compact.rs` each ≤ 500. +- [ ] **4. The tests.** `cargo test -p loopd --test tool_summary_call --test + tool_summary_messages --test compact_turn --test turn`: every test passes, 6 in + `tool_summary_call`. +- [ ] **5. Check.** `git diff --stat` lists `crates/loopd/src/summarise.rs`, + `crates/loopd/src/lib.rs`, `crates/loopd/src/compact.rs`. +- [ ] **6. Run the gate.** `cargo fmt --all`, then `make gate`, timeout 600000 ms. Last line + `gate: ok`. Test count **1561** (1555 before, plus 6). +- [ ] **7. Log and commit.** Task cell `M5d2/08-loopd-tool-summary`. Stage with + `git add crates/loopd config/summarise.md docs/implementer-log.md Cargo.lock`, then commit + as `AGENTS.md` says. + +## Done when + +- `make gate` prints `gate: ok` with 1561 tests. +- `git diff HEAD~1 --stat` lists only `crates/loopd/src/summarise.rs`, + `crates/loopd/src/lib.rs`, `crates/loopd/src/compact.rs`, + `crates/loopd/tests/tool_summary_call.rs`, `config/summarise.md` and the log. + +## Stop and report if + +- The branch check fails; a "before" block is not found exactly once; step 2 differs; `cargo + check` warns; a test fails; the gate count is not 1561; you send tools on the summary + request; you send `slots.main`; you replay the summary in `messages()`; you summarise a + name not in the list; you summarise twice; you compact before summarising. + +## What the given tests pin + +An opted-in result is summarised on slot 1 with no `tools` key; labels are copied; a name not +in the list is left alone; a second `missing` does not call again; `messages()` is unchanged +after a summary; `finish_turn` after a `read_file` turn appends a `ToolSummary` on slot 1. diff --git a/docs/plans/M5d2/09-memoryd-summary-index.md b/docs/plans/M5d2/09-memoryd-summary-index.md new file mode 100644 index 0000000..70f4cc9 --- /dev/null +++ b/docs/plans/M5d2/09-memoryd-summary-index.md @@ -0,0 +1,141 @@ +# M5d-2 task 09: index `ToolSummary` as kind `summary` + +**Branch:** `m5d2`. Check it: `git branch --show-current` prints `m5d2` and `git status --short` +is empty; otherwise stop. Never switch branch. +**Commit subject:** `memoryd: index ToolSummary as kind summary` +**Log row:** the Task cell is `M5d2/09-memoryd-summary-index`. + +## Goal + +`memoryd` indexes `LogRecord::ToolSummary` content as kind `summary`, with that record's +`class` and `untrusted` (spec 4.3, P26). `recall` finds it and tags `memory` or +`memory-tainted` as for any other session hit. Raw `ToolResult` bodies stay unindexed. + +`index_record` today has `_ => {}`. It compiles. This task adds the arm. + +## Every place that changes + +| Place | What happens there | +|---|---| +| `crates/memoryd/src/index.rs` | `ToolSummary` arm (edit I1); module doc | +| `crates/memoryd/tests/tool_summary_index.rs` | new, given | +| everything else | **no change**: not `recall.rs` headers (role `"summary"` uses the existing "the assistant said" branch) | + +## Files + +- Copy, from `docs/plans/M5d2/files/` (the same path under it): the new test + `crates/memoryd/tests/tool_summary_index.rs`. +- Modify: `crates/memoryd/src/index.rs`, `docs/implementer-log.md`. +- Do not touch: any other file. Do not edit `tests/support/logs.rs`. + +## Facts you need + +- **Kind `summary`.** Set `kind: "summary".to_string()` on the `Source`, not the log's + `"session"`. Role `"summary"`. Time from the record. `class` / `untrusted` from the + `ToolSummary`, not from `base` (the rolling session label). Skip when + `summary.class > self.max_class`, even if `base.class` is still `Private`. Skip when + `content.trim()` is empty. +- **The existing `if base.class > self.max_class` stays.** A secret `ToolResult` raises the + rolling label before the next record, so a following secret summary is also skipped that + way. Still check the summary's own class. +- **Do not add `ToolSummary` to the rolling-label update** in `index_log` (that loop still + looks at `ToolResult` only). The result already carried the same labels. +- **Recall headers.** `kind` `"summary"` is not `moved` / `dropped` / `noted`, so the tag is + `memory` or `memory-tainted`. Role `"summary"` is not `owner` or `call`, so the line says + `the assistant said`. The given tests look for `memory · session mm-a` and + `memory-tainted`. +- The items the edits use. This table is complete. + + | Item | What it does | + |---|---| + | `self.add` | FTS row plus `sources` | + | `Source` | kind / role / class / untrusted | + +## The edits, exactly + +Each quoted "before" block occurs exactly once. + +### `crates/memoryd/src/index.rs` + +**I1.** After the `Assistant` arm, before `_ => {}`. The lines + +```rust + } + _ => {} + } + Ok(()) + } +``` + +become + +```rust + } + LogRecord::ToolSummary { + time, + content, + class, + untrusted, + .. + } if !content.trim().is_empty() && *class <= self.max_class => { + let src = Source { + kind: "summary".to_string(), + time: time.to_rfc3339(), + role: "summary".to_string(), + class: *class, + untrusted: *untrusted, + ..base.clone() + }; + self.add(content, &src)?; + } + _ => {} + } + Ok(()) + } +``` + +`DataClass` already implements `Ord` (`Public < Private < Secret`). If `<=` does not +compile, stop. + +Optionally update the module doc "never a tool's result" to say summaries of opted-in +results are indexed. A comment that does not fit its code: stop. + +## Steps + +- [ ] **1. Copy the given test.** + + ```sh + cp docs/plans/M5d2/files/crates/memoryd/tests/tool_summary_index.rs crates/memoryd/tests/ + ``` + +- [ ] **2. See it fail.** `cargo test -p memoryd --test tool_summary_index`, timeout 600000 + ms: the tests compile (`ToolSummary` exists) and `a_tool_summary_is_found_as_kind_summary` + fails: `(no matches for "tea")` or the unique phrase is missing. `the_raw_tool_result_is_still_not_indexed` + may pass (results were already unindexed). +- [ ] **3. The edit.** I1. `cargo check -p memoryd` prints no error and no warning. +- [ ] **4. The tests.** `cargo test -p memoryd --test tool_summary_index --test recall`: every + test passes, 5 in `tool_summary_index`. +- [ ] **5. Check.** `git diff --stat` lists exactly `crates/memoryd/src/index.rs`. +- [ ] **6. Run the gate.** `cargo fmt --all`, then `make gate`, timeout 600000 ms. Last line + `gate: ok`. Test count **1566** (1561 before, plus 5). +- [ ] **7. Log and commit.** Task cell `M5d2/09-memoryd-summary-index`. Stage with + `git add crates/memoryd docs/implementer-log.md Cargo.lock`, then commit as `AGENTS.md` + says. + +## Done when + +- `make gate` prints `gate: ok` with 1566 tests. +- `git diff HEAD~1 --stat` lists only `crates/memoryd/src/index.rs`, + `crates/memoryd/tests/tool_summary_index.rs` and the log. + +## Stop and report if + +- The branch check fails; a "before" block is not found exactly once; step 2 differs; `cargo + check` warns; a test fails; the gate count is not 1566; you index `ToolResult` content; you + leave `kind` as `"session"`; a secret summary is searchable on a private index. + +## What the given tests pin + +A summary's content is found and tagged `memory · session …`; the raw tool body is not; an +untrusted summary taints the recall (`memory-tainted`); a trusted summary does not taint; a +secret summary is skipped when `max_class` is `Private`. diff --git a/docs/plans/M5d2/10-memoryd-tiers.md b/docs/plans/M5d2/10-memoryd-tiers.md new file mode 100644 index 0000000..049c695 --- /dev/null +++ b/docs/plans/M5d2/10-memoryd-tiers.md @@ -0,0 +1,227 @@ +# M5d-2 task 10: `MemoryCommand::Tiers` and `bxctl memory tiers` + +**Branch:** `m5d2`. Check it: `git branch --show-current` prints `m5d2` and `git status --short` +is empty; otherwise stop. Never switch branch. +**Commit subject:** `memoryd: Tiers listing; bxctl memory tiers` +**Log row:** the Task cell is `M5d2/10-memoryd-tiers`. + +## Goal + +`MemoryCommand::Tiers` lists each identity entry and each note with how many times `recalls/` +named it and when it was last found (spec 4.4). Untrusted notes are included and marked. +Identity entries list with 0 hits (README known limits). `bxctl memory tiers` prints that +listing. The starter `config/jobs/tier.toml.example` documents a weekly job; nothing in +`loopd` special-cases it. + +## Every place that changes + +| Place | What happens there | +|---|---| +| `crates/bxcore/src/memory.rs` | `MemoryCommand::Tiers` after `Sync` | +| `crates/memoryd/src/tiers.rs` | new: `list` | +| `crates/memoryd/src/lib.rs` | `pub mod tiers;` | +| `crates/memoryd/src/serve/admin.rs` | `Tiers` arm | +| `crates/bxctl/src/memory.rs` | `["tiers"]` | +| `crates/bxctl/src/cli.rs` | `USAGE` names `tiers` | +| `config/jobs/tier.toml.example` | new, given starter | +| `crates/memoryd/tests/tiers.rs` | new, given | +| `crates/bxctl/tests/cli_memory_tiers.rs` | new, given | +| everything else | **no change**: not `cli_memory.rs` (T19: new file) | + +## Files + +- Copy, from `docs/plans/M5d2/files/` (the same path under it): the new tests + `crates/memoryd/tests/tiers.rs` and `crates/bxctl/tests/cli_memory_tiers.rs`, and the + starter `config/jobs/tier.toml.example`. +- Create: `crates/memoryd/src/tiers.rs` (function below). +- Modify: `crates/bxcore/src/memory.rs`, `crates/memoryd/src/lib.rs`, + `crates/memoryd/src/serve/admin.rs`, `crates/bxctl/src/memory.rs`, + `crates/bxctl/src/cli.rs`, `docs/implementer-log.md`. +- Do not touch: any other file. Do not edit `crates/bxctl/tests/cli_memory.rs`. + +## Facts you need + +- **`MemoryCommand` is `rename_all = "snake_case"`.** JSON is `"tiers"`. Add the variant after + `Sync`, with a doc comment. `admin.rs` match is exhaustive: add the arm in this task or it + does not compile. +- **Listing format, exact.** One block per identity entry, then one per note, in file order: + + ```text + identity memory.md hits 0 last - + I like tea. + identity user.md hits 0 last - + The owner is Kyle. + note session mm-a hits 2 last 2026-10-03T07:30:00.000Z private trusted + launch window moved. + note session mm-c hits 0 last - private untrusted + example.org says the launch moved. + ``` + + Two spaces before the text line. `hits` and `last` for identity are always `0` and `-`. + For a note, `hits` is how many `recalls/*.jsonl` lines contain a hit with + `kind == "noted"`, `session == note.session`, `record ==` that note's index in + `notes::parse` (from 0). `last` is the maximum `time` string among those lines, or `-`. + Class is `public` / `private` / `secret` (`notes::class_name`). Trust is `untrusted` when + `Kind::Noted { untrusted: true }`, else `trusted`. Moved/dropped notes: still list them as + `note` using the same line, class `private`, `trusted` (they have no untrusted flag). +- **`recalls/`.** `store.path("recalls")`. Missing directory: all counts 0. A file that + exists but cannot be read is an error. Ignore a line that does not parse. Deserialize: + + ```rust + #[derive(Deserialize)] + struct Logged { + time: String, + hits: Vec, + } + #[derive(Deserialize)] + struct Hit { + kind: String, + session: String, + record: i64, + } + ``` + + (extra fields in the file are ignored: this is our format but old lines may grow; the given + tests use exactly those fields plus `session` / `query` / `epoch` on the line. Use + `deny_unknown_fields` **off** so `epoch` and `query` do not fail.) +- **Identity.** `store.entries(MemoryFile::Memory)` then `User`. Empty file: no identity + lines for that file. +- **`list` returns `Result`.** Trailing newline on the last block is fine; + `bxctl` adds another when printing `MemoryAdminReply.text` (`writeln!`). The given bxctl + test includes a trailing newline in the reply text and expects it plus `writeln!`'s + newline. +- **`admin` arm.** `MemoryCommand::Tiers => tiers::list(&self.store, now)` — `now` is unused + by `list` today; keep the signature `list(store, now)` so the given tests compile. +- **`bxctl`.** `["tiers"] => Ask::Admin(MemoryCommand::Tiers, String::new())`. `["tiers", _]` + is usage. USAGE: add `| tiers` on the `bxctl memory` line. +- rustfmt `mod` order in `memoryd/src/lib.rs`. +- The items the edits use. This table is complete. + + | Item | What it does | + |---|---| + | `Store::entries` | identity | + | `notes::parse` / `class_name` | notes | + | `Store::path("recalls")` | the jsonl directory | + | `MemoryFile::name` | `memory.md` / `user.md` | + +## The function + +`crates/memoryd/src/tiers.rs` — fill `list`. No `todo!()`. + +```rust +//! The tiering listing: identity entries and notes with recall-hit counts (M5d spec, 4.4). + +use bxcore::{MemoryFile, Timestamp}; +use serde::Deserialize; + +use crate::files::Store; +use crate::notes::{self, Kind}; + +#[derive(Deserialize)] +struct Logged { + time: String, + hits: Vec, +} + +#[derive(Deserialize)] +struct Hit { + kind: String, + session: String, + record: i64, +} + +/// One listing for `bxctl memory tiers` and for the tiering job's user text. +pub fn list(store: &Store, _now: Timestamp) -> Result { + let _ = store; + Ok(String::new()) +} +``` + +Replace the stub body in this task. + +### `crates/bxcore/src/memory.rs` + +**M1.** After `Sync,` in `MemoryCommand`: + +```rust + /// Identity entries and notes with recall-hit counts (M5d spec, section 4.4). + Tiers, +``` + +### `crates/memoryd/src/serve/admin.rs` + +**A1.** After the `Sync` arm, before the closing of the `match`: + +```rust + MemoryCommand::Tiers => crate::tiers::list(&self.store, now), +``` + +### `crates/bxctl/src/memory.rs` + +**B1.** After `["sync"] => ...`, before `_ => return Err(UsageError)`: + +```rust + ["tiers"] => Ask::Admin(MemoryCommand::Tiers, String::new()), +``` + +### `crates/bxctl/src/cli.rs` + +**U1.** The `bxctl memory` USAGE line. The line + +```text + bxctl memory log [N] | undo | pending | accept | reject | sync [--memory-socket PATH] +``` + +becomes + +```text + bxctl memory log [N] | undo | pending | accept | reject | sync | tiers [--memory-socket PATH] +``` + +## Steps + +- [ ] **1. Copy the given files.** + + ```sh + cp docs/plans/M5d2/files/crates/memoryd/tests/tiers.rs crates/memoryd/tests/ + cp docs/plans/M5d2/files/crates/bxctl/tests/cli_memory_tiers.rs crates/bxctl/tests/ + mkdir -p config/jobs + cp docs/plans/M5d2/files/config/jobs/tier.toml.example config/jobs/tier.toml.example + ``` + +- [ ] **2. See them fail.** `cargo build --tests --keep-going -p bxcore -p memoryd -p bxctl`, + timeout 600000 ms: `tiers` does not compile (`E0432` unresolved `memoryd::tiers`); + `cli_memory_tiers` does not compile (`E0599` no variant `Tiers`). After M1 alone, + `admin.rs` fails (`non-exhaustive patterns`). Do M1, A1, `tiers.rs`, B1, U1 together in + step 3. This step is the first build against the given tests. +- [ ] **3. The edits.** M1, `tiers.rs` with a real `list`, `pub mod tiers;`, A1, B1, U1. + `cargo check -p bxcore -p memoryd -p bxctl` prints no error and no warning. + `wc -l crates/memoryd/src/tiers.rs` prints a number ≤ 500. +- [ ] **4. The tests.** `cargo test -p memoryd --test tiers` (4) and + `cargo test -p bxctl --test cli_memory_tiers --test cli_memory --test cli_memory_sync` + (2 new): every test passes. +- [ ] **5. Check.** `git diff --stat` lists the files in "Every place that changes" except + the new tests and starter (those are untracked until `git add`). +- [ ] **6. Run the gate.** `cargo fmt --all`, then `make gate`, timeout 600000 ms. Last line + `gate: ok`. Test count **1572** (1566 before, plus 6). +- [ ] **7. Log and commit.** Task cell `M5d2/10-memoryd-tiers`. Stage with + `git add crates/bxcore crates/memoryd crates/bxctl config/jobs docs/implementer-log.md + Cargo.lock`, then commit as `AGENTS.md` says. + +## Done when + +- `make gate` prints `gate: ok` with 1572 tests. +- `git diff HEAD~1 --stat` lists only the files this task names plus the log. + +## Stop and report if + +- The branch check fails; a "before" block is not found exactly once; step 2 differs; `cargo + check` warns; a test fails; the gate count is not 1572; you edit `cli_memory.rs`; identity + entries show a non-zero hit count; untrusted notes are omitted; you add a cron crate; you + special-case the name `tier` in `loopd`. + +## What the given tests pin + +Identity lines are `hits 0 last -` with the entry text; notes join `recalls/` on +`(noted, session, record)` and take the latest `time`; untrusted notes are listed and marked; +a hit on note 1 does not count for note 0; `bxctl memory tiers` parses and prints the reply. diff --git a/docs/plans/M5d2/README.md b/docs/plans/M5d2/README.md new file mode 100644 index 0000000..afcc1e4 --- /dev/null +++ b/docs/plans/M5d2/README.md @@ -0,0 +1,132 @@ +# M5d-2 implementation plan: the scheduler + +> For the implementer: each task file is one closed task for one fresh session. Read `AGENTS.md`, +> then the task file, and do exactly that task. The driver is `tools/run-plan.sh`. + +**Goal:** heartbeat and cron run as isolated sessions on the background slot; live turns use the +session's slot; opted-in tool results get a `ToolSummary` that `messages()` ignores and `recall` +indexes; the tiering listing is an ordinary admin command. Answers stay in the session log. +Nothing is posted to Mattermost. + +**Architecture:** `run_turn` reads `session.slot()`. `loopd` gains `[heartbeat]`, a cron matcher, +job-file loading, a scheduler thread that shares `Context` and the busy set, and a no-tools +summary call for `http_fetch`. `bxcore` gains `LogRecord::ToolSummary` and `MemoryCommand::Tiers`. +`memoryd` indexes summaries and answers `Tiers` from `recalls/`. `bxctl memory tiers` prints that +listing. No new crate. + +**Tech stack:** Rust 1.98.1, edition 2024; no cron crate; no new dependency. + +**Spec:** `docs/specs/2026-10-03-m5d-scheduler.md`, sections 4, 5, 6, 7 (the M5d-2 tests), and 8 +(`jobs-invalid`, `heartbeat-prompt-unreadable`). M5d-1 (epochs and compaction) is already on the +branch this plan starts from. + +## The owner's decisions (2026-10-03) + +1. Heartbeat and cron answers stay in the session log. They are not posted to Mattermost. +2. Five-field UTC crontab, `{home}/jobs/.toml`, a new session each run, a tick skipped + when that job is still live. +3. Tool summaries on the background slot; a `ToolSummary` log record with the result's labels, + FTS-indexed; `http_fetch` opts in via `[compact] summarise`. + +## What the code already has, and what it lacks + +- M5d-1 writes epochs, compact after a turn, Honcho over every epoch file. Live turns still send + `slots.main`. `[compact] summarise` is stored and unused. Unknown names are not a config error. +- `LogRecord` has no `ToolSummary`. `messages()` names every variant. `support/turn.rs` `types` + is exhaustive. +- `loopd serve` binds the channel and blocks in `channel::serve`. There is no scheduler thread. +- `MemoryCommand` has no `Tiers`. `bxctl memory` has no `tiers`. +- `SessionId` is `[a-z0-9-]`, max 64. Job ids `job--t` must fit. + +## Global constraints + +- Roles depend on `bxcore` only. No `unsafe`, no async runtime, no new crate. +- No file the implementer reads holds a chat-template tag whole (T33). +- Every path a given test creates is removed by a guard (T44); no test depends on timing (T27): + the scheduler is `tick(now)`, never a sleep. +- Fail-closed states `jobs-invalid` and `heartbeat-prompt-unreadable` have runbook entries, + added by the task that adds their pointer (T49). +- Each task gives its baseline and its gate count (T50, T53); the branch line is a check, never + `git switch` (T51). +- `Cargo.lock` is in the `git add` line (T24). Gate timeout 600000 ms (T40). +- A later task that changes a shared given file ships its own copy (T19): task 02 replaces + `crates/loopd/tests/support/turn.rs` so `types` names `ToolSummary`. + +## Tasks + +Every task below has been written to be replayed from its text. The gate count is the passed +total of `cargo test --workspace` plus one, for the ignored `end_to_end` test the gate runs on +its own. Before task 01 it is **1507** (talos, rustc 1.98.1; M5d-1 task 08). + +| # | Delivers | Crates | Gate after | Tests added | +|---|---|---|---|---| +| 01 | `run_turn` sends `session.slot()`; a session created on the background slot has `id_slot` of that slot; `mm-*` stays main | loopd | 1511 | 4 | +| 02 | `LogRecord::ToolSummary`; `messages()` ignores it; `types` names it; JSON round-trip | bxcore, loopd | 1517 | 6 | +| 03 | `[heartbeat]` (`every_s` 0, `prompt` `heartbeat.md`); unknown `[compact] summarise` names are a config error; starter `heartbeat.md` | loopd | 1525 | 8 | +| 04 | Five-field UTC cron matcher: `*`, `n`, `n-m`, `*/n`; day-of-week 0 is Sunday | loopd | 1535 | 10 | +| 05 | `{home}/jobs/.toml`, `runs.jsonl`, `jobs-invalid` pointer and runbook | loopd | 1543 | 8 | +| 06 | Heartbeat tick: time line, standing session `heartbeat`, skip if busy, unreadable prompt logged once | loopd | 1549 | 6 | +| 07 | Cron tick: new session per run, skip if previous live, leftover `started` becomes `failed` | loopd | 1555 | 6 | +| 08 | After a turn, opted-in `ToolResult` without a `ToolSummary` gets a background summary appended with the result's labels | loopd | 1561 | 6 | +| 09 | `memoryd` indexes `ToolSummary` as kind `summary`; `recall` finds it and taints when untrusted | memoryd | 1566 | 5 | +| 10 | `MemoryCommand::Tiers`, `bxctl memory tiers`, starter `config/jobs/tier.toml.example` | bxcore, memoryd, bxctl | 1572 | 6 | + +After task 10, the design model runs the M5d-2 device check (spec 7) on straylight: a heartbeat +turn on the background slot leaves a following main-slot turn's `cache_n` near its previous +total; a `jobs/tier.toml` for the check promotes only a trusted note. + +## Review Focus + +These are implied by the spec and pinned in the named task. A reviewer who only reads the +happy-path tests will miss them. + +- A live `mm-*` turn still sends `slots.main` after heartbeat used the background slot (task 01). +- Inserting a `ToolSummary` between a `ToolResult` and a later `User` does not break prefix + extension (task 02). +- An unknown `[compact] summarise` name is a config error; an empty list is not (task 03). +- DOM and DOW are AND; `@daily` and a sixth field are errors (task 04). +- A leftover `started` whose session is currently held is not closed as `failed` (tasks 06, 07). +- `finish_turn` summarises before it compacts, so the `ToolResult` is still in the current epoch + (task 08). +- A secret `ToolSummary` is skipped when the index is private (task 09). +- Untrusted notes appear in `Tiers` and are marked; they cannot be omitted (task 10). + +## Known limits + +These are known, accepted for M5d-2, and not bugs to fix in a task: + +- **No Mattermost delivery.** Heartbeat and cron answers are in `sessions//` only. The owner + reads them with `bxctl chat --session` or by opening the log. +- **DOM and DOW are AND.** Both restricted (`0 7 1 * 0`) matches only a Sunday that is also the + 1st. Standard Vixie cron would OR them. The spec does not ask for OR. +- **Tool-summary inference runs inside `finish_turn`, before compaction, while the session is + still held.** That is how compaction waits for the summary, and how `missing` still sees the + current epoch's `ToolResult`. A second live turn on that session waits. The summary request + itself uses the background slot and omits `tools`. A later split that infers without the hold + is out of scope. +- **Job files are checked at start.** A file that becomes invalid after `loopd` is running is + skipped that tick and logged; it does not stop the process. +- **Identity entries in `Tiers` list with 0 hits** unless a `recalls/` line names that exact + index row. Notes carry counts from `recalls/`. That is enough for the tiering prompt. + +## Before hand-over + +Work through `docs/plans/CHECKLIST.md` with a reference in a scratch worktree with its own build +directory, replayed from its task text; mutations; `/tmp` compared around a gate run; the +reference moved to straylight as a bundle before the run (T18). Task files name their log row +(T47). + +## Running it + +The reference (`m5d2-ref`, tasks 01 to 10) is moved to straylight as a bundle and deleted here +before the run (T18). Then, from the repository root, with the OpenCode TUI closed, after M5d-1 +has landed on master: + +```sh +git switch -c m5d2 master +OPENCODE_MODEL=straylight/ornith-1.5-35b-a3b RESUME=1 \ + SLOTS_URL='http://straylight:11434/slots?model=ornith-1.5-35b-a3b' \ + tools/run-plan.sh docs/plans/M5d2 01 +``` + +After task 10, the gate count is 1572. Then the device check above. diff --git a/docs/plans/M5d2/files/config/heartbeat.md b/docs/plans/M5d2/files/config/heartbeat.md new file mode 100644 index 0000000..a457e0f --- /dev/null +++ b/docs/plans/M5d2/files/config/heartbeat.md @@ -0,0 +1,4 @@ +It is time for a heartbeat. + +Look at recent notes and the current identity. Write a note only if something needs the owner, or +if a standing fact should move. Do not use tools that fetch. Do not address the owner. diff --git a/docs/plans/M5d2/files/config/jobs/tier.toml.example b/docs/plans/M5d2/files/config/jobs/tier.toml.example new file mode 100644 index 0000000..3596b1d --- /dev/null +++ b/docs/plans/M5d2/files/config/jobs/tier.toml.example @@ -0,0 +1,6 @@ +# Copy to {BOXMAKER_HOME}/jobs/tier.toml and edit the schedule. +# Five-field UTC crontab. This example is Sunday 07:00. +schedule = "0 7 * * 0" +prompt = """Read the listing in this message. Promote only trusted notes into identity +(memory.md / user.md) with add or consolidate. Do not promote an untrusted note. +Do not fetch. Write a note of what you changed, or that nothing needed changing.""" diff --git a/docs/plans/M5d2/files/config/summarise.md b/docs/plans/M5d2/files/config/summarise.md new file mode 100644 index 0000000..652925e --- /dev/null +++ b/docs/plans/M5d2/files/config/summarise.md @@ -0,0 +1,2 @@ +Summarise this tool result for later recall. Keep facts, names, numbers and URLs. Drop boilerplate +and navigation. Do not use tools. Do not address the owner. Do not invent facts. diff --git a/docs/plans/M5d2/files/crates/bxcore/tests/tool_summary.rs b/docs/plans/M5d2/files/crates/bxcore/tests/tool_summary.rs new file mode 100644 index 0000000..6e50e6d --- /dev/null +++ b/docs/plans/M5d2/files/crates/bxcore/tests/tool_summary.rs @@ -0,0 +1,52 @@ +//! Tests for `LogRecord::ToolSummary`. Do not edit. + +use bxcore::{CallId, DataClass, LogRecord, Timestamp}; + +fn ts() -> Timestamp { + Timestamp::parse("2026-09-18T08:00:00.000Z").unwrap() +} + +#[test] +fn tool_summary_round_trips() { + let rec = LogRecord::ToolSummary { + time: ts(), + call: CallId(3), + tool_call_id: "call_x".to_string(), + content: "the page is about tea".to_string(), + class: DataClass::Public, + untrusted: true, + }; + let line = serde_json::to_string(&rec).unwrap(); + assert!(line.contains("\"type\":\"tool_summary\""), "{line}"); + let back: LogRecord = serde_json::from_str(&line).unwrap(); + assert_eq!(back, rec); +} + +#[test] +fn unknown_fields_are_refused() { + let line = r#"{"type":"tool_summary","time":"2026-09-18T08:00:00.000Z","call":3,"tool_call_id":"c","content":"x","class":"public","untrusted":false,"extra":1}"#; + assert!(serde_json::from_str::(line).is_err()); +} + +#[test] +fn field_order_is_the_struct_order() { + let rec = LogRecord::ToolSummary { + time: ts(), + call: CallId(1), + tool_call_id: "c".to_string(), + content: "x".to_string(), + class: DataClass::Private, + untrusted: false, + }; + let line = serde_json::to_string(&rec).unwrap(); + let time = line.find("\"time\"").unwrap(); + let call = line.find("\"call\"").unwrap(); + let id = line.find("\"tool_call_id\"").unwrap(); + let content = line.find("\"content\"").unwrap(); + let class = line.find("\"class\"").unwrap(); + let untrusted = line.find("\"untrusted\"").unwrap(); + assert!( + time < call && call < id && id < content && content < class && class < untrusted, + "{line}" + ); +} diff --git a/docs/plans/M5d2/files/crates/bxctl/tests/cli_memory_tiers.rs b/docs/plans/M5d2/files/crates/bxctl/tests/cli_memory_tiers.rs new file mode 100644 index 0000000..1477e9b --- /dev/null +++ b/docs/plans/M5d2/files/crates/bxctl/tests/cli_memory_tiers.rs @@ -0,0 +1,86 @@ +//! Tests for `bxctl memory tiers`. Do not edit. + +#[path = "support/temp.rs"] +mod temp; +use std::os::unix::net::UnixListener; +use std::path::{Path, PathBuf}; +use std::sync::mpsc; +use std::time::Duration; + +use bxcore::{Envelope, MemoryAdminReply, MemoryCommand, Message, read_frame, write_frame}; +use bxctl::cli::{Command, UsageError, parse}; +use bxctl::memory::{Ask, run}; + +fn args(line: &str) -> Vec { + line.split_whitespace().map(str::to_string).collect() +} + +fn home() -> PathBuf { + PathBuf::from("/srv/bx") +} + +fn fake(socket: &Path, answer: Message) -> mpsc::Receiver { + let listener = UnixListener::bind(socket).unwrap(); + let (tx, rx) = mpsc::channel(); + std::thread::spawn(move || { + let (mut stream, _) = listener.accept().unwrap(); + let env = read_frame(&mut stream).unwrap(); + tx.send(env.msg).unwrap(); + let reply = Envelope { + v: 1, + id: env.id, + r#final: true, + msg: answer, + }; + write_frame(&mut stream, &reply).unwrap(); + }); + rx +} + +#[test] +fn memory_tiers_parses() { + let h = home(); + assert_eq!( + parse(&args("memory tiers"), &h), + Ok(Command::Memory { + socket: PathBuf::from("/srv/bx/run/owner-memory/admin.sock"), + ask: Ask::Admin(MemoryCommand::Tiers, String::new()) + }) + ); + assert_eq!( + parse(&args("memory --memory-socket /r/m.sock tiers"), &h), + Ok(Command::Memory { + socket: PathBuf::from("/r/m.sock"), + ask: Ask::Admin(MemoryCommand::Tiers, String::new()) + }) + ); + assert_eq!(parse(&args("memory tiers extra"), &h), Err(UsageError)); +} + +#[test] +fn memory_tiers_prints_the_reply() { + let d = temp::TempDir::new("bx-tiers"); + let socket = d.join("admin.sock"); + let listing = "identity memory.md hits 0 last -\n I like tea.\n"; + let seen = fake( + &socket, + Message::MemoryAdminReply(MemoryAdminReply { + text: listing.to_string(), + }), + ); + let mut out = Vec::new(); + run( + &socket, + &Ask::Admin(MemoryCommand::Tiers, String::new()), + Duration::from_secs(5), + &mut out, + ) + .unwrap(); + assert_eq!(String::from_utf8(out).unwrap(), format!("{listing}\n")); + match seen.recv().unwrap() { + Message::MemoryAdmin(m) => { + assert_eq!((m.command, m.arg.as_str()), (MemoryCommand::Tiers, "")) + } + other => panic!("{other:?}"), + } +} diff --git a/docs/plans/M5d2/files/crates/loopd/tests/config_heartbeat.rs b/docs/plans/M5d2/files/crates/loopd/tests/config_heartbeat.rs new file mode 100644 index 0000000..db44748 --- /dev/null +++ b/docs/plans/M5d2/files/crates/loopd/tests/config_heartbeat.rs @@ -0,0 +1,110 @@ +//! Tests for `[heartbeat]` and unknown `[compact] summarise` names. Do not edit. + +use loopd::config::Config; +use loopd::tools::Registry; +use std::path::{Path, PathBuf}; + +fn fixture(name: &str) -> PathBuf { + Path::new(env!("CARGO_MANIFEST_DIR")) + .join("tests/fixtures/config") + .join(name) +} + +fn minimal() -> String { + std::fs::read_to_string(fixture("minimal.toml")).unwrap() +} + +struct Scratch { + dir: PathBuf, +} + +impl Scratch { + fn new() -> Scratch { + let n = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .unwrap() + .as_nanos(); + let dir = std::env::temp_dir().join(format!("loopd-hb-{}-{n}", std::process::id())); + let _ = std::fs::remove_dir_all(&dir); + std::fs::create_dir_all(&dir).unwrap(); + Scratch { dir } + } +} + +impl Drop for Scratch { + fn drop(&mut self) { + let _ = std::fs::remove_dir_all(&self.dir); + } +} + +#[test] +fn missing_heartbeat_table_is_disabled() { + let c = Config::load(&fixture("minimal.toml")).unwrap(); + assert_eq!(c.heartbeat.every_s, 0); + assert_eq!( + c.heartbeat_prompt(), + fixture("minimal.toml") + .parent() + .unwrap() + .join("heartbeat.md") + ); +} + +#[test] +fn the_table_overrides_every_default() { + let text = format!( + "{}\n[heartbeat]\nevery_s = 60\nprompt = \"pulse.md\"\n", + minimal() + ); + let c = Config::parse(&text).unwrap(); + assert_eq!(c.heartbeat.every_s, 60); + assert_eq!(c.heartbeat.prompt, PathBuf::from("pulse.md")); +} + +#[test] +fn unknown_keys_are_errors() { + let bad = format!("{}\n[heartbeat]\nzz_unknown = 1\n", minimal()); + assert!(Config::parse(&bad).is_err()); +} + +#[test] +fn heartbeat_prompt_is_beside_the_config_file() { + let home = Scratch::new(); + let cfg_path = home.dir.join("loopd.toml"); + std::fs::write(&cfg_path, minimal()).unwrap(); + std::fs::write(home.dir.join("system.md"), "sys\n").unwrap(); + let c = Config::load(&cfg_path).unwrap(); + assert_eq!(c.heartbeat_prompt(), home.dir.join("heartbeat.md")); +} + +#[test] +fn default_summarise_http_fetch_is_known_to_m5c() { + let c = Config::parse(&minimal()).unwrap(); + c.check_summarise(&Registry::m5c()).unwrap(); +} + +#[test] +fn an_unknown_summarise_name_is_an_error() { + let text = format!("{}\n[compact]\nsummarise = [\"no_such_tool\"]\n", minimal()); + let c = Config::parse(&text).unwrap(); + let err = c.check_summarise(&Registry::m5c()).unwrap_err(); + assert!(err.contains("no_such_tool"), "{err}"); + assert!(err.contains("summarise"), "{err}"); +} + +#[test] +fn empty_summarise_is_allowed() { + let text = format!("{}\n[compact]\nsummarise = []\n", minimal()); + let c = Config::parse(&text).unwrap(); + c.check_summarise(&Registry::m5c()).unwrap(); +} + +#[test] +fn http_fetch_is_known_and_clock_is_known() { + let text = format!( + "{}\n[compact]\nsummarise = [\"http_fetch\", \"clock\"]\n", + minimal() + ); + let c = Config::parse(&text).unwrap(); + c.check_summarise(&Registry::m5c()).unwrap(); +} diff --git a/docs/plans/M5d2/files/crates/loopd/tests/cron.rs b/docs/plans/M5d2/files/crates/loopd/tests/cron.rs new file mode 100644 index 0000000..0aa422b --- /dev/null +++ b/docs/plans/M5d2/files/crates/loopd/tests/cron.rs @@ -0,0 +1,85 @@ +//! Tests for the five-field UTC cron matcher. Do not edit. + +use bxcore::Timestamp; +use loopd::cron::{self, CronError}; + +fn ts(s: &str) -> Timestamp { + Timestamp::parse(s).unwrap() +} + +fn due(schedule: &str, at: &str) -> bool { + cron::due(schedule, ts(at)).unwrap() +} + +#[test] +fn a_daily_hour_matches_that_minute_only() { + assert!(due("0 7 * * *", "2026-10-03T07:00:00.000Z")); + assert!(!due("0 7 * * *", "2026-10-03T07:01:00.000Z")); + assert!(!due("0 7 * * *", "2026-10-03T06:00:00.000Z")); +} + +#[test] +fn every_minute_matches() { + assert!(due("* * * * *", "2026-10-03T00:00:00.000Z")); + assert!(due("* * * * *", "2026-10-03T23:59:00.000Z")); +} + +#[test] +fn a_step_matches_multiples_from_zero() { + assert!(due("*/5 * * * *", "2026-10-03T00:00:00.000Z")); + assert!(due("*/5 * * * *", "2026-10-03T00:05:00.000Z")); + assert!(!due("*/5 * * * *", "2026-10-03T00:06:00.000Z")); +} + +#[test] +fn a_range_is_inclusive() { + assert!(due("0-10 * * * *", "2026-10-03T00:00:00.000Z")); + assert!(due("0-10 * * * *", "2026-10-03T00:10:00.000Z")); + assert!(!due("0-10 * * * *", "2026-10-03T00:11:00.000Z")); +} + +#[test] +fn sunday_is_zero() { + // 2026-10-04 is a Sunday. + assert!(due("0 7 * * 0", "2026-10-04T07:00:00.000Z")); + // 2026-10-03 is a Saturday. + assert!(!due("0 7 * * 0", "2026-10-03T07:00:00.000Z")); +} + +#[test] +fn the_first_of_the_month() { + assert!(due("0 7 1 * *", "2026-10-01T07:00:00.000Z")); + assert!(!due("0 7 1 * *", "2026-10-02T07:00:00.000Z")); +} + +#[test] +fn four_or_six_fields_are_errors() { + assert!(matches!( + cron::due("* * * *", ts("2026-10-03T00:00:00.000Z")), + Err(CronError::Fields(_)) + )); + assert!(matches!( + cron::due("* * * * * *", ts("2026-10-03T00:00:00.000Z")), + Err(CronError::Fields(_)) + )); +} + +#[test] +fn an_out_of_range_number_is_an_error() { + assert!(cron::due("60 * * * *", ts("2026-10-03T00:00:00.000Z")).is_err()); + assert!(cron::due("* 24 * * *", ts("2026-10-03T00:00:00.000Z")).is_err()); + assert!(cron::due("* * * * 7", ts("2026-10-03T00:00:00.000Z")).is_err()); +} + +#[test] +fn nicknames_are_errors() { + let err = cron::due("@daily", ts("2026-10-03T00:00:00.000Z")).unwrap_err(); + assert!(matches!(err, CronError::Fields(_)), "{err:?}"); +} + +#[test] +fn display_names_the_schedule() { + let err = cron::due("x x x x x", ts("2026-10-03T00:00:00.000Z")).unwrap_err(); + let s = err.to_string(); + assert!(s.contains("x x x x x"), "{s}"); +} diff --git a/docs/plans/M5d2/files/crates/loopd/tests/jobs.rs b/docs/plans/M5d2/files/crates/loopd/tests/jobs.rs new file mode 100644 index 0000000..6e28f1a --- /dev/null +++ b/docs/plans/M5d2/files/crates/loopd/tests/jobs.rs @@ -0,0 +1,151 @@ +//! Tests for job files and the run ledger. Do not edit. + +use bxcore::{SessionId, Timestamp}; +use loopd::jobs::{self, JobsError, RunStatus}; + +fn ts() -> Timestamp { + Timestamp::parse("2026-10-03T07:00:00.000Z").unwrap() +} + +struct Scratch { + dir: std::path::PathBuf, +} + +impl Scratch { + fn new() -> Scratch { + let n = std::time::SystemTime::now() + .duration_since(std::time::UNIX_EPOCH) + .unwrap() + .as_nanos(); + let dir = std::env::temp_dir().join(format!("loopd-jobs-{}-{n}", std::process::id())); + let _ = std::fs::remove_dir_all(&dir); + std::fs::create_dir_all(&dir).unwrap(); + Scratch { dir } + } +} + +impl Drop for Scratch { + fn drop(&mut self) { + let _ = std::fs::remove_dir_all(&self.dir); + } +} + +#[test] +fn a_valid_job_file_loads() { + let home = Scratch::new(); + let jobs = home.dir.join("jobs"); + std::fs::create_dir(&jobs).unwrap(); + std::fs::write( + jobs.join("mail.toml"), + "schedule = \"0 7 * * *\"\nprompt = \"Check mail.\"\n", + ) + .unwrap(); + let loaded = jobs::load(&jobs).unwrap(); + assert_eq!(loaded.len(), 1); + assert_eq!(loaded[0].name, "mail"); + assert_eq!(loaded[0].schedule, "0 7 * * *"); + assert_eq!(loaded[0].prompt, "Check mail."); +} + +#[test] +fn unknown_keys_are_errors_with_the_runbook() { + let home = Scratch::new(); + let jobs = home.dir.join("jobs"); + std::fs::create_dir(&jobs).unwrap(); + std::fs::write( + jobs.join("mail.toml"), + "schedule = \"* * * * *\"\nprompt = \"x\"\nwhen = \"now\"\n", + ) + .unwrap(); + let err = jobs::load(&jobs).unwrap_err(); + let s = err.to_string(); + assert!(s.contains("mail.toml"), "{s}"); + assert!(s.ends_with("see docs/runbook.md#jobs-invalid"), "{s}"); +} + +#[test] +fn a_bad_name_is_an_error() { + let home = Scratch::new(); + let jobs = home.dir.join("jobs"); + std::fs::create_dir(&jobs).unwrap(); + std::fs::write( + jobs.join("Mail.toml"), + "schedule = \"* * * * *\"\nprompt = \"x\"\n", + ) + .unwrap(); + let err = jobs::load(&jobs).unwrap_err(); + assert!(matches!(err, JobsError::Name(_)), "{err:?}"); +} + +#[test] +fn session_id_fits_and_is_lowercase() { + let id = jobs::session_id("mail", ts()).unwrap(); + assert_eq!(id.as_str(), "job-mail-20261003t0700"); + SessionId::new(id.as_str()).unwrap(); +} + +#[test] +fn a_name_too_long_for_a_session_id_is_an_error() { + let long = "n".repeat(50); + let err = jobs::session_id(&long, ts()).unwrap_err(); + assert!(matches!(err, JobsError::Name(_)), "{err:?}"); +} + +#[test] +fn a_missing_jobs_directory_is_empty() { + let home = Scratch::new(); + let loaded = jobs::load(&home.dir.join("jobs")).unwrap(); + assert!(loaded.is_empty()); +} + +#[test] +fn leftover_started_is_closed_as_failed() { + let home = Scratch::new(); + let path = home.dir.join("runs.jsonl"); + let session = jobs::session_id("mail", ts()).unwrap(); + jobs::append( + &path, + &jobs::Run { + time: ts(), + name: "mail".to_string(), + session: session.clone(), + status: RunStatus::Started, + }, + ) + .unwrap(); + let n = jobs::close_leftovers(&path, ts()).unwrap(); + assert_eq!(n, 1); + assert!(jobs::live_session(&path, "mail").unwrap().is_none()); +} + +#[test] +fn a_started_run_without_a_close_is_live() { + let home = Scratch::new(); + let path = home.dir.join("runs.jsonl"); + let session = jobs::session_id("mail", ts()).unwrap(); + jobs::append( + &path, + &jobs::Run { + time: ts(), + name: "mail".to_string(), + session: session.clone(), + status: RunStatus::Started, + }, + ) + .unwrap(); + assert_eq!( + jobs::live_session(&path, "mail").unwrap().as_ref(), + Some(&session) + ); + jobs::append( + &path, + &jobs::Run { + time: ts(), + name: "mail".to_string(), + session, + status: RunStatus::Finished, + }, + ) + .unwrap(); + assert!(jobs::live_session(&path, "mail").unwrap().is_none()); +} diff --git a/docs/plans/M5d2/files/crates/loopd/tests/scheduler_cron.rs b/docs/plans/M5d2/files/crates/loopd/tests/scheduler_cron.rs new file mode 100644 index 0000000..bc21e08 --- /dev/null +++ b/docs/plans/M5d2/files/crates/loopd/tests/scheduler_cron.rs @@ -0,0 +1,186 @@ +//! Tests for the cron tick. Do not edit. + +mod support; + +use bxcore::{SessionId, Timestamp}; +use loopd::channel::Context; +use loopd::jobs::{self, RunStatus}; +use loopd::llama::Client; +use loopd::memory::NoMemory; +use loopd::scheduler::{Scheduler, TickWork}; +use loopd::tools::Registry; +use std::sync::Arc; +use support::{FakeServer, Home, Reply, ScriptedPort}; + +const CHAT: &str = "/v1/chat/completions"; + +fn sse() -> Reply { + let data1 = r#"data: {"choices":[{"finish_reason":null,"index":0,"delta":{"content":"done"}}],"id":"j","timings":{"cache_n":0,"prompt_n":10,"predicted_n":1}}"#; + let data2 = r#"data: {"choices":[{"finish_reason":"stop","index":0,"delta":{}}],"id":"j","timings":{"cache_n":0,"prompt_n":10,"predicted_n":1}}"#; + let body = format!("{data1}\n\n{data2}\n\ndata: [DONE]\n\n"); + Reply::raw(format!( + "HTTP/1.1 200 OK\r\nContent-Type: text/event-stream\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}", + body.len() + )) +} + +fn ts(s: &str) -> Timestamp { + Timestamp::parse(s).unwrap() +} + +fn world(schedule: &str) -> (Home, FakeServer, Arc) { + let home = Home::new(); + home.write("compact.md", "Summarise.\n"); + home.write( + "jobs/mail.toml", + &format!("schedule = \"{schedule}\"\nprompt = \"Check mail.\"\n"), + ); + let server = FakeServer::start(); + let cfg = home.config(&server.socket); + server.route(CHAT, vec![sse(), sse()]); + let ctx = Arc::new(Context::new( + cfg.clone(), + Client::new(cfg), + Box::new(ScriptedPort::new(vec![])), + Registry::m2b(), + Box::new(NoMemory), + )); + (home, server, ctx) +} + +#[test] +fn a_due_job_starts_a_new_session_on_the_background_slot() { + let (home, server, ctx) = world("0 7 * * *"); + let mut sched = Scheduler::new(ts("2026-10-03T06:00:00.000Z")); + let work = sched.tick(&ctx, ts("2026-10-03T07:00:00.000Z")); + let want = jobs::session_id("mail", ts("2026-10-03T07:00:00.000Z")).unwrap(); + assert_eq!( + work, + vec![TickWork::Job { + name: "mail".to_string(), + session: want.clone() + }] + ); + let sent = server.requests_to(CHAT); + assert_eq!(sent.len(), 1); + assert_eq!(sent[0].json()["id_slot"], 1); + let last = sent[0].json()["messages"] + .as_array() + .unwrap() + .last() + .unwrap(); + let content = last["content"].as_str().unwrap(); + assert!( + content.starts_with("It is 2026-10-03T07:00:00.000Z."), + "{content}" + ); + assert!(content.contains("Check mail."), "{content}"); + let s = loopd::session::Session::open(&home.dir, want).unwrap(); + assert_eq!(s.slot(), 1); +} + +#[test] +fn a_job_not_due_does_not_run() { + let (_home, server, ctx) = world("0 7 * * *"); + let mut sched = Scheduler::new(ts("2026-10-03T06:00:00.000Z")); + let work = sched.tick(&ctx, ts("2026-10-03T06:59:00.000Z")); + assert!(work.is_empty()); + assert!(server.requests_to(CHAT).is_empty()); +} + +#[test] +fn a_live_previous_run_skips_the_tick() { + let (home, server, ctx) = world("* * * * *"); + let ledger = home.dir.join("jobs/runs.jsonl"); + let previous = jobs::session_id("mail", ts("2026-10-03T06:59:00.000Z")).unwrap(); + jobs::append( + &ledger, + &jobs::Run { + time: ts("2026-10-03T06:59:00.000Z"), + name: "mail".to_string(), + session: previous.clone(), + status: RunStatus::Started, + }, + ) + .unwrap(); + let held = ctx.hold(previous).expect("hold"); + let mut sched = Scheduler::new(ts("2026-10-03T06:00:00.000Z")); + let work = sched.tick(&ctx, ts("2026-10-03T07:00:00.000Z")); + assert_eq!( + work, + vec![TickWork::SkippedJob { + name: "mail".to_string() + }] + ); + assert!(server.requests_to(CHAT).is_empty()); + drop(held); +} + +#[test] +fn leftover_started_is_failed_on_the_first_tick() { + let (home, _server, ctx) = world("* * * * *"); + let ledger = home.dir.join("jobs/runs.jsonl"); + let previous = jobs::session_id("mail", ts("2026-10-03T06:59:00.000Z")).unwrap(); + jobs::append( + &ledger, + &jobs::Run { + time: ts("2026-10-03T06:59:00.000Z"), + name: "mail".to_string(), + session: previous.clone(), + status: RunStatus::Started, + }, + ) + .unwrap(); + let mut sched = Scheduler::new(ts("2026-10-03T07:00:00.000Z")); + sched.tick(&ctx, ts("2026-10-03T07:00:00.000Z")); + let text = home.read("jobs/runs.jsonl"); + assert!( + text.contains("failed"), + "leftover started was closed: {text}" + ); + assert!(text.contains(previous.as_str()), "{text}"); +} + +#[test] +fn a_cron_turn_has_no_speaker() { + let (home, _server, ctx) = world("0 7 * * *"); + let mut sched = Scheduler::new(ts("2026-10-03T06:00:00.000Z")); + sched.tick(&ctx, ts("2026-10-03T07:00:00.000Z")); + let id = jobs::session_id("mail", ts("2026-10-03T07:00:00.000Z")).unwrap(); + let recs = { + let text = home.read(&format!("sessions/{}/0.jsonl", id.as_str())); + text.lines() + .map(|l| serde_json::from_str::(l).unwrap()) + .collect::>() + }; + let user = recs + .iter() + .find_map(|r| match r { + bxcore::LogRecord::User { speaker, .. } => Some(speaker), + _ => None, + }) + .unwrap(); + assert_eq!(user, &None); +} + +#[test] +fn two_due_minutes_make_two_sessions() { + let (home, server, ctx) = world("* * * * *"); + let mut sched = Scheduler::new(ts("2026-10-03T06:58:00.000Z")); + sched.tick(&ctx, ts("2026-10-03T06:59:00.000Z")); + sched.tick(&ctx, ts("2026-10-03T07:00:00.000Z")); + assert_eq!(server.requests_to(CHAT).len(), 2); + let a = jobs::session_id("mail", ts("2026-10-03T06:59:00.000Z")).unwrap(); + let b = jobs::session_id("mail", ts("2026-10-03T07:00:00.000Z")).unwrap(); + assert_ne!(a, b); + assert!( + home.dir + .join(format!("sessions/{}/0.jsonl", a.as_str())) + .is_file() + ); + assert!( + home.dir + .join(format!("sessions/{}/0.jsonl", b.as_str())) + .is_file() + ); +} diff --git a/docs/plans/M5d2/files/crates/loopd/tests/scheduler_heartbeat.rs b/docs/plans/M5d2/files/crates/loopd/tests/scheduler_heartbeat.rs new file mode 100644 index 0000000..a6b1b46 --- /dev/null +++ b/docs/plans/M5d2/files/crates/loopd/tests/scheduler_heartbeat.rs @@ -0,0 +1,148 @@ +//! Tests for the heartbeat tick. Do not edit. + +mod support; + +use bxcore::{SessionId, Timestamp}; +use loopd::channel::Context; +use loopd::llama::Client; +use loopd::memory::NoMemory; +use loopd::scheduler::{self, Scheduler, TickWork}; +use loopd::tools::Registry; +use std::sync::Arc; +use support::{FakeServer, Home, Reply, ScriptedPort}; + +const CHAT: &str = "/v1/chat/completions"; +const PROMPT: &str = "Look around.\n"; + +fn sse() -> Reply { + let data1 = r#"data: {"choices":[{"finish_reason":null,"index":0,"delta":{"content":"noted"}}],"id":"h","timings":{"cache_n":0,"prompt_n":10,"predicted_n":1}}"#; + let data2 = r#"data: {"choices":[{"finish_reason":"stop","index":0,"delta":{}}],"id":"h","timings":{"cache_n":0,"prompt_n":10,"predicted_n":1}}"#; + let body = format!("{data1}\n\n{data2}\n\ndata: [DONE]\n\n"); + Reply::raw(format!( + "HTTP/1.1 200 OK\r\nContent-Type: text/event-stream\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}", + body.len() + )) +} + +fn ts(s: &str) -> Timestamp { + Timestamp::parse(s).unwrap() +} + +fn world(every_s: u64) -> (Home, FakeServer, Arc) { + let home = Home::new(); + home.write("heartbeat.md", PROMPT); + home.write("compact.md", "Summarise.\n"); + let server = FakeServer::start(); + let mut cfg = home.config(&server.socket); + cfg.heartbeat.every_s = every_s; + cfg.heartbeat.prompt = home.dir.join("heartbeat.md"); + server.route(CHAT, vec![sse()]); + let ctx = Arc::new(Context::new( + cfg.clone(), + Client::new(cfg), + Box::new(ScriptedPort::new(vec![])), + Registry::m2b(), + Box::new(NoMemory), + )); + (home, server, ctx) +} + +#[test] +fn every_s_zero_does_nothing() { + let (_home, server, ctx) = world(0); + let mut sched = Scheduler::new(ts("2026-10-03T07:00:00.000Z")); + let work = sched.tick(&ctx, ts("2026-10-03T08:00:00.000Z")); + assert!(work.is_empty()); + assert!(server.requests_to(CHAT).is_empty()); +} + +#[test] +fn a_due_heartbeat_runs_on_the_background_slot_with_a_time_line() { + let (home, server, ctx) = world(60); + let mut sched = Scheduler::new(ts("2026-10-03T07:00:00.000Z")); + let work = sched.tick(&ctx, ts("2026-10-03T07:01:00.000Z")); + assert_eq!(work, vec![TickWork::Heartbeat]); + let sent = server.requests_to(CHAT); + assert_eq!(sent.len(), 1); + assert_eq!(sent[0].json()["id_slot"], 1); + let last = sent[0].json()["messages"] + .as_array() + .unwrap() + .last() + .unwrap(); + assert_eq!(last["role"], "user"); + let content = last["content"].as_str().unwrap(); + assert!( + content.starts_with("It is 2026-10-03T07:01:00.000Z."), + "{content}" + ); + assert!(content.contains("Look around."), "{content}"); + let s = loopd::session::Session::open(&home.dir, SessionId::new("heartbeat").unwrap()).unwrap(); + assert_eq!(s.slot(), 1); +} + +#[test] +fn a_second_tick_before_every_s_does_not_run() { + let (_home, server, ctx) = world(60); + let mut sched = Scheduler::new(ts("2026-10-03T07:00:00.000Z")); + sched.tick(&ctx, ts("2026-10-03T07:01:00.000Z")); + let work = sched.tick(&ctx, ts("2026-10-03T07:01:30.000Z")); + assert!(work.is_empty()); + assert_eq!(server.requests_to(CHAT).len(), 1); +} + +#[test] +fn a_busy_heartbeat_session_is_skipped() { + let (_home, server, ctx) = world(60); + let held = ctx + .hold(SessionId::new("heartbeat").unwrap()) + .expect("hold"); + let mut sched = Scheduler::new(ts("2026-10-03T07:00:00.000Z")); + let work = sched.tick(&ctx, ts("2026-10-03T07:01:00.000Z")); + assert_eq!( + work, + vec![TickWork::SkippedBusy { + session: SessionId::new("heartbeat").unwrap() + }] + ); + assert!(server.requests_to(CHAT).is_empty()); + drop(held); +} + +#[test] +fn an_unreadable_prompt_is_skipped_and_names_the_runbook() { + let home = Home::new(); + let path = home.dir.join("heartbeat.md"); + std::fs::create_dir(&path).unwrap(); + let server = FakeServer::start(); + let mut cfg = home.config(&server.socket); + cfg.heartbeat.every_s = 60; + cfg.heartbeat.prompt = path; + let ctx = Arc::new(Context::new( + cfg.clone(), + Client::new(cfg), + Box::new(ScriptedPort::new(vec![])), + Registry::m2b(), + Box::new(NoMemory), + )); + let mut sched = Scheduler::new(ts("2026-10-03T07:00:00.000Z")); + let work = sched.tick(&ctx, ts("2026-10-03T07:01:00.000Z")); + assert_eq!(work, vec![TickWork::HeartbeatUnreadable]); + assert!(server.requests_to(CHAT).is_empty()); +} + +#[test] +fn heartbeat_has_no_speaker() { + let (home, _server, ctx) = world(60); + let mut sched = Scheduler::new(ts("2026-10-03T07:00:00.000Z")); + sched.tick(&ctx, ts("2026-10-03T07:01:00.000Z")); + let recs = home.records("heartbeat"); + let user = recs + .iter() + .find_map(|r| match r { + bxcore::LogRecord::User { speaker, .. } => Some(speaker), + _ => None, + }) + .unwrap(); + assert_eq!(user, &None); +} diff --git a/docs/plans/M5d2/files/crates/loopd/tests/session_slot.rs b/docs/plans/M5d2/files/crates/loopd/tests/session_slot.rs new file mode 100644 index 0000000..b4db135 --- /dev/null +++ b/docs/plans/M5d2/files/crates/loopd/tests/session_slot.rs @@ -0,0 +1,170 @@ +//! Tests that a live turn uses the session's slot. Do not edit. + +mod support; + +use bxcore::SessionId; +use loopd::baseline::Baseline; +use loopd::llama::Client; +use loopd::session::Session; +use loopd::tools::Registry; +use loopd::turn::{Runtime, run_turn}; +use support::{FakeServer, Home, Reply}; + +const CHAT: &str = "/v1/chat/completions"; + +fn sse() -> Reply { + let data1 = r#"data: {"choices":[{"finish_reason":null,"index":0,"delta":{"content":"ok"}}],"id":"t","timings":{"cache_n":0,"prompt_n":10,"predicted_n":1}}"#; + let data2 = r#"data: {"choices":[{"finish_reason":"stop","index":0,"delta":{}}],"id":"t","timings":{"cache_n":0,"prompt_n":10,"predicted_n":1}}"#; + let body = format!("{data1}\n\n{data2}\n\ndata: [DONE]\n\n"); + Reply::raw(format!( + "HTTP/1.1 200 OK\r\nContent-Type: text/event-stream\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}", + body.len() + )) +} + +fn runtime<'a>( + cfg: &'a loopd::config::Config, + client: &'a Client, + port: &'a support::ScriptedPort, + registry: &'a Registry, +) -> Runtime<'a> { + Runtime { + cfg, + client, + port, + registry, + } +} + +#[test] +fn a_session_created_on_the_background_slot_sends_that_slot() { + let home = Home::new(); + let server = FakeServer::start(); + let cfg = home.config(&server.socket); + server.route(CHAT, vec![sse()]); + let client = Client::new(cfg.clone()); + let port = support::ScriptedPort::new(vec![]); + let registry = Registry::m2b(); + let baseline = Baseline::assemble(&cfg, ®istry).unwrap(); + let mut s = Session::create( + &home.dir, + SessionId::new("heartbeat").unwrap(), + baseline, + cfg.slots.background, + ) + .unwrap(); + let mut events = Vec::new(); + run_turn( + &mut s, + &runtime(&cfg, &client, &port, ®istry), + "pulse", + None, + &mut |e| events.push(e.clone()), + ) + .unwrap(); + let sent = server.requests_to(CHAT); + assert_eq!(sent.len(), 1); + assert_eq!(sent[0].json()["id_slot"], cfg.slots.background); +} + +#[test] +fn a_main_slot_session_still_sends_main() { + let home = Home::new(); + let server = FakeServer::start(); + let cfg = home.config(&server.socket); + server.route(CHAT, vec![sse()]); + let client = Client::new(cfg.clone()); + let port = support::ScriptedPort::new(vec![]); + let registry = Registry::m2b(); + let baseline = Baseline::assemble(&cfg, ®istry).unwrap(); + let mut s = Session::create( + &home.dir, + SessionId::new("mm-thread").unwrap(), + baseline, + cfg.slots.main, + ) + .unwrap(); + let mut events = Vec::new(); + run_turn( + &mut s, + &runtime(&cfg, &client, &port, ®istry), + "hi", + None, + &mut |e| events.push(e.clone()), + ) + .unwrap(); + assert_eq!( + server.requests_to(CHAT)[0].json()["id_slot"], + cfg.slots.main + ); +} + +#[test] +fn the_request_slot_follows_session_start_not_the_config_main() { + let home = Home::new(); + let server = FakeServer::start(); + let mut cfg = home.config(&server.socket); + cfg.slots.main = 0; + cfg.slots.background = 1; + server.route(CHAT, vec![sse()]); + let client = Client::new(cfg.clone()); + let port = support::ScriptedPort::new(vec![]); + let registry = Registry::m2b(); + let baseline = Baseline::assemble(&cfg, ®istry).unwrap(); + let mut s = Session::create(&home.dir, SessionId::new("a").unwrap(), baseline, 7).unwrap(); + let mut events = Vec::new(); + run_turn( + &mut s, + &runtime(&cfg, &client, &port, ®istry), + "x", + None, + &mut |e| events.push(e.clone()), + ) + .unwrap(); + assert_eq!(server.requests_to(CHAT)[0].json()["id_slot"], 7); +} + +#[test] +fn a_heartbeat_turn_between_two_main_turns_does_not_change_the_main_prefix() { + let home = Home::new(); + let server = FakeServer::start(); + let cfg = home.config(&server.socket); + server.route(CHAT, vec![sse(), sse(), sse()]); + let client = Client::new(cfg.clone()); + let port = support::ScriptedPort::new(vec![]); + let registry = Registry::m2b(); + let mut main = Session::create( + &home.dir, + SessionId::new("mm-a").unwrap(), + Baseline::assemble(&cfg, ®istry).unwrap(), + cfg.slots.main, + ) + .unwrap(); + let mut beat = Session::create( + &home.dir, + SessionId::new("heartbeat").unwrap(), + Baseline::assemble(&cfg, ®istry).unwrap(), + cfg.slots.background, + ) + .unwrap(); + let rt = runtime(&cfg, &client, &port, ®istry); + let mut events = Vec::new(); + run_turn(&mut main, &rt, "one", None, &mut |e| events.push(e.clone())).unwrap(); + run_turn(&mut beat, &rt, "pulse", None, &mut |e| { + events.push(e.clone()) + }) + .unwrap(); + run_turn(&mut main, &rt, "two", None, &mut |e| events.push(e.clone())).unwrap(); + let sent = server.requests_to(CHAT); + assert_eq!(sent.len(), 3); + assert_eq!(sent[0].json()["id_slot"], 0); + assert_eq!(sent[1].json()["id_slot"], 1); + assert_eq!(sent[2].json()["id_slot"], 0); + let first = sent[0].json()["messages"].as_array().unwrap().clone(); + let third = sent[2].json()["messages"].as_array().unwrap(); + assert!( + third.len() > first.len(), + "the main session's second turn is a longer prefix" + ); + assert_eq!(&third[..first.len()], &first[..]); +} diff --git a/docs/plans/M5d2/files/crates/loopd/tests/support/turn.rs b/docs/plans/M5d2/files/crates/loopd/tests/support/turn.rs new file mode 100644 index 0000000..fa022c8 --- /dev/null +++ b/docs/plans/M5d2/files/crates/loopd/tests/support/turn.rs @@ -0,0 +1,98 @@ +//! A turn-loop setup shared by the `turn` and `limits` tests: a home, a fake server, a scripted +//! port and a registry. Included with `#[path]` because it needs `loopd::turn`, which the earlier +//! tasks' tests must not depend on. Do not edit. + +#![allow(dead_code)] + +use crate::support::{FakeServer, Home, ScriptedPort}; +use bxcore::{LogRecord, SessionId, ToolResponse, TurnEvent}; +use loopd::baseline::Baseline; +use loopd::llama::Client; +use loopd::session::Session; +use loopd::tools::Registry; +use loopd::turn::{Runtime, TurnError, run_turn}; + +pub struct Setup { + pub home: Home, + pub server: FakeServer, + pub cfg: loopd::config::Config, + pub client: Client, + pub port: ScriptedPort, + pub registry: Registry, +} + +pub fn setup(replies: Vec) -> Setup { + let home = Home::new(); + let server = FakeServer::start(); + let cfg = home.config(&server.socket); + let client = Client::new(cfg.clone()); + Setup { + home, + server, + cfg, + client, + port: ScriptedPort::new(replies), + registry: Registry::m2b(), + } +} + +impl Setup { + pub fn session(&self, id: &str) -> Session { + let baseline = Baseline::assemble(&self.cfg, &self.registry).unwrap(); + Session::create( + &self.home.dir, + SessionId::new(id).unwrap(), + baseline, + self.cfg.slots.main, + ) + .unwrap() + } + + pub fn runtime(&self) -> Runtime<'_> { + Runtime { + cfg: &self.cfg, + client: &self.client, + port: &self.port, + registry: &self.registry, + } + } + + pub fn turn( + &self, + session: &mut Session, + text: &str, + ) -> (Result, Vec) { + let mut events = Vec::new(); + let result = run_turn(session, &self.runtime(), text, None, &mut |e| { + events.push(e.clone()) + }); + (result, events) + } +} + +/// The recordings were made in separate conversations, so their timings do not line up and the +/// loop rightly reports cache losses between them. Most tests are not about that, so `types` +/// and `records` leave `CacheLoss` out; one test checks it on purpose. +pub fn without_cache_loss(records: &[LogRecord]) -> Vec { + records + .iter() + .filter(|r| !matches!(r, LogRecord::CacheLoss { .. })) + .cloned() + .collect() +} + +pub fn types(records: &[LogRecord]) -> Vec<&'static str> { + without_cache_loss(records) + .iter() + .map(|r| match r { + LogRecord::SessionStart { .. } => "start", + LogRecord::User { .. } => "user", + LogRecord::Assistant { .. } => "assistant", + LogRecord::Usage { .. } => "usage", + LogRecord::ToolResult { .. } => "tool_result", + LogRecord::ToolSummary { .. } => "tool_summary", + LogRecord::CacheLoss { .. } => "cache_loss", + LogRecord::EpochEnd { .. } => "epoch_end", + }) + .collect() +} diff --git a/docs/plans/M5d2/files/crates/loopd/tests/tool_summary_call.rs b/docs/plans/M5d2/files/crates/loopd/tests/tool_summary_call.rs new file mode 100644 index 0000000..414b7a2 --- /dev/null +++ b/docs/plans/M5d2/files/crates/loopd/tests/tool_summary_call.rs @@ -0,0 +1,241 @@ +//! Tests for the opted-in tool-output summary. Do not edit. + +mod support; + +use bxcore::{CallId, DataClass, LogRecord, SessionId, ToolCall}; +use loopd::baseline::Baseline; +use loopd::compact; +use loopd::llama::Client; +use loopd::memory::NoMemory; +use loopd::session::Session; +use loopd::summarise; +use loopd::tools::Registry; +use loopd::turn::{Runtime, run_turn}; +use support::{FakeServer, Home, Reply, ScriptedPort}; + +const CHAT: &str = "/v1/chat/completions"; +const SUMMARISE: &str = "Summarise this tool result.\n"; + +fn sse(content: &str) -> Reply { + let content_json = serde_json::to_string(content).unwrap(); + let data1 = format!( + r#"data: {{"choices":[{{"finish_reason":null,"index":0,"delta":{{"content":{content_json}}}}}],"id":"s","timings":{{"cache_n":0,"prompt_n":10,"predicted_n":1}}}}"# + ); + let data2 = r#"data: {"choices":[{"finish_reason":"stop","index":0,"delta":{}}],"id":"s","timings":{"cache_n":0,"prompt_n":10,"predicted_n":1}}"#; + let body = format!("{data1}\n\n{data2}\n\ndata: [DONE]\n\n"); + Reply::raw(format!( + "HTTP/1.1 200 OK\r\nContent-Type: text/event-stream\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}", + body.len() + )) +} + +fn ts() -> bxcore::Timestamp { + bxcore::Timestamp::parse("2026-09-18T08:00:00.000Z").unwrap() +} + +fn assistant(name: &str) -> LogRecord { + LogRecord::Assistant { + time: ts(), + content: None, + reasoning_content: None, + tool_calls: vec![ToolCall { + id: "call_f".to_string(), + name: name.to_string(), + arguments: "{}".to_string(), + }], + } +} + +fn result(untrusted: bool) -> LogRecord { + LogRecord::ToolResult { + time: ts(), + call: CallId(1), + tool_call_id: "call_f".to_string(), + content: "the fetched page is about tea".to_string(), + class: DataClass::Public, + untrusted, + truncated: false, + } +} + +fn session(home: &Home, cfg: &loopd::config::Config) -> Session { + let registry = Registry::m2b(); + Session::create( + &home.dir, + SessionId::new("a").unwrap(), + Baseline::assemble(cfg, ®istry).unwrap(), + cfg.slots.main, + ) + .unwrap() +} + +#[test] +fn an_opted_in_result_gets_a_summary_on_the_background_slot() { + let home = Home::new(); + home.write("summarise.md", SUMMARISE); + let server = FakeServer::start(); + let mut cfg = home.config(&server.socket); + cfg.compact.summarise = vec!["http_fetch".to_string()]; + server.route(CHAT, vec![sse("tea page")]); + let client = Client::new(cfg.clone()); + let mut s = session(&home, &cfg); + s.append(assistant("http_fetch")).unwrap(); + s.append(result(false)).unwrap(); + summarise::missing(&cfg, &client, &mut s).unwrap(); + let sent = server.requests_to(CHAT); + assert_eq!(sent.len(), 1); + assert_eq!(sent[0].json()["id_slot"], 1); + assert!(sent[0].json().get("tools").is_none()); + assert!( + s.records().iter().any(|r| matches!(r, LogRecord::ToolSummary { content, call, untrusted, .. } if content == "tea page" && *call == CallId(1) && !*untrusted)), + "{:?}", + s.records() + ); +} + +#[test] +fn the_summary_copies_the_result_labels() { + let home = Home::new(); + home.write("summarise.md", SUMMARISE); + let server = FakeServer::start(); + let mut cfg = home.config(&server.socket); + cfg.compact.summarise = vec!["http_fetch".to_string()]; + server.route(CHAT, vec![sse("x")]); + let client = Client::new(cfg.clone()); + let mut s = session(&home, &cfg); + s.append(assistant("http_fetch")).unwrap(); + s.append(result(true)).unwrap(); + summarise::missing(&cfg, &client, &mut s).unwrap(); + match s.records().last() { + Some(LogRecord::ToolSummary { + class, + untrusted, + tool_call_id, + .. + }) => { + assert_eq!(*class, DataClass::Public); + assert!(*untrusted); + assert_eq!(tool_call_id, "call_f"); + } + other => panic!("{other:?}"), + } +} + +#[test] +fn a_tool_not_in_summarise_is_left_alone() { + let home = Home::new(); + home.write("summarise.md", SUMMARISE); + let server = FakeServer::start(); + let mut cfg = home.config(&server.socket); + cfg.compact.summarise = vec!["http_fetch".to_string()]; + let client = Client::new(cfg.clone()); + let mut s = session(&home, &cfg); + s.append(assistant("clock")).unwrap(); + s.append(result(false)).unwrap(); + summarise::missing(&cfg, &client, &mut s).unwrap(); + assert!(server.requests_to(CHAT).is_empty()); + assert!( + !s.records() + .iter() + .any(|r| matches!(r, LogRecord::ToolSummary { .. })) + ); +} + +#[test] +fn a_result_that_already_has_a_summary_is_not_summarised_again() { + let home = Home::new(); + home.write("summarise.md", SUMMARISE); + let server = FakeServer::start(); + let mut cfg = home.config(&server.socket); + cfg.compact.summarise = vec!["http_fetch".to_string()]; + server.route(CHAT, vec![sse("once")]); + let client = Client::new(cfg.clone()); + let mut s = session(&home, &cfg); + s.append(assistant("http_fetch")).unwrap(); + s.append(result(false)).unwrap(); + summarise::missing(&cfg, &client, &mut s).unwrap(); + summarise::missing(&cfg, &client, &mut s).unwrap(); + assert_eq!(server.requests_to(CHAT).len(), 1); + assert_eq!( + s.records() + .iter() + .filter(|r| matches!(r, LogRecord::ToolSummary { .. })) + .count(), + 1 + ); +} + +#[test] +fn messages_stay_the_same_after_a_summary() { + let home = Home::new(); + home.write("summarise.md", SUMMARISE); + let server = FakeServer::start(); + let mut cfg = home.config(&server.socket); + cfg.compact.summarise = vec!["http_fetch".to_string()]; + server.route(CHAT, vec![sse("tea page")]); + let client = Client::new(cfg.clone()); + let mut s = session(&home, &cfg); + s.append(assistant("http_fetch")).unwrap(); + s.append(result(false)).unwrap(); + let before = loopd::baseline::messages(s.baseline(), s.records()); + summarise::missing(&cfg, &client, &mut s).unwrap(); + let after = loopd::baseline::messages(s.baseline(), s.records()); + assert_eq!(before, after); +} + +#[test] +fn after_a_turn_an_opted_in_call_is_summarised() { + let home = Home::new(); + home.write("summarise.md", SUMMARISE); + home.write("compact.md", "Summarise.\n"); + let server = FakeServer::start(); + let mut cfg = home.config(&server.socket); + cfg.compact.summarise = vec!["read_file".to_string()]; + server.route( + CHAT, + vec![ + Reply::fixture("tool_call"), + Reply::fixture("plain"), + sse("hosts file"), + ], + ); + let client = Client::new(cfg.clone()); + let port = ScriptedPort::new(vec![support::ok_result("127.0.0.1 localhost\n")]); + let registry = Registry::m2b(); + let mut s = Session::create( + &home.dir, + SessionId::new("a").unwrap(), + Baseline::assemble(&cfg, ®istry).unwrap(), + cfg.slots.main, + ) + .unwrap(); + let rt = Runtime { + cfg: &cfg, + client: &client, + port: &port, + registry: ®istry, + }; + let mut events = Vec::new(); + let outcome = run_turn(&mut s, &rt, "hosts?", None, &mut |e| events.push(e.clone())); + compact::finish_turn( + &mut s, + &rt, + &NoMemory, + "hosts?", + None, + &mut |e| events.push(e.clone()), + outcome, + ) + .unwrap(); + assert!( + s.records() + .iter() + .any(|r| matches!(r, LogRecord::ToolSummary { .. })), + "{:?}", + s.records() + ); + assert_eq!( + server.requests_to(CHAT).last().unwrap().json()["id_slot"], + 1 + ); +} diff --git a/docs/plans/M5d2/files/crates/loopd/tests/tool_summary_messages.rs b/docs/plans/M5d2/files/crates/loopd/tests/tool_summary_messages.rs new file mode 100644 index 0000000..5ffc1aa --- /dev/null +++ b/docs/plans/M5d2/files/crates/loopd/tests/tool_summary_messages.rs @@ -0,0 +1,88 @@ +//! Tests that `messages()` ignores `ToolSummary`. Do not edit. + +use bxcore::{CallId, DataClass, Epoch, Hash32, LogRecord, SessionId, Timestamp}; +use loopd::baseline::{Baseline, messages}; +use loopd::llama::ChatMessage; +use loopd::llama::ToolSchema; + +fn ts() -> Timestamp { + Timestamp::parse("2026-09-18T08:00:00.000Z").unwrap() +} + +fn baseline() -> Baseline { + Baseline { + system: "sys".to_string(), + tools: vec![ToolSchema { + name: "clock".to_string(), + description: "t".to_string(), + parameters: serde_json::json!({"type": "object", "properties": {}}), + }], + } +} + +fn user(content: &str) -> LogRecord { + LogRecord::User { + time: ts(), + content: content.to_string(), + speaker: None, + } +} + +fn result() -> LogRecord { + LogRecord::ToolResult { + time: ts(), + call: CallId(1), + tool_call_id: "c".to_string(), + content: "raw body".to_string(), + class: DataClass::Public, + untrusted: false, + truncated: false, + } +} + +fn summary() -> LogRecord { + LogRecord::ToolSummary { + time: ts(), + call: CallId(1), + tool_call_id: "c".to_string(), + content: "a short account".to_string(), + class: DataClass::Public, + untrusted: false, + } +} + +#[test] +fn messages_ignore_tool_summary() { + let start = LogRecord::SessionStart { + time: ts(), + session: SessionId::new("a").unwrap(), + epoch: Epoch(0), + slot: 0, + baseline: Hash32::ZERO, + }; + let without = messages(&baseline(), &[start.clone(), user("hi"), result()]); + let with = messages(&baseline(), &[start, user("hi"), result(), summary()]); + assert_eq!(without, with); + assert!( + !with + .iter() + .any(|m| matches!(m, ChatMessage::User { content } if content == "a short account")), + "{with:?}" + ); +} + +#[test] +fn a_summary_is_not_replayed_as_a_tool_message() { + let msgs = messages(&baseline(), &[summary()]); + assert_eq!(msgs.len(), 1, "only the system message"); + assert!(matches!(msgs[0], ChatMessage::System { .. })); +} + +#[test] +fn inserting_a_summary_does_not_break_prefix_extension() { + let a = vec![user("hi"), result()]; + let b = vec![user("hi"), result(), summary(), user("next")]; + let ma = messages(&baseline(), &a); + let mb = messages(&baseline(), &b); + assert_eq!(&mb[..ma.len()], &ma[..]); +} diff --git a/docs/plans/M5d2/files/crates/memoryd/tests/tiers.rs b/docs/plans/M5d2/files/crates/memoryd/tests/tiers.rs new file mode 100644 index 0000000..5427984 --- /dev/null +++ b/docs/plans/M5d2/files/crates/memoryd/tests/tiers.rs @@ -0,0 +1,122 @@ +//! Tests for `MemoryCommand::Tiers`: identity entries and notes with recall-hit counts. Do not edit. + +#[path = "support/tmp.rs"] +mod tmp; + +use bxcore::{DataClass, MemoryFile, Timestamp}; +use memoryd::files::Store; +use memoryd::tiers; +use tmp::TempDir; + +fn now() -> Timestamp { + Timestamp::parse("2026-10-03T08:00:00.000Z").unwrap() +} + +struct Setup { + dir: TempDir, + store: Store, +} + +fn setup(tag: &str) -> Setup { + let dir = TempDir::new(tag); + let starter = dir.write("etc/soul.md", "Soul.\n"); + let store = Store::new(&dir.path().join("memory")); + store.init(&starter).unwrap(); + Setup { dir, store } +} + +fn write_recall(store: &Store, line: &str) { + std::fs::create_dir_all(store.path("recalls")).unwrap(); + store + .append("recalls/2026-10.jsonl", &format!("{line}\n")) + .unwrap(); +} + +#[test] +fn identity_entries_list_with_zero_hits() { + let s = setup("tiers-id"); + s.store + .write_entries(MemoryFile::Memory, &["I like tea.".to_string()]) + .unwrap(); + s.store + .write_entries(MemoryFile::User, &["The owner is Kyle.".to_string()]) + .unwrap(); + let text = tiers::list(&s.store, now()).unwrap(); + assert!( + text.contains("identity memory.md hits 0 last -"), + "{text}" + ); + assert!(text.contains("I like tea."), "{text}"); + assert!(text.contains("identity user.md hits 0 last -"), "{text}"); + assert!(text.contains("The owner is Kyle."), "{text}"); +} + +#[test] +fn notes_carry_counts_from_recalls() { + let s = setup("tiers-notes"); + std::fs::write( + s.store.path("notes.md"), + "§ noted 2026-09-22T18:20Z in session mm-a · private\nlaunch window moved.\n", + ) + .unwrap(); + write_recall( + &s.store, + r#"{"time":"2026-10-03T07:00:00.000Z","session":"mm-now","query":"launch","hits":[{"kind":"noted","session":"mm-a","epoch":0,"record":0}]}"#, + ); + write_recall( + &s.store, + r#"{"time":"2026-10-03T07:30:00.000Z","session":"mm-now","query":"window","hits":[{"kind":"noted","session":"mm-a","epoch":0,"record":0}]}"#, + ); + let text = tiers::list(&s.store, now()).unwrap(); + assert!( + text.contains( + "note session mm-a hits 2 last 2026-10-03T07:30:00.000Z private trusted" + ), + "{text}" + ); + assert!(text.contains("launch window moved."), "{text}"); +} + +#[test] +fn untrusted_notes_are_listed_and_marked() { + let s = setup("tiers-untrusted"); + std::fs::write( + s.store.path("notes.md"), + "§ noted 2026-09-22T18:20Z in session mm-c · private · untrusted\nexample.org says the launch moved.\n", + ) + .unwrap(); + let text = tiers::list(&s.store, now()).unwrap(); + assert!( + text.contains("note session mm-c hits 0 last - private untrusted"), + "{text}" + ); + assert!( + text.contains("example.org says the launch moved."), + "{text}" + ); +} + +#[test] +fn a_recall_of_a_different_note_does_not_count() { + let s = setup("tiers-other"); + std::fs::write( + s.store.path("notes.md"), + "§ noted 2026-09-22T18:20Z in session mm-a · private\nfirst note.\n§ noted 2026-09-22T18:21Z in session mm-b · private\nsecond note.\n", + ) + .unwrap(); + write_recall( + &s.store, + r#"{"time":"2026-10-03T07:00:00.000Z","session":"mm-now","query":"second","hits":[{"kind":"noted","session":"mm-b","epoch":0,"record":1}]}"#, + ); + let text = tiers::list(&s.store, now()).unwrap(); + assert!( + text.contains("note session mm-a hits 0 last - private trusted"), + "{text}" + ); + assert!( + text.contains( + "note session mm-b hits 1 last 2026-10-03T07:00:00.000Z private trusted" + ), + "{text}" + ); +} diff --git a/docs/plans/M5d2/files/crates/memoryd/tests/tool_summary_index.rs b/docs/plans/M5d2/files/crates/memoryd/tests/tool_summary_index.rs new file mode 100644 index 0000000..2caa109 --- /dev/null +++ b/docs/plans/M5d2/files/crates/memoryd/tests/tool_summary_index.rs @@ -0,0 +1,191 @@ +//! Tests that `ToolSummary` is indexed as kind `summary` and found by `recall`. Do not edit. + +#[path = "support/logs.rs"] +mod logs; +#[path = "support/tmp.rs"] +mod tmp; + +use bxcore::{CallId, DataClass, LogRecord, RecallArgs, Timestamp}; +use logs::{assistant, result, user, write_log}; +use memoryd::files::Store; +use memoryd::index::Index; +use memoryd::recall::recall; +use tmp::TempDir; + +fn now() -> Timestamp { + Timestamp::parse("2026-09-24T21:00:00.000Z").unwrap() +} + +fn summary(minute: u32, content: &str, untrusted: bool) -> LogRecord { + LogRecord::ToolSummary { + time: logs::at(minute), + call: CallId(1), + tool_call_id: "c0".to_string(), + content: content.to_string(), + class: DataClass::Public, + untrusted, + } +} + +struct Setup { + dir: TempDir, + store: Store, + index: Index, +} + +fn setup(tag: &str) -> Setup { + let dir = TempDir::new(tag); + let starter = dir.write("etc/soul.md", "Soul.\n"); + let store = Store::new(&dir.path().join("memory")); + store.init(&starter).unwrap(); + let index = Index::open(&dir.path().join("index/memory.sqlite3"), DataClass::Private).unwrap(); + Setup { dir, store, index } +} + +fn ask(s: &Setup, query: &str) -> memoryd::recall::Recalled { + let args = RecallArgs { + query: query.to_string(), + k: 10, + }; + recall(&s.index, &s.store, &args, "mm-now", now()).unwrap() +} + +#[test] +fn a_tool_summary_is_found_as_kind_summary() { + let s = setup("summary-found"); + write_log( + s.dir.path(), + "mm-a", + &[ + user(2, "fetch the launch page"), + assistant( + 3, + None, + &[("http_fetch", r#"{"url":"https://example.org"}"#)], + ), + result(4, "raw html body about tea", DataClass::Public, false), + summary(5, "the page is about tea and the launch window", false), + ], + ); + s.index + .catch_up(&s.dir.path().join("sessions"), &s.store.path("notes.md")) + .unwrap(); + let got = ask(&s, "tea"); + assert!( + got.text + .contains("the page is about tea and the launch window"), + "{got:?}" + ); + assert!( + got.text.contains("memory · session mm-a"), + "tagged as a session hit: {}", + got.text + ); + assert!(!got.untrusted); +} + +#[test] +fn the_raw_tool_result_is_still_not_indexed() { + let s = setup("summary-no-raw"); + write_log( + s.dir.path(), + "mm-a", + &[ + user(2, "fetch"), + assistant(3, None, &[("http_fetch", "{}")]), + result(4, "UNIQUE_RAW_BODY_XYZ", DataClass::Public, false), + summary(5, "a short account of the page", false), + ], + ); + s.index + .catch_up(&s.dir.path().join("sessions"), &s.store.path("notes.md")) + .unwrap(); + let got = ask(&s, "UNIQUE_RAW_BODY_XYZ"); + assert!( + got.text.contains("(no matches"), + "raw tool result stays unindexed: {}", + got.text + ); +} + +#[test] +fn an_untrusted_summary_taints_the_recall() { + let s = setup("summary-taint"); + write_log( + s.dir.path(), + "mm-a", + &[ + user(2, "fetch"), + assistant(3, None, &[("http_fetch", "{}")]), + result(4, "page", DataClass::Public, true), + summary(5, "the fetched page mentions the launch window", true), + ], + ); + s.index + .catch_up(&s.dir.path().join("sessions"), &s.store.path("notes.md")) + .unwrap(); + let got = ask(&s, "launch"); + assert!(got.untrusted, "{got:?}"); + assert!(got.text.contains("memory-tainted"), "{}", got.text); +} + +#[test] +fn a_trusted_summary_does_not_taint() { + let s = setup("summary-trusted"); + write_log( + s.dir.path(), + "mm-a", + &[ + user(2, "fetch"), + assistant(3, None, &[("http_fetch", "{}")]), + result(4, "page", DataClass::Public, false), + summary(5, "the fetched page mentions the launch window", false), + ], + ); + s.index + .catch_up(&s.dir.path().join("sessions"), &s.store.path("notes.md")) + .unwrap(); + let got = ask(&s, "launch"); + assert!(!got.untrusted, "{got:?}"); + assert!(got.text.contains("memory · session mm-a"), "{}", got.text); + assert!(!got.text.contains("memory-tainted"), "{}", got.text); +} + +#[test] +fn a_secret_summary_is_skipped_when_the_index_is_private() { + let s = setup("summary-secret"); + write_log( + s.dir.path(), + "mm-a", + &[ + user(2, "fetch"), + assistant(3, None, &[("http_fetch", "{}")]), + LogRecord::ToolResult { + time: logs::at(4), + call: CallId(1), + tool_call_id: "c0".to_string(), + content: "secret page", + class: DataClass::Secret, + untrusted: false, + truncated: false, + }, + LogRecord::ToolSummary { + time: logs::at(5), + call: CallId(1), + tool_call_id: "c0".to_string(), + content: "SECRET_SUMMARY_PHRASE", + class: DataClass::Secret, + untrusted: false, + }, + ], + ); + s.index + .catch_up(&s.dir.path().join("sessions"), &s.store.path("notes.md")) + .unwrap(); + let got = ask(&s, "SECRET_SUMMARY_PHRASE"); + assert!( + got.text.contains("(no matches"), + "secret summary above max_class: {}", + got.text + ); +} diff --git a/docs/plans/M5d2/files/runbook-heartbeat-prompt-unreadable.md b/docs/plans/M5d2/files/runbook-heartbeat-prompt-unreadable.md new file mode 100644 index 0000000..37c35c9 --- /dev/null +++ b/docs/plans/M5d2/files/runbook-heartbeat-prompt-unreadable.md @@ -0,0 +1,25 @@ +## heartbeat-prompt-unreadable + +**What you see.** `loopd` is serving, but heartbeat turns do not run. It logs this line once, +until the file reads: + +```text +loopd: heartbeat.md: Permission denied (os error 13); see docs/runbook.md#heartbeat-prompt-unreadable +``` + +**Why.** `[heartbeat] prompt` is next to the config file (default `heartbeat.md`). The scheduler +skips the tick rather than send an empty or guessed user message. A missing file is the same: +restore the starter from `config/heartbeat.md` in the repository. + +**Confirm.** + +```sh +ls -l "$(dirname "$LOOPD_CONFIG")/heartbeat.md" +``` + +**Fix.** Restore mode `0644` (or whatever the owner uses for the other prompts) and content. +The next tick after a successful read logs nothing extra and runs the standing `heartbeat` +session. + +**Check.** `sessions/heartbeat/0.jsonl` gains a new `User` whose text starts with `It is ` and an +RFC 3339 timestamp. diff --git a/docs/plans/M5d2/files/runbook-jobs-invalid.md b/docs/plans/M5d2/files/runbook-jobs-invalid.md new file mode 100644 index 0000000..54b43d6 --- /dev/null +++ b/docs/plans/M5d2/files/runbook-jobs-invalid.md @@ -0,0 +1,24 @@ +## jobs-invalid + +**What you see.** `loopd` does not start. It prints a line naming `{home}/jobs/.toml` and +this entry: + +```text +loopd: jobs/mail.toml: unknown field `when`, expected `schedule` or `prompt`; see docs/runbook.md#jobs-invalid +``` + +**Why.** Job files are owner-written. `loopd` reads them and cannot write them. Unknown keys, a +name that is not `[a-z0-9-]+`, a name so long that `job--t` is not a valid +session id (64 bytes, `[a-z0-9-]`), a schedule that is not five UTC fields, or a missing +`prompt`, all refuse start so a misspelt job cannot sit silently. + +**Confirm.** + +```sh +ls -l "$BOXMAKER_HOME/jobs/" +``` + +**Fix.** Rename or edit the named file so it matches the example in `config/jobs/tier.toml.example`. +Then start `loopd` again. + +**Check.** `loopd serve` prints `serving on` and `{home}/jobs/runs.jsonl` exists (empty is fine).