docs: M5d-1 and M5d-2 implementation plans for Ornith

Eight closed tasks for epochs and compaction, then ten for the scheduler,
ToolSummary, and tiers. Given tests first. Gate 1456 → 1507 → 1572.

Co-Authored-By: Grok 4.6 <noreply@x.ai>
This commit is contained in:
2026-10-02 21:54:37 -07:00
co-authored by Grok 4.6
parent 069cd1686a
commit 3726ae585f
48 changed files with 7319 additions and 0 deletions
+209
View File
@@ -0,0 +1,209 @@
# M5d-1 task 01: `[compact]` on `loopd.toml`
**Branch:** `m5d1`. Check it: `git branch --show-current` prints `m5d1` and `git status --short`
is empty; otherwise stop. Never switch branch.
**Commit subject:** `loopd: [compact] threshold, summary_chars and summarise`
**Log row:** the Task cell is `M5d1/01-loopd-compact-config`.
## Goal
M5d-1 compact after a turn whose last `Usage` is at or above a fraction of `expect.n_ctx` (spec
3.3, 5). This task adds only the `[compact]` table so later tasks can read it:
- `threshold` (float, default `0.85`)
- `summary_chars` (default `4000`)
- `summarise` (list of tool names, default `["http_fetch"]`)
Nothing in this task uses the values. `summarise` is stored so a spec-complete `loopd.toml`
loads; M5d-2 reads it. Unknown keys in `[compact]` are errors, as in every table we define.
`Config::compact_prompt` is `compact.md` beside `[baseline] system` (the same directory join
`load` already does for `system.md`). The starter file is copied in task 05.
## Every place that changes
This is the complete list.
| Place | What happens there |
|---|---|
| `crates/loopd/src/config.rs` | `Compact`; `Config.compact`; `Config::compact_prompt` (edits C1 to C3) |
| `crates/loopd/tests/config_compact.rs` | new, given |
| everything else | **no change**: not `session.rs`, not the existing `config.rs` tests or fixtures |
## Files
- Copy, from `docs/plans/M5d1/files/` (the same path under it): the new test
`crates/loopd/tests/config_compact.rs`.
- Modify: `crates/loopd/src/config.rs`, `docs/implementer-log.md`.
- Do not touch: any other file.
## Facts you need
- **`Config` already has `#[serde(default)]` on optional tables** (`sampling`, `limits`, `paths`,
`channel`, `loop`, `baseline`, `broker`, `memory`). `[compact]` is the same: missing means
`Compact::default()`.
- **`threshold` is `f64`.** `Sampling.temperature` is already `f64` on `Config`, so `PartialEq`
on `Config` still compiles. Do not add a range check: the spec does not name one.
- **`summarise` default.** `vec!["http_fetch".to_string()]`. An explicit `summarise = []` is
allowed and is empty. Unknown names are not checked here (M5d-2).
- **`compact_prompt`.** `self.baseline.system.with_file_name("compact.md")`. After `load`,
`baseline.system` is already joined to the config file's directory, so the prompt follows it
into a subdirectory when `system` does (`prompts/agent.md` → `prompts/compact.md`).
- The items the edits use. This table is complete; `serde` is already a dependency of `loopd`.
| Item | What it does |
|---|---|
| `#[serde(deny_unknown_fields, default)]` | on `Compact`: missing table uses `Default`; unknown keys error |
| `Path::with_file_name` | `compact.md` beside `system.md` |
| `Vec<String>` | `summarise` |
## The edits, exactly
Nothing else in this file changes. Each quoted "before" block occurs exactly once in its file
when its turn comes (make them in order). Keep the fields in the order shown.
### `crates/loopd/src/config.rs`
**C1.** After the `Memory` struct (the lines)
```rust
pub struct Memory {
pub socket: std::path::PathBuf,
}
```
become
```rust
pub struct Memory {
pub socket: std::path::PathBuf,
}
/// `[compact]`: when a session starts a new epoch, and how long a summary may be (M5d spec,
/// section 5). `summarise` is stored here and unused until M5d-2.
#[derive(Debug, Clone, PartialEq, Deserialize)]
#[serde(deny_unknown_fields, default)]
pub struct Compact {
pub threshold: f64,
pub summary_chars: usize,
pub summarise: Vec<String>,
}
impl Default for Compact {
fn default() -> Self {
Self {
threshold: 0.85,
summary_chars: 4_000,
summarise: vec!["http_fetch".to_string()],
}
}
}
```
**C2.** The lines
```rust
#[serde(default)]
pub memory: Memory,
}
```
become
```rust
#[serde(default)]
pub memory: Memory,
#[serde(default)]
pub compact: Compact,
}
```
**C3.** After `channel_socket` (the lines)
```rust
pub fn channel_socket(&self) -> std::path::PathBuf {
if self.channel.socket.as_os_str().is_empty() {
self.paths.home.join("run/loop/loop.sock")
} else {
self.channel.socket.clone()
}
}
}
```
become
```rust
pub fn channel_socket(&self) -> std::path::PathBuf {
if self.channel.socket.as_os_str().is_empty() {
self.paths.home.join("run/loop/loop.sock")
} else {
self.channel.socket.clone()
}
}
/// `compact.md` beside the system prompt: the summary call's last user message.
pub fn compact_prompt(&self) -> std::path::PathBuf {
self.baseline.system.with_file_name("compact.md")
}
}
```
## Steps
- [ ] **1. Copy the given test.** From the repository root, one command:
```sh
cp docs/plans/M5d1/files/crates/loopd/tests/config_compact.rs crates/loopd/tests/
```
Then `git status --short` lists one new file, `crates/loopd/tests/config_compact.rs`.
- [ ] **2. See it fail.** `cargo build --tests --keep-going -p loopd`, with a shell timeout of
600000 ms: one test target, `config_compact`, does not compile, with `E0609` (`no field
\`compact\` on type \`Config\``) and `E0599` (`no method named \`compact_prompt\`` /
`no variant or associated item named \`default\` for \`Compact\``). Every other target
compiles.
- [ ] **3. The edits.** Make edits C1 to C3, in order. `cargo check -p loopd` prints no error and
no warning.
- [ ] **4. The tests.** `cargo test -p loopd --test config_compact --test config`: every test
passes, 8 in `config_compact` and the others as before.
- [ ] **5. Check.** `git diff --stat` lists exactly `crates/loopd/src/config.rs`.
- [ ] **6. Run the gate.** `cargo fmt --all`, then `make gate`, with a shell timeout of 600000 ms.
Expected last line: `gate: ok`. The test count is **1464** after this task, 1456 before it
(1455 passed in `cargo test --workspace`, plus 1), and the 8 new ones are the tests of
`config_compact.rs`. To count, add up the `N passed` of every `test result:` line the gate's
`cargo test --workspace` printed, and add 1. Never run the gate again only to count.
- [ ] **7. Log and commit.** Add your row to `docs/implementer-log.md`; its Task cell is
`M5d1/01-loopd-compact-config`. Stage with
`git add crates/loopd docs/implementer-log.md Cargo.lock`, then commit as `AGENTS.md` says.
`Cargo.lock` does not change in this task; the `git add` is harmless.
## Done when
- `make gate` prints `gate: ok` with 1464 tests.
- `git diff HEAD~1 --stat` lists only `crates/loopd/src/config.rs`,
`crates/loopd/tests/config_compact.rs` and `docs/implementer-log.md`.
## Stop and report if
- The branch check at the top fails.
- A "before" block is not in its file exactly as shown when its turn comes, or is there more
than once.
- Step 2 shows a different result: another target failing, or other errors.
- `cargo check` in step 3 prints an error or a warning. Quote it, and stop.
- A test fails in step 4. Do not change the test: stop, and give the failing tests' names and
their messages.
- The fix seems to need a change anywhere else: a fixture, `config.rs` tests, a range check on
`threshold`. It does not; stop and say why you think so.
- A test outside `config_compact.rs` fails in the gate: check once whether it fails without your
change (`git stash -u`, run it, `git stash pop`), then stop and report both results.
- The gate passes with a count other than 1464.
- A comment in an edit does not fit the code under it. Quote it and stop.
## What the given tests pin
Each of these changes to the finished code makes at least one given test fail: the default
threshold moved; `summary_chars` default moved; `summarise` default emptied or given another
name; unknown keys accepted; a misspelt `summary_char` accepted; a partial table wiping the
other defaults; `compact_prompt` using `home` or the config file's name instead of `system.md`'s
directory; `Compact::default` disagreeing with a missing table.
+513
View File
@@ -0,0 +1,513 @@
# M5d-1 task 02: `Session::open` loads the current epoch
**Branch:** `m5d1`. Check it: `git branch --show-current` prints `m5d1` and `git status --short`
is empty; otherwise stop. Never switch branch.
**Commit subject:** `loopd: open the highest epoch and carry its summary`
**Log row:** the Task cell is `M5d1/02-loopd-session-epochs`.
## Goal
A session directory may have `0.jsonl` and, after compaction, `1.baseline.json` / `1.jsonl` and
so on (spec 3.1). `Session::open` loads the highest epoch that has **both** a baseline file and a
log. For epoch `n > 0` the previous log must end with `EpochEnd { next: n, summary }`; that
summary is the carry. `turns()` and `untrusted()` span every epoch. `next_call` is one more than
the highest `ToolResult` call in any epoch. `append` writes the current epoch's log.
`records()` stays the current epoch only: later tasks replay that file (plus the carry) as the
request.
## Every place that changes
This is the complete list.
| Place | What happens there |
|---|---|
| `crates/loopd/src/session.rs` | `NoCarry`; fields `epoch`, `slot`, `carry`, `prior_users`, `prior_untrusted`; `open`; `turns`; `untrusted`; `log_path`; accessors; helpers |
| `crates/loopd/tests/session_epochs.rs` | new, given |
| everything else | **no change**: not `session.rs`'s existing tests, not `turn.rs`, not Honcho |
## Files
- Copy, from `docs/plans/M5d1/files/` (the same path under it): the new test
`crates/loopd/tests/session_epochs.rs`.
- Modify: `crates/loopd/src/session.rs`, `docs/implementer-log.md`.
- Do not touch: any other file.
## Facts you need
- **`Epoch` is `pub struct Epoch(pub u32)`** in `bxcore`. Format a file name with
`format!("{}.jsonl", epoch.0)` and `format!("{}.baseline.json", epoch.0)`.
- **A complete epoch** is both `<n>.baseline.json` and `<n>.jsonl` in the session directory.
A baseline without a log, or a log without a baseline, does not count. The highest complete
`n` is the current epoch. If none is complete, `open` is `NotFound`, as today when `0.jsonl`
is missing.
- **Carry.** For current epoch `n > 0`, read `(n-1).jsonl`. Among its records, the last
`EpochEnd` whose `next` is `Epoch(n)` gives `summary`. If that file is missing, or has no such
record, `SessionError::NoCarry { path }` naming `(n-1).jsonl`. Epoch 0 has `carry: None`.
- **`records` on the live `Session`** is only the current epoch's log, so `messages` in task 03
can replay it. `prior_users` / `prior_untrusted` are counted from every **earlier** complete
epoch while `open` walks them. `turns()` is `prior_users` plus the `User` records in
`records`. `untrusted()` is `prior_untrusted` or any untrusted `ToolResult` in `records`.
- **`next_call`** is one more than the highest `ToolResult.call.0` in **every** complete epoch
(current included). A call id of 4 in epoch 0 and 2 in epoch 1 continues at 5.
- **`slot`** is the `slot` on the current epoch's `SessionStart`. `create` stores the `slot`
argument. If the current log has no `SessionStart`, use `0`.
- **`append` / `persist`** already write `self.log_path()`. Change `log_path` to the current
epoch's file; `create` still starts at `0.jsonl`.
- **A last `EpochEnd` on epoch 0 with no epoch-1 files** means compact stopped halfway: `open`
stays on epoch 0, `carry` is `None`, and the `EpochEnd` is in `records`. Task 07 resumes that.
- The torn-log checks stay as they are, per file, with the line number of that file.
- `session.rs` must stay at or under 500 lines (`wc -l`). After these edits it is about 430.
- The items the edits use. This table is complete.
| Item | What it does |
|---|---|
| `std::fs::read_dir` | list `<n>.jsonl` names |
| `str::strip_suffix(".jsonl")`, `str::parse::<u32>` | the epoch number |
| `Path::exists` | the matching baseline |
| `u64::try_from(count).unwrap_or(u64::MAX)` | `turns`, as today |
| `CallId(highest + 1)` | `next_call`, as today, over every epoch |
| `OpenOptions::new().append(true).open` | the current log |
## The edits, exactly
Nothing else in this file changes. Each quoted "before" block occurs exactly once. Keep struct
fields in the order shown.
### `crates/loopd/src/session.rs`
**S1.** The module doc comment, the lines
```rust
//! A session on disk: `sessions/<id>/0.baseline.json` and `0.jsonl`. The log is the state —
//! create, open, append and resume all live here, and nothing is durable until `append` syncs it.
```
become
```rust
//! A session on disk: `sessions/<id>/<epoch>.baseline.json` and `<epoch>.jsonl`. The log is the
//! state — create, open, append and resume all live here, and nothing is durable until `append`
//! syncs it. Epoch 0 is the first; later epochs appear after compaction.
```
**S2.** After `Encode(serde_json::Error),` in `SessionError`, add a variant (the lines)
```rust
Encode(serde_json::Error),
}
```
become
```rust
Encode(serde_json::Error),
NoCarry {
path: PathBuf,
},
}
```
**S3.** In `Display`, the lines
```rust
SessionError::Encode(err) => write!(f, "{err}"),
}
```
become
```rust
SessionError::Encode(err) => write!(f, "{err}"),
SessionError::NoCarry { path } => write!(
f,
"{}: compacted epoch is missing its EpochEnd; see docs/runbook.md#session-log-damaged",
path.display()
),
}
```
**S4.** The struct, the lines
```rust
pub struct Session {
id: SessionId,
dir: PathBuf,
baseline: Baseline,
records: Vec<LogRecord>,
log: File,
next_call: CallId,
}
```
become
```rust
pub struct Session {
id: SessionId,
dir: PathBuf,
baseline: Baseline,
records: Vec<LogRecord>,
log: File,
next_call: CallId,
epoch: Epoch,
slot: u32,
carry: Option<String>,
prior_users: u64,
prior_untrusted: bool,
}
```
**S5.** In `create`, the lines
```rust
let mut session = Session {
id,
dir,
baseline,
records: Vec::new(),
log,
next_call: CallId(1),
};
```
become
```rust
let mut session = Session {
id,
dir,
baseline,
records: Vec::new(),
log,
next_call: CallId(1),
epoch: Epoch(0),
slot,
carry: None,
prior_users: 0,
prior_untrusted: false,
};
```
**S6.** Replace the whole `open` function. The lines from `pub fn open(home: &Path, id: SessionId)`
through the `Ok(Session {` block that ends `next_call: CallId(highest + 1),` and `})` `}` become
the function below. Fill any `todo!()` from its comment, then delete the comment. Each comment
is at most six lines.
```rust
pub fn open(home: &Path, id: SessionId) -> Result<Session, SessionError> {
let dir = Session::dir_for(home, &id);
let epochs = complete_epochs(&dir)?;
let Some(&n) = epochs.last() else {
return Err(SessionError::NotFound(id));
};
let epoch = Epoch(n);
let baseline_path = dir.join(format!("{n}.baseline.json"));
let baseline = Baseline::load(&baseline_path).map_err(SessionError::Baseline)?;
let log_path = dir.join(format!("{n}.jsonl"));
let records = load_records(&log_path)?;
let (carry, prior_users, prior_untrusted) = priors(&dir, n)?;
let mut highest = max_call(&records);
for earlier in &epochs[..epochs.len() - 1] {
let path = dir.join(format!("{earlier}.jsonl"));
highest = highest.max(max_call(&load_records(&path)?));
}
let slot = live_slot(&records, 0);
let log = OpenOptions::new()
.append(true)
.open(&log_path)
.map_err(|e| SessionError::Io(log_path.clone(), e))?;
Ok(Session {
id,
dir,
baseline,
records,
log,
next_call: CallId(highest + 1),
epoch,
slot,
carry,
prior_users,
prior_untrusted,
})
}
```
**S7.** After `pub fn records(&self)`, add accessors (the lines)
```rust
pub fn records(&self) -> &[LogRecord] {
&self.records
}
```
become
```rust
pub fn records(&self) -> &[LogRecord] {
&self.records
}
pub fn epoch(&self) -> Epoch {
self.epoch
}
pub fn slot(&self) -> u32 {
self.slot
}
pub fn carry(&self) -> Option<&str> {
self.carry.as_deref()
}
```
**S8.** `untrusted` and `turns`, the lines
```rust
pub fn untrusted(&self) -> bool {
self.records.iter().any(|record| {
matches!(
record,
LogRecord::ToolResult {
untrusted: true,
..
}
)
})
}
/// How many turns the session has had: its `User` records, one per turn that reached the
/// log, a failed turn's too. Counted from `records`, which `open` reads back whole, so the
/// count goes on after a reopen or a restart; right after a turn it is that turn's number,
/// the first being 1. Epoch 0 is a session's only log until compaction is built.
pub fn turns(&self) -> u64 {
let users = self
.records
.iter()
.filter(|record| matches!(record, LogRecord::User { .. }))
.count();
u64::try_from(users).unwrap_or(u64::MAX)
}
```
become
```rust
pub fn untrusted(&self) -> bool {
self.prior_untrusted || any_untrusted(&self.records)
}
/// How many turns the session has had: its `User` records across every epoch, a failed
/// turn's too. The first `User` of the session is turn 1 for its life.
pub fn turns(&self) -> u64 {
self.prior_users
.saturating_add(count_users(&self.records))
}
```
**S9.** `log_path`, the lines
```rust
fn log_path(&self) -> PathBuf {
self.dir.join("0.jsonl")
}
}
```
become
```rust
fn log_path(&self) -> PathBuf {
self.dir.join(format!("{}.jsonl", self.epoch.0))
}
}
/// Epoch numbers that have both `<n>.jsonl` and `<n>.baseline.json`, sorted.
fn complete_epochs(dir: &Path) -> Result<Vec<u32>, SessionError> {
let entries = match std::fs::read_dir(dir) {
Ok(e) => e,
Err(e) if e.kind() == std::io::ErrorKind::NotFound => return Ok(Vec::new()),
Err(e) => return Err(SessionError::Io(dir.to_path_buf(), e)),
};
let mut out = Vec::new();
for entry in entries {
let entry = entry.map_err(|e| SessionError::Io(dir.to_path_buf(), e))?;
let name = entry.file_name();
let Some(name) = name.to_str() else { continue };
let Some(stem) = name.strip_suffix(".jsonl") else { continue };
let Ok(n) = stem.parse::<u32>() else { continue };
if dir.join(format!("{n}.baseline.json")).exists() {
out.push(n);
}
}
out.sort_unstable();
Ok(out)
}
fn load_records(path: &Path) -> Result<Vec<LogRecord>, SessionError> {
let text = match std::fs::read_to_string(path) {
Ok(t) => t,
Err(e) if e.kind() == std::io::ErrorKind::NotFound => {
return Err(SessionError::Io(path.to_path_buf(), e));
}
Err(e) => return Err(SessionError::Io(path.to_path_buf(), e)),
};
let mut records = Vec::new();
if text.is_empty() {
return Ok(records);
}
for (idx, line) in text.split_inclusive('\n').enumerate() {
let line_no = idx + 1;
if !line.ends_with('\n') {
return Err(SessionError::Torn {
path: path.to_path_buf(),
line: line_no,
why: "last line does not end in a newline".to_string(),
});
}
match serde_json::from_str::<LogRecord>(line) {
Ok(record) => records.push(record),
Err(why) => {
return Err(SessionError::Torn {
path: path.to_path_buf(),
line: line_no,
why: why.to_string(),
});
}
}
}
Ok(records)
}
/// Carry from epoch `n-1`, and user / untrusted counts from every epoch before `n`.
fn priors(dir: &Path, n: u32) -> Result<(Option<String>, u64, bool), SessionError> {
let mut prior_users = 0u64;
let mut prior_untrusted = false;
let mut carry = None;
if n == 0 {
return Ok((None, 0, false));
}
for earlier in 0..n {
let path = dir.join(format!("{earlier}.jsonl"));
let records = match load_records(&path) {
Ok(r) => r,
Err(SessionError::Io(path, e)) if e.kind() == std::io::ErrorKind::NotFound => {
return Err(SessionError::NoCarry { path });
}
Err(e) => return Err(e),
};
prior_users = prior_users.saturating_add(count_users(&records));
prior_untrusted = prior_untrusted || any_untrusted(&records);
if earlier + 1 == n {
carry = carry_from(&records, Epoch(n)).map(str::to_string);
if carry.is_none() {
return Err(SessionError::NoCarry { path });
}
}
}
Ok((carry, prior_users, prior_untrusted))
}
fn carry_from(records: &[LogRecord], next: Epoch) -> Option<&str> {
records.iter().rev().find_map(|record| match record {
LogRecord::EpochEnd {
next: n,
summary,
..
} if *n == next => Some(summary.as_str()),
_ => None,
})
}
fn count_users(records: &[LogRecord]) -> u64 {
let users = records
.iter()
.filter(|record| matches!(record, LogRecord::User { .. }))
.count();
u64::try_from(users).unwrap_or(u64::MAX)
}
fn any_untrusted(records: &[LogRecord]) -> bool {
records.iter().any(|record| {
matches!(
record,
LogRecord::ToolResult {
untrusted: true,
..
}
)
})
}
fn max_call(records: &[LogRecord]) -> u64 {
let mut highest = 0u64;
for record in records {
if let LogRecord::ToolResult { call, .. } = record {
highest = highest.max(call.0);
}
}
highest
}
fn live_slot(records: &[LogRecord], fallback: u32) -> u32 {
records
.iter()
.find_map(|record| match record {
LogRecord::SessionStart { slot, .. } => Some(*slot),
_ => None,
})
.unwrap_or(fallback)
}
```
The `open` body in S6 is given whole: copy it, do not leave a `todo!()` in `open`. The helpers
in S9 are given whole too.
## Steps
- [ ] **1. Copy the given test.** From the repository root:
```sh
cp docs/plans/M5d1/files/crates/loopd/tests/session_epochs.rs crates/loopd/tests/
```
- [ ] **2. See it fail.** `cargo build --tests --keep-going -p loopd`, timeout 600000 ms: test
target `session_epochs` does not compile (`E0599` no method `epoch` / `carry` / `slot`, and/or
`E0433` `NoCarry`). Every other target compiles. Existing `session` tests still compile.
- [ ] **3. The edits.** Make S1 to S9 in order. `cargo check -p loopd` prints no error and no
warning. `wc -l crates/loopd/src/session.rs` prints a number ≤ 500.
- [ ] **4. The tests.** `cargo test -p loopd --test session_epochs --test session`: every test
passes, 9 in `session_epochs` and the others as before.
- [ ] **5. Check.** `git diff --stat` lists exactly `crates/loopd/src/session.rs`.
- [ ] **6. Run the gate.** `cargo fmt --all`, then `make gate`, timeout 600000 ms. Last line
`gate: ok`. Test count **1473** (1464 before, plus 9).
- [ ] **7. Log and commit.** Task cell `M5d1/02-loopd-session-epochs`. Stage with
`git add crates/loopd docs/implementer-log.md Cargo.lock`, then commit as `AGENTS.md` says.
## Done when
- `make gate` prints `gate: ok` with 1473 tests.
- `git diff HEAD~1 --stat` lists only `crates/loopd/src/session.rs`,
`crates/loopd/tests/session_epochs.rs` and `docs/implementer-log.md`.
## Stop and report if
- The branch check fails.
- A "before" block is not found exactly once.
- Step 2 shows a different result.
- `cargo check` warns or errors, or `session.rs` exceeds 500 lines.
- A test fails in step 4. Do not change the test.
- An existing `session` test fails: stop and report. Those tests still create and open epoch 0
only.
- The gate count is not 1473.
- A comment does not fit the code under it.
## What the given tests pin
Opening a directory with epochs 0 and 1 loads 1 and the carry from 0's `EpochEnd`; a missing
`EpochEnd` on epoch 0 when epoch 1 files exist is `NoCarry` naming `0.jsonl` with the damaged-log
runbook pointer; epoch 0 has no carry and keeps its `SessionStart` slot; two `User`s in epoch 0
and one in epoch 1 is `turns() == 3` while `records()` has one `User`; an untrusted `ToolResult`
in epoch 0 taints after compact; `next_call` is one more than the highest call in any epoch;
`append` after opening epoch 1 writes `1.jsonl`; a `1.baseline.json` without `1.jsonl` is ignored;
an `EpochEnd` on epoch 0 without the next files stays on epoch 0.
+185
View File
@@ -0,0 +1,185 @@
# M5d-1 task 03: the carried summary in the next epoch's prefix
**Branch:** `m5d1`. Check it: `git branch --show-current` prints `m5d1` and `git status --short`
is empty; otherwise stop. Never switch branch.
**Commit subject:** `loopd: replay a carried summary as an unlogged user message`
**Log row:** the Task cell is `M5d1/03-loopd-carry-messages`.
## Goal
For epoch `n > 0`, the request prefix is the new system message, then one `ChatMessage::User`
whose content is a fixed heading plus the carry, then the new epoch's records (spec 3.5). That
user message is **not** a `LogRecord::User`: it is not a turn, not reported to Honcho, and not
counted by `turns()`. Epoch 0 is unchanged (`carry` is `None`).
`run_turn` must send this prefix, or a compacted session's next live turn would omit the
summary.
## Every place that changes
| Place | What happens there |
|---|---|
| `crates/loopd/src/baseline.rs` | `CARRY_HEADING`; `messages_with_carry`; `messages` calls it with `None` |
| `crates/loopd/src/turn.rs` | the request uses `messages_with_carry` and `session.carry()` |
| `crates/loopd/tests/carry_messages.rs` | new, given |
| everything else | **no change**: not the existing `baseline` tests (they still call `messages`) |
## Files
- Copy, from `docs/plans/M5d1/files/` (the same path under it): the new test
`crates/loopd/tests/carry_messages.rs`.
- Modify: `crates/loopd/src/baseline.rs`, `crates/loopd/src/turn.rs`,
`docs/implementer-log.md`.
- Do not touch: any other file.
## Facts you need
- **The heading** is built from pieces so no file holds a chat-template tag whole (T33):
```rust
pub const CARRY_HEADING: &str = concat!("Earlier ", "turns, summarised:");
```
The user content is `format!("{CARRY_HEADING}\n\n{carry}")` when `carry` is `Some`. `None`
inserts nothing.
- **`messages` stays.** Existing tests and `limits.rs` call `messages(baseline, records)`. Make
it `messages_with_carry(baseline, records, None)` so those tests do not change.
- **`run_turn` today** (one place):
```rust
messages: messages(baseline, session.records()),
```
It must pass `session.carry()`. `Session::carry` is `Option<&str>` from task 02.
- Insert the carry user **after** the system message and **before** walking `records`.
- The items the edits use. This table is complete.
| Item | What it does |
|---|---|
| `concat!` | heading from pieces |
| `Session::carry(&self) -> Option<&str>` | task 02 |
| `ChatMessage::User { content }` | the carry, and log users |
## The edits, exactly
### `crates/loopd/src/baseline.rs`
**B1.** After the `Baseline` impl, replace `messages`. The lines
```rust
/// The system message, then each log record replayed in order. Every record variant is named, so a
/// new one is a compile error rather than a silent skip.
pub fn messages(baseline: &Baseline, records: &[LogRecord]) -> Vec<ChatMessage> {
let mut out = vec![ChatMessage::System {
content: baseline.system.clone(),
}];
for record in records {
```
become
```rust
/// The heading in front of a carried summary. Built from pieces (T33).
pub const CARRY_HEADING: &str = concat!("Earlier ", "turns, summarised:");
/// The system message, then each log record replayed in order. Every record variant is named, so a
/// new one is a compile error rather than a silent skip.
pub fn messages(baseline: &Baseline, records: &[LogRecord]) -> Vec<ChatMessage> {
messages_with_carry(baseline, records, None)
}
/// Like `messages`, and when `carry` is `Some` inserts one user message after the system
/// prompt: the heading, a blank line, the summary. That user is not a `LogRecord::User`.
pub fn messages_with_carry(
baseline: &Baseline,
records: &[LogRecord],
carry: Option<&str>,
) -> Vec<ChatMessage> {
let mut out = vec![ChatMessage::System {
content: baseline.system.clone(),
}];
if let Some(summary) = carry {
out.push(ChatMessage::User {
content: format!("{CARRY_HEADING}\n\n{summary}"),
});
}
for record in records {
```
The rest of the `match` is unchanged; do not touch it.
### `crates/loopd/src/turn.rs`
**T1.** The `use` line
```rust
use crate::baseline::messages;
```
becomes
```rust
use crate::baseline::messages_with_carry;
```
**T2.** The lines
```rust
let request = ChatRequest {
slot: rt.cfg.slots.main,
messages: messages(baseline, session.records()),
tools: baseline.tools.clone(),
thinking: true,
};
```
become
```rust
let request = ChatRequest {
slot: rt.cfg.slots.main,
messages: messages_with_carry(baseline, session.records(), session.carry()),
tools: baseline.tools.clone(),
thinking: true,
};
```
## Steps
- [ ] **1. Copy the given test.**
```sh
cp docs/plans/M5d1/files/crates/loopd/tests/carry_messages.rs crates/loopd/tests/
```
- [ ] **2. See it fail.** `cargo build --tests --keep-going -p loopd`, timeout 600000 ms: test
target `carry_messages` does not compile (`E0432` unresolved import
`loopd::baseline::{CARRY_HEADING, messages_with_carry}`). Every other target compiles.
- [ ] **3. The edits.** B1, T1, T2. `cargo check -p loopd` prints no error and no warning.
- [ ] **4. The tests.** `cargo test -p loopd --test carry_messages --test baseline --test turn`:
every test passes, 5 in `carry_messages` and the others as before.
- [ ] **5. Check.** `git diff --stat` lists exactly `crates/loopd/src/baseline.rs` and
`crates/loopd/src/turn.rs`.
- [ ] **6. Run the gate.** `cargo fmt --all`, then `make gate`, timeout 600000 ms. Last line
`gate: ok`. Test count **1478** (1473 before, plus 5).
- [ ] **7. Log and commit.** Task cell `M5d1/03-loopd-carry-messages`. Stage with
`git add crates/loopd docs/implementer-log.md Cargo.lock`, then commit as `AGENTS.md` says.
## Done when
- `make gate` prints `gate: ok` with 1478 tests.
- `git diff HEAD~1 --stat` lists only `crates/loopd/src/baseline.rs`, `crates/loopd/src/turn.rs`,
`crates/loopd/tests/carry_messages.rs` and `docs/implementer-log.md`.
## Stop and report if
- The branch check fails; a "before" block is not found exactly once; step 2 differs; `cargo
check` warns; a test fails; the gate count is not 1478; a comment does not fit its code.
- You think `messages` should insert the carry. It must not: existing tests call `messages` and
would see an extra user. `run_turn` is the one call that must pass `session.carry()`.
## What the given tests pin
Epoch 0 with `None` matches `messages`; a carry is a user after the system prompt and before the
epoch's users; the carry is not in the log records; the heading is the `concat!` pieces; the
first request of epoch 1 is not a prefix-extension of epoch 0's messages.
+232
View File
@@ -0,0 +1,232 @@
# M5d-1 task 04: Honcho reads every epoch file
**Branch:** `m5d1`. Check it: `git branch --show-current` prints `m5d1` and `git status --short`
is empty; otherwise stop. Never switch branch.
**Commit subject:** `memoryd: Honcho concatenates every epoch of a session`
**Log row:** the Task cell is `M5d1/04-memoryd-honcho-epochs`.
## Goal
`memoryd` keys a turn as `Item::Turn { session, turn }` with no epoch (spec 3.2). After
compaction, turn 3 may live in `1.jsonl`. `log_path` still names `0.jsonl` (existing tests pin
that path). A new `read_session` concatenates every `<n>.jsonl` in epoch order. `TurnSync`
walks that concatenation, and the worker's `send_turn` does too, so a turn in epoch 1 is
decided and sent.
`turn_records` / `turns` do not change: they already count `User` records on a slice.
## Every place that changes
| Place | What happens there |
|---|---|
| `crates/memoryd/src/honcho/turns.rs` | `epoch_logs`; `read_session`; `records` and `resume` use it |
| `crates/memoryd/src/honcho/worker.rs` | `send_turn` reads the concatenated log |
| `crates/memoryd/tests/honcho_epochs.rs` | new, given |
| everything else | **no change**: not `log_path`'s return value, not `decide.rs` |
## Files
- Copy, from `docs/plans/M5d1/files/` (the same path under it): the new test
`crates/memoryd/tests/honcho_epochs.rs`.
- Modify: `crates/memoryd/src/honcho/turns.rs`, `crates/memoryd/src/honcho/worker.rs`,
`docs/implementer-log.md`.
- Do not touch: any other file.
## Facts you need
- **`read_records` / `LogError`** already exist in `turns.rs`. `read_session` returns
`Result<Vec<LogRecord>, LogError>` so a damaged line still names its file and
`LOG_RUNBOOK`.
- **Listing.** Under `sessions/<id>/`, every file named `<n>.jsonl` where `n` parses as `u32`,
sorted by `n`. A missing session directory is empty, not an error (same as a missing
`0.jsonl` today).
- **`log_path` stays** `.../0.jsonl`. Do not change its body. `readable` in the worker still
names that path when the concatenated read succeeds, so existing recovery log lines stay
stable.
- **`send_turn` today** calls `read_turn(&log_path(...), turn)`, which only sees epoch 0. After
this task it loads `read_session`, then `turn_records` / `user_text` / `final_reply` from
`crate::honcho::decide` (already `pub`).
- `turns.rs` after this task stays under 500 lines (it is 332 now).
- The items the edits use. This table is complete.
| Item | What it does |
|---|---|
| `read_records(&Path) -> Result<Vec<LogRecord>, LogError>` | one file, already in `turns.rs` |
| `turn_records(&[LogRecord], u64) -> Option<&[LogRecord]>` | `decide.rs` |
| `user_text(&[LogRecord]) -> (&str, Option<&str>)` | `decide.rs` |
| `final_reply(&[LogRecord]) -> Option<&str>` | `decide.rs` |
| `str::strip_suffix(".jsonl")`, `str::parse::<u32>` | epoch numbers |
| `Unsent::Unreadable(String)` | worker, `e.to_string()` |
## The edits, exactly
### `crates/memoryd/src/honcho/turns.rs`
**H1.** The doc comment on `log_path`, the lines
```rust
/// The path of `session`'s log under `sessions`: epoch 0, the only one `loopd` writes.
pub fn log_path(sessions: &Path, session: &SessionId) -> PathBuf {
```
become
```rust
/// The path of `session`'s epoch-0 log under `sessions`. Later epochs are sibling files;
/// `read_session` concatenates them. Existing tests pin this path.
pub fn log_path(sessions: &Path, session: &SessionId) -> PathBuf {
```
**H1b.** After `read_records` (the function that ends `Ok(records)` / `}`), insert these two
functions. They sit below `LogError` and `read_records`, so both names resolve.
```rust
/// Every `<n>.jsonl` of `session`, sorted by `n`. A missing directory is empty.
pub fn epoch_logs(sessions: &Path, session: &SessionId) -> Result<Vec<PathBuf>, LogError> {
let dir = sessions.join(session.as_str());
let entries = match std::fs::read_dir(&dir) {
Ok(e) => e,
Err(e) if e.kind() == std::io::ErrorKind::NotFound => return Ok(Vec::new()),
Err(e) => return Err(LogError::Io(format!("{}: {e}", dir.display()))),
};
let mut found: Vec<(u32, PathBuf)> = Vec::new();
for entry in entries {
let entry = match entry {
Ok(e) => e,
Err(e) => return Err(LogError::Io(format!("{}: {e}", dir.display()))),
};
let name = entry.file_name();
let Some(name) = name.to_str() else { continue };
let Some(stem) = name.strip_suffix(".jsonl") else { continue };
let Ok(n) = stem.parse::<u32>() else { continue };
found.push((n, entry.path()));
}
found.sort_by_key(|(n, _)| *n);
Ok(found.into_iter().map(|(_, p)| p).collect())
}
/// Every epoch's records, in epoch order. A missing session has none.
pub fn read_session(
sessions: &Path,
session: &SessionId,
) -> Result<Vec<LogRecord>, LogError> {
let mut out = Vec::new();
for path in epoch_logs(sessions, session)? {
out.extend(read_records(&path)?);
}
Ok(out)
}
```
The next function after this insert is `finished`. Do not change `finished`.
**H2.** In `resume`, the line
```rust
if let Ok(records) = read_log(&log_path(&self.sessions, &session)) {
```
becomes
```rust
if let Ok(records) = read_session(&self.sessions, &session) {
```
**H3.** In `records`, the line
```rust
match read_log(&log_path(&self.sessions, session)) {
```
becomes
```rust
match read_session(&self.sessions, session).map_err(|e| e.to_string()) {
```
`read_log` already mapped `LogError` through `Display`; this keeps the logged line the same
shape (`memoryd: {why}; {LOG_RUNBOOK}`).
### `crates/memoryd/src/honcho/worker.rs`
**W1.** The `use` line
```rust
use crate::honcho::turns::{TurnSync, log_path};
```
becomes
```rust
use crate::honcho::decide::{final_reply, turn_records, user_text};
use crate::honcho::turns::{TurnSync, log_path, read_session};
```
**W2.** The body of `send_turn`, the lines
```rust
let path = log_path(&self.sessions, session);
let read = read_turn(&path, turn).map_err(Unsent::Unreadable)?;
self.readable(&path);
let (text, reply) = read.ok_or(Unsent::Local("log"))?;
```
become
```rust
let path = log_path(&self.sessions, session);
let records = match read_session(&self.sessions, session) {
Ok(records) => {
self.readable(&path);
records
}
Err(e) => return Err(Unsent::Unreadable(e.to_string())),
};
let slice = turn_records(&records, turn).ok_or(Unsent::Local("log"))?;
let (text, _) = user_text(slice);
let reply = final_reply(slice).ok_or(Unsent::Local("log"))?;
```
The rest of `send_turn` is unchanged (`honcho.ok_or`, `batch`, `MAX_BATCH`, `add_messages`).
`text` is `&str` from `user_text`; `reply` is `&str` from `final_reply`. `batch` already takes
`&str`.
## Steps
- [ ] **1. Copy the given test.**
```sh
cp docs/plans/M5d1/files/crates/memoryd/tests/honcho_epochs.rs crates/memoryd/tests/
```
- [ ] **2. See it fail.** `cargo build --tests --keep-going -p memoryd`, timeout 600000 ms: test
target `honcho_epochs` does not compile (`E0432` unresolved import
`memoryd::honcho::turns::read_session`). Every other target compiles.
- [ ] **3. The edits.** H1, H1b, H2, H3, W1, W2. `cargo check -p memoryd`
prints no error and no warning. `wc -l crates/memoryd/src/honcho/turns.rs` and
`wc -l crates/memoryd/src/honcho/worker.rs` are each ≤ 500.
- [ ] **4. The tests.** `cargo test -p memoryd --test honcho_epochs --test honcho_turns --test
honcho_worker`: every test passes, 5 in `honcho_epochs` and the others as before.
- [ ] **5. Check.** `git diff --stat` lists exactly `crates/memoryd/src/honcho/turns.rs` and
`crates/memoryd/src/honcho/worker.rs`.
- [ ] **6. Run the gate.** `cargo fmt --all`, then `make gate`, timeout 600000 ms. Last line
`gate: ok`. Test count **1483** (1478 before, plus 5).
- [ ] **7. Log and commit.** Task cell `M5d1/04-memoryd-honcho-epochs`. Stage with
`git add crates/memoryd docs/implementer-log.md Cargo.lock`, then commit as `AGENTS.md` says.
## Done when
- `make gate` prints `gate: ok` with 1483 tests.
- `git diff HEAD~1 --stat` lists only the two `honcho` sources, `honcho_epochs.rs`, and the log.
## Stop and report if
- The branch check fails; a "before" block is not found exactly once; step 2 differs; a file
exceeds 500 lines; a test fails; the gate count is not 1483.
- You think `log_path` should return every file. It must still return `0.jsonl`.
## What the given tests pin
Two users in epoch 0 and one in epoch 1 is `turns() == 3` and `turn_records(..., 3)` is the
epoch-1 user; epoch 0 alone still reads; a damaged `1.jsonl` names that file; `TurnSync` decides
turn 3 from epoch 1; a missing session directory is empty.
+130
View File
@@ -0,0 +1,130 @@
# M5d-1 task 05: when to compact, and the fallback summary
**Branch:** `m5d1`. Check it: `git branch --show-current` prints `m5d1` and `git status --short`
is empty; otherwise stop. Never switch branch.
**Commit subject:** `loopd: compaction threshold and the fallback summary`
**Log row:** the Task cell is `M5d1/05-loopd-compact-due`.
## Goal
A session is due to compact when the current epoch's last `Usage` has
`cache_n + prompt_n` at or above `threshold × expect.n_ctx` (spec 3.3). There is no idle timer.
If the summary call is context-full (task 06), the fallback is the epoch's `User` texts, newest
last, cut to `summary_chars` on a character boundary (spec 3.4).
This task is those two pure functions, the `compact` module, and the starter `config/compact.md`.
No inference, no disk besides copying the starter.
## Every place that changes
| Place | What happens there |
|---|---|
| `crates/loopd/src/compact.rs` | new: `due`, `fallback_summary`, `cap_chars` |
| `crates/loopd/src/lib.rs` | `pub mod compact;` |
| `config/compact.md` | new, given starter |
| `crates/loopd/tests/compact_due.rs` | new, given |
| everything else | **no change** |
## Files
- Copy, from `docs/plans/M5d1/files/` (the same path under it): the new test
`crates/loopd/tests/compact_due.rs` and the starter `config/compact.md`.
- Create: `crates/loopd/src/compact.rs` (skeleton below).
- Modify: `crates/loopd/src/lib.rs`, `docs/implementer-log.md`.
- Do not touch: any other file.
## Facts you need
- **Depth is saturating.** `cache_n.saturating_add(prompt_n) as f64 >= threshold * (n_ctx as f64)`.
0.85 × 262144 is 222822.4, so 222822 is not due and 222823 is. 0.85 × 1000 is 850 exactly.
- **Cast.** `u64 as f64` is exact for these magnitudes (n_ctx is 262144). Do not use `as` on
untrusted indexes; these are our own config and the server's counts.
- **Fallback join.** `"\n\n"` between texts. Newest last: reverse the `User` contents, then join,
then cap.
- **Cap.** `s.chars().take(n).collect::<String>()`. `n == 0` is empty. A multi-byte character
is not split.
- **`config/compact.md`** is copied like `config/system.md`. Task 06 reads it from
`Config::compact_prompt`.
- The items the edits use. This table is complete.
| Item | What it does |
|---|---|
| `bxcore::Usage` | `cache_n`, `prompt_n` |
| `LogRecord::User { content, .. }` | fallback texts |
| `str::chars`, `Iterator::take` | the cap |
## Skeleton: `crates/loopd/src/compact.rs`
Write this file, then fill the two `todo!()` bodies from their comments and delete the
comments. Each comment is at most six lines.
```rust
//! Compaction: when a session starts a new epoch, the summary call, and writing the next files
//! (M5d spec, section 3). Later tasks fill the call and the write.
use bxcore::{LogRecord, Usage};
/// True when `cache_n + prompt_n` is at or above `threshold * n_ctx`.
pub fn due(usage: &Usage, threshold: f64, n_ctx: u64) -> bool {
// saturating_add the two counts; compare as f64 to `threshold * (n_ctx as f64)`.
todo!()
}
/// The epoch's `User` texts, newest last, joined with `"\n\n"`, then `cap_chars`.
pub fn fallback_summary(records: &[LogRecord], summary_chars: usize) -> String {
// Collect `&str` contents of every `User`, reverse, join `"\n\n"`, then `cap_chars`.
todo!()
}
pub(crate) fn cap_chars(s: &str, n: usize) -> String {
s.chars().take(n).collect()
}
```
### `crates/loopd/src/lib.rs`
**L1.** After `pub mod channel;` add `pub mod compact;` (keep rustfmt's order; if rustfmt
moves the line, accept that).
## Steps
- [ ] **1. Copy the given files.**
```sh
cp docs/plans/M5d1/files/crates/loopd/tests/compact_due.rs crates/loopd/tests/
cp docs/plans/M5d1/files/config/compact.md config/compact.md
```
- [ ] **2. Write the skeleton** at `crates/loopd/src/compact.rs` and make L1. See it fail:
`cargo test -p loopd --test compact_due`, timeout 600000 ms: the two `due` tests and the
three `fallback` tests panic on `todo!()` (or fail their asserts if you filled the bodies
early). If the test target does not compile (`E0432` unresolved `loopd::compact`), the
`mod` line is missing.
- [ ] **3. Fill `due` and `fallback_summary`.** Delete each instruction comment. `cargo check
-p loopd` prints no error and no warning.
- [ ] **4. The tests.** `cargo test -p loopd --test compact_due`: 6 passed.
- [ ] **5. Check.** `git diff --stat` lists `crates/loopd/src/lib.rs`,
`crates/loopd/src/compact.rs`, `config/compact.md`. `git status --short` shows
`compact_due.rs` as new.
- [ ] **6. Run the gate.** `cargo fmt --all`, then `make gate`, timeout 600000 ms. Last line
`gate: ok`. Test count **1489** (1483 before, plus 6).
- [ ] **7. Log and commit.** Task cell `M5d1/05-loopd-compact-due`. Stage with
`git add crates/loopd config/compact.md docs/implementer-log.md Cargo.lock`, then commit
as `AGENTS.md` says.
## Done when
- `make gate` prints `gate: ok` with 1489 tests.
- `git diff HEAD~1 --stat` lists only `crates/loopd/src/lib.rs`, `crates/loopd/src/compact.rs`,
`crates/loopd/tests/compact_due.rs`, `config/compact.md` and the log.
## Stop and report if
- The branch check fails; step 2 fails to compile for a reason other than the missing module
or `todo!()`; a test fails; the gate count is not 1489; you add an idle timer. There is none.
## What the given tests pin
849/1000 is not due, 850 is; 222822/262144 is not due, 222823 is; threshold 1.0 is the full
window; fallback is newest-last with `"\n\n"`; the cap is in characters on a boundary; no
users is empty.
+213
View File
@@ -0,0 +1,213 @@
# M5d-1 task 06: the summary call, and `session-compact-failed`
**Branch:** `m5d1`. Check it: `git branch --show-current` prints `m5d1` and `git status --short`
is empty; otherwise stop. Never switch branch.
**Commit subject:** `loopd: compact summary on the background slot`
**Log row:** the Task cell is `M5d1/06-loopd-compact-summary`.
## Goal
Compaction asks for a summary of the current epoch on the **background** slot, with **no tools**,
thinking on (spec 3.4). The last user message is `compact.md` (task 01's `compact_prompt`),
trimmed at the end. The completion is tag-neutralised and cut to `summary_chars`.
If that call is context-full (`exceed_context_size_error`), use task 05's `fallback_summary`.
If `compact.md` cannot be read, or the call fails for any other reason, return
`CompactError::Failed` whose `Display` ends with the runbook pointer
`see docs/runbook.md#session-compact-failed`. There is no fallback for a missing prompt.
This task does not write the next epoch. Task 07 does.
## Every place that changes
| Place | What happens there |
|---|---|
| `crates/loopd/src/compact.rs` | `CompactError`; `summarise` (edits E1, E2) |
| `docs/runbook.md` | append the given `session-compact-failed` entry |
| `crates/loopd/tests/compact_summary.rs` | new, given |
| everything else | **no change**: not `turn.rs`, not `channel.rs`, not `session.rs` |
## Files
- Copy, from `docs/plans/M5d1/files/` (the same path under it): the new test
`crates/loopd/tests/compact_summary.rs`.
- Append, from `docs/plans/M5d1/files/runbook-session-compact-failed.md`, to `docs/runbook.md`.
- Modify: `crates/loopd/src/compact.rs`, `docs/runbook.md`, `docs/implementer-log.md`.
- Do not touch: any other file. `config/compact.md` is already in the tree from task 05.
## Facts you need
- **The request.** `ChatRequest { slot: cfg.slots.background, messages, tools: Vec::new(),
thinking: true }`. Empty `tools` is omitted from the JSON (`skip_serializing_if` on the body).
The given test asserts `body.get("tools").is_none()` and `id_slot == 1` (the test config's
background slot).
- **The messages.** `messages_with_carry(session.baseline(), session.records(), session.carry())`,
then one `ChatMessage::User` whose content is `compact.md` after `trim_end`. Epoch 0 has no
carry; later epochs keep the carried summary in the prefix so the model summarises what the
session actually sends.
- **Read the prompt** with `std::fs::read_to_string(&cfg.compact_prompt())`. An I/O error is
`Failed { detail: format!("{}: {e}", path.display()) }` so the path (and `compact.md`) appears
in the message.
- **The call.** `client.chat_with_retry(&request, &mut |_| {})`. A 400 that is not context-full
is not retried (`is_retryable` is already false for it).
- **Context-full.** `crate::turn::is_context_full(&e)` then `fallback_summary(session.records(),
cfg.compact.summary_chars)`. Do not fall back for a missing file or any other `InferError`.
- **The text.** `completion.content.unwrap_or_default()`, then `bxcore::sanitize::neutralise`,
then `cap_chars(..., cfg.compact.summary_chars)`. Neutralise first so a tag is not split by
the cap.
- **`Display`.** `"{detail}; see docs/runbook.md#session-compact-failed"`. The given tests box
the error as `dyn Error` and check `to_string()`.
- **`compact.rs` stays ≤ 500 lines.** After this task it is about 120.
- The items the edits use. This table is complete.
| Item | What it does |
|---|---|
| `Config::compact_prompt` | task 01, `compact.md` beside `system.md` |
| `messages_with_carry` | task 03 |
| `Client::chat_with_retry` | same as a live turn |
| `turn::is_context_full` | the one 400 `exceed_context_size_error` |
| `bxcore::sanitize::neutralise` | tags to a visible look-alike |
| `fallback_summary`, `cap_chars` | task 05 |
| `ChatRequest.tools: Vec::new()` | omits `tools` on the wire |
## The edits, exactly
Nothing else in `compact.rs` changes. Each quoted "before" block occurs exactly once after
task 05. Keep the fields in the order shown.
### `crates/loopd/src/compact.rs`
**E1.** The module doc and the `use` line. The lines
```rust
//! Compaction: when a session starts a new epoch, the summary call, and writing the next files
//! (M5d spec, section 3). Later tasks fill the call and the write.
use bxcore::{LogRecord, Usage};
```
become
```rust
//! Compaction: when a session starts a new epoch, the summary call, and writing the next files
//! (M5d spec, section 3). Task 07 writes the next files; task 08 calls this after a turn.
use bxcore::sanitize::neutralise;
use bxcore::{LogRecord, Usage};
use crate::baseline::messages_with_carry;
use crate::config::Config;
use crate::llama::{ChatMessage, ChatRequest, Client};
use crate::session::Session;
use crate::turn::is_context_full;
/// Why a summary call or a next-epoch write failed. Every variant's `Display` ends with the
/// runbook pointer `session-compact-failed`.
#[derive(Debug)]
pub enum CompactError {
Failed { detail: String },
}
impl std::fmt::Display for CompactError {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
CompactError::Failed { detail } => write!(
f,
"{detail}; see docs/runbook.md#session-compact-failed"
),
}
}
}
impl std::error::Error for CompactError {}
```
**E2.** After `cap_chars` (the function that ends `s.chars().take(n).collect()` / `}`), add
`summarise`. Copy this function whole. There is no `todo!()`.
```rust
/// One completion on the background slot: no tools, thinking on, last user `compact.md`.
/// Context-full uses `fallback_summary`. A missing prompt, or any other infer error, is
/// `CompactError::Failed` with the runbook pointer. No fallback for a missing prompt.
pub fn summarise(cfg: &Config, client: &Client, session: &Session) -> Result<String, CompactError> {
let path = cfg.compact_prompt();
let prompt = std::fs::read_to_string(&path).map_err(|e| CompactError::Failed {
detail: format!("{}: {e}", path.display()),
})?;
let mut messages = messages_with_carry(session.baseline(), session.records(), session.carry());
messages.push(ChatMessage::User {
content: prompt.trim_end().to_string(),
});
let request = ChatRequest {
slot: cfg.slots.background,
messages,
tools: Vec::new(),
thinking: true,
};
match client.chat_with_retry(&request, &mut |_| {}) {
Ok(completion) => {
let text = completion.content.unwrap_or_default();
let text = neutralise(&text);
Ok(cap_chars(text.as_ref(), cfg.compact.summary_chars))
}
Err(e) if is_context_full(&e) => {
Ok(fallback_summary(session.records(), cfg.compact.summary_chars))
}
Err(e) => Err(CompactError::Failed {
detail: e.to_string(),
}),
}
}
```
## Steps
- [ ] **1. Copy the given test.**
```sh
cp docs/plans/M5d1/files/crates/loopd/tests/compact_summary.rs crates/loopd/tests/
```
- [ ] **2. See it fail.** `cargo build --tests --keep-going -p loopd`, timeout 600000 ms: test
target `compact_summary` does not compile (`E0432` unresolved import `loopd::compact::CompactError`
and/or `E0599` no function `summarise`). Every other target compiles. `compact_due` still
compiles.
- [ ] **3. The edits.** E1, E2. `cargo check -p loopd` prints no error and no warning.
`wc -l crates/loopd/src/compact.rs` prints a number ≤ 500.
- [ ] **4. The tests.** `cargo test -p loopd --test compact_summary --test compact_due`: every
test passes, 6 in `compact_summary` and 6 in `compact_due`.
- [ ] **5. The runbook.** From the repository root:
```sh
{ echo; cat docs/plans/M5d1/files/runbook-session-compact-failed.md; } >> docs/runbook.md
```
Then `grep -c '^## session-compact-failed$' docs/runbook.md` prints `1`.
- [ ] **6. Check.** `git diff --stat` lists `crates/loopd/src/compact.rs` and `docs/runbook.md`.
`git status --short` shows `compact_summary.rs` as new.
- [ ] **7. Run the gate.** `cargo fmt --all`, then `make gate`, timeout 600000 ms. Last line
`gate: ok`. Test count **1495** (1489 before, plus 6).
- [ ] **8. Log and commit.** Task cell `M5d1/06-loopd-compact-summary`. Stage with
`git add crates/loopd docs/runbook.md docs/implementer-log.md Cargo.lock`, then commit as
`AGENTS.md` says.
## Done when
- `make gate` prints `gate: ok` with 1495 tests.
- `git diff HEAD~1 --stat` lists only `crates/loopd/src/compact.rs`,
`crates/loopd/tests/compact_summary.rs`, `docs/runbook.md` and the log.
## Stop and report if
- The branch check fails; a "before" block is not found exactly once; step 2 differs; `cargo
check` warns; a test fails; the gate count is not 1495; a comment does not fit its code;
`compact.rs` exceeds 500 lines; you add a fallback for a missing `compact.md`; you send tools
on the summary request; you use `slots.main` for the summary.
## What the given tests pin
The summary request is `id_slot` of the background slot, omits `tools`, has thinking on, and its
last user is `compact.md` after `trim_end`; the completion is capped to `summary_chars`; a
context-full summary is the epoch's `User` texts; tags in the completion are neutralised; a
missing `compact.md` names the file and the runbook; a non-context-full infer error is
`CompactError::Failed` with the same pointer.
+268
View File
@@ -0,0 +1,268 @@
# M5d-1 task 07: writing the next epoch
**Branch:** `m5d1`. Check it: `git branch --show-current` prints `m5d1` and `git status --short`
is empty; otherwise stop. Never switch branch.
**Commit subject:** `loopd: write the next epoch from a compaction summary`
**Log row:** the Task cell is `M5d1/07-loopd-compact-advance`.
## Goal
Given a summary string, `commit` starts the next epoch (spec 3.5):
1. Append `EpochEnd { next, summary }` to the current log, unless that log already ends with an
`EpochEnd` (a compact that stopped after the end record and before the next files).
2. Ask `memoryd` for a fresh block; write `<n>.baseline.json` from `assemble_with` (so identity
edits applied in the old epoch land here, and `registry.core_schemas()` is the new tools
array).
3. Create `<n>.jsonl` with `SessionStart { epoch: n, slot: the session's live slot, baseline:
hash of the new file }`.
4. The in-memory `Session` switches: new baseline, records that are only that `SessionStart`,
`carry = Some(summary)`, `prior_users` / `prior_untrusted` accumulated from the old records.
A crash after step 1 and before the next files exist reopens epoch 0 with `EpochEnd` last (task
02). The next `commit` resumes: it uses the summary already in that `EpochEnd` as the carry and
does not write a second `EpochEnd`. The summary argument is ignored in that case.
## Every place that changes
| Place | What happens there |
|---|---|
| `crates/loopd/src/session.rs` | `begin_epoch` (edit S1) |
| `crates/loopd/src/compact.rs` | `commit` (edit C1) |
| `crates/loopd/tests/compact_advance.rs` | new, given |
| everything else | **no change**: not `turn.rs`, not `channel.rs` |
## Files
- Copy, from `docs/plans/M5d1/files/` (the same path under it): the new test
`crates/loopd/tests/compact_advance.rs`.
- Modify: `crates/loopd/src/session.rs`, `crates/loopd/src/compact.rs`,
`docs/implementer-log.md`.
- Do not touch: any other file.
## Facts you need
- **`begin_epoch` is `pub(crate)`** on `Session` so `compact.rs` can switch the private fields.
Tests call `compact::commit`, never `begin_epoch`.
- **Order inside `begin_epoch`.** Write `<n>.baseline.json` (overwrite is fine on resume),
`create_new` the log file, then set `epoch` / `baseline` / `carry` / empty `records` / the new
`File`, **then** `persist` `SessionStart`. `log_path` uses `self.epoch`, so `epoch` must be
`next` before that persist. Accumulate `prior_users` and `prior_untrusted` from the **old**
`records` before clearing them. Do not reset `next_call` or `slot`.
- **Resume.** If `session.records().last()` is `EpochEnd { next, summary }`, skip the append;
`next` and `summary` (cloned) are what `begin_epoch` gets. Otherwise append `EpochEnd` with
`next = Epoch(session.epoch().0.saturating_add(1))` and the summary argument, then
`begin_epoch`.
- **Memory.** `memory.block().map_err(|e| CompactError::Failed { detail: e })?`. Then
`Baseline::assemble_with(cfg, registry, &block)`. `assemble_with` already calls
`registry.core_schemas()`, so a new core tool lands at this boundary. Map assemble and session
errors to `Failed { detail: e.to_string() }`.
- **`NoMemory.block()`** is `Ok("")`. The given test `the_new_baseline_takes_a_fresh_memory_block`
uses a `MemorySource` that returns a distinctive sentence.
- **`session.rs` stays ≤ 500 lines.** After task 02 it is about 430; `begin_epoch` is about 40.
- **`compact.rs` stays ≤ 500 lines.** After this task it is about 180.
- The items the edits use. This table is complete.
| Item | What it does |
|---|---|
| `Baseline::assemble_with` | new system + memory block + `registry.core_schemas()` |
| `Baseline::to_json` / `hash` | the file and `SessionStart.baseline` |
| `OpenOptions::create_new(true).append(true)` | the new log, as `create` does |
| `count_users` / `any_untrusted` | task 02, module-private in `session.rs` |
| `Session::persist` | private; `begin_epoch` is in the same impl |
| `MemorySource::block` | fresh text; `Err` is `Failed` |
| `Timestamp::now` | `EpochEnd` and `SessionStart` |
| `Epoch(n.saturating_add(1))` | the next number |
## The edits, exactly
Each quoted "before" block occurs exactly once after task 06 (session.rs is as task 02 left it).
### `crates/loopd/src/session.rs`
**S1.** `log_path`, the lines
```rust
fn log_path(&self) -> PathBuf {
self.dir.join(format!("{}.jsonl", self.epoch.0))
}
}
```
become
```rust
fn log_path(&self) -> PathBuf {
self.dir.join(format!("{}.jsonl", self.epoch.0))
}
/// Write `<n>.baseline.json` and `<n>.jsonl` with `SessionStart`, and switch this session
/// onto that epoch. The live slot and `next_call` stay. Callers pass the carry that
/// `messages_with_carry` will insert.
pub(crate) fn begin_epoch(
&mut self,
next: Epoch,
baseline: Baseline,
carry: String,
) -> Result<(), SessionError> {
let n = next.0;
let baseline_path = self.dir.join(format!("{n}.baseline.json"));
let json = baseline.to_json().map_err(SessionError::Encode)?;
std::fs::write(&baseline_path, json)
.map_err(|e| SessionError::Io(baseline_path.clone(), e))?;
let log_path = self.dir.join(format!("{n}.jsonl"));
let log = OpenOptions::new()
.create_new(true)
.append(true)
.open(&log_path)
.map_err(|e| SessionError::Io(log_path.clone(), e))?;
self.prior_users = self.prior_users.saturating_add(count_users(&self.records));
self.prior_untrusted = self.prior_untrusted || any_untrusted(&self.records);
self.epoch = next;
self.baseline = baseline;
self.carry = Some(carry);
self.records = Vec::new();
self.log = log;
let hash = self.baseline.hash().map_err(SessionError::Baseline)?;
self.persist(LogRecord::SessionStart {
time: Timestamp::now(),
session: self.id.clone(),
epoch: next,
slot: self.slot,
baseline: hash,
})
}
}
```
The helpers after the `impl` (`complete_epochs` and the rest) stay where they are. Do not move
them.
### `crates/loopd/src/compact.rs`
**C1.** After `summarise` (the function that ends the `match client.chat_with_retry` / `}`), add
the `use` items this function needs by extending the existing `use crate::` group, then add
`commit`. The current crate uses (after task 06) are:
```rust
use crate::baseline::messages_with_carry;
use crate::config::Config;
use crate::llama::{ChatMessage, ChatRequest, Client};
use crate::session::Session;
use crate::turn::is_context_full;
```
They become
```rust
use crate::baseline::{messages_with_carry, Baseline};
use crate::config::Config;
use crate::llama::{ChatMessage, ChatRequest, Client};
use crate::memory::MemorySource;
use crate::session::Session;
use crate::tools::Registry;
use crate::turn::is_context_full;
```
And the bxcore use
```rust
use bxcore::{LogRecord, Usage};
```
becomes
```rust
use bxcore::{Epoch, LogRecord, Timestamp, Usage};
```
Then this function, whole, after `summarise`:
```rust
/// Append `EpochEnd` (unless it is already last), rebuild the baseline from `memory`, and
/// switch `session` onto the next epoch. When the current log already ends with `EpochEnd`,
/// that record's summary is the carry and the argument is ignored.
pub fn commit(
session: &mut Session,
cfg: &Config,
registry: &Registry,
memory: &dyn MemorySource,
summary: String,
) -> Result<(), CompactError> {
let (next, summary) = match session.records().last() {
Some(LogRecord::EpochEnd { next, summary, .. }) => (*next, summary.clone()),
_ => {
let next = Epoch(session.epoch().0.saturating_add(1));
session
.append(LogRecord::EpochEnd {
time: Timestamp::now(),
next,
summary: summary.clone(),
})
.map_err(|e| CompactError::Failed {
detail: e.to_string(),
})?;
(next, summary)
}
};
let block = memory.block().map_err(|e| CompactError::Failed { detail: e })?;
let baseline = Baseline::assemble_with(cfg, registry, &block).map_err(|e| {
CompactError::Failed {
detail: e.to_string(),
}
})?;
session
.begin_epoch(next, baseline, summary)
.map_err(|e| CompactError::Failed {
detail: e.to_string(),
})?;
Ok(())
}
```
If rustfmt splits the `map_err` closures, accept that.
## Steps
- [ ] **1. Copy the given test.**
```sh
cp docs/plans/M5d1/files/crates/loopd/tests/compact_advance.rs crates/loopd/tests/
```
- [ ] **2. See it fail.** `cargo build --tests --keep-going -p loopd`, timeout 600000 ms: test
target `compact_advance` does not compile (`E0599` no function `commit`). Every other target
compiles.
- [ ] **3. The edits.** S1, then C1. `cargo check -p loopd` prints no error and no warning.
`wc -l crates/loopd/src/session.rs` and `wc -l crates/loopd/src/compact.rs` each print a
number ≤ 500.
- [ ] **4. The tests.** `cargo test -p loopd --test compact_advance --test session_epochs --test
compact_summary`: every test passes, 6 in `compact_advance`, 9 in `session_epochs`, 6 in
`compact_summary`.
- [ ] **5. Check.** `git diff --stat` lists exactly `crates/loopd/src/session.rs` and
`crates/loopd/src/compact.rs`. `git status --short` shows `compact_advance.rs` as new.
- [ ] **6. Run the gate.** `cargo fmt --all`, then `make gate`, timeout 600000 ms. Last line
`gate: ok`. Test count **1501** (1495 before, plus 6).
- [ ] **7. Log and commit.** Task cell `M5d1/07-loopd-compact-advance`. Stage with
`git add crates/loopd docs/implementer-log.md Cargo.lock`, then commit as `AGENTS.md` says.
## Done when
- `make gate` prints `gate: ok` with 1501 tests.
- `git diff HEAD~1 --stat` lists only `crates/loopd/src/session.rs`,
`crates/loopd/src/compact.rs`, `crates/loopd/tests/compact_advance.rs` and the log.
## Stop and report if
- The branch check fails; a "before" block is not found exactly once; step 2 differs; `cargo
check` warns; a test fails; the gate count is not 1501; a file exceeds 500 lines; you write a
second `EpochEnd` on resume; you reset `slot` or `next_call`; you log the summary as a
`LogRecord::User`.
## What the given tests pin
`commit` writes `EpochEnd` on the old log, a new baseline file, and a new log whose
`SessionStart` has the next epoch; the new baseline contains a fresh memory block; reopen loads
the new epoch and its carry; a second commit leaves epoch 0 with one `EpochEnd` and writes the
next on epoch 1; resume after `EpochEnd` without next files uses that summary as carry and does
not write a second end; the new `SessionStart` keeps the live slot (including a non-default 7).
+261
View File
@@ -0,0 +1,261 @@
# M5d-1 task 08: compact after a turn, and retry once on `SessionFull`
**Branch:** `m5d1`. Check it: `git branch --show-current` prints `m5d1` and `git status --short`
is empty; otherwise stop. Never switch branch.
**Commit subject:** `loopd: compact after a full turn and retry SessionFull once`
**Log row:** the Task cell is `M5d1/08-loopd-compact-turn`.
## Goal
After `run_turn` returns, `finish_turn` decides whether to compact (spec 3.3):
- A successful turn whose last `Usage` is `due` is compacted before the caller returns. If
compact fails, the turn stays successful and the session stays on its epoch; `loopd` logs the
compact error with the runbook pointer.
- `TurnError::SessionFull` is compacted, then that user message is `run_turn`'d once in the new
epoch. If compact fails, the error stays `SessionFull`. The retry is not compacted again.
- Any other `TurnError` is returned as it is. No compact.
`Runtime` does not gain `MemorySource` (given `turn` tests build a `Runtime` without one).
`finish_turn` takes `memory` as its own argument.
The channel holds the session busy for this whole call: `handle` runs `finish_turn` **before**
`drop(held)`. A compact is not an idle timer and does not start after the busy set releases.
Live turns still send `slots.main` (task 03). Only the summary call uses the background slot.
## Every place that changes
| Place | What happens there |
|---|---|
| `crates/loopd/src/compact.rs` | `finish_turn` and its two helpers (edit F1) |
| `crates/loopd/src/channel.rs` | `handle` calls `finish_turn` before `drop(held)` (edits H1, H2) |
| `crates/loopd/tests/compact_turn.rs` | new, given |
| everything else | **no change**: not `turn.rs`, not `Runtime`, not existing `turn` / `channel` tests |
## Files
- Copy, from `docs/plans/M5d1/files/` (the same path under it): the new test
`crates/loopd/tests/compact_turn.rs`.
- Modify: `crates/loopd/src/compact.rs`, `crates/loopd/src/channel.rs`,
`docs/implementer-log.md`.
- Do not touch: any other file.
## Facts you need
- **`due` looks at `session.last_usage()`**, which is the current epoch's last `Usage`. After a
successful `run_turn` that record is there. After `SessionFull`, `run_turn` has already
appended the `User` and returned before any `Usage`; compact runs because of the error, not
because of `due`.
- **Compact helper.** `summarise` then `commit`. Log failures with
```text
loopd: could not compact session <id>: <CompactError Display>
```
`Display` already includes the runbook pointer, matching the runbook's example line.
- **Retry.** `run_turn` appends the user message again in the new epoch. The old epoch keeps its
`User` (Honcho skips it: no reply). `turns()` is 2. That is accepted (README, known limits).
- **Existing channel tests stay green.** Their `n_ctx` is 131072 and fixture `prompt_n` is tens,
so `due` is false. `SessionFull` without `compact.md` in `Home` stays `SessionFull` (the
successful-turn-when-compact-fails behaviour, on the error path).
- **`compact.rs` stays ≤ 500 lines.** After this task it is about 250.
- **`channel.rs` stays ≤ 500 lines.** After this task it is about 290.
- The items the edits use. This table is complete.
| Item | What it does |
|---|---|
| `due` | task 05 |
| `summarise` / `commit` | tasks 06, 07 |
| `run_turn` | retry the same `text` and `speaker` |
| `Session::last_usage` | current epoch |
| `Config.compact.threshold`, `expect.n_ctx` | the `due` arguments |
| `MemorySource` | `finish_turn`'s extra argument; channel passes `&*ctx.memory` |
| `Held` | already in `channel.rs`; drop it after `finish_turn` |
## The edits, exactly
Each quoted "before" block occurs exactly once after task 07 (`channel.rs` is still as on
master, except rustfmt).
### `crates/loopd/src/compact.rs`
**F1.** Extend the crate uses. After task 07 they include `MemorySource`, `Session`, `Config`,
`Registry`. Add:
```rust
use crate::turn::{Runtime, TurnError, TurnOutcome, run_turn};
use bxcore::TurnEvent;
```
Keep rustfmt's order of `use` lines; if it moves them, accept that.
Then after `commit`, add these three functions whole. There is no `todo!()`.
```rust
/// After `run_turn`: compact when the last `Usage` is due, or on `SessionFull` (then retry the
/// user message once). Compact failure on a successful turn keeps the success; on `SessionFull`
/// keeps `SessionFull`. Other errors skip compact.
pub fn finish_turn(
session: &mut Session,
rt: &Runtime<'_>,
memory: &dyn MemorySource,
text: &str,
speaker: Option<&str>,
on_event: &mut dyn FnMut(&TurnEvent),
outcome: Result<TurnOutcome, TurnError>,
) -> Result<TurnOutcome, TurnError> {
match outcome {
Err(TurnError::SessionFull) => match compact_session(session, rt, memory) {
Ok(()) => run_turn(session, rt, text, speaker, on_event),
Err(e) => {
log_compact(session, &e);
Err(TurnError::SessionFull)
}
},
Ok(done) => {
if let Some(usage) = session.last_usage()
&& due(&usage, rt.cfg.compact.threshold, rt.cfg.expect.n_ctx)
&& let Err(e) = compact_session(session, rt, memory)
{
log_compact(session, &e);
}
Ok(done)
}
Err(e) => Err(e),
}
}
fn compact_session(
session: &mut Session,
rt: &Runtime<'_>,
memory: &dyn MemorySource,
) -> Result<(), CompactError> {
let summary = summarise(rt.cfg, rt.client, session)?;
commit(session, rt.cfg, rt.registry, memory, summary)
}
fn log_compact(session: &Session, e: &CompactError) {
eprintln!(
"loopd: could not compact session {}: {e}",
session.id().as_str()
);
}
```
The `Ok(done)` arm uses `&& let Err(e) = ...` so a compact failure is logged and the original
`done` is still returned. Do not `?` that error.
### `crates/loopd/src/channel.rs`
**H1.** The `run_turn` call and the `drop(held)` that follows it. The lines
```rust
let outcome = run_turn(
&mut session,
&runtime,
&turn.content,
turn.speaker.as_deref(),
&mut |event| {
let _ = write_frame(
&mut stream,
&Envelope {
v: PROTOCOL_VERSION,
id: request.id,
r#final: false,
msg: Message::TurnEvent(event.clone()),
},
);
},
);
// 5. Release the session before the final frame, so the next turn does not find it busy.
drop(held);
```
become
```rust
let mut emit = |event: &bxcore::TurnEvent| {
let _ = write_frame(
&mut stream,
&Envelope {
v: PROTOCOL_VERSION,
id: request.id,
r#final: false,
msg: Message::TurnEvent(event.clone()),
},
);
};
let outcome = run_turn(
&mut session,
&runtime,
&turn.content,
turn.speaker.as_deref(),
&mut emit,
);
let outcome = crate::compact::finish_turn(
&mut session,
&runtime,
&*ctx.memory,
&turn.content,
turn.speaker.as_deref(),
&mut emit,
outcome,
);
// 5. Release the session before the final frame, so the next turn does not find it busy.
// Compaction ran inside the hold: a second turn on this session waits until the new
// epoch is on disk (or compact failed and the epoch is unchanged).
drop(held);
```
**H2.** Do not add `MemorySource` to `Runtime`. Do not change `turn_code`. The `memory.updated`
block after `drop(held)` stays: a successful compact-and-retry still reports `session.turns()`,
which now spans epochs.
## Steps
- [ ] **1. Copy the given test.**
```sh
cp docs/plans/M5d1/files/crates/loopd/tests/compact_turn.rs crates/loopd/tests/
```
- [ ] **2. See it fail.** `cargo build --tests --keep-going -p loopd`, timeout 600000 ms: test
target `compact_turn` does not compile (`E0599` no function `finish_turn`). Every other
target compiles, including `turn` and `channel`.
- [ ] **3. The edits.** F1, H1. `cargo check -p loopd` prints no error and no warning.
`wc -l crates/loopd/src/compact.rs` and `wc -l crates/loopd/src/channel.rs` each print a
number ≤ 500.
- [ ] **4. The tests.** `cargo test -p loopd --test compact_turn --test turn --test channel
--test compact_advance`: every test passes, 6 in `compact_turn` and the others as before.
- [ ] **5. Check.** `git diff --stat` lists exactly `crates/loopd/src/compact.rs` and
`crates/loopd/src/channel.rs`. `git status --short` shows `compact_turn.rs` as new.
- [ ] **6. Run the gate.** `cargo fmt --all`, then `make gate`, timeout 600000 ms. Last line
`gate: ok`. Test count **1507** (1501 before, plus 6).
- [ ] **7. Log and commit.** Task cell `M5d1/08-loopd-compact-turn`. Stage with
`git add crates/loopd docs/implementer-log.md Cargo.lock`, then commit as `AGENTS.md` says.
## Done when
- `make gate` prints `gate: ok` with 1507 tests.
- `git diff HEAD~1 --stat` lists only `crates/loopd/src/compact.rs`,
`crates/loopd/src/channel.rs`, `crates/loopd/tests/compact_turn.rs` and the log.
## Stop and report if
- The branch check fails; a "before" block is not found exactly once; step 2 differs; `cargo
check` warns; a test fails; the gate count is not 1507; a file exceeds 500 lines; you add
`MemorySource` to `Runtime`; you compact inside `run_turn`; you call `finish_turn` after
`drop(held)`; you add an idle timer; you compact a `TurnError` other than `SessionFull`; you
retry `SessionFull` more than once.
## What the given tests pin
Usage 849/1000 leaves epoch 0 (one chat); 850/1000 compact before return (live slot then
background, no tools on the summary); `SessionFull` compact-and-retry (three chats, user logged
in both epochs, `turns() == 2`); `SessionFull` stays `SessionFull` when compact fails; a
successful due turn stays successful when `compact.md` is missing (epoch stays 0); another
infer error is not compacted.
+119
View File
@@ -0,0 +1,119 @@
# M5d-1 implementation plan: epochs and compaction
> For the implementer: each task file is one closed task for one fresh session. Read `AGENTS.md`,
> then the task file, and do exactly that task. The driver is `tools/run-plan.sh`.
**Goal:** a live session can start a new epoch. `loopd` writes `N.jsonl` / `N.baseline.json` for
`N > 0`, turn numbers and `untrusted()` span every epoch, compaction runs after a turn whose last
`Usage` is at or above a configured fraction of `expect.n_ctx`, the summary call uses the
background slot, and `memoryd`'s Honcho walker reads every epoch file of a session.
**Architecture:** `loopd` gains a `[compact]` table and a `compact` module (when to compact, the
summary call, writing the next epoch). `Session::open` loads the highest complete epoch and the
carried summary. `messages` for epoch `n > 0` inserts that carry as a `ChatMessage::User` that is
never a `LogRecord::User`. Honcho concatenates epoch logs in order so `Item::Turn` numbers stay
monotonic. Heartbeat, cron, `ToolSummary`, and tiering are M5d-2.
**Tech stack:** Rust 1.98.1, edition 2024; no new crate.
**Spec:** `docs/specs/2026-10-03-m5d-scheduler.md`, sections 3, 5 (`[compact]` without using
`summarise`), 7 (the M5d-1 tests), and 8 (`session-compact-failed`).
## The owner's decisions (2026-10-03)
1. Compaction runs after a turn, when that turn's `cache_n + prompt_n` is at or above a configured
fraction of `expect.n_ctx` (default 0.85). A quiet session is not a reason to compact.
2. If a live request is `exceed_context_size_error`, compact and retry that user message once in
the new epoch.
3. The summary call is on the background slot, with no tools, thinking on. A context-full summary
falls back to the epoch's `User` texts, newest last, cut to `summary_chars`.
4. The carried summary is not a `LogRecord::User`, so it is not a Honcho turn.
## What the code already has, and what it lacks
- `LogRecord::EpochEnd { time, next, summary }` is in `bxcore` and unused. `messages` already
ignores it. This plan writes it.
- `Session::open` reads only `0.baseline.json` and `0.jsonl`. `turns()` and `untrusted()` look at
that file only. `next_call` is recovered from that file only.
- `run_turn` always sends `slots.main`. Live turns stay on the session's live slot (main, from
`SessionStart`). Only the summary call uses `slots.background`. Changing live turns to the
session's slot is M5d-2.
- `memoryd`'s indexer already walks every `<epoch>.jsonl`. Honcho's `log_path` is still
`sessions/<id>/0.jsonl`.
- `Runtime` has no `MemorySource`. Compaction asks `memoryd` for a fresh block from `channel.rs`
(and `compact::finish_turn`), so given `turn` tests that build a `Runtime` do not change.
- `[compact] summarise` is stored from this plan so a spec-complete `loopd.toml` loads; nothing
in M5d-1 reads the list. M5d-2 opts in `http_fetch`.
## Global constraints
- Roles depend on `bxcore` only. No `unsafe`, no async runtime, no new crate.
- No file the implementer reads holds a chat-template tag whole (T33). The carry heading is built
from pieces and named in words.
- Every path a given test creates is removed by a guard (T44); every process a test starts has a
deadline.
- The fail-closed state `session-compact-failed` has a runbook entry, added by the task that adds
its pointer (T49).
- Each task gives its baseline and its gate count with what the count includes (T50, T53); the
branch line is a check, never `git switch` (T51).
- `Cargo.lock` is in the `git add` line (T24). Gate timeout 600000 ms (T40).
- Commit trailer from the driver.
## Tasks
Every task below has been written to be replayed from its text. The gate count is the passed
total of `cargo test --workspace` plus one, for the ignored `end_to_end` test the gate runs on
its own. Before task 01 it is **1456** (talos, rustc 1.98.1, clippy of the same release; master
`069cd16`, docs-only commits after the M5c-2 review).
| # | Delivers | Crates | Gate after | Tests added |
|---|---|---|---|---|
| 01 | `[compact]` on `loopd.toml`: `threshold` 0.85, `summary_chars` 4000, `summarise` default `["http_fetch"]` (stored, unused); `Config::compact_prompt` is `compact.md` beside `system.md` | loopd | 1464 | 8 |
| 02 | `Session::open` loads the highest epoch that has both a baseline and a log, the carry from the previous `EpochEnd`, `turns` / `untrusted` / `next_call` over every epoch; persist writes the current epoch | loopd | 1473 | 9 |
| 03 | `messages_with_carry`: epoch `n > 0` is the new system message, then one `ChatMessage::User` (fixed heading plus carry, not a log record), then the new epoch's records; `run_turn` uses `session.messages()` | loopd | 1478 | 5 |
| 04 | Honcho concatenates every `<epoch>.jsonl` in epoch order; `turn_records` of turn 3 is the epoch-1 user; `send_turn` reads the concatenated log | memoryd | 1483 | 5 |
| 05 | `loopd::compact`: `due`, `fallback_summary`; starter `config/compact.md` | loopd | 1489 | 6 |
| 06 | Summary call on the background slot, no tools, thinking on; tag-neutralised and capped; context-full uses the fallback; `session-compact-failed` pointer and runbook entry | loopd | 1495 | 6 |
| 07 | Writing the next epoch: `EpochEnd`, fresh baseline from `memoryd`, `SessionStart` on the live slot; resume if `EpochEnd` is already the last record | loopd | 1501 | 6 |
| 08 | After a turn: compact when `due`; on `SessionFull`, compact and retry the user message once; compaction holds the session busy | loopd | 1507 | 6 |
After task 08, the design model runs the M5d-1 device check (spec 7) on straylight: a `bxctl`
session writes a distinctive fact into `memory.md`, continues until compaction with
`[compact] threshold` lowered, and the next turn's baseline contains the fact; `recall` of a
phrase from epoch 0 still hits; `TurnDone.untrusted` stays true across the epoch change after a
tainted recall.
## Known limits
These are known, accepted for M5d-1, and not bugs to fix in a task:
- **Live turns still use `slots.main`.** `SessionStart.slot` is recorded and copied onto the new
epoch; `run_turn` does not read it yet. Isolated sessions on the background slot are M5d-2.
- **`[compact] summarise` is unused.** Unknown tool names are not a config error until M5d-2.
- **No idle path exists.** Compaction is only called after a turn or on `SessionFull`. There is
no timer to test.
- **A crash after `EpochEnd` and before the next files exist** reopens epoch 0 with `EpochEnd` at
the end of the log. The next compact resumes without writing a second `EpochEnd` (task 07).
- **A `SessionFull` retry logs the user message twice**, once in the old epoch (Honcho skips it:
no reply) and once in the new epoch. Turn numbers stay monotonic.
## Before hand-over
Work through `docs/plans/CHECKLIST.md` with a reference in a scratch worktree with its own build
directory, replayed from its task text; mutations; `/tmp` compared around a gate run; scratch
files in the worktree's `.state/`; the reference moved to straylight as a bundle before the run
(T18). Task files name their log row (T47).
## Running it
The reference (`m5d1-ref`, tasks 01 to 08) is moved to straylight as a bundle and deleted here
before the run (T18). Then, from the repository root, with the OpenCode TUI closed:
```sh
git switch -c m5d1 master
OPENCODE_MODEL=straylight/ornith-1.5-35b-a3b RESUME=1 \
SLOTS_URL='http://straylight:11434/slots?model=ornith-1.5-35b-a3b' \
tools/run-plan.sh docs/plans/M5d1 01
```
After task 08, the gate count is 1507. Then the device check above.
+8
View File
@@ -0,0 +1,8 @@
Summarise this conversation for the next epoch of the same session.
Write a short account of:
- what the owner asked for and decided
- facts that were established
- open threads and anything still waiting
Do not use tools. Do not address the owner. Do not invent facts that are not in the messages.
@@ -0,0 +1,120 @@
//! Tests for the carried summary in the next epoch's prefix. Do not edit.
use bxcore::{Epoch, Hash32, LogRecord, SessionId, Timestamp};
use loopd::baseline::{Baseline, CARRY_HEADING, messages, messages_with_carry};
use loopd::llama::ChatMessage;
fn ts() -> Timestamp {
Timestamp::parse("2026-09-18T08:00:00.000Z").unwrap()
}
fn user(content: &str) -> LogRecord {
LogRecord::User {
time: ts(),
content: content.to_string(),
speaker: None,
}
}
fn start(epoch: u32) -> LogRecord {
LogRecord::SessionStart {
time: ts(),
session: SessionId::new("a").unwrap(),
epoch: Epoch(epoch),
slot: 0,
baseline: Hash32::ZERO,
}
}
#[test]
fn epoch_0_matches_messages_without_a_carry() {
let b = Baseline {
system: "sys".to_string(),
tools: vec![],
};
let records = vec![start(0), user("hi")];
assert_eq!(
messages_with_carry(&b, &records, None),
messages(&b, &records)
);
}
#[test]
fn a_carry_is_a_user_message_after_the_system_prompt() {
let b = Baseline {
system: "new-sys".to_string(),
tools: vec![],
};
let records = vec![start(1), user("next")];
let got = messages_with_carry(&b, &records, Some("the summary"));
assert_eq!(
got,
vec![
ChatMessage::System {
content: "new-sys".to_string(),
},
ChatMessage::User {
content: format!("{CARRY_HEADING}\n\nthe summary"),
},
ChatMessage::User {
content: "next".to_string(),
},
]
);
}
#[test]
fn the_carry_is_not_taken_from_a_log_user_record() {
let b = Baseline {
system: "sys".to_string(),
tools: vec![],
};
let records = vec![start(1)];
let got = messages_with_carry(&b, &records, Some("carried"));
assert_eq!(got.len(), 2);
assert!(
!records.iter().any(|r| matches!(r, LogRecord::User { .. })),
"the carry is not in the log"
);
}
#[test]
fn the_heading_is_built_from_pieces() {
assert_eq!(CARRY_HEADING, concat!("Earlier ", "turns, summarised:"));
assert!(
!CARRY_HEADING.contains('<'),
"the heading is words, not a tag"
);
}
#[test]
fn epoch_1_is_not_an_extension_of_epoch_0() {
let old = Baseline {
system: "old-sys".to_string(),
tools: vec![],
};
let new = Baseline {
system: "new-sys".to_string(),
tools: vec![],
};
let epoch0 = vec![start(0), user("first")];
let epoch1 = vec![start(1), user("second")];
let prev = messages(&old, &epoch0);
let next = messages_with_carry(&new, &epoch1, Some("sum of first"));
assert!(
next.len() >= 2,
"system plus carry, then the new epoch's user"
);
assert_ne!(next[0], prev[0], "the system message changed");
assert!(
!next.starts_with(&prev[..]),
"the new prefix is not an extension of the old messages"
);
match &next[1] {
ChatMessage::User { content } => {
assert!(content.contains("sum of first"), "{content}");
assert!(!content.contains("second"), "{content}");
}
other => panic!("{other:?}"),
}
}
@@ -0,0 +1,201 @@
//! Tests for writing the next epoch. Do not edit.
mod support;
use bxcore::{Epoch, LogRecord, SessionId};
use loopd::baseline::Baseline;
use loopd::compact;
use loopd::memory::{MemorySource, NoMemory};
use loopd::session::Session;
use loopd::tools::Registry;
use support::Home;
struct Block(&'static str);
impl MemorySource for Block {
fn block(&self) -> Result<String, String> {
Ok(self.0.to_string())
}
fn updated(&self, _session: &SessionId, _turn: u64) -> Result<(), String> {
Ok(())
}
}
fn id(s: &str) -> SessionId {
SessionId::new(s).unwrap()
}
fn session(home: &Home) -> (loopd::config::Config, Session) {
let cfg = home.config(std::path::Path::new("/tmp/unused.sock"));
let baseline = Baseline::assemble(&cfg, &Registry::m2b()).unwrap();
let s = Session::create(&home.dir, id("a"), baseline, cfg.slots.main).unwrap();
(cfg, s)
}
#[test]
fn commit_writes_epoch_end_a_new_baseline_and_a_new_log() {
let home = Home::new();
let (cfg, mut s) = session(&home);
s.append(LogRecord::User {
time: bxcore::Timestamp::parse("2026-09-18T08:00:00.000Z").unwrap(),
content: "hi".to_string(),
speaker: None,
})
.unwrap();
compact::commit(
&mut s,
&cfg,
&Registry::m2b(),
&NoMemory,
"the sum".to_string(),
)
.unwrap();
assert_eq!(s.epoch(), Epoch(1));
assert_eq!(s.carry(), Some("the sum"));
assert_eq!(s.slot(), cfg.slots.main);
assert_eq!(s.turns(), 1, "the old user still counts");
assert_eq!(
s.records()
.iter()
.filter(|r| matches!(r, LogRecord::User { .. }))
.count(),
0
);
let old = home.records("a");
assert!(
matches!(old.last(), Some(LogRecord::EpochEnd { next, summary, .. }) if *next == Epoch(1) && summary == "the sum")
);
assert!(home.dir.join("sessions/a/1.baseline.json").is_file());
let new_log = home.read("sessions/a/1.jsonl");
assert!(new_log.contains("\"epoch\":1"), "{new_log}");
}
#[test]
fn the_new_baseline_takes_a_fresh_memory_block() {
let home = Home::new();
let (cfg, mut s) = session(&home);
compact::commit(
&mut s,
&cfg,
&Registry::m2b(),
&Block("I take my tea strong."),
"sum".to_string(),
)
.unwrap();
assert!(
s.baseline().system.contains("I take my tea strong."),
"{}",
s.baseline().system
);
let on_disk = home.read("sessions/a/1.baseline.json");
assert!(on_disk.contains("I take my tea strong."), "{on_disk}");
}
#[test]
fn reopen_loads_the_new_epoch() {
let home = Home::new();
let (cfg, mut s) = session(&home);
compact::commit(
&mut s,
&cfg,
&Registry::m2b(),
&NoMemory,
"carry".to_string(),
)
.unwrap();
drop(s);
let s = Session::open(&home.dir, id("a")).unwrap();
assert_eq!(s.epoch(), Epoch(1));
assert_eq!(s.carry(), Some("carry"));
}
#[test]
fn a_second_commit_does_not_write_another_epoch_end_on_the_old_log() {
let home = Home::new();
let (cfg, mut s) = session(&home);
compact::commit(
&mut s,
&cfg,
&Registry::m2b(),
&NoMemory,
"first".to_string(),
)
.unwrap();
let ends = home
.records("a")
.iter()
.filter(|r| matches!(r, LogRecord::EpochEnd { .. }))
.count();
assert_eq!(ends, 1);
compact::commit(
&mut s,
&cfg,
&Registry::m2b(),
&NoMemory,
"second".to_string(),
)
.unwrap();
assert_eq!(s.epoch(), Epoch(2));
let ends = home
.records("a")
.iter()
.filter(|r| matches!(r, LogRecord::EpochEnd { .. }))
.count();
assert_eq!(ends, 1, "epoch 0 still has one EpochEnd");
let mid = home.read("sessions/a/1.jsonl");
assert!(mid.contains("\"type\":\"epoch_end\""), "{mid}");
}
#[test]
fn resume_after_epoch_end_without_next_files_does_not_write_a_second_end() {
let home = Home::new();
let (cfg, mut s) = session(&home);
s.append(LogRecord::EpochEnd {
time: bxcore::Timestamp::now(),
next: Epoch(1),
summary: "already".to_string(),
})
.unwrap();
drop(s);
let mut s = Session::open(&home.dir, id("a")).unwrap();
assert_eq!(s.epoch(), Epoch(0));
compact::commit(
&mut s,
&cfg,
&Registry::m2b(),
&NoMemory,
"ignored".to_string(),
)
.unwrap();
let ends = home
.records("a")
.iter()
.filter(|r| matches!(r, LogRecord::EpochEnd { .. }))
.count();
assert_eq!(ends, 1);
assert_eq!(s.epoch(), Epoch(1));
assert_eq!(
s.carry(),
Some("already"),
"the summary already in EpochEnd is the carry"
);
}
#[test]
fn the_new_session_start_keeps_the_live_slot() {
let home = Home::new();
let cfg = home.config(std::path::Path::new("/tmp/unused.sock"));
let baseline = Baseline::assemble(&cfg, &Registry::m2b()).unwrap();
let mut s = Session::create(&home.dir, id("a"), baseline, 7).unwrap();
compact::commit(&mut s, &cfg, &Registry::m2b(), &NoMemory, "s".to_string()).unwrap();
assert_eq!(s.slot(), 7);
match &s.records()[0] {
LogRecord::SessionStart { slot, epoch, .. } => {
assert_eq!(*slot, 7);
assert_eq!(*epoch, Epoch(1));
}
other => panic!("{other:?}"),
}
}
@@ -0,0 +1,79 @@
//! Tests for when compaction is due and for the fallback summary. Do not edit.
use bxcore::{Epoch, Hash32, LogRecord, SessionId, Timestamp, Usage};
use loopd::compact::{due, fallback_summary};
fn usage(cache_n: u64, prompt_n: u64) -> Usage {
Usage {
cache_n,
prompt_n,
predicted_n: 0,
reasoning_tokens: 0,
thinking_capped: false,
}
}
fn user(content: &str) -> LogRecord {
LogRecord::User {
time: Timestamp::parse("2026-09-18T08:00:00.000Z").unwrap(),
content: content.to_string(),
speaker: None,
}
}
#[test]
fn due_at_the_threshold_and_not_below() {
assert!(!due(&usage(400, 449), 0.85, 1_000), "849 < 850");
assert!(due(&usage(400, 450), 0.85, 1_000), "850 == 0.85 * 1000");
assert!(due(&usage(0, 851), 0.85, 1_000));
assert!(!due(&usage(0, 0), 0.85, 1_000));
}
#[test]
fn due_uses_this_agents_n_ctx() {
// 0.85 * 262144 = 222822.4, so 222823 is the first integer at or above.
assert!(!due(&usage(0, 222_822), 0.85, 262_144));
assert!(due(&usage(0, 222_823), 0.85, 262_144));
assert!(due(&usage(200_000, 22_823), 0.85, 262_144));
}
#[test]
fn a_threshold_of_one_is_the_full_window() {
assert!(!due(&usage(99, 0), 1.0, 100));
assert!(due(&usage(100, 0), 1.0, 100));
}
#[test]
fn fallback_is_user_texts_newest_last() {
let records = vec![
LogRecord::SessionStart {
time: Timestamp::parse("2026-09-18T08:00:00.000Z").unwrap(),
session: SessionId::new("a").unwrap(),
epoch: Epoch(0),
slot: 0,
baseline: Hash32::ZERO,
},
user("first"),
user("second"),
user("third"),
];
assert_eq!(
fallback_summary(&records, 4_000),
"third\n\nsecond\n\nfirst"
);
}
#[test]
fn fallback_is_cut_to_summary_chars_on_a_character_boundary() {
let records = vec![user("café"), user("ab")];
assert_eq!(fallback_summary(&records, 3), "ab\n");
assert_eq!(fallback_summary(&records, 1), "a");
assert_eq!(fallback_summary(&records, 0), "");
let long = user("ååå");
assert_eq!(fallback_summary(&[long], 2), "åå");
}
#[test]
fn fallback_with_no_users_is_empty() {
assert_eq!(fallback_summary(&[], 100), "");
}
@@ -0,0 +1,147 @@
//! Tests for the compaction summary call. Do not edit.
mod support;
use bxcore::SessionId;
use loopd::baseline::Baseline;
use loopd::compact::{self, CompactError};
use loopd::llama::Client;
use loopd::session::Session;
use loopd::tools::Registry;
use support::{FakeServer, Home, Reply};
const CHAT: &str = "/v1/chat/completions";
const PROMPT: &str = "Summarise this conversation for the next epoch.\n";
fn sse(content: &str, cache_n: u64, prompt_n: u64) -> Reply {
let content_json = serde_json::to_string(content).unwrap();
let data1 = format!(
r#"data: {{"choices":[{{"finish_reason":null,"index":0,"delta":{{"content":{content_json}}}}}],"id":"sum","timings":{{"cache_n":{cache_n},"prompt_n":{prompt_n},"predicted_n":1}}}}"#
);
let data2 = format!(
r#"data: {{"choices":[{{"finish_reason":"stop","index":0,"delta":{{}}}}],"id":"sum","timings":{{"cache_n":{cache_n},"prompt_n":{prompt_n},"predicted_n":1}}}}"#
);
let body = format!("{data1}\n\n{data2}\n\ndata: [DONE]\n\n");
Reply::raw(format!(
"HTTP/1.1 200 OK\r\nContent-Type: text/event-stream\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}",
body.len()
))
}
fn session(home: &Home, cfg: &loopd::config::Config) -> Session {
let baseline = Baseline::assemble(cfg, &Registry::m2b()).unwrap();
Session::create(
&home.dir,
SessionId::new("a").unwrap(),
baseline,
cfg.slots.main,
)
.unwrap()
}
#[test]
fn the_summary_call_is_on_the_background_slot_with_no_tools_and_thinking() {
let home = Home::new();
home.write("compact.md", PROMPT);
let server = FakeServer::start();
let cfg = home.config(&server.socket);
server.route(CHAT, vec![sse("the sum", 0, 10)]);
let client = Client::new(cfg.clone());
let s = session(&home, &cfg);
let text = compact::summarise(&cfg, &client, &s).unwrap();
assert_eq!(text, "the sum");
let sent = server.requests_to(CHAT);
assert_eq!(sent.len(), 1);
let body = sent[0].json();
assert_eq!(body["id_slot"], 1);
assert!(body.get("tools").is_none(), "{body}");
assert_eq!(body["chat_template_kwargs"]["enable_thinking"], true);
let messages = body["messages"].as_array().unwrap();
let last = messages.last().unwrap();
assert_eq!(last["role"], "user");
assert_eq!(last["content"], PROMPT.trim_end());
}
#[test]
fn the_summary_is_neutralised_and_capped() {
let home = Home::new();
home.write("compact.md", PROMPT);
let server = FakeServer::start();
let mut cfg = home.config(&server.socket);
cfg.compact.summary_chars = 4;
server.route(CHAT, vec![sse("abcdefghij", 0, 10)]);
let client = Client::new(cfg.clone());
let s = session(&home, &cfg);
let text = compact::summarise(&cfg, &client, &s).unwrap();
assert_eq!(text, "abcd");
}
#[test]
fn a_context_full_summary_uses_the_fallback() {
let home = Home::new();
home.write("compact.md", PROMPT);
let server = FakeServer::start();
let cfg = home.config(&server.socket);
server.route(CHAT, vec![Reply::fixture("context_full")]);
let client = Client::new(cfg.clone());
let mut s = session(&home, &cfg);
s.append(bxcore::LogRecord::User {
time: bxcore::Timestamp::parse("2026-09-18T08:00:00.000Z").unwrap(),
content: "remember the tea".to_string(),
speaker: None,
})
.unwrap();
let text = compact::summarise(&cfg, &client, &s).unwrap();
assert_eq!(text, "remember the tea");
}
#[test]
fn tags_in_the_completion_are_neutralised() {
let home = Home::new();
home.write("compact.md", PROMPT);
let server = FakeServer::start();
let cfg = home.config(&server.socket);
let tag = concat!("<", "think", ">");
server.route(CHAT, vec![sse(&format!("hello {tag} world"), 0, 10)]);
let client = Client::new(cfg.clone());
let s = session(&home, &cfg);
let text = compact::summarise(&cfg, &client, &s).unwrap();
assert!(!text.contains(&tag), "{text}");
assert!(text.contains("hello"), "{text}");
}
#[test]
fn a_missing_prompt_file_names_the_runbook() {
let home = Home::new();
let server = FakeServer::start();
let cfg = home.config(&server.socket);
let client = Client::new(cfg.clone());
let s = session(&home, &cfg);
let err = compact::summarise(&cfg, &client, &s).unwrap_err();
let e: Box<dyn std::error::Error> = Box::new(err);
assert!(e.to_string().contains("compact.md"), "{e}");
assert!(
e.to_string()
.ends_with("see docs/runbook.md#session-compact-failed"),
"{e}"
);
}
#[test]
fn an_inference_error_that_is_not_context_full_is_a_compact_failure() {
let home = Home::new();
home.write("compact.md", PROMPT);
let server = FakeServer::start();
let cfg = home.config(&server.socket);
server.route(CHAT, vec![Reply::fixture("bad_request")]);
let client = Client::new(cfg.clone());
let s = session(&home, &cfg);
let err = compact::summarise(&cfg, &client, &s).unwrap_err();
assert!(matches!(err, CompactError::Failed { .. }), "{err:?}");
let e: Box<dyn std::error::Error> = Box::new(err);
assert!(
e.to_string()
.ends_with("see docs/runbook.md#session-compact-failed"),
"{e}"
);
}
@@ -0,0 +1,257 @@
//! Tests for compacting after a turn and retrying a full context. Do not edit.
mod support;
use bxcore::{Epoch, LogRecord, SessionId, TurnEvent};
use loopd::baseline::Baseline;
use loopd::compact;
use loopd::llama::Client;
use loopd::memory::NoMemory;
use loopd::session::Session;
use loopd::tools::Registry;
use loopd::turn::{Runtime, TurnError, run_turn};
use support::{FakeServer, Home, Reply};
const CHAT: &str = "/v1/chat/completions";
const PROMPT: &str = "Summarise this conversation for the next epoch.\n";
fn sse(content: &str, cache_n: u64, prompt_n: u64) -> Reply {
let content_json = serde_json::to_string(content).unwrap();
let data1 = format!(
r#"data: {{"choices":[{{"finish_reason":null,"index":0,"delta":{{"content":{content_json}}}}}],"id":"t","timings":{{"cache_n":{cache_n},"prompt_n":{prompt_n},"predicted_n":1}}}}"#
);
let data2 = format!(
r#"data: {{"choices":[{{"finish_reason":"stop","index":0,"delta":{{}}}}],"id":"t","timings":{{"cache_n":{cache_n},"prompt_n":{prompt_n},"predicted_n":1}}}}"#
);
let body = format!("{data1}\n\n{data2}\n\ndata: [DONE]\n\n");
Reply::raw(format!(
"HTTP/1.1 200 OK\r\nContent-Type: text/event-stream\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}",
body.len()
))
}
struct World {
home: Home,
server: FakeServer,
cfg: loopd::config::Config,
client: Client,
port: support::ScriptedPort,
registry: Registry,
}
impl World {
fn new() -> World {
let home = Home::new();
home.write("compact.md", PROMPT);
let server = FakeServer::start();
let mut cfg = home.config(&server.socket);
cfg.expect.n_ctx = 1_000;
cfg.compact.threshold = 0.85;
let client = Client::new(cfg.clone());
World {
home,
server,
cfg,
client,
port: support::ScriptedPort::new(vec![]),
registry: Registry::m2b(),
}
}
fn session(&self) -> Session {
let baseline = Baseline::assemble(&self.cfg, &self.registry).unwrap();
Session::create(
&self.home.dir,
SessionId::new("a").unwrap(),
baseline,
self.cfg.slots.main,
)
.unwrap()
}
fn runtime(&self) -> Runtime<'_> {
Runtime {
cfg: &self.cfg,
client: &self.client,
port: &self.port,
registry: &self.registry,
}
}
fn finish(
&self,
session: &mut Session,
text: &str,
outcome: Result<loopd::turn::TurnOutcome, TurnError>,
) -> Result<loopd::turn::TurnOutcome, TurnError> {
let mut events = Vec::new();
compact::finish_turn(
session,
&self.runtime(),
&NoMemory,
text,
None,
&mut |e: &TurnEvent| events.push(e.clone()),
outcome,
)
}
}
#[test]
fn usage_under_the_threshold_leaves_the_epoch() {
let w = World::new();
w.server.route(CHAT, vec![sse("ok", 400, 449)]);
let mut s = w.session();
let mut events = Vec::new();
let outcome = run_turn(&mut s, &w.runtime(), "hi", None, &mut |e| {
events.push(e.clone())
});
let outcome = w.finish(&mut s, "hi", outcome).unwrap();
assert_eq!(outcome.content, "ok");
assert_eq!(s.epoch(), Epoch(0));
assert_eq!(w.server.requests_to(CHAT).len(), 1);
}
#[test]
fn usage_at_the_threshold_compacts_before_the_turn_returns() {
let w = World::new();
w.server
.route(CHAT, vec![sse("ok", 400, 450), sse("the sum", 0, 10)]);
let mut s = w.session();
let mut events = Vec::new();
let outcome = run_turn(&mut s, &w.runtime(), "hi", None, &mut |e| {
events.push(e.clone())
});
w.finish(&mut s, "hi", outcome).unwrap();
assert_eq!(s.epoch(), Epoch(1));
assert_eq!(s.carry(), Some("the sum"));
let sent = w.server.requests_to(CHAT);
assert_eq!(sent.len(), 2);
assert_eq!(sent[0].json()["id_slot"], 0);
assert_eq!(sent[1].json()["id_slot"], 1);
assert!(sent[1].json().get("tools").is_none());
let old = w.home.records("a");
assert!(matches!(
old.last(),
Some(LogRecord::EpochEnd { next, .. }) if *next == Epoch(1)
));
}
#[test]
fn session_full_compacts_and_retries_the_user_message_once() {
let w = World::new();
w.server.route(
CHAT,
vec![
Reply::fixture("context_full"),
sse("the sum", 0, 10),
sse("retried", 0, 20),
],
);
let mut s = w.session();
let mut events = Vec::new();
let outcome = run_turn(&mut s, &w.runtime(), "please fit", None, &mut |e| {
events.push(e.clone())
});
assert!(matches!(outcome, Err(TurnError::SessionFull)));
let done = w.finish(&mut s, "please fit", outcome).unwrap();
assert_eq!(done.content, "retried");
assert_eq!(s.epoch(), Epoch(1));
let users: Vec<_> = s
.records()
.iter()
.filter_map(|r| match r {
LogRecord::User { content, .. } => Some(content.as_str()),
_ => None,
})
.collect();
assert_eq!(users, ["please fit"]);
let old_users = w
.home
.records("a")
.iter()
.filter(|r| matches!(r, LogRecord::User { .. }))
.count();
assert_eq!(old_users, 1);
assert_eq!(s.turns(), 2);
let sent = w.server.requests_to(CHAT);
assert_eq!(sent.len(), 3);
assert_eq!(sent[1].json()["id_slot"], 1);
assert_eq!(sent[2].json()["id_slot"], 0);
}
#[test]
fn session_full_stays_session_full_when_compact_fails() {
let w = World::new();
w.server.route(
CHAT,
vec![
Reply::fixture("context_full"),
Reply::fixture("bad_request"),
],
);
let mut s = w.session();
let mut events = Vec::new();
let outcome = run_turn(&mut s, &w.runtime(), "x", None, &mut |e| {
events.push(e.clone())
});
let err = w.finish(&mut s, "x", outcome).unwrap_err();
assert!(matches!(err, TurnError::SessionFull), "{err:?}");
assert_eq!(s.epoch(), Epoch(0));
}
#[test]
fn a_successful_turn_stays_successful_when_compact_fails() {
let home = Home::new();
let server = FakeServer::start();
let mut cfg = home.config(&server.socket);
cfg.expect.n_ctx = 1_000;
cfg.compact.threshold = 0.85;
let client = Client::new(cfg.clone());
server.route(CHAT, vec![sse("ok", 400, 450)]);
let baseline = Baseline::assemble(&cfg, &Registry::m2b()).unwrap();
let mut s = Session::create(
&home.dir,
SessionId::new("a").unwrap(),
baseline,
cfg.slots.main,
)
.unwrap();
let port = support::ScriptedPort::new(vec![]);
let registry = Registry::m2b();
let rt = Runtime {
cfg: &cfg,
client: &client,
port: &port,
registry: &registry,
};
let mut events = Vec::new();
let outcome = run_turn(&mut s, &rt, "hi", None, &mut |e| events.push(e.clone()));
let done = compact::finish_turn(
&mut s,
&rt,
&NoMemory,
"hi",
None,
&mut |e| events.push(e.clone()),
outcome,
)
.unwrap();
assert_eq!(done.content, "ok");
assert_eq!(s.epoch(), Epoch(0), "compact.md is missing, so no epoch");
}
#[test]
fn finish_turn_does_not_compact_an_error_other_than_session_full() {
let w = World::new();
w.server.route(CHAT, vec![Reply::fixture("bad_request")]);
let mut s = w.session();
let mut events = Vec::new();
let outcome = run_turn(&mut s, &w.runtime(), "x", None, &mut |e| {
events.push(e.clone())
});
let err = w.finish(&mut s, "x", outcome).unwrap_err();
assert!(matches!(err, TurnError::Infer(_)), "{err:?}");
assert_eq!(s.epoch(), Epoch(0));
assert_eq!(w.server.requests_to(CHAT).len(), 1);
}
@@ -0,0 +1,119 @@
//! Tests for `[compact]` on `loopd.toml`. Do not edit.
use loopd::config::Config;
use std::path::{Path, PathBuf};
fn fixture(name: &str) -> PathBuf {
Path::new(env!("CARGO_MANIFEST_DIR"))
.join("tests/fixtures/config")
.join(name)
}
fn minimal() -> String {
std::fs::read_to_string(fixture("minimal.toml")).unwrap()
}
#[test]
fn missing_table_gets_the_documented_defaults() {
let c = Config::load(&fixture("minimal.toml")).unwrap();
assert_eq!(c.compact.threshold, 0.85);
assert_eq!(c.compact.summary_chars, 4_000);
assert_eq!(c.compact.summarise, vec!["http_fetch".to_string()]);
}
#[test]
fn the_table_overrides_every_default() {
let text = format!(
"{}\n[compact]\nthreshold = 0.5\nsummary_chars = 100\nsummarise = [\"clock\"]\n",
minimal()
);
let c = Config::parse(&text).unwrap();
assert_eq!(c.compact.threshold, 0.5);
assert_eq!(c.compact.summary_chars, 100);
assert_eq!(c.compact.summarise, vec!["clock".to_string()]);
}
#[test]
fn a_partial_table_keeps_the_other_defaults() {
let text = format!("{}\n[compact]\nthreshold = 0.9\n", minimal());
let c = Config::parse(&text).unwrap();
assert_eq!(c.compact.threshold, 0.9);
assert_eq!(c.compact.summary_chars, 4_000);
assert_eq!(c.compact.summarise, vec!["http_fetch".to_string()]);
}
#[test]
fn unknown_keys_are_errors() {
let text = minimal();
let bad = format!("{text}\n[compact]\nzz_unknown = 1\n");
assert!(
Config::parse(&bad).is_err(),
"[compact] accepted an unknown key"
);
let misspelt = format!("{text}\n[compact]\nsummary_char = 10\n");
assert!(
Config::parse(&misspelt).is_err(),
"[compact] accepted a misspelt key"
);
}
#[test]
fn empty_summarise_is_allowed() {
let text = format!("{}\n[compact]\nsummarise = []\n", minimal());
let c = Config::parse(&text).unwrap();
assert!(c.compact.summarise.is_empty());
}
#[test]
fn compact_prompt_is_compact_md_beside_system_md() {
let c = Config::load(&fixture("minimal.toml")).unwrap();
assert_eq!(c.compact_prompt(), fixture("compact.md"));
let home = Scratch::new();
let cfg_path = home.dir.join("loopd.toml");
std::fs::write(&cfg_path, minimal()).unwrap();
std::fs::write(home.dir.join("system.md"), "sys\n").unwrap();
let c = Config::load(&cfg_path).unwrap();
assert_eq!(c.compact_prompt(), home.dir.join("compact.md"));
}
struct Scratch {
dir: PathBuf,
}
impl Scratch {
fn new() -> Scratch {
let dir = std::env::temp_dir().join(format!(
"loopd-compact-cfg-{}-{}",
std::process::id(),
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap()
.as_nanos()
));
std::fs::create_dir_all(&dir).unwrap();
Scratch { dir }
}
}
impl Drop for Scratch {
fn drop(&mut self) {
let _ = std::fs::remove_dir_all(&self.dir);
}
}
#[test]
fn compact_prompt_follows_an_overridden_system_path() {
let text = std::fs::read_to_string(fixture("m2b.toml")).unwrap();
let c = Config::load(&fixture("m2b.toml")).unwrap();
assert!(
text.contains("prompts/agent.md"),
"m2b fixture names a prompt in a subdirectory"
);
assert_eq!(c.compact_prompt(), fixture("prompts/compact.md"));
}
#[test]
fn defaults_match_the_struct_default() {
let c = Config::load(&fixture("minimal.toml")).unwrap();
assert_eq!(c.compact, loopd::config::Compact::default());
}
@@ -0,0 +1,255 @@
//! Tests for multi-epoch session open, turns, trust and call ids. Do not edit.
mod support;
use bxcore::{CallId, DataClass, Epoch, LogRecord, SessionId, Timestamp};
use loopd::baseline::Baseline;
use loopd::session::{Session, SessionError};
use support::Home;
fn baseline() -> Baseline {
Baseline {
system: "sys".to_string(),
tools: vec![],
}
}
fn later() -> Baseline {
Baseline {
system: "sys-epoch-1".to_string(),
tools: vec![],
}
}
fn id(s: &str) -> SessionId {
SessionId::new(s).unwrap()
}
fn ts() -> Timestamp {
Timestamp::parse("2026-09-18T08:00:00.000Z").unwrap()
}
fn user(content: &str) -> LogRecord {
LogRecord::User {
time: ts(),
content: content.to_string(),
speaker: None,
}
}
fn write_jsonl(home: &Home, session: &str, epoch: u32, records: &[LogRecord]) {
let text: String = records
.iter()
.map(|r| format!("{}\n", serde_json::to_string(r).unwrap()))
.collect();
home.write(&format!("sessions/{session}/{epoch}.jsonl"), &text);
}
fn write_baseline(home: &Home, session: &str, epoch: u32, b: &Baseline) {
home.write(
&format!("sessions/{session}/{epoch}.baseline.json"),
&b.to_json().unwrap(),
);
}
fn start(session: &str, epoch: u32, slot: u32, b: &Baseline) -> LogRecord {
LogRecord::SessionStart {
time: ts(),
session: id(session),
epoch: Epoch(epoch),
slot,
baseline: b.hash().unwrap(),
}
}
fn epoch_end(next: u32, summary: &str) -> LogRecord {
LogRecord::EpochEnd {
time: ts(),
next: Epoch(next),
summary: summary.to_string(),
}
}
fn tool_result(call: u64, untrusted: bool) -> LogRecord {
LogRecord::ToolResult {
time: ts(),
call: CallId(call),
tool_call_id: format!("c{call}"),
content: "out".to_string(),
class: DataClass::Public,
untrusted,
truncated: false,
}
}
#[test]
fn open_loads_the_highest_complete_epoch_and_its_carry() {
let home = Home::new();
let mut s = Session::create(&home.dir, id("a"), baseline(), 3).unwrap();
s.append(user("one")).unwrap();
s.append(user("two")).unwrap();
s.append(epoch_end(1, "carry-text")).unwrap();
drop(s);
write_baseline(&home, "a", 1, &later());
write_jsonl(&home, "a", 1, &[start("a", 1, 3, &later()), user("three")]);
let s = Session::open(&home.dir, id("a")).unwrap();
assert_eq!(s.epoch(), Epoch(1));
assert_eq!(s.slot(), 3);
assert_eq!(s.carry(), Some("carry-text"));
assert_eq!(s.baseline(), &later());
assert_eq!(s.records().len(), 2);
assert!(matches!(s.records()[1], LogRecord::User { ref content, .. } if content == "three"));
}
#[test]
fn a_missing_carry_on_epoch_1_is_an_error() {
let home = Home::new();
let s = Session::create(&home.dir, id("a"), baseline(), 0).unwrap();
drop(s);
write_baseline(&home, "a", 1, &later());
write_jsonl(&home, "a", 1, &[start("a", 1, 0, &later())]);
match Session::open(&home.dir, id("a")) {
Err(SessionError::NoCarry { ref path, .. }) => {
assert!(path.ends_with("0.jsonl"), "{path:?}");
}
other => panic!("{other:?}"),
}
let e: Box<dyn std::error::Error> =
Box::new(Session::open(&home.dir, id("a")).map(|_| ()).unwrap_err());
assert!(e.to_string().contains("0.jsonl"), "{e}");
assert!(
e.to_string()
.contains("see docs/runbook.md#session-log-damaged"),
"{e}"
);
}
#[test]
fn epoch_0_has_no_carry() {
let home = Home::new();
let s = Session::create(&home.dir, id("a"), baseline(), 7).unwrap();
drop(s);
let s = Session::open(&home.dir, id("a")).unwrap();
assert_eq!(s.epoch(), Epoch(0));
assert_eq!(s.carry(), None);
assert_eq!(s.slot(), 7);
}
#[test]
fn turns_count_users_across_every_epoch() {
let home = Home::new();
let mut s = Session::create(&home.dir, id("a"), baseline(), 0).unwrap();
s.append(user("one")).unwrap();
s.append(user("two")).unwrap();
s.append(epoch_end(1, "sum")).unwrap();
drop(s);
write_baseline(&home, "a", 1, &later());
write_jsonl(&home, "a", 1, &[start("a", 1, 0, &later()), user("three")]);
let s = Session::open(&home.dir, id("a")).unwrap();
assert_eq!(s.turns(), 3);
assert_eq!(
s.records()
.iter()
.filter(|r| matches!(r, LogRecord::User { .. }))
.count(),
1,
"records() is the current epoch only"
);
}
#[test]
fn untrusted_reads_every_epoch() {
let home = Home::new();
let mut s = Session::create(&home.dir, id("a"), baseline(), 0).unwrap();
s.append(tool_result(1, true)).unwrap();
s.append(epoch_end(1, "sum")).unwrap();
drop(s);
write_baseline(&home, "a", 1, &later());
write_jsonl(
&home,
"a",
1,
&[start("a", 1, 0, &later()), tool_result(2, false)],
);
let s = Session::open(&home.dir, id("a")).unwrap();
assert!(s.untrusted(), "an untrusted result in epoch 0 still taints");
assert!(
!s.records().iter().any(|r| matches!(
r,
LogRecord::ToolResult {
untrusted: true,
..
}
)),
"the current epoch's records look clean"
);
}
#[test]
fn call_ids_continue_across_epochs() {
let home = Home::new();
let mut s = Session::create(&home.dir, id("a"), baseline(), 0).unwrap();
s.append(tool_result(4, false)).unwrap();
s.append(epoch_end(1, "sum")).unwrap();
drop(s);
write_baseline(&home, "a", 1, &later());
write_jsonl(
&home,
"a",
1,
&[start("a", 1, 0, &later()), tool_result(2, false)],
);
let mut s = Session::open(&home.dir, id("a")).unwrap();
assert_eq!(
s.next_call(),
CallId(5),
"one more than the highest call in any epoch"
);
}
#[test]
fn append_writes_the_current_epoch_file() {
let home = Home::new();
let mut s = Session::create(&home.dir, id("a"), baseline(), 0).unwrap();
s.append(epoch_end(1, "sum")).unwrap();
drop(s);
write_baseline(&home, "a", 1, &later());
write_jsonl(&home, "a", 1, &[start("a", 1, 0, &later())]);
let mut s = Session::open(&home.dir, id("a")).unwrap();
s.append(user("after")).unwrap();
let text = home.read("sessions/a/1.jsonl");
assert!(text.contains("after"), "{text}");
let old = home.read("sessions/a/0.jsonl");
assert!(!old.contains("after"), "{old}");
}
#[test]
fn an_incomplete_next_epoch_is_ignored() {
let home = Home::new();
let mut s = Session::create(&home.dir, id("a"), baseline(), 0).unwrap();
s.append(user("still-here")).unwrap();
drop(s);
write_baseline(&home, "a", 1, &later());
let s = Session::open(&home.dir, id("a")).unwrap();
assert_eq!(s.epoch(), Epoch(0));
assert_eq!(s.carry(), None);
assert_eq!(s.turns(), 1);
}
#[test]
fn epoch_end_without_the_next_files_stays_on_the_old_epoch() {
let home = Home::new();
let mut s = Session::create(&home.dir, id("a"), baseline(), 0).unwrap();
s.append(user("x")).unwrap();
s.append(epoch_end(1, "sum")).unwrap();
drop(s);
let s = Session::open(&home.dir, id("a")).unwrap();
assert_eq!(s.epoch(), Epoch(0));
assert_eq!(s.carry(), None);
assert_eq!(s.turns(), 1);
assert!(
matches!(s.records().last(), Some(LogRecord::EpochEnd { next, .. }) if *next == Epoch(1))
);
}
@@ -0,0 +1,181 @@
//! Honcho reads every epoch file of a session, in order. Do not edit.
#[path = "support/logs.rs"]
mod logs;
#[path = "support/tmp.rs"]
mod tmp;
#[path = "support/turns.rs"]
mod turns;
use std::cell::RefCell;
use bxcore::{Epoch, LogRecord, SessionId, Timestamp};
use logs::{assistant, write_log};
use memoryd::honcho::decide::{turn_records, turns};
use memoryd::honcho::ledger::{Action, Item, Ledger, Record, Workspace};
use memoryd::honcho::turns::{LOG_RUNBOOK, TurnSync, read_session};
use tmp::TempDir;
use turns::{clean, sid, spoken};
const LEDGER: &str = "honcho.jsonl";
fn epoch_end(summary: &str) -> LogRecord {
LogRecord::EpochEnd {
time: Timestamp::parse("2026-09-28T10:00:00.000Z").unwrap(),
next: Epoch(1),
summary: summary.to_string(),
}
}
fn write_epoch(home: &std::path::Path, session: &str, epoch: u32, records: &[LogRecord]) {
let dir = home.join("sessions").join(session);
std::fs::create_dir_all(&dir).unwrap();
let path = dir.join(format!("{epoch}.jsonl"));
let text: String = records
.iter()
.map(|r| format!("{}\n", serde_json::to_string(r).unwrap()))
.collect();
std::fs::write(path, text).unwrap();
}
struct World {
sync: TurnSync,
answer: RefCell<Result<bxcore::SessionStateReply, String>>,
dir: TempDir,
}
impl World {
fn new(tag: &str) -> World {
let dir = TempDir::new(tag);
std::fs::create_dir_all(dir.path().join("memory")).unwrap();
let mut ledger = Ledger::open(&dir.path().join("memory").join(LEDGER)).unwrap();
ledger
.append(Record::Began {
time: Timestamp::parse("2026-09-28T09:00:00.000Z").unwrap(),
})
.unwrap();
World {
sync: TurnSync::new(ledger, "boxmaker0", &dir.path().join("sessions")),
answer: RefCell::new(Ok(clean())),
dir,
}
}
fn updated(&mut self, session: &str, turn: u64) {
let ask = |s: &SessionId| {
let _ = s;
self.answer.borrow().clone()
};
let log = |_line: &str| {};
let now = Timestamp::parse("2026-09-28T10:00:00.000Z").unwrap();
self.sync.updated(&sid(session), turn, &ask, now, &log);
}
fn action(&self, session: &str, turn: u64) -> Option<Action> {
let item = Item::Turn {
session: sid(session),
turn,
};
let path = self.dir.path().join("memory").join(LEDGER);
let text = std::fs::read_to_string(path).unwrap_or_default();
let key = serde_json::to_string(&item).unwrap();
let line = text.lines().find(|l| l.contains(&key))?;
let value: serde_json::Value = serde_json::from_str(line).unwrap();
Some(serde_json::from_value(value["action"].clone()).unwrap())
}
}
fn to(workspace: Workspace, peer: &str) -> Action {
Action::To {
workspace,
peer: peer.to_string(),
}
}
#[test]
fn concatenated_logs_count_three_turns() {
let dir = TempDir::new("honcho-epochs-count");
let home = dir.path();
let epoch0 = vec![
spoken("one", Some("kyle")),
assistant(2, Some("ok"), &[]),
spoken("two", Some("kyle")),
assistant(3, Some("ok"), &[]),
epoch_end("sum"),
];
write_log(home, "mm-a", &epoch0);
write_epoch(
home,
"mm-a",
1,
&[
spoken("three", Some("kyle")),
assistant(4, Some("ok"), &[]),
],
);
let records = read_session(&home.join("sessions"), &sid("mm-a")).unwrap();
assert_eq!(turns(&records), 3);
let turn3 = turn_records(&records, 3).unwrap();
assert!(
matches!(turn3[0], LogRecord::User { ref content, .. } if content == "three"),
"{turn3:?}"
);
}
#[test]
fn epoch_0_alone_still_reads() {
let dir = TempDir::new("honcho-epochs-zero");
let home = dir.path();
write_log(home, "mm-a", &[spoken("only", Some("kyle"))]);
let records = read_session(&home.join("sessions"), &sid("mm-a")).unwrap();
assert_eq!(turns(&records), 1);
}
#[test]
fn a_damaged_later_epoch_is_an_error_naming_that_file() {
let dir = TempDir::new("honcho-epochs-damaged");
let home = dir.path();
write_log(home, "mm-a", &[spoken("one", Some("kyle"))]);
let path = home.join("sessions/mm-a/1.jsonl");
std::fs::write(&path, "{\"type\":\"zz\"}\n").unwrap();
let err = read_session(&home.join("sessions"), &sid("mm-a")).unwrap_err();
assert!(err.to_string().contains("1.jsonl"), "{err}");
assert!(
err.to_string().contains(LOG_RUNBOOK) || err.to_string().contains("session-log-damaged"),
"{err}"
);
}
#[test]
fn turn_sync_decides_a_turn_that_lives_in_epoch_1() {
let mut w = World::new("honcho-epochs-sync");
write_log(
w.dir.path(),
"mm-a",
&[
spoken("one", Some("kyle")),
assistant(2, Some("ok"), &[]),
spoken("two", Some("kyle")),
assistant(3, Some("ok"), &[]),
epoch_end("sum"),
],
);
write_epoch(
w.dir.path(),
"mm-a",
1,
&[
spoken("three", Some("kyle")),
assistant(4, Some("ok"), &[]),
],
);
w.updated("mm-a", 3);
assert_eq!(w.action("mm-a", 3), Some(to(Workspace::Trusted, "kyle")));
}
#[test]
fn a_missing_session_directory_is_empty() {
let dir = TempDir::new("honcho-epochs-missing");
let records = read_session(&dir.path().join("sessions"), &sid("mm-none")).unwrap();
assert!(records.is_empty());
}
@@ -0,0 +1,35 @@
## session-compact-failed
**What you see.** `loopd` logs that it could not compact a session, then this entry. The session
stays on its current epoch. A turn that was already too large to fit (`session_full`) stays
`session_full`. A turn that finished stays finished; the next turn on that session may be
`session_full` if it would not fit.
```text
loopd: could not compact session chat-0-0: compact.md: No such file or directory (os error 2); see docs/runbook.md#session-compact-failed
```
**Why.** Compaction needs a summary of the current epoch (a completion on the background slot
with `compact.md` as the last user message, or the fallback of that epoch's owner texts). If the
prompt file cannot be read, the summary call fails for a reason other than a full context, or
writing the next epoch's files fails, `loopd` refuses to guess a prefix. It does not start a new
epoch from an empty summary when the model call failed.
**Confirm.**
```sh
ls -l "$(dirname "$BOXMAKER_HOME/../config")/compact.md" 2>/dev/null
ls -l "$BOXMAKER_HOME/sessions/<id>"/
tail -n 5 "$BOXMAKER_HOME/sessions/<id>/0.jsonl"
```
`compact.md` lives next to `system.md` (the path in `loopd.toml`'s `[baseline] system`). A last
line of type `epoch_end` without `<n>.baseline.json` and `<n>.jsonl` is a compact that stopped
halfway; the next compact resumes it.
**Fix.** Restore `compact.md` beside `system.md` (the starter is `config/compact.md` in the
repository). Check that `inferproxy` still answers. If the session directory is mid-write, do not
delete `EpochEnd`; the next compact continues from it.
**Check.** The next turn on that session either runs on the current epoch, or compact succeeds
and the next turn's log is `<n>.jsonl` with `epoch` n.