docs: M5d-1 and M5d-2 implementation plans for Ornith
Eight closed tasks for epochs and compaction, then ten for the scheduler, ToolSummary, and tiers. Given tests first. Gate 1456 → 1507 → 1572. Co-Authored-By: Grok 4.6 <noreply@x.ai>
This commit is contained in:
Grok 4.6
@@ -0,0 +1,209 @@
|
||||
# M5d-1 task 01: `[compact]` on `loopd.toml`
|
||||
|
||||
**Branch:** `m5d1`. Check it: `git branch --show-current` prints `m5d1` and `git status --short`
|
||||
is empty; otherwise stop. Never switch branch.
|
||||
**Commit subject:** `loopd: [compact] threshold, summary_chars and summarise`
|
||||
**Log row:** the Task cell is `M5d1/01-loopd-compact-config`.
|
||||
|
||||
## Goal
|
||||
|
||||
M5d-1 compact after a turn whose last `Usage` is at or above a fraction of `expect.n_ctx` (spec
|
||||
3.3, 5). This task adds only the `[compact]` table so later tasks can read it:
|
||||
|
||||
- `threshold` (float, default `0.85`)
|
||||
- `summary_chars` (default `4000`)
|
||||
- `summarise` (list of tool names, default `["http_fetch"]`)
|
||||
|
||||
Nothing in this task uses the values. `summarise` is stored so a spec-complete `loopd.toml`
|
||||
loads; M5d-2 reads it. Unknown keys in `[compact]` are errors, as in every table we define.
|
||||
|
||||
`Config::compact_prompt` is `compact.md` beside `[baseline] system` (the same directory join
|
||||
`load` already does for `system.md`). The starter file is copied in task 05.
|
||||
|
||||
## Every place that changes
|
||||
|
||||
This is the complete list.
|
||||
|
||||
| Place | What happens there |
|
||||
|---|---|
|
||||
| `crates/loopd/src/config.rs` | `Compact`; `Config.compact`; `Config::compact_prompt` (edits C1 to C3) |
|
||||
| `crates/loopd/tests/config_compact.rs` | new, given |
|
||||
| everything else | **no change**: not `session.rs`, not the existing `config.rs` tests or fixtures |
|
||||
|
||||
## Files
|
||||
|
||||
- Copy, from `docs/plans/M5d1/files/` (the same path under it): the new test
|
||||
`crates/loopd/tests/config_compact.rs`.
|
||||
- Modify: `crates/loopd/src/config.rs`, `docs/implementer-log.md`.
|
||||
- Do not touch: any other file.
|
||||
|
||||
## Facts you need
|
||||
|
||||
- **`Config` already has `#[serde(default)]` on optional tables** (`sampling`, `limits`, `paths`,
|
||||
`channel`, `loop`, `baseline`, `broker`, `memory`). `[compact]` is the same: missing means
|
||||
`Compact::default()`.
|
||||
- **`threshold` is `f64`.** `Sampling.temperature` is already `f64` on `Config`, so `PartialEq`
|
||||
on `Config` still compiles. Do not add a range check: the spec does not name one.
|
||||
- **`summarise` default.** `vec!["http_fetch".to_string()]`. An explicit `summarise = []` is
|
||||
allowed and is empty. Unknown names are not checked here (M5d-2).
|
||||
- **`compact_prompt`.** `self.baseline.system.with_file_name("compact.md")`. After `load`,
|
||||
`baseline.system` is already joined to the config file's directory, so the prompt follows it
|
||||
into a subdirectory when `system` does (`prompts/agent.md` → `prompts/compact.md`).
|
||||
- The items the edits use. This table is complete; `serde` is already a dependency of `loopd`.
|
||||
|
||||
| Item | What it does |
|
||||
|---|---|
|
||||
| `#[serde(deny_unknown_fields, default)]` | on `Compact`: missing table uses `Default`; unknown keys error |
|
||||
| `Path::with_file_name` | `compact.md` beside `system.md` |
|
||||
| `Vec<String>` | `summarise` |
|
||||
|
||||
## The edits, exactly
|
||||
|
||||
Nothing else in this file changes. Each quoted "before" block occurs exactly once in its file
|
||||
when its turn comes (make them in order). Keep the fields in the order shown.
|
||||
|
||||
### `crates/loopd/src/config.rs`
|
||||
|
||||
**C1.** After the `Memory` struct (the lines)
|
||||
|
||||
```rust
|
||||
pub struct Memory {
|
||||
pub socket: std::path::PathBuf,
|
||||
}
|
||||
```
|
||||
|
||||
become
|
||||
|
||||
```rust
|
||||
pub struct Memory {
|
||||
pub socket: std::path::PathBuf,
|
||||
}
|
||||
|
||||
/// `[compact]`: when a session starts a new epoch, and how long a summary may be (M5d spec,
|
||||
/// section 5). `summarise` is stored here and unused until M5d-2.
|
||||
#[derive(Debug, Clone, PartialEq, Deserialize)]
|
||||
#[serde(deny_unknown_fields, default)]
|
||||
pub struct Compact {
|
||||
pub threshold: f64,
|
||||
pub summary_chars: usize,
|
||||
pub summarise: Vec<String>,
|
||||
}
|
||||
|
||||
impl Default for Compact {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
threshold: 0.85,
|
||||
summary_chars: 4_000,
|
||||
summarise: vec!["http_fetch".to_string()],
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**C2.** The lines
|
||||
|
||||
```rust
|
||||
#[serde(default)]
|
||||
pub memory: Memory,
|
||||
}
|
||||
```
|
||||
|
||||
become
|
||||
|
||||
```rust
|
||||
#[serde(default)]
|
||||
pub memory: Memory,
|
||||
#[serde(default)]
|
||||
pub compact: Compact,
|
||||
}
|
||||
```
|
||||
|
||||
**C3.** After `channel_socket` (the lines)
|
||||
|
||||
```rust
|
||||
pub fn channel_socket(&self) -> std::path::PathBuf {
|
||||
if self.channel.socket.as_os_str().is_empty() {
|
||||
self.paths.home.join("run/loop/loop.sock")
|
||||
} else {
|
||||
self.channel.socket.clone()
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
become
|
||||
|
||||
```rust
|
||||
pub fn channel_socket(&self) -> std::path::PathBuf {
|
||||
if self.channel.socket.as_os_str().is_empty() {
|
||||
self.paths.home.join("run/loop/loop.sock")
|
||||
} else {
|
||||
self.channel.socket.clone()
|
||||
}
|
||||
}
|
||||
|
||||
/// `compact.md` beside the system prompt: the summary call's last user message.
|
||||
pub fn compact_prompt(&self) -> std::path::PathBuf {
|
||||
self.baseline.system.with_file_name("compact.md")
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Steps
|
||||
|
||||
- [ ] **1. Copy the given test.** From the repository root, one command:
|
||||
|
||||
```sh
|
||||
cp docs/plans/M5d1/files/crates/loopd/tests/config_compact.rs crates/loopd/tests/
|
||||
```
|
||||
|
||||
Then `git status --short` lists one new file, `crates/loopd/tests/config_compact.rs`.
|
||||
- [ ] **2. See it fail.** `cargo build --tests --keep-going -p loopd`, with a shell timeout of
|
||||
600000 ms: one test target, `config_compact`, does not compile, with `E0609` (`no field
|
||||
\`compact\` on type \`Config\``) and `E0599` (`no method named \`compact_prompt\`` /
|
||||
`no variant or associated item named \`default\` for \`Compact\``). Every other target
|
||||
compiles.
|
||||
- [ ] **3. The edits.** Make edits C1 to C3, in order. `cargo check -p loopd` prints no error and
|
||||
no warning.
|
||||
- [ ] **4. The tests.** `cargo test -p loopd --test config_compact --test config`: every test
|
||||
passes, 8 in `config_compact` and the others as before.
|
||||
- [ ] **5. Check.** `git diff --stat` lists exactly `crates/loopd/src/config.rs`.
|
||||
- [ ] **6. Run the gate.** `cargo fmt --all`, then `make gate`, with a shell timeout of 600000 ms.
|
||||
Expected last line: `gate: ok`. The test count is **1464** after this task, 1456 before it
|
||||
(1455 passed in `cargo test --workspace`, plus 1), and the 8 new ones are the tests of
|
||||
`config_compact.rs`. To count, add up the `N passed` of every `test result:` line the gate's
|
||||
`cargo test --workspace` printed, and add 1. Never run the gate again only to count.
|
||||
- [ ] **7. Log and commit.** Add your row to `docs/implementer-log.md`; its Task cell is
|
||||
`M5d1/01-loopd-compact-config`. Stage with
|
||||
`git add crates/loopd docs/implementer-log.md Cargo.lock`, then commit as `AGENTS.md` says.
|
||||
`Cargo.lock` does not change in this task; the `git add` is harmless.
|
||||
|
||||
## Done when
|
||||
|
||||
- `make gate` prints `gate: ok` with 1464 tests.
|
||||
- `git diff HEAD~1 --stat` lists only `crates/loopd/src/config.rs`,
|
||||
`crates/loopd/tests/config_compact.rs` and `docs/implementer-log.md`.
|
||||
|
||||
## Stop and report if
|
||||
|
||||
- The branch check at the top fails.
|
||||
- A "before" block is not in its file exactly as shown when its turn comes, or is there more
|
||||
than once.
|
||||
- Step 2 shows a different result: another target failing, or other errors.
|
||||
- `cargo check` in step 3 prints an error or a warning. Quote it, and stop.
|
||||
- A test fails in step 4. Do not change the test: stop, and give the failing tests' names and
|
||||
their messages.
|
||||
- The fix seems to need a change anywhere else: a fixture, `config.rs` tests, a range check on
|
||||
`threshold`. It does not; stop and say why you think so.
|
||||
- A test outside `config_compact.rs` fails in the gate: check once whether it fails without your
|
||||
change (`git stash -u`, run it, `git stash pop`), then stop and report both results.
|
||||
- The gate passes with a count other than 1464.
|
||||
- A comment in an edit does not fit the code under it. Quote it and stop.
|
||||
|
||||
## What the given tests pin
|
||||
|
||||
Each of these changes to the finished code makes at least one given test fail: the default
|
||||
threshold moved; `summary_chars` default moved; `summarise` default emptied or given another
|
||||
name; unknown keys accepted; a misspelt `summary_char` accepted; a partial table wiping the
|
||||
other defaults; `compact_prompt` using `home` or the config file's name instead of `system.md`'s
|
||||
directory; `Compact::default` disagreeing with a missing table.
|
||||
@@ -0,0 +1,513 @@
|
||||
# M5d-1 task 02: `Session::open` loads the current epoch
|
||||
|
||||
**Branch:** `m5d1`. Check it: `git branch --show-current` prints `m5d1` and `git status --short`
|
||||
is empty; otherwise stop. Never switch branch.
|
||||
**Commit subject:** `loopd: open the highest epoch and carry its summary`
|
||||
**Log row:** the Task cell is `M5d1/02-loopd-session-epochs`.
|
||||
|
||||
## Goal
|
||||
|
||||
A session directory may have `0.jsonl` and, after compaction, `1.baseline.json` / `1.jsonl` and
|
||||
so on (spec 3.1). `Session::open` loads the highest epoch that has **both** a baseline file and a
|
||||
log. For epoch `n > 0` the previous log must end with `EpochEnd { next: n, summary }`; that
|
||||
summary is the carry. `turns()` and `untrusted()` span every epoch. `next_call` is one more than
|
||||
the highest `ToolResult` call in any epoch. `append` writes the current epoch's log.
|
||||
|
||||
`records()` stays the current epoch only: later tasks replay that file (plus the carry) as the
|
||||
request.
|
||||
|
||||
## Every place that changes
|
||||
|
||||
This is the complete list.
|
||||
|
||||
| Place | What happens there |
|
||||
|---|---|
|
||||
| `crates/loopd/src/session.rs` | `NoCarry`; fields `epoch`, `slot`, `carry`, `prior_users`, `prior_untrusted`; `open`; `turns`; `untrusted`; `log_path`; accessors; helpers |
|
||||
| `crates/loopd/tests/session_epochs.rs` | new, given |
|
||||
| everything else | **no change**: not `session.rs`'s existing tests, not `turn.rs`, not Honcho |
|
||||
|
||||
## Files
|
||||
|
||||
- Copy, from `docs/plans/M5d1/files/` (the same path under it): the new test
|
||||
`crates/loopd/tests/session_epochs.rs`.
|
||||
- Modify: `crates/loopd/src/session.rs`, `docs/implementer-log.md`.
|
||||
- Do not touch: any other file.
|
||||
|
||||
## Facts you need
|
||||
|
||||
- **`Epoch` is `pub struct Epoch(pub u32)`** in `bxcore`. Format a file name with
|
||||
`format!("{}.jsonl", epoch.0)` and `format!("{}.baseline.json", epoch.0)`.
|
||||
- **A complete epoch** is both `<n>.baseline.json` and `<n>.jsonl` in the session directory.
|
||||
A baseline without a log, or a log without a baseline, does not count. The highest complete
|
||||
`n` is the current epoch. If none is complete, `open` is `NotFound`, as today when `0.jsonl`
|
||||
is missing.
|
||||
- **Carry.** For current epoch `n > 0`, read `(n-1).jsonl`. Among its records, the last
|
||||
`EpochEnd` whose `next` is `Epoch(n)` gives `summary`. If that file is missing, or has no such
|
||||
record, `SessionError::NoCarry { path }` naming `(n-1).jsonl`. Epoch 0 has `carry: None`.
|
||||
- **`records` on the live `Session`** is only the current epoch's log, so `messages` in task 03
|
||||
can replay it. `prior_users` / `prior_untrusted` are counted from every **earlier** complete
|
||||
epoch while `open` walks them. `turns()` is `prior_users` plus the `User` records in
|
||||
`records`. `untrusted()` is `prior_untrusted` or any untrusted `ToolResult` in `records`.
|
||||
- **`next_call`** is one more than the highest `ToolResult.call.0` in **every** complete epoch
|
||||
(current included). A call id of 4 in epoch 0 and 2 in epoch 1 continues at 5.
|
||||
- **`slot`** is the `slot` on the current epoch's `SessionStart`. `create` stores the `slot`
|
||||
argument. If the current log has no `SessionStart`, use `0`.
|
||||
- **`append` / `persist`** already write `self.log_path()`. Change `log_path` to the current
|
||||
epoch's file; `create` still starts at `0.jsonl`.
|
||||
- **A last `EpochEnd` on epoch 0 with no epoch-1 files** means compact stopped halfway: `open`
|
||||
stays on epoch 0, `carry` is `None`, and the `EpochEnd` is in `records`. Task 07 resumes that.
|
||||
- The torn-log checks stay as they are, per file, with the line number of that file.
|
||||
- `session.rs` must stay at or under 500 lines (`wc -l`). After these edits it is about 430.
|
||||
- The items the edits use. This table is complete.
|
||||
|
||||
| Item | What it does |
|
||||
|---|---|
|
||||
| `std::fs::read_dir` | list `<n>.jsonl` names |
|
||||
| `str::strip_suffix(".jsonl")`, `str::parse::<u32>` | the epoch number |
|
||||
| `Path::exists` | the matching baseline |
|
||||
| `u64::try_from(count).unwrap_or(u64::MAX)` | `turns`, as today |
|
||||
| `CallId(highest + 1)` | `next_call`, as today, over every epoch |
|
||||
| `OpenOptions::new().append(true).open` | the current log |
|
||||
|
||||
## The edits, exactly
|
||||
|
||||
Nothing else in this file changes. Each quoted "before" block occurs exactly once. Keep struct
|
||||
fields in the order shown.
|
||||
|
||||
### `crates/loopd/src/session.rs`
|
||||
|
||||
**S1.** The module doc comment, the lines
|
||||
|
||||
```rust
|
||||
//! A session on disk: `sessions/<id>/0.baseline.json` and `0.jsonl`. The log is the state —
|
||||
//! create, open, append and resume all live here, and nothing is durable until `append` syncs it.
|
||||
```
|
||||
|
||||
become
|
||||
|
||||
```rust
|
||||
//! A session on disk: `sessions/<id>/<epoch>.baseline.json` and `<epoch>.jsonl`. The log is the
|
||||
//! state — create, open, append and resume all live here, and nothing is durable until `append`
|
||||
//! syncs it. Epoch 0 is the first; later epochs appear after compaction.
|
||||
```
|
||||
|
||||
**S2.** After `Encode(serde_json::Error),` in `SessionError`, add a variant (the lines)
|
||||
|
||||
```rust
|
||||
Encode(serde_json::Error),
|
||||
}
|
||||
```
|
||||
|
||||
become
|
||||
|
||||
```rust
|
||||
Encode(serde_json::Error),
|
||||
NoCarry {
|
||||
path: PathBuf,
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
**S3.** In `Display`, the lines
|
||||
|
||||
```rust
|
||||
SessionError::Encode(err) => write!(f, "{err}"),
|
||||
}
|
||||
```
|
||||
|
||||
become
|
||||
|
||||
```rust
|
||||
SessionError::Encode(err) => write!(f, "{err}"),
|
||||
SessionError::NoCarry { path } => write!(
|
||||
f,
|
||||
"{}: compacted epoch is missing its EpochEnd; see docs/runbook.md#session-log-damaged",
|
||||
path.display()
|
||||
),
|
||||
}
|
||||
```
|
||||
|
||||
**S4.** The struct, the lines
|
||||
|
||||
```rust
|
||||
pub struct Session {
|
||||
id: SessionId,
|
||||
dir: PathBuf,
|
||||
baseline: Baseline,
|
||||
records: Vec<LogRecord>,
|
||||
log: File,
|
||||
next_call: CallId,
|
||||
}
|
||||
```
|
||||
|
||||
become
|
||||
|
||||
```rust
|
||||
pub struct Session {
|
||||
id: SessionId,
|
||||
dir: PathBuf,
|
||||
baseline: Baseline,
|
||||
records: Vec<LogRecord>,
|
||||
log: File,
|
||||
next_call: CallId,
|
||||
epoch: Epoch,
|
||||
slot: u32,
|
||||
carry: Option<String>,
|
||||
prior_users: u64,
|
||||
prior_untrusted: bool,
|
||||
}
|
||||
```
|
||||
|
||||
**S5.** In `create`, the lines
|
||||
|
||||
```rust
|
||||
let mut session = Session {
|
||||
id,
|
||||
dir,
|
||||
baseline,
|
||||
records: Vec::new(),
|
||||
log,
|
||||
next_call: CallId(1),
|
||||
};
|
||||
```
|
||||
|
||||
become
|
||||
|
||||
```rust
|
||||
let mut session = Session {
|
||||
id,
|
||||
dir,
|
||||
baseline,
|
||||
records: Vec::new(),
|
||||
log,
|
||||
next_call: CallId(1),
|
||||
epoch: Epoch(0),
|
||||
slot,
|
||||
carry: None,
|
||||
prior_users: 0,
|
||||
prior_untrusted: false,
|
||||
};
|
||||
```
|
||||
|
||||
**S6.** Replace the whole `open` function. The lines from `pub fn open(home: &Path, id: SessionId)`
|
||||
through the `Ok(Session {` block that ends `next_call: CallId(highest + 1),` and `})` `}` become
|
||||
the function below. Fill any `todo!()` from its comment, then delete the comment. Each comment
|
||||
is at most six lines.
|
||||
|
||||
```rust
|
||||
pub fn open(home: &Path, id: SessionId) -> Result<Session, SessionError> {
|
||||
let dir = Session::dir_for(home, &id);
|
||||
let epochs = complete_epochs(&dir)?;
|
||||
let Some(&n) = epochs.last() else {
|
||||
return Err(SessionError::NotFound(id));
|
||||
};
|
||||
let epoch = Epoch(n);
|
||||
let baseline_path = dir.join(format!("{n}.baseline.json"));
|
||||
let baseline = Baseline::load(&baseline_path).map_err(SessionError::Baseline)?;
|
||||
let log_path = dir.join(format!("{n}.jsonl"));
|
||||
let records = load_records(&log_path)?;
|
||||
let (carry, prior_users, prior_untrusted) = priors(&dir, n)?;
|
||||
let mut highest = max_call(&records);
|
||||
for earlier in &epochs[..epochs.len() - 1] {
|
||||
let path = dir.join(format!("{earlier}.jsonl"));
|
||||
highest = highest.max(max_call(&load_records(&path)?));
|
||||
}
|
||||
let slot = live_slot(&records, 0);
|
||||
let log = OpenOptions::new()
|
||||
.append(true)
|
||||
.open(&log_path)
|
||||
.map_err(|e| SessionError::Io(log_path.clone(), e))?;
|
||||
Ok(Session {
|
||||
id,
|
||||
dir,
|
||||
baseline,
|
||||
records,
|
||||
log,
|
||||
next_call: CallId(highest + 1),
|
||||
epoch,
|
||||
slot,
|
||||
carry,
|
||||
prior_users,
|
||||
prior_untrusted,
|
||||
})
|
||||
}
|
||||
```
|
||||
|
||||
**S7.** After `pub fn records(&self)`, add accessors (the lines)
|
||||
|
||||
```rust
|
||||
pub fn records(&self) -> &[LogRecord] {
|
||||
&self.records
|
||||
}
|
||||
```
|
||||
|
||||
become
|
||||
|
||||
```rust
|
||||
pub fn records(&self) -> &[LogRecord] {
|
||||
&self.records
|
||||
}
|
||||
|
||||
pub fn epoch(&self) -> Epoch {
|
||||
self.epoch
|
||||
}
|
||||
|
||||
pub fn slot(&self) -> u32 {
|
||||
self.slot
|
||||
}
|
||||
|
||||
pub fn carry(&self) -> Option<&str> {
|
||||
self.carry.as_deref()
|
||||
}
|
||||
```
|
||||
|
||||
**S8.** `untrusted` and `turns`, the lines
|
||||
|
||||
```rust
|
||||
pub fn untrusted(&self) -> bool {
|
||||
self.records.iter().any(|record| {
|
||||
matches!(
|
||||
record,
|
||||
LogRecord::ToolResult {
|
||||
untrusted: true,
|
||||
..
|
||||
}
|
||||
)
|
||||
})
|
||||
}
|
||||
|
||||
/// How many turns the session has had: its `User` records, one per turn that reached the
|
||||
/// log, a failed turn's too. Counted from `records`, which `open` reads back whole, so the
|
||||
/// count goes on after a reopen or a restart; right after a turn it is that turn's number,
|
||||
/// the first being 1. Epoch 0 is a session's only log until compaction is built.
|
||||
pub fn turns(&self) -> u64 {
|
||||
let users = self
|
||||
.records
|
||||
.iter()
|
||||
.filter(|record| matches!(record, LogRecord::User { .. }))
|
||||
.count();
|
||||
u64::try_from(users).unwrap_or(u64::MAX)
|
||||
}
|
||||
```
|
||||
|
||||
become
|
||||
|
||||
```rust
|
||||
pub fn untrusted(&self) -> bool {
|
||||
self.prior_untrusted || any_untrusted(&self.records)
|
||||
}
|
||||
|
||||
/// How many turns the session has had: its `User` records across every epoch, a failed
|
||||
/// turn's too. The first `User` of the session is turn 1 for its life.
|
||||
pub fn turns(&self) -> u64 {
|
||||
self.prior_users
|
||||
.saturating_add(count_users(&self.records))
|
||||
}
|
||||
```
|
||||
|
||||
**S9.** `log_path`, the lines
|
||||
|
||||
```rust
|
||||
fn log_path(&self) -> PathBuf {
|
||||
self.dir.join("0.jsonl")
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
become
|
||||
|
||||
```rust
|
||||
fn log_path(&self) -> PathBuf {
|
||||
self.dir.join(format!("{}.jsonl", self.epoch.0))
|
||||
}
|
||||
}
|
||||
|
||||
/// Epoch numbers that have both `<n>.jsonl` and `<n>.baseline.json`, sorted.
|
||||
fn complete_epochs(dir: &Path) -> Result<Vec<u32>, SessionError> {
|
||||
let entries = match std::fs::read_dir(dir) {
|
||||
Ok(e) => e,
|
||||
Err(e) if e.kind() == std::io::ErrorKind::NotFound => return Ok(Vec::new()),
|
||||
Err(e) => return Err(SessionError::Io(dir.to_path_buf(), e)),
|
||||
};
|
||||
let mut out = Vec::new();
|
||||
for entry in entries {
|
||||
let entry = entry.map_err(|e| SessionError::Io(dir.to_path_buf(), e))?;
|
||||
let name = entry.file_name();
|
||||
let Some(name) = name.to_str() else { continue };
|
||||
let Some(stem) = name.strip_suffix(".jsonl") else { continue };
|
||||
let Ok(n) = stem.parse::<u32>() else { continue };
|
||||
if dir.join(format!("{n}.baseline.json")).exists() {
|
||||
out.push(n);
|
||||
}
|
||||
}
|
||||
out.sort_unstable();
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
fn load_records(path: &Path) -> Result<Vec<LogRecord>, SessionError> {
|
||||
let text = match std::fs::read_to_string(path) {
|
||||
Ok(t) => t,
|
||||
Err(e) if e.kind() == std::io::ErrorKind::NotFound => {
|
||||
return Err(SessionError::Io(path.to_path_buf(), e));
|
||||
}
|
||||
Err(e) => return Err(SessionError::Io(path.to_path_buf(), e)),
|
||||
};
|
||||
let mut records = Vec::new();
|
||||
if text.is_empty() {
|
||||
return Ok(records);
|
||||
}
|
||||
for (idx, line) in text.split_inclusive('\n').enumerate() {
|
||||
let line_no = idx + 1;
|
||||
if !line.ends_with('\n') {
|
||||
return Err(SessionError::Torn {
|
||||
path: path.to_path_buf(),
|
||||
line: line_no,
|
||||
why: "last line does not end in a newline".to_string(),
|
||||
});
|
||||
}
|
||||
match serde_json::from_str::<LogRecord>(line) {
|
||||
Ok(record) => records.push(record),
|
||||
Err(why) => {
|
||||
return Err(SessionError::Torn {
|
||||
path: path.to_path_buf(),
|
||||
line: line_no,
|
||||
why: why.to_string(),
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
Ok(records)
|
||||
}
|
||||
|
||||
/// Carry from epoch `n-1`, and user / untrusted counts from every epoch before `n`.
|
||||
fn priors(dir: &Path, n: u32) -> Result<(Option<String>, u64, bool), SessionError> {
|
||||
let mut prior_users = 0u64;
|
||||
let mut prior_untrusted = false;
|
||||
let mut carry = None;
|
||||
if n == 0 {
|
||||
return Ok((None, 0, false));
|
||||
}
|
||||
for earlier in 0..n {
|
||||
let path = dir.join(format!("{earlier}.jsonl"));
|
||||
let records = match load_records(&path) {
|
||||
Ok(r) => r,
|
||||
Err(SessionError::Io(path, e)) if e.kind() == std::io::ErrorKind::NotFound => {
|
||||
return Err(SessionError::NoCarry { path });
|
||||
}
|
||||
Err(e) => return Err(e),
|
||||
};
|
||||
prior_users = prior_users.saturating_add(count_users(&records));
|
||||
prior_untrusted = prior_untrusted || any_untrusted(&records);
|
||||
if earlier + 1 == n {
|
||||
carry = carry_from(&records, Epoch(n)).map(str::to_string);
|
||||
if carry.is_none() {
|
||||
return Err(SessionError::NoCarry { path });
|
||||
}
|
||||
}
|
||||
}
|
||||
Ok((carry, prior_users, prior_untrusted))
|
||||
}
|
||||
|
||||
fn carry_from(records: &[LogRecord], next: Epoch) -> Option<&str> {
|
||||
records.iter().rev().find_map(|record| match record {
|
||||
LogRecord::EpochEnd {
|
||||
next: n,
|
||||
summary,
|
||||
..
|
||||
} if *n == next => Some(summary.as_str()),
|
||||
_ => None,
|
||||
})
|
||||
}
|
||||
|
||||
fn count_users(records: &[LogRecord]) -> u64 {
|
||||
let users = records
|
||||
.iter()
|
||||
.filter(|record| matches!(record, LogRecord::User { .. }))
|
||||
.count();
|
||||
u64::try_from(users).unwrap_or(u64::MAX)
|
||||
}
|
||||
|
||||
fn any_untrusted(records: &[LogRecord]) -> bool {
|
||||
records.iter().any(|record| {
|
||||
matches!(
|
||||
record,
|
||||
LogRecord::ToolResult {
|
||||
untrusted: true,
|
||||
..
|
||||
}
|
||||
)
|
||||
})
|
||||
}
|
||||
|
||||
fn max_call(records: &[LogRecord]) -> u64 {
|
||||
let mut highest = 0u64;
|
||||
for record in records {
|
||||
if let LogRecord::ToolResult { call, .. } = record {
|
||||
highest = highest.max(call.0);
|
||||
}
|
||||
}
|
||||
highest
|
||||
}
|
||||
|
||||
fn live_slot(records: &[LogRecord], fallback: u32) -> u32 {
|
||||
records
|
||||
.iter()
|
||||
.find_map(|record| match record {
|
||||
LogRecord::SessionStart { slot, .. } => Some(*slot),
|
||||
_ => None,
|
||||
})
|
||||
.unwrap_or(fallback)
|
||||
}
|
||||
```
|
||||
|
||||
The `open` body in S6 is given whole: copy it, do not leave a `todo!()` in `open`. The helpers
|
||||
in S9 are given whole too.
|
||||
|
||||
## Steps
|
||||
|
||||
- [ ] **1. Copy the given test.** From the repository root:
|
||||
|
||||
```sh
|
||||
cp docs/plans/M5d1/files/crates/loopd/tests/session_epochs.rs crates/loopd/tests/
|
||||
```
|
||||
|
||||
- [ ] **2. See it fail.** `cargo build --tests --keep-going -p loopd`, timeout 600000 ms: test
|
||||
target `session_epochs` does not compile (`E0599` no method `epoch` / `carry` / `slot`, and/or
|
||||
`E0433` `NoCarry`). Every other target compiles. Existing `session` tests still compile.
|
||||
- [ ] **3. The edits.** Make S1 to S9 in order. `cargo check -p loopd` prints no error and no
|
||||
warning. `wc -l crates/loopd/src/session.rs` prints a number ≤ 500.
|
||||
- [ ] **4. The tests.** `cargo test -p loopd --test session_epochs --test session`: every test
|
||||
passes, 9 in `session_epochs` and the others as before.
|
||||
- [ ] **5. Check.** `git diff --stat` lists exactly `crates/loopd/src/session.rs`.
|
||||
- [ ] **6. Run the gate.** `cargo fmt --all`, then `make gate`, timeout 600000 ms. Last line
|
||||
`gate: ok`. Test count **1473** (1464 before, plus 9).
|
||||
- [ ] **7. Log and commit.** Task cell `M5d1/02-loopd-session-epochs`. Stage with
|
||||
`git add crates/loopd docs/implementer-log.md Cargo.lock`, then commit as `AGENTS.md` says.
|
||||
|
||||
## Done when
|
||||
|
||||
- `make gate` prints `gate: ok` with 1473 tests.
|
||||
- `git diff HEAD~1 --stat` lists only `crates/loopd/src/session.rs`,
|
||||
`crates/loopd/tests/session_epochs.rs` and `docs/implementer-log.md`.
|
||||
|
||||
## Stop and report if
|
||||
|
||||
- The branch check fails.
|
||||
- A "before" block is not found exactly once.
|
||||
- Step 2 shows a different result.
|
||||
- `cargo check` warns or errors, or `session.rs` exceeds 500 lines.
|
||||
- A test fails in step 4. Do not change the test.
|
||||
- An existing `session` test fails: stop and report. Those tests still create and open epoch 0
|
||||
only.
|
||||
- The gate count is not 1473.
|
||||
- A comment does not fit the code under it.
|
||||
|
||||
## What the given tests pin
|
||||
|
||||
Opening a directory with epochs 0 and 1 loads 1 and the carry from 0's `EpochEnd`; a missing
|
||||
`EpochEnd` on epoch 0 when epoch 1 files exist is `NoCarry` naming `0.jsonl` with the damaged-log
|
||||
runbook pointer; epoch 0 has no carry and keeps its `SessionStart` slot; two `User`s in epoch 0
|
||||
and one in epoch 1 is `turns() == 3` while `records()` has one `User`; an untrusted `ToolResult`
|
||||
in epoch 0 taints after compact; `next_call` is one more than the highest call in any epoch;
|
||||
`append` after opening epoch 1 writes `1.jsonl`; a `1.baseline.json` without `1.jsonl` is ignored;
|
||||
an `EpochEnd` on epoch 0 without the next files stays on epoch 0.
|
||||
@@ -0,0 +1,185 @@
|
||||
# M5d-1 task 03: the carried summary in the next epoch's prefix
|
||||
|
||||
**Branch:** `m5d1`. Check it: `git branch --show-current` prints `m5d1` and `git status --short`
|
||||
is empty; otherwise stop. Never switch branch.
|
||||
**Commit subject:** `loopd: replay a carried summary as an unlogged user message`
|
||||
**Log row:** the Task cell is `M5d1/03-loopd-carry-messages`.
|
||||
|
||||
## Goal
|
||||
|
||||
For epoch `n > 0`, the request prefix is the new system message, then one `ChatMessage::User`
|
||||
whose content is a fixed heading plus the carry, then the new epoch's records (spec 3.5). That
|
||||
user message is **not** a `LogRecord::User`: it is not a turn, not reported to Honcho, and not
|
||||
counted by `turns()`. Epoch 0 is unchanged (`carry` is `None`).
|
||||
|
||||
`run_turn` must send this prefix, or a compacted session's next live turn would omit the
|
||||
summary.
|
||||
|
||||
## Every place that changes
|
||||
|
||||
| Place | What happens there |
|
||||
|---|---|
|
||||
| `crates/loopd/src/baseline.rs` | `CARRY_HEADING`; `messages_with_carry`; `messages` calls it with `None` |
|
||||
| `crates/loopd/src/turn.rs` | the request uses `messages_with_carry` and `session.carry()` |
|
||||
| `crates/loopd/tests/carry_messages.rs` | new, given |
|
||||
| everything else | **no change**: not the existing `baseline` tests (they still call `messages`) |
|
||||
|
||||
## Files
|
||||
|
||||
- Copy, from `docs/plans/M5d1/files/` (the same path under it): the new test
|
||||
`crates/loopd/tests/carry_messages.rs`.
|
||||
- Modify: `crates/loopd/src/baseline.rs`, `crates/loopd/src/turn.rs`,
|
||||
`docs/implementer-log.md`.
|
||||
- Do not touch: any other file.
|
||||
|
||||
## Facts you need
|
||||
|
||||
- **The heading** is built from pieces so no file holds a chat-template tag whole (T33):
|
||||
|
||||
```rust
|
||||
pub const CARRY_HEADING: &str = concat!("Earlier ", "turns, summarised:");
|
||||
```
|
||||
|
||||
The user content is `format!("{CARRY_HEADING}\n\n{carry}")` when `carry` is `Some`. `None`
|
||||
inserts nothing.
|
||||
- **`messages` stays.** Existing tests and `limits.rs` call `messages(baseline, records)`. Make
|
||||
it `messages_with_carry(baseline, records, None)` so those tests do not change.
|
||||
- **`run_turn` today** (one place):
|
||||
|
||||
```rust
|
||||
messages: messages(baseline, session.records()),
|
||||
```
|
||||
|
||||
It must pass `session.carry()`. `Session::carry` is `Option<&str>` from task 02.
|
||||
- Insert the carry user **after** the system message and **before** walking `records`.
|
||||
- The items the edits use. This table is complete.
|
||||
|
||||
| Item | What it does |
|
||||
|---|---|
|
||||
| `concat!` | heading from pieces |
|
||||
| `Session::carry(&self) -> Option<&str>` | task 02 |
|
||||
| `ChatMessage::User { content }` | the carry, and log users |
|
||||
|
||||
## The edits, exactly
|
||||
|
||||
### `crates/loopd/src/baseline.rs`
|
||||
|
||||
**B1.** After the `Baseline` impl, replace `messages`. The lines
|
||||
|
||||
```rust
|
||||
/// The system message, then each log record replayed in order. Every record variant is named, so a
|
||||
/// new one is a compile error rather than a silent skip.
|
||||
pub fn messages(baseline: &Baseline, records: &[LogRecord]) -> Vec<ChatMessage> {
|
||||
let mut out = vec![ChatMessage::System {
|
||||
content: baseline.system.clone(),
|
||||
}];
|
||||
for record in records {
|
||||
```
|
||||
|
||||
become
|
||||
|
||||
```rust
|
||||
/// The heading in front of a carried summary. Built from pieces (T33).
|
||||
pub const CARRY_HEADING: &str = concat!("Earlier ", "turns, summarised:");
|
||||
|
||||
/// The system message, then each log record replayed in order. Every record variant is named, so a
|
||||
/// new one is a compile error rather than a silent skip.
|
||||
pub fn messages(baseline: &Baseline, records: &[LogRecord]) -> Vec<ChatMessage> {
|
||||
messages_with_carry(baseline, records, None)
|
||||
}
|
||||
|
||||
/// Like `messages`, and when `carry` is `Some` inserts one user message after the system
|
||||
/// prompt: the heading, a blank line, the summary. That user is not a `LogRecord::User`.
|
||||
pub fn messages_with_carry(
|
||||
baseline: &Baseline,
|
||||
records: &[LogRecord],
|
||||
carry: Option<&str>,
|
||||
) -> Vec<ChatMessage> {
|
||||
let mut out = vec![ChatMessage::System {
|
||||
content: baseline.system.clone(),
|
||||
}];
|
||||
if let Some(summary) = carry {
|
||||
out.push(ChatMessage::User {
|
||||
content: format!("{CARRY_HEADING}\n\n{summary}"),
|
||||
});
|
||||
}
|
||||
for record in records {
|
||||
```
|
||||
|
||||
The rest of the `match` is unchanged; do not touch it.
|
||||
|
||||
### `crates/loopd/src/turn.rs`
|
||||
|
||||
**T1.** The `use` line
|
||||
|
||||
```rust
|
||||
use crate::baseline::messages;
|
||||
```
|
||||
|
||||
becomes
|
||||
|
||||
```rust
|
||||
use crate::baseline::messages_with_carry;
|
||||
```
|
||||
|
||||
**T2.** The lines
|
||||
|
||||
```rust
|
||||
let request = ChatRequest {
|
||||
slot: rt.cfg.slots.main,
|
||||
messages: messages(baseline, session.records()),
|
||||
tools: baseline.tools.clone(),
|
||||
thinking: true,
|
||||
};
|
||||
```
|
||||
|
||||
become
|
||||
|
||||
```rust
|
||||
let request = ChatRequest {
|
||||
slot: rt.cfg.slots.main,
|
||||
messages: messages_with_carry(baseline, session.records(), session.carry()),
|
||||
tools: baseline.tools.clone(),
|
||||
thinking: true,
|
||||
};
|
||||
```
|
||||
|
||||
## Steps
|
||||
|
||||
- [ ] **1. Copy the given test.**
|
||||
|
||||
```sh
|
||||
cp docs/plans/M5d1/files/crates/loopd/tests/carry_messages.rs crates/loopd/tests/
|
||||
```
|
||||
|
||||
- [ ] **2. See it fail.** `cargo build --tests --keep-going -p loopd`, timeout 600000 ms: test
|
||||
target `carry_messages` does not compile (`E0432` unresolved import
|
||||
`loopd::baseline::{CARRY_HEADING, messages_with_carry}`). Every other target compiles.
|
||||
- [ ] **3. The edits.** B1, T1, T2. `cargo check -p loopd` prints no error and no warning.
|
||||
- [ ] **4. The tests.** `cargo test -p loopd --test carry_messages --test baseline --test turn`:
|
||||
every test passes, 5 in `carry_messages` and the others as before.
|
||||
- [ ] **5. Check.** `git diff --stat` lists exactly `crates/loopd/src/baseline.rs` and
|
||||
`crates/loopd/src/turn.rs`.
|
||||
- [ ] **6. Run the gate.** `cargo fmt --all`, then `make gate`, timeout 600000 ms. Last line
|
||||
`gate: ok`. Test count **1478** (1473 before, plus 5).
|
||||
- [ ] **7. Log and commit.** Task cell `M5d1/03-loopd-carry-messages`. Stage with
|
||||
`git add crates/loopd docs/implementer-log.md Cargo.lock`, then commit as `AGENTS.md` says.
|
||||
|
||||
## Done when
|
||||
|
||||
- `make gate` prints `gate: ok` with 1478 tests.
|
||||
- `git diff HEAD~1 --stat` lists only `crates/loopd/src/baseline.rs`, `crates/loopd/src/turn.rs`,
|
||||
`crates/loopd/tests/carry_messages.rs` and `docs/implementer-log.md`.
|
||||
|
||||
## Stop and report if
|
||||
|
||||
- The branch check fails; a "before" block is not found exactly once; step 2 differs; `cargo
|
||||
check` warns; a test fails; the gate count is not 1478; a comment does not fit its code.
|
||||
- You think `messages` should insert the carry. It must not: existing tests call `messages` and
|
||||
would see an extra user. `run_turn` is the one call that must pass `session.carry()`.
|
||||
|
||||
## What the given tests pin
|
||||
|
||||
Epoch 0 with `None` matches `messages`; a carry is a user after the system prompt and before the
|
||||
epoch's users; the carry is not in the log records; the heading is the `concat!` pieces; the
|
||||
first request of epoch 1 is not a prefix-extension of epoch 0's messages.
|
||||
@@ -0,0 +1,232 @@
|
||||
# M5d-1 task 04: Honcho reads every epoch file
|
||||
|
||||
**Branch:** `m5d1`. Check it: `git branch --show-current` prints `m5d1` and `git status --short`
|
||||
is empty; otherwise stop. Never switch branch.
|
||||
**Commit subject:** `memoryd: Honcho concatenates every epoch of a session`
|
||||
**Log row:** the Task cell is `M5d1/04-memoryd-honcho-epochs`.
|
||||
|
||||
## Goal
|
||||
|
||||
`memoryd` keys a turn as `Item::Turn { session, turn }` with no epoch (spec 3.2). After
|
||||
compaction, turn 3 may live in `1.jsonl`. `log_path` still names `0.jsonl` (existing tests pin
|
||||
that path). A new `read_session` concatenates every `<n>.jsonl` in epoch order. `TurnSync`
|
||||
walks that concatenation, and the worker's `send_turn` does too, so a turn in epoch 1 is
|
||||
decided and sent.
|
||||
|
||||
`turn_records` / `turns` do not change: they already count `User` records on a slice.
|
||||
|
||||
## Every place that changes
|
||||
|
||||
| Place | What happens there |
|
||||
|---|---|
|
||||
| `crates/memoryd/src/honcho/turns.rs` | `epoch_logs`; `read_session`; `records` and `resume` use it |
|
||||
| `crates/memoryd/src/honcho/worker.rs` | `send_turn` reads the concatenated log |
|
||||
| `crates/memoryd/tests/honcho_epochs.rs` | new, given |
|
||||
| everything else | **no change**: not `log_path`'s return value, not `decide.rs` |
|
||||
|
||||
## Files
|
||||
|
||||
- Copy, from `docs/plans/M5d1/files/` (the same path under it): the new test
|
||||
`crates/memoryd/tests/honcho_epochs.rs`.
|
||||
- Modify: `crates/memoryd/src/honcho/turns.rs`, `crates/memoryd/src/honcho/worker.rs`,
|
||||
`docs/implementer-log.md`.
|
||||
- Do not touch: any other file.
|
||||
|
||||
## Facts you need
|
||||
|
||||
- **`read_records` / `LogError`** already exist in `turns.rs`. `read_session` returns
|
||||
`Result<Vec<LogRecord>, LogError>` so a damaged line still names its file and
|
||||
`LOG_RUNBOOK`.
|
||||
- **Listing.** Under `sessions/<id>/`, every file named `<n>.jsonl` where `n` parses as `u32`,
|
||||
sorted by `n`. A missing session directory is empty, not an error (same as a missing
|
||||
`0.jsonl` today).
|
||||
- **`log_path` stays** `.../0.jsonl`. Do not change its body. `readable` in the worker still
|
||||
names that path when the concatenated read succeeds, so existing recovery log lines stay
|
||||
stable.
|
||||
- **`send_turn` today** calls `read_turn(&log_path(...), turn)`, which only sees epoch 0. After
|
||||
this task it loads `read_session`, then `turn_records` / `user_text` / `final_reply` from
|
||||
`crate::honcho::decide` (already `pub`).
|
||||
- `turns.rs` after this task stays under 500 lines (it is 332 now).
|
||||
- The items the edits use. This table is complete.
|
||||
|
||||
| Item | What it does |
|
||||
|---|---|
|
||||
| `read_records(&Path) -> Result<Vec<LogRecord>, LogError>` | one file, already in `turns.rs` |
|
||||
| `turn_records(&[LogRecord], u64) -> Option<&[LogRecord]>` | `decide.rs` |
|
||||
| `user_text(&[LogRecord]) -> (&str, Option<&str>)` | `decide.rs` |
|
||||
| `final_reply(&[LogRecord]) -> Option<&str>` | `decide.rs` |
|
||||
| `str::strip_suffix(".jsonl")`, `str::parse::<u32>` | epoch numbers |
|
||||
| `Unsent::Unreadable(String)` | worker, `e.to_string()` |
|
||||
|
||||
## The edits, exactly
|
||||
|
||||
### `crates/memoryd/src/honcho/turns.rs`
|
||||
|
||||
**H1.** The doc comment on `log_path`, the lines
|
||||
|
||||
```rust
|
||||
/// The path of `session`'s log under `sessions`: epoch 0, the only one `loopd` writes.
|
||||
pub fn log_path(sessions: &Path, session: &SessionId) -> PathBuf {
|
||||
```
|
||||
|
||||
become
|
||||
|
||||
```rust
|
||||
/// The path of `session`'s epoch-0 log under `sessions`. Later epochs are sibling files;
|
||||
/// `read_session` concatenates them. Existing tests pin this path.
|
||||
pub fn log_path(sessions: &Path, session: &SessionId) -> PathBuf {
|
||||
```
|
||||
|
||||
**H1b.** After `read_records` (the function that ends `Ok(records)` / `}`), insert these two
|
||||
functions. They sit below `LogError` and `read_records`, so both names resolve.
|
||||
|
||||
```rust
|
||||
/// Every `<n>.jsonl` of `session`, sorted by `n`. A missing directory is empty.
|
||||
pub fn epoch_logs(sessions: &Path, session: &SessionId) -> Result<Vec<PathBuf>, LogError> {
|
||||
let dir = sessions.join(session.as_str());
|
||||
let entries = match std::fs::read_dir(&dir) {
|
||||
Ok(e) => e,
|
||||
Err(e) if e.kind() == std::io::ErrorKind::NotFound => return Ok(Vec::new()),
|
||||
Err(e) => return Err(LogError::Io(format!("{}: {e}", dir.display()))),
|
||||
};
|
||||
let mut found: Vec<(u32, PathBuf)> = Vec::new();
|
||||
for entry in entries {
|
||||
let entry = match entry {
|
||||
Ok(e) => e,
|
||||
Err(e) => return Err(LogError::Io(format!("{}: {e}", dir.display()))),
|
||||
};
|
||||
let name = entry.file_name();
|
||||
let Some(name) = name.to_str() else { continue };
|
||||
let Some(stem) = name.strip_suffix(".jsonl") else { continue };
|
||||
let Ok(n) = stem.parse::<u32>() else { continue };
|
||||
found.push((n, entry.path()));
|
||||
}
|
||||
found.sort_by_key(|(n, _)| *n);
|
||||
Ok(found.into_iter().map(|(_, p)| p).collect())
|
||||
}
|
||||
|
||||
/// Every epoch's records, in epoch order. A missing session has none.
|
||||
pub fn read_session(
|
||||
sessions: &Path,
|
||||
session: &SessionId,
|
||||
) -> Result<Vec<LogRecord>, LogError> {
|
||||
let mut out = Vec::new();
|
||||
for path in epoch_logs(sessions, session)? {
|
||||
out.extend(read_records(&path)?);
|
||||
}
|
||||
Ok(out)
|
||||
}
|
||||
```
|
||||
|
||||
The next function after this insert is `finished`. Do not change `finished`.
|
||||
|
||||
**H2.** In `resume`, the line
|
||||
|
||||
```rust
|
||||
if let Ok(records) = read_log(&log_path(&self.sessions, &session)) {
|
||||
```
|
||||
|
||||
becomes
|
||||
|
||||
```rust
|
||||
if let Ok(records) = read_session(&self.sessions, &session) {
|
||||
```
|
||||
|
||||
**H3.** In `records`, the line
|
||||
|
||||
```rust
|
||||
match read_log(&log_path(&self.sessions, session)) {
|
||||
```
|
||||
|
||||
becomes
|
||||
|
||||
```rust
|
||||
match read_session(&self.sessions, session).map_err(|e| e.to_string()) {
|
||||
```
|
||||
|
||||
`read_log` already mapped `LogError` through `Display`; this keeps the logged line the same
|
||||
shape (`memoryd: {why}; {LOG_RUNBOOK}`).
|
||||
|
||||
### `crates/memoryd/src/honcho/worker.rs`
|
||||
|
||||
**W1.** The `use` line
|
||||
|
||||
```rust
|
||||
use crate::honcho::turns::{TurnSync, log_path};
|
||||
```
|
||||
|
||||
becomes
|
||||
|
||||
```rust
|
||||
use crate::honcho::decide::{final_reply, turn_records, user_text};
|
||||
use crate::honcho::turns::{TurnSync, log_path, read_session};
|
||||
```
|
||||
|
||||
**W2.** The body of `send_turn`, the lines
|
||||
|
||||
```rust
|
||||
let path = log_path(&self.sessions, session);
|
||||
let read = read_turn(&path, turn).map_err(Unsent::Unreadable)?;
|
||||
self.readable(&path);
|
||||
let (text, reply) = read.ok_or(Unsent::Local("log"))?;
|
||||
```
|
||||
|
||||
become
|
||||
|
||||
```rust
|
||||
let path = log_path(&self.sessions, session);
|
||||
let records = match read_session(&self.sessions, session) {
|
||||
Ok(records) => {
|
||||
self.readable(&path);
|
||||
records
|
||||
}
|
||||
Err(e) => return Err(Unsent::Unreadable(e.to_string())),
|
||||
};
|
||||
let slice = turn_records(&records, turn).ok_or(Unsent::Local("log"))?;
|
||||
let (text, _) = user_text(slice);
|
||||
let reply = final_reply(slice).ok_or(Unsent::Local("log"))?;
|
||||
```
|
||||
|
||||
The rest of `send_turn` is unchanged (`honcho.ok_or`, `batch`, `MAX_BATCH`, `add_messages`).
|
||||
`text` is `&str` from `user_text`; `reply` is `&str` from `final_reply`. `batch` already takes
|
||||
`&str`.
|
||||
|
||||
## Steps
|
||||
|
||||
- [ ] **1. Copy the given test.**
|
||||
|
||||
```sh
|
||||
cp docs/plans/M5d1/files/crates/memoryd/tests/honcho_epochs.rs crates/memoryd/tests/
|
||||
```
|
||||
|
||||
- [ ] **2. See it fail.** `cargo build --tests --keep-going -p memoryd`, timeout 600000 ms: test
|
||||
target `honcho_epochs` does not compile (`E0432` unresolved import
|
||||
`memoryd::honcho::turns::read_session`). Every other target compiles.
|
||||
- [ ] **3. The edits.** H1, H1b, H2, H3, W1, W2. `cargo check -p memoryd`
|
||||
prints no error and no warning. `wc -l crates/memoryd/src/honcho/turns.rs` and
|
||||
`wc -l crates/memoryd/src/honcho/worker.rs` are each ≤ 500.
|
||||
- [ ] **4. The tests.** `cargo test -p memoryd --test honcho_epochs --test honcho_turns --test
|
||||
honcho_worker`: every test passes, 5 in `honcho_epochs` and the others as before.
|
||||
- [ ] **5. Check.** `git diff --stat` lists exactly `crates/memoryd/src/honcho/turns.rs` and
|
||||
`crates/memoryd/src/honcho/worker.rs`.
|
||||
- [ ] **6. Run the gate.** `cargo fmt --all`, then `make gate`, timeout 600000 ms. Last line
|
||||
`gate: ok`. Test count **1483** (1478 before, plus 5).
|
||||
- [ ] **7. Log and commit.** Task cell `M5d1/04-memoryd-honcho-epochs`. Stage with
|
||||
`git add crates/memoryd docs/implementer-log.md Cargo.lock`, then commit as `AGENTS.md` says.
|
||||
|
||||
## Done when
|
||||
|
||||
- `make gate` prints `gate: ok` with 1483 tests.
|
||||
- `git diff HEAD~1 --stat` lists only the two `honcho` sources, `honcho_epochs.rs`, and the log.
|
||||
|
||||
## Stop and report if
|
||||
|
||||
- The branch check fails; a "before" block is not found exactly once; step 2 differs; a file
|
||||
exceeds 500 lines; a test fails; the gate count is not 1483.
|
||||
- You think `log_path` should return every file. It must still return `0.jsonl`.
|
||||
|
||||
## What the given tests pin
|
||||
|
||||
Two users in epoch 0 and one in epoch 1 is `turns() == 3` and `turn_records(..., 3)` is the
|
||||
epoch-1 user; epoch 0 alone still reads; a damaged `1.jsonl` names that file; `TurnSync` decides
|
||||
turn 3 from epoch 1; a missing session directory is empty.
|
||||
@@ -0,0 +1,130 @@
|
||||
# M5d-1 task 05: when to compact, and the fallback summary
|
||||
|
||||
**Branch:** `m5d1`. Check it: `git branch --show-current` prints `m5d1` and `git status --short`
|
||||
is empty; otherwise stop. Never switch branch.
|
||||
**Commit subject:** `loopd: compaction threshold and the fallback summary`
|
||||
**Log row:** the Task cell is `M5d1/05-loopd-compact-due`.
|
||||
|
||||
## Goal
|
||||
|
||||
A session is due to compact when the current epoch's last `Usage` has
|
||||
`cache_n + prompt_n` at or above `threshold × expect.n_ctx` (spec 3.3). There is no idle timer.
|
||||
If the summary call is context-full (task 06), the fallback is the epoch's `User` texts, newest
|
||||
last, cut to `summary_chars` on a character boundary (spec 3.4).
|
||||
|
||||
This task is those two pure functions, the `compact` module, and the starter `config/compact.md`.
|
||||
No inference, no disk besides copying the starter.
|
||||
|
||||
## Every place that changes
|
||||
|
||||
| Place | What happens there |
|
||||
|---|---|
|
||||
| `crates/loopd/src/compact.rs` | new: `due`, `fallback_summary`, `cap_chars` |
|
||||
| `crates/loopd/src/lib.rs` | `pub mod compact;` |
|
||||
| `config/compact.md` | new, given starter |
|
||||
| `crates/loopd/tests/compact_due.rs` | new, given |
|
||||
| everything else | **no change** |
|
||||
|
||||
## Files
|
||||
|
||||
- Copy, from `docs/plans/M5d1/files/` (the same path under it): the new test
|
||||
`crates/loopd/tests/compact_due.rs` and the starter `config/compact.md`.
|
||||
- Create: `crates/loopd/src/compact.rs` (skeleton below).
|
||||
- Modify: `crates/loopd/src/lib.rs`, `docs/implementer-log.md`.
|
||||
- Do not touch: any other file.
|
||||
|
||||
## Facts you need
|
||||
|
||||
- **Depth is saturating.** `cache_n.saturating_add(prompt_n) as f64 >= threshold * (n_ctx as f64)`.
|
||||
0.85 × 262144 is 222822.4, so 222822 is not due and 222823 is. 0.85 × 1000 is 850 exactly.
|
||||
- **Cast.** `u64 as f64` is exact for these magnitudes (n_ctx is 262144). Do not use `as` on
|
||||
untrusted indexes; these are our own config and the server's counts.
|
||||
- **Fallback join.** `"\n\n"` between texts. Newest last: reverse the `User` contents, then join,
|
||||
then cap.
|
||||
- **Cap.** `s.chars().take(n).collect::<String>()`. `n == 0` is empty. A multi-byte character
|
||||
is not split.
|
||||
- **`config/compact.md`** is copied like `config/system.md`. Task 06 reads it from
|
||||
`Config::compact_prompt`.
|
||||
- The items the edits use. This table is complete.
|
||||
|
||||
| Item | What it does |
|
||||
|---|---|
|
||||
| `bxcore::Usage` | `cache_n`, `prompt_n` |
|
||||
| `LogRecord::User { content, .. }` | fallback texts |
|
||||
| `str::chars`, `Iterator::take` | the cap |
|
||||
|
||||
## Skeleton: `crates/loopd/src/compact.rs`
|
||||
|
||||
Write this file, then fill the two `todo!()` bodies from their comments and delete the
|
||||
comments. Each comment is at most six lines.
|
||||
|
||||
```rust
|
||||
//! Compaction: when a session starts a new epoch, the summary call, and writing the next files
|
||||
//! (M5d spec, section 3). Later tasks fill the call and the write.
|
||||
|
||||
use bxcore::{LogRecord, Usage};
|
||||
|
||||
/// True when `cache_n + prompt_n` is at or above `threshold * n_ctx`.
|
||||
pub fn due(usage: &Usage, threshold: f64, n_ctx: u64) -> bool {
|
||||
// saturating_add the two counts; compare as f64 to `threshold * (n_ctx as f64)`.
|
||||
todo!()
|
||||
}
|
||||
|
||||
/// The epoch's `User` texts, newest last, joined with `"\n\n"`, then `cap_chars`.
|
||||
pub fn fallback_summary(records: &[LogRecord], summary_chars: usize) -> String {
|
||||
// Collect `&str` contents of every `User`, reverse, join `"\n\n"`, then `cap_chars`.
|
||||
todo!()
|
||||
}
|
||||
|
||||
pub(crate) fn cap_chars(s: &str, n: usize) -> String {
|
||||
s.chars().take(n).collect()
|
||||
}
|
||||
```
|
||||
|
||||
### `crates/loopd/src/lib.rs`
|
||||
|
||||
**L1.** After `pub mod channel;` add `pub mod compact;` (keep rustfmt's order; if rustfmt
|
||||
moves the line, accept that).
|
||||
|
||||
## Steps
|
||||
|
||||
- [ ] **1. Copy the given files.**
|
||||
|
||||
```sh
|
||||
cp docs/plans/M5d1/files/crates/loopd/tests/compact_due.rs crates/loopd/tests/
|
||||
cp docs/plans/M5d1/files/config/compact.md config/compact.md
|
||||
```
|
||||
|
||||
- [ ] **2. Write the skeleton** at `crates/loopd/src/compact.rs` and make L1. See it fail:
|
||||
`cargo test -p loopd --test compact_due`, timeout 600000 ms: the two `due` tests and the
|
||||
three `fallback` tests panic on `todo!()` (or fail their asserts if you filled the bodies
|
||||
early). If the test target does not compile (`E0432` unresolved `loopd::compact`), the
|
||||
`mod` line is missing.
|
||||
- [ ] **3. Fill `due` and `fallback_summary`.** Delete each instruction comment. `cargo check
|
||||
-p loopd` prints no error and no warning.
|
||||
- [ ] **4. The tests.** `cargo test -p loopd --test compact_due`: 6 passed.
|
||||
- [ ] **5. Check.** `git diff --stat` lists `crates/loopd/src/lib.rs`,
|
||||
`crates/loopd/src/compact.rs`, `config/compact.md`. `git status --short` shows
|
||||
`compact_due.rs` as new.
|
||||
- [ ] **6. Run the gate.** `cargo fmt --all`, then `make gate`, timeout 600000 ms. Last line
|
||||
`gate: ok`. Test count **1489** (1483 before, plus 6).
|
||||
- [ ] **7. Log and commit.** Task cell `M5d1/05-loopd-compact-due`. Stage with
|
||||
`git add crates/loopd config/compact.md docs/implementer-log.md Cargo.lock`, then commit
|
||||
as `AGENTS.md` says.
|
||||
|
||||
## Done when
|
||||
|
||||
- `make gate` prints `gate: ok` with 1489 tests.
|
||||
- `git diff HEAD~1 --stat` lists only `crates/loopd/src/lib.rs`, `crates/loopd/src/compact.rs`,
|
||||
`crates/loopd/tests/compact_due.rs`, `config/compact.md` and the log.
|
||||
|
||||
## Stop and report if
|
||||
|
||||
- The branch check fails; step 2 fails to compile for a reason other than the missing module
|
||||
or `todo!()`; a test fails; the gate count is not 1489; you add an idle timer. There is none.
|
||||
|
||||
## What the given tests pin
|
||||
|
||||
849/1000 is not due, 850 is; 222822/262144 is not due, 222823 is; threshold 1.0 is the full
|
||||
window; fallback is newest-last with `"\n\n"`; the cap is in characters on a boundary; no
|
||||
users is empty.
|
||||
@@ -0,0 +1,213 @@
|
||||
# M5d-1 task 06: the summary call, and `session-compact-failed`
|
||||
|
||||
**Branch:** `m5d1`. Check it: `git branch --show-current` prints `m5d1` and `git status --short`
|
||||
is empty; otherwise stop. Never switch branch.
|
||||
**Commit subject:** `loopd: compact summary on the background slot`
|
||||
**Log row:** the Task cell is `M5d1/06-loopd-compact-summary`.
|
||||
|
||||
## Goal
|
||||
|
||||
Compaction asks for a summary of the current epoch on the **background** slot, with **no tools**,
|
||||
thinking on (spec 3.4). The last user message is `compact.md` (task 01's `compact_prompt`),
|
||||
trimmed at the end. The completion is tag-neutralised and cut to `summary_chars`.
|
||||
|
||||
If that call is context-full (`exceed_context_size_error`), use task 05's `fallback_summary`.
|
||||
If `compact.md` cannot be read, or the call fails for any other reason, return
|
||||
`CompactError::Failed` whose `Display` ends with the runbook pointer
|
||||
`see docs/runbook.md#session-compact-failed`. There is no fallback for a missing prompt.
|
||||
|
||||
This task does not write the next epoch. Task 07 does.
|
||||
|
||||
## Every place that changes
|
||||
|
||||
| Place | What happens there |
|
||||
|---|---|
|
||||
| `crates/loopd/src/compact.rs` | `CompactError`; `summarise` (edits E1, E2) |
|
||||
| `docs/runbook.md` | append the given `session-compact-failed` entry |
|
||||
| `crates/loopd/tests/compact_summary.rs` | new, given |
|
||||
| everything else | **no change**: not `turn.rs`, not `channel.rs`, not `session.rs` |
|
||||
|
||||
## Files
|
||||
|
||||
- Copy, from `docs/plans/M5d1/files/` (the same path under it): the new test
|
||||
`crates/loopd/tests/compact_summary.rs`.
|
||||
- Append, from `docs/plans/M5d1/files/runbook-session-compact-failed.md`, to `docs/runbook.md`.
|
||||
- Modify: `crates/loopd/src/compact.rs`, `docs/runbook.md`, `docs/implementer-log.md`.
|
||||
- Do not touch: any other file. `config/compact.md` is already in the tree from task 05.
|
||||
|
||||
## Facts you need
|
||||
|
||||
- **The request.** `ChatRequest { slot: cfg.slots.background, messages, tools: Vec::new(),
|
||||
thinking: true }`. Empty `tools` is omitted from the JSON (`skip_serializing_if` on the body).
|
||||
The given test asserts `body.get("tools").is_none()` and `id_slot == 1` (the test config's
|
||||
background slot).
|
||||
- **The messages.** `messages_with_carry(session.baseline(), session.records(), session.carry())`,
|
||||
then one `ChatMessage::User` whose content is `compact.md` after `trim_end`. Epoch 0 has no
|
||||
carry; later epochs keep the carried summary in the prefix so the model summarises what the
|
||||
session actually sends.
|
||||
- **Read the prompt** with `std::fs::read_to_string(&cfg.compact_prompt())`. An I/O error is
|
||||
`Failed { detail: format!("{}: {e}", path.display()) }` so the path (and `compact.md`) appears
|
||||
in the message.
|
||||
- **The call.** `client.chat_with_retry(&request, &mut |_| {})`. A 400 that is not context-full
|
||||
is not retried (`is_retryable` is already false for it).
|
||||
- **Context-full.** `crate::turn::is_context_full(&e)` then `fallback_summary(session.records(),
|
||||
cfg.compact.summary_chars)`. Do not fall back for a missing file or any other `InferError`.
|
||||
- **The text.** `completion.content.unwrap_or_default()`, then `bxcore::sanitize::neutralise`,
|
||||
then `cap_chars(..., cfg.compact.summary_chars)`. Neutralise first so a tag is not split by
|
||||
the cap.
|
||||
- **`Display`.** `"{detail}; see docs/runbook.md#session-compact-failed"`. The given tests box
|
||||
the error as `dyn Error` and check `to_string()`.
|
||||
- **`compact.rs` stays ≤ 500 lines.** After this task it is about 120.
|
||||
- The items the edits use. This table is complete.
|
||||
|
||||
| Item | What it does |
|
||||
|---|---|
|
||||
| `Config::compact_prompt` | task 01, `compact.md` beside `system.md` |
|
||||
| `messages_with_carry` | task 03 |
|
||||
| `Client::chat_with_retry` | same as a live turn |
|
||||
| `turn::is_context_full` | the one 400 `exceed_context_size_error` |
|
||||
| `bxcore::sanitize::neutralise` | tags to a visible look-alike |
|
||||
| `fallback_summary`, `cap_chars` | task 05 |
|
||||
| `ChatRequest.tools: Vec::new()` | omits `tools` on the wire |
|
||||
|
||||
## The edits, exactly
|
||||
|
||||
Nothing else in `compact.rs` changes. Each quoted "before" block occurs exactly once after
|
||||
task 05. Keep the fields in the order shown.
|
||||
|
||||
### `crates/loopd/src/compact.rs`
|
||||
|
||||
**E1.** The module doc and the `use` line. The lines
|
||||
|
||||
```rust
|
||||
//! Compaction: when a session starts a new epoch, the summary call, and writing the next files
|
||||
//! (M5d spec, section 3). Later tasks fill the call and the write.
|
||||
|
||||
use bxcore::{LogRecord, Usage};
|
||||
```
|
||||
|
||||
become
|
||||
|
||||
```rust
|
||||
//! Compaction: when a session starts a new epoch, the summary call, and writing the next files
|
||||
//! (M5d spec, section 3). Task 07 writes the next files; task 08 calls this after a turn.
|
||||
|
||||
use bxcore::sanitize::neutralise;
|
||||
use bxcore::{LogRecord, Usage};
|
||||
|
||||
use crate::baseline::messages_with_carry;
|
||||
use crate::config::Config;
|
||||
use crate::llama::{ChatMessage, ChatRequest, Client};
|
||||
use crate::session::Session;
|
||||
use crate::turn::is_context_full;
|
||||
|
||||
/// Why a summary call or a next-epoch write failed. Every variant's `Display` ends with the
|
||||
/// runbook pointer `session-compact-failed`.
|
||||
#[derive(Debug)]
|
||||
pub enum CompactError {
|
||||
Failed { detail: String },
|
||||
}
|
||||
|
||||
impl std::fmt::Display for CompactError {
|
||||
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||
match self {
|
||||
CompactError::Failed { detail } => write!(
|
||||
f,
|
||||
"{detail}; see docs/runbook.md#session-compact-failed"
|
||||
),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl std::error::Error for CompactError {}
|
||||
```
|
||||
|
||||
**E2.** After `cap_chars` (the function that ends `s.chars().take(n).collect()` / `}`), add
|
||||
`summarise`. Copy this function whole. There is no `todo!()`.
|
||||
|
||||
```rust
|
||||
/// One completion on the background slot: no tools, thinking on, last user `compact.md`.
|
||||
/// Context-full uses `fallback_summary`. A missing prompt, or any other infer error, is
|
||||
/// `CompactError::Failed` with the runbook pointer. No fallback for a missing prompt.
|
||||
pub fn summarise(cfg: &Config, client: &Client, session: &Session) -> Result<String, CompactError> {
|
||||
let path = cfg.compact_prompt();
|
||||
let prompt = std::fs::read_to_string(&path).map_err(|e| CompactError::Failed {
|
||||
detail: format!("{}: {e}", path.display()),
|
||||
})?;
|
||||
let mut messages = messages_with_carry(session.baseline(), session.records(), session.carry());
|
||||
messages.push(ChatMessage::User {
|
||||
content: prompt.trim_end().to_string(),
|
||||
});
|
||||
let request = ChatRequest {
|
||||
slot: cfg.slots.background,
|
||||
messages,
|
||||
tools: Vec::new(),
|
||||
thinking: true,
|
||||
};
|
||||
match client.chat_with_retry(&request, &mut |_| {}) {
|
||||
Ok(completion) => {
|
||||
let text = completion.content.unwrap_or_default();
|
||||
let text = neutralise(&text);
|
||||
Ok(cap_chars(text.as_ref(), cfg.compact.summary_chars))
|
||||
}
|
||||
Err(e) if is_context_full(&e) => {
|
||||
Ok(fallback_summary(session.records(), cfg.compact.summary_chars))
|
||||
}
|
||||
Err(e) => Err(CompactError::Failed {
|
||||
detail: e.to_string(),
|
||||
}),
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Steps
|
||||
|
||||
- [ ] **1. Copy the given test.**
|
||||
|
||||
```sh
|
||||
cp docs/plans/M5d1/files/crates/loopd/tests/compact_summary.rs crates/loopd/tests/
|
||||
```
|
||||
|
||||
- [ ] **2. See it fail.** `cargo build --tests --keep-going -p loopd`, timeout 600000 ms: test
|
||||
target `compact_summary` does not compile (`E0432` unresolved import `loopd::compact::CompactError`
|
||||
and/or `E0599` no function `summarise`). Every other target compiles. `compact_due` still
|
||||
compiles.
|
||||
- [ ] **3. The edits.** E1, E2. `cargo check -p loopd` prints no error and no warning.
|
||||
`wc -l crates/loopd/src/compact.rs` prints a number ≤ 500.
|
||||
- [ ] **4. The tests.** `cargo test -p loopd --test compact_summary --test compact_due`: every
|
||||
test passes, 6 in `compact_summary` and 6 in `compact_due`.
|
||||
- [ ] **5. The runbook.** From the repository root:
|
||||
|
||||
```sh
|
||||
{ echo; cat docs/plans/M5d1/files/runbook-session-compact-failed.md; } >> docs/runbook.md
|
||||
```
|
||||
|
||||
Then `grep -c '^## session-compact-failed$' docs/runbook.md` prints `1`.
|
||||
- [ ] **6. Check.** `git diff --stat` lists `crates/loopd/src/compact.rs` and `docs/runbook.md`.
|
||||
`git status --short` shows `compact_summary.rs` as new.
|
||||
- [ ] **7. Run the gate.** `cargo fmt --all`, then `make gate`, timeout 600000 ms. Last line
|
||||
`gate: ok`. Test count **1495** (1489 before, plus 6).
|
||||
- [ ] **8. Log and commit.** Task cell `M5d1/06-loopd-compact-summary`. Stage with
|
||||
`git add crates/loopd docs/runbook.md docs/implementer-log.md Cargo.lock`, then commit as
|
||||
`AGENTS.md` says.
|
||||
|
||||
## Done when
|
||||
|
||||
- `make gate` prints `gate: ok` with 1495 tests.
|
||||
- `git diff HEAD~1 --stat` lists only `crates/loopd/src/compact.rs`,
|
||||
`crates/loopd/tests/compact_summary.rs`, `docs/runbook.md` and the log.
|
||||
|
||||
## Stop and report if
|
||||
|
||||
- The branch check fails; a "before" block is not found exactly once; step 2 differs; `cargo
|
||||
check` warns; a test fails; the gate count is not 1495; a comment does not fit its code;
|
||||
`compact.rs` exceeds 500 lines; you add a fallback for a missing `compact.md`; you send tools
|
||||
on the summary request; you use `slots.main` for the summary.
|
||||
|
||||
## What the given tests pin
|
||||
|
||||
The summary request is `id_slot` of the background slot, omits `tools`, has thinking on, and its
|
||||
last user is `compact.md` after `trim_end`; the completion is capped to `summary_chars`; a
|
||||
context-full summary is the epoch's `User` texts; tags in the completion are neutralised; a
|
||||
missing `compact.md` names the file and the runbook; a non-context-full infer error is
|
||||
`CompactError::Failed` with the same pointer.
|
||||
@@ -0,0 +1,268 @@
|
||||
# M5d-1 task 07: writing the next epoch
|
||||
|
||||
**Branch:** `m5d1`. Check it: `git branch --show-current` prints `m5d1` and `git status --short`
|
||||
is empty; otherwise stop. Never switch branch.
|
||||
**Commit subject:** `loopd: write the next epoch from a compaction summary`
|
||||
**Log row:** the Task cell is `M5d1/07-loopd-compact-advance`.
|
||||
|
||||
## Goal
|
||||
|
||||
Given a summary string, `commit` starts the next epoch (spec 3.5):
|
||||
|
||||
1. Append `EpochEnd { next, summary }` to the current log, unless that log already ends with an
|
||||
`EpochEnd` (a compact that stopped after the end record and before the next files).
|
||||
2. Ask `memoryd` for a fresh block; write `<n>.baseline.json` from `assemble_with` (so identity
|
||||
edits applied in the old epoch land here, and `registry.core_schemas()` is the new tools
|
||||
array).
|
||||
3. Create `<n>.jsonl` with `SessionStart { epoch: n, slot: the session's live slot, baseline:
|
||||
hash of the new file }`.
|
||||
4. The in-memory `Session` switches: new baseline, records that are only that `SessionStart`,
|
||||
`carry = Some(summary)`, `prior_users` / `prior_untrusted` accumulated from the old records.
|
||||
|
||||
A crash after step 1 and before the next files exist reopens epoch 0 with `EpochEnd` last (task
|
||||
02). The next `commit` resumes: it uses the summary already in that `EpochEnd` as the carry and
|
||||
does not write a second `EpochEnd`. The summary argument is ignored in that case.
|
||||
|
||||
## Every place that changes
|
||||
|
||||
| Place | What happens there |
|
||||
|---|---|
|
||||
| `crates/loopd/src/session.rs` | `begin_epoch` (edit S1) |
|
||||
| `crates/loopd/src/compact.rs` | `commit` (edit C1) |
|
||||
| `crates/loopd/tests/compact_advance.rs` | new, given |
|
||||
| everything else | **no change**: not `turn.rs`, not `channel.rs` |
|
||||
|
||||
## Files
|
||||
|
||||
- Copy, from `docs/plans/M5d1/files/` (the same path under it): the new test
|
||||
`crates/loopd/tests/compact_advance.rs`.
|
||||
- Modify: `crates/loopd/src/session.rs`, `crates/loopd/src/compact.rs`,
|
||||
`docs/implementer-log.md`.
|
||||
- Do not touch: any other file.
|
||||
|
||||
## Facts you need
|
||||
|
||||
- **`begin_epoch` is `pub(crate)`** on `Session` so `compact.rs` can switch the private fields.
|
||||
Tests call `compact::commit`, never `begin_epoch`.
|
||||
- **Order inside `begin_epoch`.** Write `<n>.baseline.json` (overwrite is fine on resume),
|
||||
`create_new` the log file, then set `epoch` / `baseline` / `carry` / empty `records` / the new
|
||||
`File`, **then** `persist` `SessionStart`. `log_path` uses `self.epoch`, so `epoch` must be
|
||||
`next` before that persist. Accumulate `prior_users` and `prior_untrusted` from the **old**
|
||||
`records` before clearing them. Do not reset `next_call` or `slot`.
|
||||
- **Resume.** If `session.records().last()` is `EpochEnd { next, summary }`, skip the append;
|
||||
`next` and `summary` (cloned) are what `begin_epoch` gets. Otherwise append `EpochEnd` with
|
||||
`next = Epoch(session.epoch().0.saturating_add(1))` and the summary argument, then
|
||||
`begin_epoch`.
|
||||
- **Memory.** `memory.block().map_err(|e| CompactError::Failed { detail: e })?`. Then
|
||||
`Baseline::assemble_with(cfg, registry, &block)`. `assemble_with` already calls
|
||||
`registry.core_schemas()`, so a new core tool lands at this boundary. Map assemble and session
|
||||
errors to `Failed { detail: e.to_string() }`.
|
||||
- **`NoMemory.block()`** is `Ok("")`. The given test `the_new_baseline_takes_a_fresh_memory_block`
|
||||
uses a `MemorySource` that returns a distinctive sentence.
|
||||
- **`session.rs` stays ≤ 500 lines.** After task 02 it is about 430; `begin_epoch` is about 40.
|
||||
- **`compact.rs` stays ≤ 500 lines.** After this task it is about 180.
|
||||
- The items the edits use. This table is complete.
|
||||
|
||||
| Item | What it does |
|
||||
|---|---|
|
||||
| `Baseline::assemble_with` | new system + memory block + `registry.core_schemas()` |
|
||||
| `Baseline::to_json` / `hash` | the file and `SessionStart.baseline` |
|
||||
| `OpenOptions::create_new(true).append(true)` | the new log, as `create` does |
|
||||
| `count_users` / `any_untrusted` | task 02, module-private in `session.rs` |
|
||||
| `Session::persist` | private; `begin_epoch` is in the same impl |
|
||||
| `MemorySource::block` | fresh text; `Err` is `Failed` |
|
||||
| `Timestamp::now` | `EpochEnd` and `SessionStart` |
|
||||
| `Epoch(n.saturating_add(1))` | the next number |
|
||||
|
||||
## The edits, exactly
|
||||
|
||||
Each quoted "before" block occurs exactly once after task 06 (session.rs is as task 02 left it).
|
||||
|
||||
### `crates/loopd/src/session.rs`
|
||||
|
||||
**S1.** `log_path`, the lines
|
||||
|
||||
```rust
|
||||
fn log_path(&self) -> PathBuf {
|
||||
self.dir.join(format!("{}.jsonl", self.epoch.0))
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
become
|
||||
|
||||
```rust
|
||||
fn log_path(&self) -> PathBuf {
|
||||
self.dir.join(format!("{}.jsonl", self.epoch.0))
|
||||
}
|
||||
|
||||
/// Write `<n>.baseline.json` and `<n>.jsonl` with `SessionStart`, and switch this session
|
||||
/// onto that epoch. The live slot and `next_call` stay. Callers pass the carry that
|
||||
/// `messages_with_carry` will insert.
|
||||
pub(crate) fn begin_epoch(
|
||||
&mut self,
|
||||
next: Epoch,
|
||||
baseline: Baseline,
|
||||
carry: String,
|
||||
) -> Result<(), SessionError> {
|
||||
let n = next.0;
|
||||
let baseline_path = self.dir.join(format!("{n}.baseline.json"));
|
||||
let json = baseline.to_json().map_err(SessionError::Encode)?;
|
||||
std::fs::write(&baseline_path, json)
|
||||
.map_err(|e| SessionError::Io(baseline_path.clone(), e))?;
|
||||
let log_path = self.dir.join(format!("{n}.jsonl"));
|
||||
let log = OpenOptions::new()
|
||||
.create_new(true)
|
||||
.append(true)
|
||||
.open(&log_path)
|
||||
.map_err(|e| SessionError::Io(log_path.clone(), e))?;
|
||||
self.prior_users = self.prior_users.saturating_add(count_users(&self.records));
|
||||
self.prior_untrusted = self.prior_untrusted || any_untrusted(&self.records);
|
||||
self.epoch = next;
|
||||
self.baseline = baseline;
|
||||
self.carry = Some(carry);
|
||||
self.records = Vec::new();
|
||||
self.log = log;
|
||||
let hash = self.baseline.hash().map_err(SessionError::Baseline)?;
|
||||
self.persist(LogRecord::SessionStart {
|
||||
time: Timestamp::now(),
|
||||
session: self.id.clone(),
|
||||
epoch: next,
|
||||
slot: self.slot,
|
||||
baseline: hash,
|
||||
})
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The helpers after the `impl` (`complete_epochs` and the rest) stay where they are. Do not move
|
||||
them.
|
||||
|
||||
### `crates/loopd/src/compact.rs`
|
||||
|
||||
**C1.** After `summarise` (the function that ends the `match client.chat_with_retry` / `}`), add
|
||||
the `use` items this function needs by extending the existing `use crate::` group, then add
|
||||
`commit`. The current crate uses (after task 06) are:
|
||||
|
||||
```rust
|
||||
use crate::baseline::messages_with_carry;
|
||||
use crate::config::Config;
|
||||
use crate::llama::{ChatMessage, ChatRequest, Client};
|
||||
use crate::session::Session;
|
||||
use crate::turn::is_context_full;
|
||||
```
|
||||
|
||||
They become
|
||||
|
||||
```rust
|
||||
use crate::baseline::{messages_with_carry, Baseline};
|
||||
use crate::config::Config;
|
||||
use crate::llama::{ChatMessage, ChatRequest, Client};
|
||||
use crate::memory::MemorySource;
|
||||
use crate::session::Session;
|
||||
use crate::tools::Registry;
|
||||
use crate::turn::is_context_full;
|
||||
```
|
||||
|
||||
And the bxcore use
|
||||
|
||||
```rust
|
||||
use bxcore::{LogRecord, Usage};
|
||||
```
|
||||
|
||||
becomes
|
||||
|
||||
```rust
|
||||
use bxcore::{Epoch, LogRecord, Timestamp, Usage};
|
||||
```
|
||||
|
||||
Then this function, whole, after `summarise`:
|
||||
|
||||
```rust
|
||||
/// Append `EpochEnd` (unless it is already last), rebuild the baseline from `memory`, and
|
||||
/// switch `session` onto the next epoch. When the current log already ends with `EpochEnd`,
|
||||
/// that record's summary is the carry and the argument is ignored.
|
||||
pub fn commit(
|
||||
session: &mut Session,
|
||||
cfg: &Config,
|
||||
registry: &Registry,
|
||||
memory: &dyn MemorySource,
|
||||
summary: String,
|
||||
) -> Result<(), CompactError> {
|
||||
let (next, summary) = match session.records().last() {
|
||||
Some(LogRecord::EpochEnd { next, summary, .. }) => (*next, summary.clone()),
|
||||
_ => {
|
||||
let next = Epoch(session.epoch().0.saturating_add(1));
|
||||
session
|
||||
.append(LogRecord::EpochEnd {
|
||||
time: Timestamp::now(),
|
||||
next,
|
||||
summary: summary.clone(),
|
||||
})
|
||||
.map_err(|e| CompactError::Failed {
|
||||
detail: e.to_string(),
|
||||
})?;
|
||||
(next, summary)
|
||||
}
|
||||
};
|
||||
let block = memory.block().map_err(|e| CompactError::Failed { detail: e })?;
|
||||
let baseline = Baseline::assemble_with(cfg, registry, &block).map_err(|e| {
|
||||
CompactError::Failed {
|
||||
detail: e.to_string(),
|
||||
}
|
||||
})?;
|
||||
session
|
||||
.begin_epoch(next, baseline, summary)
|
||||
.map_err(|e| CompactError::Failed {
|
||||
detail: e.to_string(),
|
||||
})?;
|
||||
Ok(())
|
||||
}
|
||||
```
|
||||
|
||||
If rustfmt splits the `map_err` closures, accept that.
|
||||
|
||||
## Steps
|
||||
|
||||
- [ ] **1. Copy the given test.**
|
||||
|
||||
```sh
|
||||
cp docs/plans/M5d1/files/crates/loopd/tests/compact_advance.rs crates/loopd/tests/
|
||||
```
|
||||
|
||||
- [ ] **2. See it fail.** `cargo build --tests --keep-going -p loopd`, timeout 600000 ms: test
|
||||
target `compact_advance` does not compile (`E0599` no function `commit`). Every other target
|
||||
compiles.
|
||||
- [ ] **3. The edits.** S1, then C1. `cargo check -p loopd` prints no error and no warning.
|
||||
`wc -l crates/loopd/src/session.rs` and `wc -l crates/loopd/src/compact.rs` each print a
|
||||
number ≤ 500.
|
||||
- [ ] **4. The tests.** `cargo test -p loopd --test compact_advance --test session_epochs --test
|
||||
compact_summary`: every test passes, 6 in `compact_advance`, 9 in `session_epochs`, 6 in
|
||||
`compact_summary`.
|
||||
- [ ] **5. Check.** `git diff --stat` lists exactly `crates/loopd/src/session.rs` and
|
||||
`crates/loopd/src/compact.rs`. `git status --short` shows `compact_advance.rs` as new.
|
||||
- [ ] **6. Run the gate.** `cargo fmt --all`, then `make gate`, timeout 600000 ms. Last line
|
||||
`gate: ok`. Test count **1501** (1495 before, plus 6).
|
||||
- [ ] **7. Log and commit.** Task cell `M5d1/07-loopd-compact-advance`. Stage with
|
||||
`git add crates/loopd docs/implementer-log.md Cargo.lock`, then commit as `AGENTS.md` says.
|
||||
|
||||
## Done when
|
||||
|
||||
- `make gate` prints `gate: ok` with 1501 tests.
|
||||
- `git diff HEAD~1 --stat` lists only `crates/loopd/src/session.rs`,
|
||||
`crates/loopd/src/compact.rs`, `crates/loopd/tests/compact_advance.rs` and the log.
|
||||
|
||||
## Stop and report if
|
||||
|
||||
- The branch check fails; a "before" block is not found exactly once; step 2 differs; `cargo
|
||||
check` warns; a test fails; the gate count is not 1501; a file exceeds 500 lines; you write a
|
||||
second `EpochEnd` on resume; you reset `slot` or `next_call`; you log the summary as a
|
||||
`LogRecord::User`.
|
||||
|
||||
## What the given tests pin
|
||||
|
||||
`commit` writes `EpochEnd` on the old log, a new baseline file, and a new log whose
|
||||
`SessionStart` has the next epoch; the new baseline contains a fresh memory block; reopen loads
|
||||
the new epoch and its carry; a second commit leaves epoch 0 with one `EpochEnd` and writes the
|
||||
next on epoch 1; resume after `EpochEnd` without next files uses that summary as carry and does
|
||||
not write a second end; the new `SessionStart` keeps the live slot (including a non-default 7).
|
||||
@@ -0,0 +1,261 @@
|
||||
# M5d-1 task 08: compact after a turn, and retry once on `SessionFull`
|
||||
|
||||
**Branch:** `m5d1`. Check it: `git branch --show-current` prints `m5d1` and `git status --short`
|
||||
is empty; otherwise stop. Never switch branch.
|
||||
**Commit subject:** `loopd: compact after a full turn and retry SessionFull once`
|
||||
**Log row:** the Task cell is `M5d1/08-loopd-compact-turn`.
|
||||
|
||||
## Goal
|
||||
|
||||
After `run_turn` returns, `finish_turn` decides whether to compact (spec 3.3):
|
||||
|
||||
- A successful turn whose last `Usage` is `due` is compacted before the caller returns. If
|
||||
compact fails, the turn stays successful and the session stays on its epoch; `loopd` logs the
|
||||
compact error with the runbook pointer.
|
||||
- `TurnError::SessionFull` is compacted, then that user message is `run_turn`'d once in the new
|
||||
epoch. If compact fails, the error stays `SessionFull`. The retry is not compacted again.
|
||||
- Any other `TurnError` is returned as it is. No compact.
|
||||
|
||||
`Runtime` does not gain `MemorySource` (given `turn` tests build a `Runtime` without one).
|
||||
`finish_turn` takes `memory` as its own argument.
|
||||
|
||||
The channel holds the session busy for this whole call: `handle` runs `finish_turn` **before**
|
||||
`drop(held)`. A compact is not an idle timer and does not start after the busy set releases.
|
||||
|
||||
Live turns still send `slots.main` (task 03). Only the summary call uses the background slot.
|
||||
|
||||
## Every place that changes
|
||||
|
||||
| Place | What happens there |
|
||||
|---|---|
|
||||
| `crates/loopd/src/compact.rs` | `finish_turn` and its two helpers (edit F1) |
|
||||
| `crates/loopd/src/channel.rs` | `handle` calls `finish_turn` before `drop(held)` (edits H1, H2) |
|
||||
| `crates/loopd/tests/compact_turn.rs` | new, given |
|
||||
| everything else | **no change**: not `turn.rs`, not `Runtime`, not existing `turn` / `channel` tests |
|
||||
|
||||
## Files
|
||||
|
||||
- Copy, from `docs/plans/M5d1/files/` (the same path under it): the new test
|
||||
`crates/loopd/tests/compact_turn.rs`.
|
||||
- Modify: `crates/loopd/src/compact.rs`, `crates/loopd/src/channel.rs`,
|
||||
`docs/implementer-log.md`.
|
||||
- Do not touch: any other file.
|
||||
|
||||
## Facts you need
|
||||
|
||||
- **`due` looks at `session.last_usage()`**, which is the current epoch's last `Usage`. After a
|
||||
successful `run_turn` that record is there. After `SessionFull`, `run_turn` has already
|
||||
appended the `User` and returned before any `Usage`; compact runs because of the error, not
|
||||
because of `due`.
|
||||
- **Compact helper.** `summarise` then `commit`. Log failures with
|
||||
|
||||
```text
|
||||
loopd: could not compact session <id>: <CompactError Display>
|
||||
```
|
||||
|
||||
`Display` already includes the runbook pointer, matching the runbook's example line.
|
||||
- **Retry.** `run_turn` appends the user message again in the new epoch. The old epoch keeps its
|
||||
`User` (Honcho skips it: no reply). `turns()` is 2. That is accepted (README, known limits).
|
||||
- **Existing channel tests stay green.** Their `n_ctx` is 131072 and fixture `prompt_n` is tens,
|
||||
so `due` is false. `SessionFull` without `compact.md` in `Home` stays `SessionFull` (the
|
||||
successful-turn-when-compact-fails behaviour, on the error path).
|
||||
- **`compact.rs` stays ≤ 500 lines.** After this task it is about 250.
|
||||
- **`channel.rs` stays ≤ 500 lines.** After this task it is about 290.
|
||||
- The items the edits use. This table is complete.
|
||||
|
||||
| Item | What it does |
|
||||
|---|---|
|
||||
| `due` | task 05 |
|
||||
| `summarise` / `commit` | tasks 06, 07 |
|
||||
| `run_turn` | retry the same `text` and `speaker` |
|
||||
| `Session::last_usage` | current epoch |
|
||||
| `Config.compact.threshold`, `expect.n_ctx` | the `due` arguments |
|
||||
| `MemorySource` | `finish_turn`'s extra argument; channel passes `&*ctx.memory` |
|
||||
| `Held` | already in `channel.rs`; drop it after `finish_turn` |
|
||||
|
||||
## The edits, exactly
|
||||
|
||||
Each quoted "before" block occurs exactly once after task 07 (`channel.rs` is still as on
|
||||
master, except rustfmt).
|
||||
|
||||
### `crates/loopd/src/compact.rs`
|
||||
|
||||
**F1.** Extend the crate uses. After task 07 they include `MemorySource`, `Session`, `Config`,
|
||||
`Registry`. Add:
|
||||
|
||||
```rust
|
||||
use crate::turn::{Runtime, TurnError, TurnOutcome, run_turn};
|
||||
use bxcore::TurnEvent;
|
||||
```
|
||||
|
||||
Keep rustfmt's order of `use` lines; if it moves them, accept that.
|
||||
|
||||
Then after `commit`, add these three functions whole. There is no `todo!()`.
|
||||
|
||||
```rust
|
||||
/// After `run_turn`: compact when the last `Usage` is due, or on `SessionFull` (then retry the
|
||||
/// user message once). Compact failure on a successful turn keeps the success; on `SessionFull`
|
||||
/// keeps `SessionFull`. Other errors skip compact.
|
||||
pub fn finish_turn(
|
||||
session: &mut Session,
|
||||
rt: &Runtime<'_>,
|
||||
memory: &dyn MemorySource,
|
||||
text: &str,
|
||||
speaker: Option<&str>,
|
||||
on_event: &mut dyn FnMut(&TurnEvent),
|
||||
outcome: Result<TurnOutcome, TurnError>,
|
||||
) -> Result<TurnOutcome, TurnError> {
|
||||
match outcome {
|
||||
Err(TurnError::SessionFull) => match compact_session(session, rt, memory) {
|
||||
Ok(()) => run_turn(session, rt, text, speaker, on_event),
|
||||
Err(e) => {
|
||||
log_compact(session, &e);
|
||||
Err(TurnError::SessionFull)
|
||||
}
|
||||
},
|
||||
Ok(done) => {
|
||||
if let Some(usage) = session.last_usage()
|
||||
&& due(&usage, rt.cfg.compact.threshold, rt.cfg.expect.n_ctx)
|
||||
&& let Err(e) = compact_session(session, rt, memory)
|
||||
{
|
||||
log_compact(session, &e);
|
||||
}
|
||||
Ok(done)
|
||||
}
|
||||
Err(e) => Err(e),
|
||||
}
|
||||
}
|
||||
|
||||
fn compact_session(
|
||||
session: &mut Session,
|
||||
rt: &Runtime<'_>,
|
||||
memory: &dyn MemorySource,
|
||||
) -> Result<(), CompactError> {
|
||||
let summary = summarise(rt.cfg, rt.client, session)?;
|
||||
commit(session, rt.cfg, rt.registry, memory, summary)
|
||||
}
|
||||
|
||||
fn log_compact(session: &Session, e: &CompactError) {
|
||||
eprintln!(
|
||||
"loopd: could not compact session {}: {e}",
|
||||
session.id().as_str()
|
||||
);
|
||||
}
|
||||
```
|
||||
|
||||
The `Ok(done)` arm uses `&& let Err(e) = ...` so a compact failure is logged and the original
|
||||
`done` is still returned. Do not `?` that error.
|
||||
|
||||
### `crates/loopd/src/channel.rs`
|
||||
|
||||
**H1.** The `run_turn` call and the `drop(held)` that follows it. The lines
|
||||
|
||||
```rust
|
||||
let outcome = run_turn(
|
||||
&mut session,
|
||||
&runtime,
|
||||
&turn.content,
|
||||
turn.speaker.as_deref(),
|
||||
&mut |event| {
|
||||
let _ = write_frame(
|
||||
&mut stream,
|
||||
&Envelope {
|
||||
v: PROTOCOL_VERSION,
|
||||
id: request.id,
|
||||
r#final: false,
|
||||
msg: Message::TurnEvent(event.clone()),
|
||||
},
|
||||
);
|
||||
},
|
||||
);
|
||||
|
||||
// 5. Release the session before the final frame, so the next turn does not find it busy.
|
||||
drop(held);
|
||||
```
|
||||
|
||||
become
|
||||
|
||||
```rust
|
||||
let mut emit = |event: &bxcore::TurnEvent| {
|
||||
let _ = write_frame(
|
||||
&mut stream,
|
||||
&Envelope {
|
||||
v: PROTOCOL_VERSION,
|
||||
id: request.id,
|
||||
r#final: false,
|
||||
msg: Message::TurnEvent(event.clone()),
|
||||
},
|
||||
);
|
||||
};
|
||||
|
||||
let outcome = run_turn(
|
||||
&mut session,
|
||||
&runtime,
|
||||
&turn.content,
|
||||
turn.speaker.as_deref(),
|
||||
&mut emit,
|
||||
);
|
||||
let outcome = crate::compact::finish_turn(
|
||||
&mut session,
|
||||
&runtime,
|
||||
&*ctx.memory,
|
||||
&turn.content,
|
||||
turn.speaker.as_deref(),
|
||||
&mut emit,
|
||||
outcome,
|
||||
);
|
||||
|
||||
// 5. Release the session before the final frame, so the next turn does not find it busy.
|
||||
// Compaction ran inside the hold: a second turn on this session waits until the new
|
||||
// epoch is on disk (or compact failed and the epoch is unchanged).
|
||||
drop(held);
|
||||
```
|
||||
|
||||
**H2.** Do not add `MemorySource` to `Runtime`. Do not change `turn_code`. The `memory.updated`
|
||||
block after `drop(held)` stays: a successful compact-and-retry still reports `session.turns()`,
|
||||
which now spans epochs.
|
||||
|
||||
## Steps
|
||||
|
||||
- [ ] **1. Copy the given test.**
|
||||
|
||||
```sh
|
||||
cp docs/plans/M5d1/files/crates/loopd/tests/compact_turn.rs crates/loopd/tests/
|
||||
```
|
||||
|
||||
- [ ] **2. See it fail.** `cargo build --tests --keep-going -p loopd`, timeout 600000 ms: test
|
||||
target `compact_turn` does not compile (`E0599` no function `finish_turn`). Every other
|
||||
target compiles, including `turn` and `channel`.
|
||||
- [ ] **3. The edits.** F1, H1. `cargo check -p loopd` prints no error and no warning.
|
||||
`wc -l crates/loopd/src/compact.rs` and `wc -l crates/loopd/src/channel.rs` each print a
|
||||
number ≤ 500.
|
||||
- [ ] **4. The tests.** `cargo test -p loopd --test compact_turn --test turn --test channel
|
||||
--test compact_advance`: every test passes, 6 in `compact_turn` and the others as before.
|
||||
- [ ] **5. Check.** `git diff --stat` lists exactly `crates/loopd/src/compact.rs` and
|
||||
`crates/loopd/src/channel.rs`. `git status --short` shows `compact_turn.rs` as new.
|
||||
- [ ] **6. Run the gate.** `cargo fmt --all`, then `make gate`, timeout 600000 ms. Last line
|
||||
`gate: ok`. Test count **1507** (1501 before, plus 6).
|
||||
- [ ] **7. Log and commit.** Task cell `M5d1/08-loopd-compact-turn`. Stage with
|
||||
`git add crates/loopd docs/implementer-log.md Cargo.lock`, then commit as `AGENTS.md` says.
|
||||
|
||||
## Done when
|
||||
|
||||
- `make gate` prints `gate: ok` with 1507 tests.
|
||||
- `git diff HEAD~1 --stat` lists only `crates/loopd/src/compact.rs`,
|
||||
`crates/loopd/src/channel.rs`, `crates/loopd/tests/compact_turn.rs` and the log.
|
||||
|
||||
## Stop and report if
|
||||
|
||||
- The branch check fails; a "before" block is not found exactly once; step 2 differs; `cargo
|
||||
check` warns; a test fails; the gate count is not 1507; a file exceeds 500 lines; you add
|
||||
`MemorySource` to `Runtime`; you compact inside `run_turn`; you call `finish_turn` after
|
||||
`drop(held)`; you add an idle timer; you compact a `TurnError` other than `SessionFull`; you
|
||||
retry `SessionFull` more than once.
|
||||
|
||||
## What the given tests pin
|
||||
|
||||
Usage 849/1000 leaves epoch 0 (one chat); 850/1000 compact before return (live slot then
|
||||
background, no tools on the summary); `SessionFull` compact-and-retry (three chats, user logged
|
||||
in both epochs, `turns() == 2`); `SessionFull` stays `SessionFull` when compact fails; a
|
||||
successful due turn stays successful when `compact.md` is missing (epoch stays 0); another
|
||||
infer error is not compacted.
|
||||
@@ -0,0 +1,119 @@
|
||||
# M5d-1 implementation plan: epochs and compaction
|
||||
|
||||
> For the implementer: each task file is one closed task for one fresh session. Read `AGENTS.md`,
|
||||
> then the task file, and do exactly that task. The driver is `tools/run-plan.sh`.
|
||||
|
||||
**Goal:** a live session can start a new epoch. `loopd` writes `N.jsonl` / `N.baseline.json` for
|
||||
`N > 0`, turn numbers and `untrusted()` span every epoch, compaction runs after a turn whose last
|
||||
`Usage` is at or above a configured fraction of `expect.n_ctx`, the summary call uses the
|
||||
background slot, and `memoryd`'s Honcho walker reads every epoch file of a session.
|
||||
|
||||
**Architecture:** `loopd` gains a `[compact]` table and a `compact` module (when to compact, the
|
||||
summary call, writing the next epoch). `Session::open` loads the highest complete epoch and the
|
||||
carried summary. `messages` for epoch `n > 0` inserts that carry as a `ChatMessage::User` that is
|
||||
never a `LogRecord::User`. Honcho concatenates epoch logs in order so `Item::Turn` numbers stay
|
||||
monotonic. Heartbeat, cron, `ToolSummary`, and tiering are M5d-2.
|
||||
|
||||
**Tech stack:** Rust 1.98.1, edition 2024; no new crate.
|
||||
|
||||
**Spec:** `docs/specs/2026-10-03-m5d-scheduler.md`, sections 3, 5 (`[compact]` without using
|
||||
`summarise`), 7 (the M5d-1 tests), and 8 (`session-compact-failed`).
|
||||
|
||||
## The owner's decisions (2026-10-03)
|
||||
|
||||
1. Compaction runs after a turn, when that turn's `cache_n + prompt_n` is at or above a configured
|
||||
fraction of `expect.n_ctx` (default 0.85). A quiet session is not a reason to compact.
|
||||
2. If a live request is `exceed_context_size_error`, compact and retry that user message once in
|
||||
the new epoch.
|
||||
3. The summary call is on the background slot, with no tools, thinking on. A context-full summary
|
||||
falls back to the epoch's `User` texts, newest last, cut to `summary_chars`.
|
||||
4. The carried summary is not a `LogRecord::User`, so it is not a Honcho turn.
|
||||
|
||||
## What the code already has, and what it lacks
|
||||
|
||||
- `LogRecord::EpochEnd { time, next, summary }` is in `bxcore` and unused. `messages` already
|
||||
ignores it. This plan writes it.
|
||||
- `Session::open` reads only `0.baseline.json` and `0.jsonl`. `turns()` and `untrusted()` look at
|
||||
that file only. `next_call` is recovered from that file only.
|
||||
- `run_turn` always sends `slots.main`. Live turns stay on the session's live slot (main, from
|
||||
`SessionStart`). Only the summary call uses `slots.background`. Changing live turns to the
|
||||
session's slot is M5d-2.
|
||||
- `memoryd`'s indexer already walks every `<epoch>.jsonl`. Honcho's `log_path` is still
|
||||
`sessions/<id>/0.jsonl`.
|
||||
- `Runtime` has no `MemorySource`. Compaction asks `memoryd` for a fresh block from `channel.rs`
|
||||
(and `compact::finish_turn`), so given `turn` tests that build a `Runtime` do not change.
|
||||
- `[compact] summarise` is stored from this plan so a spec-complete `loopd.toml` loads; nothing
|
||||
in M5d-1 reads the list. M5d-2 opts in `http_fetch`.
|
||||
|
||||
## Global constraints
|
||||
|
||||
- Roles depend on `bxcore` only. No `unsafe`, no async runtime, no new crate.
|
||||
- No file the implementer reads holds a chat-template tag whole (T33). The carry heading is built
|
||||
from pieces and named in words.
|
||||
- Every path a given test creates is removed by a guard (T44); every process a test starts has a
|
||||
deadline.
|
||||
- The fail-closed state `session-compact-failed` has a runbook entry, added by the task that adds
|
||||
its pointer (T49).
|
||||
- Each task gives its baseline and its gate count with what the count includes (T50, T53); the
|
||||
branch line is a check, never `git switch` (T51).
|
||||
- `Cargo.lock` is in the `git add` line (T24). Gate timeout 600000 ms (T40).
|
||||
- Commit trailer from the driver.
|
||||
|
||||
## Tasks
|
||||
|
||||
Every task below has been written to be replayed from its text. The gate count is the passed
|
||||
total of `cargo test --workspace` plus one, for the ignored `end_to_end` test the gate runs on
|
||||
its own. Before task 01 it is **1456** (talos, rustc 1.98.1, clippy of the same release; master
|
||||
`069cd16`, docs-only commits after the M5c-2 review).
|
||||
|
||||
| # | Delivers | Crates | Gate after | Tests added |
|
||||
|---|---|---|---|---|
|
||||
| 01 | `[compact]` on `loopd.toml`: `threshold` 0.85, `summary_chars` 4000, `summarise` default `["http_fetch"]` (stored, unused); `Config::compact_prompt` is `compact.md` beside `system.md` | loopd | 1464 | 8 |
|
||||
| 02 | `Session::open` loads the highest epoch that has both a baseline and a log, the carry from the previous `EpochEnd`, `turns` / `untrusted` / `next_call` over every epoch; persist writes the current epoch | loopd | 1473 | 9 |
|
||||
| 03 | `messages_with_carry`: epoch `n > 0` is the new system message, then one `ChatMessage::User` (fixed heading plus carry, not a log record), then the new epoch's records; `run_turn` uses `session.messages()` | loopd | 1478 | 5 |
|
||||
| 04 | Honcho concatenates every `<epoch>.jsonl` in epoch order; `turn_records` of turn 3 is the epoch-1 user; `send_turn` reads the concatenated log | memoryd | 1483 | 5 |
|
||||
| 05 | `loopd::compact`: `due`, `fallback_summary`; starter `config/compact.md` | loopd | 1489 | 6 |
|
||||
| 06 | Summary call on the background slot, no tools, thinking on; tag-neutralised and capped; context-full uses the fallback; `session-compact-failed` pointer and runbook entry | loopd | 1495 | 6 |
|
||||
| 07 | Writing the next epoch: `EpochEnd`, fresh baseline from `memoryd`, `SessionStart` on the live slot; resume if `EpochEnd` is already the last record | loopd | 1501 | 6 |
|
||||
| 08 | After a turn: compact when `due`; on `SessionFull`, compact and retry the user message once; compaction holds the session busy | loopd | 1507 | 6 |
|
||||
|
||||
After task 08, the design model runs the M5d-1 device check (spec 7) on straylight: a `bxctl`
|
||||
session writes a distinctive fact into `memory.md`, continues until compaction with
|
||||
`[compact] threshold` lowered, and the next turn's baseline contains the fact; `recall` of a
|
||||
phrase from epoch 0 still hits; `TurnDone.untrusted` stays true across the epoch change after a
|
||||
tainted recall.
|
||||
|
||||
## Known limits
|
||||
|
||||
These are known, accepted for M5d-1, and not bugs to fix in a task:
|
||||
|
||||
- **Live turns still use `slots.main`.** `SessionStart.slot` is recorded and copied onto the new
|
||||
epoch; `run_turn` does not read it yet. Isolated sessions on the background slot are M5d-2.
|
||||
- **`[compact] summarise` is unused.** Unknown tool names are not a config error until M5d-2.
|
||||
- **No idle path exists.** Compaction is only called after a turn or on `SessionFull`. There is
|
||||
no timer to test.
|
||||
- **A crash after `EpochEnd` and before the next files exist** reopens epoch 0 with `EpochEnd` at
|
||||
the end of the log. The next compact resumes without writing a second `EpochEnd` (task 07).
|
||||
- **A `SessionFull` retry logs the user message twice**, once in the old epoch (Honcho skips it:
|
||||
no reply) and once in the new epoch. Turn numbers stay monotonic.
|
||||
|
||||
## Before hand-over
|
||||
|
||||
Work through `docs/plans/CHECKLIST.md` with a reference in a scratch worktree with its own build
|
||||
directory, replayed from its task text; mutations; `/tmp` compared around a gate run; scratch
|
||||
files in the worktree's `.state/`; the reference moved to straylight as a bundle before the run
|
||||
(T18). Task files name their log row (T47).
|
||||
|
||||
## Running it
|
||||
|
||||
The reference (`m5d1-ref`, tasks 01 to 08) is moved to straylight as a bundle and deleted here
|
||||
before the run (T18). Then, from the repository root, with the OpenCode TUI closed:
|
||||
|
||||
```sh
|
||||
git switch -c m5d1 master
|
||||
OPENCODE_MODEL=straylight/ornith-1.5-35b-a3b RESUME=1 \
|
||||
SLOTS_URL='http://straylight:11434/slots?model=ornith-1.5-35b-a3b' \
|
||||
tools/run-plan.sh docs/plans/M5d1 01
|
||||
```
|
||||
|
||||
After task 08, the gate count is 1507. Then the device check above.
|
||||
@@ -0,0 +1,8 @@
|
||||
Summarise this conversation for the next epoch of the same session.
|
||||
|
||||
Write a short account of:
|
||||
- what the owner asked for and decided
|
||||
- facts that were established
|
||||
- open threads and anything still waiting
|
||||
|
||||
Do not use tools. Do not address the owner. Do not invent facts that are not in the messages.
|
||||
@@ -0,0 +1,120 @@
|
||||
//! Tests for the carried summary in the next epoch's prefix. Do not edit.
|
||||
|
||||
use bxcore::{Epoch, Hash32, LogRecord, SessionId, Timestamp};
|
||||
use loopd::baseline::{Baseline, CARRY_HEADING, messages, messages_with_carry};
|
||||
use loopd::llama::ChatMessage;
|
||||
|
||||
fn ts() -> Timestamp {
|
||||
Timestamp::parse("2026-09-18T08:00:00.000Z").unwrap()
|
||||
}
|
||||
|
||||
fn user(content: &str) -> LogRecord {
|
||||
LogRecord::User {
|
||||
time: ts(),
|
||||
content: content.to_string(),
|
||||
speaker: None,
|
||||
}
|
||||
}
|
||||
|
||||
fn start(epoch: u32) -> LogRecord {
|
||||
LogRecord::SessionStart {
|
||||
time: ts(),
|
||||
session: SessionId::new("a").unwrap(),
|
||||
epoch: Epoch(epoch),
|
||||
slot: 0,
|
||||
baseline: Hash32::ZERO,
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn epoch_0_matches_messages_without_a_carry() {
|
||||
let b = Baseline {
|
||||
system: "sys".to_string(),
|
||||
tools: vec![],
|
||||
};
|
||||
let records = vec![start(0), user("hi")];
|
||||
assert_eq!(
|
||||
messages_with_carry(&b, &records, None),
|
||||
messages(&b, &records)
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_carry_is_a_user_message_after_the_system_prompt() {
|
||||
let b = Baseline {
|
||||
system: "new-sys".to_string(),
|
||||
tools: vec![],
|
||||
};
|
||||
let records = vec![start(1), user("next")];
|
||||
let got = messages_with_carry(&b, &records, Some("the summary"));
|
||||
assert_eq!(
|
||||
got,
|
||||
vec![
|
||||
ChatMessage::System {
|
||||
content: "new-sys".to_string(),
|
||||
},
|
||||
ChatMessage::User {
|
||||
content: format!("{CARRY_HEADING}\n\nthe summary"),
|
||||
},
|
||||
ChatMessage::User {
|
||||
content: "next".to_string(),
|
||||
},
|
||||
]
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_carry_is_not_taken_from_a_log_user_record() {
|
||||
let b = Baseline {
|
||||
system: "sys".to_string(),
|
||||
tools: vec![],
|
||||
};
|
||||
let records = vec![start(1)];
|
||||
let got = messages_with_carry(&b, &records, Some("carried"));
|
||||
assert_eq!(got.len(), 2);
|
||||
assert!(
|
||||
!records.iter().any(|r| matches!(r, LogRecord::User { .. })),
|
||||
"the carry is not in the log"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_heading_is_built_from_pieces() {
|
||||
assert_eq!(CARRY_HEADING, concat!("Earlier ", "turns, summarised:"));
|
||||
assert!(
|
||||
!CARRY_HEADING.contains('<'),
|
||||
"the heading is words, not a tag"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn epoch_1_is_not_an_extension_of_epoch_0() {
|
||||
let old = Baseline {
|
||||
system: "old-sys".to_string(),
|
||||
tools: vec![],
|
||||
};
|
||||
let new = Baseline {
|
||||
system: "new-sys".to_string(),
|
||||
tools: vec![],
|
||||
};
|
||||
let epoch0 = vec![start(0), user("first")];
|
||||
let epoch1 = vec![start(1), user("second")];
|
||||
let prev = messages(&old, &epoch0);
|
||||
let next = messages_with_carry(&new, &epoch1, Some("sum of first"));
|
||||
assert!(
|
||||
next.len() >= 2,
|
||||
"system plus carry, then the new epoch's user"
|
||||
);
|
||||
assert_ne!(next[0], prev[0], "the system message changed");
|
||||
assert!(
|
||||
!next.starts_with(&prev[..]),
|
||||
"the new prefix is not an extension of the old messages"
|
||||
);
|
||||
match &next[1] {
|
||||
ChatMessage::User { content } => {
|
||||
assert!(content.contains("sum of first"), "{content}");
|
||||
assert!(!content.contains("second"), "{content}");
|
||||
}
|
||||
other => panic!("{other:?}"),
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,201 @@
|
||||
//! Tests for writing the next epoch. Do not edit.
|
||||
|
||||
mod support;
|
||||
|
||||
use bxcore::{Epoch, LogRecord, SessionId};
|
||||
use loopd::baseline::Baseline;
|
||||
use loopd::compact;
|
||||
use loopd::memory::{MemorySource, NoMemory};
|
||||
use loopd::session::Session;
|
||||
use loopd::tools::Registry;
|
||||
use support::Home;
|
||||
|
||||
struct Block(&'static str);
|
||||
|
||||
impl MemorySource for Block {
|
||||
fn block(&self) -> Result<String, String> {
|
||||
Ok(self.0.to_string())
|
||||
}
|
||||
fn updated(&self, _session: &SessionId, _turn: u64) -> Result<(), String> {
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
fn id(s: &str) -> SessionId {
|
||||
SessionId::new(s).unwrap()
|
||||
}
|
||||
|
||||
fn session(home: &Home) -> (loopd::config::Config, Session) {
|
||||
let cfg = home.config(std::path::Path::new("/tmp/unused.sock"));
|
||||
let baseline = Baseline::assemble(&cfg, &Registry::m2b()).unwrap();
|
||||
let s = Session::create(&home.dir, id("a"), baseline, cfg.slots.main).unwrap();
|
||||
(cfg, s)
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn commit_writes_epoch_end_a_new_baseline_and_a_new_log() {
|
||||
let home = Home::new();
|
||||
let (cfg, mut s) = session(&home);
|
||||
s.append(LogRecord::User {
|
||||
time: bxcore::Timestamp::parse("2026-09-18T08:00:00.000Z").unwrap(),
|
||||
content: "hi".to_string(),
|
||||
speaker: None,
|
||||
})
|
||||
.unwrap();
|
||||
compact::commit(
|
||||
&mut s,
|
||||
&cfg,
|
||||
&Registry::m2b(),
|
||||
&NoMemory,
|
||||
"the sum".to_string(),
|
||||
)
|
||||
.unwrap();
|
||||
|
||||
assert_eq!(s.epoch(), Epoch(1));
|
||||
assert_eq!(s.carry(), Some("the sum"));
|
||||
assert_eq!(s.slot(), cfg.slots.main);
|
||||
assert_eq!(s.turns(), 1, "the old user still counts");
|
||||
assert_eq!(
|
||||
s.records()
|
||||
.iter()
|
||||
.filter(|r| matches!(r, LogRecord::User { .. }))
|
||||
.count(),
|
||||
0
|
||||
);
|
||||
|
||||
let old = home.records("a");
|
||||
assert!(
|
||||
matches!(old.last(), Some(LogRecord::EpochEnd { next, summary, .. }) if *next == Epoch(1) && summary == "the sum")
|
||||
);
|
||||
assert!(home.dir.join("sessions/a/1.baseline.json").is_file());
|
||||
let new_log = home.read("sessions/a/1.jsonl");
|
||||
assert!(new_log.contains("\"epoch\":1"), "{new_log}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_new_baseline_takes_a_fresh_memory_block() {
|
||||
let home = Home::new();
|
||||
let (cfg, mut s) = session(&home);
|
||||
compact::commit(
|
||||
&mut s,
|
||||
&cfg,
|
||||
&Registry::m2b(),
|
||||
&Block("I take my tea strong."),
|
||||
"sum".to_string(),
|
||||
)
|
||||
.unwrap();
|
||||
assert!(
|
||||
s.baseline().system.contains("I take my tea strong."),
|
||||
"{}",
|
||||
s.baseline().system
|
||||
);
|
||||
let on_disk = home.read("sessions/a/1.baseline.json");
|
||||
assert!(on_disk.contains("I take my tea strong."), "{on_disk}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn reopen_loads_the_new_epoch() {
|
||||
let home = Home::new();
|
||||
let (cfg, mut s) = session(&home);
|
||||
compact::commit(
|
||||
&mut s,
|
||||
&cfg,
|
||||
&Registry::m2b(),
|
||||
&NoMemory,
|
||||
"carry".to_string(),
|
||||
)
|
||||
.unwrap();
|
||||
drop(s);
|
||||
let s = Session::open(&home.dir, id("a")).unwrap();
|
||||
assert_eq!(s.epoch(), Epoch(1));
|
||||
assert_eq!(s.carry(), Some("carry"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_second_commit_does_not_write_another_epoch_end_on_the_old_log() {
|
||||
let home = Home::new();
|
||||
let (cfg, mut s) = session(&home);
|
||||
compact::commit(
|
||||
&mut s,
|
||||
&cfg,
|
||||
&Registry::m2b(),
|
||||
&NoMemory,
|
||||
"first".to_string(),
|
||||
)
|
||||
.unwrap();
|
||||
let ends = home
|
||||
.records("a")
|
||||
.iter()
|
||||
.filter(|r| matches!(r, LogRecord::EpochEnd { .. }))
|
||||
.count();
|
||||
assert_eq!(ends, 1);
|
||||
compact::commit(
|
||||
&mut s,
|
||||
&cfg,
|
||||
&Registry::m2b(),
|
||||
&NoMemory,
|
||||
"second".to_string(),
|
||||
)
|
||||
.unwrap();
|
||||
assert_eq!(s.epoch(), Epoch(2));
|
||||
let ends = home
|
||||
.records("a")
|
||||
.iter()
|
||||
.filter(|r| matches!(r, LogRecord::EpochEnd { .. }))
|
||||
.count();
|
||||
assert_eq!(ends, 1, "epoch 0 still has one EpochEnd");
|
||||
let mid = home.read("sessions/a/1.jsonl");
|
||||
assert!(mid.contains("\"type\":\"epoch_end\""), "{mid}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn resume_after_epoch_end_without_next_files_does_not_write_a_second_end() {
|
||||
let home = Home::new();
|
||||
let (cfg, mut s) = session(&home);
|
||||
s.append(LogRecord::EpochEnd {
|
||||
time: bxcore::Timestamp::now(),
|
||||
next: Epoch(1),
|
||||
summary: "already".to_string(),
|
||||
})
|
||||
.unwrap();
|
||||
drop(s);
|
||||
let mut s = Session::open(&home.dir, id("a")).unwrap();
|
||||
assert_eq!(s.epoch(), Epoch(0));
|
||||
compact::commit(
|
||||
&mut s,
|
||||
&cfg,
|
||||
&Registry::m2b(),
|
||||
&NoMemory,
|
||||
"ignored".to_string(),
|
||||
)
|
||||
.unwrap();
|
||||
let ends = home
|
||||
.records("a")
|
||||
.iter()
|
||||
.filter(|r| matches!(r, LogRecord::EpochEnd { .. }))
|
||||
.count();
|
||||
assert_eq!(ends, 1);
|
||||
assert_eq!(s.epoch(), Epoch(1));
|
||||
assert_eq!(
|
||||
s.carry(),
|
||||
Some("already"),
|
||||
"the summary already in EpochEnd is the carry"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_new_session_start_keeps_the_live_slot() {
|
||||
let home = Home::new();
|
||||
let cfg = home.config(std::path::Path::new("/tmp/unused.sock"));
|
||||
let baseline = Baseline::assemble(&cfg, &Registry::m2b()).unwrap();
|
||||
let mut s = Session::create(&home.dir, id("a"), baseline, 7).unwrap();
|
||||
compact::commit(&mut s, &cfg, &Registry::m2b(), &NoMemory, "s".to_string()).unwrap();
|
||||
assert_eq!(s.slot(), 7);
|
||||
match &s.records()[0] {
|
||||
LogRecord::SessionStart { slot, epoch, .. } => {
|
||||
assert_eq!(*slot, 7);
|
||||
assert_eq!(*epoch, Epoch(1));
|
||||
}
|
||||
other => panic!("{other:?}"),
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,79 @@
|
||||
//! Tests for when compaction is due and for the fallback summary. Do not edit.
|
||||
|
||||
use bxcore::{Epoch, Hash32, LogRecord, SessionId, Timestamp, Usage};
|
||||
use loopd::compact::{due, fallback_summary};
|
||||
|
||||
fn usage(cache_n: u64, prompt_n: u64) -> Usage {
|
||||
Usage {
|
||||
cache_n,
|
||||
prompt_n,
|
||||
predicted_n: 0,
|
||||
reasoning_tokens: 0,
|
||||
thinking_capped: false,
|
||||
}
|
||||
}
|
||||
|
||||
fn user(content: &str) -> LogRecord {
|
||||
LogRecord::User {
|
||||
time: Timestamp::parse("2026-09-18T08:00:00.000Z").unwrap(),
|
||||
content: content.to_string(),
|
||||
speaker: None,
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn due_at_the_threshold_and_not_below() {
|
||||
assert!(!due(&usage(400, 449), 0.85, 1_000), "849 < 850");
|
||||
assert!(due(&usage(400, 450), 0.85, 1_000), "850 == 0.85 * 1000");
|
||||
assert!(due(&usage(0, 851), 0.85, 1_000));
|
||||
assert!(!due(&usage(0, 0), 0.85, 1_000));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn due_uses_this_agents_n_ctx() {
|
||||
// 0.85 * 262144 = 222822.4, so 222823 is the first integer at or above.
|
||||
assert!(!due(&usage(0, 222_822), 0.85, 262_144));
|
||||
assert!(due(&usage(0, 222_823), 0.85, 262_144));
|
||||
assert!(due(&usage(200_000, 22_823), 0.85, 262_144));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_threshold_of_one_is_the_full_window() {
|
||||
assert!(!due(&usage(99, 0), 1.0, 100));
|
||||
assert!(due(&usage(100, 0), 1.0, 100));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn fallback_is_user_texts_newest_last() {
|
||||
let records = vec![
|
||||
LogRecord::SessionStart {
|
||||
time: Timestamp::parse("2026-09-18T08:00:00.000Z").unwrap(),
|
||||
session: SessionId::new("a").unwrap(),
|
||||
epoch: Epoch(0),
|
||||
slot: 0,
|
||||
baseline: Hash32::ZERO,
|
||||
},
|
||||
user("first"),
|
||||
user("second"),
|
||||
user("third"),
|
||||
];
|
||||
assert_eq!(
|
||||
fallback_summary(&records, 4_000),
|
||||
"third\n\nsecond\n\nfirst"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn fallback_is_cut_to_summary_chars_on_a_character_boundary() {
|
||||
let records = vec![user("café"), user("ab")];
|
||||
assert_eq!(fallback_summary(&records, 3), "ab\n");
|
||||
assert_eq!(fallback_summary(&records, 1), "a");
|
||||
assert_eq!(fallback_summary(&records, 0), "");
|
||||
let long = user("ååå");
|
||||
assert_eq!(fallback_summary(&[long], 2), "åå");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn fallback_with_no_users_is_empty() {
|
||||
assert_eq!(fallback_summary(&[], 100), "");
|
||||
}
|
||||
@@ -0,0 +1,147 @@
|
||||
//! Tests for the compaction summary call. Do not edit.
|
||||
|
||||
mod support;
|
||||
|
||||
use bxcore::SessionId;
|
||||
use loopd::baseline::Baseline;
|
||||
use loopd::compact::{self, CompactError};
|
||||
use loopd::llama::Client;
|
||||
use loopd::session::Session;
|
||||
use loopd::tools::Registry;
|
||||
use support::{FakeServer, Home, Reply};
|
||||
|
||||
const CHAT: &str = "/v1/chat/completions";
|
||||
const PROMPT: &str = "Summarise this conversation for the next epoch.\n";
|
||||
|
||||
fn sse(content: &str, cache_n: u64, prompt_n: u64) -> Reply {
|
||||
let content_json = serde_json::to_string(content).unwrap();
|
||||
let data1 = format!(
|
||||
r#"data: {{"choices":[{{"finish_reason":null,"index":0,"delta":{{"content":{content_json}}}}}],"id":"sum","timings":{{"cache_n":{cache_n},"prompt_n":{prompt_n},"predicted_n":1}}}}"#
|
||||
);
|
||||
let data2 = format!(
|
||||
r#"data: {{"choices":[{{"finish_reason":"stop","index":0,"delta":{{}}}}],"id":"sum","timings":{{"cache_n":{cache_n},"prompt_n":{prompt_n},"predicted_n":1}}}}"#
|
||||
);
|
||||
let body = format!("{data1}\n\n{data2}\n\ndata: [DONE]\n\n");
|
||||
Reply::raw(format!(
|
||||
"HTTP/1.1 200 OK\r\nContent-Type: text/event-stream\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}",
|
||||
body.len()
|
||||
))
|
||||
}
|
||||
|
||||
fn session(home: &Home, cfg: &loopd::config::Config) -> Session {
|
||||
let baseline = Baseline::assemble(cfg, &Registry::m2b()).unwrap();
|
||||
Session::create(
|
||||
&home.dir,
|
||||
SessionId::new("a").unwrap(),
|
||||
baseline,
|
||||
cfg.slots.main,
|
||||
)
|
||||
.unwrap()
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_summary_call_is_on_the_background_slot_with_no_tools_and_thinking() {
|
||||
let home = Home::new();
|
||||
home.write("compact.md", PROMPT);
|
||||
let server = FakeServer::start();
|
||||
let cfg = home.config(&server.socket);
|
||||
server.route(CHAT, vec![sse("the sum", 0, 10)]);
|
||||
let client = Client::new(cfg.clone());
|
||||
let s = session(&home, &cfg);
|
||||
let text = compact::summarise(&cfg, &client, &s).unwrap();
|
||||
assert_eq!(text, "the sum");
|
||||
let sent = server.requests_to(CHAT);
|
||||
assert_eq!(sent.len(), 1);
|
||||
let body = sent[0].json();
|
||||
assert_eq!(body["id_slot"], 1);
|
||||
assert!(body.get("tools").is_none(), "{body}");
|
||||
assert_eq!(body["chat_template_kwargs"]["enable_thinking"], true);
|
||||
let messages = body["messages"].as_array().unwrap();
|
||||
let last = messages.last().unwrap();
|
||||
assert_eq!(last["role"], "user");
|
||||
assert_eq!(last["content"], PROMPT.trim_end());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_summary_is_neutralised_and_capped() {
|
||||
let home = Home::new();
|
||||
home.write("compact.md", PROMPT);
|
||||
let server = FakeServer::start();
|
||||
let mut cfg = home.config(&server.socket);
|
||||
cfg.compact.summary_chars = 4;
|
||||
server.route(CHAT, vec![sse("abcdefghij", 0, 10)]);
|
||||
let client = Client::new(cfg.clone());
|
||||
let s = session(&home, &cfg);
|
||||
let text = compact::summarise(&cfg, &client, &s).unwrap();
|
||||
assert_eq!(text, "abcd");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_context_full_summary_uses_the_fallback() {
|
||||
let home = Home::new();
|
||||
home.write("compact.md", PROMPT);
|
||||
let server = FakeServer::start();
|
||||
let cfg = home.config(&server.socket);
|
||||
server.route(CHAT, vec![Reply::fixture("context_full")]);
|
||||
let client = Client::new(cfg.clone());
|
||||
let mut s = session(&home, &cfg);
|
||||
s.append(bxcore::LogRecord::User {
|
||||
time: bxcore::Timestamp::parse("2026-09-18T08:00:00.000Z").unwrap(),
|
||||
content: "remember the tea".to_string(),
|
||||
speaker: None,
|
||||
})
|
||||
.unwrap();
|
||||
let text = compact::summarise(&cfg, &client, &s).unwrap();
|
||||
assert_eq!(text, "remember the tea");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn tags_in_the_completion_are_neutralised() {
|
||||
let home = Home::new();
|
||||
home.write("compact.md", PROMPT);
|
||||
let server = FakeServer::start();
|
||||
let cfg = home.config(&server.socket);
|
||||
let tag = concat!("<", "think", ">");
|
||||
server.route(CHAT, vec![sse(&format!("hello {tag} world"), 0, 10)]);
|
||||
let client = Client::new(cfg.clone());
|
||||
let s = session(&home, &cfg);
|
||||
let text = compact::summarise(&cfg, &client, &s).unwrap();
|
||||
assert!(!text.contains(&tag), "{text}");
|
||||
assert!(text.contains("hello"), "{text}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_missing_prompt_file_names_the_runbook() {
|
||||
let home = Home::new();
|
||||
let server = FakeServer::start();
|
||||
let cfg = home.config(&server.socket);
|
||||
let client = Client::new(cfg.clone());
|
||||
let s = session(&home, &cfg);
|
||||
let err = compact::summarise(&cfg, &client, &s).unwrap_err();
|
||||
let e: Box<dyn std::error::Error> = Box::new(err);
|
||||
assert!(e.to_string().contains("compact.md"), "{e}");
|
||||
assert!(
|
||||
e.to_string()
|
||||
.ends_with("see docs/runbook.md#session-compact-failed"),
|
||||
"{e}"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn an_inference_error_that_is_not_context_full_is_a_compact_failure() {
|
||||
let home = Home::new();
|
||||
home.write("compact.md", PROMPT);
|
||||
let server = FakeServer::start();
|
||||
let cfg = home.config(&server.socket);
|
||||
server.route(CHAT, vec![Reply::fixture("bad_request")]);
|
||||
let client = Client::new(cfg.clone());
|
||||
let s = session(&home, &cfg);
|
||||
let err = compact::summarise(&cfg, &client, &s).unwrap_err();
|
||||
assert!(matches!(err, CompactError::Failed { .. }), "{err:?}");
|
||||
let e: Box<dyn std::error::Error> = Box::new(err);
|
||||
assert!(
|
||||
e.to_string()
|
||||
.ends_with("see docs/runbook.md#session-compact-failed"),
|
||||
"{e}"
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,257 @@
|
||||
//! Tests for compacting after a turn and retrying a full context. Do not edit.
|
||||
|
||||
mod support;
|
||||
|
||||
use bxcore::{Epoch, LogRecord, SessionId, TurnEvent};
|
||||
use loopd::baseline::Baseline;
|
||||
use loopd::compact;
|
||||
use loopd::llama::Client;
|
||||
use loopd::memory::NoMemory;
|
||||
use loopd::session::Session;
|
||||
use loopd::tools::Registry;
|
||||
use loopd::turn::{Runtime, TurnError, run_turn};
|
||||
use support::{FakeServer, Home, Reply};
|
||||
|
||||
const CHAT: &str = "/v1/chat/completions";
|
||||
const PROMPT: &str = "Summarise this conversation for the next epoch.\n";
|
||||
|
||||
fn sse(content: &str, cache_n: u64, prompt_n: u64) -> Reply {
|
||||
let content_json = serde_json::to_string(content).unwrap();
|
||||
let data1 = format!(
|
||||
r#"data: {{"choices":[{{"finish_reason":null,"index":0,"delta":{{"content":{content_json}}}}}],"id":"t","timings":{{"cache_n":{cache_n},"prompt_n":{prompt_n},"predicted_n":1}}}}"#
|
||||
);
|
||||
let data2 = format!(
|
||||
r#"data: {{"choices":[{{"finish_reason":"stop","index":0,"delta":{{}}}}],"id":"t","timings":{{"cache_n":{cache_n},"prompt_n":{prompt_n},"predicted_n":1}}}}"#
|
||||
);
|
||||
let body = format!("{data1}\n\n{data2}\n\ndata: [DONE]\n\n");
|
||||
Reply::raw(format!(
|
||||
"HTTP/1.1 200 OK\r\nContent-Type: text/event-stream\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}",
|
||||
body.len()
|
||||
))
|
||||
}
|
||||
|
||||
struct World {
|
||||
home: Home,
|
||||
server: FakeServer,
|
||||
cfg: loopd::config::Config,
|
||||
client: Client,
|
||||
port: support::ScriptedPort,
|
||||
registry: Registry,
|
||||
}
|
||||
|
||||
impl World {
|
||||
fn new() -> World {
|
||||
let home = Home::new();
|
||||
home.write("compact.md", PROMPT);
|
||||
let server = FakeServer::start();
|
||||
let mut cfg = home.config(&server.socket);
|
||||
cfg.expect.n_ctx = 1_000;
|
||||
cfg.compact.threshold = 0.85;
|
||||
let client = Client::new(cfg.clone());
|
||||
World {
|
||||
home,
|
||||
server,
|
||||
cfg,
|
||||
client,
|
||||
port: support::ScriptedPort::new(vec![]),
|
||||
registry: Registry::m2b(),
|
||||
}
|
||||
}
|
||||
|
||||
fn session(&self) -> Session {
|
||||
let baseline = Baseline::assemble(&self.cfg, &self.registry).unwrap();
|
||||
Session::create(
|
||||
&self.home.dir,
|
||||
SessionId::new("a").unwrap(),
|
||||
baseline,
|
||||
self.cfg.slots.main,
|
||||
)
|
||||
.unwrap()
|
||||
}
|
||||
|
||||
fn runtime(&self) -> Runtime<'_> {
|
||||
Runtime {
|
||||
cfg: &self.cfg,
|
||||
client: &self.client,
|
||||
port: &self.port,
|
||||
registry: &self.registry,
|
||||
}
|
||||
}
|
||||
|
||||
fn finish(
|
||||
&self,
|
||||
session: &mut Session,
|
||||
text: &str,
|
||||
outcome: Result<loopd::turn::TurnOutcome, TurnError>,
|
||||
) -> Result<loopd::turn::TurnOutcome, TurnError> {
|
||||
let mut events = Vec::new();
|
||||
compact::finish_turn(
|
||||
session,
|
||||
&self.runtime(),
|
||||
&NoMemory,
|
||||
text,
|
||||
None,
|
||||
&mut |e: &TurnEvent| events.push(e.clone()),
|
||||
outcome,
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn usage_under_the_threshold_leaves_the_epoch() {
|
||||
let w = World::new();
|
||||
w.server.route(CHAT, vec![sse("ok", 400, 449)]);
|
||||
let mut s = w.session();
|
||||
let mut events = Vec::new();
|
||||
let outcome = run_turn(&mut s, &w.runtime(), "hi", None, &mut |e| {
|
||||
events.push(e.clone())
|
||||
});
|
||||
let outcome = w.finish(&mut s, "hi", outcome).unwrap();
|
||||
assert_eq!(outcome.content, "ok");
|
||||
assert_eq!(s.epoch(), Epoch(0));
|
||||
assert_eq!(w.server.requests_to(CHAT).len(), 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn usage_at_the_threshold_compacts_before_the_turn_returns() {
|
||||
let w = World::new();
|
||||
w.server
|
||||
.route(CHAT, vec![sse("ok", 400, 450), sse("the sum", 0, 10)]);
|
||||
let mut s = w.session();
|
||||
let mut events = Vec::new();
|
||||
let outcome = run_turn(&mut s, &w.runtime(), "hi", None, &mut |e| {
|
||||
events.push(e.clone())
|
||||
});
|
||||
w.finish(&mut s, "hi", outcome).unwrap();
|
||||
assert_eq!(s.epoch(), Epoch(1));
|
||||
assert_eq!(s.carry(), Some("the sum"));
|
||||
let sent = w.server.requests_to(CHAT);
|
||||
assert_eq!(sent.len(), 2);
|
||||
assert_eq!(sent[0].json()["id_slot"], 0);
|
||||
assert_eq!(sent[1].json()["id_slot"], 1);
|
||||
assert!(sent[1].json().get("tools").is_none());
|
||||
let old = w.home.records("a");
|
||||
assert!(matches!(
|
||||
old.last(),
|
||||
Some(LogRecord::EpochEnd { next, .. }) if *next == Epoch(1)
|
||||
));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn session_full_compacts_and_retries_the_user_message_once() {
|
||||
let w = World::new();
|
||||
w.server.route(
|
||||
CHAT,
|
||||
vec![
|
||||
Reply::fixture("context_full"),
|
||||
sse("the sum", 0, 10),
|
||||
sse("retried", 0, 20),
|
||||
],
|
||||
);
|
||||
let mut s = w.session();
|
||||
let mut events = Vec::new();
|
||||
let outcome = run_turn(&mut s, &w.runtime(), "please fit", None, &mut |e| {
|
||||
events.push(e.clone())
|
||||
});
|
||||
assert!(matches!(outcome, Err(TurnError::SessionFull)));
|
||||
let done = w.finish(&mut s, "please fit", outcome).unwrap();
|
||||
assert_eq!(done.content, "retried");
|
||||
assert_eq!(s.epoch(), Epoch(1));
|
||||
let users: Vec<_> = s
|
||||
.records()
|
||||
.iter()
|
||||
.filter_map(|r| match r {
|
||||
LogRecord::User { content, .. } => Some(content.as_str()),
|
||||
_ => None,
|
||||
})
|
||||
.collect();
|
||||
assert_eq!(users, ["please fit"]);
|
||||
let old_users = w
|
||||
.home
|
||||
.records("a")
|
||||
.iter()
|
||||
.filter(|r| matches!(r, LogRecord::User { .. }))
|
||||
.count();
|
||||
assert_eq!(old_users, 1);
|
||||
assert_eq!(s.turns(), 2);
|
||||
let sent = w.server.requests_to(CHAT);
|
||||
assert_eq!(sent.len(), 3);
|
||||
assert_eq!(sent[1].json()["id_slot"], 1);
|
||||
assert_eq!(sent[2].json()["id_slot"], 0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn session_full_stays_session_full_when_compact_fails() {
|
||||
let w = World::new();
|
||||
w.server.route(
|
||||
CHAT,
|
||||
vec![
|
||||
Reply::fixture("context_full"),
|
||||
Reply::fixture("bad_request"),
|
||||
],
|
||||
);
|
||||
let mut s = w.session();
|
||||
let mut events = Vec::new();
|
||||
let outcome = run_turn(&mut s, &w.runtime(), "x", None, &mut |e| {
|
||||
events.push(e.clone())
|
||||
});
|
||||
let err = w.finish(&mut s, "x", outcome).unwrap_err();
|
||||
assert!(matches!(err, TurnError::SessionFull), "{err:?}");
|
||||
assert_eq!(s.epoch(), Epoch(0));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_successful_turn_stays_successful_when_compact_fails() {
|
||||
let home = Home::new();
|
||||
let server = FakeServer::start();
|
||||
let mut cfg = home.config(&server.socket);
|
||||
cfg.expect.n_ctx = 1_000;
|
||||
cfg.compact.threshold = 0.85;
|
||||
let client = Client::new(cfg.clone());
|
||||
server.route(CHAT, vec![sse("ok", 400, 450)]);
|
||||
let baseline = Baseline::assemble(&cfg, &Registry::m2b()).unwrap();
|
||||
let mut s = Session::create(
|
||||
&home.dir,
|
||||
SessionId::new("a").unwrap(),
|
||||
baseline,
|
||||
cfg.slots.main,
|
||||
)
|
||||
.unwrap();
|
||||
let port = support::ScriptedPort::new(vec![]);
|
||||
let registry = Registry::m2b();
|
||||
let rt = Runtime {
|
||||
cfg: &cfg,
|
||||
client: &client,
|
||||
port: &port,
|
||||
registry: ®istry,
|
||||
};
|
||||
let mut events = Vec::new();
|
||||
let outcome = run_turn(&mut s, &rt, "hi", None, &mut |e| events.push(e.clone()));
|
||||
let done = compact::finish_turn(
|
||||
&mut s,
|
||||
&rt,
|
||||
&NoMemory,
|
||||
"hi",
|
||||
None,
|
||||
&mut |e| events.push(e.clone()),
|
||||
outcome,
|
||||
)
|
||||
.unwrap();
|
||||
assert_eq!(done.content, "ok");
|
||||
assert_eq!(s.epoch(), Epoch(0), "compact.md is missing, so no epoch");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn finish_turn_does_not_compact_an_error_other_than_session_full() {
|
||||
let w = World::new();
|
||||
w.server.route(CHAT, vec![Reply::fixture("bad_request")]);
|
||||
let mut s = w.session();
|
||||
let mut events = Vec::new();
|
||||
let outcome = run_turn(&mut s, &w.runtime(), "x", None, &mut |e| {
|
||||
events.push(e.clone())
|
||||
});
|
||||
let err = w.finish(&mut s, "x", outcome).unwrap_err();
|
||||
assert!(matches!(err, TurnError::Infer(_)), "{err:?}");
|
||||
assert_eq!(s.epoch(), Epoch(0));
|
||||
assert_eq!(w.server.requests_to(CHAT).len(), 1);
|
||||
}
|
||||
@@ -0,0 +1,119 @@
|
||||
//! Tests for `[compact]` on `loopd.toml`. Do not edit.
|
||||
|
||||
use loopd::config::Config;
|
||||
use std::path::{Path, PathBuf};
|
||||
|
||||
fn fixture(name: &str) -> PathBuf {
|
||||
Path::new(env!("CARGO_MANIFEST_DIR"))
|
||||
.join("tests/fixtures/config")
|
||||
.join(name)
|
||||
}
|
||||
|
||||
fn minimal() -> String {
|
||||
std::fs::read_to_string(fixture("minimal.toml")).unwrap()
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn missing_table_gets_the_documented_defaults() {
|
||||
let c = Config::load(&fixture("minimal.toml")).unwrap();
|
||||
assert_eq!(c.compact.threshold, 0.85);
|
||||
assert_eq!(c.compact.summary_chars, 4_000);
|
||||
assert_eq!(c.compact.summarise, vec!["http_fetch".to_string()]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_table_overrides_every_default() {
|
||||
let text = format!(
|
||||
"{}\n[compact]\nthreshold = 0.5\nsummary_chars = 100\nsummarise = [\"clock\"]\n",
|
||||
minimal()
|
||||
);
|
||||
let c = Config::parse(&text).unwrap();
|
||||
assert_eq!(c.compact.threshold, 0.5);
|
||||
assert_eq!(c.compact.summary_chars, 100);
|
||||
assert_eq!(c.compact.summarise, vec!["clock".to_string()]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_partial_table_keeps_the_other_defaults() {
|
||||
let text = format!("{}\n[compact]\nthreshold = 0.9\n", minimal());
|
||||
let c = Config::parse(&text).unwrap();
|
||||
assert_eq!(c.compact.threshold, 0.9);
|
||||
assert_eq!(c.compact.summary_chars, 4_000);
|
||||
assert_eq!(c.compact.summarise, vec!["http_fetch".to_string()]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn unknown_keys_are_errors() {
|
||||
let text = minimal();
|
||||
let bad = format!("{text}\n[compact]\nzz_unknown = 1\n");
|
||||
assert!(
|
||||
Config::parse(&bad).is_err(),
|
||||
"[compact] accepted an unknown key"
|
||||
);
|
||||
let misspelt = format!("{text}\n[compact]\nsummary_char = 10\n");
|
||||
assert!(
|
||||
Config::parse(&misspelt).is_err(),
|
||||
"[compact] accepted a misspelt key"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn empty_summarise_is_allowed() {
|
||||
let text = format!("{}\n[compact]\nsummarise = []\n", minimal());
|
||||
let c = Config::parse(&text).unwrap();
|
||||
assert!(c.compact.summarise.is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn compact_prompt_is_compact_md_beside_system_md() {
|
||||
let c = Config::load(&fixture("minimal.toml")).unwrap();
|
||||
assert_eq!(c.compact_prompt(), fixture("compact.md"));
|
||||
let home = Scratch::new();
|
||||
let cfg_path = home.dir.join("loopd.toml");
|
||||
std::fs::write(&cfg_path, minimal()).unwrap();
|
||||
std::fs::write(home.dir.join("system.md"), "sys\n").unwrap();
|
||||
let c = Config::load(&cfg_path).unwrap();
|
||||
assert_eq!(c.compact_prompt(), home.dir.join("compact.md"));
|
||||
}
|
||||
|
||||
struct Scratch {
|
||||
dir: PathBuf,
|
||||
}
|
||||
|
||||
impl Scratch {
|
||||
fn new() -> Scratch {
|
||||
let dir = std::env::temp_dir().join(format!(
|
||||
"loopd-compact-cfg-{}-{}",
|
||||
std::process::id(),
|
||||
std::time::SystemTime::now()
|
||||
.duration_since(std::time::UNIX_EPOCH)
|
||||
.unwrap()
|
||||
.as_nanos()
|
||||
));
|
||||
std::fs::create_dir_all(&dir).unwrap();
|
||||
Scratch { dir }
|
||||
}
|
||||
}
|
||||
|
||||
impl Drop for Scratch {
|
||||
fn drop(&mut self) {
|
||||
let _ = std::fs::remove_dir_all(&self.dir);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn compact_prompt_follows_an_overridden_system_path() {
|
||||
let text = std::fs::read_to_string(fixture("m2b.toml")).unwrap();
|
||||
let c = Config::load(&fixture("m2b.toml")).unwrap();
|
||||
assert!(
|
||||
text.contains("prompts/agent.md"),
|
||||
"m2b fixture names a prompt in a subdirectory"
|
||||
);
|
||||
assert_eq!(c.compact_prompt(), fixture("prompts/compact.md"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn defaults_match_the_struct_default() {
|
||||
let c = Config::load(&fixture("minimal.toml")).unwrap();
|
||||
assert_eq!(c.compact, loopd::config::Compact::default());
|
||||
}
|
||||
@@ -0,0 +1,255 @@
|
||||
//! Tests for multi-epoch session open, turns, trust and call ids. Do not edit.
|
||||
|
||||
mod support;
|
||||
|
||||
use bxcore::{CallId, DataClass, Epoch, LogRecord, SessionId, Timestamp};
|
||||
use loopd::baseline::Baseline;
|
||||
use loopd::session::{Session, SessionError};
|
||||
use support::Home;
|
||||
|
||||
fn baseline() -> Baseline {
|
||||
Baseline {
|
||||
system: "sys".to_string(),
|
||||
tools: vec![],
|
||||
}
|
||||
}
|
||||
|
||||
fn later() -> Baseline {
|
||||
Baseline {
|
||||
system: "sys-epoch-1".to_string(),
|
||||
tools: vec![],
|
||||
}
|
||||
}
|
||||
|
||||
fn id(s: &str) -> SessionId {
|
||||
SessionId::new(s).unwrap()
|
||||
}
|
||||
|
||||
fn ts() -> Timestamp {
|
||||
Timestamp::parse("2026-09-18T08:00:00.000Z").unwrap()
|
||||
}
|
||||
|
||||
fn user(content: &str) -> LogRecord {
|
||||
LogRecord::User {
|
||||
time: ts(),
|
||||
content: content.to_string(),
|
||||
speaker: None,
|
||||
}
|
||||
}
|
||||
|
||||
fn write_jsonl(home: &Home, session: &str, epoch: u32, records: &[LogRecord]) {
|
||||
let text: String = records
|
||||
.iter()
|
||||
.map(|r| format!("{}\n", serde_json::to_string(r).unwrap()))
|
||||
.collect();
|
||||
home.write(&format!("sessions/{session}/{epoch}.jsonl"), &text);
|
||||
}
|
||||
|
||||
fn write_baseline(home: &Home, session: &str, epoch: u32, b: &Baseline) {
|
||||
home.write(
|
||||
&format!("sessions/{session}/{epoch}.baseline.json"),
|
||||
&b.to_json().unwrap(),
|
||||
);
|
||||
}
|
||||
|
||||
fn start(session: &str, epoch: u32, slot: u32, b: &Baseline) -> LogRecord {
|
||||
LogRecord::SessionStart {
|
||||
time: ts(),
|
||||
session: id(session),
|
||||
epoch: Epoch(epoch),
|
||||
slot,
|
||||
baseline: b.hash().unwrap(),
|
||||
}
|
||||
}
|
||||
|
||||
fn epoch_end(next: u32, summary: &str) -> LogRecord {
|
||||
LogRecord::EpochEnd {
|
||||
time: ts(),
|
||||
next: Epoch(next),
|
||||
summary: summary.to_string(),
|
||||
}
|
||||
}
|
||||
|
||||
fn tool_result(call: u64, untrusted: bool) -> LogRecord {
|
||||
LogRecord::ToolResult {
|
||||
time: ts(),
|
||||
call: CallId(call),
|
||||
tool_call_id: format!("c{call}"),
|
||||
content: "out".to_string(),
|
||||
class: DataClass::Public,
|
||||
untrusted,
|
||||
truncated: false,
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn open_loads_the_highest_complete_epoch_and_its_carry() {
|
||||
let home = Home::new();
|
||||
let mut s = Session::create(&home.dir, id("a"), baseline(), 3).unwrap();
|
||||
s.append(user("one")).unwrap();
|
||||
s.append(user("two")).unwrap();
|
||||
s.append(epoch_end(1, "carry-text")).unwrap();
|
||||
drop(s);
|
||||
write_baseline(&home, "a", 1, &later());
|
||||
write_jsonl(&home, "a", 1, &[start("a", 1, 3, &later()), user("three")]);
|
||||
|
||||
let s = Session::open(&home.dir, id("a")).unwrap();
|
||||
assert_eq!(s.epoch(), Epoch(1));
|
||||
assert_eq!(s.slot(), 3);
|
||||
assert_eq!(s.carry(), Some("carry-text"));
|
||||
assert_eq!(s.baseline(), &later());
|
||||
assert_eq!(s.records().len(), 2);
|
||||
assert!(matches!(s.records()[1], LogRecord::User { ref content, .. } if content == "three"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_missing_carry_on_epoch_1_is_an_error() {
|
||||
let home = Home::new();
|
||||
let s = Session::create(&home.dir, id("a"), baseline(), 0).unwrap();
|
||||
drop(s);
|
||||
write_baseline(&home, "a", 1, &later());
|
||||
write_jsonl(&home, "a", 1, &[start("a", 1, 0, &later())]);
|
||||
|
||||
match Session::open(&home.dir, id("a")) {
|
||||
Err(SessionError::NoCarry { ref path, .. }) => {
|
||||
assert!(path.ends_with("0.jsonl"), "{path:?}");
|
||||
}
|
||||
other => panic!("{other:?}"),
|
||||
}
|
||||
let e: Box<dyn std::error::Error> =
|
||||
Box::new(Session::open(&home.dir, id("a")).map(|_| ()).unwrap_err());
|
||||
assert!(e.to_string().contains("0.jsonl"), "{e}");
|
||||
assert!(
|
||||
e.to_string()
|
||||
.contains("see docs/runbook.md#session-log-damaged"),
|
||||
"{e}"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn epoch_0_has_no_carry() {
|
||||
let home = Home::new();
|
||||
let s = Session::create(&home.dir, id("a"), baseline(), 7).unwrap();
|
||||
drop(s);
|
||||
let s = Session::open(&home.dir, id("a")).unwrap();
|
||||
assert_eq!(s.epoch(), Epoch(0));
|
||||
assert_eq!(s.carry(), None);
|
||||
assert_eq!(s.slot(), 7);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn turns_count_users_across_every_epoch() {
|
||||
let home = Home::new();
|
||||
let mut s = Session::create(&home.dir, id("a"), baseline(), 0).unwrap();
|
||||
s.append(user("one")).unwrap();
|
||||
s.append(user("two")).unwrap();
|
||||
s.append(epoch_end(1, "sum")).unwrap();
|
||||
drop(s);
|
||||
write_baseline(&home, "a", 1, &later());
|
||||
write_jsonl(&home, "a", 1, &[start("a", 1, 0, &later()), user("three")]);
|
||||
let s = Session::open(&home.dir, id("a")).unwrap();
|
||||
assert_eq!(s.turns(), 3);
|
||||
assert_eq!(
|
||||
s.records()
|
||||
.iter()
|
||||
.filter(|r| matches!(r, LogRecord::User { .. }))
|
||||
.count(),
|
||||
1,
|
||||
"records() is the current epoch only"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn untrusted_reads_every_epoch() {
|
||||
let home = Home::new();
|
||||
let mut s = Session::create(&home.dir, id("a"), baseline(), 0).unwrap();
|
||||
s.append(tool_result(1, true)).unwrap();
|
||||
s.append(epoch_end(1, "sum")).unwrap();
|
||||
drop(s);
|
||||
write_baseline(&home, "a", 1, &later());
|
||||
write_jsonl(
|
||||
&home,
|
||||
"a",
|
||||
1,
|
||||
&[start("a", 1, 0, &later()), tool_result(2, false)],
|
||||
);
|
||||
let s = Session::open(&home.dir, id("a")).unwrap();
|
||||
assert!(s.untrusted(), "an untrusted result in epoch 0 still taints");
|
||||
assert!(
|
||||
!s.records().iter().any(|r| matches!(
|
||||
r,
|
||||
LogRecord::ToolResult {
|
||||
untrusted: true,
|
||||
..
|
||||
}
|
||||
)),
|
||||
"the current epoch's records look clean"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn call_ids_continue_across_epochs() {
|
||||
let home = Home::new();
|
||||
let mut s = Session::create(&home.dir, id("a"), baseline(), 0).unwrap();
|
||||
s.append(tool_result(4, false)).unwrap();
|
||||
s.append(epoch_end(1, "sum")).unwrap();
|
||||
drop(s);
|
||||
write_baseline(&home, "a", 1, &later());
|
||||
write_jsonl(
|
||||
&home,
|
||||
"a",
|
||||
1,
|
||||
&[start("a", 1, 0, &later()), tool_result(2, false)],
|
||||
);
|
||||
let mut s = Session::open(&home.dir, id("a")).unwrap();
|
||||
assert_eq!(
|
||||
s.next_call(),
|
||||
CallId(5),
|
||||
"one more than the highest call in any epoch"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn append_writes_the_current_epoch_file() {
|
||||
let home = Home::new();
|
||||
let mut s = Session::create(&home.dir, id("a"), baseline(), 0).unwrap();
|
||||
s.append(epoch_end(1, "sum")).unwrap();
|
||||
drop(s);
|
||||
write_baseline(&home, "a", 1, &later());
|
||||
write_jsonl(&home, "a", 1, &[start("a", 1, 0, &later())]);
|
||||
let mut s = Session::open(&home.dir, id("a")).unwrap();
|
||||
s.append(user("after")).unwrap();
|
||||
let text = home.read("sessions/a/1.jsonl");
|
||||
assert!(text.contains("after"), "{text}");
|
||||
let old = home.read("sessions/a/0.jsonl");
|
||||
assert!(!old.contains("after"), "{old}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn an_incomplete_next_epoch_is_ignored() {
|
||||
let home = Home::new();
|
||||
let mut s = Session::create(&home.dir, id("a"), baseline(), 0).unwrap();
|
||||
s.append(user("still-here")).unwrap();
|
||||
drop(s);
|
||||
write_baseline(&home, "a", 1, &later());
|
||||
let s = Session::open(&home.dir, id("a")).unwrap();
|
||||
assert_eq!(s.epoch(), Epoch(0));
|
||||
assert_eq!(s.carry(), None);
|
||||
assert_eq!(s.turns(), 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn epoch_end_without_the_next_files_stays_on_the_old_epoch() {
|
||||
let home = Home::new();
|
||||
let mut s = Session::create(&home.dir, id("a"), baseline(), 0).unwrap();
|
||||
s.append(user("x")).unwrap();
|
||||
s.append(epoch_end(1, "sum")).unwrap();
|
||||
drop(s);
|
||||
let s = Session::open(&home.dir, id("a")).unwrap();
|
||||
assert_eq!(s.epoch(), Epoch(0));
|
||||
assert_eq!(s.carry(), None);
|
||||
assert_eq!(s.turns(), 1);
|
||||
assert!(
|
||||
matches!(s.records().last(), Some(LogRecord::EpochEnd { next, .. }) if *next == Epoch(1))
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,181 @@
|
||||
//! Honcho reads every epoch file of a session, in order. Do not edit.
|
||||
|
||||
#[path = "support/logs.rs"]
|
||||
mod logs;
|
||||
#[path = "support/tmp.rs"]
|
||||
mod tmp;
|
||||
#[path = "support/turns.rs"]
|
||||
mod turns;
|
||||
|
||||
use std::cell::RefCell;
|
||||
|
||||
use bxcore::{Epoch, LogRecord, SessionId, Timestamp};
|
||||
use logs::{assistant, write_log};
|
||||
use memoryd::honcho::decide::{turn_records, turns};
|
||||
use memoryd::honcho::ledger::{Action, Item, Ledger, Record, Workspace};
|
||||
use memoryd::honcho::turns::{LOG_RUNBOOK, TurnSync, read_session};
|
||||
use tmp::TempDir;
|
||||
use turns::{clean, sid, spoken};
|
||||
|
||||
const LEDGER: &str = "honcho.jsonl";
|
||||
|
||||
fn epoch_end(summary: &str) -> LogRecord {
|
||||
LogRecord::EpochEnd {
|
||||
time: Timestamp::parse("2026-09-28T10:00:00.000Z").unwrap(),
|
||||
next: Epoch(1),
|
||||
summary: summary.to_string(),
|
||||
}
|
||||
}
|
||||
|
||||
fn write_epoch(home: &std::path::Path, session: &str, epoch: u32, records: &[LogRecord]) {
|
||||
let dir = home.join("sessions").join(session);
|
||||
std::fs::create_dir_all(&dir).unwrap();
|
||||
let path = dir.join(format!("{epoch}.jsonl"));
|
||||
let text: String = records
|
||||
.iter()
|
||||
.map(|r| format!("{}\n", serde_json::to_string(r).unwrap()))
|
||||
.collect();
|
||||
std::fs::write(path, text).unwrap();
|
||||
}
|
||||
|
||||
struct World {
|
||||
sync: TurnSync,
|
||||
answer: RefCell<Result<bxcore::SessionStateReply, String>>,
|
||||
dir: TempDir,
|
||||
}
|
||||
|
||||
impl World {
|
||||
fn new(tag: &str) -> World {
|
||||
let dir = TempDir::new(tag);
|
||||
std::fs::create_dir_all(dir.path().join("memory")).unwrap();
|
||||
let mut ledger = Ledger::open(&dir.path().join("memory").join(LEDGER)).unwrap();
|
||||
ledger
|
||||
.append(Record::Began {
|
||||
time: Timestamp::parse("2026-09-28T09:00:00.000Z").unwrap(),
|
||||
})
|
||||
.unwrap();
|
||||
World {
|
||||
sync: TurnSync::new(ledger, "boxmaker0", &dir.path().join("sessions")),
|
||||
answer: RefCell::new(Ok(clean())),
|
||||
dir,
|
||||
}
|
||||
}
|
||||
|
||||
fn updated(&mut self, session: &str, turn: u64) {
|
||||
let ask = |s: &SessionId| {
|
||||
let _ = s;
|
||||
self.answer.borrow().clone()
|
||||
};
|
||||
let log = |_line: &str| {};
|
||||
let now = Timestamp::parse("2026-09-28T10:00:00.000Z").unwrap();
|
||||
self.sync.updated(&sid(session), turn, &ask, now, &log);
|
||||
}
|
||||
|
||||
fn action(&self, session: &str, turn: u64) -> Option<Action> {
|
||||
let item = Item::Turn {
|
||||
session: sid(session),
|
||||
turn,
|
||||
};
|
||||
let path = self.dir.path().join("memory").join(LEDGER);
|
||||
let text = std::fs::read_to_string(path).unwrap_or_default();
|
||||
let key = serde_json::to_string(&item).unwrap();
|
||||
let line = text.lines().find(|l| l.contains(&key))?;
|
||||
let value: serde_json::Value = serde_json::from_str(line).unwrap();
|
||||
Some(serde_json::from_value(value["action"].clone()).unwrap())
|
||||
}
|
||||
}
|
||||
|
||||
fn to(workspace: Workspace, peer: &str) -> Action {
|
||||
Action::To {
|
||||
workspace,
|
||||
peer: peer.to_string(),
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn concatenated_logs_count_three_turns() {
|
||||
let dir = TempDir::new("honcho-epochs-count");
|
||||
let home = dir.path();
|
||||
let epoch0 = vec![
|
||||
spoken("one", Some("kyle")),
|
||||
assistant(2, Some("ok"), &[]),
|
||||
spoken("two", Some("kyle")),
|
||||
assistant(3, Some("ok"), &[]),
|
||||
epoch_end("sum"),
|
||||
];
|
||||
write_log(home, "mm-a", &epoch0);
|
||||
write_epoch(
|
||||
home,
|
||||
"mm-a",
|
||||
1,
|
||||
&[
|
||||
spoken("three", Some("kyle")),
|
||||
assistant(4, Some("ok"), &[]),
|
||||
],
|
||||
);
|
||||
let records = read_session(&home.join("sessions"), &sid("mm-a")).unwrap();
|
||||
assert_eq!(turns(&records), 3);
|
||||
let turn3 = turn_records(&records, 3).unwrap();
|
||||
assert!(
|
||||
matches!(turn3[0], LogRecord::User { ref content, .. } if content == "three"),
|
||||
"{turn3:?}"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn epoch_0_alone_still_reads() {
|
||||
let dir = TempDir::new("honcho-epochs-zero");
|
||||
let home = dir.path();
|
||||
write_log(home, "mm-a", &[spoken("only", Some("kyle"))]);
|
||||
let records = read_session(&home.join("sessions"), &sid("mm-a")).unwrap();
|
||||
assert_eq!(turns(&records), 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_damaged_later_epoch_is_an_error_naming_that_file() {
|
||||
let dir = TempDir::new("honcho-epochs-damaged");
|
||||
let home = dir.path();
|
||||
write_log(home, "mm-a", &[spoken("one", Some("kyle"))]);
|
||||
let path = home.join("sessions/mm-a/1.jsonl");
|
||||
std::fs::write(&path, "{\"type\":\"zz\"}\n").unwrap();
|
||||
let err = read_session(&home.join("sessions"), &sid("mm-a")).unwrap_err();
|
||||
assert!(err.to_string().contains("1.jsonl"), "{err}");
|
||||
assert!(
|
||||
err.to_string().contains(LOG_RUNBOOK) || err.to_string().contains("session-log-damaged"),
|
||||
"{err}"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn turn_sync_decides_a_turn_that_lives_in_epoch_1() {
|
||||
let mut w = World::new("honcho-epochs-sync");
|
||||
write_log(
|
||||
w.dir.path(),
|
||||
"mm-a",
|
||||
&[
|
||||
spoken("one", Some("kyle")),
|
||||
assistant(2, Some("ok"), &[]),
|
||||
spoken("two", Some("kyle")),
|
||||
assistant(3, Some("ok"), &[]),
|
||||
epoch_end("sum"),
|
||||
],
|
||||
);
|
||||
write_epoch(
|
||||
w.dir.path(),
|
||||
"mm-a",
|
||||
1,
|
||||
&[
|
||||
spoken("three", Some("kyle")),
|
||||
assistant(4, Some("ok"), &[]),
|
||||
],
|
||||
);
|
||||
w.updated("mm-a", 3);
|
||||
assert_eq!(w.action("mm-a", 3), Some(to(Workspace::Trusted, "kyle")));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_missing_session_directory_is_empty() {
|
||||
let dir = TempDir::new("honcho-epochs-missing");
|
||||
let records = read_session(&dir.path().join("sessions"), &sid("mm-none")).unwrap();
|
||||
assert!(records.is_empty());
|
||||
}
|
||||
@@ -0,0 +1,35 @@
|
||||
## session-compact-failed
|
||||
|
||||
**What you see.** `loopd` logs that it could not compact a session, then this entry. The session
|
||||
stays on its current epoch. A turn that was already too large to fit (`session_full`) stays
|
||||
`session_full`. A turn that finished stays finished; the next turn on that session may be
|
||||
`session_full` if it would not fit.
|
||||
|
||||
```text
|
||||
loopd: could not compact session chat-0-0: compact.md: No such file or directory (os error 2); see docs/runbook.md#session-compact-failed
|
||||
```
|
||||
|
||||
**Why.** Compaction needs a summary of the current epoch (a completion on the background slot
|
||||
with `compact.md` as the last user message, or the fallback of that epoch's owner texts). If the
|
||||
prompt file cannot be read, the summary call fails for a reason other than a full context, or
|
||||
writing the next epoch's files fails, `loopd` refuses to guess a prefix. It does not start a new
|
||||
epoch from an empty summary when the model call failed.
|
||||
|
||||
**Confirm.**
|
||||
|
||||
```sh
|
||||
ls -l "$(dirname "$BOXMAKER_HOME/../config")/compact.md" 2>/dev/null
|
||||
ls -l "$BOXMAKER_HOME/sessions/<id>"/
|
||||
tail -n 5 "$BOXMAKER_HOME/sessions/<id>/0.jsonl"
|
||||
```
|
||||
|
||||
`compact.md` lives next to `system.md` (the path in `loopd.toml`'s `[baseline] system`). A last
|
||||
line of type `epoch_end` without `<n>.baseline.json` and `<n>.jsonl` is a compact that stopped
|
||||
halfway; the next compact resumes it.
|
||||
|
||||
**Fix.** Restore `compact.md` beside `system.md` (the starter is `config/compact.md` in the
|
||||
repository). Check that `inferproxy` still answers. If the session directory is mid-write, do not
|
||||
delete `EpochEnd`; the next compact continues from it.
|
||||
|
||||
**Check.** The next turn on that session either runs on the current epoch, or compact succeeds
|
||||
and the next turn's log is `<n>.jsonl` with `epoch` n.
|
||||
@@ -0,0 +1,124 @@
|
||||
# M5d-2 task 01: `run_turn` sends `session.slot()`
|
||||
|
||||
**Branch:** `m5d2`. Check it: `git branch --show-current` prints `m5d2` and `git status --short`
|
||||
is empty; otherwise stop. Never switch branch.
|
||||
**Commit subject:** `loopd: live turns use the session's slot`
|
||||
**Log row:** the Task cell is `M5d2/01-loopd-session-slot`.
|
||||
|
||||
## Goal
|
||||
|
||||
Heartbeat and cron run on the background slot (spec 4). Live Mattermost and `bxctl chat`
|
||||
sessions keep `cfg.slots.main`. Today `run_turn` always sends `slots.main`. This task makes it
|
||||
send `session.slot()`: the slot stored on that session's `SessionStart` (M5d-1 task 02).
|
||||
|
||||
Nothing else changes. Compaction's summary call already uses `slots.background`. The scheduler
|
||||
does not exist yet.
|
||||
|
||||
## Every place that changes
|
||||
|
||||
| Place | What happens there |
|
||||
|---|---|
|
||||
| `crates/loopd/src/turn.rs` | `ChatRequest.slot` is `session.slot()` (edit T1) |
|
||||
| `crates/loopd/tests/session_slot.rs` | new, given |
|
||||
| everything else | **no change**: not `session.rs`, not `compact.rs`, not channel |
|
||||
|
||||
## Files
|
||||
|
||||
- Copy, from `docs/plans/M5d2/files/` (the same path under it): the new test
|
||||
`crates/loopd/tests/session_slot.rs`.
|
||||
- Modify: `crates/loopd/src/turn.rs`, `docs/implementer-log.md`.
|
||||
- Do not touch: any other file.
|
||||
|
||||
## Facts you need
|
||||
|
||||
- **`Session::slot()`** returns the `u32` stored on create / on the current epoch's
|
||||
`SessionStart`. It is already `pub` (M5d-1 task 02).
|
||||
- **`messages_with_carry`** stays as it is. Only the `slot:` line changes.
|
||||
- **Existing turn tests** create sessions with `cfg.slots.main` (0). They still send 0.
|
||||
- The items the edits use. This table is complete.
|
||||
|
||||
| Item | What it does |
|
||||
|---|---|
|
||||
| `Session::slot` | the live slot for this session |
|
||||
| `ChatRequest.slot` | becomes `id_slot` on the wire |
|
||||
|
||||
## The edits, exactly
|
||||
|
||||
Each quoted "before" block occurs exactly once after M5d-1 task 08.
|
||||
|
||||
### `crates/loopd/src/turn.rs`
|
||||
|
||||
**T1.** The lines
|
||||
|
||||
```rust
|
||||
let request = ChatRequest {
|
||||
slot: rt.cfg.slots.main,
|
||||
messages: messages_with_carry(baseline, session.records(), session.carry()),
|
||||
tools: baseline.tools.clone(),
|
||||
thinking: true,
|
||||
};
|
||||
```
|
||||
|
||||
become
|
||||
|
||||
```rust
|
||||
let request = ChatRequest {
|
||||
slot: session.slot(),
|
||||
messages: messages_with_carry(baseline, session.records(), session.carry()),
|
||||
tools: baseline.tools.clone(),
|
||||
thinking: true,
|
||||
};
|
||||
```
|
||||
|
||||
## Steps
|
||||
|
||||
- [ ] **1. Copy the given test.** From the repository root, one command:
|
||||
|
||||
```sh
|
||||
cp docs/plans/M5d2/files/crates/loopd/tests/session_slot.rs crates/loopd/tests/
|
||||
```
|
||||
|
||||
Then `git status --short` lists one new file, `crates/loopd/tests/session_slot.rs`.
|
||||
- [ ] **2. See it fail.** `cargo test -p loopd --test session_slot`, with a shell timeout of
|
||||
600000 ms: the four tests compile (they call `session.slot()`, which exists) and
|
||||
`a_session_created_on_the_background_slot_sends_that_slot` fails: `id_slot` is `0`, not `1`.
|
||||
`a_main_slot_session_still_sends_main` may pass. `the_request_slot_follows_session_start_not_the_config_main`
|
||||
fails: `id_slot` is `0`, not `7`.
|
||||
- [ ] **3. The edit.** Make edit T1. `cargo check -p loopd` prints no error and no warning.
|
||||
- [ ] **4. The tests.** `cargo test -p loopd --test session_slot --test turn --test channel
|
||||
--test compact_turn`: every test passes, 4 in `session_slot` and the others as before.
|
||||
- [ ] **5. Check.** `git diff --stat` lists exactly `crates/loopd/src/turn.rs`.
|
||||
- [ ] **6. Run the gate.** `cargo fmt --all`, then `make gate`, with a shell timeout of 600000 ms.
|
||||
Expected last line: `gate: ok`. The test count is **1511** after this task, 1507 before it,
|
||||
and the 4 new ones are the tests of `session_slot.rs`. To count, add up the `N passed` of
|
||||
every `test result:` line the gate's `cargo test --workspace` printed, and add 1. Never run
|
||||
the gate again only to count.
|
||||
- [ ] **7. Log and commit.** Add your row to `docs/implementer-log.md`; its Task cell is
|
||||
`M5d2/01-loopd-session-slot`. Stage with
|
||||
`git add crates/loopd docs/implementer-log.md Cargo.lock`, then commit as `AGENTS.md` says.
|
||||
`Cargo.lock` does not change in this task; the `git add` is harmless.
|
||||
|
||||
## Done when
|
||||
|
||||
- `make gate` prints `gate: ok` with 1511 tests.
|
||||
- `git diff HEAD~1 --stat` lists only `crates/loopd/src/turn.rs`,
|
||||
`crates/loopd/tests/session_slot.rs` and `docs/implementer-log.md`.
|
||||
|
||||
## Stop and report if
|
||||
|
||||
- The branch check at the top fails.
|
||||
- A "before" block is not in its file exactly as shown, or is there more than once.
|
||||
- Step 2 shows a different result: the tests do not compile, or a different assertion fails.
|
||||
- `cargo check` in step 3 prints an error or a warning. Quote it, and stop.
|
||||
- A test fails in step 4. Do not change the test: stop, and give the failing tests' names and
|
||||
their messages.
|
||||
- The fix seems to need a change anywhere else. It does not; stop and say why you think so.
|
||||
- A test outside `session_slot.rs` fails in the gate: check once whether it fails without your
|
||||
change (`git stash -u`, run it, `git stash pop`), then stop and report both results.
|
||||
- The gate passes with a count other than 1511.
|
||||
|
||||
## What the given tests pin
|
||||
|
||||
A session created with `cfg.slots.background` sends that slot; one created with `cfg.slots.main`
|
||||
still sends main; a session created with slot `7` sends `7` even when `cfg.slots.main` is `0`; a
|
||||
heartbeat turn between two main turns does not change the main session's message prefix.
|
||||
@@ -0,0 +1,165 @@
|
||||
# M5d-2 task 02: `LogRecord::ToolSummary`
|
||||
|
||||
**Branch:** `m5d2`. Check it: `git branch --show-current` prints `m5d2` and `git status --short`
|
||||
is empty; otherwise stop. Never switch branch.
|
||||
**Commit subject:** `bxcore: ToolSummary log record; messages ignore it`
|
||||
**Log row:** the Task cell is `M5d2/02-bxcore-tool-summary`.
|
||||
|
||||
## Goal
|
||||
|
||||
A tool that opts in gets a second record, for the index only (spec 4.3):
|
||||
|
||||
```text
|
||||
LogRecord::ToolSummary { time, call, tool_call_id, content, class, untrusted }
|
||||
```
|
||||
|
||||
JSON type `tool_summary`. `messages()` / `messages_with_carry` ignore it, so the epoch's prefix
|
||||
does not change. `support/turn.rs` `types` names it (T19: this task ships its own copy).
|
||||
|
||||
Nothing summarises yet. That is task 08.
|
||||
|
||||
## Every place that changes
|
||||
|
||||
| Place | What happens there |
|
||||
|---|---|
|
||||
| `crates/bxcore/src/log.rs` | the variant, after `ToolResult` (edit L1) |
|
||||
| `crates/loopd/src/baseline.rs` | ignore arm (edit B1) |
|
||||
| `crates/loopd/tests/support/turn.rs` | replaced, given |
|
||||
| `crates/bxcore/tests/tool_summary.rs` | new, given |
|
||||
| `crates/loopd/tests/tool_summary_messages.rs` | new, given |
|
||||
| everything else | **no change**: not `index.rs` (task 09), not `turn.rs` |
|
||||
|
||||
`memoryd/src/index.rs` `index_record` already has `_ => {}`. It compiles. Summaries are skipped
|
||||
until task 09.
|
||||
|
||||
## Files
|
||||
|
||||
- Copy, from `docs/plans/M5d2/files/` (the same path under it): the new tests
|
||||
`crates/bxcore/tests/tool_summary.rs` and
|
||||
`crates/loopd/tests/tool_summary_messages.rs`, and the replacement
|
||||
`crates/loopd/tests/support/turn.rs`.
|
||||
- Modify: `crates/bxcore/src/log.rs`, `crates/loopd/src/baseline.rs`,
|
||||
`docs/implementer-log.md`.
|
||||
- Do not touch: any other file.
|
||||
|
||||
## Facts you need
|
||||
|
||||
- **Field order is the wire format.** `time`, `call`, `tool_call_id`, `content`, `class`,
|
||||
`untrusted`. No `truncated`: a summary is already capped.
|
||||
- **Unknown fields are errors**, as on every `LogRecord` variant (`deny_unknown_fields` is on
|
||||
the enum).
|
||||
- **`types` is exhaustive.** Adding the variant without the replacement `support/turn.rs` fails
|
||||
to compile `turn` / `limits` tests. Copy the given file over the existing one. The only
|
||||
intended change is the `ToolSummary => "tool_summary"` arm. `Setup::turn` still returns
|
||||
`(Result<TurnOutcome, TurnError>, Vec<TurnEvent>)`.
|
||||
- **`messages` names every variant.** The new arm is empty, like `Usage` / `CacheLoss` /
|
||||
`EpochEnd`.
|
||||
- The items the edits use. This table is complete.
|
||||
|
||||
| Item | What it does |
|
||||
|---|---|
|
||||
| `CallId` | already imported in `log.rs` |
|
||||
| `DataClass` | already imported in `log.rs` |
|
||||
|
||||
## The edits, exactly
|
||||
|
||||
Each quoted "before" block occurs exactly once after task 01.
|
||||
|
||||
### `crates/bxcore/src/log.rs`
|
||||
|
||||
**L1.** After the `ToolResult` variant (the lines)
|
||||
|
||||
```rust
|
||||
truncated: bool,
|
||||
},
|
||||
Usage {
|
||||
```
|
||||
|
||||
become
|
||||
|
||||
```rust
|
||||
truncated: bool,
|
||||
},
|
||||
ToolSummary {
|
||||
time: Timestamp,
|
||||
call: CallId,
|
||||
tool_call_id: String,
|
||||
content: String,
|
||||
class: DataClass,
|
||||
untrusted: bool,
|
||||
},
|
||||
Usage {
|
||||
```
|
||||
|
||||
Keep the fields in that order.
|
||||
|
||||
### `crates/loopd/src/baseline.rs`
|
||||
|
||||
**B1.** After the `ToolResult` arm (the lines)
|
||||
|
||||
```rust
|
||||
} => out.push(ChatMessage::Tool {
|
||||
tool_call_id: tool_call_id.clone(),
|
||||
content: content.clone(),
|
||||
}),
|
||||
LogRecord::Usage { .. } => {}
|
||||
```
|
||||
|
||||
become
|
||||
|
||||
```rust
|
||||
} => out.push(ChatMessage::Tool {
|
||||
tool_call_id: tool_call_id.clone(),
|
||||
content: content.clone(),
|
||||
}),
|
||||
LogRecord::ToolSummary { .. } => {}
|
||||
LogRecord::Usage { .. } => {}
|
||||
```
|
||||
|
||||
## Steps
|
||||
|
||||
- [ ] **1. Copy the given files.** From the repository root:
|
||||
|
||||
```sh
|
||||
cp docs/plans/M5d2/files/crates/bxcore/tests/tool_summary.rs crates/bxcore/tests/
|
||||
cp docs/plans/M5d2/files/crates/loopd/tests/tool_summary_messages.rs crates/loopd/tests/
|
||||
cp docs/plans/M5d2/files/crates/loopd/tests/support/turn.rs crates/loopd/tests/support/turn.rs
|
||||
```
|
||||
|
||||
- [ ] **2. See them fail.** `cargo build --tests --keep-going -p bxcore -p loopd`, timeout
|
||||
600000 ms: `tool_summary` and `tool_summary_messages` do not compile (`E0599` no variant
|
||||
`ToolSummary`). `support/turn.rs` does not compile for the same reason until L1. After L1
|
||||
and before B1, `baseline.rs` fails (`non-exhaustive patterns`). Do L1 then B1 in step 3;
|
||||
this step is the first `cargo build` against the given tests only, which is E0599.
|
||||
- [ ] **3. The edits.** Make L1 then B1. `cargo check -p bxcore -p loopd` prints no error and
|
||||
no warning.
|
||||
- [ ] **4. The tests.** `cargo test -p bxcore --test tool_summary --test records` and
|
||||
`cargo test -p loopd --test tool_summary_messages --test turn --test baseline --test
|
||||
session_slot`: every test passes, 3 in `tool_summary`, 3 in `tool_summary_messages`.
|
||||
- [ ] **5. Check.** `git diff --stat` lists `crates/bxcore/src/log.rs`,
|
||||
`crates/loopd/src/baseline.rs`, `crates/loopd/tests/support/turn.rs`.
|
||||
- [ ] **6. Run the gate.** `cargo fmt --all`, then `make gate`, timeout 600000 ms. Last line
|
||||
`gate: ok`. Test count **1517** (1511 before, plus 6).
|
||||
- [ ] **7. Log and commit.** Task cell `M5d2/02-bxcore-tool-summary`. Stage with
|
||||
`git add crates/bxcore crates/loopd docs/implementer-log.md Cargo.lock`, then commit as
|
||||
`AGENTS.md` says.
|
||||
|
||||
## Done when
|
||||
|
||||
- `make gate` prints `gate: ok` with 1517 tests.
|
||||
- `git diff HEAD~1 --stat` lists only `crates/bxcore/src/log.rs`,
|
||||
`crates/bxcore/tests/tool_summary.rs`, `crates/loopd/src/baseline.rs`,
|
||||
`crates/loopd/tests/tool_summary_messages.rs`, `crates/loopd/tests/support/turn.rs` and the
|
||||
log.
|
||||
|
||||
## Stop and report if
|
||||
|
||||
- The branch check fails; a "before" block is not found exactly once; step 2 differs; `cargo
|
||||
check` warns; a test fails; the gate count is not 1517; you replay a summary as a `User` or
|
||||
`Tool` message; you add `truncated`; you change `Setup::turn`'s return type.
|
||||
|
||||
## What the given tests pin
|
||||
|
||||
JSON type is `tool_summary`; unknown fields are refused; field order is the struct order;
|
||||
`messages()` with and without a summary are equal; a summary is not a tool message; inserting a
|
||||
summary between a result and a later user does not break prefix extension.
|
||||
@@ -0,0 +1,218 @@
|
||||
# M5d-2 task 03: `[heartbeat]` and unknown `[compact] summarise` names
|
||||
|
||||
**Branch:** `m5d2`. Check it: `git branch --show-current` prints `m5d2` and `git status --short`
|
||||
is empty; otherwise stop. Never switch branch.
|
||||
**Commit subject:** `loopd: [heartbeat] table and summarise name check`
|
||||
**Log row:** the Task cell is `M5d2/03-loopd-heartbeat-config`.
|
||||
|
||||
## Goal
|
||||
|
||||
`[heartbeat]` on `loopd.toml` (spec 4.1, 5): `every_s` (default `0`, disables) and `prompt`
|
||||
(default `heartbeat.md`, next to the config file). Unknown keys are errors.
|
||||
|
||||
Unknown names in `[compact] summarise` are a config error (spec 4.3). `Config::check_summarise`
|
||||
takes `&Registry`. `loopd` calls it after `load` with `Registry::m5c()`. `parse` still succeeds
|
||||
with an unknown name: the given tests call `check_summarise` themselves.
|
||||
|
||||
The starter `config/heartbeat.md` is copied like `config/system.md`. Nothing ticks yet.
|
||||
|
||||
## Every place that changes
|
||||
|
||||
| Place | What happens there |
|
||||
|---|---|
|
||||
| `crates/loopd/src/config.rs` | `Heartbeat`; field; `load` joins the prompt; `heartbeat_prompt`; `check_summarise` |
|
||||
| `crates/loopd/src/main.rs` | after each `Config::load`, `check_summarise(&Registry::m5c())` |
|
||||
| `config/heartbeat.md` | new, given starter |
|
||||
| `crates/loopd/tests/config_heartbeat.rs` | new, given |
|
||||
| everything else | **no change** |
|
||||
|
||||
## Files
|
||||
|
||||
- Copy, from `docs/plans/M5d2/files/` (the same path under it): the new test
|
||||
`crates/loopd/tests/config_heartbeat.rs` and the starter `config/heartbeat.md`.
|
||||
- Modify: `crates/loopd/src/config.rs`, `crates/loopd/src/main.rs`,
|
||||
`docs/implementer-log.md`.
|
||||
- Do not touch: any other file.
|
||||
|
||||
## Facts you need
|
||||
|
||||
- **`every_s` is `u64`.** `0` means off. No range check.
|
||||
- **`prompt` after `load`.** Join it to the config file's parent, the same way `load` already
|
||||
joins `baseline.system`. `heartbeat_prompt()` returns `self.heartbeat.prompt.clone()`. After
|
||||
`parse` (no file), the path stays relative.
|
||||
- **`check_summarise`.** For each name in `self.compact.summarise`, `registry.get(name)` must be
|
||||
`Some`. Empty list is allowed. The error `String` contains both `summarise` and the unknown
|
||||
name. `Registry::get` is already `pub`. `http_fetch` and `clock` are in `m5c()`.
|
||||
- **`main.rs`.** Both `run_selftest` and `run_serve` load a config. After a successful `load`,
|
||||
if `check_summarise` fails, print `loopd: {e}` and return exit 1, as for a load error.
|
||||
- **`#[serde(deny_unknown_fields, default)]`** on `Heartbeat`, like `Compact`.
|
||||
- The items the edits use. This table is complete.
|
||||
|
||||
| Item | What it does |
|
||||
|---|---|
|
||||
| `Registry::get` | known tool names |
|
||||
| `Registry::m5c` | what `loopd serve` uses |
|
||||
| `Path::join` | prompt beside the config file |
|
||||
|
||||
## The edits, exactly
|
||||
|
||||
Each quoted "before" block occurs exactly once after M5d-1 (compact is already on `Config`).
|
||||
|
||||
### `crates/loopd/src/config.rs`
|
||||
|
||||
**H1.** After the `Compact` `Default` impl (the closing `}` of that impl, then the `Config`
|
||||
struct's `memory` / `compact` fields). After
|
||||
|
||||
```rust
|
||||
#[serde(default)]
|
||||
pub compact: Compact,
|
||||
}
|
||||
```
|
||||
|
||||
become
|
||||
|
||||
```rust
|
||||
#[serde(default)]
|
||||
pub compact: Compact,
|
||||
#[serde(default)]
|
||||
pub heartbeat: Heartbeat,
|
||||
}
|
||||
```
|
||||
|
||||
**H2.** After the `Compact` `Default` impl, add `Heartbeat` (before the `Config` struct if that
|
||||
is where `Compact` sits; keep `Heartbeat` next to `Compact`). The new type:
|
||||
|
||||
```rust
|
||||
/// `[heartbeat]`: how often the standing `heartbeat` session runs (M5d spec, section 4.1).
|
||||
/// `every_s` 0 disables it.
|
||||
#[derive(Debug, Clone, PartialEq, Eq, Deserialize)]
|
||||
#[serde(deny_unknown_fields, default)]
|
||||
pub struct Heartbeat {
|
||||
pub every_s: u64,
|
||||
pub prompt: std::path::PathBuf,
|
||||
}
|
||||
|
||||
impl Default for Heartbeat {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
every_s: 0,
|
||||
prompt: std::path::PathBuf::from("heartbeat.md"),
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
If `Compact` sits just above `Config`, put `Heartbeat` immediately after `Compact`'s `Default`
|
||||
impl and **then** do H1. If a "before" block is not found exactly once, stop.
|
||||
|
||||
**H3.** In `load`, the lines
|
||||
|
||||
```rust
|
||||
if let Some(parent) = path.parent() {
|
||||
config.baseline.system = parent.join(&config.baseline.system);
|
||||
}
|
||||
Ok(config)
|
||||
```
|
||||
|
||||
become
|
||||
|
||||
```rust
|
||||
if let Some(parent) = path.parent() {
|
||||
config.baseline.system = parent.join(&config.baseline.system);
|
||||
config.heartbeat.prompt = parent.join(&config.heartbeat.prompt);
|
||||
}
|
||||
Ok(config)
|
||||
```
|
||||
|
||||
**H4.** After `compact_prompt` (the lines)
|
||||
|
||||
```rust
|
||||
pub fn compact_prompt(&self) -> std::path::PathBuf {
|
||||
self.baseline.system.with_file_name("compact.md")
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
become
|
||||
|
||||
```rust
|
||||
pub fn compact_prompt(&self) -> std::path::PathBuf {
|
||||
self.baseline.system.with_file_name("compact.md")
|
||||
}
|
||||
|
||||
/// The heartbeat prompt, already joined to the config file's directory by `load`.
|
||||
pub fn heartbeat_prompt(&self) -> std::path::PathBuf {
|
||||
self.heartbeat.prompt.clone()
|
||||
}
|
||||
|
||||
/// Unknown `[compact] summarise` names are a config error (M5d spec, section 4.3).
|
||||
pub fn check_summarise(&self, registry: &crate::tools::Registry) -> Result<(), String> {
|
||||
for name in &self.compact.summarise {
|
||||
if registry.get(name).is_none() {
|
||||
return Err(format!("[compact] summarise: unknown tool {name}"));
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### `crates/loopd/src/main.rs`
|
||||
|
||||
**M1.** In `run_selftest`, after a successful `Config::load` and before `Client::new`, add:
|
||||
|
||||
```rust
|
||||
if let Err(e) = cfg.check_summarise(&loopd::tools::Registry::m5c()) {
|
||||
eprintln!("loopd: {e}");
|
||||
return ExitCode::from(1);
|
||||
}
|
||||
```
|
||||
|
||||
`Registry` is already imported in `main.rs`. Use `Registry::m5c()`.
|
||||
|
||||
**M2.** The same check in `run_serve`, after a successful `load` and before the instance lock.
|
||||
|
||||
If `main.rs` already has `use loopd::tools::{Registry, ToolPort};`, write `Registry::m5c()`.
|
||||
|
||||
## Steps
|
||||
|
||||
- [ ] **1. Copy the given files.**
|
||||
|
||||
```sh
|
||||
cp docs/plans/M5d2/files/crates/loopd/tests/config_heartbeat.rs crates/loopd/tests/
|
||||
cp docs/plans/M5d2/files/config/heartbeat.md config/heartbeat.md
|
||||
```
|
||||
|
||||
- [ ] **2. See it fail.** `cargo build --tests --keep-going -p loopd`, timeout 600000 ms: test
|
||||
target `config_heartbeat` does not compile (`E0609` no field `heartbeat`, `E0599` no method
|
||||
`heartbeat_prompt` / `check_summarise`). Every other target compiles.
|
||||
- [ ] **3. The edits.** H1 to H4, M1, M2, in order. `cargo check -p loopd` prints no error and
|
||||
no warning.
|
||||
- [ ] **4. The tests.** `cargo test -p loopd --test config_heartbeat --test config --test
|
||||
config_compact`: every test passes, 8 in `config_heartbeat`.
|
||||
- [ ] **5. Check.** `git diff --stat` lists `crates/loopd/src/config.rs` and
|
||||
`crates/loopd/src/main.rs`. `git status --short` shows the new test and `config/heartbeat.md`.
|
||||
- [ ] **6. Run the gate.** `cargo fmt --all`, then `make gate`, timeout 600000 ms. Last line
|
||||
`gate: ok`. Test count **1525** (1517 before, plus 8).
|
||||
- [ ] **7. Log and commit.** Task cell `M5d2/03-loopd-heartbeat-config`. Stage with
|
||||
`git add crates/loopd config/heartbeat.md docs/implementer-log.md Cargo.lock`, then commit
|
||||
as `AGENTS.md` says.
|
||||
|
||||
## Done when
|
||||
|
||||
- `make gate` prints `gate: ok` with 1525 tests.
|
||||
- `git diff HEAD~1 --stat` lists only `crates/loopd/src/config.rs`, `crates/loopd/src/main.rs`,
|
||||
`crates/loopd/tests/config_heartbeat.rs`, `config/heartbeat.md` and the log.
|
||||
|
||||
## Stop and report if
|
||||
|
||||
- The branch check fails; a "before" block is not found exactly once; step 2 differs; `cargo
|
||||
check` warns; a test fails; the gate count is not 1525; you join `heartbeat.prompt` onto
|
||||
`system.md`'s directory instead of the config file's parent; you reject an empty `summarise`
|
||||
list; you check names inside `parse`.
|
||||
|
||||
## What the given tests pin
|
||||
|
||||
A missing table is disabled and `heartbeat_prompt` is beside the config file; overrides replace
|
||||
defaults; unknown keys error; default `http_fetch` is known to `m5c`; an unknown name errors
|
||||
with `summarise` and that name; empty is allowed; `http_fetch` and `clock` together are known.
|
||||
@@ -0,0 +1,192 @@
|
||||
# M5d-2 task 04: five-field UTC cron matcher
|
||||
|
||||
**Branch:** `m5d2`. Check it: `git branch --show-current` prints `m5d2` and `git status --short`
|
||||
is empty; otherwise stop. Never switch branch.
|
||||
**Commit subject:** `loopd: five-field UTC cron matcher`
|
||||
**Log row:** the Task cell is `M5d2/04-loopd-cron`.
|
||||
|
||||
## Goal
|
||||
|
||||
`loopd::cron::due(schedule, Timestamp) -> Result<bool, CronError>` (spec 4.2). Five fields:
|
||||
minute hour day-of-month month day-of-week. Each field is `*`, a single number, `n-m`, or
|
||||
`*/n`. Day-of-week `0` is Sunday. No cron crate. DOM and DOW are AND (README known limits).
|
||||
|
||||
Nothing loads job files yet.
|
||||
|
||||
## Every place that changes
|
||||
|
||||
| Place | What happens there |
|
||||
|---|---|
|
||||
| `crates/loopd/src/cron.rs` | new: `due`, `CronError` |
|
||||
| `crates/loopd/src/lib.rs` | `pub mod cron;` |
|
||||
| `crates/loopd/tests/cron.rs` | new, given |
|
||||
| everything else | **no change** |
|
||||
|
||||
## Files
|
||||
|
||||
- Copy, from `docs/plans/M5d2/files/` (the same path under it): the new test
|
||||
`crates/loopd/tests/cron.rs`.
|
||||
- Create: `crates/loopd/src/cron.rs` (skeleton below).
|
||||
- Modify: `crates/loopd/src/lib.rs`, `docs/implementer-log.md`.
|
||||
- Do not touch: any other file.
|
||||
|
||||
## Facts you need
|
||||
|
||||
- **Civil fields from RFC 3339.** `Timestamp::to_rfc3339` is `YYYY-MM-DDTHH:MM:SS.mmmZ`.
|
||||
Minute = `rfc[14..16]`, hour = `rfc[11..13]`, day = `rfc[8..10]`, month = `rfc[5..7]`,
|
||||
parsed as `u32`. If a slice is missing, `CronError::Fields` naming the schedule.
|
||||
- **Weekday.** `0` is Sunday. `(unix_millis() / 1000 / 86400 + 4) % 7` as `u32` (`1970-01-01`
|
||||
was Thursday, so `+ 4`). `unix_millis` is `pub` on `Timestamp`.
|
||||
- **Ranges.** Inclusive. `*/n` matches when `value % n == 0` (multiples from 0). `n == 0` in
|
||||
a step is `Fields`.
|
||||
- **Bounds.** Minute 0–59, hour 0–23, DOM 1–31, month 1–12, DOW 0–6. A number outside its
|
||||
field's range is `Fields`. Four or six fields, `@daily`, and any other token are `Fields`.
|
||||
- **`Display`.** Names the schedule, e.g. contains `x x x x x` when that was the input. Do not
|
||||
add a runbook pointer; this error is used by job loading in task 05.
|
||||
- **Split.** `split_whitespace`. Five pieces or `Fields`.
|
||||
- rustfmt decides `mod` order in `lib.rs`. After `pub mod compact;` alphabetically `cron`
|
||||
comes after `config` and before `http`. Insert where rustfmt will put it; run `cargo fmt`
|
||||
before the gate.
|
||||
- The items the edits use. This table is complete.
|
||||
|
||||
| Item | What it does |
|
||||
|---|---|
|
||||
| `Timestamp::to_rfc3339` | civil fields |
|
||||
| `Timestamp::unix_millis` | weekday |
|
||||
| `str::parse::<u32>` | numbers |
|
||||
|
||||
## Skeleton
|
||||
|
||||
`crates/loopd/src/cron.rs`:
|
||||
|
||||
```rust
|
||||
//! Five-field UTC crontab matcher (M5d spec, section 4.2). No cron crate.
|
||||
|
||||
use bxcore::Timestamp;
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub enum CronError {
|
||||
Fields(String),
|
||||
}
|
||||
|
||||
impl std::fmt::Display for CronError {
|
||||
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||
match self {
|
||||
CronError::Fields(schedule) => {
|
||||
write!(f, "cron schedule {schedule:?} is not five UTC fields")
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl std::error::Error for CronError {}
|
||||
|
||||
/// Whether `schedule` matches `now` in UTC.
|
||||
pub fn due(schedule: &str, now: Timestamp) -> Result<bool, CronError> {
|
||||
let fields: Vec<&str> = schedule.split_whitespace().collect();
|
||||
if fields.len() != 5 {
|
||||
return Err(CronError::Fields(schedule.to_string()));
|
||||
}
|
||||
let rfc = now.to_rfc3339();
|
||||
let minute = number(&rfc, 14, 16, schedule)?;
|
||||
let hour = number(&rfc, 11, 13, schedule)?;
|
||||
let day = number(&rfc, 8, 10, schedule)?;
|
||||
let month = number(&rfc, 5, 7, schedule)?;
|
||||
let dow = {
|
||||
let days = now.unix_millis() / 1000 / 86_400;
|
||||
((days + 4) % 7) as u32
|
||||
};
|
||||
Ok(match_field(fields[0], minute, 0, 59, schedule)?
|
||||
&& match_field(fields[1], hour, 0, 23, schedule)?
|
||||
&& match_field(fields[2], day, 1, 31, schedule)?
|
||||
&& match_field(fields[3], month, 1, 12, schedule)?
|
||||
&& match_field(fields[4], dow, 0, 6, schedule)?)
|
||||
}
|
||||
|
||||
fn number(rfc: &str, start: usize, end: usize, schedule: &str) -> Result<u32, CronError> {
|
||||
let piece = rfc
|
||||
.get(start..end)
|
||||
.ok_or_else(|| CronError::Fields(schedule.to_string()))?;
|
||||
piece
|
||||
.parse()
|
||||
.map_err(|_| CronError::Fields(schedule.to_string()))
|
||||
}
|
||||
|
||||
fn match_field(
|
||||
field: &str,
|
||||
value: u32,
|
||||
min: u32,
|
||||
max: u32,
|
||||
schedule: &str,
|
||||
) -> Result<bool, CronError> {
|
||||
let err = || CronError::Fields(schedule.to_string());
|
||||
if field == "*" {
|
||||
return Ok(true);
|
||||
}
|
||||
if let Some(step) = field.strip_prefix("*/") {
|
||||
let n: u32 = step.parse().map_err(|_| err())?;
|
||||
if n == 0 {
|
||||
return Err(err());
|
||||
}
|
||||
return Ok(value.is_multiple_of(n));
|
||||
}
|
||||
if let Some((a, b)) = field.split_once('-') {
|
||||
let lo: u32 = a.parse().map_err(|_| err())?;
|
||||
let hi: u32 = b.parse().map_err(|_| err())?;
|
||||
if lo < min || hi > max || lo > hi {
|
||||
return Err(err());
|
||||
}
|
||||
return Ok(value >= lo && value <= hi);
|
||||
}
|
||||
let n: u32 = field.parse().map_err(|_| err())?;
|
||||
if n < min || n > max {
|
||||
return Err(err());
|
||||
}
|
||||
Ok(n == value)
|
||||
}
|
||||
```
|
||||
|
||||
`u32::is_multiple_of` is stable in 1.98. Do not use `%` with a comment that `n` was checked;
|
||||
the skeleton already returns `Fields` for `n == 0`.
|
||||
|
||||
If `is_multiple_of` is not in scope on `u32` in this toolchain, use `value % n == 0` after the
|
||||
`n == 0` check, and stop only if both fail to compile.
|
||||
|
||||
## Steps
|
||||
|
||||
- [ ] **1. Copy the given test.**
|
||||
|
||||
```sh
|
||||
cp docs/plans/M5d2/files/crates/loopd/tests/cron.rs crates/loopd/tests/
|
||||
```
|
||||
|
||||
- [ ] **2. See it fail.** `cargo build --tests --keep-going -p loopd`, timeout 600000 ms: test
|
||||
target `cron` does not compile (`E0432` unresolved import `loopd::cron`). Every other
|
||||
target compiles.
|
||||
- [ ] **3. The module.** Write `cron.rs` from the skeleton. Add `pub mod cron;` to `lib.rs`
|
||||
where rustfmt will put it. `cargo check -p loopd` prints no error and no warning.
|
||||
- [ ] **4. The tests.** `cargo test -p loopd --test cron`: every test passes, 10 tests.
|
||||
- [ ] **5. Check.** `git diff --stat` lists `crates/loopd/src/cron.rs` and
|
||||
`crates/loopd/src/lib.rs`.
|
||||
- [ ] **6. Run the gate.** `cargo fmt --all`, then `make gate`, timeout 600000 ms. Last line
|
||||
`gate: ok`. Test count **1535** (1525 before, plus 10).
|
||||
- [ ] **7. Log and commit.** Task cell `M5d2/04-loopd-cron`. Stage with
|
||||
`git add crates/loopd docs/implementer-log.md Cargo.lock`, then commit as `AGENTS.md` says.
|
||||
|
||||
## Done when
|
||||
|
||||
- `make gate` prints `gate: ok` with 1535 tests.
|
||||
- `git diff HEAD~1 --stat` lists only `crates/loopd/src/cron.rs`, `crates/loopd/src/lib.rs`,
|
||||
`crates/loopd/tests/cron.rs` and the log.
|
||||
|
||||
## Stop and report if
|
||||
|
||||
- The branch check fails; step 2 differs; `cargo check` warns; a test fails; the gate count is
|
||||
not 1535; you add a dependency; you treat DOM and DOW as OR; you accept `@daily` or a sixth
|
||||
field; `cron.rs` exceeds 500 lines.
|
||||
|
||||
## What the given tests pin
|
||||
|
||||
`0 7 * * *` matches that minute only; `* * * * *` matches; `*/5` is multiples from 0; `0-10`
|
||||
is inclusive; Sunday is 0 on 2026-10-04; the 1st of the month; four or six fields error;
|
||||
minute 60, hour 24, DOW 7 error; `@daily` errors; `Display` names the schedule.
|
||||
@@ -0,0 +1,219 @@
|
||||
# M5d-2 task 05: job files and the run ledger
|
||||
|
||||
**Branch:** `m5d2`. Check it: `git branch --show-current` prints `m5d2` and `git status --short`
|
||||
is empty; otherwise stop. Never switch branch.
|
||||
**Commit subject:** `loopd: job files, run ledger, jobs-invalid`
|
||||
**Log row:** the Task cell is `M5d2/05-loopd-jobs`.
|
||||
|
||||
## Goal
|
||||
|
||||
Owner-written `{home}/jobs/<name>.toml` and a run ledger `{home}/jobs/runs.jsonl` (spec 4.2).
|
||||
`loopd` reads job files and cannot write them. Unknown keys, a name that is not `[a-z0-9-]+`,
|
||||
or a name so long that `job-<name>-<yyyymmdd>t<hhmm>` is not a `SessionId`, are
|
||||
`jobs-invalid`. A missing jobs directory is an empty list.
|
||||
|
||||
Nothing ticks yet. `loopd serve` refuses to start when `load` errors, so a misspelt job cannot
|
||||
sit silently.
|
||||
|
||||
## Every place that changes
|
||||
|
||||
| Place | What happens there |
|
||||
|---|---|
|
||||
| `crates/loopd/src/jobs.rs` | new: `Job`, `Run`, `RunStatus`, `load`, `session_id`, `append`, `close_leftovers`, `live_session` |
|
||||
| `crates/loopd/src/lib.rs` | `pub mod jobs;` |
|
||||
| `crates/loopd/src/main.rs` | `run_serve` calls `jobs::load` after `check_summarise` |
|
||||
| `docs/runbook.md` | append the given `jobs-invalid` entry |
|
||||
| `crates/loopd/tests/jobs.rs` | new, given |
|
||||
| everything else | **no change** |
|
||||
|
||||
## Files
|
||||
|
||||
- Copy, from `docs/plans/M5d2/files/` (the same path under it): the new test
|
||||
`crates/loopd/tests/jobs.rs`.
|
||||
- Append, from `docs/plans/M5d2/files/runbook-jobs-invalid.md`, to `docs/runbook.md`.
|
||||
- Create: `crates/loopd/src/jobs.rs` (skeleton below).
|
||||
- Modify: `crates/loopd/src/lib.rs`, `crates/loopd/src/main.rs`, `docs/implementer-log.md`.
|
||||
- Do not touch: any other file.
|
||||
|
||||
## Facts you need
|
||||
|
||||
- **`name` from the file stem.** `mail.toml` → `mail`. The stem must match `^[a-z0-9-]+$`
|
||||
(bytes, nonempty) and `session_id(name, now)` must succeed. `Mail.toml` is `JobsError::Name`.
|
||||
- **`session_id`.** `format!("job-{name}-{}t{}", ymd, hm)` with `ymd` = `rfc[0..4]+rfc[5..7]+rfc[8..10]`
|
||||
and `hm` = `rfc[11..13]+rfc[14..16]`, all from `now.to_rfc3339()`, then
|
||||
`SessionId::new`. `SessionId` is `[a-z0-9-]`, max 64. `"n".repeat(50)` is too long (`job-` +
|
||||
50 + `-` + 8 + `t` + 4 = 68).
|
||||
- **`load`.** `read_dir`. `ErrorKind::NotFound` → `Ok(vec![])`. Any other IO error is
|
||||
`JobsError::Invalid` naming the directory. Skip names that do not end in `.toml`. Sort by
|
||||
`name` so the order is stable. Parse with `#[serde(deny_unknown_fields)]` on a private
|
||||
`File { schedule, prompt }`.
|
||||
- **`RunStatus`.** `started` / `finished` / `failed` (`rename_all = "snake_case"`).
|
||||
- **`append`.** Create the parent directory (0700 is fine; do not weaken a tighter mode). One
|
||||
JSON object, then a newline. Open append, write, `sync_all`.
|
||||
- **`live_session`.** Walk every line of the ledger (missing file → `Ok(None)`). The last
|
||||
`Run` whose `name` matches: `Started` → `Some(session)`, anything else → `None`.
|
||||
- **`close_leftovers`.** Names whose last run is `Started` get one more `Run` with
|
||||
`status: Failed` and the same `session`, `time: now`. Return how many names were closed.
|
||||
A missing file is `Ok(0)`.
|
||||
- **`Display` for `Invalid`.** `"{path}: {detail}; see docs/runbook.md#jobs-invalid"`. The
|
||||
given unknown-keys test checks `ends_with("see docs/runbook.md#jobs-invalid")` and that
|
||||
the path's file name is in the string.
|
||||
- **`run_serve`.** After `check_summarise`, `jobs::load(&cfg.paths.home.join("jobs"))`. On
|
||||
`Err`, print `loopd: {e}` and return exit 1. Do not create the directory here.
|
||||
- A file that exists but cannot be read is an error. Only a missing directory is empty.
|
||||
- The items the edits use. This table is complete.
|
||||
|
||||
| Item | What it does |
|
||||
|---|---|
|
||||
| `toml::from_str` | already a `loopd` dependency |
|
||||
| `SessionId::new` | the 64-byte alphabet check |
|
||||
| `serde` `deny_unknown_fields` | unknown keys |
|
||||
|
||||
## Skeleton
|
||||
|
||||
`crates/loopd/src/jobs.rs` — fill every function; no `todo!()`. Keep the types in this order.
|
||||
|
||||
```rust
|
||||
//! Owner-written job files and the run ledger (M5d spec, section 4.2).
|
||||
|
||||
use std::path::{Path, PathBuf};
|
||||
|
||||
use bxcore::{SessionId, Timestamp};
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub struct Job {
|
||||
pub name: String,
|
||||
pub schedule: String,
|
||||
pub prompt: String,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
|
||||
#[serde(rename_all = "snake_case")]
|
||||
pub enum RunStatus {
|
||||
Started,
|
||||
Finished,
|
||||
Failed,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
|
||||
#[serde(deny_unknown_fields)]
|
||||
pub struct Run {
|
||||
pub time: Timestamp,
|
||||
pub name: String,
|
||||
pub session: SessionId,
|
||||
pub status: RunStatus,
|
||||
}
|
||||
|
||||
#[derive(Debug)]
|
||||
pub enum JobsError {
|
||||
Name(String),
|
||||
Invalid { path: PathBuf, detail: String },
|
||||
}
|
||||
|
||||
impl std::fmt::Display for JobsError {
|
||||
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||
match self {
|
||||
JobsError::Name(name) => write!(
|
||||
f,
|
||||
"job name {name:?} is not a valid job; see docs/runbook.md#jobs-invalid"
|
||||
),
|
||||
JobsError::Invalid { path, detail } => write!(
|
||||
f,
|
||||
"{}: {detail}; see docs/runbook.md#jobs-invalid",
|
||||
path.display()
|
||||
),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl std::error::Error for JobsError {}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
#[serde(deny_unknown_fields)]
|
||||
struct File {
|
||||
schedule: String,
|
||||
prompt: String,
|
||||
}
|
||||
|
||||
pub fn session_id(name: &str, now: Timestamp) -> Result<SessionId, JobsError> {
|
||||
// Build job-<name>-<yyyymmdd>t<hhmm> from now.to_rfc3339() as in Facts. SessionId::new
|
||||
// maps a failure to JobsError::Name(name.to_string()).
|
||||
let _ = (name, now);
|
||||
Err(JobsError::Name(name.to_string()))
|
||||
}
|
||||
|
||||
pub fn load(dir: &Path) -> Result<Vec<Job>, JobsError> {
|
||||
let _ = dir;
|
||||
Ok(Vec::new())
|
||||
}
|
||||
|
||||
pub fn append(path: &Path, run: &Run) -> Result<(), JobsError> {
|
||||
let _ = (path, run);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub fn live_session(path: &Path, name: &str) -> Result<Option<SessionId>, JobsError> {
|
||||
let _ = (path, name);
|
||||
Ok(None)
|
||||
}
|
||||
|
||||
pub fn close_leftovers(path: &Path, now: Timestamp) -> Result<usize, JobsError> {
|
||||
let _ = (path, now);
|
||||
Ok(0)
|
||||
}
|
||||
```
|
||||
|
||||
Replace the four stub bodies in the same task. Do not leave a stub that returns `Ok(None)` /
|
||||
`Ok(0)` / `Ok(vec![])` once you have written the real body. `session_id` must not stay on the
|
||||
`Err` path.
|
||||
|
||||
## Steps
|
||||
|
||||
- [ ] **1. Copy the given test.**
|
||||
|
||||
```sh
|
||||
cp docs/plans/M5d2/files/crates/loopd/tests/jobs.rs crates/loopd/tests/
|
||||
```
|
||||
|
||||
- [ ] **2. See it fail.** `cargo build --tests --keep-going -p loopd`, timeout 600000 ms: test
|
||||
target `jobs` does not compile (`E0432` unresolved `loopd::jobs`). Every other target
|
||||
compiles.
|
||||
- [ ] **3. The module.** Write `jobs.rs` from the skeleton, then fill the four bodies one at a
|
||||
time (`session_id`, `load`, `append`, `live_session` / `close_leftovers`). Add `pub mod jobs;`
|
||||
to `lib.rs`. Wire `run_serve` as in Facts. `cargo check -p loopd` prints no error and no
|
||||
warning. `wc -l crates/loopd/src/jobs.rs` prints a number ≤ 500.
|
||||
- [ ] **4. The tests.** `cargo test -p loopd --test jobs`: every test passes, 8 tests.
|
||||
- [ ] **5. The runbook.** From the repository root:
|
||||
|
||||
```sh
|
||||
{ echo; cat docs/plans/M5d2/files/runbook-jobs-invalid.md; } >> docs/runbook.md
|
||||
```
|
||||
|
||||
Then `grep -c '^## jobs-invalid$' docs/runbook.md` prints `1`.
|
||||
- [ ] **6. Check.** `git diff --stat` lists `crates/loopd/src/jobs.rs`,
|
||||
`crates/loopd/src/lib.rs`, `crates/loopd/src/main.rs`, `docs/runbook.md`.
|
||||
- [ ] **7. Run the gate.** `cargo fmt --all`, then `make gate`, timeout 600000 ms. Last line
|
||||
`gate: ok`. Test count **1543** (1535 before, plus 8).
|
||||
- [ ] **8. Log and commit.** Task cell `M5d2/05-loopd-jobs`. Stage with
|
||||
`git add crates/loopd docs/runbook.md docs/implementer-log.md Cargo.lock`, then commit as
|
||||
`AGENTS.md` says.
|
||||
|
||||
## Done when
|
||||
|
||||
- `make gate` prints `gate: ok` with 1543 tests.
|
||||
- `git diff HEAD~1 --stat` lists only `crates/loopd/src/jobs.rs`, `crates/loopd/src/lib.rs`,
|
||||
`crates/loopd/src/main.rs`, `crates/loopd/tests/jobs.rs`, `docs/runbook.md` and the log.
|
||||
|
||||
## Stop and report if
|
||||
|
||||
- The branch check fails; step 2 differs; `cargo check` warns; a test fails; the gate count is
|
||||
not 1543; you add a cron crate; a missing jobs directory is an error; `Display` for `Invalid`
|
||||
does not end with the runbook pointer; `jobs.rs` exceeds 500 lines; you write a job file.
|
||||
|
||||
## What the given tests pin
|
||||
|
||||
A valid `mail.toml` loads; unknown keys name the file and the runbook; `Mail.toml` is `Name`;
|
||||
`session_id` is `job-mail-20261003t0700`; a 50-character name is `Name`; a missing directory is
|
||||
empty; leftover `started` becomes `failed` and is no longer live; a `started` without a close
|
||||
is live until `finished`.
|
||||
@@ -0,0 +1,216 @@
|
||||
# M5d-2 task 06: heartbeat tick
|
||||
|
||||
**Branch:** `m5d2`. Check it: `git branch --show-current` prints `m5d2` and `git status --short`
|
||||
is empty; otherwise stop. Never switch branch.
|
||||
**Commit subject:** `loopd: heartbeat tick on the background slot`
|
||||
**Log row:** the Task cell is `M5d2/06-loopd-scheduler-heartbeat`.
|
||||
|
||||
## Goal
|
||||
|
||||
A `Scheduler` whose `tick(&Context, now)` runs the standing `heartbeat` session when
|
||||
`every_s` has elapsed since the last heartbeat turn, or since `Scheduler::new(started)` if
|
||||
none (spec 4.1). The user text is `It is <RFC 3339 UTC>.\n\n` plus the prompt file. Slot is
|
||||
background. Speaker is `None`. Skip if the session is busy. Skip and log once if the prompt
|
||||
cannot be read (`heartbeat-prompt-unreadable`).
|
||||
|
||||
Tests call `tick(ctx, now)` with injected time. No sleeps (T27).
|
||||
|
||||
## Every place that changes
|
||||
|
||||
| Place | What happens there |
|
||||
|---|---|
|
||||
| `crates/loopd/src/channel.rs` | `Held` is `pub`; `Context::hold` |
|
||||
| `crates/loopd/src/scheduler.rs` | new: `Scheduler`, `TickWork`, `tick` |
|
||||
| `crates/loopd/src/lib.rs` | `pub mod scheduler;` |
|
||||
| `crates/loopd/src/main.rs` | a thread that `tick`s once a second, before `channel::serve` |
|
||||
| `docs/runbook.md` | append the given `heartbeat-prompt-unreadable` entry |
|
||||
| `crates/loopd/tests/scheduler_heartbeat.rs` | new, given |
|
||||
| everything else | **no change**: not cron job runs (task 07) |
|
||||
|
||||
## Files
|
||||
|
||||
- Copy, from `docs/plans/M5d2/files/` (the same path under it): the new test
|
||||
`crates/loopd/tests/scheduler_heartbeat.rs`.
|
||||
- Append, from `docs/plans/M5d2/files/runbook-heartbeat-prompt-unreadable.md`, to
|
||||
`docs/runbook.md`.
|
||||
- Create: `crates/loopd/src/scheduler.rs` (skeleton below).
|
||||
- Modify: `crates/loopd/src/channel.rs`, `crates/loopd/src/lib.rs`,
|
||||
`crates/loopd/src/main.rs`, `docs/implementer-log.md`.
|
||||
- Do not touch: any other file.
|
||||
|
||||
## Facts you need
|
||||
|
||||
- **`Context::hold`.** Same insert as `handle`: if the id is already in `busy`, `Err(())`;
|
||||
else insert and return `Held`. `Held` must be `pub` so the given test can keep it. `Drop`
|
||||
already removes the id.
|
||||
- **Due.** `every_s == 0` → no heartbeat work. Else `now.unix_millis().saturating_sub(last.unix_millis()) >= every_s.saturating_mul(1000)`
|
||||
where `last` is `last_heartbeat.unwrap_or(started)`. `07:00` to `07:01` with `every_s = 60`
|
||||
is due. `07:01` then `07:01:30` is not.
|
||||
- **Prompt first, then hold.** If `read_to_string(cfg.heartbeat_prompt())` fails, do not hold,
|
||||
push `HeartbeatUnreadable`, log
|
||||
`loopd: {path}: {e}; see docs/runbook.md#heartbeat-prompt-unreadable` only while
|
||||
`heartbeat_unreadable_logged` is false, then set that flag. Do not update `last_heartbeat`.
|
||||
A directory at the prompt path is the given-test stand-in for an unreadable file (T27: no
|
||||
`chmod 000`).
|
||||
- **Session id `heartbeat`.** `Session::open`; `NotFound` → `Session::create` with
|
||||
`Baseline::assemble(&ctx.cfg, &ctx.registry)` and `ctx.cfg.slots.background`. Other open
|
||||
errors: log and skip (do not panic).
|
||||
- **The turn.** User text `format!("It is {}.\n\n{file}", now.to_rfc3339())` where `file` is
|
||||
the prompt contents as read (no extra trim beyond what the file has). `run_turn` then
|
||||
`compact::finish_turn` with `&*ctx.memory`, speaker `None`, `on_event` a no-op. Then
|
||||
`last_heartbeat = Some(now)`, push `Heartbeat`.
|
||||
- **`TickWork`.** Include `Job` / `SkippedJob` now so task 07 does not change the type.
|
||||
Task 06 never pushes them.
|
||||
- **`close_leftovers`.** On the first `tick` only, call
|
||||
`jobs::close_leftovers(&ctx.cfg.paths.home.join("jobs").join("runs.jsonl"), now)` and ignore
|
||||
a missing-file `Ok(0)`. A leftover that is currently `hold`ing its session is **not** closed:
|
||||
skip names whose `live_session` session is in `busy` (or: close only when `hold` of that
|
||||
session would succeed, then immediately `drop` that hold). Task 07's live-skip test depends
|
||||
on this. If `close_leftovers` has no busy-set filter, add a loop here: for each live name,
|
||||
if `hold(session)` is `Ok`, drop it and treat as leftover (append Failed yourself or call a
|
||||
helper). Simplest: **do not call `close_leftovers` blindly**. Walk: if `live_session` is
|
||||
`Some(id)` and `hold(id)` is `Ok`, drop the hold and append a `Failed` for that name (or
|
||||
call `close_leftovers` only when no job session is held). The given leftover test does not
|
||||
hold; the live-skip test does. Closing only leftovers that are not busy is required.
|
||||
- **Main thread.** After `Context::new` and before `channel::serve`:
|
||||
|
||||
```rust
|
||||
let ctx_sched = Arc::clone(&ctx);
|
||||
std::thread::spawn(move || {
|
||||
let mut sched = loopd::scheduler::Scheduler::new(bxcore::Timestamp::now());
|
||||
loop {
|
||||
sched.tick(&ctx_sched, bxcore::Timestamp::now());
|
||||
std::thread::sleep(std::time::Duration::from_secs(1));
|
||||
}
|
||||
});
|
||||
```
|
||||
|
||||
That sleep is production only. Tests never sleep.
|
||||
- **`channel.rs` stays ≤ 500 lines.** `hold` is about 15 lines.
|
||||
- The items the edits use. This table is complete.
|
||||
|
||||
| Item | What it does |
|
||||
|---|---|
|
||||
| `Session::slot` / `create` / `open` | standing session |
|
||||
| `run_turn` / `finish_turn` | the pulse |
|
||||
| `Baseline::assemble` | new session |
|
||||
| `jobs::close_leftovers` / `live_session` / `append` | leftovers |
|
||||
| `Timestamp::unix_millis` / `to_rfc3339` | due and the time line |
|
||||
|
||||
## The edits, exactly
|
||||
|
||||
### `crates/loopd/src/channel.rs`
|
||||
|
||||
**C1.** `struct Held` becomes `pub struct Held`.
|
||||
|
||||
**C2.** Inside `impl Context`, after `new`, add:
|
||||
|
||||
```rust
|
||||
/// Mark `id` busy. `Err(())` if it already is. Tests and the scheduler use this so a skip
|
||||
/// is not a sleep (T27).
|
||||
pub fn hold(&self, id: SessionId) -> Result<Held<'_>, ()> {
|
||||
let mut busy = self.busy.lock().unwrap_or_else(|p| p.into_inner());
|
||||
if busy.contains(&id) {
|
||||
return Err(());
|
||||
}
|
||||
busy.insert(id.clone());
|
||||
drop(busy);
|
||||
Ok(Held { ctx: self, id })
|
||||
}
|
||||
```
|
||||
|
||||
### `crates/loopd/src/scheduler.rs`
|
||||
|
||||
Write this module. Fill `tick`. Cron jobs stay a comment until task 07: after heartbeat, do
|
||||
not iterate job files yet.
|
||||
|
||||
```rust
|
||||
//! Isolated heartbeat (and, from task 07, cron) turns (M5d spec, section 4).
|
||||
|
||||
use crate::channel::Context;
|
||||
use bxcore::{SessionId, Timestamp};
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub enum TickWork {
|
||||
Heartbeat,
|
||||
SkippedBusy { session: SessionId },
|
||||
HeartbeatUnreadable,
|
||||
Job { name: String, session: SessionId },
|
||||
SkippedJob { name: String },
|
||||
}
|
||||
|
||||
pub struct Scheduler {
|
||||
started: Timestamp,
|
||||
last_heartbeat: Option<Timestamp>,
|
||||
leftovers_closed: bool,
|
||||
heartbeat_unreadable_logged: bool,
|
||||
}
|
||||
|
||||
impl Scheduler {
|
||||
pub fn new(started: Timestamp) -> Scheduler {
|
||||
Scheduler {
|
||||
started,
|
||||
last_heartbeat: None,
|
||||
leftovers_closed: false,
|
||||
heartbeat_unreadable_logged: false,
|
||||
}
|
||||
}
|
||||
|
||||
pub fn tick(&mut self, ctx: &Context, now: Timestamp) -> Vec<TickWork> {
|
||||
let _ = (ctx, now);
|
||||
Vec::new()
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Replace the `tick` body in this task. `leftovers_closed` is set true after the first leftover
|
||||
pass so it runs once.
|
||||
|
||||
## Steps
|
||||
|
||||
- [ ] **1. Copy the given test.**
|
||||
|
||||
```sh
|
||||
cp docs/plans/M5d2/files/crates/loopd/tests/scheduler_heartbeat.rs crates/loopd/tests/
|
||||
```
|
||||
|
||||
- [ ] **2. See it fail.** `cargo build --tests --keep-going -p loopd`, timeout 600000 ms: test
|
||||
target `scheduler_heartbeat` does not compile (`E0432` unresolved `loopd::scheduler`,
|
||||
`E0599` no method `hold`). Every other target compiles.
|
||||
- [ ] **3. The edits.** C1, C2, `scheduler.rs`, `pub mod scheduler;`, main thread. Fill
|
||||
`tick`. `cargo check -p loopd` prints no error and no warning.
|
||||
`wc -l crates/loopd/src/scheduler.rs` and `wc -l crates/loopd/src/channel.rs` each ≤ 500.
|
||||
- [ ] **4. The tests.** `cargo test -p loopd --test scheduler_heartbeat --test channel --test
|
||||
session_slot`: every test passes, 6 in `scheduler_heartbeat`.
|
||||
- [ ] **5. The runbook.**
|
||||
|
||||
```sh
|
||||
{ echo; cat docs/plans/M5d2/files/runbook-heartbeat-prompt-unreadable.md; } >> docs/runbook.md
|
||||
```
|
||||
|
||||
`grep -c '^## heartbeat-prompt-unreadable$' docs/runbook.md` prints `1`.
|
||||
- [ ] **6. Run the gate.** `cargo fmt --all`, then `make gate`, timeout 600000 ms. Last line
|
||||
`gate: ok`. Test count **1549** (1543 before, plus 6).
|
||||
- [ ] **7. Log and commit.** Task cell `M5d2/06-loopd-scheduler-heartbeat`. Stage with
|
||||
`git add crates/loopd docs/runbook.md docs/implementer-log.md Cargo.lock`, then commit as
|
||||
`AGENTS.md` says.
|
||||
|
||||
## Done when
|
||||
|
||||
- `make gate` prints `gate: ok` with 1549 tests.
|
||||
- `git diff HEAD~1 --stat` lists only `crates/loopd/src/channel.rs`,
|
||||
`crates/loopd/src/scheduler.rs`, `crates/loopd/src/lib.rs`, `crates/loopd/src/main.rs`,
|
||||
`crates/loopd/tests/scheduler_heartbeat.rs`, `docs/runbook.md` and the log.
|
||||
|
||||
## Stop and report if
|
||||
|
||||
- The branch check fails; a "before" block is not found exactly once; step 2 differs; `cargo
|
||||
check` warns; a test fails; the gate count is not 1549; you sleep in a test; you post to
|
||||
Mattermost; you send `slots.main`; you set a speaker; you `chmod 000`; `Held` stays private.
|
||||
|
||||
## What the given tests pin
|
||||
|
||||
`every_s` 0 does nothing; a due heartbeat sends `id_slot` 1 with a time line and the file
|
||||
text, session `heartbeat` has `slot() == 1`; a second tick before `every_s` does not run; a
|
||||
held session is `SkippedBusy` with no request; a directory at the prompt path is
|
||||
`HeartbeatUnreadable` with no request; the user record has `speaker: None`.
|
||||
@@ -0,0 +1,107 @@
|
||||
# M5d-2 task 07: cron tick
|
||||
|
||||
**Branch:** `m5d2`. Check it: `git branch --show-current` prints `m5d2` and `git status --short`
|
||||
is empty; otherwise stop. Never switch branch.
|
||||
**Commit subject:** `loopd: cron tick, new session per run`
|
||||
**Log row:** the Task cell is `M5d2/07-loopd-scheduler-cron`.
|
||||
|
||||
## Goal
|
||||
|
||||
After the heartbeat work, `tick` loads `{home}/jobs/*.toml` and runs each job whose schedule
|
||||
matches `now` (spec 4.2). A new session `job-<name>-<yyyymmdd>t<hhmm>` on the background slot.
|
||||
Skip the tick when a previous run of that name is still live. Leftover `started` from a dead
|
||||
process becomes `failed` on the first tick (already started in task 06: only when that session
|
||||
is not busy). Speaker `None`. Same time-line prefix as heartbeat.
|
||||
|
||||
## Every place that changes
|
||||
|
||||
| Place | What happens there |
|
||||
|---|---|
|
||||
| `crates/loopd/src/scheduler.rs` | after heartbeat, iterate jobs (edit T1) |
|
||||
| `crates/loopd/tests/scheduler_cron.rs` | new, given |
|
||||
| everything else | **no change**: not `TickWork`, not `hold`, not heartbeat |
|
||||
|
||||
## Files
|
||||
|
||||
- Copy, from `docs/plans/M5d2/files/` (the same path under it): the new test
|
||||
`crates/loopd/tests/scheduler_cron.rs`.
|
||||
- Modify: `crates/loopd/src/scheduler.rs`, `docs/implementer-log.md`.
|
||||
- Do not touch: any other file.
|
||||
|
||||
## Facts you need
|
||||
|
||||
- **Load.** `jobs::load(&ctx.cfg.paths.home.join("jobs"))`. On `Err`, log `loopd: {e}` and
|
||||
skip cron this tick (the process stays up: README known limits). On `Ok`, for each `Job` in
|
||||
order:
|
||||
1. `cron::due(&job.schedule, now)`. `Err` → log and skip that job. `Ok(false)` → skip.
|
||||
2. Ledger path `{home}/jobs/runs.jsonl`. If `live_session` is `Some(_)`, push
|
||||
`SkippedJob { name }` and continue. (Task 06 has already closed leftovers that were not
|
||||
busy.)
|
||||
3. `session_id(&job.name, now)`. `Err` → log and skip.
|
||||
4. `ctx.hold(session.clone())`. `Err` → `SkippedJob`. `Ok(held)` → create the session
|
||||
(`Session::create` with `assemble` and `slots.background`; if `Exists`, `open` instead),
|
||||
`jobs::append` `Started`, run `run_turn` then `finish_turn` with user text
|
||||
`format!("It is {}.\n\n{}", now.to_rfc3339(), job.prompt)`, speaker `None`, then
|
||||
`append` `Finished` or `Failed` (turn `Err` → `Failed`). `drop(held)`. Push
|
||||
`TickWork::Job { name, session }`.
|
||||
- **Two due minutes.** Two ticks at 06:59 and 07:00 with `* * * * *` make two session ids.
|
||||
- **Not due.** `0 7 * * *` at 06:59 pushes nothing.
|
||||
- **`scheduler.rs` stays ≤ 500 lines.** After this task it is about 200.
|
||||
- The items the edits use. This table is complete.
|
||||
|
||||
| Item | What it does |
|
||||
|---|---|
|
||||
| `jobs::load` / `session_id` / `append` / `live_session` | task 05 |
|
||||
| `cron::due` | task 04 |
|
||||
| `Context::hold` | task 06 |
|
||||
| `run_turn` / `finish_turn` | same as heartbeat |
|
||||
|
||||
## The edits, exactly
|
||||
|
||||
There is no quoted "before" for a unique heartbeat-only block: you add a loop after the
|
||||
heartbeat section of `tick`, before `work` is returned. If `tick` currently ends with
|
||||
`work` after heartbeat, insert the job loop there. If you cannot find a single place to
|
||||
insert without touching heartbeat, stop.
|
||||
|
||||
Do not change heartbeat behaviour. Do not add a sleep.
|
||||
|
||||
## Steps
|
||||
|
||||
- [ ] **1. Copy the given test.**
|
||||
|
||||
```sh
|
||||
cp docs/plans/M5d2/files/crates/loopd/tests/scheduler_cron.rs crates/loopd/tests/
|
||||
```
|
||||
|
||||
- [ ] **2. See it fail.** `cargo test -p loopd --test scheduler_cron`, timeout 600000 ms: the
|
||||
tests compile (`TickWork::Job` already exists) and `a_due_job_starts_a_new_session_on_the_background_slot`
|
||||
fails: `work` is empty, not a `Job`. `a_job_not_due_does_not_run` may pass.
|
||||
- [ ] **3. The edit.** Add the job loop. `cargo check -p loopd` prints no error and no
|
||||
warning. `wc -l crates/loopd/src/scheduler.rs` prints a number ≤ 500.
|
||||
- [ ] **4. The tests.** `cargo test -p loopd --test scheduler_cron --test scheduler_heartbeat
|
||||
--test jobs --test cron`: every test passes, 6 in `scheduler_cron`.
|
||||
- [ ] **5. Check.** `git diff --stat` lists exactly `crates/loopd/src/scheduler.rs`.
|
||||
- [ ] **6. Run the gate.** `cargo fmt --all`, then `make gate`, timeout 600000 ms. Last line
|
||||
`gate: ok`. Test count **1555** (1549 before, plus 6).
|
||||
- [ ] **7. Log and commit.** Task cell `M5d2/07-loopd-scheduler-cron`. Stage with
|
||||
`git add crates/loopd docs/implementer-log.md Cargo.lock`, then commit as `AGENTS.md` says.
|
||||
|
||||
## Done when
|
||||
|
||||
- `make gate` prints `gate: ok` with 1555 tests.
|
||||
- `git diff HEAD~1 --stat` lists only `crates/loopd/src/scheduler.rs`,
|
||||
`crates/loopd/tests/scheduler_cron.rs` and the log.
|
||||
|
||||
## Stop and report if
|
||||
|
||||
- The branch check fails; step 2 differs (tests do not compile, or a different assertion
|
||||
fails); `cargo check` warns; a test fails; the gate count is not 1555; you reuse a session
|
||||
across runs; you send `slots.main`; you set a speaker; you skip leftovers that are not busy;
|
||||
a heartbeat test fails.
|
||||
|
||||
## What the given tests pin
|
||||
|
||||
A due job creates `job-mail-…` on slot 1 with the time line and the prompt; a job not due
|
||||
does nothing; a live previous run (Started + `hold`) is `SkippedJob` with no request; leftover
|
||||
`started` is `failed` on the first tick (the due job may then run); speaker is `None`; two due
|
||||
minutes make two sessions.
|
||||
@@ -0,0 +1,257 @@
|
||||
# M5d-2 task 08: opted-in tool-output summaries
|
||||
|
||||
**Branch:** `m5d2`. Check it: `git branch --show-current` prints `m5d2` and `git status --short`
|
||||
is empty; otherwise stop. Never switch branch.
|
||||
**Commit subject:** `loopd: background ToolSummary for opted-in results`
|
||||
**Log row:** the Task cell is `M5d2/08-loopd-tool-summary`.
|
||||
|
||||
## Goal
|
||||
|
||||
After a turn that logged an opted-in `ToolResult` with no matching `ToolSummary`, run a
|
||||
no-tools completion on the background slot and append `ToolSummary` with the result's labels
|
||||
(spec 4.3). `messages()` already ignores it (task 02). `http_fetch` opts in via
|
||||
`[compact] summarise`.
|
||||
|
||||
`summarise::missing` is the function. `finish_turn` calls it on `Ok` and on `SessionFull`,
|
||||
**before** compaction, so the current epoch still holds the `ToolResult` (README known
|
||||
limits: v1 runs this inside the turn's hold).
|
||||
|
||||
The starter `config/summarise.md` is copied like `config/heartbeat.md`.
|
||||
|
||||
## Every place that changes
|
||||
|
||||
| Place | What happens there |
|
||||
|---|---|
|
||||
| `crates/loopd/src/summarise.rs` | new: `missing` |
|
||||
| `crates/loopd/src/lib.rs` | `pub mod summarise;` |
|
||||
| `crates/loopd/src/compact.rs` | `finish_turn` calls `missing` before compact (edit F1) |
|
||||
| `config/summarise.md` | new, given starter |
|
||||
| `crates/loopd/tests/tool_summary_call.rs` | new, given |
|
||||
| everything else | **no change** |
|
||||
|
||||
## Files
|
||||
|
||||
- Copy, from `docs/plans/M5d2/files/` (the same path under it): the new test
|
||||
`crates/loopd/tests/tool_summary_call.rs` and the starter `config/summarise.md`.
|
||||
- Create: `crates/loopd/src/summarise.rs` (function below).
|
||||
- Modify: `crates/loopd/src/lib.rs`, `crates/loopd/src/compact.rs`,
|
||||
`docs/implementer-log.md`.
|
||||
- Do not touch: any other file.
|
||||
|
||||
## Facts you need
|
||||
|
||||
- **Which results.** Walk `session.records()` once. From each `Assistant`, remember
|
||||
`tool_calls` by `id → name`. From each `ToolSummary`, remember `call: CallId`. Then for
|
||||
each `ToolResult` whose `tool_call_id` maps to a name in `cfg.compact.summarise` and whose
|
||||
`call` is not already summarised, queue it. Do not append while iterating `records()`.
|
||||
- **The call.** `messages_with_carry(session.baseline(), session.records(), session.carry())`,
|
||||
then one `ChatMessage::User` whose content is
|
||||
`format!("{}\n\n{}", prompt.trim_end(), crate::compact::cap_chars(result_content, cfg.r#loop.tool_result_cap))`
|
||||
where `prompt` is `read_to_string` of
|
||||
`cfg.baseline.system.with_file_name("summarise.md")`. Missing file: return `Err` naming
|
||||
the path (do not guess). `ChatRequest { slot: cfg.slots.background, tools: Vec::new(),
|
||||
thinking: true, messages }`. `client.chat_with_retry(&request, &mut |_| {})`.
|
||||
- **The record.** Completion `content.unwrap_or_default()`, `neutralise`, `cap_chars` to
|
||||
`cfg.compact.summary_chars`. Append `LogRecord::ToolSummary { time: Timestamp::now(),
|
||||
call, tool_call_id, content, class, untrusted }` copying `call`, `tool_call_id`, `class`,
|
||||
`untrusted` from that `ToolResult`.
|
||||
- **Context-full / other infer errors.** Skip that result (log `loopd: tool summary: {e}`),
|
||||
continue the queue. Do not fail the turn.
|
||||
- **`finish_turn`.** At the start of the `Ok(done)` arm and the `Err(SessionFull)` arm, before
|
||||
`compact_session`:
|
||||
|
||||
```rust
|
||||
if let Err(e) = crate::summarise::missing(rt.cfg, rt.client, session) {
|
||||
eprintln!("loopd: tool summary: {e}");
|
||||
}
|
||||
```
|
||||
|
||||
A compact-due turn summarises first, then compacts. The last given test goes through
|
||||
`finish_turn` (not `missing` alone) and expects a `ToolSummary`.
|
||||
- **`cap_chars` is `pub(crate)`** in `compact.rs` (M5d-1 task 05).
|
||||
- **`tool_call.http` calls `read_file`.** The last given test sets
|
||||
`summarise = ["read_file"]`. Default `["http_fetch"]` would leave it alone.
|
||||
- rustfmt `mod` order: `summarise` after `sse` / `session` as rustfmt decides.
|
||||
- The items the edits use. This table is complete.
|
||||
|
||||
| Item | What it does |
|
||||
|---|---|
|
||||
| `messages_with_carry` | prefix plus carry |
|
||||
| `ChatRequest.tools: Vec::new()` | omits `tools` on the wire |
|
||||
| `neutralise` | `bxcore::sanitize` |
|
||||
| `cap_chars` | character cap |
|
||||
| `Session::append` | the new record |
|
||||
|
||||
## The function
|
||||
|
||||
`crates/loopd/src/summarise.rs`:
|
||||
|
||||
```rust
|
||||
//! Opted-in tool-output summaries (M5d spec, section 4.3).
|
||||
|
||||
use crate::baseline::messages_with_carry;
|
||||
use crate::compact::cap_chars;
|
||||
use crate::config::Config;
|
||||
use crate::llama::{ChatMessage, ChatRequest, Client};
|
||||
use crate::session::Session;
|
||||
use bxcore::sanitize::neutralise;
|
||||
use bxcore::{CallId, DataClass, LogRecord, Timestamp, ToolCall};
|
||||
use std::collections::{HashMap, HashSet};
|
||||
|
||||
/// Append a `ToolSummary` for each opted-in `ToolResult` in this epoch that does not yet have
|
||||
/// one. Inference uses the background slot and no tools.
|
||||
pub fn missing(
|
||||
cfg: &Config,
|
||||
client: &Client,
|
||||
session: &mut Session,
|
||||
) -> Result<(), String> {
|
||||
let path = cfg.baseline.system.with_file_name("summarise.md");
|
||||
let prompt = std::fs::read_to_string(&path)
|
||||
.map_err(|e| format!("{}: {e}", path.display()))?;
|
||||
let mut names: HashMap<String, String> = HashMap::new();
|
||||
let mut done: HashSet<u64> = HashSet::new();
|
||||
let mut pending: Vec<(CallId, String, String, DataClass, bool)> = Vec::new();
|
||||
for record in session.records() {
|
||||
match record {
|
||||
LogRecord::Assistant { tool_calls, .. } => {
|
||||
for ToolCall { id, name, .. } in tool_calls {
|
||||
names.insert(id.clone(), name.clone());
|
||||
}
|
||||
}
|
||||
LogRecord::ToolSummary { call, .. } => {
|
||||
done.insert(call.0);
|
||||
}
|
||||
LogRecord::ToolResult {
|
||||
call,
|
||||
tool_call_id,
|
||||
content,
|
||||
class,
|
||||
untrusted,
|
||||
..
|
||||
} => {
|
||||
if done.contains(&call.0) {
|
||||
continue;
|
||||
}
|
||||
let Some(name) = names.get(tool_call_id) else {
|
||||
continue;
|
||||
};
|
||||
if !cfg.compact.summarise.iter().any(|n| n == name) {
|
||||
continue;
|
||||
}
|
||||
pending.push((
|
||||
*call,
|
||||
tool_call_id.clone(),
|
||||
content.clone(),
|
||||
*class,
|
||||
*untrusted,
|
||||
));
|
||||
}
|
||||
_ => {}
|
||||
}
|
||||
}
|
||||
for (call, tool_call_id, content, class, untrusted) in pending {
|
||||
let mut messages = messages_with_carry(session.baseline(), session.records(), session.carry());
|
||||
messages.push(ChatMessage::User {
|
||||
content: format!(
|
||||
"{}\n\n{}",
|
||||
prompt.trim_end(),
|
||||
cap_chars(&content, cfg.r#loop.tool_result_cap)
|
||||
),
|
||||
});
|
||||
let request = ChatRequest {
|
||||
slot: cfg.slots.background,
|
||||
messages,
|
||||
tools: Vec::new(),
|
||||
thinking: true,
|
||||
};
|
||||
let text = match client.chat_with_retry(&request, &mut |_| {}) {
|
||||
Ok(completion) => completion.content.unwrap_or_default(),
|
||||
Err(e) => {
|
||||
eprintln!("loopd: tool summary: {e}");
|
||||
continue;
|
||||
}
|
||||
};
|
||||
let text = cap_chars(neutralise(&text).as_ref(), cfg.compact.summary_chars);
|
||||
session
|
||||
.append(LogRecord::ToolSummary {
|
||||
time: Timestamp::now(),
|
||||
call,
|
||||
tool_call_id,
|
||||
content: text,
|
||||
class,
|
||||
untrusted,
|
||||
})
|
||||
.map_err(|e| e.to_string())?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
```
|
||||
|
||||
`CallId` is a tuple struct `CallId(pub u64)` (already used as `CallId(1)` in tests). If
|
||||
`call.0` does not compile, stop and report.
|
||||
|
||||
### `crates/loopd/src/compact.rs`
|
||||
|
||||
**F1.** The `Ok(done)` arm. The lines
|
||||
|
||||
```rust
|
||||
Ok(done) => {
|
||||
if let Some(usage) = session.last_usage()
|
||||
&& due(&usage, rt.cfg.compact.threshold, rt.cfg.expect.n_ctx)
|
||||
&& let Err(e) = compact_session(session, rt, memory)
|
||||
{
|
||||
log_compact(session, &e);
|
||||
}
|
||||
Ok(done)
|
||||
}
|
||||
```
|
||||
|
||||
become the same, with the `missing` call inserted as the first statement of the arm (see
|
||||
Facts). The `Err(TurnError::SessionFull)` arm gets the same `missing` call before
|
||||
`compact_session`.
|
||||
|
||||
## Steps
|
||||
|
||||
- [ ] **1. Copy the given files.**
|
||||
|
||||
```sh
|
||||
cp docs/plans/M5d2/files/crates/loopd/tests/tool_summary_call.rs crates/loopd/tests/
|
||||
cp docs/plans/M5d2/files/config/summarise.md config/summarise.md
|
||||
```
|
||||
|
||||
- [ ] **2. See it fail.** `cargo build --tests --keep-going -p loopd`, timeout 600000 ms: test
|
||||
target `tool_summary_call` does not compile (`E0432` unresolved `loopd::summarise`). Every
|
||||
other target compiles.
|
||||
- [ ] **3. The edits.** `summarise.rs`, `pub mod summarise;`, F1. `cargo check -p loopd`
|
||||
prints no error and no warning. `wc -l crates/loopd/src/summarise.rs` and
|
||||
`wc -l crates/loopd/src/compact.rs` each ≤ 500.
|
||||
- [ ] **4. The tests.** `cargo test -p loopd --test tool_summary_call --test
|
||||
tool_summary_messages --test compact_turn --test turn`: every test passes, 6 in
|
||||
`tool_summary_call`.
|
||||
- [ ] **5. Check.** `git diff --stat` lists `crates/loopd/src/summarise.rs`,
|
||||
`crates/loopd/src/lib.rs`, `crates/loopd/src/compact.rs`.
|
||||
- [ ] **6. Run the gate.** `cargo fmt --all`, then `make gate`, timeout 600000 ms. Last line
|
||||
`gate: ok`. Test count **1561** (1555 before, plus 6).
|
||||
- [ ] **7. Log and commit.** Task cell `M5d2/08-loopd-tool-summary`. Stage with
|
||||
`git add crates/loopd config/summarise.md docs/implementer-log.md Cargo.lock`, then commit
|
||||
as `AGENTS.md` says.
|
||||
|
||||
## Done when
|
||||
|
||||
- `make gate` prints `gate: ok` with 1561 tests.
|
||||
- `git diff HEAD~1 --stat` lists only `crates/loopd/src/summarise.rs`,
|
||||
`crates/loopd/src/lib.rs`, `crates/loopd/src/compact.rs`,
|
||||
`crates/loopd/tests/tool_summary_call.rs`, `config/summarise.md` and the log.
|
||||
|
||||
## Stop and report if
|
||||
|
||||
- The branch check fails; a "before" block is not found exactly once; step 2 differs; `cargo
|
||||
check` warns; a test fails; the gate count is not 1561; you send tools on the summary
|
||||
request; you send `slots.main`; you replay the summary in `messages()`; you summarise a
|
||||
name not in the list; you summarise twice; you compact before summarising.
|
||||
|
||||
## What the given tests pin
|
||||
|
||||
An opted-in result is summarised on slot 1 with no `tools` key; labels are copied; a name not
|
||||
in the list is left alone; a second `missing` does not call again; `messages()` is unchanged
|
||||
after a summary; `finish_turn` after a `read_file` turn appends a `ToolSummary` on slot 1.
|
||||
@@ -0,0 +1,141 @@
|
||||
# M5d-2 task 09: index `ToolSummary` as kind `summary`
|
||||
|
||||
**Branch:** `m5d2`. Check it: `git branch --show-current` prints `m5d2` and `git status --short`
|
||||
is empty; otherwise stop. Never switch branch.
|
||||
**Commit subject:** `memoryd: index ToolSummary as kind summary`
|
||||
**Log row:** the Task cell is `M5d2/09-memoryd-summary-index`.
|
||||
|
||||
## Goal
|
||||
|
||||
`memoryd` indexes `LogRecord::ToolSummary` content as kind `summary`, with that record's
|
||||
`class` and `untrusted` (spec 4.3, P26). `recall` finds it and tags `memory` or
|
||||
`memory-tainted` as for any other session hit. Raw `ToolResult` bodies stay unindexed.
|
||||
|
||||
`index_record` today has `_ => {}`. It compiles. This task adds the arm.
|
||||
|
||||
## Every place that changes
|
||||
|
||||
| Place | What happens there |
|
||||
|---|---|
|
||||
| `crates/memoryd/src/index.rs` | `ToolSummary` arm (edit I1); module doc |
|
||||
| `crates/memoryd/tests/tool_summary_index.rs` | new, given |
|
||||
| everything else | **no change**: not `recall.rs` headers (role `"summary"` uses the existing "the assistant said" branch) |
|
||||
|
||||
## Files
|
||||
|
||||
- Copy, from `docs/plans/M5d2/files/` (the same path under it): the new test
|
||||
`crates/memoryd/tests/tool_summary_index.rs`.
|
||||
- Modify: `crates/memoryd/src/index.rs`, `docs/implementer-log.md`.
|
||||
- Do not touch: any other file. Do not edit `tests/support/logs.rs`.
|
||||
|
||||
## Facts you need
|
||||
|
||||
- **Kind `summary`.** Set `kind: "summary".to_string()` on the `Source`, not the log's
|
||||
`"session"`. Role `"summary"`. Time from the record. `class` / `untrusted` from the
|
||||
`ToolSummary`, not from `base` (the rolling session label). Skip when
|
||||
`summary.class > self.max_class`, even if `base.class` is still `Private`. Skip when
|
||||
`content.trim()` is empty.
|
||||
- **The existing `if base.class > self.max_class` stays.** A secret `ToolResult` raises the
|
||||
rolling label before the next record, so a following secret summary is also skipped that
|
||||
way. Still check the summary's own class.
|
||||
- **Do not add `ToolSummary` to the rolling-label update** in `index_log` (that loop still
|
||||
looks at `ToolResult` only). The result already carried the same labels.
|
||||
- **Recall headers.** `kind` `"summary"` is not `moved` / `dropped` / `noted`, so the tag is
|
||||
`memory` or `memory-tainted`. Role `"summary"` is not `owner` or `call`, so the line says
|
||||
`the assistant said`. The given tests look for `memory · session mm-a` and
|
||||
`memory-tainted`.
|
||||
- The items the edits use. This table is complete.
|
||||
|
||||
| Item | What it does |
|
||||
|---|---|
|
||||
| `self.add` | FTS row plus `sources` |
|
||||
| `Source` | kind / role / class / untrusted |
|
||||
|
||||
## The edits, exactly
|
||||
|
||||
Each quoted "before" block occurs exactly once.
|
||||
|
||||
### `crates/memoryd/src/index.rs`
|
||||
|
||||
**I1.** After the `Assistant` arm, before `_ => {}`. The lines
|
||||
|
||||
```rust
|
||||
}
|
||||
_ => {}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
```
|
||||
|
||||
become
|
||||
|
||||
```rust
|
||||
}
|
||||
LogRecord::ToolSummary {
|
||||
time,
|
||||
content,
|
||||
class,
|
||||
untrusted,
|
||||
..
|
||||
} if !content.trim().is_empty() && *class <= self.max_class => {
|
||||
let src = Source {
|
||||
kind: "summary".to_string(),
|
||||
time: time.to_rfc3339(),
|
||||
role: "summary".to_string(),
|
||||
class: *class,
|
||||
untrusted: *untrusted,
|
||||
..base.clone()
|
||||
};
|
||||
self.add(content, &src)?;
|
||||
}
|
||||
_ => {}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
```
|
||||
|
||||
`DataClass` already implements `Ord` (`Public < Private < Secret`). If `<=` does not
|
||||
compile, stop.
|
||||
|
||||
Optionally update the module doc "never a tool's result" to say summaries of opted-in
|
||||
results are indexed. A comment that does not fit its code: stop.
|
||||
|
||||
## Steps
|
||||
|
||||
- [ ] **1. Copy the given test.**
|
||||
|
||||
```sh
|
||||
cp docs/plans/M5d2/files/crates/memoryd/tests/tool_summary_index.rs crates/memoryd/tests/
|
||||
```
|
||||
|
||||
- [ ] **2. See it fail.** `cargo test -p memoryd --test tool_summary_index`, timeout 600000
|
||||
ms: the tests compile (`ToolSummary` exists) and `a_tool_summary_is_found_as_kind_summary`
|
||||
fails: `(no matches for "tea")` or the unique phrase is missing. `the_raw_tool_result_is_still_not_indexed`
|
||||
may pass (results were already unindexed).
|
||||
- [ ] **3. The edit.** I1. `cargo check -p memoryd` prints no error and no warning.
|
||||
- [ ] **4. The tests.** `cargo test -p memoryd --test tool_summary_index --test recall`: every
|
||||
test passes, 5 in `tool_summary_index`.
|
||||
- [ ] **5. Check.** `git diff --stat` lists exactly `crates/memoryd/src/index.rs`.
|
||||
- [ ] **6. Run the gate.** `cargo fmt --all`, then `make gate`, timeout 600000 ms. Last line
|
||||
`gate: ok`. Test count **1566** (1561 before, plus 5).
|
||||
- [ ] **7. Log and commit.** Task cell `M5d2/09-memoryd-summary-index`. Stage with
|
||||
`git add crates/memoryd docs/implementer-log.md Cargo.lock`, then commit as `AGENTS.md`
|
||||
says.
|
||||
|
||||
## Done when
|
||||
|
||||
- `make gate` prints `gate: ok` with 1566 tests.
|
||||
- `git diff HEAD~1 --stat` lists only `crates/memoryd/src/index.rs`,
|
||||
`crates/memoryd/tests/tool_summary_index.rs` and the log.
|
||||
|
||||
## Stop and report if
|
||||
|
||||
- The branch check fails; a "before" block is not found exactly once; step 2 differs; `cargo
|
||||
check` warns; a test fails; the gate count is not 1566; you index `ToolResult` content; you
|
||||
leave `kind` as `"session"`; a secret summary is searchable on a private index.
|
||||
|
||||
## What the given tests pin
|
||||
|
||||
A summary's content is found and tagged `memory · session …`; the raw tool body is not; an
|
||||
untrusted summary taints the recall (`memory-tainted`); a trusted summary does not taint; a
|
||||
secret summary is skipped when `max_class` is `Private`.
|
||||
@@ -0,0 +1,227 @@
|
||||
# M5d-2 task 10: `MemoryCommand::Tiers` and `bxctl memory tiers`
|
||||
|
||||
**Branch:** `m5d2`. Check it: `git branch --show-current` prints `m5d2` and `git status --short`
|
||||
is empty; otherwise stop. Never switch branch.
|
||||
**Commit subject:** `memoryd: Tiers listing; bxctl memory tiers`
|
||||
**Log row:** the Task cell is `M5d2/10-memoryd-tiers`.
|
||||
|
||||
## Goal
|
||||
|
||||
`MemoryCommand::Tiers` lists each identity entry and each note with how many times `recalls/`
|
||||
named it and when it was last found (spec 4.4). Untrusted notes are included and marked.
|
||||
Identity entries list with 0 hits (README known limits). `bxctl memory tiers` prints that
|
||||
listing. The starter `config/jobs/tier.toml.example` documents a weekly job; nothing in
|
||||
`loopd` special-cases it.
|
||||
|
||||
## Every place that changes
|
||||
|
||||
| Place | What happens there |
|
||||
|---|---|
|
||||
| `crates/bxcore/src/memory.rs` | `MemoryCommand::Tiers` after `Sync` |
|
||||
| `crates/memoryd/src/tiers.rs` | new: `list` |
|
||||
| `crates/memoryd/src/lib.rs` | `pub mod tiers;` |
|
||||
| `crates/memoryd/src/serve/admin.rs` | `Tiers` arm |
|
||||
| `crates/bxctl/src/memory.rs` | `["tiers"]` |
|
||||
| `crates/bxctl/src/cli.rs` | `USAGE` names `tiers` |
|
||||
| `config/jobs/tier.toml.example` | new, given starter |
|
||||
| `crates/memoryd/tests/tiers.rs` | new, given |
|
||||
| `crates/bxctl/tests/cli_memory_tiers.rs` | new, given |
|
||||
| everything else | **no change**: not `cli_memory.rs` (T19: new file) |
|
||||
|
||||
## Files
|
||||
|
||||
- Copy, from `docs/plans/M5d2/files/` (the same path under it): the new tests
|
||||
`crates/memoryd/tests/tiers.rs` and `crates/bxctl/tests/cli_memory_tiers.rs`, and the
|
||||
starter `config/jobs/tier.toml.example`.
|
||||
- Create: `crates/memoryd/src/tiers.rs` (function below).
|
||||
- Modify: `crates/bxcore/src/memory.rs`, `crates/memoryd/src/lib.rs`,
|
||||
`crates/memoryd/src/serve/admin.rs`, `crates/bxctl/src/memory.rs`,
|
||||
`crates/bxctl/src/cli.rs`, `docs/implementer-log.md`.
|
||||
- Do not touch: any other file. Do not edit `crates/bxctl/tests/cli_memory.rs`.
|
||||
|
||||
## Facts you need
|
||||
|
||||
- **`MemoryCommand` is `rename_all = "snake_case"`.** JSON is `"tiers"`. Add the variant after
|
||||
`Sync`, with a doc comment. `admin.rs` match is exhaustive: add the arm in this task or it
|
||||
does not compile.
|
||||
- **Listing format, exact.** One block per identity entry, then one per note, in file order:
|
||||
|
||||
```text
|
||||
identity memory.md hits 0 last -
|
||||
I like tea.
|
||||
identity user.md hits 0 last -
|
||||
The owner is Kyle.
|
||||
note session mm-a hits 2 last 2026-10-03T07:30:00.000Z private trusted
|
||||
launch window moved.
|
||||
note session mm-c hits 0 last - private untrusted
|
||||
example.org says the launch moved.
|
||||
```
|
||||
|
||||
Two spaces before the text line. `hits` and `last` for identity are always `0` and `-`.
|
||||
For a note, `hits` is how many `recalls/*.jsonl` lines contain a hit with
|
||||
`kind == "noted"`, `session == note.session`, `record ==` that note's index in
|
||||
`notes::parse` (from 0). `last` is the maximum `time` string among those lines, or `-`.
|
||||
Class is `public` / `private` / `secret` (`notes::class_name`). Trust is `untrusted` when
|
||||
`Kind::Noted { untrusted: true }`, else `trusted`. Moved/dropped notes: still list them as
|
||||
`note` using the same line, class `private`, `trusted` (they have no untrusted flag).
|
||||
- **`recalls/`.** `store.path("recalls")`. Missing directory: all counts 0. A file that
|
||||
exists but cannot be read is an error. Ignore a line that does not parse. Deserialize:
|
||||
|
||||
```rust
|
||||
#[derive(Deserialize)]
|
||||
struct Logged {
|
||||
time: String,
|
||||
hits: Vec<Hit>,
|
||||
}
|
||||
#[derive(Deserialize)]
|
||||
struct Hit {
|
||||
kind: String,
|
||||
session: String,
|
||||
record: i64,
|
||||
}
|
||||
```
|
||||
|
||||
(extra fields in the file are ignored: this is our format but old lines may grow; the given
|
||||
tests use exactly those fields plus `session` / `query` / `epoch` on the line. Use
|
||||
`deny_unknown_fields` **off** so `epoch` and `query` do not fail.)
|
||||
- **Identity.** `store.entries(MemoryFile::Memory)` then `User`. Empty file: no identity
|
||||
lines for that file.
|
||||
- **`list` returns `Result<String, String>`.** Trailing newline on the last block is fine;
|
||||
`bxctl` adds another when printing `MemoryAdminReply.text` (`writeln!`). The given bxctl
|
||||
test includes a trailing newline in the reply text and expects it plus `writeln!`'s
|
||||
newline.
|
||||
- **`admin` arm.** `MemoryCommand::Tiers => tiers::list(&self.store, now)` — `now` is unused
|
||||
by `list` today; keep the signature `list(store, now)` so the given tests compile.
|
||||
- **`bxctl`.** `["tiers"] => Ask::Admin(MemoryCommand::Tiers, String::new())`. `["tiers", _]`
|
||||
is usage. USAGE: add `| tiers` on the `bxctl memory` line.
|
||||
- rustfmt `mod` order in `memoryd/src/lib.rs`.
|
||||
- The items the edits use. This table is complete.
|
||||
|
||||
| Item | What it does |
|
||||
|---|---|
|
||||
| `Store::entries` | identity |
|
||||
| `notes::parse` / `class_name` | notes |
|
||||
| `Store::path("recalls")` | the jsonl directory |
|
||||
| `MemoryFile::name` | `memory.md` / `user.md` |
|
||||
|
||||
## The function
|
||||
|
||||
`crates/memoryd/src/tiers.rs` — fill `list`. No `todo!()`.
|
||||
|
||||
```rust
|
||||
//! The tiering listing: identity entries and notes with recall-hit counts (M5d spec, 4.4).
|
||||
|
||||
use bxcore::{MemoryFile, Timestamp};
|
||||
use serde::Deserialize;
|
||||
|
||||
use crate::files::Store;
|
||||
use crate::notes::{self, Kind};
|
||||
|
||||
#[derive(Deserialize)]
|
||||
struct Logged {
|
||||
time: String,
|
||||
hits: Vec<Hit>,
|
||||
}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
struct Hit {
|
||||
kind: String,
|
||||
session: String,
|
||||
record: i64,
|
||||
}
|
||||
|
||||
/// One listing for `bxctl memory tiers` and for the tiering job's user text.
|
||||
pub fn list(store: &Store, _now: Timestamp) -> Result<String, String> {
|
||||
let _ = store;
|
||||
Ok(String::new())
|
||||
}
|
||||
```
|
||||
|
||||
Replace the stub body in this task.
|
||||
|
||||
### `crates/bxcore/src/memory.rs`
|
||||
|
||||
**M1.** After `Sync,` in `MemoryCommand`:
|
||||
|
||||
```rust
|
||||
/// Identity entries and notes with recall-hit counts (M5d spec, section 4.4).
|
||||
Tiers,
|
||||
```
|
||||
|
||||
### `crates/memoryd/src/serve/admin.rs`
|
||||
|
||||
**A1.** After the `Sync` arm, before the closing of the `match`:
|
||||
|
||||
```rust
|
||||
MemoryCommand::Tiers => crate::tiers::list(&self.store, now),
|
||||
```
|
||||
|
||||
### `crates/bxctl/src/memory.rs`
|
||||
|
||||
**B1.** After `["sync"] => ...`, before `_ => return Err(UsageError)`:
|
||||
|
||||
```rust
|
||||
["tiers"] => Ask::Admin(MemoryCommand::Tiers, String::new()),
|
||||
```
|
||||
|
||||
### `crates/bxctl/src/cli.rs`
|
||||
|
||||
**U1.** The `bxctl memory` USAGE line. The line
|
||||
|
||||
```text
|
||||
bxctl memory log [N] | undo <id> | pending | accept <id> | reject <id> | sync [--memory-socket PATH]
|
||||
```
|
||||
|
||||
becomes
|
||||
|
||||
```text
|
||||
bxctl memory log [N] | undo <id> | pending | accept <id> | reject <id> | sync | tiers [--memory-socket PATH]
|
||||
```
|
||||
|
||||
## Steps
|
||||
|
||||
- [ ] **1. Copy the given files.**
|
||||
|
||||
```sh
|
||||
cp docs/plans/M5d2/files/crates/memoryd/tests/tiers.rs crates/memoryd/tests/
|
||||
cp docs/plans/M5d2/files/crates/bxctl/tests/cli_memory_tiers.rs crates/bxctl/tests/
|
||||
mkdir -p config/jobs
|
||||
cp docs/plans/M5d2/files/config/jobs/tier.toml.example config/jobs/tier.toml.example
|
||||
```
|
||||
|
||||
- [ ] **2. See them fail.** `cargo build --tests --keep-going -p bxcore -p memoryd -p bxctl`,
|
||||
timeout 600000 ms: `tiers` does not compile (`E0432` unresolved `memoryd::tiers`);
|
||||
`cli_memory_tiers` does not compile (`E0599` no variant `Tiers`). After M1 alone,
|
||||
`admin.rs` fails (`non-exhaustive patterns`). Do M1, A1, `tiers.rs`, B1, U1 together in
|
||||
step 3. This step is the first build against the given tests.
|
||||
- [ ] **3. The edits.** M1, `tiers.rs` with a real `list`, `pub mod tiers;`, A1, B1, U1.
|
||||
`cargo check -p bxcore -p memoryd -p bxctl` prints no error and no warning.
|
||||
`wc -l crates/memoryd/src/tiers.rs` prints a number ≤ 500.
|
||||
- [ ] **4. The tests.** `cargo test -p memoryd --test tiers` (4) and
|
||||
`cargo test -p bxctl --test cli_memory_tiers --test cli_memory --test cli_memory_sync`
|
||||
(2 new): every test passes.
|
||||
- [ ] **5. Check.** `git diff --stat` lists the files in "Every place that changes" except
|
||||
the new tests and starter (those are untracked until `git add`).
|
||||
- [ ] **6. Run the gate.** `cargo fmt --all`, then `make gate`, timeout 600000 ms. Last line
|
||||
`gate: ok`. Test count **1572** (1566 before, plus 6).
|
||||
- [ ] **7. Log and commit.** Task cell `M5d2/10-memoryd-tiers`. Stage with
|
||||
`git add crates/bxcore crates/memoryd crates/bxctl config/jobs docs/implementer-log.md
|
||||
Cargo.lock`, then commit as `AGENTS.md` says.
|
||||
|
||||
## Done when
|
||||
|
||||
- `make gate` prints `gate: ok` with 1572 tests.
|
||||
- `git diff HEAD~1 --stat` lists only the files this task names plus the log.
|
||||
|
||||
## Stop and report if
|
||||
|
||||
- The branch check fails; a "before" block is not found exactly once; step 2 differs; `cargo
|
||||
check` warns; a test fails; the gate count is not 1572; you edit `cli_memory.rs`; identity
|
||||
entries show a non-zero hit count; untrusted notes are omitted; you add a cron crate; you
|
||||
special-case the name `tier` in `loopd`.
|
||||
|
||||
## What the given tests pin
|
||||
|
||||
Identity lines are `hits 0 last -` with the entry text; notes join `recalls/` on
|
||||
`(noted, session, record)` and take the latest `time`; untrusted notes are listed and marked;
|
||||
a hit on note 1 does not count for note 0; `bxctl memory tiers` parses and prints the reply.
|
||||
@@ -0,0 +1,132 @@
|
||||
# M5d-2 implementation plan: the scheduler
|
||||
|
||||
> For the implementer: each task file is one closed task for one fresh session. Read `AGENTS.md`,
|
||||
> then the task file, and do exactly that task. The driver is `tools/run-plan.sh`.
|
||||
|
||||
**Goal:** heartbeat and cron run as isolated sessions on the background slot; live turns use the
|
||||
session's slot; opted-in tool results get a `ToolSummary` that `messages()` ignores and `recall`
|
||||
indexes; the tiering listing is an ordinary admin command. Answers stay in the session log.
|
||||
Nothing is posted to Mattermost.
|
||||
|
||||
**Architecture:** `run_turn` reads `session.slot()`. `loopd` gains `[heartbeat]`, a cron matcher,
|
||||
job-file loading, a scheduler thread that shares `Context` and the busy set, and a no-tools
|
||||
summary call for `http_fetch`. `bxcore` gains `LogRecord::ToolSummary` and `MemoryCommand::Tiers`.
|
||||
`memoryd` indexes summaries and answers `Tiers` from `recalls/`. `bxctl memory tiers` prints that
|
||||
listing. No new crate.
|
||||
|
||||
**Tech stack:** Rust 1.98.1, edition 2024; no cron crate; no new dependency.
|
||||
|
||||
**Spec:** `docs/specs/2026-10-03-m5d-scheduler.md`, sections 4, 5, 6, 7 (the M5d-2 tests), and 8
|
||||
(`jobs-invalid`, `heartbeat-prompt-unreadable`). M5d-1 (epochs and compaction) is already on the
|
||||
branch this plan starts from.
|
||||
|
||||
## The owner's decisions (2026-10-03)
|
||||
|
||||
1. Heartbeat and cron answers stay in the session log. They are not posted to Mattermost.
|
||||
2. Five-field UTC crontab, `{home}/jobs/<name>.toml`, a new session each run, a tick skipped
|
||||
when that job is still live.
|
||||
3. Tool summaries on the background slot; a `ToolSummary` log record with the result's labels,
|
||||
FTS-indexed; `http_fetch` opts in via `[compact] summarise`.
|
||||
|
||||
## What the code already has, and what it lacks
|
||||
|
||||
- M5d-1 writes epochs, compact after a turn, Honcho over every epoch file. Live turns still send
|
||||
`slots.main`. `[compact] summarise` is stored and unused. Unknown names are not a config error.
|
||||
- `LogRecord` has no `ToolSummary`. `messages()` names every variant. `support/turn.rs` `types`
|
||||
is exhaustive.
|
||||
- `loopd serve` binds the channel and blocks in `channel::serve`. There is no scheduler thread.
|
||||
- `MemoryCommand` has no `Tiers`. `bxctl memory` has no `tiers`.
|
||||
- `SessionId` is `[a-z0-9-]`, max 64. Job ids `job-<name>-<yyyymmdd>t<hhmm>` must fit.
|
||||
|
||||
## Global constraints
|
||||
|
||||
- Roles depend on `bxcore` only. No `unsafe`, no async runtime, no new crate.
|
||||
- No file the implementer reads holds a chat-template tag whole (T33).
|
||||
- Every path a given test creates is removed by a guard (T44); no test depends on timing (T27):
|
||||
the scheduler is `tick(now)`, never a sleep.
|
||||
- Fail-closed states `jobs-invalid` and `heartbeat-prompt-unreadable` have runbook entries,
|
||||
added by the task that adds their pointer (T49).
|
||||
- Each task gives its baseline and its gate count (T50, T53); the branch line is a check, never
|
||||
`git switch` (T51).
|
||||
- `Cargo.lock` is in the `git add` line (T24). Gate timeout 600000 ms (T40).
|
||||
- A later task that changes a shared given file ships its own copy (T19): task 02 replaces
|
||||
`crates/loopd/tests/support/turn.rs` so `types` names `ToolSummary`.
|
||||
|
||||
## Tasks
|
||||
|
||||
Every task below has been written to be replayed from its text. The gate count is the passed
|
||||
total of `cargo test --workspace` plus one, for the ignored `end_to_end` test the gate runs on
|
||||
its own. Before task 01 it is **1507** (talos, rustc 1.98.1; M5d-1 task 08).
|
||||
|
||||
| # | Delivers | Crates | Gate after | Tests added |
|
||||
|---|---|---|---|---|
|
||||
| 01 | `run_turn` sends `session.slot()`; a session created on the background slot has `id_slot` of that slot; `mm-*` stays main | loopd | 1511 | 4 |
|
||||
| 02 | `LogRecord::ToolSummary`; `messages()` ignores it; `types` names it; JSON round-trip | bxcore, loopd | 1517 | 6 |
|
||||
| 03 | `[heartbeat]` (`every_s` 0, `prompt` `heartbeat.md`); unknown `[compact] summarise` names are a config error; starter `heartbeat.md` | loopd | 1525 | 8 |
|
||||
| 04 | Five-field UTC cron matcher: `*`, `n`, `n-m`, `*/n`; day-of-week 0 is Sunday | loopd | 1535 | 10 |
|
||||
| 05 | `{home}/jobs/<name>.toml`, `runs.jsonl`, `jobs-invalid` pointer and runbook | loopd | 1543 | 8 |
|
||||
| 06 | Heartbeat tick: time line, standing session `heartbeat`, skip if busy, unreadable prompt logged once | loopd | 1549 | 6 |
|
||||
| 07 | Cron tick: new session per run, skip if previous live, leftover `started` becomes `failed` | loopd | 1555 | 6 |
|
||||
| 08 | After a turn, opted-in `ToolResult` without a `ToolSummary` gets a background summary appended with the result's labels | loopd | 1561 | 6 |
|
||||
| 09 | `memoryd` indexes `ToolSummary` as kind `summary`; `recall` finds it and taints when untrusted | memoryd | 1566 | 5 |
|
||||
| 10 | `MemoryCommand::Tiers`, `bxctl memory tiers`, starter `config/jobs/tier.toml.example` | bxcore, memoryd, bxctl | 1572 | 6 |
|
||||
|
||||
After task 10, the design model runs the M5d-2 device check (spec 7) on straylight: a heartbeat
|
||||
turn on the background slot leaves a following main-slot turn's `cache_n` near its previous
|
||||
total; a `jobs/tier.toml` for the check promotes only a trusted note.
|
||||
|
||||
## Review Focus
|
||||
|
||||
These are implied by the spec and pinned in the named task. A reviewer who only reads the
|
||||
happy-path tests will miss them.
|
||||
|
||||
- A live `mm-*` turn still sends `slots.main` after heartbeat used the background slot (task 01).
|
||||
- Inserting a `ToolSummary` between a `ToolResult` and a later `User` does not break prefix
|
||||
extension (task 02).
|
||||
- An unknown `[compact] summarise` name is a config error; an empty list is not (task 03).
|
||||
- DOM and DOW are AND; `@daily` and a sixth field are errors (task 04).
|
||||
- A leftover `started` whose session is currently held is not closed as `failed` (tasks 06, 07).
|
||||
- `finish_turn` summarises before it compacts, so the `ToolResult` is still in the current epoch
|
||||
(task 08).
|
||||
- A secret `ToolSummary` is skipped when the index is private (task 09).
|
||||
- Untrusted notes appear in `Tiers` and are marked; they cannot be omitted (task 10).
|
||||
|
||||
## Known limits
|
||||
|
||||
These are known, accepted for M5d-2, and not bugs to fix in a task:
|
||||
|
||||
- **No Mattermost delivery.** Heartbeat and cron answers are in `sessions/<id>/` only. The owner
|
||||
reads them with `bxctl chat --session` or by opening the log.
|
||||
- **DOM and DOW are AND.** Both restricted (`0 7 1 * 0`) matches only a Sunday that is also the
|
||||
1st. Standard Vixie cron would OR them. The spec does not ask for OR.
|
||||
- **Tool-summary inference runs inside `finish_turn`, before compaction, while the session is
|
||||
still held.** That is how compaction waits for the summary, and how `missing` still sees the
|
||||
current epoch's `ToolResult`. A second live turn on that session waits. The summary request
|
||||
itself uses the background slot and omits `tools`. A later split that infers without the hold
|
||||
is out of scope.
|
||||
- **Job files are checked at start.** A file that becomes invalid after `loopd` is running is
|
||||
skipped that tick and logged; it does not stop the process.
|
||||
- **Identity entries in `Tiers` list with 0 hits** unless a `recalls/` line names that exact
|
||||
index row. Notes carry counts from `recalls/`. That is enough for the tiering prompt.
|
||||
|
||||
## Before hand-over
|
||||
|
||||
Work through `docs/plans/CHECKLIST.md` with a reference in a scratch worktree with its own build
|
||||
directory, replayed from its task text; mutations; `/tmp` compared around a gate run; the
|
||||
reference moved to straylight as a bundle before the run (T18). Task files name their log row
|
||||
(T47).
|
||||
|
||||
## Running it
|
||||
|
||||
The reference (`m5d2-ref`, tasks 01 to 10) is moved to straylight as a bundle and deleted here
|
||||
before the run (T18). Then, from the repository root, with the OpenCode TUI closed, after M5d-1
|
||||
has landed on master:
|
||||
|
||||
```sh
|
||||
git switch -c m5d2 master
|
||||
OPENCODE_MODEL=straylight/ornith-1.5-35b-a3b RESUME=1 \
|
||||
SLOTS_URL='http://straylight:11434/slots?model=ornith-1.5-35b-a3b' \
|
||||
tools/run-plan.sh docs/plans/M5d2 01
|
||||
```
|
||||
|
||||
After task 10, the gate count is 1572. Then the device check above.
|
||||
@@ -0,0 +1,4 @@
|
||||
It is time for a heartbeat.
|
||||
|
||||
Look at recent notes and the current identity. Write a note only if something needs the owner, or
|
||||
if a standing fact should move. Do not use tools that fetch. Do not address the owner.
|
||||
@@ -0,0 +1,6 @@
|
||||
# Copy to {BOXMAKER_HOME}/jobs/tier.toml and edit the schedule.
|
||||
# Five-field UTC crontab. This example is Sunday 07:00.
|
||||
schedule = "0 7 * * 0"
|
||||
prompt = """Read the listing in this message. Promote only trusted notes into identity
|
||||
(memory.md / user.md) with add or consolidate. Do not promote an untrusted note.
|
||||
Do not fetch. Write a note of what you changed, or that nothing needed changing."""
|
||||
@@ -0,0 +1,2 @@
|
||||
Summarise this tool result for later recall. Keep facts, names, numbers and URLs. Drop boilerplate
|
||||
and navigation. Do not use tools. Do not address the owner. Do not invent facts.
|
||||
@@ -0,0 +1,52 @@
|
||||
//! Tests for `LogRecord::ToolSummary`. Do not edit.
|
||||
|
||||
use bxcore::{CallId, DataClass, LogRecord, Timestamp};
|
||||
|
||||
fn ts() -> Timestamp {
|
||||
Timestamp::parse("2026-09-18T08:00:00.000Z").unwrap()
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn tool_summary_round_trips() {
|
||||
let rec = LogRecord::ToolSummary {
|
||||
time: ts(),
|
||||
call: CallId(3),
|
||||
tool_call_id: "call_x".to_string(),
|
||||
content: "the page is about tea".to_string(),
|
||||
class: DataClass::Public,
|
||||
untrusted: true,
|
||||
};
|
||||
let line = serde_json::to_string(&rec).unwrap();
|
||||
assert!(line.contains("\"type\":\"tool_summary\""), "{line}");
|
||||
let back: LogRecord = serde_json::from_str(&line).unwrap();
|
||||
assert_eq!(back, rec);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn unknown_fields_are_refused() {
|
||||
let line = r#"{"type":"tool_summary","time":"2026-09-18T08:00:00.000Z","call":3,"tool_call_id":"c","content":"x","class":"public","untrusted":false,"extra":1}"#;
|
||||
assert!(serde_json::from_str::<LogRecord>(line).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn field_order_is_the_struct_order() {
|
||||
let rec = LogRecord::ToolSummary {
|
||||
time: ts(),
|
||||
call: CallId(1),
|
||||
tool_call_id: "c".to_string(),
|
||||
content: "x".to_string(),
|
||||
class: DataClass::Private,
|
||||
untrusted: false,
|
||||
};
|
||||
let line = serde_json::to_string(&rec).unwrap();
|
||||
let time = line.find("\"time\"").unwrap();
|
||||
let call = line.find("\"call\"").unwrap();
|
||||
let id = line.find("\"tool_call_id\"").unwrap();
|
||||
let content = line.find("\"content\"").unwrap();
|
||||
let class = line.find("\"class\"").unwrap();
|
||||
let untrusted = line.find("\"untrusted\"").unwrap();
|
||||
assert!(
|
||||
time < call && call < id && id < content && content < class && class < untrusted,
|
||||
"{line}"
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,86 @@
|
||||
//! Tests for `bxctl memory tiers`. Do not edit.
|
||||
|
||||
#[path = "support/temp.rs"]
|
||||
mod temp;
|
||||
use std::os::unix::net::UnixListener;
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::sync::mpsc;
|
||||
use std::time::Duration;
|
||||
|
||||
use bxcore::{Envelope, MemoryAdminReply, MemoryCommand, Message, read_frame, write_frame};
|
||||
use bxctl::cli::{Command, UsageError, parse};
|
||||
use bxctl::memory::{Ask, run};
|
||||
|
||||
fn args(line: &str) -> Vec<String> {
|
||||
line.split_whitespace().map(str::to_string).collect()
|
||||
}
|
||||
|
||||
fn home() -> PathBuf {
|
||||
PathBuf::from("/srv/bx")
|
||||
}
|
||||
|
||||
fn fake(socket: &Path, answer: Message) -> mpsc::Receiver<Message> {
|
||||
let listener = UnixListener::bind(socket).unwrap();
|
||||
let (tx, rx) = mpsc::channel();
|
||||
std::thread::spawn(move || {
|
||||
let (mut stream, _) = listener.accept().unwrap();
|
||||
let env = read_frame(&mut stream).unwrap();
|
||||
tx.send(env.msg).unwrap();
|
||||
let reply = Envelope {
|
||||
v: 1,
|
||||
id: env.id,
|
||||
r#final: true,
|
||||
msg: answer,
|
||||
};
|
||||
write_frame(&mut stream, &reply).unwrap();
|
||||
});
|
||||
rx
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn memory_tiers_parses() {
|
||||
let h = home();
|
||||
assert_eq!(
|
||||
parse(&args("memory tiers"), &h),
|
||||
Ok(Command::Memory {
|
||||
socket: PathBuf::from("/srv/bx/run/owner-memory/admin.sock"),
|
||||
ask: Ask::Admin(MemoryCommand::Tiers, String::new())
|
||||
})
|
||||
);
|
||||
assert_eq!(
|
||||
parse(&args("memory --memory-socket /r/m.sock tiers"), &h),
|
||||
Ok(Command::Memory {
|
||||
socket: PathBuf::from("/r/m.sock"),
|
||||
ask: Ask::Admin(MemoryCommand::Tiers, String::new())
|
||||
})
|
||||
);
|
||||
assert_eq!(parse(&args("memory tiers extra"), &h), Err(UsageError));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn memory_tiers_prints_the_reply() {
|
||||
let d = temp::TempDir::new("bx-tiers");
|
||||
let socket = d.join("admin.sock");
|
||||
let listing = "identity memory.md hits 0 last -\n I like tea.\n";
|
||||
let seen = fake(
|
||||
&socket,
|
||||
Message::MemoryAdminReply(MemoryAdminReply {
|
||||
text: listing.to_string(),
|
||||
}),
|
||||
);
|
||||
let mut out = Vec::new();
|
||||
run(
|
||||
&socket,
|
||||
&Ask::Admin(MemoryCommand::Tiers, String::new()),
|
||||
Duration::from_secs(5),
|
||||
&mut out,
|
||||
)
|
||||
.unwrap();
|
||||
assert_eq!(String::from_utf8(out).unwrap(), format!("{listing}\n"));
|
||||
match seen.recv().unwrap() {
|
||||
Message::MemoryAdmin(m) => {
|
||||
assert_eq!((m.command, m.arg.as_str()), (MemoryCommand::Tiers, ""))
|
||||
}
|
||||
other => panic!("{other:?}"),
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,110 @@
|
||||
//! Tests for `[heartbeat]` and unknown `[compact] summarise` names. Do not edit.
|
||||
|
||||
use loopd::config::Config;
|
||||
use loopd::tools::Registry;
|
||||
use std::path::{Path, PathBuf};
|
||||
|
||||
fn fixture(name: &str) -> PathBuf {
|
||||
Path::new(env!("CARGO_MANIFEST_DIR"))
|
||||
.join("tests/fixtures/config")
|
||||
.join(name)
|
||||
}
|
||||
|
||||
fn minimal() -> String {
|
||||
std::fs::read_to_string(fixture("minimal.toml")).unwrap()
|
||||
}
|
||||
|
||||
struct Scratch {
|
||||
dir: PathBuf,
|
||||
}
|
||||
|
||||
impl Scratch {
|
||||
fn new() -> Scratch {
|
||||
let n = std::time::SystemTime::now()
|
||||
.duration_since(std::time::UNIX_EPOCH)
|
||||
.unwrap()
|
||||
.as_nanos();
|
||||
let dir = std::env::temp_dir().join(format!("loopd-hb-{}-{n}", std::process::id()));
|
||||
let _ = std::fs::remove_dir_all(&dir);
|
||||
std::fs::create_dir_all(&dir).unwrap();
|
||||
Scratch { dir }
|
||||
}
|
||||
}
|
||||
|
||||
impl Drop for Scratch {
|
||||
fn drop(&mut self) {
|
||||
let _ = std::fs::remove_dir_all(&self.dir);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn missing_heartbeat_table_is_disabled() {
|
||||
let c = Config::load(&fixture("minimal.toml")).unwrap();
|
||||
assert_eq!(c.heartbeat.every_s, 0);
|
||||
assert_eq!(
|
||||
c.heartbeat_prompt(),
|
||||
fixture("minimal.toml")
|
||||
.parent()
|
||||
.unwrap()
|
||||
.join("heartbeat.md")
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_table_overrides_every_default() {
|
||||
let text = format!(
|
||||
"{}\n[heartbeat]\nevery_s = 60\nprompt = \"pulse.md\"\n",
|
||||
minimal()
|
||||
);
|
||||
let c = Config::parse(&text).unwrap();
|
||||
assert_eq!(c.heartbeat.every_s, 60);
|
||||
assert_eq!(c.heartbeat.prompt, PathBuf::from("pulse.md"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn unknown_keys_are_errors() {
|
||||
let bad = format!("{}\n[heartbeat]\nzz_unknown = 1\n", minimal());
|
||||
assert!(Config::parse(&bad).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn heartbeat_prompt_is_beside_the_config_file() {
|
||||
let home = Scratch::new();
|
||||
let cfg_path = home.dir.join("loopd.toml");
|
||||
std::fs::write(&cfg_path, minimal()).unwrap();
|
||||
std::fs::write(home.dir.join("system.md"), "sys\n").unwrap();
|
||||
let c = Config::load(&cfg_path).unwrap();
|
||||
assert_eq!(c.heartbeat_prompt(), home.dir.join("heartbeat.md"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn default_summarise_http_fetch_is_known_to_m5c() {
|
||||
let c = Config::parse(&minimal()).unwrap();
|
||||
c.check_summarise(&Registry::m5c()).unwrap();
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn an_unknown_summarise_name_is_an_error() {
|
||||
let text = format!("{}\n[compact]\nsummarise = [\"no_such_tool\"]\n", minimal());
|
||||
let c = Config::parse(&text).unwrap();
|
||||
let err = c.check_summarise(&Registry::m5c()).unwrap_err();
|
||||
assert!(err.contains("no_such_tool"), "{err}");
|
||||
assert!(err.contains("summarise"), "{err}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn empty_summarise_is_allowed() {
|
||||
let text = format!("{}\n[compact]\nsummarise = []\n", minimal());
|
||||
let c = Config::parse(&text).unwrap();
|
||||
c.check_summarise(&Registry::m5c()).unwrap();
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn http_fetch_is_known_and_clock_is_known() {
|
||||
let text = format!(
|
||||
"{}\n[compact]\nsummarise = [\"http_fetch\", \"clock\"]\n",
|
||||
minimal()
|
||||
);
|
||||
let c = Config::parse(&text).unwrap();
|
||||
c.check_summarise(&Registry::m5c()).unwrap();
|
||||
}
|
||||
@@ -0,0 +1,85 @@
|
||||
//! Tests for the five-field UTC cron matcher. Do not edit.
|
||||
|
||||
use bxcore::Timestamp;
|
||||
use loopd::cron::{self, CronError};
|
||||
|
||||
fn ts(s: &str) -> Timestamp {
|
||||
Timestamp::parse(s).unwrap()
|
||||
}
|
||||
|
||||
fn due(schedule: &str, at: &str) -> bool {
|
||||
cron::due(schedule, ts(at)).unwrap()
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_daily_hour_matches_that_minute_only() {
|
||||
assert!(due("0 7 * * *", "2026-10-03T07:00:00.000Z"));
|
||||
assert!(!due("0 7 * * *", "2026-10-03T07:01:00.000Z"));
|
||||
assert!(!due("0 7 * * *", "2026-10-03T06:00:00.000Z"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn every_minute_matches() {
|
||||
assert!(due("* * * * *", "2026-10-03T00:00:00.000Z"));
|
||||
assert!(due("* * * * *", "2026-10-03T23:59:00.000Z"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_step_matches_multiples_from_zero() {
|
||||
assert!(due("*/5 * * * *", "2026-10-03T00:00:00.000Z"));
|
||||
assert!(due("*/5 * * * *", "2026-10-03T00:05:00.000Z"));
|
||||
assert!(!due("*/5 * * * *", "2026-10-03T00:06:00.000Z"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_range_is_inclusive() {
|
||||
assert!(due("0-10 * * * *", "2026-10-03T00:00:00.000Z"));
|
||||
assert!(due("0-10 * * * *", "2026-10-03T00:10:00.000Z"));
|
||||
assert!(!due("0-10 * * * *", "2026-10-03T00:11:00.000Z"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn sunday_is_zero() {
|
||||
// 2026-10-04 is a Sunday.
|
||||
assert!(due("0 7 * * 0", "2026-10-04T07:00:00.000Z"));
|
||||
// 2026-10-03 is a Saturday.
|
||||
assert!(!due("0 7 * * 0", "2026-10-03T07:00:00.000Z"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_first_of_the_month() {
|
||||
assert!(due("0 7 1 * *", "2026-10-01T07:00:00.000Z"));
|
||||
assert!(!due("0 7 1 * *", "2026-10-02T07:00:00.000Z"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn four_or_six_fields_are_errors() {
|
||||
assert!(matches!(
|
||||
cron::due("* * * *", ts("2026-10-03T00:00:00.000Z")),
|
||||
Err(CronError::Fields(_))
|
||||
));
|
||||
assert!(matches!(
|
||||
cron::due("* * * * * *", ts("2026-10-03T00:00:00.000Z")),
|
||||
Err(CronError::Fields(_))
|
||||
));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn an_out_of_range_number_is_an_error() {
|
||||
assert!(cron::due("60 * * * *", ts("2026-10-03T00:00:00.000Z")).is_err());
|
||||
assert!(cron::due("* 24 * * *", ts("2026-10-03T00:00:00.000Z")).is_err());
|
||||
assert!(cron::due("* * * * 7", ts("2026-10-03T00:00:00.000Z")).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn nicknames_are_errors() {
|
||||
let err = cron::due("@daily", ts("2026-10-03T00:00:00.000Z")).unwrap_err();
|
||||
assert!(matches!(err, CronError::Fields(_)), "{err:?}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn display_names_the_schedule() {
|
||||
let err = cron::due("x x x x x", ts("2026-10-03T00:00:00.000Z")).unwrap_err();
|
||||
let s = err.to_string();
|
||||
assert!(s.contains("x x x x x"), "{s}");
|
||||
}
|
||||
@@ -0,0 +1,151 @@
|
||||
//! Tests for job files and the run ledger. Do not edit.
|
||||
|
||||
use bxcore::{SessionId, Timestamp};
|
||||
use loopd::jobs::{self, JobsError, RunStatus};
|
||||
|
||||
fn ts() -> Timestamp {
|
||||
Timestamp::parse("2026-10-03T07:00:00.000Z").unwrap()
|
||||
}
|
||||
|
||||
struct Scratch {
|
||||
dir: std::path::PathBuf,
|
||||
}
|
||||
|
||||
impl Scratch {
|
||||
fn new() -> Scratch {
|
||||
let n = std::time::SystemTime::now()
|
||||
.duration_since(std::time::UNIX_EPOCH)
|
||||
.unwrap()
|
||||
.as_nanos();
|
||||
let dir = std::env::temp_dir().join(format!("loopd-jobs-{}-{n}", std::process::id()));
|
||||
let _ = std::fs::remove_dir_all(&dir);
|
||||
std::fs::create_dir_all(&dir).unwrap();
|
||||
Scratch { dir }
|
||||
}
|
||||
}
|
||||
|
||||
impl Drop for Scratch {
|
||||
fn drop(&mut self) {
|
||||
let _ = std::fs::remove_dir_all(&self.dir);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_valid_job_file_loads() {
|
||||
let home = Scratch::new();
|
||||
let jobs = home.dir.join("jobs");
|
||||
std::fs::create_dir(&jobs).unwrap();
|
||||
std::fs::write(
|
||||
jobs.join("mail.toml"),
|
||||
"schedule = \"0 7 * * *\"\nprompt = \"Check mail.\"\n",
|
||||
)
|
||||
.unwrap();
|
||||
let loaded = jobs::load(&jobs).unwrap();
|
||||
assert_eq!(loaded.len(), 1);
|
||||
assert_eq!(loaded[0].name, "mail");
|
||||
assert_eq!(loaded[0].schedule, "0 7 * * *");
|
||||
assert_eq!(loaded[0].prompt, "Check mail.");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn unknown_keys_are_errors_with_the_runbook() {
|
||||
let home = Scratch::new();
|
||||
let jobs = home.dir.join("jobs");
|
||||
std::fs::create_dir(&jobs).unwrap();
|
||||
std::fs::write(
|
||||
jobs.join("mail.toml"),
|
||||
"schedule = \"* * * * *\"\nprompt = \"x\"\nwhen = \"now\"\n",
|
||||
)
|
||||
.unwrap();
|
||||
let err = jobs::load(&jobs).unwrap_err();
|
||||
let s = err.to_string();
|
||||
assert!(s.contains("mail.toml"), "{s}");
|
||||
assert!(s.ends_with("see docs/runbook.md#jobs-invalid"), "{s}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_bad_name_is_an_error() {
|
||||
let home = Scratch::new();
|
||||
let jobs = home.dir.join("jobs");
|
||||
std::fs::create_dir(&jobs).unwrap();
|
||||
std::fs::write(
|
||||
jobs.join("Mail.toml"),
|
||||
"schedule = \"* * * * *\"\nprompt = \"x\"\n",
|
||||
)
|
||||
.unwrap();
|
||||
let err = jobs::load(&jobs).unwrap_err();
|
||||
assert!(matches!(err, JobsError::Name(_)), "{err:?}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn session_id_fits_and_is_lowercase() {
|
||||
let id = jobs::session_id("mail", ts()).unwrap();
|
||||
assert_eq!(id.as_str(), "job-mail-20261003t0700");
|
||||
SessionId::new(id.as_str()).unwrap();
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_name_too_long_for_a_session_id_is_an_error() {
|
||||
let long = "n".repeat(50);
|
||||
let err = jobs::session_id(&long, ts()).unwrap_err();
|
||||
assert!(matches!(err, JobsError::Name(_)), "{err:?}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_missing_jobs_directory_is_empty() {
|
||||
let home = Scratch::new();
|
||||
let loaded = jobs::load(&home.dir.join("jobs")).unwrap();
|
||||
assert!(loaded.is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn leftover_started_is_closed_as_failed() {
|
||||
let home = Scratch::new();
|
||||
let path = home.dir.join("runs.jsonl");
|
||||
let session = jobs::session_id("mail", ts()).unwrap();
|
||||
jobs::append(
|
||||
&path,
|
||||
&jobs::Run {
|
||||
time: ts(),
|
||||
name: "mail".to_string(),
|
||||
session: session.clone(),
|
||||
status: RunStatus::Started,
|
||||
},
|
||||
)
|
||||
.unwrap();
|
||||
let n = jobs::close_leftovers(&path, ts()).unwrap();
|
||||
assert_eq!(n, 1);
|
||||
assert!(jobs::live_session(&path, "mail").unwrap().is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_started_run_without_a_close_is_live() {
|
||||
let home = Scratch::new();
|
||||
let path = home.dir.join("runs.jsonl");
|
||||
let session = jobs::session_id("mail", ts()).unwrap();
|
||||
jobs::append(
|
||||
&path,
|
||||
&jobs::Run {
|
||||
time: ts(),
|
||||
name: "mail".to_string(),
|
||||
session: session.clone(),
|
||||
status: RunStatus::Started,
|
||||
},
|
||||
)
|
||||
.unwrap();
|
||||
assert_eq!(
|
||||
jobs::live_session(&path, "mail").unwrap().as_ref(),
|
||||
Some(&session)
|
||||
);
|
||||
jobs::append(
|
||||
&path,
|
||||
&jobs::Run {
|
||||
time: ts(),
|
||||
name: "mail".to_string(),
|
||||
session,
|
||||
status: RunStatus::Finished,
|
||||
},
|
||||
)
|
||||
.unwrap();
|
||||
assert!(jobs::live_session(&path, "mail").unwrap().is_none());
|
||||
}
|
||||
@@ -0,0 +1,186 @@
|
||||
//! Tests for the cron tick. Do not edit.
|
||||
|
||||
mod support;
|
||||
|
||||
use bxcore::{SessionId, Timestamp};
|
||||
use loopd::channel::Context;
|
||||
use loopd::jobs::{self, RunStatus};
|
||||
use loopd::llama::Client;
|
||||
use loopd::memory::NoMemory;
|
||||
use loopd::scheduler::{Scheduler, TickWork};
|
||||
use loopd::tools::Registry;
|
||||
use std::sync::Arc;
|
||||
use support::{FakeServer, Home, Reply, ScriptedPort};
|
||||
|
||||
const CHAT: &str = "/v1/chat/completions";
|
||||
|
||||
fn sse() -> Reply {
|
||||
let data1 = r#"data: {"choices":[{"finish_reason":null,"index":0,"delta":{"content":"done"}}],"id":"j","timings":{"cache_n":0,"prompt_n":10,"predicted_n":1}}"#;
|
||||
let data2 = r#"data: {"choices":[{"finish_reason":"stop","index":0,"delta":{}}],"id":"j","timings":{"cache_n":0,"prompt_n":10,"predicted_n":1}}"#;
|
||||
let body = format!("{data1}\n\n{data2}\n\ndata: [DONE]\n\n");
|
||||
Reply::raw(format!(
|
||||
"HTTP/1.1 200 OK\r\nContent-Type: text/event-stream\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}",
|
||||
body.len()
|
||||
))
|
||||
}
|
||||
|
||||
fn ts(s: &str) -> Timestamp {
|
||||
Timestamp::parse(s).unwrap()
|
||||
}
|
||||
|
||||
fn world(schedule: &str) -> (Home, FakeServer, Arc<Context>) {
|
||||
let home = Home::new();
|
||||
home.write("compact.md", "Summarise.\n");
|
||||
home.write(
|
||||
"jobs/mail.toml",
|
||||
&format!("schedule = \"{schedule}\"\nprompt = \"Check mail.\"\n"),
|
||||
);
|
||||
let server = FakeServer::start();
|
||||
let cfg = home.config(&server.socket);
|
||||
server.route(CHAT, vec![sse(), sse()]);
|
||||
let ctx = Arc::new(Context::new(
|
||||
cfg.clone(),
|
||||
Client::new(cfg),
|
||||
Box::new(ScriptedPort::new(vec![])),
|
||||
Registry::m2b(),
|
||||
Box::new(NoMemory),
|
||||
));
|
||||
(home, server, ctx)
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_due_job_starts_a_new_session_on_the_background_slot() {
|
||||
let (home, server, ctx) = world("0 7 * * *");
|
||||
let mut sched = Scheduler::new(ts("2026-10-03T06:00:00.000Z"));
|
||||
let work = sched.tick(&ctx, ts("2026-10-03T07:00:00.000Z"));
|
||||
let want = jobs::session_id("mail", ts("2026-10-03T07:00:00.000Z")).unwrap();
|
||||
assert_eq!(
|
||||
work,
|
||||
vec![TickWork::Job {
|
||||
name: "mail".to_string(),
|
||||
session: want.clone()
|
||||
}]
|
||||
);
|
||||
let sent = server.requests_to(CHAT);
|
||||
assert_eq!(sent.len(), 1);
|
||||
assert_eq!(sent[0].json()["id_slot"], 1);
|
||||
let last = sent[0].json()["messages"]
|
||||
.as_array()
|
||||
.unwrap()
|
||||
.last()
|
||||
.unwrap();
|
||||
let content = last["content"].as_str().unwrap();
|
||||
assert!(
|
||||
content.starts_with("It is 2026-10-03T07:00:00.000Z."),
|
||||
"{content}"
|
||||
);
|
||||
assert!(content.contains("Check mail."), "{content}");
|
||||
let s = loopd::session::Session::open(&home.dir, want).unwrap();
|
||||
assert_eq!(s.slot(), 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_job_not_due_does_not_run() {
|
||||
let (_home, server, ctx) = world("0 7 * * *");
|
||||
let mut sched = Scheduler::new(ts("2026-10-03T06:00:00.000Z"));
|
||||
let work = sched.tick(&ctx, ts("2026-10-03T06:59:00.000Z"));
|
||||
assert!(work.is_empty());
|
||||
assert!(server.requests_to(CHAT).is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_live_previous_run_skips_the_tick() {
|
||||
let (home, server, ctx) = world("* * * * *");
|
||||
let ledger = home.dir.join("jobs/runs.jsonl");
|
||||
let previous = jobs::session_id("mail", ts("2026-10-03T06:59:00.000Z")).unwrap();
|
||||
jobs::append(
|
||||
&ledger,
|
||||
&jobs::Run {
|
||||
time: ts("2026-10-03T06:59:00.000Z"),
|
||||
name: "mail".to_string(),
|
||||
session: previous.clone(),
|
||||
status: RunStatus::Started,
|
||||
},
|
||||
)
|
||||
.unwrap();
|
||||
let held = ctx.hold(previous).expect("hold");
|
||||
let mut sched = Scheduler::new(ts("2026-10-03T06:00:00.000Z"));
|
||||
let work = sched.tick(&ctx, ts("2026-10-03T07:00:00.000Z"));
|
||||
assert_eq!(
|
||||
work,
|
||||
vec![TickWork::SkippedJob {
|
||||
name: "mail".to_string()
|
||||
}]
|
||||
);
|
||||
assert!(server.requests_to(CHAT).is_empty());
|
||||
drop(held);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn leftover_started_is_failed_on_the_first_tick() {
|
||||
let (home, _server, ctx) = world("* * * * *");
|
||||
let ledger = home.dir.join("jobs/runs.jsonl");
|
||||
let previous = jobs::session_id("mail", ts("2026-10-03T06:59:00.000Z")).unwrap();
|
||||
jobs::append(
|
||||
&ledger,
|
||||
&jobs::Run {
|
||||
time: ts("2026-10-03T06:59:00.000Z"),
|
||||
name: "mail".to_string(),
|
||||
session: previous.clone(),
|
||||
status: RunStatus::Started,
|
||||
},
|
||||
)
|
||||
.unwrap();
|
||||
let mut sched = Scheduler::new(ts("2026-10-03T07:00:00.000Z"));
|
||||
sched.tick(&ctx, ts("2026-10-03T07:00:00.000Z"));
|
||||
let text = home.read("jobs/runs.jsonl");
|
||||
assert!(
|
||||
text.contains("failed"),
|
||||
"leftover started was closed: {text}"
|
||||
);
|
||||
assert!(text.contains(previous.as_str()), "{text}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_cron_turn_has_no_speaker() {
|
||||
let (home, _server, ctx) = world("0 7 * * *");
|
||||
let mut sched = Scheduler::new(ts("2026-10-03T06:00:00.000Z"));
|
||||
sched.tick(&ctx, ts("2026-10-03T07:00:00.000Z"));
|
||||
let id = jobs::session_id("mail", ts("2026-10-03T07:00:00.000Z")).unwrap();
|
||||
let recs = {
|
||||
let text = home.read(&format!("sessions/{}/0.jsonl", id.as_str()));
|
||||
text.lines()
|
||||
.map(|l| serde_json::from_str::<bxcore::LogRecord>(l).unwrap())
|
||||
.collect::<Vec<_>>()
|
||||
};
|
||||
let user = recs
|
||||
.iter()
|
||||
.find_map(|r| match r {
|
||||
bxcore::LogRecord::User { speaker, .. } => Some(speaker),
|
||||
_ => None,
|
||||
})
|
||||
.unwrap();
|
||||
assert_eq!(user, &None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn two_due_minutes_make_two_sessions() {
|
||||
let (home, server, ctx) = world("* * * * *");
|
||||
let mut sched = Scheduler::new(ts("2026-10-03T06:58:00.000Z"));
|
||||
sched.tick(&ctx, ts("2026-10-03T06:59:00.000Z"));
|
||||
sched.tick(&ctx, ts("2026-10-03T07:00:00.000Z"));
|
||||
assert_eq!(server.requests_to(CHAT).len(), 2);
|
||||
let a = jobs::session_id("mail", ts("2026-10-03T06:59:00.000Z")).unwrap();
|
||||
let b = jobs::session_id("mail", ts("2026-10-03T07:00:00.000Z")).unwrap();
|
||||
assert_ne!(a, b);
|
||||
assert!(
|
||||
home.dir
|
||||
.join(format!("sessions/{}/0.jsonl", a.as_str()))
|
||||
.is_file()
|
||||
);
|
||||
assert!(
|
||||
home.dir
|
||||
.join(format!("sessions/{}/0.jsonl", b.as_str()))
|
||||
.is_file()
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,148 @@
|
||||
//! Tests for the heartbeat tick. Do not edit.
|
||||
|
||||
mod support;
|
||||
|
||||
use bxcore::{SessionId, Timestamp};
|
||||
use loopd::channel::Context;
|
||||
use loopd::llama::Client;
|
||||
use loopd::memory::NoMemory;
|
||||
use loopd::scheduler::{self, Scheduler, TickWork};
|
||||
use loopd::tools::Registry;
|
||||
use std::sync::Arc;
|
||||
use support::{FakeServer, Home, Reply, ScriptedPort};
|
||||
|
||||
const CHAT: &str = "/v1/chat/completions";
|
||||
const PROMPT: &str = "Look around.\n";
|
||||
|
||||
fn sse() -> Reply {
|
||||
let data1 = r#"data: {"choices":[{"finish_reason":null,"index":0,"delta":{"content":"noted"}}],"id":"h","timings":{"cache_n":0,"prompt_n":10,"predicted_n":1}}"#;
|
||||
let data2 = r#"data: {"choices":[{"finish_reason":"stop","index":0,"delta":{}}],"id":"h","timings":{"cache_n":0,"prompt_n":10,"predicted_n":1}}"#;
|
||||
let body = format!("{data1}\n\n{data2}\n\ndata: [DONE]\n\n");
|
||||
Reply::raw(format!(
|
||||
"HTTP/1.1 200 OK\r\nContent-Type: text/event-stream\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}",
|
||||
body.len()
|
||||
))
|
||||
}
|
||||
|
||||
fn ts(s: &str) -> Timestamp {
|
||||
Timestamp::parse(s).unwrap()
|
||||
}
|
||||
|
||||
fn world(every_s: u64) -> (Home, FakeServer, Arc<Context>) {
|
||||
let home = Home::new();
|
||||
home.write("heartbeat.md", PROMPT);
|
||||
home.write("compact.md", "Summarise.\n");
|
||||
let server = FakeServer::start();
|
||||
let mut cfg = home.config(&server.socket);
|
||||
cfg.heartbeat.every_s = every_s;
|
||||
cfg.heartbeat.prompt = home.dir.join("heartbeat.md");
|
||||
server.route(CHAT, vec![sse()]);
|
||||
let ctx = Arc::new(Context::new(
|
||||
cfg.clone(),
|
||||
Client::new(cfg),
|
||||
Box::new(ScriptedPort::new(vec![])),
|
||||
Registry::m2b(),
|
||||
Box::new(NoMemory),
|
||||
));
|
||||
(home, server, ctx)
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn every_s_zero_does_nothing() {
|
||||
let (_home, server, ctx) = world(0);
|
||||
let mut sched = Scheduler::new(ts("2026-10-03T07:00:00.000Z"));
|
||||
let work = sched.tick(&ctx, ts("2026-10-03T08:00:00.000Z"));
|
||||
assert!(work.is_empty());
|
||||
assert!(server.requests_to(CHAT).is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_due_heartbeat_runs_on_the_background_slot_with_a_time_line() {
|
||||
let (home, server, ctx) = world(60);
|
||||
let mut sched = Scheduler::new(ts("2026-10-03T07:00:00.000Z"));
|
||||
let work = sched.tick(&ctx, ts("2026-10-03T07:01:00.000Z"));
|
||||
assert_eq!(work, vec![TickWork::Heartbeat]);
|
||||
let sent = server.requests_to(CHAT);
|
||||
assert_eq!(sent.len(), 1);
|
||||
assert_eq!(sent[0].json()["id_slot"], 1);
|
||||
let last = sent[0].json()["messages"]
|
||||
.as_array()
|
||||
.unwrap()
|
||||
.last()
|
||||
.unwrap();
|
||||
assert_eq!(last["role"], "user");
|
||||
let content = last["content"].as_str().unwrap();
|
||||
assert!(
|
||||
content.starts_with("It is 2026-10-03T07:01:00.000Z."),
|
||||
"{content}"
|
||||
);
|
||||
assert!(content.contains("Look around."), "{content}");
|
||||
let s = loopd::session::Session::open(&home.dir, SessionId::new("heartbeat").unwrap()).unwrap();
|
||||
assert_eq!(s.slot(), 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_second_tick_before_every_s_does_not_run() {
|
||||
let (_home, server, ctx) = world(60);
|
||||
let mut sched = Scheduler::new(ts("2026-10-03T07:00:00.000Z"));
|
||||
sched.tick(&ctx, ts("2026-10-03T07:01:00.000Z"));
|
||||
let work = sched.tick(&ctx, ts("2026-10-03T07:01:30.000Z"));
|
||||
assert!(work.is_empty());
|
||||
assert_eq!(server.requests_to(CHAT).len(), 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_busy_heartbeat_session_is_skipped() {
|
||||
let (_home, server, ctx) = world(60);
|
||||
let held = ctx
|
||||
.hold(SessionId::new("heartbeat").unwrap())
|
||||
.expect("hold");
|
||||
let mut sched = Scheduler::new(ts("2026-10-03T07:00:00.000Z"));
|
||||
let work = sched.tick(&ctx, ts("2026-10-03T07:01:00.000Z"));
|
||||
assert_eq!(
|
||||
work,
|
||||
vec![TickWork::SkippedBusy {
|
||||
session: SessionId::new("heartbeat").unwrap()
|
||||
}]
|
||||
);
|
||||
assert!(server.requests_to(CHAT).is_empty());
|
||||
drop(held);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn an_unreadable_prompt_is_skipped_and_names_the_runbook() {
|
||||
let home = Home::new();
|
||||
let path = home.dir.join("heartbeat.md");
|
||||
std::fs::create_dir(&path).unwrap();
|
||||
let server = FakeServer::start();
|
||||
let mut cfg = home.config(&server.socket);
|
||||
cfg.heartbeat.every_s = 60;
|
||||
cfg.heartbeat.prompt = path;
|
||||
let ctx = Arc::new(Context::new(
|
||||
cfg.clone(),
|
||||
Client::new(cfg),
|
||||
Box::new(ScriptedPort::new(vec![])),
|
||||
Registry::m2b(),
|
||||
Box::new(NoMemory),
|
||||
));
|
||||
let mut sched = Scheduler::new(ts("2026-10-03T07:00:00.000Z"));
|
||||
let work = sched.tick(&ctx, ts("2026-10-03T07:01:00.000Z"));
|
||||
assert_eq!(work, vec![TickWork::HeartbeatUnreadable]);
|
||||
assert!(server.requests_to(CHAT).is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn heartbeat_has_no_speaker() {
|
||||
let (home, _server, ctx) = world(60);
|
||||
let mut sched = Scheduler::new(ts("2026-10-03T07:00:00.000Z"));
|
||||
sched.tick(&ctx, ts("2026-10-03T07:01:00.000Z"));
|
||||
let recs = home.records("heartbeat");
|
||||
let user = recs
|
||||
.iter()
|
||||
.find_map(|r| match r {
|
||||
bxcore::LogRecord::User { speaker, .. } => Some(speaker),
|
||||
_ => None,
|
||||
})
|
||||
.unwrap();
|
||||
assert_eq!(user, &None);
|
||||
}
|
||||
@@ -0,0 +1,170 @@
|
||||
//! Tests that a live turn uses the session's slot. Do not edit.
|
||||
|
||||
mod support;
|
||||
|
||||
use bxcore::SessionId;
|
||||
use loopd::baseline::Baseline;
|
||||
use loopd::llama::Client;
|
||||
use loopd::session::Session;
|
||||
use loopd::tools::Registry;
|
||||
use loopd::turn::{Runtime, run_turn};
|
||||
use support::{FakeServer, Home, Reply};
|
||||
|
||||
const CHAT: &str = "/v1/chat/completions";
|
||||
|
||||
fn sse() -> Reply {
|
||||
let data1 = r#"data: {"choices":[{"finish_reason":null,"index":0,"delta":{"content":"ok"}}],"id":"t","timings":{"cache_n":0,"prompt_n":10,"predicted_n":1}}"#;
|
||||
let data2 = r#"data: {"choices":[{"finish_reason":"stop","index":0,"delta":{}}],"id":"t","timings":{"cache_n":0,"prompt_n":10,"predicted_n":1}}"#;
|
||||
let body = format!("{data1}\n\n{data2}\n\ndata: [DONE]\n\n");
|
||||
Reply::raw(format!(
|
||||
"HTTP/1.1 200 OK\r\nContent-Type: text/event-stream\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}",
|
||||
body.len()
|
||||
))
|
||||
}
|
||||
|
||||
fn runtime<'a>(
|
||||
cfg: &'a loopd::config::Config,
|
||||
client: &'a Client,
|
||||
port: &'a support::ScriptedPort,
|
||||
registry: &'a Registry,
|
||||
) -> Runtime<'a> {
|
||||
Runtime {
|
||||
cfg,
|
||||
client,
|
||||
port,
|
||||
registry,
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_session_created_on_the_background_slot_sends_that_slot() {
|
||||
let home = Home::new();
|
||||
let server = FakeServer::start();
|
||||
let cfg = home.config(&server.socket);
|
||||
server.route(CHAT, vec![sse()]);
|
||||
let client = Client::new(cfg.clone());
|
||||
let port = support::ScriptedPort::new(vec![]);
|
||||
let registry = Registry::m2b();
|
||||
let baseline = Baseline::assemble(&cfg, ®istry).unwrap();
|
||||
let mut s = Session::create(
|
||||
&home.dir,
|
||||
SessionId::new("heartbeat").unwrap(),
|
||||
baseline,
|
||||
cfg.slots.background,
|
||||
)
|
||||
.unwrap();
|
||||
let mut events = Vec::new();
|
||||
run_turn(
|
||||
&mut s,
|
||||
&runtime(&cfg, &client, &port, ®istry),
|
||||
"pulse",
|
||||
None,
|
||||
&mut |e| events.push(e.clone()),
|
||||
)
|
||||
.unwrap();
|
||||
let sent = server.requests_to(CHAT);
|
||||
assert_eq!(sent.len(), 1);
|
||||
assert_eq!(sent[0].json()["id_slot"], cfg.slots.background);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_main_slot_session_still_sends_main() {
|
||||
let home = Home::new();
|
||||
let server = FakeServer::start();
|
||||
let cfg = home.config(&server.socket);
|
||||
server.route(CHAT, vec![sse()]);
|
||||
let client = Client::new(cfg.clone());
|
||||
let port = support::ScriptedPort::new(vec![]);
|
||||
let registry = Registry::m2b();
|
||||
let baseline = Baseline::assemble(&cfg, ®istry).unwrap();
|
||||
let mut s = Session::create(
|
||||
&home.dir,
|
||||
SessionId::new("mm-thread").unwrap(),
|
||||
baseline,
|
||||
cfg.slots.main,
|
||||
)
|
||||
.unwrap();
|
||||
let mut events = Vec::new();
|
||||
run_turn(
|
||||
&mut s,
|
||||
&runtime(&cfg, &client, &port, ®istry),
|
||||
"hi",
|
||||
None,
|
||||
&mut |e| events.push(e.clone()),
|
||||
)
|
||||
.unwrap();
|
||||
assert_eq!(
|
||||
server.requests_to(CHAT)[0].json()["id_slot"],
|
||||
cfg.slots.main
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_request_slot_follows_session_start_not_the_config_main() {
|
||||
let home = Home::new();
|
||||
let server = FakeServer::start();
|
||||
let mut cfg = home.config(&server.socket);
|
||||
cfg.slots.main = 0;
|
||||
cfg.slots.background = 1;
|
||||
server.route(CHAT, vec![sse()]);
|
||||
let client = Client::new(cfg.clone());
|
||||
let port = support::ScriptedPort::new(vec![]);
|
||||
let registry = Registry::m2b();
|
||||
let baseline = Baseline::assemble(&cfg, ®istry).unwrap();
|
||||
let mut s = Session::create(&home.dir, SessionId::new("a").unwrap(), baseline, 7).unwrap();
|
||||
let mut events = Vec::new();
|
||||
run_turn(
|
||||
&mut s,
|
||||
&runtime(&cfg, &client, &port, ®istry),
|
||||
"x",
|
||||
None,
|
||||
&mut |e| events.push(e.clone()),
|
||||
)
|
||||
.unwrap();
|
||||
assert_eq!(server.requests_to(CHAT)[0].json()["id_slot"], 7);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_heartbeat_turn_between_two_main_turns_does_not_change_the_main_prefix() {
|
||||
let home = Home::new();
|
||||
let server = FakeServer::start();
|
||||
let cfg = home.config(&server.socket);
|
||||
server.route(CHAT, vec![sse(), sse(), sse()]);
|
||||
let client = Client::new(cfg.clone());
|
||||
let port = support::ScriptedPort::new(vec![]);
|
||||
let registry = Registry::m2b();
|
||||
let mut main = Session::create(
|
||||
&home.dir,
|
||||
SessionId::new("mm-a").unwrap(),
|
||||
Baseline::assemble(&cfg, ®istry).unwrap(),
|
||||
cfg.slots.main,
|
||||
)
|
||||
.unwrap();
|
||||
let mut beat = Session::create(
|
||||
&home.dir,
|
||||
SessionId::new("heartbeat").unwrap(),
|
||||
Baseline::assemble(&cfg, ®istry).unwrap(),
|
||||
cfg.slots.background,
|
||||
)
|
||||
.unwrap();
|
||||
let rt = runtime(&cfg, &client, &port, ®istry);
|
||||
let mut events = Vec::new();
|
||||
run_turn(&mut main, &rt, "one", None, &mut |e| events.push(e.clone())).unwrap();
|
||||
run_turn(&mut beat, &rt, "pulse", None, &mut |e| {
|
||||
events.push(e.clone())
|
||||
})
|
||||
.unwrap();
|
||||
run_turn(&mut main, &rt, "two", None, &mut |e| events.push(e.clone())).unwrap();
|
||||
let sent = server.requests_to(CHAT);
|
||||
assert_eq!(sent.len(), 3);
|
||||
assert_eq!(sent[0].json()["id_slot"], 0);
|
||||
assert_eq!(sent[1].json()["id_slot"], 1);
|
||||
assert_eq!(sent[2].json()["id_slot"], 0);
|
||||
let first = sent[0].json()["messages"].as_array().unwrap().clone();
|
||||
let third = sent[2].json()["messages"].as_array().unwrap();
|
||||
assert!(
|
||||
third.len() > first.len(),
|
||||
"the main session's second turn is a longer prefix"
|
||||
);
|
||||
assert_eq!(&third[..first.len()], &first[..]);
|
||||
}
|
||||
@@ -0,0 +1,98 @@
|
||||
//! A turn-loop setup shared by the `turn` and `limits` tests: a home, a fake server, a scripted
|
||||
//! port and a registry. Included with `#[path]` because it needs `loopd::turn`, which the earlier
|
||||
//! tasks' tests must not depend on. Do not edit.
|
||||
|
||||
#![allow(dead_code)]
|
||||
|
||||
use crate::support::{FakeServer, Home, ScriptedPort};
|
||||
use bxcore::{LogRecord, SessionId, ToolResponse, TurnEvent};
|
||||
use loopd::baseline::Baseline;
|
||||
use loopd::llama::Client;
|
||||
use loopd::session::Session;
|
||||
use loopd::tools::Registry;
|
||||
use loopd::turn::{Runtime, TurnError, run_turn};
|
||||
|
||||
pub struct Setup {
|
||||
pub home: Home,
|
||||
pub server: FakeServer,
|
||||
pub cfg: loopd::config::Config,
|
||||
pub client: Client,
|
||||
pub port: ScriptedPort,
|
||||
pub registry: Registry,
|
||||
}
|
||||
|
||||
pub fn setup(replies: Vec<ToolResponse>) -> Setup {
|
||||
let home = Home::new();
|
||||
let server = FakeServer::start();
|
||||
let cfg = home.config(&server.socket);
|
||||
let client = Client::new(cfg.clone());
|
||||
Setup {
|
||||
home,
|
||||
server,
|
||||
cfg,
|
||||
client,
|
||||
port: ScriptedPort::new(replies),
|
||||
registry: Registry::m2b(),
|
||||
}
|
||||
}
|
||||
|
||||
impl Setup {
|
||||
pub fn session(&self, id: &str) -> Session {
|
||||
let baseline = Baseline::assemble(&self.cfg, &self.registry).unwrap();
|
||||
Session::create(
|
||||
&self.home.dir,
|
||||
SessionId::new(id).unwrap(),
|
||||
baseline,
|
||||
self.cfg.slots.main,
|
||||
)
|
||||
.unwrap()
|
||||
}
|
||||
|
||||
pub fn runtime(&self) -> Runtime<'_> {
|
||||
Runtime {
|
||||
cfg: &self.cfg,
|
||||
client: &self.client,
|
||||
port: &self.port,
|
||||
registry: &self.registry,
|
||||
}
|
||||
}
|
||||
|
||||
pub fn turn(
|
||||
&self,
|
||||
session: &mut Session,
|
||||
text: &str,
|
||||
) -> (Result<loopd::turn::TurnOutcome, TurnError>, Vec<TurnEvent>) {
|
||||
let mut events = Vec::new();
|
||||
let result = run_turn(session, &self.runtime(), text, None, &mut |e| {
|
||||
events.push(e.clone())
|
||||
});
|
||||
(result, events)
|
||||
}
|
||||
}
|
||||
|
||||
/// The recordings were made in separate conversations, so their timings do not line up and the
|
||||
/// loop rightly reports cache losses between them. Most tests are not about that, so `types`
|
||||
/// and `records` leave `CacheLoss` out; one test checks it on purpose.
|
||||
pub fn without_cache_loss(records: &[LogRecord]) -> Vec<LogRecord> {
|
||||
records
|
||||
.iter()
|
||||
.filter(|r| !matches!(r, LogRecord::CacheLoss { .. }))
|
||||
.cloned()
|
||||
.collect()
|
||||
}
|
||||
|
||||
pub fn types(records: &[LogRecord]) -> Vec<&'static str> {
|
||||
without_cache_loss(records)
|
||||
.iter()
|
||||
.map(|r| match r {
|
||||
LogRecord::SessionStart { .. } => "start",
|
||||
LogRecord::User { .. } => "user",
|
||||
LogRecord::Assistant { .. } => "assistant",
|
||||
LogRecord::Usage { .. } => "usage",
|
||||
LogRecord::ToolResult { .. } => "tool_result",
|
||||
LogRecord::ToolSummary { .. } => "tool_summary",
|
||||
LogRecord::CacheLoss { .. } => "cache_loss",
|
||||
LogRecord::EpochEnd { .. } => "epoch_end",
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
@@ -0,0 +1,241 @@
|
||||
//! Tests for the opted-in tool-output summary. Do not edit.
|
||||
|
||||
mod support;
|
||||
|
||||
use bxcore::{CallId, DataClass, LogRecord, SessionId, ToolCall};
|
||||
use loopd::baseline::Baseline;
|
||||
use loopd::compact;
|
||||
use loopd::llama::Client;
|
||||
use loopd::memory::NoMemory;
|
||||
use loopd::session::Session;
|
||||
use loopd::summarise;
|
||||
use loopd::tools::Registry;
|
||||
use loopd::turn::{Runtime, run_turn};
|
||||
use support::{FakeServer, Home, Reply, ScriptedPort};
|
||||
|
||||
const CHAT: &str = "/v1/chat/completions";
|
||||
const SUMMARISE: &str = "Summarise this tool result.\n";
|
||||
|
||||
fn sse(content: &str) -> Reply {
|
||||
let content_json = serde_json::to_string(content).unwrap();
|
||||
let data1 = format!(
|
||||
r#"data: {{"choices":[{{"finish_reason":null,"index":0,"delta":{{"content":{content_json}}}}}],"id":"s","timings":{{"cache_n":0,"prompt_n":10,"predicted_n":1}}}}"#
|
||||
);
|
||||
let data2 = r#"data: {"choices":[{"finish_reason":"stop","index":0,"delta":{}}],"id":"s","timings":{"cache_n":0,"prompt_n":10,"predicted_n":1}}"#;
|
||||
let body = format!("{data1}\n\n{data2}\n\ndata: [DONE]\n\n");
|
||||
Reply::raw(format!(
|
||||
"HTTP/1.1 200 OK\r\nContent-Type: text/event-stream\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}",
|
||||
body.len()
|
||||
))
|
||||
}
|
||||
|
||||
fn ts() -> bxcore::Timestamp {
|
||||
bxcore::Timestamp::parse("2026-09-18T08:00:00.000Z").unwrap()
|
||||
}
|
||||
|
||||
fn assistant(name: &str) -> LogRecord {
|
||||
LogRecord::Assistant {
|
||||
time: ts(),
|
||||
content: None,
|
||||
reasoning_content: None,
|
||||
tool_calls: vec![ToolCall {
|
||||
id: "call_f".to_string(),
|
||||
name: name.to_string(),
|
||||
arguments: "{}".to_string(),
|
||||
}],
|
||||
}
|
||||
}
|
||||
|
||||
fn result(untrusted: bool) -> LogRecord {
|
||||
LogRecord::ToolResult {
|
||||
time: ts(),
|
||||
call: CallId(1),
|
||||
tool_call_id: "call_f".to_string(),
|
||||
content: "the fetched page is about tea".to_string(),
|
||||
class: DataClass::Public,
|
||||
untrusted,
|
||||
truncated: false,
|
||||
}
|
||||
}
|
||||
|
||||
fn session(home: &Home, cfg: &loopd::config::Config) -> Session {
|
||||
let registry = Registry::m2b();
|
||||
Session::create(
|
||||
&home.dir,
|
||||
SessionId::new("a").unwrap(),
|
||||
Baseline::assemble(cfg, ®istry).unwrap(),
|
||||
cfg.slots.main,
|
||||
)
|
||||
.unwrap()
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn an_opted_in_result_gets_a_summary_on_the_background_slot() {
|
||||
let home = Home::new();
|
||||
home.write("summarise.md", SUMMARISE);
|
||||
let server = FakeServer::start();
|
||||
let mut cfg = home.config(&server.socket);
|
||||
cfg.compact.summarise = vec!["http_fetch".to_string()];
|
||||
server.route(CHAT, vec![sse("tea page")]);
|
||||
let client = Client::new(cfg.clone());
|
||||
let mut s = session(&home, &cfg);
|
||||
s.append(assistant("http_fetch")).unwrap();
|
||||
s.append(result(false)).unwrap();
|
||||
summarise::missing(&cfg, &client, &mut s).unwrap();
|
||||
let sent = server.requests_to(CHAT);
|
||||
assert_eq!(sent.len(), 1);
|
||||
assert_eq!(sent[0].json()["id_slot"], 1);
|
||||
assert!(sent[0].json().get("tools").is_none());
|
||||
assert!(
|
||||
s.records().iter().any(|r| matches!(r, LogRecord::ToolSummary { content, call, untrusted, .. } if content == "tea page" && *call == CallId(1) && !*untrusted)),
|
||||
"{:?}",
|
||||
s.records()
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_summary_copies_the_result_labels() {
|
||||
let home = Home::new();
|
||||
home.write("summarise.md", SUMMARISE);
|
||||
let server = FakeServer::start();
|
||||
let mut cfg = home.config(&server.socket);
|
||||
cfg.compact.summarise = vec!["http_fetch".to_string()];
|
||||
server.route(CHAT, vec![sse("x")]);
|
||||
let client = Client::new(cfg.clone());
|
||||
let mut s = session(&home, &cfg);
|
||||
s.append(assistant("http_fetch")).unwrap();
|
||||
s.append(result(true)).unwrap();
|
||||
summarise::missing(&cfg, &client, &mut s).unwrap();
|
||||
match s.records().last() {
|
||||
Some(LogRecord::ToolSummary {
|
||||
class,
|
||||
untrusted,
|
||||
tool_call_id,
|
||||
..
|
||||
}) => {
|
||||
assert_eq!(*class, DataClass::Public);
|
||||
assert!(*untrusted);
|
||||
assert_eq!(tool_call_id, "call_f");
|
||||
}
|
||||
other => panic!("{other:?}"),
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_tool_not_in_summarise_is_left_alone() {
|
||||
let home = Home::new();
|
||||
home.write("summarise.md", SUMMARISE);
|
||||
let server = FakeServer::start();
|
||||
let mut cfg = home.config(&server.socket);
|
||||
cfg.compact.summarise = vec!["http_fetch".to_string()];
|
||||
let client = Client::new(cfg.clone());
|
||||
let mut s = session(&home, &cfg);
|
||||
s.append(assistant("clock")).unwrap();
|
||||
s.append(result(false)).unwrap();
|
||||
summarise::missing(&cfg, &client, &mut s).unwrap();
|
||||
assert!(server.requests_to(CHAT).is_empty());
|
||||
assert!(
|
||||
!s.records()
|
||||
.iter()
|
||||
.any(|r| matches!(r, LogRecord::ToolSummary { .. }))
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_result_that_already_has_a_summary_is_not_summarised_again() {
|
||||
let home = Home::new();
|
||||
home.write("summarise.md", SUMMARISE);
|
||||
let server = FakeServer::start();
|
||||
let mut cfg = home.config(&server.socket);
|
||||
cfg.compact.summarise = vec!["http_fetch".to_string()];
|
||||
server.route(CHAT, vec![sse("once")]);
|
||||
let client = Client::new(cfg.clone());
|
||||
let mut s = session(&home, &cfg);
|
||||
s.append(assistant("http_fetch")).unwrap();
|
||||
s.append(result(false)).unwrap();
|
||||
summarise::missing(&cfg, &client, &mut s).unwrap();
|
||||
summarise::missing(&cfg, &client, &mut s).unwrap();
|
||||
assert_eq!(server.requests_to(CHAT).len(), 1);
|
||||
assert_eq!(
|
||||
s.records()
|
||||
.iter()
|
||||
.filter(|r| matches!(r, LogRecord::ToolSummary { .. }))
|
||||
.count(),
|
||||
1
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn messages_stay_the_same_after_a_summary() {
|
||||
let home = Home::new();
|
||||
home.write("summarise.md", SUMMARISE);
|
||||
let server = FakeServer::start();
|
||||
let mut cfg = home.config(&server.socket);
|
||||
cfg.compact.summarise = vec!["http_fetch".to_string()];
|
||||
server.route(CHAT, vec![sse("tea page")]);
|
||||
let client = Client::new(cfg.clone());
|
||||
let mut s = session(&home, &cfg);
|
||||
s.append(assistant("http_fetch")).unwrap();
|
||||
s.append(result(false)).unwrap();
|
||||
let before = loopd::baseline::messages(s.baseline(), s.records());
|
||||
summarise::missing(&cfg, &client, &mut s).unwrap();
|
||||
let after = loopd::baseline::messages(s.baseline(), s.records());
|
||||
assert_eq!(before, after);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn after_a_turn_an_opted_in_call_is_summarised() {
|
||||
let home = Home::new();
|
||||
home.write("summarise.md", SUMMARISE);
|
||||
home.write("compact.md", "Summarise.\n");
|
||||
let server = FakeServer::start();
|
||||
let mut cfg = home.config(&server.socket);
|
||||
cfg.compact.summarise = vec!["read_file".to_string()];
|
||||
server.route(
|
||||
CHAT,
|
||||
vec![
|
||||
Reply::fixture("tool_call"),
|
||||
Reply::fixture("plain"),
|
||||
sse("hosts file"),
|
||||
],
|
||||
);
|
||||
let client = Client::new(cfg.clone());
|
||||
let port = ScriptedPort::new(vec![support::ok_result("127.0.0.1 localhost\n")]);
|
||||
let registry = Registry::m2b();
|
||||
let mut s = Session::create(
|
||||
&home.dir,
|
||||
SessionId::new("a").unwrap(),
|
||||
Baseline::assemble(&cfg, ®istry).unwrap(),
|
||||
cfg.slots.main,
|
||||
)
|
||||
.unwrap();
|
||||
let rt = Runtime {
|
||||
cfg: &cfg,
|
||||
client: &client,
|
||||
port: &port,
|
||||
registry: ®istry,
|
||||
};
|
||||
let mut events = Vec::new();
|
||||
let outcome = run_turn(&mut s, &rt, "hosts?", None, &mut |e| events.push(e.clone()));
|
||||
compact::finish_turn(
|
||||
&mut s,
|
||||
&rt,
|
||||
&NoMemory,
|
||||
"hosts?",
|
||||
None,
|
||||
&mut |e| events.push(e.clone()),
|
||||
outcome,
|
||||
)
|
||||
.unwrap();
|
||||
assert!(
|
||||
s.records()
|
||||
.iter()
|
||||
.any(|r| matches!(r, LogRecord::ToolSummary { .. })),
|
||||
"{:?}",
|
||||
s.records()
|
||||
);
|
||||
assert_eq!(
|
||||
server.requests_to(CHAT).last().unwrap().json()["id_slot"],
|
||||
1
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,88 @@
|
||||
//! Tests that `messages()` ignores `ToolSummary`. Do not edit.
|
||||
|
||||
use bxcore::{CallId, DataClass, Epoch, Hash32, LogRecord, SessionId, Timestamp};
|
||||
use loopd::baseline::{Baseline, messages};
|
||||
use loopd::llama::ChatMessage;
|
||||
use loopd::llama::ToolSchema;
|
||||
|
||||
fn ts() -> Timestamp {
|
||||
Timestamp::parse("2026-09-18T08:00:00.000Z").unwrap()
|
||||
}
|
||||
|
||||
fn baseline() -> Baseline {
|
||||
Baseline {
|
||||
system: "sys".to_string(),
|
||||
tools: vec![ToolSchema {
|
||||
name: "clock".to_string(),
|
||||
description: "t".to_string(),
|
||||
parameters: serde_json::json!({"type": "object", "properties": {}}),
|
||||
}],
|
||||
}
|
||||
}
|
||||
|
||||
fn user(content: &str) -> LogRecord {
|
||||
LogRecord::User {
|
||||
time: ts(),
|
||||
content: content.to_string(),
|
||||
speaker: None,
|
||||
}
|
||||
}
|
||||
|
||||
fn result() -> LogRecord {
|
||||
LogRecord::ToolResult {
|
||||
time: ts(),
|
||||
call: CallId(1),
|
||||
tool_call_id: "c".to_string(),
|
||||
content: "raw body".to_string(),
|
||||
class: DataClass::Public,
|
||||
untrusted: false,
|
||||
truncated: false,
|
||||
}
|
||||
}
|
||||
|
||||
fn summary() -> LogRecord {
|
||||
LogRecord::ToolSummary {
|
||||
time: ts(),
|
||||
call: CallId(1),
|
||||
tool_call_id: "c".to_string(),
|
||||
content: "a short account".to_string(),
|
||||
class: DataClass::Public,
|
||||
untrusted: false,
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn messages_ignore_tool_summary() {
|
||||
let start = LogRecord::SessionStart {
|
||||
time: ts(),
|
||||
session: SessionId::new("a").unwrap(),
|
||||
epoch: Epoch(0),
|
||||
slot: 0,
|
||||
baseline: Hash32::ZERO,
|
||||
};
|
||||
let without = messages(&baseline(), &[start.clone(), user("hi"), result()]);
|
||||
let with = messages(&baseline(), &[start, user("hi"), result(), summary()]);
|
||||
assert_eq!(without, with);
|
||||
assert!(
|
||||
!with
|
||||
.iter()
|
||||
.any(|m| matches!(m, ChatMessage::User { content } if content == "a short account")),
|
||||
"{with:?}"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_summary_is_not_replayed_as_a_tool_message() {
|
||||
let msgs = messages(&baseline(), &[summary()]);
|
||||
assert_eq!(msgs.len(), 1, "only the system message");
|
||||
assert!(matches!(msgs[0], ChatMessage::System { .. }));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn inserting_a_summary_does_not_break_prefix_extension() {
|
||||
let a = vec![user("hi"), result()];
|
||||
let b = vec![user("hi"), result(), summary(), user("next")];
|
||||
let ma = messages(&baseline(), &a);
|
||||
let mb = messages(&baseline(), &b);
|
||||
assert_eq!(&mb[..ma.len()], &ma[..]);
|
||||
}
|
||||
@@ -0,0 +1,122 @@
|
||||
//! Tests for `MemoryCommand::Tiers`: identity entries and notes with recall-hit counts. Do not edit.
|
||||
|
||||
#[path = "support/tmp.rs"]
|
||||
mod tmp;
|
||||
|
||||
use bxcore::{DataClass, MemoryFile, Timestamp};
|
||||
use memoryd::files::Store;
|
||||
use memoryd::tiers;
|
||||
use tmp::TempDir;
|
||||
|
||||
fn now() -> Timestamp {
|
||||
Timestamp::parse("2026-10-03T08:00:00.000Z").unwrap()
|
||||
}
|
||||
|
||||
struct Setup {
|
||||
dir: TempDir,
|
||||
store: Store,
|
||||
}
|
||||
|
||||
fn setup(tag: &str) -> Setup {
|
||||
let dir = TempDir::new(tag);
|
||||
let starter = dir.write("etc/soul.md", "Soul.\n");
|
||||
let store = Store::new(&dir.path().join("memory"));
|
||||
store.init(&starter).unwrap();
|
||||
Setup { dir, store }
|
||||
}
|
||||
|
||||
fn write_recall(store: &Store, line: &str) {
|
||||
std::fs::create_dir_all(store.path("recalls")).unwrap();
|
||||
store
|
||||
.append("recalls/2026-10.jsonl", &format!("{line}\n"))
|
||||
.unwrap();
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn identity_entries_list_with_zero_hits() {
|
||||
let s = setup("tiers-id");
|
||||
s.store
|
||||
.write_entries(MemoryFile::Memory, &["I like tea.".to_string()])
|
||||
.unwrap();
|
||||
s.store
|
||||
.write_entries(MemoryFile::User, &["The owner is Kyle.".to_string()])
|
||||
.unwrap();
|
||||
let text = tiers::list(&s.store, now()).unwrap();
|
||||
assert!(
|
||||
text.contains("identity memory.md hits 0 last -"),
|
||||
"{text}"
|
||||
);
|
||||
assert!(text.contains("I like tea."), "{text}");
|
||||
assert!(text.contains("identity user.md hits 0 last -"), "{text}");
|
||||
assert!(text.contains("The owner is Kyle."), "{text}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn notes_carry_counts_from_recalls() {
|
||||
let s = setup("tiers-notes");
|
||||
std::fs::write(
|
||||
s.store.path("notes.md"),
|
||||
"§ noted 2026-09-22T18:20Z in session mm-a · private\nlaunch window moved.\n",
|
||||
)
|
||||
.unwrap();
|
||||
write_recall(
|
||||
&s.store,
|
||||
r#"{"time":"2026-10-03T07:00:00.000Z","session":"mm-now","query":"launch","hits":[{"kind":"noted","session":"mm-a","epoch":0,"record":0}]}"#,
|
||||
);
|
||||
write_recall(
|
||||
&s.store,
|
||||
r#"{"time":"2026-10-03T07:30:00.000Z","session":"mm-now","query":"window","hits":[{"kind":"noted","session":"mm-a","epoch":0,"record":0}]}"#,
|
||||
);
|
||||
let text = tiers::list(&s.store, now()).unwrap();
|
||||
assert!(
|
||||
text.contains(
|
||||
"note session mm-a hits 2 last 2026-10-03T07:30:00.000Z private trusted"
|
||||
),
|
||||
"{text}"
|
||||
);
|
||||
assert!(text.contains("launch window moved."), "{text}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn untrusted_notes_are_listed_and_marked() {
|
||||
let s = setup("tiers-untrusted");
|
||||
std::fs::write(
|
||||
s.store.path("notes.md"),
|
||||
"§ noted 2026-09-22T18:20Z in session mm-c · private · untrusted\nexample.org says the launch moved.\n",
|
||||
)
|
||||
.unwrap();
|
||||
let text = tiers::list(&s.store, now()).unwrap();
|
||||
assert!(
|
||||
text.contains("note session mm-c hits 0 last - private untrusted"),
|
||||
"{text}"
|
||||
);
|
||||
assert!(
|
||||
text.contains("example.org says the launch moved."),
|
||||
"{text}"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_recall_of_a_different_note_does_not_count() {
|
||||
let s = setup("tiers-other");
|
||||
std::fs::write(
|
||||
s.store.path("notes.md"),
|
||||
"§ noted 2026-09-22T18:20Z in session mm-a · private\nfirst note.\n§ noted 2026-09-22T18:21Z in session mm-b · private\nsecond note.\n",
|
||||
)
|
||||
.unwrap();
|
||||
write_recall(
|
||||
&s.store,
|
||||
r#"{"time":"2026-10-03T07:00:00.000Z","session":"mm-now","query":"second","hits":[{"kind":"noted","session":"mm-b","epoch":0,"record":1}]}"#,
|
||||
);
|
||||
let text = tiers::list(&s.store, now()).unwrap();
|
||||
assert!(
|
||||
text.contains("note session mm-a hits 0 last - private trusted"),
|
||||
"{text}"
|
||||
);
|
||||
assert!(
|
||||
text.contains(
|
||||
"note session mm-b hits 1 last 2026-10-03T07:00:00.000Z private trusted"
|
||||
),
|
||||
"{text}"
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,191 @@
|
||||
//! Tests that `ToolSummary` is indexed as kind `summary` and found by `recall`. Do not edit.
|
||||
|
||||
#[path = "support/logs.rs"]
|
||||
mod logs;
|
||||
#[path = "support/tmp.rs"]
|
||||
mod tmp;
|
||||
|
||||
use bxcore::{CallId, DataClass, LogRecord, RecallArgs, Timestamp};
|
||||
use logs::{assistant, result, user, write_log};
|
||||
use memoryd::files::Store;
|
||||
use memoryd::index::Index;
|
||||
use memoryd::recall::recall;
|
||||
use tmp::TempDir;
|
||||
|
||||
fn now() -> Timestamp {
|
||||
Timestamp::parse("2026-09-24T21:00:00.000Z").unwrap()
|
||||
}
|
||||
|
||||
fn summary(minute: u32, content: &str, untrusted: bool) -> LogRecord {
|
||||
LogRecord::ToolSummary {
|
||||
time: logs::at(minute),
|
||||
call: CallId(1),
|
||||
tool_call_id: "c0".to_string(),
|
||||
content: content.to_string(),
|
||||
class: DataClass::Public,
|
||||
untrusted,
|
||||
}
|
||||
}
|
||||
|
||||
struct Setup {
|
||||
dir: TempDir,
|
||||
store: Store,
|
||||
index: Index,
|
||||
}
|
||||
|
||||
fn setup(tag: &str) -> Setup {
|
||||
let dir = TempDir::new(tag);
|
||||
let starter = dir.write("etc/soul.md", "Soul.\n");
|
||||
let store = Store::new(&dir.path().join("memory"));
|
||||
store.init(&starter).unwrap();
|
||||
let index = Index::open(&dir.path().join("index/memory.sqlite3"), DataClass::Private).unwrap();
|
||||
Setup { dir, store, index }
|
||||
}
|
||||
|
||||
fn ask(s: &Setup, query: &str) -> memoryd::recall::Recalled {
|
||||
let args = RecallArgs {
|
||||
query: query.to_string(),
|
||||
k: 10,
|
||||
};
|
||||
recall(&s.index, &s.store, &args, "mm-now", now()).unwrap()
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_tool_summary_is_found_as_kind_summary() {
|
||||
let s = setup("summary-found");
|
||||
write_log(
|
||||
s.dir.path(),
|
||||
"mm-a",
|
||||
&[
|
||||
user(2, "fetch the launch page"),
|
||||
assistant(
|
||||
3,
|
||||
None,
|
||||
&[("http_fetch", r#"{"url":"https://example.org"}"#)],
|
||||
),
|
||||
result(4, "raw html body about tea", DataClass::Public, false),
|
||||
summary(5, "the page is about tea and the launch window", false),
|
||||
],
|
||||
);
|
||||
s.index
|
||||
.catch_up(&s.dir.path().join("sessions"), &s.store.path("notes.md"))
|
||||
.unwrap();
|
||||
let got = ask(&s, "tea");
|
||||
assert!(
|
||||
got.text
|
||||
.contains("the page is about tea and the launch window"),
|
||||
"{got:?}"
|
||||
);
|
||||
assert!(
|
||||
got.text.contains("memory · session mm-a"),
|
||||
"tagged as a session hit: {}",
|
||||
got.text
|
||||
);
|
||||
assert!(!got.untrusted);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_raw_tool_result_is_still_not_indexed() {
|
||||
let s = setup("summary-no-raw");
|
||||
write_log(
|
||||
s.dir.path(),
|
||||
"mm-a",
|
||||
&[
|
||||
user(2, "fetch"),
|
||||
assistant(3, None, &[("http_fetch", "{}")]),
|
||||
result(4, "UNIQUE_RAW_BODY_XYZ", DataClass::Public, false),
|
||||
summary(5, "a short account of the page", false),
|
||||
],
|
||||
);
|
||||
s.index
|
||||
.catch_up(&s.dir.path().join("sessions"), &s.store.path("notes.md"))
|
||||
.unwrap();
|
||||
let got = ask(&s, "UNIQUE_RAW_BODY_XYZ");
|
||||
assert!(
|
||||
got.text.contains("(no matches"),
|
||||
"raw tool result stays unindexed: {}",
|
||||
got.text
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn an_untrusted_summary_taints_the_recall() {
|
||||
let s = setup("summary-taint");
|
||||
write_log(
|
||||
s.dir.path(),
|
||||
"mm-a",
|
||||
&[
|
||||
user(2, "fetch"),
|
||||
assistant(3, None, &[("http_fetch", "{}")]),
|
||||
result(4, "page", DataClass::Public, true),
|
||||
summary(5, "the fetched page mentions the launch window", true),
|
||||
],
|
||||
);
|
||||
s.index
|
||||
.catch_up(&s.dir.path().join("sessions"), &s.store.path("notes.md"))
|
||||
.unwrap();
|
||||
let got = ask(&s, "launch");
|
||||
assert!(got.untrusted, "{got:?}");
|
||||
assert!(got.text.contains("memory-tainted"), "{}", got.text);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_trusted_summary_does_not_taint() {
|
||||
let s = setup("summary-trusted");
|
||||
write_log(
|
||||
s.dir.path(),
|
||||
"mm-a",
|
||||
&[
|
||||
user(2, "fetch"),
|
||||
assistant(3, None, &[("http_fetch", "{}")]),
|
||||
result(4, "page", DataClass::Public, false),
|
||||
summary(5, "the fetched page mentions the launch window", false),
|
||||
],
|
||||
);
|
||||
s.index
|
||||
.catch_up(&s.dir.path().join("sessions"), &s.store.path("notes.md"))
|
||||
.unwrap();
|
||||
let got = ask(&s, "launch");
|
||||
assert!(!got.untrusted, "{got:?}");
|
||||
assert!(got.text.contains("memory · session mm-a"), "{}", got.text);
|
||||
assert!(!got.text.contains("memory-tainted"), "{}", got.text);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_secret_summary_is_skipped_when_the_index_is_private() {
|
||||
let s = setup("summary-secret");
|
||||
write_log(
|
||||
s.dir.path(),
|
||||
"mm-a",
|
||||
&[
|
||||
user(2, "fetch"),
|
||||
assistant(3, None, &[("http_fetch", "{}")]),
|
||||
LogRecord::ToolResult {
|
||||
time: logs::at(4),
|
||||
call: CallId(1),
|
||||
tool_call_id: "c0".to_string(),
|
||||
content: "secret page",
|
||||
class: DataClass::Secret,
|
||||
untrusted: false,
|
||||
truncated: false,
|
||||
},
|
||||
LogRecord::ToolSummary {
|
||||
time: logs::at(5),
|
||||
call: CallId(1),
|
||||
tool_call_id: "c0".to_string(),
|
||||
content: "SECRET_SUMMARY_PHRASE",
|
||||
class: DataClass::Secret,
|
||||
untrusted: false,
|
||||
},
|
||||
],
|
||||
);
|
||||
s.index
|
||||
.catch_up(&s.dir.path().join("sessions"), &s.store.path("notes.md"))
|
||||
.unwrap();
|
||||
let got = ask(&s, "SECRET_SUMMARY_PHRASE");
|
||||
assert!(
|
||||
got.text.contains("(no matches"),
|
||||
"secret summary above max_class: {}",
|
||||
got.text
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
## heartbeat-prompt-unreadable
|
||||
|
||||
**What you see.** `loopd` is serving, but heartbeat turns do not run. It logs this line once,
|
||||
until the file reads:
|
||||
|
||||
```text
|
||||
loopd: heartbeat.md: Permission denied (os error 13); see docs/runbook.md#heartbeat-prompt-unreadable
|
||||
```
|
||||
|
||||
**Why.** `[heartbeat] prompt` is next to the config file (default `heartbeat.md`). The scheduler
|
||||
skips the tick rather than send an empty or guessed user message. A missing file is the same:
|
||||
restore the starter from `config/heartbeat.md` in the repository.
|
||||
|
||||
**Confirm.**
|
||||
|
||||
```sh
|
||||
ls -l "$(dirname "$LOOPD_CONFIG")/heartbeat.md"
|
||||
```
|
||||
|
||||
**Fix.** Restore mode `0644` (or whatever the owner uses for the other prompts) and content.
|
||||
The next tick after a successful read logs nothing extra and runs the standing `heartbeat`
|
||||
session.
|
||||
|
||||
**Check.** `sessions/heartbeat/0.jsonl` gains a new `User` whose text starts with `It is ` and an
|
||||
RFC 3339 timestamp.
|
||||
@@ -0,0 +1,24 @@
|
||||
## jobs-invalid
|
||||
|
||||
**What you see.** `loopd` does not start. It prints a line naming `{home}/jobs/<name>.toml` and
|
||||
this entry:
|
||||
|
||||
```text
|
||||
loopd: jobs/mail.toml: unknown field `when`, expected `schedule` or `prompt`; see docs/runbook.md#jobs-invalid
|
||||
```
|
||||
|
||||
**Why.** Job files are owner-written. `loopd` reads them and cannot write them. Unknown keys, a
|
||||
name that is not `[a-z0-9-]+`, a name so long that `job-<name>-<yyyymmdd>t<hhmm>` is not a valid
|
||||
session id (64 bytes, `[a-z0-9-]`), a schedule that is not five UTC fields, or a missing
|
||||
`prompt`, all refuse start so a misspelt job cannot sit silently.
|
||||
|
||||
**Confirm.**
|
||||
|
||||
```sh
|
||||
ls -l "$BOXMAKER_HOME/jobs/"
|
||||
```
|
||||
|
||||
**Fix.** Rename or edit the named file so it matches the example in `config/jobs/tier.toml.example`.
|
||||
Then start `loopd` again.
|
||||
|
||||
**Check.** `loopd serve` prints `serving on` and `{home}/jobs/runs.jsonl` exists (empty is fine).
|
||||
Reference in New Issue
Block a user