docs: M5d-1 and M5d-2 implementation plans for Ornith

Eight closed tasks for epochs and compaction, then ten for the scheduler, ToolSummary, and tiers. Given tests first. Gate 1456 → 1507 → 1572. Co-Authored-By: Grok 4.6 <noreply@x.ai>
2026-10-02 21:54:37 -07:00
parent 069cd1686a
commit 3726ae585f
48 changed files with 7319 additions and 0 deletions
@@ -0,0 +1,261 @@
+# M5d-1 task 08: compact after a turn, and retry once on `SessionFull`
+
+**Branch:** `m5d1`. Check it: `git branch --show-current` prints `m5d1` and `git status --short`
+is empty; otherwise stop. Never switch branch.
+**Commit subject:** `loopd: compact after a full turn and retry SessionFull once`
+**Log row:** the Task cell is `M5d1/08-loopd-compact-turn`.
+
+## Goal
+
+After `run_turn` returns, `finish_turn` decides whether to compact (spec 3.3):
+
+- A successful turn whose last `Usage` is `due` is compacted before the caller returns. If
+  compact fails, the turn stays successful and the session stays on its epoch; `loopd` logs the
+  compact error with the runbook pointer.
+- `TurnError::SessionFull` is compacted, then that user message is `run_turn`'d once in the new
+  epoch. If compact fails, the error stays `SessionFull`. The retry is not compacted again.
+- Any other `TurnError` is returned as it is. No compact.
+
+`Runtime` does not gain `MemorySource` (given `turn` tests build a `Runtime` without one).
+`finish_turn` takes `memory` as its own argument.
+
+The channel holds the session busy for this whole call: `handle` runs `finish_turn` **before**
+`drop(held)`. A compact is not an idle timer and does not start after the busy set releases.
+
+Live turns still send `slots.main` (task 03). Only the summary call uses the background slot.
+
+## Every place that changes
+
+| Place | What happens there |
+|---|---|
+| `crates/loopd/src/compact.rs` | `finish_turn` and its two helpers (edit F1) |
+| `crates/loopd/src/channel.rs` | `handle` calls `finish_turn` before `drop(held)` (edits H1, H2) |
+| `crates/loopd/tests/compact_turn.rs` | new, given |
+| everything else | **no change**: not `turn.rs`, not `Runtime`, not existing `turn` / `channel` tests |
+
+## Files
+
+- Copy, from `docs/plans/M5d1/files/` (the same path under it): the new test
+  `crates/loopd/tests/compact_turn.rs`.
+- Modify: `crates/loopd/src/compact.rs`, `crates/loopd/src/channel.rs`,
+  `docs/implementer-log.md`.
+- Do not touch: any other file.
+
+## Facts you need
+
+- **`due` looks at `session.last_usage()`**, which is the current epoch's last `Usage`. After a
+  successful `run_turn` that record is there. After `SessionFull`, `run_turn` has already
+  appended the `User` and returned before any `Usage`; compact runs because of the error, not
+  because of `due`.
+- **Compact helper.** `summarise` then `commit`. Log failures with
+
+  ```text
+  loopd: could not compact session <id>: <CompactError Display>
+  ```
+
+  `Display` already includes the runbook pointer, matching the runbook's example line.
+- **Retry.** `run_turn` appends the user message again in the new epoch. The old epoch keeps its
+  `User` (Honcho skips it: no reply). `turns()` is 2. That is accepted (README, known limits).
+- **Existing channel tests stay green.** Their `n_ctx` is 131072 and fixture `prompt_n` is tens,
+  so `due` is false. `SessionFull` without `compact.md` in `Home` stays `SessionFull` (the
+  successful-turn-when-compact-fails behaviour, on the error path).
+- **`compact.rs` stays ≤ 500 lines.** After this task it is about 250.
+- **`channel.rs` stays ≤ 500 lines.** After this task it is about 290.
+- The items the edits use. This table is complete.
+
+  | Item | What it does |
+  |---|---|
+  | `due` | task 05 |
+  | `summarise` / `commit` | tasks 06, 07 |
+  | `run_turn` | retry the same `text` and `speaker` |
+  | `Session::last_usage` | current epoch |
+  | `Config.compact.threshold`, `expect.n_ctx` | the `due` arguments |
+  | `MemorySource` | `finish_turn`'s extra argument; channel passes `&*ctx.memory` |
+  | `Held` | already in `channel.rs`; drop it after `finish_turn` |
+
+## The edits, exactly
+
+Each quoted "before" block occurs exactly once after task 07 (`channel.rs` is still as on
+master, except rustfmt).
+
+### `crates/loopd/src/compact.rs`
+
+**F1.** Extend the crate uses. After task 07 they include `MemorySource`, `Session`, `Config`,
+`Registry`. Add:
+
+```rust
+use crate::turn::{Runtime, TurnError, TurnOutcome, run_turn};
+use bxcore::TurnEvent;
+```
+
+Keep rustfmt's order of `use` lines; if it moves them, accept that.
+
+Then after `commit`, add these three functions whole. There is no `todo!()`.
+
+```rust
+/// After `run_turn`: compact when the last `Usage` is due, or on `SessionFull` (then retry the
+/// user message once). Compact failure on a successful turn keeps the success; on `SessionFull`
+/// keeps `SessionFull`. Other errors skip compact.
+pub fn finish_turn(
+    session: &mut Session,
+    rt: &Runtime<'_>,
+    memory: &dyn MemorySource,
+    text: &str,
+    speaker: Option<&str>,
+    on_event: &mut dyn FnMut(&TurnEvent),
+    outcome: Result<TurnOutcome, TurnError>,
+) -> Result<TurnOutcome, TurnError> {
+    match outcome {
+        Err(TurnError::SessionFull) => match compact_session(session, rt, memory) {
+            Ok(()) => run_turn(session, rt, text, speaker, on_event),
+            Err(e) => {
+                log_compact(session, &e);
+                Err(TurnError::SessionFull)
+            }
+        },
+        Ok(done) => {
+            if let Some(usage) = session.last_usage()
+                && due(&usage, rt.cfg.compact.threshold, rt.cfg.expect.n_ctx)
+                && let Err(e) = compact_session(session, rt, memory)
+            {
+                log_compact(session, &e);
+            }
+            Ok(done)
+        }
+        Err(e) => Err(e),
+    }
+}
+
+fn compact_session(
+    session: &mut Session,
+    rt: &Runtime<'_>,
+    memory: &dyn MemorySource,
+) -> Result<(), CompactError> {
+    let summary = summarise(rt.cfg, rt.client, session)?;
+    commit(session, rt.cfg, rt.registry, memory, summary)
+}
+
+fn log_compact(session: &Session, e: &CompactError) {
+    eprintln!(
+        "loopd: could not compact session {}: {e}",
+        session.id().as_str()
+    );
+}
+```
+
+The `Ok(done)` arm uses `&& let Err(e) = ...` so a compact failure is logged and the original
+`done` is still returned. Do not `?` that error.
+
+### `crates/loopd/src/channel.rs`
+
+**H1.** The `run_turn` call and the `drop(held)` that follows it. The lines
+
+```rust
+    let outcome = run_turn(
+        &mut session,
+        &runtime,
+        &turn.content,
+        turn.speaker.as_deref(),
+        &mut |event| {
+            let _ = write_frame(
+                &mut stream,
+                &Envelope {
+                    v: PROTOCOL_VERSION,
+                    id: request.id,
+                    r#final: false,
+                    msg: Message::TurnEvent(event.clone()),
+                },
+            );
+        },
+    );
+
+    // 5. Release the session before the final frame, so the next turn does not find it busy.
+    drop(held);
+```
+
+become
+
+```rust
+    let mut emit = |event: &bxcore::TurnEvent| {
+        let _ = write_frame(
+            &mut stream,
+            &Envelope {
+                v: PROTOCOL_VERSION,
+                id: request.id,
+                r#final: false,
+                msg: Message::TurnEvent(event.clone()),
+            },
+        );
+    };
+
+    let outcome = run_turn(
+        &mut session,
+        &runtime,
+        &turn.content,
+        turn.speaker.as_deref(),
+        &mut emit,
+    );
+    let outcome = crate::compact::finish_turn(
+        &mut session,
+        &runtime,
+        &*ctx.memory,
+        &turn.content,
+        turn.speaker.as_deref(),
+        &mut emit,
+        outcome,
+    );
+
+    // 5. Release the session before the final frame, so the next turn does not find it busy.
+    //    Compaction ran inside the hold: a second turn on this session waits until the new
+    //    epoch is on disk (or compact failed and the epoch is unchanged).
+    drop(held);
+```
+
+**H2.** Do not add `MemorySource` to `Runtime`. Do not change `turn_code`. The `memory.updated`
+block after `drop(held)` stays: a successful compact-and-retry still reports `session.turns()`,
+which now spans epochs.
+
+## Steps
+
+- [ ] **1. Copy the given test.**
+
+  ```sh
+  cp docs/plans/M5d1/files/crates/loopd/tests/compact_turn.rs crates/loopd/tests/
+  ```
+
+- [ ] **2. See it fail.** `cargo build --tests --keep-going -p loopd`, timeout 600000 ms: test
+  target `compact_turn` does not compile (`E0599` no function `finish_turn`). Every other
+  target compiles, including `turn` and `channel`.
+- [ ] **3. The edits.** F1, H1. `cargo check -p loopd` prints no error and no warning.
+  `wc -l crates/loopd/src/compact.rs` and `wc -l crates/loopd/src/channel.rs` each print a
+  number ≤ 500.
+- [ ] **4. The tests.** `cargo test -p loopd --test compact_turn --test turn --test channel
+  --test compact_advance`: every test passes, 6 in `compact_turn` and the others as before.
+- [ ] **5. Check.** `git diff --stat` lists exactly `crates/loopd/src/compact.rs` and
+  `crates/loopd/src/channel.rs`. `git status --short` shows `compact_turn.rs` as new.
+- [ ] **6. Run the gate.** `cargo fmt --all`, then `make gate`, timeout 600000 ms. Last line
+  `gate: ok`. Test count **1507** (1501 before, plus 6).
+- [ ] **7. Log and commit.** Task cell `M5d1/08-loopd-compact-turn`. Stage with
+  `git add crates/loopd docs/implementer-log.md Cargo.lock`, then commit as `AGENTS.md` says.
+
+## Done when
+
+- `make gate` prints `gate: ok` with 1507 tests.
+- `git diff HEAD~1 --stat` lists only `crates/loopd/src/compact.rs`,
+  `crates/loopd/src/channel.rs`, `crates/loopd/tests/compact_turn.rs` and the log.
+
+## Stop and report if
+
+- The branch check fails; a "before" block is not found exactly once; step 2 differs; `cargo
+  check` warns; a test fails; the gate count is not 1507; a file exceeds 500 lines; you add
+  `MemorySource` to `Runtime`; you compact inside `run_turn`; you call `finish_turn` after
+  `drop(held)`; you add an idle timer; you compact a `TurnError` other than `SessionFull`; you
+  retry `SessionFull` more than once.
+
+## What the given tests pin
+
+Usage 849/1000 leaves epoch 0 (one chat); 850/1000 compact before return (live slot then
+background, no tools on the summary); `SessionFull` compact-and-retry (three chats, user logged
+in both epochs, `turns() == 2`); `SessionFull` stays `SessionFull` when compact fails; a
+successful due turn stays successful when `compact.md` is missing (epoch stays 0); another
+infer error is not compacted.