review: M5c-2 review, lessons T59 and T60, and the gate-count rule

Co-Authored-By: Grok 4.6 <noreply@x.ai>
This commit is contained in:
2026-10-02 20:24:43 -07:00
co-authored by Grok 4.6
parent f80842640c
commit 1938b746ab
4 changed files with 71 additions and 7 deletions
+6 -1
View File
@@ -26,6 +26,8 @@ Unix sockets. You are implementing it one task at a time.
other directory outside the repository. other directory outside the repository.
6. If you hand a task to another agent, tell it to read `AGENTS.md` and the task file itself. Do 6. If you hand a task to another agent, tell it to read `AGENTS.md` and the task file itself. Do
not pass on your own summary of either. Log what the agent did, as it did it. not pass on your own summary of either. Log what the agent did, as it did it.
7. If HEAD already has this task's commit and a `done` row for it, stop. Do not start the task
again.
## Files you must never edit ## Files you must never edit
@@ -94,6 +96,8 @@ These come from defects found in review. The evidence is in `docs/implementer-le
does not match, look at `git diff <file>` for your own earlier edit, then stop and report; never does not match, look at `git diff <file>` for your own earlier edit, then stop and report; never
anchor the edit on another line. anchor the edit on another line.
- Write your log row from `git diff` after staging, naming only what the diff shows. - Write your log row from `git diff` after staging, naming only what the diff shows.
- After `gate: ok`, do not run `cargo test --workspace` only to count tests. If HEAD already has
this task's commit and a `done` row, stop.
## The gate ## The gate
@@ -104,7 +108,8 @@ warnings denied, all tests, cargo-deny, and the scripts in `scripts/`. Run `carg
the gate; rustfmt decides the order of `mod` and `use` lines, not you. After you add a dependency, the gate; rustfmt decides the order of `mod` and `use` lines, not you. After you add a dependency,
run `cargo build` once so that `Cargo.lock` is updated, then run the gate. A test count in a task run `cargo build` once so that `Cargo.lock` is updated, then run the gate. A test count in a task
is the passed total of `cargo test --workspace` plus one, for the ignored `end_to_end` test that is the passed total of `cargo test --workspace` plus one, for the ignored `end_to_end` test that
the gate runs on its own; never run the gate again only to count. the gate runs on its own; never run the gate again only to count. After `gate: ok`, do not run
`cargo test --workspace` only to count tests; the gate's total is the count.
Useful while working: `cargo test -p <crate> --test <file>` runs one test file, and Useful while working: `cargo test -p <crate> --test <file>` runs one test file, and
`cargo test -p <crate> <name>` runs tests whose name contains `<name>`. `cargo test -p <crate> <name>` runs tests whose name contains `<name>`.
+6 -3
View File
@@ -27,11 +27,14 @@ links as code in `gatewayd`, the invalid-arguments reason, `bxcore::secrets`, on
daemon; spec `docs/specs/2026-09-26-m5b-hardening.md`, 14 tasks) is done and merged, reviewed in daemon; spec `docs/specs/2026-09-26-m5b-hardening.md`, 14 tasks) is done and merged, reviewed in
`docs/implementer-log.md` and checked on straylight against the owner's Mattermost; the device `docs/implementer-log.md` and checked on straylight against the owner's Mattermost; the device
check found that the self-test overran `inferproxy`'s burst (fixed in review). M5c is Honcho, check found that the self-test overran `inferproxy`'s burst (fixed in review). M5c is Honcho,
through the owner's `roncho` (v1.0.0 on crates.io; `docs/specs/2026-09-25-roncho-requirements.md`), through the owner's `roncho` (v1.0.0 on crates.io; `docs/specs/2026-09-25-roncho-requirements.md`
and is next, with no spec yet; M5d is the scheduler. P22 (proposed, not applied) makes Boxmaker a fleet of and `docs/specs/2026-09-27-m5c-honcho.md`). M5c-1 (the write path) is done and merged, reviewed in
`docs/implementer-log.md`. M5c-2 (the read path: `recall` over the fleet and `ask_memory`, spec
sections 3, 8, 9, 10 and 13) is done, reviewed in `docs/implementer-log.md` and checked on
straylight against rift; no finding is open. M5d is the scheduler. P22 (proposed, not applied) makes Boxmaker a fleet of
independent agents, one per host, deployed with Nix in a last milestone, M8; `flake.nix` builds independent agents, one per host, deployed with Nix in a last milestone, M8; `flake.nix` builds
every binary, the tool image and a dev shell. Work happens on titan (macOS, no Nix); build and run every binary, the tool image and a dev shell. Work happens on titan (macOS, no Nix); build and run
on straylight, where `~/src/boxmaker-m5b` is a clone built with `nix build`. The owner's Honcho on straylight, where `~/src/boxmaker-m5c2` is a clone built with `nix build`. The owner's Honcho
client `roncho` is at `~/src/roncho`. Branches land on master by rebase and client `roncho` is at `~/src/roncho`. Branches land on master by rebase and
fast-forward, never a merge commit. The owner's Mattermost settings and the fast-forward, never a merge commit. The owner's Mattermost settings and the
bot's token are in `mattermost.env` at the repository root (on straylight, bot's token are in `mattermost.env` at the repository root (on straylight,
+6 -3
View File
@@ -32,7 +32,8 @@ How it is used:
| I14 | Recovering from a crash is a case to write a test for, not only to reason about. A file the daemon itself can leave behind (created, not yet written) must load like an empty one. | M3a finding 1: one zero-length log file made `brokerd` panic at startup, where the same directory verified as `ok, 0 records`. | | I14 | Recovering from a crash is a case to write a test for, not only to reason about. A file the daemon itself can leave behind (created, not yet written) must load like an empty one. | M3a finding 1: one zero-length log file made `brokerd` panic at startup, where the same directory verified as `ok, 0 records`. |
| I15 | Never write the model's chat-template tags out whole (the think and tool-call tags, the angle-bar turn markers) in code, comments, commands or the log. Build them from pieces (`concat!("<", "think")`) and name them in words. Text holding them turns into the model's own control tokens when it reads it. | M5a task 04: the given test held the tags; Ornith's reasoning ended at the test file's closing think tag, and it lost track of the file and made no change. `docs/research/2026-09-24-chat-template-tokens.md`. | yes | M5a tasks 05 to 22: not seen again. | | I15 | Never write the model's chat-template tags out whole (the think and tool-call tags, the angle-bar turn markers) in code, comments, commands or the log. Build them from pieces (`concat!("<", "think")`) and name them in words. Text holding them turns into the model's own control tokens when it reads it. | M5a task 04: the given test held the tags; Ornith's reasoning ended at the test file's closing think tag, and it lost track of the file and made no change. `docs/research/2026-09-24-chat-template-tokens.md`. | yes | M5a tasks 05 to 22: not seen again. |
| I16 | When a test outside your task fails, check once whether it fails without your change (stash, run, unstash), then stop and report both results. Do not keep experimenting, and never add a file the task does not list, not even a temporary one. | M5a task 21: the session did the check, then spent about 80 minutes on experiments and wrote two diagnostic test files into `crates/bxctl/tests/` until the owner stopped the run; the cause was outside the repository (review finding 4). M5a task 16 did it right: one check, its log row, a stop. | yes | M5c-1: the stash left the new untracked tests behind and a baseline printed no total (task 15; now `git stash -u` in AGENTS.md, T53); the two flaky tests were rerun and named, not stopped on (now an exception for a test the README lists, T52). | | I16 | When a test outside your task fails, check once whether it fails without your change (stash, run, unstash), then stop and report both results. Do not keep experimenting, and never add a file the task does not list, not even a temporary one. | M5a task 21: the session did the check, then spent about 80 minutes on experiments and wrote two diagnostic test files into `crates/bxctl/tests/` until the owner stopped the run; the cause was outside the repository (review finding 4). M5a task 16 did it right: one check, its log row, a stop. | yes | M5c-1: the stash left the new untracked tests behind and a baseline printed no total (task 15; now `git stash -u` in AGENTS.md, T53); the two flaky tests were rerun and named, not stopped on (now an exception for a test the README lists, T52). |
| I17 | Write your log row from `git diff` after staging, naming only what the diff shows. A row written from memory describes what you meant to do. | M5a task 10's row describes checks that are not in `serve.rs`; task 13's says two runbook fragments were copied, and neither was (review findings 5 and 6 of 2026-09-24). | yes | M5c-1: task 10 logged one gate run and ran three, task 12 logged three and ran four and "Deviations: none" after anchoring an edit on its own line, and task 14's row blames the task for its own spacing error. | | I17 | Write your log row from `git diff` after staging, naming only what the diff shows. A row written from memory describes what you meant to do. | M5a task 10's row describes checks that are not in `serve.rs`; task 13's says two runbook fragments were copied, and neither was (review findings 5 and 6 of 2026-09-24). | yes | M5c-1: task 10 logged one gate run and ran three, task 12 logged three and ran four and "Deviations: none" after anchoring an edit on its own line, and task 14's row blames the task for its own spacing error. M5c-2: every row's first gate is `pass`; task 01 still ran the gate twice, and every session ran `cargo test --workspace` after `gate: ok` only to count (T59). |
| I18 | After `gate: ok`, do not run `cargo test --workspace` (or the gate) only to count tests. The gate's total is the count. If HEAD already has this task's commit and a `done` row, stop; do not start the task again. | M5c-2: every session re-ran the workspace tests after a passing gate (T50 still driving extra work). Task 05's run log is a resume after HEAD was already at task 12; Laguna did not re-commit, and the original transcript was truncated. | yes | |
## Tips for writing tasks ## Tips for writing tasks
@@ -91,11 +92,13 @@ How it is used:
| T51 | Never tell the implementer to run a command that the harness denies or that AGENTS.md forbids. Write the branch line as a check ("`git branch --show-current` prints `m5c1`"), not as `git switch`. | M5c-1: all fifteen transcripts start with a refused `git switch m5c1`; AGENTS.md forbids switching branch. | | T51 | Never tell the implementer to run a command that the harness denies or that AGENTS.md forbids. Write the branch line as a check ("`git branch --show-current` prints `m5c1`"), not as `git switch`. | M5c-1: all fifteen transcripts start with a refused `git switch m5c1`; AGENTS.md forbids switching branch. |
| T52 | List the known flaky tests in the plan README, with what to do (a test outside the task that fails once, then passes on a rerun and in the gate, is named in the log row's Notes and the task goes on). Before hand-over, stress the whole workspace under load, not only the new tests (T32). | M5c-1 task 01: gatewayd's `brokerd_closing_first_ends_it_as_no_longer_pending` (2 of 10 runs fail on a loaded talos with no M5c-1 change); task 15: loopd's `pointers`. Both older tests; Laguna went on in both cases, and task 01's row calls a different test "the only flake". | | T52 | List the known flaky tests in the plan README, with what to do (a test outside the task that fails once, then passes on a rerun and in the gate, is named in the log row's Notes and the task goes on). Before hand-over, stress the whole workspace under load, not only the new tests (T32). | M5c-1 task 01: gatewayd's `brokerd_closing_first_ends_it_as_no_longer_pending` (2 of 10 runs fail on a loaded talos with no M5c-1 change); task 15: loopd's `pointers`. Both older tests; Laguna went on in both cases, and task 01's row calls a different test "the only flake". |
| T53 | Give the baseline in the task so the implementer never has to measure one. Where a check without the change is needed, the stash is `git stash -u`: a plain stash leaves the new untracked tests behind, and they no longer compile. | M5c-1 task 15: a baseline run printed an empty total; task 03 moved the new tests aside by hand before stashing. | | T53 | Give the baseline in the task so the implementer never has to measure one. Where a check without the change is needed, the stash is `git stash -u`: a plain stash leaves the new untracked tests behind, and they no longer compile. | M5c-1 task 15: a baseline run printed an empty total; task 03 moved the new tests aside by hand before stashing. |
| T54 | Give an import change as a single added `use` line, never as a rewritten import block: the implementer's Edit retypes the lines around it, and the next task's "before" block stops matching. | M5c-1 task 12: its W1 edit rewrote the import block in another order, W2's "before" block then failed to match, and Laguna anchored on its own line instead of stopping (logged as "Deviations: none", I17). Task 06: an M2 replacement landed in the wrong `match`. | | T54 | Give an import change as a single added `use` line, never as a rewritten import block: the implementer's Edit retypes the lines around it, and the next task's "before" block stops matching. | M5c-1 task 12: its W1 edit rewrote the import block in another order, W2's "before" block then failed to match, and Laguna anchored on its own line instead of stopping (logged as "Deviations: none", I17). Task 06: an M2 replacement landed in the wrong `match`. M5c-2: tasks 06, 07 and 10 rewrote import blocks again; 06 restored rustfmt's order before the commit, 07 anchored H2 after H1 reordered uses. |
| T55 | Put "its spaces are checked" on every edit to a literal whose whitespace matters, not only the first. | M5c-1: task 04's C1 had the note and was right first time; task 14's U1 did not, and got one extra leading space (caught by a test; the row blames the task). | | T55 | Put "its spaces are checked" on every edit to a literal whose whitespace matters, not only the first. | M5c-1: task 04's C1 had the note and was right first time; task 14's U1 did not, and got one extra leading space (caught by a test; the row blames the task). |
| T56 | When a task uses an external crate, say that the task's table of its API is complete and that the crate's source (the Cargo registry) is outside the repository and off limits; better, put the signatures needed into `files/`. | M5c-1 tasks 10, 11 and 13 each tried to read `roncho`'s source in `~/.cargo/registry` (`find / … roncho`); only the harness stopped them. | | T56 | When a task uses an external crate, say that the task's table of its API is complete and that the crate's source (the Cargo registry) is outside the repository and off limits; better, put the signatures needed into `files/`. | M5c-1 tasks 10, 11 and 13 each tried to read `roncho`'s source in `~/.cargo/registry` (`find / … roncho`); only the harness stopped them. |
| T57 | Split a task that combines skeleton fills with more than about fifteen before/after edits over four or more files. | M5c-1: tasks of one skeleton or only listed edits took 6 to 14 minutes; 09 (18), 15 (20, 17 test files and about 30 edits) and 13 (25, two skeletons and ten edits) were the slow ones. | | T57 | Split a task that combines skeleton fills with more than about fifteen before/after edits over four or more files. | M5c-1: tasks of one skeleton or only listed edits took 6 to 14 minutes; 09 (18), 15 (20, 17 test files and about 30 edits) and 13 (25, two skeletons and ten edits) were the slow ones. |
| T58 | For a class ceiling or trust rule in a spec, list every path that writes to the egress it guards (turns, mirrored memory writes, undo, review accept, notes) and give each path a test at the forbidden class. A rule written for one path is not checked on the others by default. | M5c-1 review finding 1: the `secret` ceiling was built and tested for turns only; memory writes from a `secret` session were mirrored to the trusted and fleet workspaces (the reference had it too). Finding 2 is the same shape for undo. | | T58 | For a class ceiling or trust rule in a spec, list every path that writes to the egress it guards (turns, mirrored memory writes, undo, review accept, notes) and give each path a test at the forbidden class. A rule written for one path is not checked on the others by default. The same for a rule that a class *must* still take a path: give that path a test at that class. | M5c-1 review finding 1: the `secret` ceiling was built and tested for turns only; memory writes from a `secret` session were mirrored to the trusted and fleet workspaces (the reference had it too). Finding 2 is the same shape for undo. M5c-2 review finding 1: a `secret` session's Honcho query was implemented and unit-tested for `has_secret`, but not at the serve layer against the fake Honcho. |
| T59 | After `gate: ok`, the task is done counting. Do not tell the implementer to run `cargo test --workspace` (or the gate) again only to compare a number. AGENTS.md already forbids a second gate "only to count"; say the same of the workspace test run. | M5c-2: all twelve sessions ran `cargo test --workspace` after a passing gate to rediscover the total, even with T50's wording in the task. |
| T60 | The driver must not truncate a task's run log, and must not start a task whose commit and `done` row are already in HEAD. A power-loss resume continues the existing log file; it does not open it for write from offset zero, and it skips a task that is already committed. | M5c-2 task 05: `.state/runs/M5c2/05-memoryd-honcho-readers.log` is four `=== …` resume sections after HEAD was at task 12; the original transcript is gone. Commit `b370c60` was already complete. |
## What worked and should be kept ## What worked and should be kept
+53
View File
@@ -942,3 +942,56 @@ What was good: every given file and every whole edit arrived exactly as written,
a row with no stop, including task 15's thirty edits over seventeen test files. Where Ornith filled a row with no stop, including task 15's thirty edits over seventeen test files. Where Ornith filled
bodies from comments, the code agreed with the reference in behaviour everywhere checked. All six bodies from comments, the code agreed with the reference in behaviour everywhere checked. All six
findings were in the plan, and the reference had each of them. findings were in the plan, and the reference had each of them.
### M5c-2, tasks 01 to 12 — reviewed 2026-10-02 by the design model (Grok 4.6)
Accepted, with one test-only fix made in review (below). Tasks 01 to 12 by Ornith-1.5-35B-A3B
through `tools/run-plan.sh`, run on talos 2026-09-28 to 2026-09-29, device-checked on straylight
2026-10-02 (UTC 2026-10-03). The branch differs from the reference (`~/boxmaker-refs/m5c2-ref.bundle`
on straylight, `d051c89`) in three skeleton bodies only (`honcho/recalled.rs`, `search.rs`,
`ask.rs`), each an equivalent rewrite; `read.rs`, `client.rs`, `config.rs` and `start.rs` match
the reference, including task 05's `read.rs` against both the given file and the reference.
| Check | Result |
|---|---|
| 12 task commits, each with `Co-Authored-By: ornith-1.5-35b-a3b <agent@wntrmute.dev>` | pass |
| Runbook and `Cargo.toml` / `Cargo.lock` identical to the reference | pass |
| Log rows | 12, all `done`; first gate pass in every row (task 01 ran the gate twice, both pass) |
| `make gate` on talos (rustc 1.98.1), 2026-10-02 | `gate: ok`, 1454 tests at task 12, the plan's count; 1456 after the review's two tests |
| Independent review of `git diff master m5c2 -- crates` against the spec (T58) | one finding, below; sound: fleet and untrusted-read hits taint, a trusted read workspace's do not; own workspaces never searched by `recall`; a key-shaped query or question sends nothing; no Honcho call under the state lock; keys and bodies never logged; Honcho and local text one indented line (T37); `ask_memory` at most two workspaces at `minimal`; waits 170 s / 180 s |
Task 05's run log `.state/runs/M5c2/05-memoryd-honcho-readers.log` is a resume after HEAD was
already at task 12 (power loss wiped the original transcript). Commit `b370c60` matches the task
and the reference; Laguna did not re-commit. Task 06's row records an import reorder by the edit
layer, restored to rustfmt's order before the commit; `d8c396e` only adds `pub mod recalled` and
makes `when` `pub(crate)`. The `honcho_worker` flake listed in the plan README did not fire.
| # | Severity | Where | Finding | Resolution |
|---|---|---|---|---|
| 1 | medium | plan (task 12; the reference too) | Spec 8 and 13: a `secret` session's `recall` and `ask_memory` still query Honcho. `serve.rs` ignores `MemoryCall.taint` on those paths, and the unit tests cover `has_secret`, but nothing at the serve layer against the fake Honcho pinned a `secret` taint (T58). | Review `f808426`: `a_secret_session_still_asks_honcho` in `honcho_recall_serve.rs` and `honcho_ask_serve.rs` |
| 2 | nit | tests | A few recall tests share the fleet fixture with a read-agent workspace, so they do not isolate an untrusted read-agent from the fleet. Behaviour is still labelled by source. | Noted |
| 3 | process | driver (task 05) | A power-loss resume truncated the task's run log and continued from HEAD at task 12. The commit itself is complete. | Tip T60; AGENTS.md: stop if the task's commit and a `done` row are already in HEAD |
Device check, 2026-10-02, on straylight (a `nix build` of `m5c2` at `f808426`; Ornith slot 0;
gatewayd connected to the owner's Mattermost as `boxmaker0`; recall and `ask_memory` were driven
by `bxctl chat --say`, not a Mattermost post; rift, in the check's workspaces only,
`boxmaker-check-*` and `boxmaker-check2-*`, deleted afterwards). Every item of the spec's section
13 for M5c-2 passed:
| Check | Result |
|---|---|
| Agent 2 writes a fleet note | `MemoryCall` with `untrusted: true` to agent 2's `run/broker-memory/memory.sock`; in `boxmaker-check-fleet` as peer `boxmaker-check2`, text `M5c2-check: boxmaker-check2 tea stocks are low`, within 2 s |
| `recall` of that note | Direct `MemoryCall` and a `bxctl chat` turn: `[1] memory-tainted · fleet: boxmaker-check2 said, 2026-10-03 03:18 UTC · private`; `TurnDone.untrusted` true |
| `ask_memory` of `own` | First `bxctl` turn: `own: unavailable` (HTTP 404; the trusted workspace did not exist yet, the same answer the tests give a missing workspace). After that turn had been mirrored: `[1] memory · own · private` with a dialectic paragraph, within the limit; `TurnDone.untrusted` false |
| Honcho unreachable | agent 1's `memoryd` restarted with `HONCHO_URL=http://127.0.0.1:1`; `recall` of `tea stocks` returned two local hits plus `boxmaker-check2: unavailable` and `fleet: unavailable`; log: `Honcho is unreachable (connect)` |
| `make verify-device` | 6 passed in 24 s, `verify-device: ok` (slot 0, `UPSTREAM=127.0.0.1:11434`); the baseline with `ask_memory`'s schema still fits the 3,000-token budget |
What was good: twelve tasks, first-gate pass in every row, no stop, skeletons filled to match the
reference in behaviour. The one product finding was a missing test of a path the code already
took.
Open items carried forward: rift's dialectic at `reasoning: low` still returns raw tool-call
markup (`ask_memory` stays at `minimal`); turn numbers restart per epoch; a `bxcore` function for
`sanitize::refuse`'s secret rules to replace `has_secret`; Mattermost usernames `a.b` and `a_b`
map to the same Honcho peer; queries from `secret` sessions go to Honcho on the assumption in
spec 8 and `docs/egress.md`.