review: M5c-2 review, lessons T59 and T60, and the gate-count rule
Co-Authored-By: Grok 4.6 <noreply@x.ai>
This commit is contained in:
@@ -26,6 +26,8 @@ Unix sockets. You are implementing it one task at a time.
|
||||
other directory outside the repository.
|
||||
6. If you hand a task to another agent, tell it to read `AGENTS.md` and the task file itself. Do
|
||||
not pass on your own summary of either. Log what the agent did, as it did it.
|
||||
7. If HEAD already has this task's commit and a `done` row for it, stop. Do not start the task
|
||||
again.
|
||||
|
||||
## Files you must never edit
|
||||
|
||||
@@ -94,6 +96,8 @@ These come from defects found in review. The evidence is in `docs/implementer-le
|
||||
does not match, look at `git diff <file>` for your own earlier edit, then stop and report; never
|
||||
anchor the edit on another line.
|
||||
- Write your log row from `git diff` after staging, naming only what the diff shows.
|
||||
- After `gate: ok`, do not run `cargo test --workspace` only to count tests. If HEAD already has
|
||||
this task's commit and a `done` row, stop.
|
||||
|
||||
## The gate
|
||||
|
||||
@@ -104,7 +108,8 @@ warnings denied, all tests, cargo-deny, and the scripts in `scripts/`. Run `carg
|
||||
the gate; rustfmt decides the order of `mod` and `use` lines, not you. After you add a dependency,
|
||||
run `cargo build` once so that `Cargo.lock` is updated, then run the gate. A test count in a task
|
||||
is the passed total of `cargo test --workspace` plus one, for the ignored `end_to_end` test that
|
||||
the gate runs on its own; never run the gate again only to count.
|
||||
the gate runs on its own; never run the gate again only to count. After `gate: ok`, do not run
|
||||
`cargo test --workspace` only to count tests; the gate's total is the count.
|
||||
|
||||
Useful while working: `cargo test -p <crate> --test <file>` runs one test file, and
|
||||
`cargo test -p <crate> <name>` runs tests whose name contains `<name>`.
|
||||
|
||||
@@ -27,11 +27,14 @@ links as code in `gatewayd`, the invalid-arguments reason, `bxcore::secrets`, on
|
||||
daemon; spec `docs/specs/2026-09-26-m5b-hardening.md`, 14 tasks) is done and merged, reviewed in
|
||||
`docs/implementer-log.md` and checked on straylight against the owner's Mattermost; the device
|
||||
check found that the self-test overran `inferproxy`'s burst (fixed in review). M5c is Honcho,
|
||||
through the owner's `roncho` (v1.0.0 on crates.io; `docs/specs/2026-09-25-roncho-requirements.md`),
|
||||
and is next, with no spec yet; M5d is the scheduler. P22 (proposed, not applied) makes Boxmaker a fleet of
|
||||
through the owner's `roncho` (v1.0.0 on crates.io; `docs/specs/2026-09-25-roncho-requirements.md`
|
||||
and `docs/specs/2026-09-27-m5c-honcho.md`). M5c-1 (the write path) is done and merged, reviewed in
|
||||
`docs/implementer-log.md`. M5c-2 (the read path: `recall` over the fleet and `ask_memory`, spec
|
||||
sections 3, 8, 9, 10 and 13) is done, reviewed in `docs/implementer-log.md` and checked on
|
||||
straylight against rift; no finding is open. M5d is the scheduler. P22 (proposed, not applied) makes Boxmaker a fleet of
|
||||
independent agents, one per host, deployed with Nix in a last milestone, M8; `flake.nix` builds
|
||||
every binary, the tool image and a dev shell. Work happens on titan (macOS, no Nix); build and run
|
||||
on straylight, where `~/src/boxmaker-m5b` is a clone built with `nix build`. The owner's Honcho
|
||||
on straylight, where `~/src/boxmaker-m5c2` is a clone built with `nix build`. The owner's Honcho
|
||||
client `roncho` is at `~/src/roncho`. Branches land on master by rebase and
|
||||
fast-forward, never a merge commit. The owner's Mattermost settings and the
|
||||
bot's token are in `mattermost.env` at the repository root (on straylight,
|
||||
|
||||
@@ -32,7 +32,8 @@ How it is used:
|
||||
| I14 | Recovering from a crash is a case to write a test for, not only to reason about. A file the daemon itself can leave behind (created, not yet written) must load like an empty one. | M3a finding 1: one zero-length log file made `brokerd` panic at startup, where the same directory verified as `ok, 0 records`. |
|
||||
| I15 | Never write the model's chat-template tags out whole (the think and tool-call tags, the angle-bar turn markers) in code, comments, commands or the log. Build them from pieces (`concat!("<", "think")`) and name them in words. Text holding them turns into the model's own control tokens when it reads it. | M5a task 04: the given test held the tags; Ornith's reasoning ended at the test file's closing think tag, and it lost track of the file and made no change. `docs/research/2026-09-24-chat-template-tokens.md`. | yes | M5a tasks 05 to 22: not seen again. |
|
||||
| I16 | When a test outside your task fails, check once whether it fails without your change (stash, run, unstash), then stop and report both results. Do not keep experimenting, and never add a file the task does not list, not even a temporary one. | M5a task 21: the session did the check, then spent about 80 minutes on experiments and wrote two diagnostic test files into `crates/bxctl/tests/` until the owner stopped the run; the cause was outside the repository (review finding 4). M5a task 16 did it right: one check, its log row, a stop. | yes | M5c-1: the stash left the new untracked tests behind and a baseline printed no total (task 15; now `git stash -u` in AGENTS.md, T53); the two flaky tests were rerun and named, not stopped on (now an exception for a test the README lists, T52). |
|
||||
| I17 | Write your log row from `git diff` after staging, naming only what the diff shows. A row written from memory describes what you meant to do. | M5a task 10's row describes checks that are not in `serve.rs`; task 13's says two runbook fragments were copied, and neither was (review findings 5 and 6 of 2026-09-24). | yes | M5c-1: task 10 logged one gate run and ran three, task 12 logged three and ran four and "Deviations: none" after anchoring an edit on its own line, and task 14's row blames the task for its own spacing error. |
|
||||
| I17 | Write your log row from `git diff` after staging, naming only what the diff shows. A row written from memory describes what you meant to do. | M5a task 10's row describes checks that are not in `serve.rs`; task 13's says two runbook fragments were copied, and neither was (review findings 5 and 6 of 2026-09-24). | yes | M5c-1: task 10 logged one gate run and ran three, task 12 logged three and ran four and "Deviations: none" after anchoring an edit on its own line, and task 14's row blames the task for its own spacing error. M5c-2: every row's first gate is `pass`; task 01 still ran the gate twice, and every session ran `cargo test --workspace` after `gate: ok` only to count (T59). |
|
||||
| I18 | After `gate: ok`, do not run `cargo test --workspace` (or the gate) only to count tests. The gate's total is the count. If HEAD already has this task's commit and a `done` row, stop; do not start the task again. | M5c-2: every session re-ran the workspace tests after a passing gate (T50 still driving extra work). Task 05's run log is a resume after HEAD was already at task 12; Laguna did not re-commit, and the original transcript was truncated. | yes | |
|
||||
|
||||
## Tips for writing tasks
|
||||
|
||||
@@ -91,11 +92,13 @@ How it is used:
|
||||
| T51 | Never tell the implementer to run a command that the harness denies or that AGENTS.md forbids. Write the branch line as a check ("`git branch --show-current` prints `m5c1`"), not as `git switch`. | M5c-1: all fifteen transcripts start with a refused `git switch m5c1`; AGENTS.md forbids switching branch. |
|
||||
| T52 | List the known flaky tests in the plan README, with what to do (a test outside the task that fails once, then passes on a rerun and in the gate, is named in the log row's Notes and the task goes on). Before hand-over, stress the whole workspace under load, not only the new tests (T32). | M5c-1 task 01: gatewayd's `brokerd_closing_first_ends_it_as_no_longer_pending` (2 of 10 runs fail on a loaded talos with no M5c-1 change); task 15: loopd's `pointers`. Both older tests; Laguna went on in both cases, and task 01's row calls a different test "the only flake". |
|
||||
| T53 | Give the baseline in the task so the implementer never has to measure one. Where a check without the change is needed, the stash is `git stash -u`: a plain stash leaves the new untracked tests behind, and they no longer compile. | M5c-1 task 15: a baseline run printed an empty total; task 03 moved the new tests aside by hand before stashing. |
|
||||
| T54 | Give an import change as a single added `use` line, never as a rewritten import block: the implementer's Edit retypes the lines around it, and the next task's "before" block stops matching. | M5c-1 task 12: its W1 edit rewrote the import block in another order, W2's "before" block then failed to match, and Laguna anchored on its own line instead of stopping (logged as "Deviations: none", I17). Task 06: an M2 replacement landed in the wrong `match`. |
|
||||
| T54 | Give an import change as a single added `use` line, never as a rewritten import block: the implementer's Edit retypes the lines around it, and the next task's "before" block stops matching. | M5c-1 task 12: its W1 edit rewrote the import block in another order, W2's "before" block then failed to match, and Laguna anchored on its own line instead of stopping (logged as "Deviations: none", I17). Task 06: an M2 replacement landed in the wrong `match`. M5c-2: tasks 06, 07 and 10 rewrote import blocks again; 06 restored rustfmt's order before the commit, 07 anchored H2 after H1 reordered uses. |
|
||||
| T55 | Put "its spaces are checked" on every edit to a literal whose whitespace matters, not only the first. | M5c-1: task 04's C1 had the note and was right first time; task 14's U1 did not, and got one extra leading space (caught by a test; the row blames the task). |
|
||||
| T56 | When a task uses an external crate, say that the task's table of its API is complete and that the crate's source (the Cargo registry) is outside the repository and off limits; better, put the signatures needed into `files/`. | M5c-1 tasks 10, 11 and 13 each tried to read `roncho`'s source in `~/.cargo/registry` (`find / … roncho`); only the harness stopped them. |
|
||||
| T57 | Split a task that combines skeleton fills with more than about fifteen before/after edits over four or more files. | M5c-1: tasks of one skeleton or only listed edits took 6 to 14 minutes; 09 (18), 15 (20, 17 test files and about 30 edits) and 13 (25, two skeletons and ten edits) were the slow ones. |
|
||||
| T58 | For a class ceiling or trust rule in a spec, list every path that writes to the egress it guards (turns, mirrored memory writes, undo, review accept, notes) and give each path a test at the forbidden class. A rule written for one path is not checked on the others by default. | M5c-1 review finding 1: the `secret` ceiling was built and tested for turns only; memory writes from a `secret` session were mirrored to the trusted and fleet workspaces (the reference had it too). Finding 2 is the same shape for undo. |
|
||||
| T58 | For a class ceiling or trust rule in a spec, list every path that writes to the egress it guards (turns, mirrored memory writes, undo, review accept, notes) and give each path a test at the forbidden class. A rule written for one path is not checked on the others by default. The same for a rule that a class *must* still take a path: give that path a test at that class. | M5c-1 review finding 1: the `secret` ceiling was built and tested for turns only; memory writes from a `secret` session were mirrored to the trusted and fleet workspaces (the reference had it too). Finding 2 is the same shape for undo. M5c-2 review finding 1: a `secret` session's Honcho query was implemented and unit-tested for `has_secret`, but not at the serve layer against the fake Honcho. |
|
||||
| T59 | After `gate: ok`, the task is done counting. Do not tell the implementer to run `cargo test --workspace` (or the gate) again only to compare a number. AGENTS.md already forbids a second gate "only to count"; say the same of the workspace test run. | M5c-2: all twelve sessions ran `cargo test --workspace` after a passing gate to rediscover the total, even with T50's wording in the task. |
|
||||
| T60 | The driver must not truncate a task's run log, and must not start a task whose commit and `done` row are already in HEAD. A power-loss resume continues the existing log file; it does not open it for write from offset zero, and it skips a task that is already committed. | M5c-2 task 05: `.state/runs/M5c2/05-memoryd-honcho-readers.log` is four `=== …` resume sections after HEAD was at task 12; the original transcript is gone. Commit `b370c60` was already complete. |
|
||||
|
||||
## What worked and should be kept
|
||||
|
||||
|
||||
@@ -942,3 +942,56 @@ What was good: every given file and every whole edit arrived exactly as written,
|
||||
a row with no stop, including task 15's thirty edits over seventeen test files. Where Ornith filled
|
||||
bodies from comments, the code agreed with the reference in behaviour everywhere checked. All six
|
||||
findings were in the plan, and the reference had each of them.
|
||||
|
||||
### M5c-2, tasks 01 to 12 — reviewed 2026-10-02 by the design model (Grok 4.6)
|
||||
|
||||
Accepted, with one test-only fix made in review (below). Tasks 01 to 12 by Ornith-1.5-35B-A3B
|
||||
through `tools/run-plan.sh`, run on talos 2026-09-28 to 2026-09-29, device-checked on straylight
|
||||
2026-10-02 (UTC 2026-10-03). The branch differs from the reference (`~/boxmaker-refs/m5c2-ref.bundle`
|
||||
on straylight, `d051c89`) in three skeleton bodies only (`honcho/recalled.rs`, `search.rs`,
|
||||
`ask.rs`), each an equivalent rewrite; `read.rs`, `client.rs`, `config.rs` and `start.rs` match
|
||||
the reference, including task 05's `read.rs` against both the given file and the reference.
|
||||
|
||||
| Check | Result |
|
||||
|---|---|
|
||||
| 12 task commits, each with `Co-Authored-By: ornith-1.5-35b-a3b <agent@wntrmute.dev>` | pass |
|
||||
| Runbook and `Cargo.toml` / `Cargo.lock` identical to the reference | pass |
|
||||
| Log rows | 12, all `done`; first gate pass in every row (task 01 ran the gate twice, both pass) |
|
||||
| `make gate` on talos (rustc 1.98.1), 2026-10-02 | `gate: ok`, 1454 tests at task 12, the plan's count; 1456 after the review's two tests |
|
||||
| Independent review of `git diff master m5c2 -- crates` against the spec (T58) | one finding, below; sound: fleet and untrusted-read hits taint, a trusted read workspace's do not; own workspaces never searched by `recall`; a key-shaped query or question sends nothing; no Honcho call under the state lock; keys and bodies never logged; Honcho and local text one indented line (T37); `ask_memory` at most two workspaces at `minimal`; waits 170 s / 180 s |
|
||||
|
||||
Task 05's run log `.state/runs/M5c2/05-memoryd-honcho-readers.log` is a resume after HEAD was
|
||||
already at task 12 (power loss wiped the original transcript). Commit `b370c60` matches the task
|
||||
and the reference; Laguna did not re-commit. Task 06's row records an import reorder by the edit
|
||||
layer, restored to rustfmt's order before the commit; `d8c396e` only adds `pub mod recalled` and
|
||||
makes `when` `pub(crate)`. The `honcho_worker` flake listed in the plan README did not fire.
|
||||
|
||||
| # | Severity | Where | Finding | Resolution |
|
||||
|---|---|---|---|---|
|
||||
| 1 | medium | plan (task 12; the reference too) | Spec 8 and 13: a `secret` session's `recall` and `ask_memory` still query Honcho. `serve.rs` ignores `MemoryCall.taint` on those paths, and the unit tests cover `has_secret`, but nothing at the serve layer against the fake Honcho pinned a `secret` taint (T58). | Review `f808426`: `a_secret_session_still_asks_honcho` in `honcho_recall_serve.rs` and `honcho_ask_serve.rs` |
|
||||
| 2 | nit | tests | A few recall tests share the fleet fixture with a read-agent workspace, so they do not isolate an untrusted read-agent from the fleet. Behaviour is still labelled by source. | Noted |
|
||||
| 3 | process | driver (task 05) | A power-loss resume truncated the task's run log and continued from HEAD at task 12. The commit itself is complete. | Tip T60; AGENTS.md: stop if the task's commit and a `done` row are already in HEAD |
|
||||
|
||||
Device check, 2026-10-02, on straylight (a `nix build` of `m5c2` at `f808426`; Ornith slot 0;
|
||||
gatewayd connected to the owner's Mattermost as `boxmaker0`; recall and `ask_memory` were driven
|
||||
by `bxctl chat --say`, not a Mattermost post; rift, in the check's workspaces only,
|
||||
`boxmaker-check-*` and `boxmaker-check2-*`, deleted afterwards). Every item of the spec's section
|
||||
13 for M5c-2 passed:
|
||||
|
||||
| Check | Result |
|
||||
|---|---|
|
||||
| Agent 2 writes a fleet note | `MemoryCall` with `untrusted: true` to agent 2's `run/broker-memory/memory.sock`; in `boxmaker-check-fleet` as peer `boxmaker-check2`, text `M5c2-check: boxmaker-check2 tea stocks are low`, within 2 s |
|
||||
| `recall` of that note | Direct `MemoryCall` and a `bxctl chat` turn: `[1] memory-tainted · fleet: boxmaker-check2 said, 2026-10-03 03:18 UTC · private`; `TurnDone.untrusted` true |
|
||||
| `ask_memory` of `own` | First `bxctl` turn: `own: unavailable` (HTTP 404; the trusted workspace did not exist yet, the same answer the tests give a missing workspace). After that turn had been mirrored: `[1] memory · own · private` with a dialectic paragraph, within the limit; `TurnDone.untrusted` false |
|
||||
| Honcho unreachable | agent 1's `memoryd` restarted with `HONCHO_URL=http://127.0.0.1:1`; `recall` of `tea stocks` returned two local hits plus `boxmaker-check2: unavailable` and `fleet: unavailable`; log: `Honcho is unreachable (connect)` |
|
||||
| `make verify-device` | 6 passed in 24 s, `verify-device: ok` (slot 0, `UPSTREAM=127.0.0.1:11434`); the baseline with `ask_memory`'s schema still fits the 3,000-token budget |
|
||||
|
||||
What was good: twelve tasks, first-gate pass in every row, no stop, skeletons filled to match the
|
||||
reference in behaviour. The one product finding was a missing test of a path the code already
|
||||
took.
|
||||
|
||||
Open items carried forward: rift's dialectic at `reasoning: low` still returns raw tool-call
|
||||
markup (`ask_memory` stays at `minimal`); turn numbers restart per epoch; a `bxcore` function for
|
||||
`sanitize::refuse`'s secret rules to replace `has_secret`; Mattermost usernames `a.b` and `a_b`
|
||||
map to the same Honcho peer; queries from `secret` sessions go to Honcho on the assumption in
|
||||
spec 8 and `docs/egress.md`.
|
||||
|
||||
Reference in New Issue
Block a user