M5a review: tasks 01 to 22 and the device check; tips I16, I17, T34 to T40

Two review entries in the implementer log: tasks 01 to 15 (the header
forgery, review ids, the runbook entries, log rows written from memory)
and tasks 16 to 22 with the device check on straylight (the ledger
raising state by the grant's label, the note answers, the stale target/
that stalled task 21, the clippy 1.95 stop, the fake podman race). Two
short rules go into AGENTS.md: stop after one check when a test outside
the task fails, and write the log row from the diff.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

docs/implementer-lessons.md

@@ -30,7 +30,9 @@ How it is used:
 | I12 | A file that exists but cannot be read is an error. Only a file that does not exist may be treated as absent. | M2b finding 3. | yes | |
 | I13 | Before calling a failure "environmental", measure the thing you blame, and name the machine, OS and file system the evidence came from. A sample that shows a thread inside a system call shows where the time goes, not that the call is stuck. | M3a stop before task 20: an fsync "stall" was blamed from `__fcntl` frames that were ordinary 4 ms `F_FULLFSYNC` calls on the Mac, and the real cause, a macOS socket rule, went unfixed. | no | |
 | I14 | Recovering from a crash is a case to write a test for, not only to reason about. A file the daemon itself can leave behind (created, not yet written) must load like an empty one. | M3a finding 1: one zero-length log file made `brokerd` panic at startup, where the same directory verified as `ok, 0 records`. |
-| I15 | Never write the model's chat-template tags out whole (the think and tool-call tags, the angle-bar turn markers) in code, comments, commands or the log. Build them from pieces (`concat!("<", "think")`) and name them in words. Text holding them turns into the model's own control tokens when it reads it. | M5a task 04: the given test held the tags; Ornith's reasoning ended at the test file's closing think tag, and it lost track of the file and made no change. `docs/research/2026-09-24-chat-template-tokens.md`. | yes | |
+| I15 | Never write the model's chat-template tags out whole (the think and tool-call tags, the angle-bar turn markers) in code, comments, commands or the log. Build them from pieces (`concat!("<", "think")`) and name them in words. Text holding them turns into the model's own control tokens when it reads it. | M5a task 04: the given test held the tags; Ornith's reasoning ended at the test file's closing think tag, and it lost track of the file and made no change. `docs/research/2026-09-24-chat-template-tokens.md`. | yes | M5a tasks 05 to 22: not seen again. |
+| I16 | When a test outside your task fails, check once whether it fails without your change (stash, run, unstash), then stop and report both results. Do not keep experimenting, and never add a file the task does not list, not even a temporary one. | M5a task 21: the session did the check, then spent about 80 minutes on experiments and wrote two diagnostic test files into `crates/bxctl/tests/` until the owner stopped the run; the cause was outside the repository (review finding 4). M5a task 16 did it right: one check, its log row, a stop. | yes | |
+| I17 | Write your log row from `git diff` after staging, naming only what the diff shows. A row written from memory describes what you meant to do. | M5a task 10's row describes checks that are not in `serve.rs`; task 13's says two runbook fragments were copied, and neither was (review findings 5 and 6 of 2026-09-24). | yes | |
 
 ## Tips for writing tasks
 
@@ -69,6 +71,13 @@ How it is used:
 | T31 | In a spec, for every post or message the harness sends, name who can read it and whether that reader can act on it. Walk each kind of sender through each kind of channel (the owner, an allowed non-approver, a stranger; direct, group, channel). | M4b: an approval asked by a non-approver in a direct message went into that direct message, where no approver could see it. Every test passed, because the tests used the spec's cases; the device check with a second account found it (review finding 1, fixed by tasks 17 and 18). |
 | T32 | A given test must not assert the order of two things that run on their own threads. Check the order where it is decided (here, the sort in the catch-up parser), and compare what the threads deliver as a set. Before hand-over, stress each test with concurrency under load (`yes > /dev/null` beside a loop of 60 runs). | M4a's `serve_restart` expected two caught-up sessions' turns to reach `loopd` in order. They run on separate threads, and under load 1 run in 60 failed. M4b task 18's session hit it, proved it happens without its change, and reported it instead of editing the test. |
 | T33 | A file the implementer reads must never hold the model's chat-template tags literally: not in skeletons, given tests, fixtures, task text or specs it may open. Build them from pieces and name them in words. grep the plan for the tags before hand-over. | M5a task 04 (above); the measurements in `docs/research/2026-09-24-chat-template-tokens.md`: the think and tool-call tags are one token whatever the tokenizer's flag, and inside a chat message the turn markers become control tokens. |
+| T34 | Replay the plan with the toolchain of the host that will run it, and write the `rustc` and `clippy` versions in the plan. Each clippy release adds lints, and the workspace's `rust-version` is only a floor. | M5a task 16 was replayed on talos; on titan, clippy 1.95 failed the gate on `collapsible_match` in code from task 08, and the session stopped (review finding 6 of 2026-09-25). |
+| T35 | A scratch worktree for a reference or a replay builds into its own `target/`, never the main repository's. Test binaries carry absolute paths (`CARGO_BIN_EXE_*`, `CARGO_MANIFEST_DIR`), and Cargo's freshness check ignores the workspace's path, so the main tree reuses them. If one ever shared it, run `cargo clean` before the implementer starts. | M5a task 21: the gate failed with `NotFound` in `args_os`, and in most suites with `--no-fail-fast`, after the reference worktree was deleted; the run stalled (review finding 4). |
+| T36 | When a milestone makes a value vary that used to be fixed, list every place that reads it and give each a test with the new variation. A spec's "as today" about existing code is a claim to check against the code, not a reassurance. | M5a: results gained labels of their own. The runner and the audit record used them; the ledger kept raising the session's state by the grant's label, as it always had, and the spec said "as today". The tests checked the response and the audit record, not the state (device finding 1). |
+| T37 | For every trust boundary in a spec, write two attacks as tests: untrusted text shaped like trusted structure, and a trusted reader that takes in an untrusted result and keeps its trust. Both M5a highs were one of these, and the reference had both. | M5a: a note forging a `notes.md` header (review finding 1 of 2026-09-24); a clean session recalling a tainted note and writing it into identity (device finding 1). |
+| T38 | A tool's answer that the model relays must say the outcome in plain words, including what did not happen ("not added to memory.md: …"). A label in parentheses reads as success. | M5a device check: the model twice told the owner "Added to memory.md" for a write that became an untrusted note. After task 22 it reported it correctly. |
+| T39 | Name the source directory of copied files in every task's Files section. | M5a task 06 logged a deviation about a files directory the task never named. |
+| T40 | Give the gate's shell timeout in the task (600000 ms). On a machine that also serves the model, the gate takes longer than the implementer's default of 2 minutes, and a timed-out gate looks like a failure. | M5a tasks 18 and 19 each ran the gate twice on titan; tasks 21 and 22, which gave the timeout, ran it once. |
 
 ## What worked and should be kept
 
@@ -81,6 +90,9 @@ How it is used:
   edited tests and no touched design documents across nine tasks.
 - One task, one fresh session, one commit, with only listed paths staged. The history reads as the
   plan.
+- A device check against the real model and the owner's server before landing. M4b's non-approver
+  approval and M5a's path from a fetched page into identity were found only there; tests,
+  reference and review had all passed them.
 
 ## What we have seen of the implementers so far
 
@@ -113,3 +125,11 @@ above low. Given a skeleton whose comments name the calls and expressions, it wr
 and its code agreed with the reference on every fuzzed input. Given a `todo!()` with many branches,
 or a task that needs the whole crate's API, it plans in prose until the turn runs out; given a
 comment that contradicts its function, it notices but deliberates rather than stopping (T29).
+
+M5a was Ornith through the driver, twenty-two tasks over two machines: first gate 7 of 14 on the
+main run, and four of the five follow-ups on their first run. Its code matched the reference on
+every random input outside two findings, and every follow-up was exact. Its misses were in what it
+wrote about the code, not the code: two log rows describe work not in the diff, and one runbook
+change was its own text where the task gave a fragment (I17). It stopped correctly on a failure
+outside its task once (task 16), and once kept experimenting for over an hour (task 21, I16). The
+milestone's two serious defects were the spec's, and the reference had them too (T36, T37).