docs: M5d-2: after gate: ok, do not sum test results

Task 01 told Ornith to add up every `test result:` N passed. A greedy
sed kept the last digit (1512 became 1022) and it stopped with the
correct T1 and given test uncommitted. T50/T59: the last line of
make gate is the count; summing is not a stop.

Co-Authored-By: Grok 4.6 <noreply@x.ai>
This commit is contained in:
2026-10-03 12:56:14 -07:00
co-authored by Grok 4.6
parent ccf56c8aa7
commit 2a48b7f36d
11 changed files with 34 additions and 26 deletions
+1 -1
View File
@@ -89,7 +89,7 @@ How it is used:
| T47 | Name the log row in the task ("the row's Task cell is `M5b/14-runbook`"). The driver checks the row by the file's name; the implementer takes it from the task's heading. | M5b task 14: the row was `M5b/14-runbook-sanitizer-rules`, from the heading "the runbook entry `sanitizer-rules`", and `run-plan.sh` stopped on a finished task. |
| T48 | Run the implementer with no plugins and no borrowed rules: `opencode run --pure` and `OPENCODE_DISABLE_CLAUDE_CODE=1`. Global plugins load into every session, and a plugin that rewrites the system prompt forces a full re-read of the conversation on a model that cannot rewind its cache. | M5b run (2026-09-26): a short-term-memory plugin rewrote the system prompt every fourth request (a 35,633-token re-read in task 10), 22 plugin tools rode in every request (about 2,400 tokens), and in tasks 01 to 10 prompt reads of 10,000 tokens or more took 76 of 111 minutes of prompt processing: 42 of them, of which only about 11 were session starts (about 15,000 tokens each). Tasks 11 to 14, run after the plugins were removed, had 5 such reads (4 session starts), 4 minutes in all. |
| T49 | When a task adds requests on a path through another role, count them against that role's limits (`inferproxy`: 8 connections at once, a burst of 10, then 2 a second), and put the real proxy in at least one given test. A fake server reached directly cannot show a limit on the way to it. | M5b task 08: 66 `/tokenize` requests at startup passed every test and failed on straylight's first start, refused by `inferproxy`'s token bucket. |
| T50 | When a task gives a gate test count, give the command that produces it and say what it includes (the passed total of `cargo test --workspace` plus the ignored `end_to_end` test the gate runs on its own). Otherwise, do not make the count a stop condition. | M5c-1 tasks 01, 03, 13 and 15: each session found itself one short ("1129 passed — expected 1130") and rediscovered the rule. The count drove extra gate runs, scratch files in `/tmp` (02, 03, 05, 10, 13) and `git stash` baselines (03, 15); about a quarter of the transcripts of 03, 13 and 15 went on it. |
| T50 | When a task gives a gate test count, give the command that produces it and say what it includes (the passed total of `cargo test --workspace` plus the ignored `end_to_end` test the gate runs on its own). Otherwise, do not make the count a stop condition. Do not ask the implementer to add up `test result:` `N passed` lines: a greedy `.*([0-9]+)` keeps only the last digit (1512 becomes ~1022). After `gate: ok`, counting is done (T59). | M5c-1 tasks 01, 03, 13 and 15: each session found itself one short ("1129 passed — expected 1130") and rediscovered the rule. The count drove extra gate runs, scratch files in `/tmp` (02, 03, 05, 10, 13) and `git stash` baselines (03, 15); about a quarter of the transcripts of 03, 13 and 15 went on it. M5d-2 task 01: summed with that sed, reported 1022, stopped, left T1 and the given test uncommitted; `run-plan` then stopped on the dirty tree. |
| T51 | Never tell the implementer to run a command that the harness denies or that AGENTS.md forbids. Write the branch line as a check ("`git branch --show-current` prints `m5c1`"), not as `git switch`. | M5c-1: all fifteen transcripts start with a refused `git switch m5c1`; AGENTS.md forbids switching branch. |
| T52 | List the known flaky tests in the plan README, with what to do (a test outside the task that fails once, then passes on a rerun and in the gate, is named in the log row's Notes and the task goes on). Before hand-over, stress the whole workspace under load, not only the new tests (T32). | M5c-1 task 01: gatewayd's `brokerd_closing_first_ends_it_as_no_longer_pending` (2 of 10 runs fail on a loaded talos with no M5c-1 change); task 15: loopd's `pointers`. Both older tests; Laguna went on in both cases, and task 01's row calls a different test "the only flake". |
| T53 | Give the baseline in the task so the implementer never has to measure one. Where a check without the change is needed, the stash is `git stash -u`: a plain stash leaves the new untracked tests behind, and they no longer compile. | M5c-1 task 15: a baseline run printed an empty total; task 03 moved the new tests aside by hand before stashing. |