From 2a48b7f36d84576e1ce028808735a81d12715fa7 Mon Sep 17 00:00:00 2001 From: Kyle Isom Date: Sat, 3 Oct 2026 12:56:14 -0700 Subject: [PATCH] docs: M5d-2: after gate: ok, do not sum test results Task 01 told Ornith to add up every `test result:` N passed. A greedy sed kept the last digit (1512 became 1022) and it stopped with the correct T1 and given test uncommitted. T50/T59: the last line of make gate is the count; summing is not a stop. Co-Authored-By: Grok 4.6 --- docs/implementer-lessons.md | 2 +- docs/plans/M5d2/01-loopd-session-slot.md | 9 ++++----- docs/plans/M5d2/02-bxcore-tool-summary.md | 5 +++-- docs/plans/M5d2/03-loopd-heartbeat-config.md | 5 +++-- docs/plans/M5d2/04-loopd-cron.md | 7 ++++--- docs/plans/M5d2/05-loopd-jobs.md | 7 ++++--- docs/plans/M5d2/06-loopd-scheduler-heartbeat.md | 5 +++-- docs/plans/M5d2/07-loopd-scheduler-cron.md | 5 +++-- docs/plans/M5d2/08-loopd-tool-summary.md | 5 +++-- docs/plans/M5d2/09-memoryd-summary-index.md | 5 +++-- docs/plans/M5d2/10-memoryd-tiers.md | 5 +++-- 11 files changed, 34 insertions(+), 26 deletions(-) diff --git a/docs/implementer-lessons.md b/docs/implementer-lessons.md index 0aa7a7e..93e4386 100644 --- a/docs/implementer-lessons.md +++ b/docs/implementer-lessons.md @@ -89,7 +89,7 @@ How it is used: | T47 | Name the log row in the task ("the row's Task cell is `M5b/14-runbook`"). The driver checks the row by the file's name; the implementer takes it from the task's heading. | M5b task 14: the row was `M5b/14-runbook-sanitizer-rules`, from the heading "the runbook entry `sanitizer-rules`", and `run-plan.sh` stopped on a finished task. | | T48 | Run the implementer with no plugins and no borrowed rules: `opencode run --pure` and `OPENCODE_DISABLE_CLAUDE_CODE=1`. Global plugins load into every session, and a plugin that rewrites the system prompt forces a full re-read of the conversation on a model that cannot rewind its cache. | M5b run (2026-09-26): a short-term-memory plugin rewrote the system prompt every fourth request (a 35,633-token re-read in task 10), 22 plugin tools rode in every request (about 2,400 tokens), and in tasks 01 to 10 prompt reads of 10,000 tokens or more took 76 of 111 minutes of prompt processing: 42 of them, of which only about 11 were session starts (about 15,000 tokens each). Tasks 11 to 14, run after the plugins were removed, had 5 such reads (4 session starts), 4 minutes in all. | | T49 | When a task adds requests on a path through another role, count them against that role's limits (`inferproxy`: 8 connections at once, a burst of 10, then 2 a second), and put the real proxy in at least one given test. A fake server reached directly cannot show a limit on the way to it. | M5b task 08: 66 `/tokenize` requests at startup passed every test and failed on straylight's first start, refused by `inferproxy`'s token bucket. | -| T50 | When a task gives a gate test count, give the command that produces it and say what it includes (the passed total of `cargo test --workspace` plus the ignored `end_to_end` test the gate runs on its own). Otherwise, do not make the count a stop condition. | M5c-1 tasks 01, 03, 13 and 15: each session found itself one short ("1129 passed — expected 1130") and rediscovered the rule. The count drove extra gate runs, scratch files in `/tmp` (02, 03, 05, 10, 13) and `git stash` baselines (03, 15); about a quarter of the transcripts of 03, 13 and 15 went on it. | +| T50 | When a task gives a gate test count, give the command that produces it and say what it includes (the passed total of `cargo test --workspace` plus the ignored `end_to_end` test the gate runs on its own). Otherwise, do not make the count a stop condition. Do not ask the implementer to add up `test result:` `N passed` lines: a greedy `.*([0-9]+)` keeps only the last digit (1512 becomes ~1022). After `gate: ok`, counting is done (T59). | M5c-1 tasks 01, 03, 13 and 15: each session found itself one short ("1129 passed — expected 1130") and rediscovered the rule. The count drove extra gate runs, scratch files in `/tmp` (02, 03, 05, 10, 13) and `git stash` baselines (03, 15); about a quarter of the transcripts of 03, 13 and 15 went on it. M5d-2 task 01: summed with that sed, reported 1022, stopped, left T1 and the given test uncommitted; `run-plan` then stopped on the dirty tree. | | T51 | Never tell the implementer to run a command that the harness denies or that AGENTS.md forbids. Write the branch line as a check ("`git branch --show-current` prints `m5c1`"), not as `git switch`. | M5c-1: all fifteen transcripts start with a refused `git switch m5c1`; AGENTS.md forbids switching branch. | | T52 | List the known flaky tests in the plan README, with what to do (a test outside the task that fails once, then passes on a rerun and in the gate, is named in the log row's Notes and the task goes on). Before hand-over, stress the whole workspace under load, not only the new tests (T32). | M5c-1 task 01: gatewayd's `brokerd_closing_first_ends_it_as_no_longer_pending` (2 of 10 runs fail on a loaded talos with no M5c-1 change); task 15: loopd's `pointers`. Both older tests; Laguna went on in both cases, and task 01's row calls a different test "the only flake". | | T53 | Give the baseline in the task so the implementer never has to measure one. Where a check without the change is needed, the stash is `git stash -u`: a plain stash leaves the new untracked tests behind, and they no longer compile. | M5c-1 task 15: a baseline run printed an empty total; task 03 moved the new tests aside by hand before stashing. | diff --git a/docs/plans/M5d2/01-loopd-session-slot.md b/docs/plans/M5d2/01-loopd-session-slot.md index 107d2a0..d64d5bd 100644 --- a/docs/plans/M5d2/01-loopd-session-slot.md +++ b/docs/plans/M5d2/01-loopd-session-slot.md @@ -90,10 +90,10 @@ become --test compact_turn`: every test passes, 4 in `session_slot` and the others as before. - [ ] **5. Check.** `git diff --stat` lists exactly `crates/loopd/src/turn.rs`. - [ ] **6. Run the gate.** `cargo fmt --all`, then `make gate`, with a shell timeout of 600000 ms. - Expected last line: `gate: ok`. The test count is **1512** after this task, 1508 before it, - and the 4 new ones are the tests of `session_slot.rs`. To count, add up the `N passed` of - every `test result:` line the gate's `cargo test --workspace` printed, and add 1. Never run - the gate again only to count. + Expected last line: `gate: ok`. That is done counting: **1512** (1508 before, plus the 4 of + `session_slot.rs`). After `gate: ok`, do not add up `test result:` lines and do not run + `cargo test --workspace`. A `sed` of `.*([0-9]+) passed` keeps only the last digit and + reports about 1022. - [ ] **7. Log and commit.** Add your row to `docs/implementer-log.md`; its Task cell is `M5d2/01-loopd-session-slot`. Stage with `git add crates/loopd docs/implementer-log.md Cargo.lock`, then commit as `AGENTS.md` says. @@ -116,7 +116,6 @@ become - The fix seems to need a change anywhere else. It does not; stop and say why you think so. - A test outside `session_slot.rs` fails in the gate: check once whether it fails without your change (`git stash -u`, run it, `git stash pop`), then stop and report both results. -- The gate passes with a count other than 1512. ## What the given tests pin diff --git a/docs/plans/M5d2/02-bxcore-tool-summary.md b/docs/plans/M5d2/02-bxcore-tool-summary.md index ab4da55..db7def1 100644 --- a/docs/plans/M5d2/02-bxcore-tool-summary.md +++ b/docs/plans/M5d2/02-bxcore-tool-summary.md @@ -139,7 +139,8 @@ become - [ ] **5. Check.** `git diff --stat` lists `crates/bxcore/src/log.rs`, `crates/loopd/src/baseline.rs`, `crates/loopd/tests/support/turn.rs`. - [ ] **6. Run the gate.** `cargo fmt --all`, then `make gate`, timeout 600000 ms. Last line - `gate: ok`. Test count **1518** (1512 before, plus 6). + `gate: ok`. Test count **1518** (1512 before, plus 6). After `gate: ok`, do not add up + `test result:` lines. - [ ] **7. Log and commit.** Task cell `M5d2/02-bxcore-tool-summary`. Stage with `git add crates/bxcore crates/loopd docs/implementer-log.md Cargo.lock`, then commit as `AGENTS.md` says. @@ -155,7 +156,7 @@ become ## Stop and report if - The branch check fails; a "before" block is not found exactly once; step 2 differs; `cargo - check` warns; a test fails; the gate count is not 1518; you replay a summary as a `User` or + check` warns; a test fails; the gate's last line is not `gate: ok`; you replay a summary as a `User` or `Tool` message; you add `truncated`; you change `Setup::turn`'s return type. ## What the given tests pin diff --git a/docs/plans/M5d2/03-loopd-heartbeat-config.md b/docs/plans/M5d2/03-loopd-heartbeat-config.md index cef628a..4950c2e 100644 --- a/docs/plans/M5d2/03-loopd-heartbeat-config.md +++ b/docs/plans/M5d2/03-loopd-heartbeat-config.md @@ -216,7 +216,8 @@ The test still forbids `m2b()`, `m3a()` and `m5a()` in `main.rs`. Do not weaken `crates/loopd/src/main.rs` and `crates/loopd/tests/ask_memory_tool.rs`. `git status --short` shows the new test and `config/heartbeat.md`. - [ ] **6. Run the gate.** `cargo fmt --all`, then `make gate`, timeout 600000 ms. Last line - `gate: ok`. Test count **1526** (1518 before, plus 8). + `gate: ok`. Test count **1526** (1518 before, plus 8). After `gate: ok`, do not add up + `test result:` lines. - [ ] **7. Log and commit.** Task cell `M5d2/03-loopd-heartbeat-config`. Stage with `git add crates/loopd config/heartbeat.md docs/implementer-log.md Cargo.lock`, then commit as `AGENTS.md` says. @@ -231,7 +232,7 @@ The test still forbids `m2b()`, `m3a()` and `m5a()` in `main.rs`. Do not weaken ## Stop and report if - The branch check fails; a "before" block is not found exactly once; step 2 differs; `cargo - check` warns; a test fails; the gate count is not 1526; you leave the `m5c()` count at 1; + check` warns; a test fails; the gate's last line is not `gate: ok`; you leave the `m5c()` count at 1; you join `heartbeat.prompt` onto `system.md`'s directory instead of the config file's parent; you reject an empty `summarise` list; you check names inside `parse`. diff --git a/docs/plans/M5d2/04-loopd-cron.md b/docs/plans/M5d2/04-loopd-cron.md index 42d4ee4..db5441a 100644 --- a/docs/plans/M5d2/04-loopd-cron.md +++ b/docs/plans/M5d2/04-loopd-cron.md @@ -169,7 +169,8 @@ If `is_multiple_of` is not in scope on `u32` in this toolchain, use `value % n = - [ ] **5. Check.** `git diff --stat` lists `crates/loopd/src/cron.rs` and `crates/loopd/src/lib.rs`. - [ ] **6. Run the gate.** `cargo fmt --all`, then `make gate`, timeout 600000 ms. Last line - `gate: ok`. Test count **1537** (1526 before, plus 11). + `gate: ok`. Test count **1537** (1526 before, plus 11). After `gate: ok`, do not add up + `test result:` lines. - [ ] **7. Log and commit.** Task cell `M5d2/04-loopd-cron`. Stage with `git add crates/loopd docs/implementer-log.md Cargo.lock`, then commit as `AGENTS.md` says. @@ -181,8 +182,8 @@ If `is_multiple_of` is not in scope on `u32` in this toolchain, use `value % n = ## Stop and report if -- The branch check fails; step 2 differs; `cargo check` warns; a test fails; the gate count is - not 1537; you add a dependency; you treat DOM and DOW as OR; you accept `@daily` or a sixth +- The branch check fails; step 2 differs; `cargo check` warns; a test fails; the gate's last + line is not `gate: ok`; you add a dependency; you treat DOM and DOW as OR; you accept `@daily` or a sixth field; `cron.rs` exceeds 500 lines. ## What the given tests pin diff --git a/docs/plans/M5d2/05-loopd-jobs.md b/docs/plans/M5d2/05-loopd-jobs.md index d14c601..ceeac15 100644 --- a/docs/plans/M5d2/05-loopd-jobs.md +++ b/docs/plans/M5d2/05-loopd-jobs.md @@ -194,7 +194,8 @@ Replace the four stub bodies in the same task. Do not leave a stub that returns - [ ] **6. Check.** `git diff --stat` lists `crates/loopd/src/jobs.rs`, `crates/loopd/src/lib.rs`, `crates/loopd/src/main.rs`, `docs/runbook.md`. - [ ] **7. Run the gate.** `cargo fmt --all`, then `make gate`, timeout 600000 ms. Last line - `gate: ok`. Test count **1545** (1537 before, plus 8). + `gate: ok`. Test count **1545** (1537 before, plus 8). After `gate: ok`, do not add up + `test result:` lines. - [ ] **8. Log and commit.** Task cell `M5d2/05-loopd-jobs`. Stage with `git add crates/loopd docs/runbook.md docs/implementer-log.md Cargo.lock`, then commit as `AGENTS.md` says. @@ -207,8 +208,8 @@ Replace the four stub bodies in the same task. Do not leave a stub that returns ## Stop and report if -- The branch check fails; step 2 differs; `cargo check` warns; a test fails; the gate count is - not 1545; you add a cron crate; a missing jobs directory is an error; `Display` for `Invalid` +- The branch check fails; step 2 differs; `cargo check` warns; a test fails; the gate's last + line is not `gate: ok`; you add a cron crate; a missing jobs directory is an error; `Display` for `Invalid` does not end with the runbook pointer; `jobs.rs` exceeds 500 lines; you write a job file. ## What the given tests pin diff --git a/docs/plans/M5d2/06-loopd-scheduler-heartbeat.md b/docs/plans/M5d2/06-loopd-scheduler-heartbeat.md index ae7f6d5..7024fd2 100644 --- a/docs/plans/M5d2/06-loopd-scheduler-heartbeat.md +++ b/docs/plans/M5d2/06-loopd-scheduler-heartbeat.md @@ -197,7 +197,8 @@ pass so it runs once. `grep -c '^## heartbeat-prompt-unreadable$' docs/runbook.md` prints `1`. - [ ] **6. Run the gate.** `cargo fmt --all`, then `make gate`, timeout 600000 ms. Last line - `gate: ok`. Test count **1551** (1545 before, plus 6). + `gate: ok`. Test count **1551** (1545 before, plus 6). After `gate: ok`, do not add up + `test result:` lines. - [ ] **7. Log and commit.** Task cell `M5d2/06-loopd-scheduler-heartbeat`. Stage with `git add crates/loopd docs/runbook.md docs/implementer-log.md Cargo.lock`, then commit as `AGENTS.md` says. @@ -212,7 +213,7 @@ pass so it runs once. ## Stop and report if - The branch check fails; a "before" block is not found exactly once; step 2 differs; `cargo - check` warns; a test fails; the gate count is not 1551; you sleep in a test; you post to + check` warns; a test fails; the gate's last line is not `gate: ok`; you sleep in a test; you post to Mattermost; you send `slots.main`; you set a speaker; you `chmod 000`; `Held` stays private; `hold` returns `Result<_, ()>`. diff --git a/docs/plans/M5d2/07-loopd-scheduler-cron.md b/docs/plans/M5d2/07-loopd-scheduler-cron.md index b206e07..84d44a6 100644 --- a/docs/plans/M5d2/07-loopd-scheduler-cron.md +++ b/docs/plans/M5d2/07-loopd-scheduler-cron.md @@ -82,7 +82,8 @@ Do not change heartbeat behaviour. Do not add a sleep. --test jobs --test cron`: every test passes, 6 in `scheduler_cron`. - [ ] **5. Check.** `git diff --stat` lists exactly `crates/loopd/src/scheduler.rs`. - [ ] **6. Run the gate.** `cargo fmt --all`, then `make gate`, timeout 600000 ms. Last line - `gate: ok`. Test count **1557** (1551 before, plus 6). + `gate: ok`. Test count **1557** (1551 before, plus 6). After `gate: ok`, do not add up + `test result:` lines. - [ ] **7. Log and commit.** Task cell `M5d2/07-loopd-scheduler-cron`. Stage with `git add crates/loopd docs/implementer-log.md Cargo.lock`, then commit as `AGENTS.md` says. @@ -95,7 +96,7 @@ Do not change heartbeat behaviour. Do not add a sleep. ## Stop and report if - The branch check fails; step 2 differs (tests do not compile, or a different assertion - fails); `cargo check` warns; a test fails; the gate count is not 1557; you reuse a session + fails); `cargo check` warns; a test fails; the gate's last line is not `gate: ok`; you reuse a session across runs; you send `slots.main`; you set a speaker; you skip leftovers that are not busy; a heartbeat test fails. diff --git a/docs/plans/M5d2/08-loopd-tool-summary.md b/docs/plans/M5d2/08-loopd-tool-summary.md index ca2232b..8e5df13 100644 --- a/docs/plans/M5d2/08-loopd-tool-summary.md +++ b/docs/plans/M5d2/08-loopd-tool-summary.md @@ -230,7 +230,8 @@ Facts). The `Err(TurnError::SessionFull)` arm gets the same `missing` call befor - [ ] **5. Check.** `git diff --stat` lists `crates/loopd/src/summarise.rs`, `crates/loopd/src/lib.rs`, `crates/loopd/src/compact.rs`. - [ ] **6. Run the gate.** `cargo fmt --all`, then `make gate`, timeout 600000 ms. Last line - `gate: ok`. Test count **1563** (1557 before, plus 6). + `gate: ok`. Test count **1563** (1557 before, plus 6). After `gate: ok`, do not add up + `test result:` lines. - [ ] **7. Log and commit.** Task cell `M5d2/08-loopd-tool-summary`. Stage with `git add crates/loopd config/summarise.md docs/implementer-log.md Cargo.lock`, then commit as `AGENTS.md` says. @@ -245,7 +246,7 @@ Facts). The `Err(TurnError::SessionFull)` arm gets the same `missing` call befor ## Stop and report if - The branch check fails; a "before" block is not found exactly once; step 2 differs; `cargo - check` warns; a test fails; the gate count is not 1563; you send tools on the summary + check` warns; a test fails; the gate's last line is not `gate: ok`; you send tools on the summary request; you send `slots.main`; you replay the summary in `messages()`; you summarise a name not in the list; you summarise twice; you compact before summarising. diff --git a/docs/plans/M5d2/09-memoryd-summary-index.md b/docs/plans/M5d2/09-memoryd-summary-index.md index 890897f..dba5ee4 100644 --- a/docs/plans/M5d2/09-memoryd-summary-index.md +++ b/docs/plans/M5d2/09-memoryd-summary-index.md @@ -117,7 +117,8 @@ results are indexed. A comment that does not fit its code: stop. test passes, 5 in `tool_summary_index`. - [ ] **5. Check.** `git diff --stat` lists exactly `crates/memoryd/src/index.rs`. - [ ] **6. Run the gate.** `cargo fmt --all`, then `make gate`, timeout 600000 ms. Last line - `gate: ok`. Test count **1568** (1563 before, plus 5). + `gate: ok`. Test count **1568** (1563 before, plus 5). After `gate: ok`, do not add up + `test result:` lines. - [ ] **7. Log and commit.** Task cell `M5d2/09-memoryd-summary-index`. Stage with `git add crates/memoryd docs/implementer-log.md Cargo.lock`, then commit as `AGENTS.md` says. @@ -131,7 +132,7 @@ results are indexed. A comment that does not fit its code: stop. ## Stop and report if - The branch check fails; a "before" block is not found exactly once; step 2 differs; `cargo - check` warns; a test fails; the gate count is not 1568; you index `ToolResult` content; you + check` warns; a test fails; the gate's last line is not `gate: ok`; you index `ToolResult` content; you leave `kind` as `"session"`; a secret summary is searchable on a private index. ## What the given tests pin diff --git a/docs/plans/M5d2/10-memoryd-tiers.md b/docs/plans/M5d2/10-memoryd-tiers.md index fe8597e..c03ad20 100644 --- a/docs/plans/M5d2/10-memoryd-tiers.md +++ b/docs/plans/M5d2/10-memoryd-tiers.md @@ -226,7 +226,8 @@ Do not edit `cli_memory.rs`. This file is the M5c pin of the same USAGE string. - [ ] **5. Check.** `git diff --stat` lists the files in "Every place that changes" except the new tests and starter (those are untracked until `git add`). - [ ] **6. Run the gate.** `cargo fmt --all`, then `make gate`, timeout 600000 ms. Last line - `gate: ok`. Test count **1574** (1568 before, plus 6). + `gate: ok`. Test count **1574** (1568 before, plus 6). After `gate: ok`, do not add up + `test result:` lines. - [ ] **7. Log and commit.** Task cell `M5d2/10-memoryd-tiers`. Stage with `git add crates/bxcore crates/memoryd crates/bxctl config/jobs docs/implementer-log.md Cargo.lock`, then commit as `AGENTS.md` says. @@ -239,7 +240,7 @@ Do not edit `cli_memory.rs`. This file is the M5c pin of the same USAGE string. ## Stop and report if - The branch check fails; a "before" block is not found exactly once; step 2 differs; `cargo - check` warns; a test fails; the gate count is not 1574; you edit `cli_memory.rs`; you leave + check` warns; a test fails; the gate's last line is not `gate: ok`; you edit `cli_memory.rs`; you leave the `cli_memory_sync` USAGE pin on the old string; identity entries show a non-zero hit count; untrusted notes are omitted; you add a cron crate; you special-case the name `tier` in `loopd`.