From 182804856405262715d58b00f7d99658f243812f Mon Sep 17 00:00:00 2001 From: "K. Isom" Date: Tue, 22 Sep 2026 20:10:48 -0700 Subject: [PATCH] Check that every runbook pointer has an entry Implemented-By: Grok 4.6 --- Makefile | 1 + docs/implementer-log.md | 1 + scripts/check-runbook.sh | 112 +++++++++++++++++++++++++++++++++++ scripts/test-gate-scripts.sh | 67 ++++++++++++++++++++- 4 files changed, 180 insertions(+), 1 deletion(-) create mode 100755 scripts/check-runbook.sh diff --git a/Makefile b/Makefile index ef7a1ac..ff348f5 100644 --- a/Makefile +++ b/Makefile @@ -10,6 +10,7 @@ gate: sh scripts/check-lines.sh sh scripts/check-crate-deps.sh sh scripts/check-dep-docs.sh + sh scripts/check-runbook.sh sh scripts/test-gate-scripts.sh @echo "gate: ok" diff --git a/docs/implementer-log.md b/docs/implementer-log.md index 1bff876..3dd656b 100644 --- a/docs/implementer-log.md +++ b/docs/implementer-log.md @@ -6,6 +6,7 @@ reviewer adds findings under "Reviews" once per milestone. | Task | Date | Status | Gate runs | First gate | Deviations | Notes | Model | |---|---|---|---|---|---|---|---| +| M3a/21-runbook-check | 2026-09-22 | done | 1 | pass | none | Wrote `scripts/check-runbook.sh`: find `*.rs` under crates (prune `target/`), awk out every `docs/runbook.md#` pointer, empty anchors fail, each remaining anchor must match a whole `## ` line, every missing one is reported with its files, then one exit. Step 5: dropping `-x` from grep failed with "the entry is the whole line, at level two"; `exit 1` at a missing-anchor report failed with "both missing entries and their files are reported". Real tree exits 0. `make gate` prints `gate: ok`. | ? | | M3a/20-bxctl-chat-approvals | 2026-09-22 | done | 1 | pass | `AdminError` has no `Io` variant (task 18 maps write failures to `Protocol` via `From`), so `handle_pending` exit 8 reports every `cmd_approve`/`cmd_refuse` `Err` as `approval {id}: {e}` rather than returning `Err(AdminError::Io(e))`. Writes inside `handle_pending` itself still use `?`. | Moved `stream_turn` into `chat.rs` with `OnPending`/`Approvals`/`TurnIo`/`handle_pending`. The owner is shown `brokerd`'s list item, never the event's tool/args. Only the id typed in full (after stripping one trailing `\n` then one `\r`) approves; anything else refuses. `run` holds one `BufReader` on stdin for both modes. `Printer::event` escapes model text and tool names, prints the three fail-closed runbook lines as whole literals, and prints nothing for `ApprovalPending`. 21/12/20/9/12/8 tests five runs; `make gate` prints `gate: ok`. | ? | | M3a/19-bxctl-audit-verify | 2026-09-21 | stopped | 0 | n/a | none | The task's "The command" section says to grep `crates/bxctl/src/main.rs` for `cannot read the audit log` and stop if it is absent. `grep` returned no match: `main.rs`'s `Command::AuditVerify { home: _ }` arm (line 39) is still the placeholder that prints `bxctl: audit verify is not implemented yet`, ignores `home`, never calls `bxctl::verify::run`, and never prints the error message. The task forbids editing `main.rs` or `lib.rs`, so `bxctl audit verify` cannot be wired to `run` without that edit, and step 5 (which relies on the wiring) cannot pass. Stopped per the task's explicit instruction rather than editing a forbidden file. | ? | | M3a/19-bxctl-audit-verify | 2026-09-21 | done | 1 | pass | Edited crates/bxctl/src/main.rs to wire `audit verify` to `bxctl::verify::run` (the placeholder arm at main.rs:39 was never wired by task 18; the owner authorized this as a documented deviation). The task's step 5 shorthand `run(&home)` omits the required `out` writer, which carries the report to stdout. | Wrote crates/bxctl/src/verify.rs: `run` lists `/audit/`, keeps only `YYYY-MM-DD.jsonl` names (date dashes at 0-indexed positions 4 and 7, so the real fixture dates match), sorts them, feeds each to `proto::ChainVerifier`, and prints the report exactly (the two-line failure form, or the ok form in the task's list order); `.lock` and malformed names are ignored. A missing dir is an error, an existing empty dir is an empty log, and every io error propagates with `?`. 6 verify tests pass; `make gate` prints `gate: ok`. | ? | diff --git a/scripts/check-runbook.sh b/scripts/check-runbook.sh new file mode 100755 index 0000000..d457e12 --- /dev/null +++ b/scripts/check-runbook.sh @@ -0,0 +1,112 @@ +#!/bin/sh +# Fails if a *.rs file under ROOT/crates names a docs/runbook.md#anchor that +# has no matching `## ` line in ROOT/docs/runbook.md. +# Files under any target/ directory are ignored. Test files count. +ROOT="${1:-.}" + +if [ ! -d "$ROOT/crates" ]; then + echo "check-runbook: $ROOT/crates is not a directory" >&2 + exit 1 +fi +if [ ! -f "$ROOT/docs/runbook.md" ]; then + echo "check-runbook: $ROOT/docs/runbook.md is not a file" >&2 + exit 1 +fi + +work=$(mktemp -d) +if [ $? -ne 0 ] || [ -z "$work" ]; then + echo "check-runbook: mktemp failed" >&2 + exit 1 +fi +trap 'rm -rf "$work"' EXIT + +find "$ROOT/crates" -type d -name target -prune -o -type f -name '*.rs' -print \ + > "$work/files" +if [ $? -ne 0 ]; then + echo "check-runbook: find failed" >&2 + exit 1 +fi + +: > "$work/pointers" +while IFS= read -r f; do + if [ ! -r "$f" ]; then + echo "check-runbook: cannot read $f" >&2 + exit 1 + fi + awk '{ + s = $0 + while (match(s, /docs\/runbook\.md#[A-Za-z0-9_-]*/)) { + printf "%s\t%s\n", substr(s, RSTART + 16, RLENGTH - 16), FILENAME + s = substr(s, RSTART + RLENGTH) + } + }' "$f" >> "$work/pointers" + if [ $? -ne 0 ]; then + echo "check-runbook: awk failed on $f" >&2 + exit 1 + fi +done < "$work/files" + +if [ ! -s "$work/pointers" ]; then + echo "check-runbook: no pointer found" >&2 + exit 1 +fi + +sort "$work/pointers" > "$work/sorted" +if [ $? -ne 0 ]; then + echo "check-runbook: sort failed" >&2 + exit 1 +fi + +awk -F '\t' '{ print $1 }' "$work/sorted" > "$work/col1" +if [ $? -ne 0 ]; then + echo "check-runbook: awk failed" >&2 + exit 1 +fi +sort -u "$work/col1" > "$work/anchors" +if [ $? -ne 0 ]; then + echo "check-runbook: sort failed" >&2 + exit 1 +fi + +status=0 +while IFS= read -r anchor; do + awk -F '\t' -v a="$anchor" '$1 == a { print $2 }' "$work/sorted" > "$work/hits" + if [ $? -ne 0 ]; then + echo "check-runbook: awk failed" >&2 + exit 1 + fi + sort -u "$work/hits" > "$work/hitfiles" + if [ $? -ne 0 ]; then + echo "check-runbook: sort failed" >&2 + exit 1 + fi + files="" + while IFS= read -r hf; do + if [ -n "$files" ]; then + files="$files $hf" + else + files="$hf" + fi + done < "$work/hitfiles" + + if [ -z "$anchor" ]; then + echo "check-runbook: empty anchor in $files" >&2 + status=1 + continue + fi + + grep -q -x -F -e "## $anchor" "$ROOT/docs/runbook.md" + g=$? + if [ "$g" -eq 0 ]; then + continue + fi + if [ "$g" -eq 1 ]; then + echo "check-runbook: no entry for $anchor (in $files)" >&2 + status=1 + continue + fi + echo "check-runbook: grep failed" >&2 + exit 1 +done < "$work/anchors" + +exit "$status" diff --git a/scripts/test-gate-scripts.sh b/scripts/test-gate-scripts.sh index 9e26746..e674508 100644 --- a/scripts/test-gate-scripts.sh +++ b/scripts/test-gate-scripts.sh @@ -1,5 +1,5 @@ #!/bin/sh -# Self-test for the three gate scripts. It builds small fake trees in a temporary +# Self-test for the gate scripts. It builds small fake trees in a temporary # directory and checks that each script passes the good tree and fails the bad ones. # Do not edit: this file defines the required behaviour of the scripts. set -eu @@ -87,6 +87,65 @@ expect fail "table-form dependency without workspace = true" check-dep-docs.sh " tree "$tmp/d-sectok"; printf '\n[dependencies.serde]\nworkspace = true\nfeatures = ["derive"]\n' >> "$tmp/d-sectok/crates/loopd/Cargo.toml" expect pass "table-form dependency with workspace = true" check-dep-docs.sh "$tmp/d-sectok" +# check-runbook.sh +book() { # book ROOT [HEADING-LINES...]: a tree whose runbook has these lines + root="$1"; shift + tree "$root" + { printf '# Runbook\n\nProse that mentions grants-invalid is not an entry.\n\n' + for line in "$@"; do printf '%s\n\nText.\n\n' "$line"; done + } > "$root/docs/runbook.md" +} +src() { # src FILE TEXT: a source file holding TEXT + mkdir -p "$(dirname "$1")" + printf '%s\n' "$2" > "$1" +} +says() { # says NAME SCRIPT ROOT WORD...: the script's output must contain every WORD + name="$1"; script="$2"; root="$3"; shift 3 + output=$(sh "$here/$script" "$root" 2>&1) || true + for word in "$@"; do + case "$output" in + *"$word"*) ;; + *) echo "test-gate-scripts: $name: output lacks $word" >&2; fails=$((fails + 1)) ;; + esac + done +} + +book "$tmp/r-ok" '## grants-invalid' '## audit-unavailable' +src "$tmp/r-ok/crates/brokerd/src/a.rs" 'eprintln!("see docs/runbook.md#grants-invalid");' +src "$tmp/r-ok/crates/brokerd/src/b.rs" '// see docs/runbook.md#audit-unavailable.' +expect pass "every pointer has an entry" check-runbook.sh "$tmp/r-ok" +book "$tmp/r-miss" '## audit-unavailable' +src "$tmp/r-miss/crates/brokerd/src/a.rs" 'eprintln!("see docs/runbook.md#grants-invalid");' +expect fail "a pointer without an entry; a mention in prose is not one" check-runbook.sh "$tmp/r-miss" +book "$tmp/r-h3" '### grants-invalid' '## grants-invalid and more' ' ## grants-invalid' +src "$tmp/r-h3/crates/brokerd/src/a.rs" 'eprintln!("see docs/runbook.md#grants-invalid");' +expect fail "the entry is the whole line, at level two" check-runbook.sh "$tmp/r-h3" +book "$tmp/r-test" '## grants-invalid' +src "$tmp/r-test/crates/brokerd/src/a.rs" 'eprintln!("see docs/runbook.md#grants-invalid");' +src "$tmp/r-test/crates/brokerd/tests/t.rs" 'assert!(m.ends_with("see docs/runbook.md#no-such-entry"));' +expect fail "a pointer in a test file counts" check-runbook.sh "$tmp/r-test" +book "$tmp/r-line" '## grants-invalid' +src "$tmp/r-line/crates/brokerd/src/a.rs" 'f("docs/runbook.md#grants-invalid", "docs/runbook.md#second-on-the-line");' +expect fail "the second pointer on a line counts" check-runbook.sh "$tmp/r-line" +book "$tmp/r-tgt" '## grants-invalid' +src "$tmp/r-tgt/crates/brokerd/src/a.rs" 'eprintln!("see docs/runbook.md#grants-invalid");' +src "$tmp/r-tgt/crates/brokerd/target/debug/gen.rs" 'eprintln!("see docs/runbook.md#no-such-entry");' +expect pass "files under target/ are ignored" check-runbook.sh "$tmp/r-tgt" +book "$tmp/r-case" '## grants-invalid' +src "$tmp/r-case/crates/brokerd/src/a.rs" 'eprintln!("see docs/runbook.md#Grants-Invalid");' +expect fail "anchors are compared exactly" check-runbook.sh "$tmp/r-case" +book "$tmp/r-fmt" '## grants-invalid' +src "$tmp/r-fmt/crates/brokerd/src/a.rs" 'eprintln!("see docs/runbook.md#grants-invalid");' +src "$tmp/r-fmt/crates/brokerd/src/b.rs" 'eprintln!("see docs/runbook.md#{anchor}");' +expect fail "a pointer whose anchor is not written out" check-runbook.sh "$tmp/r-fmt" +# Report every problem, not only the first. +book "$tmp/r-all" '## grants-invalid' +src "$tmp/r-all/crates/brokerd/src/a.rs" 'eprintln!("see docs/runbook.md#first-missing");' +src "$tmp/r-all/crates/loopd/src/b.rs" 'eprintln!("see docs/runbook.md#second-missing");' +expect fail "two pointers without entries" check-runbook.sh "$tmp/r-all" +says "both missing entries and their files are reported" check-runbook.sh "$tmp/r-all" \ + first-missing second-missing crates/brokerd/src/a.rs crates/loopd/src/b.rs + # A check that cannot find what it checks must fail, not pass. mkdir -p "$tmp/empty" expect fail "check-lines without a crates directory" check-lines.sh "$tmp/empty" @@ -94,6 +153,12 @@ expect fail "check-crate-deps without a crates directory" check-crate-deps.sh "$ expect fail "check-dep-docs without a crates directory" check-dep-docs.sh "$tmp/empty" tree "$tmp/d-nodoc"; rm "$tmp/d-nodoc/docs/dependencies.md" expect fail "check-dep-docs without docs/dependencies.md" check-dep-docs.sh "$tmp/d-nodoc" +expect fail "check-runbook without a crates directory" check-runbook.sh "$tmp/empty" +book "$tmp/r-nobook" '## grants-invalid'; rm "$tmp/r-nobook/docs/runbook.md" +src "$tmp/r-nobook/crates/brokerd/src/a.rs" 'eprintln!("see docs/runbook.md#grants-invalid");' +expect fail "check-runbook without docs/runbook.md" check-runbook.sh "$tmp/r-nobook" +book "$tmp/r-none" '## grants-invalid' +expect fail "check-runbook when no source file has a pointer" check-runbook.sh "$tmp/r-none" if [ "$fails" -ne 0 ]; then echo "test-gate-scripts: $fails failure(s)" >&2