From 62aaadd17ebca1510fb679e3a3025573532b5978 Mon Sep 17 00:00:00 2001 From: "K. Isom" Date: Mon, 28 Sep 2026 21:27:53 -0700 Subject: [PATCH] M5c-2 plan: README wrap-up: the tasks with their counts, known flaky tests and known limits Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/plans/M5c2/README.md | 101 ++++++++++++++++++++++++-------------- 1 file changed, 63 insertions(+), 38 deletions(-) diff --git a/docs/plans/M5c2/README.md b/docs/plans/M5c2/README.md index 6ee8465..c335382 100644 --- a/docs/plans/M5c2/README.md +++ b/docs/plans/M5c2/README.md @@ -55,8 +55,10 @@ Honcho sources and chats outside its state lock, and answers `ask_memory`. - Roles depend on `bxcore` only; `memoryd` also on `roncho`. No `unsafe`, no async runtime. - Every `roncho` client is built with `.without_config_file()`; keys come from `bxcore::secrets` and are never logged; errors are logged by kind and status only. -- Text from Honcho is neutralised on arrival and capped like a local `recall` snippet (600 - characters); a result's labels raise the session as a local hit's do. +- Text from Honcho is neutralised on arrival, put on one indented line, and capped: 600 + characters for a `recall` hit, as a local snippet, and 2,000 for an `ask_memory` answer; a + result's labels raise the session as a local hit's do. Local `recall` items take the same form + (task 07). - No Honcho call runs under `memoryd`'s state lock. - Without a `[honcho]` section, `recall` is exactly M5b's and `ask_memory` answers that Honcho is not configured. @@ -68,23 +70,27 @@ Honcho sources and chats outside its state lock, and answers `ask_memory`. whitespace-checked literals are marked on every edit (T55); the task's table of `roncho`'s API is complete, and the Cargo registry is off limits (T56). -## Tasks (outline, for the owner's review) +## Tasks -| # | Delivers | Crates | -|---|---|---| -| 00 | (done by the design model, `718e66d`) recordings from rift: workspace search with a limit and with no match, conclusion queries for two peers, and what a missing workspace answers | memoryd fixtures | -| 01 | Wire: `AskMemoryArgs { question, workspaces }`, its name and its checks (question at most 2,000 characters; one or two workspace names, no repeats, each 1 to 512 characters as `memoryd`'s name rule) | bxcore | -| 02 | `brokerd`: `ask_memory` as a built-in memory tool (name, parsing, the grant `builtin-ask-memory`, owner grants naming it refused, routing to `memoryd`); its answer's labels raise the session | brokerd | -| 03 | `brokerd`: a per-tool wait on `memoryd`, 170 s for `ask_memory` (`ASK_MEMORY_WAIT`) and 30 s for the others, chosen by `MemoryPort::wait_for` | brokerd | -| 04 | `loopd`: `ask_memory`'s schema as a core tool in a new registry; a per-tool wait on `brokerd`, 180 s for `ask_memory` and `[broker] timeout_ms` for the others; the system prompt's line on memory tools | loopd | -| 05 | `memoryd`: read clients, one per `[[honcho.read]]` workspace, the fleet and the agent's own two, with a 5 s limit for searches and 150 s for chat; each source's label; `recall_sources` and `resolve`; `own`, `untrusted` and `fleet` reserved as read agents' names | memoryd | -| 06 | `memoryd`, pure: Honcho messages and conclusions as `recall` items (tag, source name, class, neutralised, capped), kept only when they share a term with the query, interleaved with the local hits to `k`, the local index first | memoryd | -| 07 | `memoryd`: local `recall` items in task 06's form, one indented line each, through one shared `snippet` and `item`; `recall::local` gives the items, their count and their label apart (T37) | memoryd | -| 08 | `memoryd`: `recall`'s Honcho part (`honcho/search.rs`): every source of `recall_sources` at once, each on its own thread with a limit of 5 s from the start; a read agent's workspace is a search and two conclusion queries; `: unavailable`; a query holding a secret asks nothing; the answer from the local items and Honcho's, with the stricter label | memoryd | -| 09 | `memoryd`: `recall` with `[honcho]` runs the local part under the state lock and the Honcho part outside it; the readers taken at start; without `[honcho]`, `recall` as before | memoryd | -| 10 | `memoryd`: `ask_memory`: workspace names checked against `[honcho]`, both chats at once, each answer tagged and capped, labels, `: unavailable`; a question holding a secret sends nothing | memoryd | -| 11 | `memoryd`, making room: `reindex` and `admin` move unchanged from `serve.rs` to `serve/admin.rs`, with tests for four of their behaviours no test pinned | memoryd | -| 12 | `memoryd` serves `ask_memory` beside `recall` and `memory`, never under the state lock; its arguments checked again; without `[honcho]`, a sentence as the result; no new fail-closed state, so no runbook entry, and `bxctl` unchanged | memoryd | +Every task below has been written, replayed from its text and checked; the gate count is the +passed total of `cargo test --workspace` plus one, for the ignored `end_to_end` test the gate +runs on its own. Before task 01 it is 1364 (talos, rustc 1.98.1, clippy of the same release). + +| # | Delivers | Crates | Gate after | Mutations tried, caught | +|---|---|---|---|---| +| 00 | (done by the design model, `718e66d`) recordings from rift: workspace search with a limit and with no match, conclusion queries for two peers, and what a missing workspace answers | memoryd fixtures | 1364 | (none: fixtures only) | +| 01 | Wire: `AskMemoryArgs { question, workspaces }`, its name and its checks (question at most 2,000 characters; one or two workspace names, no repeats, each 1 to 512 characters as `memoryd`'s name rule) | bxcore | 1372 | 23, 23 | +| 02 | `brokerd`: `ask_memory` as a built-in memory tool (name, parsing, the grant `builtin-ask-memory`, owner grants naming it refused, routing to `memoryd`); its answer's labels raise the session | brokerd | 1381 | 17, 17 | +| 03 | `brokerd`: a per-tool wait on `memoryd`, 170 s for `ask_memory` (`ASK_MEMORY_WAIT`) and 30 s for the others, chosen by `MemoryPort::wait_for` | brokerd | 1384 | 10, 10 | +| 04 | `loopd`: `ask_memory`'s schema as a core tool in a new registry; a per-tool wait on `brokerd`, 180 s for `ask_memory` and `[broker] timeout_ms` for the others; the system prompt's line on memory tools | loopd | 1394 | 19, 19 | +| 05 | `memoryd`: read clients, one per `[[honcho.read]]` workspace, the fleet and the agent's own two, with a 5 s limit for searches and 150 s for chat; each source's label; `recall_sources` and `resolve`; `own`, `untrusted` and `fleet` reserved as read agents' names | memoryd | 1403 | 24, 24 | +| 06 | `memoryd`, pure: Honcho messages and conclusions as `recall` items (tag, source name, class, neutralised, capped), kept only when they share a term with the query, interleaved with the local hits to `k`, the local index first | memoryd | 1414 | 22, 22 | +| 07 | `memoryd`: local `recall` items in task 06's form, one indented line each, through one shared `snippet` and `item`; `recall::local` gives the items, their count and their label apart (T37) | memoryd | 1418 | 15, 15 | +| 08 | `memoryd`: `recall`'s Honcho part (`honcho/search.rs`): every source of `recall_sources` at once, each on its own thread with a limit of 5 s from the start; a read agent's workspace is a search and two conclusion queries; `: unavailable`; a query holding a secret asks nothing; the answer from the local items and Honcho's, with the stricter label | memoryd | 1430 | 19, 19 | +| 09 | `memoryd`: `recall` with `[honcho]` runs the local part under the state lock and the Honcho part outside it; the readers taken at start; without `[honcho]`, `recall` as before | memoryd | 1434 | 8, 8 | +| 10 | `memoryd`: `ask_memory`: workspace names checked against `[honcho]`, both chats at once, each answer tagged and capped, labels, `: unavailable`; a question holding a secret sends nothing | memoryd | 1444 | 18, 18 | +| 11 | `memoryd`, making room: `reindex` and `admin` move unchanged from `serve.rs` to `serve/admin.rs`, with tests for four of their behaviours no test pinned | memoryd | 1448 | 8, 8 | +| 12 | `memoryd` serves `ask_memory` beside `recall` and `memory`, never under the state lock; its arguments checked again; without `[honcho]`, a sentence as the result; no new fail-closed state, so no runbook entry, and `bxctl` unchanged | memoryd | 1454 | 9, 9 | After task 12, the design model runs the M5c-2 device check (spec 13) on straylight, rift and the owner's Mattermost: a second `memoryd` under its own name writes a fleet note that `recall` finds, @@ -93,32 +99,45 @@ trusted workspace within its limit; with rift down, `recall` answers from the lo `unavailable` lines. The baseline is measured again against the 3,000-token budget (`make verify-device`). -## Counts, as tasks are written +## Known flaky tests -The gate count is the passed total of `cargo test --workspace` plus one, for the ignored -`end_to_end` test the gate runs on its own. Before task 01 it is 1364 (talos, rustc 1.98.1). - -| Task | Gate count after | Mutations tried, caught | -|---|---|---| -| 01 | 1372 | 23, 23 | -| 02 | 1381 | 17, 17 | -| 03 | 1384 | 10, 10 | -| 04 | 1394 | 19, 19 | -| 05 | 1403 | 24, 24 | -| 06 | 1414 | 22, 22 | -| 07 | 1418 | 15, 15 | -| 08 | 1430 | 19, 19 | -| 09 | 1434 | 8, 8 | -| 10 | 1444 | 18, 18 | -| 11 | 1448 | 8, 8 | -| 12 | 1454 | 9, 9 | - -**A flaky test, fixed before the run:** `bxctl`'s `chat_approvals` test +**Fixed before the run:** `bxctl`'s `chat_approvals` test `json_prints_only_json_and_never_asks_brokerd` failed once in 5 loaded runs of the whole workspace while task 06 was checked: its helper wrote to the standard input of a `bxctl chat` that does not read it and may already have exited, and unwrapped the broken pipe. The helper now accepts a broken pipe on that write (master, after task 06); 0 failures in 100 loaded runs. +**Seen once, not reproduced:** while task 11 was checked, one run of `memoryd`'s whole suite +failed in `honcho_worker`'s `a_turn_too_long_for_one_batch_is_marked_failed`, and another in +several unrelated `memoryd` tests at once; 18 later runs of the whole suite (12 of them beside +8 busy loops) and 10 of that test alone passed. A session that sees a `memoryd` test outside its +task fail once, then pass on a rerun and in the gate, names it in its row's Notes and goes on +(AGENTS.md). + +## Known limits + +These are known, accepted for M5c-2, and not bugs to fix in a task: + +- **A read agent's three calls share one limit.** `recall` asks another agent's workspace one + call after another (a search, then the conclusions about the agent's peer and the owner's), + all within that source's 5 s. A slow rift makes the whole source `unavailable`, not only its + last call (task 08). +- **A missing read workspace looks empty.** Honcho answers a search or a conclusion query in a + workspace that does not exist with 200 and no hits (recorded, fixtures 34 and 35), so a + `[[honcho.read]]` entry with a wrong workspace name gives `recall` nothing, silently, not an + `unavailable` line. `ask_memory`'s chat answers 404 there, so it does show `unavailable`. +- **`ask_memory` always asks at `minimal`.** A `low` chat recorded from rift on 2026-09-28 + returned the dialectic model's raw tool-call markup as its answer. Other levels stay off until + rift answers them properly (spec, section 9). +- **Sessions from before the deployment keep their tools.** A session sends the baseline it + saved when it started; one started before M5c-2 is deployed never sees `ask_memory` or the + system prompt's new sentence, and keeps its cache (spec, section 9). +- **Queries from any session reach rift.** `recall` and `ask_memory` send their query or + question to Honcho from a session of any class, `secret` included, on the assumption that + Honcho stores no query and that rift reaches only the owner's hosts (spec, section 8, + `docs/egress.md`). If rift ever uses a hosted model or exports telemetry, a `secret` session's + queries must stop going there. A query holding a known-prefix secret is never sent. + ## Before hand-over Every task is worked through `docs/plans/CHECKLIST.md` with a reference in a scratch worktree @@ -130,9 +149,15 @@ forbidden label (T58). ## Running it +The reference (`m5c2-ref`, tasks 01 to 12) is moved to straylight as a bundle and deleted here +before the run (T18). Then, from the repository root, with the OpenCode TUI closed: + ```sh git switch -c m5c2 master OPENCODE_MODEL=straylight/ornith-1.5-35b-a3b RESUME=1 \ SLOTS_URL='http://straylight:11434/slots?model=ornith-1.5-35b-a3b' \ tools/run-plan.sh docs/plans/M5c2 01 ``` + +After task 12, the gate count is 1454. Then the device check above, including +`make verify-device` for the token budget of the `m5c()` baseline (task 04).