kyle/boxmaker
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

M5c-2 plan: README wrap-up: the tasks with their counts, known flaky tests and known limits

Browse Source 
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
kyle
2026-09-28 21:27:53 -07:00
co-authored by Claude Opus 5.5
parent f6a1f1831e
commit 62aaadd17e
1 changed files with 63 additions and 38 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/plans/M5c2/README.md
+63 -38
View File
@@ -55,8 +55,10 @@ Honcho sources and chats outside its state lock, and answers `ask_memory`.
- Roles depend on `bxcore` only; `memoryd` also on `roncho`. No `unsafe`, no async runtime.
- Every `roncho` client is built with `.without_config_file()`; keys come from
`bxcore::secrets` and are never logged; errors are logged by kind and status only.
- Text from Honcho is neutralised on arrival and capped like a local `recall` snippet (600
characters); a result's labels raise the session as a local hit's do.
- Text from Honcho is neutralised on arrival, put on one indented line, and capped: 600
characters for a `recall` hit, as a local snippet, and 2,000 for an `ask_memory` answer; a
result's labels raise the session as a local hit's do. Local `recall` items take the same form
(task 07).
- No Honcho call runs under `memoryd`'s state lock.
- Without a `[honcho]` section, `recall` is exactly M5b's and `ask_memory` answers that Honcho is
not configured.
@@ -68,23 +70,27 @@ Honcho sources and chats outside its state lock, and answers `ask_memory`.
whitespace-checked literals are marked on every edit (T55); the task's table of `roncho`'s API
is complete, and the Cargo registry is off limits (T56).
## Tasks (outline, for the owner's review)
## Tasks
| # | Delivers | Crates |
|---|---|---|
| 00 | (done by the design model, `718e66d`) recordings from rift: workspace search with a limit and with no match, conclusion queries for two peers, and what a missing workspace answers | memoryd fixtures |
| 01 | Wire: `AskMemoryArgs { question, workspaces }`, its name and its checks (question at most 2,000 characters; one or two workspace names, no repeats, each 1 to 512 characters as `memoryd`'s name rule) | bxcore |
| 02 | `brokerd`: `ask_memory` as a built-in memory tool (name, parsing, the grant `builtin-ask-memory`, owner grants naming it refused, routing to `memoryd`); its answer's labels raise the session | brokerd |
| 03 | `brokerd`: a per-tool wait on `memoryd`, 170 s for `ask_memory` (`ASK_MEMORY_WAIT`) and 30 s for the others, chosen by `MemoryPort::wait_for` | brokerd |
| 04 | `loopd`: `ask_memory`'s schema as a core tool in a new registry; a per-tool wait on `brokerd`, 180 s for `ask_memory` and `[broker] timeout_ms` for the others; the system prompt's line on memory tools | loopd |
| 05 | `memoryd`: read clients, one per `[[honcho.read]]` workspace, the fleet and the agent's own two, with a 5 s limit for searches and 150 s for chat; each source's label; `recall_sources` and `resolve`; `own`, `untrusted` and `fleet` reserved as read agents' names | memoryd |
| 06 | `memoryd`, pure: Honcho messages and conclusions as `recall` items (tag, source name, class, neutralised, capped), kept only when they share a term with the query, interleaved with the local hits to `k`, the local index first | memoryd |
| 07 | `memoryd`: local `recall` items in task 06's form, one indented line each, through one shared `snippet` and `item`; `recall::local` gives the items, their count and their label apart (T37) | memoryd |
| 08 | `memoryd`: `recall`'s Honcho part (`honcho/search.rs`): every source of `recall_sources` at once, each on its own thread with a limit of 5 s from the start; a read agent's workspace is a search and two conclusion queries; `<source>: unavailable`; a query holding a secret asks nothing; the answer from the local items and Honcho's, with the stricter label | memoryd |
| 09 | `memoryd`: `recall` with `[honcho]` runs the local part under the state lock and the Honcho part outside it; the readers taken at start; without `[honcho]`, `recall` as before | memoryd |
| 10 | `memoryd`: `ask_memory`: workspace names checked against `[honcho]`, both chats at once, each answer tagged and capped, labels, `<workspace>: unavailable`; a question holding a secret sends nothing | memoryd |
| 11 | `memoryd`, making room: `reindex` and `admin` move unchanged from `serve.rs` to `serve/admin.rs`, with tests for four of their behaviours no test pinned | memoryd |
| 12 | `memoryd` serves `ask_memory` beside `recall` and `memory`, never under the state lock; its arguments checked again; without `[honcho]`, a sentence as the result; no new fail-closed state, so no runbook entry, and `bxctl` unchanged | memoryd |
Every task below has been written, replayed from its text and checked; the gate count is the
passed total of `cargo test --workspace` plus one, for the ignored `end_to_end` test the gate
runs on its own. Before task 01 it is 1364 (talos, rustc 1.98.1, clippy of the same release).
| # | Delivers | Crates | Gate after | Mutations tried, caught |
|---|---|---|---|---|
| 00 | (done by the design model, `718e66d`) recordings from rift: workspace search with a limit and with no match, conclusion queries for two peers, and what a missing workspace answers | memoryd fixtures | 1364 | (none: fixtures only) |
| 01 | Wire: `AskMemoryArgs { question, workspaces }`, its name and its checks (question at most 2,000 characters; one or two workspace names, no repeats, each 1 to 512 characters as `memoryd`'s name rule) | bxcore | 1372 | 23, 23 |
| 02 | `brokerd`: `ask_memory` as a built-in memory tool (name, parsing, the grant `builtin-ask-memory`, owner grants naming it refused, routing to `memoryd`); its answer's labels raise the session | brokerd | 1381 | 17, 17 |
| 03 | `brokerd`: a per-tool wait on `memoryd`, 170 s for `ask_memory` (`ASK_MEMORY_WAIT`) and 30 s for the others, chosen by `MemoryPort::wait_for` | brokerd | 1384 | 10, 10 |
| 04 | `loopd`: `ask_memory`'s schema as a core tool in a new registry; a per-tool wait on `brokerd`, 180 s for `ask_memory` and `[broker] timeout_ms` for the others; the system prompt's line on memory tools | loopd | 1394 | 19, 19 |
| 05 | `memoryd`: read clients, one per `[[honcho.read]]` workspace, the fleet and the agent's own two, with a 5 s limit for searches and 150 s for chat; each source's label; `recall_sources` and `resolve`; `own`, `untrusted` and `fleet` reserved as read agents' names | memoryd | 1403 | 24, 24 |
| 06 | `memoryd`, pure: Honcho messages and conclusions as `recall` items (tag, source name, class, neutralised, capped), kept only when they share a term with the query, interleaved with the local hits to `k`, the local index first | memoryd | 1414 | 22, 22 |
| 07 | `memoryd`: local `recall` items in task 06's form, one indented line each, through one shared `snippet` and `item`; `recall::local` gives the items, their count and their label apart (T37) | memoryd | 1418 | 15, 15 |
| 08 | `memoryd`: `recall`'s Honcho part (`honcho/search.rs`): every source of `recall_sources` at once, each on its own thread with a limit of 5 s from the start; a read agent's workspace is a search and two conclusion queries; `<source>: unavailable`; a query holding a secret asks nothing; the answer from the local items and Honcho's, with the stricter label | memoryd | 1430 | 19, 19 |
| 09 | `memoryd`: `recall` with `[honcho]` runs the local part under the state lock and the Honcho part outside it; the readers taken at start; without `[honcho]`, `recall` as before | memoryd | 1434 | 8, 8 |
| 10 | `memoryd`: `ask_memory`: workspace names checked against `[honcho]`, both chats at once, each answer tagged and capped, labels, `<workspace>: unavailable`; a question holding a secret sends nothing | memoryd | 1444 | 18, 18 |
| 11 | `memoryd`, making room: `reindex` and `admin` move unchanged from `serve.rs` to `serve/admin.rs`, with tests for four of their behaviours no test pinned | memoryd | 1448 | 8, 8 |
| 12 | `memoryd` serves `ask_memory` beside `recall` and `memory`, never under the state lock; its arguments checked again; without `[honcho]`, a sentence as the result; no new fail-closed state, so no runbook entry, and `bxctl` unchanged | memoryd | 1454 | 9, 9 |
After task 12, the design model runs the M5c-2 device check (spec 13) on straylight, rift and the
owner's Mattermost: a second `memoryd` under its own name writes a fleet note that `recall` finds,
@@ -93,32 +99,45 @@ trusted workspace within its limit; with rift down, `recall` answers from the lo
`unavailable` lines. The baseline is measured again against the 3,000-token budget
(`make verify-device`).
## Counts, as tasks are written
## Known flaky tests
The gate count is the passed total of `cargo test --workspace` plus one, for the ignored
`end_to_end` test the gate runs on its own. Before task 01 it is 1364 (talos, rustc 1.98.1).
| Task | Gate count after | Mutations tried, caught |
|---|---|---|
| 01 | 1372 | 23, 23 |
| 02 | 1381 | 17, 17 |
| 03 | 1384 | 10, 10 |
| 04 | 1394 | 19, 19 |
| 05 | 1403 | 24, 24 |
| 06 | 1414 | 22, 22 |
| 07 | 1418 | 15, 15 |
| 08 | 1430 | 19, 19 |
| 09 | 1434 | 8, 8 |
| 10 | 1444 | 18, 18 |
| 11 | 1448 | 8, 8 |
| 12 | 1454 | 9, 9 |
**A flaky test, fixed before the run:** `bxctl`'s `chat_approvals` test
**Fixed before the run:** `bxctl`'s `chat_approvals` test
`json_prints_only_json_and_never_asks_brokerd` failed once in 5 loaded runs of the whole workspace
while task 06 was checked: its helper wrote to the standard input of a `bxctl chat` that does not
read it and may already have exited, and unwrapped the broken pipe. The helper now accepts a
broken pipe on that write (master, after task 06); 0 failures in 100 loaded runs.
**Seen once, not reproduced:** while task 11 was checked, one run of `memoryd`'s whole suite
failed in `honcho_worker`'s `a_turn_too_long_for_one_batch_is_marked_failed`, and another in
several unrelated `memoryd` tests at once; 18 later runs of the whole suite (12 of them beside
8 busy loops) and 10 of that test alone passed. A session that sees a `memoryd` test outside its
task fail once, then pass on a rerun and in the gate, names it in its row's Notes and goes on
(AGENTS.md).
## Known limits
These are known, accepted for M5c-2, and not bugs to fix in a task:
- **A read agent's three calls share one limit.** `recall` asks another agent's workspace one
call after another (a search, then the conclusions about the agent's peer and the owner's),
all within that source's 5 s. A slow rift makes the whole source `unavailable`, not only its
last call (task 08).
- **A missing read workspace looks empty.** Honcho answers a search or a conclusion query in a
workspace that does not exist with 200 and no hits (recorded, fixtures 34 and 35), so a
`[[honcho.read]]` entry with a wrong workspace name gives `recall` nothing, silently, not an
`unavailable` line. `ask_memory`'s chat answers 404 there, so it does show `unavailable`.
- **`ask_memory` always asks at `minimal`.** A `low` chat recorded from rift on 2026-09-28
returned the dialectic model's raw tool-call markup as its answer. Other levels stay off until
rift answers them properly (spec, section 9).
- **Sessions from before the deployment keep their tools.** A session sends the baseline it
saved when it started; one started before M5c-2 is deployed never sees `ask_memory` or the
system prompt's new sentence, and keeps its cache (spec, section 9).
- **Queries from any session reach rift.** `recall` and `ask_memory` send their query or
question to Honcho from a session of any class, `secret` included, on the assumption that
Honcho stores no query and that rift reaches only the owner's hosts (spec, section 8,
`docs/egress.md`). If rift ever uses a hosted model or exports telemetry, a `secret` session's
queries must stop going there. A query holding a known-prefix secret is never sent.
## Before hand-over
Every task is worked through `docs/plans/CHECKLIST.md` with a reference in a scratch worktree
@@ -130,9 +149,15 @@ forbidden label (T58).
## Running it
The reference (`m5c2-ref`, tasks 01 to 12) is moved to straylight as a bundle and deleted here
before the run (T18). Then, from the repository root, with the OpenCode TUI closed:
```sh
git switch -c m5c2 master
OPENCODE_MODEL=straylight/ornith-1.5-35b-a3b RESUME=1 \
SLOTS_URL='http://straylight:11434/slots?model=ornith-1.5-35b-a3b' \
tools/run-plan.sh docs/plans/M5c2 01
```
After task 12, the gate count is 1454. Then the device check above, including
`make verify-device` for the token budget of the `m5c()` baseline (task 04).