kyle/boxmaker
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

memoryd: a reported turn decided after loopd has its answer, from brokerd and the log

Browse Source 
Co-Authored-By: ornith-1.5-35b-a3b <agent@wntrmute.dev>
This commit is contained in:
kyle
2026-09-28 11:59:34 -07:00
co-authored by ornith-1.5-35b-a3b
parent a339eb58b6
commit 28df2a0e3b
12 changed files with 1483 additions and 22 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/implementer-log.md
+1
View File
@@ -6,6 +6,7 @@ reviewer adds findings under "Reviews" once per milestone.
| Task | Date | Status | Gate runs | First gate | Deviations | Notes | Model |
|---|---|---|---|---|---|---|---|
| M5c1/09-memoryd-honcho-turns | 2026-09-28 | done | 1 | pass | none | Copied the five given files from `docs/plans/M5c1/files/`: the tests `crates/memoryd/tests/honcho_state.rs`, `honcho_turns.rs` (from `honcho_turns.rs-task09`) and `honcho_serve.rs` (from `honcho_serve.rs-task09`) and the support `crates/memoryd/tests/support/brokerd.rs` and `support/turns.rs` (from `turns.rs-task09`). Step 2 showed exactly the named failures: three test targets (`honcho_state`, `honcho_turns`, `honcho_serve`) with one E0432 each (`unresolved import memoryd::honcho::state`, `...::turns` and `memoryd::serve::StartError`); every other target compiled. Made H1 (`pub mod state;` and `pub mod turns;` after `pub mod ledger;` in `honcho/mod.rs`) and copied the two skeletons; `cargo check` gave 19 warnings (four unused imports: `UnixStream`; the five `bxcore` frame items plus `crate::honcho::state` and `turn_records`/`turns`; fourteen unused variables: `session` four, `log` twice, the rest once each; three fields never read: `sessions`, `broker_down`, `unreadable`). Filled its seven bodies in the order of the file, each from its comment and `cargo check` after: `ask` (`state.rs`) — connect, set read and write timeout to `wait`, write an `Envelope { v: PROTOCOL_VERSION, id: 1, r#final: true, msg: Message::SessionState(SessionState { session }) }`, match `read_frame`'s msg: `SessionStateReply` -> Ok, `Error` -> `Err(format!("brokerd: {}", e.detail))`, anything else the wrong-message error, every io error through `socket: <e>`; `log_path` (`sessions.join(id).join("0.jsonl")`); `read_log` (`fs::read`, NotFound -> empty, else Io with the runbook, walk the complete lines up to the last newline so a partial last line is left out, each line parsed and a bad line named with its number and the runbook); `finished` (last `Assistant` with empty `tool_calls`); `records` (`read_log`, remove from `unreadable` on Ok, log a damaged log once on Err); `state` (`ask`, log the outage and its return once each); `decidable` (`reported.min(count)`, hold the log's last turn when it is not finished). After the seventh `cargo check` is clean with no warning; `cargo test -p memoryd --test honcho_state --test honcho_turns` gives all 21 pass. Step 6 all pass: `honcho_serve` 6, `honcho_turns` 16, `honcho_state` 5, `serve` 8, `honcho_start` 4, the existing suites unchanged; the three step-8 greps print as expected and the other-crates `git diff --stat` is empty. `make gate` printed `gate: ok` with 1257 tests (1256 in `cargo test --workspace`, plus the 1 ignored `end_to_end` the gate runs separately with `BOXMAKER_BROKERD`). No new dependency, so `Cargo.lock` is unchanged. | ornith-1.5-35b-a3b |
| M5c1/08-memoryd-honcho-decide | 2026-09-28 | done | 1 | pass | none | Copied the given test `crates/memoryd/tests/honcho_decide.rs` and the skeleton `crates/memoryd/src/honcho/decide.rs` from `docs/plans/M5c1/files/`. Step 2 showed exactly the named failure: `honcho_decide` with one E0432 (`unresolved import memoryd::honcho::decide`); every other target compiled. Made H1 (`pub mod decide;` after `pub mod config;` in `honcho/mod.rs`) and copied the skeleton; `cargo check` gave 11 warnings (one unused import `bxcore::sanitize::refuse`, ten unused variables: `records` four, `turn` four, `speaker` and `text` once each). Step 3 `cargo test -p memoryd --test honcho_decide` gave 1 pass (`the_skip_reasons_are_these_texts`) and 17 fail with `not yet implemented`. Filled its eight bodies in the order of the file, deleting each instruction comment as it went and keeping the doc comments, each from its comment and `cargo check` after: `turns` (filter `User`, `u64::try_from(count).unwrap_or(u64::MAX)`); `bounds` (walk `enumerate`, when a second `User` follows the tracked turn-th one return `(start, index)`, else count and set `start` at `seen == turn`, then `start.map(|s| (s, len))`); `turn_records` (`bounds` then `get`); `peer_name` (`replace('.', "_")`); `has_secret` (map each char `refuse` refuses on its own to a space via a 4-byte `encode_utf8` buffer, then keep only a `secret-` rule); `user_text` (first record's `User`, `speaker.as_deref()`); `final_reply` (last `Assistant` with empty `tool_calls`, `content.as_deref()`); `labelled_untrusted` (any `ToolResult { untrusted: true }`). After the eighth `cargo check` is clean with no warning; `cargo test` gives all 18 pass. Step 6 all pass: the grep for `fs::`/`std::io`/`Timestamp::now` prints 0, the two `refuse(` lines (87, 94) are both in `has_secret`, and the other-crates `git diff --stat` is empty. `make gate` printed `gate: ok` with 1230 tests (1229 in `cargo test --workspace`, plus the 1 ignored `end_to_end` the gate runs separately with `BOXMAKER_BROKERD` set). No new dependency, so `Cargo.lock` is unchanged. | ornith-1.5-35b-a3b |
| M5c1/07-memoryd-honcho-ledger | 2026-09-28 | done | 1 | pass | none | Copied the three given files from `docs/plans/M5c1/files/`: the tests `crates/memoryd/tests/honcho_ledger.rs`, `honcho_ledger_file.rs` and the support `crates/memoryd/tests/support/ledger.rs`. Step 2 showed exactly the named failures: two test targets (`honcho_ledger` and `honcho_ledger_file`) with two E0432 each (`unresolved import memoryd::honcho::ledger`), one in each test file and one in `support/ledger.rs`; every other target compiled. Made H1 (`pub mod ledger;` after `pub mod config;` in `honcho/mod.rs`) and copied the skeleton `crates/memoryd/src/honcho/ledger.rs`; `cargo check` gave 20 warnings from the unfilled bodies. Step 3 `cargo test -p memoryd --test honcho_ledger` gave 2 pass (`each_record_is_one_line_exactly_as_given`, `an_unknown_field_or_value_is_refused`) and 10 fail with `not yet implemented`. Filled its twelve bodies in the order of the file, each from its comment and `cargo check` after: `is_decided` (`contains_key`); `outbox` (walk `order` with `.iter().filter_map`, keeping `(item, action)` when `state == Waiting`); `sent_ids` (match `State::Sent(ids)` -> `ids.as_slice()`); `conclusion` (walk `sends` in order, pushing a sent `Conclude`'s first id and popping on a matching sent `Unconclude`, returning `live.pop()` so only workspace and text match and a `Sent` with no ids pushes nothing); `counts` (one per entry, matching `&entry.state` since `State` is not `Copy`); `check` (`AlreadyDecided` for a second `Decided`, `NotWaiting` unless a `Sent`/`Failed` finds its entry `Waiting`); `apply` (`Skip` -> `Skipped` else `Waiting`, push to `order` and `entries`; `Sent` sets `Sent(ids)` and pushes to `sends`, `Failed` sets `Failed` without pushing); `write_line` (append, create, mode 0600, one `write_all` then `sync_data`); then `read` (NotFound -> None, else Io), `complete` (rposition of `\n`), `parse` (serde error -> Damaged), `cut` (`set_len` then `sync_data`). Removed the `mut` in `cut` that clippy flagged (`set_len` and `sync_data` take `&self`); after the twelfth `cargo check` was clean and `cargo fmt` left the file. Both test files passed 12 and 12; appended `runbook-honcho-ledger-damaged.md` (`grep -c '^## honcho-ledger-damaged$'` printed 1). Step 7: `sync_data` 2, `write_all` 1, one runbook pointer line in `src`, no `create_dir`, no diff in the other crates. `make gate` printed `gate: ok` with 1212 tests (1211 in `cargo test --workspace`, plus the ignored `end_to_end` the gate runs with `BOXMAKER_BROKERD` set). No new dependency, so `Cargo.lock` is unchanged. | ornith-1.5-35b-a3b |
| M5c1/06-memoryd-honcho-config | 2026-09-28 | done | 1 | pass | none | Copied the three given tests from `docs/plans/M5c1/files/crates/memoryd/tests/`: `honcho_config.rs`, `honcho_reasons.rs`, `honcho_start.rs`. Step 2 showed exactly the named failures: `honcho_config` with eight (E0433 cannot find `honcho` in `memoryd`, E0432 unresolved import `bxcore::secrets`, two E0599 for `honcho_config`, three for `state_socket`, one for the `Honcho` variant) and `honcho_reasons` with two (the same E0433 and E0432); `honcho_start` compiled. Made K1 (`bxcore` gains `features = ["secrets"]`) and L1 (`pub mod honcho;` after `pub mod files;`), created `crates/memoryd/src/honcho/mod.rs` with the given one-line module, and copied the skeleton `crates/memoryd/src/honcho/config.rs`. Filled its seven bodies in the given order, each from its comment and `cargo check` after: `is_name` (not empty, byte length `<= MAX_NAME`, every byte `is_ascii_alphanumeric()` or `_` or `-`, bytes not chars so `é` and U+0660 fail); `check_url` (strip `https://` then `http://`, `split_once(':')` into host and port, a `host_ok` helper for the non-empty ASCII-alnum/`.`/`-` host, a port that is non-empty, all digits and parses to `u16` `> 0` via `ok().is_some_and`, the `URL_REASON` which never quotes the URL); `check_own_workspaces` (the three pairs in order, first equal pair is R5 with its name); `trust_of` (match on `as_deref`, `None`/`Some("untrusted")` is `Untrusted`, `Some("trusted")` is `Trusted`, else R9 with the written text); `class_of` (same shape, `Private`/`Public`/`Secret`, else R10); `read_workspace` (check agent then workspace names, R6 the agent is this agent, R7 the agent is listed in `earlier`, R8 the workspace is one of the three, then the `ReadWorkspace` with `trust_of`/`class_of`); `check_keys` (a name other than the three is a read key only when `strip_prefix(READ_KEY)` names an agent in `read`, else R11 with the suffix or R12 without the prefix, then `spec.source()` with R13). Made C1-C8 in `config.rs` (module comment, imports with `BTreeMap`/`SecretSpec`/`honcho`, `Sockets.state`, `Config.honcho` and `.secrets`, `ConfigError::Honcho` and its Display arm, `load` calls `honcho_config` before returning, `honcho_config` and `state_socket`), and M1-M2 in `main.rs` (imports and the pointer matched by error kind on the `Config::load` arm); the pointer edit first landed on the `instance::lock` arm and I moved it to the `Config::load` arm, the diff being exactly M1-M2. Appended `runbook-honcho-config-invalid.md` to `docs/runbook.md` (`grep -c '^## honcho-config-invalid$'` printed 1) and made D1 in `docs/dependencies.md`. Step 7: `cargo build -p memoryd` builds, the runbook grep prints one line, the `secrets::load`/`read_checked` grep prints nothing, the other-crates diff is empty. `cargo test -p memoryd` every suite passes (config 4, honcho_config 8, honcho_reasons 5, honcho_start 4). `make gate` printed `gate: ok` with 1188 tests (1187 in `cargo test --workspace` plus the ignored `end_to_end` the gate runs separately with `BOXMAKER_BROKERD`; 1171 before this task plus the 17 new). No new dependency, so `Cargo.lock` is unchanged. | ornith-1.5-35b-a3b |
docs/runbook.md
+65 -16
View File
@@ -368,7 +368,9 @@ that does not parse tool calls, or whose cache does not work.
## session-log-damaged
**What you see.** A turn on an existing session fails with `<file>:<line>: <reason>` naming
`sessions/<id>/<epoch>.jsonl`, and this entry. Other sessions work.
`sessions/<id>/<epoch>.jsonl`, and this entry. Other sessions work. With `[honcho]`, `memoryd`
logs the same line once, starting `memoryd: `, and that session's turns wait undecided for Honcho
until the log reads again.
**Why.** `loopd` replays the log exactly to rebuild the conversation, so a line it cannot read
would change what the model sees. It refuses rather than guess. `last line does not end in a
@@ -926,23 +928,27 @@ tables and every `[secrets.honcho_…]`.
## honcho-ledger-damaged
**What you see.** `memoryd` logs a line naming `memory/honcho.jsonl`, a line number and a reason,
then this entry:
**What you see.** `memoryd` exits 1 at start, before it binds its sockets, with one line naming
`memory/honcho.jsonl`, a line number and a reason, then this entry:
```text
/srv/bx/memory/honcho.jsonl:12: turn 3 of session chat-1: decided already; see docs/runbook.md#honcho-ledger-damaged
memoryd: /srv/bx/memory/honcho.jsonl:12: turn 3 of session chat-1: decided already; see docs/runbook.md#honcho-ledger-damaged
```
Or, while it runs, `memoryd: <path>: <error>; see docs/runbook.md#honcho-ledger-damaged` when the
ledger cannot be written (a full disk, a wrong owner). The turn waits undecided, and is decided at
a later turn's end once the ledger can be written.
**Why.** The ledger is `memoryd`'s record of what it sends to Honcho: one line when a turn or a
memory write is decided (where it goes, or that it goes nowhere), and one when it is sent or
refused (M5c spec, section 5). What is decided and not yet sent is the outbox. `memoryd` reads the
ledger through at start so that it sends nothing twice and nothing to a workspace it was not
decided for. A line it cannot read, or one that breaks the ledger's rules, could do either, so it
does not guess: it does not start its Honcho sync, and nothing is decided or sent to Honcho until
the ledger reads through. The rules: an item is decided once (`decided already`), and it is sent
or refused once, only after it is decided and only when it was not decided to go nowhere (`not
waiting to be sent`). A last line cut off by a crash is not damage: `memoryd` drops it and says so
in its log.
does not guess: it does not start. While it is stopped no memory is served either, and `loopd`
cannot start a session (`memory-unavailable`). The rules: an item is decided once (`decided
already`), and it is sent or refused once, only after it is decided and only when it was not
decided to go nowhere (`not waiting to be sent`). A last line cut off by a crash is not damage:
`memoryd` drops it and says so in its log.
**Confirm.** Look at the line named, and at every line of the same item:
@@ -952,12 +958,55 @@ sed -n '<line>p' "$f"
grep -n '"session":"<id>","turn":<n>}' "$f" # a turn; for an edit: '"id":"<id>","part":<n>}'
```
**Fix.** Keep a copy first: `cp -a "$f" "$f.damaged-$(date +%s)"`. The ledger is written only by
`memoryd`, so a damaged line was changed by hand or by the disk. If you know what the line should
say, correct it. Otherwise remove it (`sed -i '<line>d' "$f"`), knowing what that costs: a removed
`sent` or `failed` line means the item is sent again (a turn's messages appear twice in Honcho, a
conclusion is made twice); a removed `decided` line means the item is decided again, and any
later `sent` or `failed` line of the same item then breaks the rules, so remove those too. Honcho
cannot delete a message, so a turn sent twice stays twice.
**Fix.** If the ledger cannot be written, free space or correct the owner of `memory/`
(`df -h`, `ls -l "$f"`); nothing else is needed. For a damaged line, keep a copy first:
`cp -a "$f" "$f.damaged-$(date +%s)"`. The ledger is written only by `memoryd`, so a damaged line
was changed by hand or by the disk. If you know what the line should say, correct it. Otherwise
remove it (`sed -i '<line>d' "$f"`), knowing what that costs: a removed `sent` or `failed` line
means the item is sent again (a turn's messages appear twice in Honcho, a conclusion is made
twice); a removed `decided` line means the item is decided again, and any later `sent` or
`failed` line of the same item then breaks the rules, so remove those too. Honcho cannot delete a
message, so a turn sent twice stays twice.
**Check.** `memoryd` starts, and its log has no line with this entry.
## broker-state-unavailable
**What you see.** `memoryd` logs one line when it cannot get a session's state from `brokerd`,
then this entry:
```text
memoryd: /srv/bx/run/memory-broker/state.sock: Connection refused (os error 111); turns wait undecided; see docs/runbook.md#broker-state-unavailable
```
The reason may instead be `No such file or directory` (nothing listens), a timeout
(`frame I/O error: Resource temporarily unavailable`, after 5 seconds without a byte), or
`brokerd: <its error>` when `brokerd` answered with an error. When `brokerd` answers again,
`memoryd` logs `memoryd: brokerd answers on its state socket again; turns are decided`.
Conversations are not affected.
**Why.** With `[honcho]`, `memoryd` decides where each finished turn goes in Honcho: the trusted
workspace, the untrusted one, or nowhere (M5c spec, section 5). It needs `brokerd`'s record of the
session's class and trust for that, and never guesses it: a guess could put a turn from a session
that read a web page in the trusted workspace. So nothing is decided while `brokerd` does not
answer. The turns wait, in memory, and are decided at the next turn's end of any session once
`brokerd` answers, from the session's state then (which can only have moved towards untrusted).
The outage is logged once, not at every turn. A restart of `memoryd` forgets the waiting turns
until each of their sessions has another turn.
**Confirm.**
```sh
ls -l "$BOXMAKER_HOME/run/memory-broker/state.sock"
grep -n 'state' /etc/boxmaker/memoryd.toml /etc/boxmaker/brokerd.toml
```
`memoryd`'s `[sockets] state` and `brokerd`'s `[sockets] state` must name the same path (both
default to `<home>/run/memory-broker/state.sock`). If `brokerd` answered with an error, its own
log names the problem, usually `broker-state-damaged`.
**Fix.** Start `brokerd` if it is stopped; make the two `[sockets] state` paths agree; or, for
`brokerd`'s error, follow the entry its log names.
**Check.** At the next turn's end `memoryd` logs that `brokerd` answers again, and
`memory/honcho.jsonl` gains a `decided` line for each turn that waited.