memoryd: the Honcho worker sends decided turns, waits out outages, retries undecided turns

Co-Authored-By: ornith-1.5-35b-a3b <agent@wntrmute.dev>

docs/implementer-log.md

@@ -6,6 +6,7 @@ reviewer adds findings under "Reviews" once per milestone.
 
 | Task | Date | Status | Gate runs | First gate | Deviations | Notes | Model |
 |---|---|---|---|---|---|---|---|
+| M5c1/11-memoryd-honcho-worker | 2026-09-28 | done | 1 | pass | none | Copied the four given files from `docs/plans/M5c1/files/`: the tests `crates/memoryd/tests/honcho_worker.rs` and `honcho_worker_more.rs` (from `honcho_worker_more.rs-task11`), a new version of `crates/memoryd/tests/honcho_serve.rs` (from `honcho_serve.rs-task11`, replacing the tree's) and the support `crates/memoryd/tests/support/worker.rs` (from `worker.rs-task11`), then the skeleton `crates/memoryd/src/honcho/worker.rs`. Step 2 showed exactly the named failures: `honcho_worker` with three (E0432 for `memoryd::honcho::worker` in the test and in `support/worker.rs`, E0599 for the missing `resume` on `TurnSync`); `honcho_worker_more` with four (the same three plus E0599 for the missing `Secret` on `StartError`); every other target compiled and `honcho_serve` passed its 6 already. Made D1/D2 (`user_text`/`final_reply` `pub` in `decide.rs`), H1 (`pub mod worker;` after `pub mod turns;` in `honcho/mod.rs`), T1 (`sessions` in `impl TurnSync`), T2 (`resume` walking `decisions` deduped by id then reading each log, and `append` delegating to `ledger.append` before `updated`'s doc), T3 (`updated` calling `self.retry` then the new `retry`), S1 (imports: drop `Ledger`, add `worker::{self, Wake}`), S2 (`Memoryd` holds `Arc<Mutex<TurnSync>>` and a `wake`), S3 (`StartError::Secret(String)`), S4 (the `Display` arm), S5 (`start` calling `worker::open`), S6 (`wake` built and `worker::spawn` bound after the sockets), S7 (`wake` in the `Memoryd`), S8 (`decide_turns` dropping the lock then `wake.notify()`), M1 (`main.rs` matching both `Ledger` and `Secret`). Filled its eleven bodies in the order of the file, deleting each instruction comment and keeping the doc comments, each from its comment and `cargo check` after: `notify` (lock, set flag, `notify_one`); `wait` (lock, `wait_timeout_while(..., |woken| !*woken)`, clear the flag); `parts` (collect `chars`, `chunks(MAX_PART)`); `read_turn` (`read_log` ok, `turn_records`, `user_text`, `final_reply`, both `to_string`); `batch` (a `Map` with `agent`/`boxmaker_session`/`turn` in order, then for each of `(text,speaker)`/`(reply,agent)` the neutralised `parts`, each a `MessageCreate` with the metadata); `ids` (`m.id.clone()`); `next` (lock, `outbox().find_map`, `Item::Turn` and `Action::Turn` by `let .. else`, skip when `refused.contains(workspace)`, build `Next` with `honcho_session`); `unreachable` (the `outage` match, log the start once, double-and-cap the wait, `Backoff`); `recovered` (`outage.take()` ends it, log once, `retry`); `retry` (lock, `sync.retry`); `record` (`append`, `Ok` -> `Done`, `Err` logs with `ledger::RUNBOOK`, sets `stopped`, `Stopped`). After the last body `cargo check -p memoryd` is clean with no warning. Step 5 all pass: `honcho_worker` 12, `honcho_worker_more` 10, `honcho_serve` 6, and every unchanged suite. Step 6 all pass: the `thread::spawn`/`Builder::new` grep prints one line in `spawn`, the three runbook-pointer greps print the three constants, and the other-crates `git diff --stat` is empty. Appended the three runbook fragments (`runbook-honcho-unreachable`, `-send-refused`, `-key-refused`) with `{ echo; cat ... } >>`; `grep -c '^## honcho-'` printed 5. `make gate` printed `gate: ok` with 1300 tests (1299 in `cargo test --workspace`, plus the 1 ignored `end_to_end` the gate runs separately). No new dependency, so `Cargo.lock` is unchanged. | ornith-1.5-35b-a3b |
 | M5c1/10-memoryd-honcho-client | 2026-09-28 | done | 1 | pass | none | Copied the given files from `docs/plans/M5c1/files/`: the tests `crates/memoryd/tests/honcho_client.rs`, `honcho_keys.rs` and `honcho_naming.rs` (from `honcho_naming.rs-task10`) and the support `crates/memoryd/tests/support/honcho.rs` (from `honcho.rs-task10`), and the directory `crates/memoryd/tests/fixtures/honcho/` (30 files, 29 JSON and a README). Step 2 showed exactly the named failures: three test targets (`honcho_client`, `honcho_keys`, `honcho_naming`) with E0432 for the `memoryd::honcho::client` and `...::naming` imports and E0599 for the missing `decisions` method on `Ledger`; every other target compiled. Made H1 (`pub mod client;` and `pub mod naming;` in `honcho/mod.rs`, the former first so the imports resolve and the latter last to keep the alphabetical order) and L1 (the `decisions` method before `outbox`, walking `order` with `.iter().filter_map` returning `(item, &self.entries.get(item)?.action)` for every decided item, `Skip` included), then copied the two skeletons; `cargo check` gave 26 warnings (unused `json`; unused `PeerCreate` and `WorkspaceCreate`; unused variables `workspace` five, `session` twice, the rest once each; the three `Clients` fields never read; the `Ensured::Workspace` and `Peer` variants never constructed). Filled its thirteen bodies in the order of each file, deleting each instruction comment and keeping the doc comments, each from its comment and `cargo check` after: in `client.rs` `kind` (one arm for the two `{ kind, .. }` variants, `"key-refused"` for `KeyRefused`), `status` (the two `{ status, .. }` arms, `Some(*status)` for `KeyRefused`), `fleet_configuration` (`Map::new`, insert each of `reasoning`, `summary`, `dream` -> `{ "enabled": false }`), `session_peers` (agent `observe_me(false)`, each speaker `observe_me(true)`), `created` (`Ok` and a `Refused` 409 are success), the free `client` (chain `.without_config_file()`, `.base_url`, `.workspace_id`, the two timeouts, `.api_key` only when there is a key, `.build()`), `key` (`config.keys.get(name)` else `None`, `secrets::load`, extend the warnings, return the secret), `Clients::client` (match the workspace to its field), `ensure_workspace` (the fleet one with `fleet_configuration`, the other two with defaults, remembered), `ensure_peer` (the agent's peer with `observe_me(false)`, remembered), `delete_conclusion` (Ok and a 404 are done); in `naming.rs` `session_name` (the id for `count <= 1`, `<id>-<count>` otherwise) and `honcho_session` (walk `decisions`, keep each `Turn` of the session's speakers each counted once in decision order, return when the turn-th is reached, else `None`). After the last `cargo check` is clean with no warning; `cargo test -p memoryd --test honcho_client --test honcho_keys --test honcho_naming` gives 13, 3 (one ignored, run by the third) and 5 pass. Step 6 all pass: `without_config_file` counts once, `expose()` appears once in the free `client`, the other-crates `git diff --stat` is empty. `make gate` printed `gate: ok` with 1278 tests (1257 before this task, plus the 21 new: 13, 3 and 5). No new dependency, so `Cargo.lock` is unchanged. | ornith-1.5-35b-a3b |
 | M5c1/09-memoryd-honcho-turns | 2026-09-28 | done | 1 | pass | none | Copied the five given files from `docs/plans/M5c1/files/`: the tests `crates/memoryd/tests/honcho_state.rs`, `honcho_turns.rs` (from `honcho_turns.rs-task09`) and `honcho_serve.rs` (from `honcho_serve.rs-task09`) and the support `crates/memoryd/tests/support/brokerd.rs` and `support/turns.rs` (from `turns.rs-task09`). Step 2 showed exactly the named failures: three test targets (`honcho_state`, `honcho_turns`, `honcho_serve`) with one E0432 each (`unresolved import memoryd::honcho::state`, `...::turns` and `memoryd::serve::StartError`); every other target compiled. Made H1 (`pub mod state;` and `pub mod turns;` after `pub mod ledger;` in `honcho/mod.rs`) and copied the two skeletons; `cargo check` gave 19 warnings (four unused imports: `UnixStream`; the five `bxcore` frame items plus `crate::honcho::state` and `turn_records`/`turns`; fourteen unused variables: `session` four, `log` twice, the rest once each; three fields never read: `sessions`, `broker_down`, `unreadable`). Filled its seven bodies in the order of the file, each from its comment and `cargo check` after: `ask` (`state.rs`) — connect, set read and write timeout to `wait`, write an `Envelope { v: PROTOCOL_VERSION, id: 1, r#final: true, msg: Message::SessionState(SessionState { session }) }`, match `read_frame`'s msg: `SessionStateReply` -> Ok, `Error` -> `Err(format!("brokerd: {}", e.detail))`, anything else the wrong-message error, every io error through `socket: <e>`; `log_path` (`sessions.join(id).join("0.jsonl")`); `read_log` (`fs::read`, NotFound -> empty, else Io with the runbook, walk the complete lines up to the last newline so a partial last line is left out, each line parsed and a bad line named with its number and the runbook); `finished` (last `Assistant` with empty `tool_calls`); `records` (`read_log`, remove from `unreadable` on Ok, log a damaged log once on Err); `state` (`ask`, log the outage and its return once each); `decidable` (`reported.min(count)`, hold the log's last turn when it is not finished). After the seventh `cargo check` is clean with no warning; `cargo test -p memoryd --test honcho_state --test honcho_turns` gives all 21 pass. Step 6 all pass: `honcho_serve` 6, `honcho_turns` 16, `honcho_state` 5, `serve` 8, `honcho_start` 4, the existing suites unchanged; the three step-8 greps print as expected and the other-crates `git diff --stat` is empty. `make gate` printed `gate: ok` with 1257 tests (1256 in `cargo test --workspace`, plus the 1 ignored `end_to_end` the gate runs separately with `BOXMAKER_BROKERD`). No new dependency, so `Cargo.lock` is unchanged. | ornith-1.5-35b-a3b |
 | M5c1/08-memoryd-honcho-decide | 2026-09-28 | done | 1 | pass | none | Copied the given test `crates/memoryd/tests/honcho_decide.rs` and the skeleton `crates/memoryd/src/honcho/decide.rs` from `docs/plans/M5c1/files/`. Step 2 showed exactly the named failure: `honcho_decide` with one E0432 (`unresolved import memoryd::honcho::decide`); every other target compiled. Made H1 (`pub mod decide;` after `pub mod config;` in `honcho/mod.rs`) and copied the skeleton; `cargo check` gave 11 warnings (one unused import `bxcore::sanitize::refuse`, ten unused variables: `records` four, `turn` four, `speaker` and `text` once each). Step 3 `cargo test -p memoryd --test honcho_decide` gave 1 pass (`the_skip_reasons_are_these_texts`) and 17 fail with `not yet implemented`. Filled its eight bodies in the order of the file, deleting each instruction comment as it went and keeping the doc comments, each from its comment and `cargo check` after: `turns` (filter `User`, `u64::try_from(count).unwrap_or(u64::MAX)`); `bounds` (walk `enumerate`, when a second `User` follows the tracked turn-th one return `(start, index)`, else count and set `start` at `seen == turn`, then `start.map(|s| (s, len))`); `turn_records` (`bounds` then `get`); `peer_name` (`replace('.', "_")`); `has_secret` (map each char `refuse` refuses on its own to a space via a 4-byte `encode_utf8` buffer, then keep only a `secret-` rule); `user_text` (first record's `User`, `speaker.as_deref()`); `final_reply` (last `Assistant` with empty `tool_calls`, `content.as_deref()`); `labelled_untrusted` (any `ToolResult { untrusted: true }`). After the eighth `cargo check` is clean with no warning; `cargo test` gives all 18 pass. Step 6 all pass: the grep for `fs::`/`std::io`/`Timestamp::now` prints 0, the two `refuse(` lines (87, 94) are both in `has_secret`, and the other-crates `git diff --stat` is empty. `make gate` printed `gate: ok` with 1230 tests (1229 in `cargo test --workspace`, plus the 1 ignored `end_to_end` the gate runs separately with `BOXMAKER_BROKERD` set). No new dependency, so `Cargo.lock` is unchanged. | ornith-1.5-35b-a3b |

docs/runbook.md

@@ -1010,3 +1010,91 @@ log names the problem, usually `broker-state-damaged`.
 
 **Check.** At the next turn's end `memoryd` logs that `brokerd` answers again, and
 `memory/honcho.jsonl` gains a `decided` line for each turn that waited.
+
+## honcho-unreachable
+
+**What you see.** `memoryd` logs one line when it cannot reach Honcho, then this entry:
+
+```text
+memoryd: Honcho is unreachable (connect); sends wait and are retried; see docs/runbook.md#honcho-unreachable
+```
+
+The word in parentheses is the kind: `connect` (nothing answers at `[honcho] url`), `timeout` (5
+seconds to connect, or 30 without a byte), `closed`, `io`, or `server, HTTP 5xx` (Honcho answered
+with a server error). When a send goes through again, `memoryd` logs `memoryd: Honcho answers
+again; waiting sends go on`. Conversations are not affected.
+
+**Why.** Every finished turn that is decided for Honcho waits in the ledger,
+`memory/honcho.jsonl`, until it is sent (M5c spec, section 5). While Honcho does not answer,
+`memoryd` tries the oldest waiting turn again, waiting 5 seconds, then 10, 20 and so on up to 10
+minutes between tries, and logs the outage once, not at every try. Nothing is lost: the turns go,
+in order, once Honcho answers. A restart of `memoryd` starts the waits again from 5 seconds.
+
+**Confirm.**
+
+```sh
+grep -n '^url' /etc/boxmaker/memoryd.toml
+curl -sS -m 5 "<url>/health"
+```
+
+**Fix.** Start Honcho, or make the network path to it work (on straylight, rift over the
+tailnet: `tailscale status`). If `[honcho] url` is wrong, correct it and restart `memoryd`.
+
+**Check.** `memoryd` logs `Honcho answers again`, and `memory/honcho.jsonl` gains `sent` lines.
+
+## honcho-send-refused
+
+**What you see.** `memoryd` logs one line for a turn that Honcho refused, or that could not be
+sent at all, then this entry:
+
+```text
+memoryd: turn 3 of session mm-chat-1 was not sent: Honcho refused the request (status, HTTP 422); it is marked failed; see docs/runbook.md#honcho-send-refused
+```
+
+or `memoryd: turn 3 of session mm-chat-1 cannot be sent (too-long); it is marked failed; …` for a
+turn whose text needs more than 100 messages of 25,000 characters.
+
+**Why.** Honcho answered with a 4xx other than 401 or 403, a redirect, or a body `roncho` could
+not read; sending the same again would get the same answer, or could store it twice. So the turn
+is marked `failed` in `memory/honcho.jsonl`, with the kind and the status, and the next turn goes
+on. The line never holds the turn's text or Honcho's answer (a 422 echoes the whole request).
+
+**Confirm.**
+
+```sh
+grep -n '"type":"failed"' "$BOXMAKER_HOME/memory/honcho.jsonl"
+```
+
+A 422 usually means a name Honcho does not accept, or text over its limit; a 404, a workspace or
+session removed on the Honcho side.
+
+**Fix.** A failed turn is not sent again by itself. To send it again after fixing the cause,
+stop `memoryd`, keep a copy of the ledger, remove the turn's `failed` line (see
+`honcho-ledger-damaged` for what removing a line means), and start `memoryd`.
+
+**Check.** The next turns are sent (`sent` lines follow).
+
+## honcho-key-refused
+
+**What you see.** `memoryd` logs one line when Honcho refuses a workspace's key, then this entry:
+
+```text
+memoryd: Honcho refused the key of workspace boxmaker0-trusted (HTTP 401); nothing more is sent to it until memoryd restarts; see docs/runbook.md#honcho-key-refused
+```
+
+**Why.** Each workspace is sent to with its own key, `[secrets.honcho_trusted]`,
+`honcho_untrusted` or `honcho_fleet` (M5c spec, section 10), or none. A 401 or 403 means the key
+is wrong, expired, or missing where Honcho now wants one. Trying again would be refused again, so
+`memoryd` stops sending to that workspace until it restarts; its turns stay waiting in
+`memory/honcho.jsonl`, and the other workspaces go on. The key is never printed.
+
+**Confirm.** Which workspace is named, and where its key comes from:
+
+```sh
+grep -n -A2 '^\[secrets\.honcho_' /etc/boxmaker/memoryd.toml
+```
+
+**Fix.** Put the right key where `[secrets.<name>]` points (see `secret-unavailable` for the
+sources), or remove the table if the workspace needs no key, then restart `memoryd`.
+
+**Check.** After the restart the waiting turns are sent, and no line names this entry.