review: bxctl memory sync shows what the Honcho side waits on

A shared status, kept by the worker and the walk and read by the report under
its own leaf lock, holds the workspaces whose key was refused, when an outage
of Honcho began, and when brokerd stopped answering. The report adds a line
for each while it lasts, and is unchanged when all is well. The runbook's
Honcho entries and broker-state-unavailable now say what sync shows.

Tuning and Wake move to honcho/wake.rs (re-exported from worker) to keep
worker.rs under 500 lines. honcho_sync.rs's serve test now allows the outage
line: the write wakes the worker, which finds Honcho unreachable.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-09-28 14:11:31 -07:00
co-authored by Claude Opus 5.5
parent 3dd95c5791
commit 3d266df9de
12 changed files with 439 additions and 81 deletions
+46 -7
View File
@@ -924,7 +924,8 @@ is a new peer. `agent` is this agent's own name, different on each agent; `owner
the same in every agent's file. To turn Honcho off, remove `[honcho]`, its `[[honcho.read]]`
tables and every `[secrets.honcho_…]`.
**Check.** `memoryd serve --config <path>` prints `memoryd: serving`.
**Check.** `memoryd serve --config <path>` prints `memoryd: serving`, and `bxctl memory sync`
answers with its `honcho sync:` line again.
## honcho-ledger-damaged
@@ -937,7 +938,13 @@ memoryd: /srv/bx/memory/honcho.jsonl:12: turn 3 of session chat-1: decided alrea
Or, while it runs, `memoryd: <path>: <error>; see docs/runbook.md#honcho-ledger-damaged` when the
ledger cannot be written (a full disk, a wrong owner). The turn waits undecided, and is decided at
a later turn's end once the ledger can be written.
a later turn's end once the ledger can be written. When the worker cannot record a send, the line
ends `the Honcho worker stops until memoryd restarts`, and nothing more is sent.
While `memoryd` is stopped, `bxctl memory sync` cannot reach it. While it runs with a ledger it
cannot write, `bxctl memory sync` shows the counts as they were: a turn or a memory write that
could not be decided is not counted, and an item whose send could not be recorded stays
`waiting`.
**Why.** The ledger is `memoryd`'s record of what it sends to Honcho: one line when a turn or a
memory write is decided (where it goes, or that it goes nowhere), and one when it is sent or
@@ -968,7 +975,8 @@ twice); a removed `decided` line means the item is decided again, and any later
`failed` line of the same item then breaks the rules, so remove those too. Honcho cannot delete a
message, so a turn sent twice stays twice.
**Check.** `memoryd` starts, and its log has no line with this entry.
**Check.** `memoryd` starts, its log has no line with this entry, and `bxctl memory sync`'s
`waiting` count falls as the worker sends.
## broker-state-unavailable
@@ -985,6 +993,14 @@ The reason may instead be `No such file or directory` (nothing listens), a timeo
`memoryd` logs `memoryd: brokerd answers on its state socket again; turns are decided`.
Conversations are not affected.
While it lasts, `bxctl memory sync` shows, after its counts, the line
```text
brokerd unanswered since 2026-09-28T10:00:00.000Z: turns wait undecided
```
The waiting turns are not in its counts: a turn is counted once it is decided.
**Why.** With `[honcho]`, `memoryd` decides where each finished turn goes in Honcho: the trusted
workspace, the untrusted one, or nowhere (M5c spec, section 5). It needs `brokerd`'s record of the
session's class and trust for that, and never guesses it: a guess could put a turn from a session
@@ -1008,8 +1024,9 @@ log names the problem, usually `broker-state-damaged`.
**Fix.** Start `brokerd` if it is stopped; make the two `[sockets] state` paths agree; or, for
`brokerd`'s error, follow the entry its log names.
**Check.** At the next turn's end `memoryd` logs that `brokerd` answers again, and
`memory/honcho.jsonl` gains a `decided` line for each turn that waited.
**Check.** At the next turn's end `memoryd` logs that `brokerd` answers again,
`memory/honcho.jsonl` gains a `decided` line for each turn that waited, and `bxctl memory sync`
no longer shows the `brokerd unanswered` line.
## honcho-unreachable
@@ -1024,6 +1041,12 @@ seconds to connect, or 30 without a byte), `closed`, `io`, or `server, HTTP 5xx`
with a server error). When a send goes through again, `memoryd` logs `memoryd: Honcho answers
again; waiting sends go on`. Conversations are not affected.
While it lasts, `bxctl memory sync` counts what waits and shows, after its counts, the line
```text
honcho unreachable since 2026-09-28T10:00:00.000Z: sends wait and are retried
```
**Why.** Every finished turn that is decided for Honcho waits in the ledger,
`memory/honcho.jsonl`, until it is sent (M5c spec, section 5). While Honcho does not answer,
`memoryd` tries the oldest waiting turn again, waiting 5 seconds, then 10, 20 and so on up to 10
@@ -1040,7 +1063,8 @@ curl -sS -m 5 "<url>/health"
**Fix.** Start Honcho, or make the network path to it work (on straylight, rift over the
tailnet: `tailscale status`). If `[honcho] url` is wrong, correct it and restart `memoryd`.
**Check.** `memoryd` logs `Honcho answers again`, and `memory/honcho.jsonl` gains `sent` lines.
**Check.** `memoryd` logs `Honcho answers again`, `memory/honcho.jsonl` gains `sent` lines, and
`bxctl memory sync` no longer shows the `honcho unreachable` line; its `waiting` count falls.
## honcho-send-refused
@@ -1054,6 +1078,13 @@ memoryd: turn 3 of session mm-chat-1 was not sent: Honcho refused the request (s
or `memoryd: turn 3 of session mm-chat-1 cannot be sent (too-long); it is marked failed; …` for a
turn whose text needs more than 100 messages of 25,000 characters.
`bxctl memory sync` counts it as `failed`, and lists each failed item under `failed, oldest
first:` with its time, its label, its kind, and its HTTP status when Honcho answered:
```text
2026-09-28T10:03:00.000Z turn 3 of session mm-chat-1 status (HTTP 422)
```
**Why.** Honcho answered with a 4xx other than 401 or 403, a redirect, or a body `roncho` could
not read; sending the same again would get the same answer, or could store it twice. So the turn
is marked `failed` in `memory/honcho.jsonl`, with the kind and the status, and the next turn goes
@@ -1082,6 +1113,13 @@ stop `memoryd`, keep a copy of the ledger, remove the turn's `failed` line (see
memoryd: Honcho refused the key of workspace boxmaker0-trusted (HTTP 401); nothing more is sent to it until memoryd restarts; see docs/runbook.md#honcho-key-refused
```
Until `memoryd` restarts, `bxctl memory sync` counts that workspace's items as `waiting` and
shows, after its counts, one line for each workspace whose key was refused:
```text
key refused (HTTP 401): trusted workspace boxmaker0-trusted: its sends wait until memoryd restarts
```
**Why.** Each workspace is sent to with its own key, `[secrets.honcho_trusted]`,
`honcho_untrusted` or `honcho_fleet` (M5c spec, section 10), or none. A 401 or 403 means the key
is wrong, expired, or missing where Honcho now wants one. Trying again would be refused again, so
@@ -1097,4 +1135,5 @@ grep -n -A2 '^\[secrets\.honcho_' /etc/boxmaker/memoryd.toml
**Fix.** Put the right key where `[secrets.<name>]` points (see `secret-unavailable` for the
sources), or remove the table if the workspace needs no key, then restart `memoryd`.
**Check.** After the restart the waiting turns are sent, and no line names this entry.
**Check.** After the restart the waiting turns are sent, no line names this entry, and `bxctl
memory sync` shows no `key refused` line.