memoryd: the Honcho worker sends decided turns, waits out outages, retries undecided turns

Co-Authored-By: ornith-1.5-35b-a3b <agent@wntrmute.dev>

docs/runbook.md

@@ -1010,3 +1010,91 @@ log names the problem, usually `broker-state-damaged`.
 
 **Check.** At the next turn's end `memoryd` logs that `brokerd` answers again, and
 `memory/honcho.jsonl` gains a `decided` line for each turn that waited.
+
+## honcho-unreachable
+
+**What you see.** `memoryd` logs one line when it cannot reach Honcho, then this entry:
+
+```text
+memoryd: Honcho is unreachable (connect); sends wait and are retried; see docs/runbook.md#honcho-unreachable
+```
+
+The word in parentheses is the kind: `connect` (nothing answers at `[honcho] url`), `timeout` (5
+seconds to connect, or 30 without a byte), `closed`, `io`, or `server, HTTP 5xx` (Honcho answered
+with a server error). When a send goes through again, `memoryd` logs `memoryd: Honcho answers
+again; waiting sends go on`. Conversations are not affected.
+
+**Why.** Every finished turn that is decided for Honcho waits in the ledger,
+`memory/honcho.jsonl`, until it is sent (M5c spec, section 5). While Honcho does not answer,
+`memoryd` tries the oldest waiting turn again, waiting 5 seconds, then 10, 20 and so on up to 10
+minutes between tries, and logs the outage once, not at every try. Nothing is lost: the turns go,
+in order, once Honcho answers. A restart of `memoryd` starts the waits again from 5 seconds.
+
+**Confirm.**
+
+```sh
+grep -n '^url' /etc/boxmaker/memoryd.toml
+curl -sS -m 5 "<url>/health"
+```
+
+**Fix.** Start Honcho, or make the network path to it work (on straylight, rift over the
+tailnet: `tailscale status`). If `[honcho] url` is wrong, correct it and restart `memoryd`.
+
+**Check.** `memoryd` logs `Honcho answers again`, and `memory/honcho.jsonl` gains `sent` lines.
+
+## honcho-send-refused
+
+**What you see.** `memoryd` logs one line for a turn that Honcho refused, or that could not be
+sent at all, then this entry:
+
+```text
+memoryd: turn 3 of session mm-chat-1 was not sent: Honcho refused the request (status, HTTP 422); it is marked failed; see docs/runbook.md#honcho-send-refused
+```
+
+or `memoryd: turn 3 of session mm-chat-1 cannot be sent (too-long); it is marked failed; …` for a
+turn whose text needs more than 100 messages of 25,000 characters.
+
+**Why.** Honcho answered with a 4xx other than 401 or 403, a redirect, or a body `roncho` could
+not read; sending the same again would get the same answer, or could store it twice. So the turn
+is marked `failed` in `memory/honcho.jsonl`, with the kind and the status, and the next turn goes
+on. The line never holds the turn's text or Honcho's answer (a 422 echoes the whole request).
+
+**Confirm.**
+
+```sh
+grep -n '"type":"failed"' "$BOXMAKER_HOME/memory/honcho.jsonl"
+```
+
+A 422 usually means a name Honcho does not accept, or text over its limit; a 404, a workspace or
+session removed on the Honcho side.
+
+**Fix.** A failed turn is not sent again by itself. To send it again after fixing the cause,
+stop `memoryd`, keep a copy of the ledger, remove the turn's `failed` line (see
+`honcho-ledger-damaged` for what removing a line means), and start `memoryd`.
+
+**Check.** The next turns are sent (`sent` lines follow).
+
+## honcho-key-refused
+
+**What you see.** `memoryd` logs one line when Honcho refuses a workspace's key, then this entry:
+
+```text
+memoryd: Honcho refused the key of workspace boxmaker0-trusted (HTTP 401); nothing more is sent to it until memoryd restarts; see docs/runbook.md#honcho-key-refused
+```
+
+**Why.** Each workspace is sent to with its own key, `[secrets.honcho_trusted]`,
+`honcho_untrusted` or `honcho_fleet` (M5c spec, section 10), or none. A 401 or 403 means the key
+is wrong, expired, or missing where Honcho now wants one. Trying again would be refused again, so
+`memoryd` stops sending to that workspace until it restarts; its turns stay waiting in
+`memory/honcho.jsonl`, and the other workspaces go on. The key is never printed.
+
+**Confirm.** Which workspace is named, and where its key comes from:
+
+```sh
+grep -n -A2 '^\[secrets\.honcho_' /etc/boxmaker/memoryd.toml
+```
+
+**Fix.** Put the right key where `[secrets.<name>]` points (see `secret-unavailable` for the
+sources), or remove the table if the workspace needs no key, then restart `memoryd`.
+
+**Check.** After the restart the waiting turns are sent, and no line names this entry.