brokerd: runbook pointers for startup failures, no thread panics, args_os

M3a review findings 3, 5 (the cast), 6, 7 (brokerd), 11. A config, directory
or socket failure at start now ends with docs/runbook.md#brokerd-start-failed,
and losing a listener with #brokerd-listener-lost; both entries are new.
Threads start through thread::Builder, so a refused thread is reported
instead of silently killing a listener; an aborted connection no longer stops
the daemon. brokerd reads args_os and keeps the config path as a path. The
"requester went away" result is recorded at the time it happens.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-09-22 21:07:59 -07:00
co-authored by Claude Opus 5.5
parent eed0a221ca
commit 6af89f7c60
7 changed files with 284 additions and 50 deletions
+58
View File
@@ -163,6 +163,64 @@ does not help and is not needed.
**Check.** `pgrep -a brokerd` shows one process.
## brokerd-start-failed
**What you see.** `brokerd` exits 1 at start, before it serves anything, with one line naming its
config file, a directory or a socket, then this entry. `loopd` then reports
[broker-unavailable](#broker-unavailable) for every tool call.
**Why.** `brokerd` could not read or parse its config, could not create or make private (0700) the
directory a socket lives in, or could not bind a socket or make it private (0600). It will not
serve on a socket whose permissions it could not set, because those permissions are what keep other
programs off it.
**Confirm.** The line says which:
- `<config path>: …` — the file is missing, unreadable, or not valid TOML for `brokerd.toml`
(unknown keys are errors). Check it against `docs/specs/2026-09-18-m3a-decision-path.md`,
section 2, "Configuration".
- `cannot prepare <dir>: …` — the directory cannot be made or `chmod`ed:
`ls -ld <dir> "$(dirname <dir>)"`. A path that runs through a file, or a directory owned by
another user, gives this.
- `cannot listen on <socket>: …``path must be shorter than SUN_LEN` means the socket path is
longer than 107 bytes; `Address already in use` means something still listens there
(`ss -xlp | grep <socket>`).
**Fix.** Correct the config, or the ownership of the directory, or choose a shorter socket path in
`[sockets]` (and the same path in `loopd`'s `[broker] socket`). If another process holds the
socket, stop it; `brokerd` removes a stale socket file by itself.
**Check.** `brokerd serve --config <path>` prints `brokerd: serving tools on … and approvals on …`.
## brokerd-listener-lost
**What you see.** Either `brokerd` exits 1 after it had been serving, with
`brokerd: stopped serving: <error>` and this entry; or it keeps running and prints
`brokerd: cannot start a thread for a connection, so it was closed`. In both cases `loopd` reports
[broker-unavailable](#broker-unavailable) for the calls that were refused.
**Why.** The system refused `brokerd` something it needs to serve: `accept` failed on a socket for a
reason other than one aborted connection, or a thread could not be started. The usual cause is a
limit: open files (`EMFILE`), processes or threads for the user, or memory. A connection that is
refused gets no decision, so nothing runs for it.
**Confirm.**
```sh
ulimit -n; ulimit -u
ls /proc/$(pgrep -x brokerd)/fd | wc -l # while it runs
ps -o nlwp= -p $(pgrep -x brokerd) # its thread count
```
Many threads usually means many connections waiting on approvals, or a client that opens
connections and never sends: look at `bxctl approvals` and at which process holds the sockets
(`ss -xp | grep broker`).
**Fix.** Answer or let expire the pending approvals, stop whatever is flooding the socket, or raise
the limit. Then start `brokerd` again if it exited.
**Check.** `brokerd` prints `serving tools on …`, and a tool call is decided again.
## broker-state-damaged
**What you see.** `brokerd` prints an error reading or writing