brokerd: runbook pointers for startup failures, no thread panics, args_os
M3a review findings 3, 5 (the cast), 6, 7 (brokerd), 11. A config, directory or socket failure at start now ends with docs/runbook.md#brokerd-start-failed, and losing a listener with #brokerd-listener-lost; both entries are new. Threads start through thread::Builder, so a refused thread is reported instead of silently killing a listener; an aborted connection no longer stops the daemon. brokerd reads args_os and keeps the config path as a path. The "requester went away" result is recorded at the time it happens. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -163,6 +163,64 @@ does not help and is not needed.
|
||||
|
||||
**Check.** `pgrep -a brokerd` shows one process.
|
||||
|
||||
## brokerd-start-failed
|
||||
|
||||
**What you see.** `brokerd` exits 1 at start, before it serves anything, with one line naming its
|
||||
config file, a directory or a socket, then this entry. `loopd` then reports
|
||||
[broker-unavailable](#broker-unavailable) for every tool call.
|
||||
|
||||
**Why.** `brokerd` could not read or parse its config, could not create or make private (0700) the
|
||||
directory a socket lives in, or could not bind a socket or make it private (0600). It will not
|
||||
serve on a socket whose permissions it could not set, because those permissions are what keep other
|
||||
programs off it.
|
||||
|
||||
**Confirm.** The line says which:
|
||||
|
||||
- `<config path>: …` — the file is missing, unreadable, or not valid TOML for `brokerd.toml`
|
||||
(unknown keys are errors). Check it against `docs/specs/2026-09-18-m3a-decision-path.md`,
|
||||
section 2, "Configuration".
|
||||
- `cannot prepare <dir>: …` — the directory cannot be made or `chmod`ed:
|
||||
`ls -ld <dir> "$(dirname <dir>)"`. A path that runs through a file, or a directory owned by
|
||||
another user, gives this.
|
||||
- `cannot listen on <socket>: …` — `path must be shorter than SUN_LEN` means the socket path is
|
||||
longer than 107 bytes; `Address already in use` means something still listens there
|
||||
(`ss -xlp | grep <socket>`).
|
||||
|
||||
**Fix.** Correct the config, or the ownership of the directory, or choose a shorter socket path in
|
||||
`[sockets]` (and the same path in `loopd`'s `[broker] socket`). If another process holds the
|
||||
socket, stop it; `brokerd` removes a stale socket file by itself.
|
||||
|
||||
**Check.** `brokerd serve --config <path>` prints `brokerd: serving tools on … and approvals on …`.
|
||||
|
||||
## brokerd-listener-lost
|
||||
|
||||
**What you see.** Either `brokerd` exits 1 after it had been serving, with
|
||||
`brokerd: stopped serving: <error>` and this entry; or it keeps running and prints
|
||||
`brokerd: cannot start a thread for a connection, so it was closed`. In both cases `loopd` reports
|
||||
[broker-unavailable](#broker-unavailable) for the calls that were refused.
|
||||
|
||||
**Why.** The system refused `brokerd` something it needs to serve: `accept` failed on a socket for a
|
||||
reason other than one aborted connection, or a thread could not be started. The usual cause is a
|
||||
limit: open files (`EMFILE`), processes or threads for the user, or memory. A connection that is
|
||||
refused gets no decision, so nothing runs for it.
|
||||
|
||||
**Confirm.**
|
||||
|
||||
```sh
|
||||
ulimit -n; ulimit -u
|
||||
ls /proc/$(pgrep -x brokerd)/fd | wc -l # while it runs
|
||||
ps -o nlwp= -p $(pgrep -x brokerd) # its thread count
|
||||
```
|
||||
|
||||
Many threads usually means many connections waiting on approvals, or a client that opens
|
||||
connections and never sends: look at `bxctl approvals` and at which process holds the sockets
|
||||
(`ss -xp | grep broker`).
|
||||
|
||||
**Fix.** Answer or let expire the pending approvals, stop whatever is flooding the socket, or raise
|
||||
the limit. Then start `brokerd` again if it exited.
|
||||
|
||||
**Check.** `brokerd` prints `serving tools on …`, and a tool call is decided again.
|
||||
|
||||
## broker-state-damaged
|
||||
|
||||
**What you see.** `brokerd` prints an error reading or writing
|
||||
|
||||
Reference in New Issue
Block a user