boxmaker/docs/runbook.md

Runbook

One entry for every state in which Boxmaker refuses to work or holds something back, and for events worth understanding when you see them. Every message for such a state ends with see docs/runbook.md#<entry>; a gate check makes sure the entry exists.

Each entry says what you see, why the system refuses, how to confirm, how to fix, and how to check the fix. Commands assume:

• $BOXMAKER_HOME is the harness home: /var/lib/boxmaker when deployed, .state in the repository during development.
• $GRANTS is [paths] grants from brokerd.toml: /etc/boxmaker/grants when deployed.
• Until M7 there are no service units: the daemons run in a terminal, and "stop" means Ctrl-C or pkill -x <name>.

Entries for M3a states are written ahead of the code and describe it as specified in docs/specs/2026-09-18-m3a-decision-path.md.

grants-invalid

What you see. brokerd prints one or more lines naming a grant file and a problem, then this entry. Every tool call is denied, and the model tells you "the grant files have an error".

Why. One grant file is invalid, so brokerd denies everything until it is fixed. It cannot skip the bad file: if the bad file was a deny grant, skipping it would allow what it was written to forbid.

Confirm.

bxctl grants check

It prints each problem as <file>:<line>: <problem>. Common ones: a misspelled field (unknown fields are errors); secret set (not supported until M4); patterns set (not supported); hosts on a file tool or paths on http_fetch; a relative path or one with .., . or //; a path with : or , (it cannot be mounted); the path / (grant the directories you mean); a deny grant whose max_taint is not "secret" (it would stop denying once the session read a secret); a host with uppercase letters, a port, or an IP address; a file name with characters outside [a-z0-9-].

Fix. Edit the file, or move it out of $GRANTS while you work on it:

mkdir -p ~/grants-parked && mv "$GRANTS/<file>.toml" ~/grants-parked/

Do not park a deny grant unless you are sure nothing else would then allow what it forbids.

Check. bxctl grants check prints grants: ok. No restart is needed: brokerd reads the grants again at the next call.

audit-unavailable

What you see. brokerd prints an error writing to $BOXMAKER_HOME/audit/, then this entry. Tool calls are denied, and the model says "the audit log cannot be written". Every later call prints brokerd: an earlier audit write failed; every call is denied until brokerd is restarted. A call that ran but whose result could not be recorded reaches the model as "the result could not be recorded", without its content; bxctl refuse says "the refusal could not be recorded".

A variant: brokerd: a thread panicked while holding the ledger. That is a bug in brokerd, not a disk problem; the fix below is the same, and the panic message above it in brokerd's output is worth keeping for a report.

Why. A call runs only after its decision is on disk. If the record cannot be written, nothing runs.

Confirm.

df -h "$BOXMAKER_HOME"
ls -ld "$BOXMAKER_HOME/audit"; ls -l "$BOXMAKER_HOME/audit" | tail -3

Look for a full disk, a directory or file not owned by the user brokerd runs as, or a read-only file system.

Fix. Free space or correct ownership (chown to the brokerd user; files are mode 0600, the directory 0700). Do not edit, move or delete audit files to make space: that breaks the chain. Then stop and start brokerd. It does not try again by itself: a failed write may have left part of a line, and only the startup check deals with that. Expect audit-recovered at that start.

Check. Ask for any tool call. It is decided normally, and bxctl audit verify prints audit: ok.

audit-chain-broken

What you see. brokerd refuses to start and prints <file>:<line>: <what>, then this entry. Or bxctl audit verify prints the same and exits 1. If the break is in an older file, only bxctl audit verify reports it: at an ordinary start brokerd checks the latest file alone.

Why. A record does not parse, its seq is not the next number, its prev is not the hash of the line before, or a file does not continue from the one before it. Something changed the log after it was written: an edit, a deleted or reordered line, a file restored from a backup, a partial copy. brokerd will not add records after a history it cannot vouch for.

There is one innocent cause. After a crash brokerd recovers a torn last line by itself (audit-recovered), but if it is killed a second time while writing that recovery record, the log is left with a line that nothing describes. The failure is then within the last three lines of the latest file, and the last line is cut short.

Confirm.

bxctl audit verify
sed -n '<line-1>,<line+1>p' "$BOXMAKER_HOME/audit/<file>"

Work out what happened before going on: a restore, a manual edit, a disk problem, or none you know of. If none you know of, treat it as a possible intrusion and keep the files as they are.

Fix. Keep a copy first, then accept the break explicitly:

cp -a "$BOXMAKER_HOME/audit" ~/audit-copy-$(date +%F)
brokerd serve --config <path> --accept-break

With this flag brokerd checks the whole log, not only the latest file, and appends an accepted_break record naming the file and line of the first failure. It continues from there. Nothing is repaired or deleted. The break stays visible in every later verification. One accepted break covers everything between the failure and the break record, including any further damage there, so look at that whole stretch before accepting it.

Check. bxctl audit verify prints audit: ok and lists the accepted break with its file and line. Damage that happens after the break record needs its own accepted break.

audit-recovered

What you see. brokerd prints "audit: recovered a torn final line" at start, then this entry. It starts normally.

Why. The last record was being written when brokerd or the machine stopped, so the file ended without a newline. That is expected after a crash, not an error. brokerd kept the partial line, ended it, and wrote a recovery record holding its length and hash, chained from the last complete record. The partial line may even look like a whole record; it is not one, and the recovery record takes its seq. It was never acted on: a record allows something only once it is on disk with its newline. If the partial line was the result of a call, the call did run and its result was never delivered; bxctl audit verify lists that call as unfinished.

Confirm. bxctl audit verify lists the recovery with its file and line.

Fix. None needed. If recoveries happen often, find out why brokerd is being killed.

Check. bxctl audit verify prints audit: ok.

brokerd-already-running

What you see. brokerd exits at start with "brokerd is already running" and this entry.

Why. Only one brokerd may write the audit log. It holds a lock on $BOXMAKER_HOME/audit/.lock for as long as it runs, and the lock is released when the process ends, however it ends.

Confirm.

pgrep -a brokerd

Fix. Use the running one, or stop it (pkill -x brokerd) and start again. Deleting .lock does not help and is not needed.

Check. pgrep -a brokerd shows one process.

brokerd-start-failed

What you see. brokerd exits 1 at start, before it serves anything, with one line naming its config file, a directory or a socket, then this entry. loopd then reports broker-unavailable for every tool call.

Why. brokerd could not read or parse its config, could not create or make private (0700) the directory a socket lives in, or could not bind a socket or make it private (0600). It will not serve on a socket whose permissions it could not set, because those permissions are what keep other programs off it.

Confirm. The line says which:

• <config path>: … — the file is missing, unreadable, or not valid TOML for brokerd.toml (unknown keys are errors). Check it against docs/specs/2026-09-18-m3a-decision-path.md, section 2, "Configuration".
• cannot prepare <dir>: … — the directory cannot be made or chmoded: ls -ld <dir> "$(dirname <dir>)". A path that runs through a file, or a directory owned by another user, gives this.
• cannot prepare <socket or dir>: a socket needs a directory of its own or …: is a symbolic link or not a directory — brokerd makes a socket's directory 0700, so it refuses / and a directory reached through a link (the link's target would be changed instead). Give each socket its own real directory, as the defaults under $BOXMAKER_HOME/run/ are.
• cannot listen on <socket>: … — path must be shorter than SUN_LEN means the socket path is longer than 107 bytes; Address already in use means something still listens there (ss -xlp | grep <socket>).

Fix. Correct the config, or the ownership of the directory, or choose a shorter socket path in [sockets] (and the same path in loopd's [broker] socket). If another process holds the socket, stop it; brokerd removes a stale socket file by itself.

Check. brokerd serve --config <path> prints brokerd: serving tools on … and approvals on ….

brokerd-listener-lost

What you see. One of three lines, then this entry:

• brokerd: cannot accept on accept-broker (or accept-admin) for now, retrying: <error>. It keeps running and tries every 200 ms; the line is printed once per episode.
• brokerd: cannot start a thread for a connection, so it was closed. It keeps running.
• brokerd: stopped serving: <error>, and it exits 1: any other failure of accept. In both cases loopd reports broker-unavailable for the calls that were refused.

Why. The system refused brokerd something it needs to serve: a new connection (accept failed) or a thread. The usual cause is a limit: open files (EMFILE), processes or threads for the user, or memory. A connection that is refused gets no decision, so nothing runs for it.

Confirm.

ulimit -n; ulimit -u
ls /proc/$(pgrep -x brokerd)/fd | wc -l     # while it runs
ps -o nlwp= -p $(pgrep -x brokerd)          # its thread count

Many threads usually means many connections waiting on approvals, or a client that opens connections and never sends: look at bxctl approvals and at which process holds the sockets (ss -xp | grep broker).

Fix. Answer or let expire the pending approvals, stop whatever is flooding the socket, or raise the limit. Then start brokerd again if it exited.

Check. brokerd prints serving tools on …, and a tool call is decided again.

runner-unavailable

What you see. A tool call fails with "the tool runner could not start the container", and brokerd prints what Podman said, then this entry.

Why. brokerd asked Podman to start the tool's container, or for http_fetch the egress proxy's, and Podman could not: the program is missing, the image in [runner] image is not loaded, the proxy's network does not exist, or the proxy did not create its socket in time. Nothing ran for the call.

Confirm.

podman --version                                   # the program [runner] podman names
podman image inspect --format '{{.Digest}}' <image>   # must be the digest in [runner] image
podman ps -a --filter label=boxmaker               # containers left behind, if any

Fix. Install Podman or correct [runner] podman. Load the image built from deploy/tools-image.nix (podman load < result) and put its digest in [runner] image. For the proxy, check that [runner] egress_network names a network Podman has (pasta needs the passt package). Remove leftovers with podman rm -f on the listed names.

Check. Ask for a read_file the grants allow; it returns the file.

broker-state-damaged

What you see. brokerd prints an error reading or writing $BOXMAKER_HOME/broker/sessions/<id>.json, then this entry. Either every call for that session is denied ("this session's broker state is damaged"), or one call failed with "the result could not be recorded". Such a call has no Result record, so bxctl audit verify lists it under "running or unfinished"; that is expected, not a second fault.

Why. The file holds the session's taint and untrusted flag. If brokerd cannot read it, it does not know how sensitive the session's data is, so it denies. If it cannot write it after a tool ran, it withholds the result: content that raised the taint must not reach the model unless the raised taint is on disk.

Confirm.

ls -l "$BOXMAKER_HOME/broker/sessions/<id>.json"*
cat "$BOXMAKER_HOME/broker/sessions/<id>.json"
df -h "$BOXMAKER_HOME"

A good file is one line such as {"taint":"private","untrusted":false}.

Fix. For a write failure, fix space or ownership as in audit-unavailable. For a damaged file, do not guess low: set it from the session's results. The session's audit result records show the highest taint_after, and any result from an untrusted grant means "untrusted":true. If unsure, write {"taint":"secret","untrusted":true}, which is always safe. The session only loses access to grants with a lower max_taint. A .tmp file beside it is a write that did not finish and can be removed.

Check. The next call for the session is decided normally.

socket-forbidden

What you see. brokerd prints that a message of some kind arrived on the wrong socket, then this entry.

Why. broker.sock accepts only tool requests from loopd; admin.sock accepts only admin requests from bxctl. Nothing in the harness sends a wrong kind, so this is a bug or a component doing what it should not, such as something trying to approve through the tool socket.

Confirm. Note the time, kind and socket. Check what was running: pgrep -a 'loopd|bxctl'. Look at the session logs active at that time.

Fix. If a harness version mismatch explains it (a new bxctl against an old brokerd), rebuild and restart both. Otherwise keep the logs, stop loopd, and investigate before running it again.

Check. No further messages of this kind.

broker-unavailable

What you see. loopd prints that it cannot reach broker.sock, or that brokerd did not answer in time, then this entry. Tool calls fail with "the tool broker is unavailable", the model says so, and the conversation goes on. Or, once at startup, loopd prints "no tool broker is configured" and this entry, and tool calls fail with those words.

Why. brokerd is not running, was restarted during a call, did not answer within [broker] timeout_ms, or loopd's [broker] socket points somewhere else or is not set. Pending approvals are lost when brokerd restarts. bxctl audit verify lists them as pending or abandoned.

Confirm.

pgrep -a brokerd
ls -l "$BOXMAKER_HOME/run/loop-broker/"

Compare the socket path with [broker] socket in loopd's config and [sockets] broker in brokerd.toml.

Fix. Start brokerd, or make the two paths agree. loopd needs no restart for that: it connects per call. A changed or newly set [broker] socket does need a loopd restart.

Check. Ask the model to use a tool. The call is decided (allowed, asked or denied) instead of failing.

loopd-selftest-failed

What you see. loopd prints selftest: FAILED: <reason> and this entry, and exits 1.

Why. loopd will not start against a server that differs from the one its config describes, that does not parse tool calls, or whose cache does not work.

Confirm and fix, by reason.

• chat template sha256: expected …, got …, context per slot: … or slot count: …. The server changed: a model update, or new flags in ~/src/nixos/hw/straylight/default.nix. Check what changed on straylight. If the change is intended, update [expect] in loopd's config:

  curl -s 'http://straylight:11434/props?model=ornith-1.5-35b-a3b' | jq -j .chat_template | sha256sum
  curl -s 'http://straylight:11434/props?model=ornith-1.5-35b-a3b' | jq '.default_generation_settings.n_ctx, .total_slots'
  

  A template change also invalidates the M0 measurements. Re-run the checks in docs/inference-contract.md before trusting the cache behaviour.

• the tool call did not come back parsed: …. The template or the server's tool parser changed. Treat it like a template change. Do not start loopd until tool calls parse again.

• turn 2 did not reuse turn 1's cache: …. Another client may have taken the slot between the two requests. Check curl -s 'http://straylight:11434/slots?model=ornith-1.5-35b-a3b' and run loopd selftest --config <path> again. If it fails twice with the slot otherwise idle, the cache is not working: check the server flags for cache_prompt and the KV settings.

• A connection, timeout or HTTP error. Check that inferproxy is running and that its upstream answers: curl -s http://straylight:11434/health. A model load can take tens of seconds, so retry once.

Check. loopd selftest --config <path> prints selftest: ok.

session-log-damaged

What you see. A turn on an existing session fails with <file>:<line>: <reason> naming sessions/<id>/<epoch>.jsonl, and this entry. Other sessions work.

Why. loopd replays the log exactly to rebuild the conversation, so a line it cannot read would change what the model sees. It refuses rather than guess. last line does not end in a newline means a write was cut off: loopd writes a record only after the request it describes has finished, so a cut-off line was never used.

Confirm.

f="$BOXMAKER_HOME/sessions/<id>/0.jsonl"
wc -l "$f"; tail -c 300 "$f"

Fix. Keep a copy first. Then either start a new session (always safe), or, for a cut-off last line only, drop it:

cp -a "$f" "$f.damaged-$(date +%s)"
head -n <line-1> "$f.damaged-"* > "$f"      # keep the complete lines before the damaged one

Anything else (a damaged line in the middle, an unknown record type) means the file was changed by something other than loopd. Start a new session and keep the old log for inspection.

Check. The next turn on the session succeeds. The first request re-reads the conversation if the cache was lost, which loopd logs as a cache loss.

core-memory-unreadable

What you see. Starting a new session fails with an error naming memory/core.md, and this entry. Existing sessions still work: they use the baseline saved when they started.

Why. memory/core.md exists but cannot be read. A missing file is fine, but starting without memory you wrote, and without telling you, is not.

Confirm.

ls -l "$BOXMAKER_HOME/memory/core.md"; head -c 100 "$BOXMAKER_HOME/memory/core.md"

Look for wrong ownership or mode, or a file that is not UTF-8 text.

Fix. Make it readable by the user loopd runs as (chmod 0600 and chown to that user), or save it again as UTF-8.

Check. A new session starts, and its 0.baseline.json contains the text.

gatewayd-start-failed

What you see. gatewayd exits 1 at start, before it connects, with one line naming its config file or a directory, then this entry.

Why. gatewayd could not read or parse gatewayd.toml, or a value in it is not allowed: a url that is not http:// or https:// with a host and an optional port, an id that is not 26 characters of a-z0-9, an empty [allow] users, or a secret with not exactly one of credential, env and file. Or it could not create <home>/gateway/. It will not guess at a configuration that decides who it answers.

Confirm. The line says which: <config path>: … for the file (unknown keys are errors; check it against docs/specs/2026-09-23-m4a-gateway.md, section 3), or cannot prepare <dir>: … for the directory (ls -ld <dir> "$(dirname <dir>)").

Fix. Correct the file or the directory's ownership. A user id is shown in Mattermost under the user's profile, or by GET /api/v4/users/username/<name>.

Check. gatewayd serve --config <path> prints gatewayd: connected to <url> as <user>.

secret-unavailable

What you see. gatewayd exits 1 at start with secret <name>: <reason>, then this entry. The value is never printed.

Why. gatewayd does not start without its Mattermost token, and reads it only from the one place [secrets.<name>] names. The reasons: for credential, CREDENTIALS_DIRECTORY is unset (not started by systemd with a credential) or the file in it is missing; for env, the variable is unset or empty; for file, the path is not absolute, is a symbolic link, is not a regular file, is not owned by the user gatewayd runs as, or has any group or other permission (only 0600 or 0400 are accepted). An empty value is refused in every form.

Confirm.

systemctl --user show -p LoadCredentialEncrypted gatewayd   # credential
ls -l <path>; id -u                                         # file: owner and mode

Fix. For a credential: systemd-creds --user encrypt --name=<credential> - <path>, type the token, then give the unit LoadCredentialEncrypted=<credential>:<path>. For a file: chmod 600 <path> and chown it to the user gatewayd runs as. For an environment variable, set it in the environment gatewayd starts in.

Check. gatewayd starts and prints gatewayd: connected to <url> as <user>.

secret-in-a-file

What you see. At start: gatewayd: warning: secret <name> is read in plaintext from <path>; a systemd credential keeps it encrypted at rest, then this entry. gatewayd runs normally.

Why. A file holds the token in plaintext: anyone who can read the disk, or a backup of it, can use it. A credential is encrypted to this machine's TPM and host key. Sometimes a file is right (a development machine, a system without systemd); the warning is there so the choice is deliberate.

Confirm. [secrets.<name>] in gatewayd.toml has file = ….

Fix. To keep the file: nothing; the warning stays. To move to a credential, follow the fix in secret-unavailable, change the entry to credential = "<name>", restart, then delete the file and regenerate the token if the file was ever copied elsewhere.

Check. The warning is gone at the next start.

mattermost-unreachable

What you see. gatewayd prints gatewayd: cannot reach <url>: <reason>; trying again in <n> s, then this entry, once per attempt: after 1, 2, 5 and 10 seconds, then every 30. Posts to Boxmaker go unanswered meanwhile; they are caught up when the connection returns.

Why. The TCP connection, the TLS handshake or the WebSocket upgrade failed, or the server went silent for dead_after_ms. A TLS failure means the certificate did not match the host in url or did not chain to the system's roots or ca_file; verification cannot be turned off.

Confirm.

curl -sS <url>/api/v4/system/ping           # the server answers
tailscale status                             # for a tailnet url: the tailnet is up
openssl s_client -connect <host>:443 -servername <host> </dev/null | head

Fix. Start Mattermost, or the tailnet. For a certificate error, correct url to the name on the certificate, or give the issuing CA in [mattermost] ca_file. gatewayd keeps trying by itself; no restart is needed.

Check. gatewayd: connected to <url> as <user>, then a direct message to Boxmaker is answered.

mattermost-auth-failed

What you see. gatewayd exits 1 with gatewayd: Mattermost refused the token (<status>), then this entry.

Why. Mattermost answered 401 or 403: the token is wrong, revoked, or belongs to a deactivated user. Retrying with the same token cannot help, so gatewayd stops instead.

Confirm. With the token in $T (from the same place gatewayd reads it; do not paste it into a shared shell history): curl -sS -o /dev/null -w '%{http_code}\n' -H "Authorization: Bearer $T" <url>/api/v4/users/me prints 401 or 403.

Fix. In Mattermost, under the bot's or user's access tokens, create a new token and revoke the old one. Store it as secret-unavailable describes, then start gatewayd.

Check. gatewayd: connected to <url> as <user>.

gateway-state-damaged

What you see. gatewayd exits 1 with <home>/gateway/state.json: <reason>, then this entry: at start when the file cannot be read, or while running when it cannot be written (…: cannot write: …).

Why. The state file records which posts were handled, which threads Boxmaker is in, and which turns were in flight. If it cannot be read or kept up to date, gatewayd could answer old posts twice or miss threads, so it does not guess. A missing file is a first start and is fine.

Confirm. ls -l "$BOXMAKER_HOME/gateway/state.json" and python3 -m json.tool "$BOXMAKER_HOME/gateway/state.json" >/dev/null.

Fix. For a write failure, free space or correct the directory's ownership (df -h, ls -ld "$BOXMAKER_HOME/gateway"), then start gatewayd. If only the file's ownership or mode is wrong, correct it. If the content is damaged, move it aside (mv state.json state.json.damaged) and start again. That is a first start: posts sent while gatewayd was down are not answered, and threads in channels must name Boxmaker again once.

Check. gatewayd starts, and state.json is rewritten after the next post.

loop-unavailable

What you see. In the Mattermost thread: "Boxmaker's loop is not running (see docs/runbook.md#loop-unavailable)". The messages that were waiting are dropped.

Why. gatewayd could not connect to loop.sock, or the connection closed before the turn ended. gatewayd does not retry: loopd may have finished and logged the turn, and sending it again would run it twice.

Confirm. ls -l "$BOXMAKER_HOME/run/loop/loop.sock" (or [loop] socket), and whether loopd serve is running. If it stopped, its last lines say why.

Fix. Start loopd serve --config <path>; if it failed, follow the entry its message names. Then send the message again in the thread.

Check. A direct message to Boxmaker is answered.