boxmaker/docs/runbook.md

# Runbook

One entry for every state in which Boxmaker refuses to work or holds something back, and for
events worth understanding when you see them. Every message for such a state ends with
`see docs/runbook.md#<entry>`; a gate check makes sure the entry exists.

Each entry says what you see, why the system refuses, how to confirm, how to fix, and how to check
the fix. Commands assume:

- `$BOXMAKER_HOME` is the harness home: `/var/lib/boxmaker` when deployed, `.state` in the
  repository during development.
- `$GRANTS` is `[paths] grants` from `brokerd.toml`: `/etc/boxmaker/grants` when deployed.
- Until M7 there are no service units: the daemons run in a terminal, and "stop" means Ctrl-C or
  `pkill -x <name>`.

Entries for M3a states are written ahead of the code and describe it as specified in
`docs/specs/2026-09-18-m3a-decision-path.md`.

## grants-invalid

**What you see.** `brokerd` prints one or more lines naming a grant file and a problem, then this
entry. Every tool call is denied, and the model tells you "the grant files have an error".

**Why.** One grant file is invalid, so `brokerd` denies everything until it is fixed. It cannot
skip the bad file: if the bad file was a `deny` grant, skipping it would allow what it was written
to forbid.

**Confirm.**

```sh
bxctl grants check
```

It prints each problem as `<file>:<line>: <problem>`. Common ones: a misspelled field (unknown
fields are errors); `secret` set (not supported until M4); `patterns` set (not supported); `hosts`
on a file tool or `paths` on `http_fetch`; a relative path or one with `..`, `.` or `//`; a path with `:` or `,` (it cannot be mounted); the path
`/` (grant the directories you mean); a `deny` grant whose `max_taint` is not `"secret"` (it would
stop denying once the session read a secret); a host with uppercase letters, a port, or an IP
address; a file name with characters outside `[a-z0-9-]`.

**Fix.** Edit the file, or move it out of `$GRANTS` while you work on it:

```sh
mkdir -p ~/grants-parked && mv "$GRANTS/<file>.toml" ~/grants-parked/
```

Do not park a `deny` grant unless you are sure nothing else would then allow what it forbids.

**Check.** `bxctl grants check` prints `grants: ok`. No restart is needed: `brokerd` reads the
grants again at the next call.

## audit-unavailable

**What you see.** `brokerd` prints an error writing to `$BOXMAKER_HOME/audit/`, then this entry.
Tool calls are denied, and the model says "the audit log cannot be written". Every later call
prints `brokerd: an earlier audit write failed; every call is denied until brokerd is restarted`.
A call that ran but whose result could not be recorded reaches the model as "the result could not
be recorded", without its content; `bxctl refuse` says "the refusal could not be recorded".

A variant: `brokerd: a thread panicked while holding the ledger`. That is a bug in `brokerd`, not
a disk problem; the fix below is the same, and the panic message above it in `brokerd`'s output is
worth keeping for a report.

**Why.** A call runs only after its decision is on disk. If the record cannot be written, nothing
runs.

**Confirm.**

```sh
df -h "$BOXMAKER_HOME"
ls -ld "$BOXMAKER_HOME/audit"; ls -l "$BOXMAKER_HOME/audit" | tail -3
```

Look for a full disk, a directory or file not owned by the user `brokerd` runs as, or a read-only
file system.

**Fix.** Free space or correct ownership (`chown` to the `brokerd` user; files are mode 0600, the
directory 0700). Do not edit, move or delete audit files to make space: that breaks the chain.
Then stop and start `brokerd`. It does not try again by itself: a failed write may have left part
of a line, and only the startup check deals with that. Expect
[audit-recovered](#audit-recovered) at that start.

**Check.** Ask for any tool call. It is decided normally, and `bxctl audit verify` prints
`audit: ok`.

## audit-chain-broken

**What you see.** `brokerd` refuses to start and prints `<file>:<line>: <what>`, then this entry.
Or `bxctl audit verify` prints the same and exits 1. If the break is in an older file, only
`bxctl audit verify` reports it: at an ordinary start `brokerd` checks the latest file alone.

**Why.** A record does not parse, its `seq` is not the next number, its `prev` is not the hash of
the line before, or a file does not continue from the one before it. Something changed the log
after it was written: an edit, a deleted or reordered line, a file restored from a backup, a
partial copy. `brokerd` will not add records after a history it cannot vouch for.

There is one innocent cause. After a crash `brokerd` recovers a torn last line by itself
([audit-recovered](#audit-recovered)), but if it is killed a second time while writing that
recovery record, the log is left with a line that nothing describes. The failure is then within
the last three lines of the latest file, and the last line is cut short.

**Confirm.**

```sh
bxctl audit verify
sed -n '<line-1>,<line+1>p' "$BOXMAKER_HOME/audit/<file>"
```

Work out what happened before going on: a restore, a manual edit, a disk problem, or none you know
of. If none you know of, treat it as a possible intrusion and keep the files as they are.

**Fix.** Keep a copy first, then accept the break explicitly:

```sh
cp -a "$BOXMAKER_HOME/audit" ~/audit-copy-$(date +%F)
brokerd serve --config <path> --accept-break
```

With this flag `brokerd` checks the whole log, not only the latest file, and appends an
`accepted_break` record naming the file and line of the first failure. It continues from there.
Nothing is repaired or deleted. The break stays visible in every later verification. One accepted
break covers everything between the failure and the break record, including any further damage
there, so look at that whole stretch before accepting it.

**Check.** `bxctl audit verify` prints `audit: ok` and lists the accepted break with its file and
line. Damage that happens after the break record needs its own accepted break.

## audit-recovered

**What you see.** `brokerd` prints "audit: recovered a torn final line" at start, then this entry.
It starts normally.

**Why.** The last record was being written when `brokerd` or the machine stopped, so the file ended
without a newline. That is expected after a crash, not an error. `brokerd` kept the partial line,
ended it, and wrote a `recovery` record holding its length and hash, chained from the last complete
record. The partial line may even look like a whole record; it is not one, and the recovery record
takes its `seq`. It was never acted on: a record allows something only once it is on disk with its
newline. If the partial line was the `result` of a call, the call did run and its result was
never delivered; `bxctl audit verify` lists that call as unfinished.

**Confirm.** `bxctl audit verify` lists the recovery with its file and line.

**Fix.** None needed. If recoveries happen often, find out why `brokerd` is being killed.

**Check.** `bxctl audit verify` prints `audit: ok`.

## brokerd-already-running

**What you see.** `brokerd` exits at start with "brokerd is already running" and this entry.

**Why.** Only one `brokerd` may write the audit log. It holds a lock on
`$BOXMAKER_HOME/audit/.lock` for as long as it runs, and the lock is released when the process
ends, however it ends.

**Confirm.**

```sh
pgrep -a brokerd
```

**Fix.** Use the running one, or stop it (`pkill -x brokerd`) and start again. Deleting `.lock`
does not help and is not needed.

**Check.** `pgrep -a brokerd` shows one process.

## brokerd-start-failed

**What you see.** `brokerd` exits 1 at start, before it serves anything, with one line naming its
config file, a directory or a socket, then this entry. `loopd` then reports
[broker-unavailable](#broker-unavailable) for every tool call.

**Why.** `brokerd` could not read or parse its config, could not create or make private (0700) the
directory a socket lives in, or could not bind a socket or make it private (0600). It will not
serve on a socket whose permissions it could not set, because those permissions are what keep other
programs off it.

**Confirm.** The line says which:

- `<config path>: …` — the file is missing, unreadable, or not valid TOML for `brokerd.toml`
  (unknown keys are errors). Check it against `docs/specs/2026-09-18-m3a-decision-path.md`,
  section 2, "Configuration".
- `cannot prepare <dir>: …` — the directory cannot be made or `chmod`ed:
  `ls -ld <dir> "$(dirname <dir>)"`. A path that runs through a file, or a directory owned by
  another user, gives this.
- `cannot prepare <socket or dir>: a socket needs a directory of its own` or `…: is a symbolic
  link or not a directory` — `brokerd` makes a socket's directory 0700, so it refuses `/` and a
  directory reached through a link (the link's target would be changed instead). Give each socket
  its own real directory, as the defaults under `$BOXMAKER_HOME/run/` are.
- `cannot listen on <socket>: …` — `path must be shorter than SUN_LEN` means the socket path is
  longer than 107 bytes; `Address already in use` means something still listens there
  (`ss -xlp | grep <socket>`).

**Fix.** Correct the config, or the ownership of the directory, or choose a shorter socket path in
`[sockets]` (and the same path in `loopd`'s `[broker] socket`). If another process holds the
socket, stop it; `brokerd` removes a stale socket file by itself.

**Check.** `brokerd serve --config <path>` prints `brokerd: serving tools on … and approvals on …`.

## brokerd-listener-lost

**What you see.** One of three lines, then this entry:

- `brokerd: cannot accept on accept-broker (or accept-admin, accept-state) for now, retrying:
  <error>`. It keeps running and tries every 200 ms; the line is printed once per episode.
- `brokerd: cannot start a thread for a connection, so it was closed`. It keeps running.
- `brokerd: stopped serving: <error>`, and it exits 1: any other failure of `accept`.
 In both cases `loopd` reports
[broker-unavailable](#broker-unavailable) for the calls that were refused.

**Why.** The system refused `brokerd` something it needs to serve: a new connection (`accept`
failed) or a thread. The usual cause is a
limit: open files (`EMFILE`), processes or threads for the user, or memory. A connection that is
refused gets no decision, so nothing runs for it.

**Confirm.**

```sh
ulimit -n; ulimit -u
ls /proc/$(pgrep -x brokerd)/fd | wc -l     # while it runs
ps -o nlwp= -p $(pgrep -x brokerd)          # its thread count
```

Many threads usually means many connections waiting on approvals, or a client that opens
connections and never sends: look at `bxctl approvals` and at which process holds the sockets
(`ss -xp | grep broker`).

**Fix.** Answer or let expire the pending approvals, stop whatever is flooding the socket, or raise
the limit. Then start `brokerd` again if it exited.

**Check.** `brokerd` prints `serving tools on …`, and a tool call is decided again.

## runner-unavailable

**What you see.** A tool call fails with "the tool runner could not start the container", and
`brokerd` prints what Podman said, then this entry.

**Why.** `brokerd` asked Podman to start the tool's container, or for `http_fetch` the egress
proxy's, and Podman could not: the program is missing, the image in `[runner] image` is not
loaded, the proxy's network does not exist, or the proxy did not create its socket in time.
Nothing ran for the call.

**Confirm.**

```sh
podman --version                                   # the program [runner] podman names
podman image inspect --format '{{.Digest}}' <image>   # must be the digest in [runner] image
podman ps -a --filter label=boxmaker               # containers left behind, if any
```

**Fix.** Install Podman or correct `[runner] podman`. Load the image built from
`deploy/tools-image.nix` (`podman load < result`) and put its digest in `[runner] image`. For the
proxy, check that `[runner] egress_network` names a network Podman has (`pasta` needs the `passt`
package). Remove leftovers with `podman rm -f` on the listed names.

**Check.** Ask for a `read_file` the grants allow; it returns the file.

## broker-state-damaged

**What you see.** `brokerd` prints an error reading or writing
`$BOXMAKER_HOME/broker/sessions/<id>.json`, then this entry. Either every call for that session is
denied ("this session's broker state is damaged"), or one call failed with "the result could not
be recorded". Such a call has no `Result` record, so `bxctl audit verify` lists it under "running
or unfinished"; that is expected, not a second fault.

**Why.** The file holds the session's taint and untrusted flag. If `brokerd` cannot read it, it
does not know how sensitive the session's data is, so it denies. If it cannot write it after a
tool ran, it withholds the result: content that raised the taint must not reach the model unless
the raised taint is on disk.

**Confirm.**

```sh
ls -l "$BOXMAKER_HOME/broker/sessions/<id>.json"*
cat "$BOXMAKER_HOME/broker/sessions/<id>.json"
df -h "$BOXMAKER_HOME"
```

A good file is one line such as `{"taint":"private","untrusted":false}`.

**Fix.** For a write failure, fix space or ownership as in
[audit-unavailable](#audit-unavailable). For a damaged file, do not guess low: set it from the
session's results. The session's audit `result` records show the highest `taint_after`, and any
result from an untrusted grant means `"untrusted":true`. If unsure, write
`{"taint":"secret","untrusted":true}`, which is always safe. The session only loses access to
grants with a lower `max_taint`. A `.tmp` file beside it is a write that did not finish and can
be removed.

**Check.** The next call for the session is decided normally.

## socket-forbidden

**What you see.** `brokerd` prints that a message of some kind arrived on the wrong socket, then
this entry.

**Why.** `broker.sock` accepts only tool requests from `loopd`; `admin.sock` accepts only admin
requests from `bxctl`; `state.sock` accepts only `memoryd`'s questions about a session's state.
Nothing in the harness sends a wrong kind, so this is a bug or a component
doing what it should not, such as something trying to approve through the tool socket.

**Confirm.** Note the time, kind and socket. Check what was running:
`pgrep -a 'loopd|bxctl|memoryd'`. Look at the session logs active at that time.

**Fix.** If a harness version mismatch explains it (a new `bxctl` against an old `brokerd`),
rebuild and restart both. Otherwise keep the logs, stop `loopd`, and investigate before running it
again.

**Check.** No further messages of this kind.

## broker-unavailable

**What you see.** `loopd` prints that it cannot reach `broker.sock`, or that `brokerd` did not
answer in time, then this entry. Tool calls fail with "the tool broker is unavailable", the model
says so, and the conversation goes on. Or, once at startup, `loopd` prints "no tool broker is
configured" and this entry, and tool calls fail with those words.

**Why.** `brokerd` is not running, was restarted during a call, did not answer within `[broker]
timeout_ms`, or `loopd`'s `[broker] socket` points somewhere else or is not set. Pending approvals
are lost when `brokerd` restarts. `bxctl audit verify` lists them as pending or abandoned.

**Confirm.**

```sh
pgrep -a brokerd
ls -l "$BOXMAKER_HOME/run/loop-broker/"
```

Compare the socket path with `[broker] socket` in `loopd`'s config and `[sockets] broker` in
`brokerd.toml`.

**Fix.** Start `brokerd`, or make the two paths agree. `loopd` needs no restart for that: it
connects per call. A changed or newly set `[broker] socket` does need a `loopd` restart.

**Check.** Ask the model to use a tool. The call is decided (allowed, asked or denied) instead of
failing.

## loopd-selftest-failed

**What you see.** `loopd` prints `selftest: FAILED: <reason>` and this entry, and exits 1.

**Why.** `loopd` will not start against a server that differs from the one its config describes,
that does not parse tool calls, or whose cache does not work.

**Confirm and fix, by reason.**

- `chat template sha256: expected …, got …`, `context per slot: …` or `slot count: …`. The server
  changed: a model update, or new flags in `~/src/nixos/hw/straylight/default.nix`. Check what
  changed on straylight. If the change is intended, update `[expect]` in `loopd`'s config:

  ```sh
  curl -s 'http://straylight:11434/props?model=ornith-1.5-35b-a3b' | jq -j .chat_template | sha256sum
  curl -s 'http://straylight:11434/props?model=ornith-1.5-35b-a3b' | jq '.default_generation_settings.n_ctx, .total_slots'
  ```

  A template change also invalidates the M0 measurements. Re-run the checks in
  `docs/inference-contract.md` before trusting the cache behaviour.
- `the tool call did not come back parsed: …`. The template or the server's tool parser changed.
  Treat it like a template change. Do not start `loopd` until tool calls parse again.
- `turn 2 did not reuse turn 1's cache: …`. Another client may have taken the slot between the two
  requests. Check `curl -s 'http://straylight:11434/slots?model=ornith-1.5-35b-a3b'` and run
  `loopd selftest --config <path>` again. If it fails twice with the slot otherwise idle, the cache
  is not working: check the server flags for `cache_prompt` and the KV settings.
- A connection, timeout or HTTP error. Check that `inferproxy` is running and that its upstream
  answers: `curl -s http://straylight:11434/health`. A model load can take tens of seconds, so
  retry once.

**Check.** `loopd selftest --config <path>` prints `selftest: ok`.

## session-log-damaged

**What you see.** A turn on an existing session fails with `<file>:<line>: <reason>` naming
`sessions/<id>/<epoch>.jsonl`, and this entry. Other sessions work.

**Why.** `loopd` replays the log exactly to rebuild the conversation, so a line it cannot read
would change what the model sees. It refuses rather than guess. `last line does not end in a
newline` means a write was cut off: `loopd` writes a record only after the request it describes
has finished, so a cut-off line was never used.

**Confirm.**

```sh
f="$BOXMAKER_HOME/sessions/<id>/0.jsonl"
wc -l "$f"; tail -c 300 "$f"
```

**Fix.** Keep a copy first. Then either start a new session (always safe), or, for a cut-off last
line only, drop it:

```sh
cp -a "$f" "$f.damaged-$(date +%s)"
head -n <line-1> "$f.damaged-"* > "$f"      # keep the complete lines before the damaged one
```

Anything else (a damaged line in the middle, an unknown record type) means the file was changed
by something other than `loopd`. Start a new session and keep the old log for inspection.

**Check.** The next turn on the session succeeds. The first request re-reads the conversation if
the cache was lost, which `loopd` logs as a cache loss.

## memory-unavailable

**What you see.** Starting a new session fails with a line naming `memoryd` or a file under
`memory/`, and this entry. Existing sessions still work: they use the baseline saved when they
started.

**Why.** A new session's baseline takes the memory block from `memoryd` (`soul.md`, `memory.md`,
`user.md`), and `loopd` could not get it: `memoryd` is not running or not listening at
`<home>/run/loop-memory/memory.sock` (or `[memory] socket` in `loopd.toml`), or it refused because
a file is unreadable or over its limit (`soul.md` 4,000 characters, `memory.md` 2,200, `user.md`
1,375; an owner's edit can put it over). Starting without the memory the agent is, and without
telling you, is what this prevents.

**Confirm.** `pgrep -x memoryd`; the line names the file or the socket. For a file,
`wc -m <home>/memory/<file>`.

**Fix.** Start `memoryd serve --config <path>`, or shorten the file (for `memory.md` and `user.md`,
move entries to `notes.md` by hand: an entry under a line `§ moved <time> from memory.md in session
owner`).

**Check.** Start a new conversation; it answers.

## gatewayd-start-failed

**What you see.** `gatewayd` exits 1 at start, before it connects, with one line naming its config
file or a directory, then this entry.

**Why.** `gatewayd` could not read or parse `gatewayd.toml`, or a value in it is not allowed: a
`url` that is not `http://` or `https://` with a host and an optional port, an id that is not 26
characters of `a-z0-9`, an empty `[allow] users`, an id in `[approvals] users` that is not also in
`[allow] users`, or a secret with not exactly one of `credential`, `env` and `file`. Or it could
not create `<home>/gateway/`, or could not listen on `gateway.sock` (its directory is made 0700
and the socket 0600, as `brokerd`'s are; a path of 108 bytes or more cannot be a socket). It will
not guess at a configuration that decides who it answers and who approves.

**Confirm.** The line says which: `<config path>: …` for the file (unknown keys are errors; check
it against `docs/specs/2026-09-23-m4a-gateway.md`, section 3, and
`docs/specs/2026-09-24-m4b-approvals.md`, section 3), `cannot prepare <dir>: …` for a directory
(`ls -ld <dir> "$(dirname <dir>)"`), or `cannot listen on <socket>: …` for `gateway.sock`
(`Address already in use` means another `gatewayd` runs: `pgrep -x gatewayd`).

**Fix.** Correct the file or the directory's ownership. A user id is shown in Mattermost under the
user's profile, or by `GET /api/v4/users/username/<name>`.

**Check.** `gatewayd serve --config <path>` prints `gatewayd: connected to <url> as <user>`.

## secret-unavailable

**What you see.** `gatewayd` exits 1 at start with `secret <name>: <reason>`, then this entry. The
value is never printed.

**Why.** `gatewayd` does not start without its Mattermost token, and reads it only from the one
place `[secrets.<name>]` names. The reasons: for `credential`, `CREDENTIALS_DIRECTORY` is unset
(not started by systemd with a credential) or the file in it is missing; for `env`, the variable is
unset or empty; for `file`, the path is not absolute, is a symbolic link, is not a regular file, is
not owned by the user `gatewayd` runs as, or has any group or other permission (only 0600 or 0400
are accepted). An empty value is refused in every form.

**Confirm.**

```sh
systemctl --user show -p LoadCredentialEncrypted gatewayd   # credential
ls -l <path>; id -u                                         # file: owner and mode
```

**Fix.** For a credential: `systemd-creds --user encrypt --name=<credential> - <path>`, type the
token, then give the unit `LoadCredentialEncrypted=<credential>:<path>`. For a file:
`chmod 600 <path>` and `chown` it to the user `gatewayd` runs as. For an environment variable, set
it in the environment `gatewayd` starts in.

**Check.** `gatewayd` starts and prints `gatewayd: connected to <url> as <user>`.

## secret-in-a-file

**What you see.** At start: `gatewayd: warning: secret <name> is read in plaintext from <path>; a
systemd credential keeps it encrypted at rest`, then this entry. `gatewayd` runs normally.

**Why.** A file holds the token in plaintext: anyone who can read the disk, or a backup of it, can
use it. A credential is encrypted to this machine's TPM and host key. Sometimes a file is right (a
development machine, a system without systemd); the warning is there so the choice is deliberate.

**Confirm.** `[secrets.<name>]` in `gatewayd.toml` has `file = …`.

**Fix.** To keep the file: nothing; the warning stays. To move to a credential, follow the fix in
[secret-unavailable](#secret-unavailable), change the entry to `credential = "<name>"`, restart,
then delete the file and regenerate the token if the file was ever copied elsewhere.

**Check.** The warning is gone at the next start.

## mattermost-unreachable

**What you see.** `gatewayd` prints `gatewayd: cannot reach <url>: <reason>; trying again in <n> s`,
then this entry, once per attempt: after 1, 2, 5 and 10 seconds, then every 30. Posts to Boxmaker go
unanswered meanwhile; they are caught up when the connection returns.

**Why.** The TCP connection, the TLS handshake or the WebSocket upgrade failed, or the server went
silent for `dead_after_ms`. A TLS failure means the certificate did not match the host in `url` or
did not chain to the system's roots or `ca_file`; verification cannot be turned off.

**Confirm.**

```sh
curl -sS <url>/api/v4/system/ping           # the server answers
tailscale status                             # for a tailnet url: the tailnet is up
openssl s_client -connect <host>:443 -servername <host> </dev/null | head
```

**Fix.** Start Mattermost, or the tailnet. For a certificate error, correct `url` to the name on
the certificate, or give the issuing CA in `[mattermost] ca_file`. `gatewayd` keeps trying by
itself; no restart is needed.

**Check.** `gatewayd: connected to <url> as <user>`, then a direct message to Boxmaker is answered.

## mattermost-auth-failed

**What you see.** `gatewayd` exits 1 with `gatewayd: Mattermost refused the token (<status>)`, then
this entry.

**Why.** Mattermost answered 401 or 403: the token is wrong, revoked, or belongs to a deactivated
user. Retrying with the same token cannot help, so `gatewayd` stops instead.

**Confirm.** With the token in `$T` (from the same place `gatewayd` reads it; do not paste it into a
shared shell history):
`curl -sS -o /dev/null -w '%{http_code}\n' -H "Authorization: Bearer $T" <url>/api/v4/users/me`
prints 401 or 403.

**Fix.** In Mattermost, under the bot's or user's access tokens, create a new token and revoke the
old one. Store it as [secret-unavailable](#secret-unavailable) describes, then start `gatewayd`.

**Check.** `gatewayd: connected to <url> as <user>`.

## gateway-state-damaged

**What you see.** `gatewayd` exits 1 with `<home>/gateway/state.json: <reason>`, then this entry:
at start when the file cannot be read, or while running when it cannot be written
(`…: cannot write: …`).

**Why.** The state file records which posts were handled, which threads Boxmaker is in, and which
turns were in flight. If it cannot be read or kept up to date, `gatewayd` could answer old posts
twice or miss threads, so it does not guess. A missing file is a first start and is fine.

**Confirm.** `ls -l "$BOXMAKER_HOME/gateway/state.json"` and
`python3 -m json.tool "$BOXMAKER_HOME/gateway/state.json" >/dev/null`.

**Fix.** For a write failure, free space or correct the directory's ownership (`df -h`,
`ls -ld "$BOXMAKER_HOME/gateway"`), then start `gatewayd`. If only the file's ownership or mode is
wrong, correct it. If the content is damaged, move it
aside (`mv state.json state.json.damaged`) and start again. That is a first start: posts sent while
`gatewayd` was down are not answered, and threads in channels must name Boxmaker again once.

**Check.** `gatewayd` starts, and `state.json` is rewritten after the next post.

## loop-unavailable

**What you see.** In the Mattermost thread: "Boxmaker's loop is not running
(see docs/runbook.md#loop-unavailable)". The messages that were waiting are dropped.

**Why.** `gatewayd` could not connect to `loop.sock`, or the connection closed before the turn
ended. `gatewayd` does not retry: `loopd` may have finished and logged the turn, and sending it
again would run it twice.

**Confirm.** `ls -l "$BOXMAKER_HOME/run/loop/loop.sock"` (or `[loop] socket`), and whether
`loopd serve` is running. If it stopped, its last lines say why.

**Fix.** Start `loopd serve --config <path>`; if it failed, follow the entry its message names.
Then send the message again in the thread.

**Check.** A direct message to Boxmaker is answered.

## gateway-unreachable

**What you see.** `brokerd` prints `brokerd: approval <id>: gatewayd is not reachable at <socket>:
<reason>; bxctl can still answer it`, then this entry, once per approval. No approval post
appears in Mattermost. `bxctl approvals` lists the approval as usual.

**Why.** `brokerd.toml` has a `[gateway]` section, so each pending approval is carried to
`gatewayd` to be posted in Mattermost. The courier could not connect to `gateway.sock`, or
`gatewayd` answered that it could not post (Mattermost unreachable). The approval is not lost: it
stays pending, `bxctl` can answer it, and the courier tries again every 5 seconds until it is
posted or answered. When a later try works, `brokerd` prints `brokerd: approval <id> posted in
Mattermost`.

**Confirm.**

```sh
ls -l <socket>                        # gatewayd creates it at start; [approvals] socket in gatewayd.toml
pgrep -x gatewayd                     # is it running?
```

and `gatewayd`'s own output: `cannot reach …` there means Mattermost is the problem (see
[mattermost-unreachable](#mattermost-unreachable)).

**Fix.** Start `gatewayd`, or make `[gateway] socket` in `brokerd.toml` and `[approvals] socket`
in `gatewayd.toml` name the same path (both default to `<home>/run/gateway/gateway.sock`). To turn
Mattermost approvals off, remove `[gateway]` from `brokerd.toml` and restart `brokerd`.

**Check.** Trigger an `ask` call: its approval is posted in Mattermost, and `!approve <id>` in
its thread answers it.

Where two Boxmakers share a channel thread, address approval commands to one of them
(`@<bot> !approve <id>`): a bare `!approve <id>` is taken by every Boxmaker with that id pending
in that thread.

## memoryd-start-failed

**What you see.** `memoryd` exits 1 at start with one line naming its config file, a directory,
a file under `memory/`, the index, or a socket, then this entry.

**Why.** `memoryd` could not read or parse `memoryd.toml` (unknown keys are errors), could not
read the starter `soul.md` when `memory/soul.md` was missing, could not create or open
`memory/` or `index/memory.sqlite3`, or could not listen on one of its three sockets (their
directories are made 0700 and the sockets 0600, as `brokerd`'s are; a path of 108 bytes or more
cannot be a socket). It will not start without the memory it owns.

**Confirm.** The line names the path. For the file, check it against
`docs/specs/2026-09-24-m5a-local-memory.md`, section 11. For a directory,
`ls -ld <dir> "$(dirname <dir>)"`. `Address already in use` means another `memoryd` runs:
`pgrep -x memoryd`.

**Fix.** Correct the file, the path or the ownership. The index is disposable: if it is damaged,
move it aside and start `memoryd`, which rebuilds it (see `memory-index-damaged`).

**Check.** `memoryd serve --config <path>` prints `memoryd: serving`.

## memoryd-unreachable

**What you see.** A `recall` or `memory` call answers `memory is unavailable: memoryd did not
answer (see docs/runbook.md#memoryd-unreachable)`, and the model tells you it cannot use its
memory.

**Why.** `brokerd` decided the call and sent it to `memoryd`'s `brokerd` socket
(`<home>/run/broker-memory/memory.sock`, or `[memory] socket` in `brokerd.toml`), and nothing
answered within 30 seconds, or the answer was not a tool result. `memoryd` is not running, is
listening elsewhere, or is stuck.

**Confirm.** `pgrep -x memoryd`; `ls -l <home>/run/broker-memory/`; the end of `memoryd`'s log.

**Fix.** Start `memoryd serve --config <path>`, or make `[memory] socket` in `brokerd.toml` and
`[sockets] broker` in `memoryd.toml` name the same path.

**Check.** Ask the agent to recall something; the answer lists hits or says there were none.

## memory-index-damaged

**What you see.** `recall` fails with a line starting `index:`, or `memoryd` logs one at start.

**Why.** `index/memory.sqlite3` could not be read or written. It is disposable: everything in it
comes from `sessions/` and `memory/notes.md`.

**Confirm.** `ls -l <home>/index/`; the end of `memoryd`'s log.

**Fix.** `bxctl reindex`, which builds a new index beside the old one and replaces it. If that
fails too, stop `memoryd`, move `index/memory.sqlite3` aside, and start `memoryd`, which rebuilds
it.

**Check.** `bxctl reindex` prints how many logs, records and notes it read.

## memory-write-refused

**What you see.** The agent says memory refused something: its `memory` call was answered
`memory refused this entry: it contains …` (what looks like a GitHub token, an invisible or
direction-changing character, and so on), and `memoryd` logged `memoryd: a memory write from
session <id> was refused by the rule <rule>; see docs/runbook.md#memory-write-refused`. Nothing
was written to `memory/`, staged for review or journalled.

**Why.** Every text a memory write would store is checked against the sanitizer's refuse rules
(M5b spec, section 6.1): control characters other than line feed and tab, invisible and
direction-changing characters, private-use characters, noncharacters, secrets known by their
prefix (GitHub, GitLab, `sk-` keys, Slack, AWS, private-key blocks), and the model's
chat-template tags. Such text is only ever an attack or a leaked secret, and identity goes into
every prompt. The log names the rule, never the text.

**Confirm.** The rule in the log line. `bxctl sanitize check` with the text (as an argument or on
standard input) names each rule that fires and writes nothing. The session's log holds the call's
arguments.

**Fix.** A real secret: rotate it, since it has been in the conversation; keep it out of memory.
Ordinary text refused by mistake: add it to the code-work corpus and narrow the rule, as
[`sanitizer-rules`](#sanitizer-rules) says; the change reaches the agent when it is rebuilt.

**Check.** `bxctl sanitize check` with the text prints nothing and exits 0, and the agent's next
write of it is answered `… updated` or `saved in notes.md`.

## memory-write-flagged

**What you see.** A memory write was kept one tier lower. The `memory` call was answered `not
added to memory.md: it matches the rule <rule>, so it was saved in notes.md as an untrusted note`
(or `user.md`), or, for a note, `saved in notes.md, marked untrusted: it matches the rule <rule>`.
`memoryd` logged `memoryd: a memory write from session <id> matches the rule <rule>; it was saved
in notes.md as an untrusted note (edit <edit id>); see docs/runbook.md#memory-write-flagged`, and
`bxctl memory log` shows that edit with `flagged <rule>`.

**Why.** The text is shaped like an attack (M5b spec, section 6.3): it tells the reader to ignore
or replace earlier instructions, to send data to a URL, to dump the conversation, or to write into
SSH configuration, or it holds one of M5a's six instruction phrases. A flag never refuses and
never loses text: `memory.md` and `user.md`, which are in every prompt, are not changed, and every
text of the write is saved at the end of `notes.md` under an `untrusted` header. `recall` returns
it as `memory-tainted`, so a session that recalls it becomes untrusted. With review on, a flagged
write is not staged: it is a note at once. The log names the rule, never the text.

**Confirm.** `bxctl memory log`: the edit, `flagged <rule>`. The text is at the end of
`<home>/memory/notes.md`, under a line `§ noted <time> in session <id> · <class> · untrusted`.
`bxctl sanitize check` with the text names the rule.

**Fix.** If it is an attack, leave it: the note is the record of what was tried, and it never
reaches the prompt. If it is a true fact wrongly flagged, move it into identity by hand: add it to
`<home>/memory/memory.md` (or `user.md`) with an editor, as an entry of its own, with a line
holding only `§` between it and the entry before, keeping the file within its limit. The note
stays in `notes.md`, which is append-only. If the rule catches ordinary text, narrow it as
[`sanitizer-rules`](#sanitizer-rules) says.

**Check.** Start a new conversation and ask the agent what `memory.md` holds; the entry is there.

## memory-entry-excluded

**What you see.** `memoryd` logged `memoryd: an entry of memory.md was left out of the block by
the rule <rule>; see docs/runbook.md#memory-entry-excluded` (or `user.md`), each time a session
starts. Sessions start as usual, but the agent does not know that entry.

**Why.** When a session starts, every entry of `memory.md` and `user.md` is checked again against
the sanitizer's refuse rules (M5b spec, section 7), and an entry that breaks one is left out of
the prompt. So a rule added later, or an entry an editor put in by hand, protects at once, with no
migration. The file itself is not changed. Flag rules are not checked here, so an entry you moved
from `notes.md` by hand stays in, and `soul.md` is the owner's and is not checked at all.

**Confirm.** The rule in the log line names the kind of text (`secret-github`, `invisible`, …).
Run each entry of the file through `bxctl sanitize check`: the one it reports is the one left out.

**Fix.** Edit the file with an editor: remove the entry, or the part of it that breaks the rule.
Entries are separated by a line holding only `§`. A secret does not belong in memory: rotate it
if it was real.

**Check.** Start a new conversation: `memoryd` does not log the line again.

## loopd-tags-unexpected

**What you see.** `loopd selftest` or `loopd serve` prints `selftest: tags tokenize as recorded`,
then `selftest: FAILED: <reason>; see docs/runbook.md#loopd-tags-unexpected`, and exits 1. No
prompt was sent. The reason names a tag by its token id, never by its text:

- `the tag with id <id> does not tokenize as its recorded token`
- `a neutralised tag tokenizes to the tag with id <id>`
- `could not tokenize the tags: <error>`, or `… the neutralised tags: <error>`

**Why.** `loopd` neutralises the model's chat-template tags in every text from outside, writing
their angle brackets as `&lt;` and `&gt;` (M5b spec, sections 6.2 and 8). The list of tags, each
with its token id, is compiled into `bxcore` (`crates/bxcore/src/sanitize/tags.rs`), as measured on
straylight. At startup `loopd` asks the server's `/tokenize` about all the tags at once, one per
line, then about all their neutralised forms: two requests, so that the check fits in `inferproxy`'s
burst of connections. If a tag is no longer the token recorded with it, the model or its tokenizer
has changed and the list may no longer match it; if a neutralised form comes back as a tag's token,
neutralising no longer keeps that tag from the model. Either way the model could receive text from
outside as its own control tokens, so `loopd` does not start. The check cannot find a new tag
missing from the list: the server does not list its special tokens.

**Confirm.** Find the tag: `grep -n '<id>' crates/bxcore/src/sanitize/tags.rs` shows its entry,
built from its name with `bar!`, `open!` or `close!`. Ask the server yourself, building the tag
from its name as the file does. For a `bar!` tag named `im_end`:

```sh
printf '{"model":"ornith-1.5-35b-a3b","content":"<|%s|>","parse_special":true}' im_end |
  curl -s -d @- http://straylight:11434/tokenize
printf '{"model":"ornith-1.5-35b-a3b","content":"&lt;|%s|&gt;","parse_special":true}' im_end |
  curl -s -d @- http://straylight:11434/tokenize
```

For an `open!` tag the content is `<%s>`, for a `close!` tag `</%s>`. The first answer should be
`{"tokens":[<id>]}`, the second a list of ids without `<id>` in it. Check which model file the
server loaded: `curl -s 'http://straylight:11434/props?model=ornith-1.5-35b-a3b' | jq .model_path`.

**Fix, by reason.**

- `could not tokenize …`: the server did not answer `/tokenize`. A router in front of llama-server
  may not forward it (crossbar does not, `docs/decisions.md`, P22): point `inferproxy` at
  llama-server itself. Otherwise check that the server is up:
  `curl -s http://straylight:11434/health`, and run `loopd selftest --config <path>` again.
- A tag that does not tokenize as its recorded token: the model file changed. Do not run `loopd`
  against it. Measure the new model's tags as M5b task 03 did (every vocabulary entry of type
  control or user-defined, each checked with `/tokenize`), and change `TAGS` and the tests' list in
  `crates/bxcore/tests/support/tags.rs` together; the change reaches the agent when it is rebuilt. A
  new model also invalidates the measurements in `docs/inference-contract.md`.
- A neutralised tag that tokenizes to a tag: the tokenizer reads the escaped spelling as the tag,
  and the replacement of the M5b spec, section 6.2, no longer works for it. That is a design change:
  record it in `docs/decisions.md` and do not run `loopd` until it is settled.

**Check.** `loopd selftest --config <path>` prints `selftest: ok`.

## already-running

**What you see.** A daemon exits 1 at start, before it touches its sockets, with one of these
lines:

- `<role>: another <role> is running (it holds <lock file>); see docs/runbook.md#already-running`
- `<role>: cannot take the lock <lock file>: <error>; see docs/runbook.md#already-running`

`<role>` is `loopd`, `brokerd`, `memoryd`, `gatewayd` or `inferproxy`. The lock file is
`$BOXMAKER_HOME/run/<role>.lock`; for `inferproxy` it is its `--listen` socket's path with
`.lock` added (`infer.sock.lock`, beside `infer.sock`).

**Why.** Every daemon removes a socket it finds at its path before it binds, so that a socket
left by a daemon that was killed never blocks the next start. A second copy started by mistake
would therefore take the sockets of the one that runs, which would keep running, unreachable,
and nothing would say so. So each daemon first takes an exclusive lock on its lock file, and
holds it for as long as it runs (M5b spec, section 9a). The system releases the lock when the
process ends, however it ends, so a killed daemon never leaves a lock behind. The file stays in
place and holds nothing.

**Confirm.** For `another <role> is running`, find the copy that runs, with its command line
(the `--config` path says which home it serves):

```sh
pgrep -a -x <role>
ls -l "$BOXMAKER_HOME/run/<role>.lock"
```

For `cannot take the lock`, the error says why: `Permission denied` means the daemon's user
cannot create or open the file, or create `run/` (`ls -ld "$BOXMAKER_HOME" "$BOXMAKER_HOME/run"`
and `ls -l` on the lock file show who owns them); `Not a directory` means a path above it is a
file; `Read-only file system` means what it says.

**Fix.** Two copies of a role against the same home are always a mistake: use the one that runs.
To restart it, stop it (`pkill -x <role>`, or Ctrl-C in its terminal), wait until `pgrep -x
<role>` prints nothing, and start it again. Two copies with different homes are fine (each has
its own lock), as long as their configs do not name the same socket path. For a lock that cannot
be taken, give the daemon's user the directory (`chown`), or point its home somewhere it can
write.

Never delete a lock file while its daemon runs: the lock is held on the open file, so a new copy
would create a new file, lock that, and take the sockets. Deleting it while the daemon is stopped
does no harm, and does not help either.

**Check.** `pgrep -x <role>` prints one process id, and the daemon prints its start line
(`loopd: serving on …`, `inferproxy: <socket> -> <upstream>`, `brokerd: serving tools on …`,
`memoryd: serving`, `gatewayd: connected to …`).

## sanitizer-rules

**What you see.** Not a failure: the procedure for changing a sanitizer rule, which the entries
`memory-write-refused` and `memory-write-flagged` send you to when a rule catches ordinary text,
or which you follow when an attack gets through. The rules are in `crates/bxcore/src/sanitize/`:
`refuse.rs` (6.1 of the M5b spec), `tags.rs` (the chat-template tags that are neutralised, 6.2)
and `flag.rs` with `normalise.rs` (6.3).

**Why.** The rules are compiled in rather than read from a file, so that every change passes the
two corpora and review, and a typo cannot switch a rule off (M5b spec, section 6.5). A rule that
refuses code is as much a bug as one that lets an attack through: the agent works on code
projects, and a flagged note makes every session that recalls it untrusted.

**Confirm.** The evidence, before anything changes:

- a rule's name in `bxctl memory log` (`flagged <rule>`), in `memoryd`'s log, or in a refusal the
  agent passed on;
- `bxctl sanitize check` with the text, which names each rule that fires and writes nothing;
- a device check, a review or new research that shows an attack the rules miss.

**Fix.** In this order, one commit:

1. **The case first.** Add the text to a corpus, with a comment saying where it was seen: the
   attack corpus if it must be caught, the code-work corpus if it must pass. The corpora are the
   given tests `crates/bxcore/tests/sanitize_refuse_corpus.rs` (refuse),
   `sanitize_corpus.rs` (neutralise) and `sanitize_flag_corpus.rs` (flag). Build a tag or a key
   from pieces, as the files already do: never write one whole. Run `cargo test -p bxcore` and see
   the new case fail.
2. **The rule.** Add or narrow the entry in the rule's table, with a comment naming its source.
   Keep it as narrow as the evidence: a flag rule ported from Hermes keeps its attribution.
3. **The gate.** `make gate` runs both corpora for every action, so a rule that catches code
   fails before it lands.
4. **The record.** A policy change (a new kind of refusal, or a rule moved from flag to refuse)
   gets a row in `docs/decisions.md`; an ordinary rule change is its commit.

A change reaches an agent when it is rebuilt and restarted. Stored memory is never rewritten: an
identity entry that breaks a new refuse rule leaves the memory block at once (see
`memory-entry-excluded`), and a note flagged under an old rule stays as it was written.

**Check.** `bxctl sanitize check` with the text gives the new answer (nothing, and exit 0, for a
text that must pass). After the rebuild, the agent's next write of it is answered as expected.