kyleandClaude Opus 5.5
0339dc13b2
Each task's tests were run against a reference at its end state; the end states were replayed from master in order with the gate at each step (650 to 762 tests); each skeleton compiles against its tests and fails them. The reference is kept off this machine. Lessons T27 (every wait in a test has a limit) and T28 (mutate the reference before hand-over) come from this work. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
7.0 KiB
gatewayd-start-failed
What you see. gatewayd exits 1 at start, before it connects, with one line naming its config
file or a directory, then this entry.
Why. gatewayd could not read or parse gatewayd.toml, or a value in it is not allowed: a
url that is not http:// or https:// with a host and an optional port, an id that is not 26
characters of a-z0-9, an empty [allow] users, or a secret with not exactly one of credential,
env and file. Or it could not create <home>/gateway/. It will not guess at a configuration
that decides who it answers.
Confirm. The line says which: <config path>: … for the file (unknown keys are errors; check
it against docs/specs/2026-09-23-m4a-gateway.md, section 3), or cannot prepare <dir>: … for the
directory (ls -ld <dir> "$(dirname <dir>)").
Fix. Correct the file or the directory's ownership. A user id is shown in Mattermost under the
user's profile, or by GET /api/v4/users/username/<name>.
Check. gatewayd serve --config <path> prints gatewayd: connected to <url> as <user>.
secret-unavailable
What you see. gatewayd exits 1 at start with secret <name>: <reason>, then this entry. The
value is never printed.
Why. gatewayd does not start without its Mattermost token, and reads it only from the one
place [secrets.<name>] names. The reasons: for credential, CREDENTIALS_DIRECTORY is unset
(not started by systemd with a credential) or the file in it is missing; for env, the variable is
unset or empty; for file, the path is not absolute, is a symbolic link, is not a regular file, is
not owned by the user gatewayd runs as, or has any group or other permission (only 0600 or 0400
are accepted). An empty value is refused in every form.
Confirm.
systemctl --user show -p LoadCredentialEncrypted gatewayd # credential
ls -l <path>; id -u # file: owner and mode
Fix. For a credential: systemd-creds --user encrypt --name=<credential> - <path>, type the
token, then give the unit LoadCredentialEncrypted=<credential>:<path>. For a file:
chmod 600 <path> and chown it to the user gatewayd runs as. For an environment variable, set
it in the environment gatewayd starts in.
Check. gatewayd starts and prints gatewayd: connected to <url> as <user>.
secret-in-a-file
What you see. At start: gatewayd: warning: secret <name> is read in plaintext from <path>; a systemd credential keeps it encrypted at rest, then this entry. gatewayd runs normally.
Why. A file holds the token in plaintext: anyone who can read the disk, or a backup of it, can use it. A credential is encrypted to this machine's TPM and host key. Sometimes a file is right (a development machine, a system without systemd); the warning is there so the choice is deliberate.
Confirm. [secrets.<name>] in gatewayd.toml has file = ….
Fix. To keep the file: nothing; the warning stays. To move to a credential, follow the fix in
secret-unavailable, change the entry to credential = "<name>", restart,
then delete the file and regenerate the token if the file was ever copied elsewhere.
Check. The warning is gone at the next start.
mattermost-unreachable
What you see. gatewayd prints gatewayd: cannot reach <url>: <reason>; trying again in <n> s,
then this entry, once per attempt: after 1, 2, 5 and 10 seconds, then every 30. Posts to Boxmaker go
unanswered meanwhile; they are caught up when the connection returns.
Why. The TCP connection, the TLS handshake or the WebSocket upgrade failed, or the server went
silent for dead_after_ms. A TLS failure means the certificate did not match the host in url or
did not chain to the system's roots or ca_file; verification cannot be turned off.
Confirm.
curl -sS <url>/api/v4/system/ping # the server answers
tailscale status # for a tailnet url: the tailnet is up
openssl s_client -connect <host>:443 -servername <host> </dev/null | head
Fix. Start Mattermost, or the tailnet. For a certificate error, correct url to the name on
the certificate, or give the issuing CA in [mattermost] ca_file. gatewayd keeps trying by
itself; no restart is needed.
Check. gatewayd: connected to <url> as <user>, then a direct message to Boxmaker is answered.
mattermost-auth-failed
What you see. gatewayd exits 1 with gatewayd: Mattermost refused the token (<status>), then
this entry.
Why. Mattermost answered 401 or 403: the token is wrong, revoked, or belongs to a deactivated
user. Retrying with the same token cannot help, so gatewayd stops instead.
Confirm. With the token in $T (from the same place gatewayd reads it; do not paste it into a
shared shell history):
curl -sS -o /dev/null -w '%{http_code}\n' -H "Authorization: Bearer $T" <url>/api/v4/users/me
prints 401 or 403.
Fix. In Mattermost, under the bot's or user's access tokens, create a new token and revoke the
old one. Store it as secret-unavailable describes, then start gatewayd.
Check. gatewayd: connected to <url> as <user>.
gateway-state-damaged
What you see. gatewayd exits 1 with <home>/gateway/state.json: <reason>, then this entry:
at start when the file cannot be read, or while running when it cannot be written
(…: cannot write: …).
Why. The state file records which posts were handled, which threads Boxmaker is in, and which
turns were in flight. If it cannot be read or kept up to date, gatewayd could answer old posts
twice or miss threads, so it does not guess. A missing file is a first start and is fine.
Confirm. ls -l "$BOXMAKER_HOME/gateway/state.json" and
python3 -m json.tool "$BOXMAKER_HOME/gateway/state.json" >/dev/null.
Fix. For a write failure, free space or correct the directory's ownership (df -h,
ls -ld "$BOXMAKER_HOME/gateway"), then start gatewayd. If only the file's ownership or mode is
wrong, correct it. If the content is damaged, move it
aside (mv state.json state.json.damaged) and start again. That is a first start: posts sent while
gatewayd was down are not answered, and threads in channels must name Boxmaker again once.
Check. gatewayd starts, and state.json is rewritten after the next post.
loop-unavailable
What you see. In the Mattermost thread: "Boxmaker's loop is not running (see docs/runbook.md#loop-unavailable)". The messages that were waiting are dropped.
Why. gatewayd could not connect to loop.sock, or the connection closed before the turn
ended. gatewayd does not retry: loopd may have finished and logged the turn, and sending it
again would run it twice.
Confirm. ls -l "$BOXMAKER_HOME/run/loop/loop.sock" (or [loop] socket), and whether
loopd serve is running. If it stopped, its last lines say why.
Fix. Start loopd serve --config <path>; if it failed, follow the entry its message names.
Then send the message again in the thread.
Check. A direct message to Boxmaker is answered.