gatewayd: secrets from a credential, the environment or a file; runbook entries

Implemented-By: OpenCode session (model recorded in docs/implementer-log.md)
2026-09-23 19:37:12 -07:00
parent ede25312b3
commit 608d426f95
5 changed files with 547 additions and 0 deletions
@@ -415,3 +415,146 @@ Look for wrong ownership or mode, or a file that is not UTF-8 text.
 save it again as UTF-8.

 **Check.** A new session starts, and its `0.baseline.json` contains the text.
+
+## gatewayd-start-failed
+
+**What you see.** `gatewayd` exits 1 at start, before it connects, with one line naming its config
+file or a directory, then this entry.
+
+**Why.** `gatewayd` could not read or parse `gatewayd.toml`, or a value in it is not allowed: a
+`url` that is not `http://` or `https://` with a host and an optional port, an id that is not 26
+characters of `a-z0-9`, an empty `[allow] users`, or a secret with not exactly one of `credential`,
+`env` and `file`. Or it could not create `<home>/gateway/`. It will not guess at a configuration
+that decides who it answers.
+
+**Confirm.** The line says which: `<config path>: …` for the file (unknown keys are errors; check
+it against `docs/specs/2026-09-23-m4a-gateway.md`, section 3), or `cannot prepare <dir>: …` for the
+directory (`ls -ld <dir> "$(dirname <dir>)"`).
+
+**Fix.** Correct the file or the directory's ownership. A user id is shown in Mattermost under the
+user's profile, or by `GET /api/v4/users/username/<name>`.
+
+**Check.** `gatewayd serve --config <path>` prints `gatewayd: connected to <url> as <user>`.
+
+## secret-unavailable
+
+**What you see.** `gatewayd` exits 1 at start with `secret <name>: <reason>`, then this entry. The
+value is never printed.
+
+**Why.** `gatewayd` does not start without its Mattermost token, and reads it only from the one
+place `[secrets.<name>]` names. The reasons: for `credential`, `CREDENTIALS_DIRECTORY` is unset
+(not started by systemd with a credential) or the file in it is missing; for `env`, the variable is
+unset or empty; for `file`, the path is not absolute, is a symbolic link, is not a regular file, is
+not owned by the user `gatewayd` runs as, or has any group or other permission (only 0600 or 0400
+are accepted). An empty value is refused in every form.
+
+**Confirm.**
+
+```sh
+systemctl --user show -p LoadCredentialEncrypted gatewayd   # credential
+ls -l <path>; id -u                                         # file: owner and mode
+```
+
+**Fix.** For a credential: `systemd-creds --user encrypt --name=<credential> - <path>`, type the
+token, then give the unit `LoadCredentialEncrypted=<credential>:<path>`. For a file:
+`chmod 600 <path>` and `chown` it to the user `gatewayd` runs as. For an environment variable, set
+it in the environment `gatewayd` starts in.
+
+**Check.** `gatewayd` starts and prints `gatewayd: connected to <url> as <user>`.
+
+## secret-in-a-file
+
+**What you see.** At start: `gatewayd: warning: secret <name> is read in plaintext from <path>; a
+systemd credential keeps it encrypted at rest`, then this entry. `gatewayd` runs normally.
+
+**Why.** A file holds the token in plaintext: anyone who can read the disk, or a backup of it, can
+use it. A credential is encrypted to this machine's TPM and host key. Sometimes a file is right (a
+development machine, a system without systemd); the warning is there so the choice is deliberate.
+
+**Confirm.** `[secrets.<name>]` in `gatewayd.toml` has `file = …`.
+
+**Fix.** To keep the file: nothing; the warning stays. To move to a credential, follow the fix in
+[secret-unavailable](#secret-unavailable), change the entry to `credential = "<name>"`, restart,
+then delete the file and regenerate the token if the file was ever copied elsewhere.
+
+**Check.** The warning is gone at the next start.
+
+## mattermost-unreachable
+
+**What you see.** `gatewayd` prints `gatewayd: cannot reach <url>: <reason>; trying again in <n> s`,
+then this entry, once per attempt: after 1, 2, 5 and 10 seconds, then every 30. Posts to Boxmaker go
+unanswered meanwhile; they are caught up when the connection returns.
+
+**Why.** The TCP connection, the TLS handshake or the WebSocket upgrade failed, or the server went
+silent for `dead_after_ms`. A TLS failure means the certificate did not match the host in `url` or
+did not chain to the system's roots or `ca_file`; verification cannot be turned off.
+
+**Confirm.**
+
+```sh
+curl -sS <url>/api/v4/system/ping           # the server answers
+tailscale status                             # for a tailnet url: the tailnet is up
+openssl s_client -connect <host>:443 -servername <host> </dev/null | head
+```
+
+**Fix.** Start Mattermost, or the tailnet. For a certificate error, correct `url` to the name on
+the certificate, or give the issuing CA in `[mattermost] ca_file`. `gatewayd` keeps trying by
+itself; no restart is needed.
+
+**Check.** `gatewayd: connected to <url> as <user>`, then a direct message to Boxmaker is answered.
+
+## mattermost-auth-failed
+
+**What you see.** `gatewayd` exits 1 with `gatewayd: Mattermost refused the token (<status>)`, then
+this entry.
+
+**Why.** Mattermost answered 401 or 403: the token is wrong, revoked, or belongs to a deactivated
+user. Retrying with the same token cannot help, so `gatewayd` stops instead.
+
+**Confirm.** With the token in `$T` (from the same place `gatewayd` reads it; do not paste it into a
+shared shell history):
+`curl -sS -o /dev/null -w '%{http_code}\n' -H "Authorization: Bearer $T" <url>/api/v4/users/me`
+prints 401 or 403.
+
+**Fix.** In Mattermost, under the bot's or user's access tokens, create a new token and revoke the
+old one. Store it as [secret-unavailable](#secret-unavailable) describes, then start `gatewayd`.
+
+**Check.** `gatewayd: connected to <url> as <user>`.
+
+## gateway-state-damaged
+
+**What you see.** `gatewayd` exits 1 with `<home>/gateway/state.json: <reason>`, then this entry:
+at start when the file cannot be read, or while running when it cannot be written
+(`…: cannot write: …`).
+
+**Why.** The state file records which posts were handled, which threads Boxmaker is in, and which
+turns were in flight. If it cannot be read or kept up to date, `gatewayd` could answer old posts
+twice or miss threads, so it does not guess. A missing file is a first start and is fine.
+
+**Confirm.** `ls -l "$BOXMAKER_HOME/gateway/state.json"` and
+`python3 -m json.tool "$BOXMAKER_HOME/gateway/state.json" >/dev/null`.
+
+**Fix.** For a write failure, free space or correct the directory's ownership (`df -h`,
+`ls -ld "$BOXMAKER_HOME/gateway"`), then start `gatewayd`. If only the file's ownership or mode is
+wrong, correct it. If the content is damaged, move it
+aside (`mv state.json state.json.damaged`) and start again. That is a first start: posts sent while
+`gatewayd` was down are not answered, and threads in channels must name Boxmaker again once.
+
+**Check.** `gatewayd` starts, and `state.json` is rewritten after the next post.
+
+## loop-unavailable
+
+**What you see.** In the Mattermost thread: "Boxmaker's loop is not running
+(see docs/runbook.md#loop-unavailable)". The messages that were waiting are dropped.
+
+**Why.** `gatewayd` could not connect to `loop.sock`, or the connection closed before the turn
+ended. `gatewayd` does not retry: `loopd` may have finished and logged the turn, and sending it
+again would run it twice.
+
+**Confirm.** `ls -l "$BOXMAKER_HOME/run/loop/loop.sock"` (or `[loop] socket`), and whether
+`loopd serve` is running. If it stopped, its last lines say why.
+
+**Fix.** Start `loopd serve --config <path>`; if it failed, follow the entry its message names.
+Then send the message again in the thread.
+
+**Check.** A direct message to Boxmaker is answered.