Specify M4a (conversations over Mattermost); propose P15 on secrets

M4 is split into M4a and M4b. The M4a spec rests on facts checked against
the owner's server and Mattermost's source at v11.11.0: the REST and
WebSocket shapes, and that clients will not post a message starting with
'/'. Records the design decisions, proposes P15 (secrets from a systemd
credential, the environment or an owner-only file), and adds the run-time
rows egress.md was missing since M2a and M3b.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-09-23 17:30:31 -07:00
co-authored by Claude Opus 5.5
parent f0ec0efe9d
commit 7e3783faae
5 changed files with 346 additions and 4 deletions
+2 -1
View File
@@ -12,7 +12,8 @@ M3a (`brokerd`'s decision path, spec `docs/specs/2026-09-18-m3a-decision-path.md
merged, reviewed in `docs/implementer-log.md`; its open findings (17, 18, 20) are listed there; 14 was merged, reviewed in `docs/implementer-log.md`; its open findings (17, 18, 20) are listed there; 14 was
closed by M3b task 02. M3b (the container runner and the four tools, `docs/specs/2026-09-22-m3b-runner.md`) is done and closed by M3b task 02. M3b (the container runner and the four tools, `docs/specs/2026-09-22-m3b-runner.md`) is done and
merged: tool containers run from the image `deploy/tools-image.nix` builds, checked on straylight. merged: tool containers run from the image `deploy/tools-image.nix` builds, checked on straylight.
M4 (`gatewayd` and Mattermost) is next to design. M4 is split: M4a (`gatewayd`, conversations over Mattermost) is specified in
`docs/specs/2026-09-23-m4a-gateway.md`, a draft for review; M4b (approvals over Mattermost) follows.
Straylight now serves Ornith as four slots over one 262,144-token pool; see Straylight now serves Ornith as four slots over one 262,144-token pool; see
`docs/inference-contract.md`, "Deployment change, 2026-09-20", before relying on cache behaviour. `docs/inference-contract.md`, "Deployment change, 2026-09-20", before relying on cache behaviour.
`docs/runbook.md` has an entry for every `docs/runbook.md` has an entry for every
+10 -2
View File
@@ -6,6 +6,12 @@ Newest first. A decision that changes `docs/design.md` lands in the same commit
| Date | Decision | Reason | | Date | Decision | Reason |
|---|---|---| |---|---|---|
| 2026-09-23 | M4 is split: M4a is conversations over Mattermost (`gatewayd`, the secret store), M4b is approvals over Mattermost. Spec `docs/specs/2026-09-23-m4a-gateway.md`. | As with M2 and M3: the authority-bearing half is reviewed separately, and the open question about long approvals lands in M4b. |
| 2026-09-23 | `gatewayd` reaches Mattermost over TLS, by default at the tailnet name `https://straylight.scylla-hammerhead.ts.net`; plain HTTP stays possible by URL. TLS is `rustls` 0.23 with `ring` and `rustls-native-certs` (the host's trust store, plus an optional CA file); `deny.toml` gains ISC and BSD-3-Clause. | Owner: portability, and a bigger picture that favours the tailnet name over host loopback. The host's store keeps a privately installed CA trusted; `webpki-roots` would not, and `aws-lc-rs` is a larger C build for nothing needed. |
| 2026-09-23 | Our own blocking WebSocket client (RFC 6455), adversarially tested; SHA-1 for its handshake check lives in `proto`. | `tungstenite` would add about ten crates for a few hundred lines of protocol. |
| 2026-09-23 | Secrets: one `SecretStore`, three backends chosen per secret: a systemd credential (encrypted at rest to the TPM and host key), an environment variable, or an owner-only file. Proposed as brief change P15. | Owner: a generic backend for portability. systemd 260 on straylight encrypts user-scoped credentials to its TPM (checked). |
| 2026-09-23 | In Mattermost: a thread is a session, and a top-level message starts a new one. Messages that arrive while a turn runs are sent together as the next turn. No thinking and no status in Mattermost: Boxmaker shows as typing while it works, then posts the answer. A message starting with `!` is a command to Boxmaker and never reaches the model (`!!` escapes it); M4's commands are `!approve <id>` and `!deny <id> [reason]`. | Owner's choices. `/` cannot be used: Mattermost's clients treat it as a slash command and do not post it (checked in the web client's source, v11.11.0), and a real slash command would need a listening port. |
| 2026-09-23 | M4b, agreed ahead of its spec: one `gateway.sock` connection per approval, open while it is pending; the approval posted in the asking thread or the owner's direct messages; answered only by an allowlisted ✅/❌ reaction on the approval post or `!approve`/`!deny` in its thread, after the post and while pending; long approvals in up to 4 parts with only the last answerable, else `bxctl` only; `bxctl deny` as another name for `refuse`. | Keeps `gatewayd` away from `admin.sock`; a closed connection withdraws the post; a hidden tail of an approval is where injected text would hide. |
| 2026-09-22 | M3b spec (`docs/specs/2026-09-22-m3b-runner.md`), draft for the owner's review, written for an overnight run. New: the tools' argument structs and the host rules move to `proto`, shared by `brokerd` and `toolkit`; `toolkit` exits 0/1/2 (done / the tool's own error, shown to the model / misuse, a fixed sentence); the egress proxy refuses any allowed name that resolves to a non-public address (including the tailnet's `100.64/10` and `::/96`); grant paths may not contain `:` or `,`; `[runner]` names the image by digest; tool time limits sit under `loopd`'s 120 s broker timeout, so `shell` defaults to 100 s. | A grant allows a host on the internet, not what its name points at; two programs reading the same data need one definition; `--volume` uses `:` and `,` as separators. The shell limit follows from the M3a deadline in `BrokerPort`. | | 2026-09-22 | M3b spec (`docs/specs/2026-09-22-m3b-runner.md`), draft for the owner's review, written for an overnight run. New: the tools' argument structs and the host rules move to `proto`, shared by `brokerd` and `toolkit`; `toolkit` exits 0/1/2 (done / the tool's own error, shown to the model / misuse, a fixed sentence); the egress proxy refuses any allowed name that resolves to a non-public address (including the tailnet's `100.64/10` and `::/96`); grant paths may not contain `:` or `,`; `[runner]` names the image by digest; tool time limits sit under `loopd`'s 120 s broker timeout, so `shell` defaults to 100 s. | A grant allows a host on the internet, not what its name points at; two programs reading the same data need one definition; `--volume` uses `:` and `,` as separators. The shell limit follows from the M3a deadline in `BrokerPort`. |
| 2026-09-22 | M3a review fixes. `loopd` waits at most a day after a pending frame and `brokerd` refuses `[approvals] ttl_ms` over a day. A socket whose directory is `/` or a symbolic link is refused at start. A listener out of file descriptors or memory pauses and retries instead of stopping `brokerd`. `brokerd` and `bxctl` share one rule for audit log file names (`proto::is_audit_log_name`, real months and days only). The fixes were made by the design model, and the fix commits were reviewed by a separate agent before the merge. | A far `expires` parked a turn for ever; `chmod` through a link changed its target; an idle-connection flood could stop the daemon; the two components disagreed about which files were the log. Ornith was under heavy contention. | | 2026-09-22 | M3a review fixes. `loopd` waits at most a day after a pending frame and `brokerd` refuses `[approvals] ttl_ms` over a day. A socket whose directory is `/` or a symbolic link is refused at start. A listener out of file descriptors or memory pauses and retries instead of stopping `brokerd`. `brokerd` and `bxctl` share one rule for audit log file names (`proto::is_audit_log_name`, real months and days only). The fixes were made by the design model, and the fix commits were reviewed by a separate agent before the merge. | A far `expires` parked a turn for ever; `chmod` through a link changed its target; an idle-connection flood could stop the daemon; the two components disagreed about which files were the log. Ornith was under heavy contention. |
| 2026-09-18 | M3a plan checks, `brokerd`. Every tool request gets a `Decision` record, including those denied `grants_invalid` or `state_unreadable`; forbidden kinds get none. An unreadable session state is recorded as `secret`, untrusted. A refusal that cannot be recorded is `error internal`, not `ok`. The re-decision's outcome is the matched grant's mode. A pending frame that cannot be sent is handled like a lost connection. Expiry lives in `admin`. A request frame without a read timeout is accepted for M3a. | Found while writing the reference for tasks 10 to 15: the spec left each case open, and `bxctl refuse` would have reported success for a refusal that was not on disk. | | 2026-09-18 | M3a plan checks, `brokerd`. Every tool request gets a `Decision` record, including those denied `grants_invalid` or `state_unreadable`; forbidden kinds get none. An unreadable session state is recorded as `secret`, untrusted. A refusal that cannot be recorded is `error internal`, not `ok`. The re-decision's outcome is the matched grant's mode. A pending frame that cannot be sent is handled like a lost connection. Expiry lives in `admin`. A request frame without a read timeout is accepted for M3a. | Found while writing the reference for tasks 10 to 15: the spec left each case open, and `bxctl refuse` would have reported success for a refusal that was not on disk. |
@@ -53,7 +59,7 @@ Newest first. A decision that changes `docs/design.md` lands in the same commit
| Repository licence. Crates are `publish = false` until it is chosen. | before publishing | | Repository licence. Crates are `publish = false` until it is chosen. | before publishing |
| Audit integrity beyond a self-contained hash chain (for example anchoring the chain head outside `brokerd`). Deferred by the owner on 2026-09-17 to a later revision. | after v0 | | Audit integrity beyond a self-contained hash chain (for example anchoring the chain head outside `brokerd`). Deferred by the owner on 2026-09-17 to a later revision. | after v0 |
| How an approval whose arguments do not fit one Mattermost post is shown. Settled on 2026-09-18: what is being approved is never truncated, in any channel, because the unseen tail is where injected text would put its payload; `bxctl` prints arguments in full. Leading option for Mattermost: a thread of numbered chunks split on character boundaries of the escaped text, where only a reply or reaction on the last post counts and that post repeats the id, the chunk count and the hash of the whole; above some number of chunks, the post points to `bxctl approvals` and approval is local only. The post size limit and rate limits must be checked against the Mattermost API reference first. | M4 | | How an approval whose arguments do not fit one Mattermost post is shown. Settled on 2026-09-18: what is being approved is never truncated, in any channel, because the unseen tail is where injected text would put its payload; `bxctl` prints arguments in full. Leading option for Mattermost: a thread of numbered chunks split on character boundaries of the escaped text, where only a reply or reaction on the last post counts and that post repeats the id, the chunk count and the hash of the whole; above some number of chunks, the post points to `bxctl approvals` and approval is local only. The post size limit and rate limits must be checked against the Mattermost API reference first. | M4 |
| Secret store backend, and where the v0 file's key lives. straylight has no secrets manager today. Moved from M3 on 2026-09-18. | M4 | | Secret store: settled 2026-09-23 (systemd credentials, environment, owner-only file; P15 proposed). Where `brokerd` reads tool secrets from, and whether `SecretStore` moves out of `gatewayd` into a shared place, is decided when a tool first needs a secret. | when a tool needs a secret |
| Whether cloud-led sessions are ever allowed, and for which data classes. | M6 | | Whether cloud-led sessions are ever allowed, and for which data classes. | M6 |
| Where the M5b embedding model runs. The brief says a separate `llama-server` instance, but under the shared router it would be a third model against `--models-max 2` and could unload Ornith or Laguna. Owner's note, 2026-09-17: more compute is likely by then (possibly one to three DGX Spark machines), so do not design around today's memory limit. | M5b | | Where the M5b embedding model runs. The brief says a separate `llama-server` instance, but under the shared router it would be a third model against `--models-max 2` and could unload Ornith or Laguna. Owner's note, 2026-09-17: more compute is likely by then (possibly one to three DGX Spark machines), so do not design around today's memory limit. | M5b |
@@ -86,4 +92,6 @@ Later changes to the brief:
## Proposed changes to the design brief ## Proposed changes to the design brief
None pending. | # | Change | Reason |
|---|---|---|
| P15 | State, the secrets line, becomes: "Secrets: behind a `SecretStore` trait with three backends, chosen per secret: a systemd credential (encrypted at rest; the key is the host's TPM and host key, not stored beside it), an environment variable, or a file that only the owner can read. The credential backend is the default; the other two put the secret in plaintext, in the process environment or on disk, and are for portability. None in config, none in the repo." | Owner's decision of 2026-09-23. The brief's "No plaintext secrets on disk" does not allow the file backend. |
+7
View File
@@ -8,3 +8,10 @@ allowed.
| Development | `cargo` | crates.io | Downloading the crates listed in `docs/dependencies.md` | | Development | `cargo` | crates.io | Downloading the crates listed in `docs/dependencies.md` |
| Development | `make verify-device`, `tools/check-m3a-device.sh` | straylight's `llama-server` | Checks against the real model through a private `inferproxy`; the script also reads `/slots` with `curl` first | | Development | `make verify-device`, `tools/check-m3a-device.sh` | straylight's `llama-server` | Checks against the real model through a private `inferproxy`; the script also reads `/slots` with `curl` first |
| Development | `make audit` | github.com/rustsec/advisory-db | The RustSec advisory database, fetched by `cargo deny check advisories` | | Development | `make audit` | github.com/rustsec/advisory-db | The RustSec advisory database, fetched by `cargo deny check advisories` |
| Run time (M2a) | `inferproxy` | the `llama-server` router on the host (`straylight:11434`) | Every inference request `loopd` makes, forwarded byte for byte |
| Run time (M3b) | `brokerd`'s egress proxy container, for one `http_fetch` call | only the hosts the call's grant names, on port 443, at public addresses | The tool's HTTPS request, end to end; the proxy sees only the host name |
| Run time (M4a) | `gatewayd` | the owner's Mattermost at `https://straylight.scylla-hammerhead.ts.net` (tailnet) | REST calls and one WebSocket: the owner's messages in, Boxmaker's answers out. Mattermost stores messages unencrypted in its database; the server is the owner's |
The two run-time rows for M2a and M3b were missing until 2026-09-23; they were added with the M4a
spec.
+2 -1
View File
@@ -11,7 +11,8 @@ One session per milestone. The design brief is `docs/design.md`.
| M2 | `loopd` core + `bxctl chat` | Append-only loop, baseline ≤ 3k tokens, self-test, runaway control, with one fake in-process tool | Prefix-extension property test; turn-2 cache hit on straylight | | M2 | `loopd` core + `bxctl chat` | Append-only loop, baseline ≤ 3k tokens, self-test, runaway control, with one fake in-process tool | Prefix-extension property test; turn-2 cache hit on straylight |
| M3a | `brokerd` decisions | Grants, deny by default, taint, hash-chained audit, `ask` approvals through `bxctl`, two sockets, `loopd`'s broker port; tools do not run yet | Tests for every deny path; audit tampering suite; a denial reaches Ornith on straylight as a tool result | | M3a | `brokerd` decisions | Grants, deny by default, taint, hash-chained audit, `ask` approvals through `bxctl`, two sockets, `loopd`'s broker port; tools do not run yet | Tests for every deny path; audit tampering suite; a denial reaches Ornith on straylight as a tool result |
| M3b | `brokerd` runner | Rootless container runner, four tools (read file, write file, sandboxed shell, HTTP fetch with host allowlist), egress proxy | A container with no network grant cannot reach the network; fetch refused for hosts outside the grant | | M3b | `brokerd` runner | Rootless container runner, four tools (read file, write file, sandboxed shell, HTTP fetch with host allowlist), egress proxy | A container with no network grant cannot reach the network; fetch refused for hosts outside the grant |
| M4 | `gatewayd` + Mattermost | Bot connection, user-ID allowlist, threads as sessions, async delivery, approvals by reply or reaction, no listening port | End-to-end direct message on the real server; a non-allowlisted user gets no response | | M4a | `gatewayd` + Mattermost conversations | Bot connection over TLS, user-ID allowlist, threads as sessions, bursts as one turn, typing while working, catching up after downtime, the secret store, no listening port | A direct message answered on the real server; a non-allowlisted user gets no response |
| M4b | Approvals over Mattermost | `gateway.sock`, approval posts, answers by reaction or `!approve`/`!deny`, long approvals in parts | An `ask` call approved and refused from Mattermost; the audit log names the post |
| M5 | Scheduler + memory | Heartbeat and cron on their own slot in isolated sessions; core block, `recall` and `remember` with provenance, FTS5 index, flush-then-compact at idle | Scheduled run leaves the main slot's cache intact; a fact from epoch 1 is recalled in epoch 2; a note written after reading untrusted content is marked on recall | | M5 | Scheduler + memory | Heartbeat and cron on their own slot in isolated sessions; core block, `recall` and `remember` with provenance, FTS5 index, flush-then-compact at idle | Scheduled run leaves the main slot's cache intact; a fact from epoch 1 is recalled in epoch 2; a note written after reading untrusted content is marked on recall |
| M5b | Embedding index (when justified) | Local embedding server, hybrid retrieval behind the `Retriever` trait, rebuild on model change | A paraphrased query finds a note that lexical search misses; no memory text leaves the host | | M5b | Embedding index (when justified) | Local embedding server, hybrid retrieval behind the `Retriever` trait, rebuild on model change | A paraphrased query finds a note that lexical search misses; no memory text leaves the host |
| M6 | Subagents + cloud consult | Subagents on their slot returning summaries; consult tool gated by data-class grants with full payload log | A tainted session is refused; payload log matches what was sent | | M6 | Subagents + cloud consult | Subagents on their slot returning summaries; consult tool gated by data-class grants with full payload log | A tainted session is refused; payload log matches what was sent |
+325
View File
@@ -0,0 +1,325 @@
# M4a design: `gatewayd`, conversations over Mattermost
Status: draft for owner review, 2026-09-23. M4 is split in two (`docs/decisions.md`): M4a is the
conversation path (this document), M4b is approvals over Mattermost (summarised in section 12,
specified after M4a is built). Where this document and `docs/design.md` disagree, the brief wins;
the one change it needs, P15 on secrets, is proposed in `docs/decisions.md`.
Facts this rests on, checked on 2026-09-23 against the owner's server and against Mattermost's API
source and server source at the server's version, v11.11.0:
- Mattermost runs in a rootless Podman container on straylight, plain HTTP on `127.0.0.1:8065`;
Tailscale Serve terminates TLS for `https://straylight.scylla-hammerhead.ts.net` and proxies to
it. Bot accounts are enabled. Another bot (the owner's Hermes gateway) uses the same server.
- REST: `Authorization: Bearer <token>`; `GET /api/v4/users/me`; `POST /api/v4/posts` (with
`root_id` for a reply in a thread); `GET /api/v4/channels/{id}/posts?since=<ms>`;
`POST /api/v4/channels/direct` (a direct channel between two user ids). Rate limits are reported
in `X-Ratelimit-Limit`, `-Remaining` and `-Reset`; over the limit is status 429. A post holds at
most 16,383 characters (`PostMessageMaxRunesV2`).
- WebSocket: `/api/v4/websocket`, RFC 6455, authenticated by the same header (or an
`authentication_challenge` message). Events are `{"event", "data", "broadcast", "seq"}`, starting
with `hello`. `posted` carries the post as a **JSON string** in `data.post`, and
`data.channel_type` (`"D"` for a direct message). Requests are
`{"action", "seq", "data"}`, answered by `{"status", "seq_reply"}`; `user_typing` takes
`{"channel_id", "parent_id"}` and shows the user as typing, in that thread.
- The web client does not post a message that starts with `/`: it treats it as a slash command.
Commands to Boxmaker therefore start with `!` (section 7).
## 1. What M4a proves
| Claim | Checked by |
|---|---|
| A direct message from an allowlisted user becomes a turn, and the answer is posted in its thread | Tests against a fake Mattermost; on straylight |
| Anyone else gets no response at all: no post, no typing | Tests; on straylight with a second user |
| A thread is one session; a new top-level message is a new session | Tests |
| Messages sent while a turn runs are sent together as the next turn | Tests |
| Messages sent while `gatewayd` was down are answered when it comes back; a turn cut off by a restart is reported | Tests; on straylight |
| `gatewayd` has no listening port | On straylight (`ss -ltnp`) |
| The WebSocket client survives hostile input without a panic or an unchecked allocation | Adversarial tests |
| TLS verifies: an unknown CA or a wrong name is refused | Tests with a test-only CA |
| Secrets come from a systemd credential, an environment variable or an owner-only file, and never appear in a log | Tests |
Out of scope: approvals over Mattermost (M4b), channels other than direct messages unless
allowlisted by id, files and images, edits and deletions of posts (an edited message is not
re-sent), memory and scheduled jobs (M5).
## 2. Components
```
Mattermost ── HTTPS (or HTTP) ── gatewayd ── loop.sock ── loopd
REST + WebSocket │
└── <home>/gateway/state.json
```
`gatewayd` opens **no listening socket** in M4a. (M4b adds `gateway.sock`, a Unix socket for
`brokerd`; still no TCP port.)
### Modules
Each is one file under 500 lines with one purpose.
| Module | Purpose |
|---|---|
| `config` | `gatewayd.toml` into a typed `Config`; unknown keys are errors |
| `secrets` | The `SecretStore` backends and the `Secret` type (section 4) |
| `net` | A connected stream: TCP, or TCP with TLS through `rustls` (section 5) |
| `http` | HTTP/1.1 requests over a `net` stream: `Content-Length` and chunked bodies, size caps |
| `ws` | The WebSocket client: handshake, frames, ping and close (section 6) |
| `mm` | Mattermost's JSON: the calls above and the events, typed |
| `sessions` | Which post belongs to which session; the queue per session; commands (section 7) |
| `deliver` | Turns on `loop.sock`, typing while they run, posting the answer (section 8) |
| `state` | The state file: last post seen per channel, recent post ids, turns in flight (section 9) |
| `serve` | Startup, the event loop, reconnecting |
`proto` gains `sha1` (only for the WebSocket handshake check), with the standard vectors. Base64 is
small and lives in `ws`.
### Dependencies
`gatewayd` gains `rustls` 0.23 (`default-features = false`, features `ring`, `std`, `tls12`),
`rustls-native-certs` 0.8, `zeroize` 1, and `serde`, `serde_json`, `toml` (already vetted). On
Linux, `rustls` with `ring` and native roots is 12 crates, measured 2026-09-23; `ring` builds C and
assembly. `deny.toml` gains the licences ISC and BSD-3-Clause (owner's approval, 2026-09-23).
## 3. Configuration
```toml
# gatewayd.toml
[mattermost]
url = "https://straylight.scylla-hammerhead.ts.net" # https or http; no path
ca_file = "/etc/boxmaker/extra-ca.pem" # optional; added to the system's roots
[secrets.mattermost_token]
credential = "mattermost-token" # exactly one of: credential, env, file
[allow]
users = ["abcdefghijklmnopqrstuvwxyz"] # Mattermost user ids (26 characters)
channels = [] # non-direct channels allowed, by id; default none
[loop]
socket = "" # empty: <home>/run/loop/loop.sock
[paths]
home = "/var/lib/boxmaker" # default: BOXMAKER_HOME, then /var/lib/boxmaker
[limits]
queue = 20 # messages waiting per session
typing_every_ms = 3000
ping_every_ms = 30000
dead_after_ms = 60000 # no traffic at all for this long: reconnect
```
`url` must be `http://` or `https://` with a host and an optional port, nothing else. A user id or
channel id is 26 characters of `[a-z0-9]`. An empty `allow.users` is a configuration error: a
gateway that answers nobody is a mistake.
## 4. Secrets
The brief's `SecretStore`, with three backends, chosen per secret (P15):
| Form | Reads | Refused when |
|---|---|---|
| `credential = "<name>"` | `$CREDENTIALS_DIRECTORY/<name>`, which systemd fills for a service with `LoadCredentialEncrypted=` | the variable is unset (not run by systemd with credentials), the file is missing or unreadable |
| `env = "<VAR>"` | the environment variable | unset or empty |
| `file = "<absolute path>"` | the file | not an absolute path; a symbolic link; not a regular file; not owned by the user `gatewayd` runs as; any group or other permission bit set (anything but 0600 or 0400) |
The value has one trailing newline removed; an empty value is refused. `Secret` holds it in a
`zeroize::Zeroizing<String>`, has no `Display`, and its `Debug` prints `Secret(…)`: it cannot reach
a log by accident. `Secret::expose(&self) -> &str` is the only way to the text, used only to build
the `Authorization` header.
Every refusal names the secret and the reason, never the value, and ends with
`see docs/runbook.md#secret-unavailable`. `gatewayd` does not start without its token.
For the owner: `systemd-creds --user encrypt --name=mattermost-token - <path>` encrypts the token to
the machine's TPM and host key (checked on straylight, systemd 260, 2026-09-23), and a user service
gets `LoadCredentialEncrypted=mattermost-token:<path>`.
## 5. The connection: TLS or not
`net::connect(url, ca_file) -> Stream`: for `http`, a `TcpStream`; for `https`, a `TcpStream`
wrapped in `rustls::StreamOwned<ClientConnection, TcpStream>`, verifying the server's name against
the host in `url` and its chain against the system's roots (`rustls-native-certs`) plus `ca_file`
if given. Nothing can turn verification off. A failure to load the system roots when `ca_file` is
unset is an error.
Both the REST calls and the WebSocket run over a `Stream`. REST calls use one connection per
request (`Connection: close`), as `loopd`'s HTTP client does; the WebSocket keeps one connection.
Every read has a deadline; a response header over 16 KiB or a body over 4 MiB is an error.
## 6. The WebSocket client
Our own, against RFC 6455, blocking, one connection.
- **Handshake.** `GET /api/v4/websocket` with `Upgrade: websocket`, `Connection: Upgrade`,
`Sec-WebSocket-Version: 13`, a `Sec-WebSocket-Key` of 16 bytes from `/dev/urandom` in base64, and
the `Authorization` header. The answer must be status 101 with `Upgrade: websocket` and a
`Sec-WebSocket-Accept` equal to base64(SHA-1(key + `258EAFA5-E914-47DA-95CA-C5AB0DC85B11`)).
Anything else is an error.
- **Reading.** Frames of opcodes text (1), continuation (0), close (8), ping (9), pong (10). A frame
from the server that is masked, sets a reserved bit, uses another opcode, is a control frame over
125 bytes or fragmented, is a continuation with nothing to continue, or a new data frame while a
message is unfinished, ends the connection. A message over 1 MiB, counted from the length fields
**before** anything is allocated, ends it too; so does a 64-bit length with the top bit set. Text
must be UTF-8.
- **Writing.** Every frame we send is masked with 4 fresh bytes from `/dev/urandom`. A ping is
answered with a pong carrying the same payload. We send a ping every `ping_every_ms`; no bytes
at all for `dead_after_ms` means the connection is dead.
- **Closing.** A close frame is answered with a close frame, then the connection is closed.
Any ending of the connection leads to reconnecting (section 9), never to a panic.
## 7. Sessions, the allowlist and commands
For each `posted` event: parse `data.post` (a JSON string) into `id`, `user_id`, `channel_id`,
`root_id`, `message`, `create_at`, `type`. Mattermost's JSON is not ours: unknown fields are
ignored. Then, in this order:
1. Ignore it if `user_id` is the bot's own (from `GET /api/v4/users/me` at startup), if `type` is
not empty (a system message), or if its id was handled already (section 9).
2. Ignore it, **silently**, unless `user_id` is in `allow.users` **and** the channel is a direct
message (`data.channel_type == "D"`) or its id is in `allow.channels`. No reply, no typing, no
log line with the message's text (one line with the user id and post id is enough).
3. The session: its root is `root_id`, or the post's own `id` when `root_id` is empty; the session
id is `mm-<root>` (29 characters of `[a-z0-9-]`, a valid `SessionId`). A top-level message is a
new session (`resume: false`); a reply resumes (`resume: true`). A reply in a thread `loopd`
does not know (a thread that began before Boxmaker) is created on `no_such_session`, as
`bxctl chat --session` does.
4. Commands. A message whose first character is `!`:
- `!!…` is not a command: one `!` is removed and the rest goes on as a message.
- `!approve …` and `!deny …` are M4b's; in M4a they are answered in the thread with
"approvals over Mattermost arrive in M4b; use `bxctl approvals`".
- Anything else is answered with "unknown command; the commands are !approve and !deny".
- A command is never sent to `loopd`.
5. Otherwise the message joins its session's queue. A queue already holding `queue` messages
answers "busy: too many messages are waiting in this conversation" in the thread and drops it.
## 8. Turns and delivery
One worker per session with a non-empty queue. When the session has no turn running, the worker
takes **every** message waiting, joins them with a blank line (`"\n\n"`), in the order they were
posted, and sends one `turn` on `loop.sock`.
While the turn runs, `gatewayd` sends `user_typing` over the WebSocket with the channel and the
root as `parent_id`, every `typing_every_ms`. Turn events are otherwise not shown: no thinking, no
status. One exception: an `approval_pending` event is posted once in the thread as "waiting for
approval <id>: approve or deny it with `bxctl` (Mattermost approvals arrive in M4b)".
When the turn ends:
- `turn_done` → the content is posted in the thread. Over 16,000 characters, it is split into
several replies, at the last newline before the limit (or at a character boundary if there is
none), in order.
- An `error` frame → "Error: <code>: <detail>" in the thread, with the runbook pointer when the
detail carries one.
- `loop.sock` cannot be reached, or closes early → "Boxmaker's loop is not running
(see docs/runbook.md#loop-unavailable)" in the thread, and the messages are dropped, not retried.
Then, if more messages arrived during the turn, the worker starts the next turn with them.
Several sessions can run turns at once; `loopd` serialises them at the model.
## 9. State, reconnecting and restarts
`<home>/gateway/state.json`, written atomically (write, sync, rename, sync the directory) after
every change:
```json
{"channels": {"<channel id>": 1758650000000}, "recent": ["<post id>", ],
"in_flight": [{"session": "mm-…", "channel": "<id>", "root": "<id>"}]}
```
- `channels`: the `create_at` of the last post handled in each channel.
- `recent`: the ids of the last 500 posts handled, so a post seen twice (live and in a catch-up) is
handled once.
- `in_flight`: turns sent to `loopd` and not yet answered.
A file that exists but cannot be read or parsed stops `gatewayd`, with
`see docs/runbook.md#gateway-state-damaged`; a missing file is a first start.
**Connecting** (at start and after every loss): `GET /users/me`; open the WebSocket; wait for
`hello`. Then **catch up**: for every channel in `channels`, and the direct channel with every
allowlisted user (`POST /channels/direct`), fetch the posts since the last one handled
(`GET /channels/{id}/posts?since=…`) and handle them in `create_at` order, before any live event.
On a first start, `channels` is empty and nothing is caught up: history is not answered.
**Losing the connection**: reconnect after 1, 2, 5, 10, then every 30 seconds, printing one line
per attempt. A REST call answered 429 waits until `X-Ratelimit-Reset`; a 5xx is retried twice; a
401 or 403 stops `gatewayd` with `see docs/runbook.md#mattermost-auth-failed`, since retrying with
a wrong token would not help.
**Restarting**: for every entry left in `in_flight`, `gatewayd` posts "interrupted: gatewayd
restarted before the answer arrived; ask again" in its thread and removes it. `loopd` finished the
turn and logged it; the answer is in the session log.
## 10. Runbook
New entries: `secret-unavailable`, `mattermost-unreachable` (connecting keeps failing),
`mattermost-auth-failed`, `gateway-state-damaged`, `loop-unavailable`, `gatewayd-start-failed`
(configuration). Every message for these states ends with its pointer.
## 11. Testing
Offline, as tasks with given tests, against fakes:
- **`proto::sha1` and base64**: FIPS 180 vectors and RFC 4648 vectors; lengths 55, 56, 63, 64 and
65; each also fed in pieces.
- **WebSocket**: the handshake (the exact request; RFC 6455's example key and accept; refusal of a
wrong accept, a status other than 101, a missing `Upgrade`, oversized headers). Frames: seeded
random frame streams checked against a deliberately naive decoder in the test; every hostile
frame in section 6, each ending the connection cleanly with no panic; masking of our frames;
ping and pong; fragments reassembled; the close handshake; a server that trickles one byte at a
time, and one that goes silent.
- **TLS**: a test-only CA and server certificate generated once with `openssl` and committed as
fixtures (long expiry, labelled test keys); the client reaches a local TLS server through
`ca_file`; an unknown CA and a name mismatch are refused.
- **HTTP**: `Content-Length`, chunked, the size caps, 429 with `X-Ratelimit-Reset`.
- **Secrets**: each backend, each refusal in section 4's table, and that `Debug` shows no value.
- **The gateway, against a fake Mattermost** (test support, with a reference implementation): a
scripted HTTP and WebSocket server that records every request and pushes events; and a fake
`loopd` on a Unix socket. Every rule of sections 7 to 9: the allowlist (nothing at all for anyone
else, and nothing for non-direct channels), own posts ignored, sessions from roots, the burst
joined into one turn, the queue limit, typing sent while a turn runs and stopped after, splitting
a long answer, commands and `!!`, catch-up after a reconnect, `recent` preventing a second
answer, the interrupted reply after a restart, and each runbook pointer.
On straylight, by the design model, once the owner has a bot account and a second Mattermost user:
a direct message is answered in its thread while Boxmaker shows as typing; the second user gets
nothing; a burst is one turn; messages sent while `gatewayd` was stopped are answered when it
starts; a restart mid-turn gives the interrupted reply; `ss -ltnp` shows no port for `gatewayd`;
the token comes from a systemd credential.
## 12. M4b in summary
Agreed on 2026-09-23, to be specified after M4a is built:
- `brokerd` opens a connection to `gateway.sock` (a Unix socket `gatewayd` listens on) for each
approval, with the approval's id, tool, arguments, grant, taint and expiry. `gatewayd` posts it,
answers with the post's id, and keeps the connection open until the owner answers; the final
frame is the answer (approved or refused, the Mattermost user id, the post id, a reason). If
`brokerd` closes the connection (answered in `bxctl`, expired, requester gone, restarted), the
post is edited to "no longer pending". `gatewayd` never reaches `admin.sock`.
- The approval is posted in the asking session's thread when it is a Mattermost session, otherwise
in the owner's direct messages. Arguments escaped as in `bxctl`, in a code fence longer than any
run of backticks inside them.
- An answer counts only if it is from an allowlisted user, after the approval post, while pending,
and either a ✅ or ❌ reaction on the approval post itself, or a reply in its thread of exactly
`!approve <id>` or `!deny <id> [reason]`.
- Arguments too long for one post are split into up to 4 numbered parts, the last repeating the id,
the count and the SHA-256 of the whole; only an answer on the last part counts. Longer than that:
one post with the size and the hash, answerable only in `bxctl`.
- `bxctl` gains `deny` as another name for `refuse`.
## 13. Threat model notes
- `gatewayd` holds the bot token. With the environment or file backend the token is in plaintext
(in `/proc/<pid>/environ` to the same user, or on disk under an owner-only mode); the credential
backend keeps it encrypted at rest. The owner chooses per deployment (P15).
- Only allowlisted user ids are heard. Mattermost and the tailnet are trusted (decision of
2026-09-17), so a post's `user_id` is believed.
- The model's answer is posted as it is. Mattermost renders markdown; a hostile answer can format
itself, but reaches only the owner, who asked.
- M4b gives `gatewayd` the power to answer pending approvals, as `bxctl` has; it cannot create
one, widen a grant or read one.
- M7 must let `gatewayd`'s container reach the tailnet name (and, if `url` is loopback HTTP, the
host's loopback).