kyle/boxmaker
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Specify M4a (conversations over Mattermost); propose P15 on secrets

Browse Source 
M4 is split into M4a and M4b. The M4a spec rests on facts checked against
the owner's server and Mattermost's source at v11.11.0: the REST and
WebSocket shapes, and that clients will not post a message starting with
'/'. Records the design decisions, proposes P15 (secrets from a systemd
credential, the environment or an owner-only file), and adds the run-time
rows egress.md was missing since M2a and M3b.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
kyle
2026-09-23 17:30:31 -07:00
co-authored by Claude Opus 5.5
parent f0ec0efe9d
commit 7e3783faae
5 changed files with 346 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/specs/2026-09-23-m4a-gateway.md
+325
View File
@@ -0,0 +1,325 @@
# M4a design: `gatewayd`, conversations over Mattermost
Status: draft for owner review, 2026-09-23. M4 is split in two (`docs/decisions.md`): M4a is the
conversation path (this document), M4b is approvals over Mattermost (summarised in section 12,
specified after M4a is built). Where this document and `docs/design.md` disagree, the brief wins;
the one change it needs, P15 on secrets, is proposed in `docs/decisions.md`.
Facts this rests on, checked on 2026-09-23 against the owner's server and against Mattermost's API
source and server source at the server's version, v11.11.0:
- Mattermost runs in a rootless Podman container on straylight, plain HTTP on `127.0.0.1:8065`;
Tailscale Serve terminates TLS for `https://straylight.scylla-hammerhead.ts.net` and proxies to
it. Bot accounts are enabled. Another bot (the owner's Hermes gateway) uses the same server.
- REST: `Authorization: Bearer <token>`; `GET /api/v4/users/me`; `POST /api/v4/posts` (with
`root_id` for a reply in a thread); `GET /api/v4/channels/{id}/posts?since=<ms>`;
`POST /api/v4/channels/direct` (a direct channel between two user ids). Rate limits are reported
in `X-Ratelimit-Limit`, `-Remaining` and `-Reset`; over the limit is status 429. A post holds at
most 16,383 characters (`PostMessageMaxRunesV2`).
- WebSocket: `/api/v4/websocket`, RFC 6455, authenticated by the same header (or an
`authentication_challenge` message). Events are `{"event", "data", "broadcast", "seq"}`, starting
with `hello`. `posted` carries the post as a **JSON string** in `data.post`, and
`data.channel_type` (`"D"` for a direct message). Requests are
`{"action", "seq", "data"}`, answered by `{"status", "seq_reply"}`; `user_typing` takes
`{"channel_id", "parent_id"}` and shows the user as typing, in that thread.
- The web client does not post a message that starts with `/`: it treats it as a slash command.
Commands to Boxmaker therefore start with `!` (section 7).
## 1. What M4a proves
| Claim | Checked by |
|---|---|
| A direct message from an allowlisted user becomes a turn, and the answer is posted in its thread | Tests against a fake Mattermost; on straylight |
| Anyone else gets no response at all: no post, no typing | Tests; on straylight with a second user |
| A thread is one session; a new top-level message is a new session | Tests |
| Messages sent while a turn runs are sent together as the next turn | Tests |
| Messages sent while `gatewayd` was down are answered when it comes back; a turn cut off by a restart is reported | Tests; on straylight |
| `gatewayd` has no listening port | On straylight (`ss -ltnp`) |
| The WebSocket client survives hostile input without a panic or an unchecked allocation | Adversarial tests |
| TLS verifies: an unknown CA or a wrong name is refused | Tests with a test-only CA |
| Secrets come from a systemd credential, an environment variable or an owner-only file, and never appear in a log | Tests |
Out of scope: approvals over Mattermost (M4b), channels other than direct messages unless
allowlisted by id, files and images, edits and deletions of posts (an edited message is not
re-sent), memory and scheduled jobs (M5).
## 2. Components
```
Mattermost ── HTTPS (or HTTP) ── gatewayd ── loop.sock ── loopd
REST + WebSocket │
└── <home>/gateway/state.json
```
`gatewayd` opens **no listening socket** in M4a. (M4b adds `gateway.sock`, a Unix socket for
`brokerd`; still no TCP port.)
### Modules
Each is one file under 500 lines with one purpose.
| Module | Purpose |
|---|---|
| `config` | `gatewayd.toml` into a typed `Config`; unknown keys are errors |
| `secrets` | The `SecretStore` backends and the `Secret` type (section 4) |
| `net` | A connected stream: TCP, or TCP with TLS through `rustls` (section 5) |
| `http` | HTTP/1.1 requests over a `net` stream: `Content-Length` and chunked bodies, size caps |
| `ws` | The WebSocket client: handshake, frames, ping and close (section 6) |
| `mm` | Mattermost's JSON: the calls above and the events, typed |
| `sessions` | Which post belongs to which session; the queue per session; commands (section 7) |
| `deliver` | Turns on `loop.sock`, typing while they run, posting the answer (section 8) |
| `state` | The state file: last post seen per channel, recent post ids, turns in flight (section 9) |
| `serve` | Startup, the event loop, reconnecting |
`proto` gains `sha1` (only for the WebSocket handshake check), with the standard vectors. Base64 is
small and lives in `ws`.
### Dependencies
`gatewayd` gains `rustls` 0.23 (`default-features = false`, features `ring`, `std`, `tls12`),
`rustls-native-certs` 0.8, `zeroize` 1, and `serde`, `serde_json`, `toml` (already vetted). On
Linux, `rustls` with `ring` and native roots is 12 crates, measured 2026-09-23; `ring` builds C and
assembly. `deny.toml` gains the licences ISC and BSD-3-Clause (owner's approval, 2026-09-23).
## 3. Configuration
```toml
# gatewayd.toml
[mattermost]
url = "https://straylight.scylla-hammerhead.ts.net" # https or http; no path
ca_file = "/etc/boxmaker/extra-ca.pem" # optional; added to the system's roots
[secrets.mattermost_token]
credential = "mattermost-token" # exactly one of: credential, env, file
[allow]
users = ["abcdefghijklmnopqrstuvwxyz"] # Mattermost user ids (26 characters)
channels = [] # non-direct channels allowed, by id; default none
[loop]
socket = "" # empty: <home>/run/loop/loop.sock
[paths]
home = "/var/lib/boxmaker" # default: BOXMAKER_HOME, then /var/lib/boxmaker
[limits]
queue = 20 # messages waiting per session
typing_every_ms = 3000
ping_every_ms = 30000
dead_after_ms = 60000 # no traffic at all for this long: reconnect
```
`url` must be `http://` or `https://` with a host and an optional port, nothing else. A user id or
channel id is 26 characters of `[a-z0-9]`. An empty `allow.users` is a configuration error: a
gateway that answers nobody is a mistake.
## 4. Secrets
The brief's `SecretStore`, with three backends, chosen per secret (P15):
| Form | Reads | Refused when |
|---|---|---|
| `credential = "<name>"` | `$CREDENTIALS_DIRECTORY/<name>`, which systemd fills for a service with `LoadCredentialEncrypted=` | the variable is unset (not run by systemd with credentials), the file is missing or unreadable |
| `env = "<VAR>"` | the environment variable | unset or empty |
| `file = "<absolute path>"` | the file | not an absolute path; a symbolic link; not a regular file; not owned by the user `gatewayd` runs as; any group or other permission bit set (anything but 0600 or 0400) |
The value has one trailing newline removed; an empty value is refused. `Secret` holds it in a
`zeroize::Zeroizing<String>`, has no `Display`, and its `Debug` prints `Secret(…)`: it cannot reach
a log by accident. `Secret::expose(&self) -> &str` is the only way to the text, used only to build
the `Authorization` header.
Every refusal names the secret and the reason, never the value, and ends with
`see docs/runbook.md#secret-unavailable`. `gatewayd` does not start without its token.
For the owner: `systemd-creds --user encrypt --name=mattermost-token - <path>` encrypts the token to
the machine's TPM and host key (checked on straylight, systemd 260, 2026-09-23), and a user service
gets `LoadCredentialEncrypted=mattermost-token:<path>`.
## 5. The connection: TLS or not
`net::connect(url, ca_file) -> Stream`: for `http`, a `TcpStream`; for `https`, a `TcpStream`
wrapped in `rustls::StreamOwned<ClientConnection, TcpStream>`, verifying the server's name against
the host in `url` and its chain against the system's roots (`rustls-native-certs`) plus `ca_file`
if given. Nothing can turn verification off. A failure to load the system roots when `ca_file` is
unset is an error.
Both the REST calls and the WebSocket run over a `Stream`. REST calls use one connection per
request (`Connection: close`), as `loopd`'s HTTP client does; the WebSocket keeps one connection.
Every read has a deadline; a response header over 16 KiB or a body over 4 MiB is an error.
## 6. The WebSocket client
Our own, against RFC 6455, blocking, one connection.
- **Handshake.** `GET /api/v4/websocket` with `Upgrade: websocket`, `Connection: Upgrade`,
`Sec-WebSocket-Version: 13`, a `Sec-WebSocket-Key` of 16 bytes from `/dev/urandom` in base64, and
the `Authorization` header. The answer must be status 101 with `Upgrade: websocket` and a
`Sec-WebSocket-Accept` equal to base64(SHA-1(key + `258EAFA5-E914-47DA-95CA-C5AB0DC85B11`)).
Anything else is an error.
- **Reading.** Frames of opcodes text (1), continuation (0), close (8), ping (9), pong (10). A frame
from the server that is masked, sets a reserved bit, uses another opcode, is a control frame over
125 bytes or fragmented, is a continuation with nothing to continue, or a new data frame while a
message is unfinished, ends the connection. A message over 1 MiB, counted from the length fields
**before** anything is allocated, ends it too; so does a 64-bit length with the top bit set. Text
must be UTF-8.
- **Writing.** Every frame we send is masked with 4 fresh bytes from `/dev/urandom`. A ping is
answered with a pong carrying the same payload. We send a ping every `ping_every_ms`; no bytes
at all for `dead_after_ms` means the connection is dead.
- **Closing.** A close frame is answered with a close frame, then the connection is closed.
Any ending of the connection leads to reconnecting (section 9), never to a panic.
## 7. Sessions, the allowlist and commands
For each `posted` event: parse `data.post` (a JSON string) into `id`, `user_id`, `channel_id`,
`root_id`, `message`, `create_at`, `type`. Mattermost's JSON is not ours: unknown fields are
ignored. Then, in this order:
1. Ignore it if `user_id` is the bot's own (from `GET /api/v4/users/me` at startup), if `type` is
not empty (a system message), or if its id was handled already (section 9).
2. Ignore it, **silently**, unless `user_id` is in `allow.users` **and** the channel is a direct
message (`data.channel_type == "D"`) or its id is in `allow.channels`. No reply, no typing, no
log line with the message's text (one line with the user id and post id is enough).
3. The session: its root is `root_id`, or the post's own `id` when `root_id` is empty; the session
id is `mm-<root>` (29 characters of `[a-z0-9-]`, a valid `SessionId`). A top-level message is a
new session (`resume: false`); a reply resumes (`resume: true`). A reply in a thread `loopd`
does not know (a thread that began before Boxmaker) is created on `no_such_session`, as
`bxctl chat --session` does.
4. Commands. A message whose first character is `!`:
- `!!…` is not a command: one `!` is removed and the rest goes on as a message.
- `!approve …` and `!deny …` are M4b's; in M4a they are answered in the thread with
"approvals over Mattermost arrive in M4b; use `bxctl approvals`".
- Anything else is answered with "unknown command; the commands are !approve and !deny".
- A command is never sent to `loopd`.
5. Otherwise the message joins its session's queue. A queue already holding `queue` messages
answers "busy: too many messages are waiting in this conversation" in the thread and drops it.
## 8. Turns and delivery
One worker per session with a non-empty queue. When the session has no turn running, the worker
takes **every** message waiting, joins them with a blank line (`"\n\n"`), in the order they were
posted, and sends one `turn` on `loop.sock`.
While the turn runs, `gatewayd` sends `user_typing` over the WebSocket with the channel and the
root as `parent_id`, every `typing_every_ms`. Turn events are otherwise not shown: no thinking, no
status. One exception: an `approval_pending` event is posted once in the thread as "waiting for
approval <id>: approve or deny it with `bxctl` (Mattermost approvals arrive in M4b)".
When the turn ends:
- `turn_done` → the content is posted in the thread. Over 16,000 characters, it is split into
several replies, at the last newline before the limit (or at a character boundary if there is
none), in order.
- An `error` frame → "Error: <code>: <detail>" in the thread, with the runbook pointer when the
detail carries one.
- `loop.sock` cannot be reached, or closes early → "Boxmaker's loop is not running
(see docs/runbook.md#loop-unavailable)" in the thread, and the messages are dropped, not retried.
Then, if more messages arrived during the turn, the worker starts the next turn with them.
Several sessions can run turns at once; `loopd` serialises them at the model.
## 9. State, reconnecting and restarts
`<home>/gateway/state.json`, written atomically (write, sync, rename, sync the directory) after
every change:
```json
{"channels": {"<channel id>": 1758650000000}, "recent": ["<post id>", …],
"in_flight": [{"session": "mm-…", "channel": "<id>", "root": "<id>"}]}
```
- `channels`: the `create_at` of the last post handled in each channel.
- `recent`: the ids of the last 500 posts handled, so a post seen twice (live and in a catch-up) is
handled once.
- `in_flight`: turns sent to `loopd` and not yet answered.
A file that exists but cannot be read or parsed stops `gatewayd`, with
`see docs/runbook.md#gateway-state-damaged`; a missing file is a first start.
**Connecting** (at start and after every loss): `GET /users/me`; open the WebSocket; wait for
`hello`. Then **catch up**: for every channel in `channels`, and the direct channel with every
allowlisted user (`POST /channels/direct`), fetch the posts since the last one handled
(`GET /channels/{id}/posts?since=…`) and handle them in `create_at` order, before any live event.
On a first start, `channels` is empty and nothing is caught up: history is not answered.
**Losing the connection**: reconnect after 1, 2, 5, 10, then every 30 seconds, printing one line
per attempt. A REST call answered 429 waits until `X-Ratelimit-Reset`; a 5xx is retried twice; a
401 or 403 stops `gatewayd` with `see docs/runbook.md#mattermost-auth-failed`, since retrying with
a wrong token would not help.
**Restarting**: for every entry left in `in_flight`, `gatewayd` posts "interrupted: gatewayd
restarted before the answer arrived; ask again" in its thread and removes it. `loopd` finished the
turn and logged it; the answer is in the session log.
## 10. Runbook
New entries: `secret-unavailable`, `mattermost-unreachable` (connecting keeps failing),
`mattermost-auth-failed`, `gateway-state-damaged`, `loop-unavailable`, `gatewayd-start-failed`
(configuration). Every message for these states ends with its pointer.
## 11. Testing
Offline, as tasks with given tests, against fakes:
- **`proto::sha1` and base64**: FIPS 180 vectors and RFC 4648 vectors; lengths 55, 56, 63, 64 and
65; each also fed in pieces.
- **WebSocket**: the handshake (the exact request; RFC 6455's example key and accept; refusal of a
wrong accept, a status other than 101, a missing `Upgrade`, oversized headers). Frames: seeded
random frame streams checked against a deliberately naive decoder in the test; every hostile
frame in section 6, each ending the connection cleanly with no panic; masking of our frames;
ping and pong; fragments reassembled; the close handshake; a server that trickles one byte at a
time, and one that goes silent.
- **TLS**: a test-only CA and server certificate generated once with `openssl` and committed as
fixtures (long expiry, labelled test keys); the client reaches a local TLS server through
`ca_file`; an unknown CA and a name mismatch are refused.
- **HTTP**: `Content-Length`, chunked, the size caps, 429 with `X-Ratelimit-Reset`.
- **Secrets**: each backend, each refusal in section 4's table, and that `Debug` shows no value.
- **The gateway, against a fake Mattermost** (test support, with a reference implementation): a
scripted HTTP and WebSocket server that records every request and pushes events; and a fake
`loopd` on a Unix socket. Every rule of sections 7 to 9: the allowlist (nothing at all for anyone
else, and nothing for non-direct channels), own posts ignored, sessions from roots, the burst
joined into one turn, the queue limit, typing sent while a turn runs and stopped after, splitting
a long answer, commands and `!!`, catch-up after a reconnect, `recent` preventing a second
answer, the interrupted reply after a restart, and each runbook pointer.
On straylight, by the design model, once the owner has a bot account and a second Mattermost user:
a direct message is answered in its thread while Boxmaker shows as typing; the second user gets
nothing; a burst is one turn; messages sent while `gatewayd` was stopped are answered when it
starts; a restart mid-turn gives the interrupted reply; `ss -ltnp` shows no port for `gatewayd`;
the token comes from a systemd credential.
## 12. M4b in summary
Agreed on 2026-09-23, to be specified after M4a is built:
- `brokerd` opens a connection to `gateway.sock` (a Unix socket `gatewayd` listens on) for each
approval, with the approval's id, tool, arguments, grant, taint and expiry. `gatewayd` posts it,
answers with the post's id, and keeps the connection open until the owner answers; the final
frame is the answer (approved or refused, the Mattermost user id, the post id, a reason). If
`brokerd` closes the connection (answered in `bxctl`, expired, requester gone, restarted), the
post is edited to "no longer pending". `gatewayd` never reaches `admin.sock`.
- The approval is posted in the asking session's thread when it is a Mattermost session, otherwise
in the owner's direct messages. Arguments escaped as in `bxctl`, in a code fence longer than any
run of backticks inside them.
- An answer counts only if it is from an allowlisted user, after the approval post, while pending,
and either a ✅ or ❌ reaction on the approval post itself, or a reply in its thread of exactly
`!approve <id>` or `!deny <id> [reason]`.
- Arguments too long for one post are split into up to 4 numbered parts, the last repeating the id,
the count and the SHA-256 of the whole; only an answer on the last part counts. Longer than that:
one post with the size and the hash, answerable only in `bxctl`.
- `bxctl` gains `deny` as another name for `refuse`.
## 13. Threat model notes
- `gatewayd` holds the bot token. With the environment or file backend the token is in plaintext
(in `/proc/<pid>/environ` to the same user, or on disk under an owner-only mode); the credential
backend keeps it encrypted at rest. The owner chooses per deployment (P15).
- Only allowlisted user ids are heard. Mattermost and the tailnet are trusted (decision of
2026-09-17), so a post's `user_id` is believed.
- The model's answer is posted as it is. Mattermost renders markdown; a hostile answer can format
itself, but reaches only the owner, who asked.
- M4b gives `gatewayd` the power to answer pending approvals, as `bxctl` has; it cannot create
one, widen a grant or read one.
- M7 must let `gatewayd`'s container reach the tailnet name (and, if `url` is loopback HTTP, the
host's loopback).