A secret read from a file warns at startup. In allowed channels and group messages, Boxmaker answers only posts that name it, or replies in its own threads that name nobody else; one Boxmaker per machine, each its own bot. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
373 lines
22 KiB
Markdown
373 lines
22 KiB
Markdown
# M4a design: `gatewayd`, conversations over Mattermost
|
|
|
|
Status: approved by the owner, 2026-09-23, with two changes folded in (a warning for a file
|
|
secret; channels shared with other agents, section 7). M4 is split in two (`docs/decisions.md`): M4a is the
|
|
conversation path (this document), M4b is approvals over Mattermost (summarised in section 12,
|
|
specified after M4a is built). Where this document and `docs/design.md` disagree, the brief wins;
|
|
the one change it needs, P15 on secrets, is proposed in `docs/decisions.md`.
|
|
|
|
Facts this rests on, checked on 2026-09-23 against the owner's server and against Mattermost's API
|
|
source and server source at the server's version, v11.11.0:
|
|
|
|
- Mattermost runs in a rootless Podman container on straylight, plain HTTP on `127.0.0.1:8065`;
|
|
Tailscale Serve terminates TLS for `https://straylight.scylla-hammerhead.ts.net` and proxies to
|
|
it. Bot accounts are enabled. Another bot (the owner's Hermes gateway) uses the same server.
|
|
- REST: `Authorization: Bearer <token>`; `GET /api/v4/users/me`; `POST /api/v4/posts` (with
|
|
`root_id` for a reply in a thread); `GET /api/v4/channels/{id}/posts?since=<ms>`;
|
|
`POST /api/v4/channels/direct` (a direct channel between two user ids). Rate limits are reported
|
|
in `X-Ratelimit-Limit`, `-Remaining` and `-Reset`; over the limit is status 429. A post holds at
|
|
most 16,383 characters (`PostMessageMaxRunesV2`).
|
|
- WebSocket: `/api/v4/websocket`, RFC 6455, authenticated by the same header (or an
|
|
`authentication_challenge` message). Events are `{"event", "data", "broadcast", "seq"}`, starting
|
|
with `hello`. `posted` carries the post as a **JSON string** in `data.post`, and
|
|
`data.channel_type` (`"D"` for a direct message). Requests are
|
|
`{"action", "seq", "data"}`, answered by `{"status", "seq_reply"}`; `user_typing` takes
|
|
`{"channel_id", "parent_id"}` and shows the user as typing, in that thread.
|
|
- The web client does not post a message that starts with `/`: it treats it as a slash command.
|
|
Commands to Boxmaker therefore start with `!` (section 7).
|
|
- A post's `from_bot` property is settable by any user (the server's own comment, v11.11.0), so it
|
|
does not say who is a bot. Mentions are added to events per connection; `gatewayd` finds its own
|
|
`@username` in the message text instead.
|
|
- The owner runs several agents in Mattermost channels, and may run one Boxmaker on each of several
|
|
machines (at most one per machine), each with its own bot account.
|
|
|
|
## 1. What M4a proves
|
|
|
|
| Claim | Checked by |
|
|
|---|---|
|
|
| A direct message from an allowlisted user becomes a turn, and the answer is posted in its thread | Tests against a fake Mattermost; on straylight |
|
|
| In an allowed channel shared with other agents, Boxmaker answers only posts that name it, or replies in its own threads that name nobody else | Tests; on straylight in a channel |
|
|
| Anyone else gets no response at all: no post, no typing | Tests; on straylight with a second user |
|
|
| A thread is one session; a new top-level message is a new session | Tests |
|
|
| Messages sent while a turn runs are sent together as the next turn | Tests |
|
|
| Messages sent while `gatewayd` was down are answered when it comes back; a turn cut off by a restart is reported | Tests; on straylight |
|
|
| `gatewayd` has no listening port | On straylight (`ss -ltnp`) |
|
|
| The WebSocket client survives hostile input without a panic or an unchecked allocation | Adversarial tests |
|
|
| TLS verifies: an unknown CA or a wrong name is refused | Tests with a test-only CA |
|
|
| Secrets come from a systemd credential, an environment variable or an owner-only file, and never appear in a log | Tests |
|
|
|
|
Out of scope: approvals over Mattermost (M4b), channels not allowlisted by id, the earlier posts of
|
|
a thread Boxmaker is first named in (only the naming post reaches it), files and images, edits and
|
|
deletions of posts (an edited message is not re-sent), memory and scheduled jobs (M5).
|
|
|
|
## 2. Components
|
|
|
|
```
|
|
Mattermost ── HTTPS (or HTTP) ── gatewayd ── loop.sock ── loopd
|
|
REST + WebSocket │
|
|
└── <home>/gateway/state.json
|
|
```
|
|
|
|
`gatewayd` opens **no listening socket** in M4a. (M4b adds `gateway.sock`, a Unix socket for
|
|
`brokerd`; still no TCP port.)
|
|
|
|
### Modules
|
|
|
|
Each is one file under 500 lines with one purpose.
|
|
|
|
| Module | Purpose |
|
|
|---|---|
|
|
| `config` | `gatewayd.toml` into a typed `Config`; unknown keys are errors |
|
|
| `secrets` | The `SecretStore` backends and the `Secret` type (section 4) |
|
|
| `net` | A connected stream: TCP, or TCP with TLS through `rustls` (section 5) |
|
|
| `http` | HTTP/1.1 requests over a `net` stream: `Content-Length` and chunked bodies, size caps |
|
|
| `ws` | The WebSocket client: handshake, frames, ping and close (section 6) |
|
|
| `mm` | Mattermost's JSON: the calls above and the events, typed |
|
|
| `sessions` | Which post belongs to which session; the queue per session; commands (section 7) |
|
|
| `deliver` | Turns on `loop.sock`, typing while they run, posting the answer (section 8) |
|
|
| `state` | The state file: last post seen per channel, recent post ids, turns in flight (section 9) |
|
|
| `serve` | Startup, the event loop, reconnecting |
|
|
|
|
`proto` gains `sha1` (only for the WebSocket handshake check), with the standard vectors. Base64 is
|
|
small and lives in `ws`.
|
|
|
|
### Dependencies
|
|
|
|
`gatewayd` gains `rustls` 0.23 (`default-features = false`, features `ring`, `std`, `tls12`),
|
|
`rustls-native-certs` 0.8, `zeroize` 1, and `serde`, `serde_json`, `toml` (already vetted). On
|
|
Linux, `rustls` with `ring` and native roots is 12 crates, measured 2026-09-23; `ring` builds C and
|
|
assembly. `deny.toml` gains the licences ISC and BSD-3-Clause (owner's approval, 2026-09-23).
|
|
|
|
## 3. Configuration
|
|
|
|
```toml
|
|
# gatewayd.toml
|
|
[mattermost]
|
|
url = "https://straylight.scylla-hammerhead.ts.net" # https or http; no path
|
|
ca_file = "/etc/boxmaker/extra-ca.pem" # optional; added to the system's roots
|
|
|
|
[secrets.mattermost_token]
|
|
credential = "mattermost-token" # exactly one of: credential, env, file
|
|
|
|
[allow]
|
|
users = ["abcdefghijklmnopqrstuvwxyz"] # Mattermost user ids (26 characters)
|
|
channels = [] # channels and group messages allowed, by id; default none
|
|
|
|
[loop]
|
|
socket = "" # empty: <home>/run/loop/loop.sock
|
|
|
|
[paths]
|
|
home = "/var/lib/boxmaker" # default: BOXMAKER_HOME, then /var/lib/boxmaker
|
|
|
|
[limits]
|
|
queue = 20 # messages waiting per session
|
|
typing_every_ms = 3000
|
|
ping_every_ms = 30000
|
|
dead_after_ms = 60000 # no traffic at all for this long: reconnect
|
|
```
|
|
|
|
`url` must be `http://` or `https://` with a host and an optional port, nothing else. A user id or
|
|
channel id is 26 characters of `[a-z0-9]`. An empty `allow.users` is a configuration error: a
|
|
gateway that answers nobody is a mistake.
|
|
|
|
Allowlisting another agent's user id lets that agent talk to Boxmaker. Two agents that allowlist
|
|
each other will answer each other without end; nothing prevents it, so do not.
|
|
|
|
## 4. Secrets
|
|
|
|
The brief's `SecretStore`, with three backends, chosen per secret (P15):
|
|
|
|
| Form | Reads | Refused when |
|
|
|---|---|---|
|
|
| `credential = "<name>"` | `$CREDENTIALS_DIRECTORY/<name>`, which systemd fills for a service with `LoadCredentialEncrypted=` | the variable is unset (not run by systemd with credentials), the file is missing or unreadable |
|
|
| `env = "<VAR>"` | the environment variable | unset or empty |
|
|
| `file = "<absolute path>"` | the file | not an absolute path; a symbolic link; not a regular file; not owned by the user `gatewayd` runs as; any group or other permission bit set (anything but 0600 or 0400) |
|
|
|
|
The value has one trailing newline removed; an empty value is refused.
|
|
|
|
A secret read from a **file** is in plaintext on disk. `gatewayd` starts, and prints at startup, for
|
|
each such secret: `gatewayd: warning: secret <name> is read in plaintext from <path>; a systemd
|
|
credential keeps it encrypted at rest (see docs/runbook.md#secret-in-a-file)`. It is a warning,
|
|
not a refusal: there are deployments where a file is right. `Secret` holds it in a
|
|
`zeroize::Zeroizing<String>`, has no `Display`, and its `Debug` prints `Secret(…)`: it cannot reach
|
|
a log by accident. `Secret::expose(&self) -> &str` is the only way to the text, used only to build
|
|
the `Authorization` header.
|
|
|
|
Every refusal names the secret and the reason, never the value, and ends with
|
|
`see docs/runbook.md#secret-unavailable`. `gatewayd` does not start without its token.
|
|
|
|
For the owner: `systemd-creds --user encrypt --name=mattermost-token - <path>` encrypts the token to
|
|
the machine's TPM and host key (checked on straylight, systemd 260, 2026-09-23), and a user service
|
|
gets `LoadCredentialEncrypted=mattermost-token:<path>`.
|
|
|
|
## 5. The connection: TLS or not
|
|
|
|
`net::connect(url, ca_file) -> Stream`: for `http`, a `TcpStream`; for `https`, a `TcpStream`
|
|
wrapped in `rustls::StreamOwned<ClientConnection, TcpStream>`, verifying the server's name against
|
|
the host in `url` and its chain against the system's roots (`rustls-native-certs`) plus `ca_file`
|
|
if given. Nothing can turn verification off. A failure to load the system roots when `ca_file` is
|
|
unset is an error.
|
|
|
|
Both the REST calls and the WebSocket run over a `Stream`. REST calls use one connection per
|
|
request (`Connection: close`), as `loopd`'s HTTP client does; the WebSocket keeps one connection.
|
|
Every read has a deadline; a response header over 16 KiB or a body over 4 MiB is an error.
|
|
|
|
## 6. The WebSocket client
|
|
|
|
Our own, against RFC 6455, blocking, one connection.
|
|
|
|
- **Handshake.** `GET /api/v4/websocket` with `Upgrade: websocket`, `Connection: Upgrade`,
|
|
`Sec-WebSocket-Version: 13`, a `Sec-WebSocket-Key` of 16 bytes from `/dev/urandom` in base64, and
|
|
the `Authorization` header. The answer must be status 101 with `Upgrade: websocket` and a
|
|
`Sec-WebSocket-Accept` equal to base64(SHA-1(key + `258EAFA5-E914-47DA-95CA-C5AB0DC85B11`)).
|
|
Anything else is an error.
|
|
- **Reading.** Frames of opcodes text (1), continuation (0), close (8), ping (9), pong (10). A frame
|
|
from the server that is masked, sets a reserved bit, uses another opcode, is a control frame over
|
|
125 bytes or fragmented, is a continuation with nothing to continue, or a new data frame while a
|
|
message is unfinished, ends the connection. A message over 1 MiB, counted from the length fields
|
|
**before** anything is allocated, ends it too; so does a 64-bit length with the top bit set. Text
|
|
must be UTF-8.
|
|
- **Writing.** Every frame we send is masked with 4 fresh bytes from `/dev/urandom`. A ping is
|
|
answered with a pong carrying the same payload. We send a ping every `ping_every_ms`; no bytes
|
|
at all for `dead_after_ms` means the connection is dead.
|
|
- **Closing.** A close frame is answered with a close frame, then the connection is closed.
|
|
|
|
Any ending of the connection leads to reconnecting (section 9), never to a panic.
|
|
|
|
## 7. Sessions, the allowlist and commands
|
|
|
|
For each `posted` event: parse `data.post` (a JSON string) into `id`, `user_id`, `channel_id`,
|
|
`root_id`, `message`, `create_at`, `type`. Mattermost's JSON is not ours: unknown fields are
|
|
ignored. Then, in this order:
|
|
|
|
1. Ignore it if `user_id` is the bot's own (from `GET /api/v4/users/me` at startup), if `type` is
|
|
not empty (a system message), or if its id was handled already (section 9).
|
|
2. Ignore it, **silently**, unless `user_id` is in `allow.users`. No reply, no typing, no log line
|
|
with the message's text (one line with the user id and post id is enough).
|
|
3. Is it for this Boxmaker? By the channel:
|
|
- A direct message with the bot (`data.channel_type == "D"`): yes, always.
|
|
- A channel or a group message (`"O"`, `"P"`, `"G"`) whose id is in `allow.channels`: yes if
|
|
the message **names this bot**, or if it is a reply in a thread this Boxmaker already has a
|
|
session for (section 9, `threads`) **and names no other user**. Otherwise ignore it,
|
|
silently. `@channel`, `@here` and `@all` name nobody: every agent would answer them.
|
|
- Any other channel: ignore it, silently.
|
|
|
|
"Names" means `@<username>` in the message, compared case-insensitively, where the username is
|
|
followed by the end of the message or a character that cannot be part of a username (anything
|
|
but `a-z`, `0-9`, `.`, `-`, `_`), and a trailing `.` is not part of it. This bot's username
|
|
comes from `GET /api/v4/users/me`. "Names another user" is any other `@<name>` of that shape
|
|
except `channel`, `here` and `all`.
|
|
|
|
Examples in a channel shared with an agent called Hermes, with this bot `boxmaker-straylight`:
|
|
|
|
| Post | For Boxmaker? |
|
|
|---|---|
|
|
| `@boxmaker-straylight summarise the audit log` (top level) | yes: a new session rooted here |
|
|
| a reply in that thread: `and the older files?` | yes: its thread, nobody else named |
|
|
| a reply in that thread: `@hermes what do you think?` | no |
|
|
| `@boxmaker-straylight @hermes compare notes` | yes (and Hermes answers too) |
|
|
| `@boxmaker-straylightx hello` | no: a different name |
|
|
| `@channel standup in five` | no |
|
|
|
|
4. The session: its root is `root_id`, or the post's own `id` when `root_id` is empty; the session
|
|
id is `mm-<root>` (29 characters of `[a-z0-9-]`, a valid `SessionId`). A top-level message is a
|
|
new session (`resume: false`); a reply resumes (`resume: true`). A reply in a thread `loopd`
|
|
does not know (a thread that began before Boxmaker) is created on `no_such_session`, as
|
|
`bxctl chat --session` does.
|
|
5. Commands. A message whose first character is `!`:
|
|
- `!!…` is not a command: one `!` is removed and the rest goes on as a message.
|
|
- `!approve …` and `!deny …` are M4b's; in M4a they are answered in the thread with
|
|
"approvals over Mattermost arrive in M4b; use `bxctl approvals`".
|
|
- Anything else is answered with "unknown command; the commands are !approve and !deny".
|
|
- A command is never sent to `loopd`.
|
|
6. Otherwise the message joins its session's queue. A queue already holding `queue` messages
|
|
answers "busy: too many messages are waiting in this conversation" in the thread and drops it.
|
|
|
|
## 8. Turns and delivery
|
|
|
|
One worker per session with a non-empty queue. When the session has no turn running, the worker
|
|
takes **every** message waiting, joins them with a blank line (`"\n\n"`), in the order they were
|
|
posted, and sends one `turn` on `loop.sock`.
|
|
|
|
While the turn runs, `gatewayd` sends `user_typing` over the WebSocket with the channel and the
|
|
root as `parent_id`, every `typing_every_ms`. Turn events are otherwise not shown: no thinking, no
|
|
status. One exception: an `approval_pending` event is posted once in the thread as "waiting for
|
|
approval <id>: approve or deny it with `bxctl` (Mattermost approvals arrive in M4b)".
|
|
|
|
When the turn ends:
|
|
|
|
- `turn_done` → the content is posted in the thread. Over 16,000 characters, it is split into
|
|
several replies, at the last newline before the limit (or at a character boundary if there is
|
|
none), in order.
|
|
- An `error` frame → "Error: <code>: <detail>" in the thread, with the runbook pointer when the
|
|
detail carries one.
|
|
- `loop.sock` cannot be reached, or closes early → "Boxmaker's loop is not running
|
|
(see docs/runbook.md#loop-unavailable)" in the thread, and the messages are dropped, not retried.
|
|
|
|
Then, if more messages arrived during the turn, the worker starts the next turn with them.
|
|
|
|
Several sessions can run turns at once; `loopd` serialises them at the model.
|
|
|
|
## 9. State, reconnecting and restarts
|
|
|
|
`<home>/gateway/state.json`, written atomically (write, sync, rename, sync the directory) after
|
|
every change:
|
|
|
|
```json
|
|
{"channels": {"<channel id>": 1758650000000}, "recent": ["<post id>", …],
|
|
"threads": ["<root post id>", …],
|
|
"in_flight": [{"session": "mm-…", "channel": "<id>", "root": "<id>"}]}
|
|
```
|
|
|
|
- `channels`: the `create_at` of the last post handled in each channel.
|
|
- `recent`: the ids of the last 500 posts handled, so a post seen twice (live and in a catch-up) is
|
|
handled once.
|
|
- `threads`: the roots of the threads this Boxmaker has a session for, in channels and group
|
|
messages (direct messages need none). Added when a naming post starts or joins a thread; the
|
|
newest 5,000 are kept.
|
|
- `in_flight`: turns sent to `loopd` and not yet answered.
|
|
|
|
A file that exists but cannot be read or parsed stops `gatewayd`, with
|
|
`see docs/runbook.md#gateway-state-damaged`; a missing file is a first start.
|
|
|
|
**Connecting** (at start and after every loss): `GET /users/me`; open the WebSocket; wait for
|
|
`hello`. Then **catch up**: for every channel in `channels`, and the direct channel with every
|
|
allowlisted user (`POST /channels/direct`), and every channel in `allow.channels`, fetch the posts since the last one handled
|
|
(`GET /channels/{id}/posts?since=…`) and handle them in `create_at` order, before any live event.
|
|
On a first start, `channels` is empty and nothing is caught up: history is not answered.
|
|
|
|
**Losing the connection**: reconnect after 1, 2, 5, 10, then every 30 seconds, printing one line
|
|
per attempt. A REST call answered 429 waits until `X-Ratelimit-Reset`; a 5xx is retried twice; a
|
|
401 or 403 stops `gatewayd` with `see docs/runbook.md#mattermost-auth-failed`, since retrying with
|
|
a wrong token would not help.
|
|
|
|
**Restarting**: for every entry left in `in_flight`, `gatewayd` posts "interrupted: gatewayd
|
|
restarted before the answer arrived; ask again" in its thread and removes it. `loopd` finished the
|
|
turn and logged it; the answer is in the session log.
|
|
|
|
## 10. Runbook
|
|
|
|
New entries: `secret-unavailable`, `secret-in-a-file` (the warning), `mattermost-unreachable` (connecting keeps failing),
|
|
`mattermost-auth-failed`, `gateway-state-damaged`, `loop-unavailable`, `gatewayd-start-failed`
|
|
(configuration). Every message for these states ends with its pointer.
|
|
|
|
## 11. Testing
|
|
|
|
Offline, as tasks with given tests, against fakes:
|
|
|
|
- **`proto::sha1` and base64**: FIPS 180 vectors and RFC 4648 vectors; lengths 55, 56, 63, 64 and
|
|
65; each also fed in pieces.
|
|
- **WebSocket**: the handshake (the exact request; RFC 6455's example key and accept; refusal of a
|
|
wrong accept, a status other than 101, a missing `Upgrade`, oversized headers). Frames: seeded
|
|
random frame streams checked against a deliberately naive decoder in the test; every hostile
|
|
frame in section 6, each ending the connection cleanly with no panic; masking of our frames;
|
|
ping and pong; fragments reassembled; the close handshake; a server that trickles one byte at a
|
|
time, and one that goes silent.
|
|
- **TLS**: a test-only CA and server certificate generated once with `openssl` and committed as
|
|
fixtures (long expiry, labelled test keys); the client reaches a local TLS server through
|
|
`ca_file`; an unknown CA and a name mismatch are refused.
|
|
- **HTTP**: `Content-Length`, chunked, the size caps, 429 with `X-Ratelimit-Reset`.
|
|
- **Secrets**: each backend, each refusal in section 4's table, the warning for a file secret (and
|
|
none for the other two), and that `Debug` shows no value.
|
|
- **The gateway, against a fake Mattermost** (test support, with a reference implementation): a
|
|
scripted HTTP and WebSocket server that records every request and pushes events; and a fake
|
|
`loopd` on a Unix socket. Every rule of sections 7 to 9: the allowlist (nothing at all for anyone
|
|
else, and nothing for channels not allowed), every row of the naming table and the naming rule's
|
|
edges (case, trailing punctuation, a longer name that starts with this one, `@channel`), a reply
|
|
in a joined thread with and without another name, own posts ignored, sessions from roots, the burst
|
|
joined into one turn, the queue limit, typing sent while a turn runs and stopped after, splitting
|
|
a long answer, commands and `!!`, catch-up after a reconnect, `recent` preventing a second
|
|
answer, the interrupted reply after a restart, and each runbook pointer.
|
|
|
|
On straylight, by the design model, once the owner has a bot account and a second Mattermost user:
|
|
a direct message is answered in its thread while Boxmaker shows as typing; the second user gets
|
|
nothing; in an allowed channel with another agent, Boxmaker answers only when named or in its own
|
|
thread; a burst is one turn; messages sent while `gatewayd` was stopped are answered when it
|
|
starts; a restart mid-turn gives the interrupted reply; `ss -ltnp` shows no port for `gatewayd`;
|
|
the token comes from a systemd credential.
|
|
|
|
## 12. M4b in summary
|
|
|
|
Agreed on 2026-09-23, to be specified after M4a is built:
|
|
|
|
- `brokerd` opens a connection to `gateway.sock` (a Unix socket `gatewayd` listens on) for each
|
|
approval, with the approval's id, tool, arguments, grant, taint and expiry. `gatewayd` posts it,
|
|
answers with the post's id, and keeps the connection open until the owner answers; the final
|
|
frame is the answer (approved or refused, the Mattermost user id, the post id, a reason). If
|
|
`brokerd` closes the connection (answered in `bxctl`, expired, requester gone, restarted), the
|
|
post is edited to "no longer pending". `gatewayd` never reaches `admin.sock`.
|
|
- The approval is posted in the asking session's thread when it is a Mattermost session, otherwise
|
|
in the owner's direct messages. Arguments escaped as in `bxctl`, in a code fence longer than any
|
|
run of backticks inside them.
|
|
- An answer counts only if it is from an allowlisted user, after the approval post, while pending,
|
|
and either a ✅ or ❌ reaction on the approval post itself, or a reply in its thread of exactly
|
|
`!approve <id>` or `!deny <id> [reason]`.
|
|
- Arguments too long for one post are split into up to 4 numbered parts, the last repeating the id,
|
|
the count and the SHA-256 of the whole; only an answer on the last part counts. Longer than that:
|
|
one post with the size and the hash, answerable only in `bxctl`.
|
|
- `bxctl` gains `deny` as another name for `refuse`.
|
|
|
|
## 13. Threat model notes
|
|
|
|
- `gatewayd` holds the bot token. With the environment or file backend the token is in plaintext
|
|
(in `/proc/<pid>/environ` to the same user, or on disk under an owner-only mode); the credential
|
|
backend keeps it encrypted at rest. The owner chooses per deployment (P15).
|
|
- Only allowlisted user ids are heard. Mattermost and the tailnet are trusted (decision of
|
|
2026-09-17), so a post's `user_id` is believed.
|
|
- The model's answer is posted as it is. Mattermost renders markdown; a hostile answer can format
|
|
itself, but reaches only the owner, who asked.
|
|
- M4b gives `gatewayd` the power to answer pending approvals, as `bxctl` has; it cannot create
|
|
one, widen a grant or read one.
|
|
- M7 must let `gatewayd`'s container reach the tailnet name (and, if `url` is loopback HTTP, the
|
|
host's loopback).
|