kyle/boxmaker
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Plan M4a: gatewayd in 15 tasks, with skeletons and given tests

Browse Source 
Each task's tests were run against a reference at its end state; the end states were replayed
from master in order with the gate at each step (650 to 762 tests); each skeleton compiles
against its tests and fails them. The reference is kept off this machine. Lessons T27 (every
wait in a test has a limit) and T28 (mutate the reference before hand-over) come from this work.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
kyle
2026-09-23 19:05:44 -07:00
co-authored by Claude Opus 5.5
parent f38dc8d474
commit 0339dc13b2
69 changed files with 7790 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/implementer-lessons.md
+3
View File
@@ -62,6 +62,9 @@ How it is used:
| T25 | Size a task by the largest function the model must hold in one turn, not by the task. Ornith writes one function with a few branches well; a function with half a dozen branches and threads (M3b's `run`) it plans in its head until the turn runs out, with nothing written. Give such a task a compiling skeleton with the big function already written as glue over small `todo!()` helpers, and say to fill one at a time with `cargo check` between. | M3b task 11: four sessions. Two wrote nothing; a whole-file skeleton got five of six functions; the finer skeleton finished it in ten minutes, and tasks 12 and 13 followed without a stop. |
| T26 | Replay each task's end state on its own, **and** read the task file against the reference for anything the reference has that the task does not ask for. The replay proves the tests can pass; only the reading finds a field the reference reads through a getter the task never mentions. | M3b task 11: the task stored `egress_dir` without the reference's getter, so the field was never read and clippy failed; the replay passed because the reference had the getter. |
| T27 | Every wait in a given test has a limit (`recv_timeout`, a deadline loop, `is_finished` before `join`). A test that waits forever on a broken implementation hangs the driver instead of failing, and the implementer cannot tell a hang from slowness. | M4a planning: breaking the reference's ping made `ws_conn` hang on `rx.recv()`; retrying a refused token made the serve test hang on `join()`. Both now fail within 5 s. |
| T28 | Before hand-over, break the reference on purpose, one line at a time (drop a check, move a bound by one, skip a save), and run the given tests against each change. A change the tests still pass is a missing test, unless it cannot change behaviour. Rustfmt reflows lines: match on text that survives formatting. | M4a planning: 69 changes over six modules; the tests missed 8. Five were real gaps and got tests (a reconnect that re-sent "interrupted" for a running turn was one), two could not change behaviour, and one was left (an event frame with another id, which `loopd` never sends). |
## What worked and should be kept
- Byte-exact fixtures, compared in both directions. No wire-format defect reached review.
docs/plans/M4a/01-proto-sha1.md
+60
View File
@@ -0,0 +1,60 @@
# M4a task 01: SHA-1 in `proto`
**Branch:** `m4a` (run `git switch m4a`; `git status --short` must be empty, otherwise stop)
**Commit subject:** `proto: sha1, for the WebSocket handshake check`
## Goal
`gatewayd` speaks WebSocket to Mattermost. The server proves it understood the handshake by
sending back the SHA-1 of our key (RFC 6455, section 4.2.2), so we need SHA-1. It lives in
`proto`, which has no dependencies of its own. It is used for that check and nothing else: it must
never protect anything.
## Files
- Copy: `crates/proto/tests/sha1.rs`, and the skeleton `crates/proto/src/sha1.rs`
- Modify: `crates/proto/src/lib.rs` (`pub mod sha1;`), `docs/implementer-log.md`
## Interface (in the skeleton)
```rust
pub fn sha1(data: &[u8]) -> [u8; 20]; // written: new, update, finish
pub struct Sha1 { state: [u32; 5], block: [u8; 64], filled: usize, length: u64 }
impl Sha1 {
pub fn new() -> Sha1; // written: the five initial words
pub fn update(&mut self, data: &[u8]); // todo
pub fn finish(self) -> [u8; 20]; // todo
fn compress(&mut self, block: &[u8; 64]); // todo
}
```
`length` counts **bits** fed so far. Each `todo!()` has its steps above it. No indexing that can go
out of bounds: `w[i]` on the fixed `[u32; 80]` inside `compress` is fine; slicing `data` or `block`
by a computed range is not (use `get`, `get_mut`, `split_at`). Use `as_chunks::<4>()` to read
big-endian words.
## Steps
- [ ] **1. Copy.** `git switch m4a`, then
`cp docs/plans/M4a/files/crates/proto/tests/sha1.rs crates/proto/tests/` and
`cp docs/plans/M4a/files/crates/proto/src/sha1.rs crates/proto/src/`. Add `pub mod sha1;` to
`crates/proto/src/lib.rs`.
- [ ] **2. See it fail.** `cargo test -p proto --test sha1`. Expected: it compiles, and 4 tests
fail with `not yet implemented`.
- [ ] **3. Fill the three functions**, one at a time: `compress`, then `update`, then `finish`. Run
`cargo check -p proto` after each.
- [ ] **4. See it pass.** `cargo test -p proto --test sha1`. Expected: 4 passed. The tests include
FIPS 180 and RFC 3174 vectors, a million `a`s, the lengths around the 64-byte block where the
padding changes shape (checked with `sha1sum`), and the RFC 6455 handshake example; every case is
also fed in pieces.
- [ ] **5. Run the gate.** `cargo fmt --all`, then `make gate`. Expected last line: `gate: ok`.
- [ ] **6. Log and commit.** `git add crates/proto docs/implementer-log.md Cargo.lock && git commit`
## Done when
- `cargo test -p proto --test sha1` passes; `make gate` prints `gate: ok`.
## Stop and report if
- A test vector fails and you cannot find why after checking `compress` against the comment twice.
docs/plans/M4a/02-gatewayd-deps.md
+86
View File
@@ -0,0 +1,86 @@
# M4a task 02: `gatewayd`'s dependencies and test certificates
**Branch:** `m4a` (run `git switch m4a`; `git status --short` must be empty, otherwise stop)
**Commit subject:** `gatewayd: dependencies for TLS and secrets; test-only certificates`
## Goal
`gatewayd` talks to Mattermost over TLS when the URL is `https` (the owner's decision: the tailnet
serves it that way). This task adds the dependencies it needs and nothing that uses them yet, so
that the next tasks can. It also copies in the test-only certificates the TLS tests use.
## Files
- Copy: `deny.toml` (replaces the old one), `crates/gatewayd/tests/fixtures/tls/` (the whole
directory: 9 files)
- Modify: `Cargo.toml`, `crates/gatewayd/Cargo.toml`, `crates/gatewayd/src/lib.rs`,
`docs/dependencies.md`, `docs/implementer-log.md`, `Cargo.lock`
## The changes, exactly
**`Cargo.toml`**, at the end of `[workspace.dependencies]`, after `emsha`:
```toml
rustls = { version = "0.23.45", default-features = false, features = ["ring", "std", "tls12"] }
rustls-native-certs = "0.8.4"
zeroize = "1.9.0"
```
**`crates/gatewayd/Cargo.toml`**, `[dependencies]` becomes:
```toml
[dependencies]
proto.workspace = true
rustls.workspace = true
rustls-native-certs.workspace = true
serde.workspace = true
serde_json.workspace = true
toml.workspace = true
zeroize.workspace = true
```
**`crates/gatewayd/src/lib.rs`**: replace its one comment line with
```rust
//! `gatewayd`: the Mattermost channel. It carries the owner's messages to `loopd` as turns and posts
//! the answers back, over outbound connections only. M4a spec: `docs/specs/2026-09-23-m4a-gateway.md`.
```
**`deny.toml`**: copied, not edited. It allows the licences ISC and BSD-3-Clause (owner's approval,
2026-09-23), and judges dependencies only for Linux and the Mac, since `rustls` pulls in a second
`windows-sys` for Windows that we never build.
**`docs/dependencies.md`**: in the table, the rows for `serde` and `serde_json` add `` `gatewayd` ``
to "Used by", the `toml` row adds `` `gatewayd` from M4a ``, and three rows are added at the end:
```markdown
| `rustls` | 0.23.45 | `gatewayd` | TLS to Mattermost over the tailnet (M4a, owner's decision). `default-features = false` with `ring`, `std`, `tls12`. With its dependencies on Linux: `ring` (Apache-2.0 AND ISC; builds C and assembly), `rustls-webpki` and `untrusted` (ISC), `rustls-pki-types`, `subtle` (BSD-3-Clause), `zeroize`, `once_cell`, `getrandom`, `libc`, `cfg-if`. Apache-2.0 OR ISC OR MIT. |
| `rustls-native-certs` | 0.8.4 | `gatewayd` | The host's trusted certificates, so a CA installed on the host is trusted too; adds `openssl-probe`. Apache-2.0 OR ISC OR MIT. |
| `zeroize` | 1.9.0 | `gatewayd` | Wipes a secret's memory when it is dropped (`Secret`). Already a dependency of `rustls`. Apache-2.0 OR MIT. |
```
The fixtures' `README.md` says what each certificate is. Every key in that directory is public and
for these tests only.
## Steps
- [ ] **1. Copy.** `git switch m4a`, then `cp docs/plans/M4a/files/deny.toml deny.toml` and
`mkdir -p crates/gatewayd/tests/fixtures && cp -r docs/plans/M4a/files/crates/gatewayd/tests/fixtures/tls crates/gatewayd/tests/fixtures/`
- [ ] **2. Make the four edits above.**
- [ ] **3. Build once.** `cargo build -p gatewayd`, so that `Cargo.lock` is updated. Expected: it
builds (the crates are already in the local cache; if cargo says it needs the network and cannot
reach it, stop and report).
- [ ] **4. Check.** `git status --short` lists `Cargo.lock`, `Cargo.toml`, `deny.toml`, the two
`gatewayd` files, `docs/dependencies.md` and the fixtures directory, and nothing else.
- [ ] **5. Run the gate.** `cargo fmt --all`, then `make gate`. Expected last line: `gate: ok`.
`cargo-deny` must pass: if it reports a licence or a duplicate, stop and report it word for word.
- [ ] **6. Log and commit.**
`git add Cargo.toml Cargo.lock deny.toml crates/gatewayd docs/dependencies.md docs/implementer-log.md && git commit`
## Done when
- `make gate` prints `gate: ok` with the new dependencies in `Cargo.lock`.
## Stop and report if
- `cargo-deny` fails, or cargo wants a version other than the ones above.
docs/plans/M4a/03-gatewayd-config.md
+95
View File
@@ -0,0 +1,95 @@
# M4a task 03: `gatewayd.toml`
**Branch:** `m4a` (run `git switch m4a`; `git status --short` must be empty, otherwise stop)
**Commit subject:** `gatewayd: config, gatewayd.toml into a checked Config`
## Goal
`gatewayd.toml` says where Mattermost is, where the token comes from, who may talk to Boxmaker and
in which channels, and a few limits. It is **our** format: an unknown key anywhere is an error, and
a value that is wrong is refused at start with a message that names it. Spec section 3.
```toml
[mattermost]
url = "https://straylight.scylla-hammerhead.ts.net" # https or http; no path
ca_file = "/etc/boxmaker/extra-ca.pem" # optional; added to the system's roots
[secrets.mattermost_token]
credential = "mattermost-token" # exactly one of: credential, env, file
[allow]
users = ["abcdefghijklmnopqrstuvwxyz"] # Mattermost user ids (26 characters)
channels = [] # channels and group messages allowed, by id; default none
[loop]
socket = "" # empty: <home>/run/loop/loop.sock
[paths]
home = "/var/lib/boxmaker" # default: BOXMAKER_HOME, then /var/lib/boxmaker
[limits]
queue = 20
typing_every_ms = 3000
ping_every_ms = 30000
dead_after_ms = 60000
```
## Files
- Copy: `crates/gatewayd/tests/config.rs`, `crates/gatewayd/tests/support/tmp.rs`, and the
skeleton `crates/gatewayd/src/config.rs`
- Modify: `crates/gatewayd/src/lib.rs` (`pub mod config;`), `docs/implementer-log.md`
## The skeleton
Every type is written, with its serde attributes: `Config`, `MattermostConfig`, `SecretSpec`,
`SecretSource`, `AllowConfig`, `LoopConfig`, `Paths`, `Limits` (with their defaults), `ServerUrl`,
`ConfigError`, and the constant `MATTERMOST_TOKEN = "mattermost_token"`. Do not change them. The
functions are `todo!()`, each with its steps above it.
## The messages, exactly
`Config::problem` returns the first problem, in this order:
| Problem | Message |
|---|---|
| `url` fails `parse_url` | `parse_url`'s message: `[mattermost] url "<url>" must be http:// or https://, a host, an optional port, and nothing else` (the url with `{:?}`) |
| `ca_file` not absolute | `[mattermost] ca_file "<path>" must be an absolute path` (`{:?}`) |
| no `[secrets.mattermost_token]` | `[secrets.mattermost_token] is missing` |
| a secret whose `source()` fails | `[secrets.<name>] <source's message>` |
| `allow.users` empty | `[allow] users is empty: a gateway that answers nobody is a mistake` |
| an id in `allow.users` or `allow.channels` that is not `valid_id` | `[allow] "<id>" is not a Mattermost id (26 characters of a-z and 0-9)` (`{:?}`) |
| a limit that is 0 (checked in the order queue, typing_every_ms, ping_every_ms, dead_after_ms) | `[limits] <name> must be at least 1` |
`SecretSpec::source`'s messages:
| Case | Message |
|---|---|
| not exactly one of the three | `needs exactly one of credential, env and file` |
| `credential` empty or with a byte that is not an ASCII letter, digit, `_`, `.` or `-` | `credential "<name>" is not a credential name (letters, digits, _ . -)` |
| `env` empty or with a byte that is not `A-Z`, `0-9` or `_` | `env "<var>" is not a variable name (A-Z, 0-9, _)` |
| `file` not absolute | `file "<path>" must be an absolute path` |
`Config::token_source` gives `source()` of `mattermost_token`, or `[secrets.mattermost_token] is
missing`. `ConfigError`'s `Display` is `<path>: <error or message>`.
## Steps
- [ ] **1. Copy.** `git switch m4a`, then
`mkdir -p crates/gatewayd/tests/support && cp docs/plans/M4a/files/crates/gatewayd/tests/config.rs crates/gatewayd/tests/ && cp docs/plans/M4a/files/crates/gatewayd/tests/support/tmp.rs crates/gatewayd/tests/support/ && cp docs/plans/M4a/files/crates/gatewayd/src/config.rs crates/gatewayd/src/`.
Add `pub mod config;` to `crates/gatewayd/src/lib.rs`.
- [ ] **2. See it fail.** `cargo test -p gatewayd --test config`. Expected: it compiles and 7 tests
fail with `not yet implemented`.
- [ ] **3. Fill the functions one at a time**: `valid_id`, `parse_url`, `SecretSpec::source`,
`ConfigError`'s `fmt`, then the `Config` methods. `cargo check -p gatewayd` after each.
- [ ] **4. See it pass.** `cargo test -p gatewayd --test config`. Expected: 7 passed.
- [ ] **5. Run the gate.** `cargo fmt --all`, then `make gate`. Expected last line: `gate: ok`.
- [ ] **6. Log and commit.** `git add crates/gatewayd docs/implementer-log.md Cargo.lock && git commit`
## Done when
- `cargo test -p gatewayd --test config` passes; `make gate` prints `gate: ok`.
## Stop and report if
- A test wants a type changed. The types are the format; report instead.
docs/plans/M4a/04-gatewayd-secrets.md
+85
View File
@@ -0,0 +1,85 @@
# M4a task 04: secrets, and the runbook entries for `gatewayd`
**Branch:** `m4a` (run `git switch m4a`; `git status --short` must be empty, otherwise stop)
**Commit subject:** `gatewayd: secrets from a credential, the environment or a file; runbook entries`
## Goal
`gatewayd`'s one secret is the Mattermost token. It can come from three places, chosen per secret
in `gatewayd.toml` (spec section 4, P15 in the brief):
| Form | Reads | Refused when |
|---|---|---|
| `credential = "<name>"` | `$CREDENTIALS_DIRECTORY/<name>`, which systemd fills for a service with `LoadCredentialEncrypted=` | the variable is unset, or the file cannot be read |
| `env = "<VAR>"` | the environment variable | unset, or not UTF-8 |
| `file = "<absolute path>"` | the file | not absolute; a symbolic link; not a regular file; not owned by the user `gatewayd` runs as; any group or other permission bit set |
In every form one trailing newline is removed, and an empty value is refused. A secret read from a
**file** is in plaintext on disk: `gatewayd` still starts, but `load` returns this warning for it,
exactly (`<name>` and `<path>` filled in, `path.display()`):
```text
gatewayd: warning: secret <name> is read in plaintext from <path>; a systemd credential keeps it encrypted at rest (see docs/runbook.md#secret-in-a-file)
```
The value must never reach a log: `Secret` has no `Display`, its `Debug` prints `Secret(…)`, it is
wiped when dropped (`zeroize::Zeroizing`), and `expose()` is the only way to the text.
This task also adds the runbook entries for every fail-closed state of `gatewayd`, all seven at
once, so that later tasks can point at them.
## Files
- Copy: `crates/gatewayd/tests/secrets.rs`, and the skeleton `crates/gatewayd/src/secrets.rs`
- Append: `docs/plans/M4a/files/runbook-gatewayd.md` to the end of `docs/runbook.md`
- Modify: `crates/gatewayd/src/lib.rs` (`pub mod secrets;`), `docs/implementer-log.md`
## The skeleton
Written: `Secret` (with `new`, `expose` and its `Debug`), `Loaded { secret, warning }`,
`SecretError { name, why }` and its `Display` (`secret <name>: <why>` then a newline and
`see docs/runbook.md#secret-unavailable`), and the constants `RUNBOOK` and `RUNBOOK_FILE`. To fill:
`load`, `check_file` and `value`.
`load` takes the environment as a function, `env: &dyn Fn(&str) -> Option<OsString>`, so that tests
can pass their own: read `CREDENTIALS_DIRECTORY` and `env` variables through it, never through
`std::env`. The uid of the user `gatewayd` runs as is the owner of `/proc/self`
(`std::fs::metadata("/proc/self")?.uid()`, from `std::os::unix::fs::MetadataExt`); no `unsafe`, no
`libc`.
The `why` of each error, for the reader: say what is wrong and never the value. The reference used
these, and you may too:
| Case | `why` |
|---|---|
| credential, variable unset | `CREDENTIALS_DIRECTORY is not set: gatewayd was not started by systemd with LoadCredentialEncrypted=` |
| credential, unreadable | `cannot read the credential <path>: <error>` |
| env unset | `the environment variable <VAR> is not set` |
| env not UTF-8 | `the environment variable <VAR> is not UTF-8` |
| file checks | `<path> is not an absolute path`, `cannot read <path>: <error>`, `<path> is a symbolic link`, `<path> is not a regular file`, `<path> is not owned by the user gatewayd runs as`, `<path> has mode <mode as {:03o}>; only the owner may read it (0600 or 0400)` |
| the value | `the value is not UTF-8`, `the value is empty` |
## Steps
- [ ] **1. Copy.** `git switch m4a`, then
`cp docs/plans/M4a/files/crates/gatewayd/tests/secrets.rs crates/gatewayd/tests/ && cp docs/plans/M4a/files/crates/gatewayd/src/secrets.rs crates/gatewayd/src/`
and `cat docs/plans/M4a/files/runbook-gatewayd.md >> docs/runbook.md`. Add `pub mod secrets;` to
`crates/gatewayd/src/lib.rs`.
- [ ] **2. See it fail.** `cargo test -p gatewayd --test secrets`. Expected: it compiles and 8
tests fail.
- [ ] **3. Fill `value`, then `check_file`, then `load`.** `cargo check -p gatewayd` after each.
- [ ] **4. See it pass.** `cargo test -p gatewayd --test secrets`. Expected: 8 passed.
- [ ] **5. Check the runbook.** `grep -c '^## ' docs/runbook.md` is 7 more than before step 1, and
`sh scripts/check-runbook.sh` prints nothing and exits 0.
- [ ] **6. Run the gate.** `cargo fmt --all`, then `make gate`. Expected last line: `gate: ok`.
- [ ] **7. Log and commit.**
`git add crates/gatewayd docs/runbook.md docs/implementer-log.md Cargo.lock && git commit`
## Done when
- `cargo test -p gatewayd --test secrets` passes; the seven entries are in `docs/runbook.md`;
`make gate` prints `gate: ok`.
## Stop and report if
- A test needs the secret's value in an error message, or `unsafe` looks necessary.
docs/plans/M4a/05-gatewayd-net.md
+63
View File
@@ -0,0 +1,63 @@
# M4a task 05: a connection, plain or TLS
**Branch:** `m4a` (run `git switch m4a`; `git status --short` must be empty, otherwise stop)
**Commit subject:** `gatewayd: net, TCP or verified TLS to the Mattermost server`
## Goal
Every REST call and the WebSocket run over one kind of stream: plain TCP for an `http` URL, or TCP
wrapped in TLS through `rustls` for `https`. TLS verifies the server's name against the host in the
URL and its chain against the host's trusted certificates (`rustls-native-certs`) plus `ca_file` if
given. **Nothing can turn verification off.** Spec section 5.
## Files
- Copy: `crates/gatewayd/tests/net.rs`, `crates/gatewayd/tests/support/tls_server.rs`, and the
skeleton `crates/gatewayd/src/net.rs`
- Modify: `crates/gatewayd/src/lib.rs` (`pub mod net;`), `crates/gatewayd/Cargo.toml` (below),
`docs/implementer-log.md`
The tests' TLS server needs `rustls` too. Add to the end of `crates/gatewayd/Cargo.toml`:
```toml
[dev-dependencies]
rustls.workspace = true
```
## The skeleton
Written, because they are mostly `rustls` API (checked against rustls 0.23.45):
`client_config(ca_file)`, which builds the root store and the `ClientConfig` with the `ring`
provider, and `Connector::connect`, which connects to each address in turn, sets the read and
write timeouts, and for TLS runs the handshake to its end so that a bad certificate is an error
from `connect` and not from a later read. Read both before you start.
Also written: `NetError` (`Roots`, `Connect`, `Tls`) and its `Display`, and the types `Stream`
(`Plain(TcpStream)` or `Tls(Box<StreamOwned<ClientConnection, TcpStream>>)`) and `Connector`.
To fill: `Stream::tcp`, `Stream::set_read_timeout`, `Read` and `Write` for `Stream` (forward to
the inner stream in each variant), `Connector::new` and `Connector::server`.
## Steps
- [ ] **1. Copy.** `git switch m4a`, then
`cp docs/plans/M4a/files/crates/gatewayd/tests/net.rs crates/gatewayd/tests/ && cp docs/plans/M4a/files/crates/gatewayd/tests/support/tls_server.rs crates/gatewayd/tests/support/ && cp docs/plans/M4a/files/crates/gatewayd/src/net.rs crates/gatewayd/src/`.
Add `pub mod net;` to `lib.rs` and the `[dev-dependencies]` above.
- [ ] **2. See it fail.** `cargo test -p gatewayd --test net`. Expected: it compiles and 7 tests
fail.
- [ ] **3. Fill the functions**, `cargo check -p gatewayd` after each.
- [ ] **4. See it pass.** `cargo test -p gatewayd --test net`. Expected: 7 passed: plain TCP; TLS
with the test CA; an unknown CA and a wrong name refused at connect; TLS to a plain server fails
without hanging; a bad `ca_file` refused before any connection; nobody listening.
- [ ] **5. Run the gate.** `cargo fmt --all`, then `make gate`. Expected last line: `gate: ok`.
- [ ] **6. Log and commit.** `git add crates/gatewayd docs/implementer-log.md Cargo.lock && git commit`
## Done when
- `cargo test -p gatewayd --test net` passes; `make gate` prints `gate: ok`.
## Stop and report if
- A test passes only with verification weakened in any way (a custom verifier, a skipped name
check). That is never the fix.
docs/plans/M4a/06-gatewayd-http.md
+57
View File
@@ -0,0 +1,57 @@
# M4a task 06: HTTP/1.1 requests
**Branch:** `m4a` (run `git switch m4a`; `git status --short` must be empty, otherwise stop)
**Commit subject:** `gatewayd: http, requests over a stream with size caps and rate-limit waits`
## Goal
Mattermost's REST calls are HTTP/1.1, one connection per request (`Connection: close`), over a
`net::Stream`. This module writes a request and reads the response. The server is not trusted: a
response head over 16 KiB or a body over 4 MiB is an error, and every limit is checked **before**
reading or allocating. The WebSocket handshake (task 07) reuses `read_head`, which therefore must
not read a byte past the blank line that ends the head: the WebSocket's first frame may follow at
once.
## Files
- Copy: `crates/gatewayd/tests/http.rs`, and the skeleton `crates/gatewayd/src/http.rs`
- Modify: `crates/gatewayd/src/lib.rs` (`pub mod http;`), `docs/implementer-log.md`
## The skeleton
Written: the constants `MAX_HEAD` (16 KiB), `MAX_BODY` (4 MiB) and `MAX_RATE_WAIT` (60 s);
`HttpError` (`Io`, `Protocol(String)`, `TooLarge(&'static str)`) with `Display` and
`From<std::io::Error>`; `Head { status, headers }` and `Response { head, body }`.
To fill, each with its steps above it: `Head::header`, `write_request`, `request`, `read_head`,
`read_body`, `read_line`, `read_chunked`, `rate_limit_wait`.
Rules that apply to all of them:
- `read_head` reads **one byte at a time**. That is slow and it is right: it must stop exactly at
the end of the head.
- A retry on `ErrorKind::Interrupted`, and nowhere else.
- No subtraction that can wrap and no indexing that can go out of bounds: `saturating_sub`, `get`,
`get_mut`. `raw.truncate(raw.len() - 2)` is fine only right after checking `raw.ends_with(b"\r\n")`.
- `rate_limit_wait` is about Mattermost's `X-Ratelimit-Reset`, which is a Unix time in seconds; the
test also gives it a small number of seconds, which some servers send.
## Steps
- [ ] **1. Copy.** `git switch m4a`, then
`cp docs/plans/M4a/files/crates/gatewayd/tests/http.rs crates/gatewayd/tests/ && cp docs/plans/M4a/files/crates/gatewayd/src/http.rs crates/gatewayd/src/`.
Add `pub mod http;` to `lib.rs`.
- [ ] **2. See it fail.** `cargo test -p gatewayd --test http`. Expected: it compiles and 6 tests
fail.
- [ ] **3. Fill the functions in the order above**, `cargo check -p gatewayd` after each.
- [ ] **4. See it pass.** `cargo test -p gatewayd --test http`. Expected: 6 passed.
- [ ] **5. Run the gate.** `cargo fmt --all`, then `make gate`. Expected last line: `gate: ok`.
- [ ] **6. Log and commit.** `git add crates/gatewayd docs/implementer-log.md Cargo.lock && git commit`
## Done when
- `cargo test -p gatewayd --test http` passes; `make gate` prints `gate: ok`.
## Stop and report if
- A test needs `read_head` to read ahead into a buffer.
docs/plans/M4a/07-gatewayd-ws-handshake.md
+63
View File
@@ -0,0 +1,63 @@
# M4a task 07: the WebSocket handshake
**Branch:** `m4a` (run `git switch m4a`; `git status --short` must be empty, otherwise stop)
**Commit subject:** `gatewayd: ws, the WebSocket error type and the opening handshake`
## Goal
Mattermost pushes events over a WebSocket at `/api/v4/websocket` (RFC 6455). Our client is our own:
blocking, one connection, tested against hostile input. This task is its first part: the error
type, base64, and the opening handshake (RFC 6455, section 4.1; spec section 6).
The request, exactly (it is written for you in `request_text`):
```text
GET <path> HTTP/1.1\r\n
Host: <host>\r\n
Upgrade: websocket\r\n
Connection: Upgrade\r\n
Sec-WebSocket-Key: <16 random bytes in base64>\r\n
Sec-WebSocket-Version: 13\r\n
Authorization: Bearer <token>\r\n
\r\n
```
The answer must be status 101, with `Upgrade: websocket`, a `Connection` header holding the token
`upgrade`, and `Sec-WebSocket-Accept` equal to base64(SHA-1(key + the GUID
`258EAFA5-E914-47DA-95CA-C5AB0DC85B11`)). Anything else is an error. SHA-1 is `proto::sha1::sha1`
(task 01); the head is read with `http::read_head` (task 06), which stops at the blank line.
## Files
- Copy: `crates/gatewayd/tests/ws_handshake.rs`, and `crates/gatewayd/src/ws/mod.rs` (complete)
and the skeleton `crates/gatewayd/src/ws/handshake.rs`
- Modify: `crates/gatewayd/src/lib.rs` (`pub mod ws;`), `docs/implementer-log.md`
`ws/mod.rs` is complete: `WsError` (`Handshake`, `Protocol`, `TooLarge`, `Closed`, `Dead`, `Io`)
with `Display` and `From<std::io::Error>`, and `pub mod handshake;`. Tasks 08 and 09 add their
`pub mod` lines to it.
## The skeleton
Written: `GUID`, `ALPHABET`, `request_text`. To fill: `base64`, `accept_for`, `new_key`,
`check_response`, `handshake`. The key's 16 bytes come from `random: &mut dyn Read` (in `gatewayd`,
`/dev/urandom`; in the tests, fixed bytes), so the tests know the key.
## Steps
- [ ] **1. Copy.** `git switch m4a`, then
`mkdir -p crates/gatewayd/src/ws && cp docs/plans/M4a/files/crates/gatewayd/src/ws/mod.rs docs/plans/M4a/files/crates/gatewayd/src/ws/handshake.rs crates/gatewayd/src/ws/ && cp docs/plans/M4a/files/crates/gatewayd/tests/ws_handshake.rs crates/gatewayd/tests/`.
Add `pub mod ws;` to `lib.rs`.
- [ ] **2. See it fail.** `cargo test -p gatewayd --test ws_handshake`. Expected: it compiles; 6
fail and 1 passes (`the_request_is_exactly_this`, since `request_text` is given).
- [ ] **3. Fill `base64`, `accept_for`, `new_key`, `check_response`, `handshake`**, in that order,
`cargo check -p gatewayd` after each.
- [ ] **4. See it pass.** `cargo test -p gatewayd --test ws_handshake`. Expected: 7 passed,
including the RFC 6455 example (`dGhlIHNhbXBsZSBub25jZQ==` gives `s3pPLMBiTxaQ9kYGzzhZRbK+xOo=`)
and a handshake whose first frame arrives in the same packet as the head and is left unread.
- [ ] **5. Run the gate.** `cargo fmt --all`, then `make gate`. Expected last line: `gate: ok`.
- [ ] **6. Log and commit.** `git add crates/gatewayd docs/implementer-log.md Cargo.lock && git commit`
## Done when
- `cargo test -p gatewayd --test ws_handshake` passes; `make gate` prints `gate: ok`.
docs/plans/M4a/08-gatewayd-ws-frames.md
+73
View File
@@ -0,0 +1,73 @@
# M4a task 08: WebSocket frames
**Branch:** `m4a` (run `git switch m4a`; `git status --short` must be empty, otherwise stop)
**Commit subject:** `gatewayd: ws frames, a strict decoder and a masked encoder`
## Goal
After the handshake, everything is frames (RFC 6455, section 5). This task is the codec, without
I/O: a `Decoder` that is fed bytes as they arrive, in any pieces, and yields whole messages, and
`encode`, which builds our frames. Everything the server sends is untrusted, so the decoder is
strict and **checks every length before it allocates** (spec section 6):
- A frame from the server that is masked, sets a reserved bit, uses an opcode other than
continuation (0), text (1), close (8), ping (9) or pong (10), is a control frame (8 to 10) over
125 bytes or not final, is a continuation with nothing to continue, or a new text frame while a
message is unfinished, is a `Protocol` error.
- A length not in its shortest form (126 for a length under 126; 127 for one that fits in 16
bits), or a 64-bit length with its top bit set, is a `Protocol` error.
- A message over `MAX_MESSAGE` (1 MiB), counted from the length fields of its frames **before**
their payloads arrive, is `TooLarge`.
- Text must be UTF-8 once whole (a character may be split across frames).
- A close frame's payload is empty, or a 2-byte code and a UTF-8 reason; one byte is an error.
Every frame **we** send is final, masked with the 4 bytes we are given.
## Files
- Copy: `crates/gatewayd/tests/ws_frame.rs`, and the skeleton `crates/gatewayd/src/ws/frame.rs`
- Modify: `crates/gatewayd/src/ws/mod.rs` (`pub mod frame;`), `docs/implementer-log.md`
## The skeleton
Written: `MAX_MESSAGE`, the opcode constants, `Incoming` (`Text`, `Ping`, `Pong`,
`Close(Option<u16>, String)`), `Decoder { buf, partial }` with `new` and `feed`, and the private
`Header { fin, opcode, header_len, payload_len }`.
To fill, in this order, each with its steps above it:
1. `Decoder::header(&self) -> Result<Option<Header>, WsError>`: the next frame's header once all of
it is in `buf`, checked against every rule above that does not need the payload. `Ok(None)`
means "wait for more bytes". It only reads `buf`; it removes nothing.
2. `close(payload) -> Result<Incoming, WsError>`.
3. `Decoder::next_message(&mut self)`: uses `header`, and when the whole frame is in `buf`, takes it
out (`drain`) and acts on it. Control frames may come between the frames of a text message and
are returned at once.
4. `encode(opcode, payload, mask) -> Vec<u8>`.
No indexing that can go out of bounds, no `as` casts: read lengths with `get(2..4)` and
`u16::from_be_bytes`, and convert with `usize::try_from` / `u8::try_from`.
## Steps
- [ ] **1. Copy.** `git switch m4a`, then
`cp docs/plans/M4a/files/crates/gatewayd/tests/ws_frame.rs crates/gatewayd/tests/ && cp docs/plans/M4a/files/crates/gatewayd/src/ws/frame.rs crates/gatewayd/src/ws/`.
Add `pub mod frame;` to `crates/gatewayd/src/ws/mod.rs`.
- [ ] **2. See it fail.** `cargo test -p gatewayd --test ws_frame`. Expected: it compiles and 7
tests fail.
- [ ] **3. Fill the four functions**, `cargo check -p gatewayd` after each.
- [ ] **4. See it pass.** `cargo test -p gatewayd --test ws_frame`. Expected: 7 passed. The last
test feeds 300 random streams, half of them with bits flipped, in random pieces, and compares
your decoder with a simple one written inside the test: they must return the same messages, and
both fail or both not.
- [ ] **5. Run the gate.** `cargo fmt --all`, then `make gate`. Expected last line: `gate: ok`.
- [ ] **6. Log and commit.** `git add crates/gatewayd docs/implementer-log.md Cargo.lock && git commit`
## Done when
- `cargo test -p gatewayd --test ws_frame` passes; `make gate` prints `gate: ok`.
## Stop and report if
- The random-stream test fails on a case you believe the naive decoder gets wrong. Report the seed
and the bytes; do not change the test.
docs/plans/M4a/09-gatewayd-ws-conn.md
+71
View File
@@ -0,0 +1,71 @@
# M4a task 09: the WebSocket connection
**Branch:** `m4a` (run `git switch m4a`; `git status --short` must be empty, otherwise stop)
**Commit subject:** `gatewayd: ws conn, messages, pings, closing and a dead peer`
## Goal
The connection that uses tasks 07 and 08: open it, send text, and wait for the next message while
keeping the connection alive (spec section 6):
- A ping from the server is answered with a pong carrying the same payload.
- We send a ping every `ping_every`. No bytes at all for `dead_after` means the peer is dead.
- A close frame is answered with a close frame, and the connection ends (`Closed`); so does the end
of the stream.
- Any protocol error ends the connection. `gatewayd` then reconnects (task 14); nothing panics.
## Files
- Copy: `crates/gatewayd/tests/ws_conn.rs`, `crates/gatewayd/tests/support/ws_server.rs`, and the
skeleton `crates/gatewayd/src/ws/conn.rs`
- Modify: `crates/gatewayd/src/ws/mod.rs` (`pub mod conn;`), `docs/implementer-log.md`
## The skeleton
```rust
pub const PATH: &str = "/api/v4/websocket";
pub struct Timing { pub ping_every: Duration, pub dead_after: Duration } // Clone, Copy
pub struct Ws { stream, decoder, random, timing, last_heard, last_ping }
impl Ws {
pub fn open(connector: &Connector, token: &str, timing: Timing,
mut random: Box<dyn Read + Send>) -> Result<Ws, WsError>;
fn send(&mut self, opcode: u8, payload: &[u8]) -> Result<(), WsError>;
pub fn send_text(&mut self, text: &str) -> Result<(), WsError>;
pub fn poll(&mut self, wait: Duration) -> Result<Option<String>, WsError>;
pub fn close(mut self);
}
pub fn host_header(server: &ServerUrl) -> String;
```
All are `todo!()`, each with its steps above it. `poll` is the one with the most in it: it returns
the next text message, or `None` after about `wait` with none, and does the pinging, the pong
answers and the dead-peer check on the way. Its read timeout is always the time to the **next
thing it must do** (the end of `wait`, the next ping, or the dead-after limit), so a quiet
connection neither spins nor oversleeps. Write it as its comment says, step by step; a helper
function for step 1 is fine.
`random` gives the handshake key and a fresh 4-byte mask for every frame we send (in `gatewayd`,
`/dev/urandom`).
## Steps
- [ ] **1. Copy.** `git switch m4a`, then
`cp docs/plans/M4a/files/crates/gatewayd/tests/ws_conn.rs crates/gatewayd/tests/ && cp docs/plans/M4a/files/crates/gatewayd/tests/support/ws_server.rs crates/gatewayd/tests/support/ && cp docs/plans/M4a/files/crates/gatewayd/src/ws/conn.rs crates/gatewayd/src/ws/`.
Add `pub mod conn;` to `crates/gatewayd/src/ws/mod.rs`.
- [ ] **2. See it fail.** `cargo test -p gatewayd --test ws_conn`. Expected: it compiles and 10
tests fail.
- [ ] **3. Fill `host_header`, `send`, `send_text`, `open`, `close`, then `poll`**,
`cargo check -p gatewayd` after each.
- [ ] **4. See it pass.** `cargo test -p gatewayd --test ws_conn`, five times. Expected: 10 passed
each time, in under a second.
- [ ] **5. Run the gate.** `cargo fmt --all`, then `make gate`. Expected last line: `gate: ok`.
- [ ] **6. Log and commit.** `git add crates/gatewayd docs/implementer-log.md Cargo.lock && git commit`
## Done when
- `cargo test -p gatewayd --test ws_conn` passes five times running; `make gate` prints `gate: ok`.
## Stop and report if
- A test passes only sometimes, or takes seconds.
docs/plans/M4a/10-gatewayd-mm.md
+70
View File
@@ -0,0 +1,70 @@
# M4a task 10: Mattermost's events and REST calls
**Branch:** `m4a` (run `git switch m4a`; `git status --short` must be empty, otherwise stop)
**Commit subject:** `gatewayd: mm, Mattermost's events and the four REST calls, typed`
## Goal
Mattermost's JSON, typed, and the REST calls `gatewayd` makes. This is **another program's
format**: unknown fields are ignored. But every id we keep ends up in a session id or a URL path,
so every id must be a valid Mattermost id (26 characters of `a-z0-9`), or the whole post is refused.
Facts from Mattermost's source at the owner's server version, v11.11.0 (checked by the design
model; the tests use these shapes):
- A WebSocket event is `{"event", "data", "broadcast", "seq"}`. The first is `hello`. `posted`
carries the post as a **JSON string** in `data.post` (a string holding JSON, not an object), and
the channel's type in `data.channel_type`: `D` direct, `G` group, `O` open, `P` private. A reply
to one of our requests has no `event`.
- `user_typing` is a request we send: `{"action": "user_typing", "seq": <n>, "data":
{"channel_id", "parent_id"}}`; it shows the bot as typing in that thread.
- `GET /api/v4/channels/{id}/posts?since=<ms>` returns `{"order": [ids], "posts": {id: post}}`
with the posts **changed** after that time. Only the ids in `order` changed; `posts` also holds
the roots of their threads, which may be old. Edited and deleted posts come back too. The server
takes at most 1000 (`SINCE_LIMIT`); a full answer may have left some out.
- REST errors: 401 or 403 is a refused token, 429 is a rate limit (wait for `X-Ratelimit-Reset`),
5xx is worth trying again.
## Files
- Copy: `crates/gatewayd/tests/mm_json.rs`, `crates/gatewayd/tests/mm_rest.rs`,
`crates/gatewayd/tests/support/http_server.rs`, and the skeletons
`crates/gatewayd/src/mm/mod.rs` and `crates/gatewayd/src/mm/rest.rs`
- Modify: `crates/gatewayd/src/lib.rs` (`pub mod mm;`), `docs/implementer-log.md`
## The skeletons
`mm/mod.rs`, written: `SINCE_LIMIT`; `MmError` (`Net`, `Auth(u16)`, `RateLimited(Duration)`,
`Status(u16, String)`, `Json`) and its `Display`, which **quotes** a server's body (`{:?}`) so that
a newline in it cannot forge a log line; `Me { id, username }`; `Post { id, user_id, channel_id,
root_id, message, create_at, delete_at, kind }` (`kind` is the JSON `type`); `Event` (`Hello`,
`Posted { post, channel_type }`, `Other(String)`); `Since { posts, full }`; and the private serde
shapes `RawEvent` and `PostList`. To fill: `Post::check`, `json`, `parse_event`, `typing`,
`since_list`.
`mm/rest.rs`, written: `RETRY_5XX` (500 ms), `RETRIES` (2), `BODY_KEPT` (200), `Client`. To fill:
`new`, `connector`, `token`, `once`, `call`, `me`, `create_post`, `posts_since`, `direct_channel`.
The token reaches the wire only in `once`, as `Authorization: Bearer <token>`; nowhere else calls
`expose()`.
## Steps
- [ ] **1. Copy.** `git switch m4a`, then
`mkdir -p crates/gatewayd/src/mm && cp docs/plans/M4a/files/crates/gatewayd/src/mm/mod.rs docs/plans/M4a/files/crates/gatewayd/src/mm/rest.rs crates/gatewayd/src/mm/ && cp docs/plans/M4a/files/crates/gatewayd/tests/mm_json.rs docs/plans/M4a/files/crates/gatewayd/tests/mm_rest.rs crates/gatewayd/tests/ && cp docs/plans/M4a/files/crates/gatewayd/tests/support/http_server.rs crates/gatewayd/tests/support/`.
Add `pub mod mm;` to `lib.rs`.
- [ ] **2. See it fail.** `cargo test -p gatewayd --no-fail-fast --test mm_json --test mm_rest`.
Expected: it compiles; `mm_json` 6 fail and 1 passes (the `Display` test), `mm_rest` 10 fail.
- [ ] **3. Fill `mm/mod.rs`, then `mm/rest.rs`**, one function at a time, `cargo check -p gatewayd`
after each.
- [ ] **4. See it pass.** `cargo test -p gatewayd --test mm_json --test mm_rest`. Expected: 7 and
10 passed. `mm_rest` takes about 3 s: two tests wait out a rate limit on purpose.
- [ ] **5. Run the gate.** `cargo fmt --all`, then `make gate`. Expected last line: `gate: ok`.
- [ ] **6. Log and commit.** `git add crates/gatewayd docs/implementer-log.md Cargo.lock && git commit`
## Done when
- Both suites pass; `make gate` prints `gate: ok`.
## Stop and report if
- A test needs a post with an invalid id to be accepted.
docs/plans/M4a/11-gatewayd-sessions.md
+80
View File
@@ -0,0 +1,80 @@
# M4a task 11: which posts become turns, and the queue per session
**Branch:** `m4a` (run `git switch m4a`; `git status --short` must be empty, otherwise stop)
**Commit subject:** `gatewayd: sessions, routing posts to sessions, commands and the queue`
## Goal
The decisions, as pure code: no network, no files. For each new post, **in this order** (spec
section 7):
1. Ignore it if it is the bot's own, or a system message (`kind` not empty).
2. Ignore it, **silently**, unless the author is in `allow.users`.
3. Is it for this Boxmaker? A direct message (`"D"`): always. A channel or group message (`"O"`,
`"P"`, `"G"`) whose id is in `allow.channels`: if it **names this bot**, or if it is a reply in a
thread this Boxmaker already has a session for **and names nobody else**. Anything else: ignore.
`@channel`, `@here` and `@all` name nobody: every agent would answer them.
4. The session: its root is `root_id`, or the post's own id when that is empty; the session id is
`mm-<root>`. A top-level post is a new session (`resume: false`); a reply resumes it.
5. Commands: a message starting with `!`. `!!…` is not a command: one `!` is removed and the rest
goes on as a message. `!approve …` and `!deny …` are answered with `M4B_COMMAND`; any other `!`
with `UNKNOWN_COMMAND`. A command never reaches `loopd`.
6. Otherwise the message joins its session's queue.
"Names" means `@<username>` in the message, case-insensitive, where the name is the longest run of
`a-z`, `0-9`, `.`, `-` and `_` after the `@`, without trailing dots. The spec's examples, with this
bot called `boxmaker-straylight` and another agent called Hermes in the channel (the first test
checks every row):
| Post | For Boxmaker? |
|---|---|
| `@boxmaker-straylight summarise the audit log` (top level) | yes: a new session rooted here |
| a reply in that thread: `and the older files?` | yes: its thread, nobody else named |
| a reply in that thread: `@hermes what do you think?` | no |
| `@boxmaker-straylight @hermes compare notes` | yes (and Hermes answers too) |
| `@boxmaker-straylightx hello` | no: a different name |
| `@channel standup in five` | no |
**The queue.** One turn at a time per session. A message for a session with no turn running
starts one at once. Messages that arrive while it runs wait; when it ends, **all** waiting messages
go together as the next turn, joined with a blank line (`"\n\n"`), in the order they came. A
session with `limit` messages already waiting drops the next one, and the caller answers `BUSY`.
## Files
- Copy: `crates/gatewayd/tests/sessions.rs`, and the skeleton `crates/gatewayd/src/sessions.rs`
- Modify: `crates/gatewayd/src/lib.rs` (`pub mod sessions;`), `docs/implementer-log.md`
## The skeleton
Written: the three answer texts (`M4B_COMMAND`, `UNKNOWN_COMMAND`, `BUSY`), `EVERYONE`, `Ignored`
(`Own`, `System`, `NotAllowed`, `NotForUs`), `Thread { channel, root }`, `Message { session,
thread, resume, text, joins_thread }`, `Route` (`Ignore`, `Reply { thread, text }`, `Queue`),
`Router`, `Batch { session, thread, resume, text }`, `Pushed` (`Start(Batch)`, `Waiting`,
`Full(Thread)`), `Pending`, `Queues`.
To fill: `named`, `Router::new`, `Router::for_us`, `Router::route`, `Queues::new`, `push`,
`finish`, `running`, `threads`. `route` is a straight line of early returns in the order above; the
`known` function it is given answers "does this Boxmaker have a session for this thread root?"
(task 14 answers it from the state file). `SessionId` has no `Ord`, so `Queues` keys a `HashMap`.
## Steps
- [ ] **1. Copy.** `git switch m4a`, then
`cp docs/plans/M4a/files/crates/gatewayd/tests/sessions.rs crates/gatewayd/tests/ && cp docs/plans/M4a/files/crates/gatewayd/src/sessions.rs crates/gatewayd/src/`.
Add `pub mod sessions;` to `lib.rs`.
- [ ] **2. See it fail.** `cargo test -p gatewayd --test sessions`. Expected: it compiles and 8
tests fail.
- [ ] **3. Fill `named`, then the `Router`, then the `Queues`**, `cargo check -p gatewayd` after
each function.
- [ ] **4. See it pass.** `cargo test -p gatewayd --test sessions`. Expected: 8 passed.
- [ ] **5. Run the gate.** `cargo fmt --all`, then `make gate`. Expected last line: `gate: ok`.
- [ ] **6. Log and commit.** `git add crates/gatewayd docs/implementer-log.md Cargo.lock && git commit`
## Done when
- `cargo test -p gatewayd --test sessions` passes; `make gate` prints `gate: ok`.
## Stop and report if
- A row of the table above seems to need a different order of the steps.
docs/plans/M4a/12-gatewayd-state.md
+63
View File
@@ -0,0 +1,63 @@
# M4a task 12: the state file
**Branch:** `m4a` (run `git switch m4a`; `git status --short` must be empty, otherwise stop)
**Commit subject:** `gatewayd: state, what was handled, our threads and turns in flight`
## Goal
`<home>/gateway/state.json` lets `gatewayd` pick up where it left off (spec section 9):
```json
{"channels": {"<channel id>": 1758650000000}, "recent": ["<post id>", …],
"threads": ["<root post id>", …],
"in_flight": [{"session": "mm-…", "channel": "<id>", "root": "<id>"}]}
```
- `channels`: the `create_at` of the last post handled in each channel; catching up starts there.
A mark never moves back.
- `recent`: the ids of the last 500 posts handled (`RECENT_KEPT`), so a post seen twice (live and
in a catch-up) is handled once.
- `threads`: the roots of the threads this Boxmaker takes part in, in channels; the newest 5,000
(`THREADS_KEPT`).
- `in_flight`: turns sent to `loopd` and not yet answered; after a restart each gets an
"interrupted" reply.
It is **our** format: unknown fields are errors, and every id must be a valid Mattermost id. It is
written atomically after every change, in the same six steps as `brokerd/src/state.rs`'s
`persist` (read it). A file that exists but cannot be read, parsed or written stops `gatewayd`
with `see docs/runbook.md#gateway-state-damaged`: guessing would answer posts twice. **Only a
missing file** is a first start.
## Files
- Copy: `crates/gatewayd/tests/state.rs`, and the skeleton `crates/gatewayd/src/state.rs`
- Modify: `crates/gatewayd/src/lib.rs` (`pub mod state;`), `docs/implementer-log.md`
## The skeleton
Written: `RECENT_KEPT`, `THREADS_KEPT`, `RUNBOOK`, `StateError` (`Read(PathBuf, String)`,
`Write(PathBuf, io::Error)`) and its `Display` (the path, the problem, and the pointer on its own
line), `InFlight`, the private `StateFile`, `State { path, file }`.
To fill: `StateFile::problem`, `State::load`, `save`, `persist`, and the methods `seen`, `handled`,
`since`, `channels`, `mark`, `knows_thread`, `join_thread`, `start_turn`, `end_turn`,
`take_in_flight`. Every method that changes the state saves it before it returns, and returns the
save's error.
## Steps
- [ ] **1. Copy.** `git switch m4a`, then
`cp docs/plans/M4a/files/crates/gatewayd/tests/state.rs crates/gatewayd/tests/ && cp docs/plans/M4a/files/crates/gatewayd/src/state.rs crates/gatewayd/src/`.
Add `pub mod state;` to `lib.rs`.
- [ ] **2. See it fail.** `cargo test -p gatewayd --test state`. Expected: it compiles and 4 tests
fail.
- [ ] **3. Fill `problem`, `load`, `persist`, `save`, then the methods**, `cargo check -p gatewayd`
after each.
- [ ] **4. See it pass.** `cargo test -p gatewayd --test state`. Expected: 4 passed, in well under a
second.
- [ ] **5. Run the gate.** `cargo fmt --all`, then `make gate`. Expected last line: `gate: ok`.
- [ ] **6. Log and commit.** `git add crates/gatewayd docs/implementer-log.md Cargo.lock && git commit`
## Done when
- `cargo test -p gatewayd --test state` passes; `make gate` prints `gate: ok`.
docs/plans/M4a/13-gatewayd-deliver.md
+61
View File
@@ -0,0 +1,61 @@
# M4a task 13: a turn on `loop.sock`, and its answer in the thread
**Branch:** `m4a` (run `git switch m4a`; `git status --short` must be empty, otherwise stop)
**Commit subject:** `gatewayd: deliver, turns on loop.sock and answers posted in their thread`
## Goal
A batch of messages (task 11) becomes one `turn` on `loopd`'s socket, exactly as `bxctl chat` sends
one (read `crates/bxctl/src/chat.rs`, `run_turn`): one frame `{"kind": "turn", "body": {"session",
"content", "resume"}}` with id 1, then events until a final `turn_done` or `error`. What comes back
is posted in the thread (spec section 8):
- No thinking and no status reach Mattermost. The one event that is shown: `approval_pending`,
posted once as `waiting for approval <id>: approve or deny it with `bxctl` (Mattermost approvals
arrive in M4b)` (the skeleton's `approval_text`).
- `turn_done` → its content, split into posts of at most 16,000 **characters** (a Mattermost post
holds 16,383), at the last newline before the limit, or at the limit when there is none, in
order. An empty answer is posted as `(the answer was empty)`: Mattermost refuses an empty post.
- An `error` frame → `Error: <code>: <detail>`, where the code is the snake_case name
(`no_such_session`), and the detail carries `loopd`'s runbook pointer when it has one.
- `loop.sock` cannot be reached, or closes early, or sends a frame that does not belong →
`LOOP_DOWN`, and the messages are not sent again: `loopd` may have run them.
- A reply in a thread `loopd` does not know (`no_such_session` with `resume: true`) is sent once
more with `resume: false`, creating the session, as `bxctl chat --session` does. Only then.
## Files
- Copy: `crates/gatewayd/tests/deliver.rs`, `crates/gatewayd/tests/support/fake_loop.rs`, and the
skeleton `crates/gatewayd/src/deliver.rs`
- Modify: `crates/gatewayd/src/lib.rs` (`pub mod deliver;`), `docs/implementer-log.md`
## The skeleton
Written: `MAX_POST` (16,000), `LOOP_DOWN`, `EMPTY_ANSWER`, the trait `Poster` (somewhere to post:
Mattermost's `Client`, or a test's record) and its `impl` for `Client`, `Outcome` (`Answer`,
`Refused`, `LoopDown(why)`), `approval_text`.
To fill: `error_text`, `split_answer`, `one_turn`, `run_turn`, `deliver`. `split_answer` counts
**characters**, not bytes, and never cuts inside one: find the byte index of the 16,000th character
with `char_indices().nth(MAX_POST)`.
## Steps
- [ ] **1. Copy.** `git switch m4a`, then
`cp docs/plans/M4a/files/crates/gatewayd/tests/deliver.rs crates/gatewayd/tests/ && cp docs/plans/M4a/files/crates/gatewayd/tests/support/fake_loop.rs crates/gatewayd/tests/support/ && cp docs/plans/M4a/files/crates/gatewayd/src/deliver.rs crates/gatewayd/src/`.
Add `pub mod deliver;` to `lib.rs`.
- [ ] **2. See it fail.** `cargo test -p gatewayd --test deliver`. Expected: it compiles and 9
tests fail.
- [ ] **3. Fill `error_text`, `split_answer`, `one_turn`, `run_turn`, `deliver`**,
`cargo check -p gatewayd` after each.
- [ ] **4. See it pass.** `cargo test -p gatewayd --test deliver`. Expected: 9 passed.
- [ ] **5. Run the gate.** `cargo fmt --all`, then `make gate`. Expected last line: `gate: ok`.
- [ ] **6. Log and commit.** `git add crates/gatewayd docs/implementer-log.md Cargo.lock && git commit`
## Done when
- `cargo test -p gatewayd --test deliver` passes; `make gate` prints `gate: ok`.
## Stop and report if
- A test seems to need a turn sent twice in any case other than the one above.
docs/plans/M4a/14-gatewayd-serve.md
+79
View File
@@ -0,0 +1,79 @@
# M4a task 14: the serve loop
**Branch:** `m4a` (run `git switch m4a`; `git status --short` must be empty, otherwise stop)
**Commit subject:** `gatewayd: serve, the event loop, typing, catch-up and reconnecting`
## Goal
The loop that ties tasks 03 to 13 together (spec sections 8 and 9):
- **Connecting**, at start and after every loss: `GET /users/me`, open the WebSocket, wait for
`hello`, then log `gatewayd: connected to <url> as <username>`. A connection that fails is tried
again after 1, 2, 5, 10, then every 30 seconds, with one log line per attempt:
`gatewayd: cannot reach <url>: <error>; trying again in <n> s`, then a newline and
`see docs/runbook.md#mattermost-unreachable`. A refused token (401 or 403) is never tried again:
`run` returns `Stop::Auth`.
- **After a restart** (the first connection only): every turn left in flight gets
`interrupted: gatewayd restarted before the answer arrived; ask again` in its thread. A later
reconnect must not do this: those turns are still running.
- **Catching up**: the direct channel with each allowed user, and each allowed channel, from its
mark. A channel without a mark is marked "now" and not caught up: history is not answered.
- **Each post**, live or caught up: posts in channels we do not track are not recorded at all. A
tracked post is recorded as handled **before** it is acted on, so after a crash it is not
answered twice (the in-flight record reports it instead). Then it is routed (task 11): a stranger
is logged by user id and post id only, **never with the text**.
- **Turns** run on their own threads (task 13), recorded in flight while they run. When one ends,
its session's waiting messages start the next.
- **Typing**: every `typing_every_ms`, `user_typing` for every thread with a turn running.
## Files
- Copy: `crates/gatewayd/tests/serve.rs`, `crates/gatewayd/tests/serve_restart.rs`,
`crates/gatewayd/tests/support/fake_mm.rs`, `crates/gatewayd/tests/support/gateway.rs`, and the
skeletons `crates/gatewayd/src/serve/mod.rs` and `crates/gatewayd/src/serve/handle.rs`
- Modify: `crates/gatewayd/src/lib.rs` (`pub mod serve;`), `docs/implementer-log.md`
## The skeletons
`serve/mod.rs`, written: the pointers, `INTERRUPTED`, `Log`, `Stop` (`Auth`, `State`, `Start`,
`Asked`) and its `Display`, `Tuning` and its default, the `Gateway` struct, and the two functions
that are glue: **`run`** (the connect-or-back-off loop) and **`Gateway::event_loop`** (finish turns,
send typing, wait for an event, handle it). Read both first: they call everything you write.
To fill: `From<StateError> for Stop`, `backoff`, `sleep_unless`, `connect`, `Gateway::new`,
`Gateway::connected`.
`serve/handle.rs`, all to fill: `now_ms`, `post`, `tracked`, `handle_post`, `start`, `finished`,
`typing`, `catch_up`.
Each `todo!()` has its steps above it, with the exact log lines. `run` takes the token already
loaded and a `stop` flag, so the tests need no secrets and can end it; task 15's `main` passes a
flag that is never set.
## Steps
- [ ] **1. Copy.** `git switch m4a`, then
`mkdir -p crates/gatewayd/src/serve && cp docs/plans/M4a/files/crates/gatewayd/src/serve/mod.rs docs/plans/M4a/files/crates/gatewayd/src/serve/handle.rs crates/gatewayd/src/serve/`,
`cp docs/plans/M4a/files/crates/gatewayd/tests/serve.rs docs/plans/M4a/files/crates/gatewayd/tests/serve_restart.rs crates/gatewayd/tests/`
and
`cp docs/plans/M4a/files/crates/gatewayd/tests/support/fake_mm.rs docs/plans/M4a/files/crates/gatewayd/tests/support/gateway.rs crates/gatewayd/tests/support/`.
Add `pub mod serve;` to `lib.rs`.
- [ ] **2. See it fail.** `cargo test -p gatewayd --no-fail-fast --test serve --test serve_restart`.
Expected: it compiles; `serve` 6 fail; `serve_restart` 6 fail and 1 passes (a damaged state file
stops `run` before anything you write is called).
- [ ] **3. Fill `mod.rs` first** (`from`, `backoff`, `sleep_unless`, `Gateway::new`, `connect`,
`connected`), **then `handle.rs`** (`now_ms`, `post`, `tracked`, `finished`, `typing`, `start`,
`handle_post`, `catch_up`). `cargo check -p gatewayd` after each function.
- [ ] **4. See it pass.** `cargo test -p gatewayd --test serve --test serve_restart`, five times.
Expected: 6 and 7 passed each time, in about 2 s.
- [ ] **5. Run the gate.** `cargo fmt --all`, then `make gate`. Expected last line: `gate: ok`.
- [ ] **6. Log and commit.** `git add crates/gatewayd docs/implementer-log.md Cargo.lock && git commit`
## Done when
- Both suites pass five times running; `make gate` prints `gate: ok`.
## Stop and report if
- A test passes only sometimes, or a test takes 5 s or more (that is a wait that timed out, not a
pass).
- `run` or `event_loop` seems to need a change. They are given; report instead.
docs/plans/M4a/15-gatewayd-main.md
+57
View File
@@ -0,0 +1,57 @@
# M4a task 15: `gatewayd serve --config <path>`
**Branch:** `m4a` (run `git switch m4a`; `git status --short` must be empty, otherwise stop)
**Commit subject:** `gatewayd: main, serve --config with its start checks`
## Goal
The program. `gatewayd serve --config <path>` loads the configuration and the token, prepares its
directory, and runs the serve loop (task 14) until it must stop, then exits 1. Anything else on the
command line prints `usage: gatewayd serve --config <path>` and exits 2. Every failure at start
prints one line and its runbook pointer and exits 1, **before** anything is created on disk and
before any connection:
| Failure | Output |
|---|---|
| the config cannot be read, parsed or checked | `gatewayd: <error>` then `see docs/runbook.md#gatewayd-start-failed` |
| the token's source is wrong | `gatewayd: <path>: <why>` then the same pointer |
| the token cannot be loaded | `gatewayd: secret mattermost_token: <why>` then `see docs/runbook.md#secret-unavailable` (the error's own `Display`) |
| `<home>/gateway` cannot be created | `gatewayd: cannot prepare <dir>: <error>` then the start-failed pointer |
A token read from a file prints task 04's warning, and `gatewayd` goes on. The token itself is
never printed: the last test runs the real program and looks for it in everything it wrote.
## Files
- Copy: `crates/gatewayd/tests/main.rs`, and the skeleton `crates/gatewayd/src/main.rs` (replaces
the placeholder that printed "not implemented")
- Modify: `docs/implementer-log.md`
## The skeleton
Written: `main`, which parses the arguments like `loopd`'s. To fill: `serve(path)`, whose steps
are above its `todo!()`. The directory is `config.state_path()`'s parent, created with
`std::fs::DirBuilder`, `recursive(true)` and `mode(0o700)` (`std::os::unix::fs::DirBuilderExt`).
## Steps
- [ ] **1. Copy.** `git switch m4a`, then
`cp docs/plans/M4a/files/crates/gatewayd/tests/main.rs crates/gatewayd/tests/ && cp docs/plans/M4a/files/crates/gatewayd/src/main.rs crates/gatewayd/src/`
- [ ] **2. See it fail.** `cargo test -p gatewayd --test main`. Expected: it compiles; 4 fail and 1
passes (`usage`, since `main` is given).
- [ ] **3. Fill `serve`.** `cargo check -p gatewayd`.
- [ ] **4. See it pass.** `cargo test -p gatewayd --test main`. Expected: 5 passed.
- [ ] **5. Run everything.** `cargo test -p gatewayd`. Expected: every suite passes.
- [ ] **6. Run the gate.** `cargo fmt --all`, then `make gate`. Expected last line: `gate: ok`,
with about 762 tests.
- [ ] **7. Log and commit.** `git add crates/gatewayd docs/implementer-log.md Cargo.lock && git commit`
This is the last task of M4a. Stop after the commit.
## Done when
- `cargo test -p gatewayd` passes; `make gate` prints `gate: ok`.
## Stop and report if
- A test's output contains the token.
docs/plans/M4a/README.md
+75
View File
@@ -0,0 +1,75 @@
# M4a implementation plan: `gatewayd`, conversations over Mattermost
> **For the implementing model:** do not work from this file. The owner gives you one task file at
> a time (`01-…` to `15-…`). This file is the index for the owner and the reviewer.
**Goal:** a direct message to Boxmaker's bot account on Mattermost becomes a turn in `loopd`, and
the answer is posted in its thread. In channels shared with other agents, Boxmaker answers only
posts that name it, and replies in its own threads that name nobody else. Anyone not on the
allowlist gets nothing at all. `gatewayd` opens no listening port.
**Architecture:** `proto` gains SHA-1 (01). `gatewayd` gets its dependencies and test certificates
(02), its configuration (03), its secrets and the runbook entries (04), then a network layer built
bottom-up: TCP or TLS (05), HTTP/1.1 (06), the WebSocket handshake (07), frames (08) and the
connection (09). Mattermost's JSON and REST calls (10), the routing of posts to sessions (11), the
state file (12) and delivery on `loop.sock` (13) are each one module. The serve loop ties them
together (14), and `main` starts it (15).
**Spec:** `docs/specs/2026-09-23-m4a-gateway.md`. Brief: `docs/design.md` (P15 applied). Every
fail-closed message ends with a pointer into `docs/runbook.md`; task 04 adds all seven entries
this milestone needs.
**No task needs Mattermost or a network.** Everything runs against fakes: TLS test servers with a
test-only CA, a scripted WebSocket server, a scripted HTTP server, a fake Mattermost and a fake
`loopd`. The checks against the owner's server are done by the design model afterwards.
## Global constraints
- Everything in `AGENTS.md`, including "Lessons from earlier reviews".
- New dependencies, all in task 02 and no others: `rustls` 0.23.45 (no default features; `ring`,
`std`, `tls12`), `rustls-native-certs` 0.8.4, `zeroize` 1.9.0; `gatewayd` also uses `serde`,
`serde_json` and `toml`, already vetted.
- Branch `m4a`. One task, one fresh OpenCode session, one commit. Run `cargo fmt --all` before the
gate. Review happens once, after task 15.
- Most tasks hand over a **skeleton**: the file with its types, constants and signatures written,
and `todo!()` bodies with the steps as comments above them. Fill one function at a time and run
`cargo check -p <crate>` after each (tip T25). Keep the comments; they say why.
- Mattermost's JSON is another program's format: unknown fields are ignored. Our formats
(`gatewayd.toml`, `state.json`) reject them.
## Tasks
The last column is how the given tests were checked before hand-over (tip T17). Every task had a
reference implementation, and the given tests were run against it at that task's end state. Then
each task's end state was rebuilt on its own from master in this order, with the gate at every
step (tip T26), and the reference was deleted so it cannot be read (tip T18). Each skeleton was
checked to compile against its task's tests and fail them.
| # | File | Delivers | Tests | Check |
|---|---|---|---|---|
| 01 | `01-proto-sha1.md` | `proto::sha1` | `proto/tests/sha1.rs` | reference; vectors checked with `sha1sum` |
| 02 | `02-gatewayd-deps.md` | dependencies, `deny.toml`, TLS test certificates | none new | reference |
| 03 | `03-gatewayd-config.md` | `gatewayd.toml` | `config.rs`, `support/tmp.rs` | reference |
| 04 | `04-gatewayd-secrets.md` | `SecretStore`: credential, env, file; seven runbook entries | `secrets.rs` | reference |
| 05 | `05-gatewayd-net.md` | TCP or TLS, verified | `net.rs`, `support/tls_server.rs` | reference |
| 06 | `06-gatewayd-http.md` | HTTP/1.1 requests, size caps, rate-limit waits | `http.rs` | reference |
| 07 | `07-gatewayd-ws-handshake.md` | base64, the WebSocket handshake | `ws_handshake.rs` | reference |
| 08 | `08-gatewayd-ws-frames.md` | the frame decoder and encoder | `ws_frame.rs` (a naive decoder as oracle, 300 seeds) | reference |
| 09 | `09-gatewayd-ws-conn.md` | the connection: pings, close, a dead peer | `ws_conn.rs`, `support/ws_server.rs` | reference; 7 mutations, all caught once a test for the `Host` header was added |
| 10 | `10-gatewayd-mm.md` | Mattermost's events and REST calls | `mm_json.rs`, `mm_rest.rs`, `support/http_server.rs` | reference; 11 mutations, 10 caught, 1 that cannot change behaviour |
| 11 | `11-gatewayd-sessions.md` | routing, commands, the queue per session | `sessions.rs` | reference; 14 mutations, 13 caught, 1 that cannot change behaviour |
| 12 | `12-gatewayd-state.md` | `state.json` | `state.rs` | reference; 12 mutations, all caught |
| 13 | `13-gatewayd-deliver.md` | turns on `loop.sock`, answers in the thread | `deliver.rs`, `support/fake_loop.rs` | reference; 11 mutations, 10 caught; the one left accepts an event frame with another id, which `loopd` never sends |
| 14 | `14-gatewayd-serve.md` | the serve loop: routing, typing, catch-up, reconnecting | `serve.rs`, `serve_restart.rs`, `support/fake_mm.rs`, `support/gateway.rs` | reference; 14 mutations, all caught once four tests were added; 8 runs clean |
| 15 | `15-gatewayd-main.md` | `gatewayd serve --config <path>` | `main.rs` | reference; run against the owner's server, token never printed |
At the end of task 15: about 762 tests (650 before task 01).
## Running it
```sh
git switch m4a
BOXMAKER_MODEL=straylight/ornith-1.5-35b-a3b tools/run-plan.sh docs/plans/M4a
```
Keep the OpenCode TUI closed while it runs.
docs/plans/M4a/files/crates/gatewayd/src/config.rs
+194
View File
@@ -0,0 +1,194 @@
//! `gatewayd.toml` into a typed `Config`. Our own format: unknown keys are errors in every table.
use std::collections::BTreeMap;
use std::path::{Path, PathBuf};
use serde::Deserialize;
#[derive(Debug, Clone, PartialEq, Eq, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct Config {
pub mattermost: MattermostConfig,
pub secrets: BTreeMap<String, SecretSpec>,
pub allow: AllowConfig,
#[serde(default, rename = "loop")]
pub loop_: LoopConfig,
#[serde(default)]
pub paths: Paths,
#[serde(default)]
pub limits: Limits,
}
#[derive(Debug, Clone, PartialEq, Eq, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct MattermostConfig {
pub url: String,
#[serde(default)]
pub ca_file: Option<PathBuf>,
}
/// Where one secret comes from: exactly one of the three is set (checked by `Config::load`).
#[derive(Debug, Clone, PartialEq, Eq, Deserialize, Default)]
#[serde(deny_unknown_fields)]
pub struct SecretSpec {
#[serde(default)]
pub credential: Option<String>,
#[serde(default)]
pub env: Option<String>,
#[serde(default)]
pub file: Option<PathBuf>,
}
/// A checked `SecretSpec`.
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum SecretSource {
Credential(String),
Env(String),
File(PathBuf),
}
#[derive(Debug, Clone, PartialEq, Eq, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct AllowConfig {
pub users: Vec<String>,
#[serde(default)]
pub channels: Vec<String>,
}
#[derive(Debug, Clone, PartialEq, Eq, Deserialize, Default)]
#[serde(deny_unknown_fields, default)]
pub struct LoopConfig {
pub socket: PathBuf,
}
#[derive(Debug, Clone, PartialEq, Eq, Deserialize)]
#[serde(deny_unknown_fields, default)]
pub struct Paths {
pub home: PathBuf,
}
impl Default for Paths {
fn default() -> Self {
Paths {
home: std::env::var_os("BOXMAKER_HOME")
.map(PathBuf::from)
.unwrap_or_else(|| PathBuf::from("/var/lib/boxmaker")),
}
}
}
#[derive(Debug, Clone, PartialEq, Eq, Deserialize)]
#[serde(deny_unknown_fields, default)]
pub struct Limits {
pub queue: u32,
pub typing_every_ms: u64,
pub ping_every_ms: u64,
pub dead_after_ms: u64,
}
impl Default for Limits {
fn default() -> Self {
Limits {
queue: 20,
typing_every_ms: 3_000,
ping_every_ms: 30_000,
dead_after_ms: 60_000,
}
}
}
/// `url` taken apart.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct ServerUrl {
pub tls: bool,
pub host: String,
pub port: u16,
}
#[derive(Debug)]
pub enum ConfigError {
Read(PathBuf, std::io::Error),
Parse(PathBuf, toml::de::Error),
Invalid(PathBuf, String),
}
impl std::fmt::Display for ConfigError {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
// Each variant: "<path>: <error or why>", with `path.display()`.
todo!()
}
}
impl std::error::Error for ConfigError {}
/// The one secret M4a needs.
pub const MATTERMOST_TOKEN: &str = "mattermost_token";
impl Config {
/// Parse without the checks `load` makes.
pub fn parse(text: &str) -> Result<Config, toml::de::Error> {
// `toml::from_str`.
todo!()
}
pub fn load(path: &Path) -> Result<Config, ConfigError> {
// Read the file (else Read), parse (else Parse), then `problem()` (Some(why) is Invalid).
todo!()
}
/// The first thing wrong with the values, or `None`.
pub fn problem(&self) -> Option<String> {
// The first of these, in this order, with the exact messages in the task: the url
// (`parse_url`); ca_file not absolute; [secrets.mattermost_token] missing; each secret
// whose `source()` fails; allow.users empty; any id in allow.users or allow.channels not
// `valid_id`; any limit that is 0.
todo!()
}
/// The server's address, from `url`. Call only on a checked `Config`.
pub fn server(&self) -> Result<ServerUrl, String> {
// `parse_url` of the url.
todo!()
}
pub fn token_source(&self) -> Result<SecretSource, String> {
// The `source()` of [secrets.mattermost_token], or an error if it is missing.
todo!()
}
pub fn loop_socket(&self) -> PathBuf {
// [loop] socket, or <home>/run/loop/loop.sock when it is empty.
todo!()
}
pub fn state_path(&self) -> PathBuf {
// <home>/gateway/state.json.
todo!()
}
}
impl SecretSpec {
/// Exactly one source, well formed.
pub fn source(&self) -> Result<SecretSource, String> {
// Exactly one of the three set, else "needs exactly one of credential, env and file".
// credential: not empty, only ASCII letters, digits, _ . -. env: not empty, only A-Z, 0-9,
// _. file: an absolute path. The messages are in the task.
todo!()
}
}
/// A Mattermost id: 26 characters of `a-z0-9`.
pub fn valid_id(id: &str) -> bool {
// 26 bytes, each a-z or 0-9.
todo!()
}
/// `http://host[:port]` or `https://host[:port]`, nothing else.
pub fn parse_url(url: &str) -> Result<ServerUrl, String> {
// http:// or https://, then a host, then optionally ":" and a port. The port: 1..=65535 written
// exactly as `port.to_string()` (so no "+1", no "080"). Default 443 for https, 80 for http. The
// host: 1..=253 bytes of a-z, 0-9, "." and "-", not starting or ending with "." or "-".
// Anything else, including a path, a user or an upper-case letter, is the one error message in
// the task.
todo!()
}
docs/plans/M4a/files/crates/gatewayd/src/deliver.rs
+96
View File
@@ -0,0 +1,96 @@
//! One turn on `loop.sock` and its answer in the thread (M4a spec, section 8). Typing is shown by
//! `serve`, which owns the WebSocket; this module only sends the turn and posts what comes back.
use std::os::unix::net::UnixStream;
use std::path::Path;
use proto::{
Envelope, ErrorCode, Message, PROTOCOL_VERSION, Turn, TurnEvent, WireError, read_frame,
write_frame,
};
use crate::mm::MmError;
use crate::mm::rest::Client;
use crate::sessions::Batch;
/// The longest reply we post, in characters (a post holds at most 16,383).
pub const MAX_POST: usize = 16_000;
pub const LOOP_DOWN: &str = "Boxmaker's loop is not running (see docs/runbook.md#loop-unavailable)";
pub const EMPTY_ANSWER: &str = "(the answer was empty)";
/// Somewhere to post: Mattermost, or a test's record.
pub trait Poster: Send + Sync {
fn post(&self, channel: &str, root: &str, text: &str) -> Result<(), MmError>;
}
impl Poster for Client {
fn post(&self, channel: &str, root: &str, text: &str) -> Result<(), MmError> {
self.create_post(channel, root, text).map(|_| ())
}
}
/// How a turn ended.
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum Outcome {
Answer(String),
Refused(WireError),
/// `loop.sock` could not be reached or closed early; why, for the log.
LoopDown(String),
}
pub fn approval_text(approval: u64) -> String {
format!(
"waiting for approval {approval}: approve or deny it with `bxctl` (Mattermost approvals arrive in M4b)"
)
}
/// "Error: <code>: <detail>"; the detail carries `loopd`'s runbook pointer when there is one.
pub fn error_text(error: &WireError) -> String {
// "Error: <code>: <detail>", where <code> is the snake_case name serde gives the ErrorCode
// (`serde_json::to_value(code)` is a JSON string, e.g. "no_such_session").
todo!()
}
/// An answer in posts of at most `MAX_POST` characters: each cut at the last newline before the
/// limit (the newline is dropped), or at the limit when there is none.
pub fn split_answer(text: &str) -> Vec<String> {
// Blank (only whitespace) -> [EMPTY_ANSWER]. Otherwise, while the rest is longer than MAX_POST
// *characters*: take the first MAX_POST characters; if they hold a newline past position 0, cut
// at the last one and drop that newline; else cut at MAX_POST characters. The last part is the
// rest.
todo!()
}
/// Send one turn and read it to its end; `on_event` sees every event.
fn one_turn(
socket: &Path,
batch: &Batch,
resume: bool,
on_event: &mut dyn FnMut(&TurnEvent),
) -> Outcome {
// 1. Connect to the socket (else LoopDown("cannot connect to <path>: <e>")).
// 2. `write_frame` one Envelope: v PROTOCOL_VERSION, id 1, final true, msg Turn (else
// LoopDown).
// 3. `read_frame` until the end: (id 1, not final, TurnEvent) -> on_event; (1, final, TurnDone)
// -> Answer with its content; (1, final, Error) -> Refused; a read error -> LoopDown("the
// turn ended early: <e>"); anything else -> LoopDown("an unexpected frame").
todo!()
}
/// A turn for a batch. A reply in a thread `loopd` does not know creates the session, as
/// `bxctl chat --session` does.
pub fn run_turn(socket: &Path, batch: &Batch, on_event: &mut dyn FnMut(&TurnEvent)) -> Outcome {
// `one_turn` with batch.resume. When that is Refused with NoSuchSession and batch.resume was
// true, one more `one_turn` with resume false. Otherwise the first outcome.
todo!()
}
/// Run a batch's turn and post what comes of it in its thread. A post that fails is logged.
pub fn deliver(poster: &dyn Poster, socket: &Path, batch: &Batch, log: &dyn Fn(&str)) {
// Post in the batch thread. An ApprovalPending event posts `approval_text(approval)` at once.
// Then: Answer -> every part of `split_answer`, in order; Refused -> `error_text`;
// LoopDown(why) -> log "gatewayd: <session>: <why>" and post LOOP_DOWN. A post that fails is
// logged, exactly "gatewayd: cannot post in <channel> (thread <root>): <error>", and the rest
// goes on.
todo!()
}
docs/plans/M4a/files/crates/gatewayd/src/http.rs
+127
View File
@@ -0,0 +1,127 @@
//! HTTP/1.1 over a connected stream: one request, one response, `Connection: close` (M4a spec,
//! section 5). The response head is read a byte at a time, so nothing past it is consumed: the
//! WebSocket handshake reads its frames after it from the same stream.
use std::io::{Read, Write};
use std::time::{Duration, SystemTime, UNIX_EPOCH};
pub const MAX_HEAD: usize = 16 * 1024;
pub const MAX_BODY: usize = 4 * 1024 * 1024;
/// The longest a rate limit is waited out, whatever the server says.
pub const MAX_RATE_WAIT: Duration = Duration::from_secs(60);
#[derive(Debug)]
pub enum HttpError {
Io(std::io::Error),
/// The response is not HTTP/1.1 as we read it.
Protocol(String),
/// A head or body over its cap.
TooLarge(&'static str),
}
impl std::fmt::Display for HttpError {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
HttpError::Io(e) => write!(f, "{e}"),
HttpError::Protocol(why) => write!(f, "bad HTTP response: {why}"),
HttpError::TooLarge(what) => write!(f, "the response {what} is too large"),
}
}
}
impl std::error::Error for HttpError {}
impl From<std::io::Error> for HttpError {
fn from(e: std::io::Error) -> Self {
HttpError::Io(e)
}
}
/// A status line and headers.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Head {
pub status: u16,
pub headers: Vec<(String, String)>,
}
impl Head {
/// The first header named `name`, compared case-insensitively.
pub fn header(&self, name: &str) -> Option<&str> {
// The value of the first header whose name matches, ignoring ASCII case.
todo!()
}
}
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Response {
pub head: Head,
pub body: Vec<u8>,
}
/// Write one request. `host` is the `Host` header; `headers` come after it, then
/// `Content-Length` when there is a body, then `Connection: close`.
pub fn write_request(
stream: &mut dyn Write,
method: &str,
host: &str,
path: &str,
headers: &[(&str, &str)],
body: Option<&[u8]>,
) -> Result<(), HttpError> {
// Exactly: "<method> <path> HTTP/1.1\r\nHost: <host>\r\n", each header as "<k>: <v>\r\n",
// "Content-Length: <n>\r\n" when there is a body, "Connection: close\r\n\r\n", then the body.
// Flush.
todo!()
}
/// One request and its whole response.
pub fn request(
stream: &mut (impl Read + Write),
method: &str,
host: &str,
path: &str,
headers: &[(&str, &str)],
body: Option<&[u8]>,
) -> Result<Response, HttpError> {
// `write_request`, then `read_head`, then `read_body`.
todo!()
}
/// The status line and headers, up to and including the blank line, and not a byte more.
pub fn read_head(stream: &mut dyn Read) -> Result<Head, HttpError> {
// One byte at a time until CRLF CRLF, never more; over MAX_HEAD is TooLarge("head"); end of
// stream is Protocol; retry Interrupted. Then: UTF-8; status line "HTTP/1.1" or "HTTP/1.0", a
// 3-digit status in 100..=599; each header line "name: value" with a non-empty name without
// spaces, the value trimmed. Anything else is Protocol.
todo!()
}
/// The body after `head`: chunked, `Content-Length`, or to the end; at most `MAX_BODY`.
pub fn read_body(stream: &mut dyn Read, head: &Head) -> Result<Vec<u8>, HttpError> {
// Transfer-Encoding: chunked (any case) -> `read_chunked`. Else Content-Length: parse (else
// Protocol), over MAX_BODY is TooLarge("body"), then read_exact that many. Else read to the end
// through `take(MAX_BODY + 1)`; more than MAX_BODY is TooLarge("body").
todo!()
}
fn read_line(stream: &mut dyn Read, cap: usize) -> Result<String, HttpError> {
// Bytes up to CRLF (dropped), at most `cap` (else TooLarge("chunk header")); end of stream is
// Protocol.
todo!()
}
fn read_chunked(stream: &mut dyn Read) -> Result<Vec<u8>, HttpError> {
// Loop: a size line (hex, before any ";"), at most 1024 bytes. Size 0: read trailer lines (8
// KiB each) until an empty one, and return. Otherwise the size must fit in MAX_BODY minus what
// is already read (else TooLarge("body")), read it, then exactly CRLF (else Protocol).
todo!()
}
/// How long a 429 asks us to wait, from `X-Ratelimit-Reset`: a Unix time if it is one, else a
/// number of seconds; never more than `MAX_RATE_WAIT`. One second if the header is missing or bad.
pub fn rate_limit_wait(head: &Head, now: SystemTime) -> Duration {
// X-Ratelimit-Reset as u64: above 1_000_000_000 it is a Unix time (wait = it - now, at least 1
// s), otherwise seconds (at least 1). Missing or not a number: 1 s. Never more than
// MAX_RATE_WAIT.
todo!()
}
docs/plans/M4a/files/crates/gatewayd/src/main.rs
+37
View File
@@ -0,0 +1,37 @@
//! `gatewayd serve --config <path>`: the Mattermost channel. It loads its configuration and its
//! token, prepares its directory, then serves until it must stop (exit 1).
use std::os::unix::fs::DirBuilderExt;
use std::path::Path;
use std::process::ExitCode;
use std::sync::Arc;
use std::sync::atomic::AtomicBool;
use gatewayd::config::{Config, MATTERMOST_TOKEN};
use gatewayd::secrets;
use gatewayd::serve::{START_FAILED, Tuning, run};
fn main() -> ExitCode {
// `args_os`: the config path need not be UTF-8, and `args` would panic on one that is not.
let args: Vec<std::ffi::OsString> = std::env::args_os().skip(1).collect();
let words: Vec<Option<&str>> = args.iter().map(|a| a.to_str()).collect();
match (words.as_slice(), args.get(2)) {
([Some("serve"), Some("--config"), _], Some(path)) => serve(Path::new(path)),
_ => {
eprintln!("usage: gatewayd serve --config <path>");
ExitCode::from(2)
}
}
}
fn serve(path: &Path) -> ExitCode {
// Each failure prints one line (plus its pointer) and returns ExitCode::from(1):
// 1. `Config::load`: "gatewayd: <error>\n<START_FAILED>". 2. `token_source`: "gatewayd: <path>:
// <why>\n<START_FAILED>".
// 3. `secrets::load(MATTERMOST_TOKEN, &source, &|k| std::env::var_os(k))`: "gatewayd: <error>"
// (it carries its pointer). Print the warning, if any, as it is. 4. Create the state file
// directory, recursive, 0700: "gatewayd: cannot prepare <dir>: <e>\n<START_FAILED>". 5.
// `run` with Tuning::default(), a log that prints each line to standard error, and a stop
// flag that is never set; print the Stop it returns.
todo!()
}
docs/plans/M4a/files/crates/gatewayd/src/mm/mod.rs
+149
View File
@@ -0,0 +1,149 @@
//! Mattermost's JSON, typed: the posts and users we read, the events that arrive over the
//! WebSocket, and the requests we send over it. Mattermost's JSON is not ours: unknown fields are
//! ignored, but every id we keep must be a valid id.
pub mod rest;
use std::time::Duration;
use serde::Deserialize;
use crate::config::valid_id;
/// At most this many changed posts come back from one `posts?since` call (Mattermost v11.11.0,
/// `SqlPostStore::GetPostsSince`); a full answer may have left some out.
pub const SINCE_LIMIT: usize = 1000;
#[derive(Debug)]
pub enum MmError {
/// No answer: connecting, TLS, or the HTTP exchange failed.
Net(String),
/// 401 or 403: the token is refused.
Auth(u16),
/// 429, still, after waiting as asked.
RateLimited(Duration),
/// Any other status that is not 2xx, with the start of the body.
Status(u16, String),
/// The answer is not the JSON we expect.
Json(String),
}
impl std::fmt::Display for MmError {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
MmError::Net(why) => write!(f, "{why}"),
MmError::Auth(status) => write!(f, "Mattermost refused the token ({status})"),
MmError::RateLimited(wait) => {
write!(f, "rate limited; asked to wait {} s", wait.as_secs())
}
// Quoted: the body is the server's text, and must not forge a log line.
MmError::Status(status, body) => write!(f, "status {status}: {body:?}"),
MmError::Json(why) => write!(f, "unexpected JSON: {why}"),
}
}
}
impl std::error::Error for MmError {}
/// This bot, from `GET /users/me`.
#[derive(Debug, Clone, PartialEq, Eq, Deserialize)]
pub struct Me {
pub id: String,
pub username: String,
}
/// The fields of a post that `gatewayd` uses.
#[derive(Debug, Clone, PartialEq, Eq, Deserialize)]
pub struct Post {
pub id: String,
pub user_id: String,
pub channel_id: String,
#[serde(default)]
pub root_id: String,
#[serde(default)]
pub message: String,
pub create_at: i64,
#[serde(default)]
pub delete_at: i64,
/// Empty for a message a user wrote; anything else is a system message.
#[serde(default, rename = "type")]
pub kind: String,
}
impl Post {
/// Every id is a Mattermost id (the root may be empty): they end up in session ids and paths.
pub(crate) fn check(self) -> Result<Post, MmError> {
// id, user_id and channel_id must be `valid_id`, and root_id empty or `valid_id`; otherwise
// Json("a post with an invalid id: <id quoted with {:?}>").
todo!()
}
}
/// An event from the WebSocket.
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum Event {
Hello,
/// A new post, and the type of its channel: `D` direct, `G` group, `O` open, `P` private.
Posted {
post: Post,
channel_type: String,
},
/// Any other event, or a reply to one of our requests, by its name (empty for a reply).
Other(String),
}
#[derive(Deserialize)]
struct RawEvent {
#[serde(default)]
event: String,
#[serde(default)]
data: serde_json::Map<String, serde_json::Value>,
}
pub(crate) fn json<T: serde::de::DeserializeOwned>(bytes: &[u8]) -> Result<T, MmError> {
// `serde_json::from_slice`; its error becomes Json(e.to_string()).
todo!()
}
/// One WebSocket text message as an event. `posted` carries the post as a JSON **string**.
pub fn parse_event(text: &str) -> Result<Event, MmError> {
// Parse into RawEvent. "hello" -> Hello. "posted" -> data.post must be a JSON *string*; parse
// that string as a Post and `check` it (a missing or non-string post is Json). channel_type is
// data.channel_type when it is a string, else "". Any other event name -> Other(name) (a reply
// has no event: Other("")).
todo!()
}
/// The WebSocket request that shows this bot as typing in a thread (`parent` is the root).
pub fn typing(seq: u64, channel: &str, parent: &str) -> String {
// serde_json::json!({"action": "user_typing", "seq": seq, "data": {"channel_id": channel,
// "parent_id": parent}}) as a string.
todo!()
}
/// What a `posts?since` call gave back.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Since {
/// Posts created after the time, not deleted, oldest first.
pub posts: Vec<Post>,
/// Mattermost's limit was reached: some posts may be missing.
pub full: bool,
}
#[derive(Deserialize)]
struct PostList {
#[serde(default)]
order: Vec<String>,
#[serde(default)]
posts: std::collections::HashMap<String, Post>,
}
/// The body of `GET /channels/{id}/posts?since=<since>`. Only ids in `order` changed after
/// `since`; `posts` also holds the roots of their threads, which may be older. Edited and deleted
/// posts come back too: only new posts count.
pub fn since_list(body: &[u8], since: i64) -> Result<Since, MmError> {
// Parse a PostList. full = order.len() >= SINCE_LIMIT. For each id in `order` (never the keys
// of `posts`), skip it if it is not in `posts`; keep the post if create_at > since and
// delete_at == 0, after `check`. Sort by (create_at, id), remove repeated ids.
todo!()
}
docs/plans/M4a/files/crates/gatewayd/src/mm/rest.rs
+95
View File
@@ -0,0 +1,95 @@
//! Mattermost's REST calls, one connection per request, with the token in the `Authorization`
//! header. A 429 waits as asked; a 5xx is tried twice more; a 401 or 403 is `Auth` at once.
use std::time::{Duration, SystemTime};
use serde::Deserialize;
use crate::config::valid_id;
use crate::http::{self, Response, rate_limit_wait};
use crate::mm::{Me, MmError, Post, Since, json, since_list};
use crate::net::Connector;
use crate::secrets::Secret;
use crate::ws::conn::host_header;
/// The pause before trying a call again after a 5xx.
pub const RETRY_5XX: Duration = Duration::from_millis(500);
/// How many times a call is tried again after a 5xx, and waits after a 429.
pub const RETRIES: u32 = 2;
/// How much of an error body is kept for the message.
const BODY_KEPT: usize = 200;
pub struct Client {
connector: Connector,
token: Secret,
timeout: Duration,
}
#[derive(Deserialize)]
struct Channel {
id: String,
}
impl Client {
/// `timeout` bounds connecting and each read: no bytes for that long is an error.
pub fn new(connector: Connector, token: Secret, timeout: Duration) -> Client {
// Store the three.
todo!()
}
pub fn connector(&self) -> &Connector {
// The Connector.
todo!()
}
pub fn token(&self) -> &Secret {
// The Secret.
todo!()
}
fn once(&self, method: &str, path: &str, body: Option<&[u8]>) -> Result<Response, MmError> {
// 1. `self.connector.connect(self.timeout)`, then `set_read_timeout(Some(self.timeout))`.
// 2. Headers: Authorization "Bearer <token>" (the only use of `expose`), Accept
// "application/json", and Content-Type "application/json" when there is a body. Host is
// `host_header(self.connector.server())`.
// 3. `http::request`. Every error on the way is Net("<method> <path>: <error>").
todo!()
}
/// One call, tried again as the module comment says; the body of a 2xx answer.
fn call(&self, method: &str, path: &str, body: Option<&[u8]>) -> Result<Vec<u8>, MmError> {
// Loop over `once`: 2xx -> the body. 401 or 403 -> Auth(status) at once. 429 -> after
// RETRIES waits already, RateLimited(wait); else sleep `rate_limit_wait(&head,
// SystemTime::now())` and try again. 5xx -> up to RETRIES more tries, sleeping RETRY_5XX
// before each. Anything else -> Status(status, the first BODY_KEPT characters of the body,
// lossy UTF-8).
todo!()
}
/// `GET /api/v4/users/me`: who this token is.
pub fn me(&self) -> Result<Me, MmError> {
// GET /api/v4/users/me into Me; an id that is not `valid_id` or an empty username is Json.
todo!()
}
/// `POST /api/v4/posts`: a post in `channel`, in the thread of `root` (empty: top level).
pub fn create_post(&self, channel: &str, root: &str, message: &str) -> Result<Post, MmError> {
// POST /api/v4/posts with {"channel_id", "root_id", "message"}; the answer is a Post,
// `check`ed.
todo!()
}
/// `GET /api/v4/channels/{channel}/posts?since=<ms>`: the posts created after `since`.
pub fn posts_since(&self, channel: &str, since: i64) -> Result<Since, MmError> {
// A channel that is not `valid_id` is Json, and nothing is sent. Otherwise GET
// /api/v4/channels/<channel>/posts?since=<since>, through `since_list`.
todo!()
}
/// `POST /api/v4/channels/direct`: the id of the direct channel between two users.
pub fn direct_channel(&self, a: &str, b: &str) -> Result<String, MmError> {
// POST /api/v4/channels/direct with the JSON array [a, b]; the answer has an "id", which
// must be `valid_id` (else Json).
todo!()
}
}
docs/plans/M4a/files/crates/gatewayd/src/net.rs
+168
View File
@@ -0,0 +1,168 @@
//! A connection to the Mattermost server: TCP, or TCP with TLS through `rustls`, verified against
//! the host's trusted certificates plus an optional CA file (M4a spec, section 5). Verification
//! cannot be turned off.
use std::io::{Read, Write};
use std::net::{TcpStream, ToSocketAddrs};
use std::path::Path;
use std::sync::Arc;
use std::time::Duration;
use rustls::pki_types::pem::PemObject;
use rustls::pki_types::{CertificateDer, ServerName};
use rustls::{ClientConfig, ClientConnection, RootCertStore, StreamOwned};
use crate::config::ServerUrl;
#[derive(Debug)]
pub enum NetError {
/// The CA file or the host's certificates could not be loaded.
Roots(String),
/// No address of the server accepted a connection.
Connect(String),
/// The TLS handshake failed: an unknown CA, a wrong name, an old protocol.
Tls(String),
}
impl std::fmt::Display for NetError {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
NetError::Roots(why) => write!(f, "cannot load trusted certificates: {why}"),
NetError::Connect(why) => write!(f, "cannot connect: {why}"),
NetError::Tls(why) => write!(f, "TLS failed: {why}"),
}
}
}
impl std::error::Error for NetError {}
/// A connected stream, plain or TLS.
pub enum Stream {
Plain(TcpStream),
Tls(Box<StreamOwned<ClientConnection, TcpStream>>),
}
impl Stream {
/// The TCP socket underneath, for timeouts and shutdown.
pub fn tcp(&self) -> &TcpStream {
// The TcpStream: itself for Plain; `get_ref()` for Tls.
todo!()
}
pub fn set_read_timeout(&self, timeout: Option<Duration>) -> std::io::Result<()> {
// On `self.tcp()`.
todo!()
}
}
impl Read for Stream {
fn read(&mut self, buf: &mut [u8]) -> std::io::Result<usize> {
// Forward to the inner stream for each variant.
todo!()
}
}
impl Write for Stream {
fn write(&mut self, buf: &[u8]) -> std::io::Result<usize> {
// Forward to the inner stream for each variant.
todo!()
}
fn flush(&mut self) -> std::io::Result<()> {
// Forward to the inner stream for each variant.
todo!()
}
}
/// Makes connections to one server.
#[derive(Clone)]
pub struct Connector {
server: ServerUrl,
tls: Option<Arc<ClientConfig>>,
}
impl Connector {
/// For `https`, loads the host's trusted certificates and `ca_file`; an error in either is an
/// error here, before any connection.
pub fn new(server: ServerUrl, ca_file: Option<&Path>) -> Result<Connector, NetError> {
// For tls, `client_config(ca_file)?` in an Arc; for plain, None.
todo!()
}
pub fn server(&self) -> &ServerUrl {
// The ServerUrl.
todo!()
}
/// Connect, and for TLS complete the handshake, within `timeout` for each step.
pub fn connect(&self, timeout: Duration) -> Result<Stream, NetError> {
let addrs = (self.server.host.as_str(), self.server.port)
.to_socket_addrs()
.map_err(|e| NetError::Connect(format!("{}: {e}", self.server.host)))?;
let mut last = format!("{} has no address", self.server.host);
let mut tcp = None;
for addr in addrs {
match TcpStream::connect_timeout(&addr, timeout) {
Ok(s) => {
tcp = Some(s);
break;
}
Err(e) => last = format!("{addr}: {e}"),
}
}
let tcp = tcp.ok_or(NetError::Connect(last))?;
tcp.set_read_timeout(Some(timeout))
.map_err(|e| NetError::Connect(e.to_string()))?;
tcp.set_write_timeout(Some(timeout))
.map_err(|e| NetError::Connect(e.to_string()))?;
let _ = tcp.set_nodelay(true);
let Some(config) = &self.tls else {
return Ok(Stream::Plain(tcp));
};
let name = ServerName::try_from(self.server.host.clone())
.map_err(|e| NetError::Tls(e.to_string()))?;
let conn = ClientConnection::new(Arc::clone(config), name)
.map_err(|e| NetError::Tls(e.to_string()))?;
let mut stream = StreamOwned::new(conn, tcp);
while stream.conn.is_handshaking() {
stream
.conn
.complete_io(&mut stream.sock)
.map_err(|e| NetError::Tls(e.to_string()))?;
}
Ok(Stream::Tls(Box::new(stream)))
}
}
fn client_config(ca_file: Option<&Path>) -> Result<ClientConfig, NetError> {
let mut roots = RootCertStore::empty();
let native = rustls_native_certs::load_native_certs();
let (added, _ignored) = roots.add_parsable_certificates(native.certs);
if let Some(path) = ca_file {
let certs: Vec<CertificateDer<'static>> = CertificateDer::pem_file_iter(path)
.map_err(|e| NetError::Roots(format!("{}: {e}", path.display())))?
.collect::<Result<_, _>>()
.map_err(|e| NetError::Roots(format!("{}: {e}", path.display())))?;
if certs.is_empty() {
return Err(NetError::Roots(format!(
"{} holds no certificate",
path.display()
)));
}
for cert in certs {
roots
.add(cert)
.map_err(|e| NetError::Roots(format!("{}: {e}", path.display())))?;
}
} else if added == 0 {
return Err(NetError::Roots(
"the host has no trusted certificates and no ca_file is set".to_string(),
));
}
let provider = Arc::new(rustls::crypto::ring::default_provider());
let config = ClientConfig::builder_with_provider(provider)
.with_safe_default_protocol_versions()
.map_err(|e| NetError::Roots(e.to_string()))?
.with_root_certificates(roots)
.with_no_client_auth();
Ok(config)
}
docs/plans/M4a/files/crates/gatewayd/src/secrets.rs
+83
View File
@@ -0,0 +1,83 @@
//! The `SecretStore`: a secret from a systemd credential, an environment variable, or an owner-only
//! file (M4a spec, section 4; the brief after P15). A `Secret` cannot be printed.
use std::ffi::OsString;
use std::os::unix::fs::{MetadataExt, PermissionsExt};
use std::path::{Path, PathBuf};
use zeroize::Zeroizing;
use crate::config::SecretSource;
pub const RUNBOOK: &str = "see docs/runbook.md#secret-unavailable";
pub const RUNBOOK_FILE: &str = "see docs/runbook.md#secret-in-a-file";
/// A secret's text. No `Display`; `Debug` shows nothing of it; wiped when dropped.
pub struct Secret(Zeroizing<String>);
impl Secret {
pub fn new(text: String) -> Secret {
Secret(Zeroizing::new(text))
}
/// The text, for the one place that must send it.
pub fn expose(&self) -> &str {
&self.0
}
}
impl std::fmt::Debug for Secret {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
f.write_str("Secret(…)")
}
}
/// A loaded secret, and the warning to print for it, if any.
#[derive(Debug)]
pub struct Loaded {
pub secret: Secret,
pub warning: Option<String>,
}
#[derive(Debug)]
pub struct SecretError {
pub name: String,
pub why: String,
}
impl std::fmt::Display for SecretError {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
write!(f, "secret {}: {}\n{RUNBOOK}", self.name, self.why)
}
}
impl std::error::Error for SecretError {}
/// Load secret `name` from `source`. `env` reads an environment variable (in `gatewayd`,
/// `std::env::var_os`); tests pass their own.
pub fn load(
name: &str,
source: &SecretSource,
env: &dyn Fn(&str) -> Option<OsString>,
) -> Result<Loaded, SecretError> {
// By source (spec section 4). Credential: read $CREDENTIALS_DIRECTORY/<name> (through `env`,
// not std::env); an unset variable, or a file that cannot be read, is an error. Env: the
// variable, UTF-8; unset is an error. File: `check_file` first, then read it, and set `warning`
// to the exact text in the task. Every value goes through `value`. Every error is a
// `SecretError` naming the secret, never the value.
todo!()
}
/// An owner-only regular file, not a link.
fn check_file(path: &Path) -> Result<(), String> {
// In this order, each its own error: not absolute; `symlink_metadata` fails; a symbolic link;
// not a regular file; owner uid differs from the uid of /proc/self; mode & 0o077 != 0.
todo!()
}
/// The text without one trailing newline; not empty; UTF-8.
fn value(bytes: Vec<u8>) -> Result<Secret, String> {
// UTF-8 (else an error), one trailing newline removed, not empty. Keep it in `Zeroizing`
// throughout.
todo!()
}
docs/plans/M4a/files/crates/gatewayd/src/serve/handle.rs
+78
View File
@@ -0,0 +1,78 @@
//! What the event loop does with a post, a finished turn, and the time between: routing, starting
//! turns, typing, and catching up after a gap.
use std::sync::Arc;
use std::time::{SystemTime, UNIX_EPOCH};
use crate::deliver::{LOOP_DOWN, deliver};
use crate::mm::{Post, typing};
use crate::serve::{Gateway, Stop};
use crate::sessions::{BUSY, Batch, Ignored, Pushed, Route};
use crate::state::InFlight;
use crate::ws::WsError;
use crate::ws::conn::Ws;
/// Now, in Mattermost's milliseconds.
fn now_ms() -> i64 {
// Milliseconds since the Unix epoch as i64, with try_from (i64::MAX if it does not fit).
todo!()
}
impl Gateway {
/// Post in a thread; a failure is logged, not fatal.
pub(super) fn post(&self, channel: &str, root: &str, text: &str) {
// `self.client.create_post`; an error is logged, "gatewayd: cannot post in <channel>
// (thread <root>): <error>".
todo!()
}
/// Is this channel one whose posts `gatewayd` keeps track of?
fn tracked(&self, channel: &str, channel_type: &str) -> bool {
// channel_type "D", or the channel is in allow.channels.
todo!()
}
/// One new post, live or caught up.
pub(super) fn handle_post(&mut self, post: &Post, channel_type: &str) -> Result<(), Stop> {
// 1. Not tracked, or seen -> nothing. 2. `state.handled(...)?` before anything else. 3.
// Route it: NotAllowed -> log "gatewayd: ignored post <id> from <user>: not allowed"
// (never the message); other Ignore -> nothing; Reply -> post it; Queue -> join the
// thread when joins_thread, then push: Start -> `start`, Waiting -> nothing, Full ->
// post BUSY.
todo!()
}
/// Record the turn as in flight and run it on its own thread.
fn start(&mut self, batch: Batch) -> Result<(), Stop> {
// `state.start_turn` (session, channel, root). Spawn with std::thread::Builder: `deliver`
// with the client, the loop socket, the batch and the log, then send the session on
// done_tx. If spawning fails: log "gatewayd: cannot start a thread for <session>: <e>",
// post LOOP_DOWN in the thread, and send the session on done_tx.
todo!()
}
/// Turns that ended: out of flight, and the next batch of each session started.
pub(super) fn finished(&mut self) -> Result<(), Stop> {
// For each session on done_rx (try_recv, never blocking): `end_turn`, then `queues.finish`;
// a batch it returns is started.
todo!()
}
/// Show this bot as typing in every thread with a turn running.
pub(super) fn typing(&mut self, ws: &mut Ws) -> Result<(), WsError> {
// For each of `queues.threads()`: seq += 1, send `typing(seq, channel, root)` as text.
todo!()
}
/// Answer what arrived while `gatewayd` was away: the direct channel with each allowed user,
/// and each allowed channel. A channel seen for the first time starts from now.
pub(super) fn catch_up(&mut self) -> Result<(), Stop> {
// Channels: the direct channel with each allowed user ("D"; an error is logged "gatewayd:
// no direct channel with <user>: <e>" and skipped), then each allowed channel ("O"). For
// each: no mark -> `mark(channel, now_ms())` and skip; else `posts_since(channel, mark)`
// (an error is logged "gatewayd: cannot catch up <channel>: <e>" and skipped); when full,
// log "gatewayd: <channel>: too many posts to catch up; some may be missed"; `handle_post`
// each post in order.
todo!()
}
}
docs/plans/M4a/files/crates/gatewayd/src/serve/mod.rs
+232
View File
@@ -0,0 +1,232 @@
//! Startup, the event loop and reconnecting (M4a spec, section 9). `run` returns only when
//! `gatewayd` must stop: a refused token, a state file it cannot keep, or a stop asked by a test.
mod handle;
use std::fs::File;
use std::path::PathBuf;
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::mpsc::{self, Receiver, Sender};
use std::time::{Duration, Instant};
use proto::SessionId;
use crate::config::Config;
use crate::mm::rest::Client;
use crate::mm::{Event, Me, MmError, parse_event};
use crate::net::Connector;
use crate::secrets::Secret;
use crate::sessions::{Queues, Router};
use crate::state::{State, StateError};
use crate::ws::conn::{Timing, Ws};
pub const UNREACHABLE: &str = "see docs/runbook.md#mattermost-unreachable";
pub const AUTH_FAILED: &str = "see docs/runbook.md#mattermost-auth-failed";
pub const START_FAILED: &str = "see docs/runbook.md#gatewayd-start-failed";
pub const INTERRUPTED: &str =
"interrupted: gatewayd restarted before the answer arrived; ask again";
/// A log line: `stderr` in `main`, a record in tests.
pub type Log = Arc<dyn Fn(&str) + Send + Sync>;
/// Why `run` returned.
#[derive(Debug)]
pub enum Stop {
/// Mattermost refused the token (401 or 403).
Auth(u16),
State(StateError),
/// Something `gatewayd` needs at start is missing; the message names it.
Start(String),
/// The stop flag was set.
Asked,
}
impl std::fmt::Display for Stop {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
Stop::Auth(status) => {
write!(
f,
"gatewayd: Mattermost refused the token ({status})\n{AUTH_FAILED}"
)
}
Stop::State(e) => write!(f, "gatewayd: {e}"),
Stop::Start(why) => write!(f, "gatewayd: {why}\n{START_FAILED}"),
Stop::Asked => write!(f, "gatewayd: stopped"),
}
}
}
impl From<StateError> for Stop {
fn from(e: StateError) -> Stop {
// Stop::State(e).
todo!()
}
}
/// Timings a test shortens.
#[derive(Debug, Clone)]
pub struct Tuning {
/// The waits between connection attempts; the last repeats.
pub backoff: Vec<Duration>,
/// How long one wait for a WebSocket message lasts, at most.
pub poll: Duration,
/// Connecting and each REST read.
pub rest_timeout: Duration,
}
impl Default for Tuning {
fn default() -> Self {
let secs = [1, 2, 5, 10, 30].map(Duration::from_secs);
Tuning {
backoff: secs.to_vec(),
poll: Duration::from_millis(200),
rest_timeout: Duration::from_secs(30),
}
}
}
/// Everything the event loop works with.
pub(crate) struct Gateway {
config: Config,
client: Arc<Client>,
router: Router,
me: Me,
state: State,
queues: Queues,
loop_socket: PathBuf,
done_tx: Sender<SessionId>,
done_rx: Receiver<SessionId>,
log: Log,
/// The typing requests' sequence number.
seq: u64,
/// The turns a restart cut off have been answered.
restarted: bool,
}
/// The wait before attempt `n` (from 0) after a loss.
pub fn backoff(tuning: &Tuning, n: usize) -> Duration {
// tuning.backoff[n], or its last entry when n is past the end (30 s if the list is empty). No
// indexing.
todo!()
}
/// Sleep for `d`, in short steps, unless `stop` is set.
fn sleep_unless(stop: &AtomicBool, d: Duration) {
// Sleep in steps of at most 20 ms until `d` has passed or `stop` is set.
todo!()
}
/// Run `gatewayd` with its token, until it must stop.
pub fn run(config: Config, token: Secret, tuning: Tuning, log: Log, stop: &AtomicBool) -> Stop {
let server = match config.server() {
Ok(s) => s,
Err(why) => return Stop::Start(why),
};
let connector = match Connector::new(server, config.mattermost.ca_file.as_deref()) {
Ok(c) => c,
Err(e) => return Stop::Start(e.to_string()),
};
let state = match State::load(&config.state_path()) {
Ok(s) => s,
Err(e) => return Stop::State(e),
};
let client = Arc::new(Client::new(connector, token, tuning.rest_timeout));
let mut g = Gateway::new(config, client, state, log);
let mut failures = 0;
loop {
if stop.load(Ordering::SeqCst) {
return Stop::Asked;
}
let stopped = match connect(&g.config, &g.client, &tuning) {
Ok((me, mut ws)) => {
failures = 0;
g.connected(me)
.and_then(|()| g.catch_up())
.and_then(|()| g.event_loop(&mut ws, &tuning, stop))
}
Err(MmError::Auth(status)) => Err(Stop::Auth(status)),
Err(e) => {
let wait = backoff(&tuning, failures);
failures += 1;
(g.log)(&format!(
"gatewayd: cannot reach {}: {e}; trying again in {} s\n{UNREACHABLE}",
g.config.mattermost.url,
wait.as_secs()
));
sleep_unless(stop, wait);
Ok(())
}
};
if let Err(stop) = stopped {
return stop;
}
}
}
/// Who we are, and a WebSocket that has said hello.
fn connect(config: &Config, client: &Client, tuning: &Tuning) -> Result<(Me, Ws), MmError> {
// 1. `client.me()?`. 2. Timing from limits (ping_every_ms, dead_after_ms). 3. Open /dev/urandom
// (an error is Net("/dev/urandom: <e>")). 4. `Ws::open(client.connector(),
// client.token().expose(), timing, ..)`.
// 5. Poll with tuning.poll until a text that `parse_event`s to Hello, for at most dead_after.
// Every WebSocket error, and no hello in time, is Net("websocket: <why>").
todo!()
}
impl Gateway {
fn new(config: Config, client: Arc<Client>, state: State, log: Log) -> Gateway {
// A done channel, a Router with empty ids (set on connect), an empty Me, loop_socket from
// config, Queues with limit limits.queue (usize::try_from), seq 0, restarted false.
todo!()
}
/// A connection is up: say so, route as this user, and on the first one, answer the turns a
/// restart cut off.
fn connected(&mut self, me: Me) -> Result<(), Stop> {
// Log exactly "gatewayd: connected to <url> as <username>". A new Router from me and the
// allow lists; store me. The first time only (`restarted`): for each turn `take_in_flight`
// gives back, post INTERRUPTED in its channel and root. A later reconnect must not: those
// turns are still running.
todo!()
}
/// Read events until the connection is lost (`Ok`) or `gatewayd` must stop.
fn event_loop(&mut self, ws: &mut Ws, tuning: &Tuning, stop: &AtomicBool) -> Result<(), Stop> {
let typing_every = Duration::from_millis(self.config.limits.typing_every_ms);
let mut last_typing = Instant::now();
loop {
if stop.load(Ordering::SeqCst) {
return Err(Stop::Asked);
}
self.finished()?;
if last_typing.elapsed() >= typing_every {
last_typing = Instant::now();
if let Err(e) = self.typing(ws) {
(self.log)(&format!(
"gatewayd: lost the connection: {e}\n{UNREACHABLE}"
));
return Ok(());
}
}
let text = match ws.poll(tuning.poll.min(typing_every)) {
Ok(Some(text)) => text,
Ok(None) => continue,
Err(e) => {
(self.log)(&format!(
"gatewayd: lost the connection: {e}\n{UNREACHABLE}"
));
return Ok(());
}
};
match parse_event(&text) {
Ok(Event::Posted { post, channel_type }) => {
self.handle_post(&post, &channel_type)?
}
Ok(_) => {}
Err(e) => (self.log)(&format!("gatewayd: ignored an event: {e}")),
}
}
}
}
docs/plans/M4a/files/crates/gatewayd/src/sessions.rs
+162
View File
@@ -0,0 +1,162 @@
//! Which posts `gatewayd` acts on, which session each belongs to, commands, and the queue of
//! messages per session (M4a spec, section 7). Pure: the state file and the network are elsewhere.
use std::collections::{BTreeSet, HashMap};
use proto::SessionId;
use crate::mm::Post;
pub const M4B_COMMAND: &str = "approvals over Mattermost arrive in M4b; use `bxctl approvals`";
pub const UNKNOWN_COMMAND: &str = "unknown command; the commands are !approve and !deny";
pub const BUSY: &str = "busy: too many messages are waiting in this conversation";
/// Names that name nobody: every agent in the channel would answer them.
const EVERYONE: [&str; 3] = ["channel", "here", "all"];
/// Why a post was not acted on. Only `NotAllowed` is logged, by user id and post id.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum Ignored {
Own,
System,
NotAllowed,
NotForUs,
}
/// Where an answer goes: a channel, and the root of the thread in it.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Thread {
pub channel: String,
pub root: String,
}
/// A message for a session.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Message {
pub session: SessionId,
pub thread: Thread,
/// A reply in a thread: the session should exist already.
pub resume: bool,
pub text: String,
/// A thread in a channel or group message that this Boxmaker now takes part in.
pub joins_thread: bool,
}
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum Route {
Ignore(Ignored),
/// Answer in the thread without a turn (a command).
Reply {
thread: Thread,
text: String,
},
Queue(Message),
}
pub struct Router {
me_id: String,
me_name: String,
users: BTreeSet<String>,
channels: BTreeSet<String>,
}
/// Every `@name` in a message, lower-cased: `a-z`, `0-9`, `.`, `-` and `_` after an `@`, without
/// trailing dots.
pub fn named(message: &str) -> Vec<String> {
// Find each "@". The name after it is the longest run of ASCII letters, digits, ".", "-" and
// "_", with trailing "." removed and lower-cased. Skip empty names. Continue after the name.
todo!()
}
impl Router {
pub fn new(me_id: &str, me_name: &str, users: &[String], channels: &[String]) -> Router {
// Store the ids, the username lower-cased, and the two lists as sets.
todo!()
}
/// Is a post in a channel or group message for this Boxmaker? `known` says whether it has a
/// session for a thread root.
fn for_us(&self, post: &Post, known: &dyn Fn(&str) -> bool) -> bool {
// Named this bot -> true. Otherwise true only for a reply (root_id not empty) in a known
// thread (`known(root_id)`) that names nobody but channel, here or all.
todo!()
}
/// What to do with a new post (a post seen before is dropped by the caller first).
pub fn route(&self, post: &Post, channel_type: &str, known: &dyn Fn(&str) -> bool) -> Route {
// Section 7 of the spec, in this order: own post -> Ignore(Own); kind not empty ->
// Ignore(System); user not allowed -> Ignore(NotAllowed); then the channel: "D" is always
// ours; "O", "P" or "G" is ours when its id is allowed and `for_us`; anything else ->
// Ignore(NotForUs). The thread root is root_id, or the post id when root_id is empty. Then
// commands: "!!..." drops one "!" and goes on as a message; "!" then a first word approve
// or deny -> Reply M4B_COMMAND; any other "!" -> Reply UNKNOWN_COMMAND. Then Queue: session
// "mm-<root>" (if SessionId::new fails, Ignore(NotForUs)), resume = root_id not empty,
// joins_thread = not "D".
todo!()
}
}
/// A turn to send: every message that was waiting, joined with a blank line.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Batch {
pub session: SessionId,
pub thread: Thread,
pub resume: bool,
pub text: String,
}
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum Pushed {
/// No turn was running: send this one now.
Start(Batch),
/// A turn is running; the message waits for the next.
Waiting,
/// Too many are waiting: the message is dropped, answer `BUSY`.
Full(Thread),
}
struct Pending {
thread: Thread,
resume: bool,
waiting: Vec<String>,
}
/// The sessions with a turn running, and the messages waiting for each.
pub struct Queues {
limit: usize,
running: HashMap<SessionId, Pending>,
}
impl Queues {
/// `limit` is the most messages that may wait per session.
pub fn new(limit: usize) -> Queues {
// An empty map.
todo!()
}
pub fn push(&mut self, message: Message) -> Pushed {
// If the session is running: with `limit` messages already waiting -> Full(thread); else
// add the text to waiting -> Waiting. Otherwise insert it as running (its later batches
// resume: true) and return Start with this message alone.
todo!()
}
/// A session's turn ended: the next batch, or `None`, and the session is no longer running.
pub fn finish(&mut self, session: &SessionId) -> Option<Batch> {
// Not running -> None. Nothing waiting -> remove it, None. Otherwise take every waiting
// text, joined with "\n\n", as the next Batch (it stays running).
todo!()
}
/// How many sessions have a turn running.
pub fn running(&self) -> usize {
// How many sessions are running.
todo!()
}
/// The threads with a turn running, for showing this bot as typing in them.
pub fn threads(&self) -> Vec<Thread> {
// The thread of every running session.
todo!()
}
}
docs/plans/M4a/files/crates/gatewayd/src/state.rs
+158
View File
@@ -0,0 +1,158 @@
//! `<home>/gateway/state.json`: what `gatewayd` has handled, the threads it takes part in, and the
//! turns in flight (M4a spec, section 9). Written atomically after every change. A file that
//! cannot be read, parsed or written stops `gatewayd`: guessing would answer posts twice.
use std::collections::BTreeMap;
use std::fs;
use std::io::{self, Write};
use std::os::unix::fs::{DirBuilderExt, OpenOptionsExt};
use std::path::{Path, PathBuf};
use serde::{Deserialize, Serialize};
use crate::config::valid_id;
pub const RECENT_KEPT: usize = 500;
pub const THREADS_KEPT: usize = 5000;
pub const RUNBOOK: &str = "see docs/runbook.md#gateway-state-damaged";
#[derive(Debug)]
pub enum StateError {
Read(PathBuf, String),
Write(PathBuf, io::Error),
}
impl std::fmt::Display for StateError {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
StateError::Read(path, why) => write!(f, "{}: {why}\n{RUNBOOK}", path.display()),
StateError::Write(path, err) => {
write!(f, "{}: cannot write: {err}\n{RUNBOOK}", path.display())
}
}
}
}
impl std::error::Error for StateError {}
/// A turn sent to `loopd` and not yet answered.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct InFlight {
pub session: String,
pub channel: String,
pub root: String,
}
#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)]
#[serde(deny_unknown_fields)]
struct StateFile {
channels: BTreeMap<String, i64>,
recent: Vec<String>,
threads: Vec<String>,
in_flight: Vec<InFlight>,
}
impl StateFile {
/// Every id is a Mattermost id; a session is `mm-<id>`.
fn problem(&self) -> Option<String> {
// Every key of channels, every entry of recent and threads must be `valid_id` ("not a
// Mattermost id: <id with {:?}>"). Each in_flight entry: session is "mm-" + a valid id,
// channel and root valid ids ("a turn in flight is not valid: <session with {:?}>"). None
// when all are good.
todo!()
}
}
pub struct State {
path: PathBuf,
file: StateFile,
}
impl State {
/// Read the state; a missing file is a first start.
pub fn load(path: &Path) -> Result<State, StateError> {
// Read the file. Missing (NotFound only) -> an empty StateFile. Any other read error, a
// parse error (serde_json), or `problem()` -> Read(path, why).
todo!()
}
/// Write the state to its path atomically, in six steps; an error leaves the old file.
fn save(&self) -> Result<(), StateError> {
// `persist`, its error as Write(path, error).
todo!()
}
fn persist(&self) -> io::Result<()> {
// The six steps of brokerd/src/state.rs, persist: 1. the parent directory, recursive, 0700;
// 2. serde_json::to_string plus "\n"; 3. "<path>.tmp" (with_extension("json.tmp")), create
// + truncate, mode 0600; 4. write_all and sync_all; 5. rename over the path; 6. open the
// directory and sync_all.
todo!()
}
/// Was this post handled already?
pub fn seen(&self, post_id: &str) -> bool {
// Is the id in recent?
todo!()
}
/// A post was handled (acted on or ignored): remember its id and move its channel's mark.
pub fn handled(
&mut self,
post_id: &str,
channel: &str,
create_at: i64,
) -> Result<(), StateError> {
// If not seen: push the id to recent, then drop the oldest beyond RECENT_KEPT. The channel
// mark becomes the larger of its old value and create_at (a new channel starts at
// create_at). Save.
todo!()
}
/// The `create_at` of the last post handled in a channel, if any.
pub fn since(&self, channel: &str) -> Option<i64> {
// The mark of the channel.
todo!()
}
/// The channels with a mark, for catching up.
pub fn channels(&self) -> Vec<String> {
// The channel ids that have a mark.
todo!()
}
/// Start catching up a channel from `at` (milliseconds), if it has no mark yet.
pub fn mark(&mut self, channel: &str, at: i64) -> Result<(), StateError> {
// A channel that has a mark is left alone (nothing saved). Otherwise set it to `at` and
// save.
todo!()
}
pub fn knows_thread(&self, root: &str) -> bool {
// Is the root in threads?
todo!()
}
/// This Boxmaker takes part in a thread; the newest `THREADS_KEPT` are kept.
pub fn join_thread(&mut self, root: &str) -> Result<(), StateError> {
// Known -> nothing. Else push it, drop the oldest beyond THREADS_KEPT, save.
todo!()
}
pub fn start_turn(&mut self, turn: InFlight) -> Result<(), StateError> {
// Remove any entry of the same session, push this one, save.
todo!()
}
pub fn end_turn(&mut self, session: &str) -> Result<(), StateError> {
// Remove the entries of the session, save.
todo!()
}
/// The turns left in flight by the last run, removed from the state.
pub fn take_in_flight(&mut self) -> Result<Vec<InFlight>, StateError> {
// Take the whole list out (std::mem::take); save only when it was not empty.
todo!()
}
}
docs/plans/M4a/files/crates/gatewayd/src/ws/conn.rs
+89
View File
@@ -0,0 +1,89 @@
//! One WebSocket connection to Mattermost (M4a spec, section 6): open it, send text, and poll for
//! the next text message while answering pings, sending our own, and noticing a dead peer.
use std::io::{Read, Write};
use std::time::{Duration, Instant};
use crate::config::ServerUrl;
use crate::net::{Connector, Stream};
use crate::ws::WsError;
use crate::ws::frame::{CLOSE, Decoder, Incoming, PING, PONG, TEXT, encode};
use crate::ws::handshake::handshake;
/// Mattermost's WebSocket path.
pub const PATH: &str = "/api/v4/websocket";
/// How often we ping, and how long silence may last.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct Timing {
pub ping_every: Duration,
pub dead_after: Duration,
}
pub struct Ws {
stream: Stream,
decoder: Decoder,
random: Box<dyn Read + Send>,
timing: Timing,
last_heard: Instant,
last_ping: Instant,
}
impl Ws {
/// Connect, and complete the handshake with `token`. `random` supplies the key and every mask
/// (in `gatewayd`, `/dev/urandom`).
pub fn open(
connector: &Connector,
token: &str,
timing: Timing,
mut random: Box<dyn Read + Send>,
) -> Result<Ws, WsError> {
// 1. `connector.connect(timing.dead_after)`; its error becomes
// WsError::Handshake(e.to_string()).
// 2. `handshake(&mut stream, &host_header(connector.server()), PATH, token, &mut random)?`.
// 3. A Ws with a new Decoder, and last_heard and last_ping both now.
todo!()
}
fn send(&mut self, opcode: u8, payload: &[u8]) -> Result<(), WsError> {
// 4 mask bytes from `random` (read_exact), then write `encode(opcode, payload, mask)` and
// flush.
todo!()
}
pub fn send_text(&mut self, text: &str) -> Result<(), WsError> {
// `send` with TEXT.
todo!()
}
/// The next text message, or `None` after about `wait` with none. Pings are answered and sent
/// here; a close frame is answered and ends the connection (`Closed`); silence past the
/// dead-after limit is `Dead`.
pub fn poll(&mut self, wait: Duration) -> Result<Option<String>, WsError> {
// Loop, with `until = now + wait`:
// 1. Every whole message the decoder has: Text -> return it; Ping(p) -> send PONG with p;
// Pong -> nothing; Close(code, _) -> send CLOSE with the code as 2 bytes (or empty),
// ignore that error, return Closed.
// 2. Silence since last_heard >= dead_after -> Dead.
// 3. Since last_ping >= ping_every -> send an empty PING, last_ping = now.
// 4. now >= until -> Ok(None).
// 5. Read timeout: the least of (until - now), (last_ping + ping_every - now) and
// (last_heard + dead_after - now), at least 1 ms. Read into a 16 KiB buffer: 0 bytes ->
// Closed; n bytes -> feed them, last_heard = now; WouldBlock, TimedOut or Interrupted ->
// go round; any other error -> Io.
todo!()
}
/// Send a close frame, best effort, and drop the connection.
pub fn close(mut self) {
// Send CLOSE with 1000 as 2 big-endian bytes; ignore the error.
todo!()
}
}
/// The `Host` header for a server: the port is written only when it is not the scheme's default.
pub fn host_header(server: &ServerUrl) -> String {
// The host alone when the port is the default for the scheme (443 for tls, 80 otherwise), else
// "host:port".
todo!()
}
docs/plans/M4a/files/crates/gatewayd/src/ws/frame.rs
+91
View File
@@ -0,0 +1,91 @@
//! WebSocket frames (RFC 6455, section 5), without I/O. `Decoder` is fed the bytes as they arrive,
//! in any pieces, and yields whole messages; `encode` builds our masked frames. Everything the
//! server sends is untrusted: every length is checked before anything is allocated.
use crate::ws::WsError;
/// The largest message we accept, counted from the length fields.
pub const MAX_MESSAGE: usize = 1 << 20;
pub const CONTINUATION: u8 = 0x0;
pub const TEXT: u8 = 0x1;
pub const CLOSE: u8 = 0x8;
pub const PING: u8 = 0x9;
pub const PONG: u8 = 0xA;
/// A whole message from the server.
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum Incoming {
Text(String),
Ping(Vec<u8>),
Pong(Vec<u8>),
/// A close frame: the status code if there is one, and the reason.
Close(Option<u16>, String),
}
/// Reassembles frames into messages.
#[derive(Debug, Default)]
pub struct Decoder {
buf: Vec<u8>,
/// A text message whose first frame has come and whose last has not.
partial: Option<Vec<u8>>,
}
/// A parsed header: what it says and how long it is.
struct Header {
fin: bool,
opcode: u8,
header_len: usize,
payload_len: usize,
}
impl Decoder {
pub fn new() -> Decoder {
Decoder::default()
}
/// Append bytes as they arrived.
pub fn feed(&mut self, bytes: &[u8]) {
self.buf.extend_from_slice(bytes);
}
/// The next whole message, `None` if more bytes are needed, or the error that ends the
/// connection. After an error, do not call again.
pub fn next_message(&mut self) -> Result<Option<Incoming>, WsError> {
// Loop: `self.header()?`, None -> Ok(None). If the buffer holds less than header_len +
// payload_len, Ok(None). Otherwise take the payload out and drain the frame from the
// buffer, then by opcode: PING -> Ping, PONG -> Pong, CLOSE -> `close(&payload)`. TEXT
// starts a new message, CONTINUATION extends `partial`; with fin the whole message must be
// UTF-8 (else Protocol) and is returned as Text; without fin it is kept in `partial` and
// the loop goes on. Any other opcode is Protocol.
todo!()
}
/// The next frame's header, once it is all here, checked against every rule that does not need
/// the payload.
fn header(&self) -> Result<Option<Header>, WsError> {
// Return Ok(None) while the bytes the header needs have not all arrived. The rules, each an
// error: a reserved bit (0x70 of byte 0) is Protocol; the mask bit (0x80 of byte 1) is
// Protocol; an opcode that is not CONTINUATION, TEXT, CLOSE, PING or PONG is Protocol.
// Length 126: a u16 in the next 2 bytes, below 126 is Protocol (not the shortest form).
// 127: a u64 in the next 8 bytes; top bit set is Protocol; <= 0xFFFF is Protocol. Control
// frames (opcode & 0x8): not fin, or over 125 bytes, is Protocol. Data frames: TEXT while
// `partial` is Some, or CONTINUATION while it is None, is Protocol; a payload over
// MAX_MESSAGE minus what `partial` holds is TooLarge. All of it before any allocation.
todo!()
}
}
fn close(payload: &[u8]) -> Result<Incoming, WsError> {
// Empty: Close(None, ""). One byte: Protocol. Otherwise a big-endian u16 code and a UTF-8
// reason (else Protocol).
todo!()
}
/// One whole frame from us: FIN set, masked with `mask`.
pub fn encode(opcode: u8, payload: &[u8], mask: [u8; 4]) -> Vec<u8> {
// Byte 0: 0x80 | opcode. Byte 1: 0x80 | length, where the length is the 7-bit form below 126,
// 126 then a u16 up to 0xFFFF, else 127 then a u64. Then the 4 mask bytes, then each payload
// byte XOR mask[i % 4]. No `as` casts: use try_from.
todo!()
}
docs/plans/M4a/files/crates/gatewayd/src/ws/handshake.rs
+60
View File
@@ -0,0 +1,60 @@
//! The opening handshake (RFC 6455, section 4.1) and the base64 it needs.
use std::io::{Read, Write};
use crate::http::{Head, read_head};
use crate::ws::WsError;
/// RFC 6455's magic string, appended to the key before hashing.
pub const GUID: &str = "258EAFA5-E914-47DA-95CA-C5AB0DC85B11";
const ALPHABET: &[u8; 64] = b"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/";
/// Standard base64 with padding (RFC 4648, section 4).
pub fn base64(bytes: &[u8]) -> String {
// Standard alphabet (ALPHABET), "=" padding: each 3 bytes become 4 characters; a last group of
// 1 or 2 bytes becomes 2 or 3 characters and 2 or 1 "=". No indexing that can go out of bounds.
todo!()
}
/// The `Sec-WebSocket-Accept` a server must send for `key`.
pub fn accept_for(key: &str) -> String {
// base64(sha1(key + GUID)), with `proto::sha1::sha1`.
todo!()
}
/// A fresh key: 16 bytes from `random` (in `gatewayd`, `/dev/urandom`), in base64.
pub fn new_key(random: &mut dyn Read) -> std::io::Result<String> {
// 16 bytes read from `random` with read_exact, then base64.
todo!()
}
/// The request, exactly.
pub fn request_text(host: &str, path: &str, key: &str, token: &str) -> String {
format!(
"GET {path} HTTP/1.1\r\nHost: {host}\r\nUpgrade: websocket\r\nConnection: Upgrade\r\n\
Sec-WebSocket-Key: {key}\r\nSec-WebSocket-Version: 13\r\nAuthorization: Bearer {token}\r\n\r\n"
)
}
/// Is `head` a server's acceptance of `key`? Status 101, `Upgrade: websocket`, a `Connection`
/// holding the token `upgrade`, and the right `Sec-WebSocket-Accept` (case matters there).
pub fn check_response(head: &Head, key: &str) -> Result<(), WsError> {
// In this order, each a Handshake error: status is not 101 ("status <n>"); no Upgrade header
// equal to "websocket" ignoring case; no Connection header with a comma-separated token equal
// to "upgrade" ignoring case; Sec-WebSocket-Accept missing, or not exactly `accept_for(key)`.
todo!()
}
/// The whole handshake on `stream`. Nothing after the server's head is read.
pub fn handshake(
stream: &mut (impl Read + Write),
host: &str,
path: &str,
token: &str,
random: &mut dyn Read,
) -> Result<(), WsError> {
// A new key; write `request_text` and flush; `read_head` (its error is a Handshake error); then
// `check_response`. Read nothing after the head.
todo!()
}
docs/plans/M4a/files/crates/gatewayd/src/ws/mod.rs
+42
View File
@@ -0,0 +1,42 @@
//! The WebSocket client (RFC 6455; M4a spec, section 6): the handshake, the frame codec, and the
//! connection that uses them.
pub mod handshake;
/// Why a WebSocket ended or could not start. Every one of these ends the connection; `gatewayd`
/// then reconnects.
#[derive(Debug)]
pub enum WsError {
/// The server's answer to the handshake was not an upgrade to a WebSocket.
Handshake(String),
/// A frame broke the protocol.
Protocol(String),
/// A message over `MAX_MESSAGE`, refused from its length fields.
TooLarge,
/// The server closed the connection (a close frame, or the end of the stream).
Closed,
/// Nothing was heard for the dead-after limit.
Dead,
Io(std::io::Error),
}
impl std::fmt::Display for WsError {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
WsError::Handshake(why) => write!(f, "WebSocket handshake failed: {why}"),
WsError::Protocol(why) => write!(f, "WebSocket protocol error: {why}"),
WsError::TooLarge => write!(f, "WebSocket message too large"),
WsError::Closed => write!(f, "WebSocket closed"),
WsError::Dead => write!(f, "WebSocket silent for too long"),
WsError::Io(e) => write!(f, "WebSocket I/O: {e}"),
}
}
}
impl std::error::Error for WsError {}
impl From<std::io::Error> for WsError {
fn from(e: std::io::Error) -> Self {
WsError::Io(e)
}
}
docs/plans/M4a/files/crates/gatewayd/tests/config.rs
+246
View File
@@ -0,0 +1,246 @@
//! `gatewayd.toml` (M4a spec, section 3). Do not edit.
#[path = "support/tmp.rs"]
mod tmp;
use std::path::PathBuf;
use gatewayd::config::{Config, ConfigError, SecretSource, ServerUrl, parse_url, valid_id};
use tmp::TempDir;
const OWNER: &str = "abcdefghijklmnopqrstuvwxyz";
const CHANNEL: &str = "0123456789abcdefghijklmnop";
fn minimal() -> String {
format!(
"[mattermost]\nurl = \"https://straylight.scylla-hammerhead.ts.net\"\n\
[secrets.mattermost_token]\ncredential = \"mattermost-token\"\n\
[allow]\nusers = [\"{OWNER}\"]\n"
)
}
fn load(text: &str) -> Result<Config, ConfigError> {
let dir = TempDir::new("cfg");
let path = dir.write("gatewayd.toml", text);
Config::load(&path)
}
fn invalid(text: &str) -> String {
match load(text) {
Err(ConfigError::Invalid(_, why)) => why,
other => panic!("expected Invalid for {text:?}, got {other:?}"),
}
}
#[test]
fn a_minimal_config_gets_every_default() {
let c = load(&minimal()).unwrap();
assert_eq!(
c.server().unwrap(),
ServerUrl {
tls: true,
host: "straylight.scylla-hammerhead.ts.net".to_string(),
port: 443
}
);
assert_eq!(c.mattermost.ca_file, None);
assert_eq!(
c.token_source().unwrap(),
SecretSource::Credential("mattermost-token".to_string())
);
assert_eq!(c.allow.users, vec![OWNER.to_string()]);
assert!(c.allow.channels.is_empty());
assert_eq!((c.limits.queue, c.limits.typing_every_ms), (20, 3_000));
assert_eq!(
(c.limits.ping_every_ms, c.limits.dead_after_ms),
(30_000, 60_000)
);
let home = std::env::var_os("BOXMAKER_HOME")
.map(PathBuf::from)
.unwrap_or_else(|| PathBuf::from("/var/lib/boxmaker"));
assert_eq!(c.loop_socket(), home.join("run/loop/loop.sock"));
assert_eq!(c.state_path(), home.join("gateway/state.json"));
}
#[test]
fn every_value_can_be_set() {
let text = format!(
"[mattermost]\nurl = \"http://127.0.0.1:8065\"\nca_file = \"/etc/boxmaker/ca.pem\"\n\
[secrets.mattermost_token]\nfile = \"/home/k/.config/boxmaker/token\"\n\
[allow]\nusers = [\"{OWNER}\"]\nchannels = [\"{CHANNEL}\"]\n\
[loop]\nsocket = \"/run/l.sock\"\n[paths]\nhome = \"/h\"\n\
[limits]\nqueue = 5\ntyping_every_ms = 1\nping_every_ms = 2\ndead_after_ms = 3\n"
);
let c = load(&text).unwrap();
assert_eq!(
c.server().unwrap(),
ServerUrl {
tls: false,
host: "127.0.0.1".to_string(),
port: 8065
}
);
assert_eq!(
c.mattermost.ca_file,
Some(PathBuf::from("/etc/boxmaker/ca.pem"))
);
assert_eq!(
c.token_source().unwrap(),
SecretSource::File(PathBuf::from("/home/k/.config/boxmaker/token"))
);
assert_eq!(c.allow.channels, vec![CHANNEL.to_string()]);
assert_eq!(c.loop_socket(), PathBuf::from("/run/l.sock"));
assert_eq!(c.state_path(), PathBuf::from("/h/gateway/state.json"));
assert_eq!(
(
c.limits.queue,
c.limits.typing_every_ms,
c.limits.ping_every_ms,
c.limits.dead_after_ms
),
(5, 1, 2, 3)
);
}
#[test]
fn an_env_secret() {
let text = minimal().replace(
"credential = \"mattermost-token\"",
"env = \"BOXMAKER_MM_TOKEN\"",
);
assert_eq!(
load(&text).unwrap().token_source().unwrap(),
SecretSource::Env("BOXMAKER_MM_TOKEN".to_string())
);
}
#[test]
fn urls() {
for (url, tls, host, port) in [
("https://a.example", true, "a.example", 443),
("https://a.example:8443", true, "a.example", 8443),
("http://localhost", false, "localhost", 80),
("http://127.0.0.1:8065", false, "127.0.0.1", 8065),
] {
assert_eq!(
parse_url(url),
Ok(ServerUrl {
tls,
host: host.to_string(),
port
}),
"{url}"
);
}
for url in [
"",
"a.example",
"ftp://a.example",
"https://",
"https://a.example/",
"https://a.example/api",
"https://A.example",
"https://a.example:0",
"https://a.example:65536",
"https://a.example:0443",
"https://a.example:",
"https://user@a.example",
"https://a.example?x",
"https://.a.example",
"https://a.example.",
"https://[::1]:443",
"https:// a.example",
] {
assert!(parse_url(url).is_err(), "{url:?} must be refused");
}
}
#[test]
fn ids() {
assert!(valid_id(OWNER));
assert!(valid_id(CHANNEL));
for id in [
"",
"abc",
"abcdefghijklmnopqrstuvwxyZ",
"abcdefghijklmnopqrstuvwxy-",
"abcdefghijklmnopqrstuvwxyza",
] {
assert!(!valid_id(id), "{id:?}");
}
}
#[test]
fn bad_values_are_named() {
let token = "credential = \"mattermost-token\"";
let cases: Vec<(String, &str)> = vec![
(
minimal().replace(
"https://straylight.scylla-hammerhead.ts.net",
"https://x.example/path",
),
"url",
),
(
minimal()
.replace(
"[secrets.mattermost_token]",
"[mattermost2]\n[secrets.other]",
)
.replace("[mattermost2]\n", ""),
"mattermost_token",
),
(
minimal().replace(token, "credential = \"a b\""),
"credential",
),
(minimal().replace(token, "env = \"lower\""), "env"),
(
minimal().replace(token, "file = \"relative/token\""),
"absolute",
),
(
minimal().replace(token, "credential = \"x\"\nenv = \"Y\""),
"exactly one",
),
(minimal().replace(token, ""), "exactly one"),
(
minimal().replace(&format!("[\"{OWNER}\"]"), "[]"),
"users is empty",
),
(minimal().replace(OWNER, "tooshort"), "not a Mattermost id"),
(
format!("{}channels = [\"NOTANID\"]\n", minimal()),
"not a Mattermost id",
),
(
minimal().replace("[mattermost]\n", "[mattermost]\nca_file = \"ca.pem\"\n"),
"ca_file",
),
(format!("{}[limits]\nqueue = 0\n", minimal()), "queue"),
(
format!("{}[limits]\ndead_after_ms = 0\n", minimal()),
"dead_after_ms",
),
];
for (text, word) in cases {
let why = invalid(&text);
assert!(why.contains(word), "{word}: {why}");
}
}
#[test]
fn unknown_keys_and_missing_tables_are_parse_errors() {
for text in [
format!("{}[allow2]\n", minimal()),
minimal().replace("[mattermost]\n", "[mattermost]\nproxy = \"x\"\n"),
minimal().replace(
"credential = \"mattermost-token\"",
"credential = \"t\"\nkeyring = \"x\"",
),
minimal().replace(&format!("[allow]\nusers = [\"{OWNER}\"]\n"), ""),
format!("{}[limits]\nqueue = -1\n", minimal()),
] {
assert!(matches!(load(&text), Err(ConfigError::Parse(..))), "{text}");
}
}
docs/plans/M4a/files/crates/gatewayd/tests/deliver.rs
+304
View File
@@ -0,0 +1,304 @@
//! A turn on `loop.sock` and its answer in the thread, against a fake `loopd`: the answer, long
//! answers, approvals, errors, an unknown session, and a loop that is not there (M4a spec, section
//! 8). Do not edit.
#[path = "support/fake_loop.rs"]
mod fake_loop;
#[path = "support/tmp.rs"]
mod tmp;
use std::sync::Mutex;
use std::time::Duration;
use fake_loop::{Reply, done, error, event, serve_loop};
use gatewayd::deliver::{EMPTY_ANSWER, LOOP_DOWN, MAX_POST, Poster, deliver, split_answer};
use gatewayd::mm::MmError;
use gatewayd::sessions::{Batch, Thread};
use proto::{DataClass, ErrorCode, SessionId, Timestamp, TurnEvent};
use tmp::TempDir;
const DM: &str = "d0000000000000000000000000";
const ROOT: &str = "r0000000000000000000000000";
#[derive(Default)]
struct Record {
posts: Mutex<Vec<(String, String, String)>>,
fail: bool,
}
impl Poster for Record {
fn post(&self, channel: &str, root: &str, text: &str) -> Result<(), MmError> {
if self.fail {
return Err(MmError::Status(500, "down".to_string()));
}
self.posts
.lock()
.unwrap()
.push((channel.to_string(), root.to_string(), text.to_string()));
Ok(())
}
}
impl Record {
fn texts(&self) -> Vec<String> {
let posts = self.posts.lock().unwrap();
assert!(
posts.iter().all(|(c, r, _)| c == DM && r == ROOT),
"every post in the thread"
);
posts.iter().map(|(_, _, t)| t.clone()).collect()
}
}
fn batch(resume: bool, text: &str) -> Batch {
Batch {
session: SessionId::new(&format!("mm-{ROOT}")).unwrap(),
thread: Thread {
channel: DM.to_string(),
root: ROOT.to_string(),
},
resume,
text: text.to_string(),
}
}
fn run(dir: &TempDir, poster: &Record, batch: &Batch) -> Vec<String> {
let log = Mutex::new(Vec::new());
deliver(poster, &dir.path().join("loop.sock"), batch, &|line| {
log.lock().unwrap().push(line.to_string())
});
log.into_inner().unwrap()
}
#[test]
fn the_answer_is_posted_and_nothing_else() {
let dir = TempDir::new("deliver-answer");
let turns = serve_loop(&dir.path().join("loop.sock"), |_, _| {
vec![
event(TurnEvent::Progress {
total: 10,
cache: 0,
processed: 10,
}),
event(TurnEvent::Reasoning {
text: "private thoughts".to_string(),
}),
event(TurnEvent::ToolCallStarted {
name: "read_file".to_string(),
}),
event(TurnEvent::ToolResult {
name: "read_file".to_string(),
class: DataClass::Private,
truncated: false,
}),
event(TurnEvent::Content {
text: "The ans".to_string(),
}),
done("The answer."),
]
});
let poster = Record::default();
let log = run(&dir, &poster, &batch(true, "one\n\ntwo"));
assert_eq!(poster.texts(), ["The answer."]);
assert!(log.is_empty(), "{log:?}");
let turn = turns.recv_timeout(Duration::from_secs(5)).unwrap();
assert_eq!(
(turn.session.as_str(), turn.content.as_str(), turn.resume),
(format!("mm-{ROOT}").as_str(), "one\n\ntwo", true)
);
}
#[test]
fn an_approval_is_announced_once_before_the_answer() {
let dir = TempDir::new("deliver-approval");
let expires = Timestamp::from_unix_millis(1_758_650_000_000).unwrap();
let _turns = serve_loop(&dir.path().join("loop.sock"), move |_, _| {
vec![
event(TurnEvent::ApprovalPending {
approval: 42,
tool: "shell".to_string(),
expires,
}),
done("done"),
]
});
let poster = Record::default();
run(&dir, &poster, &batch(false, "go"));
assert_eq!(
poster.texts(),
[
"waiting for approval 42: approve or deny it with `bxctl` (Mattermost approvals arrive in M4b)",
"done"
]
);
}
#[test]
fn errors_are_posted_with_their_code() {
let dir = TempDir::new("deliver-error");
let _turns = serve_loop(&dir.path().join("loop.sock"), |_, _| {
vec![error(
ErrorCode::Inference,
"the model server failed\nsee docs/runbook.md#loopd-selftest-failed",
)]
});
let poster = Record::default();
run(&dir, &poster, &batch(false, "go"));
assert_eq!(
poster.texts(),
["Error: inference: the model server failed\nsee docs/runbook.md#loopd-selftest-failed"]
);
}
#[test]
fn a_reply_in_a_thread_loopd_does_not_know_creates_the_session() {
let dir = TempDir::new("deliver-unknown");
let turns = serve_loop(&dir.path().join("loop.sock"), |n, _| {
if n == 0 {
vec![error(ErrorCode::NoSuchSession, "no such session")]
} else {
vec![done("hello")]
}
});
let poster = Record::default();
run(&dir, &poster, &batch(true, "hi"));
assert_eq!(poster.texts(), ["hello"]);
let first = turns.recv_timeout(Duration::from_secs(5)).unwrap();
let second = turns.recv_timeout(Duration::from_secs(5)).unwrap();
assert_eq!(
(first.resume, second.resume, second.content.as_str()),
(true, false, "hi")
);
}
#[test]
fn a_new_session_is_not_retried() {
let dir = TempDir::new("deliver-noretry");
let turns = serve_loop(&dir.path().join("loop.sock"), |_, _| {
vec![error(ErrorCode::NoSuchSession, "odd")]
});
let poster = Record::default();
run(&dir, &poster, &batch(false, "hi"));
assert_eq!(poster.texts(), ["Error: no_such_session: odd"]);
assert!(turns.recv_timeout(Duration::from_secs(5)).is_ok());
assert!(
turns.recv_timeout(Duration::from_millis(200)).is_err(),
"one turn only"
);
}
#[test]
fn a_loop_that_is_not_there_or_goes_away() {
let dir = TempDir::new("deliver-down");
let poster = Record::default();
let log = run(&dir, &poster, &batch(false, "hi"));
assert_eq!(poster.texts(), [LOOP_DOWN]);
assert_eq!(log.len(), 1, "{log:?}");
assert!(
log[0].starts_with(&format!("gatewayd: mm-{ROOT}: cannot connect to ")),
"{log:?}"
);
for (n, replies) in [
vec![
event(TurnEvent::Content {
text: "x".to_string(),
}),
Reply::Close,
],
vec![Reply::Bytes(b"\x00\x00\x00\x05{bad}".to_vec())],
vec![
Reply::Frame(proto::Envelope {
v: 1,
id: 2,
r#final: true,
msg: proto::Message::Ok(proto::Empty {}),
}),
done("late"),
],
]
.into_iter()
.enumerate()
{
let dir = TempDir::new(&format!("deliver-early-{n}"));
let replies = Mutex::new(Some(replies));
let _turns = serve_loop(&dir.path().join("loop.sock"), move |_, _| {
replies.lock().unwrap().take().unwrap_or_default()
});
let poster = Record::default();
let log = run(&dir, &poster, &batch(false, "hi"));
assert_eq!(poster.texts(), [LOOP_DOWN], "case {n}");
assert_eq!(log.len(), 1, "case {n}: {log:?}");
}
}
#[test]
fn a_post_that_fails_is_logged() {
let dir = TempDir::new("deliver-postfail");
let _turns = serve_loop(&dir.path().join("loop.sock"), |_, _| vec![done("lost")]);
let poster = Record {
fail: true,
..Record::default()
};
let log = run(&dir, &poster, &batch(false, "hi"));
assert_eq!(
log,
[format!(
"gatewayd: cannot post in {DM} (thread {ROOT}): status 500: \"down\""
)]
);
}
#[test]
fn long_answers_are_split_at_newlines() {
let short = "a".repeat(MAX_POST);
assert_eq!(split_answer(&short), [short.as_str()]);
let over = "a".repeat(MAX_POST + 1);
assert_eq!(split_answer(&over), ["a".repeat(MAX_POST), "a".to_string()]);
let lines = format!(
"{}\n{}\n{}",
"a".repeat(10_000),
"b".repeat(5_000),
"c".repeat(2_000)
);
assert_eq!(
split_answer(&lines),
[
format!("{}\n{}", "a".repeat(10_000), "b".repeat(5_000)),
"c".repeat(2_000)
]
);
let wide = "é".repeat(MAX_POST + 5);
let parts = split_answer(&wide);
assert_eq!(
parts.iter().map(|p| p.chars().count()).collect::<Vec<_>>(),
[MAX_POST, 5],
"characters, not bytes"
);
let leading = format!("\n{}", "x".repeat(MAX_POST + 1));
let parts = split_answer(&leading);
assert!(
parts
.iter()
.all(|p| !p.is_empty() && p.chars().count() <= MAX_POST),
"{:?}",
parts.iter().map(|p| p.len()).collect::<Vec<_>>()
);
assert_eq!(parts.concat(), leading, "a hard cut drops nothing");
assert_eq!(split_answer(""), [EMPTY_ANSWER]);
assert_eq!(split_answer(" \n "), [EMPTY_ANSWER]);
}
#[test]
fn a_long_answer_is_posted_in_order() {
let dir = TempDir::new("deliver-long");
let answer = format!("{}\n{}", "a".repeat(MAX_POST - 1), "b".repeat(MAX_POST));
let sent = answer.clone();
let _turns = serve_loop(&dir.path().join("loop.sock"), move |_, _| vec![done(&sent)]);
let poster = Record::default();
run(&dir, &poster, &batch(false, "hi"));
assert_eq!(
poster.texts(),
["a".repeat(MAX_POST - 1), "b".repeat(MAX_POST)]
);
}
docs/plans/M4a/files/crates/gatewayd/tests/fixtures/tls/README.md
Vendored
+12
View File
@@ -0,0 +1,12 @@
# TEST-ONLY TLS fixtures
Generated once with `openssl` on 2026-09-23 for `gatewayd`'s TLS tests. Every key here is public
and must never be trusted anywhere but these tests. The two CA private keys were deleted after
signing; the server keys are kept because the test servers need them.
- `test-ca.pem`: the CA the tests trust (through `ca_file`).
- `server.pem`/`server.key`: `localhost` and `127.0.0.1`, signed by `test-ca`.
- `wrong-name.pem`/`wrong-name.key`: `wrong.example` only, signed by `test-ca` (a name mismatch).
- `other-ca.pem`, `other-server.pem`/`other-server.key`: a CA the tests do not trust.
Valid for 100 years from 2026-09-23.
docs/plans/M4a/files/crates/gatewayd/tests/fixtures/tls/other-ca.pem
Vendored
+12
View File
@@ -0,0 +1,12 @@
-----BEGIN CERTIFICATE-----
MIIBszCCAVmgAwIBAgIUZK7y645vuYezo+uB8S2ktvXjLBUwCgYIKoZIzj0EAwIw
JjEkMCIGA1UEAwwbQm94bWFrZXIgVEVTVC1PTkxZIG90aGVyIENBMCAXDTI2MDky
NDAwNDgxN1oYDzIxMjYwODMxMDA0ODE3WjAmMSQwIgYDVQQDDBtCb3htYWtlciBU
RVNULU9OTFkgb3RoZXIgQ0EwWTATBgcqhkjOPQIBBggqhkjOPQMBBwNCAAQV1sRm
ePvJps71wo1/QVUi8Y0Ra4kFhJ0WI7VLIqeINPpaQtBKHUH+SWrjR4mromEtJ8ZR
d3frK7jBFmI+AVmLo2MwYTAdBgNVHQ4EFgQUUBg/R1eSkXP7swTjnoEFSvcIGcww
HwYDVR0jBBgwFoAUUBg/R1eSkXP7swTjnoEFSvcIGcwwDwYDVR0TAQH/BAUwAwEB
/zAOBgNVHQ8BAf8EBAMCAgQwCgYIKoZIzj0EAwIDSAAwRQIhAIKFnNomDrIwpeOG
wdsm8NfXWydx7Mp2/ujRCXCqMyrAAiAo2hoprQhU3uRmyrTtokBAqE5kWFSKOa6K
BlnVrcFSvw==
-----END CERTIFICATE-----
docs/plans/M4a/files/crates/gatewayd/tests/fixtures/tls/other-server.key
Vendored
+5
View File
@@ -0,0 +1,5 @@
-----BEGIN PRIVATE KEY-----
MIGHAgEAMBMGByqGSM49AgEGCCqGSM49AwEHBG0wawIBAQQg940CxyChSA5oYsx2
W6tmkg66INWcYxOOfUcqkvi9TBehRANCAAQBOwVQrucb42OCoWScE/Grn6DnmMBk
6yceR+ZNU9wvYwMKBovg6sErdvjACNlYIsAkjjRuO7xYQbrxJ4ixoY3O
-----END PRIVATE KEY-----
docs/plans/M4a/files/crates/gatewayd/tests/fixtures/tls/other-server.pem
Vendored
+12
View File
@@ -0,0 +1,12 @@
-----BEGIN CERTIFICATE-----
MIIB2zCCAYCgAwIBAgIUWxsnA8gxRlvuY1SrXg+eQxRDq7cwCgYIKoZIzj0EAwIw
JjEkMCIGA1UEAwwbQm94bWFrZXIgVEVTVC1PTkxZIG90aGVyIENBMCAXDTI2MDky
NDAwNDgxN1oYDzIxMjYwODMxMDA0ODE3WjAuMSwwKgYDVQQDDCNsb2NhbGhvc3Qg
ZnJvbSBvdGhlciBDQSAoVEVTVCBPTkxZKTBZMBMGByqGSM49AgEGCCqGSM49AwEH
A0IABAE7BVCu5xvjY4KhZJwT8aufoOeYwGTrJx5H5k1T3C9jAwoGi+DqwSt2+MAI
2VgiwCSONG47vFhBuvEniLGhjc6jgYEwfzAaBgNVHREEEzARgglsb2NhbGhvc3SH
BH8AAAEwDAYDVR0TAQH/BAIwADATBgNVHSUEDDAKBggrBgEFBQcDATAdBgNVHQ4E
FgQUdpTjXgMrZvKV54i+QnheUhd17IswHwYDVR0jBBgwFoAUUBg/R1eSkXP7swTj
noEFSvcIGcwwCgYIKoZIzj0EAwIDSQAwRgIhAJqpsTc14FSZpyWvmn6G0Ar2bxLz
CYQNanzxCPLMDGTCAiEA/F1wQjxrCikZAfuQKBKL5cc2MHf2dsSjZq2Sg5OC6z0=
-----END CERTIFICATE-----
docs/plans/M4a/files/crates/gatewayd/tests/fixtures/tls/server.key
Vendored
+5
View File
@@ -0,0 +1,5 @@
-----BEGIN PRIVATE KEY-----
MIGHAgEAMBMGByqGSM49AgEGCCqGSM49AwEHBG0wawIBAQQgtj+G5HUdct3YHcZ2
rvTnI3blyTjkfEiwVwGTqOINYZChRANCAAQcwnbJi8KAjVQEQd/mIXFCSDGNcy9V
XRx5uZ+wqnAUqbmsj+LHl6q9KM1Y3bowFBIHQOjpBWvy8JA0oPRJPLWM
-----END PRIVATE KEY-----
docs/plans/M4a/files/crates/gatewayd/tests/fixtures/tls/server.pem
Vendored
+13
View File
@@ -0,0 +1,13 @@
-----BEGIN CERTIFICATE-----
MIIB4zCCAYigAwIBAgIUNQfYYHxiZvBYa5P4sYqm4yJqf9MwCgYIKoZIzj0EAwIw
PDE6MDgGA1UEAwwxQm94bWFrZXIgVEVTVC1PTkxZIENBIChuZXZlciB0cnVzdCBv
dXRzaWRlIHRlc3RzKTAgFw0yNjA5MjQwMDQ4MTdaGA8yMTI2MDgzMTAwNDgxN1ow
IDEeMBwGA1UEAwwVbG9jYWxob3N0IChURVNUIE9OTFkpMFkwEwYHKoZIzj0CAQYI
KoZIzj0DAQcDQgAEHMJ2yYvCgI1UBEHf5iFxQkgxjXMvVV0cebmfsKpwFKm5rI/i
x5eqvSjNWN26MBQSB0Do6QVr8vCQNKD0STy1jKOBgTB/MBoGA1UdEQQTMBGCCWxv
Y2FsaG9zdIcEfwAAATAMBgNVHRMBAf8EAjAAMBMGA1UdJQQMMAoGCCsGAQUFBwMB
MB0GA1UdDgQWBBRtebrcLsFuJloYqBEhrWKh4ghINTAfBgNVHSMEGDAWgBRZLO26
Eow46wSsnj/mQBJ5Hi7VZDAKBggqhkjOPQQDAgNJADBGAiEAn215O/7cosHkI5n4
7Kuq+30BXfrqBHnZ6FznHQIgIjsCIQDW8om0qRjIo5dXNIY4DLj757+KaleqaQdE
oFc079H45g==
-----END CERTIFICATE-----
docs/plans/M4a/files/crates/gatewayd/tests/fixtures/tls/test-ca.pem
Vendored
+13
View File
@@ -0,0 +1,13 @@
-----BEGIN CERTIFICATE-----
MIIB3zCCAYWgAwIBAgIURpYQTJ2pH+M6c7wb2aMwpr96TQUwCgYIKoZIzj0EAwIw
PDE6MDgGA1UEAwwxQm94bWFrZXIgVEVTVC1PTkxZIENBIChuZXZlciB0cnVzdCBv
dXRzaWRlIHRlc3RzKTAgFw0yNjA5MjQwMDQ4MTdaGA8yMTI2MDgzMTAwNDgxN1ow
PDE6MDgGA1UEAwwxQm94bWFrZXIgVEVTVC1PTkxZIENBIChuZXZlciB0cnVzdCBv
dXRzaWRlIHRlc3RzKTBZMBMGByqGSM49AgEGCCqGSM49AwEHA0IABMeSQ30pi+FZ
85pHjd7+q6bo30eJGcdwmiK2MwlASDejQb0nA4cOWqPLGdlNO4o5679DwiigSUnv
yh/V1yJ4KyqjYzBhMB0GA1UdDgQWBBRZLO26Eow46wSsnj/mQBJ5Hi7VZDAfBgNV
HSMEGDAWgBRZLO26Eow46wSsnj/mQBJ5Hi7VZDAPBgNVHRMBAf8EBTADAQH/MA4G
A1UdDwEB/wQEAwICBDAKBggqhkjOPQQDAgNIADBFAiA+LzwUA1QvGOcDNxMbnbb8
ycfuH+i16pebeH3rcJIwDAIhALgKBj1r2ItuB/Rag8Y0sYs9rx5Arlikzg2VGWoT
CYbm
-----END CERTIFICATE-----
docs/plans/M4a/files/crates/gatewayd/tests/fixtures/tls/wrong-name.key
Vendored
+5
View File
@@ -0,0 +1,5 @@
-----BEGIN PRIVATE KEY-----
MIGHAgEAMBMGByqGSM49AgEGCCqGSM49AwEHBG0wawIBAQQgflra3VFKl15oCyVi
0KDJ52JphSZfIFDqAmFIUdVow0ShRANCAAQLaVnu5yblt9VdhunVTXzxk4k1ZIAv
qs0WEHCiNRfR+Wex5GpMfRCDcHH6fFlqyq5YpFV0/ripVSlt3RnH9Ok5
-----END PRIVATE KEY-----
docs/plans/M4a/files/crates/gatewayd/tests/fixtures/tls/wrong-name.pem
Vendored
+13
View File
@@ -0,0 +1,13 @@
-----BEGIN CERTIFICATE-----
MIIB4jCCAYmgAwIBAgIUNQfYYHxiZvBYa5P4sYqm4yJqf9QwCgYIKoZIzj0EAwIw
PDE6MDgGA1UEAwwxQm94bWFrZXIgVEVTVC1PTkxZIENBIChuZXZlciB0cnVzdCBv
dXRzaWRlIHRlc3RzKTAgFw0yNjA5MjQwMDQ4MTdaGA8yMTI2MDgzMTAwNDgxN1ow
JDEiMCAGA1UEAwwZd3JvbmcuZXhhbXBsZSAoVEVTVCBPTkxZKTBZMBMGByqGSM49
AgEGCCqGSM49AwEHA0IABAtpWe7nJuW31V2G6dVNfPGTiTVkgC+qzRYQcKI1F9H5
Z7Hkakx9EINwcfp8WWrKrlikVXT+uKlVKW3dGcf06TmjfzB9MBgGA1UdEQQRMA+C
DXdyb25nLmV4YW1wbGUwDAYDVR0TAQH/BAIwADATBgNVHSUEDDAKBggrBgEFBQcD
ATAdBgNVHQ4EFgQU7JXvioL6xNp1Xd8wEN1vHw32ZC4wHwYDVR0jBBgwFoAUWSzt
uhKMOOsErJ4/5kASeR4u1WQwCgYIKoZIzj0EAwIDRwAwRAIgJ+BxEK1QVQUeI/PM
Ap1A7fHECE5GTgKazmJ79DiRBa4CIEEAw9AxKBjNn5gXcQWe/zSs+cGwD6jAxdAe
hbIC76Kx
-----END CERTIFICATE-----
docs/plans/M4a/files/crates/gatewayd/tests/http.rs
+207
View File
@@ -0,0 +1,207 @@
//! The HTTP client over an in-memory stream: what it writes, how it reads each kind of body, its
//! caps, and the wait a 429 asks for (M4a spec, section 5). Do not edit.
use std::io::{Cursor, Read, Write};
use std::time::{Duration, UNIX_EPOCH};
use gatewayd::http::{Head, HttpError, MAX_BODY, MAX_HEAD, rate_limit_wait, read_head, request};
/// Reads from `input`, records what is written.
struct Duplex {
input: Cursor<Vec<u8>>,
output: Vec<u8>,
}
impl Duplex {
fn new(input: &[u8]) -> Duplex {
Duplex {
input: Cursor::new(input.to_vec()),
output: Vec::new(),
}
}
}
impl Read for Duplex {
fn read(&mut self, buf: &mut [u8]) -> std::io::Result<usize> {
self.input.read(buf)
}
}
impl Write for Duplex {
fn write(&mut self, buf: &[u8]) -> std::io::Result<usize> {
self.output.write(buf)
}
fn flush(&mut self) -> std::io::Result<()> {
Ok(())
}
}
fn get(response: &[u8]) -> Result<(u16, Vec<u8>), HttpError> {
let mut d = Duplex::new(response);
let r = request(&mut d, "GET", "a.example", "/api/v4/users/me", &[], None)?;
Ok((r.head.status, r.body))
}
#[test]
fn the_request_is_exactly_this() {
let mut d = Duplex::new(b"HTTP/1.1 201 Created\r\nContent-Length: 2\r\n\r\n{}");
let r = request(
&mut d,
"POST",
"straylight.example",
"/api/v4/posts",
&[
("Authorization", "Bearer t"),
("Content-Type", "application/json"),
],
Some(b"{\"message\":\"hi\"}"),
)
.unwrap();
assert_eq!(
String::from_utf8(d.output).unwrap(),
"POST /api/v4/posts HTTP/1.1\r\nHost: straylight.example\r\nAuthorization: Bearer t\r\n\
Content-Type: application/json\r\nContent-Length: 16\r\nConnection: close\r\n\r\n{\"message\":\"hi\"}"
);
assert_eq!((r.head.status, r.body), (201, b"{}".to_vec()));
}
#[test]
fn bodies_by_length_chunks_or_close() {
assert_eq!(
get(b"HTTP/1.1 200 OK\r\nContent-Length: 5\r\n\r\nhello").unwrap(),
(200, b"hello".to_vec())
);
assert_eq!(
get(b"HTTP/1.1 200 OK\r\nTransfer-Encoding: chunked\r\n\r\n5\r\nhello\r\n6;x=y\r\n world\r\n0\r\nTrailer: z\r\n\r\n").unwrap(),
(200, b"hello world".to_vec())
);
assert_eq!(
get(b"HTTP/1.1 200 OK\r\ntransfer-encoding: CHUNKED\r\n\r\nA\r\n0123456789\r\n0\r\n\r\n")
.unwrap()
.1,
b"0123456789".to_vec()
);
assert_eq!(
get(b"HTTP/1.0 200 OK\r\n\r\nuntil the end").unwrap(),
(200, b"until the end".to_vec())
);
assert_eq!(
get(b"HTTP/1.1 204 No Content\r\nContent-Length: 0\r\n\r\n").unwrap(),
(204, Vec::new())
);
}
#[test]
fn headers_are_found_whatever_their_case_and_trimmed() {
let mut d = Duplex::new(
b"HTTP/1.1 101 Switching Protocols\r\nUPGRADE: websocket \r\nX-A: 1\r\n\r\nFRAMES",
);
let head = read_head(&mut d).unwrap();
assert_eq!(head.status, 101);
assert_eq!(head.header("upgrade"), Some("websocket"));
assert_eq!(head.header("x-a"), Some("1"));
assert_eq!(head.header("missing"), None);
let mut rest = String::new();
d.read_to_string(&mut rest).unwrap();
assert_eq!(
rest, "FRAMES",
"read_head reads nothing past the blank line"
);
}
#[test]
fn malformed_responses_are_errors_not_panics() {
for bad in [
&b""[..],
b"HTTP/1.1 200 OK\r\nContent-Length: 5\r\n",
b"HTTP/2 200 OK\r\n\r\n",
b"HTTP/1.1 2000 OK\r\n\r\n",
b"HTTP/1.1 abc OK\r\n\r\n",
b"HTTP/1.1 99 OK\r\n\r\n",
b"HTTP/1.1 200 OK\r\nno colon here\r\n\r\n",
b"HTTP/1.1 200 OK\r\nContent-Length: five\r\n\r\n",
b"HTTP/1.1 200 OK\r\nContent-Length: 10\r\n\r\nshort",
b"HTTP/1.1 200 OK\r\nTransfer-Encoding: chunked\r\n\r\nzz\r\n",
b"HTTP/1.1 200 OK\r\nTransfer-Encoding: chunked\r\n\r\n5\r\nhelloXX0\r\n\r\n",
b"HTTP/1.1 200 OK\r\nTransfer-Encoding: chunked\r\n\r\n5\r\nhel",
b"HTTP/1.1 200 OK\r\n\xff\xfe: x\r\n\r\n",
] {
assert!(get(bad).is_err(), "{:?}", String::from_utf8_lossy(bad));
}
}
#[test]
fn caps_are_checked_before_reading_or_allocating() {
let mut long_head = b"HTTP/1.1 200 OK\r\nX: ".to_vec();
long_head.extend(std::iter::repeat_n(b'a', MAX_HEAD));
long_head.extend(b"\r\n\r\n");
assert!(matches!(get(&long_head), Err(HttpError::TooLarge("head"))));
let huge = format!(
"HTTP/1.1 200 OK\r\nContent-Length: {}\r\n\r\n",
MAX_BODY + 1
);
assert!(
matches!(get(huge.as_bytes()), Err(HttpError::TooLarge("body"))),
"refused from the header alone"
);
let huge_chunk = format!(
"HTTP/1.1 200 OK\r\nTransfer-Encoding: chunked\r\n\r\n{:x}\r\n",
MAX_BODY + 1
);
assert!(matches!(
get(huge_chunk.as_bytes()),
Err(HttpError::TooLarge("body"))
));
let overflow = "HTTP/1.1 200 OK\r\nTransfer-Encoding: chunked\r\n\r\nffffffffffffffffffff\r\n";
assert!(get(overflow.as_bytes()).is_err());
let mut to_close = b"HTTP/1.0 200 OK\r\n\r\n".to_vec();
to_close.extend(std::iter::repeat_n(b'b', MAX_BODY + 1));
assert!(matches!(get(&to_close), Err(HttpError::TooLarge("body"))));
}
fn head_with(reset: Option<&str>) -> Head {
let mut headers = vec![("X-Ratelimit-Limit".to_string(), "10".to_string())];
if let Some(r) = reset {
headers.push(("X-Ratelimit-Reset".to_string(), r.to_string()));
}
Head {
status: 429,
headers,
}
}
#[test]
fn a_rate_limit_is_waited_out_within_bounds() {
let now = UNIX_EPOCH + Duration::from_secs(1_800_000_000);
assert_eq!(
rate_limit_wait(&head_with(Some("3")), now),
Duration::from_secs(3)
);
assert_eq!(
rate_limit_wait(&head_with(Some("1800000005")), now),
Duration::from_secs(5),
"a Unix time"
);
assert_eq!(
rate_limit_wait(&head_with(Some("1799999999")), now),
Duration::from_secs(1),
"already past"
);
assert_eq!(
rate_limit_wait(&head_with(Some("0")), now),
Duration::from_secs(1)
);
assert_eq!(
rate_limit_wait(&head_with(Some("999999")), now),
Duration::from_secs(60),
"capped"
);
assert_eq!(
rate_limit_wait(&head_with(None), now),
Duration::from_secs(1)
);
assert_eq!(
rate_limit_wait(&head_with(Some("soon")), now),
Duration::from_secs(1)
);
}
docs/plans/M4a/files/crates/gatewayd/tests/main.rs
+171
View File
@@ -0,0 +1,171 @@
//! The `gatewayd` program: its usage, what stops it at start, the warning for a secret in a file,
//! and that the token never reaches its output (M4a spec, sections 3, 4 and 10). Do not edit.
#[path = "support/tmp.rs"]
mod tmp;
use std::io::{BufRead, BufReader};
use std::os::unix::fs::PermissionsExt;
use std::process::{Command, Stdio};
use std::time::{Duration, Instant};
use tmp::TempDir;
const KYLE: &str = "k0000000000000000000000000";
const TOKEN: &str = "tok-3f9a1c7e5b2d4f6a8c0e";
fn gatewayd() -> Command {
let mut c = Command::new(env!("CARGO_BIN_EXE_gatewayd"));
c.env_remove("GW_TEST_TOKEN")
.env_remove("CREDENTIALS_DIRECTORY");
c
}
fn closed_url() -> String {
let port = std::net::TcpListener::bind("127.0.0.1:0")
.unwrap()
.local_addr()
.unwrap()
.port();
format!("http://127.0.0.1:{port}")
}
fn write_config(dir: &TempDir, secret: &str) -> std::path::PathBuf {
let text = format!(
"[mattermost]\nurl = \"{}\"\n[secrets.mattermost_token]\n{secret}\n[allow]\nusers = [\"{KYLE}\"]\n[paths]\nhome = \"{}\"\n",
closed_url(),
dir.path().join("home").display()
);
dir.write("gatewayd.toml", &text)
}
/// Run until `want` appears on standard error or 5 s pass, then kill it; all it printed.
fn stderr_until(mut cmd: Command, want: &str) -> String {
let mut child = cmd
.stderr(Stdio::piped())
.stdout(Stdio::null())
.spawn()
.unwrap();
let mut reader = BufReader::new(child.stderr.take().unwrap());
let until = Instant::now() + Duration::from_secs(5);
let mut all = String::new();
while Instant::now() < until && !all.contains(want) {
let mut line = String::new();
if reader.read_line(&mut line).unwrap_or(0) == 0 {
break;
}
all.push_str(&line);
}
let _ = child.kill();
let _ = child.wait();
all
}
#[test]
fn usage() {
for args in [
&[][..],
&["serve"][..],
&["serve", "--config"][..],
&["run", "--config", "x"][..],
] {
let out = gatewayd().args(args).output().unwrap();
assert_eq!(out.status.code(), Some(2), "{args:?}");
assert_eq!(
String::from_utf8_lossy(&out.stderr),
"usage: gatewayd serve --config <path>\n"
);
}
}
#[test]
fn a_bad_config_stops_at_start() {
let dir = TempDir::new("main-config");
let missing = dir.path().join("nope.toml");
let bad = dir.write("bad.toml", "[mattermost]\nurl = \"ftp://x\"\n");
for path in [missing, bad] {
let out = gatewayd()
.arg("serve")
.arg("--config")
.arg(&path)
.output()
.unwrap();
let err = String::from_utf8_lossy(&out.stderr);
assert_eq!(out.status.code(), Some(1), "{err}");
assert!(
err.starts_with(&format!("gatewayd: {}: ", path.display())),
"{err}"
);
assert!(
err.ends_with("\nsee docs/runbook.md#gatewayd-start-failed\n"),
"{err}"
);
}
}
#[test]
fn a_missing_secret_stops_at_start() {
let dir = TempDir::new("main-secret");
let config = write_config(&dir, "env = \"GW_TEST_TOKEN\"");
let out = gatewayd()
.arg("serve")
.arg("--config")
.arg(&config)
.output()
.unwrap();
let err = String::from_utf8_lossy(&out.stderr);
assert_eq!(out.status.code(), Some(1), "{err}");
assert!(
err.starts_with("gatewayd: secret mattermost_token: "),
"{err}"
);
assert!(
err.ends_with("\nsee docs/runbook.md#secret-unavailable\n"),
"{err}"
);
assert!(
!dir.path().join("home").exists(),
"nothing is made before the secret is read"
);
}
#[test]
fn a_secret_in_a_file_warns_and_the_token_is_never_printed() {
let dir = TempDir::new("main-file");
let secret = dir.write("token", &format!("{TOKEN}\n"));
std::fs::set_permissions(&secret, std::fs::Permissions::from_mode(0o600)).unwrap();
let config = write_config(&dir, &format!("file = \"{}\"", secret.display()));
let mut cmd = gatewayd();
cmd.arg("serve").arg("--config").arg(&config);
let err = stderr_until(cmd, "trying again");
let warning = format!(
"gatewayd: warning: secret mattermost_token is read in plaintext from {}; a systemd credential keeps it encrypted at rest (see docs/runbook.md#secret-in-a-file)\n",
secret.display()
);
assert!(err.starts_with(&warning), "{err}");
assert!(
err.contains("gatewayd: cannot reach http://127.0.0.1:"),
"{err}"
);
assert!(!err.contains(TOKEN), "{err}");
let mode = std::fs::metadata(dir.path().join("home/gateway"))
.unwrap()
.permissions()
.mode()
& 0o777;
assert_eq!(mode, 0o700);
}
#[test]
fn a_secret_from_the_environment_has_no_warning() {
let dir = TempDir::new("main-env");
let config = write_config(&dir, "env = \"GW_TEST_TOKEN\"");
let mut cmd = gatewayd();
cmd.arg("serve")
.arg("--config")
.arg(&config)
.env("GW_TEST_TOKEN", TOKEN);
let err = stderr_until(cmd, "trying again");
assert!(err.starts_with("gatewayd: cannot reach "), "{err}");
assert!(!err.contains("warning") && !err.contains(TOKEN), "{err}");
}
docs/plans/M4a/files/crates/gatewayd/tests/mm_json.rs
+173
View File
@@ -0,0 +1,173 @@
//! Mattermost's JSON: events as the server sends them (v11.11.0 shapes), the typing request, and
//! the `posts?since` list, whose edited, deleted and root-only posts must not be answered (M4a spec,
//! sections 7 and 9). Do not edit.
use gatewayd::mm::{Event, MmError, SINCE_LIMIT, parse_event, since_list, typing};
use serde_json::json;
const KYLE: &str = "k0000000000000000000000000";
const DM: &str = "d0000000000000000000000000";
fn post(id: &str, create_at: i64) -> serde_json::Value {
json!({
"id": id, "create_at": create_at, "update_at": create_at, "edit_at": 0, "delete_at": 0,
"is_pinned": false, "user_id": KYLE, "channel_id": DM, "root_id": "", "original_id": "",
"message": "hello", "type": "", "props": {"from_bot": "true"}, "hashtags": "",
"pending_post_id": "", "reply_count": 0, "metadata": {}
})
}
fn id(n: u32) -> String {
format!("p{n:025}")
}
fn posted(post: &serde_json::Value, channel_type: &str) -> String {
json!({
"event": "posted",
"data": {
"channel_display_name": "@kyle", "channel_name": "x__y", "channel_type": channel_type,
"post": post.to_string(), "sender_name": "@kyle", "set_online": true, "team_id": ""
},
"broadcast": {"omit_users": null, "user_id": "", "channel_id": DM, "team_id": ""},
"seq": 3
})
.to_string()
}
#[test]
fn hello_replies_and_other_events() {
let hello =
json!({"event": "hello", "data": {"server_version": "11.11.0"}, "broadcast": {}, "seq": 0});
assert_eq!(parse_event(&hello.to_string()).unwrap(), Event::Hello);
assert_eq!(
parse_event(r#"{"status":"OK","seq_reply":1}"#).unwrap(),
Event::Other(String::new())
);
let typing = json!({"event": "typing", "data": {"parent_id": ""}, "broadcast": {}, "seq": 4});
assert_eq!(
parse_event(&typing.to_string()).unwrap(),
Event::Other("typing".to_string())
);
}
#[test]
fn a_posted_event_carries_the_post_as_a_string() {
let mut p = post(&id(1), 1_758_650_000_123);
p["root_id"] = json!(id(0));
p["message"] = json!("line one\nline two");
let Event::Posted { post, channel_type } = parse_event(&posted(&p, "D")).unwrap() else {
panic!("not a post")
};
assert_eq!(channel_type, "D");
assert_eq!(
(post.id.as_str(), post.root_id.as_str()),
(id(1).as_str(), id(0).as_str())
);
assert_eq!(
(post.user_id.as_str(), post.channel_id.as_str()),
(KYLE, DM)
);
assert_eq!(
(post.message.as_str(), post.create_at, post.kind.as_str()),
("line one\nline two", 1_758_650_000_123, "")
);
let mut s = self::post(&id(2), 5);
s["type"] = json!("system_join_channel");
let Event::Posted { post, .. } = parse_event(&posted(&s, "O")).unwrap() else {
panic!()
};
assert_eq!(post.kind, "system_join_channel");
}
#[test]
fn bad_posts_are_errors_not_panics() {
let p = post(&id(1), 5);
let as_object =
json!({"event": "posted", "data": {"post": p, "channel_type": "D"}}).to_string();
let mut cases = vec![
as_object,
"not json".to_string(),
"[1,2]".to_string(),
"{\"event\":5}".to_string(),
];
for (field, value) in [
("id", json!("../../etc")),
("user_id", json!("")),
("channel_id", json!("A0000000000000000000000000")),
("root_id", json!("short")),
("create_at", json!("soon")),
] {
let mut bad = post(&id(1), 5);
bad[field] = value;
cases.push(posted(&bad, "D"));
}
let mut missing = post(&id(1), 5);
missing.as_object_mut().unwrap().remove("create_at");
cases.push(posted(&missing, "D"));
for case in cases {
assert!(
matches!(parse_event(&case), Err(MmError::Json(_))),
"{case}"
);
}
}
#[test]
fn the_typing_request() {
let got: serde_json::Value = serde_json::from_str(&typing(7, DM, &id(0))).unwrap();
assert_eq!(
got,
json!({"action": "user_typing", "seq": 7, "data": {"channel_id": DM, "parent_id": id(0)}})
);
}
#[test]
fn since_keeps_new_posts_in_order_and_nothing_else() {
let since = 1000;
let mut edited = post(&id(1), 900);
edited["update_at"] = json!(1500);
let mut deleted = post(&id(2), 1100);
deleted["delete_at"] = json!(1200);
let root_only = post(&id(3), 10);
let later = post(&id(4), 1300);
let earlier = post(&id(5), 1200);
let at_since = post(&id(6), 1000);
let mut posts = serde_json::Map::new();
for p in [&edited, &deleted, &root_only, &later, &earlier, &at_since] {
posts.insert(p["id"].as_str().unwrap().to_string(), p.clone());
}
let order = json!([id(4), id(2), id(1), id(5), id(9), id(6)]);
let body = json!({"order": order, "posts": posts, "next_post_id": "", "prev_post_id": "", "has_next": false});
let got = since_list(body.to_string().as_bytes(), since).unwrap();
let ids: Vec<&str> = got.posts.iter().map(|p| p.id.as_str()).collect();
assert_eq!(ids, [id(5), id(4)]);
assert!(!got.full);
}
#[test]
fn a_full_since_answer_says_so() {
let mut posts = serde_json::Map::new();
let mut order = Vec::new();
for n in 0..SINCE_LIMIT {
let n = u32::try_from(n).unwrap();
posts.insert(id(n), post(&id(n), 2000 + i64::from(n)));
order.push(id(n));
}
let body = json!({"order": order, "posts": posts});
let got = since_list(body.to_string().as_bytes(), 1000).unwrap();
assert!(got.full);
assert_eq!(got.posts.len(), SINCE_LIMIT);
let empty = since_list(br#"{"order":[],"posts":{}}"#, 0).unwrap();
assert!(empty.posts.is_empty() && !empty.full);
}
#[test]
fn a_server_error_body_is_quoted_in_messages() {
let e = MmError::Status(500, "boom\ngatewayd: a forged line".to_string());
let text = e.to_string();
assert!(!text.contains('\n'), "{text}");
assert_eq!(
MmError::Auth(401).to_string(),
"Mattermost refused the token (401)"
);
}
docs/plans/M4a/files/crates/gatewayd/tests/mm_rest.rs
+242
View File
@@ -0,0 +1,242 @@
//! Mattermost's REST calls against a scripted server: the requests on the wire, and what a 401,
//! 403, 429, 5xx or 404 does (M4a spec, sections 5 and 9). Do not edit.
#[path = "support/http_server.rs"]
mod http_server;
#[path = "support/tls_server.rs"]
mod tls_server;
use std::net::SocketAddr;
use std::sync::mpsc::Receiver;
use std::time::{Duration, Instant};
use gatewayd::config::ServerUrl;
use gatewayd::mm::MmError;
use gatewayd::mm::rest::Client;
use gatewayd::net::Connector;
use gatewayd::secrets::Secret;
use http_server::{Request, reply, serve_http};
use serde_json::json;
use tls_server::{fixture, server_config};
const BOT: &str = "b0000000000000000000000000";
const KYLE: &str = "k0000000000000000000000000";
const DM: &str = "d0000000000000000000000000";
const POST: &str = "p0000000000000000000000001";
fn client(addr: SocketAddr, tls: bool) -> Client {
let url = ServerUrl {
tls,
host: "localhost".to_string(),
port: addr.port(),
};
let ca = tls.then(|| fixture("test-ca.pem"));
let connector = Connector::new(url, ca.as_deref()).unwrap();
Client::new(
connector,
Secret::new("TOKEN".to_string()),
Duration::from_secs(5),
)
}
fn me_body() -> String {
json!({"id": BOT, "username": "boxmaker-straylight", "roles": "system_user", "is_bot": true})
.to_string()
}
fn requests(rx: &Receiver<Request>) -> Vec<Request> {
rx.try_iter().collect()
}
#[test]
fn me_plain_and_over_tls() {
for tls in [false, true] {
let config = tls.then(|| server_config("server"));
let (addr, rx) = serve_http(config, |_, _| reply(200, "", &me_body()));
let me = client(addr, tls).me().unwrap();
assert_eq!(
(me.id.as_str(), me.username.as_str()),
(BOT, "boxmaker-straylight")
);
let r = rx.recv_timeout(Duration::from_secs(5)).unwrap();
assert_eq!(
(r.method.as_str(), r.path.as_str()),
("GET", "/api/v4/users/me")
);
assert!(
r.head.contains("Authorization: Bearer TOKEN\r\n"),
"{}",
r.head
);
assert!(
r.head
.contains(&format!("Host: localhost:{}\r\n", addr.port())),
"{}",
r.head
);
}
}
#[test]
fn create_post_sends_the_thread_and_reads_the_post_back() {
let (addr, rx) = serve_http(None, |_, r| {
let mut p = r.json();
p["id"] = json!(POST);
p["user_id"] = json!(BOT);
p["create_at"] = json!(5);
reply(201, "", &p.to_string())
});
let post = client(addr, false)
.create_post(DM, KYLE, "an answer\nin two lines")
.unwrap();
assert_eq!((post.id.as_str(), post.root_id.as_str()), (POST, KYLE));
let r = rx.recv_timeout(Duration::from_secs(5)).unwrap();
assert_eq!(
(r.method.as_str(), r.path.as_str()),
("POST", "/api/v4/posts")
);
assert!(
r.head.contains("Content-Type: application/json\r\n"),
"{}",
r.head
);
assert_eq!(
r.json(),
json!({"channel_id": DM, "root_id": KYLE, "message": "an answer\nin two lines"})
);
}
#[test]
fn posts_since_and_the_direct_channel() {
let (addr, rx) = serve_http(None, |_, r| {
if r.path.contains("/posts?since=") {
reply(200, "", r#"{"order":[],"posts":{}}"#)
} else {
reply(201, "", &json!({"id": DM, "type": "D"}).to_string())
}
});
let c = client(addr, false);
assert!(
c.posts_since(DM, 1_758_650_000_000)
.unwrap()
.posts
.is_empty()
);
assert_eq!(c.direct_channel(BOT, KYLE).unwrap(), DM);
let rs = requests(&rx);
assert_eq!(
rs[0].path,
format!("/api/v4/channels/{DM}/posts?since=1758650000000")
);
assert_eq!(
(rs[1].method.as_str(), rs[1].path.as_str()),
("POST", "/api/v4/channels/direct")
);
assert_eq!(rs[1].json(), json!([BOT, KYLE]));
}
#[test]
fn a_channel_that_is_not_an_id_is_never_sent() {
let (addr, rx) = serve_http(None, |_, _| reply(200, "", r#"{"order":[],"posts":{}}"#));
let err = client(addr, false)
.posts_since("../users/me?x=", 0)
.unwrap_err();
assert!(matches!(err, MmError::Json(_)), "{err}");
std::thread::sleep(Duration::from_millis(100));
assert!(requests(&rx).is_empty());
}
#[test]
fn a_refused_token_stops_at_once() {
for status in [401, 403] {
let (addr, rx) = serve_http(None, move |_, _| {
reply(
status,
"",
r#"{"id":"api.context.session_expired.app_error"}"#,
)
});
let err = client(addr, false).me().unwrap_err();
assert!(matches!(err, MmError::Auth(s) if s == status), "{err}");
std::thread::sleep(Duration::from_millis(100));
assert_eq!(requests(&rx).len(), 1, "no retry on {status}");
}
}
#[test]
fn a_server_error_is_tried_twice_more() {
let (addr, rx) = serve_http(None, |n, _| {
if n < 2 {
reply(503, "", "{}")
} else {
reply(200, "", &me_body())
}
});
assert!(client(addr, false).me().is_ok());
assert_eq!(requests(&rx).len(), 3);
let (addr, rx) = serve_http(None, |_, _| reply(502, "", "{}"));
let err = client(addr, false).me().unwrap_err();
assert!(matches!(err, MmError::Status(502, _)), "{err}");
std::thread::sleep(Duration::from_millis(100));
assert_eq!(requests(&rx).len(), 3);
}
#[test]
fn a_rate_limit_is_waited_out() {
let (addr, rx) = serve_http(None, |n, _| {
if n == 0 {
reply(429, "X-Ratelimit-Reset: 1\r\n", "{}")
} else {
reply(200, "", &me_body())
}
});
let started = Instant::now();
assert!(client(addr, false).me().is_ok());
assert!(
started.elapsed() >= Duration::from_millis(900),
"{:?}",
started.elapsed()
);
assert_eq!(requests(&rx).len(), 2);
let (addr, _rx) = serve_http(None, |_, _| reply(429, "X-Ratelimit-Reset: 1\r\n", "{}"));
assert!(matches!(
client(addr, false).me(),
Err(MmError::RateLimited(_))
));
}
#[test]
fn other_statuses_keep_the_start_of_the_body() {
let long = format!("{{\"message\":\"{}\"}}", "x".repeat(1000));
let (addr, _rx) = serve_http(None, move |_, _| reply(404, "", &long));
let Err(MmError::Status(404, body)) = client(addr, false).me() else {
panic!("not a 404")
};
assert_eq!(body.chars().count(), 200);
}
#[test]
fn answers_that_are_not_what_we_expect() {
for body in [
"not json",
r#"{"id":"short","username":"x"}"#,
r#"{"id":"b0000000000000000000000000","username":""}"#,
] {
let (addr, _rx) = serve_http(None, move |_, _| reply(200, "", body));
assert!(
matches!(client(addr, false).me(), Err(MmError::Json(_))),
"{body}"
);
}
}
#[test]
fn nobody_listening_is_a_network_error() {
let port = std::net::TcpListener::bind("127.0.0.1:0")
.unwrap()
.local_addr()
.unwrap();
assert!(matches!(client(port, false).me(), Err(MmError::Net(_))));
}
docs/plans/M4a/files/crates/gatewayd/tests/net.rs
+120
View File
@@ -0,0 +1,120 @@
//! Connections, plain and TLS, against local servers with TEST-ONLY certificates (M4a spec,
//! section 5): the right CA is trusted through `ca_file`; an unknown CA and a wrong name are
//! refused, so verification is on. Do not edit.
#[path = "support/tls_server.rs"]
mod tls_server;
use std::io::{BufRead, BufReader, Write};
use std::time::Duration;
use gatewayd::config::ServerUrl;
use gatewayd::net::{Connector, NetError, Stream};
use tls_server::{echo_line, fixture, serve, server_config};
const T: Duration = Duration::from_secs(5);
fn url(tls: bool, host: &str, port: u16) -> ServerUrl {
ServerUrl {
tls,
host: host.to_string(),
port,
}
}
fn round_trip(mut stream: Stream) -> String {
stream.write_all(b"hello over the wire\n").unwrap();
stream.flush().unwrap();
let mut line = String::new();
BufReader::new(stream).read_line(&mut line).unwrap();
line
}
#[test]
fn plain_tcp() {
let addr = serve(None, echo_line);
let c = Connector::new(url(false, "127.0.0.1", addr.port()), None).unwrap();
let stream = c.connect(T).unwrap();
assert!(matches!(stream, Stream::Plain(_)));
assert_eq!(round_trip(stream), "hello over the wire\n");
}
#[test]
fn tls_to_a_server_signed_by_the_ca_file() {
let addr = serve(Some(server_config("server")), echo_line);
for host in ["localhost", "127.0.0.1"] {
let c =
Connector::new(url(true, host, addr.port()), Some(&fixture("test-ca.pem"))).unwrap();
let stream = c.connect(T).unwrap_or_else(|e| panic!("{host}: {e}"));
assert!(matches!(stream, Stream::Tls(_)));
assert_eq!(round_trip(stream), "hello over the wire\n", "{host}");
}
}
#[test]
fn an_unknown_ca_is_refused_at_connect() {
let addr = serve(Some(server_config("other-server")), echo_line);
let c = Connector::new(
url(true, "localhost", addr.port()),
Some(&fixture("test-ca.pem")),
)
.unwrap();
match c.connect(T) {
Err(NetError::Tls(why)) => assert!(why.to_lowercase().contains("certificate"), "{why}"),
Err(e) => panic!("expected a TLS error, got {e}"),
Ok(_) => panic!("a server signed by an unknown CA was accepted"),
}
}
#[test]
fn a_wrong_name_is_refused_at_connect() {
let addr = serve(Some(server_config("wrong-name")), echo_line);
let c = Connector::new(
url(true, "localhost", addr.port()),
Some(&fixture("test-ca.pem")),
)
.unwrap();
assert!(
matches!(c.connect(T), Err(NetError::Tls(_))),
"a certificate for wrong.example was accepted for localhost"
);
}
#[test]
fn tls_to_a_plain_server_fails_and_does_not_hang() {
let addr = serve(None, |_conn| std::thread::sleep(Duration::from_secs(30)));
let c = Connector::new(
url(true, "localhost", addr.port()),
Some(&fixture("test-ca.pem")),
)
.unwrap();
let started = std::time::Instant::now();
assert!(c.connect(Duration::from_millis(300)).is_err());
assert!(started.elapsed() < Duration::from_secs(3));
}
#[test]
fn a_bad_ca_file_is_an_error_before_any_connection() {
let missing = fixture("no-such.pem");
assert!(matches!(
Connector::new(url(true, "localhost", 1), Some(&missing)),
Err(NetError::Roots(_))
));
let not_pem = fixture("README.md");
assert!(matches!(
Connector::new(url(true, "localhost", 1), Some(&not_pem)),
Err(NetError::Roots(_))
));
// A plain server needs no certificates at all.
assert!(Connector::new(url(false, "localhost", 1), Some(&missing)).is_ok());
}
#[test]
fn nothing_listening_is_a_connect_error() {
let port = {
let l = std::net::TcpListener::bind("127.0.0.1:0").unwrap();
l.local_addr().unwrap().port()
};
let c = Connector::new(url(false, "127.0.0.1", port), None).unwrap();
assert!(matches!(c.connect(T), Err(NetError::Connect(_))));
}
docs/plans/M4a/files/crates/gatewayd/tests/secrets.rs
+200
View File
@@ -0,0 +1,200 @@
//! The secret store (M4a spec, section 4). The environment is passed in as a function, so no test
//! changes the process's environment. Do not edit.
#[path = "support/tmp.rs"]
mod tmp;
use std::collections::HashMap;
use std::ffi::OsString;
use std::os::unix::fs::PermissionsExt;
use std::path::PathBuf;
use gatewayd::config::SecretSource;
use gatewayd::secrets::{RUNBOOK, RUNBOOK_FILE, load};
use tmp::TempDir;
const TOKEN: &str = "s3cret-t0ken-value";
fn env_of(pairs: &[(&str, &str)]) -> impl Fn(&str) -> Option<OsString> + use<> {
let map: HashMap<String, OsString> = pairs
.iter()
.map(|(k, v)| (k.to_string(), OsString::from(v)))
.collect();
move |k| map.get(k).cloned()
}
fn owner_file(dir: &TempDir, name: &str, text: &str, mode: u32) -> PathBuf {
let path = dir.write(name, text);
std::fs::set_permissions(&path, std::fs::Permissions::from_mode(mode)).unwrap();
path
}
fn refused(source: &SecretSource, env: &dyn Fn(&str) -> Option<OsString>, word: &str) {
let e = load("mattermost_token", source, env).expect_err(word);
let text = e.to_string();
assert!(text.contains(word), "{word}: {text}");
assert!(text.starts_with("secret mattermost_token: "), "{text}");
assert!(text.ends_with(RUNBOOK), "{text}");
assert!(
!text.contains(TOKEN),
"a refusal never shows the value: {text}"
);
}
#[test]
fn a_systemd_credential() {
let dir = TempDir::new("cred");
dir.write("creds/mattermost-token", &format!("{TOKEN}\n"));
let creds = dir.path().join("creds");
let env = env_of(&[("CREDENTIALS_DIRECTORY", creds.to_str().unwrap())]);
let got = load(
"mattermost_token",
&SecretSource::Credential("mattermost-token".into()),
&env,
)
.unwrap();
assert_eq!(
got.secret.expose(),
TOKEN,
"one trailing newline is removed"
);
assert_eq!(got.warning, None);
}
#[test]
fn a_credential_outside_systemd_or_missing_is_refused() {
let src = SecretSource::Credential("mattermost-token".into());
refused(&src, &env_of(&[]), "CREDENTIALS_DIRECTORY is not set");
let dir = TempDir::new("cred-missing");
refused(
&src,
&env_of(&[("CREDENTIALS_DIRECTORY", dir.path().to_str().unwrap())]),
"cannot read the credential",
);
}
#[test]
fn an_environment_variable() {
let got = load(
"mattermost_token",
&SecretSource::Env("MM".into()),
&env_of(&[("MM", TOKEN)]),
)
.unwrap();
assert_eq!(got.secret.expose(), TOKEN);
assert_eq!(got.warning, None, "only a file warns");
refused(&SecretSource::Env("MM".into()), &env_of(&[]), "is not set");
refused(
&SecretSource::Env("MM".into()),
&env_of(&[("MM", "")]),
"empty",
);
}
#[test]
fn an_owner_only_file_is_read_with_a_warning() {
let dir = TempDir::new("file");
for mode in [0o600, 0o400] {
let path = owner_file(
&dir,
&format!("token-{mode:o}"),
&format!("{TOKEN}\n"),
mode,
);
let got = load(
"mattermost_token",
&SecretSource::File(path.clone()),
&env_of(&[]),
)
.unwrap();
assert_eq!(got.secret.expose(), TOKEN);
let warning = got.warning.expect("a file secret warns");
assert!(
warning.starts_with(
"gatewayd: warning: secret mattermost_token is read in plaintext from "
),
"{warning}"
);
assert!(warning.contains(path.to_str().unwrap()), "{warning}");
assert!(warning.contains(RUNBOOK_FILE), "{warning}");
assert!(!warning.contains(TOKEN));
}
}
#[test]
fn a_file_anyone_else_can_read_or_that_is_not_a_plain_file_is_refused() {
let dir = TempDir::new("file-bad");
for (mode, _) in [
(0o640, "group"),
(0o604, "other"),
(0o644, "both"),
(0o660, "group write"),
] {
let path = owner_file(&dir, &format!("t-{mode:o}"), TOKEN, mode);
refused(
&SecretSource::File(path),
&env_of(&[]),
"only the owner may read it",
);
}
let target = owner_file(&dir, "real", TOKEN, 0o600);
let link = dir.path().join("link");
std::os::unix::fs::symlink(&target, &link).unwrap();
refused(&SecretSource::File(link), &env_of(&[]), "symbolic link");
std::fs::create_dir(dir.path().join("adir")).unwrap();
refused(
&SecretSource::File(dir.path().join("adir")),
&env_of(&[]),
"not a regular file",
);
refused(
&SecretSource::File(dir.path().join("missing")),
&env_of(&[]),
"cannot read",
);
refused(
&SecretSource::File(PathBuf::from("relative/token")),
&env_of(&[]),
"absolute",
);
}
#[test]
fn empty_and_non_utf8_values_are_refused() {
let dir = TempDir::new("value");
refused(
&SecretSource::File(owner_file(&dir, "empty", "", 0o600)),
&env_of(&[]),
"empty",
);
refused(
&SecretSource::File(owner_file(&dir, "nl", "\n", 0o600)),
&env_of(&[]),
"empty",
);
let path = dir.path().join("bin");
std::fs::write(&path, [0xff, 0xfe]).unwrap();
std::fs::set_permissions(&path, std::fs::Permissions::from_mode(0o600)).unwrap();
refused(&SecretSource::File(path), &env_of(&[]), "not UTF-8");
}
#[test]
fn only_one_trailing_newline_is_removed_and_spaces_stay() {
let dir = TempDir::new("trim");
let path = owner_file(&dir, "t", " a b \n\n", 0o600);
let got = load("x", &SecretSource::File(path), &env_of(&[])).unwrap();
assert_eq!(got.secret.expose(), " a b \n");
}
#[test]
fn a_secret_prints_nothing_of_itself() {
let got = load(
"mattermost_token",
&SecretSource::Env("MM".into()),
&env_of(&[("MM", TOKEN)]),
)
.unwrap();
let shown = format!("{:?} {:?}", got.secret, got);
assert!(!shown.contains(TOKEN), "{shown}");
assert!(shown.contains("Secret(…)"), "{shown}");
}
docs/plans/M4a/files/crates/gatewayd/tests/serve.rs
+213
View File
@@ -0,0 +1,213 @@
//! `gatewayd` end to end, against a fake Mattermost and a fake `loopd`: who is answered, where,
//! and how (M4a spec, sections 7 and 8). Do not edit.
#[path = "support/fake_loop.rs"]
mod fake_loop;
#[path = "support/fake_mm.rs"]
mod fake_mm;
#[path = "support/gateway.rs"]
mod gateway;
#[path = "support/tmp.rs"]
mod tmp;
use std::sync::{Mutex, mpsc};
use std::time::Duration;
use fake_loop::{done, serve_loop};
use fake_mm::{BOT, BOT_NAME, DM, EVE, EVE_DM, FakeMm, KYLE, id, post};
use gateway::{OTHER, SHARED, WAIT, config, loop_dir, read_state, start, up};
use gatewayd::serve::Stop;
use gatewayd::sessions::{BUSY, M4B_COMMAND};
use tmp::TempDir;
#[test]
fn a_direct_message_is_answered_in_its_thread_while_typing() {
let home = TempDir::new("serve-dm");
let (mm, running, turns, mut ws) = up(&home, Duration::from_millis(600));
let p1 = id('p', 1);
ws.posted(&post(&p1, KYLE, DM, "", "hello there", 5), "D");
let turn = turns.recv_timeout(WAIT).unwrap();
assert_eq!(
(turn.session.as_str(), turn.content.as_str(), turn.resume),
(format!("mm-{p1}").as_str(), "hello there", false)
);
let typing = ws.typing_within(Duration::from_millis(500));
assert!(typing.len() >= 2, "{typing:?}");
assert!(
typing.iter().all(|(c, p)| c == DM && *p == p1),
"{typing:?}"
);
let posts = mm.wait_posts(1, WAIT);
assert_eq!(
posts,
[(
DM.to_string(),
p1.clone(),
"answer to hello there".to_string()
)]
);
// Typing sent just before the answer may still be on its way; after that, it stops.
ws.typing_within(Duration::from_millis(300));
assert!(
ws.typing_within(Duration::from_millis(400)).is_empty(),
"typing stops after the answer"
);
let log = running.log();
assert!(
log.iter()
.any(|l| l == &format!("gatewayd: connected to {} as {BOT_NAME}", mm.url())),
"{log:?}"
);
assert!(matches!(running.finish(), Stop::Asked));
}
#[test]
fn anyone_else_gets_nothing_at_all() {
let home = TempDir::new("serve-stranger");
let (mm, running, turns, mut ws) = up(&home, Duration::ZERO);
let p1 = id('p', 1);
ws.posted(&post(&p1, EVE, EVE_DM, "", "secret words", 5), "D");
let mut own = post(&id('p', 2), BOT, DM, "", "my own post", 6);
own["user_id"] = serde_json::json!(BOT);
ws.posted(&own, "D");
let log = running.wait_log("not allowed");
assert!(turns.recv_timeout(Duration::from_millis(300)).is_err());
assert!(ws.typing_within(Duration::from_millis(200)).is_empty());
assert!(mm.posts().is_empty());
assert!(
log.contains(&format!(
"gatewayd: ignored post {p1} from {EVE}: not allowed"
)),
"{log:?}"
);
assert!(!log.iter().any(|l| l.contains("secret words")), "{log:?}");
}
#[test]
fn messages_during_a_turn_go_together_in_the_next() {
let home = TempDir::new("serve-burst");
loop_dir(&home);
let (release_tx, release_rx) = mpsc::channel::<()>();
let release = Mutex::new(release_rx);
let turns = serve_loop(&home.path().join("run/loop/loop.sock"), move |n, turn| {
if n == 0 {
let _ = release.lock().unwrap().recv_timeout(WAIT);
}
vec![done(&format!("answer to {}", turn.content))]
});
let mm = FakeMm::start();
let _running = start(config(&home, &mm.url(), ""));
let mut ws = mm.next_ws(WAIT);
let p1 = id('p', 1);
ws.posted(&post(&p1, KYLE, DM, "", "one", 5), "D");
assert_eq!(turns.recv_timeout(WAIT).unwrap().content, "one");
let saved = read_state(&home);
assert_eq!(
saved["in_flight"],
serde_json::json!([{"session": format!("mm-{p1}"), "channel": DM, "root": p1}])
);
ws.posted(&post(&id('p', 2), KYLE, DM, &p1, "two", 6), "D");
ws.posted(&post(&id('p', 3), KYLE, DM, &p1, "three", 7), "D");
std::thread::sleep(Duration::from_millis(200));
release_tx.send(()).unwrap();
let second = turns.recv_timeout(WAIT).unwrap();
assert_eq!(
(
second.session.as_str(),
second.content.as_str(),
second.resume
),
(format!("mm-{p1}").as_str(), "two\n\nthree", true)
);
let posts = mm.wait_posts(2, WAIT);
let texts: Vec<&str> = posts.iter().map(|(_, _, t)| t.as_str()).collect();
assert_eq!(texts, ["answer to one", "answer to two\n\nthree"]);
}
#[test]
fn a_full_queue_says_busy() {
let home = TempDir::new("serve-busy");
loop_dir(&home);
let (release_tx, release_rx) = mpsc::channel::<()>();
let release = Mutex::new(release_rx);
let _turns = serve_loop(&home.path().join("run/loop/loop.sock"), move |_, _| {
let _ = release.lock().unwrap().recv_timeout(WAIT);
vec![done("ok")]
});
let mm = FakeMm::start();
let _running = start(config(&home, &mm.url(), "queue = 1"));
let mut ws = mm.next_ws(WAIT);
let p1 = id('p', 1);
for (n, text) in ["run", "waits", "too many"].iter().enumerate() {
let n = u32::try_from(n).unwrap();
let root = if n == 0 { String::new() } else { p1.clone() };
ws.posted(
&post(&id('p', n + 1), KYLE, DM, &root, text, i64::from(n) + 5),
"D",
);
}
let posts = mm.wait_posts(1, WAIT);
assert_eq!(posts, [(DM.to_string(), p1, BUSY.to_string())]);
release_tx.send(()).unwrap();
release_tx.send(()).unwrap();
}
#[test]
fn channels_are_answered_only_when_named_or_in_our_thread() {
let home = TempDir::new("serve-channel");
let (mm, _running, turns, mut ws) = up(&home, Duration::ZERO);
let p1 = id('p', 1);
ws.posted(
&post(&id('p', 9), KYLE, SHARED, "", "hello everyone", 4),
"O",
);
ws.posted(
&post(
&id('p', 8),
KYLE,
OTHER,
"",
"@boxmaker-straylight elsewhere",
4,
),
"O",
);
ws.posted(
&post(&p1, KYLE, SHARED, "", "@boxmaker-straylight start", 5),
"O",
);
assert_eq!(
turns.recv_timeout(WAIT).unwrap().content,
"@boxmaker-straylight start"
);
mm.wait_posts(1, WAIT);
ws.posted(
&post(&id('p', 2), KYLE, SHARED, &p1, "@hermes your turn", 6),
"O",
);
ws.posted(&post(&id('p', 3), KYLE, SHARED, &p1, "and more", 7), "O");
let next = turns.recv_timeout(WAIT).unwrap();
assert_eq!((next.content.as_str(), next.resume), ("and more", true));
let posts = mm.wait_posts(2, WAIT);
assert!(
posts.iter().all(|(c, r, _)| c == SHARED && *r == p1),
"{posts:?}"
);
assert!(turns.recv_timeout(Duration::from_millis(200)).is_err());
let saved = read_state(&home);
assert!(saved["channels"].get(OTHER).is_none(), "{saved}");
assert_eq!(saved["threads"], serde_json::json!([p1]));
}
#[test]
fn commands_are_answered_without_a_turn() {
let home = TempDir::new("serve-command");
let (mm, _running, turns, mut ws) = up(&home, Duration::ZERO);
let p1 = id('p', 1);
ws.posted(&post(&p1, KYLE, DM, "", "!approve 3", 5), "D");
assert_eq!(
mm.wait_posts(1, WAIT),
[(DM.to_string(), p1, M4B_COMMAND.to_string())]
);
assert!(turns.recv_timeout(Duration::from_millis(200)).is_err());
}
docs/plans/M4a/files/crates/gatewayd/tests/serve_restart.rs
+192
View File
@@ -0,0 +1,192 @@
//! `gatewayd` end to end across gaps: a restart, a lost connection, an unreachable server, a
//! refused token and a damaged state file (M4a spec, section 9). Do not edit.
#[path = "support/fake_loop.rs"]
mod fake_loop;
#[path = "support/fake_mm.rs"]
mod fake_mm;
#[path = "support/gateway.rs"]
mod gateway;
#[path = "support/tmp.rs"]
mod tmp;
use std::sync::{Mutex, mpsc};
use std::time::Duration;
use fake_loop::{done, serve_loop};
use fake_mm::{DM, FakeMm, KYLE, id, post};
use gateway::{SHARED, WAIT, answering, config, loop_dir, start, up};
use gatewayd::serve::{INTERRUPTED, Stop};
use tmp::TempDir;
#[test]
fn a_restart_reports_the_cut_off_turn_and_catches_up() {
let home = TempDir::new("serve-restart");
let (cut, seen, new1, new2) = (id('r', 1), id('p', 2), id('p', 3), id('p', 4));
let state = serde_json::json!({
"channels": {DM: 1000}, "recent": [seen], "threads": [],
"in_flight": [{"session": format!("mm-{cut}"), "channel": DM, "root": cut}]
});
home.write("gateway/state.json", &state.to_string());
loop_dir(&home);
let turns = answering(&home, Duration::ZERO);
let mm = FakeMm::start();
mm.set_since(
DM,
&[
post(&new2, KYLE, DM, "", "second", 2000),
post(&seen, KYLE, DM, "", "already answered", 1500),
post(&new1, KYLE, DM, "", "first", 1800),
],
);
let _running = start(config(&home, &mm.url(), ""));
let _ws = mm.next_ws(WAIT);
let first = turns.recv_timeout(WAIT).unwrap();
let second = turns.recv_timeout(WAIT).unwrap();
assert_eq!(
(first.content.as_str(), second.content.as_str()),
("first", "second")
);
let posts = mm.wait_posts(3, WAIT);
assert_eq!(posts[0], (DM.to_string(), cut, INTERRUPTED.to_string()));
assert_eq!(posts.len(), 3, "{posts:?}");
std::thread::sleep(Duration::from_millis(100));
let saved: serde_json::Value = serde_json::from_str(
&std::fs::read_to_string(home.path().join("gateway/state.json")).unwrap(),
)
.unwrap();
assert_eq!(saved["in_flight"], serde_json::json!([]));
assert_eq!(saved["channels"][DM], 2000);
}
#[test]
fn a_first_start_answers_no_history() {
let home = TempDir::new("serve-first");
loop_dir(&home);
let turns = answering(&home, Duration::ZERO);
let mm = FakeMm::start();
mm.set_since(DM, &[post(&id('p', 1), KYLE, DM, "", "old", 5)]);
let running = start(config(&home, &mm.url(), ""));
let _ws = mm.next_ws(WAIT);
running.wait_log("connected to");
assert!(turns.recv_timeout(Duration::from_millis(300)).is_err());
assert!(
!mm.calls().iter().any(|(_, p)| p.contains("since=")),
"{:?}",
mm.calls()
);
let saved: serde_json::Value = serde_json::from_str(
&std::fs::read_to_string(home.path().join("gateway/state.json")).unwrap(),
)
.unwrap();
assert!(
saved["channels"][DM].as_i64().unwrap() > 1_700_000_000_000,
"marked from now"
);
assert!(saved["channels"][SHARED].as_i64().is_some());
}
#[test]
fn a_lost_connection_is_made_again() {
let home = TempDir::new("serve-reconnect");
let (mm, running, turns, ws) = up(&home, Duration::ZERO);
ws.drop_connection();
let mut again = mm.next_ws(WAIT);
let log = running.wait_log("lost the connection");
assert!(
log.iter()
.any(|l| l.ends_with("see docs/runbook.md#mattermost-unreachable")),
"{log:?}"
);
let p1 = id('p', 1);
again.posted(&post(&p1, KYLE, DM, "", "still there?", 5), "D");
assert_eq!(turns.recv_timeout(WAIT).unwrap().content, "still there?");
assert_eq!(mm.wait_posts(1, WAIT).len(), 1);
}
#[test]
fn a_refused_token_stops_gatewayd() {
let home = TempDir::new("serve-auth");
let mm = FakeMm::start();
mm.refuse_token();
let running = start(config(&home, &mm.url(), ""));
let stop = running.join_within();
assert!(matches!(stop, Stop::Auth(401)), "{stop}");
assert_eq!(
stop.to_string(),
"gatewayd: Mattermost refused the token (401)\nsee docs/runbook.md#mattermost-auth-failed"
);
}
#[test]
fn an_unreachable_server_is_tried_again() {
let home = TempDir::new("serve-unreachable");
let port = std::net::TcpListener::bind("127.0.0.1:0")
.unwrap()
.local_addr()
.unwrap()
.port();
let url = format!("http://127.0.0.1:{port}");
let running = start(config(&home, &url, ""));
std::thread::sleep(Duration::from_millis(300));
let log = running.log();
let tries: Vec<&String> = log
.iter()
.filter(|l| l.starts_with(&format!("gatewayd: cannot reach {url}: ")))
.collect();
assert!(tries.len() >= 2, "{log:?}");
assert!(
tries
.iter()
.all(|l| l
.ends_with("; trying again in 0 s\nsee docs/runbook.md#mattermost-unreachable")),
"{tries:?}"
);
assert!(matches!(running.finish(), Stop::Asked));
}
#[test]
fn a_damaged_state_file_stops_at_once() {
let home = TempDir::new("serve-damaged");
home.write("gateway/state.json", "{not json");
let mm = FakeMm::start();
let running = start(config(&home, &mm.url(), ""));
let stop = running.join_within();
assert!(matches!(stop, Stop::State(_)), "{stop}");
assert!(
stop.to_string()
.ends_with("see docs/runbook.md#gateway-state-damaged"),
"{stop}"
);
assert!(mm.calls().is_empty(), "nothing is asked of Mattermost");
}
#[test]
fn a_reconnect_does_not_interrupt_a_running_turn() {
let home = TempDir::new("serve-reconnect-turn");
loop_dir(&home);
let (release_tx, release_rx) = mpsc::channel::<()>();
let release = Mutex::new(release_rx);
let turns = serve_loop(&home.path().join("run/loop/loop.sock"), move |_, _| {
let _ = release.lock().unwrap().recv_timeout(WAIT);
vec![done("the answer")]
});
let mm = FakeMm::start();
let running = start(config(&home, &mm.url(), ""));
let mut ws = mm.next_ws(WAIT);
let p1 = id('p', 1);
ws.posted(&post(&p1, KYLE, DM, "", "a long one", 5), "D");
turns.recv_timeout(WAIT).unwrap();
ws.drop_connection();
let _again = mm.next_ws(WAIT);
running.wait_log("lost the connection");
std::thread::sleep(Duration::from_millis(100));
release_tx.send(()).unwrap();
let posts = mm.wait_posts(1, WAIT);
std::thread::sleep(Duration::from_millis(200));
assert_eq!(
mm.posts(),
[(DM.to_string(), p1, "the answer".to_string())],
"{posts:?}"
);
}
docs/plans/M4a/files/crates/gatewayd/tests/sessions.rs
+298
View File
@@ -0,0 +1,298 @@
//! Which posts become turns, in which session, and how messages wait for a running turn (M4a spec,
//! section 7, including its table of examples). Do not edit.
use gatewayd::mm::Post;
use gatewayd::sessions::{
BUSY, Batch, Ignored, M4B_COMMAND, Message, Pushed, Queues, Route, Router, Thread,
UNKNOWN_COMMAND, named,
};
use proto::SessionId;
const BOT: &str = "b0000000000000000000000000";
const KYLE: &str = "k0000000000000000000000000";
const EVE: &str = "e0000000000000000000000000";
const DM: &str = "d0000000000000000000000000";
const SHARED: &str = "c0000000000000000000000000";
const OTHER: &str = "o0000000000000000000000000";
const ROOT: &str = "r0000000000000000000000000";
const POST: &str = "p0000000000000000000000000";
fn router() -> Router {
Router::new(
BOT,
"boxmaker-straylight",
&[KYLE.to_string()],
&[SHARED.to_string()],
)
}
fn post(channel: &str, root: &str, message: &str) -> Post {
Post {
id: POST.to_string(),
user_id: KYLE.to_string(),
channel_id: channel.to_string(),
root_id: root.to_string(),
message: message.to_string(),
create_at: 5,
delete_at: 0,
kind: String::new(),
}
}
fn known(root: &str) -> bool {
root == ROOT
}
fn unknown(_: &str) -> bool {
false
}
fn queued(route: Route) -> Message {
match route {
Route::Queue(m) => m,
other => panic!("not queued: {other:?}"),
}
}
#[test]
fn the_examples_in_the_spec() {
let r = router();
let cases = [
("", "@boxmaker-straylight summarise the audit log", true),
(ROOT, "and the older files?", true),
(ROOT, "@hermes what do you think?", false),
("", "@boxmaker-straylight @hermes compare notes", true),
("", "@boxmaker-straylightx hello", false),
("", "@channel standup in five", false),
];
for (root, message, yes) in cases {
let got = r.route(&post(SHARED, root, message), "O", &known);
assert_eq!(matches!(got, Route::Queue(_)), yes, "{message}: {got:?}");
if !yes {
assert_eq!(got, Route::Ignore(Ignored::NotForUs), "{message}");
}
}
}
#[test]
fn naming() {
assert_eq!(named("@Boxmaker-Straylight, look"), ["boxmaker-straylight"]);
assert_eq!(named("ask @boxmaker-straylight."), ["boxmaker-straylight"]);
assert_eq!(named("@a.b_c-d... and @e"), ["a.b_c-d", "e"]);
assert_eq!(named("@ alone, @@x, trailing @"), ["x"]);
assert_eq!(named("ünïcødé @ʙob @bob"), ["bob"]);
assert!(named("no names here").is_empty());
let r = router();
for message in [
"hi @BOXMAKER-STRAYLIGHT",
"@boxmaker-straylight.",
"(@boxmaker-straylight)",
] {
assert!(
matches!(
r.route(&post(SHARED, "", message), "P", &unknown),
Route::Queue(_)
),
"{message}"
);
}
}
#[test]
fn replies_in_our_thread_that_name_everyone_are_still_ours() {
let r = router();
for message in ["@here any news?", "@all done", "thanks @channel"] {
assert!(
matches!(
r.route(&post(SHARED, ROOT, message), "O", &known),
Route::Queue(_)
),
"{message}"
);
}
let got = r.route(
&post(
SHARED,
"q0000000000000000000000000",
"a reply in someone else's thread",
),
"O",
&known,
);
assert_eq!(got, Route::Ignore(Ignored::NotForUs));
}
#[test]
fn who_and_where() {
let r = router();
let mut own = post(DM, "", "hello");
own.user_id = BOT.to_string();
assert_eq!(r.route(&own, "D", &unknown), Route::Ignore(Ignored::Own));
let mut system = post(DM, "", "joined");
system.kind = "system_join_channel".to_string();
assert_eq!(
r.route(&system, "D", &unknown),
Route::Ignore(Ignored::System)
);
let mut stranger = post(DM, "", "@boxmaker-straylight hello");
stranger.user_id = EVE.to_string();
assert_eq!(
r.route(&stranger, "D", &unknown),
Route::Ignore(Ignored::NotAllowed)
);
let mut stranger_cmd = post(DM, "", "!approve 1");
stranger_cmd.user_id = EVE.to_string();
assert_eq!(
r.route(&stranger_cmd, "D", &unknown),
Route::Ignore(Ignored::NotAllowed)
);
let naming = "@boxmaker-straylight hello";
assert_eq!(
r.route(&post(OTHER, "", naming), "O", &unknown),
Route::Ignore(Ignored::NotForUs)
);
assert_eq!(
r.route(&post(SHARED, "", naming), "X", &unknown),
Route::Ignore(Ignored::NotForUs)
);
assert!(matches!(
r.route(&post(SHARED, "", naming), "G", &unknown),
Route::Queue(_)
));
assert!(matches!(
r.route(&post(DM, "", "no name needed"), "D", &unknown),
Route::Queue(_)
));
}
#[test]
fn sessions_and_threads() {
let r = router();
let top = queued(r.route(&post(DM, "", "hello"), "D", &unknown));
assert_eq!(top.session.as_str(), format!("mm-{POST}"));
assert_eq!(
top.thread,
Thread {
channel: DM.to_string(),
root: POST.to_string()
}
);
assert!(!top.resume && !top.joins_thread);
assert_eq!(top.text, "hello");
let reply = queued(r.route(&post(DM, ROOT, "more"), "D", &unknown));
assert_eq!(reply.session.as_str(), format!("mm-{ROOT}"));
assert_eq!(reply.thread.root, ROOT);
assert!(reply.resume);
let channel = queued(r.route(&post(SHARED, "", "@boxmaker-straylight hi"), "O", &unknown));
assert!(channel.joins_thread && !channel.resume);
assert_eq!(channel.text, "@boxmaker-straylight hi");
}
#[test]
fn commands() {
let r = router();
let thread = Thread {
channel: DM.to_string(),
root: ROOT.to_string(),
};
let reply = |text: &str| Route::Reply {
thread: thread.clone(),
text: text.to_string(),
};
for (message, answer) in [
("!approve 42", M4B_COMMAND),
("!deny 42 not now", M4B_COMMAND),
("!deny", M4B_COMMAND),
("!approved", UNKNOWN_COMMAND),
("!help", UNKNOWN_COMMAND),
("!", UNKNOWN_COMMAND),
] {
assert_eq!(
r.route(&post(DM, ROOT, message), "D", &unknown),
reply(answer),
"{message}"
);
}
assert_eq!(
queued(r.route(&post(DM, ROOT, "!!approve is a word"), "D", &unknown)).text,
"!approve is a word"
);
assert_eq!(
queued(r.route(&post(DM, ROOT, "!!"), "D", &unknown)).text,
"!"
);
assert_eq!(
queued(r.route(&post(DM, ROOT, " !help"), "D", &unknown)).text,
" !help"
);
}
fn message(root: &str, resume: bool, text: &str) -> Message {
Message {
session: SessionId::new(&format!("mm-{root}")).unwrap(),
thread: Thread {
channel: DM.to_string(),
root: root.to_string(),
},
resume,
text: text.to_string(),
joins_thread: false,
}
}
#[test]
fn messages_wait_for_the_running_turn_and_go_together() {
let mut q = Queues::new(3);
let first = message(ROOT, false, "one");
let Pushed::Start(batch) = q.push(first.clone()) else {
panic!("not started")
};
assert_eq!(
batch,
Batch {
session: first.session.clone(),
thread: first.thread.clone(),
resume: false,
text: "one".to_string()
}
);
assert_eq!(q.push(message(ROOT, true, "two")), Pushed::Waiting);
assert_eq!(q.push(message(ROOT, true, "three\nlines")), Pushed::Waiting);
let other = message(POST, false, "elsewhere");
assert!(
matches!(q.push(other.clone()), Pushed::Start(_)),
"another session starts at once"
);
assert_eq!(q.running(), 2);
let mut roots: Vec<String> = q.threads().into_iter().map(|t| t.root).collect();
roots.sort();
assert_eq!(roots, [POST, ROOT]);
let next = q.finish(&first.session).unwrap();
assert_eq!(
(next.text.as_str(), next.resume),
("two\n\nthree\nlines", true)
);
assert_eq!(q.finish(&first.session), None);
assert_eq!(q.finish(&other.session), None);
assert_eq!(q.running(), 0);
assert!(
matches!(q.push(message(ROOT, true, "later")), Pushed::Start(_)),
"idle again"
);
}
#[test]
fn a_full_queue_drops_the_message() {
let mut q = Queues::new(2);
let m = message(ROOT, false, "run");
assert!(matches!(q.push(m.clone()), Pushed::Start(_)));
assert_eq!(q.push(message(ROOT, true, "a")), Pushed::Waiting);
assert_eq!(q.push(message(ROOT, true, "b")), Pushed::Waiting);
assert_eq!(
q.push(message(ROOT, true, "c")),
Pushed::Full(m.thread.clone())
);
assert_eq!(q.finish(&m.session).unwrap().text, "a\n\nb");
assert!(!BUSY.is_empty());
assert_eq!(q.finish(&SessionId::new("mm-never").unwrap()), None);
}
docs/plans/M4a/files/crates/gatewayd/tests/state.rs
+178
View File
@@ -0,0 +1,178 @@
//! The state file: a first start, surviving a restart, its limits, and refusing a damaged file
//! instead of guessing (M4a spec, section 9). Do not edit.
#[path = "support/tmp.rs"]
mod tmp;
use std::os::unix::fs::PermissionsExt;
use gatewayd::state::{InFlight, RECENT_KEPT, State, StateError, THREADS_KEPT};
use tmp::TempDir;
const CHAN: &str = "c0000000000000000000000000";
const DM: &str = "d0000000000000000000000000";
fn id(n: usize) -> String {
format!("p{n:025}")
}
fn turn(n: usize) -> InFlight {
InFlight {
session: format!("mm-{}", id(n)),
channel: DM.to_string(),
root: id(n),
}
}
#[test]
fn a_first_start_then_a_restart() {
let dir = TempDir::new("state-restart");
let path = dir.path().join("gateway/state.json");
let mut s = State::load(&path).unwrap();
assert!(!path.exists(), "loading writes nothing");
assert_eq!(
(s.since(CHAN), s.channels().len(), s.seen(&id(1))),
(None, 0, false)
);
s.handled(&id(1), CHAN, 2000).unwrap();
s.handled(&id(2), CHAN, 1500).unwrap();
s.mark(DM, 3000).unwrap();
s.mark(DM, 9000).unwrap();
s.join_thread(&id(1)).unwrap();
s.start_turn(turn(1)).unwrap();
s.start_turn(turn(2)).unwrap();
s.end_turn(&format!("mm-{}", id(2))).unwrap();
let mode = std::fs::metadata(&path).unwrap().permissions().mode() & 0o777;
let dir_mode = std::fs::metadata(path.parent().unwrap())
.unwrap()
.permissions()
.mode()
& 0o777;
assert_eq!((mode, dir_mode), (0o600, 0o700));
assert!(!dir.path().join("gateway/state.json.tmp").exists());
let mut again = State::load(&path).unwrap();
assert_eq!(again.since(CHAN), Some(2000), "the mark never moves back");
assert_eq!(
again.since(DM),
Some(3000),
"mark only sets a channel without one"
);
assert_eq!(again.channels(), [CHAN, DM]);
assert!(again.seen(&id(1)) && again.seen(&id(2)));
assert!(again.knows_thread(&id(1)) && !again.knows_thread(&id(2)));
assert_eq!(again.take_in_flight().unwrap(), [turn(1)]);
assert!(
State::load(&path)
.unwrap()
.take_in_flight()
.unwrap()
.is_empty(),
"taking is saved"
);
}
#[test]
fn only_the_newest_posts_and_threads_are_kept() {
let dir = TempDir::new("state-limits");
// A full file to start from: posts 0.. and threads 0.., at their limits.
let recent: Vec<String> = (0..RECENT_KEPT).map(id).collect();
let threads: Vec<String> = (0..THREADS_KEPT).map(id).collect();
let full =
serde_json::json!({"channels": {}, "recent": recent, "threads": threads, "in_flight": []});
let path = dir.write("state.json", &full.to_string());
let mut s = State::load(&path).unwrap();
s.handled(&id(RECENT_KEPT), CHAN, 1).unwrap();
s.handled(&id(RECENT_KEPT + 1), CHAN, 1).unwrap();
s.handled(&id(RECENT_KEPT + 1), CHAN, 1).unwrap();
s.join_thread(&id(THREADS_KEPT)).unwrap();
s.join_thread(&id(3)).unwrap();
let s = State::load(&path).unwrap();
assert!(!s.seen(&id(0)) && !s.seen(&id(1)) && s.seen(&id(2)) && s.seen(&id(RECENT_KEPT + 1)));
assert!(!s.knows_thread(&id(0)) && s.knows_thread(&id(1)) && s.knows_thread(&id(THREADS_KEPT)));
let text = std::fs::read_to_string(&path).unwrap();
let v: serde_json::Value = serde_json::from_str(&text).unwrap();
assert_eq!(
v["recent"].as_array().unwrap().len(),
RECENT_KEPT,
"a repeat is not stored twice"
);
assert_eq!(v["threads"].as_array().unwrap().len(), THREADS_KEPT);
}
#[test]
fn a_damaged_file_stops_with_its_pointer() {
let dir = TempDir::new("state-damaged");
let good_turn = r#"{"session":"mm-p0000000000000000000000001","channel":"d0000000000000000000000000","root":"p0000000000000000000000001"}"#;
let cases = [
"".to_string(),
"{".to_string(),
"[]".to_string(),
r#"{"channels":{},"recent":[],"threads":[]}"#.to_string(),
r#"{"channels":{},"recent":[],"threads":[],"in_flight":[],"extra":1}"#.to_string(),
r#"{"channels":{"../x":1},"recent":[],"threads":[],"in_flight":[]}"#.to_string(),
r#"{"channels":{},"recent":["short"],"threads":[],"in_flight":[]}"#.to_string(),
r#"{"channels":{},"recent":[],"threads":["P0000000000000000000000000"],"in_flight":[]}"#
.to_string(),
format!(
r#"{{"channels":{{}},"recent":[],"threads":[],"in_flight":[{}]}}"#,
good_turn.replace("mm-", "xx-")
),
format!(
r#"{{"channels":{{}},"recent":[],"threads":[],"in_flight":[{}]}}"#,
good_turn.replace("\"d0", "\"D0")
),
];
for (n, text) in cases.iter().enumerate() {
let path = dir.write(&format!("s{n}.json"), text);
match State::load(&path) {
Err(e @ StateError::Read(..)) => {
let message = e.to_string();
assert!(
message.starts_with(&path.display().to_string()),
"{message}"
);
assert!(
message.ends_with("\nsee docs/runbook.md#gateway-state-damaged"),
"{message}"
);
}
Err(e) => panic!("{text}: {e}"),
Ok(_) => panic!("accepted: {text}"),
}
}
let ok = dir.write(
"ok.json",
&format!(
r#"{{"channels":{{"{CHAN}":5}},"recent":[],"threads":[],"in_flight":[{good_turn}]}}"#
),
);
assert_eq!(State::load(&ok).unwrap().since(CHAN), Some(5));
std::fs::create_dir(dir.path().join("adir")).unwrap();
assert!(matches!(
State::load(&dir.path().join("adir")),
Err(StateError::Read(..))
));
}
#[test]
fn a_failed_write_is_an_error_and_keeps_the_old_file() {
let dir = TempDir::new("state-readonly");
let sub = dir.path().join("gateway");
let path = sub.join("state.json");
let mut s = State::load(&path).unwrap();
s.handled(&id(1), CHAN, 5).unwrap();
let before = std::fs::read(&path).unwrap();
std::fs::set_permissions(&sub, std::fs::Permissions::from_mode(0o500)).unwrap();
let got = s.handled(&id(2), CHAN, 6);
std::fs::set_permissions(&sub, std::fs::Permissions::from_mode(0o700)).unwrap();
let e = got.unwrap_err();
assert!(matches!(e, StateError::Write(..)), "{e}");
assert!(
e.to_string()
.ends_with("see docs/runbook.md#gateway-state-damaged"),
"{e}"
);
assert_eq!(std::fs::read(&path).unwrap(), before);
}
docs/plans/M4a/files/crates/gatewayd/tests/support/fake_loop.rs
+101
View File
@@ -0,0 +1,101 @@
//! A fake `loopd` on a Unix socket: it records each turn and answers with the frames the test's
//! function gives for it. Do not edit.
#![allow(dead_code)] // each test file uses a different part of this module
use std::io::Write;
use std::os::unix::net::UnixListener;
use std::path::Path;
use std::sync::mpsc;
use proto::{
Envelope, ErrorCode, Message, PROTOCOL_VERSION, Turn, TurnDone, TurnEvent, Usage, WireError,
read_frame, write_frame,
};
const USAGE: Usage = Usage {
cache_n: 0,
prompt_n: 1,
predicted_n: 1,
reasoning_tokens: 0,
thinking_capped: false,
};
/// What the fake sends back for a turn.
pub enum Reply {
Frame(Envelope),
/// Raw bytes, for broken frames.
Bytes(Vec<u8>),
/// Stop answering this connection (the caller sees it close).
Close,
}
pub fn event(e: TurnEvent) -> Reply {
Reply::Frame(Envelope {
v: PROTOCOL_VERSION,
id: 1,
r#final: false,
msg: Message::TurnEvent(e),
})
}
pub fn done(content: &str) -> Reply {
let msg = Message::TurnDone(TurnDone {
content: content.to_string(),
usage: USAGE,
});
Reply::Frame(Envelope {
v: PROTOCOL_VERSION,
id: 1,
r#final: true,
msg,
})
}
pub fn error(code: ErrorCode, detail: &str) -> Reply {
let msg = Message::Error(WireError {
code,
detail: detail.to_string(),
});
Reply::Frame(Envelope {
v: PROTOCOL_VERSION,
id: 1,
r#final: true,
msg,
})
}
/// Listen on `socket`; `script(n, &turn)` gives the replies to the n-th turn (from 0).
pub fn serve_loop<F>(socket: &Path, script: F) -> mpsc::Receiver<Turn>
where
F: Fn(usize, &Turn) -> Vec<Reply> + Send + 'static,
{
let listener = UnixListener::bind(socket).unwrap();
let (tx, rx) = mpsc::channel();
std::thread::spawn(move || {
for (n, stream) in listener.incoming().enumerate() {
let Ok(mut stream) = stream else { continue };
let Ok(Envelope {
msg: Message::Turn(turn),
..
}) = read_frame(&mut stream)
else {
continue;
};
// Reported before the script runs: a script may wait for the test.
let _ = tx.send(turn.clone());
let replies = script(n, &turn);
for reply in replies {
let ok = match reply {
Reply::Frame(env) => write_frame(&mut stream, &env).is_ok(),
Reply::Bytes(b) => stream.write_all(&b).is_ok(),
Reply::Close => false,
};
if !ok {
break;
}
}
}
});
rx
}
docs/plans/M4a/files/crates/gatewayd/tests/support/fake_mm.rs
+331
View File
@@ -0,0 +1,331 @@
//! A fake Mattermost on 127.0.0.1, plain TCP: the four REST calls `gatewayd` makes, and the
//! WebSocket, whose events the test sends and whose requests it reads. Do not edit.
#![allow(dead_code)] // each test file uses a different part of this module
use std::collections::HashMap;
use std::io::{Read, Write};
use std::net::{Shutdown, SocketAddr, TcpListener, TcpStream};
use std::sync::{Arc, Mutex, mpsc};
use std::time::{Duration, Instant};
use gatewayd::ws::handshake::accept_for;
use serde_json::{Value, json};
pub const BOT: &str = "b0000000000000000000000000";
pub const BOT_NAME: &str = "boxmaker-straylight";
pub const KYLE: &str = "k0000000000000000000000000";
pub const EVE: &str = "e0000000000000000000000000";
/// The direct channel between the bot and Kyle, and between the bot and anyone else.
pub const DM: &str = "d0000000000000000000000000";
pub const EVE_DM: &str = "f0000000000000000000000000";
#[derive(Default)]
struct Inner {
/// Status for `users/me`: 200 unless a test sets another.
me_status: u16,
/// The body for `channels/<id>/posts?since=`, by channel.
since: HashMap<String, Value>,
/// Every post made: channel, root, message.
posts: Vec<(String, String, String)>,
/// Every REST call: method and path.
calls: Vec<(String, String)>,
}
pub struct FakeMm {
pub addr: SocketAddr,
inner: Arc<Mutex<Inner>>,
sockets: Mutex<mpsc::Receiver<WsPeer>>,
}
/// One WebSocket connection from `gatewayd`.
pub struct WsPeer {
writer: TcpStream,
/// The text of every text frame `gatewayd` sends.
pub texts: mpsc::Receiver<String>,
}
/// A post as Mattermost sends it.
pub fn post(
id: &str,
user: &str,
channel: &str,
root: &str,
message: &str,
create_at: i64,
) -> Value {
json!({
"id": id, "create_at": create_at, "update_at": create_at, "delete_at": 0, "user_id": user,
"channel_id": channel, "root_id": root, "message": message, "type": "", "props": {}
})
}
pub fn id(prefix: char, n: u32) -> String {
format!("{prefix}{n:025}")
}
fn frame(opcode: u8, payload: &[u8]) -> Vec<u8> {
let mut out = vec![0x80 | opcode];
match payload.len() {
n if n < 126 => out.push(n as u8),
n => {
out.push(126);
out.extend_from_slice(&(n as u16).to_be_bytes());
}
}
out.extend_from_slice(payload);
out
}
impl WsPeer {
pub fn event(&mut self, value: &Value) {
let _ = self
.writer
.write_all(&frame(0x1, value.to_string().as_bytes()));
}
pub fn posted(&mut self, post: &Value, channel_type: &str) {
let data = json!({"post": post.to_string(), "channel_type": channel_type, "team_id": ""});
self.event(&json!({"event": "posted", "data": data, "broadcast": {}, "seq": 1}));
}
/// End the connection without a close frame.
pub fn drop_connection(self) {
let _ = self.writer.shutdown(Shutdown::Both);
}
/// The `user_typing` requests received within `wait`, as (channel, parent).
pub fn typing_within(&self, wait: Duration) -> Vec<(String, String)> {
let until = Instant::now() + wait;
let mut got = Vec::new();
while let Ok(text) = self
.texts
.recv_timeout(until.saturating_duration_since(Instant::now()))
{
let v: Value = serde_json::from_str(&text).unwrap();
if v["action"] == "user_typing" {
let data = &v["data"];
got.push((
data["channel_id"].as_str().unwrap().to_string(),
data["parent_id"].as_str().unwrap().to_string(),
));
}
}
got
}
}
fn read_head(stream: &mut TcpStream) -> Option<String> {
let mut head = Vec::new();
let mut byte = [0u8; 1];
while !head.ends_with(b"\r\n\r\n") {
if stream.read(&mut byte).ok()? == 0 {
return None;
}
head.push(byte[0]);
}
String::from_utf8(head).ok()
}
/// Unmask the client's frames and send each text on `tx`, until the connection ends.
fn read_frames(mut stream: TcpStream, tx: mpsc::Sender<String>) {
let mut exact = |n: usize| -> Option<Vec<u8>> {
let mut buf = vec![0u8; n];
stream.read_exact(&mut buf).ok().map(|()| buf)
};
loop {
let Some(head) = exact(2) else { return };
let len = match head[1] & 0x7F {
126 => u16::from_be_bytes(exact(2).unwrap().try_into().unwrap()) as usize,
127 => return,
n => n as usize,
};
let Some(mask) = exact(4) else { return };
let Some(raw) = exact(len) else { return };
let payload: Vec<u8> = raw
.iter()
.zip(mask.iter().cycle())
.map(|(b, m)| b ^ m)
.collect();
if head[0] & 0x0F == 0x1 {
let _ = tx.send(String::from_utf8(payload).unwrap());
}
}
}
impl FakeMm {
pub fn start() -> FakeMm {
let listener = TcpListener::bind("127.0.0.1:0").unwrap();
let addr = listener.local_addr().unwrap();
let inner = Arc::new(Mutex::new(Inner {
me_status: 200,
..Inner::default()
}));
let (ws_tx, ws_rx) = mpsc::channel();
let shared = Arc::clone(&inner);
std::thread::spawn(move || {
for stream in listener.incoming() {
let Ok(stream) = stream else { continue };
let (inner, ws_tx) = (Arc::clone(&shared), ws_tx.clone());
std::thread::spawn(move || connection(stream, &inner, &ws_tx));
}
});
FakeMm {
addr,
inner,
sockets: Mutex::new(ws_rx),
}
}
pub fn url(&self) -> String {
format!("http://127.0.0.1:{}", self.addr.port())
}
pub fn refuse_token(&self) {
self.inner.lock().unwrap().me_status = 401;
}
pub fn set_since(&self, channel: &str, posts: &[Value]) {
let order: Vec<Value> = posts.iter().map(|p| p["id"].clone()).collect();
let map: serde_json::Map<String, Value> = posts
.iter()
.map(|p| (p["id"].as_str().unwrap().to_string(), p.clone()))
.collect();
self.inner
.lock()
.unwrap()
.since
.insert(channel.to_string(), json!({"order": order, "posts": map}));
}
/// The next WebSocket `gatewayd` opens, after its hello.
pub fn next_ws(&self, wait: Duration) -> WsPeer {
self.sockets
.lock()
.unwrap()
.recv_timeout(wait)
.expect("no WebSocket connection")
}
pub fn posts(&self) -> Vec<(String, String, String)> {
self.inner.lock().unwrap().posts.clone()
}
/// Wait until at least `n` posts were made, for at most `wait`.
pub fn wait_posts(&self, n: usize, wait: Duration) -> Vec<(String, String, String)> {
let until = Instant::now() + wait;
while self.posts().len() < n && Instant::now() < until {
std::thread::sleep(Duration::from_millis(10));
}
self.posts()
}
pub fn calls(&self) -> Vec<(String, String)> {
self.inner.lock().unwrap().calls.clone()
}
}
fn connection(mut stream: TcpStream, inner: &Mutex<Inner>, ws_tx: &mpsc::Sender<WsPeer>) {
let Some(head) = read_head(&mut stream) else {
return;
};
let mut words = head.split_whitespace();
let (method, path) = (
words.next().unwrap_or("").to_string(),
words.next().unwrap_or("").to_string(),
);
if path == "/api/v4/websocket" {
let key = head
.lines()
.find_map(|l| l.strip_prefix("Sec-WebSocket-Key: "))
.unwrap_or("")
.trim()
.to_string();
let reply = format!(
"HTTP/1.1 101 Switching Protocols\r\nUpgrade: websocket\r\nConnection: Upgrade\r\nSec-WebSocket-Accept: {}\r\n\r\n",
accept_for(&key)
);
let _ = stream.write_all(reply.as_bytes());
let _ = stream.write_all(&frame(
0x1,
br#"{"event":"hello","data":{},"broadcast":{},"seq":0}"#,
));
let (tx, texts) = mpsc::channel();
let reader = stream.try_clone().unwrap();
std::thread::spawn(move || read_frames(reader, tx));
let _ = ws_tx.send(WsPeer {
writer: stream,
texts,
});
return;
}
let length = head
.lines()
.find_map(|l| {
l.to_ascii_lowercase()
.strip_prefix("content-length:")
.map(|v| v.trim().parse::<usize>().unwrap_or(0))
})
.unwrap_or(0);
let mut body = vec![0u8; length];
let _ = stream.read_exact(&mut body);
let (status, answer) = rest(inner, &method, &path, &body);
let text = answer.to_string();
let reply = format!(
"HTTP/1.1 {status} X\r\nContent-Type: application/json\r\nContent-Length: {}\r\n\r\n{text}",
text.len()
);
let _ = stream.write_all(reply.as_bytes());
}
fn rest(inner: &Mutex<Inner>, method: &str, path: &str, body: &[u8]) -> (u16, Value) {
let mut inner = inner.lock().unwrap();
inner.calls.push((method.to_string(), path.to_string()));
match (method, path) {
("GET", "/api/v4/users/me") if inner.me_status == 200 => {
(200, json!({"id": BOT, "username": BOT_NAME}))
}
("GET", "/api/v4/users/me") => (
inner.me_status,
json!({"id": "api.context.session_expired.app_error"}),
),
("POST", "/api/v4/channels/direct") => {
let users: Vec<String> = serde_json::from_slice(body).unwrap();
let channel = if users.iter().any(|u| u == KYLE) {
DM
} else {
EVE_DM
};
(201, json!({"id": channel, "type": "D"}))
}
("POST", "/api/v4/posts") => {
let p: Value = serde_json::from_slice(body).unwrap();
let n = u32::try_from(inner.posts.len()).unwrap();
let (channel, root, message) = (
p["channel_id"].as_str().unwrap(),
p["root_id"].as_str().unwrap(),
p["message"].as_str().unwrap(),
);
inner
.posts
.push((channel.to_string(), root.to_string(), message.to_string()));
(201, post(&id('x', n), BOT, channel, root, message, 1))
}
("GET", p) if p.contains("/posts?since=") => {
let channel = p
.trim_start_matches("/api/v4/channels/")
.split('/')
.next()
.unwrap_or("");
(
200,
inner
.since
.get(channel)
.cloned()
.unwrap_or(json!({"order": [], "posts": {}})),
)
}
_ => (404, json!({"message": "not found"})),
}
}
docs/plans/M4a/files/crates/gatewayd/tests/support/gateway.rs
+141
View File
@@ -0,0 +1,141 @@
//! Running `gatewayd`'s serve loop in a test, against the fake Mattermost and the fake `loopd`.
//! Do not edit.
#![allow(dead_code)] // each test file uses a different part of this module
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::{Arc, Mutex, mpsc};
use std::thread::JoinHandle;
use std::time::Duration;
use gatewayd::config::Config;
use gatewayd::secrets::Secret;
use gatewayd::serve::{Stop, Tuning, run};
use proto::Turn;
use crate::fake_loop::{done, serve_loop};
use crate::fake_mm::{FakeMm, KYLE, WsPeer};
use crate::tmp::TempDir;
pub const SHARED: &str = "c0000000000000000000000000";
pub const OTHER: &str = "o0000000000000000000000000";
pub const WAIT: Duration = Duration::from_secs(5);
pub struct Running {
pub stop: Arc<AtomicBool>,
pub log: Arc<Mutex<Vec<String>>>,
pub handle: Option<JoinHandle<Stop>>,
}
impl Running {
pub fn log(&self) -> Vec<String> {
self.log.lock().unwrap().clone()
}
pub fn wait_log(&self, part: &str) -> Vec<String> {
let until = std::time::Instant::now() + WAIT;
while !self.log().iter().any(|l| l.contains(part)) {
assert!(
std::time::Instant::now() < until,
"no log line with {part:?}: {:?}",
self.log()
);
std::thread::sleep(Duration::from_millis(10));
}
self.log()
}
/// The `Stop` `run` returns by itself within 5 s; after that it is stopped, and the test fails.
pub fn join_within(mut self) -> Stop {
let handle = self.handle.take().unwrap();
let until = std::time::Instant::now() + WAIT;
while !handle.is_finished() && std::time::Instant::now() < until {
std::thread::sleep(Duration::from_millis(10));
}
self.stop.store(true, Ordering::SeqCst);
let stop = handle.join().unwrap();
assert!(!matches!(stop, Stop::Asked), "run did not stop by itself");
stop
}
pub fn finish(mut self) -> Stop {
self.stop.store(true, Ordering::SeqCst);
self.handle.take().unwrap().join().unwrap()
}
}
impl Drop for Running {
fn drop(&mut self) {
self.stop.store(true, Ordering::SeqCst);
}
}
pub fn config(home: &TempDir, url: &str, extra: &str) -> Config {
let text = format!(
r#"
[mattermost]
url = "{url}"
[secrets.mattermost_token]
env = "NOT_READ_BY_RUN"
[allow]
users = ["{KYLE}"]
channels = ["{SHARED}"]
[paths]
home = "{}"
[limits]
typing_every_ms = 100
{extra}
"#,
home.path().display()
);
Config::parse(&text).unwrap()
}
pub fn start(config: Config) -> Running {
let stop = Arc::new(AtomicBool::new(false));
let log = Arc::new(Mutex::new(Vec::new()));
let tuning = Tuning {
backoff: vec![Duration::from_millis(50)],
poll: Duration::from_millis(20),
rest_timeout: WAIT,
};
let (s, l) = (Arc::clone(&stop), Arc::clone(&log));
let handle = std::thread::spawn(move || {
let sink: gatewayd::serve::Log =
Arc::new(move |line: &str| l.lock().unwrap().push(line.to_string()));
run(config, Secret::new("TOKEN".to_string()), tuning, sink, &s)
});
Running {
stop,
log,
handle: Some(handle),
}
}
/// A fake loop that answers every turn with "answer to <content>", after `delay`.
pub fn answering(home: &TempDir, delay: Duration) -> mpsc::Receiver<Turn> {
serve_loop(&home.path().join("run/loop/loop.sock"), move |_, turn| {
std::thread::sleep(delay);
vec![done(&format!("answer to {}", turn.content))]
})
}
pub fn read_state(home: &TempDir) -> serde_json::Value {
let text = std::fs::read_to_string(home.path().join("gateway/state.json")).unwrap();
serde_json::from_str(&text).unwrap()
}
pub fn loop_dir(home: &TempDir) {
std::fs::create_dir_all(home.path().join("run/loop")).unwrap();
}
/// Start with a fake Mattermost and a fake loop; the first WebSocket is returned.
pub fn up(home: &TempDir, delay: Duration) -> (FakeMm, Running, mpsc::Receiver<Turn>, WsPeer) {
loop_dir(home);
let turns = answering(home, delay);
let mm = FakeMm::start();
let running = start(config(home, &mm.url(), ""));
let ws = mm.next_ws(WAIT);
running.wait_log("gatewayd: connected to ");
(mm, running, turns, ws)
}
docs/plans/M4a/files/crates/gatewayd/tests/support/http_server.rs
+93
View File
@@ -0,0 +1,93 @@
//! A scripted HTTP server for tests: it records each request and answers with what the test's
//! function returns for it. Built on `tls_server`. Do not edit.
#![allow(dead_code)] // each test file uses a different part of this module
use std::io::{Read, Write};
use std::net::SocketAddr;
use std::sync::atomic::{AtomicUsize, Ordering};
use std::sync::{Arc, Mutex, mpsc};
use rustls::ServerConfig;
use crate::tls_server::serve;
#[derive(Debug, Clone)]
pub struct Request {
pub method: String,
pub path: String,
/// The whole head, for tests that look for a header.
pub head: String,
pub body: Vec<u8>,
}
impl Request {
pub fn json(&self) -> serde_json::Value {
serde_json::from_slice(&self.body).unwrap()
}
}
/// A response with a JSON body.
pub fn reply(status: u16, extra_headers: &str, body: &str) -> String {
format!(
"HTTP/1.1 {status} X\r\nContent-Type: application/json\r\n{extra_headers}Content-Length: {}\r\n\r\n{body}",
body.len()
)
}
fn read_request(conn: &mut dyn Read) -> Option<Request> {
let mut head = Vec::new();
let mut byte = [0u8; 1];
while !head.ends_with(b"\r\n\r\n") {
if conn.read(&mut byte).ok()? == 0 {
return None;
}
head.push(byte[0]);
}
let head = String::from_utf8(head).ok()?;
let mut first = head.split_whitespace();
let method = first.next()?.to_string();
let path = first.next()?.to_string();
let length = head
.lines()
.find_map(|l| {
l.to_ascii_lowercase()
.strip_prefix("content-length:")
.map(|v| v.trim().parse().ok())
})
.flatten()
.unwrap_or(0);
let mut body = vec![0u8; length];
conn.read_exact(&mut body).ok()?;
Some(Request {
method,
path,
head,
body,
})
}
/// Serve requests: `answer(n, &request)` gives the response to the n-th request (from 0). Every
/// request is sent on the returned channel.
pub fn serve_http<F>(
tls: Option<Arc<ServerConfig>>,
answer: F,
) -> (SocketAddr, mpsc::Receiver<Request>)
where
F: Fn(usize, &Request) -> String + Send + Sync + 'static,
{
let (tx, rx) = mpsc::channel();
let tx = Mutex::new(tx);
let count = AtomicUsize::new(0);
let addr = serve(tls, move |mut conn| {
let Some(request) = read_request(&mut conn) else {
return;
};
let n = count.fetch_add(1, Ordering::SeqCst);
let response = answer(n, &request);
let _ = tx.lock().unwrap().send(request);
let _ = conn.write_all(response.as_bytes());
let _ = conn.flush();
});
(addr, rx)
}
docs/plans/M4a/files/crates/gatewayd/tests/support/tls_server.rs
+82
View File
@@ -0,0 +1,82 @@
//! Small TCP and TLS servers for tests, using the TEST-ONLY certificates in `fixtures/tls/`.
//! Each serves connections on its own thread with a function of the connection. Do not edit.
#![allow(dead_code)] // each test file uses a different part of this module
use std::io::{Read, Write};
use std::net::{SocketAddr, TcpListener, TcpStream};
use std::path::PathBuf;
use std::sync::Arc;
use rustls::pki_types::pem::PemObject;
use rustls::pki_types::{CertificateDer, PrivateKeyDer};
use rustls::{ServerConfig, ServerConnection, StreamOwned};
pub fn fixture(name: &str) -> PathBuf {
PathBuf::from(env!("CARGO_MANIFEST_DIR"))
.join("tests/fixtures/tls")
.join(name)
}
/// A server certificate (`server`, `wrong-name`, `other-server`) and its key.
pub fn server_config(which: &str) -> Arc<ServerConfig> {
let certs: Vec<CertificateDer<'static>> =
CertificateDer::pem_file_iter(fixture(&format!("{which}.pem")))
.unwrap()
.collect::<Result<_, _>>()
.unwrap();
let key = PrivateKeyDer::from_pem_file(fixture(&format!("{which}.key"))).unwrap();
let provider = Arc::new(rustls::crypto::ring::default_provider());
let config = ServerConfig::builder_with_provider(provider)
.with_safe_default_protocol_versions()
.unwrap()
.with_no_client_auth()
.with_single_cert(certs, key)
.unwrap();
Arc::new(config)
}
/// Anything a test server can serve: plain TCP, or TLS over it.
pub trait Conn: Read + Write + Send {}
impl<T: Read + Write + Send> Conn for T {}
/// Serve every connection on 127.0.0.1 with `handle`, in plain TCP (`tls` None) or TLS.
pub fn serve<F>(tls: Option<Arc<ServerConfig>>, handle: F) -> SocketAddr
where
F: Fn(Box<dyn Conn>) + Send + Sync + 'static,
{
let listener = TcpListener::bind("127.0.0.1:0").unwrap();
let addr = listener.local_addr().unwrap();
let handle = Arc::new(handle);
std::thread::spawn(move || {
for stream in listener.incoming() {
let Ok(stream) = stream else { continue };
let handle = Arc::clone(&handle);
let tls = tls.clone();
std::thread::spawn(move || match tls {
None => handle(Box::new(stream)),
Some(config) => {
let conn = ServerConnection::new(config).unwrap();
let tls_stream: StreamOwned<ServerConnection, TcpStream> =
StreamOwned::new(conn, stream);
handle(Box::new(tls_stream));
}
});
}
});
addr
}
/// A handler that reads one line and writes it back.
pub fn echo_line(mut conn: Box<dyn Conn>) {
let mut line = Vec::new();
let mut byte = [0u8; 1];
while conn.read(&mut byte).map(|n| n == 1).unwrap_or(false) {
line.push(byte[0]);
if byte[0] == b'\n' {
break;
}
}
let _ = conn.write_all(&line);
let _ = conn.flush();
}
docs/plans/M4a/files/crates/gatewayd/tests/support/tmp.rs
+40
View File
@@ -0,0 +1,40 @@
//! Temporary directories for tests. Do not edit.
#![allow(dead_code)] // each test file uses a different part of this module
use std::path::{Path, PathBuf};
use std::sync::atomic::{AtomicU32, Ordering};
static NEXT: AtomicU32 = AtomicU32::new(0);
pub struct TempDir(PathBuf);
impl TempDir {
pub fn new(tag: &str) -> TempDir {
let n = NEXT.fetch_add(1, Ordering::SeqCst);
let path = std::env::temp_dir().join(format!("gw-{tag}-{}-{n}", std::process::id()));
let _ = std::fs::remove_dir_all(&path);
std::fs::create_dir_all(&path).unwrap();
TempDir(path)
}
pub fn path(&self) -> &Path {
&self.0
}
/// Writes `text` to `name` inside the directory and returns the full path.
pub fn write(&self, name: &str, text: &str) -> PathBuf {
let path = self.0.join(name);
if let Some(parent) = path.parent() {
std::fs::create_dir_all(parent).unwrap();
}
std::fs::write(&path, text).unwrap();
path
}
}
impl Drop for TempDir {
fn drop(&mut self) {
let _ = std::fs::remove_dir_all(&self.0);
}
}
docs/plans/M4a/files/crates/gatewayd/tests/support/ws_server.rs
+132
View File
@@ -0,0 +1,132 @@
//! A scripted WebSocket server for tests: it accepts one handshake per connection and then lets
//! the test send raw frames and read the client's. Built on `tls_server`. Do not edit.
#![allow(dead_code)] // each test file uses a different part of this module
use std::io::{Read, Write};
use std::net::SocketAddr;
use std::sync::Arc;
use std::time::Duration;
use gatewayd::ws::handshake::accept_for;
use rustls::ServerConfig;
use crate::tls_server::{Conn, serve};
pub struct Peer {
pub conn: Box<dyn Conn>,
/// The request head the client sent, for tests that check it.
pub request: String,
}
/// A frame from the client: opcode, whether it was masked, and the unmasked payload.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct ClientFrame {
pub opcode: u8,
pub masked: bool,
pub mask: [u8; 4],
pub payload: Vec<u8>,
}
impl Peer {
pub fn send(&mut self, bytes: &[u8]) {
let _ = self.conn.write_all(bytes);
let _ = self.conn.flush();
}
/// An unmasked server frame, FIN set.
pub fn frame(&mut self, opcode: u8, payload: &[u8]) {
let mut out = vec![0x80 | opcode];
if payload.len() < 126 {
out.push(payload.len() as u8);
} else {
out.push(126);
out.extend_from_slice(&(payload.len() as u16).to_be_bytes());
}
out.extend_from_slice(payload);
self.send(&out);
}
pub fn text(&mut self, text: &str) {
self.frame(0x1, text.as_bytes());
}
fn read_exact(&mut self, n: usize) -> Option<Vec<u8>> {
let mut buf = vec![0u8; n];
self.conn.read_exact(&mut buf).ok()?;
Some(buf)
}
/// The next frame from the client, or `None` when it has gone.
pub fn read_frame(&mut self) -> Option<ClientFrame> {
let head = self.read_exact(2)?;
let opcode = head[0] & 0x0F;
let masked = head[1] & 0x80 != 0;
let len = match head[1] & 0x7F {
126 => u16::from_be_bytes(self.read_exact(2)?.try_into().ok()?) as usize,
127 => u64::from_be_bytes(self.read_exact(8)?.try_into().ok()?) as usize,
n => n as usize,
};
let mask: [u8; 4] = if masked {
self.read_exact(4)?.try_into().ok()?
} else {
[0; 4]
};
let raw = self.read_exact(len)?;
let payload = raw
.iter()
.zip(mask.iter().cycle())
.map(|(b, m)| b ^ m)
.collect();
Some(ClientFrame {
opcode,
masked,
mask,
payload,
})
}
pub fn pause(&self, d: Duration) {
std::thread::sleep(d);
}
}
/// Serve WebSocket connections: complete the handshake (or answer `refuse_with` instead), then run
/// `script` on the connection.
pub fn serve_ws<F>(
tls: Option<Arc<ServerConfig>>,
refuse_with: Option<&'static str>,
script: F,
) -> SocketAddr
where
F: Fn(Peer) + Send + Sync + 'static,
{
serve(tls, move |mut conn| {
let mut head = Vec::new();
let mut byte = [0u8; 1];
while !head.ends_with(b"\r\n\r\n") {
if conn.read(&mut byte).map(|n| n == 0).unwrap_or(true) {
return;
}
head.push(byte[0]);
}
let request = String::from_utf8_lossy(&head).into_owned();
if let Some(reply) = refuse_with {
let _ = conn.write_all(reply.as_bytes());
return;
}
let key = request
.lines()
.find_map(|l| l.strip_prefix("Sec-WebSocket-Key: "))
.unwrap_or_default()
.trim()
.to_string();
let reply = format!(
"HTTP/1.1 101 Switching Protocols\r\nUpgrade: websocket\r\nConnection: Upgrade\r\nSec-WebSocket-Accept: {}\r\n\r\n",
accept_for(&key)
);
let _ = conn.write_all(reply.as_bytes());
let _ = conn.flush();
script(Peer { conn, request });
})
}
docs/plans/M4a/files/crates/gatewayd/tests/ws_conn.rs
+241
View File
@@ -0,0 +1,241 @@
//! A WebSocket connection against a scripted server, plain and TLS: messages in order, pings both
//! ways, a dead peer, closing, and hostile input (M4a spec, section 6). Do not edit.
#[path = "support/tls_server.rs"]
mod tls_server;
#[path = "support/ws_server.rs"]
mod ws_server;
use std::io::Cursor;
use std::sync::mpsc;
use std::time::{Duration, Instant};
use gatewayd::config::ServerUrl;
use gatewayd::net::Connector;
use gatewayd::ws::WsError;
use gatewayd::ws::conn::{Timing, Ws, host_header};
use tls_server::{fixture, server_config};
use ws_server::{ClientFrame, serve_ws};
const SLOW: Timing = Timing {
ping_every: Duration::from_secs(60),
dead_after: Duration::from_secs(60),
};
/// Plenty of deterministic "random" bytes: the key, then masks.
fn random() -> Box<Cursor<Vec<u8>>> {
Box::new(Cursor::new(
(0..4096u32).map(|i| (i * 37 % 251) as u8).collect(),
))
}
fn open(port: u16, tls: bool, timing: Timing) -> Result<Ws, WsError> {
let url = ServerUrl {
tls,
host: "localhost".to_string(),
port,
};
let ca = tls.then(|| fixture("test-ca.pem"));
let c = Connector::new(url, ca.as_deref()).unwrap();
Ws::open(&c, "TOKEN", timing, random())
}
/// What the server saw, within 5 s: a missing frame fails the test instead of hanging it.
fn got<T>(rx: &mpsc::Receiver<T>) -> T {
rx.recv_timeout(Duration::from_secs(5))
.expect("the server saw nothing within 5 s")
}
/// Poll until a text message or an error, for at most 5 s.
fn next(ws: &mut Ws) -> Result<String, WsError> {
let until = Instant::now() + Duration::from_secs(5);
loop {
if let Some(t) = ws.poll(Duration::from_millis(200))? {
return Ok(t);
}
assert!(Instant::now() < until, "no message within 5 s");
}
}
#[test]
fn messages_arrive_in_order_plain_and_over_tls() {
for tls in [false, true] {
let (tx, rx) = mpsc::channel();
let config = tls.then(|| server_config("server"));
let addr = serve_ws(config, None, move |mut p| {
tx.send(p.request.clone()).unwrap();
p.text("{\"event\":\"hello\"}");
p.send(&[0x01, 0x03, b'o', b'n', b'e']);
p.send(&[0x80, 0x04, b'-', b't', b'w', b'o']);
p.text("three");
p.pause(Duration::from_secs(2));
});
let mut ws = open(addr.port(), tls, SLOW).unwrap();
let request = got(&rx);
assert!(
request.contains("Authorization: Bearer TOKEN\r\n"),
"{request}"
);
assert!(
request.starts_with("GET /api/v4/websocket HTTP/1.1\r\n"),
"{request}"
);
assert!(
request.contains(&format!("Host: localhost:{}\r\n", addr.port())),
"{request}"
);
assert_eq!(next(&mut ws).unwrap(), "{\"event\":\"hello\"}");
assert_eq!(next(&mut ws).unwrap(), "one-two");
assert_eq!(next(&mut ws).unwrap(), "three");
}
}
#[test]
fn a_ping_is_answered_with_the_same_payload() {
let (tx, rx) = mpsc::channel();
let addr = serve_ws(None, None, move |mut p| {
p.frame(0x9, b"are you there");
tx.send(p.read_frame()).unwrap();
p.text("after");
p.pause(Duration::from_secs(2));
});
let mut ws = open(addr.port(), false, SLOW).unwrap();
assert_eq!(next(&mut ws).unwrap(), "after");
let pong: ClientFrame = got(&rx).expect("a pong");
assert_eq!(
(pong.opcode, pong.masked, pong.payload.as_slice()),
(0xA, true, &b"are you there"[..])
);
}
#[test]
fn we_ping_on_schedule_and_every_frame_is_masked_differently() {
let (tx, rx) = mpsc::channel();
let addr = serve_ws(None, None, move |mut p| {
for _ in 0..3 {
tx.send(p.read_frame()).unwrap();
p.frame(0xA, b"");
}
p.pause(Duration::from_secs(2));
});
let timing = Timing {
ping_every: Duration::from_millis(100),
dead_after: Duration::from_secs(5),
};
let mut ws = open(addr.port(), false, timing).unwrap();
ws.send_text("first").unwrap();
let started = Instant::now();
while started.elapsed() < Duration::from_millis(350) {
let _ = ws.poll(Duration::from_millis(50)).unwrap();
}
let frames: Vec<ClientFrame> = (0..3).map(|_| got(&rx).unwrap()).collect();
assert_eq!(
(frames[0].opcode, frames[0].payload.as_slice()),
(0x1, &b"first"[..])
);
assert_eq!(frames[1].opcode, 0x9, "a ping after ping_every");
assert_eq!(frames[2].opcode, 0x9);
assert!(frames.iter().all(|f| f.masked));
assert_ne!(
frames[0].mask, frames[1].mask,
"a fresh mask for every frame"
);
}
#[test]
fn silence_is_a_dead_peer() {
let addr = serve_ws(None, None, |p| p.pause(Duration::from_secs(10)));
let timing = Timing {
ping_every: Duration::from_secs(60),
dead_after: Duration::from_millis(300),
};
let mut ws = open(addr.port(), false, timing).unwrap();
let started = Instant::now();
let err = loop {
match ws.poll(Duration::from_millis(100)) {
Ok(_) => assert!(
started.elapsed() < Duration::from_secs(3),
"never declared dead"
),
Err(e) => break e,
}
};
assert!(matches!(err, WsError::Dead), "{err}");
assert!(started.elapsed() < Duration::from_secs(1));
}
#[test]
fn a_peer_that_trickles_is_alive_and_its_message_arrives() {
let addr = serve_ws(None, None, |mut p| {
for b in [0x81u8, 0x05, b'd', b'r', b'i', b'p', b's'] {
p.send(&[b]);
p.pause(Duration::from_millis(100));
}
p.pause(Duration::from_secs(2));
});
let timing = Timing {
ping_every: Duration::from_secs(60),
dead_after: Duration::from_millis(400),
};
let mut ws = open(addr.port(), false, timing).unwrap();
assert_eq!(next(&mut ws).unwrap(), "drips");
}
#[test]
fn a_close_frame_is_answered_and_ends_the_connection() {
let (tx, rx) = mpsc::channel();
let addr = serve_ws(None, None, move |mut p| {
p.frame(0x8, &[0x03, 0xE8]);
tx.send(p.read_frame()).unwrap();
});
let mut ws = open(addr.port(), false, SLOW).unwrap();
assert!(matches!(next(&mut ws), Err(WsError::Closed)));
let reply = got(&rx).expect("a close in reply");
assert_eq!(
(reply.opcode, reply.payload.as_slice()),
(0x8, &[0x03u8, 0xE8][..])
);
}
#[test]
fn a_dropped_connection_is_closed() {
let addr = serve_ws(None, None, drop);
let mut ws = open(addr.port(), false, SLOW).unwrap();
assert!(matches!(next(&mut ws), Err(WsError::Closed)));
}
#[test]
fn a_hostile_frame_is_an_error_not_a_panic() {
let addr = serve_ws(None, None, |mut p| {
p.send(&[0x81, 0xFF, 0x80, 0, 0, 0, 0, 0, 0, 0]);
p.pause(Duration::from_secs(2));
});
let mut ws = open(addr.port(), false, SLOW).unwrap();
assert!(matches!(next(&mut ws), Err(WsError::Protocol(_))));
}
#[test]
fn a_refused_handshake() {
let addr = serve_ws(
None,
Some("HTTP/1.1 401 Unauthorized\r\nContent-Length: 0\r\n\r\n"),
|_| {},
);
assert!(matches!(
open(addr.port(), false, SLOW),
Err(WsError::Handshake(_))
));
}
#[test]
fn the_host_header_names_the_port_only_when_it_is_not_the_default() {
let url = |tls, port| ServerUrl {
tls,
host: "chat.example".to_string(),
port,
};
assert_eq!(host_header(&url(true, 443)), "chat.example");
assert_eq!(host_header(&url(false, 80)), "chat.example");
assert_eq!(host_header(&url(true, 80)), "chat.example:80");
assert_eq!(host_header(&url(false, 8065)), "chat.example:8065");
}
docs/plans/M4a/files/crates/gatewayd/tests/ws_frame.rs
+469
View File
@@ -0,0 +1,469 @@
//! WebSocket frames, adversarially (M4a spec, section 6). Everything a server sends is untrusted:
//! each hostile frame must end the connection with an error, never a panic, and a length must be
//! refused before anything waits for or allocates its payload. A seeded property test compares the
//! decoder with a deliberately naive one written here, on valid streams and on random mutations of
//! them, fed in random pieces. The seed is printed on failure. Do not edit.
use gatewayd::ws::WsError;
use gatewayd::ws::frame::{
CLOSE, CONTINUATION, Decoder, Incoming, MAX_MESSAGE, PING, PONG, TEXT, encode,
};
/// How a server frame's length is written: the shortest form, or a longer one on purpose.
#[derive(Clone, Copy)]
enum Len {
Short,
Force16,
Force64,
}
/// A frame as a server sends it (unmasked unless `masked`).
fn frame(fin: bool, rsv: u8, opcode: u8, masked: bool, payload: &[u8], form: Len) -> Vec<u8> {
let mut out = vec![(if fin { 0x80 } else { 0 }) | (rsv << 4) | opcode];
let m = if masked { 0x80 } else { 0 };
let len = payload.len();
match form {
Len::Short if len < 126 => out.push(m | len as u8),
Len::Short if len <= 0xFFFF => {
out.push(m | 126);
out.extend_from_slice(&(len as u16).to_be_bytes());
}
Len::Force16 => {
out.push(m | 126);
out.extend_from_slice(&(len as u16).to_be_bytes());
}
_ => {
out.push(m | 127);
out.extend_from_slice(&(len as u64).to_be_bytes());
}
}
if masked {
out.extend_from_slice(&[1, 2, 3, 4]);
}
out.extend_from_slice(payload);
out
}
fn text(s: &str) -> Vec<u8> {
frame(true, 0, TEXT, false, s.as_bytes(), Len::Short)
}
/// Feed `bytes` in pieces of `step` and collect every message, stopping at the first error.
fn decode(bytes: &[u8], step: usize) -> (Vec<Incoming>, Option<String>) {
let mut d = Decoder::new();
let mut got = Vec::new();
for piece in bytes.chunks(step.max(1)) {
d.feed(piece);
loop {
match d.next_message() {
Ok(Some(m)) => got.push(m),
Ok(None) => break,
Err(e) => return (got, Some(e.to_string())),
}
}
}
(got, None)
}
fn fails(bytes: &[u8], why: &str) {
for step in [1, 2, 3, 7, bytes.len().max(1)] {
let (_, err) = decode(bytes, step);
assert!(err.is_some(), "{why} (fed {step} at a time) was accepted");
}
}
#[test]
fn plain_messages() {
let mut bytes = text("hello");
bytes.extend(frame(true, 0, PING, false, b"p1", Len::Short));
bytes.extend(frame(true, 0, PONG, false, b"", Len::Short));
bytes.extend(frame(
true,
0,
CLOSE,
false,
&[0x03, 0xE8, b'b', b'y', b'e'],
Len::Short,
));
let (got, err) = decode(&bytes, bytes.len());
assert_eq!(err, None);
assert_eq!(
got,
[
Incoming::Text("hello".into()),
Incoming::Ping(b"p1".to_vec()),
Incoming::Pong(Vec::new()),
Incoming::Close(Some(1000), "bye".into()),
]
);
assert_eq!(
decode(&frame(true, 0, CLOSE, false, b"", Len::Short), 1).0,
[Incoming::Close(None, String::new())]
);
}
#[test]
fn fragments_reassemble_with_control_frames_between_and_utf8_split_across_them() {
let snow = "snow ☃ man";
let bytes_of = snow.as_bytes();
let cut = snow.find('☃').unwrap() + 1; // inside the three-byte character
let mut bytes = frame(false, 0, TEXT, false, &bytes_of[..cut], Len::Short);
bytes.extend(frame(true, 0, PING, false, b"mid", Len::Short));
bytes.extend(frame(
false,
0,
CONTINUATION,
false,
&bytes_of[cut..cut + 1],
Len::Short,
));
bytes.extend(frame(
true,
0,
CONTINUATION,
false,
&bytes_of[cut + 1..],
Len::Short,
));
for step in 1..=bytes.len() {
let (got, err) = decode(&bytes, step);
assert_eq!(err, None, "step {step}");
assert_eq!(
got,
[Incoming::Ping(b"mid".to_vec()), Incoming::Text(snow.into())],
"step {step}"
);
}
}
#[test]
fn lengths_in_every_form() {
for len in [0usize, 1, 125, 126, 127, 65_535, 65_536, 100_000] {
let body = "x".repeat(len);
let (got, err) = decode(&text(&body), 4096);
assert_eq!(err, None, "{len}");
assert_eq!(got, [Incoming::Text(body)], "{len}");
}
}
#[test]
fn hostile_frames_end_the_connection() {
for rsv in [1, 2, 4] {
fails(
&frame(true, rsv, TEXT, false, b"x", Len::Short),
"a reserved bit",
);
}
fails(
&frame(true, 0, TEXT, true, b"x", Len::Short),
"a masked frame from the server",
);
for op in [2u8, 3, 7, 11, 15] {
fails(
&frame(true, 0, op, false, b"x", Len::Short),
"an unknown or binary opcode",
);
}
fails(
&frame(true, 0, PING, false, &[0u8; 126], Len::Short),
"a control frame over 125 bytes",
);
fails(
&frame(false, 0, PING, false, b"x", Len::Short),
"a fragmented control frame",
);
fails(
&frame(true, 0, CONTINUATION, false, b"x", Len::Short),
"a continuation with nothing to continue",
);
let mut inside = frame(false, 0, TEXT, false, b"a", Len::Short);
inside.extend(text("b"));
fails(&inside, "a new message inside an unfinished one");
fails(
&frame(true, 0, TEXT, false, b"x", Len::Force16),
"a 16-bit length for 1 byte",
);
fails(
&frame(true, 0, TEXT, false, &[b'y'; 200], Len::Force64),
"a 64-bit length for 200 bytes",
);
fails(
&frame(true, 0, TEXT, false, &[0xff, 0xfe], Len::Short),
"text that is not UTF-8",
);
fails(
&frame(true, 0, CLOSE, false, &[3], Len::Short),
"a close frame of one byte",
);
fails(
&frame(true, 0, CLOSE, false, &[3, 232, 0xff], Len::Short),
"a close reason that is not UTF-8",
);
}
#[test]
fn huge_lengths_are_refused_from_the_header_alone() {
// Only the header is fed: the decoder must refuse without waiting for a payload.
let top_bit = [0x81u8, 127, 0x80, 0, 0, 0, 0, 0, 0, 1];
let mut d = Decoder::new();
d.feed(&top_bit);
assert!(
d.next_message().is_err(),
"a 64-bit length with its top bit set"
);
let too_big = (MAX_MESSAGE as u64) + 1;
let mut head = vec![0x81u8, 127];
head.extend_from_slice(&too_big.to_be_bytes());
let mut d = Decoder::new();
d.feed(&head);
assert!(matches!(d.next_message(), Err(WsError::TooLarge)));
let mut head = vec![0x81u8, 127];
head.extend_from_slice(&0x7FFF_FFFF_FFFF_FFFFu64.to_be_bytes());
let mut d = Decoder::new();
d.feed(&head);
assert!(matches!(d.next_message(), Err(WsError::TooLarge)));
// Across fragments: the sum counts.
let half = MAX_MESSAGE / 2 + 1;
let mut d = Decoder::new();
d.feed(&frame(false, 0, TEXT, false, &vec![b'a'; half], Len::Short));
assert!(matches!(d.next_message(), Ok(None)));
let mut second = vec![0x00u8, 127];
second.extend_from_slice(&(half as u64).to_be_bytes());
d.feed(&second);
assert!(matches!(d.next_message(), Err(WsError::TooLarge)));
// Exactly the limit is fine.
let (got, err) = decode(&text(&"z".repeat(MAX_MESSAGE)), 65_536);
assert_eq!(err, None);
assert_eq!(got.len(), 1);
}
#[test]
fn our_frames_are_masked_and_decode_back() {
let mask = [0x11, 0x22, 0x33, 0x44];
for len in [0usize, 5, 125, 126, 65_535, 65_536] {
let payload: Vec<u8> = (0..len).map(|i| (i % 251) as u8).collect();
let bytes = encode(TEXT, &payload, mask);
assert_eq!(bytes[0], 0x80 | TEXT, "FIN and the opcode");
assert_ne!(bytes[1] & 0x80, 0, "the mask bit");
let (len_field, header) = match bytes[1] & 0x7F {
126 => (u16::from_be_bytes([bytes[2], bytes[3]]) as usize, 4),
127 => (
u64::from_be_bytes(bytes[2..10].try_into().unwrap()) as usize,
10,
),
n => (n as usize, 2),
};
assert_eq!(len_field, len);
let shortest = if len < 126 {
2
} else if len <= 0xFFFF {
4
} else {
10
};
assert_eq!(header, shortest, "the shortest length form");
assert_eq!(&bytes[header..header + 4], &mask);
let unmasked: Vec<u8> = bytes[header + 4..]
.iter()
.zip(mask.iter().cycle())
.map(|(b, m)| b ^ m)
.collect();
assert_eq!(unmasked, payload);
}
assert_eq!(encode(PONG, b"p", mask)[0], 0x80 | PONG);
}
// ---------- the property test ----------
struct Rng(u64);
impl Rng {
fn next(&mut self) -> u64 {
let mut x = self.0;
x ^= x << 13;
x ^= x >> 7;
x ^= x << 17;
self.0 = x;
x
}
fn below(&mut self, n: usize) -> usize {
(self.next() % n.max(1) as u64) as usize
}
}
/// The naive decoder: the whole buffer at once, the rules written out plainly.
fn naive(bytes: &[u8]) -> (Vec<Incoming>, bool) {
let mut out = Vec::new();
let mut i = 0usize;
let mut partial: Option<Vec<u8>> = None;
while i < bytes.len() {
if bytes.len() - i < 2 {
return (out, false);
}
let (b0, b1) = (bytes[i], bytes[i + 1]);
let (fin, rsv, op, masked, short) = (
b0 >> 7 == 1,
(b0 >> 4) & 7,
b0 & 15,
b1 >> 7 == 1,
(b1 & 127) as usize,
);
if rsv != 0 || masked || ![0, 1, 8, 9, 10].contains(&op) {
return (out, true);
}
let (hl, len) = if short == 126 {
if bytes.len() - i < 4 {
return (out, false);
}
let l = u16::from_be_bytes([bytes[i + 2], bytes[i + 3]]) as usize;
if l < 126 {
return (out, true);
}
(4, l)
} else if short == 127 {
if bytes.len() - i < 10 {
return (out, false);
}
let l = u64::from_be_bytes(bytes[i + 2..i + 10].try_into().unwrap());
if l >> 63 == 1 || l <= 0xFFFF {
return (out, true);
}
(10, l as usize)
} else {
(2, short)
};
let control = op >= 8;
if control && (!fin || len > 125) {
return (out, true);
}
if !control {
if (op == 1 && partial.is_some()) || (op == 0 && partial.is_none()) {
return (out, true);
}
if partial.as_ref().map_or(0, |p| p.len()) + len > MAX_MESSAGE {
return (out, true);
}
}
if bytes.len() - i - hl < len {
return (out, false);
}
let payload = bytes[i + hl..i + hl + len].to_vec();
i += hl + len;
match op {
9 => out.push(Incoming::Ping(payload)),
10 => out.push(Incoming::Pong(payload)),
8 => {
if payload.len() == 1 {
return (out, true);
}
if payload.is_empty() {
out.push(Incoming::Close(None, String::new()));
} else {
match String::from_utf8(payload[2..].to_vec()) {
Ok(r) => out.push(Incoming::Close(
Some(u16::from_be_bytes([payload[0], payload[1]])),
r,
)),
Err(_) => return (out, true),
}
}
}
_ => {
let mut m = if op == 1 {
Vec::new()
} else {
partial.take().unwrap()
};
m.extend_from_slice(&payload);
if fin {
match String::from_utf8(m) {
Ok(t) => out.push(Incoming::Text(t)),
Err(_) => return (out, true),
}
} else {
partial = Some(m);
}
}
}
}
(out, false)
}
/// A random valid stream: text messages split into random fragments, with control frames between.
fn valid_stream(rng: &mut Rng) -> Vec<u8> {
let mut bytes = Vec::new();
for _ in 0..1 + rng.below(6) {
let len = [0, 1, 50, 125, 126, 300, 70_000][rng.below(7)];
let body: String = (0..len)
.map(|k| {
if (k + rng.below(3)).is_multiple_of(29) {
'é'
} else {
'a'
}
})
.collect();
let raw = body.as_bytes();
let parts = 1 + rng.below(3);
let mut cuts: Vec<usize> = (0..parts - 1).map(|_| rng.below(raw.len() + 1)).collect();
cuts.sort();
let mut start = 0;
for (k, cut) in cuts
.iter()
.copied()
.chain(std::iter::once(raw.len()))
.enumerate()
{
let op = if k == 0 { TEXT } else { CONTINUATION };
bytes.extend(frame(
k == parts - 1,
0,
op,
false,
&raw[start..cut],
Len::Short,
));
start = cut;
if rng.below(3) == 0 {
bytes.extend(frame(
true,
0,
PING,
false,
&[rng.below(256) as u8; 3],
Len::Short,
));
}
}
}
bytes
}
#[test]
fn random_streams_agree_with_the_naive_decoder() {
for case in 0..300u64 {
let seed = 0x9E37_79B9_7F4A_7C15 ^ (case * 7919 + 1);
let mut rng = Rng(seed);
let mut bytes = valid_stream(&mut rng);
if case % 2 == 1 {
// Mutate: flip a few random bits, so most streams break somewhere different.
for _ in 0..1 + rng.below(4) {
let at = rng.below(bytes.len());
bytes[at] ^= 1 << rng.below(8);
}
}
let (want, want_err) = naive(&bytes);
let step = 1 + rng.below(4096);
let (got, got_err) = decode(&bytes, step);
assert_eq!(got, want, "seed {seed:#x}, step {step}: messages differ");
assert_eq!(
got_err.is_some(),
want_err,
"seed {seed:#x}, step {step}: {got_err:?} vs naive error {want_err}"
);
}
}
docs/plans/M4a/files/crates/gatewayd/tests/ws_handshake.rs
+200
View File
@@ -0,0 +1,200 @@
//! Base64 and the WebSocket opening handshake (RFC 4648; RFC 6455, section 4). Do not edit.
use std::io::{Cursor, Read, Write};
use gatewayd::http::Head;
use gatewayd::ws::WsError;
use gatewayd::ws::handshake::{
accept_for, base64, check_response, handshake, new_key, request_text,
};
#[test]
fn base64_vectors() {
for (input, want) in [
("", ""),
("f", "Zg=="),
("fo", "Zm8="),
("foo", "Zm9v"),
("foob", "Zm9vYg=="),
("fooba", "Zm9vYmE="),
("foobar", "Zm9vYmFy"),
] {
assert_eq!(base64(input.as_bytes()), want, "{input:?}");
}
assert_eq!(base64(&[0xff, 0xfe, 0xfd, 0x00, 0x3f]), "//79AD8=");
assert_eq!(
base64(&(0u8..=15).collect::<Vec<_>>()),
"AAECAwQFBgcICQoLDA0ODw=="
);
}
#[test]
fn the_rfc_example_accept() {
assert_eq!(
accept_for("dGhlIHNhbXBsZSBub25jZQ=="),
"s3pPLMBiTxaQ9kYGzzhZRbK+xOo="
);
}
#[test]
fn a_key_is_sixteen_random_bytes() {
let mut random = Cursor::new((0u8..=15).collect::<Vec<_>>());
assert_eq!(new_key(&mut random).unwrap(), "AAECAwQFBgcICQoLDA0ODw==");
let mut short = Cursor::new(vec![1u8; 15]);
assert!(
new_key(&mut short).is_err(),
"too few random bytes is an error, not a weak key"
);
}
#[test]
fn the_request_is_exactly_this() {
assert_eq!(
request_text("a.example", "/api/v4/websocket", "KEY==", "TOKEN"),
"GET /api/v4/websocket HTTP/1.1\r\nHost: a.example\r\nUpgrade: websocket\r\nConnection: Upgrade\r\n\
Sec-WebSocket-Key: KEY==\r\nSec-WebSocket-Version: 13\r\nAuthorization: Bearer TOKEN\r\n\r\n"
);
}
fn head(status: u16, headers: &[(&str, &str)]) -> Head {
Head {
status,
headers: headers
.iter()
.map(|(k, v)| (k.to_string(), v.to_string()))
.collect(),
}
}
const KEY: &str = "dGhlIHNhbXBsZSBub25jZQ==";
const ACCEPT: &str = "s3pPLMBiTxaQ9kYGzzhZRbK+xOo=";
#[test]
fn only_a_proper_upgrade_is_accepted() {
let good = [
("Upgrade", "websocket"),
("Connection", "Upgrade"),
("Sec-WebSocket-Accept", ACCEPT),
];
assert!(check_response(&head(101, &good), KEY).is_ok());
let loose = [
("upgrade", "WebSocket"),
("connection", "keep-alive, Upgrade"),
("sec-websocket-accept", ACCEPT),
];
assert!(
check_response(&head(101, &loose), KEY).is_ok(),
"names and tokens are case-insensitive"
);
let cases: [(u16, &[(&str, &str)]); 7] = [
(200, &good),
(401, &good),
(
101,
&[("Connection", "Upgrade"), ("Sec-WebSocket-Accept", ACCEPT)],
),
(
101,
&[
("Upgrade", "h2c"),
("Connection", "Upgrade"),
("Sec-WebSocket-Accept", ACCEPT),
],
),
(
101,
&[("Upgrade", "websocket"), ("Sec-WebSocket-Accept", ACCEPT)],
),
(101, &[("Upgrade", "websocket"), ("Connection", "Upgrade")]),
(
101,
&[
("Upgrade", "websocket"),
("Connection", "Upgrade"),
("Sec-WebSocket-Accept", "s3pplmbitxaq9kygzzhzrbk+xoo="),
],
),
];
for (status, headers) in cases {
assert!(
matches!(
check_response(&head(status, headers), KEY),
Err(WsError::Handshake(_))
),
"{status} {headers:?}"
);
}
}
/// A server side scripted as bytes; records what the client wrote.
struct Scripted {
input: Cursor<Vec<u8>>,
output: Vec<u8>,
}
impl Read for Scripted {
fn read(&mut self, buf: &mut [u8]) -> std::io::Result<usize> {
self.input.read(buf)
}
}
impl Write for Scripted {
fn write(&mut self, buf: &[u8]) -> std::io::Result<usize> {
self.output.write(buf)
}
fn flush(&mut self) -> std::io::Result<()> {
Ok(())
}
}
#[test]
fn a_whole_handshake_leaves_the_first_frame_unread() {
let key_bytes: Vec<u8> = (0u8..=15).collect();
let key = base64(&key_bytes);
let reply = format!(
"HTTP/1.1 101 Switching Protocols\r\nUpgrade: websocket\r\nConnection: Upgrade\r\nSec-WebSocket-Accept: {}\r\n\r\n",
accept_for(&key)
);
let mut bytes = reply.into_bytes();
bytes.extend_from_slice(&[0x81, 0x02, b'h', b'i']);
let mut s = Scripted {
input: Cursor::new(bytes),
output: Vec::new(),
};
handshake(
&mut s,
"a.example",
"/api/v4/websocket",
"TOKEN",
&mut Cursor::new(key_bytes),
)
.unwrap();
assert_eq!(
String::from_utf8(s.output).unwrap(),
request_text("a.example", "/api/v4/websocket", &key, "TOKEN")
);
let mut rest = Vec::new();
s.input.read_to_end(&mut rest).unwrap();
assert_eq!(rest, [0x81, 0x02, b'h', b'i']);
}
#[test]
fn a_refused_or_broken_handshake_is_a_handshake_error() {
for reply in [
&b"HTTP/1.1 401 Unauthorized\r\nContent-Length: 0\r\n\r\n"[..],
b"HTTP/1.1 101 Switching",
b"not http at all\r\n\r\n",
b"",
] {
let mut s = Scripted {
input: Cursor::new(reply.to_vec()),
output: Vec::new(),
};
let got = handshake(&mut s, "h", "/p", "t", &mut Cursor::new(vec![7u8; 16]));
assert!(
matches!(got, Err(WsError::Handshake(_))),
"{:?}: {got:?}",
String::from_utf8_lossy(reply)
);
}
}
docs/plans/M4a/files/crates/proto/src/sha1.rs
+65
View File
@@ -0,0 +1,65 @@
//! SHA-1 (FIPS 180-4), used only to check the `Sec-WebSocket-Accept` header of a WebSocket
//! handshake (RFC 6455, section 4.2.2). Never use it for anything that needs to resist attack.
/// The digest of `data`.
pub fn sha1(data: &[u8]) -> [u8; 20] {
let mut h = Sha1::new();
h.update(data);
h.finish()
}
/// SHA-1 fed in pieces.
#[derive(Debug, Clone)]
pub struct Sha1 {
state: [u32; 5],
block: [u8; 64],
filled: usize,
length: u64,
}
impl Default for Sha1 {
fn default() -> Self {
Self::new()
}
}
impl Sha1 {
pub fn new() -> Sha1 {
Sha1 {
state: [
0x6745_2301,
0xEFCD_AB89,
0x98BA_DCFE,
0x1032_5476,
0xC3D2_E1F0,
],
block: [0; 64],
filled: 0,
length: 0,
}
}
pub fn update(&mut self, mut data: &[u8]) {
// Add 8 * data.len() to `length` (wrapping; `u64::try_from`, never `as`). Copy bytes into
// `block` from `filled` on; each time it is full (64), `compress` it and set `filled` to 0.
// Use `split_at` and `get_mut(..)`, no indexing that can go out of bounds.
todo!()
}
pub fn finish(mut self) -> [u8; 20] {
// Save `length`. Feed 0x80 then zeros so that 56 bytes of the block are filled (56 -
// filled, or 120 - filled when filled >= 56), then the saved length as 8 big-endian bytes,
// through `update`. `update` adds to `length`: put the saved value back after. Then the
// five state words, big-endian.
todo!()
}
fn compress(&mut self, block: &[u8; 64]) {
// FIPS 180-4, section 6.1.2: w[0..16] are the block as big-endian u32s; w[i] = (w[i-3] ^
// w[i-8] ^ w[i-14] ^ w[i-16]).rotate_left(1) for 16..80. Eighty rounds with f and k by
// range: 0..=19 (b & c) | (!b & d), 0x5A827999; 20..=39 b ^ c ^ d, 0x6ED9EBA1; 40..=59 (b &
// c) | (b & d) | (c & d), 0x8F1BBCDC; 60..=79 b ^ c ^ d, 0xCA62C1D6. All additions
// wrapping. Add a..e into state.
todo!()
}
}
docs/plans/M4a/files/crates/proto/tests/sha1.rs
+78
View File
@@ -0,0 +1,78 @@
//! SHA-1 against FIPS 180 and RFC 3174 vectors, and against `sha1sum` for the lengths around the
//! 64-byte block where padding changes shape. Every case is also fed in pieces. Do not edit.
use proto::sha1::{Sha1, sha1};
fn hex(bytes: &[u8]) -> String {
bytes.iter().map(|b| format!("{b:02x}")).collect()
}
fn check(data: &[u8], want: &str) {
assert_eq!(hex(&sha1(data)), want, "whole, {} bytes", data.len());
for split in [
0,
1,
data.len() / 2,
data.len().saturating_sub(1),
data.len(),
] {
let split = split.min(data.len());
let mut h = Sha1::new();
h.update(&data[..split]);
h.update(&data[split..]);
assert_eq!(hex(&h.finish()), want, "split at {split} of {}", data.len());
}
let mut h = Sha1::new();
for b in data {
h.update(&[*b]);
}
assert_eq!(hex(&h.finish()), want, "byte by byte, {} bytes", data.len());
}
#[test]
fn standard_vectors() {
check(b"", "da39a3ee5e6b4b0d3255bfef95601890afd80709");
check(b"abc", "a9993e364706816aba3e25717850c26c9cd0d89d");
check(
b"abcdbcdecdefdefgefghfghighijhijkijkljklmklmnlmnomnopnopq",
"84983e441c3bd26ebaae4aa1f95129e5e54670f1",
);
check(
b"The quick brown fox jumps over the lazy dog",
"2fd4e1c67a2d28fced849ee1bb76e7391b93eb12",
);
}
#[test]
fn a_million_a() {
let data = vec![b'a'; 1_000_000];
assert_eq!(
hex(&sha1(&data)),
"34aa973cd4c4daa4f61eeb2bdbad27316534016f"
);
}
#[test]
fn lengths_around_the_block_boundary() {
// `printf 'a%.0s' $(seq N) | sha1sum`, N = 55, 56, 63, 64, 65, 119, 120.
let cases = [
(55, "c1c8bbdc22796e28c0e15163d20899b65621d65a"),
(56, "c2db330f6083854c99d4b5bfb6e8f29f201be699"),
(63, "03f09f5b158a7a8cdad920bddc29b81c18a551f5"),
(64, "0098ba824b5c16427bd7a1122a5a442a25ec644d"),
(65, "11655326c708d70319be2610e8a57d9a5b959d3b"),
(119, "ee971065aaa017e0632a8ca6c77bb3bf8b1dfc56"),
(120, "f34c1488385346a55709ba056ddd08280dd4c6d6"),
];
for (n, want) in cases {
check(&vec![b'a'; n], want);
}
}
#[test]
fn the_websocket_handshake_example() {
// RFC 6455, section 1.3: the key and the GUID give this digest (its base64 is
// "s3pPLMBiTxaQ9kYGzzhZRbK+xOo=").
let digest = sha1(b"dGhlIHNhbXBsZSBub25jZQ==258EAFA5-E914-47DA-95CA-C5AB0DC85B11");
assert_eq!(hex(&digest), "b37a4f2cc0624f1690f64606cf385945b2bec4ea");
}
docs/plans/M4a/files/deny.toml
+32
View File
@@ -0,0 +1,32 @@
# cargo-deny configuration. `make gate` runs bans, licenses and sources offline.
# `make audit` runs advisories, which fetches the RustSec database.
# Only the platforms Boxmaker is built on: straylight and the development machines (Linux), and
# the owner's Mac. Dependencies for other targets (Windows) are not judged.
[graph]
all-features = true
targets = [
"x86_64-unknown-linux-gnu",
"x86_64-unknown-linux-musl",
"aarch64-apple-darwin",
]
[licenses]
allow = ["MIT", "Apache-2.0", "Unicode-3.0", "ISC", "BSD-3-Clause"]
confidence-threshold = 0.9
[licenses.private]
ignore = true
[bans]
multiple-versions = "deny"
wildcards = "deny"
allow-wildcard-paths = true
[sources]
unknown-registry = "deny"
unknown-git = "deny"
allow-registry = ["https://github.com/rust-lang/crates.io-index"]
[advisories]
yanked = "deny"
docs/plans/M4a/files/runbook-gatewayd.md
+143
View File
@@ -0,0 +1,143 @@
## gatewayd-start-failed
**What you see.** `gatewayd` exits 1 at start, before it connects, with one line naming its config
file or a directory, then this entry.
**Why.** `gatewayd` could not read or parse `gatewayd.toml`, or a value in it is not allowed: a
`url` that is not `http://` or `https://` with a host and an optional port, an id that is not 26
characters of `a-z0-9`, an empty `[allow] users`, or a secret with not exactly one of `credential`,
`env` and `file`. Or it could not create `<home>/gateway/`. It will not guess at a configuration
that decides who it answers.
**Confirm.** The line says which: `<config path>: …` for the file (unknown keys are errors; check
it against `docs/specs/2026-09-23-m4a-gateway.md`, section 3), or `cannot prepare <dir>: …` for the
directory (`ls -ld <dir> "$(dirname <dir>)"`).
**Fix.** Correct the file or the directory's ownership. A user id is shown in Mattermost under the
user's profile, or by `GET /api/v4/users/username/<name>`.
**Check.** `gatewayd serve --config <path>` prints `gatewayd: connected to <url> as <user>`.
## secret-unavailable
**What you see.** `gatewayd` exits 1 at start with `secret <name>: <reason>`, then this entry. The
value is never printed.
**Why.** `gatewayd` does not start without its Mattermost token, and reads it only from the one
place `[secrets.<name>]` names. The reasons: for `credential`, `CREDENTIALS_DIRECTORY` is unset
(not started by systemd with a credential) or the file in it is missing; for `env`, the variable is
unset or empty; for `file`, the path is not absolute, is a symbolic link, is not a regular file, is
not owned by the user `gatewayd` runs as, or has any group or other permission (only 0600 or 0400
are accepted). An empty value is refused in every form.
**Confirm.**
```sh
systemctl --user show -p LoadCredentialEncrypted gatewayd # credential
ls -l <path>; id -u # file: owner and mode
```
**Fix.** For a credential: `systemd-creds --user encrypt --name=<credential> - <path>`, type the
token, then give the unit `LoadCredentialEncrypted=<credential>:<path>`. For a file:
`chmod 600 <path>` and `chown` it to the user `gatewayd` runs as. For an environment variable, set
it in the environment `gatewayd` starts in.
**Check.** `gatewayd` starts and prints `gatewayd: connected to <url> as <user>`.
## secret-in-a-file
**What you see.** At start: `gatewayd: warning: secret <name> is read in plaintext from <path>; a
systemd credential keeps it encrypted at rest`, then this entry. `gatewayd` runs normally.
**Why.** A file holds the token in plaintext: anyone who can read the disk, or a backup of it, can
use it. A credential is encrypted to this machine's TPM and host key. Sometimes a file is right (a
development machine, a system without systemd); the warning is there so the choice is deliberate.
**Confirm.** `[secrets.<name>]` in `gatewayd.toml` has `file = …`.
**Fix.** To keep the file: nothing; the warning stays. To move to a credential, follow the fix in
[secret-unavailable](#secret-unavailable), change the entry to `credential = "<name>"`, restart,
then delete the file and regenerate the token if the file was ever copied elsewhere.
**Check.** The warning is gone at the next start.
## mattermost-unreachable
**What you see.** `gatewayd` prints `gatewayd: cannot reach <url>: <reason>; trying again in <n> s`,
then this entry, once per attempt: after 1, 2, 5 and 10 seconds, then every 30. Posts to Boxmaker go
unanswered meanwhile; they are caught up when the connection returns.
**Why.** The TCP connection, the TLS handshake or the WebSocket upgrade failed, or the server went
silent for `dead_after_ms`. A TLS failure means the certificate did not match the host in `url` or
did not chain to the system's roots or `ca_file`; verification cannot be turned off.
**Confirm.**
```sh
curl -sS <url>/api/v4/system/ping # the server answers
tailscale status # for a tailnet url: the tailnet is up
openssl s_client -connect <host>:443 -servername <host> </dev/null | head
```
**Fix.** Start Mattermost, or the tailnet. For a certificate error, correct `url` to the name on
the certificate, or give the issuing CA in `[mattermost] ca_file`. `gatewayd` keeps trying by
itself; no restart is needed.
**Check.** `gatewayd: connected to <url> as <user>`, then a direct message to Boxmaker is answered.
## mattermost-auth-failed
**What you see.** `gatewayd` exits 1 with `gatewayd: Mattermost refused the token (<status>)`, then
this entry.
**Why.** Mattermost answered 401 or 403: the token is wrong, revoked, or belongs to a deactivated
user. Retrying with the same token cannot help, so `gatewayd` stops instead.
**Confirm.** With the token in `$T` (from the same place `gatewayd` reads it; do not paste it into a
shared shell history):
`curl -sS -o /dev/null -w '%{http_code}\n' -H "Authorization: Bearer $T" <url>/api/v4/users/me`
prints 401 or 403.
**Fix.** In Mattermost, under the bot's or user's access tokens, create a new token and revoke the
old one. Store it as [secret-unavailable](#secret-unavailable) describes, then start `gatewayd`.
**Check.** `gatewayd: connected to <url> as <user>`.
## gateway-state-damaged
**What you see.** `gatewayd` exits 1 with `<home>/gateway/state.json: <reason>`, then this entry:
at start when the file cannot be read, or while running when it cannot be written
(`…: cannot write: …`).
**Why.** The state file records which posts were handled, which threads Boxmaker is in, and which
turns were in flight. If it cannot be read or kept up to date, `gatewayd` could answer old posts
twice or miss threads, so it does not guess. A missing file is a first start and is fine.
**Confirm.** `ls -l "$BOXMAKER_HOME/gateway/state.json"` and
`python3 -m json.tool "$BOXMAKER_HOME/gateway/state.json" >/dev/null`.
**Fix.** For a write failure, free space or correct the directory's ownership (`df -h`,
`ls -ld "$BOXMAKER_HOME/gateway"`), then start `gatewayd`. If only the file's ownership or mode is
wrong, correct it. If the content is damaged, move it
aside (`mv state.json state.json.damaged`) and start again. That is a first start: posts sent while
`gatewayd` was down are not answered, and threads in channels must name Boxmaker again once.
**Check.** `gatewayd` starts, and `state.json` is rewritten after the next post.
## loop-unavailable
**What you see.** In the Mattermost thread: "Boxmaker's loop is not running
(see docs/runbook.md#loop-unavailable)". The messages that were waiting are dropped.
**Why.** `gatewayd` could not connect to `loop.sock`, or the connection closed before the turn
ended. `gatewayd` does not retry: `loopd` may have finished and logged the turn, and sending it
again would run it twice.
**Confirm.** `ls -l "$BOXMAKER_HOME/run/loop/loop.sock"` (or `[loop] socket`), and whether
`loopd serve` is running. If it stopped, its last lines say why.
**Fix.** Start `loopd serve --config <path>`; if it failed, follow the entry its message names.
Then send the message again in the thread.
**Check.** A direct message to Boxmaker is answered.