Specify and plan M3b: the runner and the tools

A draft spec for the owner's review and 13 offline tasks with their given
tests: shared tool arguments and host rules in proto, the sealed fetch
target (M3a finding 14), the toolkit tools and SOCKS5 egress proxy, and
brokerd's [runner], podman argument lists, runtime and proxy lifecycle. Each
task's tests were run against a reference at that task's end state (560 to
638 tests, clippy clean); the reference is not in the repository. Adds the
runner-unavailable runbook entry and tip T23 (ETXTBSY in script tests).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-09-22 22:29:27 -07:00
co-authored by Claude Opus 5.5
parent d988edac4a
commit b426ca1958
47 changed files with 4491 additions and 2 deletions
+3 -1
View File
@@ -10,7 +10,9 @@ time (M0 to M7, table in `docs/milestones.md`). M0 (measurements), M1 (workspace
startup self-test) and M2b (sessions, the turn loop, `loopd serve`, `bxctl chat`) are done. M3 is split: startup self-test) and M2b (sessions, the turn loop, `loopd serve`, `bxctl chat`) are done. M3 is split:
M3a (`brokerd`'s decision path, spec `docs/specs/2026-09-18-m3a-decision-path.md`) is done and M3a (`brokerd`'s decision path, spec `docs/specs/2026-09-18-m3a-decision-path.md`) is done and
merged, reviewed in `docs/implementer-log.md`; its open findings (17, 18, 20, and 14, which M3b's merged, reviewed in `docs/implementer-log.md`; its open findings (17, 18, 20, and 14, which M3b's
first task must take) are listed there. M3b (the container runner and tools) is next to plan. first task must take) are listed there. M3b (the container runner and tools) is specified in
`docs/specs/2026-09-22-m3b-runner.md` (a draft for review) and planned in `docs/plans/M3b/`
(13 tasks, all testable offline); the image and the checks on straylight are the design model's.
Straylight now serves Ornith as four slots over one 262,144-token pool; see Straylight now serves Ornith as four slots over one 262,144-token pool; see
`docs/inference-contract.md`, "Deployment change, 2026-09-20", before relying on cache behaviour. `docs/inference-contract.md`, "Deployment change, 2026-09-20", before relying on cache behaviour.
`docs/runbook.md` has an entry for every `docs/runbook.md` has an entry for every
+1
View File
@@ -6,6 +6,7 @@ Newest first. A decision that changes `docs/design.md` lands in the same commit
| Date | Decision | Reason | | Date | Decision | Reason |
|---|---|---| |---|---|---|
| 2026-09-22 | M3b spec (`docs/specs/2026-09-22-m3b-runner.md`), draft for the owner's review, written for an overnight run. New: the tools' argument structs and the host rules move to `proto`, shared by `brokerd` and `toolkit`; `toolkit` exits 0/1/2 (done / the tool's own error, shown to the model / misuse, a fixed sentence); the egress proxy refuses any allowed name that resolves to a non-public address (including the tailnet's `100.64/10` and `::/96`); grant paths may not contain `:` or `,`; `[runner]` names the image by digest; tool time limits sit under `loopd`'s 120 s broker timeout, so `shell` defaults to 100 s. | A grant allows a host on the internet, not what its name points at; two programs reading the same data need one definition; `--volume` uses `:` and `,` as separators. The shell limit follows from the M3a deadline in `BrokerPort`. |
| 2026-09-22 | M3a review fixes. `loopd` waits at most a day after a pending frame and `brokerd` refuses `[approvals] ttl_ms` over a day. A socket whose directory is `/` or a symbolic link is refused at start. A listener out of file descriptors or memory pauses and retries instead of stopping `brokerd`. `brokerd` and `bxctl` share one rule for audit log file names (`proto::is_audit_log_name`, real months and days only). The fixes were made by the design model, and the fix commits were reviewed by a separate agent before the merge. | A far `expires` parked a turn for ever; `chmod` through a link changed its target; an idle-connection flood could stop the daemon; the two components disagreed about which files were the log. Ornith was under heavy contention. | | 2026-09-22 | M3a review fixes. `loopd` waits at most a day after a pending frame and `brokerd` refuses `[approvals] ttl_ms` over a day. A socket whose directory is `/` or a symbolic link is refused at start. A listener out of file descriptors or memory pauses and retries instead of stopping `brokerd`. `brokerd` and `bxctl` share one rule for audit log file names (`proto::is_audit_log_name`, real months and days only). The fixes were made by the design model, and the fix commits were reviewed by a separate agent before the merge. | A far `expires` parked a turn for ever; `chmod` through a link changed its target; an idle-connection flood could stop the daemon; the two components disagreed about which files were the log. Ornith was under heavy contention. |
| 2026-09-18 | M3a plan checks, `brokerd`. Every tool request gets a `Decision` record, including those denied `grants_invalid` or `state_unreadable`; forbidden kinds get none. An unreadable session state is recorded as `secret`, untrusted. A refusal that cannot be recorded is `error internal`, not `ok`. The re-decision's outcome is the matched grant's mode. A pending frame that cannot be sent is handled like a lost connection. Expiry lives in `admin`. A request frame without a read timeout is accepted for M3a. | Found while writing the reference for tasks 10 to 15: the spec left each case open, and `bxctl refuse` would have reported success for a refusal that was not on disk. | | 2026-09-18 | M3a plan checks, `brokerd`. Every tool request gets a `Decision` record, including those denied `grants_invalid` or `state_unreadable`; forbidden kinds get none. An unreadable session state is recorded as `secret`, untrusted. A refusal that cannot be recorded is `error internal`, not `ok`. The re-decision's outcome is the matched grant's mode. A pending frame that cannot be sent is handled like a lost connection. Expiry lives in `admin`. A request frame without a read timeout is accepted for M3a. | Found while writing the reference for tasks 10 to 15: the spec left each case open, and `bxctl refuse` would have reported success for a refusal that was not on disk. |
| 2026-09-18 | M3a plan checks, audit. `DecisionRecord::Allowed {}` and `Ask {}` are empty struct variants. The resumed verifier accepts a break naming an older file inside a failed region too. When the previous file's last line does not parse, an ordinary start verifies the whole log. A `Recovery` that describes no line is a failure. `abandoned` and `unfinished` both name the decision's `seq`. The report carries what a `Recovery` or `AcceptedBreak` must hold. `--accept-break` with nothing to accept writes nothing. | serde ignores `deny_unknown_fields` on unit variants of an internally tagged enum, so `{"outcome":"allowed","x":1}` decoded; found by `strict.rs` in two areas, and the struct-variant fix was kept over a hand-written `try_from` as the smaller one that also refuses `"reason":null`. As specified, a correctly accepted break could stop every later start while `--accept-break` said "nothing to accept". | | 2026-09-18 | M3a plan checks, audit. `DecisionRecord::Allowed {}` and `Ask {}` are empty struct variants. The resumed verifier accepts a break naming an older file inside a failed region too. When the previous file's last line does not parse, an ordinary start verifies the whole log. A `Recovery` that describes no line is a failure. `abandoned` and `unfinished` both name the decision's `seq`. The report carries what a `Recovery` or `AcceptedBreak` must hold. `--accept-break` with nothing to accept writes nothing. | serde ignores `deny_unknown_fields` on unit variants of an internally tagged enum, so `{"outcome":"allowed","x":1}` decoded; found by `strict.rs` in two areas, and the struct-variant fix was kept over a hand-written `try_from` as the smaller one that also refuses `"reason":null`. As specified, a correctly accepted break could stop every later start while `--accept-break` said "nothing to accept". |
+1
View File
@@ -57,6 +57,7 @@ How it is used:
| T20 | The gate runs on two platforms, Talos (Linux) and the Mac (macOS), and they differ where the tests touch the OS: sockets, file sync, `/tmp`. Accept a task only when the gate passes on both. Code that sets a socket option after the peer may have closed breaks on macOS only. | M3a: `BrokerPort`, the HTTP client and the brokerd test client passed on Talos and failed on the Mac from task 13 on; it was found only when the gate was first run on the Mac. | | T20 | The gate runs on two platforms, Talos (Linux) and the Mac (macOS), and they differ where the tests touch the OS: sockets, file sync, `/tmp`. Accept a task only when the gate passes on both. Code that sets a socket option after the peer may have closed breaks on macOS only. | M3a: `BrokerPort`, the HTTP client and the brokerd test client passed on Talos and failed on the Mac from task 13 on; it was found only when the gate was first run on the Mac. |
| T21 | When two components must agree on a set (which files are the log, which names are ids), give the tests one case that walks both. Each half was tested alone and they still disagreed. | M3a finding 2: `brokerd` accepted `2026-0x-18.jsonl` as a log file and `bxctl audit verify` ignored it, calling the log `ok` while leaving out half its records. | | T21 | When two components must agree on a set (which files are the log, which names are ids), give the tests one case that walks both. Each half was tested alone and they still disagreed. | M3a finding 2: `brokerd` accepted `2026-0x-18.jsonl` as a log file and `bxctl audit verify` ignored it, calling the log `ok` while leaving out half its records. |
| T22 | List the fail-closed states a task creates, with their runbook anchors, in the task itself. A script can check that a pointer names an existing entry; nothing can check for a pointer that was never written. | M3a finding 3: four startup failures in `serve` and `main` print no pointer, and the spec's own list of pointers omitted them. | | T22 | List the fail-closed states a task creates, with their runbook anchors, in the task itself. A script can check that a pointer names an existing entry; nothing can check for a pointer that was never written. | M3a finding 3: four startup failures in `serve` and `main` print no pointer, and the spec's own list of pointers omitted them. |
| T23 | A test that writes a script and then runs it must hold a lock shared by every test in its binary that starts a process. Otherwise another test's fork can hold the new script open for writing at that moment, and running it fails with "Text file busy" (ETXTBSY), about once in seven runs. Give the lock in the support file and say why. | M3b plan checks: the fake `curl` tests failed 8 times in 40 until every forking test took `serial()`; then 0 in 60. |
## What worked and should be kept ## What worked and should be kept
+93
View File
@@ -0,0 +1,93 @@
# M3b task 01: the tools' arguments and the host rules move to `proto`
**Branch:** `m3b` (create it: `git switch master && git switch -c m3b`; `git status --short` must be
empty first, otherwise stop)
**Commit subject:** `Move the tools' arguments and the host rules to proto`
## Goal
Two programs will now read the same things: `brokerd` parses the model's tool arguments and checks
hosts, and `toolkit` (inside the container) reads the same arguments and checks the same hosts.
Each must have **one** definition. Move them into `proto`; `brokerd` uses them from there. Nothing
changes in behaviour: every existing test passes unchanged.
## Files
- Copy: `crates/proto/tests/tools.rs`, `crates/proto/tests/hosts.rs`
- Create: `crates/proto/src/tools.rs`, `crates/proto/src/hosts.rs`
- Modify: `crates/proto/src/lib.rs`, `crates/brokerd/src/args.rs`, `docs/implementer-log.md`
## Interfaces
`crates/proto/src/tools.rs` (module doc: the four tools' arguments, shared by `brokerd` and
`toolkit`):
```rust
use serde::{Deserialize, Serialize};
// Each: #[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] #[serde(deny_unknown_fields)]
pub struct ReadFileArgs { pub path: String }
pub struct WriteFileArgs { pub path: String, pub content: String }
pub struct ShellArgs {
pub command: String,
#[serde(default, skip_serializing_if = "Option::is_none")]
pub cwd: Option<String>,
}
pub struct HttpFetchArgs { pub url: String }
```
These are the four private structs at the bottom of `brokerd/src/args.rs` today (`ReadFileArgs`,
`WriteFileArgs`, `ShellArgs`, `HttpFetchArgs`), made public, with `Debug, Clone, PartialEq, Eq`
added. **Move** them: delete them from `args.rs` and `use proto::tools::{…}` there instead.
`crates/proto/src/hosts.rs` (module doc: host names and patterns, shared by `brokerd` and the
egress proxy):
```rust
pub fn valid_host(host: &str) -> bool;
pub fn valid_host_pattern(pattern: &str) -> bool;
pub fn host_matches(pattern: &str, host: &str) -> bool;
fn valid_label(label: &str) -> bool; // private, as it is now
```
**Move** these four functions from `brokerd/src/args.rs`, bodies and doc comments unchanged. In
`args.rs`, where they were, put:
```rust
/// Moved to `proto::hosts` in M3b, so `toolkit` checks hosts with the same rules.
pub use proto::hosts::{host_matches, valid_host, valid_host_pattern};
```
so that `brokerd::args::valid_host` and the others still exist for `brokerd`'s code and tests.
`url_host`, `valid_path`, `inside` and `MAX_URL` stay in `args.rs`.
In `crates/proto/src/lib.rs`: add `pub mod hosts;` and `pub mod tools;` in alphabetical order with
the other `pub mod` lines. No `pub use` for them: callers write `proto::tools::ShellArgs` and
`proto::hosts::host_matches`.
`args.rs` no longer needs `use serde::{Deserialize, Serialize};` once the structs are gone; remove
it if the compiler says it is unused.
## Steps
- [ ] **1. Branch and copy.** `git switch master && git switch -c m3b`, then
`cp docs/plans/M3b/files/crates/proto/tests/tools.rs docs/plans/M3b/files/crates/proto/tests/hosts.rs crates/proto/tests/`
- [ ] **2. See them fail.** `cargo test -p proto --test tools --test hosts`. Expected: they do not
compile (`proto::tools` and `proto::hosts` do not exist).
- [ ] **3. Move the code** as above. Run `cargo fmt --all`.
- [ ] **4. See them pass.** `cargo test -p proto --test tools --test hosts`. Expected: 3 and 3
passed. Then `cargo test -p brokerd --test args --test grants --test policy`: all pass, unchanged.
- [ ] **5. Check nothing is defined twice.** `grep -rn "fn valid_host\|struct ShellArgs" crates/`
must show only `crates/proto/src/`.
- [ ] **6. Run the gate.** `make gate`. Expected last line: `gate: ok`.
- [ ] **7. Log and commit.** Add your row to `docs/implementer-log.md`, then
`git add crates/proto crates/brokerd docs/implementer-log.md && git commit`
## Done when
- Both new suites pass, every existing suite passes, `make gate` prints `gate: ok`.
## Stop and report if
- A given test needs a change to pass.
- An existing test fails after the move.
+104
View File
@@ -0,0 +1,104 @@
# M3b task 02: seal the fetch target
**Branch:** `m3b` (run `git switch m3b`; `git status --short` must be empty, otherwise stop)
**Commit subject:** `Seal the fetch target: one value holds the URL and its host`
## Goal
Today `ToolArgs::HttpFetch { url, host }` has two public fields. Policy matches `host` against the
grant; from M3b on, the runtime fetches `url`. Any code could build one where the two disagree
(`url` of `evil.example`, `host` of `example.com`), and policy would allow a fetch of the wrong
host. Make that impossible: one sealed value, built only by `args::parse` after the URL checks.
This is M3a review finding 14.
## Files
- Copy: `crates/brokerd/tests/args.rs` (replaces the old one; one test now reads the parsed value
through getters, because it can no longer build one)
- Modify: `crates/brokerd/src/args.rs`, `crates/brokerd/src/policy.rs`,
`crates/brokerd/src/runner.rs`, `docs/implementer-log.md`
## Interfaces
In `args.rs`, the variant becomes a tuple variant holding a new struct:
```rust
pub enum ToolArgs {
ReadFile { path: String },
WriteFile { path: String, content: String },
Shell { command: String, cwd: Option<String> },
HttpFetch(FetchUrl),
}
/// A URL that passed the checks, and its host. Only `parse` makes one, so the host policy matched
/// is always the host of the URL the tool fetches.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct FetchUrl {
url: String, // private
host: String, // private
}
impl FetchUrl {
pub fn url(&self) -> &str;
pub fn host(&self) -> &str;
}
```
No `pub fn new`, no `Default`, no `Deserialize`, no public field: `parse` (in the same module) is
the only code that writes `FetchUrl { url, host }`.
Put these two doctests in `FetchUrl`'s doc comment, word for word. The first proves the struct
cannot be built outside `args`; the second proves the first fails for that reason and not a typo:
````rust
/// ```compile_fail
/// let _ = brokerd::args::FetchUrl {
/// url: "https://evil.example/".to_string(),
/// host: "example.com".to_string(),
/// };
/// ```
///
/// ```
/// let args = brokerd::args::parse(brokerd::args::ToolName::HttpFetch, r#"{"url":"https://example.com/a"}"#);
/// let Ok(brokerd::args::ToolArgs::HttpFetch(target)) = args else { panic!("{args:?}") };
/// assert_eq!((target.url(), target.host()), ("https://example.com/a", "example.com"));
/// ```
````
Every place that matched the old variant changes. There are four:
1. `ToolArgs::tool`: `ToolArgs::HttpFetch(_) => ToolName::HttpFetch`.
2. `ToolArgs::canonical_json`: `ToolArgs::HttpFetch(target) =>` serialise
`HttpFetchArgs { url: target.url.clone() }` (still only the URL: the host is not an argument).
3. `parse`'s `ToolName::HttpFetch` arm: `Ok(ToolArgs::HttpFetch(FetchUrl { url: value.url, host }))`.
4. `policy.rs`, in `covers`: `ToolArgs::HttpFetch(target) =>` and match the grant's patterns with
`host_matches(pattern, target.host())`.
In `runner.rs`: `ToolArgs::HttpFetch { .. }` becomes `ToolArgs::HttpFetch(_)`.
Search for any other: `grep -rn "HttpFetch {" crates/` must show nothing when you are done.
## Steps
- [ ] **1. Copy.** `git switch m3b`, then
`cp docs/plans/M3b/files/crates/brokerd/tests/args.rs crates/brokerd/tests/`
- [ ] **2. See it fail.** `cargo test -p brokerd --test args`. Expected: it does not compile
(`ToolArgs::HttpFetch(target)` is not a tuple variant yet).
- [ ] **3. Make the change.** Run `cargo fmt --all`.
- [ ] **4. See it pass.** `cargo test -p brokerd --test args --test policy --test runner`.
Expected: `args` 13 passed; the others pass unchanged. `cargo test -p brokerd --doc`: every
doctest passes, including your two new ones (`compile_fail` counts as passing when it fails to
compile).
- [ ] **5. Check the seal.** `grep -rn "FetchUrl {" crates/brokerd/src/` shows the struct
definition, the `parse` line, and the doctest only.
- [ ] **6. Run the gate.** `make gate`. Expected last line: `gate: ok`.
- [ ] **7. Log and commit.** `git add crates/brokerd docs/implementer-log.md && git commit`
## Done when
- `cargo test -p brokerd` and `cargo test -p brokerd --doc` pass; `make gate` prints `gate: ok`.
## Stop and report if
- Some code outside `args.rs` needs to build a `FetchUrl`: that would break the seal.
- A test other than `args.rs` needs a change.
@@ -0,0 +1,53 @@
# M3b task 03: a grant path that cannot be mounted is invalid
**Branch:** `m3b` (run `git switch m3b`; `git status --short` must be empty, otherwise stop)
**Commit subject:** `Refuse grant paths that cannot be mounted`
## Goal
From M3b, a grant's paths are mounted into the tool container as `--volume=<path>:<path>:ro`. In
that argument `:` and `,` are separators, so a path containing either would be read as something
else. A grant with such a path is **invalid**, which (as for every invalid grant) makes the whole
set invalid and denies every call until it is fixed.
## Files
- Copy: `crates/brokerd/tests/grants_mount.rs`
- Modify: `crates/brokerd/src/grants.rs`, `docs/implementer-log.md`
## The rule
In `check_grant`, the loop over `inner.constraints.paths` has two cases today: the path is `/`, or
it is not a valid path. Add a third, checked **only when the first two did not apply** (a path that
is already reported as invalid is not reported twice):
```rust
} else if path.contains([':', ',']) {
// Mounted as `--volume=<path>:<path>:ro`, where both are separators.
push(problems, file.clone(), None,
format!("{:?} cannot be mounted: it contains ':' or ','", path));
}
```
So a grant file with `paths = ["/home/kyle/a:b"]` gives exactly one problem, for that file, whose
text contains `cannot be mounted`. Other punctuation (space, `;`, `=`, `.`, `-`, `_`) stays fine.
## Steps
- [ ] **1. Copy.** `git switch m3b`, then
`cp docs/plans/M3b/files/crates/brokerd/tests/grants_mount.rs crates/brokerd/tests/`
- [ ] **2. See it fail.** `cargo test -p brokerd --test grants_mount`. Expected: 1 of 2 fails
(`a_path_with_a_colon_or_a_comma_makes_the_set_invalid`).
- [ ] **3. Add the rule.** Run `cargo fmt --all`.
- [ ] **4. See it pass.** `cargo test -p brokerd --test grants_mount --test grants`. Expected: 2 and
17 passed.
- [ ] **5. Run the gate.** `make gate`. Expected last line: `gate: ok`.
- [ ] **6. Log and commit.** `git add crates/brokerd docs/implementer-log.md && git commit`
## Done when
- Both suites pass; `make gate` prints `gate: ok`.
## Stop and report if
- A given test wants a path with `:` or `,` to be accepted, or wants two problems for one path.
+151
View File
@@ -0,0 +1,151 @@
# M3b task 04: the `toolkit` program, `read_file` and `write_file`
**Branch:** `m3b` (run `git switch m3b`; `git status --short` must be empty, otherwise stop)
**Commit subject:** `toolkit: the tool program, read_file and write_file`
## Goal
`toolkit` is the one binary inside a tool container. `brokerd` will run `toolkit <tool>` with the
tool's arguments as JSON on standard input (the `proto::tools` structs from task 01). `toolkit`
prints what the model should see on standard output and exits 0 (done), 1 (the tool could not do
it, for a reason the model should read) or 2 (it was run wrongly). It decides nothing: whether the
call may run was decided before the container existed. Spec section 4.
This task makes the program and its first two tools. `shell` and `http_fetch` come in tasks 05 and
06; until then they are unknown tools (exit 2).
## Files
- Copy: `crates/toolkit/tests/support/mod.rs`, `crates/toolkit/tests/files.rs`
- Create: `crates/toolkit/src/input.rs`, `crates/toolkit/src/files.rs`
- Replace: `crates/toolkit/src/lib.rs`, `crates/toolkit/src/main.rs` (both are stubs today)
- Modify: `crates/toolkit/Cargo.toml`, `docs/dependencies.md`, `docs/implementer-log.md`
`Cargo.toml`: under `[dependencies]`, after `proto.workspace = true`, add
`serde.workspace = true` and `serde_json.workspace = true`, each on its own line.
`docs/dependencies.md`: in the `serde` and `serde_json` rows, the "Used by" column becomes
`` `proto`, `brokerd`, `toolkit` ``.
## Interfaces
`lib.rs` (module doc: the programs that run inside tool containers):
```rust
pub mod files;
pub mod input;
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum Exit { Done, ToolError, Misuse }
impl Exit {
/// 0, 1, 2. No `as` cast: a `match`.
pub fn code(self) -> u8;
}
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Outcome { pub exit: Exit, pub stdout: String, pub stderr: String }
impl Outcome {
pub fn done(stdout: String) -> Outcome; // Done, stderr empty
pub fn tool_error(stdout: String) -> Outcome; // ToolError, stderr empty
pub fn misuse(stderr: String) -> Outcome; // Misuse, stdout EMPTY: the model sees nothing
}
/// Run tool `name` with the arguments on `stdin`.
pub fn run(name: &str, stdin: &mut dyn std::io::Read) -> Outcome;
```
`run`:
1. `input::read_input(stdin)`; an error `e` → `Outcome::misuse(format!("toolkit: {e}"))`.
2. By `name`: `"read_file"` → `input::parse::<ReadFileArgs>(name, &text)` then
`files::read_file(&args)`; `"write_file"` → the same with `WriteFileArgs` and
`files::write_file`. A parse error `e` (already a whole line) → `Outcome::misuse(e)`.
3. Any other name → `Outcome::misuse(format!("toolkit: unknown tool {name:?}"))`.
The input is read **before** the name is looked at, for every name.
`input.rs`:
```rust
pub const MAX_INPUT: usize = 2 * 1024 * 1024;
#[derive(Debug)]
pub enum InputError { TooLarge, NotUtf8, Io(std::io::Error) }
// Display, by hand:
// TooLarge → "the arguments are larger than 2097152 bytes" (write it with {MAX_INPUT})
// NotUtf8 → "the arguments are not UTF-8"
// Io(e) → "cannot read the arguments: {e}"
/// All of `stdin`, at most MAX_INPUT bytes, as UTF-8. Read at most MAX_INPUT + 1 bytes
/// (`Read::take`); more than MAX_INPUT is TooLarge.
pub fn read_input(stdin: &mut dyn std::io::Read) -> Result<String, InputError>;
/// Parse `text` as the arguments of tool `name`; the error is the whole line for stderr:
/// "toolkit: {name}: the arguments do not parse: {serde's error}".
pub fn parse<T: serde::de::DeserializeOwned>(name: &str, text: &str) -> Result<T, String>;
```
`files.rs`:
```rust
pub const MAX_READ: usize = 1024 * 1024;
pub fn read_file(args: &ReadFileArgs) -> Outcome;
pub fn write_file(args: &WriteFileArgs) -> Outcome;
```
## `read_file`: every exit
Every failure is `Outcome::tool_error(format!("read_file: {path}: {why}"))`, one line, no newline
at the end.
1. `std::fs::metadata(path)` fails with `ErrorKind::NotFound` → why = `no such file`.
2. It fails any other way → why = the error's `Display` text.
3. It is a directory → why = `is a directory`.
4. Opening fails → why = the error's text.
5. Read at most `MAX_READ + 1` bytes (`take`). Reading fails → the error's text. More than
`MAX_READ` bytes → why = `larger than 1048576 bytes` (write it with `{MAX_READ}`). Exactly
`MAX_READ` is fine.
6. Not UTF-8 → why = `not UTF-8 text`.
7. Otherwise `Outcome::done(text)`: the content exactly, nothing added.
## `write_file`: every exit
Every failure is `Outcome::tool_error(format!("write_file: {path}: {why}"))`.
1. The path's parent is not an existing directory (`Path::parent` is `None`, or not `is_dir()`) →
why = `the directory does not exist`. **No directory is ever created.**
2. The path is a directory → why = `is a directory`.
3. `std::fs::write(path, content)` fails → why = the error's text.
4. Otherwise `Outcome::done(format!("wrote {} bytes to {path}", content.len()))` (bytes, not
characters).
## `main.rs`
Read the arguments with `std::env::args_os()` (never `args()`, which panics on non-UTF-8). Skip the
program name. If there is exactly one argument, it is the tool name (`to_str()`, or `""` if it is
not UTF-8); otherwise the name is `""`, which `run` answers as an unknown tool. Call
`toolkit::run(name, &mut std::io::stdin().lock())`, write `stdout` to standard output and `stderr`
to standard error with `write_all` (ignore their errors), and return
`ExitCode::from(outcome.exit.code())`.
## Steps
- [ ] **1. Copy.** `git switch m3b`, then `mkdir -p crates/toolkit/tests/support` and
`cp docs/plans/M3b/files/crates/toolkit/tests/support/mod.rs crates/toolkit/tests/support/` and
`cp docs/plans/M3b/files/crates/toolkit/tests/files.rs crates/toolkit/tests/`
- [ ] **2. See it fail.** `cargo test -p toolkit --test files`. Expected: it does not compile.
- [ ] **3. Write the code** and the two manifest and doc changes. Run `cargo fmt --all`.
- [ ] **4. See it pass.** `cargo test -p toolkit --test files`. Expected: 7 passed. One of them
returns early and passes if you are root; you should not be.
- [ ] **5. Walk the exits.** Point at the line of your code for each numbered exit above.
- [ ] **6. Run the gate.** `make gate`. Expected last line: `gate: ok`.
- [ ] **7. Log and commit.** `git add crates/toolkit docs/dependencies.md docs/implementer-log.md && git commit`
## Done when
- `cargo test -p toolkit --test files` reports 7 passed; `make gate` prints `gate: ok`.
## Stop and report if
- A test wants a misuse to print anything on standard output.
- A test wants `write_file` to create a directory.
+80
View File
@@ -0,0 +1,80 @@
# M3b task 05: `toolkit shell`
**Branch:** `m3b` (run `git switch m3b`; `git status --short` must be empty, otherwise stop)
**Commit subject:** `toolkit: shell`
## Goal
`toolkit shell` runs one command under `/bin/sh -c`, in the call's `cwd` or in `/tmp`, and prints
its output (standard output and standard error together, in the order written) followed by how it
ended. The command's own exit status is part of the result, not `toolkit`'s: `toolkit` exits 0
whenever the command ran. Spec section 4.
## Files
- Copy: `crates/toolkit/tests/shell.rs`
- Create: `crates/toolkit/src/shell.rs`
- Modify: `crates/toolkit/src/lib.rs`, `docs/implementer-log.md`
## Interfaces
```rust
pub const SHELL: &str = "/bin/sh";
pub const DEFAULT_CWD: &str = "/tmp";
pub const MAX_OUTPUT: usize = 1024 * 1024;
pub fn shell(args: &proto::tools::ShellArgs) -> crate::Outcome;
```
In `lib.rs`: add `pub mod shell;` and, in `run`, a `"shell"` arm before the unknown-tool arm,
parsing `ShellArgs` exactly as the other two tools do and calling `shell::shell(&args)`.
## `shell`: every step and exit
1. `cwd` is `args.cwd`, or `DEFAULT_CWD` when it is `None`. If `Path::new(cwd).is_dir()` is false →
`Outcome::tool_error(format!("shell: {cwd}: no such directory"))`.
2. Make **one** pipe for both output streams: `let (mut reader, writer) = std::io::pipe()?`, and a
second write end with `writer.try_clone()`. (`std::io::pipe` is stable since Rust 1.87.) A
failure here → `tool_error(format!("shell: cannot make a pipe: {e}"))`.
3. `Command::new(SHELL).arg("-c").arg(&args.command).current_dir(cwd)`, standard input
`Stdio::null()`, standard output the first write end, standard error the second. Spawn it. A
failure → `tool_error(format!("shell: cannot start {SHELL}: {e}"))`.
4. **Drop the `Command`** right after spawning (`drop(command)`): it still holds the write ends, and
while any write end is open, reading never reaches the end.
5. Read `reader` to the end in a loop with an 8 KiB buffer. Keep the first `MAX_OUTPUT` bytes;
**keep reading after that and throw the rest away**, remembering that something was dropped. Do
not stop reading: the command would block on a full pipe. Retry `ErrorKind::Interrupted`; stop
on any other error. No indexing that can panic: use `buf.get(..n)` and the like.
6. `child.wait()`. A failure → `tool_error(format!("shell: cannot wait for the command: {e}"))`.
7. The text is the kept bytes decoded with `String::from_utf8_lossy` (invalid bytes become
U+FFFD, never an error). Then append, in this order:
- if something was dropped: `format!("\n[output after {MAX_OUTPUT} bytes dropped]")`;
- `format!("\n[exit {code}]")` if `status.code()` is `Some`, else
`format!("\n[killed by signal {n}]")` if `status.signal()` is `Some`
(`std::os::unix::process::ExitStatusExt`), else `"\n[ended without a status]"`.
8. `Outcome::done(text)`.
A command that leaves a background process holding the pipe (`sleep 100 &`) makes step 5 wait for
it; that is expected, and `brokerd`'s time limit ends it.
## Steps
- [ ] **1. Copy.** `git switch m3b`, then
`cp docs/plans/M3b/files/crates/toolkit/tests/shell.rs crates/toolkit/tests/`
- [ ] **2. See it fail.** `cargo test -p toolkit --test shell`. Expected: it does not compile.
- [ ] **3. Write `shell.rs`** and the `lib.rs` changes. Run `cargo fmt --all`.
- [ ] **4. See it pass.** `cargo test -p toolkit --test shell`. Expected: 8 passed. Run it five
times; it must pass every time.
- [ ] **5. Walk the steps.** Point at the line for steps 4 and 5 in particular.
- [ ] **6. Run the gate.** `make gate`. Expected last line: `gate: ok`.
- [ ] **7. Log and commit.** `git add crates/toolkit docs/implementer-log.md && git commit`
## Done when
- `cargo test -p toolkit --test shell` reports 8 passed five times running; `make gate` prints
`gate: ok`.
## Stop and report if
- A test hangs: that is almost always step 4 (a write end still open). Report it if dropping the
`Command` does not fix it.
+91
View File
@@ -0,0 +1,91 @@
# M3b task 06: `toolkit http_fetch`
**Branch:** `m3b` (run `git switch m3b`; `git status --short` must be empty, otherwise stop)
**Commit subject:** `toolkit: http_fetch through curl`
## Goal
`toolkit http_fetch` runs `/bin/curl` with a **fixed** argument list and the URL, through the egress
proxy's socket that `brokerd` mounts at `/run/egress/egress.sock` for this call. It prints the body
and a status line, or one line saying why the fetch failed. We do not write an HTTP or TLS client:
`curl` does that, inside the container. Spec sections 4 and 6.
## Files
- Copy: `crates/toolkit/tests/fetch.rs`
- Create: `crates/toolkit/src/fetch.rs`
- Modify: `crates/toolkit/src/lib.rs`, `docs/implementer-log.md`
## Interfaces
```rust
pub const CURL: &str = "/bin/curl";
pub const PROXY: &str = "socks5h://localhost/run/egress/egress.sock";
pub const CA_BUNDLE: &str = "/etc/ssl/certs/ca-certificates.crt";
/// `curl`'s arguments for `url`, in order, without the program name.
pub fn curl_args(url: &str) -> Vec<String>;
pub fn fetch(args: &proto::tools::HttpFetchArgs) -> crate::Outcome; // fetch_with(Path::new(CURL), args)
/// `fetch` with another `curl`, for tests.
pub fn fetch_with(curl: &std::path::Path, args: &proto::tools::HttpFetchArgs) -> crate::Outcome;
```
In `lib.rs`: `pub mod fetch;`, and an `"http_fetch"` arm in `run` like the others, calling
`fetch::fetch(&args)`.
## `curl_args`
Exactly these 21 strings, in this order (the test compares them one by one):
```
--silent --show-error --proto =https --proto-redir =https --location --max-redirs 5
--max-time 50 --max-filesize 8388608 --cacert <CA_BUNDLE> --proxy <PROXY>
--write-out "\n[http %{response_code}]" --url <url>
```
In Rust, `--write-out`'s value is the literal `"\n[http %{response_code}]"`: a real newline
character, then `[http %{response_code}]` (curl fills in the status). `--url <url>` (not a bare
URL) keeps a URL from ever being read as an option.
## `fetch_with`: every exit
1. Spawn `curl` with `curl_args(&args.url)`, standard input `Stdio::null()`, standard output and
standard error piped. A failure →
`Outcome::tool_error(format!("http_fetch: cannot start {}: {e}", curl.display()))`.
2. Read standard error **on its own thread**, keeping at most 64 KiB, while the main thread reads
standard output to the end. (Reading one and then the other can deadlock when both are full:
the given test writes 300,000 bytes to each.)
3. Wait for `curl`, then join the thread.
4. `curl` succeeded → `Outcome::done(body)`, the body decoded with `from_utf8_lossy`. The status
line is already in it, from `--write-out`.
5. It failed → `Outcome::tool_error(format!("http_fetch: {url}: {why}"))`, where `why` is the first
line of standard error that is not blank, trimmed; if there is none, `curl exited {code}`, or
`curl was killed` when there is no code. The body, if any, is not shown.
6. Waiting failed → `tool_error(format!("http_fetch: cannot wait for curl: {e}"))`.
## About the given tests
`fetch.rs` writes small shell scripts that stand in for `curl` and runs them. Every test that
starts a process takes a lock first (`let _serial = serial();`). Without it, another test's fork
can hold a just-written script open, and running it fails with "Text file busy" (ETXTBSY) about
once in seven runs. If you add a test that writes and runs a script, take the lock too.
## Steps
- [ ] **1. Copy.** `git switch m3b`, then
`cp docs/plans/M3b/files/crates/toolkit/tests/fetch.rs crates/toolkit/tests/`
- [ ] **2. See it fail.** `cargo test -p toolkit --test fetch`. Expected: it does not compile.
- [ ] **3. Write `fetch.rs`** and the `lib.rs` changes. Run `cargo fmt --all`.
- [ ] **4. See it pass.** `cargo test -p toolkit --test fetch`. Expected: 7 passed. Run it ten
times; it must pass every time.
- [ ] **5. Run the gate.** `make gate`. Expected last line: `gate: ok`.
- [ ] **6. Log and commit.** `git add crates/toolkit docs/implementer-log.md && git commit`
## Done when
- `cargo test -p toolkit --test fetch` reports 7 passed ten times running; `make gate` prints
`gate: ok`.
## Stop and report if
- A test fails with "Text file busy" even with the lock: report the run and the test.
+82
View File
@@ -0,0 +1,82 @@
# M3b task 07: which addresses are public
**Branch:** `m3b` (run `git switch m3b`; `git status --short` must be empty, otherwise stop)
**Commit subject:** `toolkit: is_public, the addresses the egress proxy may reach`
## Goal
The egress proxy (task 08) connects only to **public** addresses. A grant allows a host on the
internet; if its name resolves into the tailnet (`100.64.0.0/10`), the host (`127.0.0.1`) or a
private network, the connection is refused even though the name is allowed. This task is the one
pure function that decides it. Spec section 5, "Public".
## Files
- Copy: `crates/toolkit/tests/addr.rs`
- Create: `crates/toolkit/src/addr.rs`
- Modify: `crates/toolkit/src/lib.rs` (`pub mod addr;`), `docs/implementer-log.md`
## Interface
```rust
/// True if `ip` is a public unicast address.
pub fn is_public(ip: std::net::IpAddr) -> bool;
```
## Refused ranges: every one of these is **not** public
IPv4 (with `[a, b, c, _] = ip.octets()`):
| Range | Test |
|---|---|
| `0.0.0.0/8` | `a == 0` |
| `10.0.0.0/8` | `a == 10` |
| `100.64.0.0/10` (the tailnet) | `a == 100 && (64..=127).contains(&b)` |
| `127.0.0.0/8` | `a == 127` |
| `169.254.0.0/16` | `a == 169 && b == 254` |
| `172.16.0.0/12` | `a == 172 && (16..=31).contains(&b)` |
| `192.0.0.0/24` | `a == 192 && b == 0 && c == 0` |
| `192.0.2.0/24` | `a == 192 && b == 0 && c == 2` |
| `192.168.0.0/16` | `a == 192 && b == 168` |
| `198.18.0.0/15` | `a == 198 && (b == 18 \|\| b == 19)` |
| `198.51.100.0/24` | `a == 198 && b == 51 && c == 100` |
| `203.0.113.0/24` | `a == 203 && b == 0 && c == 113` |
| `224.0.0.0/4` and `240.0.0.0/4` | `a >= 224` |
IPv6 (with `s = ip.segments()`, eight `u16`s):
| Range | Test |
|---|---|
| `::/96` (holds `::`, `::1`, and the old IPv4-compatible form) | the first six segments are all 0 |
| IPv4-mapped `::ffff:0:0/96` | `s[0..5]` all 0 and `s[5] == 0xffff`: judge the last 32 bits as IPv4 |
| NAT64 `64:ff9b::/96` | `s[0..6] == [0x64, 0xff9b, 0, 0, 0, 0]`: judge the last 32 bits as IPv4 |
| `fc00::/7` | `s[0] & 0xfe00 == 0xfc00` |
| `fe80::/10` | `s[0] & 0xffc0 == 0xfe80` |
| `ff00::/8` | `s[0] & 0xff00 == 0xff00` |
| `2001:db8::/32` | `s[0] == 0x2001 && s[1] == 0x0db8` |
Check the three "judge as IPv4" rows **before** the others. The last 32 bits as IPv4 are
`Ipv4Addr::from(((s[6] as u32) << 16) | s[7] as u32)` in spirit, but **no `as` casts**: use
`u32::from(s[6])` and `u32::from(s[7])`, or `s[6].to_be_bytes()` and `s[7].to_be_bytes()`.
Everything else is public. Do not use the standard library's `is_global` (unstable) or add ranges
that are not in these tables: the test checks public neighbours just outside each range too.
## Steps
- [ ] **1. Copy.** `git switch m3b`, then
`cp docs/plans/M3b/files/crates/toolkit/tests/addr.rs crates/toolkit/tests/`
- [ ] **2. See it fail.** `cargo test -p toolkit --test addr`. Expected: it does not compile.
- [ ] **3. Write `addr.rs`.** Run `cargo fmt --all`.
- [ ] **4. See it pass.** `cargo test -p toolkit --test addr`. Expected: 4 passed.
- [ ] **5. Walk the tables.** Point at the line for each row of both tables.
- [ ] **6. Run the gate.** `make gate`. Expected last line: `gate: ok`.
- [ ] **7. Log and commit.** `git add crates/toolkit docs/implementer-log.md && git commit`
## Done when
- `cargo test -p toolkit --test addr` reports 4 passed; `make gate` prints `gate: ok`.
## Stop and report if
- A test's expectation disagrees with these tables.
+160
View File
@@ -0,0 +1,160 @@
# M3b task 08: the egress proxy
**Branch:** `m3b` (run `git switch m3b`; `git status --short` must be empty, otherwise stop)
**Commit subject:** `toolkit: the egress proxy`
## Goal
`http_fetch`'s container has no network. Its only way out is a Unix socket to this proxy, which
runs in a second container that does have a network. The proxy speaks the part of SOCKS5
(RFC 1928) that `curl --proxy socks5h://` uses, allows only the call's hosts, only port 443, only
public addresses (task 07), and after the handshake copies bytes both ways without looking at them
(TLS runs end to end between `curl` and the server). **Everything it reads during the handshake
comes from the untrusted side**: read exactly what the protocol says and nothing more, and never
index or allocate by a number you have not checked. Spec section 5.
## Files
- Copy: `crates/toolkit/tests/egress.rs`
- Create: `crates/toolkit/src/egress.rs`
- Modify: `crates/toolkit/src/lib.rs` (`pub mod egress;`), `crates/toolkit/src/main.rs`,
`docs/implementer-log.md`
## Interfaces
```rust
pub const HANDSHAKE_TIMEOUT: Duration = Duration::from_secs(10);
pub const CONNECT_TIMEOUT: Duration = Duration::from_secs(10);
pub const MAX_CONNECTIONS: usize = 8;
// SOCKS5 reply codes the proxy sends:
pub const NOT_ALLOWED: u8 = 2;
pub const HOST_UNREACHABLE: u8 = 4;
pub const CONNECTION_REFUSED: u8 = 5;
pub const COMMAND_NOT_SUPPORTED: u8 = 7;
pub const ADDRESS_TYPE_NOT_SUPPORTED: u8 = 8;
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Allow { /* patterns: Vec<String>, private */ }
impl Allow {
/// One or more host patterns separated by ','. Every piece must pass
/// `proto::hosts::valid_host_pattern`; an empty piece (",", "a.com,", "") is an error.
pub fn parse(list: &str) -> Result<Allow, String>;
/// `valid_host(host)` and some pattern `host_matches` it.
pub fn permits(&self, host: &str) -> bool;
}
/// Name resolution and connecting, so tests need no network.
pub trait Dial: Send + Sync {
fn resolve(&self, host: &str, port: u16) -> std::io::Result<Vec<SocketAddr>>;
fn connect(&self, addr: SocketAddr, timeout: Duration) -> std::io::Result<TcpStream>;
}
/// The system's: `(host, port).to_socket_addrs()` and `TcpStream::connect_timeout`.
pub struct SystemDial;
impl Dial for SystemDial { }
pub struct Proxy { /* allow: Allow, dial: Arc<dyn Dial>, handshake: Duration — private */ }
impl Proxy {
pub fn new(allow: Allow, dial: Arc<dyn Dial>) -> Proxy; // handshake = HANDSHAKE_TIMEOUT
pub fn with_handshake_timeout(self, handshake: Duration) -> Proxy; // for tests
/// Accept for ever; at most MAX_CONNECTIONS handled at once, more are closed at once.
pub fn serve(self: Arc<Self>, listener: UnixListener) -> std::io::Result<()>;
/// One connection, from greeting to the end of the copy.
pub fn handle(&self, client: UnixStream);
}
```
## `handle`: the handshake, every exit
The whole handshake shares **one deadline**: `now + handshake`. Before every read, set the read
timeout to the time left (`deadline.checked_duration_since(Instant::now())`); if none is left, or a
read times out, fails, or reads 0 bytes (the peer closed), stop: return without a reply. A client
that sends one byte every 100 ms must still be cut off at the deadline (a test does exactly that).
Read with a loop that fills a buffer of the exact length the protocol gives, never more.
"Refuse with `code`" means: write `[5, code, 0, 1, 0, 0, 0, 0, 0, 0]` (ignore a write error) and
return, which closes the connection.
1. Read 2 bytes: `version`, `count`. `version != 5` → return, no reply.
2. Read `count` bytes of methods. If none of them is 0: write `[5, 0xff]`, return. Else write
`[5, 0]`.
3. Read 4 bytes: `version`, `command`, `reserved`, `kind`. `version != 5` or `reserved != 0`
return, no reply.
4. `command != 1` → refuse with `COMMAND_NOT_SUPPORTED`.
5. `kind != 3` (not a domain name; 1 is IPv4, 4 is IPv6) → refuse with
`ADDRESS_TYPE_NOT_SUPPORTED`. Do not read the address.
6. Read 1 byte: `length`. `length == 0` → refuse with `NOT_ALLOWED`.
7. Read `length` bytes of name, then 2 bytes of port (`u16::from_be_bytes`).
8. The name is not UTF-8, or `port != 443`, or `!allow.permits(&name)` → refuse with
`NOT_ALLOWED`.
9. `dial.resolve(&name, port)`. An error → refuse with `HOST_UNREACHABLE`. Take the **first**
address for which `crate::addr::is_public(addr.ip())` is true; none → refuse with
`HOST_UNREACHABLE`. Non-public addresses are skipped, **never tried**.
10. `dial.connect(addr, CONNECT_TIMEOUT)`. An error → refuse with `CONNECTION_REFUSED`.
11. Write the success reply `[5, 0, 0, 1, 0, 0, 0, 0, 0, 0]`; if that write fails, return.
## After the handshake: the copy
Clear the client's read timeout (`set_read_timeout(None)`). Copy both ways until both directions
are done, passing a half-close on:
- A second thread copies client → server (`std::io::copy`), then `shutdown(Shutdown::Write)` on
the server side.
- The handler's own thread copies server → client, then `shutdown(Shutdown::Write)` on the
client side, then joins the second thread.
Use `try_clone` for the second handle of each stream. If a `try_clone` fails, return.
## `serve`
For each accepted stream: if the number of connections being handled is already
`MAX_CONNECTIONS`, drop the stream at once (the client reads the end). Otherwise count it, handle
it on its own thread (`std::thread::Builder`, not `spawn`, which panics; if the thread cannot be
started, uncount it), and uncount it when `handle` returns. An `accept` error ends `serve` with
that error. Use an `AtomicUsize` for the count.
## `main.rs`
Add one form before the tool form: the arguments (after the program name) are exactly
`egress-proxy --socket <path> --allow <list>`, in that order, all UTF-8. Then:
1. `Allow::parse(list)`; an error `e` → print `toolkit: egress-proxy: --allow: {e}` to standard
error, exit 2.
2. `UnixListener::bind(path)`. **Do not remove anything first**: the directory is fresh for each
call, so a file already there is a mistake. An error → print
`toolkit: egress-proxy: cannot listen on {path}: {e}`, exit 2.
3. `Arc::new(Proxy::new(allow, Arc::new(SystemDial))).serve(listener)`. If it returns an error →
print `toolkit: egress-proxy: {e}`, exit 2.
Any other argument list that starts with `egress-proxy` is not this form, and goes to the tool form
as before, which answers it as an unknown tool (exit 2).
## About the given tests
`egress.rs` handles one connection per test over `UnixStream::pair()`, with a fake `Dial` whose
names resolve from a table and whose connections all go to a local echo server. A proxy that closes
with some of the client's bytes unread makes the client's next read fail with "connection reset"
rather than read the end; the tests treat both as closed. That is normal and needs nothing from
you.
## Steps
- [ ] **1. Copy.** `git switch m3b`, then
`cp docs/plans/M3b/files/crates/toolkit/tests/egress.rs crates/toolkit/tests/`
- [ ] **2. See it fail.** `cargo test -p toolkit --test egress`. Expected: it does not compile.
- [ ] **3. Write `egress.rs`** and the `lib.rs` and `main.rs` changes. Run `cargo fmt --all`.
- [ ] **4. See it pass.** `cargo test -p toolkit --test egress`. Expected: 15 passed. Run it ten
times; it must pass every time.
- [ ] **5. Walk the exits.** Point at the line of your code for each of the 11 handshake steps.
- [ ] **6. Run the gate.** `make gate`. Expected last line: `gate: ok`.
- [ ] **7. Log and commit.** `git add crates/toolkit docs/implementer-log.md && git commit`
## Done when
- `cargo test -p toolkit --test egress` reports 15 passed ten times running; `make gate` prints
`gate: ok`.
## Stop and report if
- A test wants the proxy to try a non-public address, or to answer an IP-literal request with
anything but `ADDRESS_TYPE_NOT_SUPPORTED`.
- A test hangs.
+100
View File
@@ -0,0 +1,100 @@
# M3b task 09: `[runner]` in `brokerd.toml`
**Branch:** `m3b` (run `git switch m3b`; `git status --short` must be empty, otherwise stop)
**Commit subject:** `brokerd: the [runner] section`
## Goal
`[runner]` says how tools run: which `podman`, which image, and the limits. Without the section,
`brokerd` keeps M3a's runtime, which refuses every call (task 13 does that choice). The image must
be named **by digest**, so what runs is exactly what was built. Spec section 6, "Configuration".
## Files
- Copy: `crates/brokerd/tests/config_runner.rs`, the five files
`crates/brokerd/tests/fixtures/config/runner_*.toml`, and `crates/brokerd/tests/support/rig.rs`
(replaces the old one: it builds a `Config` with a struct literal, which now needs
`runner: None`)
- Modify: `crates/brokerd/src/config.rs`, `docs/implementer-log.md`
## Interfaces
In `config.rs`:
```rust
#[derive(Debug, Clone, PartialEq, Eq, Deserialize)]
#[serde(deny_unknown_fields)]
pub struct Runner {
#[serde(default = "default_podman")] pub podman: PathBuf, // "podman"
pub image: String, // required, no default
#[serde(default = "default_egress_network")] pub egress_network: String, // "pasta"
#[serde(default = "default_output_cap")] pub output_cap: u64, // 262_144
#[serde(default = "default_memory")] pub memory: String, // "512m"
#[serde(default = "default_pids")] pub pids: u32, // 128
#[serde(default = "default_read_file_ms")] pub read_file_ms: u64, // 30_000
#[serde(default = "default_write_file_ms")] pub write_file_ms: u64, // 30_000
#[serde(default = "default_shell_ms")] pub shell_ms: u64, // 100_000
#[serde(default = "default_http_fetch_ms")] pub http_fetch_ms: u64, // 60_000
}
impl Runner {
/// The time limit for one call of `tool`: the matching `_ms` field.
pub fn time_limit(&self, tool: crate::args::ToolName) -> std::time::Duration;
}
pub struct Config {
// paths, sockets, approvals as now, then:
#[serde(default)]
pub runner: Option<Runner>,
}
impl Config {
// … as now, plus:
/// `<home>/run/egress`: where each `http_fetch` call gets a directory for the proxy's socket.
pub fn egress_dir(&self) -> PathBuf;
}
```
Write one small private `fn default_…() -> …` per defaulted field (serde needs a function path).
Put the struct attributes on separate lines as usual; the table above only saves space.
## Checks in `Config::load`
After the `ttl_ms` check, if `runner` is `Some`, check it and return the first problem as
`ConfigError::Invalid(path, text)`. In this order:
1. `image` is not `<name>@sha256:<hex>` where `<name>` is not empty and `<hex>` is exactly 64 of
`0-9a-f` (lowercase). Split with `rsplit_once("@sha256:")`. Text:
`[runner] image is {image:?}; it must be named by digest: <name>@sha256:<64 hex digits>`.
2. `memory` is not digits followed by one of `b`, `k`, `m`, `g` (at least one digit). Text:
`[runner] memory is {memory:?}; it must be a number and one of b, k, m, g`.
3. `egress_network` or `podman` is empty. Text:
`[runner] podman and egress_network must not be empty`.
4. Any of `output_cap`, `pids`, `read_file_ms`, `write_file_ms`, `shell_ms`, `http_fetch_ms` is 0,
checked in that order. Text: `[runner] <name> must be at least 1`, with the field's name.
A missing `image` or an unknown key is already a `ConfigError::Parse` from serde; do not add
checks for those. `Config::parse` (the string version the tests use for plain values) does not run
these checks; only `load` does, as for `ttl_ms` today.
## Steps
- [ ] **1. Copy.** `git switch m3b`, then
`cp docs/plans/M3b/files/crates/brokerd/tests/config_runner.rs crates/brokerd/tests/`,
`cp docs/plans/M3b/files/crates/brokerd/tests/fixtures/config/runner_*.toml crates/brokerd/tests/fixtures/config/`,
`cp docs/plans/M3b/files/crates/brokerd/tests/support/rig.rs crates/brokerd/tests/support/`
- [ ] **2. See it fail.** `cargo test -p brokerd --test config_runner`. Expected: it does not
compile.
- [ ] **3. Write the code.** Run `cargo fmt --all`.
- [ ] **4. See it pass.** `cargo test -p brokerd --test config_runner --test config`. Expected: 7
and 7 passed. Then `cargo test -p brokerd`: everything passes.
- [ ] **5. Run the gate.** `make gate`. Expected last line: `gate: ok`.
- [ ] **6. Log and commit.** `git add crates/brokerd docs/implementer-log.md && git commit`
## Done when
- `cargo test -p brokerd` passes; `make gate` prints `gate: ok`.
## Stop and report if
- A test wants an image named by tag (`:latest`) to be accepted.
+107
View File
@@ -0,0 +1,107 @@
# M3b task 10: the `podman` argument lists
**Branch:** `m3b` (run `git switch m3b`; `git status --short` must be empty, otherwise stop)
**Commit subject:** `brokerd: the podman argument lists`
## Goal
Build, without running anything, the exact `podman` arguments for one call's container, and for
`http_fetch`'s egress proxy. Every argument is its own `OsString`: nothing is ever passed through a
shell, and the tool's arguments go on standard input, never on the command line. The lists are
tested as golden files, one argument per line. Spec section 6.
The container's name says whose call it is, so `RunSpec` gains the session and the call.
## Files
- Copy: `crates/brokerd/tests/podman_args.rs` and the six files
`crates/brokerd/tests/fixtures/podman/*.args`
- Create: `crates/brokerd/src/podman.rs`
- Modify: `crates/brokerd/src/runner.rs`, `crates/brokerd/src/lib.rs` (`pub mod podman;`),
`docs/implementer-log.md`
## `RunSpec` gains two fields
In `runner.rs`, `RunSpec` gets `session: SessionId` and `call: CallId` as its **first** two
fields (private, like the others), with getters:
```rust
pub fn session(&self) -> &SessionId;
pub fn call(&self) -> CallId;
```
`run` fills them from `decision.request().session.clone()` and `decision.request().call`. The
`compile_fail` doctest at the top of `runner.rs` builds a `RunSpec` with a struct literal: add the
two fields to it, first, so it still fails **only** because the fields are private:
```rust
//! session: proto::SessionId::new("s1").unwrap(),
//! call: proto::CallId(1),
```
## `podman.rs`
```rust
pub const EGRESS_MOUNT: &str = "/run/egress";
pub const EGRESS_SOCKET: &str = "/run/egress/egress.sock";
pub const TOOLKIT: &str = "/bin/toolkit";
/// "boxmaker-<session>-<call>-<n>"
pub fn container_name(session: &SessionId, call: CallId, n: u64) -> String;
/// The tool's container. `egress` is the call's egress directory, for `http_fetch` only.
pub fn tool_args(spec: &RunSpec, runner: &Runner, name: &str, egress: Option<&Path>) -> Vec<OsString>;
/// The egress proxy's container.
pub fn egress_args(runner: &Runner, name: &str, dir: &Path, hosts: &[String]) -> Vec<OsString>;
```
`tool_args`, in this order:
```
run --rm -i --name=<name> --label=boxmaker=tool --network=none
--read-only --cap-drop=all --security-opt=no-new-privileges --userns=keep-id
--pids-limit=<runner.pids> --memory=<runner.memory>
--tmpfs=/tmp:rw,size=64m,mode=1777
--volume=<path>:<path>:ro (or :rw when writable) one per spec.mounts(), in order
--volume=<egress dir>:/run/egress:rw only when `egress` is Some
<runner.image> /bin/toolkit <spec.tool().as_str()>
```
`egress_args`, in this order:
```
run -d --rm --name=<name>-egress --label=boxmaker=egress --network=<runner.egress_network>
--read-only --cap-drop=all --security-opt=no-new-privileges --userns=keep-id
--pids-limit=64 --memory=128m
--volume=<dir>:/run/egress:rw
<runner.image> /bin/toolkit egress-proxy --socket /run/egress/egress.sock --allow <hosts joined with ",">
```
The six hardening flags (`--read-only` to `--memory=…`) are the same in both, so write them once in
a private function. Build the `--volume=<dir>:…` argument as an `OsString` with `push`, so a
directory need not be UTF-8 (`dir.as_os_str()`), not with `format!` on `display()`.
## Steps
- [ ] **1. Copy.** `git switch m3b`, then
`cp docs/plans/M3b/files/crates/brokerd/tests/podman_args.rs crates/brokerd/tests/`,
`mkdir -p crates/brokerd/tests/fixtures/podman`,
`cp docs/plans/M3b/files/crates/brokerd/tests/fixtures/podman/*.args crates/brokerd/tests/fixtures/podman/`
- [ ] **2. See it fail.** `cargo test -p brokerd --test podman_args`. Expected: it does not
compile.
- [ ] **3. Write the code.** Run `cargo fmt --all`.
- [ ] **4. See it pass.** `cargo test -p brokerd --test podman_args --test runner`. Expected: 7
and 8 passed. `cargo test -p brokerd --doc`: every doctest passes.
- [ ] **5. Run the gate.** `make gate`. Expected last line: `gate: ok`.
- [ ] **6. Log and commit.** `git add crates/brokerd docs/implementer-log.md && git commit`
## Done when
- `cargo test -p brokerd` and `cargo test -p brokerd --doc` pass; `make gate` prints `gate: ok`.
## Stop and report if
- A golden file disagrees with the lists above: report both, do not edit the golden file.
- Anything would put the tool's arguments (the command, the path, the URL) on `podman`'s command
line.
+110
View File
@@ -0,0 +1,110 @@
# M3b task 11: the Podman runtime
**Branch:** `m3b` (run `git switch m3b`; `git status --short` must be empty, otherwise stop)
**Commit subject:** `brokerd: the Podman runtime`
## Goal
`Podman` implements `Runtime`: each call runs in a fresh container, its arguments go on standard
input, its output is kept up to a cap, and it is stopped at its time limit. Whatever the tool
prints is the result's content, labelled by the grant; every failure is a **fixed sentence**, and
what Podman itself said goes only to `brokerd`'s log. Spec section 6, "One call". `http_fetch`'s
proxy is task 12; in this task every call runs its container with no egress directory.
## Files
- Copy: `crates/brokerd/tests/support/fake_podman.rs`, `crates/brokerd/tests/container.rs`
- Create: `crates/brokerd/src/container.rs`
- Modify: `crates/brokerd/src/lib.rs` (`pub mod container;`), `docs/implementer-log.md`
## Interfaces
```rust
pub const RUNBOOK: &str = "see docs/runbook.md#runner-unavailable";
pub const COULD_NOT_RUN: &str = "the tool could not run";
pub const CANNOT_START: &str = "the tool runner could not start the container";
pub const KILLED: &str = "the tool was stopped: it ran out of memory or was killed";
pub const TIMED_OUT: &str = "the tool ran past its time limit";
pub const UNEXPECTED: &str = "the tool failed with an unexpected status";
pub const POLL: Duration = Duration::from_millis(50);
pub const STDERR_KEPT: usize = 4096;
pub type Log = Arc<dyn Fn(&str) + Send + Sync>;
pub struct Podman { /* runner: Runner, egress_dir: PathBuf, log: Log, next: AtomicU64 — private */ }
impl Podman {
/// `egress_dir` is `Config::egress_dir()`; task 12 uses it.
pub fn new(runner: Runner, egress_dir: PathBuf, log: Log) -> Podman;
}
impl Runtime for Podman {
fn run(&self, spec: &RunSpec) -> Result<RunOutput, RunError>;
}
```
## `run`: every step and exit
1. `n = next.fetch_add(1, SeqCst)`; `name = podman::container_name(spec.session(), spec.call(), n)`.
The first call of a `Podman` is number 0.
2. `args = podman::tool_args(spec, &runner, &name, None)`;
`input = spec.arguments().canonical_json()`; `limit = runner.time_limit(spec.tool())`.
3. Spawn `Command::new(&runner.podman).args(args)` with standard input, output and error all
piped. A failure → log `brokerd: cannot start {podman path}: {e}` + `"\n"` + `RUNBOOK`, and
return `Err(RunError::Unavailable(CANNOT_START))`.
4. Three threads, so no pipe can block another: one **writes** `input` to standard input and then
drops it (closing it); one **reads** standard output, keeping the first `output_cap` bytes and
**reading on past the cap**, throwing the rest away, and remembering that there was more; one
reads standard error the same way, keeping `STDERR_KEPT` bytes. (Stopping to read would leave
the tool blocked on a full pipe; closing the pipe would make Podman fail. Both give the wrong
answer.) Write the reading loop once, as a private function returning `(Vec<u8>, bool)`.
5. Wait with a loop: `child.try_wait()`; if it has ended, go to 6. If `limit` has passed since the
spawn, **stop the container**: run `podman kill <name>`, then `podman rm -f <name>` (each with
`.status()`, standard streams null; log a line if one does not succeed, and go on), then
`child.kill()` and `child.wait()`. Otherwise sleep `POLL` and try again.
6. Join the three threads (a thread that panicked counts as empty output).
7. The answer:
| How it ended | Answer | Log |
|---|---|---|
| past the time limit (step 5) | `Err(RunError::Failed(TIMED_OUT))` | a line naming the container |
| exit 0 or 1 | `Ok(RunOutput { content, truncated })`, content = the kept output decoded with `from_utf8_lossy` | — |
| exit 2 | `Err(RunError::Failed(COULD_NOT_RUN))` | the kept standard error |
| exit 125, 126 or 127 | `Err(RunError::Unavailable(CANNOT_START))` | the kept standard error, then `"\n"` + `RUNBOOK` |
| exit 137 | `Err(RunError::Failed(KILLED))` | — |
| anything else, including killed by a signal | `Err(RunError::Failed(UNEXPECTED))` | the status and the kept standard error |
**The tool's output never goes into a `RunError`**: only the six constants do.
Also add `pub(crate) fn podman(&self, args: &[&str])`, the helper step 5 uses to run a short
`podman` command and log if it fails; task 12 uses it too.
## About the given tests
The tests run `brokerd`'s real `runner::run` against a fake `podman`: a shell script that records
every call's arguments in a file and then does what the test says. As in task 06, every test takes
the `serial()` lock first, because writing a script and running it at once races with other tests'
forks ("Text file busy"). The time-limit test's fake ends with `exec sleep 30`, so killing the
process kills the sleep and nothing keeps the pipes open.
## Steps
- [ ] **1. Copy.** `git switch m3b`, then
`cp docs/plans/M3b/files/crates/brokerd/tests/support/fake_podman.rs crates/brokerd/tests/support/`
and `cp docs/plans/M3b/files/crates/brokerd/tests/container.rs crates/brokerd/tests/`
- [ ] **2. See it fail.** `cargo test -p brokerd --test container`. Expected: it does not compile.
- [ ] **3. Write `container.rs`.** Run `cargo fmt --all`.
- [ ] **4. See it pass.** `cargo test -p brokerd --test container`. Expected: 11 passed. Run it ten
times; it must pass every time.
- [ ] **5. Walk the table.** Point at the line of your code for each row, and check that no row
puts output into a `RunError`.
- [ ] **6. Run the gate.** `make gate`. Expected last line: `gate: ok`.
- [ ] **7. Log and commit.** `git add crates/brokerd docs/implementer-log.md && git commit`
## Done when
- `cargo test -p brokerd --test container` reports 11 passed ten times running; `make gate` prints
`gate: ok`.
## Stop and report if
- A test hangs, or fails only sometimes: report which, and how often, rather than add sleeps.
- A test wants tool output inside a failure message.
+89
View File
@@ -0,0 +1,89 @@
# M3b task 12: the egress proxy's lifecycle
**Branch:** `m3b` (run `git switch m3b`; `git status --short` must be empty, otherwise stop)
**Commit subject:** `brokerd: start and remove the egress proxy for http_fetch`
## Goal
For an `http_fetch` call, the tool's container still has no network. Before it runs, `brokerd`
makes a directory for this call, starts the egress proxy (task 08) in its own container with a
network and that directory mounted, and waits for the proxy's socket. The tool's container gets the
same directory. Afterwards, **on every path**, the proxy's container and the directory are removed.
Spec section 6, "`http_fetch`".
## Files
- Copy: `crates/brokerd/tests/container_egress.rs`
- Modify: `crates/brokerd/src/container.rs`, `docs/implementer-log.md`
## Interfaces (added to `container.rs`)
```rust
pub const EGRESS_WAIT: Duration = Duration::from_secs(5);
impl Podman {
/// The same runtime with another wait for the proxy's socket, for tests.
pub fn with_egress_wait(self, egress_wait: Duration) -> Podman;
}
```
`Podman` gains a private field `egress_wait`, set to `EGRESS_WAIT` by `new`.
## `run`, now
After step 2 of task 11 (name, input, limit):
- `spec.egress()` is `None` → exactly as before: `tool_args(…, None)` and run the container.
- `spec.egress()` is `Some(hosts)` → start the proxy (below), and on success run the container
with `tool_args(spec, &runner, &name, Some(&dir))`.
### Starting the proxy: every step and exit
Use a **guard**: a private struct holding the proxy's container name (`<name>-egress`) and the
directory, whose `Drop` runs `podman rm -f <name>-egress` (the helper from task 11) and then
`remove_dir_all(dir)` (log an error other than `NotFound`). Create the guard **first**, before
anything can fail, so that every return below cleans up by dropping it, and keep it alive until
the tool's container has finished.
1. `dir = egress_dir.join(&name)`. Make `egress_dir` (and its parents) with mode 0700
(`DirBuilder::new().recursive(true).mode(0o700)`), then set its mode to 0700 anyway (it may
have existed). If `dir` already exists, it was left by a crash: `remove_dir_all` it (a
`NotFound` is fine). Then create `dir` with mode 0700, not recursively. Any failure → log
`brokerd: cannot make {dir}: {e}` + `"\n"` + `RUNBOOK`, return
`Err(RunError::Unavailable(CANNOT_START))`.
2. Run `podman` with `podman::egress_args(&runner, &name, &dir, hosts)`, standard input and output
null, standard error piped, with `.output()`. It does not succeed → log
`brokerd: podman could not start {name}-egress: {stderr}` + `"\n"` + `RUNBOOK`, return
`Unavailable(CANNOT_START)`. It cannot be started at all → the same log as task 11 step 3, and
the same answer.
3. Wait until `dir.join("egress.sock")` exists, checking every 20 ms, for at most `egress_wait`.
It does not appear → log `brokerd: {name}-egress did not make its socket within {ms} ms` +
`"\n"` + `RUNBOOK`, return `Unavailable(CANNOT_START)`.
4. Return the guard. Run the tool's container as in task 11. Whatever it answers, the guard is
dropped after it, removing the proxy and the directory.
So for a successful call the fake sees exactly three `podman` calls: `run -d …` (the proxy),
`run --rm -i …` (the tool), `rm -f <name>-egress`. For a tool past its time limit it sees the
proxy's run, the tool's run, `kill <name>`, `rm -f <name>`, then `rm -f <name>-egress`.
## Steps
- [ ] **1. Copy.** `git switch m3b`, then
`cp docs/plans/M3b/files/crates/brokerd/tests/container_egress.rs crates/brokerd/tests/`
- [ ] **2. See it fail.** `cargo test -p brokerd --test container_egress`. Expected: it does not
compile (`with_egress_wait` does not exist).
- [ ] **3. Write the code.** Run `cargo fmt --all`. `container.rs` must stay under 500 lines.
- [ ] **4. See it pass.** `cargo test -p brokerd --test container_egress --test container`.
Expected: 6 and 11 passed. Run them ten times; they must pass every time.
- [ ] **5. Walk the exits.** For each of the four steps, say how the guard cleans up after it.
- [ ] **6. Run the gate.** `make gate`. Expected last line: `gate: ok`.
- [ ] **7. Log and commit.** `git add crates/brokerd docs/implementer-log.md && git commit`
## Done when
- Both suites pass ten times running; `make gate` prints `gate: ok`.
## Stop and report if
- A path leaves the proxy's container or the directory behind, and a guard does not fix it.
- The tool's container would get a network.
+56
View File
@@ -0,0 +1,56 @@
# M3b task 13: `brokerd serve` uses the runtime
**Branch:** `m3b` (run `git switch m3b`; `git status --short` must be empty, otherwise stop)
**Commit subject:** `brokerd serve: run tools in containers when [runner] is set`
## Goal
With a `[runner]` section, `brokerd serve` runs allowed calls through `Podman`; without it, it keeps
M3a's `Refusing` runtime. It prints one line saying which, so the owner can see it at a glance.
## Files
- Copy: `crates/brokerd/tests/serve_runner.rs`
- Modify: `crates/brokerd/src/main.rs`, `docs/implementer-log.md`
## The change
In `main.rs`, after the config is loaded and before `serve::start`:
```rust
let log: Arc<dyn Fn(&str) + Send + Sync> = Arc::new(|line: &str| eprintln!("{line}"));
```
and choose the runtime and its notice:
- `cfg.runner` is `Some(runner)``Box::new(Podman::new(runner, cfg.egress_dir(), Arc::clone(&log)))`,
notice `brokerd: tools run in containers from {runner.image}` (take the image before `runner`
moves).
- `None``Box::new(Refusing)`, notice `brokerd: no [runner] section: every tool call is refused`.
Pass the runtime and `log` to `serve::start` (instead of `Box::new(Refusing)` and the closure made
there today). Print the notice with `eprintln!` right after the existing
`brokerd: serving tools on … and approvals on …` line. Nothing else changes.
## Steps
- [ ] **1. Copy.** `git switch m3b`, then
`cp docs/plans/M3b/files/crates/brokerd/tests/serve_runner.rs crates/brokerd/tests/`
- [ ] **2. See it fail.** `cargo test -p brokerd --test serve_runner`. Expected: 2 fail (the
notices are missing, and the call is refused).
- [ ] **3. Make the change.** Run `cargo fmt --all`.
- [ ] **4. See it pass.** `cargo test -p brokerd --test serve_runner --test serve`. Expected: 2
and 9 passed.
- [ ] **5. Run the gate.** `make gate`. Expected last line: `gate: ok`, with about 638 tests in
all.
- [ ] **6. Log and commit.** `git add crates/brokerd docs/implementer-log.md && git commit`
This is the last task of M3b. Stop after the commit; the review comes next.
## Done when
- `cargo test -p brokerd` passes; `make gate` prints `gate: ok`.
## Stop and report if
- `serve::start`'s signature would have to change.
+63
View File
@@ -0,0 +1,63 @@
# M3b implementation plan: the runner and the tools
> **For the implementing model:** do not work from this file. The owner gives you one task file at
> a time (`01-…` to `13-…`). This file is the index for the owner and the reviewer.
**Goal:** an allowed tool call runs in a fresh rootless Podman container, with no network unless
it is `http_fetch`, which reaches the grant's hosts only through an egress proxy of our own. The
four tools and the proxy are programs in `toolkit`; `brokerd` builds the `podman` argument lists
and runs them.
**Architecture:** shared definitions move to `proto` first (01), the fetch target is sealed (02),
and grant paths that cannot be mounted are refused (03). `toolkit` gets its four tools (04 to 06),
the public-address check (07) and the proxy (08). `brokerd` gets `[runner]` (09), the argument
lists (10), the runtime (11), the proxy's lifecycle (12) and the wiring in `serve` (13).
**Spec:** `docs/specs/2026-09-22-m3b-runner.md`. Brief: `docs/design.md`. Every fail-closed
message ends with a pointer into `docs/runbook.md`; the `runner-unavailable` entry is already
there.
**No task needs Podman, Nix or a network.** Everything runs against fakes: fake `curl` and
`podman` scripts, a fake resolver, `UnixStream::pair()`. The image and the checks on straylight
are done by the design model afterwards.
## Global constraints
- Everything in `AGENTS.md`, including "Lessons from earlier reviews".
- No new dependency. `toolkit` gains `serde` and `serde_json`, both already vetted (task 04).
- Branch `m3b`. One task, one fresh OpenCode session, one commit. Run `cargo fmt --all` before the
gate. Review happens once, after task 13.
- Tests that write a script and run it take a lock (`serial()`): otherwise another test's fork
can make running it fail with "text file busy". Keep that pattern in anything you add.
## Tasks
The last column is how the given tests were checked before hand-over (decision of 2026-09-18, tip
T17). For M3b every task had a reference implementation: the tests of each task were run against
it at that task's end state, then the reference was deleted so it cannot be read (tip T18).
| # | File | Delivers | Tests | Check |
|---|---|---|---|---|
| 01 | `01-proto-tools-hosts.md` | `proto::tools`, `proto::hosts`; `brokerd::args` uses them | `proto/tests/tools.rs`, `hosts.rs` | reference; the given test pinned an accepted serde weakness, removed |
| 02 | `02-brokerd-fetch-url.md` | `ToolArgs::HttpFetch(FetchUrl)`, sealed (M3a finding 14) | `brokerd/tests/args.rs` (changed), a `compile_fail` doctest | reference |
| 03 | `03-brokerd-grant-mount-rule.md` | grant paths with `:` or `,` are invalid | `brokerd/tests/grants_mount.rs` | reference |
| 04 | `04-toolkit-files.md` | the `toolkit` program, `read_file`, `write_file` | `toolkit/tests/files.rs`, `support/mod.rs` | reference |
| 05 | `05-toolkit-shell.md` | `shell` | `toolkit/tests/shell.rs` | reference |
| 06 | `06-toolkit-fetch.md` | `http_fetch` through `curl` | `toolkit/tests/fetch.rs` | reference; found the ETXTBSY race (1 in 7 runs), fixed with `serial()` |
| 07 | `07-toolkit-addr.md` | `is_public` | `toolkit/tests/addr.rs` | reference; found `::/96` missing from the spec, added |
| 08 | `08-toolkit-egress-proxy.md` | the SOCKS5 egress proxy | `toolkit/tests/egress.rs` | reference; 20 runs clean |
| 09 | `09-brokerd-runner-config.md` | `[runner]` | `config_runner.rs`, 5 fixtures, `support/rig.rs` (changed) | reference |
| 10 | `10-brokerd-podman-args.md` | the `podman` argument lists; `RunSpec` gains session and call | `podman_args.rs`, 6 golden files | reference |
| 11 | `11-brokerd-container.md` | the Podman runtime | `container.rs`, `support/fake_podman.rs` | reference; 20 runs clean |
| 12 | `12-brokerd-egress.md` | the proxy's lifecycle for `http_fetch` | `container_egress.rs` | reference; 20 runs clean |
| 13 | `13-brokerd-serve-runner.md` | `brokerd serve` uses the runtime | `serve_runner.rs` | reference |
At the end: `make gate` prints `gate: ok` with about 638 tests.
## Running it
```sh
BOXMAKER_MODEL=straylight/ornith-1.5-35b-a3b tools/run-plan.sh docs/plans/M3b
```
Keep the OpenCode TUI closed while it runs.
@@ -0,0 +1,428 @@
//! Tests for tool arguments: their shape, and the form of paths, hosts and URLs. Do not edit.
//!
//! Every row of the "Paths" and "Hosts" tables in the M3a spec is a case here.
use brokerd::args::{
ArgsError, MAX_PATH, MAX_URL, ToolArgs, ToolName, host_matches, inside, parse, url_host,
valid_host, valid_host_pattern, valid_path,
};
#[test]
fn the_four_tool_names() {
let names = ["read_file", "write_file", "shell", "http_fetch"];
for (tool, name) in ToolName::ALL.into_iter().zip(names) {
assert_eq!(tool.as_str(), name);
assert_eq!(ToolName::parse(name), Some(tool));
}
for other in [
"",
"echo",
"clock",
"call_tool",
"Read_File",
"read_file ",
"readfile",
] {
assert_eq!(ToolName::parse(other), None, "{other:?}");
}
}
#[test]
fn valid_paths() {
let longest = format!("/{}", "a".repeat(MAX_PATH - 1));
assert_eq!(longest.len(), MAX_PATH);
for path in [
"/",
"/etc",
"/home/kyle/notes/a.md",
"/home/kyle/notes",
"/with space/and\ttab",
"/dots.in.names/..hidden/...",
"/unicode/\u{e9}t\u{e9}",
longest.as_str(),
] {
assert!(valid_path(path), "{path:?} should be valid");
}
}
#[test]
fn invalid_paths() {
let too_long = format!("/{}", "a".repeat(MAX_PATH));
assert_eq!(too_long.len(), MAX_PATH + 1);
for path in [
"",
"notes/a.md",
"./notes",
"~/notes",
"/home/kyle/notes/../.ssh/id",
"/home/kyle//notes/./a.md",
"/home//kyle",
"/home/./kyle",
"/home/kyle/",
"/home/kyle/..",
"/..",
"/.",
"//",
"/nul\0byte",
too_long.as_str(),
] {
assert!(!valid_path(path), "{path:?} should be invalid");
}
}
/// The table in the spec, row by row, for the rows about form and containment.
#[test]
fn inside_is_by_whole_components() {
let grant = "/home/kyle/notes";
assert!(inside(grant, "/home/kyle/notes/a.md"));
assert!(inside(grant, "/home/kyle/notes"));
assert!(inside(grant, "/home/kyle/notes/deep/er/b.md"));
assert!(!inside(grant, "/home/kyle/notes2/a.md"));
assert!(!inside(grant, "/home/kyle/note"));
assert!(!inside(grant, "/home/kyle"));
assert!(!inside(grant, "/"));
assert!(!inside(grant, "/other/home/kyle/notes/a.md"));
// A grant of the root is refused when grants are loaded, but the function is still right.
assert!(inside("/", "/etc/passwd"));
assert!(inside("/", "/"));
}
#[test]
fn valid_hosts_and_patterns() {
let label63 = "a".repeat(63);
let long = format!("{label63}.{label63}.{label63}.{}.com", "b".repeat(57));
assert_eq!(long.len(), 253);
for host in [
"example.com",
"www.example.com",
"a.b.example.com",
"xn--bcher-kva.example",
"1password.com",
"3.example.org",
"a-b.c-d.io",
long.as_str(),
] {
assert!(valid_host(host), "{host:?} should be a valid host");
assert!(
valid_host_pattern(host),
"{host:?} should be a valid pattern"
);
let wild = format!("*.{host}");
assert!(!valid_host(&wild), "{wild:?} is a pattern, not a host");
}
assert!(valid_host_pattern("*.example.com"));
assert!(valid_host_pattern("*.a.b.example.com"));
}
#[test]
fn invalid_hosts_and_patterns() {
let label64 = format!("{}.com", "a".repeat(64));
let too_long = format!("{}.com", ["a".repeat(63).as_str(); 4].join("."));
assert!(too_long.len() > 253);
for host in [
"",
"localhost",
"com",
"Example.com",
"example.COM",
"example.com.",
".example.com",
"example..com",
"-example.com",
"example-.com",
"exa_mple.com",
"example.com:443",
"example.com/path",
"user@example.com",
"exa mple.com",
"[::1]",
"::1",
// Every spelling of an IPv4 address: the last label does not start with a letter.
"127.0.0.1",
"127.1",
"10.0.0.0x1",
"1.2.3.4",
"example.123",
"b\u{fc}cher.example",
label64.as_str(),
too_long.as_str(),
] {
assert!(!valid_host(host), "{host:?} should not be a valid host");
assert!(
!valid_host_pattern(host),
"{host:?} should not be a valid pattern"
);
}
for pattern in [
"*",
"*.",
"*.com",
"*example.com",
"www.*.com",
"*.*.example.com",
"**.example.com",
"*.Example.com",
"*.127.0.0.1",
] {
assert!(!valid_host_pattern(pattern), "{pattern:?}");
}
}
/// The host table in the spec, row by row.
#[test]
fn host_matching() {
assert!(host_matches("example.com", "example.com"));
assert!(!host_matches("example.com", "www.example.com"));
assert!(host_matches("*.example.com", "www.example.com"));
assert!(host_matches("*.example.com", "a.b.example.com"));
assert!(!host_matches("*.example.com", "example.com"));
// A suffix is not enough: the match is by whole labels.
assert!(!host_matches("*.example.com", "badexample.com"));
assert!(!host_matches("*.example.com", "www.example.com.evil.org"));
assert!(!host_matches("example.com", "example.com.evil.org"));
assert!(!host_matches("*.example.com", ".example.com"));
}
#[test]
fn valid_urls_and_their_hosts() {
let base = "https://example.com/";
let longest = format!("{base}{}", "a".repeat(MAX_URL - base.len()));
assert_eq!(longest.len(), MAX_URL);
for (url, host) in [
("https://example.com", "example.com"),
("https://example.com/", "example.com"),
("https://example.com:443", "example.com"),
("https://example.com:443/", "example.com"),
("https://www.example.com/a/b.html", "www.example.com"),
("https://example.com/search?q=a+b&x=%20#frag", "example.com"),
("https://example.com/@user", "example.com"),
("https://example.com/a:8080/b", "example.com"),
("https://example.com/https://other.org/", "example.com"),
("https://example.com/back\\slash", "example.com"),
(longest.as_str(), "example.com"),
] {
assert_eq!(url_host(url), Some(host), "{url}");
}
}
#[test]
fn invalid_urls() {
let base = "https://example.com/";
let too_long = format!("{base}{}", "a".repeat(MAX_URL - base.len() + 1));
assert_eq!(too_long.len(), MAX_URL + 1);
for url in [
"",
"example.com",
"http://example.com/",
"HTTPS://example.com/",
"https:/example.com/",
"https://",
"https:///path",
"ftp://example.com/",
"file:///etc/passwd",
// userinfo
"https://user@example.com/",
"https://user:pw@example.com/",
"https://example.com@evil.org/",
// ports
"https://example.com:8443/",
"https://example.com:80/",
"https://example.com:/",
"https://example.com:443x/",
"https://example.com:4433/",
"https://example.com:443:443/",
// what follows the host must be the end, `:443` or `/`
"https://example.com?q=1",
"https://example.com#frag",
"https://example.com\\@evil.org/",
// hosts that are not host names
"https://localhost/",
"https://127.0.0.1/",
"https://127.1/",
"https://[::1]/",
"https://Example.com/",
"https://example.com./",
"https://b\u{fc}cher.example/",
// the rest must be printable ASCII with no space
"https://example.com/a b",
"https://example.com/a\tb",
"https://example.com/a\nb",
"https://example.com/caf\u{e9}",
"https://example.com/\u{7f}",
" https://example.com/",
"https://example.com/ ",
too_long.as_str(),
] {
assert_eq!(url_host(url), None, "{url:?} should be invalid");
}
}
#[test]
fn each_tool_parses_its_own_arguments() {
assert_eq!(
parse(ToolName::ReadFile, r#"{"path":"/home/kyle/notes/a.md"}"#),
Ok(ToolArgs::ReadFile {
path: "/home/kyle/notes/a.md".to_string()
})
);
assert_eq!(
parse(
ToolName::WriteFile,
r#"{"path":"/home/kyle/notes/a.md","content":"line\n"}"#
),
Ok(ToolArgs::WriteFile {
path: "/home/kyle/notes/a.md".to_string(),
content: "line\n".to_string()
})
);
assert_eq!(
parse(ToolName::Shell, r#"{"command":"ls -l"}"#),
Ok(ToolArgs::Shell {
command: "ls -l".to_string(),
cwd: None
})
);
assert_eq!(
parse(ToolName::Shell, r#"{"command":"ls","cwd":"/home/kyle"}"#),
Ok(ToolArgs::Shell {
command: "ls".to_string(),
cwd: Some("/home/kyle".to_string())
})
);
// A `FetchUrl` cannot be built outside `args`, so the parsed value is read through its getters.
let fetched = parse(
ToolName::HttpFetch,
r#"{"url":"https://www.example.com/a"}"#,
);
let Ok(ToolArgs::HttpFetch(target)) = fetched else {
panic!("{fetched:?}")
};
assert_eq!(target.url(), "https://www.example.com/a");
assert_eq!(target.host(), "www.example.com");
// Field order and white space in the request do not matter.
assert_eq!(
parse(
ToolName::WriteFile,
" { \"content\" : \"x\" , \"path\" : \"/a/b\" } "
),
Ok(ToolArgs::WriteFile {
path: "/a/b".to_string(),
content: "x".to_string()
})
);
// `command` and `content` are not inspected.
assert!(parse(ToolName::Shell, r#"{"command":""}"#).is_ok());
assert!(parse(ToolName::Shell, r#"{"command":"rm -rf / ; ../../x"}"#).is_ok());
assert!(parse(ToolName::WriteFile, r#"{"path":"/a/b","content":""}"#).is_ok());
}
#[test]
fn arguments_of_the_wrong_shape_are_refused() {
let cases: [(ToolName, &str); 17] = [
(ToolName::ReadFile, ""),
(ToolName::ReadFile, "null"),
(ToolName::ReadFile, "[]"),
(ToolName::ReadFile, r#""/etc/hosts""#),
(ToolName::ReadFile, "{}"),
(ToolName::ReadFile, r#"{"path":"/a","mode":"r"}"#),
(ToolName::ReadFile, r#"{"path":"/a","path":"/b"}"#),
(ToolName::ReadFile, r#"{"path":7}"#),
(ToolName::ReadFile, r#"{"path":null}"#),
(ToolName::ReadFile, r#"{"path":"/a"} trailing"#),
(ToolName::WriteFile, r#"{"path":"/a/b"}"#),
(ToolName::WriteFile, r#"{"content":"x"}"#),
(
ToolName::WriteFile,
r#"{"path":"/a/b","content":"x","append":true}"#,
),
(ToolName::Shell, r#"{"cwd":"/a"}"#),
(ToolName::Shell, r#"{"command":"ls","cwd":"/a","env":{}}"#),
(ToolName::Shell, r#"{"command":["ls"]}"#),
(
ToolName::HttpFetch,
r#"{"url":"https://example.com/","method":"POST"}"#,
),
];
for (tool, text) in cases {
match parse(tool, text) {
Err(ArgsError::Shape(_)) => {}
other => panic!("{tool:?} {text:?}: expected a shape error, got {other:?}"),
}
}
// One tool's arguments do not fit another tool.
assert!(parse(ToolName::Shell, r#"{"path":"/a"}"#).is_err());
assert!(parse(ToolName::ReadFile, r#"{"url":"https://example.com/"}"#).is_err());
}
#[test]
fn a_bad_path_or_url_is_refused_in_every_tool_that_takes_one() {
for bad in ["notes/a.md", "/home/kyle/notes/../.ssh/id", "/a//b", "/a/"] {
let quoted = serde_json::to_string(bad).unwrap();
let read = format!(r#"{{"path":{quoted}}}"#);
let write = format!(r#"{{"path":{quoted},"content":"x"}}"#);
let shell = format!(r#"{{"command":"ls","cwd":{quoted}}}"#);
assert_eq!(
parse(ToolName::ReadFile, &read),
Err(ArgsError::Path(bad.to_string()))
);
assert_eq!(
parse(ToolName::WriteFile, &write),
Err(ArgsError::Path(bad.to_string()))
);
assert_eq!(
parse(ToolName::Shell, &shell),
Err(ArgsError::Path(bad.to_string()))
);
}
assert_eq!(
parse(ToolName::HttpFetch, r#"{"url":"http://example.com/"}"#),
Err(ArgsError::Url("http://example.com/".to_string()))
);
// A NUL can only arrive as a JSON escape; it is refused once decoded.
let nul = format!("{{\"path\":\"/a{}u0000b\"}}", '\\');
assert!(matches!(
parse(ToolName::ReadFile, &nul),
Err(ArgsError::Path(_))
));
// `cwd: null` is the same as no `cwd`.
assert_eq!(
parse(ToolName::Shell, r#"{"command":"ls","cwd":null}"#),
Ok(ToolArgs::Shell {
command: "ls".to_string(),
cwd: None
})
);
}
/// What the owner is shown is the parsed value written out again, so two spellings of one path
/// look the same. The escape is built from pieces so that no tool rewrites it on the way here.
#[test]
fn canonical_json_shows_what_was_parsed() {
let escaped_slash = format!("{}u002f", '\\');
let sneaky = format!("{{ \"path\" : \"{escaped_slash}etc{escaped_slash}hosts\" }}");
assert!(sneaky.contains("u002fetc"));
let args = parse(ToolName::ReadFile, &sneaky).unwrap();
assert_eq!(
args,
ToolArgs::ReadFile {
path: "/etc/hosts".to_string()
}
);
assert_eq!(args.canonical_json(), r#"{"path":"/etc/hosts"}"#);
// Fields come out in the spec's order whatever order they came in.
let write = parse(ToolName::WriteFile, r#"{"content":"x\ny","path":"/a/b"}"#).unwrap();
assert_eq!(
write.canonical_json(),
r#"{"path":"/a/b","content":"x\ny"}"#
);
let shell = parse(ToolName::Shell, r#"{"cwd":"/a","command":"ls"}"#).unwrap();
assert_eq!(shell.canonical_json(), r#"{"command":"ls","cwd":"/a"}"#);
// An absent cwd is left out, and the host is never written: it is not an argument.
let bare = parse(ToolName::Shell, r#"{"command":"ls","cwd":null}"#).unwrap();
assert_eq!(bare.canonical_json(), r#"{"command":"ls"}"#);
let fetch = parse(ToolName::HttpFetch, r#"{"url":"https://example.com/a"}"#).unwrap();
assert_eq!(fetch.canonical_json(), r#"{"url":"https://example.com/a"}"#);
assert_eq!(fetch.tool(), ToolName::HttpFetch);
assert_eq!(write.tool(), ToolName::WriteFile);
}
@@ -0,0 +1,140 @@
//! `[runner]` in `brokerd.toml` (M3b spec, section 6). Do not edit.
use std::path::{Path, PathBuf};
use std::time::Duration;
use brokerd::args::ToolName;
use brokerd::config::{Config, ConfigError};
const DIGEST: &str = "localhost/boxmaker-tools@sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef";
fn fixture(name: &str) -> PathBuf {
Path::new(env!("CARGO_MANIFEST_DIR"))
.join("tests/fixtures/config")
.join(name)
}
fn invalid(name: &str) -> String {
match Config::load(&fixture(name)) {
Err(ConfigError::Invalid(_, why)) => why,
other => panic!("{name}: expected Invalid, got {other:?}"),
}
}
#[test]
fn without_a_runner_section_there_is_no_runner() {
assert_eq!(Config::load(&fixture("empty.toml")).unwrap().runner, None);
}
#[test]
fn a_runner_with_only_an_image_gets_every_default() {
let r = Config::load(&fixture("runner_minimal.toml"))
.unwrap()
.runner
.unwrap();
assert_eq!(r.podman, PathBuf::from("podman"));
assert_eq!(r.image, DIGEST);
assert_eq!(r.egress_network, "pasta");
assert_eq!(r.output_cap, 262_144);
assert_eq!(r.memory, "512m");
assert_eq!(r.pids, 128);
assert_eq!(
r.time_limit(ToolName::ReadFile),
Duration::from_millis(30_000)
);
assert_eq!(
r.time_limit(ToolName::WriteFile),
Duration::from_millis(30_000)
);
assert_eq!(
r.time_limit(ToolName::Shell),
Duration::from_millis(100_000)
);
assert_eq!(
r.time_limit(ToolName::HttpFetch),
Duration::from_millis(60_000)
);
}
#[test]
fn every_runner_value_can_be_set() {
let r = Config::load(&fixture("runner_full.toml"))
.unwrap()
.runner
.unwrap();
assert_eq!(r.podman, PathBuf::from("/run/current-system/sw/bin/podman"));
assert_eq!(r.egress_network, "slirp4netns");
assert_eq!((r.output_cap, r.memory.as_str(), r.pids), (1000, "1g", 64));
let limits: Vec<Duration> = ToolName::ALL.iter().map(|t| r.time_limit(*t)).collect();
assert_eq!(limits, [1, 2, 3, 4].map(Duration::from_millis));
}
#[test]
fn a_runner_needs_an_image_named_by_digest() {
assert!(matches!(
Config::load(&fixture("runner_no_image.toml")),
Err(ConfigError::Parse(..))
));
assert!(invalid("runner_tag.toml").contains("by digest"));
}
#[test]
fn unknown_runner_keys_are_errors() {
assert!(matches!(
Config::load(&fixture("runner_unknown_key.toml")),
Err(ConfigError::Parse(..))
));
}
#[test]
fn bad_values_are_errors_that_name_them() {
let dir = std::env::temp_dir().join(format!("bx-runner-cfg-{}", std::process::id()));
std::fs::create_dir_all(&dir).unwrap();
let cases = [
(
format!("image = \"x@sha256:{}\"", "0".repeat(63)),
"by digest",
),
(
format!("image = \"x@sha256:{}\"", "A".repeat(64)),
"by digest",
),
(
format!("image = \"@sha256:{}\"", "0".repeat(64)),
"by digest",
),
(format!("image = \"{DIGEST}\"\nmemory = \"lots\""), "memory"),
(format!("image = \"{DIGEST}\"\nmemory = \"512\""), "memory"),
(format!("image = \"{DIGEST}\"\nmemory = \"m\""), "memory"),
(
format!("image = \"{DIGEST}\"\noutput_cap = 0"),
"output_cap",
),
(format!("image = \"{DIGEST}\"\npids = 0"), "pids"),
(format!("image = \"{DIGEST}\"\nshell_ms = 0"), "shell_ms"),
(
format!("image = \"{DIGEST}\"\nhttp_fetch_ms = 0"),
"http_fetch_ms",
),
(
format!("image = \"{DIGEST}\"\negress_network = \"\""),
"egress_network",
),
(format!("image = \"{DIGEST}\"\npodman = \"\""), "podman"),
];
for (n, (body, word)) in cases.iter().enumerate() {
let path = dir.join(format!("c{n}.toml"));
std::fs::write(&path, format!("[runner]\n{body}\n")).unwrap();
match Config::load(&path) {
Err(ConfigError::Invalid(_, why)) => assert!(why.contains(word), "{body}: {why}"),
other => panic!("{body}: expected Invalid, got {other:?}"),
}
}
let _ = std::fs::remove_dir_all(&dir);
}
#[test]
fn the_egress_directory_is_under_home() {
let c = Config::parse("[paths]\nhome = \"/h\"\n").unwrap();
assert_eq!(c.egress_dir(), PathBuf::from("/h/run/egress"));
}
@@ -0,0 +1,271 @@
//! The Podman runtime against a fake `podman`: what it is given, and what each way a container can
//! end becomes (M3b spec, section 6). Every call goes through `runner::run`, as in `brokerd`.
//! Do not edit.
#[path = "support/build.rs"]
mod build;
#[path = "support/fake_podman.rs"]
mod fake_podman;
use std::time::{Duration, Instant};
use brokerd::container::{
CANNOT_START, COULD_NOT_RUN, KILLED, Podman, RUNBOOK, TIMED_OUT, UNEXPECTED,
};
use brokerd::policy::{Outcome, SessionState, decide};
use brokerd::runner::run;
use build::{grant, now, read, request, set};
use fake_podman::{Fake, Lines, serial};
use proto::{DataClass, Mode, ToolRequest, ToolResponse};
fn call(podman: &Podman, req: ToolRequest, grants: Vec<build::Build>) -> ToolResponse {
let decision = match decide(req, &set(grants), SessionState::default(), now()) {
Outcome::Allowed(d) => d,
other => panic!("not allowed: {other:?}"),
};
run(decision, podman)
}
fn notes() -> Vec<build::Build> {
vec![grant("notes", "read_file", Mode::Auto).paths(&["/n"])]
}
fn podman(fake: &Fake, extra: &str, log: &Lines) -> Podman {
Podman::new(fake.runner(extra), fake.dir.join("egress"), log.sink())
}
fn failed(message: &str) -> ToolResponse {
ToolResponse::Failed {
message: message.to_string(),
}
}
#[test]
fn a_tool_that_succeeds_is_a_result_labelled_by_its_grant() {
let _s = serial();
let fake = Fake::new("ok", r#"cat > "$D/stdin"; printf 'the file text'; exit 0"#);
let log = Lines::default();
let got = call(&podman(&fake, "", &log), read("/n/a.md"), notes());
assert_eq!(
got,
ToolResponse::Result {
content: "the file text".to_string(),
class: DataClass::Private,
untrusted: true,
truncated: false,
}
);
assert_eq!(
fake.stdin(),
r#"{"path":"/n/a.md"}"#,
"the arguments go on standard input"
);
}
#[test]
fn podman_is_given_the_tool_argument_list_and_the_first_container_is_numbered_0() {
let _s = serial();
let fake = Fake::new("args", r#"cat > /dev/null; exit 0"#);
let log = Lines::default();
let p = podman(&fake, "", &log);
call(&p, read("/n/a.md"), notes());
call(&p, read("/n/b.md"), notes());
let calls = fake.calls();
assert_eq!(
calls.len(),
2,
"one podman run per call and nothing else: {calls:?}"
);
assert_eq!(calls[0][3], "--name=boxmaker-s1-1-0");
assert_eq!(calls[1][3], "--name=boxmaker-s1-1-1");
// The whole list is `podman::tool_args`, tested as golden files in podman_args.rs.
assert_eq!(calls[0].first().map(String::as_str), Some("run"));
assert_eq!(calls[0].last().map(String::as_str), Some("read_file"));
assert!(calls[0].contains(&"--volume=/n:/n:ro".to_string()));
}
#[test]
fn exit_1_is_the_tools_own_error_and_still_a_result() {
let _s = serial();
let fake = Fake::new(
"e1",
"cat > /dev/null; printf 'read_file: /n/x: no such file'; exit 1",
);
let log = Lines::default();
let got = call(&podman(&fake, "", &log), read("/n/x"), notes());
assert!(
matches!(&got, ToolResponse::Result { content, .. } if content == "read_file: /n/x: no such file"),
"{got:?}"
);
}
#[test]
fn every_other_ending_is_a_fixed_sentence() {
let cases = [
("2", COULD_NOT_RUN),
("125", CANNOT_START),
("126", CANNOT_START),
("127", CANNOT_START),
("137", KILLED),
("3", UNEXPECTED),
("124", UNEXPECTED),
];
for (code, sentence) in cases {
let _s = serial();
let body = format!(
"cat > /dev/null; printf 'secret tool output'; echo 'podman said this' >&2; exit {code}"
);
let fake = Fake::new("codes", &body);
let log = Lines::default();
let got = call(&podman(&fake, "", &log), read("/n/a"), notes());
assert_eq!(got, failed(sentence), "exit {code}");
}
}
#[test]
fn a_podman_failure_is_logged_with_the_runbook_pointer_and_its_stderr() {
let _s = serial();
let fake = Fake::new(
"125",
"cat > /dev/null; echo 'Error: image not known' >&2; exit 125",
);
let log = Lines::default();
call(&podman(&fake, "", &log), read("/n/a"), notes());
let text = log.all();
assert!(text.contains("Error: image not known"), "{text}");
assert!(text.contains(RUNBOOK), "{text}");
}
#[test]
fn a_podman_that_cannot_be_started_is_unavailable_and_logged() {
let _s = serial();
let fake = Fake::new("missing", "exit 0");
let mut runner = fake.runner("");
runner.podman = fake.dir.join("no-such-podman");
let log = Lines::default();
let p = Podman::new(runner, fake.dir.join("egress"), log.sink());
assert_eq!(call(&p, read("/n/a"), notes()), failed(CANNOT_START));
assert!(log.all().contains(RUNBOOK), "{}", log.all());
}
#[test]
fn output_past_the_cap_is_cut_and_marked() {
let _s = serial();
let fake = Fake::new(
"cap",
"cat > /dev/null; head -c 1000 /dev/zero | tr '\\0' x; exit 0",
);
let log = Lines::default();
let got = call(
&podman(&fake, "output_cap = 100", &log),
read("/n/a"),
notes(),
);
assert!(
matches!(&got, ToolResponse::Result { content, truncated: true, .. } if *content == "x".repeat(100)),
"{got:?}"
);
let exact = Fake::new(
"cap-exact",
"cat > /dev/null; head -c 100 /dev/zero | tr '\\0' x; exit 0",
);
let got = call(
&podman(&exact, "output_cap = 100", &log),
read("/n/a"),
notes(),
);
assert!(
matches!(
&got,
ToolResponse::Result {
truncated: false,
..
}
),
"{got:?}"
);
}
#[test]
fn output_that_is_not_utf8_is_replaced() {
let _s = serial();
let fake = Fake::new("utf8", "cat > /dev/null; printf 'a\\377b'; exit 0");
let log = Lines::default();
let got = call(&podman(&fake, "", &log), read("/n/a"), notes());
assert!(
matches!(&got, ToolResponse::Result { content, .. } if content == "a\u{fffd}b"),
"{got:?}"
);
}
#[test]
fn a_tool_past_its_time_limit_is_killed_removed_and_failed() {
let _s = serial();
// `exec`, so killing the process kills the sleep and nothing holds the pipes open.
let fake = Fake::new("slow", "cat > /dev/null; exec sleep 30");
let log = Lines::default();
let started = Instant::now();
let got = call(
&podman(&fake, "read_file_ms = 300", &log),
read("/n/a"),
notes(),
);
let took = started.elapsed();
assert_eq!(got, failed(TIMED_OUT));
assert!(took >= Duration::from_millis(300), "{took:?}");
assert!(took < Duration::from_secs(5), "{took:?}");
let calls = fake.calls();
assert_eq!(calls.len(), 3, "{calls:?}");
assert_eq!(calls[1], ["kill", "boxmaker-s1-1-0"]);
assert_eq!(calls[2], ["rm", "-f", "boxmaker-s1-1-0"]);
}
#[test]
fn a_large_argument_is_written_whole_while_the_tool_reads_it() {
let _s = serial();
let fake = Fake::new("big", r#"cat > "$D/stdin"; printf done; exit 0"#);
let log = Lines::default();
let content = "y".repeat(900_000);
let req = request(
"write_file",
&format!(r#"{{"path":"/w/big.txt","content":"{content}"}}"#),
);
let got = call(
&podman(&fake, "", &log),
req,
vec![grant("w", "write_file", Mode::Auto).paths(&["/w"])],
);
assert!(
matches!(&got, ToolResponse::Result { content, .. } if content == "done"),
"{got:?}"
);
assert_eq!(
fake.stdin().len(),
content.len() + r#"{"path":"/w/big.txt","content":""}"#.len()
);
}
#[test]
fn a_tool_that_never_reads_its_input_still_ends() {
let _s = serial();
let fake = Fake::new("noread", "printf ignored; exit 0");
let log = Lines::default();
let req = request(
"write_file",
&format!(
r#"{{"path":"/w/big.txt","content":"{}"}}"#,
"z".repeat(900_000)
),
);
let started = Instant::now();
let got = call(
&podman(&fake, "write_file_ms = 5000", &log),
req,
vec![grant("w", "write_file", Mode::Auto).paths(&["/w"])],
);
assert!(
matches!(&got, ToolResponse::Result { content, .. } if content == "ignored"),
"{got:?}"
);
assert!(started.elapsed() < Duration::from_secs(4));
}
@@ -0,0 +1,213 @@
//! `http_fetch` through the Podman runtime: the proxy is started first, its socket awaited, the tool
//! run with the directory mounted, and the proxy and directory removed afterwards on every path
//! (M3b spec, section 6, "http_fetch"). Against a fake `podman`. Do not edit.
#[path = "support/build.rs"]
mod build;
#[path = "support/fake_podman.rs"]
mod fake_podman;
use std::os::unix::fs::PermissionsExt;
use std::time::{Duration, Instant};
use brokerd::container::{CANNOT_START, Podman, RUNBOOK, TIMED_OUT};
use brokerd::policy::{Outcome, SessionState, decide};
use brokerd::runner::run;
use build::{fetch, grant, now, set};
use fake_podman::{Fake, Lines, path, serial};
use proto::{Mode, ToolResponse};
/// For `run -d` (the proxy): make the socket file in the mounted directory, as the proxy does.
const PROXY_OK: &str = r#"for a in "$@"; do case "$a" in --volume=*:/run/egress:rw) v=${a#--volume=}; v=${v%:/run/egress:rw};; esac; done"#;
fn body(proxy: &str, tool: &str) -> String {
format!("{PROXY_OK}\nif [ \"$2\" = -d ]; then\n{proxy}\nfi\n{tool}")
}
fn call(podman: &Podman) -> ToolResponse {
let grants = set(vec![
grant("web", "http_fetch", Mode::Auto).hosts(&["example.com", "*.example.org"]),
]);
let decision = match decide(
fetch("https://example.com/a"),
&grants,
SessionState::default(),
now(),
) {
Outcome::Allowed(d) => d,
other => panic!("not allowed: {other:?}"),
};
run(decision, podman)
}
fn failed(message: &str) -> ToolResponse {
ToolResponse::Failed {
message: message.to_string(),
}
}
#[test]
fn the_proxy_starts_first_the_tool_gets_its_socket_and_both_are_cleaned_up() {
let _s = serial();
let fake = Fake::new(
"eg-ok",
&body(
r#": > "$v/egress.sock"; exit 0"#,
r#"cat > "$D/stdin"; printf 'body\n[http 200]'; exit 0"#,
),
);
let log = Lines::default();
let egress = fake.dir.join("egress");
let got = call(&Podman::new(fake.runner(""), egress.clone(), log.sink()));
assert!(
matches!(&got, ToolResponse::Result { content, .. } if content == "body\n[http 200]"),
"{got:?} {}",
log.all()
);
let calls = fake.calls();
assert_eq!(calls.len(), 3, "{calls:?}");
let dir = egress.join("boxmaker-s1-1-0");
// 1. The proxy, detached, with the call's hosts.
assert_eq!(&calls[0][..2], ["run", "-d"]);
assert!(calls[0].contains(&"--name=boxmaker-s1-1-0-egress".to_string()));
assert!(calls[0].contains(&format!("--volume={}:/run/egress:rw", path(&dir))));
assert_eq!(calls[0].last().unwrap(), "example.com,*.example.org");
// 2. The tool, with the same directory and no network.
assert_eq!(&calls[1][..3], ["run", "--rm", "-i"]);
assert!(calls[1].contains(&format!("--volume={}:/run/egress:rw", path(&dir))));
assert!(calls[1].contains(&"--network=none".to_string()));
assert_eq!(fake.stdin(), r#"{"url":"https://example.com/a"}"#);
// 3. The proxy removed, and its directory with it.
assert_eq!(calls[2], ["rm", "-f", "boxmaker-s1-1-0-egress"]);
assert!(!dir.exists(), "the call's directory is removed");
let mode = std::fs::metadata(&egress).unwrap().permissions().mode() & 0o777;
assert_eq!(mode, 0o700);
}
#[test]
fn a_proxy_that_podman_cannot_start_means_no_tool_and_nothing_left() {
let _s = serial();
let fake = Fake::new(
"eg-fail",
&body(
"echo 'Error: network pasta not found' >&2; exit 125",
"cat > /dev/null; exit 0",
),
);
let log = Lines::default();
let egress = fake.dir.join("egress");
let got = call(&Podman::new(fake.runner(""), egress.clone(), log.sink()));
assert_eq!(got, failed(CANNOT_START));
let calls = fake.calls();
assert!(
calls
.iter()
.all(|c| c.get(2).map(String::as_str) != Some("-i")),
"no tool ran: {calls:?}"
);
assert_eq!(
calls.last().unwrap(),
&["rm", "-f", "boxmaker-s1-1-0-egress"]
);
assert!(!egress.join("boxmaker-s1-1-0").exists());
assert!(
log.all().contains("network pasta not found"),
"{}",
log.all()
);
assert!(log.all().contains(RUNBOOK));
}
#[test]
fn a_proxy_that_makes_no_socket_in_time_means_no_tool_and_nothing_left() {
let _s = serial();
let fake = Fake::new("eg-nosock", &body("exit 0", "cat > /dev/null; exit 0"));
let log = Lines::default();
let egress = fake.dir.join("egress");
let podman = Podman::new(fake.runner(""), egress.clone(), log.sink())
.with_egress_wait(Duration::from_millis(200));
let started = Instant::now();
assert_eq!(call(&podman), failed(CANNOT_START));
assert!(started.elapsed() < Duration::from_secs(3));
let calls = fake.calls();
assert_eq!(calls.len(), 2, "{calls:?}");
assert_eq!(calls[1], ["rm", "-f", "boxmaker-s1-1-0-egress"]);
assert!(!egress.join("boxmaker-s1-1-0").exists());
assert!(log.all().contains(RUNBOOK));
}
#[test]
fn a_tool_that_runs_too_long_still_leaves_nothing_behind() {
let _s = serial();
let fake = Fake::new(
"eg-slow",
&body(
r#": > "$v/egress.sock"; exit 0"#,
"cat > /dev/null; exec sleep 30",
),
);
let log = Lines::default();
let egress = fake.dir.join("egress");
let got = call(&Podman::new(
fake.runner("http_fetch_ms = 300"),
egress.clone(),
log.sink(),
));
assert_eq!(got, failed(TIMED_OUT));
let calls = fake.calls();
let tail: Vec<Vec<String>> = calls[2..].to_vec();
assert_eq!(
tail,
[
vec!["kill", "boxmaker-s1-1-0"],
vec!["rm", "-f", "boxmaker-s1-1-0"],
vec!["rm", "-f", "boxmaker-s1-1-0-egress"],
]
.map(|c| c.into_iter().map(String::from).collect::<Vec<_>>())
);
assert!(!egress.join("boxmaker-s1-1-0").exists());
}
#[test]
fn a_tool_that_podman_cannot_start_still_leaves_nothing_behind() {
let _s = serial();
let fake = Fake::new(
"eg-toolfail",
&body(
r#": > "$v/egress.sock"; exit 0"#,
"cat > /dev/null; exit 125",
),
);
let log = Lines::default();
let egress = fake.dir.join("egress");
assert_eq!(
call(&Podman::new(fake.runner(""), egress.clone(), log.sink())),
failed(CANNOT_START)
);
assert_eq!(
fake.calls().last().unwrap(),
&["rm", "-f", "boxmaker-s1-1-0-egress"]
);
assert!(!egress.join("boxmaker-s1-1-0").exists());
}
#[test]
fn a_directory_left_by_a_crash_is_replaced() {
let _s = serial();
let fake = Fake::new(
"eg-stale",
&body(
r#"[ -e "$v/old.sock" ] && exit 9; : > "$v/egress.sock"; exit 0"#,
"cat > /dev/null; printf ok; exit 0",
),
);
let log = Lines::default();
let egress = fake.dir.join("egress");
std::fs::create_dir_all(egress.join("boxmaker-s1-1-0")).unwrap();
std::fs::write(egress.join("boxmaker-s1-1-0/old.sock"), "").unwrap();
let got = call(&Podman::new(fake.runner(""), egress.clone(), log.sink()));
assert!(
matches!(&got, ToolResponse::Result { content, .. } if content == "ok"),
"{got:?}"
);
}
@@ -0,0 +1,11 @@
[runner]
podman = "/run/current-system/sw/bin/podman"
image = "localhost/boxmaker-tools@sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef"
egress_network = "slirp4netns"
output_cap = 1000
memory = "1g"
pids = 64
read_file_ms = 1
write_file_ms = 2
shell_ms = 3
http_fetch_ms = 4
@@ -0,0 +1,2 @@
[runner]
image = "localhost/boxmaker-tools@sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef"
@@ -0,0 +1,2 @@
[runner]
podman = "podman"
@@ -0,0 +1,2 @@
[runner]
image = "localhost/boxmaker-tools:latest"
@@ -0,0 +1,3 @@
[runner]
image = "localhost/boxmaker-tools@sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef"
network = "host"
@@ -0,0 +1,20 @@
run
-d
--rm
--name=boxmaker-s1-1-7-egress
--label=boxmaker=egress
--network=pasta
--read-only
--cap-drop=all
--security-opt=no-new-privileges
--userns=keep-id
--pids-limit=64
--memory=128m
--volume=/h/run/egress/boxmaker-s1-1-7:/run/egress:rw
localhost/boxmaker-tools@sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef
/bin/toolkit
egress-proxy
--socket
/run/egress/egress.sock
--allow
example.com,*.example.org
@@ -0,0 +1,17 @@
run
--rm
-i
--name=boxmaker-s1-1-7
--label=boxmaker=tool
--network=none
--read-only
--cap-drop=all
--security-opt=no-new-privileges
--userns=keep-id
--pids-limit=128
--memory=512m
--tmpfs=/tmp:rw,size=64m,mode=1777
--volume=/h/run/egress/boxmaker-s1-1-7:/run/egress:rw
localhost/boxmaker-tools@sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef
/bin/toolkit
http_fetch
@@ -0,0 +1,17 @@
run
--rm
-i
--name=boxmaker-s1-1-7
--label=boxmaker=tool
--network=none
--read-only
--cap-drop=all
--security-opt=no-new-privileges
--userns=keep-id
--pids-limit=128
--memory=512m
--tmpfs=/tmp:rw,size=64m,mode=1777
--volume=/home/kyle/notes:/home/kyle/notes:ro
localhost/boxmaker-tools@sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef
/bin/toolkit
read_file
@@ -0,0 +1,18 @@
run
--rm
-i
--name=boxmaker-s1-1-7
--label=boxmaker=tool
--network=none
--read-only
--cap-drop=all
--security-opt=no-new-privileges
--userns=keep-id
--pids-limit=128
--memory=512m
--tmpfs=/tmp:rw,size=64m,mode=1777
--volume=/srv/a:/srv/a:rw
--volume=/srv/b:/srv/b:rw
localhost/boxmaker-tools@sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef
/bin/toolkit
shell
@@ -0,0 +1,16 @@
run
--rm
-i
--name=boxmaker-s1-1-7
--label=boxmaker=tool
--network=none
--read-only
--cap-drop=all
--security-opt=no-new-privileges
--userns=keep-id
--pids-limit=128
--memory=512m
--tmpfs=/tmp:rw,size=64m,mode=1777
localhost/boxmaker-tools@sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef
/bin/toolkit
shell
@@ -0,0 +1,17 @@
run
--rm
-i
--name=boxmaker-s1-1-7
--label=boxmaker=tool
--network=none
--read-only
--cap-drop=all
--security-opt=no-new-privileges
--userns=keep-id
--pids-limit=128
--memory=512m
--tmpfs=/tmp:rw,size=64m,mode=1777
--volume=/home/kyle/out:/home/kyle/out:rw
localhost/boxmaker-tools@sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef
/bin/toolkit
write_file
@@ -0,0 +1,45 @@
//! A grant path is mounted into the tool container as `--volume=<path>:<path>:ro`, so a path with
//! `:` or `,` in it cannot be granted: the set is invalid, as for any other bad grant (M3b spec,
//! section 3). Do not edit.
#[path = "support/tmp.rs"]
mod tmp;
use brokerd::grants::load;
use tmp::TempDir;
fn grant_with_path(path: &str) -> String {
format!(
"tool = \"read_file\"\nmode = \"auto\"\nmax_taint = \"secret\"\n[constraints]\npaths = [{path:?}]\n"
)
}
#[test]
fn a_path_with_a_colon_or_a_comma_makes_the_set_invalid() {
for path in ["/home/kyle/a:b", "/home/kyle/a,b", "/x:/y", "/n,ro"] {
let dir = TempDir::new("mount-bad");
dir.write("notes.toml", &grant_with_path(path));
let problems = load(dir.path()).expect_err(path);
assert_eq!(problems.len(), 1, "{path}: {problems:?}");
assert_eq!(problems[0].file, "notes.toml");
assert!(
problems[0].problem.contains("cannot be mounted"),
"{path}: {}",
problems[0].problem
);
}
}
#[test]
fn other_punctuation_is_still_fine() {
for path in [
"/home/kyle/a b",
"/home/kyle/a;b",
"/home/kyle/a=b",
"/home/kyle/a.b-c_d",
] {
let dir = TempDir::new("mount-ok");
dir.write("notes.toml", &grant_with_path(path));
assert!(load(dir.path()).is_ok(), "{path}");
}
}
@@ -0,0 +1,155 @@
//! The `podman` argument lists, as golden files: one argument per line, compared exactly. A
//! runtime that only builds the list stands in for Podman, so the lists are built from real
//! `RunSpec`s, which only `runner::run` can make. Do not edit.
#[path = "support/build.rs"]
mod build;
use std::ffi::OsString;
use std::path::{Path, PathBuf};
use std::sync::Mutex;
use brokerd::config::Runner;
use brokerd::podman::{container_name, egress_args, tool_args};
use brokerd::policy::{Outcome, SessionState, decide};
use brokerd::runner::{RunError, RunOutput, RunSpec, Runtime, run};
use build::{fetch, grant, now, read, request, set, shell, write};
use proto::{CallId, Mode, SessionId, ToolRequest};
const IMAGE: &str = "localhost/boxmaker-tools@sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef";
fn runner() -> Runner {
let text = format!("[runner]\nimage = \"{IMAGE}\"\n");
brokerd::config::Config::parse(&text)
.unwrap()
.runner
.unwrap()
}
/// Builds the tool's argument list inside `run`, as the real runtime will.
struct Lists {
egress: Option<PathBuf>,
got: Mutex<Vec<Vec<OsString>>>,
}
impl Runtime for Lists {
fn run(&self, spec: &RunSpec) -> Result<RunOutput, RunError> {
let name = container_name(spec.session(), spec.call(), 7);
let args = tool_args(spec, &runner(), &name, self.egress.as_deref());
self.got.lock().unwrap().push(args);
Ok(RunOutput {
content: String::new(),
truncated: false,
})
}
}
fn list_for(grants: Vec<build::Build>, req: ToolRequest, egress: Option<&Path>) -> Vec<String> {
let decision = match decide(req, &set(grants), SessionState::default(), now()) {
Outcome::Allowed(d) => d,
other => panic!("not allowed: {other:?}"),
};
let lists = Lists {
egress: egress.map(Path::to_path_buf),
got: Mutex::new(Vec::new()),
};
run(decision, &lists);
let got = lists.got.into_inner().unwrap();
assert_eq!(got.len(), 1);
got[0]
.iter()
.map(|a| a.to_str().unwrap().to_string())
.collect()
}
fn golden(name: &str) -> Vec<String> {
let path = Path::new(env!("CARGO_MANIFEST_DIR"))
.join("tests/fixtures/podman")
.join(name);
let text = std::fs::read_to_string(&path).unwrap_or_else(|e| panic!("{}: {e}", path.display()));
text.lines().map(str::to_string).collect()
}
#[test]
fn the_container_name_says_whose_call_it_is() {
let s = SessionId::new("chat-17").unwrap();
assert_eq!(container_name(&s, CallId(42), 3), "boxmaker-chat-17-42-3");
}
#[test]
fn read_file_gets_its_one_directory_read_only_and_no_network() {
let got = list_for(
vec![grant("notes", "read_file", Mode::Auto).paths(&["/home/kyle/notes"])],
read("/home/kyle/notes/a.md"),
None,
);
assert_eq!(got, golden("read_file.args"));
}
#[test]
fn write_file_gets_its_one_directory_writable() {
let got = list_for(
vec![grant("out", "write_file", Mode::Auto).paths(&["/home/kyle/out"])],
write("/home/kyle/out/b.md"),
None,
);
assert_eq!(got, golden("write_file.args"));
}
#[test]
fn shell_gets_every_grant_directory_writable_in_order() {
let got = list_for(
vec![grant("sh", "shell", Mode::Auto).paths(&["/srv/a", "/srv/b"])],
shell(Some("/srv/b")),
None,
);
assert_eq!(got, golden("shell.args"));
let bare = list_for(
vec![grant("sh", "shell", Mode::Auto)],
request("shell", r#"{"command":"ls"}"#),
None,
);
assert_eq!(bare, golden("shell_no_paths.args"));
}
#[test]
fn http_fetch_gets_the_egress_directory_and_still_no_network() {
let got = list_for(
vec![grant("web", "http_fetch", Mode::Auto).hosts(&["example.com"])],
fetch("https://example.com/a"),
Some(Path::new("/h/run/egress/boxmaker-s1-1-7")),
);
assert_eq!(got, golden("http_fetch.args"));
assert!(got.contains(&"--network=none".to_string()));
}
#[test]
fn the_proxy_gets_a_network_the_socket_and_the_hosts() {
let got: Vec<String> = egress_args(
&runner(),
"boxmaker-s1-1-7",
Path::new("/h/run/egress/boxmaker-s1-1-7"),
&["example.com".to_string(), "*.example.org".to_string()],
)
.iter()
.map(|a| a.to_str().unwrap().to_string())
.collect();
assert_eq!(got, golden("egress.args"));
}
#[test]
fn no_argument_holds_a_shell_string_or_a_second_network() {
let got = list_for(
vec![grant("sh", "shell", Mode::Auto).paths(&["/srv/a"])],
request(
"shell",
r#"{"command":"curl evil.test; rm -rf /","cwd":"/srv/a"}"#,
),
None,
);
assert!(
got.iter().all(|a| !a.contains("evil.test")),
"the command goes on standard input"
);
assert_eq!(got.iter().filter(|a| a.starts_with("--network")).count(), 1);
}
@@ -0,0 +1,143 @@
//! `brokerd serve` with a `[runner]` section runs allowed calls through Podman (here a fake), and
//! says which runtime it uses. Without the section it refuses every call, as in M3a. Do not edit.
#[path = "support/fake_podman.rs"]
mod fake_podman;
use std::io::Read;
use std::os::unix::net::UnixStream;
use std::path::Path;
use std::process::{Child, Command, Stdio};
use std::time::{Duration, Instant};
use fake_podman::{Fake, IMAGE, serial};
use proto::{
CallId, DataClass, Envelope, Message, PROTOCOL_VERSION, SessionId, ToolRequest, ToolResponse,
read_frame, write_frame,
};
struct Running(Child);
impl Drop for Running {
fn drop(&mut self) {
let _ = self.0.kill();
let _ = self.0.wait();
}
}
impl Running {
fn stop(mut self) -> String {
let _ = self.0.kill();
let _ = self.0.wait();
let mut err = String::new();
if let Some(mut stderr) = self.0.stderr.take() {
let _ = stderr.read_to_string(&mut err);
}
err
}
}
fn start(home: &Path, runner: &str) -> Running {
std::fs::create_dir_all(home.join("grants")).unwrap();
std::fs::write(
home.join("grants/notes.toml"),
"tool = \"read_file\"\nmode = \"auto\"\nmax_taint = \"secret\"\n[constraints]\npaths = [\"/n\"]\n",
)
.unwrap();
let config = home.join("brokerd.toml");
std::fs::write(
&config,
format!(
"[paths]\nhome = \"{0}\"\ngrants = \"{0}/grants\"\n{runner}\n",
home.display()
),
)
.unwrap();
let child = Command::new(env!("CARGO_BIN_EXE_brokerd"))
.args(["serve", "--config"])
.arg(&config)
.stdout(Stdio::null())
.stderr(Stdio::piped())
.spawn()
.unwrap();
let running = Running(child);
let until = Instant::now() + Duration::from_secs(10);
while UnixStream::connect(home.join("run/loop-broker/broker.sock")).is_err() {
assert!(Instant::now() < until, "brokerd never listened");
std::thread::sleep(Duration::from_millis(20));
}
running
}
fn read_call(home: &Path) -> ToolResponse {
let mut stream = UnixStream::connect(home.join("run/loop-broker/broker.sock")).unwrap();
stream
.set_read_timeout(Some(Duration::from_secs(10)))
.unwrap();
let request = ToolRequest {
session: SessionId::new("s1").unwrap(),
call: CallId(1),
tool: "read_file".to_string(),
arguments: r#"{"path":"/n/a.md"}"#.to_string(),
};
let envelope = Envelope {
v: PROTOCOL_VERSION,
id: 1,
r#final: true,
msg: Message::ToolRequest(request),
};
write_frame(&mut stream, &envelope).unwrap();
match read_frame(&mut stream).unwrap().msg {
Message::ToolResponse(response) => response,
other => panic!("{other:?}"),
}
}
#[test]
fn with_a_runner_an_allowed_call_runs_in_a_container() {
let _s = serial();
let fake = Fake::new(
"serve",
"cat > /dev/null; printf 'from the container'; exit 0",
);
let home = fake.dir.join("home");
let running = start(
&home,
&format!(
"[runner]\npodman = \"{}\"\nimage = \"{IMAGE}\"\n",
fake.script.display()
),
);
assert_eq!(
read_call(&home),
ToolResponse::Result {
content: "from the container".to_string(),
class: DataClass::Private,
untrusted: true,
truncated: false,
}
);
let printed = running.stop();
assert!(
printed.contains(&format!("tools run in containers from {IMAGE}")),
"{printed}"
);
assert_eq!(fake.calls().len(), 1);
}
#[test]
fn without_a_runner_every_call_is_refused_and_it_says_so() {
let _s = serial();
let fake = Fake::new("serve-none", "exit 0");
let home = fake.dir.join("home");
let running = start(&home, "");
assert_eq!(
read_call(&home),
ToolResponse::Failed {
message: brokerd::runner::REFUSING.to_string()
}
);
let printed = running.stop();
assert!(printed.contains("no [runner] section"), "{printed}");
assert!(fake.calls().is_empty());
}
@@ -0,0 +1,102 @@
//! A fake `podman` for the runtime tests: a shell script that records every call's arguments and,
//! for `run`, does what the test says. Do not edit.
//!
//! Included with `#[path = "support/fake_podman.rs"] mod fake_podman;`.
#![allow(dead_code)] // each test file uses a different part of this module
use std::os::unix::fs::PermissionsExt;
use std::path::{Path, PathBuf};
use std::sync::atomic::{AtomicU32, Ordering};
use std::sync::{Arc, Mutex, MutexGuard};
use brokerd::config::{Config, Runner};
static NEXT: AtomicU32 = AtomicU32::new(0);
static SERIAL: Mutex<()> = Mutex::new(());
/// Tests that write a script and run it take turns. Otherwise another test's fork can hold the
/// script open for writing at the moment it is run, and running it fails with "text file busy"
/// (ETXTBSY), which has nothing to do with the code under test.
pub fn serial() -> MutexGuard<'static, ()> {
SERIAL.lock().unwrap_or_else(|p| p.into_inner())
}
pub const IMAGE: &str = "localhost/boxmaker-tools@sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef";
pub struct Fake {
pub dir: PathBuf,
pub script: PathBuf,
}
impl Drop for Fake {
fn drop(&mut self) {
let _ = std::fs::remove_dir_all(&self.dir);
}
}
impl Fake {
/// A fake whose `run` does `run_body` (a shell fragment; `$D` is the fake's directory). Every
/// other command (`kill`, `rm`) is recorded and succeeds.
pub fn new(tag: &str, run_body: &str) -> Fake {
let n = NEXT.fetch_add(1, Ordering::SeqCst);
let dir = std::env::temp_dir().join(format!("bx-fp-{tag}-{}-{n}", std::process::id()));
let _ = std::fs::remove_dir_all(&dir);
std::fs::create_dir_all(&dir).unwrap();
let script = dir.join("podman");
let text = format!(
"#!/bin/sh\nD='{}'\nfor a in \"$@\"; do printf '%s\\n' \"$a\"; done >> \"$D/calls\"\necho --- >> \"$D/calls\"\n[ \"$1\" = run ] || exit 0\n{run_body}\n",
dir.display()
);
std::fs::write(&script, text).unwrap();
std::fs::set_permissions(&script, std::fs::Permissions::from_mode(0o755)).unwrap();
Fake { dir, script }
}
/// Every call so far, each as its arguments.
pub fn calls(&self) -> Vec<Vec<String>> {
let text = std::fs::read_to_string(self.dir.join("calls")).unwrap_or_default();
let mut calls = Vec::new();
let mut current = Vec::new();
for line in text.lines() {
if line == "---" {
calls.push(std::mem::take(&mut current));
} else {
current.push(line.to_string());
}
}
calls
}
/// What the last `run` read on standard input, if the body saved it to `$D/stdin`.
pub fn stdin(&self) -> String {
std::fs::read_to_string(self.dir.join("stdin")).unwrap_or_default()
}
/// A `[runner]` using this fake, with `extra` lines added.
pub fn runner(&self, extra: &str) -> Runner {
let text = format!(
"[runner]\npodman = \"{}\"\nimage = \"{IMAGE}\"\n{extra}\n",
self.script.display()
);
Config::parse(&text).unwrap().runner.unwrap()
}
}
/// A log that keeps its lines.
#[derive(Clone, Default)]
pub struct Lines(pub Arc<Mutex<Vec<String>>>);
impl Lines {
pub fn sink(&self) -> Arc<dyn Fn(&str) + Send + Sync> {
let lines = Arc::clone(&self.0);
Arc::new(move |l: &str| lines.lock().unwrap().push(l.to_string()))
}
pub fn all(&self) -> String {
self.0.lock().unwrap().join("\n")
}
}
pub fn path(p: &Path) -> String {
p.to_str().unwrap().to_string()
}
@@ -0,0 +1,119 @@
//! A home for ledger and broker tests: grants, audit log, session state, a flaky audit sink and
//! a log to read. Do not edit.
//!
//! Included with `#[path = "support/rig.rs"] mod rig;`, beside `tmp` and `sink`. The broker
//! tests add `client`.
#![allow(dead_code)] // each test file uses a different part of this module
use std::path::PathBuf;
use brokerd::audit::Writer;
use brokerd::config::{Approvals, Config, Paths, Sockets};
use brokerd::ledger::Ledger;
use brokerd::state::StateStore;
use proto::{AuditEvent, AuditRecord, CallId, SessionId, ToolRequest};
use crate::sink::{Flaky, Lines, Switch};
use crate::tmp::TempDir;
pub struct Rig {
pub dir: TempDir,
pub cfg: Config,
pub switch: Switch,
pub lines: Lines,
}
impl Rig {
pub fn new(tag: &str) -> Rig {
Rig::with_ttl(tag, 900_000)
}
pub fn with_ttl(tag: &str, ttl_ms: u64) -> Rig {
let dir = TempDir::new(tag);
let grants = dir.path().join("grants");
std::fs::create_dir_all(&grants).unwrap();
let cfg = Config {
paths: Paths {
home: dir.path().to_path_buf(),
grants,
},
sockets: Sockets::default(),
approvals: Approvals { ttl_ms },
runner: None,
};
Rig {
dir,
cfg,
switch: Switch::default(),
lines: Lines::default(),
}
}
pub fn state(&self) -> StateStore {
StateStore::new(&self.cfg.state_dir())
}
/// Opens the audit log (once: the writer holds its lock) behind the flaky sink.
pub fn ledger(&self) -> Ledger {
let opened = Writer::open(&self.cfg.audit_dir(), false).unwrap();
let sink = Flaky {
writer: opened.writer,
switch: self.switch.clone(),
};
Ledger::new(Box::new(sink), self.state(), self.lines.sink())
}
/// Writes `grants/<id>.toml`.
pub fn grant(&self, id: &str, text: &str) {
std::fs::write(self.cfg.paths.grants.join(format!("{id}.toml")), text).unwrap();
}
pub fn remove_grant(&self, id: &str) {
std::fs::remove_file(self.cfg.paths.grants.join(format!("{id}.toml"))).unwrap();
}
pub fn state_file(&self, session: &str) -> PathBuf {
self.cfg.state_dir().join(format!("{session}.json"))
}
/// Every record in the audit log, in order.
pub fn records(&self) -> Vec<AuditRecord> {
let dir = self.cfg.audit_dir();
let mut names: Vec<String> = std::fs::read_dir(&dir)
.unwrap()
.map(|e| e.unwrap().file_name().into_string().unwrap())
.filter(|n| n.ends_with(".jsonl"))
.collect();
names.sort();
let mut out = Vec::new();
for name in names {
let text = std::fs::read_to_string(dir.join(name)).unwrap();
for line in text.lines() {
out.push(serde_json::from_str(line).unwrap());
}
}
out
}
pub fn events(&self) -> Vec<AuditEvent> {
self.records().into_iter().map(|r| r.event).collect()
}
}
/// A grant file's text. `extra` goes before `[constraints]`, `constraints` after it.
pub fn grant_text(tool: &str, mode: &str, extra: &str, constraints: &str) -> String {
format!(
"tool = \"{tool}\"\nmode = \"{mode}\"\nmax_taint = \"secret\"\nresult_class = \"private\"\n\
untrusted = false\n{extra}\n[constraints]\n{constraints}\n"
)
}
pub fn request(session: &str, call: u64, tool: &str, arguments: &str) -> ToolRequest {
ToolRequest {
session: SessionId::new(session).unwrap(),
call: CallId(call),
tool: tool.to_string(),
arguments: arguments.to_string(),
}
}
@@ -0,0 +1,50 @@
//! Host names and patterns, shared by `brokerd` and `toolkit`'s egress proxy. The full tables are
//! in `crates/brokerd/tests/args.rs`, which reaches these through `brokerd::args`. Do not edit.
use proto::hosts::{host_matches, valid_host, valid_host_pattern};
#[test]
fn hosts() {
for good in ["example.com", "a.b.example.com", "x-1.example.org", "a.b"] {
assert!(valid_host(good), "{good}");
}
for bad in [
"",
"example",
"Example.com",
"-a.com",
"a-.com",
"a..com",
".a.com",
"a.com.",
"127.0.0.1",
"127.1",
"1.2.3.4x",
"[::1]",
"a_b.com",
"a.com:443",
"*.a.com",
] {
assert!(!valid_host(bad), "{bad}");
}
}
#[test]
fn patterns() {
assert!(valid_host_pattern("example.com"));
assert!(valid_host_pattern("*.example.com"));
for bad in ["*", "*.", "*.*.a.com", "a.*.com", "**.a.com", "*a.com"] {
assert!(!valid_host_pattern(bad), "{bad}");
}
}
#[test]
fn matching() {
assert!(host_matches("example.com", "example.com"));
assert!(!host_matches("example.com", "www.example.com"));
assert!(host_matches("*.example.com", "www.example.com"));
assert!(host_matches("*.example.com", "a.b.example.com"));
assert!(!host_matches("*.example.com", "example.com"));
assert!(!host_matches("*.example.com", "badexample.com"));
assert!(!host_matches("*.example.com", ".example.com"));
}
@@ -0,0 +1,60 @@
//! The four tools' arguments: `brokerd` writes them to a container's standard input, `toolkit`
//! reads them back, and both use these types. Do not edit.
use proto::tools::{HttpFetchArgs, ReadFileArgs, ShellArgs, WriteFileArgs};
#[test]
fn each_type_round_trips_in_field_order() {
let read = ReadFileArgs {
path: "/n/a.md".to_string(),
};
assert_eq!(
serde_json::to_string(&read).unwrap(),
r#"{"path":"/n/a.md"}"#
);
let write = WriteFileArgs {
path: "/n/a.md".to_string(),
content: "hi\n".to_string(),
};
let text = serde_json::to_string(&write).unwrap();
assert_eq!(text, r#"{"path":"/n/a.md","content":"hi\n"}"#);
assert_eq!(serde_json::from_str::<WriteFileArgs>(&text).unwrap(), write);
let fetch = HttpFetchArgs {
url: "https://example.com/".to_string(),
};
assert_eq!(
serde_json::to_string(&fetch).unwrap(),
r#"{"url":"https://example.com/"}"#
);
}
#[test]
fn a_shell_cwd_is_left_out_when_absent_and_may_be_null_or_missing() {
let bare = ShellArgs {
command: "ls".to_string(),
cwd: None,
};
assert_eq!(serde_json::to_string(&bare).unwrap(), r#"{"command":"ls"}"#);
for text in [r#"{"command":"ls"}"#, r#"{"command":"ls","cwd":null}"#] {
assert_eq!(serde_json::from_str::<ShellArgs>(text).unwrap(), bare);
}
let with = ShellArgs {
command: "ls".to_string(),
cwd: Some("/n".to_string()),
};
assert_eq!(
serde_json::to_string(&with).unwrap(),
r#"{"command":"ls","cwd":"/n"}"#
);
}
#[test]
fn unknown_and_missing_fields_are_refused() {
assert!(serde_json::from_str::<ReadFileArgs>(r#"{"path":"/a","mode":"x"}"#).is_err());
assert!(serde_json::from_str::<ReadFileArgs>(r#"{}"#).is_err());
assert!(serde_json::from_str::<WriteFileArgs>(r#"{"path":"/a"}"#).is_err());
assert!(serde_json::from_str::<ShellArgs>(r#"{"command":"ls","env":{}}"#).is_err());
assert!(
serde_json::from_str::<HttpFetchArgs>(r#"{"url":"https://a.b/","method":"POST"}"#).is_err()
);
}
@@ -0,0 +1,116 @@
//! `is_public`: the egress proxy connects only to public addresses. Every range in the M3b spec,
//! section 5, has a case at each end, and a public neighbour just outside it. Do not edit.
use std::net::IpAddr;
use toolkit::addr::is_public;
fn ip(s: &str) -> IpAddr {
s.parse().unwrap()
}
#[test]
fn refused_ipv4() {
for s in [
"0.0.0.0",
"0.255.255.255",
"10.0.0.0",
"10.255.255.255",
"100.64.0.0",
"100.100.100.100",
"100.127.255.255",
"127.0.0.1",
"127.255.255.255",
"169.254.0.1",
"169.254.255.255",
"172.16.0.0",
"172.31.255.255",
"192.0.0.0",
"192.0.0.255",
"192.0.2.1",
"192.168.0.1",
"192.168.255.255",
"198.18.0.0",
"198.19.255.255",
"198.51.100.7",
"203.0.113.9",
"224.0.0.1",
"239.255.255.255",
"240.0.0.0",
"255.255.255.255",
] {
assert!(!is_public(ip(s)), "{s} must be refused");
}
}
#[test]
fn public_ipv4() {
for s in [
"1.1.1.1",
"8.8.8.8",
"9.255.255.255",
"11.0.0.0",
"100.63.255.255",
"100.128.0.0",
"126.255.255.255",
"128.0.0.0",
"169.253.255.255",
"172.15.255.255",
"172.32.0.0",
"192.0.1.0",
"192.0.3.0",
"192.167.255.255",
"192.169.0.0",
"198.17.255.255",
"198.20.0.0",
"198.51.99.255",
"203.0.112.255",
"223.255.255.255",
"93.184.216.34",
] {
assert!(is_public(ip(s)), "{s} is public");
}
}
#[test]
fn refused_ipv6() {
for s in [
"::",
"::1",
"fc00::1",
"fdff:ffff::1",
"fd7a:115c:a1e0::1",
"fe80::1",
"febf::1",
"ff02::1",
"ff00::",
"2001:db8::1",
"2001:db8:ffff::1",
"::ffff:127.0.0.1",
"::ffff:10.1.2.3",
"::ffff:100.100.100.100",
"64:ff9b::7f00:1",
"64:ff9b::a01:203",
"::ffff:0.0.0.0",
"::127.0.0.1",
"::1.1.1.1",
"::ffff",
] {
assert!(!is_public(ip(s)), "{s} must be refused");
}
}
#[test]
fn public_ipv6() {
for s in [
"2606:4700:4700::1111",
"2a00:1450::1",
"fbff::1",
"fec0::1",
"2001:db9::1",
"::ffff:1.1.1.1",
"64:ff9b::101:101",
] {
assert!(is_public(ip(s)), "{s} is public");
}
}
@@ -0,0 +1,451 @@
//! The egress proxy against a fake resolver and a local echo server: every reply code, the host,
//! port and address checks, the byte copy both ways with half-close, the handshake deadline, the
//! connection limit, and `toolkit egress-proxy` as a program. No test needs a network. Do not edit.
mod support;
use std::collections::HashMap;
use std::io::{Read, Write};
use std::net::{Shutdown, SocketAddr, TcpListener, TcpStream};
use std::os::unix::net::{UnixListener, UnixStream};
use std::sync::{Arc, Mutex};
use std::time::{Duration, Instant};
use support::TempDir;
use toolkit::egress::{
ADDRESS_TYPE_NOT_SUPPORTED, Allow, COMMAND_NOT_SUPPORTED, CONNECTION_REFUSED, Dial,
HOST_UNREACHABLE, MAX_CONNECTIONS, NOT_ALLOWED, Proxy,
};
/// Names resolve from a table; every connection goes to one local echo server, except to
/// addresses listed as refusing. Records every address it was asked to connect to.
struct FakeDial {
names: HashMap<String, Vec<SocketAddr>>,
refusing: Vec<SocketAddr>,
echo: SocketAddr,
connected: Mutex<Vec<SocketAddr>>,
}
impl Dial for FakeDial {
fn resolve(&self, host: &str, _port: u16) -> std::io::Result<Vec<SocketAddr>> {
self.names
.get(host)
.cloned()
.ok_or_else(|| std::io::Error::other("no such name"))
}
fn connect(&self, addr: SocketAddr, _timeout: Duration) -> std::io::Result<TcpStream> {
self.connected.lock().unwrap().push(addr);
if self.refusing.contains(&addr) {
return Err(std::io::Error::from(std::io::ErrorKind::ConnectionRefused));
}
TcpStream::connect(self.echo)
}
}
fn sa(s: &str) -> SocketAddr {
s.parse().unwrap()
}
/// An echo server: copies back what it reads, and closes its side after reading the end.
fn echo_server() -> SocketAddr {
let listener = TcpListener::bind("127.0.0.1:0").unwrap();
let addr = listener.local_addr().unwrap();
std::thread::spawn(move || {
for stream in listener.incoming() {
let Ok(mut stream) = stream else { continue };
std::thread::spawn(move || {
let mut back = stream.try_clone().unwrap();
let _ = std::io::copy(&mut stream, &mut back);
let _ = back.shutdown(Shutdown::Write);
});
}
});
addr
}
fn dial() -> Arc<FakeDial> {
let mut names = HashMap::new();
names.insert("example.com".to_string(), vec![sa("93.184.216.34:443")]);
names.insert(
"www.example.org".to_string(),
vec![sa("[2606:2800::1]:443")],
);
names.insert(
"mixed.example.com".to_string(),
vec![
sa("10.0.0.1:443"),
sa("100.100.100.100:443"),
sa("1.1.1.1:443"),
],
);
names.insert(
"inside.example.com".to_string(),
vec![
sa("100.101.102.103:443"),
sa("127.0.0.1:443"),
sa("[::1]:443"),
],
);
names.insert("refusing.example.com".to_string(), vec![sa("8.8.8.8:443")]);
Arc::new(FakeDial {
names,
refusing: vec![sa("8.8.8.8:443")],
echo: echo_server(),
connected: Mutex::new(Vec::new()),
})
}
fn allow() -> Allow {
Allow::parse(
"example.com,*.example.org,mixed.example.com,inside.example.com,refusing.example.com,nowhere.example.com",
)
.unwrap()
}
/// A proxy on one end of a socket pair, handling that one connection on its own thread.
fn start(dial: Arc<FakeDial>) -> (UnixStream, std::thread::JoinHandle<()>) {
let (client, server) = UnixStream::pair().unwrap();
client
.set_read_timeout(Some(Duration::from_secs(5)))
.unwrap();
let proxy = Proxy::new(allow(), dial).with_handshake_timeout(Duration::from_millis(300));
let handle = std::thread::spawn(move || proxy.handle(server));
(client, handle)
}
fn request(host: &str, port: u16) -> Vec<u8> {
let mut bytes = vec![5, 1, 0, 3, u8::try_from(host.len()).unwrap()];
bytes.extend_from_slice(host.as_bytes());
bytes.extend_from_slice(&port.to_be_bytes());
bytes
}
fn read_exactly(client: &mut UnixStream, n: usize) -> Vec<u8> {
let mut buf = vec![0u8; n];
client.read_exact(&mut buf).unwrap();
buf
}
/// Read until the proxy closes; the bytes read. A close that leaves some of our bytes unread by
/// the proxy arrives as "connection reset" rather than as the end: both mean closed. A timeout
/// does not: the proxy did not close.
fn read_to_close(client: &mut UnixStream) -> Vec<u8> {
let mut rest = Vec::new();
let mut buf = [0u8; 4096];
loop {
match client.read(&mut buf) {
Ok(0) => return rest,
Ok(n) => rest.extend_from_slice(&buf[..n]),
Err(e) if e.kind() == std::io::ErrorKind::ConnectionReset => return rest,
Err(e) => panic!("the proxy did not close: {e}"),
}
}
}
/// Greeting, then `req`; the reply's code, and that the proxy then closed (for failures).
fn reply_code(req: &[u8]) -> u8 {
let (mut client, handle) = start(dial());
client.write_all(&[5, 1, 0]).unwrap();
assert_eq!(read_exactly(&mut client, 2), [5, 0]);
client.write_all(req).unwrap();
let reply = read_exactly(&mut client, 10);
assert_eq!(reply[0], 5);
assert_eq!(&reply[2..], &[0, 1, 0, 0, 0, 0, 0, 0]);
if reply[1] != 0 {
assert_eq!(
read_to_close(&mut client),
b"",
"a refusal is followed by the close"
);
handle.join().unwrap();
}
reply[1]
}
#[test]
fn an_allowed_host_is_connected_and_bytes_flow_both_ways_until_both_ends_close() {
let dial = dial();
let (mut client, handle) = start(Arc::clone(&dial));
client.write_all(&[5, 2, 2, 0]).unwrap();
assert_eq!(
read_exactly(&mut client, 2),
[5, 0],
"method 0 among others"
);
client.write_all(&request("example.com", 443)).unwrap();
assert_eq!(
read_exactly(&mut client, 10),
[5, 0, 0, 1, 0, 0, 0, 0, 0, 0]
);
client.write_all(b"hello through the tunnel").unwrap();
assert_eq!(read_exactly(&mut client, 24), b"hello through the tunnel");
client.shutdown(Shutdown::Write).unwrap();
assert_eq!(
read_to_close(&mut client),
b"",
"the half-close reached the server and back"
);
handle.join().unwrap();
assert_eq!(
*dial.connected.lock().unwrap(),
vec![sa("93.184.216.34:443")]
);
}
#[test]
fn a_wildcard_pattern_allows_a_name_under_it() {
assert_eq!(reply_code(&request("www.example.org", 443)), 0);
}
#[test]
fn a_greeting_without_method_zero_is_answered_ff_and_closed() {
let (mut client, handle) = start(dial());
client.write_all(&[5, 2, 1, 2]).unwrap();
assert_eq!(read_exactly(&mut client, 2), [5, 0xff]);
assert_eq!(read_to_close(&mut client), b"");
handle.join().unwrap();
}
#[test]
fn another_version_is_closed_without_a_word() {
let (mut client, handle) = start(dial());
client.write_all(&[4, 1, 0]).unwrap();
assert_eq!(read_to_close(&mut client), b"");
handle.join().unwrap();
}
#[test]
fn only_connect_by_name_to_an_allowed_host_on_443() {
let mut bind = request("example.com", 443);
bind[1] = 2;
assert_eq!(reply_code(&bind), COMMAND_NOT_SUPPORTED);
let mut udp = request("example.com", 443);
udp[1] = 3;
assert_eq!(reply_code(&udp), COMMAND_NOT_SUPPORTED);
assert_eq!(
reply_code(&[5, 1, 0, 1, 93, 184, 216, 34, 1, 187]),
ADDRESS_TYPE_NOT_SUPPORTED
);
let mut v6 = vec![5, 1, 0, 4];
v6.extend_from_slice(&[0; 16]);
v6.extend_from_slice(&443u16.to_be_bytes());
assert_eq!(reply_code(&v6), ADDRESS_TYPE_NOT_SUPPORTED);
for (host, port) in [
("example.com", 80),
("example.com", 8443),
("evil.test", 443),
("example.org", 443), // *.example.org does not cover example.org
("Example.com", 443), // not a valid host name
("example.com.", 443),
("127.0.0.1", 443), // an IP literal sent as a name
("wwwexample.com", 443),
] {
assert_eq!(
reply_code(&request(host, port)),
NOT_ALLOWED,
"{host}:{port}"
);
}
}
#[test]
fn an_empty_or_non_utf8_name_is_not_allowed() {
assert_eq!(reply_code(&[5, 1, 0, 3, 0, 1, 187]), NOT_ALLOWED);
assert_eq!(
reply_code(&[5, 1, 0, 3, 2, 0xff, 0xfe, 1, 187]),
NOT_ALLOWED
);
}
#[test]
fn only_public_addresses_are_used() {
let dial = dial();
let (mut client, handle) = start(Arc::clone(&dial));
client.write_all(&[5, 1, 0]).unwrap();
read_exactly(&mut client, 2);
client
.write_all(&request("mixed.example.com", 443))
.unwrap();
assert_eq!(read_exactly(&mut client, 10)[1], 0);
drop(client);
handle.join().unwrap();
assert_eq!(
*dial.connected.lock().unwrap(),
vec![sa("1.1.1.1:443")],
"the private and tailnet addresses were skipped, not tried"
);
}
#[test]
fn a_name_with_no_public_address_or_no_address_is_unreachable() {
assert_eq!(
reply_code(&request("inside.example.com", 443)),
HOST_UNREACHABLE
);
assert_eq!(
reply_code(&request("nowhere.example.com", 443)),
HOST_UNREACHABLE
);
}
#[test]
fn a_refused_connection_is_reply_5() {
assert_eq!(
reply_code(&request("refusing.example.com", 443)),
CONNECTION_REFUSED
);
}
#[test]
fn a_stalled_handshake_is_closed_at_the_deadline() {
let (mut client, handle) = start(dial());
client.write_all(&[5, 1, 0]).unwrap();
assert_eq!(read_exactly(&mut client, 2), [5, 0]);
client.write_all(&[5, 1]).unwrap(); // half a request, then nothing
let started = Instant::now();
assert_eq!(read_to_close(&mut client), b"");
let took = started.elapsed();
handle.join().unwrap();
assert!(
took < Duration::from_secs(2),
"one deadline for the handshake: {took:?}"
);
}
#[test]
fn a_handshake_that_trickles_still_ends_at_the_deadline() {
let (mut client, handle) = start(dial());
let started = Instant::now();
// One byte every 100 ms: each read is quick, but the handshake as a whole is not.
for byte in [5u8, 1, 0, 5, 1, 0, 3, 11] {
if client.write_all(&[byte]).is_err() {
break;
}
std::thread::sleep(Duration::from_millis(100));
}
let _ = read_to_close(&mut client);
handle.join().unwrap();
assert!(started.elapsed() < Duration::from_secs(2));
}
#[test]
fn allow_lists_parse_strictly() {
assert!(Allow::parse("example.com").is_ok());
assert!(Allow::parse("example.com,*.example.org").is_ok());
for bad in [
"",
",",
"example.com,",
"a.com,,b.com",
"Example.com",
"*",
"10.0.0.1",
] {
assert!(Allow::parse(bad).is_err(), "{bad:?}");
}
let allow = Allow::parse("*.example.org").unwrap();
assert!(allow.permits("a.example.org"));
assert!(!allow.permits("example.org"));
assert!(!allow.permits("A.example.org"));
}
#[test]
fn more_than_the_limit_of_connections_are_closed_at_once() {
let dir = TempDir::new("egress-limit");
let path = dir.path().join("egress.sock");
let listener = UnixListener::bind(&path).unwrap();
let proxy =
Arc::new(Proxy::new(allow(), dial()).with_handshake_timeout(Duration::from_secs(5)));
std::thread::spawn(move || {
let _ = proxy.serve(listener);
});
// MAX_CONNECTIONS clients that greet and then wait, holding their places.
let mut held = Vec::new();
for _ in 0..MAX_CONNECTIONS {
let mut c = UnixStream::connect(&path).unwrap();
c.set_read_timeout(Some(Duration::from_secs(5))).unwrap();
c.write_all(&[5, 1, 0]).unwrap();
assert_eq!(read_exactly(&mut c, 2), [5, 0]);
held.push(c);
}
let mut extra = UnixStream::connect(&path).unwrap();
extra
.set_read_timeout(Some(Duration::from_secs(2)))
.unwrap();
let _ = extra.write_all(&[5, 1, 0]);
let mut buf = [0u8; 2];
assert!(
matches!(extra.read(&mut buf), Ok(0) | Err(_)),
"the connection over the limit gets no answer"
);
// When one place is freed, a new connection is served again.
drop(held.pop());
std::thread::sleep(Duration::from_millis(200));
let mut again = UnixStream::connect(&path).unwrap();
again
.set_read_timeout(Some(Duration::from_secs(5)))
.unwrap();
again.write_all(&[5, 1, 0]).unwrap();
assert_eq!(read_exactly(&mut again, 2), [5, 0]);
}
#[test]
fn the_program_listens_where_it_is_told_and_refuses_what_is_not_allowed() {
let dir = TempDir::new("egress-prog");
let path = dir.path().join("egress.sock");
let mut child = std::process::Command::new(env!("CARGO_BIN_EXE_toolkit"))
.args([
"egress-proxy",
"--socket",
path.to_str().unwrap(),
"--allow",
"example.com",
])
.spawn()
.unwrap();
let until = Instant::now() + Duration::from_secs(5);
let mut client = loop {
if let Ok(c) = UnixStream::connect(&path) {
break c;
}
assert!(Instant::now() < until, "the proxy never listened");
std::thread::sleep(Duration::from_millis(20));
};
client
.set_read_timeout(Some(Duration::from_secs(5)))
.unwrap();
client.write_all(&[5, 1, 0]).unwrap();
assert_eq!(read_exactly(&mut client, 2), [5, 0]);
client.write_all(&request("evil.test", 443)).unwrap();
assert_eq!(read_exactly(&mut client, 10)[1], NOT_ALLOWED);
child.kill().unwrap();
child.wait().unwrap();
}
#[test]
fn the_program_refuses_bad_arguments() {
let dir = TempDir::new("egress-args");
let path = dir.path().join("egress.sock");
let sock = path.to_str().unwrap();
std::fs::write(dir.path().join("taken"), "").unwrap();
let taken = dir.path().join("taken");
for args in [
vec!["egress-proxy", "--socket", sock, "--allow", "Example.com"],
vec!["egress-proxy", "--socket", sock, "--allow", ""],
vec![
"egress-proxy",
"--socket",
taken.to_str().unwrap(),
"--allow",
"example.com",
],
vec!["egress-proxy", "--socket", sock],
vec!["egress-proxy", "--allow", "example.com", "--socket", sock],
] {
let out = std::process::Command::new(env!("CARGO_BIN_EXE_toolkit"))
.args(&args)
.output()
.unwrap();
assert_eq!(out.status.code(), Some(2), "{args:?}");
}
}
@@ -0,0 +1,146 @@
//! `http_fetch`: the fixed `curl` arguments, and what `toolkit` makes of `curl`'s answer, against
//! fake `curl` scripts. Do not edit.
mod support;
use std::os::unix::fs::PermissionsExt;
use std::path::PathBuf;
use std::sync::{Mutex, MutexGuard};
use proto::tools::HttpFetchArgs;
use support::TempDir;
use toolkit::Exit;
use toolkit::fetch::{curl_args, fetch_with};
const URL: &str = "https://example.com/a?b=c";
/// Every test that starts a process takes its turn. Otherwise another test's fork can hold
/// the script open for writing at the moment it is run, and running it fails with "text file
/// busy" (ETXTBSY), which has nothing to do with the code under test.
static SERIAL: Mutex<()> = Mutex::new(());
fn serial() -> MutexGuard<'static, ()> {
SERIAL.lock().unwrap_or_else(|p| p.into_inner())
}
fn fake_curl(dir: &TempDir, body: &str) -> PathBuf {
let path = dir.path().join("curl");
std::fs::write(&path, format!("#!/bin/sh\n{body}\n")).unwrap();
std::fs::set_permissions(&path, std::fs::Permissions::from_mode(0o755)).unwrap();
path
}
fn args() -> HttpFetchArgs {
HttpFetchArgs {
url: URL.to_string(),
}
}
#[test]
fn the_argument_list_is_fixed_and_ends_with_the_url() {
let expected: Vec<&str> = vec![
"--silent",
"--show-error",
"--proto",
"=https",
"--proto-redir",
"=https",
"--location",
"--max-redirs",
"5",
"--max-time",
"50",
"--max-filesize",
"8388608",
"--cacert",
"/etc/ssl/certs/ca-certificates.crt",
"--proxy",
"socks5h://localhost/run/egress/egress.sock",
"--write-out",
"\n[http %{response_code}]",
"--url",
URL,
];
assert_eq!(curl_args(URL), expected);
}
#[test]
fn curl_is_given_exactly_those_arguments() {
let _serial = serial();
let dir = TempDir::new("fetch-args");
let curl = fake_curl(&dir, r#"for a in "$@"; do printf '%s|' "$a"; done"#);
let got = fetch_with(&curl, &args());
assert_eq!(got.exit, Exit::Done);
let expected: String = curl_args(URL).iter().map(|a| format!("{a}|")).collect();
assert_eq!(got.stdout, expected);
}
#[test]
fn a_successful_fetch_is_the_body_and_the_status_line() {
let _serial = serial();
let dir = TempDir::new("fetch-ok");
let curl = fake_curl(&dir, r#"printf 'hello\n\n[http 404]'"#);
let got = fetch_with(&curl, &args());
assert_eq!(
(got.exit, got.stdout.as_str()),
(Exit::Done, "hello\n\n[http 404]")
);
}
#[test]
fn a_failed_fetch_is_exit_1_with_curls_first_error_line() {
let _serial = serial();
let dir = TempDir::new("fetch-fail");
let curl = fake_curl(
&dir,
"printf 'partial'; printf '\\ncurl: (97) cannot complete SOCKS5 connection to evil.test. (2)\\nmore\\n' >&2; exit 97",
);
let got = fetch_with(&curl, &args());
assert_eq!(got.exit, Exit::ToolError);
assert_eq!(
got.stdout,
format!(
"http_fetch: {URL}: curl: (97) cannot complete SOCKS5 connection to evil.test. (2)"
)
);
}
#[test]
fn a_failure_without_a_message_names_the_exit() {
let _serial = serial();
let dir = TempDir::new("fetch-quiet");
let curl = fake_curl(&dir, "exit 28");
let got = fetch_with(&curl, &args());
assert_eq!(
(got.exit, got.stdout.as_str()),
(
Exit::ToolError,
"http_fetch: https://example.com/a?b=c: curl exited 28"
)
);
}
#[test]
fn a_curl_that_cannot_start_is_exit_1() {
let _serial = serial();
let got = fetch_with(&PathBuf::from("/no/such/curl"), &args());
assert_eq!(got.exit, Exit::ToolError);
assert!(
got.stdout
.starts_with("http_fetch: cannot start /no/such/curl: "),
"{}",
got.stdout
);
}
#[test]
fn much_output_on_both_streams_does_not_stop_curl() {
let _serial = serial();
let dir = TempDir::new("fetch-both");
let curl = fake_curl(
&dir,
"head -c 300000 /dev/zero | tr '\\0' e >&2; head -c 300000 /dev/zero | tr '\\0' o; exit 0",
);
let got = fetch_with(&curl, &args());
assert_eq!((got.exit, got.stdout.len()), (Exit::Done, 300000));
}
@@ -0,0 +1,159 @@
//! `toolkit read_file` and `toolkit write_file`, run as `brokerd` runs them. Do not edit.
mod support;
use std::os::unix::fs::PermissionsExt;
use proto::tools::{ReadFileArgs, WriteFileArgs};
use support::{TempDir, json, toolkit};
use toolkit::files::MAX_READ;
fn read(path: &str) -> support::Ran {
toolkit(
&["read_file"],
&json(&ReadFileArgs {
path: path.to_string(),
}),
)
}
fn write(path: &str, content: &str) -> support::Ran {
let args = WriteFileArgs {
path: path.to_string(),
content: content.to_string(),
};
toolkit(&["write_file"], &json(&args))
}
#[test]
fn a_file_is_read_exactly() {
let dir = TempDir::new("read");
let text = "line one\nline two, no newline at the end: ✓";
std::fs::write(dir.at("a.md"), text).unwrap();
let ran = read(&dir.at("a.md"));
assert_eq!((ran.code, ran.stdout.as_str()), (0, text));
assert_eq!(ran.stderr, "");
}
#[test]
fn an_empty_file_is_empty_output() {
let dir = TempDir::new("read-empty");
std::fs::write(dir.at("e"), "").unwrap();
let ran = read(&dir.at("e"));
assert_eq!((ran.code, ran.stdout.as_str()), (0, ""));
}
#[test]
fn what_cannot_be_read_is_exit_1_with_one_line_for_the_model() {
let dir = TempDir::new("read-bad");
std::fs::write(dir.at("bin"), [0xff, 0xfe, 0x00]).unwrap();
std::fs::write(dir.at("big"), vec![b'a'; MAX_READ + 1]).unwrap();
std::fs::write(dir.at("exact"), vec![b'a'; MAX_READ]).unwrap();
let cases = [
(dir.at("missing"), "no such file"),
(dir.at(""), "is a directory"),
(dir.at("bin"), "not UTF-8 text"),
(dir.at("big"), "larger than 1048576 bytes"),
];
for (path, why) in cases {
let ran = read(&path);
assert_eq!(ran.code, 1, "{path}");
assert_eq!(ran.stdout, format!("read_file: {path}: {why}"));
}
let ran = read(&dir.at("exact"));
assert_eq!(
(ran.code, ran.stdout.len()),
(0, MAX_READ),
"exactly the limit is fine"
);
}
#[test]
fn an_unreadable_file_names_the_error() {
if is_root() {
return; // root reads anything
}
let dir = TempDir::new("read-perm");
std::fs::write(dir.at("secret"), "x").unwrap();
std::fs::set_permissions(dir.at("secret"), std::fs::Permissions::from_mode(0o000)).unwrap();
let ran = read(&dir.at("secret"));
assert_eq!(ran.code, 1);
assert!(
ran.stdout
.starts_with(&format!("read_file: {}: ", dir.at("secret"))),
"{}",
ran.stdout
);
assert!(ran.stdout.contains("ermission denied"), "{}", ran.stdout);
}
#[test]
fn a_file_is_written_created_or_replaced() {
let dir = TempDir::new("write");
let ran = write(&dir.at("new.md"), "hello ✓\n");
assert_eq!(ran.code, 0);
assert_eq!(
ran.stdout,
format!("wrote 10 bytes to {}", dir.at("new.md"))
);
assert_eq!(
std::fs::read_to_string(dir.at("new.md")).unwrap(),
"hello ✓\n"
);
let ran = write(&dir.at("new.md"), "");
assert_eq!(ran.stdout, format!("wrote 0 bytes to {}", dir.at("new.md")));
assert_eq!(std::fs::read_to_string(dir.at("new.md")).unwrap(), "");
}
#[test]
fn what_cannot_be_written_is_exit_1() {
let dir = TempDir::new("write-bad");
std::fs::create_dir(dir.at("sub")).unwrap();
let cases = [
(dir.at("nope/a.md"), "the directory does not exist"),
(dir.at("sub"), "is a directory"),
];
for (path, why) in cases {
let ran = write(&path, "x");
assert_eq!(ran.code, 1, "{path}");
assert_eq!(ran.stdout, format!("write_file: {path}: {why}"));
}
assert!(!dir.path().join("nope").exists(), "no directory is created");
}
#[test]
fn misuse_is_exit_2_with_nothing_on_standard_output() {
let long = vec![b' '; toolkit::input::MAX_INPUT + 1];
let cases: [(&[&str], &[u8], &str); 7] = [
(
&["read_file"],
br#"{"path":"/a","mode":1}"#,
"the arguments do not parse",
),
(&["read_file"], b"", "the arguments do not parse"),
(
&["write_file"],
br#"{"path":"/a"}"#,
"the arguments do not parse",
),
(&["read_file"], &[0xff, 0xfe], "not UTF-8"),
(&["read_file"], &long, "larger than 2097152 bytes"),
(&["format_disk"], b"{}", "unknown tool"),
(&[], b"{}", "unknown tool"),
];
for (args, input, why) in cases {
let ran = toolkit(args, input);
assert_eq!(ran.code, 2, "{args:?}");
assert_eq!(
ran.stdout, "",
"{args:?}: the model sees nothing of a misuse"
);
assert!(ran.stderr.contains(why), "{args:?}: {}", ran.stderr);
}
}
fn is_root() -> bool {
std::fs::read_to_string("/proc/self/status")
.map(|s| s.lines().any(|l| l.starts_with("Uid:\t0\t")))
.unwrap_or(false)
}
@@ -0,0 +1,89 @@
//! `toolkit shell`, run as `brokerd` runs it, with the host's `/bin/sh`. Do not edit.
mod support;
use proto::tools::ShellArgs;
use support::{TempDir, json, toolkit};
use toolkit::shell::MAX_OUTPUT;
fn sh(command: &str, cwd: Option<&str>) -> support::Ran {
let args = ShellArgs {
command: command.to_string(),
cwd: cwd.map(str::to_string),
};
toolkit(&["shell"], &json(&args))
}
#[test]
fn output_and_errors_come_back_in_order_then_the_exit_status() {
let ran = sh("echo one; echo two >&2; echo three", None);
assert_eq!(ran.code, 0);
assert_eq!(ran.stdout, "one\ntwo\nthree\n\n[exit 0]");
}
#[test]
fn a_failing_command_is_still_a_result_with_its_status() {
let ran = sh("echo nope; exit 3", None);
assert_eq!((ran.code, ran.stdout.as_str()), (0, "nope\n\n[exit 3]"));
let ran = sh("true", None);
assert_eq!(ran.stdout, "\n[exit 0]");
}
#[test]
fn a_command_killed_by_a_signal_says_so() {
let ran = sh("kill -9 $$", None);
assert_eq!(
(ran.code, ran.stdout.as_str()),
(0, "\n[killed by signal 9]")
);
}
#[test]
fn the_command_runs_in_cwd_or_in_tmp() {
let dir = TempDir::new("cwd");
let here = dir.path().to_str().unwrap();
let ran = sh("pwd", Some(here));
assert_eq!(ran.stdout, format!("{here}\n\n[exit 0]"));
let ran = sh("pwd", None);
assert_eq!(ran.stdout, "/tmp\n\n[exit 0]");
}
#[test]
fn a_missing_cwd_is_exit_1() {
let ran = sh("pwd", Some("/no/such/dir"));
assert_eq!(
(ran.code, ran.stdout.as_str()),
(1, "shell: /no/such/dir: no such directory")
);
}
#[test]
fn standard_input_is_empty() {
let ran = sh("cat; echo done", None);
assert_eq!(ran.stdout, "done\n\n[exit 0]");
}
#[test]
fn output_past_the_limit_is_dropped_and_the_command_still_finishes() {
let ran = sh(
&format!(
"head -c {} /dev/zero | tr '\\0' a; echo; echo end >&2; exit 4",
MAX_OUTPUT + 5000
),
None,
);
assert_eq!(ran.code, 0);
let expected_tail = format!("\n[output after {MAX_OUTPUT} bytes dropped]\n[exit 4]");
assert!(
ran.stdout.ends_with(&expected_tail),
"{}",
&ran.stdout[ran.stdout.len() - 80..]
);
assert_eq!(ran.stdout.len(), MAX_OUTPUT + expected_tail.len());
}
#[test]
fn output_that_is_not_utf8_is_replaced_not_refused() {
let ran = sh("printf 'a\\377b'", None);
assert_eq!(ran.stdout, "a\u{fffd}b\n[exit 0]");
}
@@ -0,0 +1,68 @@
//! Running the `toolkit` binary as `brokerd` does: arguments on standard input. Do not edit.
#![allow(dead_code)] // each test file uses its own part
use std::io::Write;
use std::path::{Path, PathBuf};
use std::process::{Command, Stdio};
use std::sync::atomic::{AtomicU32, Ordering};
static NEXT: AtomicU32 = AtomicU32::new(0);
/// A temporary directory, removed when dropped.
pub struct TempDir(pub PathBuf);
impl TempDir {
pub fn new(tag: &str) -> TempDir {
let n = NEXT.fetch_add(1, Ordering::SeqCst);
let path = std::env::temp_dir().join(format!("tk-{tag}-{}-{n}", std::process::id()));
let _ = std::fs::remove_dir_all(&path);
std::fs::create_dir_all(&path).unwrap();
TempDir(path)
}
pub fn path(&self) -> &Path {
&self.0
}
/// The path of `name` inside, as a string (the tools take strings).
pub fn at(&self, name: &str) -> String {
self.0.join(name).to_str().unwrap().to_string()
}
}
impl Drop for TempDir {
fn drop(&mut self) {
let _ = std::fs::remove_dir_all(&self.0);
}
}
pub struct Ran {
pub code: i32,
pub stdout: String,
pub stderr: String,
}
/// Run `toolkit <args…>` with `input` on standard input.
pub fn toolkit(args: &[&str], input: &[u8]) -> Ran {
let mut child = Command::new(env!("CARGO_BIN_EXE_toolkit"))
.args(args)
.stdin(Stdio::piped())
.stdout(Stdio::piped())
.stderr(Stdio::piped())
.spawn()
.unwrap();
let mut stdin = child.stdin.take().unwrap();
// A broken pipe here only means toolkit stopped reading, which some tests expect.
let _ = stdin.write_all(input);
drop(stdin);
let out = child.wait_with_output().unwrap();
Ran {
code: out.status.code().unwrap_or(-1),
stdout: String::from_utf8(out.stdout).unwrap(),
stderr: String::from_utf8_lossy(&out.stderr).into_owned(),
}
}
/// The JSON for one argument struct.
pub fn json<T: serde::Serialize>(value: &T) -> Vec<u8> {
serde_json::to_vec(value).unwrap()
}
+26 -1
View File
@@ -33,7 +33,7 @@ bxctl grants check
It prints each problem as `<file>:<line>: <problem>`. Common ones: a misspelled field (unknown It prints each problem as `<file>:<line>: <problem>`. Common ones: a misspelled field (unknown
fields are errors); `secret` set (not supported until M4); `patterns` set (not supported); `hosts` fields are errors); `secret` set (not supported until M4); `patterns` set (not supported); `hosts`
on a file tool or `paths` on `http_fetch`; a relative path or one with `..`, `.` or `//`; the path on a file tool or `paths` on `http_fetch`; a relative path or one with `..`, `.` or `//`; a path with `:` or `,` (it cannot be mounted); the path
`/` (grant the directories you mean); a `deny` grant whose `max_taint` is not `"secret"` (it would `/` (grant the directories you mean); a `deny` grant whose `max_taint` is not `"secret"` (it would
stop denying once the session read a secret); a host with uppercase letters, a port, or an IP stop denying once the session read a secret); a host with uppercase letters, a port, or an IP
address; a file name with characters outside `[a-z0-9-]`. address; a file name with characters outside `[a-z0-9-]`.
@@ -229,6 +229,31 @@ the limit. Then start `brokerd` again if it exited.
**Check.** `brokerd` prints `serving tools on …`, and a tool call is decided again. **Check.** `brokerd` prints `serving tools on …`, and a tool call is decided again.
## runner-unavailable
**What you see.** A tool call fails with "the tool runner could not start the container", and
`brokerd` prints what Podman said, then this entry.
**Why.** `brokerd` asked Podman to start the tool's container, or for `http_fetch` the egress
proxy's, and Podman could not: the program is missing, the image in `[runner] image` is not
loaded, the proxy's network does not exist, or the proxy did not create its socket in time.
Nothing ran for the call.
**Confirm.**
```sh
podman --version # the program [runner] podman names
podman image inspect --format '{{.Digest}}' <image> # must be the digest in [runner] image
podman ps -a --filter label=boxmaker # containers left behind, if any
```
**Fix.** Install Podman or correct `[runner] podman`. Load the image built from
`deploy/tools-image.nix` (`podman load < result`) and put its digest in `[runner] image`. For the
proxy, check that `[runner] egress_network` names a network Podman has (`pasta` needs the `passt`
package). Remove leftovers with `podman rm -f` on the listed names.
**Check.** Ask for a `read_file` the grants allow; it returns the file.
## broker-state-damaged ## broker-state-damaged
**What you see.** `brokerd` prints an error reading or writing **What you see.** `brokerd` prints an error reading or writing
+241
View File
@@ -0,0 +1,241 @@
# M3b design: the runner and the tools
Status: draft, 2026-09-22, written for an overnight implementation run; the owner reviews it the
next morning. The golden files of plan task 10 (`docs/plans/M3b/files/crates/brokerd/tests/fixtures/podman/`)
are the exact argument lists. The shape was agreed in the M3 design discussion (the split, rootless Podman, an
image built by Nix and named by digest, `curl` behind a SOCKS5 proxy of our own). The details
below that are new are marked **(new)** and listed in section 10 for review. Where this document
and `docs/design.md` disagree, the brief wins. M3a's spec (`2026-09-18-m3a-decision-path.md`)
still holds for everything before a `RunSpec` exists.
## 1. What M3b proves
| Claim | Checked by |
|---|---|
| Each allowed call runs in a fresh container with no network, a read-only root, only the grant's directories mounted, and limits on time, memory, processes and output | Golden `podman` argument lists; the runtime against a fake `podman`; on straylight |
| A container with no network grant cannot reach the network | On straylight: the tailnet, the host and the internet are all unreachable from `shell` |
| `http_fetch` reaches only the grant's hosts, redirects included, and never an address inside the tailnet, the host or a private network | Proxy tests against a fake resolver; on straylight |
| A tool's output reaches the model labelled by its grant, capped, and a runner failure reaches it as a fixed sentence | Runtime tests |
| No container outlives its call | Runtime tests with a fake `podman` (the kill and remove calls); on straylight (`podman ps` by label) |
Out of scope: secrets (M4), anything but the four tools, a registry (the image is loaded locally),
the NixOS module and service users (M7).
## 2. Shared definitions move to `proto` (new)
Two programs now read the same things, so each has one definition (AGENTS: "when two programs
must agree about a set of things, one of them deciding alone is a bug"):
- `proto::tools`: the argument structs of the four tools, as `brokerd` parses them today:
`ReadFileArgs { path }`, `WriteFileArgs { path, content }`, `ShellArgs { command, cwd: Option }`,
`HttpFetchArgs { url }`, all `deny_unknown_fields`, `cwd` omitted when `None`. `brokerd` parses
the model's arguments with them and writes them to the container's standard input; `toolkit`
reads them back.
- `proto::hosts`: `valid_host`, `valid_host_pattern`, `host_matches`, moved from `brokerd::args`
unchanged. `brokerd` checks grants and URLs with them; the egress proxy checks each connection.
`brokerd::args` re-exports the moved functions, so its callers and tests do not change.
## 3. Sealed fetch target (M3a finding 14)
`ToolArgs::HttpFetch { url, host }` becomes `ToolArgs::HttpFetch(FetchUrl)`. `FetchUrl` has
private fields, `url()` and `host()` getters, no public constructor, and is built only by
`args::parse` after the URL checks. Policy matches `host()`; the runner hands `url()` to the tool.
They cannot disagree.
Grant paths gain one rule **(new)**: a path in `paths` may not contain `:` or `,`, because it is
mounted with `podman --volume=<path>:<path>:ro` and those characters are separators there. The
problem text is "cannot be mounted: it contains ':' or ','".
## 4. `toolkit`: the programs inside the container
One static binary, `/bin/toolkit`, with one subcommand per tool and one for the proxy:
```
toolkit read_file | write_file | shell | http_fetch arguments as JSON on standard input
toolkit egress-proxy --socket <path> --allow <pattern>[,<pattern>…]
```
For the four tools, `toolkit` reads all of standard input (at most 2 MiB), parses it strictly as
the tool's `proto::tools` struct, does the work, writes what the model should see to standard
output, and exits:
| Exit | Meaning | `brokerd` answers |
|---|---|---|
| 0 | The tool did its work | `Result`, content = standard output |
| 1 | The tool could not do it, for a reason the model should read (no such file, not text) | `Result`, content = standard output, which is one line starting with the tool's name |
| 2 | `toolkit` was run wrongly: unknown subcommand, input that does not parse | `Failed`, "the tool could not run" |
- **`read_file`**: reads the file. More than 1 MiB, or not UTF-8, or a directory, is exit 1 with
`read_file: <path>: <reason>`. Otherwise the content, unchanged.
- **`write_file`**: the parent directory must exist; the file is created or truncated and written.
A directory at the path is exit 1. Output: `wrote <n> bytes to <path>`.
- **`shell`**: runs `/bin/sh -c <command>` in `cwd` (or `/tmp` without one), standard input empty,
standard output and standard error on one pipe in the order written, and then prints
`\n[exit <code>]`, or `\n[killed by signal <n>]`. `toolkit` exits 0 whatever the command did: the
tool ran. A `cwd` that does not exist is exit 1.
- **`http_fetch`**: runs `/bin/curl` with a fixed argument list (section 6) and the URL, prints
the body and then `\n[http <status>]`. A `curl` failure is exit 1 with
`http_fetch: <url>: <curl's first line of error>`.
`toolkit` never decides what may run: that was decided before the container existed. It is also
not trusted by `brokerd`: whatever it prints is labelled by the grant, and a bad exit is a fixed
sentence.
## 5. The egress proxy
`toolkit egress-proxy` listens on a Unix socket and speaks the part of SOCKS5 (RFC 1928) that
`curl --proxy socks5h://` uses: no authentication, `CONNECT`, a domain name. Each connection:
1. Greeting: version 5, then the method list. It must offer method 0 (none); the answer is
`05 00`, otherwise `05 FF` and close.
2. Request: version 5, command 1 (`CONNECT`), reserved 0, address type 3 (domain), a length of 1
to 255, the name, a port. Another command is reply 7; another address type (an IP literal) is
reply 8.
3. The name must be a valid host name (`proto::hosts::valid_host`), match one of the `--allow`
patterns (`host_matches`), and the port must be 443. Otherwise reply 2 ("not allowed").
4. Resolve the name. Keep only **public** addresses (below). None left, or resolution failed:
reply 4 ("host unreachable"). A name that resolves into the tailnet or the host is refused
here even though the owner allowed the name: the grant allows a host on the internet.
5. Connect to the first public address, 10 s timeout. Failure: reply 5. Success: reply
`05 00 00 01 00 00 00 00 00 00`, then copy bytes both ways until either side closes, passing a
half-close on.
Every reply other than success is followed by closing the connection. The proxy reads at most 262
bytes of greeting and request, with a 10 s timeout for the whole handshake, and serves at most 8
connections at once (more are closed at once). It never parses what flows after the handshake:
TLS runs end to end between `curl` and the server.
**Public** means not any of: IPv4 `0.0.0.0/8`, `10/8`, `100.64/10` (the tailnet's range),
`127/8`, `169.254/16`, `172.16/12`, `192.0.0/24`, `192.0.2/24`, `192.168/16`, `198.18/15`,
`198.51.100/24`, `203.0.113/24`, `224/4`, `240/4`; IPv6 `::/96` (which holds `::`, `::1` and
the old IPv4-compatible form), `fc00::/7`, `fe80::/10`,
`ff00::/8`, `2001:db8::/32`, and any IPv4-mapped (`::ffff:0:0/96`) or NAT64 (`64:ff9b::/96`)
address whose IPv4 part is not public. `toolkit::addr::is_public(IpAddr) -> bool`.
For tests, name resolution and connecting are behind a trait, so no test needs a network:
```rust
pub trait Dial: Send + Sync {
fn resolve(&self, host: &str, port: u16) -> std::io::Result<Vec<SocketAddr>>;
fn connect(&self, addr: SocketAddr, timeout: Duration) -> std::io::Result<TcpStream>;
}
```
## 6. `brokerd`: the Podman runtime
### Configuration (new)
```toml
[runner] # absent: the M3a runtime, which refuses every call
podman = "podman" # the program; a bare name is looked up in PATH
image = "localhost/boxmaker-tools@sha256:…" # required; by digest
egress_network = "pasta" # the proxy container's network
output_cap = 262144 # bytes of standard output kept; more is truncated
memory = "512m"
pids = 128
read_file_ms = 30000
write_file_ms = 30000
shell_ms = 100000
http_fetch_ms = 60000
```
Every time limit must stay under `loopd`'s `[broker] timeout_ms` (default 120,000), because `loopd`
stops waiting then. To give `shell` longer, raise both.
### One call
The container is named `boxmaker-<session>-<call>-<n>`, where `n` counts calls within this
`brokerd` process, and labelled `boxmaker=tool`. Arguments, in this order, as separate `OsString`s
(never a shell string):
```
run --rm -i --name=<name> --label=boxmaker=tool --network=none --read-only --cap-drop=all
--security-opt=no-new-privileges --userns=keep-id --pids-limit=<pids> --memory=<memory>
--tmpfs=/tmp:rw,size=64m,mode=1777
[--volume=<path>:<path>:ro | :rw for each mount, in RunSpec order]
[--volume=<egress dir>:/run/egress:rw http_fetch only]
<image> /bin/toolkit <tool>
```
`brokerd` writes the tool's `proto::tools` JSON to standard input and closes it, reads standard
output up to `output_cap` bytes (one byte more means `truncated`, and the rest is not read), and
keeps the first 4 KiB of standard error for its own log only. It waits for the tool's time limit,
polling. Then:
| What happened | Answer |
|---|---|
| exit 0 or 1 | `Ok(RunOutput { content, truncated })`, content decoded as UTF-8 with replacement |
| exit 2 | `Failed("the tool could not run")` |
| exit 125, 126 or 127 (`podman` itself failed: no image, bad option) | `Unavailable("the tool runner could not start the container")`, and `brokerd` prints standard error and `see docs/runbook.md#runner-unavailable` |
| `podman` cannot be started at all | the same |
| exit 137 (killed: out of memory, or the limit below) | `Failed("the tool was stopped: it ran out of memory or was killed")` |
| past the time limit | `podman kill <name>`, then `podman rm -f <name>`, then `Failed("the tool ran past its time limit")` |
| any other exit | `Failed("the tool failed with an unexpected status")` |
Every text above is fixed: tool output never reaches the model through a `RunError`.
### `http_fetch`
Before the tool container, `brokerd` makes `<home>/run/egress/<name>/` (mode 0700) and starts the
proxy:
```
run -d --rm --name=<name>-egress --label=boxmaker=egress --network=<egress_network> --read-only
--cap-drop=all --security-opt=no-new-privileges --userns=keep-id --pids-limit=64
--memory=128m --volume=<egress dir>:/run/egress:rw
<image> /bin/toolkit egress-proxy --socket /run/egress/egress.sock --allow <hosts joined by ,>
```
It waits up to 5 s for `<egress dir>/egress.sock` to exist, runs the tool container with the
directory mounted, and afterwards, **whatever happened**, runs `podman rm -f <name>-egress` and
removes the directory. A proxy that does not start in time is `Unavailable` with the
`runner-unavailable` pointer. The tool container itself still has `--network=none`: its only way
out is the socket.
`curl`'s arguments inside the tool container are fixed:
```
/bin/curl --silent --show-error --proto =https --proto-redir =https --location --max-redirs 5
--max-time 50 --max-filesize 8388608 --cacert /etc/ssl/certs/ca-certificates.crt
--proxy socks5h://localhost/run/egress/egress.sock --write-out "\n[http %{response_code}]"
--url <url>
```
## 7. The image (new detail; P13)
Built by Nix on straylight (`deploy/tools-image.nix`, `dockerTools.buildLayeredImage`), loaded with
`podman load`, and named in `[runner] image` by digest. It holds `/bin/toolkit` (static musl),
`/bin/busybox` with `/bin/sh` linking to it, `/bin/curl` (static), and
`/etc/ssl/certs/ca-certificates.crt`. The design model builds and checks it on straylight; it is not
an implementer task (this machine has neither Nix nor Podman).
## 8. Runbook
A new entry, `runner-unavailable`: `podman` missing, the image not loaded or not the digest in the
config, or the proxy not starting. Every such message ends with its pointer.
## 9. Testing
- **Offline (the implementer's tasks)**: `proto::tools` and `proto::hosts` round trips; `FetchUrl`
sealed (a `compile_fail` doctest); the grant path rule; `toolkit` run as a program against temporary
directories (read, write, shell, the exit codes); `http_fetch`'s argument list; `is_public` as a
table; the proxy over `UnixStream::pair()` against a fake `Dial` (every reply code, the host and
address checks, the byte copy both ways, the handshake limits); `brokerd`'s argument lists as
golden files; the runtime against fake `podman` scripts that record their arguments (exit codes,
output cap, time limit with its kill and remove, standard input, the egress lifecycle and its
clean-up on every path).
- **On straylight (the design model)**: build and load the image; run each tool through
`brokerd`; a `shell` call cannot reach `100.100.100.100`, the host or the internet;
`http_fetch` reaches an allowed host, is refused for another, for a redirect to another, and for
an allowed name that resolves into the tailnet; limits hold; no container with a `boxmaker`
label remains.
## 10. For the owner's review
1. Tool time limits sit under `loopd`'s 120 s broker timeout, so `shell` defaults to 100 s, not the
10 minutes discussed. Longer shells mean raising both values.
2. The proxy refuses any allowed name that resolves to a non-public address.
3. Grant paths may not contain `:` or `,`.
4. `shell` has busybox and nothing else in the image: no compilers, no `git`.
5. `toolkit`'s exit 1 output is shown to the model as a result, labelled like any other.