# M3b design: the runner and the tools Status: draft, 2026-09-22, written for an overnight implementation run; the owner reviews it the next morning. The golden files of plan task 10 (`docs/plans/M3b/files/crates/brokerd/tests/fixtures/podman/`) are the exact argument lists. The shape was agreed in the M3 design discussion (the split, rootless Podman, an image built by Nix and named by digest, `curl` behind a SOCKS5 proxy of our own). The details below that are new are marked **(new)** and listed in section 10 for review. Where this document and `docs/design.md` disagree, the brief wins. M3a's spec (`2026-09-18-m3a-decision-path.md`) still holds for everything before a `RunSpec` exists. ## 1. What M3b proves | Claim | Checked by | |---|---| | Each allowed call runs in a fresh container with no network, a read-only root, only the grant's directories mounted, and limits on time, memory, processes and output | Golden `podman` argument lists; the runtime against a fake `podman`; on straylight | | A container with no network grant cannot reach the network | On straylight: the tailnet, the host and the internet are all unreachable from `shell` | | `http_fetch` reaches only the grant's hosts, redirects included, and never an address inside the tailnet, the host or a private network | Proxy tests against a fake resolver; on straylight | | A tool's output reaches the model labelled by its grant, capped, and a runner failure reaches it as a fixed sentence | Runtime tests | | No container outlives its call | Runtime tests with a fake `podman` (the kill and remove calls); on straylight (`podman ps` by label) | Out of scope: secrets (M4), anything but the four tools, a registry (the image is loaded locally), the NixOS module and service users (M7). ## 2. Shared definitions move to `proto` (new) Two programs now read the same things, so each has one definition (AGENTS: "when two programs must agree about a set of things, one of them deciding alone is a bug"): - `proto::tools`: the argument structs of the four tools, as `brokerd` parses them today: `ReadFileArgs { path }`, `WriteFileArgs { path, content }`, `ShellArgs { command, cwd: Option }`, `HttpFetchArgs { url }`, all `deny_unknown_fields`, `cwd` omitted when `None`. `brokerd` parses the model's arguments with them and writes them to the container's standard input; `toolkit` reads them back. - `proto::hosts`: `valid_host`, `valid_host_pattern`, `host_matches`, moved from `brokerd::args` unchanged. `brokerd` checks grants and URLs with them; the egress proxy checks each connection. `brokerd::args` re-exports the moved functions, so its callers and tests do not change. ## 3. Sealed fetch target (M3a finding 14) `ToolArgs::HttpFetch { url, host }` becomes `ToolArgs::HttpFetch(FetchUrl)`. `FetchUrl` has private fields, `url()` and `host()` getters, no public constructor, and is built only by `args::parse` after the URL checks. Policy matches `host()`; the runner hands `url()` to the tool. They cannot disagree. Grant paths gain one rule **(new)**: a path in `paths` may not contain `:` or `,`, because it is mounted with `podman --volume=::ro` and those characters are separators there. The problem text is "cannot be mounted: it contains ':' or ','". ## 4. `toolkit`: the programs inside the container One static binary, `/bin/toolkit`, with one subcommand per tool and one for the proxy: ``` toolkit read_file | write_file | shell | http_fetch arguments as JSON on standard input toolkit egress-proxy --socket --allow [,…] ``` For the four tools, `toolkit` reads all of standard input (at most 2 MiB), parses it strictly as the tool's `proto::tools` struct, does the work, writes what the model should see to standard output, and exits: | Exit | Meaning | `brokerd` answers | |---|---|---| | 0 | The tool did its work | `Result`, content = standard output | | 1 | The tool could not do it, for a reason the model should read (no such file, not text) | `Result`, content = standard output, which is one line starting with the tool's name | | 2 | `toolkit` was run wrongly: unknown subcommand, input that does not parse | `Failed`, "the tool could not run" | - **`read_file`**: reads the file. More than 1 MiB, or not UTF-8, or a directory, is exit 1 with `read_file: : `. Otherwise the content, unchanged. - **`write_file`**: the parent directory must exist; the file is created or truncated and written. A directory at the path is exit 1. Output: `wrote bytes to `. - **`shell`**: runs `/bin/sh -c ` in `cwd` (or `/tmp` without one), standard input empty, standard output and standard error on one pipe in the order written, and then prints `\n[exit ]`, or `\n[killed by signal ]`. `toolkit` exits 0 whatever the command did: the tool ran. A `cwd` that does not exist is exit 1. - **`http_fetch`**: runs `/bin/curl` with a fixed argument list (section 6) and the URL, prints the body and then `\n[http ]`. A `curl` failure is exit 1 with `http_fetch: : `. `toolkit` never decides what may run: that was decided before the container existed. It is also not trusted by `brokerd`: whatever it prints is labelled by the grant, and a bad exit is a fixed sentence. ## 5. The egress proxy `toolkit egress-proxy` listens on a Unix socket and speaks the part of SOCKS5 (RFC 1928) that `curl --proxy socks5h://` uses: no authentication, `CONNECT`, a domain name. Each connection: 1. Greeting: version 5, then the method list. It must offer method 0 (none); the answer is `05 00`, otherwise `05 FF` and close. 2. Request: version 5, command 1 (`CONNECT`), reserved 0, address type 3 (domain), a length of 1 to 255, the name, a port. Another command is reply 7; another address type (an IP literal) is reply 8. 3. The name must be a valid host name (`proto::hosts::valid_host`), match one of the `--allow` patterns (`host_matches`), and the port must be 443. Otherwise reply 2 ("not allowed"). 4. Resolve the name. Keep only **public** addresses (below). None left, or resolution failed: reply 4 ("host unreachable"). A name that resolves into the tailnet or the host is refused here even though the owner allowed the name: the grant allows a host on the internet. 5. Connect to the first public address, 10 s timeout. Failure: reply 5. Success: reply `05 00 00 01 00 00 00 00 00 00`, then copy bytes both ways until either side closes, passing a half-close on. Every reply other than success is followed by closing the connection. The proxy reads at most 262 bytes of greeting and request, with a 10 s timeout for the whole handshake, and serves at most 8 connections at once (more are closed at once). It never parses what flows after the handshake: TLS runs end to end between `curl` and the server. **Public** means not any of: IPv4 `0.0.0.0/8`, `10/8`, `100.64/10` (the tailnet's range), `127/8`, `169.254/16`, `172.16/12`, `192.0.0/24`, `192.0.2/24`, `192.168/16`, `198.18/15`, `198.51.100/24`, `203.0.113/24`, `224/4`, `240/4`; IPv6 `::/96` (which holds `::`, `::1` and the old IPv4-compatible form), `fc00::/7`, `fe80::/10`, `ff00::/8`, `2001:db8::/32`, and any IPv4-mapped (`::ffff:0:0/96`) or NAT64 (`64:ff9b::/96`) address whose IPv4 part is not public. `toolkit::addr::is_public(IpAddr) -> bool`. For tests, name resolution and connecting are behind a trait, so no test needs a network: ```rust pub trait Dial: Send + Sync { fn resolve(&self, host: &str, port: u16) -> std::io::Result>; fn connect(&self, addr: SocketAddr, timeout: Duration) -> std::io::Result; } ``` ## 6. `brokerd`: the Podman runtime ### Configuration (new) ```toml [runner] # absent: the M3a runtime, which refuses every call podman = "podman" # the program; a bare name is looked up in PATH image = "localhost/boxmaker-tools@sha256:…" # required; by digest egress_network = "pasta" # the proxy container's network output_cap = 262144 # bytes of standard output kept; more is truncated memory = "512m" pids = 128 read_file_ms = 30000 write_file_ms = 30000 shell_ms = 100000 http_fetch_ms = 60000 ``` Every time limit must stay under `loopd`'s `[broker] timeout_ms` (default 120,000), because `loopd` stops waiting then. To give `shell` longer, raise both. ### One call The container is named `boxmaker---`, where `n` counts calls within this `brokerd` process, and labelled `boxmaker=tool`. Arguments, in this order, as separate `OsString`s (never a shell string): ``` run --rm -i --name= --label=boxmaker=tool --network=none --pull=never --read-only --cap-drop=all --security-opt=no-new-privileges --userns=keep-id --pids-limit= --memory= --tmpfs=/tmp:rw,size=64m,mode=1777 [--volume=::ro | :rw for each mount, in RunSpec order] [--volume=:/run/egress:rw http_fetch only] /bin/toolkit ``` `brokerd` writes the tool's `proto::tools` JSON to standard input and closes it, reads standard output up to `output_cap` bytes (one byte more means `truncated`, and the rest is not read), and keeps the first 4 KiB of standard error for its own log only. It waits for the tool's time limit, polling. Then: | What happened | Answer | |---|---| | exit 0 or 1 | `Ok(RunOutput { content, truncated })`, content decoded as UTF-8 with replacement | | exit 2 | `Failed("the tool could not run")` | | exit 125, 126 or 127 (`podman` itself failed: no image, bad option) | `Unavailable("the tool runner could not start the container")`, and `brokerd` prints standard error and `see docs/runbook.md#runner-unavailable` | | `podman` cannot be started at all | the same | | exit 137 (killed: out of memory, or the limit below) | `Failed("the tool was stopped: it ran out of memory or was killed")` | | past the time limit | `podman kill `, then `podman rm -f `, then `Failed("the tool ran past its time limit")` | | any other exit | `Failed("the tool failed with an unexpected status")` | Every text above is fixed: tool output never reaches the model through a `RunError`. `--pull=never` (added after the M3b review): an image that is not loaded fails the call at once (exit 125) instead of Podman trying to pull it, which would be unlisted egress. ### `http_fetch` Before the tool container, `brokerd` makes `/run/egress//` (mode 0700) and starts the proxy: ``` run -d --rm --name=-egress --label=boxmaker=egress --network= --pull=never --read-only --cap-drop=all --security-opt=no-new-privileges --userns=keep-id --pids-limit=64 --memory=128m --volume=:/run/egress:rw /bin/toolkit egress-proxy --socket /run/egress/egress.sock --allow ``` It waits up to 5 s for `/egress.sock` to exist, runs the tool container with the directory mounted, and afterwards, **whatever happened**, runs `podman rm -f -egress` and removes the directory. A proxy that does not start in time is `Unavailable` with the `runner-unavailable` pointer. The tool container itself still has `--network=none`: its only way out is the socket. `curl`'s arguments inside the tool container are fixed: ``` /bin/curl --disable --silent --show-error --globoff --proto =https --proto-redir =https --location --max-redirs 5 --max-time 50 --max-filesize 8388608 --cacert /etc/ssl/certs/ca-certificates.crt --proxy socks5h://localhost/run/egress/egress.sock --write-out "\n[http %{response_code}]" --url ``` ## 7. The image (new detail; P13) Built by Nix on straylight (`deploy/tools-image.nix`, `dockerTools.buildLayeredImage`), loaded with `podman load`, and named in `[runner] image` by digest. It holds `/bin/toolkit` (static musl), `/bin/busybox` with `/bin/sh` linking to it, `/bin/curl` (static), and `/etc/ssl/certs/ca-certificates.crt`. The design model builds and checks it on straylight; it is not an implementer task (this machine has neither Nix nor Podman). ## 8. Runbook A new entry, `runner-unavailable`: `podman` missing, the image not loaded or not the digest in the config, or the proxy not starting. Every such message ends with its pointer. ## 9. Testing - **Offline (the implementer's tasks)**: `proto::tools` and `proto::hosts` round trips; `FetchUrl` sealed (a `compile_fail` doctest); the grant path rule; `toolkit` run as a program against temporary directories (read, write, shell, the exit codes); `http_fetch`'s argument list; `is_public` as a table; the proxy over `UnixStream::pair()` against a fake `Dial` (every reply code, the host and address checks, the byte copy both ways, the handshake limits); `brokerd`'s argument lists as golden files; the runtime against fake `podman` scripts that record their arguments (exit codes, output cap, time limit with its kill and remove, standard input, the egress lifecycle and its clean-up on every path). - **On straylight (the design model)**: build and load the image; run each tool through `brokerd`; a `shell` call cannot reach `100.100.100.100`, the host or the internet; `http_fetch` reaches an allowed host, is refused for another, for a redirect to another, and for an allowed name that resolves into the tailnet; limits hold; no container with a `boxmaker` label remains. ## 10. For the owner's review 1. Tool time limits sit under `loopd`'s 120 s broker timeout, so `shell` defaults to 100 s, not the 10 minutes discussed. Longer shells mean raising both values. 2. The proxy refuses any allowed name that resolves to a non-public address. 3. Grant paths may not contain `:` or `,`. 4. `shell` has busybox and nothing else in the image: no compilers, no `git`. 5. `toolkit`'s exit 1 output is shown to the model as a result, labelled like any other.