boxmaker/docs/specs/2026-09-22-m3b-runner.md

# M3b design: the runner and the tools

Status: draft, 2026-09-22, written for an overnight implementation run; the owner reviews it the
next morning. The golden files of plan task 10 (`docs/plans/M3b/files/crates/brokerd/tests/fixtures/podman/`)
are the exact argument lists. The shape was agreed in the M3 design discussion (the split, rootless Podman, an
image built by Nix and named by digest, `curl` behind a SOCKS5 proxy of our own). The details
below that are new are marked **(new)** and listed in section 10 for review. Where this document
and `docs/design.md` disagree, the brief wins. M3a's spec (`2026-09-18-m3a-decision-path.md`)
still holds for everything before a `RunSpec` exists.

## 1. What M3b proves

| Claim | Checked by |
|---|---|
| Each allowed call runs in a fresh container with no network, a read-only root, only the grant's directories mounted, and limits on time, memory, processes and output | Golden `podman` argument lists; the runtime against a fake `podman`; on straylight |
| A container with no network grant cannot reach the network | On straylight: the tailnet, the host and the internet are all unreachable from `shell` |
| `http_fetch` reaches only the grant's hosts, redirects included, and never an address inside the tailnet, the host or a private network | Proxy tests against a fake resolver; on straylight |
| A tool's output reaches the model labelled by its grant, capped, and a runner failure reaches it as a fixed sentence | Runtime tests |
| No container outlives its call | Runtime tests with a fake `podman` (the kill and remove calls); on straylight (`podman ps` by label) |

Out of scope: secrets (M4), anything but the four tools, a registry (the image is loaded locally),
the NixOS module and service users (M7).

## 2. Shared definitions move to `proto` (new)

Two programs now read the same things, so each has one definition (AGENTS: "when two programs
must agree about a set of things, one of them deciding alone is a bug"):

- `proto::tools`: the argument structs of the four tools, as `brokerd` parses them today:
  `ReadFileArgs { path }`, `WriteFileArgs { path, content }`, `ShellArgs { command, cwd: Option }`,
  `HttpFetchArgs { url }`, all `deny_unknown_fields`, `cwd` omitted when `None`. `brokerd` parses
  the model's arguments with them and writes them to the container's standard input; `toolkit`
  reads them back.
- `proto::hosts`: `valid_host`, `valid_host_pattern`, `host_matches`, moved from `brokerd::args`
  unchanged. `brokerd` checks grants and URLs with them; the egress proxy checks each connection.

`brokerd::args` re-exports the moved functions, so its callers and tests do not change.

## 3. Sealed fetch target (M3a finding 14)

`ToolArgs::HttpFetch { url, host }` becomes `ToolArgs::HttpFetch(FetchUrl)`. `FetchUrl` has
private fields, `url()` and `host()` getters, no public constructor, and is built only by
`args::parse` after the URL checks. Policy matches `host()`; the runner hands `url()` to the tool.
They cannot disagree.

Grant paths gain one rule **(new)**: a path in `paths` may not contain `:` or `,`, because it is
mounted with `podman --volume=<path>:<path>:ro` and those characters are separators there. The
problem text is "cannot be mounted: it contains ':' or ','".

## 4. `toolkit`: the programs inside the container

One static binary, `/bin/toolkit`, with one subcommand per tool and one for the proxy:

```
toolkit read_file | write_file | shell | http_fetch      arguments as JSON on standard input
toolkit egress-proxy --socket <path> --allow <pattern>[,<pattern>…]
```

For the four tools, `toolkit` reads all of standard input (at most 2 MiB), parses it strictly as
the tool's `proto::tools` struct, does the work, writes what the model should see to standard
output, and exits:

| Exit | Meaning | `brokerd` answers |
|---|---|---|
| 0 | The tool did its work | `Result`, content = standard output |
| 1 | The tool could not do it, for a reason the model should read (no such file, not text) | `Result`, content = standard output, which is one line starting with the tool's name |
| 2 | `toolkit` was run wrongly: unknown subcommand, input that does not parse | `Failed`, "the tool could not run" |

- **`read_file`**: reads the file. More than 1 MiB, or not UTF-8, or a directory, is exit 1 with
  `read_file: <path>: <reason>`. Otherwise the content, unchanged.
- **`write_file`**: the parent directory must exist; the file is created or truncated and written.
  A directory at the path is exit 1. Output: `wrote <n> bytes to <path>`.
- **`shell`**: runs `/bin/sh -c <command>` in `cwd` (or `/tmp` without one), standard input empty,
  standard output and standard error on one pipe in the order written, and then prints
  `\n[exit <code>]`, or `\n[killed by signal <n>]`. `toolkit` exits 0 whatever the command did: the
  tool ran. A `cwd` that does not exist is exit 1.
- **`http_fetch`**: runs `/bin/curl` with a fixed argument list (section 6) and the URL, prints
  the body and then `\n[http <status>]`. A `curl` failure is exit 1 with
  `http_fetch: <url>: <curl's first line of error>`.

`toolkit` never decides what may run: that was decided before the container existed. It is also
not trusted by `brokerd`: whatever it prints is labelled by the grant, and a bad exit is a fixed
sentence.

## 5. The egress proxy

`toolkit egress-proxy` listens on a Unix socket and speaks the part of SOCKS5 (RFC 1928) that
`curl --proxy socks5h://` uses: no authentication, `CONNECT`, a domain name. Each connection:

1. Greeting: version 5, then the method list. It must offer method 0 (none); the answer is
   `05 00`, otherwise `05 FF` and close.
2. Request: version 5, command 1 (`CONNECT`), reserved 0, address type 3 (domain), a length of 1
   to 255, the name, a port. Another command is reply 7; another address type (an IP literal) is
   reply 8.
3. The name must be a valid host name (`proto::hosts::valid_host`), match one of the `--allow`
   patterns (`host_matches`), and the port must be 443. Otherwise reply 2 ("not allowed").
4. Resolve the name. Keep only **public** addresses (below). None left, or resolution failed:
   reply 4 ("host unreachable"). A name that resolves into the tailnet or the host is refused
   here even though the owner allowed the name: the grant allows a host on the internet.
5. Connect to the first public address, 10 s timeout. Failure: reply 5. Success: reply
   `05 00 00 01 00 00 00 00 00 00`, then copy bytes both ways until either side closes, passing a
   half-close on.

Every reply other than success is followed by closing the connection. The proxy reads at most 262
bytes of greeting and request, with a 10 s timeout for the whole handshake, and serves at most 8
connections at once (more are closed at once). It never parses what flows after the handshake:
TLS runs end to end between `curl` and the server.

**Public** means not any of: IPv4 `0.0.0.0/8`, `10/8`, `100.64/10` (the tailnet's range),
`127/8`, `169.254/16`, `172.16/12`, `192.0.0/24`, `192.0.2/24`, `192.168/16`, `198.18/15`,
`198.51.100/24`, `203.0.113/24`, `224/4`, `240/4`; IPv6 `::/96` (which holds `::`, `::1` and
the old IPv4-compatible form), `fc00::/7`, `fe80::/10`,
`ff00::/8`, `2001:db8::/32`, and any IPv4-mapped (`::ffff:0:0/96`) or NAT64 (`64:ff9b::/96`)
address whose IPv4 part is not public. `toolkit::addr::is_public(IpAddr) -> bool`.

For tests, name resolution and connecting are behind a trait, so no test needs a network:

```rust
pub trait Dial: Send + Sync {
    fn resolve(&self, host: &str, port: u16) -> std::io::Result<Vec<SocketAddr>>;
    fn connect(&self, addr: SocketAddr, timeout: Duration) -> std::io::Result<TcpStream>;
}
```

## 6. `brokerd`: the Podman runtime

### Configuration (new)

```toml
[runner]                                  # absent: the M3a runtime, which refuses every call
podman = "podman"                         # the program; a bare name is looked up in PATH
image = "localhost/boxmaker-tools@sha256:…"   # required; by digest
egress_network = "pasta"                  # the proxy container's network
output_cap = 262144                       # bytes of standard output kept; more is truncated
memory = "512m"
pids = 128
read_file_ms = 30000
write_file_ms = 30000
shell_ms = 100000
http_fetch_ms = 60000
```

Every time limit must stay under `loopd`'s `[broker] timeout_ms` (default 120,000), because `loopd`
stops waiting then. To give `shell` longer, raise both.

### One call

The container is named `boxmaker-<session>-<call>-<n>`, where `n` counts calls within this
`brokerd` process, and labelled `boxmaker=tool`. Arguments, in this order, as separate `OsString`s
(never a shell string):

```
run --rm -i --name=<name> --label=boxmaker=tool --network=none --read-only --cap-drop=all
    --security-opt=no-new-privileges --userns=keep-id --pids-limit=<pids> --memory=<memory>
    --tmpfs=/tmp:rw,size=64m,mode=1777
    [--volume=<path>:<path>:ro | :rw   for each mount, in RunSpec order]
    [--volume=<egress dir>:/run/egress:rw   http_fetch only]
    <image> /bin/toolkit <tool>
```

`brokerd` writes the tool's `proto::tools` JSON to standard input and closes it, reads standard
output up to `output_cap` bytes (one byte more means `truncated`, and the rest is not read), and
keeps the first 4 KiB of standard error for its own log only. It waits for the tool's time limit,
polling. Then:

| What happened | Answer |
|---|---|
| exit 0 or 1 | `Ok(RunOutput { content, truncated })`, content decoded as UTF-8 with replacement |
| exit 2 | `Failed("the tool could not run")` |
| exit 125, 126 or 127 (`podman` itself failed: no image, bad option) | `Unavailable("the tool runner could not start the container")`, and `brokerd` prints standard error and `see docs/runbook.md#runner-unavailable` |
| `podman` cannot be started at all | the same |
| exit 137 (killed: out of memory, or the limit below) | `Failed("the tool was stopped: it ran out of memory or was killed")` |
| past the time limit | `podman kill <name>`, then `podman rm -f <name>`, then `Failed("the tool ran past its time limit")` |
| any other exit | `Failed("the tool failed with an unexpected status")` |

Every text above is fixed: tool output never reaches the model through a `RunError`.

### `http_fetch`

Before the tool container, `brokerd` makes `<home>/run/egress/<name>/` (mode 0700) and starts the
proxy:

```
run -d --rm --name=<name>-egress --label=boxmaker=egress --network=<egress_network> --read-only
    --cap-drop=all --security-opt=no-new-privileges --userns=keep-id --pids-limit=64
    --memory=128m --volume=<egress dir>:/run/egress:rw
    <image> /bin/toolkit egress-proxy --socket /run/egress/egress.sock --allow <hosts joined by ,>
```

It waits up to 5 s for `<egress dir>/egress.sock` to exist, runs the tool container with the
directory mounted, and afterwards, **whatever happened**, runs `podman rm -f <name>-egress` and
removes the directory. A proxy that does not start in time is `Unavailable` with the
`runner-unavailable` pointer. The tool container itself still has `--network=none`: its only way
out is the socket.

`curl`'s arguments inside the tool container are fixed:

```
/bin/curl --silent --show-error --proto =https --proto-redir =https --location --max-redirs 5
    --max-time 50 --max-filesize 8388608 --cacert /etc/ssl/certs/ca-certificates.crt
    --proxy socks5h://localhost/run/egress/egress.sock --write-out "\n[http %{response_code}]"
    --url <url>
```

## 7. The image (new detail; P13)

Built by Nix on straylight (`deploy/tools-image.nix`, `dockerTools.buildLayeredImage`), loaded with
`podman load`, and named in `[runner] image` by digest. It holds `/bin/toolkit` (static musl),
`/bin/busybox` with `/bin/sh` linking to it, `/bin/curl` (static), and
`/etc/ssl/certs/ca-certificates.crt`. The design model builds and checks it on straylight; it is not
an implementer task (this machine has neither Nix nor Podman).

## 8. Runbook

A new entry, `runner-unavailable`: `podman` missing, the image not loaded or not the digest in the
config, or the proxy not starting. Every such message ends with its pointer.

## 9. Testing

- **Offline (the implementer's tasks)**: `proto::tools` and `proto::hosts` round trips; `FetchUrl`
  sealed (a `compile_fail` doctest); the grant path rule; `toolkit` run as a program against temporary
  directories (read, write, shell, the exit codes); `http_fetch`'s argument list; `is_public` as a
  table; the proxy over `UnixStream::pair()` against a fake `Dial` (every reply code, the host and
  address checks, the byte copy both ways, the handshake limits); `brokerd`'s argument lists as
  golden files; the runtime against fake `podman` scripts that record their arguments (exit codes,
  output cap, time limit with its kill and remove, standard input, the egress lifecycle and its
  clean-up on every path).
- **On straylight (the design model)**: build and load the image; run each tool through
  `brokerd`; a `shell` call cannot reach `100.100.100.100`, the host or the internet;
  `http_fetch` reaches an allowed host, is refused for another, for a redirect to another, and for
  an allowed name that resolves into the tailnet; limits hold; no container with a `boxmaker`
  label remains.

## 10. For the owner's review

1. Tool time limits sit under `loopd`'s 120 s broker timeout, so `shell` defaults to 100 s, not the
   10 minutes discussed. Longer shells mean raising both values.
2. The proxy refuses any allowed name that resolves to a non-public address.
3. Grant paths may not contain `:` or `,`.
4. `shell` has busybox and nothing else in the image: no compilers, no `git`.
5. `toolkit`'s exit 1 output is shown to the model as a result, labelled like any other.