boxmaker/docs/specs/2026-09-22-m3b-runner.md

M3b design: the runner and the tools

Status: draft, 2026-09-22, written for an overnight implementation run; the owner reviews it the next morning. The golden files of plan task 10 (docs/plans/M3b/files/crates/brokerd/tests/fixtures/podman/) are the exact argument lists. The shape was agreed in the M3 design discussion (the split, rootless Podman, an image built by Nix and named by digest, curl behind a SOCKS5 proxy of our own). The details below that are new are marked (new) and listed in section 10 for review. Where this document and docs/design.md disagree, the brief wins. M3a's spec (2026-09-18-m3a-decision-path.md) still holds for everything before a RunSpec exists.

1. What M3b proves

Claim	Checked by
Each allowed call runs in a fresh container with no network, a read-only root, only the grant's directories mounted, and limits on time, memory, processes and output	Golden podman argument lists; the runtime against a fake podman; on straylight
A container with no network grant cannot reach the network	On straylight: the tailnet, the host and the internet are all unreachable from shell
http_fetch reaches only the grant's hosts, redirects included, and never an address inside the tailnet, the host or a private network	Proxy tests against a fake resolver; on straylight
A tool's output reaches the model labelled by its grant, capped, and a runner failure reaches it as a fixed sentence	Runtime tests
No container outlives its call	Runtime tests with a fake podman (the kill and remove calls); on straylight (podman ps by label)

Out of scope: secrets (M4), anything but the four tools, a registry (the image is loaded locally), the NixOS module and service users (M7).

2. Shared definitions move to proto (new)

Two programs now read the same things, so each has one definition (AGENTS: "when two programs must agree about a set of things, one of them deciding alone is a bug"):

• proto::tools: the argument structs of the four tools, as brokerd parses them today: ReadFileArgs { path }, WriteFileArgs { path, content }, ShellArgs { command, cwd: Option }, HttpFetchArgs { url }, all deny_unknown_fields, cwd omitted when None. brokerd parses the model's arguments with them and writes them to the container's standard input; toolkit reads them back.
• proto::hosts: valid_host, valid_host_pattern, host_matches, moved from brokerd::args unchanged. brokerd checks grants and URLs with them; the egress proxy checks each connection.

brokerd::args re-exports the moved functions, so its callers and tests do not change.

3. Sealed fetch target (M3a finding 14)

ToolArgs::HttpFetch { url, host } becomes ToolArgs::HttpFetch(FetchUrl). FetchUrl has private fields, url() and host() getters, no public constructor, and is built only by args::parse after the URL checks. Policy matches host(); the runner hands url() to the tool. They cannot disagree.

Grant paths gain one rule (new): a path in paths may not contain : or ,, because it is mounted with podman --volume=<path>:<path>:ro and those characters are separators there. The problem text is "cannot be mounted: it contains ':' or ','".

4. toolkit: the programs inside the container

One static binary, /bin/toolkit, with one subcommand per tool and one for the proxy:

toolkit read_file | write_file | shell | http_fetch      arguments as JSON on standard input
toolkit egress-proxy --socket <path> --allow <pattern>[,<pattern>…]

For the four tools, toolkit reads all of standard input (at most 2 MiB), parses it strictly as the tool's proto::tools struct, does the work, writes what the model should see to standard output, and exits:

Exit	Meaning	brokerd answers
0	The tool did its work	Result, content = standard output
1	The tool could not do it, for a reason the model should read (no such file, not text)	Result, content = standard output, which is one line starting with the tool's name
2	toolkit was run wrongly: unknown subcommand, input that does not parse	Failed, "the tool could not run"

• read_file: reads the file. More than 1 MiB, or not UTF-8, or a directory, is exit 1 with read_file: <path>: <reason>. Otherwise the content, unchanged.
• write_file: the parent directory must exist; the file is created or truncated and written. A directory at the path is exit 1. Output: wrote <n> bytes to <path>.
• shell: runs /bin/sh -c <command> in cwd (or /tmp without one), standard input empty, standard output and standard error on one pipe in the order written, and then prints \n[exit <code>], or \n[killed by signal <n>]. toolkit exits 0 whatever the command did: the tool ran. A cwd that does not exist is exit 1.
• http_fetch: runs /bin/curl with a fixed argument list (section 6) and the URL, prints the body and then \n[http <status>]. A curl failure is exit 1 with http_fetch: <url>: <curl's first line of error>.

toolkit never decides what may run: that was decided before the container existed. It is also not trusted by brokerd: whatever it prints is labelled by the grant, and a bad exit is a fixed sentence.

5. The egress proxy

toolkit egress-proxy listens on a Unix socket and speaks the part of SOCKS5 (RFC 1928) that curl --proxy socks5h:// uses: no authentication, CONNECT, a domain name. Each connection:

1. Greeting: version 5, then the method list. It must offer method 0 (none); the answer is 05 00, otherwise 05 FF and close.
2. Request: version 5, command 1 (CONNECT), reserved 0, address type 3 (domain), a length of 1 to 255, the name, a port. Another command is reply 7; another address type (an IP literal) is reply 8.
3. The name must be a valid host name (proto::hosts::valid_host), match one of the --allow patterns (host_matches), and the port must be 443. Otherwise reply 2 ("not allowed").
4. Resolve the name. Keep only public addresses (below). None left, or resolution failed: reply 4 ("host unreachable"). A name that resolves into the tailnet or the host is refused here even though the owner allowed the name: the grant allows a host on the internet.
5. Connect to the first public address, 10 s timeout. Failure: reply 5. Success: reply 05 00 00 01 00 00 00 00 00 00, then copy bytes both ways until either side closes, passing a half-close on.

Every reply other than success is followed by closing the connection. The proxy reads at most 262 bytes of greeting and request, with a 10 s timeout for the whole handshake, and serves at most 8 connections at once (more are closed at once). It never parses what flows after the handshake: TLS runs end to end between curl and the server.

Public means not any of: IPv4 0.0.0.0/8, 10/8, 100.64/10 (the tailnet's range), 127/8, 169.254/16, 172.16/12, 192.0.0/24, 192.0.2/24, 192.168/16, 198.18/15, 198.51.100/24, 203.0.113/24, 224/4, 240/4; IPv6 ::/96 (which holds ::, ::1 and the old IPv4-compatible form), fc00::/7, fe80::/10, ff00::/8, 2001:db8::/32, and any IPv4-mapped (::ffff:0:0/96) or NAT64 (64:ff9b::/96) address whose IPv4 part is not public. toolkit::addr::is_public(IpAddr) -> bool.

For tests, name resolution and connecting are behind a trait, so no test needs a network:

pub trait Dial: Send + Sync {
    fn resolve(&self, host: &str, port: u16) -> std::io::Result<Vec<SocketAddr>>;
    fn connect(&self, addr: SocketAddr, timeout: Duration) -> std::io::Result<TcpStream>;
}

6. brokerd: the Podman runtime

Configuration (new)

[runner]                                  # absent: the M3a runtime, which refuses every call
podman = "podman"                         # the program; a bare name is looked up in PATH
image = "localhost/boxmaker-tools@sha256:…"   # required; by digest
egress_network = "pasta"                  # the proxy container's network
output_cap = 262144                       # bytes of standard output kept; more is truncated
memory = "512m"
pids = 128
read_file_ms = 30000
write_file_ms = 30000
shell_ms = 100000
http_fetch_ms = 60000

Every time limit must stay under loopd's [broker] timeout_ms (default 120,000), because loopd stops waiting then. To give shell longer, raise both.

One call

The container is named boxmaker-<session>-<call>-<n>, where n counts calls within this brokerd process, and labelled boxmaker=tool. Arguments, in this order, as separate OsStrings (never a shell string):

run --rm -i --name=<name> --label=boxmaker=tool --network=none --pull=never --read-only --cap-drop=all
    --security-opt=no-new-privileges --userns=keep-id --pids-limit=<pids> --memory=<memory>
    --tmpfs=/tmp:rw,size=64m,mode=1777
    [--volume=<path>:<path>:ro | :rw   for each mount, in RunSpec order]
    [--volume=<egress dir>:/run/egress:rw   http_fetch only]
    <image> /bin/toolkit <tool>

brokerd writes the tool's proto::tools JSON to standard input and closes it, reads standard output up to output_cap bytes (one byte more means truncated, and the rest is not read), and keeps the first 4 KiB of standard error for its own log only. It waits for the tool's time limit, polling. Then:

What happened	Answer
exit 0 or 1	Ok(RunOutput { content, truncated }), content decoded as UTF-8 with replacement
exit 2	Failed("the tool could not run")
exit 125, 126 or 127 (podman itself failed: no image, bad option)	Unavailable("the tool runner could not start the container"), and brokerd prints standard error and see docs/runbook.md#runner-unavailable
podman cannot be started at all	the same
exit 137 (killed: out of memory, or the limit below)	Failed("the tool was stopped: it ran out of memory or was killed")
past the time limit	podman kill <name>, then podman rm -f <name>, then Failed("the tool ran past its time limit")
any other exit	Failed("the tool failed with an unexpected status")

Every text above is fixed: tool output never reaches the model through a RunError.

--pull=never (added after the M3b review): an image that is not loaded fails the call at once (exit 125) instead of Podman trying to pull it, which would be unlisted egress.

http_fetch

Before the tool container, brokerd makes <home>/run/egress/<name>/ (mode 0700) and starts the proxy:

run -d --rm --name=<name>-egress --label=boxmaker=egress --network=<egress_network> --pull=never --read-only
    --cap-drop=all --security-opt=no-new-privileges --userns=keep-id --pids-limit=64
    --memory=128m --volume=<egress dir>:/run/egress:rw
    <image> /bin/toolkit egress-proxy --socket /run/egress/egress.sock --allow <hosts joined by ,>

It waits up to 5 s for <egress dir>/egress.sock to exist, runs the tool container with the directory mounted, and afterwards, whatever happened, runs podman rm -f <name>-egress and removes the directory. A proxy that does not start in time is Unavailable with the runner-unavailable pointer. The tool container itself still has --network=none: its only way out is the socket.

curl's arguments inside the tool container are fixed:

/bin/curl --disable --silent --show-error --globoff --proto =https --proto-redir =https --location --max-redirs 5
    --max-time 50 --max-filesize 8388608 --cacert /etc/ssl/certs/ca-certificates.crt
    --proxy socks5h://localhost/run/egress/egress.sock --write-out "\n[http %{response_code}]"
    --url <url>

7. The image (new detail; P13)

Built by Nix on straylight (deploy/tools-image.nix, dockerTools.buildLayeredImage), loaded with podman load, and named in [runner] image by digest. It holds /bin/toolkit (static musl), /bin/busybox with /bin/sh linking to it, /bin/curl (static), and /etc/ssl/certs/ca-certificates.crt. The design model builds and checks it on straylight; it is not an implementer task (this machine has neither Nix nor Podman).

8. Runbook

A new entry, runner-unavailable: podman missing, the image not loaded or not the digest in the config, or the proxy not starting. Every such message ends with its pointer.

9. Testing

• Offline (the implementer's tasks): proto::tools and proto::hosts round trips; FetchUrl sealed (a compile_fail doctest); the grant path rule; toolkit run as a program against temporary directories (read, write, shell, the exit codes); http_fetch's argument list; is_public as a table; the proxy over UnixStream::pair() against a fake Dial (every reply code, the host and address checks, the byte copy both ways, the handshake limits); brokerd's argument lists as golden files; the runtime against fake podman scripts that record their arguments (exit codes, output cap, time limit with its kill and remove, standard input, the egress lifecycle and its clean-up on every path).
• On straylight (the design model): build and load the image; run each tool through brokerd; a shell call cannot reach 100.100.100.100, the host or the internet; http_fetch reaches an allowed host, is refused for another, for a redirect to another, and for an allowed name that resolves into the tailnet; limits hold; no container with a boxmaker label remains.

10. For the owner's review

1. Tool time limits sit under loopd's 120 s broker timeout, so shell defaults to 100 s, not the 10 minutes discussed. Longer shells mean raising both values.
2. The proxy refuses any allowed name that resolves to a non-public address.
3. Grant paths may not contain : or ,.
4. shell has busybox and nothing else in the image: no compilers, no git.
5. toolkit's exit 1 output is shown to the model as a result, labelled like any other.