Specify and plan M3b: the runner and the tools

A draft spec for the owner's review and 13 offline tasks with their given
tests: shared tool arguments and host rules in proto, the sealed fetch
target (M3a finding 14), the toolkit tools and SOCKS5 egress proxy, and
brokerd's [runner], podman argument lists, runtime and proxy lifecycle. Each
task's tests were run against a reference at that task's end state (560 to
638 tests, clippy clean); the reference is not in the repository. Adds the
runner-unavailable runbook entry and tip T23 (ETXTBSY in script tests).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-09-22 22:29:27 -07:00
co-authored by Claude Opus 5.5
parent d988edac4a
commit b426ca1958
47 changed files with 4491 additions and 2 deletions
+91
View File
@@ -0,0 +1,91 @@
# M3b task 06: `toolkit http_fetch`
**Branch:** `m3b` (run `git switch m3b`; `git status --short` must be empty, otherwise stop)
**Commit subject:** `toolkit: http_fetch through curl`
## Goal
`toolkit http_fetch` runs `/bin/curl` with a **fixed** argument list and the URL, through the egress
proxy's socket that `brokerd` mounts at `/run/egress/egress.sock` for this call. It prints the body
and a status line, or one line saying why the fetch failed. We do not write an HTTP or TLS client:
`curl` does that, inside the container. Spec sections 4 and 6.
## Files
- Copy: `crates/toolkit/tests/fetch.rs`
- Create: `crates/toolkit/src/fetch.rs`
- Modify: `crates/toolkit/src/lib.rs`, `docs/implementer-log.md`
## Interfaces
```rust
pub const CURL: &str = "/bin/curl";
pub const PROXY: &str = "socks5h://localhost/run/egress/egress.sock";
pub const CA_BUNDLE: &str = "/etc/ssl/certs/ca-certificates.crt";
/// `curl`'s arguments for `url`, in order, without the program name.
pub fn curl_args(url: &str) -> Vec<String>;
pub fn fetch(args: &proto::tools::HttpFetchArgs) -> crate::Outcome; // fetch_with(Path::new(CURL), args)
/// `fetch` with another `curl`, for tests.
pub fn fetch_with(curl: &std::path::Path, args: &proto::tools::HttpFetchArgs) -> crate::Outcome;
```
In `lib.rs`: `pub mod fetch;`, and an `"http_fetch"` arm in `run` like the others, calling
`fetch::fetch(&args)`.
## `curl_args`
Exactly these 21 strings, in this order (the test compares them one by one):
```
--silent --show-error --proto =https --proto-redir =https --location --max-redirs 5
--max-time 50 --max-filesize 8388608 --cacert <CA_BUNDLE> --proxy <PROXY>
--write-out "\n[http %{response_code}]" --url <url>
```
In Rust, `--write-out`'s value is the literal `"\n[http %{response_code}]"`: a real newline
character, then `[http %{response_code}]` (curl fills in the status). `--url <url>` (not a bare
URL) keeps a URL from ever being read as an option.
## `fetch_with`: every exit
1. Spawn `curl` with `curl_args(&args.url)`, standard input `Stdio::null()`, standard output and
standard error piped. A failure →
`Outcome::tool_error(format!("http_fetch: cannot start {}: {e}", curl.display()))`.
2. Read standard error **on its own thread**, keeping at most 64 KiB, while the main thread reads
standard output to the end. (Reading one and then the other can deadlock when both are full:
the given test writes 300,000 bytes to each.)
3. Wait for `curl`, then join the thread.
4. `curl` succeeded → `Outcome::done(body)`, the body decoded with `from_utf8_lossy`. The status
line is already in it, from `--write-out`.
5. It failed → `Outcome::tool_error(format!("http_fetch: {url}: {why}"))`, where `why` is the first
line of standard error that is not blank, trimmed; if there is none, `curl exited {code}`, or
`curl was killed` when there is no code. The body, if any, is not shown.
6. Waiting failed → `tool_error(format!("http_fetch: cannot wait for curl: {e}"))`.
## About the given tests
`fetch.rs` writes small shell scripts that stand in for `curl` and runs them. Every test that
starts a process takes a lock first (`let _serial = serial();`). Without it, another test's fork
can hold a just-written script open, and running it fails with "Text file busy" (ETXTBSY) about
once in seven runs. If you add a test that writes and runs a script, take the lock too.
## Steps
- [ ] **1. Copy.** `git switch m3b`, then
`cp docs/plans/M3b/files/crates/toolkit/tests/fetch.rs crates/toolkit/tests/`
- [ ] **2. See it fail.** `cargo test -p toolkit --test fetch`. Expected: it does not compile.
- [ ] **3. Write `fetch.rs`** and the `lib.rs` changes. Run `cargo fmt --all`.
- [ ] **4. See it pass.** `cargo test -p toolkit --test fetch`. Expected: 7 passed. Run it ten
times; it must pass every time.
- [ ] **5. Run the gate.** `make gate`. Expected last line: `gate: ok`.
- [ ] **6. Log and commit.** `git add crates/toolkit docs/implementer-log.md && git commit`
## Done when
- `cargo test -p toolkit --test fetch` reports 7 passed ten times running; `make gate` prints
`gate: ok`.
## Stop and report if
- A test fails with "Text file busy" even with the lock: report the run and the test.