Specify and plan M3b: the runner and the tools
A draft spec for the owner's review and 13 offline tasks with their given tests: shared tool arguments and host rules in proto, the sealed fetch target (M3a finding 14), the toolkit tools and SOCKS5 egress proxy, and brokerd's [runner], podman argument lists, runtime and proxy lifecycle. Each task's tests were run against a reference at that task's end state (560 to 638 tests, clippy clean); the reference is not in the repository. Adds the runner-unavailable runbook entry and tip T23 (ETXTBSY in script tests). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,91 @@
|
||||
# M3b task 06: `toolkit http_fetch`
|
||||
|
||||
**Branch:** `m3b` (run `git switch m3b`; `git status --short` must be empty, otherwise stop)
|
||||
**Commit subject:** `toolkit: http_fetch through curl`
|
||||
|
||||
## Goal
|
||||
|
||||
`toolkit http_fetch` runs `/bin/curl` with a **fixed** argument list and the URL, through the egress
|
||||
proxy's socket that `brokerd` mounts at `/run/egress/egress.sock` for this call. It prints the body
|
||||
and a status line, or one line saying why the fetch failed. We do not write an HTTP or TLS client:
|
||||
`curl` does that, inside the container. Spec sections 4 and 6.
|
||||
|
||||
## Files
|
||||
|
||||
- Copy: `crates/toolkit/tests/fetch.rs`
|
||||
- Create: `crates/toolkit/src/fetch.rs`
|
||||
- Modify: `crates/toolkit/src/lib.rs`, `docs/implementer-log.md`
|
||||
|
||||
## Interfaces
|
||||
|
||||
```rust
|
||||
pub const CURL: &str = "/bin/curl";
|
||||
pub const PROXY: &str = "socks5h://localhost/run/egress/egress.sock";
|
||||
pub const CA_BUNDLE: &str = "/etc/ssl/certs/ca-certificates.crt";
|
||||
|
||||
/// `curl`'s arguments for `url`, in order, without the program name.
|
||||
pub fn curl_args(url: &str) -> Vec<String>;
|
||||
pub fn fetch(args: &proto::tools::HttpFetchArgs) -> crate::Outcome; // fetch_with(Path::new(CURL), args)
|
||||
/// `fetch` with another `curl`, for tests.
|
||||
pub fn fetch_with(curl: &std::path::Path, args: &proto::tools::HttpFetchArgs) -> crate::Outcome;
|
||||
```
|
||||
|
||||
In `lib.rs`: `pub mod fetch;`, and an `"http_fetch"` arm in `run` like the others, calling
|
||||
`fetch::fetch(&args)`.
|
||||
|
||||
## `curl_args`
|
||||
|
||||
Exactly these 21 strings, in this order (the test compares them one by one):
|
||||
|
||||
```
|
||||
--silent --show-error --proto =https --proto-redir =https --location --max-redirs 5
|
||||
--max-time 50 --max-filesize 8388608 --cacert <CA_BUNDLE> --proxy <PROXY>
|
||||
--write-out "\n[http %{response_code}]" --url <url>
|
||||
```
|
||||
|
||||
In Rust, `--write-out`'s value is the literal `"\n[http %{response_code}]"`: a real newline
|
||||
character, then `[http %{response_code}]` (curl fills in the status). `--url <url>` (not a bare
|
||||
URL) keeps a URL from ever being read as an option.
|
||||
|
||||
## `fetch_with`: every exit
|
||||
|
||||
1. Spawn `curl` with `curl_args(&args.url)`, standard input `Stdio::null()`, standard output and
|
||||
standard error piped. A failure →
|
||||
`Outcome::tool_error(format!("http_fetch: cannot start {}: {e}", curl.display()))`.
|
||||
2. Read standard error **on its own thread**, keeping at most 64 KiB, while the main thread reads
|
||||
standard output to the end. (Reading one and then the other can deadlock when both are full:
|
||||
the given test writes 300,000 bytes to each.)
|
||||
3. Wait for `curl`, then join the thread.
|
||||
4. `curl` succeeded → `Outcome::done(body)`, the body decoded with `from_utf8_lossy`. The status
|
||||
line is already in it, from `--write-out`.
|
||||
5. It failed → `Outcome::tool_error(format!("http_fetch: {url}: {why}"))`, where `why` is the first
|
||||
line of standard error that is not blank, trimmed; if there is none, `curl exited {code}`, or
|
||||
`curl was killed` when there is no code. The body, if any, is not shown.
|
||||
6. Waiting failed → `tool_error(format!("http_fetch: cannot wait for curl: {e}"))`.
|
||||
|
||||
## About the given tests
|
||||
|
||||
`fetch.rs` writes small shell scripts that stand in for `curl` and runs them. Every test that
|
||||
starts a process takes a lock first (`let _serial = serial();`). Without it, another test's fork
|
||||
can hold a just-written script open, and running it fails with "Text file busy" (ETXTBSY) about
|
||||
once in seven runs. If you add a test that writes and runs a script, take the lock too.
|
||||
|
||||
## Steps
|
||||
|
||||
- [ ] **1. Copy.** `git switch m3b`, then
|
||||
`cp docs/plans/M3b/files/crates/toolkit/tests/fetch.rs crates/toolkit/tests/`
|
||||
- [ ] **2. See it fail.** `cargo test -p toolkit --test fetch`. Expected: it does not compile.
|
||||
- [ ] **3. Write `fetch.rs`** and the `lib.rs` changes. Run `cargo fmt --all`.
|
||||
- [ ] **4. See it pass.** `cargo test -p toolkit --test fetch`. Expected: 7 passed. Run it ten
|
||||
times; it must pass every time.
|
||||
- [ ] **5. Run the gate.** `make gate`. Expected last line: `gate: ok`.
|
||||
- [ ] **6. Log and commit.** `git add crates/toolkit docs/implementer-log.md && git commit`
|
||||
|
||||
## Done when
|
||||
|
||||
- `cargo test -p toolkit --test fetch` reports 7 passed ten times running; `make gate` prints
|
||||
`gate: ok`.
|
||||
|
||||
## Stop and report if
|
||||
|
||||
- A test fails with "Text file busy" even with the lock: report the run and the test.
|
||||
Reference in New Issue
Block a user