boxmaker/docs/plans/M3b/06-toolkit-fetch.md

M3b task 06: toolkit http_fetch

Branch: m3b (run git switch m3b; git status --short must be empty, otherwise stop) Commit subject: toolkit: http_fetch through curl

Goal

toolkit http_fetch runs /bin/curl with a fixed argument list and the URL, through the egress proxy's socket that brokerd mounts at /run/egress/egress.sock for this call. It prints the body and a status line, or one line saying why the fetch failed. We do not write an HTTP or TLS client: curl does that, inside the container. Spec sections 4 and 6.

Files

• Copy: crates/toolkit/tests/fetch.rs
• Create: crates/toolkit/src/fetch.rs
• Modify: crates/toolkit/src/lib.rs, docs/implementer-log.md

Interfaces

pub const CURL: &str = "/bin/curl";
pub const PROXY: &str = "socks5h://localhost/run/egress/egress.sock";
pub const CA_BUNDLE: &str = "/etc/ssl/certs/ca-certificates.crt";

/// `curl`'s arguments for `url`, in order, without the program name.
pub fn curl_args(url: &str) -> Vec<String>;
pub fn fetch(args: &proto::tools::HttpFetchArgs) -> crate::Outcome;   // fetch_with(Path::new(CURL), args)
/// `fetch` with another `curl`, for tests.
pub fn fetch_with(curl: &std::path::Path, args: &proto::tools::HttpFetchArgs) -> crate::Outcome;

In lib.rs: pub mod fetch;, and an "http_fetch" arm in run like the others, calling fetch::fetch(&args).

curl_args

Exactly these 21 strings, in this order (the test compares them one by one):

--silent  --show-error  --proto  =https  --proto-redir  =https  --location  --max-redirs  5
--max-time  50  --max-filesize  8388608  --cacert  <CA_BUNDLE>  --proxy  <PROXY>
--write-out  "\n[http %{response_code}]"  --url  <url>

In Rust, --write-out's value is the literal "\n[http %{response_code}]": a real newline character, then [http %{response_code}] (curl fills in the status). --url <url> (not a bare URL) keeps a URL from ever being read as an option.

fetch_with: every exit

1. Spawn curl with curl_args(&args.url), standard input Stdio::null(), standard output and standard error piped. A failure → Outcome::tool_error(format!("http_fetch: cannot start {}: {e}", curl.display())).
2. Read standard error on its own thread, keeping at most 64 KiB, while the main thread reads standard output to the end. (Reading one and then the other can deadlock when both are full: the given test writes 300,000 bytes to each.)
3. Wait for curl, then join the thread.
4. curl succeeded → Outcome::done(body), the body decoded with from_utf8_lossy. The status line is already in it, from --write-out.
5. It failed → Outcome::tool_error(format!("http_fetch: {url}: {why}")), where why is the first line of standard error that is not blank, trimmed; if there is none, curl exited {code}, or curl was killed when there is no code. The body, if any, is not shown.
6. Waiting failed → tool_error(format!("http_fetch: cannot wait for curl: {e}")).

About the given tests

fetch.rs writes small shell scripts that stand in for curl and runs them. Every test that starts a process takes a lock first (let _serial = serial();). Without it, another test's fork can hold a just-written script open, and running it fails with "Text file busy" (ETXTBSY) about once in seven runs. If you add a test that writes and runs a script, take the lock too.

Steps

• 1. Copy. git switch m3b, then cp docs/plans/M3b/files/crates/toolkit/tests/fetch.rs crates/toolkit/tests/
• 2. See it fail. cargo test -p toolkit --test fetch. Expected: it does not compile.
• 3. Write fetch.rs and the lib.rs changes. Run cargo fmt --all.
• 4. See it pass. cargo test -p toolkit --test fetch. Expected: 7 passed. Run it ten times; it must pass every time.
• 5. Run the gate. make gate. Expected last line: gate: ok.
• 6. Log and commit. git add crates/toolkit docs/implementer-log.md Cargo.lock && git commit

Done when

• cargo test -p toolkit --test fetch reports 7 passed ten times running; make gate prints gate: ok.

Stop and report if

• A test fails with "Text file busy" even with the lock: report the run and the test.