# M3b task 06: `toolkit http_fetch`

**Branch:** `m3b` (run `git switch m3b`; `git status --short` must be empty, otherwise stop)
**Commit subject:** `toolkit: http_fetch through curl`

## Goal

`toolkit http_fetch` runs `/bin/curl` with a **fixed** argument list and the URL, through the egress
proxy's socket that `brokerd` mounts at `/run/egress/egress.sock` for this call. It prints the body
and a status line, or one line saying why the fetch failed. We do not write an HTTP or TLS client:
`curl` does that, inside the container. Spec sections 4 and 6.

## Files

- Copy: `crates/toolkit/tests/fetch.rs`
- Create: `crates/toolkit/src/fetch.rs`
- Modify: `crates/toolkit/src/lib.rs`, `docs/implementer-log.md`

## Interfaces

```rust
pub const CURL: &str = "/bin/curl";
pub const PROXY: &str = "socks5h://localhost/run/egress/egress.sock";
pub const CA_BUNDLE: &str = "/etc/ssl/certs/ca-certificates.crt";

/// `curl`'s arguments for `url`, in order, without the program name.
pub fn curl_args(url: &str) -> Vec<String>;
pub fn fetch(args: &proto::tools::HttpFetchArgs) -> crate::Outcome;   // fetch_with(Path::new(CURL), args)
/// `fetch` with another `curl`, for tests.
pub fn fetch_with(curl: &std::path::Path, args: &proto::tools::HttpFetchArgs) -> crate::Outcome;
```

In `lib.rs`: `pub mod fetch;`, and an `"http_fetch"` arm in `run` like the others, calling
`fetch::fetch(&args)`.

## `curl_args`

Exactly these 21 strings, in this order (the test compares them one by one):

```
--silent  --show-error  --proto  =https  --proto-redir  =https  --location  --max-redirs  5
--max-time  50  --max-filesize  8388608  --cacert  <CA_BUNDLE>  --proxy  <PROXY>
--write-out  "\n[http %{response_code}]"  --url  <url>
```

In Rust, `--write-out`'s value is the literal `"\n[http %{response_code}]"`: a real newline
character, then `[http %{response_code}]` (curl fills in the status). `--url <url>` (not a bare
URL) keeps a URL from ever being read as an option.

## `fetch_with`: every exit

1. Spawn `curl` with `curl_args(&args.url)`, standard input `Stdio::null()`, standard output and
   standard error piped. A failure →
   `Outcome::tool_error(format!("http_fetch: cannot start {}: {e}", curl.display()))`.
2. Read standard error **on its own thread**, keeping at most 64 KiB, while the main thread reads
   standard output to the end. (Reading one and then the other can deadlock when both are full:
   the given test writes 300,000 bytes to each.)
3. Wait for `curl`, then join the thread.
4. `curl` succeeded → `Outcome::done(body)`, the body decoded with `from_utf8_lossy`. The status
   line is already in it, from `--write-out`.
5. It failed → `Outcome::tool_error(format!("http_fetch: {url}: {why}"))`, where `why` is the first
   line of standard error that is not blank, trimmed; if there is none, `curl exited {code}`, or
   `curl was killed` when there is no code. The body, if any, is not shown.
6. Waiting failed → `tool_error(format!("http_fetch: cannot wait for curl: {e}"))`.

## About the given tests

`fetch.rs` writes small shell scripts that stand in for `curl` and runs them. Every test that
starts a process takes a lock first (`let _serial = serial();`). Without it, another test's fork
can hold a just-written script open, and running it fails with "Text file busy" (ETXTBSY) about
once in seven runs. If you add a test that writes and runs a script, take the lock too.

## Steps

- [ ] **1. Copy.** `git switch m3b`, then
  `cp docs/plans/M3b/files/crates/toolkit/tests/fetch.rs crates/toolkit/tests/`
- [ ] **2. See it fail.** `cargo test -p toolkit --test fetch`. Expected: it does not compile.
- [ ] **3. Write `fetch.rs`** and the `lib.rs` changes. Run `cargo fmt --all`.
- [ ] **4. See it pass.** `cargo test -p toolkit --test fetch`. Expected: 7 passed. Run it ten
  times; it must pass every time.
- [ ] **5. Run the gate.** `make gate`. Expected last line: `gate: ok`.
- [ ] **6. Log and commit.** `git add crates/toolkit docs/implementer-log.md && git commit`

## Done when

- `cargo test -p toolkit --test fetch` reports 7 passed ten times running; `make gate` prints
  `gate: ok`.

## Stop and report if

- A test fails with "Text file busy" even with the lock: report the run and the test.