kyleandClaude Opus 5.5
5e55fe4c66
Task 04 added dependencies to toolkit and its git add line left out the lock file, so the driver stopped on an unclean tree. The lock change is folded into task 04's commit. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
92 lines
4.2 KiB
Markdown
92 lines
4.2 KiB
Markdown
# M3b task 06: `toolkit http_fetch`
|
|
|
|
**Branch:** `m3b` (run `git switch m3b`; `git status --short` must be empty, otherwise stop)
|
|
**Commit subject:** `toolkit: http_fetch through curl`
|
|
|
|
## Goal
|
|
|
|
`toolkit http_fetch` runs `/bin/curl` with a **fixed** argument list and the URL, through the egress
|
|
proxy's socket that `brokerd` mounts at `/run/egress/egress.sock` for this call. It prints the body
|
|
and a status line, or one line saying why the fetch failed. We do not write an HTTP or TLS client:
|
|
`curl` does that, inside the container. Spec sections 4 and 6.
|
|
|
|
## Files
|
|
|
|
- Copy: `crates/toolkit/tests/fetch.rs`
|
|
- Create: `crates/toolkit/src/fetch.rs`
|
|
- Modify: `crates/toolkit/src/lib.rs`, `docs/implementer-log.md`
|
|
|
|
## Interfaces
|
|
|
|
```rust
|
|
pub const CURL: &str = "/bin/curl";
|
|
pub const PROXY: &str = "socks5h://localhost/run/egress/egress.sock";
|
|
pub const CA_BUNDLE: &str = "/etc/ssl/certs/ca-certificates.crt";
|
|
|
|
/// `curl`'s arguments for `url`, in order, without the program name.
|
|
pub fn curl_args(url: &str) -> Vec<String>;
|
|
pub fn fetch(args: &proto::tools::HttpFetchArgs) -> crate::Outcome; // fetch_with(Path::new(CURL), args)
|
|
/// `fetch` with another `curl`, for tests.
|
|
pub fn fetch_with(curl: &std::path::Path, args: &proto::tools::HttpFetchArgs) -> crate::Outcome;
|
|
```
|
|
|
|
In `lib.rs`: `pub mod fetch;`, and an `"http_fetch"` arm in `run` like the others, calling
|
|
`fetch::fetch(&args)`.
|
|
|
|
## `curl_args`
|
|
|
|
Exactly these 21 strings, in this order (the test compares them one by one):
|
|
|
|
```
|
|
--silent --show-error --proto =https --proto-redir =https --location --max-redirs 5
|
|
--max-time 50 --max-filesize 8388608 --cacert <CA_BUNDLE> --proxy <PROXY>
|
|
--write-out "\n[http %{response_code}]" --url <url>
|
|
```
|
|
|
|
In Rust, `--write-out`'s value is the literal `"\n[http %{response_code}]"`: a real newline
|
|
character, then `[http %{response_code}]` (curl fills in the status). `--url <url>` (not a bare
|
|
URL) keeps a URL from ever being read as an option.
|
|
|
|
## `fetch_with`: every exit
|
|
|
|
1. Spawn `curl` with `curl_args(&args.url)`, standard input `Stdio::null()`, standard output and
|
|
standard error piped. A failure →
|
|
`Outcome::tool_error(format!("http_fetch: cannot start {}: {e}", curl.display()))`.
|
|
2. Read standard error **on its own thread**, keeping at most 64 KiB, while the main thread reads
|
|
standard output to the end. (Reading one and then the other can deadlock when both are full:
|
|
the given test writes 300,000 bytes to each.)
|
|
3. Wait for `curl`, then join the thread.
|
|
4. `curl` succeeded → `Outcome::done(body)`, the body decoded with `from_utf8_lossy`. The status
|
|
line is already in it, from `--write-out`.
|
|
5. It failed → `Outcome::tool_error(format!("http_fetch: {url}: {why}"))`, where `why` is the first
|
|
line of standard error that is not blank, trimmed; if there is none, `curl exited {code}`, or
|
|
`curl was killed` when there is no code. The body, if any, is not shown.
|
|
6. Waiting failed → `tool_error(format!("http_fetch: cannot wait for curl: {e}"))`.
|
|
|
|
## About the given tests
|
|
|
|
`fetch.rs` writes small shell scripts that stand in for `curl` and runs them. Every test that
|
|
starts a process takes a lock first (`let _serial = serial();`). Without it, another test's fork
|
|
can hold a just-written script open, and running it fails with "Text file busy" (ETXTBSY) about
|
|
once in seven runs. If you add a test that writes and runs a script, take the lock too.
|
|
|
|
## Steps
|
|
|
|
- [ ] **1. Copy.** `git switch m3b`, then
|
|
`cp docs/plans/M3b/files/crates/toolkit/tests/fetch.rs crates/toolkit/tests/`
|
|
- [ ] **2. See it fail.** `cargo test -p toolkit --test fetch`. Expected: it does not compile.
|
|
- [ ] **3. Write `fetch.rs`** and the `lib.rs` changes. Run `cargo fmt --all`.
|
|
- [ ] **4. See it pass.** `cargo test -p toolkit --test fetch`. Expected: 7 passed. Run it ten
|
|
times; it must pass every time.
|
|
- [ ] **5. Run the gate.** `make gate`. Expected last line: `gate: ok`.
|
|
- [ ] **6. Log and commit.** `git add crates/toolkit docs/implementer-log.md Cargo.lock && git commit`
|
|
|
|
## Done when
|
|
|
|
- `cargo test -p toolkit --test fetch` reports 7 passed ten times running; `make gate` prints
|
|
`gate: ok`.
|
|
|
|
## Stop and report if
|
|
|
|
- A test fails with "Text file busy" even with the lock: report the run and the test.
|