kyleandClaude Opus 5.5
b426ca1958
A draft spec for the owner's review and 13 offline tasks with their given tests: shared tool arguments and host rules in proto, the sealed fetch target (M3a finding 14), the toolkit tools and SOCKS5 egress proxy, and brokerd's [runner], podman argument lists, runtime and proxy lifecycle. Each task's tests were run against a reference at that task's end state (560 to 638 tests, clippy clean); the reference is not in the repository. Adds the runner-unavailable runbook entry and tip T23 (ETXTBSY in script tests). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
92 lines
4.1 KiB
Markdown
92 lines
4.1 KiB
Markdown
# M3b task 06: `toolkit http_fetch`
|
|
|
|
**Branch:** `m3b` (run `git switch m3b`; `git status --short` must be empty, otherwise stop)
|
|
**Commit subject:** `toolkit: http_fetch through curl`
|
|
|
|
## Goal
|
|
|
|
`toolkit http_fetch` runs `/bin/curl` with a **fixed** argument list and the URL, through the egress
|
|
proxy's socket that `brokerd` mounts at `/run/egress/egress.sock` for this call. It prints the body
|
|
and a status line, or one line saying why the fetch failed. We do not write an HTTP or TLS client:
|
|
`curl` does that, inside the container. Spec sections 4 and 6.
|
|
|
|
## Files
|
|
|
|
- Copy: `crates/toolkit/tests/fetch.rs`
|
|
- Create: `crates/toolkit/src/fetch.rs`
|
|
- Modify: `crates/toolkit/src/lib.rs`, `docs/implementer-log.md`
|
|
|
|
## Interfaces
|
|
|
|
```rust
|
|
pub const CURL: &str = "/bin/curl";
|
|
pub const PROXY: &str = "socks5h://localhost/run/egress/egress.sock";
|
|
pub const CA_BUNDLE: &str = "/etc/ssl/certs/ca-certificates.crt";
|
|
|
|
/// `curl`'s arguments for `url`, in order, without the program name.
|
|
pub fn curl_args(url: &str) -> Vec<String>;
|
|
pub fn fetch(args: &proto::tools::HttpFetchArgs) -> crate::Outcome; // fetch_with(Path::new(CURL), args)
|
|
/// `fetch` with another `curl`, for tests.
|
|
pub fn fetch_with(curl: &std::path::Path, args: &proto::tools::HttpFetchArgs) -> crate::Outcome;
|
|
```
|
|
|
|
In `lib.rs`: `pub mod fetch;`, and an `"http_fetch"` arm in `run` like the others, calling
|
|
`fetch::fetch(&args)`.
|
|
|
|
## `curl_args`
|
|
|
|
Exactly these 21 strings, in this order (the test compares them one by one):
|
|
|
|
```
|
|
--silent --show-error --proto =https --proto-redir =https --location --max-redirs 5
|
|
--max-time 50 --max-filesize 8388608 --cacert <CA_BUNDLE> --proxy <PROXY>
|
|
--write-out "\n[http %{response_code}]" --url <url>
|
|
```
|
|
|
|
In Rust, `--write-out`'s value is the literal `"\n[http %{response_code}]"`: a real newline
|
|
character, then `[http %{response_code}]` (curl fills in the status). `--url <url>` (not a bare
|
|
URL) keeps a URL from ever being read as an option.
|
|
|
|
## `fetch_with`: every exit
|
|
|
|
1. Spawn `curl` with `curl_args(&args.url)`, standard input `Stdio::null()`, standard output and
|
|
standard error piped. A failure →
|
|
`Outcome::tool_error(format!("http_fetch: cannot start {}: {e}", curl.display()))`.
|
|
2. Read standard error **on its own thread**, keeping at most 64 KiB, while the main thread reads
|
|
standard output to the end. (Reading one and then the other can deadlock when both are full:
|
|
the given test writes 300,000 bytes to each.)
|
|
3. Wait for `curl`, then join the thread.
|
|
4. `curl` succeeded → `Outcome::done(body)`, the body decoded with `from_utf8_lossy`. The status
|
|
line is already in it, from `--write-out`.
|
|
5. It failed → `Outcome::tool_error(format!("http_fetch: {url}: {why}"))`, where `why` is the first
|
|
line of standard error that is not blank, trimmed; if there is none, `curl exited {code}`, or
|
|
`curl was killed` when there is no code. The body, if any, is not shown.
|
|
6. Waiting failed → `tool_error(format!("http_fetch: cannot wait for curl: {e}"))`.
|
|
|
|
## About the given tests
|
|
|
|
`fetch.rs` writes small shell scripts that stand in for `curl` and runs them. Every test that
|
|
starts a process takes a lock first (`let _serial = serial();`). Without it, another test's fork
|
|
can hold a just-written script open, and running it fails with "Text file busy" (ETXTBSY) about
|
|
once in seven runs. If you add a test that writes and runs a script, take the lock too.
|
|
|
|
## Steps
|
|
|
|
- [ ] **1. Copy.** `git switch m3b`, then
|
|
`cp docs/plans/M3b/files/crates/toolkit/tests/fetch.rs crates/toolkit/tests/`
|
|
- [ ] **2. See it fail.** `cargo test -p toolkit --test fetch`. Expected: it does not compile.
|
|
- [ ] **3. Write `fetch.rs`** and the `lib.rs` changes. Run `cargo fmt --all`.
|
|
- [ ] **4. See it pass.** `cargo test -p toolkit --test fetch`. Expected: 7 passed. Run it ten
|
|
times; it must pass every time.
|
|
- [ ] **5. Run the gate.** `make gate`. Expected last line: `gate: ok`.
|
|
- [ ] **6. Log and commit.** `git add crates/toolkit docs/implementer-log.md && git commit`
|
|
|
|
## Done when
|
|
|
|
- `cargo test -p toolkit --test fetch` reports 7 passed ten times running; `make gate` prints
|
|
`gate: ok`.
|
|
|
|
## Stop and report if
|
|
|
|
- A test fails with "Text file busy" even with the lock: report the run and the test.
|