boxmaker/docs/plans/M2a/04-loopd-http.md

M2a task 04: the HTTP client

Branch: m2a (run git switch m2a; git status --short must be empty, otherwise stop) Commit subject: Add a minimal HTTP/1.1 client over a Unix socket to loopd

Goal

Write the HTTP/1.1 client loopd uses to talk to inferproxy. It is hand-written so that loopd needs no HTTP crate. It supports exactly what llama-server sends and nothing more. It reads bytes from another process, so everything it reads is bounded and nothing it reads can make it panic.

This task also brings in the fake server and the recordings that every later task uses.

Context

One request per connection. The client sends Connection: close, reads the status line and headers, and then offers the body as a std::io::Read. A body arrives in one of three framings:

Framing	How the head says so	Where the body ends
chunked	Transfer-Encoding: chunked (any letter case)	after the zero-size chunk and its trailers
length	Content-Length: N, when not chunked	after exactly N bytes
close	neither	when the peer closes

Chunked looks like this on the wire. Sizes are hexadecimal, may be followed by ;extension, and each chunk's data is followed by \r\n:

5;ext=1\r\nhello\r\n1\r\n,\r\n6\r\n world\r\n0\r\nX-Trailer: t\r\n\r\n

The real server's head has two oddities, both in the recordings: Connection: close appears twice, and Access-Control-Allow-Origin: has an empty value.

The caller sets a read timeout and may call read_head again after a timeout. That is how the next tasks wait for a busy server: a timeout with nothing received means "not started yet".

Files

• Copy: crates/loopd/tests/http.rs, crates/loopd/tests/support/mod.rs, crates/loopd/tests/fixtures/http/ (13 files)
• Create: crates/loopd/src/http.rs
• Modify: crates/loopd/src/lib.rs, docs/implementer-log.md

Interfaces

Produces, in crates/loopd/src/http.rs:

pub const MAX_HEAD: usize = 16 * 1024;      // status line plus headers

#[derive(Debug)]
pub enum HttpError {
    Connect(std::io::Error),                // the socket could not be opened
    Io(std::io::Error),
    Timeout,                                // the read timeout passed
    Closed,                                 // the peer closed before the head, or the body, was complete
    TooLarge(&'static str),                 // names what was too large
    Malformed(&'static str),                // names what was malformed
}                                           // Display and std::error::Error

/// True for the two error kinds a read timeout produces: WouldBlock and TimedOut.
pub fn is_timeout(e: &std::io::Error) -> bool;

pub struct Request<'a> { pub method: &'a str, pub path: &'a str, pub body: Option<&'a [u8]> }

#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Head { pub status: u16, pub headers: Vec<(String, String)> }   // names lowercased; order and duplicates kept
impl Head { pub fn header(&self, name: &str) -> Option<&str>; }           // first match; `name` is lowercase

pub struct Connection { /* private: the UnixStream, a buffer of unread bytes, a flag */ }
impl Connection {
    pub fn open(socket: &std::path::Path) -> Result<Self, HttpError>;
    pub fn send(&mut self, req: &Request<'_>) -> Result<(), HttpError>;
    pub fn set_read_timeout(&self, timeout: std::time::Duration) -> Result<(), HttpError>;
    pub fn received_any(&self) -> bool;     // true once one byte of the response has arrived
    pub fn read_head(&mut self) -> Result<Head, HttpError>;
    pub fn body(&mut self, head: &Head) -> Result<Body<'_>, HttpError>;
}

pub struct Body<'a> { /* private */ }        // impl std::io::Read

/// Reads a whole body, refusing to hold more than `cap` bytes.
pub fn read_capped(body: &mut impl std::io::Read, cap: usize) -> Result<Vec<u8>, HttpError>;

Rules the tests check:

1. send writes <METHOD> <path> HTTP/1.1, then Host: localhost, Connection: close, Accept: */*, and, only when there is a body, Content-Type: application/json and Content-Length. Then a blank line, then the body. Line endings are \r\n.
2. read_head reads until \r\n\r\n. Bytes that arrive after the head stay buffered for the body. On a read timeout it returns Timeout and keeps what has arrived, so calling it again continues. If the peer closes first: Closed. More than MAX_HEAD bytes without the end of the head: TooLarge. The status line must be HTTP/1.1 or HTTP/1.0, a space, and a number from 100 to 599; a header line needs a colon and a non-empty name without spaces; else Malformed. Header values are trimmed.
3. Body::read follows the table above. The result must not depend on how the bytes arrive: one byte at a time gives the same body as all at once.
   • A peer that closes early is ErrorKind::UnexpectedEof in the chunked and length framings.
   • A chunk-size line that is not hex, that is longer than 64 bytes, or chunk data not followed by \r\n, is ErrorKind::InvalidData. A size too big for usize is not hex as far as we care.
   • A Content-Length that is not a non-negative number makes body() return Malformed.
   • A read timeout passes through as the io::Error it is.
4. read_capped maps: timeout (is_timeout) to Timeout, UnexpectedEof to Closed, more than cap bytes to TooLarge, anything else to Io.
5. No unwrap, expect, slice indexing that can go out of range, or as cast on a value that came from the peer. Use get(..), checked_*, try_from, usize::from_str_radix.

The fake server (given; read it, do not edit it)

tests/support/mod.rs. FakeServer::start() listens on a Unix socket in a temporary directory. server.route(path, vec![reply, …]) sets the replies for a path; they are served in order and the last one repeats. Reply::fixture("props") replays tests/fixtures/http/props.http, a response recorded from the real server, byte for byte. A reply can be shaped: .head_delay(ms), .trickle(piece_bytes, delay_ms), .cut_after(bytes), .hang_after(bytes, ms). server.requests() returns what the client sent.

Steps

• 1. Copy.

git switch m2a
cp docs/plans/M2a/files/crates/loopd/tests/http.rs crates/loopd/tests/
cp -r docs/plans/M2a/files/crates/loopd/tests/support crates/loopd/tests/
cp -r docs/plans/M2a/files/crates/loopd/tests/fixtures/http crates/loopd/tests/fixtures/

Look at one recording: head -c 600 crates/loopd/tests/fixtures/http/props.http.

• 2. See the test fail. cargo test -p loopd --test http. Expected: it does not compile.
• 3. Write http.rs, and add pub mod http; to lib.rs. Run cargo fmt --all.
• 4. See the test pass. cargo test -p loopd --test http. Expected: 15 passed. It takes a few seconds, because one test trickles a whole response a byte at a time.
• 5. Check the "everywhere" rule yourself. Find every place http.rs reads from the socket or takes a number from the peer, and confirm each is bounded and cannot panic. List them in your log row.
• 6. Run the gate. make gate. Expected last line: gate: ok.
• 7. Log and commit.

git add crates/loopd docs/implementer-log.md
git commit

Done when

• cargo test -p loopd --test http reports 15 passed; make gate prints gate: ok.
• diff -r crates/loopd/tests/support docs/plans/M2a/files/crates/loopd/tests/support and the same for fixtures/http print nothing.

Stop and report if

• A recording cannot be read correctly by a client that follows the rules above.