boxmaker/docs/plans/M2a/09-llama-chat.md

M2a task 09: one chat request, with its waits and its errors

Branch: m2a (run git switch m2a; git status --short must be empty, otherwise stop) Commit subject: Add Client::chat with the first-byte wait, liveness and error mapping

Goal

Send one chat request and return its Completion, or a precise error. This is the centre of M2a: it decides what happens when the server is slow, busy, loading, silent or gone. There is no total deadline anywhere; the only question asked is whether bytes are still arriving.

Context

Measured on the real server: a request for a slot that another client is using receives no bytes at all, not even headers, until the slot is free. So silence before the first byte means "queued" or "dead", and only GET /slots can tell which. Once bytes flow, the longest silence seen was 2.2 seconds.

The thinking cap and the slot gate are not part of this task; they are tasks 10 and 11.

Files

• Copy: crates/loopd/tests/chat.rs
• Create: crates/loopd/src/llama/chat.rs
• Modify: crates/loopd/src/llama/mod.rs (add pub mod chat;), docs/implementer-log.md

Interfaces

Consumes: http::{Connection, Request, HttpError, Head, read_capped}, sse::{Events, SseItem, SseError}, request::build_body, assemble::Assembler, info::{map_http, error_text, MAX_BODY}, Client::slots.

Produces, in crates/loopd/src/llama/chat.rs:

pub const MAX_SSE_LINE: usize = 1024 * 1024;

impl Client {
    pub fn chat(&self, req: &ChatRequest, on_event: &mut dyn FnMut(&ChatEvent)) -> Result<Completion, InferError>;
}

What chat does

All limits are fields of self.cfg.limits.

1. Send. Build the body (build_body; an error is Protocol). Open a connection (map_http turns a failure into Connect). Send POST /v1/chat/completions.

2. Wait for the head. Set the read timeout to poll_ms and call read_head in a loop.

• Ok(head): go to step 3.
• Err(HttpError::Timeout): handle it as below, then call read_head again.
• any other error: return map_http(e). (Closed becomes StreamClosedEarly.)

On each timeout, work out what state the wait is in:

State	When
Idle	conn.received_any() is true (the head has started to arrive; no poll is needed)
Busy	nothing received, self.slots() succeeded, and the entry whose id == req.slot has is_processing: true
Idle	nothing received, self.slots() succeeded, and that entry is not processing, or is missing
Unavailable	nothing received and self.slots() failed for any reason (the server is loading the model, or restarting)

When a poll was made, emit ChatEvent::Waiting { slot_busy } (true only for Busy). Keep how long the same state has lasted without a break: add poll_ms when the state equals the previous one, and start again at poll_ms when it differs. Then:

State	It has lasted at least	Return
Busy	busy_wait_ms	WaitTimeout
Unavailable	load_wait_ms	LoadTimeout
Idle	idle_grace_ms	Stalled

3. Status. Set the read timeout to liveness_ms. If head.status != 200: read the body with read_capped(…, MAX_BODY) and return Http { status, body: error_text(&bytes) }.

4. Stream. Wrap the body in Events::new(body, MAX_SSE_LINE) and make an Assembler. Loop on next_item():

next_item() gives	Do
Ok(Some(SseItem::Data(text)))	assembler.push(&text)?, and pass each event to on_event
Ok(Some(SseItem::Done)) or Ok(None)	leave the loop
Err(SseError::Timeout)	return Stalled
Err(SseError::Truncated)	return StreamClosedEarly
any other Err(e)	return Protocol(e.to_string())

5. Finish. assembler.finish(false). It returns StreamClosedEarly by itself when the stream ended without a finish_reason, which is what a server that dies between two events looks like.

Returning early drops the connection. That is deliberate: the real server stops generating when the client goes away.

Steps

• 1. Copy. git switch m2a, then cp docs/plans/M2a/files/crates/loopd/tests/chat.rs crates/loopd/tests/. Read the test names; each is one row of the tables above.
• 2. See the test fail. cargo test -p loopd --test chat. Expected: it does not compile.
• 3. Write chat.rs and register the module. Keep the head wait in its own function. Run cargo fmt --all.
• 4. See the test pass. cargo test -p loopd --test chat. Expected: 13 passed, in about a second. Run it five times in a row: timing tests must pass every time.
• 5. Check the tables yourself. For each row of the three tables above, name the test in chat.rs that covers it. Put the list in your log row. A row without a test is worth a note.
• 6. Run the gate. make gate. Expected last line: gate: ok.
• 7. Log and commit. git add crates/loopd docs/implementer-log.md && git commit

Done when

• cargo test -p loopd --test chat reports 13 passed, five runs in a row; make gate prints gate: ok.
• chat.rs contains no std::thread, no sleep and no Instant: the waits are read timeouts.

Stop and report if

• A test only passes with a sleep or a thread in chat.rs.