boxmaker/docs/plans/M2a/09-llama-chat.md

# M2a task 09: one chat request, with its waits and its errors

**Branch:** `m2a` (run `git switch m2a`; `git status --short` must be empty, otherwise stop)
**Commit subject:** `Add Client::chat with the first-byte wait, liveness and error mapping`

## Goal

Send one chat request and return its `Completion`, or a precise error. This is the centre of M2a:
it decides what happens when the server is slow, busy, loading, silent or gone. There is no total
deadline anywhere; the only question asked is whether bytes are still arriving.

## Context

Measured on the real server: a request for a slot that another client is using receives **no
bytes at all**, not even headers, until the slot is free. So silence before the first byte means
"queued" or "dead", and only `GET /slots` can tell which. Once bytes flow, the longest silence seen
was 2.2 seconds.

The thinking cap and the slot gate are **not** part of this task; they are tasks 10 and 11.

## Files

- Copy: `crates/loopd/tests/chat.rs`
- Create: `crates/loopd/src/llama/chat.rs`
- Modify: `crates/loopd/src/llama/mod.rs` (add `pub mod chat;`), `docs/implementer-log.md`

## Interfaces

Consumes: `http::{Connection, Request, HttpError, Head, read_capped}`, `sse::{Events, SseItem,
SseError}`, `request::build_body`, `assemble::Assembler`, `info::{map_http, error_text, MAX_BODY}`,
`Client::slots`.

Produces, in `crates/loopd/src/llama/chat.rs`:

```rust
pub const MAX_SSE_LINE: usize = 1024 * 1024;

impl Client {
    pub fn chat(&self, req: &ChatRequest, on_event: &mut dyn FnMut(&ChatEvent)) -> Result<Completion, InferError>;
}
```

## What `chat` does

All limits are fields of `self.cfg.limits`.

**1. Send.** Build the body (`build_body`; an error is `Protocol`). Open a connection (`map_http`
turns a failure into `Connect`). Send `POST /v1/chat/completions`.

**2. Wait for the head.** Set the read timeout to `poll_ms` and call `read_head` in a loop.

- `Ok(head)`: go to step 3.
- `Err(HttpError::Timeout)`: handle it as below, then call `read_head` again.
- any other error: return `map_http(e)`. (`Closed` becomes `StreamClosedEarly`.)

On each timeout, work out what state the wait is in:

| State | When |
|---|---|
| `Idle` | `conn.received_any()` is true (the head has started to arrive; no poll is needed) |
| `Busy` | nothing received, `self.slots()` succeeded, and the entry whose `id == req.slot` has `is_processing: true` |
| `Idle` | nothing received, `self.slots()` succeeded, and that entry is not processing, or is missing |
| `Unavailable` | nothing received and `self.slots()` failed for any reason (the server is loading the model, or restarting) |

When a poll was made, emit `ChatEvent::Waiting { slot_busy }` (true only for `Busy`). Keep how long
the **same state** has lasted without a break: add `poll_ms` when the state equals the previous
one, and start again at `poll_ms` when it differs. Then:

| State | It has lasted at least | Return |
|---|---|---|
| `Busy` | `busy_wait_ms` | `WaitTimeout` |
| `Unavailable` | `load_wait_ms` | `LoadTimeout` |
| `Idle` | `idle_grace_ms` | `Stalled` |

**3. Status.** Set the read timeout to `liveness_ms`. If `head.status != 200`: read the body with
`read_capped(…, MAX_BODY)` and return `Http { status, body: error_text(&bytes) }`.

**4. Stream.** Wrap the body in `Events::new(body, MAX_SSE_LINE)` and make an `Assembler`. Loop on
`next_item()`:

| `next_item()` gives | Do |
|---|---|
| `Ok(Some(SseItem::Data(text)))` | `assembler.push(&text)?`, and pass each event to `on_event` |
| `Ok(Some(SseItem::Done))` or `Ok(None)` | leave the loop |
| `Err(SseError::Timeout)` | return `Stalled` |
| `Err(SseError::Truncated)` | return `StreamClosedEarly` |
| any other `Err(e)` | return `Protocol(e.to_string())` |

**5. Finish.** `assembler.finish(false)`. It returns `StreamClosedEarly` by itself when the stream
ended without a `finish_reason`, which is what a server that dies between two events looks like.

Returning early drops the connection. That is deliberate: the real server stops generating when
the client goes away.

## Steps

- [ ] **1. Copy.** `git switch m2a`, then
  `cp docs/plans/M2a/files/crates/loopd/tests/chat.rs crates/loopd/tests/`.
  Read the test names; each is one row of the tables above.
- [ ] **2. See the test fail.** `cargo test -p loopd --test chat`. Expected: it does not compile.
- [ ] **3. Write `chat.rs`** and register the module. Keep the head wait in its own function. Run
  `cargo fmt --all`.
- [ ] **4. See the test pass.** `cargo test -p loopd --test chat`. Expected: `13 passed`, in about a
  second. Run it five times in a row: timing tests must pass every time.
- [ ] **5. Check the tables yourself.** For each row of the three tables above, name the test in
  `chat.rs` that covers it. Put the list in your log row. A row without a test is worth a note.
- [ ] **6. Run the gate.** `make gate`. Expected last line: `gate: ok`.
- [ ] **7. Log and commit.** `git add crates/loopd docs/implementer-log.md && git commit`

## Done when

- `cargo test -p loopd --test chat` reports 13 passed, five runs in a row; `make gate` prints
  `gate: ok`.
- `chat.rs` contains no `std::thread`, no `sleep` and no `Instant`: the waits are read timeouts.

## Stop and report if

- A test only passes with a sleep or a thread in `chat.rs`.