Add M2a plan: thirteen tasks, tests, fake server and recordings
The tasks build the inference path: emsha-backed SHA-256, inferproxy, config, a hand-written HTTP and SSE client, request building, delta assembly, the chat state machine, the thinking cap, the slot gate with retry, the startup self-test and on-device verification. Everything the tasks copy in was checked against a private reference implementation: the gate passes after each task in order, the timing tests pass repeatedly under CPU load, and the reference passes the self-test and all four device checks on straylight. Expected results for the recorded streams were derived by a separate script. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,118 @@
|
||||
# M2a task 09: one chat request, with its waits and its errors
|
||||
|
||||
**Branch:** `m2a` (run `git switch m2a`; `git status --short` must be empty, otherwise stop)
|
||||
**Commit subject:** `Add Client::chat with the first-byte wait, liveness and error mapping`
|
||||
|
||||
## Goal
|
||||
|
||||
Send one chat request and return its `Completion`, or a precise error. This is the centre of M2a:
|
||||
it decides what happens when the server is slow, busy, loading, silent or gone. There is no total
|
||||
deadline anywhere; the only question asked is whether bytes are still arriving.
|
||||
|
||||
## Context
|
||||
|
||||
Measured on the real server: a request for a slot that another client is using receives **no
|
||||
bytes at all**, not even headers, until the slot is free. So silence before the first byte means
|
||||
"queued" or "dead", and only `GET /slots` can tell which. Once bytes flow, the longest silence seen
|
||||
was 2.2 seconds.
|
||||
|
||||
The thinking cap and the slot gate are **not** part of this task; they are tasks 10 and 11.
|
||||
|
||||
## Files
|
||||
|
||||
- Copy: `crates/loopd/tests/chat.rs`
|
||||
- Create: `crates/loopd/src/llama/chat.rs`
|
||||
- Modify: `crates/loopd/src/llama/mod.rs` (add `pub mod chat;`), `docs/implementer-log.md`
|
||||
|
||||
## Interfaces
|
||||
|
||||
Consumes: `http::{Connection, Request, HttpError, Head, read_capped}`, `sse::{Events, SseItem,
|
||||
SseError}`, `request::build_body`, `assemble::Assembler`, `info::{map_http, error_text, MAX_BODY}`,
|
||||
`Client::slots`.
|
||||
|
||||
Produces, in `crates/loopd/src/llama/chat.rs`:
|
||||
|
||||
```rust
|
||||
pub const MAX_SSE_LINE: usize = 1024 * 1024;
|
||||
|
||||
impl Client {
|
||||
pub fn chat(&self, req: &ChatRequest, on_event: &mut dyn FnMut(&ChatEvent)) -> Result<Completion, InferError>;
|
||||
}
|
||||
```
|
||||
|
||||
## What `chat` does
|
||||
|
||||
All limits are fields of `self.cfg.limits`.
|
||||
|
||||
**1. Send.** Build the body (`build_body`; an error is `Protocol`). Open a connection (`map_http`
|
||||
turns a failure into `Connect`). Send `POST /v1/chat/completions`.
|
||||
|
||||
**2. Wait for the head.** Set the read timeout to `poll_ms` and call `read_head` in a loop.
|
||||
|
||||
- `Ok(head)`: go to step 3.
|
||||
- `Err(HttpError::Timeout)`: handle it as below, then call `read_head` again.
|
||||
- any other error: return `map_http(e)`. (`Closed` becomes `StreamClosedEarly`.)
|
||||
|
||||
On each timeout, work out what state the wait is in:
|
||||
|
||||
| State | When |
|
||||
|---|---|
|
||||
| `Idle` | `conn.received_any()` is true (the head has started to arrive; no poll is needed) |
|
||||
| `Busy` | nothing received, `self.slots()` succeeded, and the entry whose `id == req.slot` has `is_processing: true` |
|
||||
| `Idle` | nothing received, `self.slots()` succeeded, and that entry is not processing, or is missing |
|
||||
| `Unavailable` | nothing received and `self.slots()` failed for any reason (the server is loading the model, or restarting) |
|
||||
|
||||
When a poll was made, emit `ChatEvent::Waiting { slot_busy }` (true only for `Busy`). Keep how long
|
||||
the **same state** has lasted without a break: add `poll_ms` when the state equals the previous
|
||||
one, and start again at `poll_ms` when it differs. Then:
|
||||
|
||||
| State | It has lasted at least | Return |
|
||||
|---|---|---|
|
||||
| `Busy` | `busy_wait_ms` | `WaitTimeout` |
|
||||
| `Unavailable` | `load_wait_ms` | `LoadTimeout` |
|
||||
| `Idle` | `idle_grace_ms` | `Stalled` |
|
||||
|
||||
**3. Status.** Set the read timeout to `liveness_ms`. If `head.status != 200`: read the body with
|
||||
`read_capped(…, MAX_BODY)` and return `Http { status, body: error_text(&bytes) }`.
|
||||
|
||||
**4. Stream.** Wrap the body in `Events::new(body, MAX_SSE_LINE)` and make an `Assembler`. Loop on
|
||||
`next_item()`:
|
||||
|
||||
| `next_item()` gives | Do |
|
||||
|---|---|
|
||||
| `Ok(Some(SseItem::Data(text)))` | `assembler.push(&text)?`, and pass each event to `on_event` |
|
||||
| `Ok(Some(SseItem::Done))` or `Ok(None)` | leave the loop |
|
||||
| `Err(SseError::Timeout)` | return `Stalled` |
|
||||
| `Err(SseError::Truncated)` | return `StreamClosedEarly` |
|
||||
| any other `Err(e)` | return `Protocol(e.to_string())` |
|
||||
|
||||
**5. Finish.** `assembler.finish(false)`. It returns `StreamClosedEarly` by itself when the stream
|
||||
ended without a `finish_reason`, which is what a server that dies between two events looks like.
|
||||
|
||||
Returning early drops the connection. That is deliberate: the real server stops generating when
|
||||
the client goes away.
|
||||
|
||||
## Steps
|
||||
|
||||
- [ ] **1. Copy.** `git switch m2a`, then
|
||||
`cp docs/plans/M2a/files/crates/loopd/tests/chat.rs crates/loopd/tests/`.
|
||||
Read the test names; each is one row of the tables above.
|
||||
- [ ] **2. See the test fail.** `cargo test -p loopd --test chat`. Expected: it does not compile.
|
||||
- [ ] **3. Write `chat.rs`** and register the module. Keep the head wait in its own function. Run
|
||||
`cargo fmt --all`.
|
||||
- [ ] **4. See the test pass.** `cargo test -p loopd --test chat`. Expected: `13 passed`, in about a
|
||||
second. Run it five times in a row: timing tests must pass every time.
|
||||
- [ ] **5. Check the tables yourself.** For each row of the three tables above, name the test in
|
||||
`chat.rs` that covers it. Put the list in your log row. A row without a test is worth a note.
|
||||
- [ ] **6. Run the gate.** `make gate`. Expected last line: `gate: ok`.
|
||||
- [ ] **7. Log and commit.** `git add crates/loopd docs/implementer-log.md && git commit`
|
||||
|
||||
## Done when
|
||||
|
||||
- `cargo test -p loopd --test chat` reports 13 passed, five runs in a row; `make gate` prints
|
||||
`gate: ok`.
|
||||
- `chat.rs` contains no `std::thread`, no `sleep` and no `Instant`: the waits are read timeouts.
|
||||
|
||||
## Stop and report if
|
||||
|
||||
- A test only passes with a sleep or a thread in `chat.rs`.
|
||||
Reference in New Issue
Block a user