kyleandClaude Fable 5.1
76ccc251cd
The tasks build the inference path: emsha-backed SHA-256, inferproxy, config, a hand-written HTTP and SSE client, request building, delta assembly, the chat state machine, the thinking cap, the slot gate with retry, the startup self-test and on-device verification. Everything the tasks copy in was checked against a private reference implementation: the gate passes after each task in order, the timing tests pass repeatedly under CPU load, and the reference passes the self-test and all four device checks on straylight. Expected results for the recorded streams were derived by a separate script. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
99 lines
4.7 KiB
Markdown
99 lines
4.7 KiB
Markdown
# M2a task 08: small server calls and the cache-loss check
|
|
|
|
**Branch:** `m2a` (run `git switch m2a`; `git status --short` must be empty, otherwise stop)
|
|
**Commit subject:** `Add props, slots, tokenize and the cache-loss check to the llama client`
|
|
|
|
## Goal
|
|
|
|
Three small calls to the server that are not streams, and one pure function that says whether the
|
|
server's prompt cache was lost between two requests.
|
|
|
|
## Context
|
|
|
|
The server is a router for several models. For GET calls the model goes in the query
|
|
(`/props?model=<id>`); for POST calls it goes in the JSON body. Responses are the server's format:
|
|
**ignore unknown fields**. `/props` alone has dozens.
|
|
|
|
From the measurements: after each request the slot holds `cache_n + prompt_n + predicted_n`
|
|
tokens. If the next request of the same conversation reuses about that many (`cache_n`), the cache
|
|
held. A normal turn loses one to four tokens at the boundary. If it reuses far fewer, the cache was
|
|
lost: another client took the slot, the model was reloaded, or the prompt did not repeat exactly.
|
|
|
|
## Files
|
|
|
|
- Copy: `crates/loopd/tests/info.rs`
|
|
- Create: `crates/loopd/src/llama/info.rs`
|
|
- Modify: `crates/loopd/src/llama/mod.rs` (add `pub mod info;`), `docs/implementer-log.md`
|
|
|
|
## Interfaces
|
|
|
|
Consumes: `loopd::http::{Connection, Request, HttpError, read_capped}`, `Client`, `InferError`,
|
|
`Timings`.
|
|
|
|
Produces, in `crates/loopd/src/llama/info.rs`:
|
|
|
|
```rust
|
|
pub const MAX_BODY: usize = 4 * 1024 * 1024; // bodies that are not streams
|
|
pub const MAX_ERROR_BODY: usize = 4 * 1024; // how much of an error body is kept
|
|
pub const CACHE_TOLERANCE: u64 = 64;
|
|
|
|
#[derive(Debug, Clone, PartialEq, Eq)]
|
|
pub struct Props { pub chat_template: String, pub n_ctx: u64, pub total_slots: u32 }
|
|
|
|
#[derive(Debug, Clone, Copy, PartialEq, Eq, serde::Deserialize)]
|
|
pub struct SlotInfo { pub id: u32, pub is_processing: bool }
|
|
|
|
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
|
pub enum CacheOutcome { Hit, Loss { expected: u64, got: u64 } }
|
|
|
|
pub fn cache_outcome(previous: &Timings, current: &Timings) -> CacheOutcome;
|
|
|
|
impl Client {
|
|
pub fn props(&self) -> Result<Props, InferError>; // GET /props?model=<id>
|
|
pub fn slots(&self) -> Result<Vec<SlotInfo>, InferError>; // GET /slots?model=<id>
|
|
pub fn tokenize(&self, text: &str) -> Result<usize, InferError>; // POST /tokenize; the token count
|
|
}
|
|
|
|
// Used again by the next tasks, so make them pub(crate):
|
|
pub(crate) fn map_http(e: HttpError) -> InferError;
|
|
pub(crate) fn error_text(bytes: &[u8]) -> String;
|
|
impl Client { pub(crate) fn call(&self, method: &str, path: &str, body: Option<&[u8]>) -> Result<Vec<u8>, InferError>; }
|
|
```
|
|
|
|
Rules the tests check:
|
|
|
|
1. `call` is one whole exchange: open `cfg.infer.socket`, set the read timeout to
|
|
`cfg.limits.liveness_ms`, send, read the head, read the body with `read_capped(…, MAX_BODY)`.
|
|
A status other than 200 is `InferError::Http { status, body: error_text(&bytes) }`.
|
|
2. `map_http`: `Connect(e)` to `Connect(e)`; `Timeout` to `Stalled`; `Closed` to
|
|
`StreamClosedEarly`; everything else to `Protocol` with the error's message.
|
|
3. `error_text` keeps at most `MAX_ERROR_BODY` bytes and cuts on a character boundary (use
|
|
`String::from_utf8_lossy`, then `is_char_boundary`). It must not panic on a body whose 4,096th
|
|
byte is in the middle of a character; one test sends exactly that.
|
|
4. `props` reads three things from the response: `chat_template`, `total_slots`, and
|
|
`default_generation_settings.n_ctx` (the context of one slot).
|
|
5. `tokenize` sends `{"model": <id>, "content": <text>}` and returns the length of `tokens`.
|
|
6. A 200 response that does not have the expected shape is `Protocol`.
|
|
7. `cache_outcome`: `expected` is `previous.cache_n + previous.prompt_n + previous.predicted_n`.
|
|
It is a `Hit` when `current.cache_n + CACHE_TOLERANCE >= expected`, else
|
|
`Loss { expected, got: current.cache_n }`. 64 short is a hit, 65 short is a loss.
|
|
|
|
## Steps
|
|
|
|
- [ ] **1. Copy.** `git switch m2a`, then
|
|
`cp docs/plans/M2a/files/crates/loopd/tests/info.rs crates/loopd/tests/`
|
|
- [ ] **2. See the test fail.** `cargo test -p loopd --test info`. Expected: it does not compile.
|
|
- [ ] **3. Write `info.rs`** and register the module. Run `cargo fmt --all`.
|
|
- [ ] **4. See the test pass.** `cargo test -p loopd --test info`. Expected: `7 passed`.
|
|
- [ ] **5. Run the gate.** `make gate`. Expected last line: `gate: ok`. If clippy reports
|
|
`map_http`, `error_text` or `call` as unused, check that `props` really goes through `call`.
|
|
- [ ] **6. Log and commit.** `git add crates/loopd docs/implementer-log.md && git commit`
|
|
|
|
## Done when
|
|
|
|
- `cargo test -p loopd --test info` reports 7 passed; `make gate` prints `gate: ok`.
|
|
|
|
## Stop and report if
|
|
|
|
- The recorded `/props` response has no `default_generation_settings.n_ctx`.
|