boxmaker/docs/plans/M2a/08-llama-info.md

M2a task 08: small server calls and the cache-loss check

Branch: m2a (run git switch m2a; git status --short must be empty, otherwise stop) Commit subject: Add props, slots, tokenize and the cache-loss check to the llama client

Goal

Three small calls to the server that are not streams, and one pure function that says whether the server's prompt cache was lost between two requests.

Context

The server is a router for several models. For GET calls the model goes in the query (/props?model=<id>); for POST calls it goes in the JSON body. Responses are the server's format: ignore unknown fields. /props alone has dozens.

From the measurements: after each request the slot holds cache_n + prompt_n + predicted_n tokens. If the next request of the same conversation reuses about that many (cache_n), the cache held. A normal turn loses one to four tokens at the boundary. If it reuses far fewer, the cache was lost: another client took the slot, the model was reloaded, or the prompt did not repeat exactly.

Files

• Copy: crates/loopd/tests/info.rs
• Create: crates/loopd/src/llama/info.rs
• Modify: crates/loopd/src/llama/mod.rs (add pub mod info;), docs/implementer-log.md

Interfaces

Consumes: loopd::http::{Connection, Request, HttpError, read_capped}, Client, InferError, Timings.

Produces, in crates/loopd/src/llama/info.rs:

pub const MAX_BODY: usize = 4 * 1024 * 1024;   // bodies that are not streams
pub const MAX_ERROR_BODY: usize = 4 * 1024;    // how much of an error body is kept
pub const CACHE_TOLERANCE: u64 = 64;

#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Props { pub chat_template: String, pub n_ctx: u64, pub total_slots: u32 }

#[derive(Debug, Clone, Copy, PartialEq, Eq, serde::Deserialize)]
pub struct SlotInfo { pub id: u32, pub is_processing: bool }

#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum CacheOutcome { Hit, Loss { expected: u64, got: u64 } }

pub fn cache_outcome(previous: &Timings, current: &Timings) -> CacheOutcome;

impl Client {
    pub fn props(&self) -> Result<Props, InferError>;         // GET /props?model=<id>
    pub fn slots(&self) -> Result<Vec<SlotInfo>, InferError>; // GET /slots?model=<id>
    pub fn tokenize(&self, text: &str) -> Result<usize, InferError>;   // POST /tokenize; the token count
}

// Used again by the next tasks, so make them pub(crate):
pub(crate) fn map_http(e: HttpError) -> InferError;
pub(crate) fn error_text(bytes: &[u8]) -> String;
impl Client { pub(crate) fn call(&self, method: &str, path: &str, body: Option<&[u8]>) -> Result<Vec<u8>, InferError>; }

Rules the tests check:

1. call is one whole exchange: open cfg.infer.socket, set the read timeout to cfg.limits.liveness_ms, send, read the head, read the body with read_capped(…, MAX_BODY). A status other than 200 is InferError::Http { status, body: error_text(&bytes) }.
2. map_http: Connect(e) to Connect(e); Timeout to Stalled; Closed to StreamClosedEarly; everything else to Protocol with the error's message.
3. error_text keeps at most MAX_ERROR_BODY bytes and cuts on a character boundary (use String::from_utf8_lossy, then is_char_boundary). It must not panic on a body whose 4,096th byte is in the middle of a character; one test sends exactly that.
4. props reads three things from the response: chat_template, total_slots, and default_generation_settings.n_ctx (the context of one slot).
5. tokenize sends {"model": <id>, "content": <text>} and returns the length of tokens.
6. A 200 response that does not have the expected shape is Protocol.
7. cache_outcome: expected is previous.cache_n + previous.prompt_n + previous.predicted_n. It is a Hit when current.cache_n + CACHE_TOLERANCE >= expected, else Loss { expected, got: current.cache_n }. 64 short is a hit, 65 short is a loss.

Steps

• 1. Copy. git switch m2a, then cp docs/plans/M2a/files/crates/loopd/tests/info.rs crates/loopd/tests/
• 2. See the test fail. cargo test -p loopd --test info. Expected: it does not compile.
• 3. Write info.rs and register the module. Run cargo fmt --all.
• 4. See the test pass. cargo test -p loopd --test info. Expected: 7 passed.
• 5. Run the gate. make gate. Expected last line: gate: ok. If clippy reports map_http, error_text or call as unused, check that props really goes through call.
• 6. Log and commit. git add crates/loopd docs/implementer-log.md && git commit

Done when

• cargo test -p loopd --test info reports 7 passed; make gate prints gate: ok.

Stop and report if

• The recorded /props response has no default_generation_settings.n_ctx.