boxmaker/docs/plans/M2a/06-llama-request.md

M2a task 06: the llama client's types, and the request body

Branch: m2a (run git switch m2a; git status --short must be empty, otherwise stop) Commit subject: Add the llama client types and the chat request builder

Goal

Define what goes into a chat request and what comes back, and build the JSON body the server expects. The streaming and the waiting come in later tasks; this one has no I/O.

Context

The server speaks the OpenAI chat-completions format with some fields of its own. Two facts from the project's measurements shape this task:

• The server caches the prompt. A request only gets that cache if it repeats the earlier messages exactly, so an assistant message is sent back with the same three fields it arrived with.
• tools is rendered at the very top of the prompt. It is sent only when there are tools.

Files

• Copy: crates/loopd/tests/request.rs
• Create: crates/loopd/src/llama/mod.rs, crates/loopd/src/llama/request.rs
• Modify: crates/loopd/src/lib.rs, docs/implementer-log.md

Interfaces

Produces, in crates/loopd/src/llama/mod.rs. Copy these definitions as they are; later tasks and their tests depend on every name.

pub mod request;

#[derive(Debug, Clone, PartialEq, Eq)]
pub enum ChatMessage {
    System { content: String },
    User { content: String },
    Assistant { content: Option<String>, reasoning_content: Option<String>, tool_calls: Vec<proto::ToolCall> },
    Tool { tool_call_id: String, content: String },
}

#[derive(Debug, Clone, PartialEq)]
pub struct ToolSchema { pub name: String, pub description: String, pub parameters: serde_json::Value }

#[derive(Debug, Clone, PartialEq)]
pub struct ChatRequest { pub slot: u32, pub messages: Vec<ChatMessage>, pub tools: Vec<ToolSchema>, pub thinking: bool }

#[derive(Debug, Clone, PartialEq, Eq)]
pub enum ChatEvent {
    Queued { ahead: usize },
    Waiting { slot_busy: bool },
    Progress { total: u64, cache: u64, processed: u64 },
    Reasoning(String),
    Content(String),
    ToolCallDelta { index: u32, id: Option<String>, name: Option<String>, arguments: String },
    ThinkingCapped { tokens: u64 },
    Retrying { attempt: u32, after_ms: u64, error: String },
}

#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum FinishReason { Stop, ToolCalls, Length }

/// The three counts loopd uses. The server sends more fields beside them; they are ignored.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Default, serde::Deserialize)]
pub struct Timings { pub cache_n: u64, pub prompt_n: u64, pub predicted_n: u64 }

#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Completion {
    pub id: String,
    pub content: Option<String>,
    pub reasoning_content: Option<String>,
    pub tool_calls: Vec<proto::ToolCall>,
    pub finish_reason: FinishReason,
    pub timings: Timings,
    pub reasoning_tokens: u64,
    pub thinking_capped: bool,
}

#[derive(Debug)]
pub enum InferError {
    Busy, Connect(std::io::Error), WaitTimeout, LoadTimeout, Stalled, StreamClosedEarly,
    ThinkingOverrun, Http { status: u16, body: String }, Protocol(String),
}   // Display (one short sentence per variant) and std::error::Error

pub struct Client { pub(crate) cfg: crate::config::Config }
impl Client {
    pub fn new(cfg: crate::config::Config) -> Self;
    pub fn config(&self) -> &crate::config::Config;
}

Timings parses the server's JSON, so it must not have deny_unknown_fields: the server sends six more timing fields beside these three.

And in crates/loopd/src/llama/request.rs:

/// The request body as JSON text. The same inputs always give the same bytes.
pub fn build_body(cfg: &crate::config::Config, req: &super::ChatRequest) -> Result<String, serde_json::Error>;

The body, for a request with one tool and thinking on:

{"model":"<cfg.infer.model>","messages":[…],"tools":[…],"id_slot":<req.slot>,"cache_prompt":true,
 "stream":true,"return_progress":true,"timings_per_token":true,"reasoning_control":true,
 "max_tokens":<cfg.limits.max_tokens>,"temperature":…,"top_p":…,"top_k":…,
 "chat_template_kwargs":{"enable_thinking":true}}

One example of each kind of message, because each is rendered differently:

ChatMessage	JSON
System { content }	{"role":"system","content":"…"}
User { content }	{"role":"user","content":"…"}
Tool { tool_call_id, content }	{"role":"tool","tool_call_id":"…","content":"…"}
Assistant with text only	{"role":"assistant","content":"…"}
Assistant with reasoning and a tool call, no text	{"role":"assistant","content":"","reasoning_content":"…","tool_calls":[{"id":"…","type":"function","function":{"name":"…","arguments":"…"}}]}

Rules: an assistant content of None is sent as "" (the template needs a string). reasoning_content is left out when None. tool_calls is left out when empty. The top-level tools is left out when empty; each tool is {"type":"function","function":{"name","description","parameters"}}. arguments stays the string it is; do not parse it. The tests compare bodies as JSON values, so key order is yours to choose, but it must be the same every time: build the body from structs with #[derive(Serialize)], not from a map.

API notes (serde 1.0, verified)

#[serde(skip_serializing_if = "Option::is_none")] and #[serde(skip_serializing_if = "Vec::is_empty")] leave a field out. A field named r#type serializes as type.

Steps

• 1. Copy. git switch m2a, then cp docs/plans/M2a/files/crates/loopd/tests/request.rs crates/loopd/tests/
• 2. See the test fail. cargo test -p loopd --test request. Expected: it does not compile.
• 3. Write llama/mod.rs and llama/request.rs, and add pub mod llama; to lib.rs. Run cargo fmt --all.
• 4. See the test pass. cargo test -p loopd --test request. Expected: 5 passed.
• 5. Run the gate. make gate. Expected last line: gate: ok.
• 6. Log and commit. git add crates/loopd docs/implementer-log.md && git commit

Done when

• cargo test -p loopd --test request reports 5 passed; make gate prints gate: ok.

Stop and report if

• A name or a derive above cannot be written as given.