Add M2a plan: thirteen tasks, tests, fake server and recordings
The tasks build the inference path: emsha-backed SHA-256, inferproxy, config, a hand-written HTTP and SSE client, request building, delta assembly, the chat state machine, the thinking cap, the slot gate with retry, the startup self-test and on-device verification. Everything the tasks copy in was checked against a private reference implementation: the gate passes after each task in order, the timing tests pass repeatedly under CPU load, and the reference passes the self-test and all four device checks on straylight. Expected results for the recorded streams were derived by a separate script. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,151 @@
|
||||
# M2a task 06: the llama client's types, and the request body
|
||||
|
||||
**Branch:** `m2a` (run `git switch m2a`; `git status --short` must be empty, otherwise stop)
|
||||
**Commit subject:** `Add the llama client types and the chat request builder`
|
||||
|
||||
## Goal
|
||||
|
||||
Define what goes into a chat request and what comes back, and build the JSON body the server
|
||||
expects. The streaming and the waiting come in later tasks; this one has no I/O.
|
||||
|
||||
## Context
|
||||
|
||||
The server speaks the OpenAI chat-completions format with some fields of its own. Two facts from
|
||||
the project's measurements shape this task:
|
||||
|
||||
- The server caches the prompt. A request only gets that cache if it repeats the earlier messages
|
||||
exactly, so an assistant message is sent back with the same three fields it arrived with.
|
||||
- `tools` is rendered at the very top of the prompt. It is sent only when there are tools.
|
||||
|
||||
## Files
|
||||
|
||||
- Copy: `crates/loopd/tests/request.rs`
|
||||
- Create: `crates/loopd/src/llama/mod.rs`, `crates/loopd/src/llama/request.rs`
|
||||
- Modify: `crates/loopd/src/lib.rs`, `docs/implementer-log.md`
|
||||
|
||||
## Interfaces
|
||||
|
||||
Produces, in `crates/loopd/src/llama/mod.rs`. Copy these definitions as they are; later tasks and
|
||||
their tests depend on every name.
|
||||
|
||||
```rust
|
||||
pub mod request;
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub enum ChatMessage {
|
||||
System { content: String },
|
||||
User { content: String },
|
||||
Assistant { content: Option<String>, reasoning_content: Option<String>, tool_calls: Vec<proto::ToolCall> },
|
||||
Tool { tool_call_id: String, content: String },
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, PartialEq)]
|
||||
pub struct ToolSchema { pub name: String, pub description: String, pub parameters: serde_json::Value }
|
||||
|
||||
#[derive(Debug, Clone, PartialEq)]
|
||||
pub struct ChatRequest { pub slot: u32, pub messages: Vec<ChatMessage>, pub tools: Vec<ToolSchema>, pub thinking: bool }
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub enum ChatEvent {
|
||||
Queued { ahead: usize },
|
||||
Waiting { slot_busy: bool },
|
||||
Progress { total: u64, cache: u64, processed: u64 },
|
||||
Reasoning(String),
|
||||
Content(String),
|
||||
ToolCallDelta { index: u32, id: Option<String>, name: Option<String>, arguments: String },
|
||||
ThinkingCapped { tokens: u64 },
|
||||
Retrying { attempt: u32, after_ms: u64, error: String },
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
pub enum FinishReason { Stop, ToolCalls, Length }
|
||||
|
||||
/// The three counts loopd uses. The server sends more fields beside them; they are ignored.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Default, serde::Deserialize)]
|
||||
pub struct Timings { pub cache_n: u64, pub prompt_n: u64, pub predicted_n: u64 }
|
||||
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub struct Completion {
|
||||
pub id: String,
|
||||
pub content: Option<String>,
|
||||
pub reasoning_content: Option<String>,
|
||||
pub tool_calls: Vec<proto::ToolCall>,
|
||||
pub finish_reason: FinishReason,
|
||||
pub timings: Timings,
|
||||
pub reasoning_tokens: u64,
|
||||
pub thinking_capped: bool,
|
||||
}
|
||||
|
||||
#[derive(Debug)]
|
||||
pub enum InferError {
|
||||
Busy, Connect(std::io::Error), WaitTimeout, LoadTimeout, Stalled, StreamClosedEarly,
|
||||
ThinkingOverrun, Http { status: u16, body: String }, Protocol(String),
|
||||
} // Display (one short sentence per variant) and std::error::Error
|
||||
|
||||
pub struct Client { pub(crate) cfg: crate::config::Config }
|
||||
impl Client {
|
||||
pub fn new(cfg: crate::config::Config) -> Self;
|
||||
pub fn config(&self) -> &crate::config::Config;
|
||||
}
|
||||
```
|
||||
|
||||
`Timings` parses **the server's** JSON, so it must **not** have `deny_unknown_fields`: the server
|
||||
sends six more timing fields beside these three.
|
||||
|
||||
And in `crates/loopd/src/llama/request.rs`:
|
||||
|
||||
```rust
|
||||
/// The request body as JSON text. The same inputs always give the same bytes.
|
||||
pub fn build_body(cfg: &crate::config::Config, req: &super::ChatRequest) -> Result<String, serde_json::Error>;
|
||||
```
|
||||
|
||||
The body, for a request with one tool and thinking on:
|
||||
|
||||
```json
|
||||
{"model":"<cfg.infer.model>","messages":[…],"tools":[…],"id_slot":<req.slot>,"cache_prompt":true,
|
||||
"stream":true,"return_progress":true,"timings_per_token":true,"reasoning_control":true,
|
||||
"max_tokens":<cfg.limits.max_tokens>,"temperature":…,"top_p":…,"top_k":…,
|
||||
"chat_template_kwargs":{"enable_thinking":true}}
|
||||
```
|
||||
|
||||
One example of each kind of message, because each is rendered differently:
|
||||
|
||||
| `ChatMessage` | JSON |
|
||||
|---|---|
|
||||
| `System { content }` | `{"role":"system","content":"…"}` |
|
||||
| `User { content }` | `{"role":"user","content":"…"}` |
|
||||
| `Tool { tool_call_id, content }` | `{"role":"tool","tool_call_id":"…","content":"…"}` |
|
||||
| `Assistant` with text only | `{"role":"assistant","content":"…"}` |
|
||||
| `Assistant` with reasoning and a tool call, no text | `{"role":"assistant","content":"","reasoning_content":"…","tool_calls":[{"id":"…","type":"function","function":{"name":"…","arguments":"…"}}]}` |
|
||||
|
||||
Rules: an assistant `content` of `None` is sent as `""` (the template needs a string).
|
||||
`reasoning_content` is left out when `None`. `tool_calls` is left out when empty. The top-level
|
||||
`tools` is left out when empty; each tool is
|
||||
`{"type":"function","function":{"name","description","parameters"}}`. `arguments` stays the string
|
||||
it is; do not parse it. The tests compare bodies as JSON values, so key order is yours to choose,
|
||||
but it must be the same every time: build the body from structs with `#[derive(Serialize)]`, not
|
||||
from a map.
|
||||
|
||||
## API notes (serde 1.0, verified)
|
||||
|
||||
`#[serde(skip_serializing_if = "Option::is_none")]` and `#[serde(skip_serializing_if =
|
||||
"Vec::is_empty")]` leave a field out. A field named `r#type` serializes as `type`.
|
||||
|
||||
## Steps
|
||||
|
||||
- [ ] **1. Copy.** `git switch m2a`, then
|
||||
`cp docs/plans/M2a/files/crates/loopd/tests/request.rs crates/loopd/tests/`
|
||||
- [ ] **2. See the test fail.** `cargo test -p loopd --test request`. Expected: it does not compile.
|
||||
- [ ] **3. Write `llama/mod.rs` and `llama/request.rs`,** and add `pub mod llama;` to `lib.rs`.
|
||||
Run `cargo fmt --all`.
|
||||
- [ ] **4. See the test pass.** `cargo test -p loopd --test request`. Expected: `5 passed`.
|
||||
- [ ] **5. Run the gate.** `make gate`. Expected last line: `gate: ok`.
|
||||
- [ ] **6. Log and commit.** `git add crates/loopd docs/implementer-log.md && git commit`
|
||||
|
||||
## Done when
|
||||
|
||||
- `cargo test -p loopd --test request` reports 5 passed; `make gate` prints `gate: ok`.
|
||||
|
||||
## Stop and report if
|
||||
|
||||
- A name or a derive above cannot be written as given.
|
||||
Reference in New Issue
Block a user