boxmaker/docs/specs/2026-09-17-m2a-inference-path.md

M2a design: the inference path

Status: draft for owner review, 2026-09-17. M2 is split in two (docs/decisions.md). M2a is everything between loopd and llama-server: the forwarder, the HTTP and SSE client, the llama client with its timers and caps, a fake server for tests, and the startup self-test. M2b builds sessions, the turn loop and bxctl chat on top of it. Where this document and docs/design.md disagree, the brief wins. Measurements it relies on are in docs/inference-contract.md.

1. What M2a proves

Claim	Checked by
loopd reaches the model only through a Unix socket	inferproxy tests; loopd has no TCP code
A request is a faithful render of its inputs, and streamed deltas reassemble exactly	Fixture tests against transcripts recorded from straylight
Silence, a dead server, a busy slot and a runaway thinking block each end in a defined way	Timing tests against the fake server
A restart of the server under a request is survived	Retry tests; on device, the test kills and restarts its own inferproxy
The deployed server is the one the config describes, tool calls parse, and turn 2 hits the cache	Startup self-test; make verify-device

Out of scope: session logs, the turn loop, tools, the channel protocol, bxctl chat, the baseline token budget test (all M2b).

2. Components

loopd ── infer.sock ── inferproxy ── TCP ── llama-server router (straylight:11434)

inferproxy (crate inferproxy)

inferproxy --listen <socket path> --upstream <host:port>. No config file. It logs nothing about traffic; it prints one line at start and one line per refused connection.

• Removes a stale socket file, binds, sets the socket file mode to 0600.
• For each accepted connection: open one TCP connection upstream, copy bytes both ways on two threads, and pass a half-close on. When either side fails, close both.
• At most 8 connections at a time. A token bucket on accept: burst 10, refill 2 per second. A connection over either limit is closed at once.
• It never parses what it forwards.

proto::hash

pub fn sha256(data: &[u8]) -> Result<Hash32, HashError>, a wrapper around the owner's emsha crate (1.0.4 or later). proto's tests carry their own vectors: the empty message, abc, the million-a message, and messages of 55, 56, 63, 64 and 65 bytes, each also fed in two pieces.

loopd modules

Each is one file under 500 lines with one purpose. None of them starts a thread.

Module	Purpose
config	config.toml into a typed Config. Unknown keys are errors. Every limit in section 4 is a field with the default given there.
http	HTTP/1.1 client over a UnixStream. One request per connection, Connection: close. Reads the status line and headers, then offers the body as a reader that handles chunked, content-length and read-to-close bodies.
sse	Turns a body reader into a sequence of data: payloads. Handles lines split across reads, CRLF, comment lines and [DONE].
llama::request	Builds the chat-completions JSON from typed inputs: messages, tools, slot, sampling.
llama::assemble	Folds streamed deltas into one assistant message, and keeps the latest per-chunk timings.
llama::chat	One request from gate to completion: the waits, liveness, the thinking cap.
llama::gate	The slot gate: one request in flight per slot, first come first served, bounded queue.
llama::retry	chat_with_retry, what is retryable, and the backoff schedule.
llama::info	props, slots, tokenize, and the cache-loss function.
selftest	The three startup checks. loopd selftest --config <path> runs them and exits.

loopd gains the dependencies serde, serde_json and toml, all already vetted.

3. Interfaces

// What goes in. Messages are the four kinds the chat template knows.
pub enum ChatMessage {
    System { content: String },
    User { content: String },
    Assistant { content: Option<String>, reasoning_content: Option<String>, tool_calls: Vec<proto::ToolCall> },
    Tool { tool_call_id: String, content: String },
}
pub struct ToolSchema { pub name: String, pub description: String, pub parameters: serde_json::Value }
pub struct ChatRequest {
    pub slot: u32,
    pub messages: Vec<ChatMessage>,
    pub tools: Vec<ToolSchema>,          // fixed for the epoch; may be empty
    pub thinking: bool,
}

// What comes back while it runs.
pub enum ChatEvent {
    Queued { ahead: usize },             // waiting for loopd's own slot gate
    Waiting { slot_busy: bool },         // sent, no byte yet; one event per /slots poll
    Progress { total: u64, cache: u64, processed: u64 },
    Reasoning(String),
    Content(String),
    ToolCallDelta { index: u32, id: Option<String>, name: Option<String>, arguments: String },
    ThinkingCapped { tokens: u64 },
    Retrying { attempt: u32, after_ms: u64, error: String },
}

// What comes back at the end. Nothing is final before this.
pub struct Completion {
    pub id: String,
    pub content: Option<String>,
    pub reasoning_content: Option<String>,
    pub tool_calls: Vec<proto::ToolCall>,
    pub finish_reason: FinishReason,     // Stop | ToolCalls | Length
    pub timings: Timings,                // cache_n, prompt_n, predicted_n
    pub reasoning_tokens: u64,
    pub thinking_capped: bool,
}

pub enum InferError {
    Busy,                                // loopd's queue for the slot is full
    Connect(std::io::Error),
    WaitTimeout,                         // the slot stayed busy past the limit
    LoadTimeout,                         // the server stayed unavailable past the limit
    Stalled,                             // silence past the liveness limit
    StreamClosedEarly,
    ThinkingOverrun,
    Http { status: u16, body: String },  // body capped at 4 KiB
    Protocol(String),
}

impl Client {
    pub fn chat(&self, req: &ChatRequest, on_event: &mut dyn FnMut(&ChatEvent)) -> Result<Completion, InferError>;
    pub fn chat_with_retry(&self, req: &ChatRequest, on_event: &mut dyn FnMut(&ChatEvent)) -> Result<Completion, InferError>;
}

Assistant uses the same three fields as proto::LogRecord::Assistant, so M2b can replay a log without conversion. Every request also carries, from config: the model id, id_slot, cache_prompt: true, stream: true, return_progress: true, timings_per_token: true, reasoning_control: true, max_tokens, and the sampling settings.

4. The life of one request

1. Slot gate. One request in flight per slot, inside loopd. Others queue, at most 8; beyond that the caller gets Busy. The gate is held for one request, never across a tool call or an approval, so a long tool does not block other sessions.
2. Send, then wait for the first byte. Reads time out every 5 s (poll_ms). On each timeout the client asks /slots on a second connection and emits Waiting:
   • the slot is busy: keep waiting, up to 10 min (busy_wait_ms), then WaitTimeout
   • /slots fails or answers 503: the model is loading or the server is restarting; keep waiting, up to 3 min (load_wait_ms), then LoadTimeout
   • the slot is idle and still no byte for 30 s (idle_grace_ms): Stalled
3. After the first byte. The read timeout becomes the liveness limit, 30 s (liveness_ms). Progress events and deltas both count. Silence past it is Stalled. There is no total deadline.
4. Thinking cap. Reasoning tokens are counted from the per-chunk predicted_n. At 4,096 (thinking_cap) the client posts reasoning_end on a second connection, emits ThinkingCapped, and keeps reading. If reasoning continues for 256 more tokens, or the control call does not return success: true, the client closes the connection, which cancels generation, and returns ThinkingOverrun. max_tokens is 8,192 as a backstop.
5. Done. The final chunk carries finish_reason. The client returns the Completion. A stream that ends without one is StreamClosedEarly.

Retry. chat_with_retry retries Connect, StreamClosedEarly, Stalled, LoadTimeout and HTTP 503: 4 attempts in all, waiting 2 s, 8 s and 30 s with ±25% jitter, and never past 5 min in total. Each retry goes back through the slot gate and emits Retrying. Everything else is returned at once: WaitTimeout has already waited, HTTP 400 includes "the prompt does not fit", which M2b answers with compaction, and the rest are bugs. Retrying is safe because nothing is recorded until a completion is final, so a retry sends the same bytes again.

Size caps in the client. 16 KiB of headers, 1 MiB per SSE line, 4 MiB for a body that is not a stream. Exceeding one is Protocol.

Cache loss. cache_outcome(previous: &Timings, current: &Timings) -> CacheOutcome compares current.cache_n with the previous request's total (cache_n + prompt_n + predicted_n). More than 64 tokens short is a loss, reported with both numbers. It is a pure function; M2b logs it.

5. Startup self-test

loopd refuses to start unless all three pass. They run on the main slot and cost about 1,000 tokens. If the router has unloaded the model, the first request loads it and the load wait covers that.

1. The server is the one the config describes. From /props: the SHA-256 of chat_template, per-slot n_ctx and total_slots equal the [expect] section of the config.
2. Tool calls parse. One tool, a fixed prompt, and the completion must contain that tool call with valid JSON arguments and finish_reason ToolCalls.
3. Turn 2 hits the cache. A two-turn exchange; cache_outcome for turn 2 must not be a loss.

6. Testing

The fake server is test support inside loopd (tests/support/), written by the design model. It listens on a Unix socket in a temporary directory and serves each connection from a script: a status, headers, and body pieces with a delay before each, ending in a clean close, a cut connection or silence. It records every request it receives, which M2b will use to check that each request extends the one before.

Transcripts. Bodies are real SSE transcripts recorded from straylight with synthetic prompts: a plain answer, thinking then an answer, a streamed tool call, progress events during a long prompt, a forced end to reasoning, and one raw chunked HTTP response for the http tests.

Timing tests set every limit to between 50 and 200 ms through Config, so none of them sleeps for a second.

On device. make verify-device builds the workspace, starts inferproxy against straylight:11434, runs loopd selftest, and then runs the ignored tests in crates/loopd/tests/device.rs: a capped thinking block, and a stream that survives its inferproxy being killed and restarted. The tests get the path of the inferproxy binary from the environment, so loopd does not depend on that crate. They use slot 0 and never touch the real llama-server process.

7. Threat model addition

A compromised loopd can degrade the shared llama-server for its other clients, by sending large prompts or requests that are not pinned to its own slots. This is accepted for v0. The harm is to availability only, the owner would see it, and preventing it would mean parsing untrusted input in inferproxy. The limits in inferproxy and the slot gate exist to stop accidents: a retry storm after a server restart, several sessions talking at once, a loop that spins.

8. How the work is handed over

As in M1 (docs/specs/2026-09-17-pre-m1-design.md, section 9), with the lessons in docs/implementer-lessons.md applied: an example for every kind of place a rule applies, tests that walk the data, and every syntactic form named. Tests, fixtures and the fake server are given; the implementer writes the code. Review happens once, after the last M2a task, before M2b is planned in detail.