boxmaker/docs/specs/2026-09-17-m2a-inference-path.md

# M2a design: the inference path

Status: draft for owner review, 2026-09-17. M2 is split in two (`docs/decisions.md`). M2a is
everything between `loopd` and `llama-server`: the forwarder, the HTTP and SSE client, the llama
client with its timers and caps, a fake server for tests, and the startup self-test. M2b builds
sessions, the turn loop and `bxctl chat` on top of it. Where this document and `docs/design.md`
disagree, the brief wins. Measurements it relies on are in `docs/inference-contract.md`.

## 1. What M2a proves

| Claim | Checked by |
|---|---|
| `loopd` reaches the model only through a Unix socket | `inferproxy` tests; `loopd` has no TCP code |
| A request is a faithful render of its inputs, and streamed deltas reassemble exactly | Fixture tests against transcripts recorded from straylight |
| Silence, a dead server, a busy slot and a runaway thinking block each end in a defined way | Timing tests against the fake server |
| A restart of the server under a request is survived | Retry tests; on device, the test kills and restarts its own `inferproxy` |
| The deployed server is the one the config describes, tool calls parse, and turn 2 hits the cache | Startup self-test; `make verify-device` |

Out of scope: session logs, the turn loop, tools, the channel protocol, `bxctl chat`, the baseline
token budget test (all M2b).

## 2. Components

```
loopd ── infer.sock ── inferproxy ── TCP ── llama-server router (straylight:11434)
```

### `inferproxy` (crate `inferproxy`)

`inferproxy --listen <socket path> --upstream <host:port>`. No config file. It logs nothing about
traffic; it prints one line at start and one line per refused connection.

- Removes a stale socket file, binds, sets the socket file mode to 0600.
- For each accepted connection: open one TCP connection upstream, copy bytes both ways on two
  threads, and pass a half-close on. When either side fails, close both.
- At most 8 connections at a time. A token bucket on `accept`: burst 10, refill 2 per second. A
  connection over either limit is closed at once.
- It never parses what it forwards.

### `proto::hash`

`pub fn sha256(data: &[u8]) -> Result<Hash32, HashError>`, a wrapper around the owner's `emsha`
crate (1.0.4 or later). `proto`'s tests carry their own vectors: the empty message, `abc`, the
million-`a` message, and messages of 55, 56, 63, 64 and 65 bytes, each also fed in two pieces.

### `loopd` modules

Each is one file under 500 lines with one purpose. None of them starts a thread.

| Module | Purpose |
|---|---|
| `config` | `config.toml` into a typed `Config`. Unknown keys are errors. Every limit in section 4 is a field with the default given there. |
| `http` | HTTP/1.1 client over a `UnixStream`. One request per connection, `Connection: close`. Reads the status line and headers, then offers the body as a reader that handles chunked, content-length and read-to-close bodies. |
| `sse` | Turns a body reader into a sequence of `data:` payloads. Handles lines split across reads, CRLF, comment lines and `[DONE]`. |
| `llama::request` | Builds the chat-completions JSON from typed inputs: messages, tools, slot, sampling. |
| `llama::assemble` | Folds streamed deltas into one assistant message, and keeps the latest per-chunk timings. |
| `llama::chat` | One request from gate to completion: the waits, liveness, the thinking cap. |
| `llama::gate` | The slot gate: one request in flight per slot, first come first served, bounded queue. |
| `llama::retry` | `chat_with_retry`, what is retryable, and the backoff schedule. |
| `llama::info` | `props`, `slots`, `tokenize`, and the cache-loss function. |
| `selftest` | The three startup checks. `loopd selftest --config <path>` runs them and exits. |

`loopd` gains the dependencies `serde`, `serde_json` and `toml`, all already vetted.

## 3. Interfaces

```rust
// What goes in. Messages are the four kinds the chat template knows.
pub enum ChatMessage {
    System { content: String },
    User { content: String },
    Assistant { content: Option<String>, reasoning_content: Option<String>, tool_calls: Vec<proto::ToolCall> },
    Tool { tool_call_id: String, content: String },
}
pub struct ToolSchema { pub name: String, pub description: String, pub parameters: serde_json::Value }
pub struct ChatRequest {
    pub slot: u32,
    pub messages: Vec<ChatMessage>,
    pub tools: Vec<ToolSchema>,          // fixed for the epoch; may be empty
    pub thinking: bool,
}

// What comes back while it runs.
pub enum ChatEvent {
    Queued { ahead: usize },             // waiting for loopd's own slot gate
    Waiting { slot_busy: bool },         // sent, no byte yet; one event per /slots poll
    Progress { total: u64, cache: u64, processed: u64 },
    Reasoning(String),
    Content(String),
    ToolCallDelta { index: u32, id: Option<String>, name: Option<String>, arguments: String },
    ThinkingCapped { tokens: u64 },
    Retrying { attempt: u32, after_ms: u64, error: String },
}

// What comes back at the end. Nothing is final before this.
pub struct Completion {
    pub id: String,
    pub content: Option<String>,
    pub reasoning_content: Option<String>,
    pub tool_calls: Vec<proto::ToolCall>,
    pub finish_reason: FinishReason,     // Stop | ToolCalls | Length
    pub timings: Timings,                // cache_n, prompt_n, predicted_n
    pub reasoning_tokens: u64,
    pub thinking_capped: bool,
}

pub enum InferError {
    Busy,                                // loopd's queue for the slot is full
    Connect(std::io::Error),
    WaitTimeout,                         // the slot stayed busy past the limit
    LoadTimeout,                         // the server stayed unavailable past the limit
    Stalled,                             // silence past the liveness limit
    StreamClosedEarly,
    ThinkingOverrun,
    Http { status: u16, body: String },  // body capped at 4 KiB
    Protocol(String),
}

impl Client {
    pub fn chat(&self, req: &ChatRequest, on_event: &mut dyn FnMut(&ChatEvent)) -> Result<Completion, InferError>;
    pub fn chat_with_retry(&self, req: &ChatRequest, on_event: &mut dyn FnMut(&ChatEvent)) -> Result<Completion, InferError>;
}
```

`Assistant` uses the same three fields as `proto::LogRecord::Assistant`, so M2b can replay a log
without conversion. Every request also carries, from config: the model id, `id_slot`,
`cache_prompt: true`, `stream: true`, `return_progress: true`, `timings_per_token: true`,
`reasoning_control: true`, `max_tokens`, and the sampling settings.

## 4. The life of one request

1. **Slot gate.** One request in flight per slot, inside `loopd`. Others queue, at most 8; beyond
   that the caller gets `Busy`. The gate is held for one request, never across a tool call or an
   approval, so a long tool does not block other sessions.
2. **Send, then wait for the first byte.** Reads time out every 5 s (`poll_ms`). On each timeout
   the client asks `/slots` on a second connection and emits `Waiting`:
   - the slot is busy: keep waiting, up to 10 min (`busy_wait_ms`), then `WaitTimeout`
   - `/slots` fails or answers 503: the model is loading or the server is restarting; keep waiting,
     up to 3 min (`load_wait_ms`), then `LoadTimeout`
   - the slot is idle and still no byte for 30 s (`idle_grace_ms`): `Stalled`
3. **After the first byte.** The read timeout becomes the liveness limit, 30 s (`liveness_ms`).
   Progress events and deltas both count. Silence past it is `Stalled`. There is no total deadline.
4. **Thinking cap.** Reasoning tokens are counted from the per-chunk `predicted_n`. At 4,096
   (`thinking_cap`) the client posts `reasoning_end` on a second connection, emits
   `ThinkingCapped`, and keeps reading. If reasoning continues for 256 more tokens, or the control
   call does not return `success: true`, the client closes the connection, which cancels
   generation, and returns `ThinkingOverrun`. `max_tokens` is 8,192 as a backstop.
5. **Done.** The final chunk carries `finish_reason`. The client returns the `Completion`. A stream
   that ends without one is `StreamClosedEarly`.

**Retry.** `chat_with_retry` retries `Connect`, `StreamClosedEarly`, `Stalled`, `LoadTimeout` and
HTTP 503: 4 attempts in all, waiting 2 s, 8 s and 30 s with ±25% jitter, and never past 5 min in
total. Each retry goes back through the slot gate and emits `Retrying`. Everything else is
returned at once: `WaitTimeout` has already waited, HTTP 400 includes "the prompt does not fit",
which M2b answers with compaction, and the rest are bugs. Retrying is safe because nothing is
recorded until a completion is final, so a retry sends the same bytes again.

**Size caps in the client.** 16 KiB of headers, 1 MiB per SSE line, 4 MiB for a body that is not a
stream. Exceeding one is `Protocol`.

**Cache loss.** `cache_outcome(previous: &Timings, current: &Timings) -> CacheOutcome` compares
`current.cache_n` with the previous request's total (`cache_n + prompt_n + predicted_n`). More
than 64 tokens short is a loss, reported with both numbers. It is a pure function; M2b logs it.

## 5. Startup self-test

`loopd` refuses to start unless all three pass. They run on the main slot and cost about 1,000
tokens. If the router has unloaded the model, the first request loads it and the load wait covers
that.

1. **The server is the one the config describes.** From `/props`: the SHA-256 of `chat_template`,
   per-slot `n_ctx` and `total_slots` equal the `[expect]` section of the config.
2. **Tool calls parse.** One tool, a fixed prompt, and the completion must contain that tool call
   with valid JSON arguments and `finish_reason` `ToolCalls`.
3. **Turn 2 hits the cache.** A two-turn exchange; `cache_outcome` for turn 2 must not be a loss.

## 6. Testing

**The fake server** is test support inside `loopd` (`tests/support/`), written by the design
model. It listens on a Unix socket in a temporary directory and serves each connection from a
script: a status, headers, and body pieces with a delay before each, ending in a clean close, a
cut connection or silence. It records every request it receives, which M2b will use to check that
each request extends the one before.

**Transcripts.** Bodies are real SSE transcripts recorded from straylight with synthetic prompts:
a plain answer, thinking then an answer, a streamed tool call, progress events during a long
prompt, a forced end to reasoning, and one raw chunked HTTP response for the `http` tests.

**Timing tests** set every limit to between 50 and 200 ms through `Config`, so none of them sleeps
for a second.

**On device.** `make verify-device` builds the workspace, starts `inferproxy` against
`straylight:11434`, runs `loopd selftest`, and then runs the ignored tests in
`crates/loopd/tests/device.rs`: a capped thinking block, and a stream that survives its
`inferproxy` being killed and restarted. The tests get the path of the `inferproxy` binary from
the environment, so `loopd` does not depend on that crate. They use slot 0 and never touch the
real `llama-server` process.

## 7. Threat model addition

A compromised `loopd` can degrade the shared `llama-server` for its other clients, by sending large
prompts or requests that are not pinned to its own slots. This is accepted for v0. The harm is to
availability only, the owner would see it, and preventing it would mean parsing untrusted input in
`inferproxy`. The limits in `inferproxy` and the slot gate exist to stop accidents: a retry storm
after a server restart, several sessions talking at once, a loop that spins.

## 8. How the work is handed over

As in M1 (`docs/specs/2026-09-17-pre-m1-design.md`, section 9), with the lessons in
`docs/implementer-lessons.md` applied: an example for every kind of place a rule applies, tests
that walk the data, and every syntactic form named. Tests, fixtures and the fake server are given;
the implementer writes the code. Review happens once, after the last M2a task, before M2b is
planned in detail.