Add M2a plan: thirteen tasks, tests, fake server and recordings
The tasks build the inference path: emsha-backed SHA-256, inferproxy, config, a hand-written HTTP and SSE client, request building, delta assembly, the chat state machine, the thinking cap, the slot gate with retry, the startup self-test and on-device verification. Everything the tasks copy in was checked against a private reference implementation: the gate passes after each task in order, the timing tests pass repeatedly under CPU load, and the reference passes the self-test and all four device checks on straylight. Expected results for the recorded streams were derived by a separate script. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,110 @@
|
||||
# M2a task 12: the startup self-test
|
||||
|
||||
**Branch:** `m2a` (run `git switch m2a`; `git status --short` must be empty, otherwise stop)
|
||||
**Commit subject:** `Add the startup self-test and the loopd selftest command`
|
||||
|
||||
## Goal
|
||||
|
||||
Before `loopd` serves anyone it checks that the server is the one its config describes, that tool
|
||||
calls come back parsed, and that a second turn reuses the first turn's cache. If any check fails,
|
||||
`loopd` refuses to start. This task writes the checks and a command that runs them.
|
||||
|
||||
## Context
|
||||
|
||||
From the design brief: "On boot `loopd` checks: tool-call round trip parses, turn-2 prompt
|
||||
processing count shows a cache hit, configured context matches what the server reports. It refuses
|
||||
to start if any check fails."
|
||||
|
||||
## Files
|
||||
|
||||
- Copy: `crates/loopd/tests/selftest.rs`
|
||||
- Create: `crates/loopd/src/selftest.rs`
|
||||
- Modify: `crates/loopd/src/lib.rs`, `crates/loopd/src/main.rs`, `docs/implementer-log.md`
|
||||
|
||||
## Interfaces
|
||||
|
||||
```rust
|
||||
// crates/loopd/src/selftest.rs
|
||||
#[derive(Debug)]
|
||||
pub enum SelfTestError {
|
||||
Mismatch { what: &'static str, expected: String, got: String },
|
||||
ToolCall(String),
|
||||
CacheMiss { expected: u64, got: u64 },
|
||||
Infer(InferError),
|
||||
Hash,
|
||||
} // Display, std::error::Error, and From<InferError>
|
||||
|
||||
/// Runs the three checks in order. `on_step` is told the name of each check as it starts.
|
||||
pub fn run(client: &Client, on_step: &mut dyn FnMut(&str)) -> Result<(), SelfTestError>;
|
||||
```
|
||||
|
||||
The checks. All chat requests use `client.config().slots.main`, `thinking: false`, and
|
||||
`chat_with_retry` with an event callback that does nothing.
|
||||
|
||||
**1. `on_step("server matches config")`.** Call `props()`. Compare, in this order, and return
|
||||
`Mismatch` with the given `what` for the first that differs (`expected` from the config, `got` from
|
||||
the server, both as text):
|
||||
|
||||
| `what` | Config | Server |
|
||||
|---|---|---|
|
||||
| `"chat template sha256"` | `expect.template_sha256` (as hex) | `proto::sha256(chat_template.as_bytes())` (as hex); a hash error is `SelfTestError::Hash` |
|
||||
| `"context per slot"` | `expect.n_ctx` | `props.n_ctx` |
|
||||
| `"slot count"` | `expect.slots` | `props.total_slots` |
|
||||
|
||||
A server that is not the expected one is sent no prompt at all.
|
||||
|
||||
**2. `on_step("tool call round trip")`.** Messages: `System` "You are Boxmaker, a careful personal
|
||||
agent." and `User` "Read /etc/hostname and tell me what it says." One tool:
|
||||
|
||||
```rust
|
||||
ToolSchema {
|
||||
name: "read_file".to_string(),
|
||||
description: "Read a text file and return its contents.".to_string(),
|
||||
parameters: serde_json::json!({
|
||||
"type": "object",
|
||||
"properties": { "path": { "type": "string", "description": "Absolute path" } },
|
||||
"required": ["path"],
|
||||
}),
|
||||
}
|
||||
```
|
||||
|
||||
Return `ToolCall(reason)` unless: `finish_reason` is `ToolCalls`, there is a first tool call, its
|
||||
name is `read_file`, its `arguments` parse as JSON, and the JSON has a string `path`.
|
||||
|
||||
**3. `on_step("turn 2 cache hit")`.** No tools. Turn 1: the same `System` message and `User` "Name
|
||||
one colour. One word." Turn 2: those two messages, then an `Assistant` message made from turn 1's
|
||||
completion (its `content`, `reasoning_content` and `tool_calls`, unchanged), then `User` "Name
|
||||
another. One word." If `cache_outcome(&turn1.timings, &turn2.timings)` is a `Loss`, return
|
||||
`CacheMiss` with its two numbers.
|
||||
|
||||
**`main.rs`.** `loopd selftest --config <path>` loads the config (`Config::load`), makes a `Client`,
|
||||
runs the self-test printing `selftest: <step>` to stderr for each step, then `selftest: ok` and
|
||||
exit code 0, or `selftest: FAILED: <error>` and exit code 1. A config that cannot be loaded prints
|
||||
`loopd: <error>` and exits 1. Any other arguments print a usage line and exit 2.
|
||||
|
||||
## Steps
|
||||
|
||||
- [ ] **1. Copy.** `git switch m2a`, then
|
||||
`cp docs/plans/M2a/files/crates/loopd/tests/selftest.rs crates/loopd/tests/`
|
||||
- [ ] **2. See the test fail.** `cargo test -p loopd --test selftest`. Expected: it does not compile.
|
||||
- [ ] **3. Write `selftest.rs`, register it in `lib.rs`, and rewrite `main.rs`.** Run
|
||||
`cargo fmt --all`.
|
||||
- [ ] **4. See the test pass.** `cargo test -p loopd --test selftest`. Expected: `6 passed`.
|
||||
- [ ] **5. Try it on the real server,** if straylight can be reached. In one terminal:
|
||||
`cargo run -p inferproxy -- --listen /tmp/ip.sock --upstream straylight:11434`. Write
|
||||
`/tmp/loopd.toml` by copying `crates/loopd/tests/fixtures/config/minimal.toml` and changing
|
||||
`socket` to `/tmp/ip.sock`. Then `cargo run -p loopd -- selftest --config /tmp/loopd.toml`.
|
||||
Expected: three step lines and `selftest: ok`, in a few seconds. Then set `slots = 3` in the
|
||||
file and run it again. Expected: `selftest: FAILED: slot count: …` and exit code 1. Put what you
|
||||
saw in your log row. Remove both files in `/tmp` afterwards.
|
||||
- [ ] **6. Run the gate.** `make gate`. Expected last line: `gate: ok`.
|
||||
- [ ] **7. Log and commit.** `git add crates/loopd docs/implementer-log.md && git commit`
|
||||
|
||||
## Done when
|
||||
|
||||
- `cargo test -p loopd --test selftest` reports 6 passed; `make gate` prints `gate: ok`.
|
||||
|
||||
## Stop and report if
|
||||
|
||||
- The real server fails check 1 with the unchanged `minimal.toml`. That means the server's chat
|
||||
template or settings changed, which the owner needs to know; it is not something to fix in code.
|
||||
Reference in New Issue
Block a user