boxmaker/docs/design.md

Boxmaker design brief

Boxmaker is a sovereign personal agent harness. The name is from Count Zero: the remnant AI in the Straylight cores that patiently assembles small boxes from fragments.

This brief is binding. Changes to it land as their own commit and are recorded in docs/decisions.md. Milestones and session prompts are in docs/milestones.md.

Crate names and llama.cpp server parameters below are from memory or secondhand research. Verify each one against primary docs before use.

Goal

A personal agent harness that maximizes sovereignty: local inference by default, no ambient authority, state in plain files the owner can read, a codebase small enough to audit.

Target environment

• Inference host: straylight, AMD Strix Halo, 128 GB unified memory, of which the GPU may address 104 GiB. The harness runs on the same host.
• Serving: llama.cpp llama-server in router mode. One endpoint, one child server per model, at most two models loaded. The owner's coding agents and Open WebUI use the same router and the same Ornith instance. Details and measurements are in docs/inference-contract.md.
• Model: Ornith-1.5-35B-A3B (Qwen3.5-family MoE, hybrid linear/full attention, Qwen XML tool calls with thinking blocks). As deployed: Q4_K_M weights, q8_0 KV cache, two slots of 131,072 tokens. Weights and cache are unloaded after six idle hours, or when a third model is loaded.
• Memory is not abundant: Laguna S 2.1 (69 GB) and Ornith (22 GB) are normally both loaded. Generation is fast. Prompt processing is slow and cache invalidation is the dominant cost. Hybrid-attention models cannot partially rewind their cache; M0 confirmed it.
• Measured in M0 with the GPU otherwise idle: prompt processing 1,170 tokens/s at depth 0 and 669 at 32k depth; generation 69 tokens/s falling to 60. A second active session on Ornith roughly halves generation speed. An uncached 32k-token prompt took 36 s, so a 64k re-read costs well over a minute.
• Deployment: rootless containers on a host that is already a tailnet node. Containers do not get their own tailnet identities, so tailnet ACLs cannot tell one role from another. Isolation between roles is done with container networking on the host (see "Network isolation"). Tailnet ACLs restrict what the host as a whole may reach.
• First channel: the owner's existing self-hosted Mattermost server, reached over the tailnet.

Non-goals (v0)

Plugin registry, skill marketplace, web dashboard, multi-user, voice, more than one chat channel, cloud-led orchestration as the default, in-process third-party code of any kind.

Roles (separate binaries, narrow interfaces)

Role	Purpose	Holds	Can reach	Never holds
gatewayd	Carries messages between the owner's Mattermost and the harness.	Mattermost bot token, allowlist, message queue	Mattermost server (outbound only), loopd socket	Tool secrets, grants
loopd	Runs the conversation: builds each prompt, calls the model, and asks brokerd for every tool call.	Session logs, prompt assembly, memory files	Three Unix sockets: gatewayd, brokerd, inferproxy	Any credential, any network interface
inferproxy	Lets loopd reach the model without having a network.	The llama-server address	llama-server only	Anything else; it forwards bytes and logs nothing
brokerd	Decides whether each tool call may run, runs it in a fresh container, and records the decision.	Grants (read-only), secrets, audit log	Container runtime, gatewayd socket (for approvals)	Conversation history
tool containers	Do one approved tool call and then cease to exist.	Only what one call was granted	Only what one call was granted	Anything persistent

v0 runs all roles on one host. The interfaces must not assume co-location with llama-server.

Network isolation

The host's tailnet identity is shared by every container that has a network, so a container with ordinary outbound networking can reach the whole tailnet and the internet as the host.

1. loopd runs with no network namespace interfaces except loopback (--network=none). Its only I/O is three Unix sockets on a shared volume.
2. inferproxy is a forwarder of about 100 lines: one Unix socket in, one fixed upstream out. The upstream is the shared router's TCP listener on the host. It is required: loopd has no network, and the router stays on TCP for its other clients.
3. gatewayd has outbound network only. It opens no listening TCP port. v0 approvals use replies or emoji reactions, which arrive over the Mattermost WebSocket, so no callback URL is needed.
4. Tool containers get no network unless the grant names hosts. Granted network access goes through an allowlisting egress proxy that brokerd configures per call.
5. Tailnet ACLs (kept in deploy/) limit the host's node to llama-server and Mattermost, plus whatever tool grants need. This is defence in depth, not the role boundary.

Channel: Mattermost

• Bot account with a token held in the SecretStore. REST API for posting, WebSocket for events. Implement directly over HTTP and WebSocket; do not adopt a community SDK without checking its maintenance. Fetch the Mattermost API reference before writing any request.
• Identity is the Mattermost user ID. The allowlist is a config file of user IDs. Direct messages only by default; channels must be allowlisted by ID. Authentication is delegated to the server.
• Long tasks: post a placeholder, edit or reply when finished. Threads map to sessions.
• Mattermost stores messages unencrypted in its database. That is acceptable because the server is the owner's, and it is recorded in docs/egress.md.

Inference contract (the part existing harnesses get wrong)

1. Baseline budget. System prompt plus tool schemas at session start: 3,000 tokens or less, measured with the server's tokenizer (/tokenize), enforced by a test. That test needs straylight, so it runs in make verify-device, not in the offline make gate.
2. Append-only. The request for turn N+1 is a strict extension of the request for turn N. Nothing volatile (time, heartbeat notes, memory refreshes) goes anywhere but the newest message. Enforced by a property test on the serialized message array. The session log stores each assistant message exactly as the server returned it (content, reasoning_content, tool_calls) and replays it unchanged. Ornith's template keeps every thinking block, so dropping or editing one changes the prefix.
3. No in-place pruning. Tool results are size-capped when first appended, never trimmed later.
4. Compaction is an epoch change. It happens only when the session is idle, writes a summary, and starts a new epoch whose prefix is the baseline plus the summary. The old log is kept.
5. Pinned slots, not reserved. Every request carries id_slot, and a session always uses the same slot: moving it costs a full re-read. Slot assignment is configuration. With the two slots deployed today, main sessions (one per Mattermost thread) share one slot, and subagents and scheduled runs share the other. The server is shared, so another client can take a harness slot and the router can unload Ornith. An evicted session is normally restored from the server's host-RAM prompt cache when it returns on its slot; an unloaded model is not. Cache loss is therefore an expected event: loopd detects it (cache_n far below the previous request's total), records it in the session log and carries on. It is never an error.
6. Progressive disclosure. The tools array is fixed for the whole epoch. The template renders it at the top of the prompt, so any change re-reads everything. Only a small core tool set has schemas in it. Everything else is found through find_tool, which returns schemas as an ordinary tool result, and is called through one fixed call_tool(name, arguments) meta-tool. brokerd applies grants to the target tool, not to call_tool. Never prompt the model to call a tool that is not in the tools array: the server's grammar forces such a call into a declared tool.
7. Liveness, not deadlines. Streaming always, with return_progress: true so that prompt processing produces events. The timeout is "no bytes for N seconds", never a total-request deadline. Progress events count as bytes. Before the first byte, a request may be queued behind another client on its slot or waiting for the router to load the model. A separate, longer limit covers that wait, and the liveness timer starts at the first byte. M2 measures what the stream sends while a request is queued.
8. Runaway control. Per-turn thinking-token cap, per-turn tool-iteration cap, detection of repeated identical tool calls. loopd enforces the thinking cap: it counts streamed reasoning tokens and ends the block with the server's reasoning_control mechanism (POST /v1/chat/completions/control, action reasoning_end). M2 exercises this on straylight before relying on it.
9. Startup self-test. On boot loopd checks: tool-call round trip parses, turn-2 prompt processing count shows a cache hit, configured context matches what the server reports. It refuses to start if any check fails.
10. Serving settings are recorded, not owned. The launch flags live in the owner's NixOS configuration (~/src/nixos on straylight). This repo records the expected values: chat template hash, per-slot context, slot count, q8_0 KV cache, no speculative decoding. The startup self-test compares them with /props and /slots. Sampling settings (temp 0.6, top-p 0.95, top-k 20) are sent with every request, because the server's default temperature is 1.0.

Settled by M0 (docs/inference-contract.md): loopd uses the server's chat-completions endpoint with server-side tool parsing. It does not render the template in-process. Cache reuse held across a multi-turn tool conversation and tool parsing had no failures in 20 trials.

Authority contract

1. loopd has no authority. Every tool call goes to brokerd as a request.
2. A grant is a file the owner writes: tool, argument constraints (paths, hosts, patterns), allowed data classes, expiry, mode (auto, ask, deny). brokerd reads grants and cannot write them. loopd cannot see the directory.
3. No matching grant means deny. ask routes an approval request to the owner through gatewayd and the turn suspends until answered or expired.
4. Each approved call runs in a fresh rootless container: no network unless granted, only granted paths mounted, only the granted secret injected.
5. Audit log: append-only JSONL, hash-chained, one record per decision including denials.
6. Data classes: every tool result is labelled public, private, or secret. A session's taint is the union of labels it has seen. The cloud-consult tool is refused for tainted sessions unless a grant allows that class out. Every outbound cloud payload is logged in full.
7. Scheduled jobs are re-validated against current grants on every run.

Cloud consult (M6)

• One tool, consult, behind brokerd like any other. The endpoint is any OpenAI-compatible URL, set in config. Nothing in the code is specific to one vendor.
• Candidate as of 2026-09-16: GLM-5.3-Flash (open weights, MIT, 320B total / 18B active). It does not run usefully on the inference host, so it is cloud-only here. Hosting choices: Z.ai's international API (Singapore entity, Singapore law, states it does not store API content), or a third-party host chosen by jurisdiction with zero-data-retention routing. Z.ai's subscription coding plan is restricted to its officially supported tools; use pay-as-you-go.
• Call it at temperature 0.6 or lower. Community tests show malformed tool calls at the default of 1.0. Prefer giving consult no tools at all: it answers a self-contained question in text.
• Hosted GLM always thinks. Treat consult as slow and asynchronous.
• Payloads are built from an explicit allowlist of fields, never from raw session history.

Memory (M5)

Memory is what carries continuity across epoch compaction, so it is required. It must not break the inference contract, and it must not become a way for injected text to persist.

1. Core block. memory/core.md, owner-curated, 300 tokens or less, part of the baseline. Edits take effect at the next epoch, never mid-session. The agent can propose changes; only an ask approval or the owner's editor applies them.
2. Everything else is retrieved, not injected. A recall(query, k) tool returns size-capped snippets with file, line and provenance. Results are appended like any tool result, so the cache prefix is untouched.
3. Notes. memory/notes/YYYY-MM-DD.md, append-only, written by a remember tool. Each entry records its session, time, data-class label and whether the session had seen untrusted content.
4. Flush before compaction. When a session goes idle and is about to change epoch, a job on the scheduled slot extracts durable facts from the log into notes. The main slot is not used.
5. Provenance on recall. Recalled text is presented as data with its provenance, never placed in the system prompt. Entries written from sessions that saw untrusted content are marked as such when recalled. Recalled entries carry their data-class label and taint the session.
6. Both tools go through brokerd under ordinary grants: recall and note-writing auto, core and topic edits ask.
7. Index. Lexical search first: SQLite FTS5 over the Markdown files, rebuildable with bxctl reindex. Retrieval sits behind a Retriever trait.
8. Embeddings are a later, additive index (M5b), never a store. Local model only, served by a separate llama-server embedding instance so chat slots are unaffected. Vectors live in a flat file or SQLite table keyed by chunk hash and embedding-model hash; a model change triggers a rebuild. Combine with lexical results by rank fusion. Add M5b when logged recall misses show lexical search failing, not before.
9. No vector database service. Search is exact brute-force cosine, in-process. Revisit only if the index passes about one million chunks or measured recall latency exceeds one second, and then prefer an embedded, file-backed approximate index over a server.

State

Files are the source of truth. SQLite is allowed only for rebuildable indexes and queues.

• sessions/<id>/<epoch>.jsonl — append-only session log
• memory/core.md, memory/notes/*.md, memory/topics/*.md — owner-readable memory
• index/ — FTS5 and, later, vectors; disposable and rebuildable from memory/
• grants/*.toml — owner-written
• audit/*.jsonl — hash-chained
• Secrets: behind a SecretStore trait. v0 backend is an encrypted file whose key is not stored beside it. No plaintext secrets on disk, none in config, none in the repo.

Code constraints

• Rust stable, Cargo workspace. One crate per role in the table above (loopd, brokerd, gatewayd, inferproxy), plus three that are not roles:

  Crate	Purpose
  proto	Defines the data every role shares and the frame format they exchange it in.
  toolkit	Holds the programs that run inside tool containers.
  bxctl	Is the owner's command line: chat, local approvals, checks and maintenance.

• No source file over 500 lines. No crate depends on another role's crate, only on proto.
• Authority is encoded in types: a tool cannot execute without a Decision value that only brokerd's policy module can construct. Decision is defined in brokerd, has a private field and does not implement Deserialize, so no other crate or wire message can produce one. proto carries a plain DecisionRecord for the audit log and the wire.
• Dependencies are few and justified in docs/dependencies.md. cargo-deny runs in the gate.
• make gate runs offline. Checks that need straylight run in make verify-device.
• No telemetry, no update checks, no outbound call not listed in docs/egress.md.