From 817fb9d47ff1bb58e995e090da6373bb38a99d6c Mon Sep 17 00:00:00 2001 From: "K. Isom" Date: Thu, 24 Sep 2026 13:33:37 -0700 Subject: [PATCH] M5a spec: built-in policy for recall and memory; tool calls indexed; output summaries in M5c Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/decisions.md | 1 + docs/milestones.md | 2 +- docs/specs/2026-09-24-m5a-local-memory.md | 39 +++++++++++++++-------- 3 files changed, 28 insertions(+), 14 deletions(-) diff --git a/docs/decisions.md b/docs/decisions.md index 39c1ee5..50b9fe7 100644 --- a/docs/decisions.md +++ b/docs/decisions.md @@ -6,6 +6,7 @@ Newest first. A decision that changes `docs/design.md` lands in the same commit | Date | Decision | Reason | |---|---|---| +| 2026-09-24 | M5a spec review: the policy of `recall` (`auto`) and `memory` (`ask`) is built into `brokerd`, and grant files cannot name them; tool calls are indexed by name and arguments, tool results are not; summaries of tool output, where a tool opts in, come with M5c's summariser. | Owner: an agent that cannot write memories is no better than a plain chat window, and a grant file must not be able to turn edits off or make them unattended. A web fetch's summary is worth keeping; most tools' output is not. | | 2026-09-24 | Compaction (new epochs) comes with the scheduler in M5c, not in M5a. Until then a session has one epoch, and M5a's rules say "the next session" where the brief says "the next epoch". `soul.md` starts from a generic file in the repository; `memory.md` and `user.md` start empty. M5a spec drafted: `docs/specs/2026-09-24-m5a-local-memory.md`. | Owner. `loopd` has no epochs yet (every session is epoch 0; a full context is `SessionFull`); compaction is an idle-time job on the background slot, which is what M5c builds. Over Mattermost each thread is a session, so a new thread already picks up approved edits. | | 2026-09-24 | M5 direction, before a spec. Memory is three local files and Honcho. The files: `memory/soul.md` (owner-written), `memory/memory.md` and `memory/user.md` (agent-edited, `ask`), read once per epoch, as Hermes does. Honcho, the fleet's memory service on rift (v3.2.0; models on dixie), reached by a new role, `memoryd`, which also owns the memory files and the SQLite index (owner, same day): each Boxmaker writes to its own workspace, only turns from sessions that saw no untrusted content, and can read other workspaces such as `hermes-fleet`, each with an owner-set trust level (untrusted by default). Local full-text search over the session logs stays. `consolidate` moves entries out of `memory.md` into `memory/archive.md` (searched by `recall`) and, from M5b, into Honcho. From M5c a scheduled job on the background slot proposes which entries stay at which tier, from counts of `recall` hits that `memoryd` keeps from M5a; its proposals are ordinary `ask` edits. The owner can mark any workspace trusted or untrusted; which ones is configuration, not design (the owner trusts the Hermes agents' workspace today). M5 splits into M5a (files and search, no network), M5b (`memoryd`, Honcho through the owner's `roncho` client) and M5c (the scheduler). Brief changes P17 to P20 proposed. | Owner, 2026-09-24: Honcho lets several Boxmakers share memory across the fleet; the Hermes agents do Boxmaker's job today and may be trusted. `brokerd` never holds conversation history and `loopd` has no network, so syncing turns needs a role of its own. Honcho's derivation replaces the brief's notes, pre-compaction flush and embedding index. | | 2026-09-24 | M4b, changing the agreed outline (row of 2026-09-23): no reactions, an approval is answered only by `!approve ` or `!deny [reason]` in its thread; an `[approvals] users` list in `gatewayd.toml`, which falls back to `allow.users` when empty; approvals are posted in the asking thread, channel threads included; on restart, `gatewayd` marks its old approval posts dead and `brokerd`'s courier posts pending approvals again. Spec `docs/specs/2026-09-24-m4b-approvals.md`. Brief change P16 proposed. Also: `gatewayd` strips a leading mention of its own bot from every message before handling it; in a thread shared by several Boxmakers the operator addresses approval commands, and a bare one is taken by every Boxmaker with that id pending there. | Owner: reactions are not used. Approvers are separate because an agent allowed to talk to Boxmaker should not be able to approve for it, but the default keeps one list for one person. Redacting secrets from arguments comes with the tools work. A restart must not leave posts that look answerable and are not. The mention is how the operator chooses the bot; the frontend removes it so neither the commands nor the model see it. | diff --git a/docs/milestones.md b/docs/milestones.md index 6f45066..cd9a540 100644 --- a/docs/milestones.md +++ b/docs/milestones.md @@ -15,7 +15,7 @@ One session per milestone. The design brief is `docs/design.md`. | M4b | Approvals over Mattermost | `gateway.sock`, approval posts, answers by `!approve`/`!deny` in the thread, approvers list, long approvals in parts | An `ask` call approved and refused from Mattermost; the audit log names the post | | M5a | Local memory | `memoryd` without a network: the three memory files served to `loopd` once per epoch, the `memory` tool (add, replace, remove, `consolidate`) at `ask`, `memory/archive.md`, FTS5 `recall` over session logs and the archive, `recall` hit counts, `bxctl reindex` | A fact from one session is recalled in another; an entry moved by `consolidate` is found by `recall`; a memory edit changes the prompt only in the next session (next epoch once M5c adds epochs); the baseline with the owner's files stays under 3,000 tokens | | M5b | Honcho | `memoryd` syncs finished turns to its own Honcho workspace through `roncho`, `recall` and `ask_memory` over Honcho, other workspaces read with the owner's trust labels, moved entries written as conclusions | A turn from a tainted session never reaches Honcho; a result from an untrusted workspace taints the session; Honcho's contents are rebuilt from the logs | -| M5c | Scheduler and compaction | Heartbeat and cron on their own slot in isolated sessions; compaction at idle, starting a new epoch from the baseline and a summary; the tiering job, which proposes what stays at which tier | A scheduled run leaves the main slot's cache intact; a fact from epoch 1 is recalled in epoch 2, and a memory edit reaches the next epoch; the tiering job's proposals reach the owner as ordinary approvals and change nothing on their own | +| M5c | Scheduler and compaction | Heartbeat and cron on their own slot in isolated sessions; compaction at idle, starting a new epoch from the baseline and a summary; summaries of tool output for the tools that opt in (a web fetch, say), indexed with the output's label; the tiering job, which proposes what stays at which tier | A scheduled run leaves the main slot's cache intact; a fact from epoch 1 is recalled in epoch 2, and a memory edit reaches the next epoch; the tiering job's proposals reach the owner as ordinary approvals and change nothing on their own | | M6 | Subagents + cloud consult | Subagents on their slot returning summaries; consult tool gated by data-class grants with full payload log | A tainted session is refused; payload log matches what was sent | | M7 | Split deployment | Each role in its own rootless container; `loopd` with no network; socket volume permissions; egress proxy for tools; host ACL policy in `deploy/` | From inside the `loopd` container, requests to the internet, the tailnet and the host all fail; a tool container without a host grant cannot reach the tailnet | diff --git a/docs/specs/2026-09-24-m5a-local-memory.md b/docs/specs/2026-09-24-m5a-local-memory.md index 3373af1..c5d3c37 100644 --- a/docs/specs/2026-09-24-m5a-local-memory.md +++ b/docs/specs/2026-09-24-m5a-local-memory.md @@ -121,6 +121,8 @@ its source and label, and each snippet at most 600 characters: Owner prefers answers in UTC. [3] session chat-1789…, 2026-09-21 09:15 UTC, the assistant said · private · untrusted … +[4] session mm-dh45qg…, 2026-09-22 18:17 UTC, the assistant called · private +http_fetch {"url":"https://example.org/notes"} ``` `(no matches for "…")` when nothing is found. The query goes into FTS5 only as quoted terms: it @@ -177,9 +179,12 @@ appended. `{home}/index/memory.sqlite3`, directory 0700, file 0600, owned by `memoryd`. Disposable: deleting it and restarting `memoryd`, or `bxctl reindex`, rebuilds it from `sessions/` and `archive.md`. -**What is indexed.** From session logs: the `content` of `User` records and of `Assistant` -records that have content. Not tool results (the bulk of every log, and the way injected text -arrives), not `reasoning_content`, not tool call arguments. From `archive.md`: every entry. +**What is indexed.** From session logs: the `content` of `User` records, of `Assistant` records +that have content, and each tool call the assistant made, as its tool name and arguments +(`read_file {"path":"/srv/notes/q3.txt"}`), so `recall` can answer "did we read that, and when". +Not tool results (the bulk of every log, and the way injected text arrives) and not +`reasoning_content`. A summary of a tool's output is M5c's (owner, 2026-09-24): it needs a model +to write it. From `archive.md`: every entry. `memory.md` and `user.md` are not indexed: they are already in every prompt. **Labels.** Walking a session log in order, `memoryd` keeps the session's label so far: the @@ -230,10 +235,11 @@ connection, the existing frames (`proto::frame`, `MAX_FRAME`). New message kinds - **Tools.** `ToolName` gains `Recall` and `Memory`; `ToolArgs` gains their parsed arguments (`proto` holds the serde structs, as for the other tools). Parsing enforces the shapes and limits in section 5. Neither tool has constraints (`paths`, `hosts`, `patterns` must be empty). -- **Grants.** Ordinary grant files. The shipped examples: +- **Grants: built in** (owner, 2026-09-24: an agent that cannot write memory is not worth + running). `brokerd` holds two fixed grants, always present whatever is in `grants/`: ```toml - # grants/recall.toml + # built in: recall tool = "recall" mode = "auto" max_taint = "secret" @@ -242,7 +248,7 @@ connection, the existing frames (`proto::frame`, `MAX_FRAME`). New message kinds ``` ```toml - # grants/memory.toml + # built in: memory tool = "memory" mode = "ask" max_taint = "secret" @@ -250,8 +256,13 @@ connection, the existing frames (`proto::frame`, `MAX_FRAME`). New message kinds untrusted = false ``` - A grant of `memory` at `auto` is refused when grants are loaded: every memory edit is the - owner's (brief, memory 7). The fixed sentence names the grant. + Their names are `builtin-recall` and `builtin-memory`, and their `grant_sha256` is the hash of + that text, so the audit log names them like any other grant. A file in `grants/` whose `tool` + is `recall` or `memory` is a grant problem at load (`grants-invalid`), with the fixed sentence + `the policy of recall and memory is built in; remove this grant`. So no grant file can turn + memory edits off, or make them unattended; every edit is the owner's (brief, memory 7). With + section 9 (no session without the memory block), an agent that can talk can always read its + memory and propose edits to it. - **Where a decided call runs.** A composite runtime sends `Recall` and `Memory` specs to `memoryd`'s `brokerd` socket as a `tool_request`, and everything else to the configured runtime (Podman, or the refusing one). `[memory] socket` in `brokerd.toml` names the socket; without it, @@ -326,7 +337,8 @@ Offline, in `make gate`: archive and journal lines; the block; the quoted-terms query builder against FTS5 operators (`NEAR`, `*`, `^`, quotes, column filters); labels walked from a log; `max_class`; catching up from `progress`; a reindex equal to the incremental index; the recall log. -- `brokerd`: parsing both tools; `memory` at `auto` refused at load; the composite runtime; the +- `brokerd`: parsing both tools; the built-in grants present with an empty `grants/`; a grant + file naming either tool refused at load; the composite runtime; the stricter label, with a runtime that returns a stricter and a laxer one. - `loopd`: the baseline with a fake `memoryd`; the refusal and its pointer; `session_updated` sent after a finished turn and not after a turn ending in tool calls. @@ -339,10 +351,11 @@ which `recall` then finds; a hit from a session that read a fetched page coming ## 15. For the owner's review -1. Tool results are not indexed (section 6). It keeps injected text out of `recall`, and keeps - the index small; the price is that `recall` cannot find something only a tool returned, such - as a file's contents, unless the assistant repeated it. -2. `memory` at `auto` is refused at load (section 8), not merely discouraged. +1. Decided: tool results are not indexed; tool calls are, by name and arguments (section 6). + Summaries of a tool's output, where the tool opts in (a web fetch, say), come with M5c's + summariser, labelled like the output they summarise. +2. Decided: the policy of `recall` (`auto`) and `memory` (`ask`) is built in, and grant files + cannot name them (section 8): memory edits can be neither turned off nor made unattended. 3. Entries are named by their full text, not by number (section 5), so approvals are readable and stale edits fail; the arguments of a `consolidate` of a full `memory.md` are about 4,500 characters, one approval post.