Apply P17: memory is three files read once per epoch; the memory tool with consolidate

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

docs/decisions.md

@@ -95,6 +95,7 @@ Later changes to the brief:
 | 2026-09-22 | P13, Authority contract 4: tool containers run from one image built from source by Nix and named by digest; nothing is pulled at call time. | Approved with the M3 design (2026-09-18); applied when the M3b spec came to rely on it. A pull at call time would be unlisted egress. |
 | 2026-09-23 | P15, State, the secrets line: three `SecretStore` backends chosen per secret (a systemd credential by default, an environment variable, an owner-only file), the last two in plaintext and for portability, a file secret warning at startup. | Owner's decision and approval, 2026-09-23 (M4a). The brief's "no plaintext secrets on disk" did not allow the file backend. |
 | 2026-09-24 | P16, Network isolation 3: approvals are commands (`!approve`, `!deny`) replied in the approval's thread; emoji reactions are not used. | Owner's decision and approval, 2026-09-24; M4b spec `docs/specs/2026-09-24-m4b-approvals.md`. |
+| 2026-09-24 | P17, Memory 1, the baseline and State: the core block becomes three files (`soul.md`, owner-written; `memory.md` and `user.md`, agent-edited at `ask` with add, replace, remove and `consolidate`), read once per epoch; moved entries go to `memory/archive.md`. | Owner's approval, 2026-09-24; see the row of that date under Decided. |
 
 ## Proposed changes to the design brief
 
@@ -102,7 +103,6 @@ Proposed 2026-09-24 for M5 (row of that date under Decided). Not yet applied.
 
 | # | Change | Reason |
 |---|---|---|
-| P17 | Memory 1 and State: the core block becomes three files, read once per epoch into the baseline, in this order after the system prompt: `memory/soul.md` (persona; owner-written only; at most 4,000 characters), `memory/memory.md` and `memory/user.md` (the agent's core notes on its work and on the owner; at most 2,200 and 1,375 characters; each shown with how full it is). The agent edits the last two with a `memory` tool: `add`, `replace` and `remove` an entry, and `consolidate`, one edit that marks every entry of a file kept, rewritten, moved or dropped. Moved entries are appended to `memory/archive.md` (append-only, each entry with the date and session it was first approved in), where `recall` finds them; nothing leaves `memory.md` silently. Every edit is `ask`, and the approval shows `consolidate` as a diff. An approved edit takes effect at the next epoch. A write over the cap is refused with a pointer to `consolidate`. The baseline stays at 3,000 tokens or less, measured on the device. `core.md` and its 300-token limit go. | Owner: as Hermes does (`~/.hermes/SOUL.md`, `memories/MEMORY.md`, `USER.md`, frozen at session start for the prompt cache); and `consolidate` moves memories out of the fast path rather than losing them (owner, 2026-09-24). The owner's current files are 2,870, 2,151 and 790 characters, about 1,500 tokens; the rest of the baseline measured 251 tokens at M2b. With every edit at `ask`, one consolidation is one approval, not one per entry. |
 | P18 | Roles: a sixth role, `memoryd`, the one owner of memory. Purpose: keeps the memory files and the index, syncs finished turns to Honcho, and answers memory tool calls. Holds: `memory/` (the three files and `archive.md`), `index/` (SQLite), Honcho's address and workspace list (and a key, if Honcho's authentication is on). Can reach: Honcho only (from M5b); a socket from `loopd` (finished turns in; the three files out, once per epoch) and one from `brokerd` (memory tool calls it has decided). Reads `sessions/` for `bxctl reindex`, never writes it. Never holds: grants, tool secrets, the Mattermost token. Sessions above a configured data class are not indexed. `loopd` loses "memory files" from its Holds column, keeps no network, and gains a fourth socket. | `brokerd` never holds conversation history and `loopd` never has a network; syncing turns needs both conversation text and the network. One owner for every memory tool means `brokerd` decides and `memoryd` executes, with no call from `brokerd` into `loopd`, and SQLite is linked by one crate. The cost: a copy of session text in a process that reaches Honcho; `memoryd` is not steered by a model, its network is Honcho only (enforced in M7), and the most sensitive classes are not indexed. |
 | P19 | Memory 2 to 9: notes files, the pre-compaction flush, the embedding index (M5b) and "no vector database service" are replaced. `recall` searches the local full-text index (SQLite FTS5 over the session logs and `memory/archive.md`, rebuildable with `bxctl reindex`) and Honcho; `ask_memory` asks Honcho's dialectic. Both go through `brokerd` at `auto` and run in `memoryd`. Entries found in `archive.md` come back as approved memory. `memoryd` counts `recall` hits per archive entry and when each was last found, for the tiering job (P20, M5c). From M5b, entries moved by `consolidate` are also written to Boxmaker's Honcho workspace as conclusions. `memoryd` writes to Honcho only turns from sessions that saw no untrusted content and are within a configured data class, and it enforces that itself. Each workspace read has an owner-set trust level; results from an untrusted workspace (the default) are marked untrusted and taint the session. Trust is per workspace because its writers differ; the brief names no workspace. Every result carries its source (the file and line, or the workspace) and data class. "No memory text leaves the host" becomes "no memory text leaves the owner's hosts": Honcho runs on rift with its models on dixie. | Honcho's deriver, summaries and dreamer do the extraction the flush job was for, and `honcho-embed` on dixie is the embedding index. Honcho's conclusions carry no per-entry provenance, so the untrusted filter must be applied when writing, and each workspace is labelled as a whole. |
 | P20 | Milestones: M5 becomes M5a (`memoryd` without a network: the three files, the `memory` tool with `consolidate`, the archive, full-text `recall`), M5b (`memoryd` and Honcho) and M5c (heartbeat and cron on their own slot). The old M5b row goes. Acceptance moves with the parts: a fact from epoch 1 is recalled in epoch 2, and an entry moved by `consolidate` is found by `recall` (M5a); a turn from a tainted session never reaches Honcho, and a result from an untrusted workspace taints the session (M5b); a scheduled run leaves the main slot's cache intact, and the tiering job's proposals reach the owner as ordinary approvals and change nothing on their own (M5c). M5c's tiering job: a scheduled session that reads the memory files, the archive with its hit counts, and Boxmaker's Honcho summaries, and proposes a `consolidate` (moving entries down) or an `add` naming its source (bringing one up), when a file passes about 80% full or weekly, at most one per file per run. | As for M2 to M4: the parts are reviewed separately, and M5a needs no network. |

docs/design.md

@@ -85,9 +85,9 @@ ordinary outbound networking can reach the whole tailnet and the internet as the
 
 ## Inference contract (the part existing harnesses get wrong)
 
-1. **Baseline budget.** System prompt plus tool schemas at session start: 3,000 tokens or less,
-   measured with the server's tokenizer (`/tokenize`), enforced by a test. That test needs
-   straylight, so it runs in `make verify-device`, not in the offline `make gate`.
+1. **Baseline budget.** System prompt, tool schemas and the three memory files at session start:
+   3,000 tokens or less, measured with the server's tokenizer (`/tokenize`), enforced by a test.
+   That test needs straylight, so it runs in `make verify-device`, not in the offline `make gate`.
 2. **Append-only.** The request for turn N+1 is a strict extension of the request for turn N.
    Nothing volatile (time, heartbeat notes, memory refreshes) goes anywhere but the newest message.
    Enforced by a property test on the serialized message array. The session log stores each
@@ -173,9 +173,17 @@ a multi-turn tool conversation and tool parsing had no failures in 20 trials.
 Memory is what carries continuity across epoch compaction, so it is required. It must not break
 the inference contract, and it must not become a way for injected text to persist.
 
-1. **Core block.** `memory/core.md`, owner-curated, 300 tokens or less, part of the baseline.
-   Edits take effect at the next epoch, never mid-session. The agent can propose changes; only an
-   `ask` approval or the owner's editor applies them.
+1. **Memory files.** Three files, read once per epoch into the baseline, in this order after the
+   system prompt: `memory/soul.md`, the persona, written only by the owner (at most 4,000
+   characters); `memory/memory.md`, the agent's core notes on its work (at most 2,200 characters);
+   and `memory/user.md`, its core notes on the owner (at most 1,375 characters). Each file is shown
+   with how full it is. The agent edits the last two with a `memory` tool: `add`, `replace` and
+   `remove` an entry, and `consolidate`, one edit that marks every entry of a file kept,
+   rewritten, moved or dropped. Moved entries are appended to `memory/archive.md` (append-only,
+   each entry with the date and session it was first approved in), where `recall` finds them;
+   nothing leaves `memory.md` silently. Every edit is `ask`, and the approval shows `consolidate`
+   as a diff. An approved edit takes effect at the next epoch, never mid-session. A write over a
+   file's limit is refused with a pointer to `consolidate`.
 2. **Everything else is retrieved, not injected.** A `recall(query, k)` tool returns size-capped
    snippets with file, line and provenance. Results are appended like any tool result, so the
    cache prefix is untouched.
@@ -204,7 +212,7 @@ the inference contract, and it must not become a way for injected text to persis
 Files are the source of truth. SQLite is allowed only for rebuildable indexes and queues.
 
 - `sessions/<id>/<epoch>.jsonl` — append-only session log
-- `memory/core.md`, `memory/notes/*.md`, `memory/topics/*.md` — owner-readable memory
+- `memory/soul.md`, `memory/memory.md`, `memory/user.md`, `memory/archive.md` — owner-readable memory
 - `index/` — FTS5 and, later, vectors; disposable and rebuildable from `memory/`
 - `grants/*.toml` — owner-written
 - `audit/*.jsonl` — hash-chained