From 64a5931354963c4def2184a42cbfa6d31dddd3e4 Mon Sep 17 00:00:00 2001 From: "K. Isom" Date: Thu, 24 Sep 2026 13:49:28 -0700 Subject: [PATCH] P21 and the M5a spec: identity and experience; the archive becomes notes, and note is a memory action Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/decisions.md | 4 +- docs/specs/2026-09-24-m5a-local-memory.md | 94 +++++++++++++---------- 2 files changed, 56 insertions(+), 42 deletions(-) diff --git a/docs/decisions.md b/docs/decisions.md index 3b03918..8a92a0b 100644 --- a/docs/decisions.md +++ b/docs/decisions.md @@ -7,7 +7,7 @@ Newest first. A decision that changes `docs/design.md` lands in the same commit | Date | Decision | Reason | |---|---|---| | 2026-09-24 | M5a's check on memory writes stays short and fixed: control characters, invisible characters, chat-template markup, six phrases. A research thread on prompt-injection defences feeds a sanitizer module for memory writes (and possibly other persisted text), built as hardening in M5b. Memory writes are not posted to the chat; `bxctl memory log` lists them. | Owner. The archive routing of untrusted sessions' writes is the main defence; a pattern list is defence in depth and should rest on research, not guesses. | -| 2026-09-24 | Memory writes need no approval, as in Hermes (the owner's `~/.hermes` has no `write_approval`). Writes from a session that has seen untrusted content go to `memory/archive.md`, labelled untrusted, not into the prompt. Every write is scanned, journalled (listed by `bxctl memory log`, not posted anywhere) and can be undone; nothing is deleted. Brief change P21 proposed; the M5a spec is revised to match. Session-scoped approvals for other tools become an open decision. | Owner: an agent harness should act without a stream of questions, and reading and writing memory is what an agent is. The only risk the approval guarded is injected text reaching every future prompt, and that needs a session that read untrusted content; `brokerd` knows which sessions those are. | +| 2026-09-24 | Memory writes need no approval, as in Hermes (the owner's `~/.hermes` has no `write_approval`). What a session that has seen untrusted content wants remembered becomes a note, labelled untrusted, not identity in the prompt (refined the same day: identity and experience are separate kinds of memory). Every write is scanned, journalled (listed by `bxctl memory log`, not posted anywhere) and can be undone; nothing is deleted. Brief change P21 proposed; the M5a spec is revised to match. Session-scoped approvals for other tools become an open decision. | Owner: an agent harness should act without a stream of questions, and reading and writing memory is what an agent is. The only risk the approval guarded is injected text reaching every future prompt, and that needs a session that read untrusted content; `brokerd` knows which sessions those are. | | 2026-09-24 | M5a spec review: the policy of `recall` (`auto`) and `memory` (`ask`) is built into `brokerd`, and grant files cannot name them; tool calls are indexed by name and arguments, tool results are not; summaries of tool output, where a tool opts in, come with M5c's summariser. | Owner: an agent that cannot write memories is no better than a plain chat window, and a grant file must not be able to turn edits off or make them unattended. A web fetch's summary is worth keeping; most tools' output is not. | | 2026-09-24 | Compaction (new epochs) comes with the scheduler in M5c, not in M5a. Until then a session has one epoch, and M5a's rules say "the next session" where the brief says "the next epoch". `soul.md` starts from a generic file in the repository; `memory.md` and `user.md` start empty. M5a spec drafted: `docs/specs/2026-09-24-m5a-local-memory.md`. | Owner. `loopd` has no epochs yet (every session is epoch 0; a full context is `SessionFull`); compaction is an idle-time job on the background slot, which is what M5c builds. Over Mattermost each thread is a session, so a new thread already picks up approved edits. | | 2026-09-24 | M5 direction, before a spec. Memory is three local files and Honcho. The files: `memory/soul.md` (owner-written), `memory/memory.md` and `memory/user.md` (agent-edited, `ask`), read once per epoch, as Hermes does. Honcho, the fleet's memory service on rift (v3.2.0; models on dixie), reached by a new role, `memoryd`, which also owns the memory files and the SQLite index (owner, same day): each Boxmaker writes to its own workspace, only turns from sessions that saw no untrusted content, and can read other workspaces such as `hermes-fleet`, each with an owner-set trust level (untrusted by default). Local full-text search over the session logs stays. `consolidate` moves entries out of `memory.md` into `memory/archive.md` (searched by `recall`) and, from M5b, into Honcho. From M5c a scheduled job on the background slot proposes which entries stay at which tier, from counts of `recall` hits that `memoryd` keeps from M5a; its proposals are ordinary `ask` edits. The owner can mark any workspace trusted or untrusted; which ones is configuration, not design (the owner trusts the Hermes agents' workspace today). M5 splits into M5a (files and search, no network), M5b (`memoryd`, Honcho through the owner's `roncho` client) and M5c (the scheduler). Brief changes P17 to P20 proposed. | Owner, 2026-09-24: Honcho lets several Boxmakers share memory across the fleet; the Hermes agents do Boxmaker's job today and may be trusted. `brokerd` never holds conversation history and `loopd` has no network, so syncing turns needs a role of its own. Honcho's derivation replaces the brief's notes, pre-compaction flush and embedding index. | @@ -108,4 +108,4 @@ Later changes to the brief: | # | Change | Reason | |---|---|---| -| P21 | Memory 1 and 7: memory edits are not approved. `memory` writes are automatic; in a session that has seen untrusted content they are appended to `memory/archive.md`, labelled untrusted, and never reach `memory.md` or `user.md` (only `add` is possible there). Every write is checked (no control or invisible characters, no chat-template tokens, no instruction phrases from a short fixed list), and entries are checked again when the block is built. Every write is recorded in the edit journal, listed by `bxctl memory log` (not posted to the chat), and the latest edit of a file can be undone with `bxctl`. `consolidate` archives what it drops. An owner setting stages writes for review instead of applying them (off by default). The policy of `recall` and `memory` is built into `brokerd` and no grant file can change it. `soul.md` stays the owner's alone. | Owner, 2026-09-24, pointing to Hermes: writes without questions, a scan, a snapshot frozen per session, size limits. The archive routing covers what a pattern scan cannot: a plausible fact planted by a page. | +| P21 | Memory 1 to 8, State and Milestones: memory is **identity** and **experience** (owner, 2026-09-24). Identity is who the agent is and what it holds true about its work and owner: `soul.md` (the owner's), `memory.md`, `user.md`, in every prompt, written only by the owner or by sessions that have not read untrusted content. Experience is what it has seen or been told: `memory/notes.md` (replacing `archive.md`) and the session logs, never in the prompt, found with `recall`, each item labelled with its source and trust; not everything remembered is believed. Memory writes are not approved. The `memory` tool gains `note`, which records something seen in any session, labelled by the session's state. In a session that has read untrusted content an identity `add` becomes a `note` and the other identity edits are refused. Every write is checked (no control or invisible characters, no chat-template tokens, no instruction phrases from a short fixed list), and identity entries are checked again when the block is built. Every write is recorded in the edit journal and listed by `bxctl memory log` (not posted to the chat), and the latest edit of a file can be undone with `bxctl`. `consolidate` and `remove` put what they take out into `notes.md` as moved or dropped; nothing is deleted. An owner setting stages identity writes for review instead of applying them (off by default). The policy of `recall` and `memory` is built into `brokerd` and no grant file can change it. `recall` tells identity ("you knew") from notes ("you noted, from an untrusted session"). Milestones: M5a's row loses `ask` and gains notes; M5c's tiering job may promote only trusted notes into identity, and its writes are journalled like any other. | Owner, 2026-09-24, pointing to Hermes: writes without questions, a scan, a snapshot frozen per session, size limits; and memory is two things, who the agent is and what it has seen. Turning an untrusted session's writes into notes covers what a pattern scan cannot, a plausible fact planted by a page, without losing what was seen. | diff --git a/docs/specs/2026-09-24-m5a-local-memory.md b/docs/specs/2026-09-24-m5a-local-memory.md index f39e5c0..e97d971 100644 --- a/docs/specs/2026-09-24-m5a-local-memory.md +++ b/docs/specs/2026-09-24-m5a-local-memory.md @@ -28,8 +28,8 @@ Facts this rests on, checked on 2026-09-24: 2. An entry moved out of `memory.md` by `consolidate` is found by `recall`. 3. A memory write reaches the prompt only in the next session; a session already running keeps the bytes it started with. No write asks the owner anything. -4. A write from a session that has seen untrusted content lands in the archive, labelled - untrusted, and never in the prompt. +4. What a session that has seen untrusted content wants to remember becomes a note, labelled + untrusted: remembered and found by `recall`, never in the prompt. 5. The baseline with the starter `soul.md` stays at 3,000 tokens or less, measured on the device. 6. A hit from a session that had seen untrusted content comes back marked untrusted. 7. The owner can list every write (`bxctl memory log`), and undo the latest one of a file. @@ -57,7 +57,7 @@ any of them with an editor; `soul.md` only ever that way. | `soul.md` | the owner | 4,000 characters | a copy of the starter (`[memory] soul_starter`, default `soul.md` next to the config file) | | `memory.md` | `memoryd`, applying the agent's writes | 2,200 characters | empty | | `user.md` | `memoryd`, applying the agent's writes | 1,375 characters | empty | -| `archive.md` | `memoryd`, appending moved and dropped entries, and writes from untrusted sessions | none | empty | +| `notes.md` | `memoryd`, appending notes (things seen) and entries moved or dropped from the files above | none | empty | | `edits/YYYY-MM.jsonl` | `memoryd`, appending one line per applied edit | none | absent | | `pending.jsonl` | `memoryd`, when review is on (section 5) | none | absent | | `recalls/YYYY-MM.jsonl` | `memoryd`, appending one line per `recall` | none | absent | @@ -69,23 +69,31 @@ counted with `chars()`. Limits are on the file's text. (as Hermes does). An entry is trimmed, is not empty, and may span lines, but may not contain a line holding only `§`. Two identical entries in one file are refused. -**`archive.md`.** Append-only. Each entry starts with one header line saying how it got there: +**Two kinds of memory** (owner, 2026-09-24). **Identity** is who the agent is and what it holds +true about its work and its owner: `soul.md`, `memory.md`, `user.md`, in every prompt, written +only by the owner or by sessions that have not read untrusted content. **Experience** is what it +has seen or been told: `notes.md` and the session logs, never in the prompt, found with `recall`, +each item with its own label. Not everything remembered is believed, but it is remembered. + +**`notes.md`.** Append-only. Each entry starts with one header line saying how it got there: ```text +§ noted 2026-09-24T21:02Z in session mm-dh45qg… · untrusted + § moved 2026-09-24T20:14Z from memory.md in session mm-y73sxj3popgwumyccptb56p1ah § dropped 2026-09-24T20:14Z from memory.md in session mm-y73sxj3popgwumyccptb56p1ah -§ written 2026-09-24T21:02Z for user.md in session mm-dh45qg… · untrusted - ``` -Nothing a write removes is lost: moved and dropped entries are archived, and the journal keeps +A `noted` entry is labelled by the session that wrote it: untrusted if it had read untrusted +content, trusted otherwise; its class is the session's. `moved` and `dropped` entries were +identity, so they are trusted. Nothing a write removes is lost: moved and dropped entries go to `notes.md`, and the journal keeps every before and after. **The edit journal** (`edits/`) records every applied edit: its id (`YYYY-MM-DD-NNNN`), time, -session, call, action, file, where it went (`memory.md`, `user.md` or the archive), the entries -before and after, and whether the session had seen untrusted content. It is the history `archive.md` alone cannot give (when +session, call, action, file, where it went (`memory.md`, `user.md` or `notes.md`), the entries +before and after, and whether the session had seen untrusted content. It is the history `notes.md` alone cannot give (when an entry first appeared), and the record an owner reads when a memory looks wrong. **The recall log** (`recalls/`) records time, session, query, and the source of every hit, and the queries that found nothing: hit counts for M5c's tiering job are worked out from it, so they survive a reindex. @@ -132,11 +140,13 @@ its source and label, and each snippet at most 600 characters: ```text [1] session mm-8ass9m…, 2026-09-20 14:02 UTC, the owner said · private …the launch window moved to Thursday… -[2] archive.md, moved 2026-09-24 from memory.md · private +[2] you knew this: moved 2026-09-24 from memory.md · private Owner prefers answers in UTC. [3] session chat-1789…, 2026-09-21 09:15 UTC, the assistant said · private · untrusted … -[4] session mm-dh45qg…, 2026-09-22 18:17 UTC, the assistant called · private +[4] you noted this, 2026-09-22 18:20 UTC, in a session that had read untrusted content · private · untrusted +example.org says the release moved to March. +[5] session mm-dh45qg…, 2026-09-22 18:17 UTC, the assistant called · private http_fetch {"url":"https://example.org/notes"} ``` @@ -151,12 +161,14 @@ stricter of each (section 8). ### `memory` -Arguments, one of (`file` is `"memory"` or `"user"`): +Arguments, one of (`file` is `"memory"` or `"user"`). `note` records something seen or told +(experience); the others edit identity: ```json {"action": "add", "file": "memory", "text": "…"} {"action": "replace", "file": "memory", "old": "", "text": "…"} {"action": "remove", "file": "user", "old": ""} +{"action": "note", "text": "…"} {"action": "consolidate", "file": "memory", "entries": [ {"old": "", "keep": true}, {"old": "", "text": ""}, @@ -172,25 +184,27 @@ the file fails safely: every `old` must match an entry of the file as it is now, **Where a write goes.** `brokerd` sends each call with the session's state (section 7): -- In a session that has **not** seen untrusted content, the write applies to `memory.md` or - `user.md`. -- In a session that **has**, only `add` is possible, and the entry is appended to `archive.md` - with the header `§ written … · untrusted`, never to the file named. `recall` finds it, labelled - untrusted; it never reaches the prompt on its own. The answer says so: `saved to the archive, - not memory.md: this session has read untrusted content`. `replace`, `remove` and `consolidate` - are refused with `this session has read untrusted content; it can add to the archive but not - change memory.md`. The owner can move an archived entry into a file by editing it. +- `note` is appended to `notes.md` in any session, labelled by the session (above). The answer: + `noted (untrusted)` or `noted`. +- In a session that has **not** seen untrusted content, the identity actions apply to `memory.md` + or `user.md`. +- In a session that **has**, an identity `add` becomes a `note`, and the answer says so: `saved as + a note (untrusted): this session has read untrusted content`. It is remembered and `recall` + finds it; it never reaches the prompt on its own. `replace`, `remove` and `consolidate` are + refused with `this session has read untrusted content; it can take notes but not change + memory.md`. The owner can move a note into a file by editing it. **Rules**, checked by `brokerd` when parsing (shape) and by `memoryd` when applying (content): - `add`: the new file must fit its limit; a duplicate is refused. -- `replace`, `remove`: `old` must match exactly one entry. A removed entry is archived as dropped. +- `replace`, `remove`: `old` must match exactly one entry. A removed entry goes to `notes.md` as dropped. +- `note`: 1 to 1,000 characters; the check applies. - `consolidate`: every current entry of the file must appear exactly once as an `old`, each with exactly one of `keep`, `text`, `move`, `drop`; entries with only `text` are new. The resulting - file must fit its limit. Moved and dropped entries are appended to `archive.md` in the order + file must fit its limit. Moved and dropped entries are appended to `notes.md` in the order given, with their headers. - An edit that would go over the limit is refused with: `memory.md would be 2,350 of 2,200 - characters; use consolidate to move entries to the archive`. That sentence is fixed apart from + characters; use consolidate to move entries to notes`. That sentence is fixed apart from the numbers and the file. **The check.** Every new or rewritten entry is refused, with a fixed sentence naming which rule, @@ -204,13 +218,13 @@ if it contains: `ignore all previous`, `disregard previous instructions`, `disregard all previous`, `new instructions:`, `system prompt:`. -The list is short and fixed on purpose: it is defence in depth, as Hermes' scan is. The archive -routing above is the real defence, because a pattern cannot tell a plausible planted fact from a +The list is short and fixed on purpose: it is defence in depth, as Hermes' scan is. Turning an untrusted +session's writes into notes is the real defence, because a pattern cannot tell a plausible planted fact from a true one. **Applying.** The file is written to a temporary file and renamed, then the journal line is appended. The answer to the model: `memory.md updated (1,510 of 2,200 characters); it takes -effect in the next session`, or the archive sentence above. +effect in the next session`, or the note sentences above. **Seeing writes.** Writes are not posted anywhere (owner, 2026-09-24): the journal is the record, and `bxctl memory log` shows it. @@ -218,34 +232,33 @@ record, and `bxctl memory log` shows it. **Undo.** `bxctl memory log [N]` prints the last N edits (default 10) from the journal: id, time, session, action, file. `bxctl memory undo ` reverts an edit if it is the **latest** edit of its file and the file still holds exactly what that edit left: the file gets its entries back from -the journal, entries the edit archived stay in the archive (append-only), and the undo is itself +the journal, entries the edit moved to `notes.md` stay there (append-only), and the undo is itself journalled. Anything else is refused with the reason, so an undo never overwrites a later change. **Review (off by default).** With `[memory] review = true`, a write that would change `memory.md` or `user.md` is appended to `pending.jsonl` instead, and the answer says it is waiting for the owner. `bxctl memory pending` lists them; `bxctl memory accept ` applies one (with the same checks, against the file as it is then); `bxctl memory reject ` discards it (journalled). -Writes to the archive are never staged. +Notes are never staged. ## 6. The index `{home}/index/memory.sqlite3`, directory 0700, file 0600, owned by `memoryd`. Disposable: deleting -it and restarting `memoryd`, or `bxctl reindex`, rebuilds it from `sessions/` and `archive.md`. +it and restarting `memoryd`, or `bxctl reindex`, rebuilds it from `sessions/` and `notes.md`. **What is indexed.** From session logs: the `content` of `User` records, of `Assistant` records that have content, and each tool call the assistant made, as its tool name and arguments (`read_file {"path":"/srv/notes/q3.txt"}`), so `recall` can answer "did we read that, and when". Not tool results (the bulk of every log, and the way injected text arrives) and not `reasoning_content`. A summary of a tool's output is M5c's (owner, 2026-09-24): it needs a model -to write it. From `archive.md`: every entry. +to write it. From `notes.md`: every entry. `memory.md` and `user.md` are not indexed: they are already in every prompt. **Labels.** Walking a session log in order, `memoryd` keeps the session's label so far: the highest `class` of any `ToolResult` before this record (at least `private`), and whether any was untrusted. Each indexed record gets the label in force when it was written. A record whose class -is above `[index] max_class` (default `private`) is not indexed at all. Archive entries are -labelled `private`, and untrusted when their header says so; moved and dropped entries came from -`memory.md` or `user.md`, which only clean sessions write, so they are trusted. +is above `[index] max_class` (default `private`) is not indexed at all. `notes.md` entries carry +the label their header gives (section 3). **Keeping up.** The index stores, per session log, how many records it has read. After each finished turn `loopd` sends `memoryd` `session_updated` with the session id (section 7), and @@ -281,7 +294,7 @@ connection, the existing frames (`proto::frame`, `MAX_FRAME`). New message kinds | `session_updated` | `{"session": SessionId}` | | `memory_call` | `{"request": ToolRequest, "taint": DataClass, "untrusted": bool}`: the decided call, with the session's state when `brokerd` decided it | | `reindex` | `{}` | -| `reindexed` | `{"logs": u64, "records": u64, "archive": u64}` | +| `reindexed` | `{"logs": u64, "records": u64, "notes": u64}` | | `memory_admin` | `{"command": "log" \| "undo" \| "pending" \| "accept" \| "reject", "arg": string}` | | `memory_admin_reply` | `{"text": string}` | @@ -347,7 +360,7 @@ connection, the existing frames (`proto::frame`, `MAX_FRAME`). New message kinds ## 10. `bxctl` - `bxctl reindex [--admin-socket PATH]`: sends `reindex`, waits, and prints `reindexed: 42 logs, - 1,208 records, 7 archive entries`. `memoryd` rebuilds into a new file and renames it over the + 1,208 records, 7 notes`. `memoryd` rebuilds into a new file and renames it over the old one, so `recall` keeps working meanwhile. - `bxctl memory log [N]`, `bxctl memory undo `, and with review on, `bxctl memory pending`, `accept `, `reject ` (section 5). Entries are printed escaped, as approvals are. @@ -375,8 +388,8 @@ max_class = "private" # records above this class are not indexed - `memoryd` has no network in M5a and never writes `sessions/`, grants or the audit log. - What reaches the prompt: `soul.md`, the owner's file; and `memory.md` and `user.md`, which only sessions that have not read untrusted content write, through the check, frozen per session. - Everything a session that read untrusted content writes goes to the archive, labelled - untrusted, where it can be recalled but never lands in the prompt by itself. + Everything a session that read untrusted content wants remembered becomes a note, labelled + untrusted, which can be recalled but never lands in the prompt by itself. - Memory writes are the one tool with no approval anywhere. The trade is deliberate (owner, 2026-09-24): the risk it leaves is a clean session writing something false, which is the agent's judgement of what the owner said; every such write is journalled, listed by `bxctl @@ -402,7 +415,8 @@ Offline, in `make gate`: - `memoryd`: entries parsing and rendering; each edit's rules, including a stale `old`, a duplicate, the limit sentence, and `consolidate` missing an entry or naming one twice; every rule of the check, on write and on building the block; routing by the session's state (an - untrusted `add` to the archive, the other actions refused); the archive headers and journal + untrusted `add` becoming a note, the other identity actions refused; `note` in both kinds of + session); the `notes.md` headers and labels, and the journal lines; undo of the latest edit, and its refusal when the file changed; review on and off; the block; the quoted-terms query builder against FTS5 operators (`NEAR`, `*`, `^`, quotes, column filters); labels walked from a log; `max_class`; catching up from `progress`; a reindex equal to the incremental index; the recall log. @@ -416,7 +430,7 @@ Offline, in `make gate`: On the device (the owner's Mattermost, as for M4b): the baseline token count with the starter soul; a fact told in one thread and recalled in another; an `add`, made with no question and listed by `bxctl memory log`, that appears in a new thread and not in the thread that wrote it; a `consolidate` that -moves an entry, which `recall` then finds; a write after a fetched page that lands in the archive, +moves an entry, which `recall` then finds; a write after a fetched page that becomes an untrusted note, untrusted; `bxctl memory undo` of the latest edit. ## 15. For the owner's review @@ -426,7 +440,7 @@ untrusted; `bxctl memory undo` of the latest edit. summariser, labelled like the output they summarise. 2. Decided: memory writes need no approval (P21); the policy of `recall` and `memory` is built in and no grant file can change it (section 8). A session that read untrusted content writes only - to the archive (section 5). + to `notes.md`, as notes (section 5). Identity and experience are separate (section 3). 3. Decided: entries are named by their full text, not by number (section 5). 4. Decided: the starter `soul.md` is generic (section 3). 5. Decided: M5a starts with the check as written, the short fixed phrase list and rules 1 to 3