Draft the M5a spec (local memory); compaction moves to M5c
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -6,6 +6,7 @@ Newest first. A decision that changes `docs/design.md` lands in the same commit
|
|||||||
|
|
||||||
| Date | Decision | Reason |
|
| Date | Decision | Reason |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
|
| 2026-09-24 | Compaction (new epochs) comes with the scheduler in M5c, not in M5a. Until then a session has one epoch, and M5a's rules say "the next session" where the brief says "the next epoch". `soul.md` starts from a generic file in the repository; `memory.md` and `user.md` start empty. M5a spec drafted: `docs/specs/2026-09-24-m5a-local-memory.md`. | Owner. `loopd` has no epochs yet (every session is epoch 0; a full context is `SessionFull`); compaction is an idle-time job on the background slot, which is what M5c builds. Over Mattermost each thread is a session, so a new thread already picks up approved edits. |
|
||||||
| 2026-09-24 | M5 direction, before a spec. Memory is three local files and Honcho. The files: `memory/soul.md` (owner-written), `memory/memory.md` and `memory/user.md` (agent-edited, `ask`), read once per epoch, as Hermes does. Honcho, the fleet's memory service on rift (v3.2.0; models on dixie), reached by a new role, `memoryd`, which also owns the memory files and the SQLite index (owner, same day): each Boxmaker writes to its own workspace, only turns from sessions that saw no untrusted content, and can read other workspaces such as `hermes-fleet`, each with an owner-set trust level (untrusted by default). Local full-text search over the session logs stays. `consolidate` moves entries out of `memory.md` into `memory/archive.md` (searched by `recall`) and, from M5b, into Honcho. From M5c a scheduled job on the background slot proposes which entries stay at which tier, from counts of `recall` hits that `memoryd` keeps from M5a; its proposals are ordinary `ask` edits. The owner can mark any workspace trusted or untrusted; which ones is configuration, not design (the owner trusts the Hermes agents' workspace today). M5 splits into M5a (files and search, no network), M5b (`memoryd`, Honcho through the owner's `roncho` client) and M5c (the scheduler). Brief changes P17 to P20 proposed. | Owner, 2026-09-24: Honcho lets several Boxmakers share memory across the fleet; the Hermes agents do Boxmaker's job today and may be trusted. `brokerd` never holds conversation history and `loopd` has no network, so syncing turns needs a role of its own. Honcho's derivation replaces the brief's notes, pre-compaction flush and embedding index. |
|
| 2026-09-24 | M5 direction, before a spec. Memory is three local files and Honcho. The files: `memory/soul.md` (owner-written), `memory/memory.md` and `memory/user.md` (agent-edited, `ask`), read once per epoch, as Hermes does. Honcho, the fleet's memory service on rift (v3.2.0; models on dixie), reached by a new role, `memoryd`, which also owns the memory files and the SQLite index (owner, same day): each Boxmaker writes to its own workspace, only turns from sessions that saw no untrusted content, and can read other workspaces such as `hermes-fleet`, each with an owner-set trust level (untrusted by default). Local full-text search over the session logs stays. `consolidate` moves entries out of `memory.md` into `memory/archive.md` (searched by `recall`) and, from M5b, into Honcho. From M5c a scheduled job on the background slot proposes which entries stay at which tier, from counts of `recall` hits that `memoryd` keeps from M5a; its proposals are ordinary `ask` edits. The owner can mark any workspace trusted or untrusted; which ones is configuration, not design (the owner trusts the Hermes agents' workspace today). M5 splits into M5a (files and search, no network), M5b (`memoryd`, Honcho through the owner's `roncho` client) and M5c (the scheduler). Brief changes P17 to P20 proposed. | Owner, 2026-09-24: Honcho lets several Boxmakers share memory across the fleet; the Hermes agents do Boxmaker's job today and may be trusted. `brokerd` never holds conversation history and `loopd` has no network, so syncing turns needs a role of its own. Honcho's derivation replaces the brief's notes, pre-compaction flush and embedding index. |
|
||||||
| 2026-09-24 | M4b, changing the agreed outline (row of 2026-09-23): no reactions, an approval is answered only by `!approve <id>` or `!deny <id> [reason]` in its thread; an `[approvals] users` list in `gatewayd.toml`, which falls back to `allow.users` when empty; approvals are posted in the asking thread, channel threads included; on restart, `gatewayd` marks its old approval posts dead and `brokerd`'s courier posts pending approvals again. Spec `docs/specs/2026-09-24-m4b-approvals.md`. Brief change P16 proposed. Also: `gatewayd` strips a leading mention of its own bot from every message before handling it; in a thread shared by several Boxmakers the operator addresses approval commands, and a bare one is taken by every Boxmaker with that id pending there. | Owner: reactions are not used. Approvers are separate because an agent allowed to talk to Boxmaker should not be able to approve for it, but the default keeps one list for one person. Redacting secrets from arguments comes with the tools work. A restart must not leave posts that look answerable and are not. The mention is how the operator chooses the bot; the frontend removes it so neither the commands nor the model see it. |
|
| 2026-09-24 | M4b, changing the agreed outline (row of 2026-09-23): no reactions, an approval is answered only by `!approve <id>` or `!deny <id> [reason]` in its thread; an `[approvals] users` list in `gatewayd.toml`, which falls back to `allow.users` when empty; approvals are posted in the asking thread, channel threads included; on restart, `gatewayd` marks its old approval posts dead and `brokerd`'s courier posts pending approvals again. Spec `docs/specs/2026-09-24-m4b-approvals.md`. Brief change P16 proposed. Also: `gatewayd` strips a leading mention of its own bot from every message before handling it; in a thread shared by several Boxmakers the operator addresses approval commands, and a bare one is taken by every Boxmaker with that id pending there. | Owner: reactions are not used. Approvers are separate because an agent allowed to talk to Boxmaker should not be able to approve for it, but the default keeps one list for one person. Redacting secrets from arguments comes with the tools work. A restart must not leave posts that look answerable and are not. The mention is how the operator chooses the bot; the frontend removes it so neither the commands nor the model see it. |
|
||||||
| 2026-09-23 | M4a spec approved with two changes: a secret read from a file prints a warning at startup (and still starts); and in allowed channels and group messages shared with other agents, Boxmaker answers a post only if it names this bot, or it is a reply in one of this Boxmaker's threads that names no other user (`@channel`, `@here`, `@all` name nobody). One Boxmaker per machine, each with its own bot account; nothing is shared between them. | Owner's notes. The owner runs several agents in channels, possibly one Boxmaker per machine. A post's `from_bot` property is user-settable in v11.11.0, so bots are told apart by the allowlist, not by that property. |
|
| 2026-09-23 | M4a spec approved with two changes: a secret read from a file prints a warning at startup (and still starts); and in allowed channels and group messages shared with other agents, Boxmaker answers a post only if it names this bot, or it is a reply in one of this Boxmaker's threads that names no other user (`@channel`, `@here`, `@all` name nobody). One Boxmaker per machine, each with its own bot account; nothing is shared between them. | Owner's notes. The owner runs several agents in channels, possibly one Boxmaker per machine. A post's `from_bot` property is user-settable in v11.11.0, so bots are told apart by the allowlist, not by that property. |
|
||||||
|
|||||||
+2
-2
@@ -13,9 +13,9 @@ One session per milestone. The design brief is `docs/design.md`.
|
|||||||
| M3b | `brokerd` runner | Rootless container runner, four tools (read file, write file, sandboxed shell, HTTP fetch with host allowlist), egress proxy | A container with no network grant cannot reach the network; fetch refused for hosts outside the grant |
|
| M3b | `brokerd` runner | Rootless container runner, four tools (read file, write file, sandboxed shell, HTTP fetch with host allowlist), egress proxy | A container with no network grant cannot reach the network; fetch refused for hosts outside the grant |
|
||||||
| M4a | `gatewayd` + Mattermost conversations | Bot connection over TLS, user-ID allowlist, threads as sessions, bursts as one turn, typing while working, catching up after downtime, the secret store, no listening port | A direct message answered on the real server; a non-allowlisted user gets no response |
|
| M4a | `gatewayd` + Mattermost conversations | Bot connection over TLS, user-ID allowlist, threads as sessions, bursts as one turn, typing while working, catching up after downtime, the secret store, no listening port | A direct message answered on the real server; a non-allowlisted user gets no response |
|
||||||
| M4b | Approvals over Mattermost | `gateway.sock`, approval posts, answers by `!approve`/`!deny` in the thread, approvers list, long approvals in parts | An `ask` call approved and refused from Mattermost; the audit log names the post |
|
| M4b | Approvals over Mattermost | `gateway.sock`, approval posts, answers by `!approve`/`!deny` in the thread, approvers list, long approvals in parts | An `ask` call approved and refused from Mattermost; the audit log names the post |
|
||||||
| M5a | Local memory | `memoryd` without a network: the three memory files served to `loopd` once per epoch, the `memory` tool (add, replace, remove, `consolidate`) at `ask`, `memory/archive.md`, FTS5 `recall` over session logs and the archive, `recall` hit counts, `bxctl reindex` | A fact from epoch 1 is recalled in epoch 2; an entry moved by `consolidate` is found by `recall`; a memory edit changes the prompt only at the next epoch; the baseline with the owner's files stays under 3,000 tokens |
|
| M5a | Local memory | `memoryd` without a network: the three memory files served to `loopd` once per epoch, the `memory` tool (add, replace, remove, `consolidate`) at `ask`, `memory/archive.md`, FTS5 `recall` over session logs and the archive, `recall` hit counts, `bxctl reindex` | A fact from one session is recalled in another; an entry moved by `consolidate` is found by `recall`; a memory edit changes the prompt only in the next session (next epoch once M5c adds epochs); the baseline with the owner's files stays under 3,000 tokens |
|
||||||
| M5b | Honcho | `memoryd` syncs finished turns to its own Honcho workspace through `roncho`, `recall` and `ask_memory` over Honcho, other workspaces read with the owner's trust labels, moved entries written as conclusions | A turn from a tainted session never reaches Honcho; a result from an untrusted workspace taints the session; Honcho's contents are rebuilt from the logs |
|
| M5b | Honcho | `memoryd` syncs finished turns to its own Honcho workspace through `roncho`, `recall` and `ask_memory` over Honcho, other workspaces read with the owner's trust labels, moved entries written as conclusions | A turn from a tainted session never reaches Honcho; a result from an untrusted workspace taints the session; Honcho's contents are rebuilt from the logs |
|
||||||
| M5c | Scheduler | Heartbeat and cron on their own slot in isolated sessions; the tiering job, which proposes what stays at which tier | A scheduled run leaves the main slot's cache intact; the tiering job's proposals reach the owner as ordinary approvals and change nothing on their own |
|
| M5c | Scheduler and compaction | Heartbeat and cron on their own slot in isolated sessions; compaction at idle, starting a new epoch from the baseline and a summary; the tiering job, which proposes what stays at which tier | A scheduled run leaves the main slot's cache intact; a fact from epoch 1 is recalled in epoch 2, and a memory edit reaches the next epoch; the tiering job's proposals reach the owner as ordinary approvals and change nothing on their own |
|
||||||
| M6 | Subagents + cloud consult | Subagents on their slot returning summaries; consult tool gated by data-class grants with full payload log | A tainted session is refused; payload log matches what was sent |
|
| M6 | Subagents + cloud consult | Subagents on their slot returning summaries; consult tool gated by data-class grants with full payload log | A tainted session is refused; payload log matches what was sent |
|
||||||
| M7 | Split deployment | Each role in its own rootless container; `loopd` with no network; socket volume permissions; egress proxy for tools; host ACL policy in `deploy/` | From inside the `loopd` container, requests to the internet, the tailnet and the host all fail; a tool container without a host grant cannot reach the tailnet |
|
| M7 | Split deployment | Each role in its own rootless container; `loopd` with no network; socket volume permissions; egress proxy for tools; host ACL policy in `deploy/` | From inside the `loopd` container, requests to the internet, the tailnet and the host all fail; a tool container without a host grant cannot reach the tailnet |
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,350 @@
|
|||||||
|
# M5a design: local memory
|
||||||
|
|
||||||
|
Status: draft for the owner's review, 2026-09-24. It rests on the brief as changed by P17 to P20
|
||||||
|
(applied 2026-09-24, `docs/decisions.md`): memory is three files read into the baseline, edited
|
||||||
|
through a `memory` tool at `ask`; a new role, `memoryd`, owns the files and a full-text index; and
|
||||||
|
M5 is split into M5a (this), M5b (Honcho) and M5c (the scheduler). Owner's decisions of
|
||||||
|
2026-09-24, beyond those: `soul.md` starts from a generic file shipped in the repository, and
|
||||||
|
`memory.md` and `user.md` start empty; and compaction (new epochs) comes with the scheduler in
|
||||||
|
M5c. Until then every session has one epoch, so this spec says "the next session" where the
|
||||||
|
brief says "the next epoch". Over Mattermost every thread is a session, so a new thread is a new
|
||||||
|
session.
|
||||||
|
|
||||||
|
Facts this rests on, checked on 2026-09-24:
|
||||||
|
|
||||||
|
- `rusqlite` 0.40.2 (MIT, 2026-08-08; crates.io) with the `bundled` feature compiles SQLite from
|
||||||
|
source, and its build script always defines `SQLITE_ENABLE_FTS5` in that mode (the
|
||||||
|
`libsqlite3-sys` `build.rs`). No system SQLite is needed.
|
||||||
|
- `loopd` has no epochs yet: `Session::create` writes `0.baseline.json` and `0.jsonl`, and a full
|
||||||
|
context ends in `TurnError::SessionFull`. The baseline is saved once per session and replayed
|
||||||
|
from that file, so whatever goes into it is frozen for the session's life.
|
||||||
|
- Every tool result in a session log records its `class` and `untrusted` label
|
||||||
|
(`LogRecord::ToolResult`), so a session's taint at any record can be worked out from the log.
|
||||||
|
|
||||||
|
## 1. What M5a proves
|
||||||
|
|
||||||
|
1. A fact from one session is recalled in another.
|
||||||
|
2. An entry moved out of `memory.md` by `consolidate` is found by `recall`.
|
||||||
|
3. A memory edit reaches the prompt only in the next session, and only after the owner approved
|
||||||
|
it; a session already running keeps the bytes it started with.
|
||||||
|
4. The baseline with the starter `soul.md` stays at 3,000 tokens or less, measured on the device.
|
||||||
|
5. A hit from a session that had seen untrusted content comes back marked untrusted.
|
||||||
|
|
||||||
|
## 2. Components
|
||||||
|
|
||||||
|
| Piece | Change |
|
||||||
|
|---|---|
|
||||||
|
| `memoryd` (new crate) | Owns `memory/` and `index/`. Serves the memory block to `loopd`, runs `recall` and `memory` calls for `brokerd`, indexes session logs, rebuilds the index for `bxctl`. No network. |
|
||||||
|
| `proto` | Five new message kinds (section 7); the `memory` tool's argument structs. |
|
||||||
|
| `loopd` | The baseline takes the memory block from `memoryd` instead of reading `memory/core.md`. `recall` and `memory` are core tools. After each finished turn it tells `memoryd` the session has new records. |
|
||||||
|
| `brokerd` | `recall` and `memory` are tools it decides like any other; a decided call goes to `memoryd` instead of a container. A result's label can only be made stricter by the runtime. |
|
||||||
|
| `bxctl` | `bxctl reindex`. |
|
||||||
|
| `config/soul.md` (new) | The generic starter persona. |
|
||||||
|
|
||||||
|
`memoryd` depends only on `proto` (and `rusqlite`), like every role.
|
||||||
|
|
||||||
|
## 3. The files
|
||||||
|
|
||||||
|
All under `{home}/memory/`, directory 0700, files 0600, owned by `memoryd`. The owner may edit
|
||||||
|
any of them with an editor; `soul.md` only ever that way.
|
||||||
|
|
||||||
|
| File | Written by | Limit | Starts as |
|
||||||
|
|---|---|---|---|
|
||||||
|
| `soul.md` | the owner | 4,000 characters | a copy of the starter (`[memory] soul_starter`, default `soul.md` next to the config file) |
|
||||||
|
| `memory.md` | `memoryd`, applying approved edits | 2,200 characters | empty |
|
||||||
|
| `user.md` | `memoryd`, applying approved edits | 1,375 characters | empty |
|
||||||
|
| `archive.md` | `memoryd`, appending moved entries | none | empty |
|
||||||
|
| `edits/YYYY-MM.jsonl` | `memoryd`, appending one line per applied edit | none | absent |
|
||||||
|
| `recalls/YYYY-MM.jsonl` | `memoryd`, appending one line per `recall` | none | absent |
|
||||||
|
|
||||||
|
At startup `memoryd` creates what is missing and never overwrites anything. Characters are
|
||||||
|
counted with `chars()`. Limits are on the file's text.
|
||||||
|
|
||||||
|
**Entries.** `memory.md` and `user.md` are lists of entries separated by a line holding only `§`
|
||||||
|
(as Hermes does). An entry is trimmed, is not empty, and may span lines, but may not contain a
|
||||||
|
line holding only `§`. Two identical entries in one file are refused.
|
||||||
|
|
||||||
|
**`archive.md`.** Append-only. Each moved entry is written as:
|
||||||
|
|
||||||
|
```text
|
||||||
|
§ moved 2026-09-24T20:14Z from memory.md in session mm-y73sxj3popgwumyccptb56p1ah
|
||||||
|
<the entry>
|
||||||
|
```
|
||||||
|
|
||||||
|
**The edit journal** (`edits/`) records every applied edit: time, session, call, approval id,
|
||||||
|
action, file, the entries before and after. It is the history `archive.md` alone cannot give (when
|
||||||
|
an entry first appeared), and the record an owner reads when a memory looks wrong. **The recall
|
||||||
|
log** (`recalls/`) records time, session, query, and the source of every hit, and the queries that
|
||||||
|
found nothing: hit counts for M5c's tiering job are worked out from it, so they survive a reindex.
|
||||||
|
|
||||||
|
## 4. The memory block
|
||||||
|
|
||||||
|
At the start of each session `loopd` asks `memoryd` for the block and appends it to the system
|
||||||
|
prompt, after `system.md` and a blank line. `memoryd` renders it:
|
||||||
|
|
||||||
|
```text
|
||||||
|
# soul.md
|
||||||
|
<soul.md, trimmed>
|
||||||
|
|
||||||
|
# memory.md (1,474 of 2,200 characters)
|
||||||
|
<entries, separated by a line holding only §>
|
||||||
|
|
||||||
|
# user.md (0 of 1,375 characters)
|
||||||
|
(empty)
|
||||||
|
```
|
||||||
|
|
||||||
|
The block goes into the session's saved baseline, so it is frozen for the session: a restarted
|
||||||
|
`loopd` replays the saved bytes whatever the files say by then (inference contract 2, "strict
|
||||||
|
extension"). `memoryd` refuses to serve the block when a file is over its limit (the owner edited
|
||||||
|
it) or unreadable; `loopd` then refuses to start the session, with the error and a pointer to the
|
||||||
|
runbook entry `memory-unavailable`. Existing sessions are unaffected. A starting session with no
|
||||||
|
memory, and no sign of it, is what this prevents; the M2b rule for `core.md` stays, moved.
|
||||||
|
|
||||||
|
`loopd` keeps no memory code: `memory/core.md` and the `core-memory-unreadable` entry go.
|
||||||
|
|
||||||
|
## 5. The tools
|
||||||
|
|
||||||
|
Both are core tools: declared in the `tools` array of every session, so the model never needs
|
||||||
|
`find_tool` for them. Their schemas count toward the baseline.
|
||||||
|
|
||||||
|
### `recall`
|
||||||
|
|
||||||
|
Arguments: `{"query": string, "k": integer}`; `query` 1 to 500 characters, `k` 1 to 10, default 5.
|
||||||
|
|
||||||
|
`memoryd` searches the index (section 6) and answers with at most `k` hits, best first, each with
|
||||||
|
its source and label, and each snippet at most 600 characters:
|
||||||
|
|
||||||
|
```text
|
||||||
|
[1] session mm-8ass9m…, 2026-09-20 14:02 UTC, the owner said · private
|
||||||
|
…the launch window moved to Thursday…
|
||||||
|
[2] archive.md, moved 2026-09-24 from memory.md · approved memory · private
|
||||||
|
Owner prefers answers in UTC.
|
||||||
|
[3] session chat-1789…, 2026-09-21 09:15 UTC, the assistant said · private · untrusted
|
||||||
|
…
|
||||||
|
```
|
||||||
|
|
||||||
|
`(no matches for "…")` when nothing is found. The query goes into FTS5 only as quoted terms: it
|
||||||
|
is split on anything that is not a letter or digit, each term is wrapped in double quotes, and the
|
||||||
|
terms are joined with `OR`, so no FTS5 operator from the model ever reaches SQLite. Ranking is
|
||||||
|
FTS5's `bm25`.
|
||||||
|
|
||||||
|
**The label.** The result's class is the highest class among the hits (at least `private`), and
|
||||||
|
it is untrusted if any hit is. `brokerd` combines that with the grant's label and keeps the
|
||||||
|
stricter of each (section 8).
|
||||||
|
|
||||||
|
### `memory`
|
||||||
|
|
||||||
|
Arguments, one of (`file` is `"memory"` or `"user"`):
|
||||||
|
|
||||||
|
```json
|
||||||
|
{"action": "add", "file": "memory", "text": "…"}
|
||||||
|
{"action": "replace", "file": "memory", "old": "<an entry, exactly>", "text": "…"}
|
||||||
|
{"action": "remove", "file": "user", "old": "<an entry, exactly>"}
|
||||||
|
{"action": "consolidate", "file": "memory", "entries": [
|
||||||
|
{"old": "<an entry>", "keep": true},
|
||||||
|
{"old": "<an entry>", "text": "<its rewrite>"},
|
||||||
|
{"old": "<an entry>", "move": true},
|
||||||
|
{"old": "<an entry>", "drop": true},
|
||||||
|
{"text": "<a new entry>"}
|
||||||
|
]}
|
||||||
|
```
|
||||||
|
|
||||||
|
Entries are named by their **full text**, never by number or version. That makes every
|
||||||
|
approval readable on its own: the owner sees exactly which text is kept, rewritten, moved or
|
||||||
|
dropped, and what replaces it. It also makes a stale edit fail safely: when `memoryd` applies an
|
||||||
|
approved edit, every `old` must match an entry of the file as it is **now**, exactly; if the file
|
||||||
|
changed in between and one does not, the edit is refused and nothing changes.
|
||||||
|
|
||||||
|
Rules, checked by `brokerd` when parsing (shape) and by `memoryd` when applying (content):
|
||||||
|
|
||||||
|
- `add`: the new file must fit its limit; a duplicate is refused.
|
||||||
|
- `replace`, `remove`: `old` must match exactly one entry.
|
||||||
|
- `consolidate`: every current entry of the file must appear exactly once as an `old`, each with
|
||||||
|
exactly one of `keep`, `text`, `move`, `drop`; entries with only `text` are new. The resulting
|
||||||
|
file must fit its limit. Moved entries are appended to `archive.md` in the order given.
|
||||||
|
- An edit that would go over the limit is refused with: `memory.md would be 2,350 of 2,200
|
||||||
|
characters; use consolidate to move entries to the archive`. That sentence is fixed apart from
|
||||||
|
the numbers and the file.
|
||||||
|
|
||||||
|
Every `memory` call is `ask` (section 8), so every edit passes the owner's approval, over
|
||||||
|
Mattermost or `bxctl`. The approval shows the arguments, which name every entry by its text.
|
||||||
|
An applied edit answers `memory.md updated (1,510 of 2,200 characters); it takes effect in the
|
||||||
|
next session`. The file is written to a temporary file and renamed, then the journal line is
|
||||||
|
appended.
|
||||||
|
|
||||||
|
## 6. The index
|
||||||
|
|
||||||
|
`{home}/index/memory.sqlite3`, directory 0700, file 0600, owned by `memoryd`. Disposable: deleting
|
||||||
|
it and restarting `memoryd`, or `bxctl reindex`, rebuilds it from `sessions/` and `archive.md`.
|
||||||
|
|
||||||
|
**What is indexed.** From session logs: the `content` of `User` records and of `Assistant`
|
||||||
|
records that have content. Not tool results (the bulk of every log, and the way injected text
|
||||||
|
arrives), not `reasoning_content`, not tool call arguments. From `archive.md`: every entry.
|
||||||
|
`memory.md` and `user.md` are not indexed: they are already in every prompt.
|
||||||
|
|
||||||
|
**Labels.** Walking a session log in order, `memoryd` keeps the session's label so far: the
|
||||||
|
highest `class` of any `ToolResult` before this record (at least `private`), and whether any was
|
||||||
|
untrusted. Each indexed record gets the label in force when it was written. A record whose class
|
||||||
|
is above `[index] max_class` (default `private`) is not indexed at all. Archive entries are
|
||||||
|
labelled `private`, trusted: they passed the owner's approval.
|
||||||
|
|
||||||
|
**Keeping up.** The index stores, per session log, how many records it has read. After each
|
||||||
|
finished turn `loopd` sends `memoryd` `session_updated` with the session id (section 7), and
|
||||||
|
`memoryd` reads the records past that count. At startup it does the same for every log under
|
||||||
|
`sessions/`, so a turn missed while it was down is caught up. `memoryd` only ever reads
|
||||||
|
`sessions/`. A log it cannot parse is skipped with a log line naming it; it is `loopd`'s file, and
|
||||||
|
`loopd` reports damage itself.
|
||||||
|
|
||||||
|
Schema (one FTS5 table with the text, and a plain table for its metadata):
|
||||||
|
|
||||||
|
```sql
|
||||||
|
CREATE VIRTUAL TABLE hits USING fts5(text, tokenize = 'unicode61');
|
||||||
|
CREATE TABLE sources (rowid INTEGER PRIMARY KEY, kind TEXT, session TEXT, epoch INTEGER,
|
||||||
|
record INTEGER, time TEXT, role TEXT, class TEXT, untrusted INTEGER);
|
||||||
|
CREATE TABLE progress (log TEXT PRIMARY KEY, records INTEGER);
|
||||||
|
```
|
||||||
|
|
||||||
|
## 7. Sockets and messages
|
||||||
|
|
||||||
|
| Socket | Default path | Peer | Messages |
|
||||||
|
|---|---|---|---|
|
||||||
|
| `loopd` | `{home}/run/loop-memory/memory.sock` | `loopd` | `memory_block` → `memory_block_reply`; `session_updated` → `ok` |
|
||||||
|
| `brokerd` | `{home}/run/broker-memory/memory.sock` | `brokerd` | `tool_request` → `tool_response` (the existing kinds) |
|
||||||
|
| owner | `{home}/run/owner-memory/admin.sock` | `bxctl` | `reindex` → `reindexed` |
|
||||||
|
|
||||||
|
Directories 0700, sockets 0600, stale sockets removed, as `brokerd`'s. One request per
|
||||||
|
connection, the existing frames (`proto::frame`, `MAX_FRAME`). New message kinds in `proto`:
|
||||||
|
|
||||||
|
| Kind | Body |
|
||||||
|
|---|---|
|
||||||
|
| `memory_block` | `{}` |
|
||||||
|
| `memory_block_reply` | `{"text": string}` |
|
||||||
|
| `session_updated` | `{"session": SessionId}` |
|
||||||
|
| `reindex` | `{}` |
|
||||||
|
| `reindexed` | `{"logs": u64, "records": u64, "archive": u64}` |
|
||||||
|
|
||||||
|
`brokerd`'s list of kinds grows from eighteen to twenty-three.
|
||||||
|
|
||||||
|
## 8. `brokerd`
|
||||||
|
|
||||||
|
- **Tools.** `ToolName` gains `Recall` and `Memory`; `ToolArgs` gains their parsed arguments
|
||||||
|
(`proto` holds the serde structs, as for the other tools). Parsing enforces the shapes and
|
||||||
|
limits in section 5. Neither tool has constraints (`paths`, `hosts`, `patterns` must be empty).
|
||||||
|
- **Grants.** Ordinary grant files. The shipped examples:
|
||||||
|
|
||||||
|
```toml
|
||||||
|
# grants/recall.toml
|
||||||
|
tool = "recall"
|
||||||
|
mode = "auto"
|
||||||
|
max_taint = "secret"
|
||||||
|
result_class = "private"
|
||||||
|
untrusted = false
|
||||||
|
```
|
||||||
|
|
||||||
|
```toml
|
||||||
|
# grants/memory.toml
|
||||||
|
tool = "memory"
|
||||||
|
mode = "ask"
|
||||||
|
max_taint = "secret"
|
||||||
|
result_class = "private"
|
||||||
|
untrusted = false
|
||||||
|
```
|
||||||
|
|
||||||
|
A grant of `memory` at `auto` is refused when grants are loaded: every memory edit is the
|
||||||
|
owner's (brief, memory 7). The fixed sentence names the grant.
|
||||||
|
- **Where a decided call runs.** A composite runtime sends `Recall` and `Memory` specs to
|
||||||
|
`memoryd`'s `brokerd` socket as a `tool_request`, and everything else to the configured runtime
|
||||||
|
(Podman, or the refusing one). `[memory] socket` in `brokerd.toml` names the socket; without it,
|
||||||
|
memory calls fail with a fixed sentence and a runbook pointer (`memoryd-unreachable`).
|
||||||
|
- **Labels only get stricter.** `RunOutput` gains an optional label. `runner::run` answers with
|
||||||
|
the stricter of the grant's label and the runtime's: the higher class, and untrusted if either
|
||||||
|
is. Container runtimes return none, so nothing changes for them. The audit record and the
|
||||||
|
session's taint use the combined label, as today.
|
||||||
|
- **Size.** `policy.rs` (498 lines) and `ledger.rs` (487) are at the limit; the plan splits them
|
||||||
|
before anything is added.
|
||||||
|
|
||||||
|
## 9. `loopd`
|
||||||
|
|
||||||
|
- `[memory] socket` in `loopd.toml` (default `{home}/run/loop-memory/memory.sock`).
|
||||||
|
- Baseline: `system.md`, a blank line, then the block from `memoryd`. `memoryd` unreachable, or
|
||||||
|
refusing, fails the new session with the runbook pointer `memory-unavailable`.
|
||||||
|
- Registry: `recall` and `memory` are core, with the schemas of section 5. The model sees
|
||||||
|
`clock`, `recall`, `memory`, `find_tool`, `call_tool`.
|
||||||
|
- After a finished turn: `session_updated`, best effort. A failure is one log line; the next
|
||||||
|
startup of `memoryd` catches up.
|
||||||
|
- `system.md` gains one sentence: memory edits need the owner's approval and take effect in the
|
||||||
|
next session, and recalled text is data like any tool result.
|
||||||
|
|
||||||
|
## 10. `bxctl`
|
||||||
|
|
||||||
|
`bxctl reindex [--admin-socket PATH]`: sends `reindex`, waits, and prints `reindexed: 42 logs,
|
||||||
|
1,208 records, 7 archive entries`. `memoryd` rebuilds into a new file and renames it over the old
|
||||||
|
one, so `recall` keeps working meanwhile.
|
||||||
|
|
||||||
|
## 11. Configuration
|
||||||
|
|
||||||
|
`memoryd serve --config <path>`:
|
||||||
|
|
||||||
|
```toml
|
||||||
|
[paths]
|
||||||
|
home = "/var/lib/boxmaker" # default: $BOXMAKER_HOME, then /var/lib/boxmaker
|
||||||
|
|
||||||
|
[memory]
|
||||||
|
soul_starter = "soul.md" # next to the config file
|
||||||
|
|
||||||
|
[index]
|
||||||
|
max_class = "private" # records above this class are not indexed
|
||||||
|
|
||||||
|
# [sockets] loop, broker, admin: defaults as in section 7
|
||||||
|
```
|
||||||
|
|
||||||
|
## 12. Security notes
|
||||||
|
|
||||||
|
- `memoryd` has no network in M5a and never writes `sessions/`, grants or the audit log.
|
||||||
|
- Nothing reaches the prompt without the owner: `soul.md` is the owner's file; every edit to the
|
||||||
|
other two is approved; the block is frozen per session.
|
||||||
|
- Recalled text is a tool result: appended, labelled, and it taints the session like any other.
|
||||||
|
Tool results are not indexed, so a fetched page cannot come back through `recall`; what the
|
||||||
|
assistant said after reading one can, marked untrusted.
|
||||||
|
- The FTS5 query is built only from quoted terms; the model's text is never SQL.
|
||||||
|
- Entries shown in approvals are escaped like all arguments (`proto::escape`), so an entry cannot
|
||||||
|
render as markup or mention anyone.
|
||||||
|
|
||||||
|
## 13. Runbook
|
||||||
|
|
||||||
|
New entries: `memoryd-start-failed` (configuration, `memory/` or `index/` unusable),
|
||||||
|
`memory-unavailable` (a new session refused: `memoryd` down, a file unreadable or over its limit),
|
||||||
|
`memoryd-unreachable` (`brokerd` cannot reach it: memory calls fail), `memory-index-damaged`
|
||||||
|
(rebuild with `bxctl reindex`). `core-memory-unreadable` is removed.
|
||||||
|
|
||||||
|
## 14. Testing
|
||||||
|
|
||||||
|
Offline, in `make gate`:
|
||||||
|
|
||||||
|
- `memoryd`: entries parsing and rendering; each edit's rules, including a stale `old`, a
|
||||||
|
duplicate, the limit sentence, and `consolidate` missing an entry or naming one twice; the
|
||||||
|
archive and journal lines; the block; the quoted-terms query builder against FTS5 operators
|
||||||
|
(`NEAR`, `*`, `^`, quotes, column filters); labels walked from a log; `max_class`; catching up
|
||||||
|
from `progress`; a reindex equal to the incremental index; the recall log.
|
||||||
|
- `brokerd`: parsing both tools; `memory` at `auto` refused at load; the composite runtime; the
|
||||||
|
stricter label, with a runtime that returns a stricter and a laxer one.
|
||||||
|
- `loopd`: the baseline with a fake `memoryd`; the refusal and its pointer; `session_updated`
|
||||||
|
sent after a finished turn and not after a turn ending in tool calls.
|
||||||
|
- `bxctl`: `reindex` against a fake socket.
|
||||||
|
|
||||||
|
On the device (the owner's Mattermost, as for M4b): the baseline token count with the starter
|
||||||
|
soul; a fact told in one thread and recalled in another; an `add` approved from Mattermost that
|
||||||
|
appears in a new thread and not in the thread that asked; a `consolidate` that moves an entry,
|
||||||
|
which `recall` then finds; a hit from a session that read a fetched page coming back untrusted.
|
||||||
|
|
||||||
|
## 15. For the owner's review
|
||||||
|
|
||||||
|
1. Tool results are not indexed (section 6). It keeps injected text out of `recall`, and keeps
|
||||||
|
the index small; the price is that `recall` cannot find something only a tool returned, such
|
||||||
|
as a file's contents, unless the assistant repeated it.
|
||||||
|
2. `memory` at `auto` is refused at load (section 8), not merely discouraged.
|
||||||
|
3. Entries are named by their full text, not by number (section 5), so approvals are readable and
|
||||||
|
stale edits fail; the arguments of a `consolidate` of a full `memory.md` are about 4,500
|
||||||
|
characters, one approval post.
|
||||||
|
4. The starter `soul.md` is generic (section 3); the owner writes it before or after the first
|
||||||
|
start.
|
||||||
Reference in New Issue
Block a user