From 774807aa81a03b7570896b257d819587c87c8531 Mon Sep 17 00:00:00 2001 From: "K. Isom" Date: Thu, 24 Sep 2026 13:12:03 -0700 Subject: [PATCH] Apply P19: recall over the local index and Honcho; trust per workspace Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/decisions.md | 3 +-- docs/design.md | 46 ++++++++++++++++++++++++---------------------- 2 files changed, 25 insertions(+), 24 deletions(-) diff --git a/docs/decisions.md b/docs/decisions.md index a8707e5..20138d5 100644 --- a/docs/decisions.md +++ b/docs/decisions.md @@ -65,7 +65,6 @@ Newest first. A decision that changes `docs/design.md` lands in the same commit | Secret store: settled 2026-09-23 (systemd credentials, environment, owner-only file; P15 proposed). Where `brokerd` reads tool secrets from, and whether `SecretStore` moves out of `gatewayd` into a shared place, is decided when a tool first needs a secret. | when a tool needs a secret | | Whether cloud-led sessions are ever allowed, and for which data classes. | M6 | | Projects in memory. Honcho can separate projects by a workspace per project (hard walls, shared naturally by every agent on the project, but Honcho's picture of the owner splits per workspace) or by scopes inside one workspace (one picture, recall filtered by project; semantics not yet read). Either way a project is a `memoryd` setting: its channels and threads, its workspace or scope, `write` and `trust`. A shared project workspace is written by several agents, so its trust is the owner's call per project. Test both on rift before choosing. Owner's idea, 2026-09-24. | M5b or a later M5d | -| Where the M5b embedding model runs. Superseded by P19 if it is approved: Honcho's embeddings run on dixie (`honcho-embed`). | M5b | ## Applied to the design brief @@ -97,6 +96,7 @@ Later changes to the brief: | 2026-09-24 | P16, Network isolation 3: approvals are commands (`!approve`, `!deny`) replied in the approval's thread; emoji reactions are not used. | Owner's decision and approval, 2026-09-24; M4b spec `docs/specs/2026-09-24-m4b-approvals.md`. | | 2026-09-24 | P17, Memory 1, the baseline and State: the core block becomes three files (`soul.md`, owner-written; `memory.md` and `user.md`, agent-edited at `ask` with add, replace, remove and `consolidate`), read once per epoch; moved entries go to `memory/archive.md`. | Owner's approval, 2026-09-24; see the row of that date under Decided. | | 2026-09-24 | P18, Roles and Network isolation: a sixth role, `memoryd`, owns `memory/` and `index/`, syncs turns to Honcho and runs memory tool calls; `loopd` has four sockets and no longer holds the memory files. | Owner's approval, 2026-09-24; see the row of that date under Decided. | +| 2026-09-24 | P19, Memory 2 to 8: `recall` over a local FTS5 index (session logs, archive) and Honcho, `ask_memory`; Honcho writes only from clean sessions; per-workspace trust set by the owner; notes, the flush job and the embedding index go; memory text stays on the owner's hosts. | Owner's approval, 2026-09-24; see the row of that date under Decided. | ## Proposed changes to the design brief @@ -104,5 +104,4 @@ Proposed 2026-09-24 for M5 (row of that date under Decided). Not yet applied. | # | Change | Reason | |---|---|---| -| P19 | Memory 2 to 9: notes files, the pre-compaction flush, the embedding index (M5b) and "no vector database service" are replaced. `recall` searches the local full-text index (SQLite FTS5 over the session logs and `memory/archive.md`, rebuildable with `bxctl reindex`) and Honcho; `ask_memory` asks Honcho's dialectic. Both go through `brokerd` at `auto` and run in `memoryd`. Entries found in `archive.md` come back as approved memory. `memoryd` counts `recall` hits per archive entry and when each was last found, for the tiering job (P20, M5c). From M5b, entries moved by `consolidate` are also written to Boxmaker's Honcho workspace as conclusions. `memoryd` writes to Honcho only turns from sessions that saw no untrusted content and are within a configured data class, and it enforces that itself. Each workspace read has an owner-set trust level; results from an untrusted workspace (the default) are marked untrusted and taint the session. Trust is per workspace because its writers differ; the brief names no workspace. Every result carries its source (the file and line, or the workspace) and data class. "No memory text leaves the host" becomes "no memory text leaves the owner's hosts": Honcho runs on rift with its models on dixie. | Honcho's deriver, summaries and dreamer do the extraction the flush job was for, and `honcho-embed` on dixie is the embedding index. Honcho's conclusions carry no per-entry provenance, so the untrusted filter must be applied when writing, and each workspace is labelled as a whole. | | P20 | Milestones: M5 becomes M5a (`memoryd` without a network: the three files, the `memory` tool with `consolidate`, the archive, full-text `recall`), M5b (`memoryd` and Honcho) and M5c (heartbeat and cron on their own slot). The old M5b row goes. Acceptance moves with the parts: a fact from epoch 1 is recalled in epoch 2, and an entry moved by `consolidate` is found by `recall` (M5a); a turn from a tainted session never reaches Honcho, and a result from an untrusted workspace taints the session (M5b); a scheduled run leaves the main slot's cache intact, and the tiering job's proposals reach the owner as ordinary approvals and change nothing on their own (M5c). M5c's tiering job: a scheduled session that reads the memory files, the archive with its hit counts, and Boxmaker's Honcho summaries, and proposes a `consolidate` (moving entries down) or an `add` naming its source (bringing one up), when a file passes about 80% full or weekly, at most one per file per run. | As for M2 to M4: the parts are reviewed separately, and M5a needs no network. | diff --git a/docs/design.md b/docs/design.md index 80397d6..cda3730 100644 --- a/docs/design.md +++ b/docs/design.md @@ -188,27 +188,29 @@ the inference contract, and it must not become a way for injected text to persis as a diff. An approved edit takes effect at the next epoch, never mid-session. A write over a file's limit is refused with a pointer to `consolidate`. 2. **Everything else is retrieved, not injected.** A `recall(query, k)` tool returns size-capped - snippets with file, line and provenance. Results are appended like any tool result, so the - cache prefix is untouched. -3. **Notes.** `memory/notes/YYYY-MM-DD.md`, append-only, written by a `remember` tool. Each entry - records its session, time, data-class label and whether the session had seen untrusted content. -4. **Flush before compaction.** When a session goes idle and is about to change epoch, a job on - the scheduled slot extracts durable facts from the log into notes. The main slot is not used. -5. **Provenance on recall.** Recalled text is presented as data with its provenance, never placed - in the system prompt. Entries written from sessions that saw untrusted content are marked as - such when recalled. Recalled entries carry their data-class label and taint the session. -6. **Both tools go through `brokerd`** under ordinary grants: `recall` and note-writing `auto`, - core and topic edits `ask`. -7. **Index.** Lexical search first: SQLite FTS5 over the Markdown files, rebuildable with - `bxctl reindex`. Retrieval sits behind a `Retriever` trait. -8. **Embeddings are a later, additive index** (M5b), never a store. Local model only, served by a - separate `llama-server` embedding instance so chat slots are unaffected. Vectors live in a flat - file or SQLite table keyed by chunk hash and embedding-model hash; a model change triggers a - rebuild. Combine with lexical results by rank fusion. Add M5b when logged `recall` misses show - lexical search failing, not before. -9. **No vector database service.** Search is exact brute-force cosine, in-process. Revisit only if - the index passes about one million chunks or measured `recall` latency exceeds one second, and - then prefer an embedded, file-backed approximate index over a server. + snippets with their source (a file and line, or a Honcho workspace). An `ask_memory` tool asks + Honcho's dialectic. Results are appended like any tool result, so the cache prefix is untouched. +3. **Local index.** SQLite FTS5 over the session logs and `memory/archive.md`, kept by `memoryd` + and rebuildable with `bxctl reindex`. It needs no network and no model, so it still answers + when Honcho is down. `memoryd` counts `recall` hits per archive entry and when each was last + found. Retrieval sits behind a `Retriever` trait. +4. **Honcho.** The fleet's memory service; it and the models it calls run on the owner's hosts. Each + Boxmaker writes finished turns to its own workspace, and only turns from sessions that saw no + untrusted content and are within a configured data class; `memoryd` enforces this itself. Entries + moved by `consolidate` are written there too. Honcho's own extraction, summaries and + consolidation replace a flush job of ours. What Honcho holds is derived from the session logs and + `memory/archive.md`, and can be rebuilt from them. +5. **Workspaces and trust.** `memoryd` can read other workspaces. The owner marks each one + trusted or untrusted; untrusted is the default. Results from an untrusted workspace are marked + untrusted and taint the session. Honcho's conclusions carry no per-entry provenance, so the + filter is applied when writing and the label is per workspace. +6. **Provenance on recall.** Recalled text is presented as data with its source and data class, + never placed in the system prompt, and taints the session with that class. Entries from + `memory/archive.md` come back as approved memory. +7. **The tools go through `brokerd`** under ordinary grants and run in `memoryd`: `recall` and + `ask_memory` `auto`, every `memory` edit `ask`. +8. **No memory text leaves the owner's hosts.** Honcho and the models it calls run on the owner's + hosts, and each is listed in `docs/egress.md`. ## State @@ -216,7 +218,7 @@ Files are the source of truth. SQLite is allowed only for rebuildable indexes an - `sessions//.jsonl` — append-only session log - `memory/soul.md`, `memory/memory.md`, `memory/user.md`, `memory/archive.md` — owner-readable memory -- `index/` — FTS5 and, later, vectors; disposable and rebuildable from `memory/` +- `index/` — FTS5, kept by `memoryd`; disposable and rebuildable from `memory/` and `sessions/` - `grants/*.toml` — owner-written - `audit/*.jsonl` — hash-chained - `broker/sessions/.json` — each session's taint and untrusted flag, written only by