diff --git a/docs/plans/M5a/01-brokerd-split.md b/docs/plans/M5a/01-brokerd-split.md new file mode 100644 index 0000000..dadc299 --- /dev/null +++ b/docs/plans/M5a/01-brokerd-split.md @@ -0,0 +1,106 @@ +# M5a task 01: split `brokerd`'s policy and ledger into modules + +**Branch:** `m5a` (run `git switch m5a`; `git status --short` must be empty, otherwise stop) +**Commit subject:** `brokerd: policy and ledger as modules, before the memory tools` + +## Goal + +`crates/brokerd/src/policy.rs` is 498 lines and `crates/brokerd/src/ledger.rs` 487; the limit is +500, and M5a adds to both crates' paths. Move each into a directory module, with part of it in a +child file. **Nothing else changes**: no line of code is edited, only moved, plus the visibility +and imports listed below. A child module can see its parent's private items, so `Decision` keeps +its private fields and its compile-fail doc tests still pass. + +## Files + +- Move: `crates/brokerd/src/policy.rs` to `crates/brokerd/src/policy/mod.rs`, and + `crates/brokerd/src/ledger.rs` to `crates/brokerd/src/ledger/mod.rs` (with `git mv`) +- New: `crates/brokerd/src/policy/matching.rs`, `crates/brokerd/src/ledger/answer.rs` +- Modify: `docs/implementer-log.md` + +## The policy split, exactly + +1. `mkdir crates/brokerd/src/policy && git mv crates/brokerd/src/policy.rs crates/brokerd/src/policy/mod.rs` +2. In `policy/mod.rs`, find the line `/// The grants left after matching, and the two flags that pick + the denial reason when none are` (the doc comment of `struct MatchResult`). Cut from that line + to the end of the file, and paste it into the new file `policy/matching.rs` below this header: + +```rust +//! Matching a call against the grants: which grants cover it, and which one wins. + +use super::{Label, Matched, SessionState}; +use crate::args::{ToolArgs, host_matches, inside}; +use crate::grants::GrantSet; +use proto::{DataClass, Grant, Hash32, Mode, Timestamp}; + +``` + +3. In `policy/matching.rs`, and nowhere else, add `pub(super)` to: `struct MatchResult` and its + three fields; `struct Candidate` and every one of its fields; `fn match_grants`, `fn winner` + and `fn build_matched`. Leave `covers`, `best_path`, `better` and `combined_label` private. +4. In `policy/mod.rs`, the three `use` lines + +```rust +use crate::args::{self, ToolArgs, ToolName, host_matches, inside}; +use crate::grants::GrantSet; +use proto::{DataClass, DenyReason, Grant, Hash32, Mode, Timestamp, ToolRequest}; +``` + + become + +```rust +mod matching; + +use crate::args::{self, ToolArgs, ToolName}; +use crate::grants::GrantSet; +use matching::{MatchResult, build_matched, match_grants, winner}; +use proto::{DataClass, DenyReason, Hash32, Mode, Timestamp, ToolRequest}; +``` + +## The ledger split, exactly + +1. `mkdir crates/brokerd/src/ledger && git mv crates/brokerd/src/ledger.rs crates/brokerd/src/ledger/mod.rs` +2. In `ledger/mod.rs`, inside `impl Ledger`, find ` /// Re-decide an approval and record it.` + (the doc comment of `pub fn answer`). Cut from that line down to the end of `pub fn finish`, + which is everything up to, but not including, the `}` that closes `impl Ledger` (the last line + of the file). Paste it into the new file `ledger/answer.rs` inside an `impl Ledger { … }`, below + this header: + +```rust +//! Answering an approval, and recording a call's result. + +use std::sync::MutexGuard; + +use proto::{ApprovalAnswer, AuditEvent, DataClass, DenyReason, Timestamp, ToolResponse}; + +use super::{Answer, Answered, Call, Grants, Inner, Ledger}; +use crate::approvals::Verdict; +use crate::policy::{Ask, Denial, SessionState}; + +impl Ledger { +``` + + and close the file with `}` after `finish`. +3. In `ledger/mod.rs`: add `mod answer;` and a blank line just before + `use std::sync::{Mutex, MutexGuard};`, and remove `ApprovalAnswer, ` from the `use proto::{…}` + list (it is used only in `answer.rs` now). + +## Steps + +- [ ] **1. Move** the two files as above, one split at a time, with `cargo check -p brokerd` + after each. `cargo fmt --all`. +- [ ] **2. See nothing change.** `cargo test -p brokerd`: every suite passes, the doc tests + included (`policy`'s two `compile_fail` examples still fail to compile, which is a pass). + `wc -l crates/brokerd/src/policy/*.rs crates/brokerd/src/ledger/*.rs`: every file under 500. +- [ ] **3. Run the gate.** `make gate`. Expected last line: `gate: ok`, with the same number of + tests as before this task (about 854). +- [ ] **4. Log and commit.** `git add -A crates/brokerd docs/implementer-log.md Cargo.lock && git commit` + +## Done when + +- `make gate` prints `gate: ok` with no change in the number of tests. + +## Stop and report if + +- The compiler asks for a visibility change or an import not listed here. +- Any test changes result. diff --git a/docs/plans/M5a/02-proto-memory.md b/docs/plans/M5a/02-proto-memory.md new file mode 100644 index 0000000..e8cef3c --- /dev/null +++ b/docs/plans/M5a/02-proto-memory.md @@ -0,0 +1,92 @@ +# M5a task 02: `proto`, the memory tools' arguments and messages + +**Branch:** `m5a` (run `git switch m5a`; `git status --short` must be empty, otherwise stop) +**Commit subject:** `proto: memory, the arguments of recall and memory and eight message kinds` + +## Goal + +The shared types of M5a (spec, sections 5 and 7): the arguments of `recall` and `memory` with +their shape checks, and the eight messages `memoryd` exchanges with `loopd`, `brokerd` and +`bxctl`. `brokerd`'s list of message kinds grows to twenty-six. + +## Files + +- Copy: the skeleton `crates/proto/src/memory.rs`; `crates/proto/tests/memory_args.rs`, + `crates/proto/tests/memory_wire.rs`, and the eight fixtures + `crates/proto/tests/fixtures/wire/{memory_block,memory_block_reply,session_updated,memory_call,reindex,reindexed,memory_admin,memory_admin_reply}.json` +- Modify: `crates/proto/src/lib.rs`, `crates/proto/src/wire.rs`, `crates/brokerd/src/broker.rs`, + `docs/implementer-log.md` + +## The skeleton + +Written: the constants, every type with its serde attributes, and the message structs. To fill, +each with the code in its comment: `ConsolidateEntry::action`, `entry_problem`, +`RecallArgs::check`, `MemoryArgs::check`. Count characters with `chars().count()`, never `len()`. + +## The edits, exactly + +**`proto/src/lib.rs`:** add `pub mod memory;` after `pub mod log;`, and after +`pub use log::{LogRecord, ToolCall, Usage};` add: + +```rust +pub use memory::{ + ConsolidateEntry, EntryAction, MemoryAdmin, MemoryAdminReply, MemoryArgs, MemoryBlockReply, + MemoryCall, MemoryCommand, MemoryFile, RecallArgs, Reindexed, SessionUpdated, +}; +``` + +**`proto/src/wire.rs`:** after `use crate::gateway::{ApprovalOutcome, ApprovalPosted, ApprovalReply, ApprovalRequest};` add + +```rust +use crate::memory::{ + MemoryAdmin, MemoryAdminReply, MemoryBlockReply, MemoryCall, Reindexed, SessionUpdated, +}; +``` + +and in `enum Message`, after `ApprovalOutcome(ApprovalOutcome),`, add these eight variants, in +this order: + +```rust + MemoryBlock(Empty), + MemoryBlockReply(MemoryBlockReply), + SessionUpdated(SessionUpdated), + MemoryCall(MemoryCall), + Reindex(Empty), + Reindexed(Reindexed), + MemoryAdmin(MemoryAdmin), + MemoryAdminReply(MemoryAdminReply), +``` + +**`brokerd/src/broker.rs`, `kind`:** the doc comment's "all eighteen kinds" becomes "all +twenty-six kinds", and after `Message::ApprovalOutcome(_) => "approval_outcome",` add: + +```rust + Message::MemoryBlock(_) => "memory_block", + Message::MemoryBlockReply(_) => "memory_block_reply", + Message::SessionUpdated(_) => "session_updated", + Message::MemoryCall(_) => "memory_call", + Message::Reindex(_) => "reindex", + Message::Reindexed(_) => "reindexed", + Message::MemoryAdmin(_) => "memory_admin", + Message::MemoryAdminReply(_) => "memory_admin_reply", +``` + +## Steps + +- [ ] **1. Copy** the skeleton, the two tests and the eight fixtures from `docs/plans/M5a/files/` + to the same paths. +- [ ] **2. Make the edits** above; `cargo check --workspace`. +- [ ] **3. See it fail.** `cargo test -p proto --test memory_args --test memory_wire`. Expected: + `memory_wire` 2 passed (the types are given); `memory_args` 4 failed, 2 passed. +- [ ] **4. Fill the four functions.** `cargo fmt --all`. +- [ ] **5. See it pass.** The same command: 6 and 2 passed. +- [ ] **6. Run the gate.** `make gate`: `gate: ok`, with 8 more tests than before (about 862). +- [ ] **7. Log and commit.** `git add -A crates docs/implementer-log.md Cargo.lock && git commit` + +## Done when + +- `make gate` prints `gate: ok`. + +## Stop and report if + +- A given test or fixture fails after the four functions follow their comments exactly. diff --git a/docs/plans/M5a/03-memoryd-crate.md b/docs/plans/M5a/03-memoryd-crate.md new file mode 100644 index 0000000..e6941c3 --- /dev/null +++ b/docs/plans/M5a/03-memoryd-crate.md @@ -0,0 +1,73 @@ +# M5a task 03: the `memoryd` crate, its configuration and sockets + +**Branch:** `m5a` (run `git switch m5a`; `git status --short` must be empty, otherwise stop) +**Commit subject:** `memoryd: the crate, memoryd.toml, and listening on a socket` + +## Goal + +A new role crate, `memoryd` (M5a spec, sections 2, 7 and 11), with its configuration and the +socket helper its three sockets will use. It depends on `proto` and on one new crate, `rusqlite` +(the index, from task 08), which is added to the workspace now so the dependency checks pass +once. `memoryd serve --config ` reads its configuration and says "not implemented" until +task 10. + +## Files + +- Copy (whole, nothing to fill): `crates/memoryd/Cargo.toml`, `crates/memoryd/src/lib.rs`, + `crates/memoryd/src/main.rs` +- Copy (skeletons): `crates/memoryd/src/config.rs`, `crates/memoryd/src/socket.rs` +- Copy (tests): `crates/memoryd/tests/config.rs`, `crates/memoryd/tests/socket.rs`, + `crates/memoryd/tests/support/tmp.rs` +- Append: `docs/plans/M5a/files/runbook-memoryd-start-failed.md` to the end of `docs/runbook.md` + (a blank line before it) +- Modify: `Cargo.toml`, `deny.toml`, `docs/dependencies.md`, `docs/implementer-log.md` + +## The edits, exactly + +**`Cargo.toml` (the workspace):** add `"crates/memoryd",` to `members` after +`"crates/gatewayd",`, and at the end of `[workspace.dependencies]` add + +```toml +rusqlite = { version = "0.40.2", features = ["bundled"] } +``` + +**`deny.toml`:** the line `allow = ["MIT", "Apache-2.0", "Unicode-3.0", "ISC", "BSD-3-Clause"]` +becomes `allow = ["MIT", "Apache-2.0", "Unicode-3.0", "ISC", "BSD-3-Clause", "Zlib"]` +(`foldhash`, a dependency of `rusqlite`, is Zlib-licensed). + +**`docs/dependencies.md`:** add this row at the end of the table: + +```markdown +| `rusqlite` | 0.40.2 | `memoryd` from M5a | The full-text index (`index/memory.sqlite3`, FTS5), disposable and rebuilt from files. Feature `bundled`: `libsqlite3-sys` compiles SQLite from source with FTS5 on, so no system SQLite is needed. With its dependencies: `libsqlite3-sys` (MIT; build: `cc`, `pkg-config`, `vcpkg`), `hashlink`, `hashbrown`, `foldhash` (Zlib, added to `deny.toml` for it), `fallible-iterator`, `fallible-streaming-iterator`, `smallvec`, `bitflags`. MIT. | +``` + +## The skeletons + +`config.rs`: the types, `Config::parse` and the path methods are written; fill `Config::load` and +`Config::socket`. `socket.rs`: fill `listen`, the same five steps as `listen` in +`crates/brokerd/src/serve.rs` (read it; no crate may depend on another role's crate, so this is a +copy with `String` errors). + +## Steps + +- [ ] **1. Copy** every file above from `docs/plans/M5a/files/` to the same path; make the three + edits; append the runbook entry. +- [ ] **2. Build.** `cargo build -p memoryd`. The first build downloads `rusqlite` and compiles + SQLite; it takes a few minutes. Then `cargo deny check`: expected `advisories ok, bans ok, + licenses ok, sources ok` (it needs the network). +- [ ] **3. See it fail.** `cargo test -p memoryd --test config --test socket`: expected + `config` 3 failed, 1 passed; `socket` 2 failed. +- [ ] **4. Fill** `Config::load`, `Config::socket` and `listen`. `cargo fmt --all`. +- [ ] **5. See it pass.** The same command: 4 and 2 passed. +- [ ] **6. Run the gate.** `make gate`: `gate: ok`, with 6 more tests than before (about 868). +- [ ] **7. Log and commit.** `git add -A Cargo.toml Cargo.lock deny.toml crates/memoryd docs/runbook.md docs/dependencies.md docs/implementer-log.md && git commit` + +## Done when + +- `make gate` prints `gate: ok`, and `cargo deny check` passes. + +## Stop and report if + +- `cargo deny check` reports anything but the four "ok"s: a licence other than Zlib, or two + versions of one crate. Do not edit `deny.toml` beyond the one line above. +- The build cannot download `rusqlite`. diff --git a/docs/plans/M5a/04-memoryd-entries-check.md b/docs/plans/M5a/04-memoryd-entries-check.md new file mode 100644 index 0000000..4eb39f4 --- /dev/null +++ b/docs/plans/M5a/04-memoryd-entries-check.md @@ -0,0 +1,51 @@ +# M5a task 04: entries and the check + +**Branch:** `m5a` (run `git switch m5a`; `git status --short` must be empty, otherwise stop) +**Commit subject:** `memoryd: entries, and the check on every memory write` + +## Goal + +Two small modules (M5a spec, section 3 and section 5, "The check"): + +- `entries`: `memory.md` and `user.md` are entries separated by a line holding only `§`. Parse, + render, and measure a file in characters. +- `check`: every entry the agent writes, and every identity entry going into the prompt, is + refused if it holds a control character (other than newline and tab), an invisible or + direction-changing character, chat-template markup, or one of six instruction phrases. The + table of invisible characters is written in the skeleton, from Unicode 16's + `Default_Ignorable_Code_Point` property, plus six more. + +## Files + +- Copy (skeletons): `crates/memoryd/src/entries.rs`, `crates/memoryd/src/check.rs` +- Copy (tests): `crates/memoryd/tests/entries.rs`, `crates/memoryd/tests/check.rs` +- Modify: `crates/memoryd/src/lib.rs`, `docs/implementer-log.md` + +## The edits, exactly + +**`crates/memoryd/src/lib.rs`:** add `pub mod check;` before `pub mod config;`, and +`pub mod entries;` after it. The file then lists `check`, `config`, `entries`, `socket`. + +## The skeletons + +`entries.rs`: fill `parse`, `push`, `render`, `size`. `check.rs`: `Refused` and its sentences, +`DEFAULT_IGNORABLE`, `ALSO_INVISIBLE`, `MARKUP` and `PHRASES` are written; fill `invisible` and +`check`. Do not change the tables. + +## Steps + +- [ ] **1. Copy** the four files; edit `lib.rs`. +- [ ] **2. See it fail.** `cargo test -p memoryd --test entries --test check`: `entries` 3 failed, + `check` 6 failed. +- [ ] **3. Fill** the six functions. `cargo fmt --all`. +- [ ] **4. See it pass.** The same command: 3 and 6 passed. +- [ ] **5. Run the gate.** `make gate`: `gate: ok`, 9 more tests than before (about 877). +- [ ] **6. Log and commit.** `git add -A crates/memoryd docs/implementer-log.md Cargo.lock && git commit` + +## Done when + +- `make gate` prints `gate: ok`. + +## Stop and report if + +- A test fails after the functions follow their comments exactly. diff --git a/docs/plans/M5a/05-memoryd-files.md b/docs/plans/M5a/05-memoryd-files.md new file mode 100644 index 0000000..0f9a8ab --- /dev/null +++ b/docs/plans/M5a/05-memoryd-files.md @@ -0,0 +1,42 @@ +# M5a task 05: the memory files and the block + +**Branch:** `m5a` (run `git switch m5a`; `git status --short` must be empty, otherwise stop) +**Commit subject:** `memoryd: the memory files, and the block a new session starts with` + +## Goal + +`Store`, the directory `memory/` (M5a spec, sections 3 and 4): created at 0700 with each missing +file at 0600 (`soul.md` from the starter, the others empty), never overwriting anything; reading +and rewriting an identity file's entries (through a temporary file and a rename); appending; and +the **block**, the text a new session's prompt gets: `soul.md`, then `memory.md` and `user.md` +with how full each is. A file over its limit stops the block; an identity entry that fails the +check is left out of it and reported. + +## Files + +- Copy (skeleton): `crates/memoryd/src/files.rs` +- Copy (test): `crates/memoryd/tests/files.rs` +- Modify: `crates/memoryd/src/lib.rs` (`pub mod files;` after `pub mod entries;`), + `docs/implementer-log.md` + +## The skeleton + +Written: `SOUL_LIMIT`, `Store`, `Block`, `fail`, `Store::new`, `Store::path`. Fill: `grouped`, +`init`, `create`, `entries`, `write_entries`, `append`, `block`. + +## Steps + +- [ ] **1. Copy** the two files; edit `lib.rs`. +- [ ] **2. See it fail.** `cargo test -p memoryd --test files`: 8 failed. +- [ ] **3. Fill** the seven functions, `block` last. `cargo fmt --all`. +- [ ] **4. See it pass.** The same command: 8 passed. +- [ ] **5. Run the gate.** `make gate`: `gate: ok`, 8 more tests than before (about 885). +- [ ] **6. Log and commit.** `git add -A crates/memoryd docs/implementer-log.md Cargo.lock && git commit` + +## Done when + +- `make gate` prints `gate: ok`. + +## Stop and report if + +- A test fails after the functions follow their comments exactly. diff --git a/docs/plans/M5a/06-memoryd-edits.md b/docs/plans/M5a/06-memoryd-edits.md new file mode 100644 index 0000000..a978974 --- /dev/null +++ b/docs/plans/M5a/06-memoryd-edits.md @@ -0,0 +1,53 @@ +# M5a task 06: notes and the memory edits + +**Branch:** `m5a` (run `git switch m5a`; `git status --short` must be empty, otherwise stop) +**Commit subject:** `memoryd: notes.md, and the five memory actions with routing by trust` + +## Goal + +What a `memory` call does (M5a spec, section 5): + +- `note` appends to `notes.md`, in any session, labelled by the session: its class, and + `untrusted` when the session has read untrusted content. +- `add`, `replace`, `remove` and `consolidate` edit `memory.md` or `user.md` (**identity**), and + only a session that has not read untrusted content may use them. In one that has, `add` becomes + a note (`SAVED_AS_NOTE`) and the other three are refused. +- Entries are named by their full text. An edit made against an old view of the file fails + safely; an entry named twice, a duplicate, or a file over its limit is refused, and nothing + changes. +- `remove` and `consolidate` never lose anything: what they take out goes to `notes.md` as + `dropped` or `moved`. + +## Files + +- Copy (skeletons): `crates/memoryd/src/notes.rs`, `crates/memoryd/src/edit.rs` +- Copy (tests): `crates/memoryd/tests/notes.rs`, `crates/memoryd/tests/edit.rs` +- Modify: `crates/memoryd/src/lib.rs` (`pub mod edit;` after `pub mod config;`, `pub mod notes;` + after `pub mod files;`), `docs/implementer-log.md` + +## The skeletons + +`notes.rs`: `Kind`, `Note`, `class_name`, `file_of` are written; fill `minute`, `Note::header`, +`Note::render`, `parse_header`, `parse`. `edit.rs`: `Context`, `Change`, `Outcome`, the sentences +(`over_limit`, `untrusted_refusal`, `SAVED_AS_NOTE`, `stale`), and the helpers `checked`, `note`, +`position` are written; fill `apply`, `edited`, `noted`, `consolidate`. `edited` still needs +an arm for a note, which `apply` never passes it: its comment says what that arm returns. + +## Steps + +- [ ] **1. Copy** the four files; edit `lib.rs`. +- [ ] **2. See it fail.** `cargo test -p memoryd --test notes --test edit`: `notes` 3 failed, + `edit` 8 failed. +- [ ] **3. Fill** `notes.rs` first, then `edit.rs` (`noted`, `consolidate`, `edited`, `apply`). + `cargo fmt --all`. +- [ ] **4. See it pass.** The same command: 3 and 8 passed. +- [ ] **5. Run the gate.** `make gate`: `gate: ok`, 11 more tests than before (about 896). +- [ ] **6. Log and commit.** `git add -A crates/memoryd docs/implementer-log.md Cargo.lock && git commit` + +## Done when + +- `make gate` prints `gate: ok`. + +## Stop and report if + +- A test fails after the functions follow their comments exactly. Do not edit a test. diff --git a/docs/plans/M5a/07-memoryd-journal.md b/docs/plans/M5a/07-memoryd-journal.md new file mode 100644 index 0000000..970f81c --- /dev/null +++ b/docs/plans/M5a/07-memoryd-journal.md @@ -0,0 +1,46 @@ +# M5a task 07: the journal, undo and review + +**Branch:** `m5a` (run `git switch m5a`; `git status --short` must be empty, otherwise stop) +**Commit subject:** `memoryd: the edit journal, undo of the latest edit, and review` + +## Goal + +M5a spec, section 5, "Seeing writes", "Undo" and "Review": + +- Every applied write is one JSON line in `memory/edits/YYYY-MM.jsonl`, with an id numbered from 1 + each day (`2026-09-24-0001`), the entries before and after, and where it went. `log` lists the + last N. +- `undo` reverts an edit only if it is the latest edit of its identity file **and** the file still + holds exactly what it left; the undo is journalled too. Notes are append-only and never undone. +- With review on, identity writes wait in `memory/pending.jsonl` until the owner accepts one + (applied then, against the file as it is) or rejects it (journalled). + +## Files + +- Copy (skeletons): `crates/memoryd/src/journal.rs`, `crates/memoryd/src/review.rs` +- Copy (test): `crates/memoryd/tests/journal.rs` +- Modify: `crates/memoryd/src/lib.rs` (`pub mod journal;` after `pub mod files;`, + `pub mod review;` after `pub mod notes;`), `docs/implementer-log.md` + +## The skeletons + +`journal.rs`: `Record` and `file_of` are written; fill `records`, `append`, `record`, `log`, +`undo`. `append` is `pub(crate)` because `review::reject` journals through it. `review.rs`: +`Line` is written; fill `lines`, `add`, `waiting`, `submit`, `list`, `accept`, `reject`. + +## Steps + +- [ ] **1. Copy** the three files; edit `lib.rs`. +- [ ] **2. See it fail.** `cargo test -p memoryd --test journal`: 4 failed. +- [ ] **3. Fill** `journal.rs`, then `review.rs`. `cargo fmt --all`. +- [ ] **4. See it pass.** The same command: 4 passed. +- [ ] **5. Run the gate.** `make gate`: `gate: ok`, 4 more tests than before (about 900). +- [ ] **6. Log and commit.** `git add -A crates/memoryd docs/implementer-log.md Cargo.lock && git commit` + +## Done when + +- `make gate` prints `gate: ok`. + +## Stop and report if + +- A test fails after the functions follow their comments exactly. Do not edit a test. diff --git a/docs/plans/M5a/08-memoryd-index.md b/docs/plans/M5a/08-memoryd-index.md new file mode 100644 index 0000000..ad1c901 --- /dev/null +++ b/docs/plans/M5a/08-memoryd-index.md @@ -0,0 +1,49 @@ +# M5a task 08: the index + +**Branch:** `m5a` (run `git switch m5a`; `git status --short` must be empty, otherwise stop) +**Commit subject:** `memoryd: the FTS5 index over session logs and notes, with labels` + +## Goal + +`index/memory.sqlite3` (M5a spec, section 6): an SQLite FTS5 table of text and a table of where +each came from. Indexed: the owner's messages, the assistant's replies, and each tool call by name +and arguments; **never a tool's result**. Each record is labelled with the session's label **at +that point in its log** (the highest class of any earlier tool result, and whether any was +untrusted). Records above `[index] max_class` are left out. It keeps, per log, how many records +it has read, so it only reads what is new; and it can be rebuilt whole into a new file renamed +over the old one. + +The `rusqlite` calls you need, checked on docs.rs for 0.40.2: `Connection::open(path)`, +`conn.execute_batch(sql)`, `conn.execute(sql, params![..])`, `conn.query_row(sql, params![..], +|row| row.get(0))`, `conn.last_insert_rowid()`. `u64` and `usize` are **not** SQL types in +rusqlite: pass `i64` (the skeleton's `int` does that). + +## Files + +- Copy (skeleton): `crates/memoryd/src/index.rs` +- Copy (tests): `crates/memoryd/tests/index.rs`, `crates/memoryd/tests/support/logs.rs` +- Modify: `crates/memoryd/src/lib.rs` (`pub mod index;` after `pub mod files;`), + `docs/implementer-log.md` + +## The skeleton + +Written: `SCHEMA`, `Source`, `Counts`, `Index`, `db`, `class_name`, `class_of`, `int`, and +`Index::conn`. Fill: `open`, `progress`, `add`, `set_progress`, `index_record`, `index_log`, +`index_notes`, `catch_up`, `logs`, `rebuild`. + +## Steps + +- [ ] **1. Copy** the three files; edit `lib.rs`. +- [ ] **2. See it fail.** `cargo test -p memoryd --test index`: 6 failed. +- [ ] **3. Fill** the functions in the order listed. `cargo fmt --all`. +- [ ] **4. See it pass.** The same command, three times: 6 passed each time. +- [ ] **5. Run the gate.** `make gate`: `gate: ok`, 6 more tests than before (about 906). +- [ ] **6. Log and commit.** `git add -A crates/memoryd docs/implementer-log.md Cargo.lock && git commit` + +## Done when + +- `make gate` prints `gate: ok`. + +## Stop and report if + +- A test fails after the functions follow their comments exactly. Do not edit a test. diff --git a/docs/plans/M5a/09-memoryd-recall.md b/docs/plans/M5a/09-memoryd-recall.md new file mode 100644 index 0000000..beef7aa --- /dev/null +++ b/docs/plans/M5a/09-memoryd-recall.md @@ -0,0 +1,46 @@ +# M5a task 09: `recall` + +**Branch:** `m5a` (run `git switch m5a`; `git status --short` must be empty, otherwise stop) +**Commit subject:** `memoryd: recall, with source tags and the recall log` + +## Goal + +`recall` (M5a spec, section 5): search the index and answer with at most `k` hits, best first +(FTS5's `bm25`), each cut to 600 characters and headed by a **source tag**: `identity` (an entry +moved out of an identity file), `memory` (trusted experience, and dropped entries), or +`memory-tainted` (from a session that had read untrusted content). The result's label is the +highest class among the hits, and untrusted if any hit is tagged `memory-tainted`. The model's +query reaches SQLite **only as quoted terms**, so no FTS5 operator it writes is ever parsed. Every +recall, and every query that found nothing, is logged in `memory/recalls/YYYY-MM.jsonl`. + +The one `rusqlite` call not used before: `stmt.query_map(params![..], |r| ..)`, collected into a +`Result, _>`. + +## Files + +- Copy (skeleton): `crates/memoryd/src/recall.rs` +- Copy (test): `crates/memoryd/tests/recall.rs` +- Modify: `crates/memoryd/src/lib.rs` (`pub mod recall;` after `pub mod notes;`), + `docs/implementer-log.md` + +## The skeleton + +Written: `SNIPPET`, `Recalled`, `Hit`, `LoggedHit`, `Logged`. Fill: `terms`, `when`, `day`, +`Hit::tag`, `Hit::header`, `recall`, `search`. + +## Steps + +- [ ] **1. Copy** the two files; edit `lib.rs`. +- [ ] **2. See it fail.** `cargo test -p memoryd --test recall`: 6 failed. +- [ ] **3. Fill** the seven functions. `cargo fmt --all`. +- [ ] **4. See it pass.** The same command, three times: 6 passed each time. +- [ ] **5. Run the gate.** `make gate`: `gate: ok`, 6 more tests than before (about 912). +- [ ] **6. Log and commit.** `git add -A crates/memoryd docs/implementer-log.md Cargo.lock && git commit` + +## Done when + +- `make gate` prints `gate: ok`. + +## Stop and report if + +- A test fails after the functions follow their comments exactly. Do not edit a test. diff --git a/docs/plans/M5a/10-memoryd-serve.md b/docs/plans/M5a/10-memoryd-serve.md new file mode 100644 index 0000000..eb47177 --- /dev/null +++ b/docs/plans/M5a/10-memoryd-serve.md @@ -0,0 +1,79 @@ +# M5a task 10: serving + +**Branch:** `m5a` (run `git switch m5a`; `git status --short` must be empty, otherwise stop) +**Commit subject:** `memoryd: serve, three sockets and one lock over the files and the index` + +## Goal + +`memoryd serve --config ` (M5a spec, section 7): at start it creates what is missing under +`memory/`, opens the index and catches it up, then listens on three sockets, one request per +connection, each answered by one frame: + +| Socket | Peer | Takes | +|---|---|---| +| `run/loop-memory/memory.sock` | `loopd` | `memory_block`, `session_updated` | +| `run/broker-memory/memory.sock` | `brokerd` | `memory_call` (`recall` or `memory`) | +| `run/owner-memory/admin.sock` | `bxctl` | `reindex`, `memory_admin` | + +A message on the wrong socket is refused (`BadMessage`). One lock covers the files and the index, +so writes never interleave. + +## Files + +- Copy (skeleton): `crates/memoryd/src/serve.rs` +- Copy (test): `crates/memoryd/tests/serve.rs` +- Replace: `crates/memoryd/src/main.rs` with `docs/plans/M5a/files/crates/memoryd/src/main.rs-task10` + (task 03 gave it whole, and nothing has changed it since; this version starts the server) +- Modify: `crates/memoryd/src/lib.rs` (`pub mod serve;` after `pub mod review;`), + `crates/memoryd/src/index.rs`, `docs/implementer-log.md` + +## `index.rs`, exactly + +Add this function just before the doc comment `/// Every session log under `sessions`: `(session, +epoch, path)`, sorted.`: + +```rust +/// The logs of one session: `(epoch, path)`, sorted. A session with no directory has none. +pub fn session_logs(sessions: &Path, session: &str) -> Result, String> { + let dir = sessions.join(session); + let files = match std::fs::read_dir(&dir) { + Ok(files) => files, + Err(e) if e.kind() == std::io::ErrorKind::NotFound => return Ok(Vec::new()), + Err(e) => return Err(format!("{}: {e}", dir.display())), + }; + let mut found = Vec::new(); + for file in files.filter_map(|f| f.ok()) { + let name = file.file_name().to_string_lossy().into_owned(); + if let Some(Ok(epoch)) = name.strip_suffix(".jsonl").map(str::parse::) { + found.push((epoch, file.path())); + } + } + found.sort(); + Ok(found) +} +``` + +## The skeleton + +Written: `READ_WAIT`, `Log`, `State`, `Memoryd`, `Running`, `Which`, `error`, `bad`, +`Memoryd::lock`, and **`Memoryd::handle`**, the table of which socket takes which message: read +it first. Fill: `start`, `Running::run`, `connection`, `block`, `session_updated`, `call`, +`recall`, `memory`, `index_notes`, `reindex`, `admin`. + +## Steps + +- [ ] **1. Copy** the two files, replace `main.rs`; edit `lib.rs` and `index.rs`. +- [ ] **2. See it fail.** `cargo test -p memoryd --test serve`: 8 failed. +- [ ] **3. Fill** the functions. `cargo fmt --all`. +- [ ] **4. See it pass.** The same command, three times: 8 passed each time, in about a second. +- [ ] **5. Run the gate.** `make gate`: `gate: ok`, 8 more tests than before (about 920). +- [ ] **6. Log and commit.** `git add -A crates/memoryd docs/implementer-log.md Cargo.lock && git commit` + +## Done when + +- `make gate` prints `gate: ok`. + +## Stop and report if + +- A test fails after the functions follow their comments exactly. Do not edit a test. +- A test waits more than a few seconds, or passes only sometimes. diff --git a/docs/plans/M5a/11-brokerd-memory-tools.md b/docs/plans/M5a/11-brokerd-memory-tools.md new file mode 100644 index 0000000..2d59836 --- /dev/null +++ b/docs/plans/M5a/11-brokerd-memory-tools.md @@ -0,0 +1,199 @@ +# M5a task 11: `brokerd` knows the memory tools, under built-in grants + +**Branch:** `m5a` (run `git switch m5a`; `git status --short` must be empty, otherwise stop) +**Commit subject:** `brokerd: recall and memory, decided under built-in grants` + +## Goal + +M5a spec, section 8: + +- `recall` and `memory` are tools `brokerd` parses and decides, with their shape checks from + `proto` (task 02). +- Their policy is **built in**: two fixed grants, `builtin-recall` and `builtin-memory`, both + `auto`, in every set of grants. They are kept apart from the owner's grants (`grants()` lists + only the owner's; matching sees both through `matchable()`). +- A grant file naming `recall` or `memory` is a grant problem at load, and grant ids starting + `builtin-` are reserved. So no grant file can turn memory off or put questions in front of it. + +Where a decided memory call goes (`memoryd`, not a container) is task 12. Until then, like every +tool, it reaches the configured runtime. + +## Files + +- Copy (skeleton): `crates/brokerd/src/builtin.rs` +- Copy (tests): `crates/brokerd/tests/memory_tools.rs`, and a **new version** of the given + `crates/brokerd/tests/grants.rs` (an empty directory now loads the two built-ins, which the old + test compared against an empty set) +- Modify: `crates/brokerd/src/lib.rs` (`pub mod builtin;` after `pub mod broker;`), + `crates/brokerd/src/args.rs`, `crates/brokerd/src/grants.rs`, + `crates/brokerd/src/policy/matching.rs`, `crates/brokerd/src/runner.rs`, + `crates/brokerd/src/config.rs`, `docs/implementer-log.md` + +## The skeleton + +`builtin.rs`: `RESERVED`, the two grants' text `RECALL` and `MEMORY`, and the sentence `BUILT_IN` +are written. Fill `grants`. + +## `args.rs`, exactly + +1. After `use proto::tools::{HttpFetchArgs, ReadFileArgs, ShellArgs, WriteFileArgs};` add + `use proto::{MemoryArgs, RecallArgs};`. +2. The doc comment of `enum ToolName`, `/// The four tools the broker can dispatch to.`, becomes + `/// The tools the broker can dispatch to: four that run in a container, and the two memory + tools` / `/// that `memoryd` runs (M5a spec, section 8).` (two lines), and add the variants + `Recall,` and `Memory,` after `HttpFetch,`. +3. The doc comment of `ALL`, `/// The four tools, in the order listed in the interface.`, becomes + `/// The four container tools, in the order listed in the interface.` `ALL` itself is + unchanged: it stays the four container tools. +4. In `ToolName::parse`, add `"recall" => Some(ToolName::Recall),` and + `"memory" => Some(ToolName::Memory),` before `_ => None,`. In `as_str`, add + `ToolName::Recall => "recall",` and `ToolName::Memory => "memory",` after the `HttpFetch` arm. +5. After `as_str`, still inside `impl ToolName`, add: + +```rust + + /// Whether `memoryd` runs this tool rather than a container. + pub fn is_memory(self) -> bool { + matches!(self, ToolName::Recall | ToolName::Memory) + } +``` + +6. In `enum ToolArgs`, add `Recall(RecallArgs),` and `Memory(MemoryArgs),` after + `HttpFetch(FetchUrl),`. In `ToolArgs::tool`, add `ToolArgs::Recall(_) => ToolName::Recall,` + and `ToolArgs::Memory(_) => ToolName::Memory,` after the `HttpFetch` arm. In + `canonical_json`, after the `HttpFetch` arm, add: + +```rust + ToolArgs::Recall(value) => { + serde_json::to_string(value).unwrap_or_else(|_| "{}".to_string()) + } + ToolArgs::Memory(value) => { + serde_json::to_string(value).unwrap_or_else(|_| "{}".to_string()) + } +``` + +7. In `parse`, after the `ToolName::HttpFetch => { … }` arm, add: + +```rust + ToolName::Recall => { + let value: RecallArgs = serde_json::from_str(arguments) + .map_err(|error| ArgsError::Shape(error.to_string()))?; + value.check().map_err(ArgsError::Shape)?; + Ok(ToolArgs::Recall(value)) + } + ToolName::Memory => { + let value: MemoryArgs = serde_json::from_str(arguments) + .map_err(|error| ArgsError::Shape(error.to_string()))?; + value.check().map_err(ArgsError::Shape)?; + Ok(ToolArgs::Memory(value)) + } +``` + +## `grants.rs`, exactly + +1. In `struct GrantSet`, after `grants: Vec,` add: + +```rust + /// The built-in grants of the memory tools, which every set from `from_grants` has. They are + /// not the owner's, so `grants` does not list them; matching sees them (`matchable`). + builtin: Vec, +``` + +2. At the end of `from_grants`, the block + +```rust + if problems.is_empty() { + Ok(GrantSet { grants }) + } else { + Err(problems) + } +``` + + becomes + +```rust + if !problems.is_empty() { + return Err(problems); + } + // The built-ins join only a set that passed: an owner's grant for a memory tool is a + // problem above, never a replacement. + let builtin = crate::builtin::grants().map_err(|problem| vec![problem])?; + Ok(GrantSet { grants, builtin }) +``` + +3. The doc comment of `grants()`, `/// The grants in id order.`, becomes + `/// The owner's grants in id order.`, and after `grants()` add: + +```rust + + /// Every grant a call is matched against: the owner's, then the built-ins. + pub fn matchable(&self) -> impl Iterator { + self.grants.iter().chain(self.builtin.iter()) + } +``` + +4. In `check_grant`, just before `// Rule 3: the tool must be one of the four; …`, add: + +```rust + // Rule 2b: ids starting `builtin-` are the built-in grants'. + if id.starts_with(crate::builtin::RESERVED) { + push( + problems, + file.clone(), + None, + "grant ids starting builtin- are reserved", + ); + } + +``` + + and under the Rule 3 comment add the line `// The memory tools' grants are built in (M5a spec, + section 8).`; in its `match`, before `Some(tool) => check_tool_constraints(…)`, add: + +```rust + Some(tool) if tool.is_memory() => { + push(problems, file.clone(), None, crate::builtin::BUILT_IN) + } +``` + +## The other three files, exactly + +- `policy/matching.rs`, in `match_grants`: `for loaded in grants.grants() {` becomes + `for loaded in grants.matchable() {`. In `covers`, just before the `ToolArgs::HttpFetch(target)` + arm, add: + +```rust + // The memory tools take no constraints: their grants are built in (M5a spec, section 8). + ToolArgs::Recall(_) | ToolArgs::Memory(_) => Some(None), +``` + +- `runner.rs`, in `run`, after the arm + `ToolArgs::HttpFetch(_) => (Vec::new(), Some(decision.hosts().to_vec())),` add + `ToolArgs::Recall(_) | ToolArgs::Memory(_) => (Vec::new(), None),`. +- `config.rs`, in `Runner::time_limit`, after `crate::args::ToolName::HttpFetch => self.http_fetch_ms,` add: + +```rust + // `memoryd` runs these; they never reach a container. + crate::args::ToolName::Recall | crate::args::ToolName::Memory => self.read_file_ms, +``` + +## Steps + +- [ ] **1. Copy** the skeleton and the two tests; make the edits, file by file, with + `cargo check -p brokerd` after each. +- [ ] **2. See it fail.** `cargo test -p brokerd --test memory_tools --test grants`: both fail + (`builtin::grants` is still a `todo!()`). +- [ ] **3. Fill** `builtin::grants`. `cargo fmt --all`. +- [ ] **4. See it pass.** The same command: 4 and 17 passed. Then `cargo test -p brokerd`: every + suite passes. +- [ ] **5. Run the gate.** `make gate`: `gate: ok`, 4 more tests than before (about 924). +- [ ] **6. Log and commit.** `git add -A crates/brokerd docs/implementer-log.md Cargo.lock && git commit` + +## Done when + +- `make gate` prints `gate: ok`. + +## Stop and report if + +- Any other `brokerd` test fails. The given `policy_property` test matches random grants against + the four container tools; the built-ins must not change what it sees. diff --git a/docs/plans/M5a/12-brokerd-memory-port.md b/docs/plans/M5a/12-brokerd-memory-port.md new file mode 100644 index 0000000..be80b8e --- /dev/null +++ b/docs/plans/M5a/12-brokerd-memory-port.md @@ -0,0 +1,202 @@ +# M5a task 12: decided memory calls go to `memoryd`, and labels only get stricter + +**Branch:** `m5a` (run `git switch m5a`; `git status --short` must be empty, otherwise stop) +**Commit subject:** `brokerd: memory calls to memoryd with the session's state; labels only stricter` + +## Goal + +M5a spec, section 8: + +- A decided `recall` or `memory` call goes to `memoryd`'s `brokerd` socket as a `memory_call`, + with the session's taint and untrusted flag **as `brokerd` held them when it decided**. Every + other call goes to the configured runtime, as before (`Routed`). +- `memoryd` knows the label of what it returns (a `recall` hit from an untrusted session). The + result keeps the **stricter** of that and the grant's label: the higher class, and untrusted if + either says so. A runtime can never make a result laxer. +- The container runtime refuses a memory tool rather than start a container for it. +- `memoryd` not answering in 30 seconds is a failed result with a fixed sentence and a runbook + pointer. + +To carry the session's state, `Decision` gains a private field (`decide` and `redecide` already +receive the state), and `RunSpec` gains one too. `Runtime` gains `run_labelled`, with a default +that calls `run` and returns no label, so no existing runtime changes. + +## Files + +- Copy (skeleton): `crates/brokerd/src/memory_port.rs` +- Copy (tests): `crates/brokerd/tests/memory_port.rs`, and a **new version** of the given + `crates/brokerd/tests/support/rig.rs` (its `Config` literal gains `memory`) +- Append: `docs/plans/M5a/files/runbook-memoryd-unreachable.md` to the end of `docs/runbook.md` + (a blank line before it) +- Modify: `crates/brokerd/src/lib.rs` (`pub mod memory_port;` after `pub mod ledger;`), + `crates/brokerd/src/policy/mod.rs`, `crates/brokerd/src/runner.rs`, + `crates/brokerd/src/container.rs`, `crates/brokerd/src/config.rs`, `crates/brokerd/src/main.rs`, + `docs/implementer-log.md` + +## The skeleton + +Written: `UNREACHABLE`, `MEMORY_WAIT`, `MemoryPort` and `new`, `Routed`, and both `run` methods +(they call `run_labelled`). Fill: `MemoryPort::ask`, `MemoryPort::run_labelled`, +`Routed::run_labelled`. + +## `policy/mod.rs`, exactly + +1. In `struct Decision`, after `matched: Matched,` add: + +```rust + /// The session's state when the call was decided; `memoryd` needs it (M5a spec, section 8). + state: SessionState, +``` + + (Only `Decision`: `Ask` is unchanged.) +2. In `impl Decision`, after `request()` add: + +```rust + pub fn state(&self) -> SessionState { + self.state + } +``` + +3. In `decide`, the `Mode::Auto => Outcome::Allowed(Decision { request, args, matched, })` + literal gains `state,` after `matched,`; in `redecide`, the `Ok(Decision { … matched, })` + literal gains `state,` after `matched,`. Both functions already have `state` as a parameter. + +## `runner.rs`, exactly + +1. `use crate::policy::Decision;` becomes `use crate::policy::{Decision, Label, SessionState};`. +2. In `struct RunSpec`, after `egress: Option>,` add `state: SessionState,`; in + `impl RunSpec`, after `egress()` add: + +```rust + /// The session's state when the call was decided. + pub fn state(&self) -> SessionState { + self.state + } +``` + +3. In `trait Runtime`, after `fn run(…);` add: + +```rust + + /// `run`, and a label for what came back when the runtime knows one (`memoryd` does: a + /// `recall` hit from an untrusted session). The runner keeps the stricter of this and the + /// grant's label. Container runtimes know none. + fn run_labelled(&self, spec: &RunSpec) -> Result<(RunOutput, Option