boxmaker/docs/plans/M5c1/10-memoryd-honcho-client.md

M5c-1 task 10: memoryd's Honcho clients, and making sure a workspace, its peers and a session exist

Branch: m5c1 (run git switch m5c1; git status --short must be empty, otherwise stop) Commit subject: memoryd: a Honcho client per workspace, and its workspace, peers and sessions made sure of Log row: the Task cell is M5c1/10-memoryd-honcho-client.

Goal

Tasks 11 and 13 send turns, conclusions and fleet messages to Honcho. This task gives them what they send through (M5c spec, sections 3 and 6):

• A client per workspace. One roncho blocking client for each of this agent's three workspaces ([honcho] trusted, untrusted, fleet), each with its own key when [secrets.honcho_trusted] (and so on) is configured. Every client is built with .without_config_file(): without it roncho reads ~/.config/roncho/roncho.toml and takes an api_key from it. Timeouts: connect 5 s, read ("no byte for") 30 s; chat, which M5c-2 uses, 180 s (the constant only, here).
• Making sure things exist before anything is written: the workspace, the agent's peer, the speakers' peers, then the session. Each is Honcho's get-or-create (POST: 201 when made, 200 when it was there; both are success, and so is a 409). A session must be made before a message is added to it: Honcho makes a missing one on the fly, with no peers configured (recorded: 20-messages-add-missing-session.json).
  • The fleet workspace is made with reasoning, summary and dream all enabled: false; the other two with Honcho's defaults (no configuration sent).
  • The agent's peer is made with observe_me: false; a speaker's peer with nothing.
  • A session holds the agent's peer (observe_me: false, observe_others: false) and each speaker's (observe_me: true, observe_others: false).
  • What was made sure of is remembered in memory for the run. After a restart everything is made sure of again, which costs one request each and changes nothing.
• Which Honcho session a turn goes to (spec 3): the Boxmaker session's id; a speaker new to it starts the next, <id>-2, <id>-3, holding every speaker so far (a Honcho session's peers are fixed when it is made). The numbering comes from the ledger alone: the speakers of that session's decided Turn actions, in any workspace, in the order of the decisions, each counted once, up to and including the turn. A restart numbers the same.
• Errors by kind and status, never a body. A 422 echoes the whole request, memory text included, and roncho's Status and Decode errors carry up to 4 KiB of the response. So a HonchoError keeps only a kind (a fixed word) and a status.
• Deleting a conclusion (for task 13, which uses it): 204 is done, and 404 (deleted before) is done too.

Nothing calls this yet: task 11 builds the clients at start and sends turns. No request goes to a real Honcho in any test: a fake Honcho on 127.0.0.1 answers from the responses the design model recorded from rift in task 00, crates/memoryd/tests/fixtures/honcho/, which this task copies in.

Interfaces

In crates/memoryd/src/honcho/client.rs (the module memoryd::honcho::client):

pub const CONNECT: Duration = Duration::from_secs(5);
pub const READ: Duration = Duration::from_secs(30);
pub const CHAT_READ: Duration = Duration::from_secs(180);
pub struct Timeouts { pub connect: Duration, pub read: Duration } // Timeouts::WRITES = (CONNECT, READ)
pub enum HonchoError {
    Unreachable { kind: &'static str, status: Option<u16> },
    Refused { kind: &'static str, status: Option<u16> },
    KeyRefused { status: u16 },
}
impl HonchoError { pub fn kind(&self) -> &'static str; pub fn status(&self) -> Option<u16>; }
pub fn classify(error: &roncho::blocking::Error) -> HonchoError;
pub fn fleet_configuration() -> serde_json::Map<String, Value>;
pub fn session_peers(agent: &str, speakers: &[String]) -> HashMap<String, SessionPeerConfig>;
pub struct Clients { /* private */ }
impl Clients {
    pub fn new(config: &HonchoConfig, env: &dyn Fn(&str) -> Option<OsString>, timeouts: Timeouts)
        -> Result<(Clients, Vec<String>), String>;
    pub fn client(&self, workspace: Workspace) -> &roncho::blocking::Client;
    pub fn ensure_workspace(&mut self, workspace: Workspace) -> Result<(), HonchoError>;
    pub fn ensure_peer(&mut self, workspace: Workspace, peer: &str) -> Result<(), HonchoError>;
    pub fn ensure_session(&mut self, workspace: Workspace, session: &HonchoSession)
        -> Result<(), HonchoError>;
    pub fn delete_conclusion(&self, workspace: Workspace, id: &str) -> Result<(), HonchoError>;
}

In crates/memoryd/src/honcho/naming.rs (the module memoryd::honcho::naming):

pub struct HonchoSession { pub name: String, pub speakers: Vec<String> }
pub fn session_name(session: &SessionId, count: usize) -> String;
pub fn honcho_session<'a, I>(decisions: I, session: &SessionId, turn: u64) -> Option<HonchoSession>
where I: IntoIterator<Item = (&'a Item, &'a Action)>;

In crates/memoryd/src/honcho/ledger.rs, one new method (edit L1): pub fn decisions(&self) -> impl Iterator<Item = (&Item, &Action)> + '_, every decided item and its action (Skip included) in the order of the Decided records.

Clients::new returns the key files' plaintext warnings for the caller to log. Its error is one line naming the key (bxcore::secrets' own text, with its pointer) or the workspace; never a key's value.

The errors

classify is given whole in the skeleton. It maps roncho's error so:

roncho::blocking::Error	HonchoError
Connect, Timeout, Closed, Io	Unreachable, kind connect, timeout, closed, io; no status
Status with a status of 500 or more	Unreachable, kind server, the status
Status below 500	Refused, kind status, the status
Unauthorized (401 or 403)	KeyRefused, the status
Redirect	Refused, kind redirect, the status
ActiveSessions	Refused, kind active-sessions, 409
TooLarge, Decode, Stream, Config, any other	Refused, kind too-large, decode, stream, config, other; no status

Display (given): Honcho is unreachable (<kind>[, HTTP <status>]), Honcho refused the request (<kind>[, HTTP <status>]), Honcho refused the key (key-refused, HTTP <status>). For a create, a Refused with status 409 is success (created); for a delete, one with 404 is.

The requests

Exactly what the given tests expect (Value bodies, compared as JSON):

Step	Request	Body
a workspace	POST /v3/workspaces	{"id":"<name>"}; the fleet: {"id":"<name>","configuration":{"reasoning":{"enabled":false},"summary":{"enabled":false},"dream":{"enabled":false}}}
the agent's peer	POST /v3/workspaces/<name>/peers	{"id":"<agent>","configuration":{"observe_me":false}}
a speaker's peer	the same	{"id":"<speaker>"}
a session	POST /v3/workspaces/<name>/sessions	{"id":"<session>","peers":{"<agent>":{"observe_me":false,"observe_others":false},"<speaker>":{"observe_me":true,"observe_others":false},…}}
delete a conclusion	DELETE /v3/workspaces/<name>/conclusions/<id>	none

ensure_session (given whole) makes sure of the workspace, the agent's peer, each speaker's peer, and then the session, each only if not yet made sure of in this run. A failed step is not remembered, so the next call tries it again.

Every place that changes

This is the complete list.

Place	What happens there
crates/memoryd/src/honcho/mod.rs	pub mod client; and pub mod naming; (edit H1)
crates/memoryd/src/honcho/ledger.rs	decisions (edit L1)
crates/memoryd/src/honcho/client.rs	new: a skeleton you copy, then fill 11 bodies
crates/memoryd/src/honcho/naming.rs	new: a skeleton you copy, then fill 2 bodies
crates/memoryd/tests/fixtures/honcho/	new: the recorded responses, copied
everything else (serve.rs, turns.rs, config.rs, Cargo.toml, other crates)	no change

Files

• Copy, from docs/plans/M5c1/files/ (the same path under it):
  • new tests: crates/memoryd/tests/honcho_client.rs, crates/memoryd/tests/honcho_keys.rs, crates/memoryd/tests/honcho_naming.rs
  • new test support: crates/memoryd/tests/support/honcho.rs (the fake Honcho)
  • the directory crates/memoryd/tests/fixtures/honcho/ (29 JSON files and a README), whole
  • the skeletons crates/memoryd/src/honcho/client.rs and crates/memoryd/src/honcho/naming.rs
• Modify: crates/memoryd/src/honcho/mod.rs, crates/memoryd/src/honcho/ledger.rs, docs/implementer-log.md.
• Do not touch: any other source file, any other test or support file, the fixtures (never edit one, and never record new ones: there is no Honcho to reach from here), any Cargo.toml, and in the skeletons anything but the 13 todo!() bodies.

Facts you need

• roncho 1.0.0 is already a dependency (roncho.workspace = true, the feature tls, which brings blocking). Its blocking client is HTTP/1.1 over std::net, one request per connection. The calls this task uses, all on roncho::blocking::Client:

  Call	Signature, and what it gives
  Client::builder()	-> ClientBuilder; then .without_config_file(), .base_url(url), .workspace_id(name), .connect_timeout(d), .read_timeout(d), .api_key(key) (each self -> Self), .build() -> Result<Client, Error>
  workspace_id	client.workspace_id() -> &str
  get_or_create_workspace	client.get_or_create_workspace(&WorkspaceCreate) -> Result<roncho::Workspace, Error>
  peer	client.peer(&PeerCreate) -> Result<roncho::models::Peer, Error>
  session	client.session(&SessionCreate) -> Result<roncho::models::Session, Error>
  delete_conclusion	client.delete_conclusion(id: &str) -> Result<(), Error>
  WorkspaceCreate	roncho::WorkspaceCreate::new(id), .with_configuration(Map<String, Value>)
  PeerCreate	roncho::PeerCreate::new(id), .with_configuration(Map<String, Value>)
  SessionCreate	roncho::SessionCreate::new(id), .with_peers(HashMap<String, SessionPeerConfig>)
  SessionPeerConfig	roncho::SessionPeerConfig::new().observe_me(bool).observe_others(bool)

  roncho's Client and ClientBuilder Debug show api_key_present, never the key; the client keeps the key in a Zeroizing<String>.

• Keys. HonchoConfig.keys (task 06) maps honcho_trusted, honcho_untrusted, honcho_fleet to a checked SecretSource. bxcore::secrets::load("memoryd", name, source, env) -> Result<Loaded, SecretError> reads it (Loaded { secret: Secret, warning: Option<String> }; a file source always has a warning). Secret::expose() -> &str is the one place the value is taken, to hand it to roncho. SecretError's Display names the key and ends with see docs/runbook.md#secret-unavailable. env is std::env::var_os in memoryd; tests pass their own.

• serde_json. serde_json::Map::new(), map.insert(String, Value), and serde_json::json!({ "enabled": false }) (imported as json).

• The fake Honcho (tests/support/honcho.rs) answers by method and the shape of the path: the first create of a thing with its 201 fixture, a repeat with its 200 one; the first delete of a conclusion 204, a repeat 404; anything else 404. It can be set to answer every request with one status (a 422 echoing the request's body, as Honcho does) or to answer nothing, and it records every request with its headers and body.

• The given tests.

  • honcho_client.rs (13 tests): the timeouts; each client's workspace and timeouts (from roncho's Debug); the fleet's configuration equal to the recorded one; the four requests of a session, in order and exactly; nothing asked twice in a run, a new speaker's peer and session only, another workspace on its own; everything asked again after a restart (200 is success); the fleet's workspace and a session with the agent alone; 409 on create; errors by kind and status, not remembered, and a timeout (the fake answers nothing, the read timeout is 300 ms); nothing listening; no response text in an error; classify for every error; deleting a conclusion twice.
  • honcho_keys.rs (3 tests, and 1 ignored that the third runs): each key sent as a bearer token to its own workspace only, and shown by no client, error or warning; a key that cannot be read; a roncho.toml with an api_key under HOME not read (a child process of the test binary, since HOME is the process's).
  • honcho_naming.rs (5 tests): the names; each new speaker, in any workspace, starting the next session; a skipped or undecided turn; the order of decision; a restart; decisions.

• Where the delete lives. delete_conclusion is here, with its two recorded answers; task 13 calls it for an undo.

The edits, exactly

crates/memoryd/src/honcho/mod.rs

H1. The lines

pub mod config;
pub mod decide;
pub mod ledger;

become

pub mod client;
pub mod config;
pub mod decide;
pub mod ledger;
pub mod naming;

crates/memoryd/src/honcho/ledger.rs

L1. The lines

    /// The items waiting to be sent, with their actions, in the order they were decided.
    pub fn outbox(&self) -> Vec<(Item, Action)> {

become

    /// Every decided item with its action, `Skip` included, in the order they were decided.
    pub fn decisions(&self) -> impl Iterator<Item = (&Item, &Action)> + '_ {
        self.order
            .iter()
            .filter_map(|item| Some((item, &self.entries.get(item)?.action)))
    }

    /// The items waiting to be sent, with their actions, in the order they were decided.
    pub fn outbox(&self) -> Vec<(Item, Action)> {

Steps

• 1. Copy the given files. From the repository root:

  mkdir -p crates/memoryd/tests/fixtures
  cp -r docs/plans/M5c1/files/crates/memoryd/tests/fixtures/honcho crates/memoryd/tests/fixtures/
  cp docs/plans/M5c1/files/crates/memoryd/tests/honcho_client.rs crates/memoryd/tests/
  cp docs/plans/M5c1/files/crates/memoryd/tests/honcho_keys.rs crates/memoryd/tests/
  cp docs/plans/M5c1/files/crates/memoryd/tests/honcho_naming.rs crates/memoryd/tests/
  cp docs/plans/M5c1/files/crates/memoryd/tests/support/honcho.rs crates/memoryd/tests/support/
  

  ls crates/memoryd/tests/fixtures/honcho | wc -l prints 30.

• 2. See them fail. cargo build --tests --keep-going -p memoryd, with a shell timeout of 600000 ms: three test targets do not compile. honcho_client, 2 errors: E0432 (unresolved import `memoryd::honcho::client` ) and E0432 (the same for memoryd::honcho::naming). honcho_keys, 1 error: the E0432 for client. honcho_naming, 8 errors: the E0432 for naming, and seven E0599 (no method named `decisions` found for struct `memoryd::honcho::ledger::Ledger` ). Every other target compiles.

• 3. The modules. Make edits H1 and L1, then copy the two skeletons:

  cp docs/plans/M5c1/files/crates/memoryd/src/honcho/client.rs crates/memoryd/src/honcho/
  cp docs/plans/M5c1/files/crates/memoryd/src/honcho/naming.rs crates/memoryd/src/honcho/
  

  cargo check -p memoryd: no error, and 26 warnings, all from the unfilled bodies: unused import: `json` ; unused imports: `PeerCreate` and `WorkspaceCreate` ; twenty-two unused variable (workspace five times, session twice, and agent, config, count, decisions, env, id, key, name, peer, result, speakers, timeouts, turn, url, warnings once each); fields `trusted`, `untrusted`, and `fleet` are never read; and variants `Workspace` and `Peer` are never constructed. They go as you fill the bodies.

• 4. Fill the thirteen bodies, one at a time, in the order of each file, each from its comment, and run cargo check -p memoryd after each. In client.rs: kind, status, fleet_configuration, session_peers, created, client (the free function), key, Clients::client, ensure_workspace, ensure_peer, delete_conclusion. In naming.rs: session_name, honcho_session. When a body is filled, delete its instruction comment (the // lines inside the body) and keep the function's doc comment (///) as it is. Change nothing else: classify, Display, Clients::new and ensure_session are given. After the last, cargo check -p memoryd prints no warning.

• 5. The tests. With a shell timeout of 600000 ms, and every test passes (13, 3 and 5; one test is ignored: the child the third test of honcho_keys runs):

  cargo test -p memoryd --test honcho_client --test honcho_keys --test honcho_naming
  

• 6. Check. Three commands:

  • grep -c 'without_config_file' crates/memoryd/src/honcho/client.rs prints 1.
  • grep -n 'expose()' crates/memoryd/src -r prints exactly one line, in client (the free function): the one place a key's value is taken.
  • git diff --stat -- crates/bxcore crates/loopd crates/brokerd crates/gatewayd crates/bxctl Cargo.toml Cargo.lock crates/memoryd/Cargo.toml crates/memoryd/src/serve.rs prints nothing.

• 7. Run the gate. cargo fmt --all, then make gate, with a shell timeout of 600000 ms. Expected last line: gate: ok, with 1278 tests: 1257 before this task, plus 21 new ones (13 in honcho_client.rs, 3 in honcho_keys.rs, 5 in honcho_naming.rs).

• 8. Log and commit. Add your row to docs/implementer-log.md; its Task cell is M5c1/10-memoryd-honcho-client. git status --short lists honcho/mod.rs, honcho/ledger.rs and your log as modified, and honcho/client.rs, honcho/naming.rs, the three test files, tests/support/honcho.rs and tests/fixtures/ as new. Stage with git add crates/memoryd docs/implementer-log.md Cargo.lock, then commit as AGENTS.md says. Cargo.lock does not change in this task; the git add is harmless.

Done when

• make gate prints gate: ok with 1278 tests.
• git diff HEAD~1 --stat lists only the files of step 8.

Stop and report if

• A "before" block of H1 or L1 is not in its file exactly as shown, or is there more than once.
• Step 1 does not give 30 files, or step 2 or 3 shows a different result: other test targets, errors of another kind, or other warnings.
• cargo check reports an error after a body you filled, and the comment does not tell you how to fix it. Quote the error, and stop.
• A comment in a skeleton does not fit the code under it, or disagrees with this task's tables. Quote both and stop.
• After step 4 a test fails although each body follows its comment. Do not change a test, the fake, a fixture or the skeletons' given code: stop, and give the failing tests' names and their messages.
• A test seems to need a fixture that is not there, or a request to a real Honcho. It does not; stop and say why you think so. Never record a response.
• The fix seems to need a change anywhere else: serve.rs, turns.rs, a Cargo.toml, another crate. It does not; stop and say why you think so.
• A test outside the new files fails in the gate: check once whether it fails without your change, then stop and report both results.
• The gate passes with a count other than 1278.

What the given tests pin

Each of these changes to the finished code makes at least one given test fail (44 tried, one cannot change behaviour): either timeout ignored or changed; roncho.toml read; each error mapped to the other side (a 5xx refused, a 401 refused, a timeout, a connect failure or a closed socket refused, a bad body unreachable); a 409 on create an error, or any refusal on create taken as success; a 404 on delete an error; the fleet's settings on another workspace, missing one, or on; the agent observed, or every peer unobserved; the workspace, a peer or a session not remembered, or remembered for every workspace, or before it was made; the session made without its workspace, its agent or its speakers first; the agent observed or left out of a session, the speakers unobserved or observing others; a key sent to the wrong workspace or dropped; a key file's warning dropped; a key that cannot be read taken as no key; the wrong client for a workspace; an error shown without its status; a session named <id>-1; a speaker counted twice; another session's speakers counted; the speakers of later turns counted; decisions without the skipped items or in reverse.

After this task

Task 11 builds the Clients at start (after the ledger, logging the warnings, a key that cannot be read stopping memoryd with secret-unavailable), and sends turns with ensure_session and honcho_session. Task 13 uses delete_conclusion, and makes the fleet's sessions with no speakers.