kyle/boxmaker
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

memoryd: a reported turn decided after loopd has its answer, from brokerd and the log

Browse Source 
Co-Authored-By: ornith-1.5-35b-a3b <agent@wntrmute.dev>
This commit is contained in:
kyle
2026-09-28 11:59:34 -07:00
co-authored by ornith-1.5-35b-a3b
parent a339eb58b6
commit 28df2a0e3b
12 changed files with 1483 additions and 22 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/runbook.md
+65 -16
View File
@@ -368,7 +368,9 @@ that does not parse tool calls, or whose cache does not work.
## session-log-damaged
**What you see.** A turn on an existing session fails with `<file>:<line>: <reason>` naming
`sessions/<id>/<epoch>.jsonl`, and this entry. Other sessions work.
`sessions/<id>/<epoch>.jsonl`, and this entry. Other sessions work. With `[honcho]`, `memoryd`
logs the same line once, starting `memoryd: `, and that session's turns wait undecided for Honcho
until the log reads again.
**Why.** `loopd` replays the log exactly to rebuild the conversation, so a line it cannot read
would change what the model sees. It refuses rather than guess. `last line does not end in a
@@ -926,23 +928,27 @@ tables and every `[secrets.honcho_…]`.
## honcho-ledger-damaged
**What you see.** `memoryd` logs a line naming `memory/honcho.jsonl`, a line number and a reason,
then this entry:
**What you see.** `memoryd` exits 1 at start, before it binds its sockets, with one line naming
`memory/honcho.jsonl`, a line number and a reason, then this entry:
```text
/srv/bx/memory/honcho.jsonl:12: turn 3 of session chat-1: decided already; see docs/runbook.md#honcho-ledger-damaged
memoryd: /srv/bx/memory/honcho.jsonl:12: turn 3 of session chat-1: decided already; see docs/runbook.md#honcho-ledger-damaged
```
Or, while it runs, `memoryd: <path>: <error>; see docs/runbook.md#honcho-ledger-damaged` when the
ledger cannot be written (a full disk, a wrong owner). The turn waits undecided, and is decided at
a later turn's end once the ledger can be written.
**Why.** The ledger is `memoryd`'s record of what it sends to Honcho: one line when a turn or a
memory write is decided (where it goes, or that it goes nowhere), and one when it is sent or
refused (M5c spec, section 5). What is decided and not yet sent is the outbox. `memoryd` reads the
ledger through at start so that it sends nothing twice and nothing to a workspace it was not
decided for. A line it cannot read, or one that breaks the ledger's rules, could do either, so it
does not guess: it does not start its Honcho sync, and nothing is decided or sent to Honcho until
the ledger reads through. The rules: an item is decided once (`decided already`), and it is sent
or refused once, only after it is decided and only when it was not decided to go nowhere (`not
waiting to be sent`). A last line cut off by a crash is not damage: `memoryd` drops it and says so
in its log.
does not guess: it does not start. While it is stopped no memory is served either, and `loopd`
cannot start a session (`memory-unavailable`). The rules: an item is decided once (`decided
already`), and it is sent or refused once, only after it is decided and only when it was not
decided to go nowhere (`not waiting to be sent`). A last line cut off by a crash is not damage:
`memoryd` drops it and says so in its log.
**Confirm.** Look at the line named, and at every line of the same item:
@@ -952,12 +958,55 @@ sed -n '<line>p' "$f"
grep -n '"session":"<id>","turn":<n>}' "$f" # a turn; for an edit: '"id":"<id>","part":<n>}'
```
**Fix.** Keep a copy first: `cp -a "$f" "$f.damaged-$(date +%s)"`. The ledger is written only by
`memoryd`, so a damaged line was changed by hand or by the disk. If you know what the line should
say, correct it. Otherwise remove it (`sed -i '<line>d' "$f"`), knowing what that costs: a removed
`sent` or `failed` line means the item is sent again (a turn's messages appear twice in Honcho, a
conclusion is made twice); a removed `decided` line means the item is decided again, and any
later `sent` or `failed` line of the same item then breaks the rules, so remove those too. Honcho
cannot delete a message, so a turn sent twice stays twice.
**Fix.** If the ledger cannot be written, free space or correct the owner of `memory/`
(`df -h`, `ls -l "$f"`); nothing else is needed. For a damaged line, keep a copy first:
`cp -a "$f" "$f.damaged-$(date +%s)"`. The ledger is written only by `memoryd`, so a damaged line
was changed by hand or by the disk. If you know what the line should say, correct it. Otherwise
remove it (`sed -i '<line>d' "$f"`), knowing what that costs: a removed `sent` or `failed` line
means the item is sent again (a turn's messages appear twice in Honcho, a conclusion is made
twice); a removed `decided` line means the item is decided again, and any later `sent` or
`failed` line of the same item then breaks the rules, so remove those too. Honcho cannot delete a
message, so a turn sent twice stays twice.
**Check.** `memoryd` starts, and its log has no line with this entry.
## broker-state-unavailable
**What you see.** `memoryd` logs one line when it cannot get a session's state from `brokerd`,
then this entry:
```text
memoryd: /srv/bx/run/memory-broker/state.sock: Connection refused (os error 111); turns wait undecided; see docs/runbook.md#broker-state-unavailable
```
The reason may instead be `No such file or directory` (nothing listens), a timeout
(`frame I/O error: Resource temporarily unavailable`, after 5 seconds without a byte), or
`brokerd: <its error>` when `brokerd` answered with an error. When `brokerd` answers again,
`memoryd` logs `memoryd: brokerd answers on its state socket again; turns are decided`.
Conversations are not affected.
**Why.** With `[honcho]`, `memoryd` decides where each finished turn goes in Honcho: the trusted
workspace, the untrusted one, or nowhere (M5c spec, section 5). It needs `brokerd`'s record of the
session's class and trust for that, and never guesses it: a guess could put a turn from a session
that read a web page in the trusted workspace. So nothing is decided while `brokerd` does not
answer. The turns wait, in memory, and are decided at the next turn's end of any session once
`brokerd` answers, from the session's state then (which can only have moved towards untrusted).
The outage is logged once, not at every turn. A restart of `memoryd` forgets the waiting turns
until each of their sessions has another turn.
**Confirm.**
```sh
ls -l "$BOXMAKER_HOME/run/memory-broker/state.sock"
grep -n 'state' /etc/boxmaker/memoryd.toml /etc/boxmaker/brokerd.toml
```
`memoryd`'s `[sockets] state` and `brokerd`'s `[sockets] state` must name the same path (both
default to `<home>/run/memory-broker/state.sock`). If `brokerd` answered with an error, its own
log names the problem, usually `broker-state-damaged`.
**Fix.** Start `brokerd` if it is stopped; make the two `[sockets] state` paths agree; or, for
`brokerd`'s error, follow the entry its log names.
**Check.** At the next turn's end `memoryd` logs that `brokerd` answers again, and
`memory/honcho.jsonl` gains a `decided` line for each turn that waited.