boxmaker/docs/specs/2026-10-03-m5d-scheduler.md

M5d design: epochs, compaction, and the scheduler

Status: approved by the owner, 2026-10-03, with section 11's answers. The implementer is Ornith (through tools/run-plan.sh), as for M5c.

It rests on the brief (docs/design.md) as changed by P17 to P24, on M5a's files and index (docs/specs/2026-09-24-m5a-local-memory.md), and on M5c's Honcho ledger (docs/specs/2026-09-27-m5c-honcho.md). Where this document and the brief disagree, section 10 names the brief change.

1. What M5d proves

Continuity inside a live session, and work that runs when the owner is not talking, without breaking the inference contract or the main slot's cache.

The milestone is checked by:

• a scheduled run leaves the main slot's cache intact;
• a fact from epoch 1 is recalled in epoch 2, and a memory edit applied in epoch 1 is in epoch 2's baseline;
• the tiering job promotes only trusted notes into identity, and its writes are journalled like any other.

2. Two halves

One spec, two plans, each run and device-checked on its own (owner):

1. Epochs and compaction (M5d-1): loopd writes N.jsonl / N.baseline.json for N > 0; turn numbers and untrusted() span every epoch; compaction after a turn whose last Usage is at or above a configured fraction of expect.n_ctx; the summary call on the background slot; memoryd's Honcho walker reads every epoch file of a session.
2. The scheduler (M5d-2): heartbeat and cron as isolated sessions on the background slot; tool-output summaries for tools that opt in; the tiering job.

3. Epochs and compaction (M5d-1)

3.1 Disk

A session directory already has 0.baseline.json and 0.jsonl. After compaction it also has:

sessions/<id>/0.jsonl              ends with EpochEnd { next: 1, summary }
sessions/<id>/1.baseline.json      rebuilt from memoryd at compaction
sessions/<id>/1.jsonl              starts with SessionStart { epoch: 1, … }

LogRecord::EpochEnd { time, next, summary } is already in bxcore and unused. M5d-1 writes it. The old log is kept. memoryd's indexer already walks every <epoch>.jsonl under a session (memoryd::index::logs).

Session::open loads the highest epoch that has both a baseline file and a log, and the summary carried from the previous epoch's EpochEnd (none for epoch 0). untrusted() is true if any ToolResult in any epoch of the session is labelled untrusted. turns() counts User records across every epoch, in epoch order then file order; the first User of the session is turn 1 for its life.

3.2 Turn numbers and Honcho

memoryd keys a turn as Item::Turn { session, turn } and today reads only 0.jsonl (honcho::turns::log_path). After M5d-1 it concatenates every epoch's records in epoch order and uses the same turn_records / turns helpers. Because User records are never written for the carried summary, those helpers stay correct.

loopd reports SessionUpdated { turn: session.turns() } as it does now; that number is monotonic.

3.3 When compaction runs

After a turn has finished (the session is not in the busy set for a live turn), if the current epoch's last Usage has cache_n + prompt_n at or above threshold × expect.n_ctx, loopd compacts that session before it accepts another turn on it.

Default threshold is 0.85. It is a float in loopd.toml ([compact] threshold) so it can move once this agent's own Usage records exist. There is no idle timer: a session that sits for hours under the threshold stays on its current epoch.

If a live request returns exceed_context_size_error (TurnError::SessionFull today), loopd compacts and retries that user message once in the new epoch. If the retry is still too large, the turn ends session_full as today.

Compaction holds the session busy. Incoming channel turns wait, as they do for any other turn.

3.4 The summary call

On the background slot, with no tools, thinking on. The messages are the current epoch's replay (messages(baseline, records)), then one user line whose text is config/compact.md (next to system.md; a starter ships in the repository). The completion's content is the summary: tag-neutralised, capped to [compact] summary_chars (default 4,000 characters). It is not stuffed into the baseline, so it does not count against the 3,000-token budget.

The session's live slot (the one on SessionStart, main for a Mattermost thread) is unchanged. The next live turn of the new epoch is a cold start on that slot: new prefix.

If the summary call itself is refused as context-full, loopd writes a fallback summary: the epoch's User texts, newest last, cut to summary_chars. Compaction still proceeds.

The large prompt shares the server's unified KV pool. That pressure is accepted; running it on the background slot keeps it off a live main-slot request.

3.5 The new epoch's prefix

1. Append EpochEnd to the old log.
2. Ask memoryd for a fresh memory block; write <n+1>.baseline.json (system.md + block + the same core tool schemas). Identity edits applied during the old epoch land here.
3. Create <n+1>.jsonl with SessionStart { epoch: n+1, slot: the session's live slot, baseline: hash of the new file }.
4. The in-memory Session switches to the new baseline, empty records (the SessionStart), and carry = Some(summary).

messages for epoch n > 0 is: the new system message, then one ChatMessage::User whose content is a fixed heading plus carry (the heading is built from pieces in code, named in words), then the new epoch's records. That user message is not a LogRecord::User, so it is not a turn, not reported to Honcho, and not counted by turns().

Within an epoch the request is still a strict extension of the previous one. A new epoch is a new prefix by definition.

3.6 Trust across compaction

Session::untrusted() reads every epoch file, so a compacted session that had read untrusted content still does. TurnDone.untrusted follows that, and gatewayd keeps escaping links. brokerd's broker/sessions/<id>.json is already per session, not per epoch, and stays the authority for taint on tool calls.

4. Isolated sessions and the scheduler (M5d-2)

Scheduled work is a session like any other: same core tools, same brokerd path, same memory tools. It runs on the background slot (SessionStart.slot = cfg.slots.background). Live Mattermost and bxctl chat sessions keep cfg.slots.main. run_turn uses the session's slot, not always slots.main (today's turn.rs).

The scheduler is a thread inside loopd serve (the only role that can start inference). It shares Context and the busy set. It never talks to gatewayd. Answers stay in sessions/<id>/. The owner reads them with bxctl chat --session <id> or by opening the log.

Session ids:

Kind	Id	Lives
Heartbeat	heartbeat	one standing session, compacted like any other
A cron run	job-<name>-<yyyymmdd>t<hhmm> ([a-z0-9-])	one session per run
Mattermost	mm-<root> as today	unchanged

A heartbeat or cron turn has no speaker, so Honcho skips it (M5c: no single speaker). The local index still holds the log.

Priority, when the background slot is contended: a compaction summary first, then a cron run that is due, then heartbeat, then a tool-output summary. A live turn on the main slot is independent. SlotGate already serialises requests on one slot.

4.1 Heartbeat

[heartbeat] in loopd.toml: every_s (0, the default, disables it) and prompt (a path next to the config file, default heartbeat.md). A starter config/heartbeat.md ships in the repository.

When every_s has elapsed since the last heartbeat turn (or since start, if none), the scheduler runs one turn on session heartbeat whose user text is the file's contents. Volatile time belongs in that newest message: the scheduler prepends one line It is <RFC 3339 UTC>. ahead of the file.

4.2 Cron

Owner-written files {home}/jobs/<name>.toml. name is [a-z0-9-]+, short enough that job-<name>-<yyyymmdd>t<hhmm> fits in a SessionId (64 bytes). Unknown keys are errors. loopd reads them and cannot write them.

schedule = "0 7 * * *"   # five-field UTC crontab: minute hour day-of-month month day-of-week
prompt = "Check overnight mail and write a note of anything that needs the owner."

prompt is the user text of that run, with the same time line prepended as heartbeat. A new session each run. If a previous run of the same name is still busy, this tick is skipped (the schedule does not queue).

The matcher is a small module in loopd (UTC, no cron crate): *, a single number, n-m, */n. Day-of-week 0 is Sunday. Tested against a table of (schedule, timestamp, due) cases. loopd wakes at least once a minute while any job file exists.

A run ledger {home}/jobs/runs.jsonl records {time, name, session, status} (started, finished, failed). It is how "previous run still busy" is answered after a restart: a started line with no later finished/failed for that name means skip until the process that owns the busy set is this one and the session is not busy. On start, a leftover started from a dead process is closed as failed.

Scheduled jobs re-check current grants on every tool call: that is already brokerd's rule.

4.3 Tool-output summaries

M5a's index records owner text, assistant replies, and tool calls; it does not record tool results. A fetch's body is therefore invisible to recall except as whatever the assistant restated.

A tool that opts in gets a second record, for the index only:

LogRecord::ToolSummary { time, call, tool_call_id, content, class, untrusted }

messages() ignores it, so the epoch's prefix does not change. class and untrusted are copied from the ToolResult they summarise. memoryd indexes content as kind summary, same labels; recall tags it memory or memory-tainted as for any other session hit.

M5d opts in http_fetch only. [compact] summarise = ["http_fetch"] is the list; unknown names are a config error.

After a turn that logged an opted-in ToolResult with no matching ToolSummary yet, the scheduler runs a no-tools completion on the background slot: the user text is config/summarise.md plus the result, capped to the tool-result cap. The completion is tag-neutralised, capped to summary_chars, and appended as ToolSummary. The model call does not hold the session busy; appending the record does (the same log lock as any append). A live turn on that session is never delayed by summarising. Compaction of a session waits until in-flight summaries for it have appended or failed.

4.4 The tiering job

An ordinary cron job. Nothing in loopd special-cases it. A starter {repo}/config/jobs/tier.toml.example documents a weekly schedule and a prompt that says: read the listing, then consolidate / add / remove / note as needed; promote only trusted notes; do not fetch.

The listing is the job's user text after the time line, from a new memoryd admin request MemoryCommand::Tiers: each identity entry and each note, with how many times recalls/ named it and when it was last found (M5a already writes that log so counts survive a reindex). Untrusted notes are included and marked; the prompt forbids promoting them, and memoryd still refuses identity edits from an untrusted session, so a job that fetches a page cannot promote.

bxctl memory tiers prints the same listing for the owner.

5. Config

New tables on loopd.toml. Unknown keys are errors. Defaults disable scheduled work.

[compact]
threshold = 0.85
summary_chars = 4000
summarise = ["http_fetch"]

[heartbeat]
every_s = 0
prompt = "heartbeat.md"    # next to this config file

expect.n_ctx is already required. Depth is cache_n + prompt_n of the current epoch's last Usage.

Starter files in the repository, copied like soul.md / system.md: config/compact.md, config/heartbeat.md, config/summarise.md, config/jobs/tier.toml.example.

6. Components

Piece	Change
bxcore	LogRecord::ToolSummary; MemoryCommand::Tiers and its reply. EpochEnd stays as it is.
loopd	Current epoch on open; turns / untrusted over every epoch; run_turn uses the session's slot; compaction (section 3); scheduler thread (section 4).
memoryd	Honcho reads every epoch file of a session; indexes ToolSummary; Tiers from recalls/.
gatewayd	None. TurnDone.untrusted keeps doing the work if loopd is right.
brokerd	None beyond using the session's existing state.
bxctl	bxctl memory tiers. Heartbeat and job sessions are ordinary --session ids.

loopd still depends only on bxcore. No new crate for cron.

7. Testing

Given tests, first, against the fake llama server and a fake memoryd:

• Open / resume: a directory with 0 and 1 loads epoch 1 and the carry from 0's EpochEnd; a missing carry on epoch 1 is an error (the file exists but has no EpochEnd).
• Turns: two User records in epoch 0 and one in epoch 1 → turns() == 3; Honcho turn_records of turn 3 is the epoch-1 user.
• Trust: an untrusted ToolResult in epoch 0, none in epoch 1 → untrusted() is true (the open item in docs/decisions.md).
• Prefix: property test per epoch; after compaction the first request of epoch 1 is the new baseline plus carry, not an extension of epoch 0.
• Trigger: last Usage at 84% of n_ctx leaves the epoch; at 85% the next turn waits on compaction (the fake server records a no-tools request on the background slot) and then runs on epoch 1. An idle gap with no turn does nothing.
• SessionFull: a context-full error on epoch 0 compacts and retries once.
• Slot: a heartbeat turn's request carries id_slot = background; a mm-* turn still carries main. A heartbeat turn in between two main-slot turns of another session leaves the fake server's recorded cache prefix of the main session intact.
• Heartbeat / cron: every_s = 1 fires; schedule = "* * * * *" fires; a still-busy run of the same name is skipped; a leftover started after a restart is failed; unknown keys in a job file refuse start; a due run with no speaker is not a Honcho item (existing skip).
• ToolSummary: an http_fetch result produces a ToolSummary with the same labels; messages() is unchanged; recall finds the summary text and taints when untrusted.
• Tiers: a trusted note with hits is listed; an untrusted note is listed and marked; a tiering session that is untrusted cannot add to memory.md (existing M5a rule).

On the device (straylight, Ornith slot 0 for the live session, slot 2 for background), with [compact] threshold lowered so the check does not need 220k tokens:

• M5d-1: a bxctl session writes a distinctive fact into memory.md, continues until compaction, and the next turn's baseline contains the fact; recall of a phrase from epoch 0 still hits; TurnDone.untrusted stays true across the epoch change after a tainted recall.
• M5d-2: a heartbeat turn runs on the background slot; a following main-slot turn of a second session shows a cache hit on its own prefix (cache_n near its previous total). A jobs/tier.toml for the check promotes only a trusted note; the journal has the edit; an untrusted note is still a note.

Heartbeat and cron answers are not posted to Mattermost. The check reads the session log.

8. Runbook

New entries, each with a pointer from the message:

• session-compact-failed: the summary call and the fallback both failed; the session stays on the old epoch and the next turn is session_full if it would not fit.
• jobs-invalid: a job file failed to load; loopd does not start.
• heartbeat-prompt-unreadable: the prompt file exists but cannot be read; heartbeat is skipped and the line is logged once until it reads.

9. Out of scope

• Subagents (M6) share the background slot; the scheduler's queue is the one they will join.
• Optional delivery of a cron answer to Mattermost.
• Session-scoped approvals (open row in docs/decisions.md).
• Changing Item::Turn to carry an epoch: turn numbers are monotonic instead.
• A cron crate; nicknames (@daily); seconds fields; non-UTC zones.

10. Changes to the brief and decisions

With this spec, as their own commits:

• docs/design.md, Inference contract 4: compaction runs after a turn, when cache_n + prompt_n is at or above a configured fraction of n_ctx (default 0.85), while the session is not in another turn. It does not run because the session has been quiet.
• docs/design.md, Memory 3 / M5a index: recall also indexes ToolSummary records of tools that opt in, labelled as the tool result was. Raw tool results stay unindexed.
• docs/decisions.md: close "A session's trust across compaction" (section 3.6); row for monotonic turn numbers; row for scheduled sessions staying off Mattermost; P20's "M5c scheduler" wording already moved to M5d by P24.

11. The owner's answers (2026-10-03)

1. Heartbeat delivery: answers stay in the session log. They are not posted to Mattermost.
2. Cron: five-field UTC crontab, {home}/jobs/<name>.toml, a new session each run, a tick skipped when that job is still live.
3. Compaction trigger: 85% of expect.n_ctx from the last Usage, configurable. Retune from this agent's Usage records once they exist. A quiet session is not a reason to compact (the owner may have gone to sleep).
4. Tool summaries: on the background slot; a ToolSummary log record with the result's labels, FTS-indexed; http_fetch opts in.
5. Split: M5d-1 (epochs and compaction), then M5d-2 (the scheduler), two plans.