boxmaker/docs/specs/2026-10-03-m5d-scheduler.md

# M5d design: epochs, compaction, and the scheduler

Status: approved by the owner, 2026-10-03, with section 11's answers. The implementer is Ornith
(through `tools/run-plan.sh`), as for M5c.

It rests on the brief (`docs/design.md`) as changed by P17 to P24, on M5a's files and index
(`docs/specs/2026-09-24-m5a-local-memory.md`), and on M5c's Honcho ledger (`docs/specs/2026-09-27-m5c-honcho.md`).
Where this document and the brief disagree, section 10 names the brief change.

## 1. What M5d proves

Continuity inside a live session, and work that runs when the owner is not talking, without
breaking the inference contract or the main slot's cache.

The milestone is checked by:

- a scheduled run leaves the main slot's cache intact;
- a fact from epoch 1 is recalled in epoch 2, and a memory edit applied in epoch 1 is in epoch 2's
  baseline;
- the tiering job promotes only trusted notes into identity, and its writes are journalled like
  any other.

## 2. Two halves

One spec, two plans, each run and device-checked on its own (owner):

1. **Epochs and compaction (M5d-1)**: `loopd` writes `N.jsonl` / `N.baseline.json` for `N > 0`;
   turn numbers and `untrusted()` span every epoch; compaction after a turn whose last `Usage`
   is at or above a configured fraction of `expect.n_ctx`; the summary call on the background
   slot; `memoryd`'s Honcho walker reads every epoch file of a session.
2. **The scheduler (M5d-2)**: heartbeat and cron as isolated sessions on the background slot;
   tool-output summaries for tools that opt in; the tiering job.

## 3. Epochs and compaction (M5d-1)

### 3.1 Disk

A session directory already has `0.baseline.json` and `0.jsonl`. After compaction it also has:

```
sessions/<id>/0.jsonl              ends with EpochEnd { next: 1, summary }
sessions/<id>/1.baseline.json      rebuilt from memoryd at compaction
sessions/<id>/1.jsonl              starts with SessionStart { epoch: 1, … }
```

`LogRecord::EpochEnd { time, next, summary }` is already in `bxcore` and unused. M5d-1 writes it.
The old log is kept. `memoryd`'s indexer already walks every `<epoch>.jsonl` under a session
(`memoryd::index::logs`).

`Session::open` loads the highest epoch that has both a baseline file and a log, and the summary
carried from the previous epoch's `EpochEnd` (none for epoch 0). `untrusted()` is true if any
`ToolResult` in any epoch of the session is labelled untrusted. `turns()` counts `User` records
across every epoch, in epoch order then file order; the first `User` of the session is turn 1 for
its life.

### 3.2 Turn numbers and Honcho

`memoryd` keys a turn as `Item::Turn { session, turn }` and today reads only `0.jsonl`
(`honcho::turns::log_path`). After M5d-1 it concatenates every epoch's records in epoch order
and uses the same `turn_records` / `turns` helpers. Because `User` records are never written for
the carried summary, those helpers stay correct.

`loopd` reports `SessionUpdated { turn: session.turns() }` as it does now; that number is
monotonic.

### 3.3 When compaction runs

After a turn has finished (the session is not in the busy set for a live turn), if the current
epoch's last `Usage` has `cache_n + prompt_n` at or above `threshold × expect.n_ctx`, `loopd`
compacts that session before it accepts another turn on it.

Default `threshold` is `0.85`. It is a float in `loopd.toml` (`[compact] threshold`) so it can
move once this agent's own `Usage` records exist. There is no idle timer: a session that sits
for hours under the threshold stays on its current epoch.

If a live request returns `exceed_context_size_error` (`TurnError::SessionFull` today), `loopd`
compacts and retries that user message once in the new epoch. If the retry is still too large,
the turn ends `session_full` as today.

Compaction holds the session busy. Incoming channel turns wait, as they do for any other turn.

### 3.4 The summary call

On the **background** slot, with **no tools**, thinking on. The messages are the current epoch's
replay (`messages(baseline, records)`), then one user line whose text is `config/compact.md`
(next to `system.md`; a starter ships in the repository). The completion's `content` is the
summary: tag-neutralised, capped to `[compact] summary_chars` (default 4,000 characters). It is
not stuffed into the baseline, so it does not count against the 3,000-token budget.

The session's live slot (the one on `SessionStart`, main for a Mattermost thread) is unchanged.
The next live turn of the new epoch is a cold start on that slot: new prefix.

If the summary call itself is refused as context-full, `loopd` writes a fallback summary: the
epoch's `User` texts, newest last, cut to `summary_chars`. Compaction still proceeds.

The large prompt shares the server's unified KV pool. That pressure is accepted; running it on
the background slot keeps it off a live main-slot request.

### 3.5 The new epoch's prefix

1. Append `EpochEnd` to the old log.
2. Ask `memoryd` for a fresh memory block; write `<n+1>.baseline.json` (system.md + block + the
   same core tool schemas). Identity edits applied during the old epoch land here.
3. Create `<n+1>.jsonl` with `SessionStart { epoch: n+1, slot: the session's live slot, baseline:
   hash of the new file }`.
4. The in-memory `Session` switches to the new baseline, empty records (the `SessionStart`), and
   `carry = Some(summary)`.

`messages` for epoch `n > 0` is: the new system message, then one `ChatMessage::User` whose
content is a fixed heading plus `carry` (the heading is built from pieces in code, named in
words), then the new epoch's records. That user message is **not** a `LogRecord::User`, so it is
not a turn, not reported to Honcho, and not counted by `turns()`.

Within an epoch the request is still a strict extension of the previous one. A new epoch is a
new prefix by definition.

### 3.6 Trust across compaction

`Session::untrusted()` reads every epoch file, so a compacted session that had read untrusted
content still does. `TurnDone.untrusted` follows that, and `gatewayd` keeps escaping links.
`brokerd`'s `broker/sessions/<id>.json` is already per session, not per epoch, and stays the
authority for taint on tool calls.

## 4. Isolated sessions and the scheduler (M5d-2)

Scheduled work is a session like any other: same core tools, same `brokerd` path, same memory
tools. It runs on the **background** slot (`SessionStart.slot = cfg.slots.background`). Live
Mattermost and `bxctl chat` sessions keep `cfg.slots.main`. `run_turn` uses the session's slot,
not always `slots.main` (today's `turn.rs`).

The scheduler is a thread inside `loopd serve` (the only role that can start inference). It
shares `Context` and the busy set. It never talks to `gatewayd`. Answers stay in
`sessions/<id>/`. The owner reads them with `bxctl chat --session <id>` or by opening the log.

Session ids:

| Kind | Id | Lives |
|---|---|---|
| Heartbeat | `heartbeat` | one standing session, compacted like any other |
| A cron run | `job-<name>-<yyyymmdd>t<hhmm>` (`[a-z0-9-]`) | one session per run |
| Mattermost | `mm-<root>` as today | unchanged |

A heartbeat or cron turn has no speaker, so Honcho skips it (M5c: no single speaker). The local
index still holds the log.

Priority, when the background slot is contended: a compaction summary first, then a cron run
that is due, then heartbeat, then a tool-output summary. A live turn on the main slot is
independent. `SlotGate` already serialises requests on one slot.

### 4.1 Heartbeat

`[heartbeat]` in `loopd.toml`: `every_s` (0, the default, disables it) and `prompt` (a path next
to the config file, default `heartbeat.md`). A starter `config/heartbeat.md` ships in the
repository.

When `every_s` has elapsed since the last heartbeat turn (or since start, if none), the
scheduler runs one turn on session `heartbeat` whose user text is the file's contents. Volatile
time belongs in that newest message: the scheduler prepends one line `It is <RFC 3339 UTC>.`
ahead of the file.

### 4.2 Cron

Owner-written files `{home}/jobs/<name>.toml`. `name` is `[a-z0-9-]+`, short enough that
`job-<name>-<yyyymmdd>t<hhmm>` fits in a `SessionId` (64 bytes). Unknown keys are errors.
`loopd` reads them and cannot write them.

```toml
schedule = "0 7 * * *"   # five-field UTC crontab: minute hour day-of-month month day-of-week
prompt = "Check overnight mail and write a note of anything that needs the owner."
```

`prompt` is the user text of that run, with the same time line prepended as heartbeat. A new
session each run. If a previous run of the same `name` is still busy, this tick is skipped (the
schedule does not queue).

The matcher is a small module in `loopd` (UTC, no cron crate): `*`, a single number, `n-m`,
`*/n`. Day-of-week 0 is Sunday. Tested against a table of (schedule, timestamp, due) cases.
`loopd` wakes at least once a minute while any job file exists.

A run ledger `{home}/jobs/runs.jsonl` records `{time, name, session, status}` (`started`,
`finished`, `failed`). It is how "previous run still busy" is answered after a restart: a
`started` line with no later `finished`/`failed` for that `name` means skip until the process
that owns the busy set is this one and the session is not busy. On start, a leftover `started`
from a dead process is closed as `failed`.

Scheduled jobs re-check current grants on every tool call: that is already `brokerd`'s rule.

### 4.3 Tool-output summaries

M5a's index records owner text, assistant replies, and tool **calls**; it does not record tool
**results**. A fetch's body is therefore invisible to `recall` except as whatever the assistant
restated.

A tool that opts in gets a second record, for the index only:

```text
LogRecord::ToolSummary { time, call, tool_call_id, content, class, untrusted }
```

`messages()` ignores it, so the epoch's prefix does not change. `class` and `untrusted` are
copied from the `ToolResult` they summarise. `memoryd` indexes `content` as kind `summary`, same
labels; `recall` tags it `memory` or `memory-tainted` as for any other session hit.

M5d opts in `http_fetch` only. `[compact] summarise = ["http_fetch"]` is the list; unknown names
are a config error.

After a turn that logged an opted-in `ToolResult` with no matching `ToolSummary` yet, the
scheduler runs a no-tools completion on the background slot: the user text is
`config/summarise.md` plus the result, capped to the tool-result cap. The completion is
tag-neutralised, capped to `summary_chars`, and appended as `ToolSummary`. The model call does
not hold the session busy; appending the record does (the same log lock as any append). A live
turn on that session is never delayed by summarising. Compaction of a session waits until
in-flight summaries for it have appended or failed.

### 4.4 The tiering job

An ordinary cron job. Nothing in `loopd` special-cases it. A starter `{repo}/config/jobs/tier.toml.example`
documents a weekly schedule and a prompt that says: read the listing, then `consolidate` /
`add` / `remove` / `note` as needed; promote only trusted notes; do not fetch.

The listing is the job's user text after the time line, from a new `memoryd` admin request
`MemoryCommand::Tiers`: each identity entry and each note, with how many times `recalls/` named
it and when it was last found (M5a already writes that log so counts survive a reindex).
Untrusted notes are included and marked; the prompt forbids promoting them, and `memoryd` still
refuses identity edits from an untrusted session, so a job that fetches a page cannot promote.

`bxctl memory tiers` prints the same listing for the owner.

## 5. Config

New tables on `loopd.toml`. Unknown keys are errors. Defaults disable scheduled work.

```toml
[compact]
threshold = 0.85
summary_chars = 4000
summarise = ["http_fetch"]

[heartbeat]
every_s = 0
prompt = "heartbeat.md"    # next to this config file
```

`expect.n_ctx` is already required. Depth is `cache_n + prompt_n` of the current epoch's last
`Usage`.

Starter files in the repository, copied like `soul.md` / `system.md`: `config/compact.md`,
`config/heartbeat.md`, `config/summarise.md`, `config/jobs/tier.toml.example`.

## 6. Components

| Piece | Change |
|---|---|
| `bxcore` | `LogRecord::ToolSummary`; `MemoryCommand::Tiers` and its reply. `EpochEnd` stays as it is. |
| `loopd` | Current epoch on open; `turns` / `untrusted` over every epoch; `run_turn` uses the session's slot; compaction (section 3); scheduler thread (section 4). |
| `memoryd` | Honcho reads every epoch file of a session; indexes `ToolSummary`; `Tiers` from `recalls/`. |
| `gatewayd` | None. `TurnDone.untrusted` keeps doing the work if `loopd` is right. |
| `brokerd` | None beyond using the session's existing state. |
| `bxctl` | `bxctl memory tiers`. Heartbeat and job sessions are ordinary `--session` ids. |

`loopd` still depends only on `bxcore`. No new crate for cron.

## 7. Testing

Given tests, first, against the fake llama server and a fake `memoryd`:

- **Open / resume**: a directory with `0` and `1` loads epoch 1 and the carry from `0`'s
  `EpochEnd`; a missing carry on epoch 1 is an error (the file exists but has no `EpochEnd`).
- **Turns**: two `User` records in epoch 0 and one in epoch 1 → `turns() == 3`; Honcho
  `turn_records` of turn 3 is the epoch-1 user.
- **Trust**: an untrusted `ToolResult` in epoch 0, none in epoch 1 → `untrusted()` is true
  (the open item in `docs/decisions.md`).
- **Prefix**: property test per epoch; after compaction the first request of epoch 1 is the new
  baseline plus carry, not an extension of epoch 0.
- **Trigger**: last `Usage` at 84% of `n_ctx` leaves the epoch; at 85% the next turn waits on
  compaction (the fake server records a no-tools request on the background slot) and then runs
  on epoch 1. An idle gap with no turn does nothing.
- **SessionFull**: a context-full error on epoch 0 compacts and retries once.
- **Slot**: a `heartbeat` turn's request carries `id_slot = background`; a `mm-*` turn still
  carries `main`. A heartbeat turn in between two main-slot turns of another session leaves the
  fake server's recorded cache prefix of the main session intact.
- **Heartbeat / cron**: `every_s = 1` fires; `schedule = "* * * * *"` fires; a still-busy run of
  the same name is skipped; a leftover `started` after a restart is `failed`; unknown keys in a
  job file refuse start; a due run with no speaker is not a Honcho item (existing skip).
- **ToolSummary**: an `http_fetch` result produces a `ToolSummary` with the same labels;
  `messages()` is unchanged; `recall` finds the summary text and taints when `untrusted`.
- **Tiers**: a trusted note with hits is listed; an untrusted note is listed and marked; a
  tiering session that is untrusted cannot `add` to `memory.md` (existing M5a rule).

**On the device** (straylight, Ornith slot 0 for the live session, slot 2 for background), with
`[compact] threshold` lowered so the check does not need 220k tokens:

- M5d-1: a `bxctl` session writes a distinctive fact into `memory.md`, continues until
  compaction, and the next turn's baseline contains the fact; `recall` of a phrase from epoch 0
  still hits; `TurnDone.untrusted` stays true across the epoch change after a tainted recall.
- M5d-2: a heartbeat turn runs on the background slot; a following main-slot turn of a second
  session shows a cache hit on its own prefix (`cache_n` near its previous total). A
  `jobs/tier.toml` for the check promotes only a trusted note; the journal has the edit; an
  untrusted note is still a note.

Heartbeat and cron answers are not posted to Mattermost. The check reads the session log.

## 8. Runbook

New entries, each with a pointer from the message:

- `session-compact-failed`: the summary call and the fallback both failed; the session stays on
  the old epoch and the next turn is `session_full` if it would not fit.
- `jobs-invalid`: a job file failed to load; `loopd` does not start.
- `heartbeat-prompt-unreadable`: the prompt file exists but cannot be read; heartbeat is skipped
  and the line is logged once until it reads.

## 9. Out of scope

- Subagents (M6) share the background slot; the scheduler's queue is the one they will join.
- Optional delivery of a cron answer to Mattermost.
- Session-scoped approvals (open row in `docs/decisions.md`).
- Changing `Item::Turn` to carry an epoch: turn numbers are monotonic instead.
- A cron crate; nicknames (`@daily`); seconds fields; non-UTC zones.

## 10. Changes to the brief and decisions

With this spec, as their own commits:

- `docs/design.md`, Inference contract 4: compaction runs after a turn, when
  `cache_n + prompt_n` is at or above a configured fraction of `n_ctx` (default 0.85), while the
  session is not in another turn. It does not run because the session has been quiet.
- `docs/design.md`, Memory 3 / M5a index: `recall` also indexes `ToolSummary` records of tools
  that opt in, labelled as the tool result was. Raw tool results stay unindexed.
- `docs/decisions.md`: close "A session's trust across compaction" (section 3.6); row for
  monotonic turn numbers; row for scheduled sessions staying off Mattermost; P20's "M5c
  scheduler" wording already moved to M5d by P24.

## 11. The owner's answers (2026-10-03)

1. **Heartbeat delivery**: answers stay in the session log. They are not posted to Mattermost.
2. **Cron**: five-field UTC crontab, `{home}/jobs/<name>.toml`, a new session each run, a tick
   skipped when that job is still live.
3. **Compaction trigger**: 85% of `expect.n_ctx` from the last `Usage`, configurable. Retune
   from this agent's `Usage` records once they exist. A quiet session is not a reason to
   compact (the owner may have gone to sleep).
4. **Tool summaries**: on the background slot; a `ToolSummary` log record with the result's
   labels, FTS-indexed; `http_fetch` opts in.
5. **Split**: M5d-1 (epochs and compaction), then M5d-2 (the scheduler), two plans.