From bbe5697193b81c547ab28c3a37d2953e7ab69605 Mon Sep 17 00:00:00 2001 From: Kyle Isom Date: Sun, 4 Oct 2026 03:17:03 -0700 Subject: [PATCH] docs: M7 split-deployment spec draft --- CLAUDE.md | 3 +- docs/specs/2026-10-04-m7-split-deployment.md | 202 +++++++++++++++++++ 2 files changed, 204 insertions(+), 1 deletion(-) create mode 100644 docs/specs/2026-10-04-m7-split-deployment.md diff --git a/CLAUDE.md b/CLAUDE.md index c268128..57f876d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -48,7 +48,8 @@ device-checked on straylight (child `sub-m6a-live-2` on slot 2; following slot-0 `docs/plans/M6b/`) is done, reviewed in `docs/implementer-log.md`; no finding is open; spec 7 device-checked on straylight (clean consult audit body matched the captured loopback POST; a private untrusted recall still consulted under `max_taint = "private"`, matching spec 4.1's -class gate). +class gate). M6 is on `origin/master` (`3063c3e`). M7 (split deployment) has a draft spec +`docs/specs/2026-10-04-m7-split-deployment.md` awaiting the owner's answers; no plan yet. P22 (proposed, not applied) makes Boxmaker a fleet of independent agents, one per host, deployed with Nix in a last milestone, M8; `flake.nix` builds every binary, the tool image and a dev shell. Work happens on titan (macOS, no Nix); build and run diff --git a/docs/specs/2026-10-04-m7-split-deployment.md b/docs/specs/2026-10-04-m7-split-deployment.md new file mode 100644 index 0000000..ab6e285 --- /dev/null +++ b/docs/specs/2026-10-04-m7-split-deployment.md @@ -0,0 +1,202 @@ +# M7 design: split deployment + +Status: draft, 2026-10-04, awaiting the owner's answers in section 11. The implementer is Ornith +(through `tools/run-plan.sh`), as for M6. No plan is written until this spec is approved. + +It rests on the brief (`docs/design.md`, Network isolation, Roles, Authority 4, Target +environment), on the 2026-09-18 decisions (rootless Podman, separate users per role, `broker.sock` +and `admin.sock` enforced with mounts in M7), and on what already exists: `inferproxy`, the M3b +tool image and egress proxy, `flake.nix` packages and `deploy/tools-image.nix`. P22 / M8 (a fleet +module and `make install`) stays out of this milestone. + +Where this document and the brief disagree, section 10 names the brief change. There is none yet. + +## 1. What M7 proves + +Each role already has a narrow binary. Until now they all run as the owner's user on the host. +M7 puts each role in its own rootless container so a compromise of one process is contained by +namespaces, mounts and users, not only by code. + +The milestone is checked by: + +- from inside the `loopd` container, requests to the internet, the tailnet and the host all fail +- a tool container without a host grant cannot reach the tailnet (already true of M3b; repeated + from the new layout) +- `loopd` cannot open `admin.sock`; `bxctl` on the host still can + +## 2. What already holds + +| Piece | Today | +|---|---| +| Tool containers | Fresh rootless Podman per call, image from Nix, named by digest, `--network=none` unless the grant names hosts, egress proxy for `http_fetch` | +| `inferproxy` | Unix socket in, one TCP upstream to the host `llama-server` | +| `loopd` | No credentials, no network in process terms; it still runs with the host network namespace | +| Sockets | Pair directories under `$HOME/run/`; both ends are the same Unix user, so the split is code (`forbidden`) | +| `flake.nix` | Builds every binary and the tool image; no NixOS module (P22 / M8) | +| Production on straylight | Host processes, owner's user, Mattermost bot untouched by this spec's device check | + +M7 does not rewrite the IPC, grants, audit, or inference contract. It changes *where* the +binaries run and *who* can see which sockets and files. + +## 3. Isolation model + +One long-lived container per role: `loopd`, `brokerd`, `gatewayd`, `memoryd`, `inferproxy`. +Tool containers stay as they are (one shot, from `brokerd`). `bxctl` stays on the host: it is +the owner's CLI, not a role. + +Each role container: + +- runs as a dedicated user on the host (the 2026-09-18 decision: separate users per role) +- is started rootless Podman (`--userns=keep-id` so the role user owns its files) +- is `--read-only` with tmpfs for `/tmp` and the lock/socket directories it must write +- drops capabilities (`--cap-drop=all`, `no-new-privileges`) +- mounts only the directories that role may see, with the mode that role needs + +Network per role (brief, Network isolation): + +| Role | Network | +|---|---| +| `loopd` | `--network=none`. Loopback only. Four Unix sockets. | +| `inferproxy` | Outbound to the configured `llama-server` host:port and nothing else | +| `gatewayd` | Outbound to Mattermost. No listening TCP port | +| `memoryd` | Outbound to Honcho and nothing else | +| `brokerd` | No outbound internet in M7. It talks to the container runtime, to its sockets, and (for `consult`) to a loopback URL already required to be `127.0.0.1` or `localhost`. How loopback `consult` is reached from inside the container is question 4 | +| tool containers | Unchanged: none, or the per-call egress proxy | + +## 4. Sockets and mounts + +Peer authentication stays directory permissions, now across users. + +Each pair of roles shares one directory that only those two users can enter. The listening +socket lives there, mode `0600`, owned by the listener. The other role's container mounts that +directory. A third role does not get the directory. + +The pairs that already exist under `run/`: + +| Directory | Listener | Client | M7 extra | +|---|---|---|---| +| `run/loop/` | `loopd` (`loop.sock`) | `gatewayd`, `bxctl` (host) | `bxctl` is in the owner's group or ACL for this dir | +| `run/loop-broker/` | `brokerd` (`broker.sock`) | `loopd` | `loopd` does **not** mount `run/owner-broker/` | +| `run/owner-broker/` | `brokerd` (`admin.sock`) | `bxctl` (host) | not mounted into `loopd` | +| `run/infer/` | `inferproxy` | `loopd` | | +| `run/loop-memory/` | `memoryd` | `loopd` | | +| `run/broker-memory/` | `memoryd` | `brokerd` | | +| `run/memory-broker/` | `brokerd` (`state.sock`) | `memoryd` | | +| `run/owner-memory/` | `memoryd` (`admin.sock`) | `bxctl` (host) | not mounted into `loopd` | +| `run/gateway/` | `brokerd` (`gateway.sock`), when `[gateway]` is set | `gatewayd` | | + +Files: + +| Tree | Who writes | Who reads | +|---|---|---| +| `sessions/` | `loopd` | `memoryd` (reindex, Honcho send), `bxctl` | +| `memory/` | `memoryd` | `loopd` (baseline copies at epoch start, through the socket, as today) | +| `grants/` | the owner, from the host | `brokerd` (read-only) | +| `audit/` | `brokerd` | `bxctl audit verify` on the host | +| `broker/sessions/` | `brokerd` | `brokerd` | +| config `*.toml`, `system.md`, `soul.md`, `heartbeat.md`, `compact.md` | the owner | the role that loads them | + +`loopd` never sees `grants/`, `audit/`, or `admin.sock`. That is the mount half of the 2026-09-18 +socket split. + +## 5. Images and how they start + +`flake.nix` already builds one package of every binary. M7 adds, under `deploy/`: + +- a rootless image per role **or** one image holding all role binaries (question 2), still + built by Nix, loaded with `podman load`, named by digest +- a host ACL file for the tailnet node (question 5): `deploy/acl.hujson` or the owner's + existing Tailscale ACL repo, copied or pointed at from here so the gate can see it +- unit files or a small supervisor script that starts the five role containers with the + mounts in section 4 (question 3) + +No NixOS module in M7 (P22 / M8). No `make install` for Omarchy in M7. + +Secrets stay `bxcore::secrets`: systemd credential, env, or owner-only file. Inside a +container the credential backend needs the host to pass the credential in; the env and file +backends already work if the file is mounted mode `0600`. Default remains credential. + +One instance per daemon still holds: the existing lock files live on the mounted `run/` dirs. + +## 6. Components + +| Piece | Change | +|---|---| +| `loopd` | none required in Rust if `--network=none` plus mounts are enough. A start failure when a required socket dir is missing already has runbook entries; add any new pointer M7 creates | +| `brokerd` | Podman is invoked from inside the `brokerd` container to start tool containers (nested rootless, question 1). `consult` stays loopback HTTP/1.1 | +| `gatewayd` | none in Rust | +| `memoryd` | none in Rust | +| `inferproxy` | none in Rust; its upstream is still host `llama-server` | +| `bxctl` | stays on the host; needs group/ACL on `loop.sock` and `admin.sock` dirs | +| `bxcore` | none | +| `deploy/` | role image(s), start units or script, tailnet ACL policy, example mounts | +| `docs/egress.md` | no new run-time destination. `consult` is still loopback until a later live URL | + +## 7. Testing + +Given tests first, against a fake `podman` / a recorded argument list where the start script is +code, and against permission checks on a scratch tree of `run/` dirs: + +- **loopd has no net:** a helper in the `loopd` image (or `busybox wget`) to `1.1.1.1:443`, to + the tailnet Mattermost host, and to the host's `llama-server` port, all fail. The four Unix + sockets still work. +- **admin.sock is not in the loopd mount:** opening it from the `loopd` container is + `ENOENT` or `EACCES`. `bxctl approvals` from the host still lists. +- **broker.sock is in the loopd mount:** a tool request still reaches `brokerd`. +- **Tool without hosts:** `shell` or `http_fetch` with no host grant cannot reach the tailnet + (repeat of M3b from the new `brokerd` container). +- **inferproxy:** `loopd` still completes a self-test through the socket. + +**On the device** (straylight, private `$HOME`, production daemons untouched, slot 1 left busy): + +- start the five role containers from `deploy/` +- from inside `loopd`, the three negative probes in the first bullet fail +- `bxctl chat --session m7-live --no-thinking` completes one turn on slot 0 +- a tool call with no host grant cannot reach the tailnet +- production Mattermost bot stays down for this check + +## 8. Runbook + +New entries, each with a pointer from the message, for every start failure M7 adds. Candidates: + +- a role container cannot bind its socket (directory missing or wrong user) +- nested Podman from `brokerd` is unavailable +- `loopd` is missing one of its four sockets at start + +Existing entries (`brokerd-start-failed`, `secret-unavailable`, `consult-not-configured`, …) +keep their text. If a path in an entry assumes host processes, the task that adds the unit +updates the check steps. + +## 9. Out of scope + +- P22 / M8: NixOS module, `dist/` units for Omarchy, fleet of agents, crossbar +- A live `consult` URL off-host (own egress row, grant, secret, later) +- HTTPS for `consult` +- Cloud-led sessions +- Changing the inference contract or slot layout +- Moving `bxctl` into a container +- Replacing rootless Podman + +## 10. Changes to the brief and decisions + +None required if section 11 stays inside the brief. The 2026-09-18 rows already say M7 +enforces the socket split with mounts and gives each role its own user. + +## 11. The owner's answers (open) + +1. **Nested Podman.** `brokerd` today starts tool containers as the owner's user. In M7 it + runs inside a container. Is nested rootless Podman (socket from the host, or Podman-in-Podman) + acceptable, or does `brokerd` stay a host process that only the tools and `loopd` move? +2. **Images.** One image per role, or one image with every role binary and a different + `command`? +3. **How they start on straylight.** systemd user units wrapping `podman run`, Podman + quadlets, or a script in `deploy/` the device check invokes? (NixOS module is M8.) +4. **`consult` loopback from the `brokerd` container.** `slirp4netns`/`pasta` localhost is not + the host's `127.0.0.1`. Keep `consult` on host-gateway loopback, put the dummy in the + `brokerd` netns, or leave `brokerd` on the host until a live URL exists? +5. **Tailnet ACL source of truth.** A file in `deploy/` that the owner's Tailscale ACL is + generated from, or a pointer plus a checked-in copy of the policy this host should have? +6. **Production cutover.** Device check on a private `$HOME` only in M7, or also move the + live Mattermost bot into the new layout as part of the milestone? +7. **Split.** One plan, or M7a (loopd `--network=none` + socket mounts, still one user) then + M7b (users, images, ACLs)?