boxmaker/docs/specs/2026-10-04-m7-split-deployment.md

# M7 design: split deployment

Status: draft, 2026-10-04, awaiting the owner's answers in section 11. The implementer is Ornith
(through `tools/run-plan.sh`), as for M6. No plan is written until this spec is approved.

It rests on the brief (`docs/design.md`, Network isolation, Roles, Authority 4, Target
environment), on the 2026-09-18 decisions (rootless Podman, separate users per role, `broker.sock`
and `admin.sock` enforced with mounts in M7), and on what already exists: `inferproxy`, the M3b
tool image and egress proxy, `flake.nix` packages and `deploy/tools-image.nix`. P22 / M8 (a fleet
module and `make install`) stays out of this milestone.

Where this document and the brief disagree, section 10 names the brief change. There is none yet.

## 1. What M7 proves

Each role already has a narrow binary. Until now they all run as the owner's user on the host.
M7 puts each role in its own rootless container so a compromise of one process is contained by
namespaces, mounts and users, not only by code.

The milestone is checked by:

- from inside the `loopd` container, requests to the internet, the tailnet and the host all fail
- a tool container without a host grant cannot reach the tailnet (already true of M3b; repeated
  from the new layout)
- `loopd` cannot open `admin.sock`; `bxctl` on the host still can

## 2. What already holds

| Piece | Today |
|---|---|
| Tool containers | Fresh rootless Podman per call, image from Nix, named by digest, `--network=none` unless the grant names hosts, egress proxy for `http_fetch` |
| `inferproxy` | Unix socket in, one TCP upstream to the host `llama-server` |
| `loopd` | No credentials, no network in process terms; it still runs with the host network namespace |
| Sockets | Pair directories under `$HOME/run/`; both ends are the same Unix user, so the split is code (`forbidden`) |
| `flake.nix` | Builds every binary and the tool image; no NixOS module (P22 / M8) |
| Production on straylight | Host processes, owner's user, Mattermost bot untouched by this spec's device check |

M7 does not rewrite the IPC, grants, audit, or inference contract. It changes *where* the
binaries run and *who* can see which sockets and files.

## 3. Isolation model

One long-lived container per role: `loopd`, `brokerd`, `gatewayd`, `memoryd`, `inferproxy`.
Tool containers stay as they are (one shot, from `brokerd`). `bxctl` stays on the host: it is
the owner's CLI, not a role.

Each role container:

- runs as a dedicated user on the host (the 2026-09-18 decision: separate users per role)
- is started rootless Podman (`--userns=keep-id` so the role user owns its files)
- is `--read-only` with tmpfs for `/tmp` and the lock/socket directories it must write
- drops capabilities (`--cap-drop=all`, `no-new-privileges`)
- mounts only the directories that role may see, with the mode that role needs

Network per role (brief, Network isolation):

| Role | Network |
|---|---|
| `loopd` | `--network=none`. Loopback only. Four Unix sockets. |
| `inferproxy` | Outbound to the configured `llama-server` host:port and nothing else |
| `gatewayd` | Outbound to Mattermost. No listening TCP port |
| `memoryd` | Outbound to Honcho and nothing else |
| `brokerd` | No outbound internet in M7. It talks to the container runtime, to its sockets, and (for `consult`) to a loopback URL already required to be `127.0.0.1` or `localhost`. How loopback `consult` is reached from inside the container is question 4 |
| tool containers | Unchanged: none, or the per-call egress proxy |

## 4. Sockets and mounts

Peer authentication stays directory permissions, now across users.

Each pair of roles shares one directory that only those two users can enter. The listening
socket lives there, mode `0600`, owned by the listener. The other role's container mounts that
directory. A third role does not get the directory.

The pairs that already exist under `run/`:

| Directory | Listener | Client | M7 extra |
|---|---|---|---|
| `run/loop/` | `loopd` (`loop.sock`) | `gatewayd`, `bxctl` (host) | `bxctl` is in the owner's group or ACL for this dir |
| `run/loop-broker/` | `brokerd` (`broker.sock`) | `loopd` | `loopd` does **not** mount `run/owner-broker/` |
| `run/owner-broker/` | `brokerd` (`admin.sock`) | `bxctl` (host) | not mounted into `loopd` |
| `run/infer/` | `inferproxy` | `loopd` | |
| `run/loop-memory/` | `memoryd` | `loopd` | |
| `run/broker-memory/` | `memoryd` | `brokerd` | |
| `run/memory-broker/` | `brokerd` (`state.sock`) | `memoryd` | |
| `run/owner-memory/` | `memoryd` (`admin.sock`) | `bxctl` (host) | not mounted into `loopd` |
| `run/gateway/` | `brokerd` (`gateway.sock`), when `[gateway]` is set | `gatewayd` | |

Files:

| Tree | Who writes | Who reads |
|---|---|---|
| `sessions/` | `loopd` | `memoryd` (reindex, Honcho send), `bxctl` |
| `memory/` | `memoryd` | `loopd` (baseline copies at epoch start, through the socket, as today) |
| `grants/` | the owner, from the host | `brokerd` (read-only) |
| `audit/` | `brokerd` | `bxctl audit verify` on the host |
| `broker/sessions/` | `brokerd` | `brokerd` |
| config `*.toml`, `system.md`, `soul.md`, `heartbeat.md`, `compact.md` | the owner | the role that loads them |

`loopd` never sees `grants/`, `audit/`, or `admin.sock`. That is the mount half of the 2026-09-18
socket split.

## 5. Images and how they start

`flake.nix` already builds one package of every binary. M7 adds, under `deploy/`:

- a rootless image per role **or** one image holding all role binaries (question 2), still
  built by Nix, loaded with `podman load`, named by digest
- a host ACL file for the tailnet node (question 5): `deploy/acl.hujson` or the owner's
  existing Tailscale ACL repo, copied or pointed at from here so the gate can see it
- unit files or a small supervisor script that starts the five role containers with the
  mounts in section 4 (question 3)

No NixOS module in M7 (P22 / M8). No `make install` for Omarchy in M7.

Secrets stay `bxcore::secrets`: systemd credential, env, or owner-only file. Inside a
container the credential backend needs the host to pass the credential in; the env and file
backends already work if the file is mounted mode `0600`. Default remains credential.

One instance per daemon still holds: the existing lock files live on the mounted `run/` dirs.

## 6. Components

| Piece | Change |
|---|---|
| `loopd` | none required in Rust if `--network=none` plus mounts are enough. A start failure when a required socket dir is missing already has runbook entries; add any new pointer M7 creates |
| `brokerd` | Podman is invoked from inside the `brokerd` container to start tool containers (nested rootless, question 1). `consult` stays loopback HTTP/1.1 |
| `gatewayd` | none in Rust |
| `memoryd` | none in Rust |
| `inferproxy` | none in Rust; its upstream is still host `llama-server` |
| `bxctl` | stays on the host; needs group/ACL on `loop.sock` and `admin.sock` dirs |
| `bxcore` | none |
| `deploy/` | role image(s), start units or script, tailnet ACL policy, example mounts |
| `docs/egress.md` | no new run-time destination. `consult` is still loopback until a later live URL |

## 7. Testing

Given tests first, against a fake `podman` / a recorded argument list where the start script is
code, and against permission checks on a scratch tree of `run/` dirs:

- **loopd has no net:** a helper in the `loopd` image (or `busybox wget`) to `1.1.1.1:443`, to
  the tailnet Mattermost host, and to the host's `llama-server` port, all fail. The four Unix
  sockets still work.
- **admin.sock is not in the loopd mount:** opening it from the `loopd` container is
  `ENOENT` or `EACCES`. `bxctl approvals` from the host still lists.
- **broker.sock is in the loopd mount:** a tool request still reaches `brokerd`.
- **Tool without hosts:** `shell` or `http_fetch` with no host grant cannot reach the tailnet
  (repeat of M3b from the new `brokerd` container).
- **inferproxy:** `loopd` still completes a self-test through the socket.

**On the device** (straylight, private `$HOME`, production daemons untouched, slot 1 left busy):

- start the five role containers from `deploy/`
- from inside `loopd`, the three negative probes in the first bullet fail
- `bxctl chat --session m7-live --no-thinking` completes one turn on slot 0
- a tool call with no host grant cannot reach the tailnet
- production Mattermost bot stays down for this check

## 8. Runbook

New entries, each with a pointer from the message, for every start failure M7 adds. Candidates:

- a role container cannot bind its socket (directory missing or wrong user)
- nested Podman from `brokerd` is unavailable
- `loopd` is missing one of its four sockets at start

Existing entries (`brokerd-start-failed`, `secret-unavailable`, `consult-not-configured`, …)
keep their text. If a path in an entry assumes host processes, the task that adds the unit
updates the check steps.

## 9. Out of scope

- P22 / M8: NixOS module, `dist/` units for Omarchy, fleet of agents, crossbar
- A live `consult` URL off-host (own egress row, grant, secret, later)
- HTTPS for `consult`
- Cloud-led sessions
- Changing the inference contract or slot layout
- Moving `bxctl` into a container
- Replacing rootless Podman

## 10. Changes to the brief and decisions

None required if section 11 stays inside the brief. The 2026-09-18 rows already say M7
enforces the socket split with mounts and gives each role its own user.

## 11. The owner's answers (open)

1. **Nested Podman.** `brokerd` today starts tool containers as the owner's user. In M7 it
   runs inside a container. Is nested rootless Podman (socket from the host, or Podman-in-Podman)
   acceptable, or does `brokerd` stay a host process that only the tools and `loopd` move?
2. **Images.** One image per role, or one image with every role binary and a different
   `command`?
3. **How they start on straylight.** systemd user units wrapping `podman run`, Podman
   quadlets, or a script in `deploy/` the device check invokes? (NixOS module is M8.)
4. **`consult` loopback from the `brokerd` container.** `slirp4netns`/`pasta` localhost is not
   the host's `127.0.0.1`. Keep `consult` on host-gateway loopback, put the dummy in the
   `brokerd` netns, or leave `brokerd` on the host until a live URL exists?
5. **Tailnet ACL source of truth.** A file in `deploy/` that the owner's Tailscale ACL is
   generated from, or a pointer plus a checked-in copy of the policy this host should have?
6. **Production cutover.** Device check on a private `$HOME` only in M7, or also move the
   live Mattermost bot into the new layout as part of the milestone?
7. **Split.** One plan, or M7a (loopd `--network=none` + socket mounts, still one user) then
   M7b (users, images, ACLs)?