metacircular/docs/hypervisor-design.md

# Hypervisor-Based Service Isolation -- Design Notes

> **Status**: Brainstorming / future direction. This document is NOT
> an active work item. Agents should ignore this document unless
> specifically asked to consider it.

## Context

The metacircular platform runs Go services as rootless podman
containers orchestrated by MCP. This is a pragmatic execution of ideas
originally explored in a series of 2015 papers on security kernels,
environment isolation, and unikernels (see References). Those papers
describe a richer model than what containers provide: hardware-enforced
isolation, mandatory inter-environment communication mediation, and
capability-based access control. This document explores bridging the
gap by running services as unikernel VMs (specifically Nanos) on the
MCP control plane.

## Motivation: What Containers Don't Give Us

The current platform has the W7 security kernel's three properties --
isolated environments, inter-environment communication (IEC), and
access mediation -- but implemented cooperatively rather than enforced:

| W7 Property | Metacircular Today | Enforcement |
|---|---|---|
| Isolated environments | Rootless podman (namespaces/cgroups) | OS-cooperative -- shared kernel, escape CVEs exist |
| IEC | gRPC/TLS through mc-proxy | Application-cooperative -- services *choose* to route through mc-proxy |
| Access mediation | MCIAS tokens + per-service policies | Application-level -- services check tokens voluntarily |

The topology is right. The enforcement mechanism is weak. Containers
share a kernel, and any service could bypass mc-proxy to reach the
Tailnet directly.

## What Unikernels Buy Us

### Hardware-Enforced Isolation

Each service runs in its own VM with its own kernel. There is no
shared kernel to escape from. The security boundary is the hypervisor
(KVM), not Linux namespaces. This is the W7 "isolated environments"
model enforced by hardware, not convention.

### Mandatory IEC

This is the subtle but powerful part. A container on the Tailnet can
talk to anything. A unikernel VM with no direct network interface --
only a virtio-net device connected to a host-only bridge that the
agent controls -- cannot bypass the mediation layer. If the agent is
the only Tailnet citizen on the node and VMs can only reach the
agent's bridge, then mc-proxy stops being a routing convenience and
becomes the IEC mechanism. Communication between environments is
mediated by design, not by trust.

### Reduced TCB

Container TCB: Linux kernel + podman runtime + container image (often
a full distro). Unikernel TCB: KVM + Nanos runtime + the Go binary.
No shell, no package manager, no multi-user, no unnecessary syscalls.

## The Agent as Security Kernel

The MCP agent is already structurally positioned to be the W7 security
kernel for its node. It manages environment lifecycle, controls the
IEC layer (mc-proxy routes), provisions credentials (Metacrypt certs),
and reports to a central authority (master).

With unikernels, this role is formalized. The agent becomes the only
entity with host access. Services exist in VMs that can only
communicate through agent-controlled channels:

- **Network access**: virtio-net bridge under agent control; the agent
  decides what each VM can reach.
- **Storage access**: 9p/virtio-fs mounts; the agent controls what
  each VM sees on disk.
- **Credential access**: the agent provisions certs into the VM's
  filesystem before boot.
- **Identity**: the agent attests to the master what image hash is
  running in each VM.

## Why This Is Feasible for Metacircular

Several properties of the existing platform make this tractable:

- **Go + CGO_ENABLED=0**: Every service already produces a static ELF
  binary. Nanos needs exactly this. The `ops` tool packages them with
  minimal friction.

- **Single-process services**: Each service is one Go binary -- no
  sidecars, no shell scripts, no multi-process orchestration. That is
  the unikernel sweet spot.

- **Single-operator trust domain**: No multi-tenant capability
  delegation or federated attestation needed. The agent is the
  security kernel for its node; the master is the coordination point.

- **mc-proxy already mediates traffic**: The routing mesh is already
  in place. Making it mandatory (rather than optional) for unikernel
  VMs is an incremental change, not a new system.

## Design Sketch

### Runtime Abstraction

The agent gains a `Runtime` interface. Podman is one implementation;
QEMU/KVM is another. Service definitions gain a `runtime` field:

```toml
name    = "mcq"
runtime = "unikernel"    # or "container" (default)
tier    = "worker"
```

Both runtimes coexist. Services can be converted incrementally.

### Networking: Host-Only Bridge

Each unikernel VM gets a virtio-net device on a host-only bridge. The
agent runs on the bridge and controls forwarding. VMs cannot reach the
Tailnet directly. All external communication flows through mc-proxy on
the host.

This is structurally similar to how rootless podman already works
(container ports are localhost-only, mc-proxy routes to them), but
with the enforcement moved from convention to network topology.

### Storage: 9p Passthrough

Unikernel VMs mount `/srv/<service>/` via QEMU's `-virtfs` 9p
passthrough. Writes go directly to the host filesystem. This makes
snapshots work the same way as containers -- the agent tars the host
directory.

### Image Building

Two options (not mutually exclusive):

1. **Build on agent**: Agent extracts the ELF binary from the OCI
   image (pulled from MCR) and runs `ops build` locally.
2. **Store unikernel images in MCR**: OCI supports arbitrary media
   types. Unikernel `.img` files could be stored as OCI artifacts.

Option 1 is simpler to start with. Option 2 is cleaner long-term.

### Image Attestation

Before booting a unikernel, the agent hashes the image and reports it
to the master. The master compares against expected hashes from the
service definition. This is software attestation -- not TPM-based, but
it closes the "is this what I deployed?" question. It is a stepping
stone toward measured boot with hardware TPM.

### Snapshot Constraints

Unikernels have no shell. The `cli` and `exec:` snapshot methods
don't work. Only `grpc` snapshots are viable for unikernel services
(the service implements the standard `SnapshotService` RPC). The
default snapshot method (tar config/db/certs from the host-side 9p
mount) works unchanged since the agent tars the host directory, not
the VM filesystem.

### Debugging

No `podman exec`, no shell. Debugging relies on:

- Serial console output from QEMU
- gRPC health/status endpoints
- Structured logging to a file on the 9p mount
- The agent can snapshot and inspect VM state

This is a real loss of convenience. It is the price of proper
isolation -- as noted in the 2015 hypervisor paper, "the nature of
debugging means that isolation is broken."

## Difficulty Assessment

| Aspect | Difficulty | Notes |
|---|---|---|
| Building unikernel images from Go binaries | Easy | Already static ELF, `ops` handles it |
| QEMU lifecycle management in agent | Medium | Replace podman calls with qemu-system calls |
| Networking (host-only bridge + mc-proxy) | Medium | Similar to rootless podman model |
| Persistent storage via 9p | Medium | Well-supported in QEMU, maps to existing `/srv/` layout |
| Snapshots | Medium | `grpc` method works; `cli`/`exec` don't |
| Image attestation | Medium-Low | SHA-256 of image before boot |
| mc-proxy integration | Low | Just needs a reachable IP:port |
| Debugging/observability | Annoying | Loss of exec/shell access |

The minimum meaningful change is the runtime abstraction + isolated
networking together. Running a unikernel with full Tailnet access is
just a heavier container with worse debugging. The isolation properties
only kick in when the agent mediates all communication.

## Progression Path

1. **Runtime abstraction in the agent.** `Runtime` interface with
   podman and qemu implementations. Service definitions gain a
   `runtime` field. Both coexist.

2. **Isolated networking for unikernel VMs.** Host-only bridge per
   node, agent controls forwarding. mc-proxy becomes the mandatory
   IEC layer for unikernel services.

3. **Image attestation.** Agent hashes images before boot, reports to
   master. Master compares against expected values.

4. **Capability tokens (longer-term).** MCIAS issues operation-scoped
   tokens instead of identity tokens. The agent's mediation layer
   enforces them at the network boundary. This is independent of
   unikernels but synergizes with mandatory mediation.

## Open Questions

- **Tailscale integration**: Should unikernel VMs ever be first-class
  Tailnet citizens (via tsnet compiled into the binary), or should the
  agent always mediate? Mandatory mediation is more secure but means
  the agent is on the critical path for all traffic.

- **Resource limits**: QEMU VMs need explicit memory and CPU
  allocation. The current container model doesn't declare resource
  requirements. Unikernels would force this.

- **Mixed fleet**: During transition, some services run as containers
  and some as unikernels. mc-proxy routes to both. Does the master
  need to know the runtime type for placement decisions?

- **ARM support**: Nanos supports aarch64 but the QEMU/KVM story on
  Raspberry Pi (no KVM on all models) may limit unikernels to amd64
  nodes.

## References

- Rees, J. "A Security Kernel Based on the Lambda Calculus" (W7
  security kernel model -- isolated environments, IEC, access
  mediation)
- "Containers, isolation, and operating systems for network spaces"
  (2015) -- argues the OS must provide a security kernel; unikernels
  as viable isolation mechanism
- "A hypervisor for the modern age" (2015) -- problem statement for a
  hypervisor providing proper isolation, IEC, and access mediation
  with a programmatic administrative interface
- "A content-addressable data store with object capabilities" (Nebula,
  2015) -- capability-based access control model
- MCP v2 Architecture (`docs/architecture-v2.md`) -- current platform
  design this document builds on