A unikernel VM has no runtime restart policy, so if it exits — including
when an agent restart's cgroup kill takes it down — nothing restarts it,
and it sits in drift. Recover() already handles this (and unikernels, via
runtimeFor), but only ran inside RunBootSequence, which is gated on a
[boot] sequence that worker nodes don't define. Now the agent also runs
Recover once in the background on startup when there is no boot sequence,
so desired=running components (VMs especially) come back after an agent
or host restart without delaying registration.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two drift-reporting bugs:
1. The monitor listed only the podman runtime, so unikernel VMs always
showed observed=unknown (false drift). It now takes a ContainerLister
and the agent passes a merged lister (containers + VMs), mirroring
listAllContainers.
2. The monitor computed the lookup name as service+"-"+component, which
is wrong when component==service (the name collapses to just the
service, e.g. "uktest"/"mc-proxy"). It now uses the canonical
naming.ContainerNameFor — extracted to a shared package so the agent
and monitor can't disagree.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Enables migrating real services (config/cert dirs, stateless) to
unikernels. Volume host dirs are copied into a per-VM staging tree
mirroring guest paths; the ops config goes in the staging root with the
top-level dirs in Dirs, so ops bakes them at the right absolute paths.
(Staging is required — an absolute /srv MapDirs source makes ops descend
into the agent's podman overlay storage and fail.) A component may set
network = "user" to use QEMU user-mode NAT instead of the isolated
bridge (Phase-1 networking for first migrations, before a gateway proxy).
Verified: mcat (the MCIAS policy tester) deployed as a Nanos unikernel
via 'mcp deploy', booting with its baked /srv/mcat config+certs, serving
HTTPS verified against the platform CA, configured against MCIAS.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
When the mcp-br0 bridge exists, the agent runs unikernels on it instead
of QEMU user-mode networking: each VM gets a TAP device on the bridge
and a static 10.99.0.0/24 IP (baked into the Nanos image via ops
RunConfig). With the host firewall dropping off-bridge VM traffic and no
NAT, a VM can reach only the gateway -- making mc-proxy mediation
mandatory by topology rather than convention.
- runtime/qemu.go: bridge mode (createTAP/destroyTAP, IP allocator,
deterministic MAC, static-IP ops config, VMAddr for proxy backends).
- agent auto-enables bridge mode when /sys/class/net/mcp-br0 exists.
Verified on straylight: uktest unikernel boots on mcp-br0 at 10.99.0.2,
serves via the gateway, TAP enslaved to the bridge; bridge has no uplink
and off-bridge forwarding is dropped.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Implements the hypervisor design's Phase 1: a second runtime.Runtime
backend (QEMU) that runs each service component as a Nanos unikernel VM
instead of a podman container, selected per-component via a new
runtime = "unikernel" service-def field.
- internal/runtime/qemu.go: QEMURuntime. Pull extracts the ELF from the
OCI image; Run does `ops build` + boots qemu-system-x86_64 with KVM,
user-mode net port-forwards, QMP control socket and serial console log;
Stop/Remove/Inspect/List/Logs map onto VM lifecycle + state dir.
- proto/registry/servicedef: add runtime, memory_mb, vcpus fields
(registry migration 5).
- agent: holds both runtimes; runtimeFor() selects per component;
listAllContainers() merges containers + VMs so drift/status see both.
Unikernel runtime auto-enables on nodes with /dev/kvm + ops.
Validated end-to-end on straylight: a test service deploys via
`mcp deploy --direct`, boots as a Nanos unikernel, serves HTTP through
the agent port-forward, and reports running via `mcp status`/`mcp logs`.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Heartbeat client now reads master connection settings from the agent
config ([master] section) with env var fallback. Includes address,
ca_cert, token_path, and role fields.
Agent's Run() creates and starts the heartbeat client automatically
when [master] is configured.
Tested on all three nodes: rift (master), svc (edge), orion (worker)
all registered with the master and sending heartbeats every 30s.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The agent reads [[boot.sequence]] stages from its config and starts
services in dependency order before accepting gRPC connections. Each
stage waits for its services to pass health checks before proceeding:
- tcp: TCP connect to the container's mapped port
- grpc: standard gRPC health check
Foundation stage (stage 0): blocks and retries indefinitely if health
fails — all downstream services depend on it.
Non-foundation stages: log warning and proceed on failure.
Uses the recover logic to start containers from the registry, then
health-checks to verify readiness.
Config example:
[[boot.sequence]]
name = "foundation"
services = ["mcias", "mcns"]
timeout = "120s"
health = "tcp"
Architecture v2 Phase 4 feature.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Recreates containers from the agent's SQLite registry when podman's
database is lost (UID change, podman reset, reboot). For each service
with desired_state="running" that doesn't have a running container:
- Removes any stale container with the same name
- Recreates the container from the stored spec (image, ports, volumes,
cmd, network, user, restart policy)
- Allocates route ports and injects PORT env vars
- Re-registers mc-proxy routes
- Provisions TLS certs for L7 routes
Does NOT pull images — assumes local cache.
Root cause action item from the 2026-04-03 UID incident.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
NodeConfig and MasterNodeConfig gain an optional addresses[] field
for fallback addresses tried in order after the primary address.
Provides resilience when Tailscale DNS is down or a node is only
reachable via LAN.
- dialAgentMulti: tries each address with a 3s health check, returns
first success
- forEachNode: uses multi-address dialing
- AgentPool.AddNodeMulti: master tries all addresses when connecting
- AllAddresses(): deduplicates primary + fallback addresses
Config example:
[[nodes]]
name = "rift"
address = "rift.scylla-hammerhead.ts.net:9444"
addresses = ["100.95.252.120:9444", "192.168.88.181:9444"]
Existing configs without addresses[] work unchanged.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Two-stage build: golang:1.25-alpine builder, alpine:3.21 runtime.
Produces a minimal container image for mcp-master.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
RouteDef gains Public field (bool) for edge routing. ServiceDef gains
Tier field. Node validation relaxed: defaults to tier=worker when both
node and tier are empty (v2 compatibility).
ToProto/FromProto updated to round-trip all new fields. Without this,
public=true in TOML was silently dropped and edge routing never triggered.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Node addresses may be Tailscale DNS names (e.g., rift.scylla-hammerhead.ts.net:9444)
but MCNS needs an IPv4 address for A records. The master now resolves
the hostname via net.LookupHost before passing it to the DNS client.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
New cmd/mcp-master/ entry point following the agent pattern:
cobra CLI with --config, version, and server commands.
Makefile: add mcp-master target, update all and clean targets.
Example config: deploy/examples/mcp-master.toml with all sections.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Master struct with Run() lifecycle following the agent pattern exactly:
open DB → bootstrap nodes → create agent pool → DNS client → TLS →
auth interceptor → gRPC server → signal handler.
RPC handlers:
- Deploy: place service (tier-aware), forward to agent, register DNS
with Tailnet IP, detect public routes, validate against allowed
domains, coordinate edge routing via SetupEdgeRoute, record placement
and edge routes in master DB, return structured per-step results.
- Undeploy: undeploy on worker first, then remove edge routes, DNS,
and DB records. Best-effort cleanup on failure.
- Status: query agents for service status, aggregate with placements
and edge route info from master DB.
- ListNodes: return all nodes with placement counts.
Placement algorithm: fewest services, ties broken alphabetically.
DNS client: extracted from agent's DNSRegistrar with explicit nodeAddr
parameter (master registers for different nodes).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
AgentClient wraps a gRPC connection to a single agent with typed
forwarding methods (Deploy, UndeployService, SetupEdgeRoute, etc.).
AgentPool manages connections to multiple agents keyed by node name.
Follows the same TLS 1.3 + token interceptor pattern as cmd/mcp/dial.go
but runs server-side with the master's own MCIAS service token.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
New internal/masterdb/ package for mcp-master cluster state. Separate
from the agent's registry because the schemas are fundamentally
different (cluster-wide placement vs node-local containers).
Tables: nodes, placements, edge_routes. Full CRUD with tests.
Follows the same Open/migrate pattern as internal/registry/.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Temporary CLI commands for testing edge routing RPCs directly
(before the master exists):
mcp edge list -n svc
mcp edge setup <hostname> -n svc --backend-hostname ... --backend-port ...
mcp edge remove <hostname> -n svc
Verified end-to-end on svc: setup provisions route in mc-proxy and
persists in agent registry, remove cleans up both, list shows routes
with cert metadata.
Finding: MCNS registers LAN IPs for .svc.mcp. hostnames, not Tailnet
IPs. The v2 master needs to register Tailnet IPs in deploy flow step 3.
These commands will be removed or replaced when the master is built
(Phase 3).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
New agent RPCs for v2 multi-node orchestration:
- SetupEdgeRoute: provisions TLS cert from Metacrypt, resolves backend
hostname to Tailnet IP, validates it's in 100.64.0.0/10, registers
L7 route in mc-proxy. Rejects backend_tls=false.
- RemoveEdgeRoute: removes mc-proxy route, cleans up TLS cert, removes
registry entry.
- ListEdgeRoutes: returns all edge routes with cert serial/expiry.
- HealthCheck: returns agent health and container count.
New database table (migration 4): edge_routes stores hostname, backend
info, and cert paths for persistence across agent restarts.
ProxyRouter gains CertPath/KeyPath helpers for consistent cert path
construction.
Security:
- Backend hostname must resolve to a Tailnet IP (100.64.0.0/10)
- backend_tls=false is rejected (no cleartext to backends)
- Cert provisioning failure fails the setup (no route to missing cert)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The v2 architecture doc is platform-wide (covers master, agents,
edge routing, snapshots, migration across all nodes). Moved to
docs/architecture-v2.md in the metacircular workspace repo.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The git fetcher doesn't provide gitDescribe, so the Nix build was
falling through to shortRev and producing commit-hash versions instead
of tag-based ones.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Allow start/stop/restart to target a single component via
<service>/<component> syntax, matching deploy/logs/purge. When a
component is specified, start/stop skip toggling the service-level
active flag. Agent-side filtering returns NotFound for unknown
components.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The --mode flag was defined but never wired through to the RPC.
Add tls_cert and tls_key fields to AddProxyRouteRequest so L7
routes can be created via mcp route add.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Documents the planned v2 architecture: mcp-master on straylight
coordinates deployments across worker (rift) and edge (svc) nodes.
Includes edge routing flow, agent RPCs, migration plan, and
operational issues from v1 that motivate the redesign.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Previously, explicit port mappings from the service definition were
ignored when routes were present. Now both are included, allowing
services to have stable external port bindings alongside dynamic
route-allocated ports.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
New top-level command with list, add, remove subcommands. Supports
-n/--node to target a specific node. Adds AddProxyRoute and
RemoveProxyRoute RPCs to the agent. Moves route listing from
mcp node routes to mcp route list.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
mcp dns queries MCNS via an agent to list all zones and DNS records.
mcp node routes queries mc-proxy on each node for listener/route status,
matching the mcproxyctl status output format.
New agent RPCs: ListDNSRecords, ListProxyRoutes.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Probe journalctl with -n 0 before committing to it. When the journal
is not readable (e.g. rootless podman without user journal storage),
fall back to podman logs instead of streaming the permission error.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Thread the linker-injected version string into the Agent struct and
return it in the NodeStatus RPC. The CLI now dials each node and
displays the agent version alongside name and address.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Rootless podman writes container logs to the user journal, but
journalctl without --user only reads the system journal. Add --user
when the agent is running as a non-root user.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Eliminates the manual version bump in flake.nix on each release.
Uses self.shortRev (or dirtyShortRev) since self.gitDescribe is not
yet available in this Nix version. Makefile builds still get the full
git describe output via ldflags.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
mcp ps now uses the actual container image and version from the runtime
instead of the registry, which could be stale after a failed deploy.
Deploy now returns an error when the component filter matches nothing
instead of silently succeeding with zero results.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Skip empty lines from the scanner that result from double newlines
(application slog trailing newline + container runtime newline).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
New server-streaming Logs RPC streams container output to the CLI.
Supports --tail/-n, --follow/-f, --timestamps/-t, --since.
Detects journald log driver and falls back to journalctl (podman logs
can't read journald outside the originating user session). New containers
default to k8s-file via mcp user's containers.conf.
Also adds stream auth interceptor for the agent gRPC server (required
for streaming RPCs).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replaces the "admin required for all operations" model with the new
three-tier identity model: human operators for CLI, mcp-agent system
account for infrastructure automation, admin reserved for MCIAS-level
administration. Documents agent-to-service token paths and per-service
authorization policies.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The agent now accepts any authenticated user or system account, except
those with the guest role. Admin is reserved for MCIAS account management
and policy changes, not routine deploy/stop/start operations.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
mcp build and mcp deploy (auto-build path) now authenticate to the
container registry using the CLI's stored MCIAS token before pushing.
MCR accepts JWTs as passwords, so this works with both human and
service account tokens. Falls back silently to existing podman auth.
Eliminates the need for a separate interactive `podman login` step.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Replaces raw bufio.Scanner password reading (which echoed to terminal)
with the new mcdsl terminal package that suppresses echo via x/term.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
List() was not extracting the StartedAt field from podman's JSON
output, so LiveCheck always returned zero timestamps and the CLI
showed "-" for every container's uptime.
podman ps --format json includes StartedAt as a Unix timestamp
(int64). Parse it into ContainerInfo.Started so the existing
LiveCheck → CLI uptime display chain works.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Implements `mcp undeploy <service>` which tears down all infrastructure
for a service: removes mc-proxy routes, DNS records, TLS certificates,
stops and removes containers, releases allocated ports, and marks the
service inactive.
This fills the gap between `stop` (temporary pause) and `purge` (registry
cleanup). Undeploy is the complete teardown that returns the node to the
state before the service was deployed.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
