Compare commits
7
Commits
master
..
95bec6a095
| Author | SHA1 | Date | |
|---|---|---|---|
  kyleandClaude Opus 4.6
|
95bec6a095 | ||
|
kyleandClaude Opus 4.6
|
faf58ceb72 | ||
|
kyleandClaude Opus 4.6
|
bce32654e1 | ||
|
kyleandClaude Opus 4.6
|
0123e6e29a | ||
|
kyleandClaude Opus 4.6
|
86bbfa640f | ||
|
kyleandClaude Opus 4.6
|
cadbb3f234 | ||
|
kyleandClaude Opus 4.6
|
a777c3ff8b |
-11
@@ -11,15 +11,4 @@
|
||||
/mcns
|
||||
/mcp
|
||||
/mcdeploy
|
||||
/mcdoc
|
||||
|
||||
|
||||
# Local service configs and tool settings; these have held credentials.
|
||||
/mcat.toml
|
||||
/mcq.toml
|
||||
/mcr.toml
|
||||
/metacrypt.toml
|
||||
/.claude/
|
||||
|
||||
# V1/V2 source checkouts, archived 2026-09-20. Each is its own git repo.
|
||||
/archive/
|
||||
|
||||
@@ -5,7 +5,7 @@
|
||||
"command": "/home/kyle/.local/bin/gitea-mcp",
|
||||
"env": {
|
||||
"GITEA_HOST": "https://git.wntrmute.dev",
|
||||
"GITEA_ACCESS_TOKEN": "${GITEA_ACCESS_TOKEN}"
|
||||
"GITEA_ACCESS_TOKEN": "873c9bf3ef872e11ca8811621dba0d7e3762cad9"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -2,11 +2,6 @@
|
||||
|
||||
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
||||
|
||||
> **2026-09-20 — V1/V2 are archived.** The platform was parked and is being
|
||||
> redesigned as V3 (see `docs/v3/`). The source repos described below now live
|
||||
> under `archive/v2/` (each still its own git repo, gitignored here). Paths in
|
||||
> this document that name a component directory are relative to `archive/v2/`.
|
||||
|
||||
## Overview
|
||||
|
||||
Metacircular is a multi-service personal infrastructure platform. This root repository is a workspace container — each subdirectory is a separate Git repo (gitignored here). The authoritative platform-wide standards live in `engineering-standards.md`.
|
||||
@@ -22,16 +17,12 @@ Metacircular is a multi-service personal infrastructure platform. This root repo
|
||||
| `mcat/` | MCIAS login policy tester — lightweight web app to test and audit login policies | Go |
|
||||
| `mcdsl/` | Standard library — shared packages for auth, db, config, HTTP/gRPC servers, CSRF, snapshots | Go |
|
||||
| `mcdoc/` | Documentation server — renders markdown from Gitea, serves public docs via mc-proxy | Go |
|
||||
| `mcp/` | Control plane — service deployment, container lifecycle, multi-node fleet management (CLI/agent, master in development) | Go |
|
||||
| `mcp/` | Control plane — operator-driven deployment, service registry, container lifecycle (master/agent) | Go |
|
||||
| `mcns/` | Networking service — custom Go DNS server, authoritative for internal zones | Go |
|
||||
| `ca/` | PKI infrastructure and secrets for dev/test (not source code, gitignored) | — |
|
||||
| `docs/` | Platform-wide documentation (architecture overview, deployment guide) | Markdown |
|
||||
| `log/` | Operational work logs and session notes | Markdown |
|
||||
|
||||
Each subproject has its own `CLAUDE.md`, `ARCHITECTURE.md`, `Makefile`, and `go.mod`. When working in a subproject, read its own CLAUDE.md first.
|
||||
|
||||
Some subprojects live at sibling paths rather than as subdirectories. For example, `mcq` (document review queue) lives at `../mcq/`. This repository contains only core infrastructure services.
|
||||
|
||||
## Service Dependencies
|
||||
|
||||
MCIAS is the root dependency — every other service authenticates through it. No service maintains its own user database. The dependency graph:
|
||||
@@ -59,7 +50,6 @@ make proto # regenerate gRPC code from .proto files
|
||||
make proto-lint # buf lint + buf breaking
|
||||
make devserver # build and run locally against srv/ config
|
||||
make docker # build container image
|
||||
make push # push container image to MCR
|
||||
make clean # remove binaries
|
||||
```
|
||||
|
||||
|
||||
@@ -5,7 +5,7 @@ from its current manually-wired state to fully declarative deployment.
|
||||
It is a living design document — not a spec, not a commitment, but a
|
||||
record of where we are, where we want to be, and what's between.
|
||||
|
||||
Last updated: 2026-04-02 (Phases A–D complete, Phase E in progress)
|
||||
Last updated: 2026-03-28 (Phases A + B + C + D complete)
|
||||
|
||||
---
|
||||
|
||||
@@ -324,11 +324,9 @@ mcproxyctl, TOML editing, cert generation, or DNS zone editing.
|
||||
|
||||
### Immediate Next Steps
|
||||
|
||||
1. **Phase E: Multi-node agent management** — see below. See
|
||||
`docs/phase-e-plan.md` for the detailed Phase E design and
|
||||
`docs/architecture-v2.md` for the v2 master/agent architecture.
|
||||
2. **mcdoc** — deployed (v0.1.0 on rift). Serves rendered markdown
|
||||
from Gitea via mc-proxy.
|
||||
1. **Phase E: Multi-node agent management** — see below.
|
||||
2. **mcdoc implementation** — fully designed, no platform evolution
|
||||
dependency. Deployable now with the new route system.
|
||||
|
||||
#### 10. Agent Binary Location Convention
|
||||
|
||||
@@ -382,9 +380,8 @@ Different for NixOS vs Debian.
|
||||
| Node | OS | Arch | Status |
|
||||
|------|----|------|--------|
|
||||
| rift | NixOS | amd64 | Operational, single MCP agent |
|
||||
| hyperborea | Debian (RPi) | arm64 | Online, agent provisioning planned |
|
||||
| hyperborea | Debian (RPi) | arm64 | Online, needs agent provisioning |
|
||||
| svc | Debian | amd64 | Runs MCIAS, needs agent for public edge services |
|
||||
| orion | NixOS | amd64 | Provisioned, offline for maintenance |
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -15,11 +15,6 @@ All services are written in Go and follow shared
|
||||
[engineering standards](engineering-standards.md). Full platform documentation
|
||||
lives in [docs/metacircular.md](docs/metacircular.md).
|
||||
|
||||
> **2026-09-20 — V1/V2 are archived.** The platform was parked and is being
|
||||
> redesigned as V3 (see `docs/v3/`). The source repos described below now live
|
||||
> under `archive/v2/` (each still its own git repo, gitignored here). Paths in
|
||||
> this document that name a component directory are relative to `archive/v2/`.
|
||||
|
||||
## Components
|
||||
|
||||
| Component | Purpose | Status |
|
||||
@@ -30,9 +25,7 @@ lives in [docs/metacircular.md](docs/metacircular.md).
|
||||
| **MC-Proxy** | Node ingress — TLS proxy and router. L4 passthrough or L7 terminating (per-route), PROXY protocol, firewall with rate limiting and GeoIP. | Implemented |
|
||||
| **MCNS** | Networking — authoritative DNS for internal platform zones, upstream forwarding. | Implemented |
|
||||
| **MCP** | Control plane — operator-driven deployment, service registry, data transfer, master/agent container lifecycle. | Implemented |
|
||||
| **MCDoc** | Documentation server — renders markdown from Gitea, serves public docs. | Implemented |
|
||||
| **MCQ** | Document review queue — push docs for review, MCP server for Claude Code integration. | Implemented |
|
||||
| **MCDeploy** | Deployment CLI — tactical bridge tool, now deprecated and archived. Superseded by MCP. | Deprecated |
|
||||
| **MCDoc** | Documentation server — renders markdown from Gitea, serves public docs. | In progress |
|
||||
|
||||
Shared library: **MCDSL** — standard library for all services (auth, db,
|
||||
config, TLS server, CSRF, snapshots).
|
||||
@@ -109,9 +102,7 @@ metacircular/
|
||||
├── mcns/ DNS server
|
||||
├── mcat/ Login policy tester
|
||||
├── mcdsl/ Standard library (shared packages)
|
||||
├── mcdeploy/ Deployment CLI (deprecated, archived)
|
||||
├── mcdoc/ Documentation server
|
||||
├── mcq/ Document review queue
|
||||
├── ca/ PKI infrastructure (dev/test, not source code)
|
||||
└── docs/ Platform-wide documentation
|
||||
```
|
||||
|
||||
@@ -1,45 +1,41 @@
|
||||
# Metacircular Platform Status
|
||||
|
||||
Last updated: 2026-04-02
|
||||
Last updated: 2026-03-28
|
||||
|
||||
## Platform Overview
|
||||
|
||||
Two nodes operational (**rift** + **svc**), with **orion** provisioned but
|
||||
offline for maintenance. Core infrastructure services run as containers on
|
||||
rift, fronted by MC-Proxy. Svc operates as an MCP edge node managing
|
||||
mc-proxy routing only (no containers); MCIAS runs on svc separately as a
|
||||
systemd service. Bootstrap phases 0–4 complete (MCIAS, Metacrypt, MC-Proxy,
|
||||
MCR all operational). MCP is deployed and managing all platform containers
|
||||
on rift, with multi-node capability (svc as edge node). MCNS is deployed on
|
||||
rift, serving authoritative DNS. Platform evolution Phases A–D complete
|
||||
(automated port assignment, route registration, TLS cert provisioning, and
|
||||
DNS registration). Phase E (multi-node expansion) is in planning, with v2
|
||||
architecture in development.
|
||||
One node operational (**rift**), running core infrastructure services as
|
||||
containers fronted by MC-Proxy. MCIAS runs separately (not on rift).
|
||||
Bootstrap phases 0–4 complete (MCIAS, Metacrypt, MC-Proxy, MCR all
|
||||
operational). MCP is deployed and managing all platform containers. MCNS is
|
||||
deployed on rift, serving authoritative DNS. Platform evolution Phases A–D
|
||||
complete (automated port assignment, route registration, TLS cert
|
||||
provisioning, and DNS registration). Multi-node deployment is being planned
|
||||
(Phase E).
|
||||
|
||||
## Service Status
|
||||
|
||||
| Service | Version | SDLC Phase | Deployed | Node |
|
||||
|---------|---------|------------|----------|------|
|
||||
| MCIAS | v1.10.5 | Maintenance | Yes | svc (systemd) |
|
||||
| Metacrypt | v1.4.1 | Production | Yes | rift |
|
||||
| MC-Proxy | v1.2.2 | Maintenance | Yes | rift |
|
||||
| MCR | v1.3.2 | Production | Yes | rift |
|
||||
| MCAT | v1.2.0 | Production | Yes | rift |
|
||||
| MCDSL | v1.7.0 | Stable | N/A (library) | — |
|
||||
| MCNS | v1.2.0 | Production | Yes | rift |
|
||||
| MCDoc | v0.1.0 | Production | Yes | rift |
|
||||
| MCQ | v0.4.2 | Production | Yes | rift |
|
||||
| MCP | v0.9.0 | Production | Yes | rift |
|
||||
| MCIAS | v1.9.0 | Maintenance | Yes | (separate) |
|
||||
| Metacrypt | v1.3.1 | Production | Yes | rift |
|
||||
| MC-Proxy | v1.2.1 | Maintenance | Yes | rift |
|
||||
| MCR | v1.2.1 | Production | Yes | rift |
|
||||
| MCAT | v1.1.1 | Complete | Unknown | — |
|
||||
| MCDSL | v1.4.0 | Stable | N/A (library) | — |
|
||||
| MCNS | v1.1.1 | Production | Yes | rift |
|
||||
| MCP | v0.7.6 | Production | Yes | rift |
|
||||
| MCDoc | v0.1.0 | Active dev | No | — |
|
||||
|
||||
## Service Details
|
||||
|
||||
### MCIAS — Identity and Access Service
|
||||
|
||||
- **Version:** v1.10.5 (client library: clients/go/v0.2.0)
|
||||
- **Version:** v1.9.0 (client library: clients/go/v0.2.0)
|
||||
- **Phase:** Maintenance. Phases 0-14 complete. Feature-complete with active
|
||||
refinement.
|
||||
- **Deployment:** Running in production on svc as a systemd service. All
|
||||
other services authenticate against it.
|
||||
- **Deployment:** Running in production. All other services authenticate
|
||||
against it.
|
||||
- **Recent work:** WebAuthn/FIDO2 passkeys, TOTP 2FA, service-context login
|
||||
policies, Nix flake for CLI tools.
|
||||
- **Artifacts:** systemd units (service + backup timer), install script,
|
||||
@@ -47,7 +43,7 @@ architecture in development.
|
||||
|
||||
### Metacrypt — Cryptographic Service Engine
|
||||
|
||||
- **Version:** v1.4.1 (API v1.3.1, Web v1.4.1).
|
||||
- **Version:** v1.3.1.
|
||||
- **Phase:** Production. All four engine types implemented (CA, SSH CA, transit,
|
||||
user-to-user). Active work on integration test coverage.
|
||||
- **Deployment:** Running on rift as a container, fronted by MC-Proxy on
|
||||
@@ -59,21 +55,19 @@ architecture in development.
|
||||
|
||||
### MC-Proxy — TLS Proxy and Router
|
||||
|
||||
- **Version:** v1.2.2.
|
||||
- **Phase:** Maintenance. Stable and actively routing traffic on rift and svc.
|
||||
- **Version:** v1.2.1.
|
||||
- **Phase:** Maintenance. Stable and actively routing traffic on rift.
|
||||
- **Deployment:** Running on rift. Fronts Metacrypt, MCR, and sgard on ports
|
||||
443, 8443, and 9443. Prometheus metrics on 127.0.0.1:9091. Routes persisted
|
||||
in SQLite and managed via gRPC API. Svc runs its own mc-proxy on :443 with
|
||||
public-facing routes.
|
||||
- **Recent work:** Route persistence (SQLite), idempotent AddRoute (upsert),
|
||||
golangci-lint v2 compliance, module path migration to mc/ org.
|
||||
in SQLite and managed via gRPC API.
|
||||
- **Recent work:** MCR route additions, Nix flake, L7 backend cert handling,
|
||||
Prometheus metrics, L7 policies.
|
||||
- **Artifacts:** systemd units (service + backup timer), Docker Compose
|
||||
(standard + rift), install and backup scripts, rift config.
|
||||
|
||||
### MCR — Container Registry
|
||||
|
||||
- **Version:** v1.3.2 (API v1.2.1, Web v1.3.2). All implementation phases
|
||||
complete.
|
||||
- **Version:** v1.2.1. All implementation phases complete.
|
||||
- **Phase:** Production. Deployed on rift, serving container images.
|
||||
- **Deployment:** Running on rift as two containers (mcr API + mcr-web),
|
||||
fronted by MC-Proxy on ports 443 (web, L7), 8443 (API, L4), and
|
||||
@@ -86,26 +80,26 @@ architecture in development.
|
||||
|
||||
### MCAT — Login Policy Tester
|
||||
|
||||
- **Version:** v1.2.0.
|
||||
- **Phase:** Production. Deployed on rift as a container managed by MCP.
|
||||
- **Deployment:** Running on rift. Lightweight tool for testing MCIAS login
|
||||
policy rules.
|
||||
- **Version:** v1.1.1.
|
||||
- **Phase:** Complete. Diagnostic tool, not core infrastructure.
|
||||
- **Deployment:** Available for ad-hoc use. Lightweight tool for testing
|
||||
MCIAS login policy rules.
|
||||
- **Recent work:** Migrated to mcdsl for auth, config, CSRF, and web.
|
||||
- **Artifacts:** systemd unit, install script, example config.
|
||||
|
||||
### MCDSL — Standard Library
|
||||
|
||||
- **Version:** v1.7.0.
|
||||
- **Version:** v1.4.0.
|
||||
- **Phase:** Stable. All 9 packages implemented and tested. Being adopted
|
||||
across the platform.
|
||||
- **Deployment:** N/A (Go library, imported by other services).
|
||||
- **Packages:** auth, db, config, httpserver, grpcserver, csrf, web, health,
|
||||
archive.
|
||||
- **Adoption:** All services except mcias on v1.7.0. mcias pending.
|
||||
- **Adoption:** All services except mcias on v1.4.0. mcias pending.
|
||||
|
||||
### MCNS — Networking Service
|
||||
|
||||
- **Version:** v1.2.0.
|
||||
- **Version:** v1.1.1.
|
||||
- **Phase:** Production. Custom Go DNS server replacing CoreDNS precursor.
|
||||
- **Deployment:** Running on rift as a container managed by MCP. Serves two
|
||||
authoritative zones plus upstream forwarding. REST + gRPC APIs with MCIAS
|
||||
@@ -115,75 +109,52 @@ architecture in development.
|
||||
- **Artifacts:** Dockerfile, Docker Compose (rift), MCP service definition,
|
||||
systemd units, install script, example config.
|
||||
|
||||
### MCP — Control Plane
|
||||
|
||||
- **Version:** v0.7.6.
|
||||
- **Phase:** Production. Phases A–D complete (automated port assignment, route
|
||||
registration, TLS cert provisioning, DNS registration).
|
||||
- **Deployment:** Running on rift. Agent as systemd service under `mcp` user
|
||||
with rootless podman. Manages metacrypt, mc-proxy, mcr, and mcns containers.
|
||||
- **Architecture:** Two components — `mcp` CLI (thin client on vade) and
|
||||
`mcp-agent` (per-node daemon with SQLite registry, podman management,
|
||||
monitoring with drift/flap detection). gRPC-only (no REST). 15 RPCs, 17+
|
||||
CLI commands.
|
||||
- **Recent work:** Phase C (automated TLS cert provisioning via Metacrypt CA),
|
||||
Phase D (automated DNS registration via MCNS), undeploy command, logs
|
||||
command, edit command, auto-login to MCR, system account auth model.
|
||||
- **Artifacts:** systemd service (NixOS), TLS cert from Metacrypt, service
|
||||
definition files, design docs.
|
||||
|
||||
### MCDoc — Documentation Server
|
||||
|
||||
- **Version:** v0.1.0.
|
||||
- **Phase:** Production. Fetches and renders markdown documentation from Gitea.
|
||||
- **Deployment:** Running on rift as a container, fronted by MC-Proxy on
|
||||
port 443 (L7).
|
||||
- **Recent work:** Initial implementation, Gitea content fetching, goldmark
|
||||
rendering with syntax highlighting, webhook-driven refresh.
|
||||
- **Artifacts:** Dockerfile, MCP service definition.
|
||||
|
||||
### MCQ — Document Review Queue
|
||||
|
||||
- **Version:** v0.4.2.
|
||||
- **Phase:** Production. Document review queue with MCP server for Claude
|
||||
integration.
|
||||
- **Deployment:** Running on rift as a container managed by MCP.
|
||||
- **Recent work:** Claude MCP server integration, document review workflow.
|
||||
- **Artifacts:** Dockerfile, MCP service definition.
|
||||
|
||||
### MCP — Control Plane
|
||||
|
||||
- **Version:** v0.9.0 (agent on rift: v0.8.3-dirty, agent on svc: v0.9.0).
|
||||
- **Phase:** Production. Phases A–D complete. Multi-node capable with svc
|
||||
operating as an edge node. V2 architecture in development, Phase E planning
|
||||
underway.
|
||||
- **Deployment:** Running on rift. Agent as systemd service under `mcp` user
|
||||
with rootless podman. Manages metacrypt, mc-proxy, mcr, mcns, mcdoc, mcat,
|
||||
mcq, and non-platform containers. Svc runs an MCP agent for edge mc-proxy
|
||||
route management.
|
||||
- **Architecture:** Two components — `mcp` CLI (thin client on vade) and
|
||||
`mcp-agent` (per-node daemon with SQLite registry, podman management,
|
||||
monitoring with drift/flap detection, route registration with mc-proxy,
|
||||
automated TLS cert provisioning for L7 routes via Metacrypt CA, automated
|
||||
DNS registration in MCNS). gRPC-only (no REST). 15 RPCs, 17+ CLI commands.
|
||||
- **Recent work:** Phase C (automated TLS cert provisioning), Phase D
|
||||
(automated DNS registration via MCNS), undeploy command, logs command,
|
||||
edit command, auto-login to MCR, system account auth model, module path
|
||||
migration, multi-node edge support (svc).
|
||||
- **Artifacts:** systemd service (NixOS), TLS cert from Metacrypt, service
|
||||
definition files, design docs.
|
||||
- **Phase:** Active development.
|
||||
- **Deployment:** Not yet deployed.
|
||||
- **Description:** Documentation server — fetches markdown from Gitea, renders
|
||||
HTML, serves public docs via mc-proxy. No MCIAS auth required.
|
||||
|
||||
## Node Inventory
|
||||
|
||||
| Node | Address (LAN) | Address (Tailscale) | Role |
|
||||
|------|---------------|---------------------|------|
|
||||
| rift | 192.168.88.181 | 100.95.252.120 | Infrastructure services |
|
||||
| svc | — | 100.106.232.4 | Edge — public mc-proxy, MCIAS (systemd) |
|
||||
| orion | 192.168.88.214 | — | Worker (provisioned, offline for maintenance) |
|
||||
|
||||
## Rift Port Map
|
||||
|
||||
Note: Services deployed via MCP receive dynamically assigned host ports
|
||||
(10000–60000). The ports below are for infrastructure services with static
|
||||
assignments or well-known ports.
|
||||
assignments.
|
||||
|
||||
| Port | Protocol | Services |
|
||||
|------|----------|----------|
|
||||
| 53 | DNS (LAN + Tailscale) | mcns |
|
||||
| 443 | L7 (TLS termination) | metacrypt-web, mcr-web, mcdoc, mcat, kls |
|
||||
| 443 | L7 (TLS termination) | metacrypt-web, mcr-web |
|
||||
| 8080 | HTTP (all interfaces) | exod |
|
||||
| 8443 | L4 (SNI passthrough) | metacrypt API, mcr API |
|
||||
| 9090 | HTTP (all interfaces) | exod |
|
||||
| 9443 | L4 (SNI passthrough) | metacrypt gRPC, mcr gRPC, sgard |
|
||||
| 9091 | HTTP (loopback) | MC-Proxy Prometheus metrics |
|
||||
|
||||
Svc runs its own mc-proxy on :443 (L7) with public-facing routes:
|
||||
kls.metacircular.net, mcq.metacircular.net, metacrypt.metacircular.net,
|
||||
docs.metacircular.net, git.metacircular.net, git.wntrmute.dev.
|
||||
|
||||
Non-platform services also running on rift: **exod** (v0.1.0, ports
|
||||
8080/9090), **sgardd** (v3.2.0, port 19473, fronted by MC-Proxy on 9443),
|
||||
**kls** (v0.2.0).
|
||||
Non-platform services also running on rift: **exod** (ports 8080/9090),
|
||||
**sgardd** (port 19473, fronted by MC-Proxy on 9443).
|
||||
|
||||
-158
@@ -1,158 +0,0 @@
|
||||
# svc.metacircular.net — Phase 1 Work Log
|
||||
|
||||
Date: 2026-04-02
|
||||
Purpose: Deploy mcp-agent to svc (edge node) for MCP v2 Phase 1.
|
||||
|
||||
## Changes Made
|
||||
|
||||
### 1. Created `mcp` system user
|
||||
```
|
||||
useradd --system --home-dir /srv/mcp --create-home --shell /usr/sbin/nologin mcp
|
||||
usermod -aG mc-proxy mcp
|
||||
```
|
||||
- UID 992, GID 991
|
||||
- Member of `mc-proxy` group for socket access
|
||||
|
||||
### 2. Created `/srv/mcp/` directory structure
|
||||
```
|
||||
/srv/mcp/
|
||||
├── mcp-agent # binary (v0.8.3-1-gfa8ba6f, linux/amd64)
|
||||
├── mcp-agent.toml # agent config
|
||||
├── mcp.db # SQLite registry (created on first run)
|
||||
└── certs/
|
||||
├── cert.pem # TLS cert (SAN: IP:100.106.232.4, DNS:svc.svc.mcp.metacircular.net)
|
||||
├── key.pem # TLS private key
|
||||
└── ca.pem # Metacircular CA cert
|
||||
```
|
||||
- Owned by `mcp:mcp`, key file mode 0600
|
||||
|
||||
### 3. TLS certificate
|
||||
- Issued from the Metacircular CA (`ca/ca.pem` + `ca/ca.key`)
|
||||
- Subject: `CN=mcp-agent-svc`
|
||||
- SANs: `IP:100.106.232.4`, `DNS:svc.svc.mcp.metacircular.net`
|
||||
- Validity: 365 days
|
||||
- Stored at `/srv/mcp/certs/{cert,key,ca}.pem`
|
||||
|
||||
### 4. Agent configuration
|
||||
- File: `/srv/mcp/mcp-agent.toml`
|
||||
- gRPC listen: `100.106.232.4:9555` (port 9444 in use by MCNS)
|
||||
- MCIAS: `https://mcias.metacircular.net:8443`
|
||||
- mc-proxy socket: `/srv/mc-proxy/mc-proxy.sock`
|
||||
- Node name: `svc`
|
||||
- Runtime: `podman` (not used on edge, but required by config)
|
||||
|
||||
### 5. systemd unit
|
||||
- File: `/etc/systemd/system/mcp-agent.service`
|
||||
- Runs as `mcp:mcp`
|
||||
- Security hardened (NoNewPrivileges, ProtectSystem=strict, etc.)
|
||||
- ReadWritePaths: `/srv/mcp`, `/srv/mc-proxy/mc-proxy.sock`
|
||||
- Enabled and started
|
||||
|
||||
### 6. mc-proxy directory permissions
|
||||
- Changed `/srv/mc-proxy/` from `drwx------` to `drwxr-x---` (group traversal)
|
||||
- Changed `/srv/mc-proxy/mc-proxy.sock` from `srw-------` to `srw-rw----` (group read/write)
|
||||
- Required for `mcp` user (in `mc-proxy` group) to access the socket
|
||||
|
||||
### 7. MCP CLI config update (on rift)
|
||||
- Added svc node to `~/.config/mcp/mcp.toml`:
|
||||
```toml
|
||||
[[nodes]]
|
||||
name = "svc"
|
||||
address = "100.106.232.4:9555"
|
||||
```
|
||||
|
||||
## Verification
|
||||
```
|
||||
$ mcp node list
|
||||
NAME ADDRESS VERSION
|
||||
rift 100.95.252.120:9444 v0.8.3-dirty
|
||||
svc 100.106.232.4:9555 v0.8.3-1-gfa8ba6f
|
||||
|
||||
$ mcp route list -n svc
|
||||
NODE: svc
|
||||
mc-proxy v1.2.1-2-g82fce41-dirty
|
||||
:443 routes=6
|
||||
l7 git.wntrmute.dev → 127.0.0.1:3000
|
||||
l7 kls.metacircular.net → 100.95.252.120:58080
|
||||
l7 mcq.metacircular.net → 100.95.252.120:48080
|
||||
l7 metacrypt.metacircular.net → 100.95.252.120:18080 (re-encrypt)
|
||||
l7 docs.metacircular.net → 100.95.252.120:38080
|
||||
l7 git.metacircular.net → 127.0.0.1:3000
|
||||
```
|
||||
|
||||
## Agent Cert Reissue (2026-04-02)
|
||||
|
||||
Both agent certs reissued with comprehensive SANs:
|
||||
|
||||
**Rift agent** (`/srv/mcp/certs/cert.pem`):
|
||||
- DNS: `rift.scylla-hammerhead.ts.net`, `mcp-agent.svc.mcp.metacircular.net`
|
||||
- IP: `100.95.252.120`, `192.168.88.181`
|
||||
|
||||
**Svc agent** (`/srv/mcp/certs/cert.pem`):
|
||||
- DNS: `svc.scylla-hammerhead.ts.net`, `svc.svc.mcp.metacircular.net`
|
||||
- IP: `100.106.232.4`
|
||||
|
||||
Both agents upgraded to v0.10.0 (Phase 2 edge routing RPCs + v2 proto fields).
|
||||
|
||||
## MCP Master Deployment (2026-04-02)
|
||||
|
||||
**Binary**: `/srv/mcp-master/mcp-master` (v0.10.0) on rift
|
||||
**Config**: `/srv/mcp-master/mcp-master.toml`
|
||||
**Database**: `/srv/mcp-master/master.db`
|
||||
**Certs**: `/srv/mcp-master/certs/{cert,key,ca}.pem`
|
||||
- SAN: `rift.scylla-hammerhead.ts.net`, `mcp-master.svc.mcp.metacircular.net`, IP `100.95.252.120`
|
||||
**Service token**: `/srv/mcp-master/mcias-token` (MCIAS identity: `mcp-master`, expires 2027-04-03)
|
||||
**Listen**: `100.95.252.120:9555`
|
||||
**Bootstrap nodes**: rift (master), svc (edge)
|
||||
|
||||
**Status**: Running via `doas` (ad-hoc). NixOS read-only /etc prevents
|
||||
direct systemd unit creation — needs NixOS config update for persistent
|
||||
service.
|
||||
|
||||
**Tested**:
|
||||
- `mcp deploy mcq` → master places on rift, forwards to agent ✓
|
||||
- `mcp undeploy mcq` → master forwards to agent, cleans up placement ✓
|
||||
- `mcp ps` → fleet-wide status through agents ✓
|
||||
- `mcp node list` → both nodes visible with versions ✓
|
||||
|
||||
## CLI Config Changes (vade)
|
||||
|
||||
Updated `~/.config/mcp/mcp.toml`:
|
||||
- Added `[master]` section: `address = "rift.scylla-hammerhead.ts.net:9555"`
|
||||
- All node addresses switched to Tailscale DNS names
|
||||
- Added CA cert path
|
||||
|
||||
## Known Limitations
|
||||
- ~~mc-proxy socket permissions will reset on restart~~ **FIXED**: mc-proxy
|
||||
now creates the socket with 0660 (was 0600). Committed to mc-proxy master.
|
||||
- Master runs ad-hoc via `doas` on rift. Needs NixOS systemd config for
|
||||
persistent service (rift has read-only /etc).
|
||||
- DNS registration not configured on master (MCNS config omitted for now).
|
||||
- Edge routing not yet tested end-to-end through master (svc cert provisioning
|
||||
not configured).
|
||||
- The TLS cert was issued from the local CA directly, not via Metacrypt API.
|
||||
Should be re-issued via Metacrypt once the agent has cert provisioning.
|
||||
- Container runtime is set to `podman` but podman is not installed on svc
|
||||
(Docker is). Edge agents don't run containers so this is benign.
|
||||
- Metacrypt and MCNS integrations not configured (not needed for edge role).
|
||||
|
||||
## Edge Routing E2E Test (2026-04-02)
|
||||
|
||||
Full edge routing flow tested successfully through the master:
|
||||
|
||||
1. Added `public = true` route for `mcq.metacircular.net` to mcq service def
|
||||
2. `mcp deploy mcq` → master placed on rift, deployed, registered DNS, set up edge route on svc
|
||||
3. Svc agent provisioned TLS cert from Metacrypt (expires 2026-06-28)
|
||||
4. mc-proxy route created: `mcq.metacircular.net → 100.95.252.120:8443` (re-encrypt)
|
||||
5. Edge route persisted in both master DB and svc agent registry
|
||||
|
||||
**Fix required**: `RouteDef` in servicedef.go was missing `Public` field —
|
||||
TOML `public = true` was silently dropped. Fixed in v0.10.2 along with
|
||||
`Tier` field and relaxed `Node` validation.
|
||||
|
||||
## DNS Registration Working (2026-04-02)
|
||||
|
||||
Master registers Tailnet IPs in MCNS (not LAN IPs). Fix in v0.10.1
|
||||
resolves Tailscale DNS names to IPs before passing to MCNS.
|
||||
|
||||
`mcq.svc.mcp.metacircular.net → 100.95.252.120` (Tailnet IP) ✓
|
||||
+7
-12
@@ -8,11 +8,12 @@ Go binaries built with `CGO_ENABLED=0` and installed via Nix flakes.
|
||||
| Tool | Project | Purpose | Install target |
|
||||
|------|---------|---------|---------------|
|
||||
| `mcp` | mcp | Control plane CLI — deploy, status, lifecycle, file transfer | vade, orion |
|
||||
| `mcp-agent` | mcp | Control plane agent — per-node container management daemon | rift, svc (systemd) |
|
||||
| `mcp-agent` | mcp | Control plane agent — per-node container management daemon | rift (systemd) |
|
||||
| `mciasctl` | mcias | MCIAS admin CLI — accounts, tokens, policies | vade, orion, rift |
|
||||
| `mciasgrpcctl` | mcias | MCIAS gRPC debug CLI | vade, orion, rift |
|
||||
| `mcproxyctl` | mc-proxy | MC-Proxy admin CLI — routes, firewall, status | vade, orion, rift |
|
||||
| `mcrctl` | mcr | MCR admin CLI — repositories, policies, audit | vade, orion, rift |
|
||||
| `mcdeploy` | mcdeploy | Deployment CLI — build, push, deploy (bridge tool) | vade |
|
||||
|
||||
### Server-only binaries (not installed as tools)
|
||||
|
||||
@@ -28,8 +29,6 @@ These run inside containers and are not installed on operator workstations:
|
||||
| `mc-proxy` | mc-proxy | TLS proxy server |
|
||||
| `mcns` | mcns | DNS server |
|
||||
| `mcat` | mcat | Login policy tester web app |
|
||||
| `mcdoc` | mcdoc | Documentation server |
|
||||
| `mcq` | mcq | Document review queue |
|
||||
|
||||
## Installation
|
||||
|
||||
@@ -47,18 +46,15 @@ via `mcpkg.nix` in the NixOS configuration. Adding a tool:
|
||||
- Build: `pkgs.buildGoModule` with `vendorHash = null` (vendored deps).
|
||||
- ldflags: `-s -w -X main.version=${version}`.
|
||||
- `subPackages`: list only the client binaries, not servers.
|
||||
- `system`: `x86_64-linux` for rift, svc, and orion; `aarch64-linux`
|
||||
for hyperborea. Flakes that target the full fleet should support both.
|
||||
- `system`: `x86_64-linux` (all machines are x86_64).
|
||||
|
||||
### MCP agent
|
||||
|
||||
The `mcp-agent` is a special case: it runs as a systemd service on
|
||||
managed nodes (not as a container, since it manages containers). Its
|
||||
flake exposes both `mcp` (client) and `mcp-agent` (server). Phase E is
|
||||
moving the agent binary to `/srv/mcp/mcp-agent` on all nodes — NixOS
|
||||
`ExecStart` will point there instead of a nix store path, and Debian
|
||||
nodes use the same layout. svc already follows this convention. See
|
||||
`docs/phase-e-plan.md` for details.
|
||||
flake exposes both `mcp` (client) and `mcp-agent` (server). The NixOS
|
||||
config in `configs/mcp.nix` references the Nix package path for the
|
||||
agent binary.
|
||||
|
||||
## Flake status
|
||||
|
||||
@@ -67,10 +63,9 @@ nodes use the same layout. svc already follows this convention. See
|
||||
| mcias | Yes | mciasctl, mciasgrpcctl | Yes | |
|
||||
| mc-proxy | Yes | mcproxyctl | Yes | |
|
||||
| mcr | Yes | mcrctl | Yes | |
|
||||
| mcdeploy | Yes | mcdeploy | Yes | |
|
||||
| mcp | Yes | mcp, mcp-agent | Yes | Agent also used by mcp.nix systemd unit |
|
||||
| mcns | No | — | No | Server-only, no client tool yet |
|
||||
| metacrypt | No | — | No | Server-only, no client tool yet |
|
||||
| mcat | No | — | No | Server-only, no client tool yet |
|
||||
| mcdoc | No | — | No | Server-only, deployed as container |
|
||||
| mcq | No | — | No | Server-only, document review queue |
|
||||
| mcdsl | No | — | No | Library, no binaries |
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,344 +0,0 @@
|
||||
# Disaster Recovery: Bootstrap from Zero
|
||||
|
||||
This document covers recovering the Metacircular platform when all
|
||||
services on rift are down and no containers are running. It assumes:
|
||||
|
||||
- The machine boots and NixOS is functional
|
||||
- The mcp-agent systemd service starts automatically
|
||||
- Tailscale is configured and connects on boot
|
||||
- Service data directories (`/srv/<service>/`) are intact on disk
|
||||
- Container images are cached in podman's overlay storage
|
||||
|
||||
If images are NOT cached (fresh machine, disk wipe), see the
|
||||
"Cold Start" section at the end.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Before starting recovery, verify:
|
||||
|
||||
```bash
|
||||
# 1. Machine is up
|
||||
hostname # should print "rift"
|
||||
|
||||
# 2. Tailscale is connected
|
||||
tailscale status --self
|
||||
# Should show the Tailnet IP (100.95.252.120)
|
||||
|
||||
# 3. The mcp user exists
|
||||
id mcp
|
||||
# Should show uid=850(mcp) gid=850(mcp)
|
||||
|
||||
# 4. The agent is running
|
||||
systemctl status mcp-agent
|
||||
# Should be active
|
||||
|
||||
# 5. Images are cached
|
||||
su -s /bin/sh mcp -c "XDG_RUNTIME_DIR=/run/user/850 HOME=/srv/mcp podman images" | wc -l
|
||||
# Should be > 0
|
||||
```
|
||||
|
||||
If Tailscale is not running: `doas systemctl start tailscaled && doas tailscale up`
|
||||
|
||||
If the agent is not running: check `/srv/mcp/mcp-agent` exists and
|
||||
`/srv/mcp/mcp-agent.toml` is correct, then `doas systemctl restart mcp-agent`.
|
||||
|
||||
## Recovery Order
|
||||
|
||||
Services must be started in dependency order. Each stage must be
|
||||
healthy before the next starts.
|
||||
|
||||
```
|
||||
Stage 1 (Foundation): MCNS → DNS works
|
||||
Stage 2 (Core): mc-proxy, MCR, Metacrypt → routing + images + certs
|
||||
Stage 3 (Management): mcp-master → orchestration
|
||||
Stage 4 (Services): mcq, mcdoc, mcat, kls, sgard, exo → applications
|
||||
```
|
||||
|
||||
## Manual Recovery Commands
|
||||
|
||||
All commands run as the mcp user. Use this shell prefix:
|
||||
|
||||
```bash
|
||||
# Set up the environment
|
||||
export PODMAN_CMD='doas sh -c "cd /srv/mcp && XDG_RUNTIME_DIR=/run/user/850 HOME=/srv/mcp su -s /bin/sh mcp -c"'
|
||||
# Or SSH as mcp directly (if SSH login is enabled):
|
||||
ssh mcp@rift
|
||||
```
|
||||
|
||||
For brevity, commands below show the `podman run` portion only. Prefix
|
||||
with the environment setup above.
|
||||
|
||||
### Stage 1: MCNS (DNS)
|
||||
|
||||
MCNS must start first. Without it, no hostname resolution works.
|
||||
|
||||
```bash
|
||||
podman run -d --name mcns --restart unless-stopped \
|
||||
-p 192.168.88.181:53:53/tcp \
|
||||
-p 192.168.88.181:53:53/udp \
|
||||
-p 100.95.252.120:53:53/tcp \
|
||||
-p 100.95.252.120:53:53/udp \
|
||||
-p 127.0.0.1:38443:8443 \
|
||||
-v /srv/mcns:/srv/mcns \
|
||||
mcr.svc.mcp.metacircular.net:8443/mcns:v1.2.0 \
|
||||
server --config /srv/mcns/mcns.toml
|
||||
```
|
||||
|
||||
**Verify:**
|
||||
```bash
|
||||
dig @192.168.88.181 google.com +short
|
||||
# Should return an IP address
|
||||
dig @192.168.88.181 mcq.svc.mcp.metacircular.net +short
|
||||
# Should return a Tailnet IP
|
||||
```
|
||||
|
||||
**Note:** MCNS binds to specific IPs, not `0.0.0.0`, because
|
||||
systemd-resolved holds port 53 on localhost. The explicit bindings
|
||||
avoid the conflict.
|
||||
|
||||
### Stage 2: Core Infrastructure
|
||||
|
||||
#### mc-proxy (TLS routing)
|
||||
|
||||
```bash
|
||||
podman run -d --name mc-proxy --restart unless-stopped \
|
||||
--network host \
|
||||
-v /srv/mc-proxy:/srv/mc-proxy \
|
||||
mcr.svc.mcp.metacircular.net:8443/mc-proxy:v1.2.2 \
|
||||
server --config /srv/mc-proxy/mc-proxy.toml
|
||||
```
|
||||
|
||||
**Verify:** `curl -sk https://localhost:443/ 2>&1 | head -1`
|
||||
(should get a response, even if 404)
|
||||
|
||||
#### MCR (Container Registry)
|
||||
|
||||
```bash
|
||||
# API server
|
||||
podman run -d --name mcr-api --restart unless-stopped \
|
||||
-v /srv/mcr:/srv/mcr \
|
||||
-p 127.0.0.1:28443:8443 \
|
||||
-p 127.0.0.1:29443:9443 \
|
||||
mcr.svc.mcp.metacircular.net:8443/mcr:v1.2.1 \
|
||||
server --config /srv/mcr/mcr.toml
|
||||
|
||||
# Web UI
|
||||
podman run -d --name mcr-web --restart unless-stopped \
|
||||
--user 0:0 \
|
||||
-v /srv/mcr:/srv/mcr \
|
||||
-p 127.0.0.1:28080:8080 \
|
||||
mcr.svc.mcp.metacircular.net:8443/mcr-web:v1.3.2 \
|
||||
server --config /srv/mcr/mcr.toml
|
||||
```
|
||||
|
||||
**If MCR fails with "chmod" or "readonly database":**
|
||||
```bash
|
||||
podman stop mcr-api
|
||||
rm -f /srv/mcr/mcr.db /srv/mcr/mcr.db-wal /srv/mcr/mcr.db-shm
|
||||
podman start mcr-api
|
||||
```
|
||||
This recreates the database empty. Image blobs in `/srv/mcr/layers/`
|
||||
are preserved but tag metadata is lost. Re-push images to rebuild the
|
||||
registry.
|
||||
|
||||
#### Metacrypt (PKI / Secrets)
|
||||
|
||||
```bash
|
||||
# API server
|
||||
podman run -d --name metacrypt-api --restart unless-stopped \
|
||||
-v /srv/metacrypt:/srv/metacrypt \
|
||||
-p 127.0.0.1:18443:8443 \
|
||||
-p 127.0.0.1:19443:9443 \
|
||||
mcr.svc.mcp.metacircular.net:8443/metacrypt:v1.3.1 \
|
||||
server --config /srv/metacrypt/metacrypt.toml
|
||||
|
||||
# Web UI
|
||||
podman run -d --name metacrypt-web --restart unless-stopped \
|
||||
-v /srv/metacrypt:/srv/metacrypt \
|
||||
-p 127.0.0.1:18080:8080 \
|
||||
mcr.svc.mcp.metacircular.net:8443/metacrypt-web:v1.4.1 \
|
||||
--config /srv/metacrypt/metacrypt.toml
|
||||
```
|
||||
|
||||
**If Metacrypt fails with "chmod" or "readonly database":**
|
||||
Same fix as MCR — delete the database files. **Warning:** this loses
|
||||
all encrypted secrets, issued certs tracking, and CA state. The CA
|
||||
key itself is in the sealed vault (password-protected), not in SQLite.
|
||||
|
||||
### Stage 3: MCP Master
|
||||
|
||||
```bash
|
||||
podman run -d --name mcp-master --restart unless-stopped \
|
||||
--network host \
|
||||
-v /srv/mcp-master:/srv/mcp-master \
|
||||
mcr.svc.mcp.metacircular.net:8443/mcp-master:v0.10.3 \
|
||||
server --config /srv/mcp-master/mcp-master.toml
|
||||
```
|
||||
|
||||
**Verify:**
|
||||
```bash
|
||||
# From vade (operator workstation):
|
||||
mcp node list
|
||||
# Should show rift, svc, orion
|
||||
```
|
||||
|
||||
### Stage 4: Application Services
|
||||
|
||||
Once the master is running, deploy applications through MCP:
|
||||
|
||||
```bash
|
||||
mcp deploy mcq --direct
|
||||
mcp deploy mcdoc --direct
|
||||
mcp deploy mcat --direct
|
||||
mcp deploy kls --direct
|
||||
```
|
||||
|
||||
Or start them manually:
|
||||
|
||||
```bash
|
||||
# MCQ
|
||||
podman run -d --name mcq --restart unless-stopped \
|
||||
-v /srv/mcq:/srv/mcq \
|
||||
-p 127.0.0.1:48080:8080 -p 100.95.252.120:48080:8080 \
|
||||
mcr.svc.mcp.metacircular.net:8443/mcq:v0.4.2 \
|
||||
server --config /srv/mcq/mcq.toml
|
||||
|
||||
# MCDoc
|
||||
podman run -d --name mcdoc --restart unless-stopped \
|
||||
-v /srv/mcdoc:/srv/mcdoc \
|
||||
-p 127.0.0.1:38080:8080 \
|
||||
mcr.svc.mcp.metacircular.net:8443/mcdoc:v0.1.0 \
|
||||
server --config /srv/mcdoc/mcdoc.toml
|
||||
|
||||
# MCAT
|
||||
podman run -d --name mcat --restart unless-stopped \
|
||||
-v /srv/mcat:/srv/mcat \
|
||||
-p 127.0.0.1:48116:8443 \
|
||||
mcr.svc.mcp.metacircular.net:8443/mcat:v1.2.0 \
|
||||
server --config /srv/mcat/mcat.toml
|
||||
|
||||
# KLS
|
||||
podman run -d --name kls --restart unless-stopped \
|
||||
-v /srv/kls:/srv/kls \
|
||||
-p 127.0.0.1:58080:8080 -p 100.95.252.120:58080:8080 \
|
||||
mcr.svc.mcp.metacircular.net:8443/kls:v0.2.0 \
|
||||
-f /srv/kls/kls.conf
|
||||
|
||||
# Sgard
|
||||
podman run -d --name sgardd --restart unless-stopped \
|
||||
-v /srv/sgard:/srv/sgard \
|
||||
-p 127.0.0.1:19473:9473 \
|
||||
mcr.svc.mcp.metacircular.net:8443/sgardd:v3.2.0 \
|
||||
--repo /srv/sgard --authorized-keys /srv/sgard/authorized_keys \
|
||||
--tls-cert /srv/sgard/certs/sgard.pem --tls-key /srv/sgard/certs/sgard.key
|
||||
```
|
||||
|
||||
## Verification Checklist
|
||||
|
||||
After all services are running:
|
||||
|
||||
```bash
|
||||
# Fleet status
|
||||
mcp ps
|
||||
# All services should show "running"
|
||||
|
||||
# DNS
|
||||
dig @192.168.88.181 google.com +short
|
||||
dig @192.168.88.181 mcq.svc.mcp.metacircular.net +short
|
||||
|
||||
# MCIAS (runs on svc, should be unaffected by rift outage)
|
||||
curl -sk https://mcias.metacircular.net:8443/v1/health
|
||||
|
||||
# MCR
|
||||
curl -sk https://mcr.svc.mcp.metacircular.net:8443/v2/
|
||||
|
||||
# Metacrypt
|
||||
curl -sk https://metacrypt.svc.mcp.metacircular.net:8443/v1/health
|
||||
|
||||
# Public routes via svc
|
||||
curl -sk https://mcq.metacircular.net/
|
||||
curl -sk https://docs.metacircular.net/
|
||||
```
|
||||
|
||||
## Common Errors
|
||||
|
||||
### "chmod: operation not permitted"
|
||||
|
||||
modernc.org/sqlite calls `fchmod()` on database files. This is denied
|
||||
inside rootless podman user namespaces. Fix:
|
||||
|
||||
```bash
|
||||
# Delete the database and let the service recreate it
|
||||
podman stop <container>
|
||||
rm -f /srv/<service>/<service>.db*
|
||||
podman start <container>
|
||||
```
|
||||
|
||||
The `fchmod` error will still appear in logs as a warning but is
|
||||
non-fatal for newly created databases.
|
||||
|
||||
### "address already in use" on port 53
|
||||
|
||||
systemd-resolved holds port 53 on localhost. MCNS must bind to
|
||||
specific IPs, not `0.0.0.0:53`. Use explicit port bindings:
|
||||
`-p 192.168.88.181:53:53 -p 100.95.252.120:53:53`
|
||||
|
||||
### "connection refused" to MCR
|
||||
|
||||
MCR is down. Images are cached locally — you can start services that
|
||||
use cached images without MCR. MCR itself starts from its cached
|
||||
image.
|
||||
|
||||
### Agent shows "error" for all nodes
|
||||
|
||||
Check:
|
||||
1. Tailscale is running on both the CLI machine and the target node
|
||||
2. The agent is listening: `ss -tlnp | grep 9444`
|
||||
3. The CLI config has the correct addresses
|
||||
4. TLS certs have the right SANs for the Tailnet IP
|
||||
|
||||
### "podman: executable file not found"
|
||||
|
||||
This warning appears for svc (which uses Docker, not podman). It's
|
||||
benign — svc is an edge node that doesn't run containers.
|
||||
|
||||
## Cold Start (No Cached Images)
|
||||
|
||||
If the machine was wiped and no images are cached:
|
||||
|
||||
1. **MCIAS** runs on svc (Docker container), not rift. It should be
|
||||
unaffected by a rift failure. Verify: `ssh svc.metacircular.net
|
||||
"docker ps | grep mcias"`.
|
||||
|
||||
2. **Pre-stage images** by pulling from a backup or building locally:
|
||||
```bash
|
||||
# On vade (operator workstation), build and push to a temp location
|
||||
cd ~/src/metacircular/mcns && make docker
|
||||
podman save mcr.svc.mcp.metacircular.net:8443/mcns:v1.2.0 | \
|
||||
ssh rift "podman load"
|
||||
```
|
||||
Repeat for each service.
|
||||
|
||||
3. Alternatively, if another node has MCR access, push images there
|
||||
first, then pull from the running MCR instance.
|
||||
|
||||
## Service Reference
|
||||
|
||||
Quick reference for all services, their images, and critical flags:
|
||||
|
||||
| Service | Image | Network | Key Ports | Config Path |
|
||||
|---------|-------|---------|-----------|-------------|
|
||||
| mcns | mcns:v1.2.0 | bridge | 53/tcp+udp, 38443→8443 | /srv/mcns/mcns.toml |
|
||||
| mc-proxy | mc-proxy:v1.2.2 | host | 443, 8443, 9443 | /srv/mc-proxy/mc-proxy.toml |
|
||||
| mcr (api) | mcr:v1.2.1 | bridge | 28443→8443, 29443→9443 | /srv/mcr/mcr.toml |
|
||||
| mcr (web) | mcr-web:v1.3.2 | bridge | 28080→8080 | /srv/mcr/mcr.toml |
|
||||
| metacrypt (api) | metacrypt:v1.3.1 | bridge | 18443→8443, 19443→9443 | /srv/metacrypt/metacrypt.toml |
|
||||
| metacrypt (web) | metacrypt-web:v1.4.1 | bridge | 18080→8080 | /srv/metacrypt/metacrypt.toml |
|
||||
| mcp-master | mcp-master:v0.10.3 | host | 9555 | /srv/mcp-master/mcp-master.toml |
|
||||
| mcq | mcq:v0.4.2 | bridge | 48080→8080 | /srv/mcq/mcq.toml |
|
||||
| mcdoc | mcdoc:v0.1.0 | bridge | 38080→8080 | /srv/mcdoc/mcdoc.toml |
|
||||
| mcat | mcat:v1.2.0 | bridge | 48116→8443 | /srv/mcat/mcat.toml |
|
||||
| kls | kls:v0.2.0 | bridge | 58080→8080 | /srv/kls/kls.conf |
|
||||
| sgard | sgardd:v3.2.0 | bridge | 19473→9473 | (flags, see above) |
|
||||
|
||||
All images are prefixed with `mcr.svc.mcp.metacircular.net:8443/`.
|
||||
@@ -1,244 +0,0 @@
|
||||
# Hypervisor-Based Service Isolation -- Design Notes
|
||||
|
||||
> **Status**: Brainstorming / future direction. This document is NOT
|
||||
> an active work item. Agents should ignore this document unless
|
||||
> specifically asked to consider it.
|
||||
|
||||
## Context
|
||||
|
||||
The metacircular platform runs Go services as rootless podman
|
||||
containers orchestrated by MCP. This is a pragmatic execution of ideas
|
||||
originally explored in a series of 2015 papers on security kernels,
|
||||
environment isolation, and unikernels (see References). Those papers
|
||||
describe a richer model than what containers provide: hardware-enforced
|
||||
isolation, mandatory inter-environment communication mediation, and
|
||||
capability-based access control. This document explores bridging the
|
||||
gap by running services as unikernel VMs (specifically Nanos) on the
|
||||
MCP control plane.
|
||||
|
||||
## Motivation: What Containers Don't Give Us
|
||||
|
||||
The current platform has the W7 security kernel's three properties --
|
||||
isolated environments, inter-environment communication (IEC), and
|
||||
access mediation -- but implemented cooperatively rather than enforced:
|
||||
|
||||
| W7 Property | Metacircular Today | Enforcement |
|
||||
|---|---|---|
|
||||
| Isolated environments | Rootless podman (namespaces/cgroups) | OS-cooperative -- shared kernel, escape CVEs exist |
|
||||
| IEC | gRPC/TLS through mc-proxy | Application-cooperative -- services *choose* to route through mc-proxy |
|
||||
| Access mediation | MCIAS tokens + per-service policies | Application-level -- services check tokens voluntarily |
|
||||
|
||||
The topology is right. The enforcement mechanism is weak. Containers
|
||||
share a kernel, and any service could bypass mc-proxy to reach the
|
||||
Tailnet directly.
|
||||
|
||||
## What Unikernels Buy Us
|
||||
|
||||
### Hardware-Enforced Isolation
|
||||
|
||||
Each service runs in its own VM with its own kernel. There is no
|
||||
shared kernel to escape from. The security boundary is the hypervisor
|
||||
(KVM), not Linux namespaces. This is the W7 "isolated environments"
|
||||
model enforced by hardware, not convention.
|
||||
|
||||
### Mandatory IEC
|
||||
|
||||
This is the subtle but powerful part. A container on the Tailnet can
|
||||
talk to anything. A unikernel VM with no direct network interface --
|
||||
only a virtio-net device connected to a host-only bridge that the
|
||||
agent controls -- cannot bypass the mediation layer. If the agent is
|
||||
the only Tailnet citizen on the node and VMs can only reach the
|
||||
agent's bridge, then mc-proxy stops being a routing convenience and
|
||||
becomes the IEC mechanism. Communication between environments is
|
||||
mediated by design, not by trust.
|
||||
|
||||
### Reduced TCB
|
||||
|
||||
Container TCB: Linux kernel + podman runtime + container image (often
|
||||
a full distro). Unikernel TCB: KVM + Nanos runtime + the Go binary.
|
||||
No shell, no package manager, no multi-user, no unnecessary syscalls.
|
||||
|
||||
## The Agent as Security Kernel
|
||||
|
||||
The MCP agent is already structurally positioned to be the W7 security
|
||||
kernel for its node. It manages environment lifecycle, controls the
|
||||
IEC layer (mc-proxy routes), provisions credentials (Metacrypt certs),
|
||||
and reports to a central authority (master).
|
||||
|
||||
With unikernels, this role is formalized. The agent becomes the only
|
||||
entity with host access. Services exist in VMs that can only
|
||||
communicate through agent-controlled channels:
|
||||
|
||||
- **Network access**: virtio-net bridge under agent control; the agent
|
||||
decides what each VM can reach.
|
||||
- **Storage access**: 9p/virtio-fs mounts; the agent controls what
|
||||
each VM sees on disk.
|
||||
- **Credential access**: the agent provisions certs into the VM's
|
||||
filesystem before boot.
|
||||
- **Identity**: the agent attests to the master what image hash is
|
||||
running in each VM.
|
||||
|
||||
## Why This Is Feasible for Metacircular
|
||||
|
||||
Several properties of the existing platform make this tractable:
|
||||
|
||||
- **Go + CGO_ENABLED=0**: Every service already produces a static ELF
|
||||
binary. Nanos needs exactly this. The `ops` tool packages them with
|
||||
minimal friction.
|
||||
|
||||
- **Single-process services**: Each service is one Go binary -- no
|
||||
sidecars, no shell scripts, no multi-process orchestration. That is
|
||||
the unikernel sweet spot.
|
||||
|
||||
- **Single-operator trust domain**: No multi-tenant capability
|
||||
delegation or federated attestation needed. The agent is the
|
||||
security kernel for its node; the master is the coordination point.
|
||||
|
||||
- **mc-proxy already mediates traffic**: The routing mesh is already
|
||||
in place. Making it mandatory (rather than optional) for unikernel
|
||||
VMs is an incremental change, not a new system.
|
||||
|
||||
## Design Sketch
|
||||
|
||||
### Runtime Abstraction
|
||||
|
||||
The agent gains a `Runtime` interface. Podman is one implementation;
|
||||
QEMU/KVM is another. Service definitions gain a `runtime` field:
|
||||
|
||||
```toml
|
||||
name = "mcq"
|
||||
runtime = "unikernel" # or "container" (default)
|
||||
tier = "worker"
|
||||
```
|
||||
|
||||
Both runtimes coexist. Services can be converted incrementally.
|
||||
|
||||
### Networking: Host-Only Bridge
|
||||
|
||||
Each unikernel VM gets a virtio-net device on a host-only bridge. The
|
||||
agent runs on the bridge and controls forwarding. VMs cannot reach the
|
||||
Tailnet directly. All external communication flows through mc-proxy on
|
||||
the host.
|
||||
|
||||
This is structurally similar to how rootless podman already works
|
||||
(container ports are localhost-only, mc-proxy routes to them), but
|
||||
with the enforcement moved from convention to network topology.
|
||||
|
||||
### Storage: 9p Passthrough
|
||||
|
||||
Unikernel VMs mount `/srv/<service>/` via QEMU's `-virtfs` 9p
|
||||
passthrough. Writes go directly to the host filesystem. This makes
|
||||
snapshots work the same way as containers -- the agent tars the host
|
||||
directory.
|
||||
|
||||
### Image Building
|
||||
|
||||
Two options (not mutually exclusive):
|
||||
|
||||
1. **Build on agent**: Agent extracts the ELF binary from the OCI
|
||||
image (pulled from MCR) and runs `ops build` locally.
|
||||
2. **Store unikernel images in MCR**: OCI supports arbitrary media
|
||||
types. Unikernel `.img` files could be stored as OCI artifacts.
|
||||
|
||||
Option 1 is simpler to start with. Option 2 is cleaner long-term.
|
||||
|
||||
### Image Attestation
|
||||
|
||||
Before booting a unikernel, the agent hashes the image and reports it
|
||||
to the master. The master compares against expected hashes from the
|
||||
service definition. This is software attestation -- not TPM-based, but
|
||||
it closes the "is this what I deployed?" question. It is a stepping
|
||||
stone toward measured boot with hardware TPM.
|
||||
|
||||
### Snapshot Constraints
|
||||
|
||||
Unikernels have no shell. The `cli` and `exec:` snapshot methods
|
||||
don't work. Only `grpc` snapshots are viable for unikernel services
|
||||
(the service implements the standard `SnapshotService` RPC). The
|
||||
default snapshot method (tar config/db/certs from the host-side 9p
|
||||
mount) works unchanged since the agent tars the host directory, not
|
||||
the VM filesystem.
|
||||
|
||||
### Debugging
|
||||
|
||||
No `podman exec`, no shell. Debugging relies on:
|
||||
|
||||
- Serial console output from QEMU
|
||||
- gRPC health/status endpoints
|
||||
- Structured logging to a file on the 9p mount
|
||||
- The agent can snapshot and inspect VM state
|
||||
|
||||
This is a real loss of convenience. It is the price of proper
|
||||
isolation -- as noted in the 2015 hypervisor paper, "the nature of
|
||||
debugging means that isolation is broken."
|
||||
|
||||
## Difficulty Assessment
|
||||
|
||||
| Aspect | Difficulty | Notes |
|
||||
|---|---|---|
|
||||
| Building unikernel images from Go binaries | Easy | Already static ELF, `ops` handles it |
|
||||
| QEMU lifecycle management in agent | Medium | Replace podman calls with qemu-system calls |
|
||||
| Networking (host-only bridge + mc-proxy) | Medium | Similar to rootless podman model |
|
||||
| Persistent storage via 9p | Medium | Well-supported in QEMU, maps to existing `/srv/` layout |
|
||||
| Snapshots | Medium | `grpc` method works; `cli`/`exec` don't |
|
||||
| Image attestation | Medium-Low | SHA-256 of image before boot |
|
||||
| mc-proxy integration | Low | Just needs a reachable IP:port |
|
||||
| Debugging/observability | Annoying | Loss of exec/shell access |
|
||||
|
||||
The minimum meaningful change is the runtime abstraction + isolated
|
||||
networking together. Running a unikernel with full Tailnet access is
|
||||
just a heavier container with worse debugging. The isolation properties
|
||||
only kick in when the agent mediates all communication.
|
||||
|
||||
## Progression Path
|
||||
|
||||
1. **Runtime abstraction in the agent.** `Runtime` interface with
|
||||
podman and qemu implementations. Service definitions gain a
|
||||
`runtime` field. Both coexist.
|
||||
|
||||
2. **Isolated networking for unikernel VMs.** Host-only bridge per
|
||||
node, agent controls forwarding. mc-proxy becomes the mandatory
|
||||
IEC layer for unikernel services.
|
||||
|
||||
3. **Image attestation.** Agent hashes images before boot, reports to
|
||||
master. Master compares against expected values.
|
||||
|
||||
4. **Capability tokens (longer-term).** MCIAS issues operation-scoped
|
||||
tokens instead of identity tokens. The agent's mediation layer
|
||||
enforces them at the network boundary. This is independent of
|
||||
unikernels but synergizes with mandatory mediation.
|
||||
|
||||
## Open Questions
|
||||
|
||||
- **Tailscale integration**: Should unikernel VMs ever be first-class
|
||||
Tailnet citizens (via tsnet compiled into the binary), or should the
|
||||
agent always mediate? Mandatory mediation is more secure but means
|
||||
the agent is on the critical path for all traffic.
|
||||
|
||||
- **Resource limits**: QEMU VMs need explicit memory and CPU
|
||||
allocation. The current container model doesn't declare resource
|
||||
requirements. Unikernels would force this.
|
||||
|
||||
- **Mixed fleet**: During transition, some services run as containers
|
||||
and some as unikernels. mc-proxy routes to both. Does the master
|
||||
need to know the runtime type for placement decisions?
|
||||
|
||||
- **ARM support**: Nanos supports aarch64 but the QEMU/KVM story on
|
||||
Raspberry Pi (no KVM on all models) may limit unikernels to amd64
|
||||
nodes.
|
||||
|
||||
## References
|
||||
|
||||
- Rees, J. "A Security Kernel Based on the Lambda Calculus" (W7
|
||||
security kernel model -- isolated environments, IEC, access
|
||||
mediation)
|
||||
- "Containers, isolation, and operating systems for network spaces"
|
||||
(2015) -- argues the OS must provide a security kernel; unikernels
|
||||
as viable isolation mechanism
|
||||
- "A hypervisor for the modern age" (2015) -- problem statement for a
|
||||
hypervisor providing proper isolation, IEC, and access mediation
|
||||
with a programmatic administrative interface
|
||||
- "A content-addressable data store with object capabilities" (Nebula,
|
||||
2015) -- capability-based access control model
|
||||
- MCP v2 Architecture (`docs/architecture-v2.md`) -- current platform
|
||||
design this document builds on
|
||||
@@ -1,852 +0,0 @@
|
||||
# Unikernel Migration Plan
|
||||
|
||||
> **Status**: Detailed work plan. Not an active work item. Agents
|
||||
> should ignore this document unless specifically asked to consider it.
|
||||
>
|
||||
> **Prerequisite**: MCP v2 phase 6 complete -- master running, agents
|
||||
> on all nodes, edge routing, snapshots, and migration all operational.
|
||||
|
||||
## Starting Point
|
||||
|
||||
The MCP agent already has a `runtime.Runtime` interface
|
||||
(`mcp/internal/runtime/runtime.go`) with methods for Pull, Run, Stop,
|
||||
Remove, Inspect, List, Build, Push, ImageExists, and Logs. The only
|
||||
implementation is `Podman` (`mcp/internal/runtime/podman.go`). The
|
||||
agent struct holds `Runtime runtime.Runtime` and all lifecycle
|
||||
operations (deploy, stop, start, undeploy, status) go through this
|
||||
interface.
|
||||
|
||||
This means the runtime abstraction layer is already in place. The
|
||||
migration is primarily: implement a QEMU/Nanos backend for the
|
||||
existing interface, add isolated networking, and extend service
|
||||
definitions with runtime-specific fields.
|
||||
|
||||
## Terminology
|
||||
|
||||
| Term | Meaning |
|
||||
|------|---------|
|
||||
| **VM** | A QEMU/KVM virtual machine running a Nanos unikernel |
|
||||
| **bridge** | A Linux bridge device (`mcp-br0`) on the host for VM networking |
|
||||
| **TAP** | A TAP device attached to the bridge, one per VM |
|
||||
| **9p mount** | QEMU's `-virtfs` passthrough for host directory access |
|
||||
| **ops** | The Nanos toolchain CLI for building unikernel images |
|
||||
|
||||
---
|
||||
|
||||
## Phase 1: QEMU Runtime Implementation
|
||||
|
||||
**Goal**: A second `runtime.Runtime` implementation that can start and
|
||||
stop Nanos unikernel VMs with basic networking. No isolation
|
||||
enforcement yet -- VMs get host-forwarded ports like containers do.
|
||||
|
||||
### 1.1 NixOS Host Prerequisites
|
||||
|
||||
Add QEMU/KVM packages to the NixOS configuration on rift and orion.
|
||||
svc (Debian) gets equivalent packages via apt.
|
||||
|
||||
Required on all nodes that will run unikernels:
|
||||
|
||||
- `qemu` (specifically `qemu-system-x86_64`)
|
||||
- `ops` CLI (Nanos toolchain) -- install from GitHub release or build
|
||||
from source
|
||||
- KVM access: the `mcp` user needs `/dev/kvm` access. On NixOS, add
|
||||
the user to the `kvm` group. On Debian, same.
|
||||
- `bridge-utils` or `iproute2` for bridge management (Phase 2)
|
||||
|
||||
Verify KVM works: `qemu-system-x86_64 -enable-kvm -nographic
|
||||
-no-reboot` should boot and exit.
|
||||
|
||||
**Deliverable**: All amd64 nodes can run QEMU with KVM acceleration.
|
||||
RPi nodes (arm64, no KVM) are excluded from unikernel support.
|
||||
|
||||
### 1.2 Image Building Pipeline
|
||||
|
||||
The agent needs to produce a Nanos `.img` file from a Go binary. Two
|
||||
paths, implemented in order:
|
||||
|
||||
**1.2a -- Local build from OCI image (initial approach)**
|
||||
|
||||
The agent already pulls OCI images via `Runtime.Pull()`. For
|
||||
unikernels:
|
||||
|
||||
1. Pull the OCI image from MCR (reuse existing podman pull or use
|
||||
`skopeo copy` to a local directory).
|
||||
2. Extract the ELF binary from the image. Convention: the binary is at
|
||||
`/usr/local/bin/<service>` in the image (same path the Dockerfiles
|
||||
use).
|
||||
3. Run `ops build <binary> -c <config.json>` to produce a `.img` file.
|
||||
4. Store the image at `/srv/mcp/images/<service>-<component>.img`.
|
||||
|
||||
The `ops` config JSON specifies:
|
||||
|
||||
```json
|
||||
{
|
||||
"Args": ["server", "--config", "/srv/mcq/mcq.toml"],
|
||||
"Dirs": ["srv"],
|
||||
"Mounts": {
|
||||
"/srv/<service>": "/srv/<service>"
|
||||
},
|
||||
"ManifestPassthrough": {
|
||||
"mem": "256m",
|
||||
"smp": 1
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**1.2b -- Pre-built unikernel images in MCR (later)**
|
||||
|
||||
Store `.img` files as OCI artifacts in MCR with a distinct media type
|
||||
(`application/vnd.metacircular.unikernel.nanos.v1`). The agent pulls
|
||||
the artifact and writes it directly to
|
||||
`/srv/mcp/images/<service>-<component>.img`. This skips the
|
||||
extract-and-build step and ensures the deployed image is identical to
|
||||
what was built.
|
||||
|
||||
MCR already stores OCI artifacts; this requires adding the media type
|
||||
to MCR's accepted list and adding an `mcp build --unikernel` command
|
||||
that builds the image locally and pushes it.
|
||||
|
||||
**Deliverable**: Agent can produce a bootable Nanos image from an
|
||||
existing OCI container image.
|
||||
|
||||
### 1.3 QEMU Runtime Type
|
||||
|
||||
Implement `QEMURuntime` satisfying `runtime.Runtime`:
|
||||
|
||||
```go
|
||||
type QEMURuntime struct {
|
||||
imageDir string // /srv/mcp/images/
|
||||
stateDir string // /srv/mcp/vm-state/
|
||||
opsPath string // path to ops binary
|
||||
qemuPath string // path to qemu-system-x86_64
|
||||
logger *slog.Logger
|
||||
mu sync.Mutex
|
||||
vms map[string]*vmState // name → running VM state
|
||||
}
|
||||
|
||||
type vmState struct {
|
||||
pid int
|
||||
qmpSocket string // QMP control socket
|
||||
serial string // serial console log path
|
||||
ip string // VM IP on bridge (Phase 2)
|
||||
ports map[int]int // guest port → host port
|
||||
}
|
||||
```
|
||||
|
||||
**Method mapping:**
|
||||
|
||||
| Runtime Method | QEMU Implementation |
|
||||
|---|---|
|
||||
| `Pull(image)` | Pull OCI image, extract ELF, run `ops build`, store `.img` |
|
||||
| `Run(spec)` | Start `qemu-system-x86_64` with KVM, virtio-net, 9p mounts, QMP socket |
|
||||
| `Stop(name)` | Send `system_powerdown` via QMP, wait 10s, then SIGKILL |
|
||||
| `Remove(name)` | Kill process if running, remove state files |
|
||||
| `Inspect(name)` | Check process liveness + read QMP status |
|
||||
| `List()` | Enumerate `/srv/mcp/vm-state/*/qemu.pid`, check liveness |
|
||||
| `Build(...)` | Not applicable for unikernels (image built during Pull) |
|
||||
| `Push(...)` | Not applicable (future: push `.img` to MCR as OCI artifact) |
|
||||
| `ImageExists(image)` | Check if `.img` file exists in imageDir |
|
||||
| `Logs(name)` | Read serial console log file |
|
||||
|
||||
**QEMU invocation** (Phase 1 -- user-mode networking with port
|
||||
forwards, no bridge yet):
|
||||
|
||||
```
|
||||
qemu-system-x86_64 \
|
||||
-enable-kvm \
|
||||
-m 256 \
|
||||
-smp 1 \
|
||||
-nographic \
|
||||
-serial file:/srv/mcp/vm-state/<name>/console.log \
|
||||
-qmp unix:/srv/mcp/vm-state/<name>/qmp.sock,server,nowait \
|
||||
-drive file=/srv/mcp/images/<name>.img,format=raw,if=virtio \
|
||||
-virtfs local,path=/srv/<service>,mount_tag=srvdata,security_model=mapped-xattr,id=srvdata \
|
||||
-device virtio-net-pci,netdev=net0 \
|
||||
-netdev user,id=net0,hostfwd=tcp:127.0.0.1:<host_port>-:<guest_port>
|
||||
```
|
||||
|
||||
This gives user-mode networking with port forwards to localhost --
|
||||
functionally identical to how rootless podman works. mc-proxy routes
|
||||
to `127.0.0.1:<host_port>` the same way it does for containers.
|
||||
|
||||
**Deliverable**: A `QEMURuntime` that passes the same interface as
|
||||
`Podman`. Agent can deploy, stop, inspect, and undeploy unikernel
|
||||
services using QEMU user-mode networking.
|
||||
|
||||
### 1.4 Service Definition Changes
|
||||
|
||||
Add `runtime` field to service definitions, proto specs, and registry
|
||||
schema.
|
||||
|
||||
**TOML** (`servicedef.go`):
|
||||
|
||||
```toml
|
||||
name = "mcq"
|
||||
runtime = "unikernel"
|
||||
tier = "worker"
|
||||
active = true
|
||||
|
||||
[[components]]
|
||||
name = "mcq"
|
||||
image = "mcr.svc.mcp.metacircular.net:8443/mcq:v0.4.0"
|
||||
memory = 256 # MB, required for unikernels
|
||||
vcpus = 1 # default 1
|
||||
volumes = ["/srv/mcq:/srv/mcq"]
|
||||
cmd = ["server", "--config", "/srv/mcq/mcq.toml"]
|
||||
```
|
||||
|
||||
**Proto** (`mcp.proto`):
|
||||
|
||||
```protobuf
|
||||
message ServiceSpec {
|
||||
string name = 1;
|
||||
bool active = 2;
|
||||
repeated ComponentSpec components = 3;
|
||||
string comment = 4;
|
||||
string runtime = 5; // "container" (default) or "unikernel"
|
||||
}
|
||||
|
||||
message ComponentSpec {
|
||||
// ... existing fields ...
|
||||
int32 memory_mb = 11; // required for unikernel runtime
|
||||
int32 vcpus = 12; // default 1
|
||||
}
|
||||
```
|
||||
|
||||
**Registry schema** (new migration):
|
||||
|
||||
```sql
|
||||
ALTER TABLE components ADD COLUMN runtime TEXT NOT NULL DEFAULT 'container';
|
||||
ALTER TABLE components ADD COLUMN memory_mb INTEGER NOT NULL DEFAULT 0;
|
||||
ALTER TABLE components ADD COLUMN vcpus INTEGER NOT NULL DEFAULT 1;
|
||||
```
|
||||
|
||||
**Agent runtime selection**: In `agent.go`, the agent holds both
|
||||
runtimes:
|
||||
|
||||
```go
|
||||
type Agent struct {
|
||||
// ... existing fields ...
|
||||
ContainerRuntime runtime.Runtime // podman
|
||||
UnikernelRuntime runtime.Runtime // qemu (nil if not configured)
|
||||
}
|
||||
|
||||
func (a *Agent) runtimeFor(comp *registry.Component) runtime.Runtime {
|
||||
if comp.Runtime == "unikernel" {
|
||||
return a.UnikernelRuntime
|
||||
}
|
||||
return a.ContainerRuntime
|
||||
}
|
||||
```
|
||||
|
||||
All lifecycle operations call `a.runtimeFor(comp)` instead of
|
||||
`a.Runtime` directly.
|
||||
|
||||
**Validation rules**:
|
||||
- `runtime = "unikernel"` requires `memory_mb > 0`.
|
||||
- `runtime = "unikernel"` requires the node to have KVM
|
||||
(`/dev/kvm` exists). Agent rejects deploys on nodes without KVM.
|
||||
- `runtime = "unikernel"` is incompatible with `exec:` and `cli`
|
||||
snapshot methods. Validation rejects these combinations.
|
||||
|
||||
**Deliverable**: Service definitions can declare `runtime =
|
||||
"unikernel"`. The agent selects the correct runtime per component.
|
||||
Container services are completely unaffected.
|
||||
|
||||
### 1.5 Resource Tracking
|
||||
|
||||
The agent needs to track allocated VM resources to avoid overcommit.
|
||||
The v2 heartbeat already reports CPU, memory, and disk. Add tracking
|
||||
of allocated-to-VMs resources:
|
||||
|
||||
```go
|
||||
type ResourceTracker struct {
|
||||
mu sync.Mutex
|
||||
totalMemMB int64 // from /proc/meminfo
|
||||
totalCPUs int32 // from runtime.NumCPU()
|
||||
allocMemMB int64 // sum of running VM memory_mb
|
||||
allocCPUs int32 // sum of running VM vcpus
|
||||
}
|
||||
|
||||
func (r *ResourceTracker) CanFit(memMB int64, vcpus int32) bool
|
||||
func (r *ResourceTracker) Allocate(memMB int64, vcpus int32)
|
||||
func (r *ResourceTracker) Release(memMB int64, vcpus int32)
|
||||
```
|
||||
|
||||
The master's placement algorithm gains a resource check: before
|
||||
placing a unikernel service on a node, verify the node has enough
|
||||
unallocated memory and CPUs. Container services continue to use
|
||||
container-count placement.
|
||||
|
||||
**Deliverable**: Agent tracks VM resource allocation. Master rejects
|
||||
placements that would overcommit a node.
|
||||
|
||||
### 1.6 Phase 1 Validation
|
||||
|
||||
Deploy a test service (a minimal Go HTTP server, not a real platform
|
||||
service) as a unikernel:
|
||||
|
||||
1. Build a trivial Go binary that serves HTTP on port 8080.
|
||||
2. Package it as an OCI image, push to MCR.
|
||||
3. Write a service definition with `runtime = "unikernel"`.
|
||||
4. `mcp deploy test-unikernel` -- verify it starts, mc-proxy routes
|
||||
to it, health checks pass.
|
||||
5. `mcp undeploy test-unikernel` -- verify clean shutdown.
|
||||
6. Verify container services are completely unaffected.
|
||||
|
||||
**Phase 1 complete when**: A unikernel service can be deployed,
|
||||
health-checked, and undeployed through the normal `mcp deploy`/
|
||||
`mcp undeploy` flow, alongside running container services.
|
||||
|
||||
---
|
||||
|
||||
## Phase 2: Isolated Networking
|
||||
|
||||
**Goal**: Replace QEMU user-mode networking with a host-only bridge.
|
||||
VMs can only communicate through mc-proxy. This is the phase that
|
||||
delivers the security properties -- without it, unikernels are just
|
||||
heavier containers.
|
||||
|
||||
### 2.1 Bridge Setup
|
||||
|
||||
Create a persistent Linux bridge on each unikernel-capable node:
|
||||
|
||||
**NixOS** (`networking.bridges` in NixOS config):
|
||||
|
||||
```nix
|
||||
networking.bridges.mcp-br0.interfaces = [];
|
||||
networking.interfaces.mcp-br0.ipv4.addresses = [{
|
||||
address = "10.99.0.1";
|
||||
prefixLength = 24;
|
||||
}];
|
||||
```
|
||||
|
||||
**Debian** (svc -- if svc ever runs unikernels, which is unlikely
|
||||
given its edge role, but document for completeness):
|
||||
|
||||
```
|
||||
# /etc/network/interfaces.d/mcp-br0
|
||||
auto mcp-br0
|
||||
iface mcp-br0 inet static
|
||||
address 10.99.0.1/24
|
||||
bridge_ports none
|
||||
bridge_stp off
|
||||
```
|
||||
|
||||
The bridge uses the `10.99.0.0/24` subnet. This is a host-only
|
||||
network -- no default route, no NAT to the internet or Tailnet. VMs
|
||||
can only reach `10.99.0.1` (the agent/mc-proxy host).
|
||||
|
||||
**Deliverable**: Each unikernel-capable node has a `mcp-br0` bridge
|
||||
with address `10.99.0.1/24`.
|
||||
|
||||
### 2.2 TAP Device Management
|
||||
|
||||
Each VM gets a TAP device attached to the bridge. The agent creates
|
||||
and destroys TAP devices as part of the VM lifecycle:
|
||||
|
||||
```go
|
||||
func (q *QEMURuntime) createTAP(name string) (string, error) {
|
||||
tapName := fmt.Sprintf("tap-%s", name) // max 15 chars for IFNAMSIZ
|
||||
// ip tuntap add dev <tap> mode tap user mcp
|
||||
// ip link set <tap> master mcp-br0
|
||||
// ip link set <tap> up
|
||||
return tapName, nil
|
||||
}
|
||||
|
||||
func (q *QEMURuntime) destroyTAP(name string) error {
|
||||
tapName := fmt.Sprintf("tap-%s", name)
|
||||
// ip link del <tap>
|
||||
return nil
|
||||
}
|
||||
```
|
||||
|
||||
TAP creation requires `CAP_NET_ADMIN` or `ip tuntap` permissions for
|
||||
the `mcp` user. On NixOS, grant this via a udev rule or by running
|
||||
the agent with ambient capabilities:
|
||||
|
||||
```nix
|
||||
systemd.services.mcp-agent.serviceConfig.AmbientCapabilities = [
|
||||
"CAP_NET_ADMIN"
|
||||
];
|
||||
```
|
||||
|
||||
**QEMU invocation changes** (bridge networking replaces user-mode):
|
||||
|
||||
```
|
||||
qemu-system-x86_64 \
|
||||
... \
|
||||
-device virtio-net-pci,netdev=net0,mac=52:54:00:xx:xx:xx \
|
||||
-netdev tap,id=net0,ifname=tap-<name>,script=no,downscript=no
|
||||
```
|
||||
|
||||
Each VM gets a deterministic MAC address derived from the service
|
||||
name (e.g., SHA-256 of service name, take 5 bytes, prepend `52:54:00`).
|
||||
|
||||
### 2.3 VM IP Assignment
|
||||
|
||||
VMs need static IPs on the bridge. No DHCP server -- the agent
|
||||
assigns IPs and passes them to Nanos via the ops config.
|
||||
|
||||
```go
|
||||
type IPAllocator struct {
|
||||
mu sync.Mutex
|
||||
subnet net.IPNet // 10.99.0.0/24
|
||||
gateway net.IP // 10.99.0.1
|
||||
assigned map[string]net.IP // service name → IP
|
||||
next byte // next octet to try (2-254)
|
||||
}
|
||||
```
|
||||
|
||||
The ops config passes networking to Nanos:
|
||||
|
||||
```json
|
||||
{
|
||||
"RunConfig": {
|
||||
"IPAddress": "10.99.0.5",
|
||||
"NetMask": "255.255.255.0",
|
||||
"Gateway": "10.99.0.1"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Assigned IPs are persisted in the agent's registry:
|
||||
|
||||
```sql
|
||||
ALTER TABLE components ADD COLUMN vm_ip TEXT;
|
||||
```
|
||||
|
||||
**Deliverable**: Each VM gets a static IP on the bridge. The agent
|
||||
tracks assignments in its registry.
|
||||
|
||||
### 2.4 mc-proxy Route Update
|
||||
|
||||
With bridge networking, mc-proxy routes change from
|
||||
`127.0.0.1:<host_port>` to `10.99.0.<n>:<guest_port>`:
|
||||
|
||||
- L7 routes: mc-proxy terminates TLS, forwards to
|
||||
`10.99.0.<n>:<port>` (plaintext on the bridge).
|
||||
- L4 routes: mc-proxy passes through to `10.99.0.<n>:<port>` (TLS
|
||||
end-to-end).
|
||||
|
||||
The `ProxyRouter.RegisterRoutes()` method needs to use the VM's
|
||||
bridge IP instead of `127.0.0.1` for unikernel components. Port
|
||||
allocation changes: unikernel VMs expose their actual service port
|
||||
on the bridge (no random host port needed), so `host_port` equals
|
||||
the route's declared port.
|
||||
|
||||
### 2.5 Firewall Rules
|
||||
|
||||
The bridge must be locked down so VMs can only reach mc-proxy:
|
||||
|
||||
```bash
|
||||
# Allow established connections back to VMs
|
||||
iptables -A FORWARD -i mcp-br0 -o mcp-br0 -m state --state ESTABLISHED,RELATED -j ACCEPT
|
||||
|
||||
# Allow VMs to reach the host (mc-proxy) on the bridge IP
|
||||
iptables -A INPUT -i mcp-br0 -d 10.99.0.1 -j ACCEPT
|
||||
|
||||
# Block VM-to-VM traffic on the bridge
|
||||
ebtables -A FORWARD -i tap-+ -o tap-+ -j DROP
|
||||
|
||||
# Block VMs from reaching anything outside the bridge
|
||||
iptables -A FORWARD -i mcp-br0 ! -o mcp-br0 -j DROP
|
||||
```
|
||||
|
||||
These rules enforce mandatory mediation: VMs can reach the host
|
||||
(where mc-proxy listens) but nothing else. No Tailnet, no internet,
|
||||
no other VMs. All inter-service communication goes through mc-proxy.
|
||||
|
||||
On NixOS, these rules go in `networking.firewall` or
|
||||
`networking.nftables`. On Debian, `/etc/iptables/rules.v4`.
|
||||
|
||||
**Deliverable**: VMs are network-isolated. They can only reach
|
||||
mc-proxy on the host. VM-to-VM and VM-to-Tailnet traffic is blocked.
|
||||
|
||||
### 2.6 Phase 2 Validation
|
||||
|
||||
1. Deploy the test unikernel from Phase 1 with bridge networking.
|
||||
2. Verify mc-proxy routes to it via the bridge IP.
|
||||
3. From inside the VM (via the service's own gRPC or HTTP endpoint),
|
||||
attempt to reach a Tailnet IP directly -- must fail.
|
||||
4. Attempt to reach another VM on the bridge -- must fail.
|
||||
5. Verify the service can reach its dependencies (MCIAS, Metacrypt)
|
||||
only via mc-proxy on the host.
|
||||
6. Verify container services are completely unaffected by the bridge.
|
||||
|
||||
**Phase 2 complete when**: Unikernel VMs are fully network-isolated
|
||||
and can only communicate through mc-proxy. The agent enforces this
|
||||
structurally, not cooperatively.
|
||||
|
||||
---
|
||||
|
||||
## Phase 3: Snapshots and Observability
|
||||
|
||||
**Goal**: Ensure unikernel services participate in the snapshot and
|
||||
monitoring systems. Adapt debugging tools for the no-shell environment.
|
||||
|
||||
### 3.1 Snapshot Adaptation
|
||||
|
||||
The default snapshot method (tar `*.toml`, `*.db`, `*.pem` from the
|
||||
host-side `/srv/<service>/`) works unchanged for unikernels because
|
||||
the agent tars the host directory, not the VM filesystem. The 9p
|
||||
passthrough means writes from the VM appear on the host immediately.
|
||||
|
||||
The `grpc` snapshot method also works unchanged -- the agent calls the
|
||||
service's `SnapshotService.Snapshot` RPC over mc-proxy, which reaches
|
||||
the VM the same way any other gRPC call does.
|
||||
|
||||
**What doesn't work**: `cli` and `exec:` methods, because there is
|
||||
no shell inside the VM. Validation (from Phase 1.4) already rejects
|
||||
these combinations, but the snapshot scheduler should also log a
|
||||
warning if it encounters a unikernel service with an incompatible
|
||||
snapshot method.
|
||||
|
||||
**Deliverable**: Snapshots work for unikernel services using the
|
||||
default or `grpc` methods.
|
||||
|
||||
### 3.2 Serial Console Log Collection
|
||||
|
||||
QEMU writes serial console output to
|
||||
`/srv/mcp/vm-state/<name>/console.log`. The `Logs()` method on
|
||||
`QEMURuntime` reads this file. But the agent's `Logs` gRPC RPC
|
||||
currently streams from podman/journalctl.
|
||||
|
||||
Extend the `Logs` RPC to detect the component's runtime and read from
|
||||
the serial console log instead:
|
||||
|
||||
```go
|
||||
func (a *Agent) Logs(req *pb.LogsRequest, stream pb.McpAgent_LogsServer) error {
|
||||
comp := a.registryComponent(req)
|
||||
if comp.Runtime == "unikernel" {
|
||||
return a.streamSerialLog(comp, req, stream)
|
||||
}
|
||||
return a.streamContainerLog(comp, req, stream)
|
||||
}
|
||||
```
|
||||
|
||||
For Nanos, configure the Go binary's logging to write to stdout/stderr
|
||||
(which Nanos routes to the serial console). This is the default Go
|
||||
behavior, so no changes needed in the services themselves.
|
||||
|
||||
**Deliverable**: `mcp logs <service>` works for unikernel services,
|
||||
streaming the serial console output.
|
||||
|
||||
### 3.3 Health Check Adaptation
|
||||
|
||||
The v2 health check types (tcp, grpc, http) all work over the network
|
||||
and don't require shell access. No changes needed -- the agent's
|
||||
monitoring loop connects to the VM's port via mc-proxy or the bridge
|
||||
IP the same way it does for containers.
|
||||
|
||||
### 3.4 Drift Detection
|
||||
|
||||
The agent's `LiveCheck()` currently calls `Runtime.List()` and
|
||||
reconciles with the registry. The QEMU `List()` implementation
|
||||
enumerates running VMs by checking PIDs in
|
||||
`/srv/mcp/vm-state/*/qemu.pid`. This needs to be reliable:
|
||||
|
||||
- On agent restart, rebuild the `vms` map from the state directory.
|
||||
- QEMU processes started with `--daemonize` survive agent restarts.
|
||||
- The QMP socket reconnects on agent restart.
|
||||
|
||||
**Deliverable**: Drift detection works for unikernel VMs. Agent
|
||||
restart does not lose track of running VMs.
|
||||
|
||||
### 3.5 Phase 3 Validation
|
||||
|
||||
1. Deploy a unikernel service with `[snapshot] method = "grpc"`.
|
||||
2. `mcp snapshot <service>` -- verify snapshot succeeds.
|
||||
3. Verify scheduled snapshots include the unikernel service.
|
||||
4. `mcp logs <service>` -- verify serial console output streams.
|
||||
5. Kill the QEMU process manually. Verify drift detection catches it
|
||||
and reports the service as unhealthy.
|
||||
6. Restart the agent. Verify it rediscovers running VMs.
|
||||
|
||||
---
|
||||
|
||||
## Phase 4: Image Attestation
|
||||
|
||||
**Goal**: The agent verifies that the image it boots matches what the
|
||||
operator deployed. The master records expected image hashes.
|
||||
|
||||
### 4.1 Image Hashing
|
||||
|
||||
After building the `.img` file (Phase 1.2), the agent computes its
|
||||
SHA-256 hash and stores it in the registry:
|
||||
|
||||
```sql
|
||||
ALTER TABLE components ADD COLUMN image_hash TEXT;
|
||||
```
|
||||
|
||||
Before every VM boot, the agent re-hashes the `.img` file and
|
||||
compares against the stored value. If they don't match, the deploy
|
||||
fails with an attestation error. This detects:
|
||||
|
||||
- Accidental image corruption.
|
||||
- Tampering with the image file on disk.
|
||||
- Stale images from a previous deploy.
|
||||
|
||||
### 4.2 Master-Side Hash Verification
|
||||
|
||||
The agent reports the image hash to the master in the deploy response
|
||||
and in heartbeats. The master stores expected hashes in its placements
|
||||
table:
|
||||
|
||||
```sql
|
||||
ALTER TABLE placements ADD COLUMN image_hash TEXT;
|
||||
```
|
||||
|
||||
On reconciliation, the master compares the agent-reported hash against
|
||||
its stored value. Mismatches are flagged in `mcp status` output.
|
||||
|
||||
### 4.3 Build Reproducibility
|
||||
|
||||
For attestation to be meaningful, image builds must be reproducible:
|
||||
the same ELF binary + the same ops config must produce the same `.img`
|
||||
hash. Nanos/ops builds are deterministic if the config is fixed and
|
||||
the binary is identical. Document and test this property.
|
||||
|
||||
If builds are not reproducible (timestamps, random padding), hash the
|
||||
ELF binary instead of the `.img` and accept that the image-level hash
|
||||
is a weaker check.
|
||||
|
||||
### 4.4 Phase 4 Validation
|
||||
|
||||
1. Deploy a unikernel service. Note the image hash in `mcp status`.
|
||||
2. Manually modify the `.img` file on disk.
|
||||
3. Attempt to restart the service -- must fail with attestation error.
|
||||
4. Redeploy (rebuilds the image) -- must succeed with a new hash.
|
||||
5. Verify master reconciliation flags hash mismatches.
|
||||
|
||||
---
|
||||
|
||||
## Phase 5: Service Migration
|
||||
|
||||
**Goal**: Convert real platform services from containers to
|
||||
unikernels, starting with the lowest-risk services and working toward
|
||||
core infrastructure.
|
||||
|
||||
### 5.1 Migration Order
|
||||
|
||||
Services are migrated in order of increasing criticality and
|
||||
decreasing tolerance for disruption:
|
||||
|
||||
**Wave 1 -- Stateless/low-risk worker services:**
|
||||
|
||||
| Service | Why first | Risk |
|
||||
|---|---|---|
|
||||
| mcdoc | Stateless doc renderer. No database. Public-facing but read-only. Failure means docs are down, not data loss. | Very low |
|
||||
| mcat | MCIAS policy tester. Internal only. No persistent state. | Very low |
|
||||
|
||||
**Wave 2 -- Stateful worker services:**
|
||||
|
||||
| Service | Why second | Risk |
|
||||
|---|---|---|
|
||||
| mcq | Review queue. SQLite database. Has gRPC snapshot support. Good test of 9p + SQLite under unikernel. | Low-medium |
|
||||
|
||||
**Wave 3 -- Core infrastructure (only after Waves 1-2 are stable):**
|
||||
|
||||
| Service | Considerations | Risk |
|
||||
|---|---|---|
|
||||
| mcns | DNS server. Failure affects all name resolution. Must validate that Nanos's network stack handles DNS UDP correctly. | Medium |
|
||||
| metacrypt | Seal/unseal lifecycle. Sensitive key material in memory. The reduced TCB is most valuable here. | Medium-high |
|
||||
| mcr | Container registry. Must continue serving OCI images for container-based services that haven't migrated. | Medium |
|
||||
| mcias | Root dependency. Every other service authenticates through it. Last to migrate. Must be thoroughly validated. | High |
|
||||
|
||||
**Not migrated:**
|
||||
|
||||
| Service | Reason |
|
||||
|---|---|
|
||||
| mc-proxy | Node infrastructure, not a deployed service. Runs on the host. |
|
||||
| mcp-agent | Node infrastructure. Must have host access. Unikernel isolation is the opposite of what it needs. |
|
||||
| mcp-master | Same as agent -- needs full host/network access. |
|
||||
|
||||
### 5.2 Per-Service Migration Procedure
|
||||
|
||||
For each service:
|
||||
|
||||
1. **Validate the binary under Nanos locally.** Before touching the
|
||||
control plane, run `ops run <binary> -c config.json` on a dev
|
||||
machine. Verify:
|
||||
- The service starts and passes health checks.
|
||||
- SQLite opens in WAL mode (if applicable).
|
||||
- TLS connections work (Nanos's TLS stack handles the Metacrypt CA
|
||||
cert).
|
||||
- 9p-mounted files are readable and writable.
|
||||
|
||||
2. **Deploy as unikernel on a worker node alongside the container
|
||||
version.** Use a different service name (e.g., `mcq-uk`) to run
|
||||
both versions simultaneously. Route test traffic to the unikernel
|
||||
version via a temporary mc-proxy route.
|
||||
|
||||
3. **Validate under real traffic.**
|
||||
- Health checks pass consistently.
|
||||
- gRPC and HTTP endpoints respond correctly.
|
||||
- Snapshots succeed.
|
||||
- Logs are readable via `mcp logs`.
|
||||
- SQLite performance is acceptable under 9p (benchmark IOPS).
|
||||
|
||||
4. **Cut over.** Update the real service definition to `runtime =
|
||||
"unikernel"` and redeploy. The master handles the transition:
|
||||
stop old container, start new unikernel, update routes and DNS.
|
||||
|
||||
5. **Soak.** Run for at least one full snapshot cycle (24h) before
|
||||
declaring stable. Monitor for:
|
||||
- Memory growth (unikernels have fixed memory, no swap).
|
||||
- 9p filesystem performance under sustained writes.
|
||||
- Clock drift (Nanos uses KVM clock, should be fine).
|
||||
|
||||
6. **Remove the container fallback.** Once stable, remove the
|
||||
parallel container deployment.
|
||||
|
||||
### 5.3 Rollback
|
||||
|
||||
If a unikernel service fails in production:
|
||||
|
||||
1. Change `runtime` back to `"container"` in the service definition.
|
||||
2. `mcp deploy <service>` -- the agent deploys via podman using the
|
||||
same OCI image (still in MCR).
|
||||
3. Routes and DNS update automatically.
|
||||
|
||||
Both runtimes use the same `/srv/<service>/` data directory, so no
|
||||
data migration is needed for rollback. The 9p mount is just a view
|
||||
of the same host directory that containers bind-mount.
|
||||
|
||||
### 5.4 Phase 5 Validation
|
||||
|
||||
Per wave:
|
||||
- All services in the wave are running as unikernels.
|
||||
- Snapshots complete successfully for all migrated services.
|
||||
- `mcp status` shows all services healthy.
|
||||
- Edge routing works for public services (mcdoc, mcq).
|
||||
- No performance regression in SQLite operations.
|
||||
- Successful `mcp migrate` of a unikernel service between nodes.
|
||||
|
||||
---
|
||||
|
||||
## Phase 6: Hardening and Long-Term
|
||||
|
||||
**Goal**: Operational maturity. The platform is running a mixed fleet
|
||||
of containers and unikernels reliably.
|
||||
|
||||
### 6.1 Agent Upgrade for Unikernel Nodes
|
||||
|
||||
`mcp agent upgrade` currently cross-compiles and SCPs the agent
|
||||
binary. No changes needed -- the agent is host software, not a
|
||||
unikernel. Running VMs survive agent restarts because QEMU processes
|
||||
are independent.
|
||||
|
||||
### 6.2 Boot Sequence for Unikernel Core Services
|
||||
|
||||
If core services (Wave 3) are migrated to unikernels, the agent's
|
||||
boot sequence config needs to handle QEMU instead of podman for those
|
||||
stages. The boot sequence already uses service definitions; adding
|
||||
`runtime = "unikernel"` to a boot-stage service is sufficient.
|
||||
|
||||
**Consideration**: QEMU VMs take slightly longer to boot than
|
||||
containers (BIOS/kernel init). Adjust stage timeouts if needed.
|
||||
|
||||
### 6.3 MCR Unikernel Image Storage (Phase 1.2b)
|
||||
|
||||
Once the pipeline is stable, implement pre-built unikernel images in
|
||||
MCR. This eliminates the extract-and-build step on the agent and
|
||||
ensures image reproducibility.
|
||||
|
||||
Add `mcp build` subcommand:
|
||||
|
||||
```
|
||||
mcp build mcq --unikernel # build .img, push to MCR as OCI artifact
|
||||
mcp build mcq --container # existing behavior
|
||||
mcp build mcq --all # both
|
||||
```
|
||||
|
||||
### 6.4 Monitoring Dashboard
|
||||
|
||||
Add unikernel-specific metrics to `mcp status`:
|
||||
|
||||
- VM memory usage (from QMP `query-memory`)
|
||||
- VM CPU usage (from QMP `query-cpus`)
|
||||
- 9p I/O statistics
|
||||
- Image hash and attestation status
|
||||
- Serial console tail (last N lines)
|
||||
|
||||
### 6.5 Future: Capability Tokens
|
||||
|
||||
Independent of unikernels but synergistic. With mandatory mediation
|
||||
(Phase 2), the agent can enforce capability tokens at the network
|
||||
boundary. This is an MCIAS redesign, not an MCP change:
|
||||
|
||||
- MCIAS issues operation-scoped tokens ("bearer may read from mcq
|
||||
review queue") instead of identity tokens ("bearer is kyle").
|
||||
- mc-proxy (or agent-level proxy) inspects tokens on forwarded
|
||||
requests and enforces capabilities.
|
||||
- Services no longer need to implement their own policy engines --
|
||||
the mediation layer handles it.
|
||||
|
||||
This is a significant design effort and should be its own design
|
||||
document when the time comes.
|
||||
|
||||
---
|
||||
|
||||
## Dependency Graph
|
||||
|
||||
```
|
||||
Phase 1.1 (NixOS/KVM setup)
|
||||
│
|
||||
├── Phase 1.2 (image building)
|
||||
│ │
|
||||
│ └── Phase 1.3 (QEMU runtime)
|
||||
│ │
|
||||
│ ├── Phase 1.4 (service def changes)
|
||||
│ │
|
||||
│ └── Phase 1.5 (resource tracking)
|
||||
│ │
|
||||
│ └── Phase 1.6 (validation) ─── PHASE 1 DONE
|
||||
│
|
||||
└── Phase 2.1 (bridge setup)
|
||||
│
|
||||
├── Phase 2.2 (TAP management)
|
||||
│ │
|
||||
│ └── Phase 2.3 (IP assignment)
|
||||
│ │
|
||||
│ └── Phase 2.4 (mc-proxy routes)
|
||||
│
|
||||
└── Phase 2.5 (firewall) ─── Phase 2.6 (validation) ─── PHASE 2 DONE
|
||||
│
|
||||
├── Phase 3 (snapshots/observability) ─── PHASE 3 DONE
|
||||
│
|
||||
└── Phase 4 (attestation) ─── PHASE 4 DONE
|
||||
│
|
||||
└── Phase 5 (service migration)
|
||||
│
|
||||
└── Phase 6 (hardening)
|
||||
```
|
||||
|
||||
Phases 1 and 2 can be partially parallelized: bridge setup (2.1) only
|
||||
depends on the NixOS/KVM setup (1.1), not on the QEMU runtime being
|
||||
complete. However, Phase 2 validation requires Phase 1 to be done.
|
||||
|
||||
## Risks and Mitigations
|
||||
|
||||
| Risk | Impact | Mitigation |
|
||||
|---|---|---|
|
||||
| Nanos doesn't support a Go stdlib feature a service uses | Service won't start | Validate each binary under Nanos before committing to migration (Phase 5.2 step 1) |
|
||||
| 9p performance too slow for SQLite WAL mode | Database operations degrade | Benchmark during Wave 2 (mcq). Fallback: use virtio-blk disk image instead of 9p |
|
||||
| QEMU memory overhead per VM | Node runs out of memory with many services | Resource tracking (Phase 1.5) prevents overcommit. Budget ~50MB overhead per VM beyond declared memory |
|
||||
| Bridge networking adds latency | Service response times increase | Measure during Phase 2 validation. The bridge is a software switch -- overhead should be microseconds |
|
||||
| `ops` tool or Nanos has breaking changes | Image builds fail | Pin ops/Nanos versions. Treat as a dependency like Go or podman |
|
||||
| KVM not available (RPi, nested virt) | Can't run unikernels on some nodes | Runtime field allows per-service opt-in. Container remains the default. Nodes without KVM simply don't get unikernel placements |
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- **Replacing containers entirely.** Containers remain the default
|
||||
runtime. Unikernels are opt-in for services where the isolation
|
||||
properties justify the debugging trade-offs.
|
||||
- **Multi-process unikernels.** Services that need sidecars (none
|
||||
currently) stay as containers.
|
||||
- **Custom Nanos kernel builds.** Use stock Nanos. If a service needs
|
||||
kernel customization, it stays as a container.
|
||||
- **Internet access from VMs.** VMs communicate only through mc-proxy.
|
||||
If a service needs to reach external APIs, it goes through a
|
||||
host-side proxy (future work, not in scope).
|
||||
@@ -9,8 +9,8 @@ the platform's internals.
|
||||
## Platform Overview
|
||||
|
||||
Metacircular is a multi-service infrastructure platform. Services are
|
||||
Go binaries running as containers across a fleet of Linux nodes,
|
||||
managed by these core components:
|
||||
Go binaries running as containers on Linux nodes, managed by these core
|
||||
components:
|
||||
|
||||
| Component | Role |
|
||||
|-----------|------|
|
||||
@@ -21,27 +21,8 @@ managed by these core components:
|
||||
| **MCNS** (DNS) | Authoritative DNS for `*.svc.mcp.metacircular.net` |
|
||||
|
||||
The operator workflow is: **build image → push to MCR → write service
|
||||
definition → deploy via MCP**. MCP handles port assignment, TLS cert
|
||||
provisioning, route registration, DNS registration, and container
|
||||
lifecycle.
|
||||
|
||||
### Fleet Topology
|
||||
|
||||
The platform runs across multiple nodes connected via Tailnet:
|
||||
|
||||
| Node | Role | OS | Arch | Purpose |
|
||||
|------|------|----|------|---------|
|
||||
| **rift** | Compute + core infra | NixOS | amd64 | Runs most services (Metacrypt, MCR, MCNS, etc.) |
|
||||
| **svc** | Edge | Debian | amd64 | Public-facing mc-proxy, routes traffic over Tailnet to compute nodes |
|
||||
| **orion** | Compute | NixOS | amd64 | Provisioned, currently offline |
|
||||
|
||||
**Node roles:**
|
||||
|
||||
- **Compute nodes** (rift, orion, future RPis) run the full container
|
||||
lifecycle via rootless Podman.
|
||||
- **Edge nodes** (svc) run mc-proxy for public traffic routing only.
|
||||
The MCP agent on edge nodes manages mc-proxy routes but does not run
|
||||
application containers.
|
||||
definition → deploy via MCP**. MCP handles port assignment, route
|
||||
registration, and container lifecycle.
|
||||
|
||||
---
|
||||
|
||||
@@ -53,7 +34,7 @@ The platform runs across multiple nodes connected via Tailnet:
|
||||
| Container engine | Docker or Podman (for building images) |
|
||||
| `mcp` CLI | Installed on the operator workstation |
|
||||
| MCR access | Credentials to push images to `mcr.svc.mcp.metacircular.net:8443` |
|
||||
| MCP agent | Running on the target node (`rift` for services, `svc` for edge routing) |
|
||||
| MCP agent | Running on the target node (currently `rift`) |
|
||||
| MCIAS account | For `mcp` CLI authentication to the agent |
|
||||
|
||||
---
|
||||
@@ -339,20 +320,14 @@ recreates the containers.
|
||||
### What Happens During Deploy
|
||||
|
||||
1. Agent assigns a free host port (10000–60000) for each declared route.
|
||||
2. For L7 routes, agent provisions a TLS certificate from Metacrypt CA
|
||||
(via `POST /v1/engine/request`). Certs are written to
|
||||
`/srv/mc-proxy/certs/<service>.pem` and `.key`. Existing valid certs
|
||||
(more than 30 days from expiry) are reused.
|
||||
3. Agent starts containers with `$PORT` / `$PORT_<NAME>` environment
|
||||
2. Agent starts containers with `$PORT` / `$PORT_<NAME>` environment
|
||||
variables set to the assigned ports.
|
||||
4. Agent registers routes with mc-proxy via gRPC (hostname →
|
||||
`<node-address>:<port>`, mode, TLS cert paths).
|
||||
5. Agent registers DNS entries in MCNS for
|
||||
`<service>.svc.mcp.metacircular.net`.
|
||||
6. Agent records the full state in its SQLite registry.
|
||||
3. Agent registers routes with mc-proxy (hostname → `127.0.0.1:<port>`,
|
||||
mode, TLS cert paths).
|
||||
4. Agent records the full state in its SQLite registry.
|
||||
|
||||
On stop (`mcp stop <service>`), the agent reverses the process: removes
|
||||
DNS entries, removes mc-proxy routes, then stops containers.
|
||||
mc-proxy routes, then stops containers.
|
||||
|
||||
---
|
||||
|
||||
@@ -410,14 +385,7 @@ tags = []
|
||||
level = "info"
|
||||
```
|
||||
|
||||
For services with SSO-enabled web UIs, add:
|
||||
|
||||
```toml
|
||||
[sso]
|
||||
redirect_uri = "https://<service>.svc.mcp.metacircular.net/sso/callback"
|
||||
```
|
||||
|
||||
For services with a separate web UI binary, add:
|
||||
For services with a web UI, add:
|
||||
|
||||
```toml
|
||||
[web]
|
||||
@@ -465,72 +433,18 @@ these.
|
||||
## 6. Authentication (MCIAS Integration)
|
||||
|
||||
Every service delegates authentication to MCIAS. No service maintains
|
||||
its own user database. Services support two login modes: **SSO
|
||||
redirect** (recommended for web UIs) and **direct credentials**
|
||||
(fallback / API clients).
|
||||
its own user database.
|
||||
|
||||
### SSO Login (Web UIs)
|
||||
|
||||
SSO is the preferred login method for web UIs. The flow is an OAuth
|
||||
2.0-style authorization code exchange:
|
||||
|
||||
1. User visits the service and is redirected to `/login`.
|
||||
2. Login page shows a "Sign in with MCIAS" button.
|
||||
3. Click redirects to MCIAS (`/sso/authorize`), which authenticates the
|
||||
user.
|
||||
4. MCIAS redirects back to the service's `/sso/callback` with an
|
||||
authorization code.
|
||||
5. The service exchanges the code for a JWT via a server-to-server call
|
||||
to MCIAS `POST /v1/sso/token`.
|
||||
6. The JWT is stored in a session cookie.
|
||||
|
||||
SSO is enabled by adding an `[sso]` section to the service config and
|
||||
registering the service as an SSO client in MCIAS.
|
||||
|
||||
**Service config:**
|
||||
|
||||
```toml
|
||||
[sso]
|
||||
redirect_uri = "https://<service>.svc.mcp.metacircular.net/sso/callback"
|
||||
```
|
||||
|
||||
**MCIAS config** (add to the `[[sso_clients]]` list):
|
||||
|
||||
```toml
|
||||
[[sso_clients]]
|
||||
client_id = "<service>"
|
||||
redirect_uri = "https://<service>.svc.mcp.metacircular.net/sso/callback"
|
||||
service_name = "<service>"
|
||||
```
|
||||
|
||||
The `redirect_uri` must match exactly between the service config and
|
||||
the MCIAS client registration.
|
||||
|
||||
When `[sso].redirect_uri` is empty or absent, the service falls back to
|
||||
the direct credentials form.
|
||||
|
||||
**Implementation:** Services use `mcdsl/sso` (v1.7.0+) which handles
|
||||
state management, CSRF-safe cookies, and the code exchange. The web
|
||||
server registers three routes:
|
||||
|
||||
| Route | Purpose |
|
||||
|-------|---------|
|
||||
| `GET /login` | Renders landing page with "Sign in with MCIAS" button |
|
||||
| `GET /sso/redirect` | Sets state cookies, redirects to MCIAS |
|
||||
| `GET /sso/callback` | Validates state, exchanges code for JWT, sets session |
|
||||
|
||||
### Direct Credentials (API / Fallback)
|
||||
### Auth Flow
|
||||
|
||||
1. Client sends credentials to the service's `POST /v1/auth/login`.
|
||||
2. Service forwards them to MCIAS via `mcdsl/auth.Authenticator.Login()`.
|
||||
2. Service forwards them to MCIAS via the client library
|
||||
(`git.wntrmute.dev/mc/mcias/clients/go`).
|
||||
3. MCIAS validates and returns a bearer token.
|
||||
4. Subsequent requests include `Authorization: Bearer <token>`.
|
||||
5. Service validates tokens via `ValidateToken()`, cached for 30s
|
||||
5. Service validates tokens via MCIAS `ValidateToken()`, cached for 30s
|
||||
(keyed by SHA-256 of the token).
|
||||
|
||||
Web UIs use this mode when SSO is not configured, presenting a
|
||||
username/password/TOTP form instead of the SSO button.
|
||||
|
||||
### Roles
|
||||
|
||||
| Role | Access |
|
||||
@@ -771,10 +685,9 @@ For reference, these services are operational on the platform:
|
||||
| Service | Version | Node | Purpose |
|
||||
|---------|---------|------|---------|
|
||||
| MCIAS | v1.9.0 | (separate) | Identity and access |
|
||||
| Metacrypt | v1.4.1 | rift | Cryptographic service, PKI/CA |
|
||||
| MC-Proxy | v1.2.1 | rift, svc | TLS proxy and router (svc handles public edge) |
|
||||
| Metacrypt | v1.3.1 | rift | Cryptographic service, PKI/CA |
|
||||
| MC-Proxy | v1.2.1 | rift | TLS proxy and router |
|
||||
| MCR | v1.2.1 | rift | Container registry |
|
||||
| MCNS | v1.1.1 | rift | Authoritative DNS |
|
||||
| MCDoc | v0.1.0 | rift | Documentation server |
|
||||
| MCQ | v0.4.0 | rift | Document review queue |
|
||||
| MCP | v0.7.6 | rift, svc | Control plane agent |
|
||||
| MCP | v0.7.6 | rift | Control plane agent |
|
||||
|
||||
@@ -1,127 +0,0 @@
|
||||
# Phase E: Multi-Node Orchestration
|
||||
|
||||
Phase D (automated DNS registration) is complete. Phase E extends MCP from
|
||||
a single-node agent on rift to a multi-node fleet with a central master
|
||||
process.
|
||||
|
||||
## Goal
|
||||
|
||||
Deploy and manage services across multiple nodes from a single control
|
||||
plane. The operator runs `mcp deploy` and the system places the workload on
|
||||
the right node, provisions certs, registers DNS, and configures routing --
|
||||
same as today on rift, but across the fleet.
|
||||
|
||||
## Fleet Topology
|
||||
|
||||
| Node | OS | Arch | Role |
|
||||
|------|----|------|------|
|
||||
| rift | NixOS | amd64 | Master + worker -- runs mcp-master, core infra, and application services |
|
||||
| orion | NixOS | amd64 | Worker |
|
||||
| hyperborea | Debian | arm64 | Worker (Raspberry Pi) |
|
||||
| svc | Debian | amd64 | Edge -- mc-proxy for public traffic, no containers |
|
||||
|
||||
Tailnet is the interconnect between all nodes. Public traffic enters via
|
||||
mc-proxy on svc, which forwards over Tailnet to worker nodes.
|
||||
|
||||
## Key Architecture Decisions
|
||||
|
||||
These were resolved in the 2026-04-01 design session:
|
||||
|
||||
1. **Rift is the master node.** No separate straylight machine. Core infra
|
||||
stays on rift, which gains mcp-master alongside its existing agent.
|
||||
|
||||
2. **Master-mediated coordination.** Agents never talk to each other. All
|
||||
cross-node operations go through the master. Agents only dial the master
|
||||
(for registration and heartbeats) and respond to master RPCs.
|
||||
|
||||
3. **Agent self-registration.** Agents register with the master on startup
|
||||
(name, role, address, arch). The master maintains the live node registry.
|
||||
No static `[[nodes]]` config required except for bootstrap.
|
||||
|
||||
4. **Heartbeats with fallback probe.** Agents push heartbeats every 30s
|
||||
(with resource data). If the master misses 3 heartbeats (90s), it
|
||||
actively probes the agent. Failed probe marks the node unhealthy.
|
||||
|
||||
5. **Tier-based placement.** `tier = "core"` runs on the master node.
|
||||
`tier = "worker"` (default) is auto-placed on a worker with capacity.
|
||||
Explicit `node = "orion"` overrides tier for pinned services.
|
||||
|
||||
6. **Two separate certs for public services.** Internal cert
|
||||
(`svc.mcp.metacircular.net`) issued by worker agent. Public cert
|
||||
(`metacircular.net`) issued by edge agent. Internal names never
|
||||
appear on edge certs.
|
||||
|
||||
7. **`public = true` on routes.** Public routes declare intent with a
|
||||
boolean flag. The master assigns the route to an edge node (currently
|
||||
always svc). No explicit `edge` field in service definitions.
|
||||
|
||||
## Components
|
||||
|
||||
### Master (`mcp-master`)
|
||||
|
||||
Long-lived orchestrator on rift. Responsibilities:
|
||||
|
||||
- Accept CLI commands and dispatch to the correct agent
|
||||
- Maintain node registry from agent self-registration
|
||||
- Place services based on tier, explicit node, and resource availability
|
||||
- Detect `public = true` routes and coordinate edge setup
|
||||
- Validate public hostnames against allowed domain list
|
||||
- Aggregate status from all agents (fleet-wide view)
|
||||
- Probe agents on missed heartbeats
|
||||
|
||||
The master is stateless in the durable sense -- it rebuilds its world view
|
||||
from agents on startup. If the master goes down, running services continue
|
||||
unaffected; only new deploys and rescheduling stop.
|
||||
|
||||
### Agent upgrades
|
||||
|
||||
The fleet is heterogeneous (NixOS + Debian, amd64 + arm64), so NixOS flake
|
||||
inputs don't work as a universal update mechanism.
|
||||
|
||||
**Design:** MCP owns the binary at `/srv/mcp/mcp-agent` on all nodes.
|
||||
|
||||
- `mcp agent upgrade [node]` -- CLI cross-compiles for the target's
|
||||
GOARCH, SCPs the binary, restarts via SSH
|
||||
- Node config gains `ssh` (user@host) and `arch` (amd64/arm64) fields
|
||||
- rift's NixOS `ExecStart` changes from nix store path to
|
||||
`/srv/mcp/mcp-agent`
|
||||
- All nodes: binary at `/srv/mcp/mcp-agent`, systemd unit
|
||||
`mcp-agent.service`
|
||||
|
||||
### Edge agents
|
||||
|
||||
svc runs an agent but does NOT run containers. Its agent manages mc-proxy
|
||||
routing only: when the master tells it to set up an edge route, it
|
||||
provisions a TLS cert from Metacrypt and registers the route in its local
|
||||
mc-proxy via the gRPC admin API.
|
||||
|
||||
## Migration Plan
|
||||
|
||||
### Phase 1: Agent on svc
|
||||
Deploy mcp-agent to svc. Verify with `mcp node list`.
|
||||
|
||||
### Phase 2: Edge routing RPCs
|
||||
Implement SetupEdgeRoute/RemoveEdgeRoute/ListEdgeRoutes on the agent.
|
||||
Test by calling directly from CLI.
|
||||
|
||||
### Phase 3: Build mcp-master
|
||||
Core loop: registration, heartbeats, deploy routing, placement, edge
|
||||
coordination.
|
||||
|
||||
### Phase 4: Agent registration and health
|
||||
Self-registration, heartbeat loop, master probe fallback, fleet status.
|
||||
|
||||
### Phase 5: Cut over
|
||||
Point CLI at master, add tier fields to service defs, deploy agents to
|
||||
orion and hyperborea.
|
||||
|
||||
## What Phase E Does NOT Include
|
||||
|
||||
These remain future work:
|
||||
|
||||
- Auto-reconciliation (agent auto-restarting drifted containers)
|
||||
- Live migration (snapshot streaming between nodes)
|
||||
- Web UI for fleet management
|
||||
- Observability / log aggregation
|
||||
- Object store
|
||||
- Multiple edge nodes / master HA
|
||||
@@ -184,7 +184,7 @@ require git.wntrmute.dev/mc/mcdsl v1.2.0
|
||||
Every repository has a Makefile with these standard targets:
|
||||
|
||||
```makefile
|
||||
.PHONY: build test vet lint proto-lint clean docker push all
|
||||
.PHONY: build test vet lint proto-lint clean docker all
|
||||
|
||||
LDFLAGS := -trimpath -ldflags="-s -w -X main.version=$(shell git describe --tags --always --dirty)"
|
||||
|
||||
@@ -218,9 +218,6 @@ clean:
|
||||
docker:
|
||||
docker build -t <service> -f Dockerfile.api .
|
||||
|
||||
push: docker
|
||||
docker push $(MCR)/<service>:$(VERSION)
|
||||
|
||||
all: vet lint test <service>
|
||||
```
|
||||
|
||||
@@ -233,7 +230,6 @@ all: vet lint test <service>
|
||||
| `test` | Every change | Yes |
|
||||
| `proto-lint` | Any proto change | Yes |
|
||||
| `proto` | After editing `.proto` files | No (manual) |
|
||||
| `push` | After building container image | No (manual) |
|
||||
| `all` | Pre-push verification | Yes |
|
||||
|
||||
The `all` target is the CI pipeline: `vet → lint → test → build`. If any
|
||||
|
||||
@@ -1,627 +0,0 @@
|
||||
# Building MCQ: A Reading Queue in a Session
|
||||
|
||||
*A document reading queue goes from design discussion to production
|
||||
deployment — including the deployment's sharp edges.*
|
||||
|
||||
*Written by Claude (Opus 4.6), reflecting on a collaborative session with
|
||||
Kyle. This log focuses heavily on the deployment side, including what
|
||||
went wrong, what was done manually versus what should have been MCP-managed,
|
||||
and why `mcp ps` doesn't know mcq exists.*
|
||||
|
||||
---
|
||||
|
||||
## The Idea
|
||||
|
||||
Kyle was out and about, away from his tailnet, and wanted to review
|
||||
platform documentation on his phone. The existing tools — mcdoc (which
|
||||
renders docs from Gitea repos) and the repos themselves — require either
|
||||
tailnet access or a desktop workflow.
|
||||
|
||||
The concept: a **document queue**. Push raw markdown from inside the
|
||||
infrastructure, read rendered HTML from anywhere via a browser. Like a
|
||||
self-hosted Pocket, but for internal docs you're actively iterating on.
|
||||
|
||||
After a design discussion, we settled on:
|
||||
|
||||
- **Name**: mcq (Metacircular Document Queue)
|
||||
- **Data model**: Documents keyed by slug, upsert semantics (re-push
|
||||
replaces content, resets read flag)
|
||||
- **Auth**: MCIAS on everything — any user including guest can read, any
|
||||
user including system accounts can push
|
||||
- **Rendering**: Goldmark with GFM + syntax highlighting, rendered on
|
||||
each page view
|
||||
- **Architecture**: Single binary, REST API + gRPC + web UI
|
||||
|
||||
## Building the Service
|
||||
|
||||
### Codebase Exploration
|
||||
|
||||
Before writing any code, I explored the existing platform services to
|
||||
understand the patterns:
|
||||
|
||||
- **mcat** (`~/src/metacircular/mcat/`): Reference for the web UI pattern —
|
||||
chi router, CSRF, session cookies, htmx, embedded templates, cobra CLI,
|
||||
config loading via `mcdsl/config`.
|
||||
- **mcns** (`~/src/metacircular/mcns/`): Reference for REST + gRPC pattern —
|
||||
separate `internal/server/` (REST) and `internal/grpcserver/` (gRPC),
|
||||
method maps for auth interceptors, SQLite via `mcdsl/db`.
|
||||
- **mcdoc** (`~/src/metacircular/mcdoc/`): Reference for goldmark rendering
|
||||
and plain HTTP serving (mcdoc doesn't use mcdsl for config or HTTP — it
|
||||
has its own, because it serves plain HTTP behind mc-proxy).
|
||||
- **mcdsl** (`~/src/metacircular/mcdsl/`): The shared library — auth,
|
||||
config, db, httpserver, grpcserver, csrf, web packages.
|
||||
|
||||
### Implementation (on vade, Kyle's workstation)
|
||||
|
||||
Created `~/src/mcq/` with the standard platform layout:
|
||||
|
||||
```
|
||||
cmd/mcq/ main.go, server.go (cobra CLI)
|
||||
internal/
|
||||
config/ custom config (TLS optional, see below)
|
||||
db/ SQLite schema, migrations, document CRUD
|
||||
server/ REST API routes and handlers
|
||||
grpcserver/ gRPC server, interceptors, service handlers
|
||||
webserver/ Web UI routes, templates, session management
|
||||
render/ goldmark markdown-to-HTML renderer
|
||||
proto/mcq/v1/ Protobuf definitions
|
||||
gen/mcq/v1/ Generated Go code
|
||||
web/ Embedded templates + static files
|
||||
deploy/ systemd, examples
|
||||
```
|
||||
|
||||
Key files:
|
||||
|
||||
- **Proto** (`proto/mcq/v1/mcq.proto`): DocumentService (ListDocuments,
|
||||
GetDocument, PutDocument, DeleteDocument, MarkRead, MarkUnread),
|
||||
AuthService (Login, Logout), AdminService (Health).
|
||||
- **DB** (`internal/db/documents.go`): Single `documents` table with slug
|
||||
as unique key. PutDocument uses `INSERT ... ON CONFLICT(slug) DO UPDATE`.
|
||||
- **REST** (`internal/server/routes.go`): All routes under `/v1/` —
|
||||
`PUT /v1/documents/{slug}` for upsert, standard CRUD otherwise.
|
||||
- **Web UI** (`internal/webserver/server.go`): Login page, document list
|
||||
at `/`, rendered markdown reader at `/d/{slug}`.
|
||||
- **gRPC** (`internal/grpcserver/`): Mirrors REST exactly. Method map puts
|
||||
all document operations in `authRequiredMethods`, nothing in
|
||||
`adminRequiredMethods`.
|
||||
|
||||
Proto generation ran on vade:
|
||||
|
||||
```bash
|
||||
cd ~/src/mcq
|
||||
protoc --go_out=. --go_opt=module=git.wntrmute.dev/mc/mcq \
|
||||
--go-grpc_out=. --go-grpc_opt=module=git.wntrmute.dev/mc/mcq \
|
||||
proto/mcq/v1/*.proto
|
||||
```
|
||||
|
||||
### The .gitignore Bug
|
||||
|
||||
First `git add -A` missed `cmd/mcq/`, `proto/mcq/`, and `gen/mcq/`. The
|
||||
`.gitignore` had:
|
||||
|
||||
```
|
||||
mcq
|
||||
srv/
|
||||
```
|
||||
|
||||
The pattern `mcq` (without a leading slash) matches any file or directory
|
||||
named `mcq` at any level — so it was ignoring `cmd/mcq/`, `gen/mcq/`, and
|
||||
`proto/mcq/`. Fixed to:
|
||||
|
||||
```
|
||||
/mcq
|
||||
/srv/
|
||||
```
|
||||
|
||||
### The TLS Decision
|
||||
|
||||
This was the most consequential design decision for deployment.
|
||||
|
||||
The standard platform pattern (mcdsl's `httpserver`) enforces TLS 1.3
|
||||
minimum. But mc-proxy on svc terminates TLS at the edge and forwards to
|
||||
backends as plain HTTP (for localhost services) or HTTPS (for remote
|
||||
backends like rift). Gitea on svc runs plain HTTP on port 3000 behind
|
||||
mc-proxy. mcdoc on rift runs plain HTTP on port 38080 behind mc-proxy.
|
||||
|
||||
mcdsl's `config.Load` validates that `tls_cert` and `tls_key` are present
|
||||
— they're required fields. So I couldn't use `config.Base` with empty TLS
|
||||
fields.
|
||||
|
||||
**Solution**: Created `internal/config/config.go` — mcq's own config
|
||||
package, modeled after mcdoc's. Same TOML loading, env var overrides, and
|
||||
validation, but TLS fields are optional. When empty, the server uses
|
||||
`http.ListenAndServe()` instead of `httpserver.ListenAndServeTLS()`.
|
||||
|
||||
This meant giving up the mcdsl httpserver (with its logging middleware and
|
||||
TLS enforcement) for the plain HTTP path. The gRPC server was also dropped
|
||||
from the svc deployment since it requires TLS. The REST API and web UI
|
||||
are sufficient for the use case.
|
||||
|
||||
### Build and Test (on vade)
|
||||
|
||||
```bash
|
||||
cd ~/src/mcq
|
||||
go mod tidy
|
||||
go build ./... # clean
|
||||
go vet ./... # clean
|
||||
go test ./... # 6 tests pass (all in internal/db)
|
||||
|
||||
# Production binary
|
||||
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 \
|
||||
go build -trimpath -ldflags="-s -w -X main.version=v0.1.0" \
|
||||
-o mcq ./cmd/mcq
|
||||
|
||||
# Result: 21MB static binary
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Deployment
|
||||
|
||||
### Why mcq is NOT in `mcp ps`
|
||||
|
||||
**This is the most important thing in this log.**
|
||||
|
||||
mcq was deployed as a **manual systemd service on svc**, not as an
|
||||
MCP-managed container. This means:
|
||||
|
||||
- `mcp ps` doesn't know about it
|
||||
- `mcp stop mcq` won't work
|
||||
- `mcp deploy mcq` won't work
|
||||
- There's no service definition in `~/.config/mcp/services/`
|
||||
- There's no container image in MCR
|
||||
- The binary was `scp`'d directly to svc and `install`'d to `/usr/local/bin/`
|
||||
|
||||
**Why?** Three reasons:
|
||||
|
||||
1. **svc has no MCP agent.** The MCP agent (`mcp-agent`) only runs on rift.
|
||||
svc is a Debian VPS that hosts MCIAS, mc-proxy, MCNS, and Gitea — all
|
||||
deployed as manual systemd services, not via MCP. Getting mcq into MCP
|
||||
would require deploying an MCP agent to svc first (Phase E in
|
||||
PLATFORM_EVOLUTION.md, items #10-#12).
|
||||
|
||||
2. **mcq runs as a native binary, not a container.** MCP manages containers
|
||||
(podman). mcq on svc is a bare binary under systemd, like MCIAS and
|
||||
mc-proxy on svc. To make it MCP-managed, it would need to be
|
||||
containerized and pushed to MCR first.
|
||||
|
||||
3. **The deployment followed the existing svc pattern.** Every service on
|
||||
svc was deployed this way: build on vade, scp to svc, install, write
|
||||
config, write systemd unit, enable. This was a deliberate choice to
|
||||
match the existing operational model rather than block on MCP agent
|
||||
deployment.
|
||||
|
||||
### What MCP-managed deployment would look like
|
||||
|
||||
Once svc has an MCP agent, mcq could be managed like services on rift:
|
||||
|
||||
```toml
|
||||
# ~/.config/mcp/services/mcq.toml
|
||||
name = "mcq"
|
||||
node = "svc"
|
||||
version = "v0.1.0"
|
||||
|
||||
[[components]]
|
||||
name = "api"
|
||||
|
||||
[[components.routes]]
|
||||
port = 8090
|
||||
mode = "l7"
|
||||
hostname = "mcq.metacircular.net"
|
||||
```
|
||||
|
||||
This would require:
|
||||
- MCP agent running on svc
|
||||
- mcq containerized (Dockerfile) and pushed to MCR
|
||||
- Agent handles port assignment, mc-proxy route registration, lifecycle
|
||||
|
||||
### The Actual Deployment Steps
|
||||
|
||||
All commands below were run from vade (Kyle's workstation) via SSH to svc,
|
||||
unless otherwise noted.
|
||||
|
||||
#### 1. Push repo to Gitea (from vade)
|
||||
|
||||
```bash
|
||||
cd ~/src/mcq
|
||||
git remote add origin git@git.wntrmute.dev:mc/mcq.git
|
||||
git push -u origin master
|
||||
```
|
||||
|
||||
The mc/mcq repo was created manually in Gitea (the MCP tool's API token
|
||||
lacked `write:organization` scope for creating repos under the mc org).
|
||||
|
||||
#### 2. Copy binary to svc (from vade)
|
||||
|
||||
```bash
|
||||
scp ~/src/mcq/mcq kyle@svc:/tmp/mcq
|
||||
```
|
||||
|
||||
SSH to svc uses Tailscale hostname resolution — `svc` resolves to
|
||||
`100.106.232.4` via tailscale. No SSH config entry was needed. Had to
|
||||
accept the host key on first connection:
|
||||
|
||||
```bash
|
||||
ssh -o StrictHostKeyChecking=accept-new kyle@svc
|
||||
```
|
||||
|
||||
#### 3. Create user and install binary (on svc, as root via sudo)
|
||||
|
||||
```bash
|
||||
sudo useradd --system --no-create-home --shell /usr/sbin/nologin mcq
|
||||
sudo mkdir -p /srv/mcq
|
||||
sudo chown mcq:mcq /srv/mcq
|
||||
sudo chmod 0700 /srv/mcq
|
||||
sudo install -m 0755 /tmp/mcq /usr/local/bin/mcq
|
||||
```
|
||||
|
||||
Verified: `/usr/local/bin/mcq --version` → `mcq version v0.1.0`
|
||||
|
||||
#### 4. Write config (on svc)
|
||||
|
||||
Created `/srv/mcq/mcq.toml`:
|
||||
|
||||
```toml
|
||||
[server]
|
||||
listen_addr = "127.0.0.1:8090"
|
||||
|
||||
[database]
|
||||
path = "/srv/mcq/mcq.db"
|
||||
|
||||
[mcias]
|
||||
server_url = "https://mcias.metacircular.net:8443"
|
||||
ca_cert = "/srv/mcq/ca.pem"
|
||||
service_name = "mcq"
|
||||
tags = []
|
||||
|
||||
[log]
|
||||
level = "info"
|
||||
```
|
||||
|
||||
**Important detail**: The first attempt used `server_url = "https://127.0.0.1:8443"`
|
||||
which failed because MCIAS's TLS cert has SANs for `mcias.wntrmute.dev`
|
||||
and `mcias.metacircular.net` but **not** `127.0.0.1` or `localhost`. Token
|
||||
validation returned "invalid or expired token" because the mcdsl auth
|
||||
client couldn't establish a TLS connection to MCIAS.
|
||||
|
||||
Fixed by copying the pattern from MCNS on svc:
|
||||
- `server_url = "https://mcias.metacircular.net:8443"` (uses the hostname
|
||||
that matches the cert's SAN)
|
||||
- `ca_cert = "/srv/mcq/ca.pem"` (the WNTRMUTE root CA cert, copied from
|
||||
`/srv/mcns/certs/ca.pem`)
|
||||
|
||||
The hostname `mcias.metacircular.net` resolves to svc's public IP, so
|
||||
this still connects to localhost MCIAS — it just goes through the public
|
||||
IP for TLS hostname verification. (On a locked-down firewall this could
|
||||
be an issue, but svc allows loopback through its public IP.)
|
||||
|
||||
#### 5. Create systemd unit (on svc)
|
||||
|
||||
Created `/etc/systemd/system/mcq.service`:
|
||||
|
||||
```ini
|
||||
[Unit]
|
||||
Description=MCQ Document Queue
|
||||
After=network-online.target mcias.service
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
ExecStart=/usr/local/bin/mcq server --config /srv/mcq/mcq.toml
|
||||
WorkingDirectory=/srv/mcq
|
||||
Restart=on-failure
|
||||
RestartSec=5
|
||||
User=mcq
|
||||
Group=mcq
|
||||
|
||||
NoNewPrivileges=yes
|
||||
ProtectSystem=strict
|
||||
ProtectHome=yes
|
||||
ReadWritePaths=/srv/mcq
|
||||
PrivateTmp=yes
|
||||
ProtectKernelTunables=yes
|
||||
ProtectControlGroups=yes
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
```
|
||||
|
||||
```bash
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable --now mcq
|
||||
```
|
||||
|
||||
Verified running: PID 3765144, memory 7.8MB, started cleanly.
|
||||
|
||||
#### 6. Generate TLS certificate for mc-proxy (on vade)
|
||||
|
||||
mc-proxy needs a TLS cert for the `mcq.metacircular.net` hostname (it
|
||||
terminates TLS at the edge). Generated using the local WNTRMUTE root CA:
|
||||
|
||||
```bash
|
||||
cd /tmp
|
||||
openssl ecparam -name prime256v1 -genkey -noout -out mcq.key
|
||||
|
||||
openssl req -new -key mcq.key -out mcq.csr \
|
||||
-subj "/CN=mcq.metacircular.net/O=Metacircular Dynamics" \
|
||||
-addext "subjectAltName=DNS:mcq.metacircular.net"
|
||||
|
||||
openssl x509 -req -in mcq.csr \
|
||||
-CA ~/src/metacircular/ca/ca.pem \
|
||||
-CAkey ~/src/metacircular/ca/ca.key \
|
||||
-CAcreateserial -out mcq.pem -days 365 -sha256 \
|
||||
-extfile <(echo "subjectAltName=DNS:mcq.metacircular.net
|
||||
keyUsage=digitalSignature
|
||||
extendedKeyUsage=serverAuth")
|
||||
```
|
||||
|
||||
The CA key and cert are at `~/src/metacircular/ca/` — this is the
|
||||
WNTRMUTE Issuing Authority root CA. Not Metacrypt (which has its own
|
||||
intermediate CA for automated issuance). The existing mc-proxy certs
|
||||
(docs, git, metacrypt) were all signed by this same root CA.
|
||||
|
||||
Copied to svc:
|
||||
|
||||
```bash
|
||||
scp /tmp/mcq.pem /tmp/mcq.key kyle@svc:/tmp/
|
||||
```
|
||||
|
||||
Installed on svc:
|
||||
|
||||
```bash
|
||||
sudo cp /tmp/mcq.pem /srv/mc-proxy/certs/mcq.metacircular.net.pem
|
||||
sudo cp /tmp/mcq.key /srv/mc-proxy/certs/mcq.metacircular.net.key
|
||||
sudo chown mc-proxy:mc-proxy /srv/mc-proxy/certs/mcq.metacircular.net.*
|
||||
sudo chmod 0600 /srv/mc-proxy/certs/mcq.metacircular.net.key
|
||||
```
|
||||
|
||||
#### 7. Add mc-proxy route (on svc)
|
||||
|
||||
mc-proxy on svc uses SQLite for route persistence. The TOML config only
|
||||
seeds the database on first run (`store.IsEmpty()` check). After that,
|
||||
routes are loaded from SQLite. So editing the TOML alone doesn't add a
|
||||
route — you must also insert into the database.
|
||||
|
||||
I did both (TOML for documentation/re-seeding, SQLite for immediate effect):
|
||||
|
||||
**TOML** (added via `sed` to `/srv/mc-proxy/mc-proxy.toml`):
|
||||
|
||||
```toml
|
||||
[[listeners.routes]]
|
||||
hostname = "mcq.metacircular.net"
|
||||
backend = "127.0.0.1:8090"
|
||||
mode = "l7"
|
||||
tls_cert = "/srv/mc-proxy/certs/mcq.metacircular.net.pem"
|
||||
tls_key = "/srv/mc-proxy/certs/mcq.metacircular.net.key"
|
||||
backend_tls = false
|
||||
```
|
||||
|
||||
**SQLite** (direct insert):
|
||||
|
||||
```bash
|
||||
sudo sqlite3 /srv/mc-proxy/mc-proxy.db "
|
||||
INSERT INTO routes (listener_id, hostname, backend, mode, tls_cert, tls_key, backend_tls)
|
||||
VALUES (1, 'mcq.metacircular.net', '127.0.0.1:8090', 'l7',
|
||||
'/srv/mc-proxy/certs/mcq.metacircular.net.pem',
|
||||
'/srv/mc-proxy/certs/mcq.metacircular.net.key', 0);
|
||||
"
|
||||
```
|
||||
|
||||
The `listener_id = 1` is the `:443` listener (only listener on svc's
|
||||
mc-proxy).
|
||||
|
||||
**Note on `backend_tls = false`**: mcq serves plain HTTP on localhost.
|
||||
mc-proxy terminates TLS for the client and forwards as plain HTTP to
|
||||
`127.0.0.1:8090`. This is the same pattern as Gitea (`127.0.0.1:3000`)
|
||||
and mcdoc (`100.95.252.120:38080`). Only metacrypt uses `backend_tls = true`
|
||||
because its backend is on rift over Tailscale.
|
||||
|
||||
#### 8. Restart mc-proxy (on svc)
|
||||
|
||||
```bash
|
||||
sudo systemctl restart mc-proxy
|
||||
```
|
||||
|
||||
This was messy. mc-proxy's graceful shutdown waits for in-flight
|
||||
connections to drain, and the 30-second shutdown timeout was exceeded
|
||||
(lingering connections from internet scanners hitting git.metacircular.net).
|
||||
The shutdown hung for ~30 seconds before logging "shutdown timeout exceeded,
|
||||
forcing close". systemd then moved to `deactivating (stop-sigterm)` state.
|
||||
|
||||
Had to force it:
|
||||
|
||||
```bash
|
||||
sudo systemctl kill mc-proxy
|
||||
sleep 2
|
||||
sudo systemctl start mc-proxy
|
||||
```
|
||||
|
||||
After restart: `routes=5` (was 4 before mcq). Confirmed:
|
||||
|
||||
```bash
|
||||
curl -sk https://mcq.metacircular.net/v1/health
|
||||
# {"status":"ok"}
|
||||
```
|
||||
|
||||
#### 9. Push documents (from vade)
|
||||
|
||||
Used the mcp-agent service account token (from
|
||||
`~/data/downloads/service-account-76d35a82-77ca-422f-85a3-b9f9360d5164.token`)
|
||||
to authenticate API calls. This is a long-lived JWT issued by MCIAS with
|
||||
`admin` role, `exp` in 2027.
|
||||
|
||||
```bash
|
||||
TOKEN=$(cat ~/data/downloads/service-account-*.token)
|
||||
|
||||
# Push MCP Architecture
|
||||
python3 -c "
|
||||
import json
|
||||
body = open('mcp/ARCHITECTURE.md').read()
|
||||
print(json.dumps({'title': 'MCP Architecture', 'body': body}))
|
||||
" | curl -sk -X PUT https://mcq.metacircular.net/v1/documents/mcp-architecture \
|
||||
-H "Authorization: Bearer $TOKEN" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d @-
|
||||
|
||||
# Push Platform Evolution
|
||||
python3 -c "
|
||||
import json
|
||||
body = open('PLATFORM_EVOLUTION.md').read()
|
||||
print(json.dumps({'title': 'Platform Evolution', 'body': body}))
|
||||
" | curl -sk -X PUT https://mcq.metacircular.net/v1/documents/platform-evolution \
|
||||
-H "Authorization: Bearer $TOKEN" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d @-
|
||||
|
||||
# Push Packaging doc
|
||||
python3 -c "
|
||||
import json
|
||||
body = open('docs/packaging-and-deployment.md').read()
|
||||
print(json.dumps({'title': 'Packaging and Deployment', 'body': body}))
|
||||
" | curl -sk -X PUT https://mcq.metacircular.net/v1/documents/packaging-and-deployment \
|
||||
-H "Authorization: Bearer $TOKEN" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d @-
|
||||
```
|
||||
|
||||
Used `python3` for JSON encoding because `jq` isn't installed on vade
|
||||
(NixOS — would need to add it to the system config or use `nix-shell`).
|
||||
|
||||
All three documents pushed successfully. The token identifies as
|
||||
`mcp-agent` (the service account name), so `pushed_by` shows `mcp-agent`
|
||||
on each document.
|
||||
|
||||
### Subsequent Update: Tufte Theme
|
||||
|
||||
Kyle wanted a wider reading area (70%) and a Tufte-inspired theme. Updated
|
||||
`web/static/style.css`:
|
||||
|
||||
- Serif font stack (Georgia, Palatino)
|
||||
- Cream background (`#fffff8`)
|
||||
- Italic headings, small-caps labels
|
||||
- `width: 70%` on `.page-container` (was `max-width: 720px`)
|
||||
- Minimal chrome — document list uses ruled lines instead of cards,
|
||||
tables use bottom-borders only
|
||||
- Mobile fallback: full width below 768px
|
||||
|
||||
Rebuilt, deployed same way:
|
||||
|
||||
```bash
|
||||
# On vade
|
||||
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 \
|
||||
go build -trimpath -ldflags="-s -w -X main.version=v0.1.1" \
|
||||
-o mcq ./cmd/mcq
|
||||
scp mcq kyle@svc:/tmp/mcq
|
||||
|
||||
# On svc
|
||||
sudo install -m 0755 /tmp/mcq /usr/local/bin/mcq
|
||||
sudo systemctl restart mcq
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## State After This Session
|
||||
|
||||
### What's running on svc
|
||||
|
||||
| Service | Port | Managed by | Notes |
|
||||
|---------|------|------------|-------|
|
||||
| MCIAS | :8443/:9443 | systemd | Identity/auth, been here longest |
|
||||
| mc-proxy | :443 | systemd | L7 TLS termination, 5 routes |
|
||||
| MCNS | :53/:8444/:9444 | systemd | Authoritative DNS |
|
||||
| Gitea | :3000 | systemd | Git hosting |
|
||||
| **mcq** | **:8090** | **systemd** | **NEW: document queue** |
|
||||
|
||||
None of these are MCP-managed. svc has no MCP agent.
|
||||
|
||||
### mc-proxy routes on svc
|
||||
|
||||
| Hostname | Backend | Mode | TLS Backend |
|
||||
|----------|---------|------|-------------|
|
||||
| metacrypt.metacircular.net | 100.95.252.120:18080 | L7 | yes (rift) |
|
||||
| git.metacircular.net | 127.0.0.1:3000 | L7 | no |
|
||||
| git.wntrmute.dev | 127.0.0.1:3000 | L7 | no |
|
||||
| docs.metacircular.net | 100.95.252.120:38080 | L7 | no |
|
||||
| **mcq.metacircular.net** | **127.0.0.1:8090** | **L7** | **no** |
|
||||
|
||||
### DNS
|
||||
|
||||
`mcq.metacircular.net` is a CNAME to `svc.metacircular.net` (set up by
|
||||
Kyle at the DNS registrar before this session). mc-proxy's SNI-based
|
||||
routing handles the rest.
|
||||
|
||||
### Documents in queue
|
||||
|
||||
| Slug | Title | Pushed By |
|
||||
|------|-------|-----------|
|
||||
| mcp-architecture | MCP Architecture | mcp-agent |
|
||||
| platform-evolution | Platform Evolution | mcp-agent |
|
||||
| packaging-and-deployment | Packaging and Deployment | mcp-agent |
|
||||
|
||||
### Git
|
||||
|
||||
Repo: `mc/mcq` on Gitea (`git.wntrmute.dev:mc/mcq.git`)
|
||||
|
||||
Commits:
|
||||
1. `bc16279` — Initial implementation
|
||||
2. `648e9dc` — Support plain HTTP mode for mc-proxy L7 deployment
|
||||
3. `a5b90b6` — Switch to Tufte-inspired reading theme
|
||||
|
||||
---
|
||||
|
||||
## What Would Be Different with MCP
|
||||
|
||||
If svc had an MCP agent and mcq were containerized:
|
||||
|
||||
1. **No manual SSH** — `mcp deploy mcq` from vade would push the service
|
||||
definition, agent would pull the image from MCR.
|
||||
2. **No manual port picking** — agent assigns a free port from 10000-60000.
|
||||
3. **No manual mc-proxy route** — agent calls mc-proxy's gRPC API to
|
||||
register the route (Phase B, already working on rift).
|
||||
4. **No manual TLS cert** — agent provisions from Metacrypt CA
|
||||
(Phase C, already working on rift).
|
||||
5. **No manual systemd unit** — agent manages the container lifecycle.
|
||||
6. **`mcp ps` would show mcq** — because the agent tracks it in its
|
||||
registry.
|
||||
7. **`mcp stop mcq` / `mcp restart mcq` would work** — standard lifecycle.
|
||||
|
||||
The gap is: svc has no agent. That's Phase E work (items #10-#12 in
|
||||
PLATFORM_EVOLUTION.md). The prerequisites are the agent binary location
|
||||
convention, SSH-based upgrade tooling, and node provisioning for Debian.
|
||||
|
||||
---
|
||||
|
||||
## Rough Edges and Lessons
|
||||
|
||||
1. **MCIAS cert hostname**: Every new service on svc will hit this. The
|
||||
MCIAS cert doesn't include localhost as a SAN. Services must use
|
||||
`server_url = "https://mcias.metacircular.net:8443"` (which routes
|
||||
through the public IP back to localhost) and include the CA cert.
|
||||
Could fix by reissuing the MCIAS cert with a localhost SAN.
|
||||
|
||||
2. **mc-proxy route persistence**: The TOML-seeds-once-then-SQLite model
|
||||
means you have to touch two places (TOML for future re-seeds, SQLite
|
||||
for immediate effect). On rift this is handled by the agent's gRPC
|
||||
calls. On svc without an agent, it's manual database surgery.
|
||||
|
||||
3. **mc-proxy shutdown timeout**: The 30-second timeout isn't enough when
|
||||
internet scanners maintain persistent connections to git.metacircular.net.
|
||||
Had to force-kill on restart. Should increase `shutdown_timeout` or
|
||||
add a SIGKILL escalation in the systemd unit (`TimeoutStopSec=45`,
|
||||
which sends SIGKILL after 45s).
|
||||
|
||||
4. **No jq on vade**: NixOS doesn't have jq in the default system config.
|
||||
Used python3 as a workaround for JSON encoding. Minor friction.
|
||||
|
||||
5. **mcdsl httpserver assumes TLS**: Services behind mc-proxy L7 can't use
|
||||
`mcdsl/httpserver` because it enforces TLS 1.3. mcdoc solved this with
|
||||
its own config/server. mcq now does the same. This is a recurring
|
||||
pattern — might warrant adding a plain HTTP mode to mcdsl httpserver,
|
||||
or a separate `mcdsl/httpserver/plain` package.
|
||||
|
||||
6. **Session cookie Secure flag behind plain HTTP**: The mcdsl `web`
|
||||
package always sets `Secure: true` on session cookies. This works
|
||||
behind mc-proxy L7 because the *browser* sees HTTPS (mc-proxy
|
||||
terminates TLS) — the `Secure` flag is about the browser's view of
|
||||
the connection, not the backend. If mcq were ever accessed directly
|
||||
(not through mc-proxy), cookies would silently fail.
|
||||
@@ -1,158 +0,0 @@
|
||||
# MCP Goes Multi-Node: Debugging the Edge
|
||||
|
||||
*A day of operational firefighting leads to an architecture redesign.
|
||||
What started as "why can't I see container logs" ended with a v2
|
||||
architecture document and a plan to introduce mcp-master.*
|
||||
|
||||
*Written by Claude (Opus 4.6), reflecting on a collaborative session with
|
||||
Kyle.*
|
||||
|
||||
---
|
||||
|
||||
## It Started with Logs
|
||||
|
||||
The first problem was simple: `mcp logs mcns` returned "No journal files
|
||||
were opened due to insufficient permissions." The mcns container uses
|
||||
podman's journald log driver, so the agent runs `journalctl` to read
|
||||
logs. But the `mcp` user — running as a system service — didn't have
|
||||
permission to read the system journal.
|
||||
|
||||
The fix was two-part. First, code: add `--user` to `journalctl` for
|
||||
non-root users, then fall back to `podman logs` when `journalctl` fails
|
||||
entirely (v0.7.7–v0.7.9). Second, operational: add the `mcp` user to the
|
||||
`systemd-journal` group in the NixOS config so it can actually read the
|
||||
journal. Neither `journalctl` nor `podman logs` works without the group
|
||||
membership — `podman logs` silently returns empty because it uses the
|
||||
journal API internally.
|
||||
|
||||
Along the way, we added `mcp node list` showing the agent version
|
||||
(v0.7.8), which required threading the linker-injected version string
|
||||
through the Agent struct into the NodeStatus RPC.
|
||||
|
||||
## The mcq Deployment Saga
|
||||
|
||||
Then Kyle tried to check mcns status and hit a TLS EOF. This led us down
|
||||
the mcns certificate rabbit hole (self-signed cert instead of one from
|
||||
Metacrypt), which led to adding a `mcns cert` command for provisioning
|
||||
certs from Metacrypt's CA API (mcns v1.2.0). But the real story was mcq.
|
||||
|
||||
Kyle had deployed an updated mcq earlier, and it broke the public route
|
||||
at mcq.metacircular.net. What followed was a multi-hour debugging session
|
||||
that touched every layer of the stack:
|
||||
|
||||
**Problem 1: Stale route on rift.** mc-proxy on rift had an old
|
||||
`mcq.metacircular.net` route pointing to a wrong port. Rift shouldn't
|
||||
have been routing the public hostname at all — that's svc's job. We
|
||||
added `mcp route add/remove` commands (v0.8.0) to manage mc-proxy routes
|
||||
directly, and cleaned up the stale route.
|
||||
|
||||
**Problem 2: Dynamic ports.** The route system assigns ephemeral host
|
||||
ports that change on every deploy. svc's mc-proxy pointed at
|
||||
`100.95.252.120:48080`, which was a port from a previous deployment.
|
||||
The new container was listening on a completely different port.
|
||||
|
||||
**Problem 3: Rootless podman ports are localhost-only.** Even after
|
||||
getting the right port, svc couldn't reach it — rootless podman binds
|
||||
mapped ports to `127.0.0.1`. We added explicit Tailscale IP bindings to
|
||||
the service definition: `100.95.252.120:48080:8080`.
|
||||
|
||||
**Problem 4: $PORT env override conflict.** The mcdsl config loader
|
||||
overrides `listen_addr` from `$PORT` when routes are present. Adding a
|
||||
route made the container stop listening on port 8080 and listen on the
|
||||
route-allocated port instead, breaking the explicit port mapping. We had
|
||||
to drop the route and manage mc-proxy manually.
|
||||
|
||||
**Problem 5: mc-proxy database overrides TOML.** After updating svc's
|
||||
mc-proxy TOML config, the route still didn't change. mc-proxy persists
|
||||
routes in SQLite, and the database entry (added via the admin API) took
|
||||
precedence over the config file. We had to `sqlite3` into the database
|
||||
and update the route directly. This one took the longest to diagnose —
|
||||
debug logging finally revealed it was proxying to the old backend.
|
||||
|
||||
**Problem 6: Missing cert chain.** The mcq TLS cert on svc was leaf-only
|
||||
(16 lines). mc-proxy requires full chains (leaf + intermediates). The
|
||||
cert loaded fine in Go's `tls.LoadX509KeyPair` but mc-proxy's
|
||||
`GetCertificate` callback failed silently — `client_bytes=7
|
||||
backend_bytes=0` with no error. We issued a proper cert from Metacrypt
|
||||
with the full chain.
|
||||
|
||||
**Problem 7: Old mc-proxy on svc.** Even with the correct cert, TLS
|
||||
still failed. svc was running mc-proxy `v1.0.0-dirty` while rift had
|
||||
`v1.2.1`. We rebuilt and deployed the current version. (This turned out
|
||||
not to be the actual fix — it was the database issue — but svc needed
|
||||
the update anyway.)
|
||||
|
||||
## The Route Command
|
||||
|
||||
Out of the debugging came a useful new tool: `mcp route list/add/remove`
|
||||
(v0.8.0–v0.8.2). It wraps mc-proxy's admin gRPC API through the
|
||||
mcp-agent, so you can manage routes from the operator workstation:
|
||||
|
||||
```
|
||||
mcp route list -n rift
|
||||
mcp route add -n rift :443 mcq.svc.mcp.metacircular.net 127.0.0.1:48080 \
|
||||
--mode l7 --tls-cert /srv/mc-proxy/certs/mcq.pem \
|
||||
--tls-key /srv/mc-proxy/certs/mcq.key
|
||||
mcp route remove -n rift :443 mcq.metacircular.net
|
||||
```
|
||||
|
||||
The `--mode` flag wasn't wired through initially (defined on the cobra
|
||||
command but never passed to the RPC), which we caught when the first L7
|
||||
route add silently created an L4 route instead.
|
||||
|
||||
## Architecture v2
|
||||
|
||||
The operational pain made the case for a redesign. Every public route
|
||||
required hand-editing configs, provisioning certs, debugging database
|
||||
divergence, and manually coordinating between rift and svc. Kyle laid
|
||||
out the target architecture:
|
||||
|
||||
**mcp-master** on a new node (straylight) becomes the coordination
|
||||
point. The CLI talks to the master, not agents directly. The master
|
||||
routes deployments to the correct worker agent (rift), detects public
|
||||
hostnames, and tells the edge agent (svc) to set up forwarding and
|
||||
provision certs.
|
||||
|
||||
The key insight: the service definition already declares everything
|
||||
needed. A route with `hostname = "mcq.metacircular.net"` is
|
||||
unambiguously public (no `.svc.mcp.` prefix). The master can detect this,
|
||||
resolve the CNAME to find which edge node handles it, and orchestrate the
|
||||
whole thing — no manual config editing, no database poking, no separate
|
||||
cert provisioning step.
|
||||
|
||||
Core infrastructure (mcns, metacrypt, mcr) moves to straylight. Rift
|
||||
becomes a pure application worker. svc stays as the public edge, running
|
||||
only mc-proxy and the routes the master tells it to set up.
|
||||
|
||||
The full design is in `ARCHITECTURE_V2.md`, pushed to both git and the
|
||||
mcq reading queue.
|
||||
|
||||
## What Shipped
|
||||
|
||||
| Version | Change |
|
||||
|---------|--------|
|
||||
| mcp v0.7.7 | Fix journald log permissions for rootless podman |
|
||||
| mcp v0.7.8 | Add agent version to `mcp node list` |
|
||||
| mcp v0.7.9 | Fall back to `podman logs` when journalctl inaccessible |
|
||||
| mcp v0.8.0 | Add `mcp route list/add/remove` with `-n/--node` |
|
||||
| mcp v0.8.1 | Merge explicit ports with route-allocated ports during deploy |
|
||||
| mcp v0.8.2 | Wire --mode, --tls-cert, --tls-key through route add |
|
||||
| mcns v1.2.0 | Add `mcns cert` command for Metacrypt TLS provisioning |
|
||||
| mc-proxy on svc | Updated from v1.0.0-dirty to v1.2.1 |
|
||||
| NixOS | Added `systemd-journal` group to mcp user |
|
||||
|
||||
## Lessons
|
||||
|
||||
The deployment pitfalls doc grew significantly. The key additions for
|
||||
the future Debian deployment:
|
||||
|
||||
1. `mcp` user needs `systemd-journal` group for container logs.
|
||||
2. Routes and explicit ports conflict via `$PORT` env override.
|
||||
3. Rootless podman ports need explicit Tailscale IP bindings.
|
||||
4. mc-proxy certs must include the full chain.
|
||||
5. mc-proxy's SQLite database overrides the TOML config.
|
||||
6. Always check the database first when debugging mc-proxy routing.
|
||||
|
||||
Every one of these was a surprise. None was documented before today.
|
||||
The v2 architecture exists specifically so that nobody has to debug
|
||||
these by hand again.
|
||||
@@ -1,231 +0,0 @@
|
||||
# Incident Report: UID Change Cascading Failure
|
||||
|
||||
**Date**: 2026-04-03
|
||||
**Duration**: ~2 hours (08:45–10:45 PDT)
|
||||
**Severity**: Full platform outage on rift (all containers lost)
|
||||
**Root cause**: Changing the `mcp` system user UID from 995 to 850
|
||||
|
||||
## Timeline
|
||||
|
||||
### Background
|
||||
|
||||
Orion was being provisioned as a new worker node. Its NixOS config
|
||||
imports `mcp.nix` which pins the mcp user to UID 995. On orion, UID 995
|
||||
was already assigned to the `sshd` user, causing a UID collision:
|
||||
|
||||
```
|
||||
uid=995(sshd) gid=988(mcp) groups=988(mcp),62(systemd-journal),992(sshd)
|
||||
```
|
||||
|
||||
Both `sshd` and `mcp` had UID 995 on orion. The `newuidmap` tool
|
||||
rejected rootless podman operations because the calling process's UID
|
||||
(995) belonged to `sshd`, not `mcp`, in `/etc/passwd`.
|
||||
|
||||
### The UID Change
|
||||
|
||||
To resolve the collision, `mcp.nix` was updated to pin UID 850 (in
|
||||
the 800–899 range, empty on all nodes). Both rift and orion were
|
||||
rebuilt with `nixos-rebuild switch`.
|
||||
|
||||
**Problem 1: NixOS doesn't change UIDs for existing users.** The
|
||||
rebuild created the NixOS config with `uid = 850` but the existing
|
||||
`mcp` user on both nodes kept UID 995. Manual `usermod -u 850 mcp`
|
||||
was required on each node.
|
||||
|
||||
**Problem 2: Rootless podman caches the UID everywhere.**
|
||||
- Podman's SQLite database (`db.sql`) stores absolute paths like
|
||||
`/run/user/995/libpod/tmp` and `/run/user/995/containers`
|
||||
- The systemd user session (`/run/user/995/`) is tied to the UID
|
||||
- subuid/subgid mappings reference the user by name but the kernel
|
||||
checks the actual UID
|
||||
- Container storage overlay directories have file ownership based on
|
||||
the old UID namespace mapping (995 → 100000)
|
||||
|
||||
After changing the UID, `podman` operations failed with:
|
||||
```
|
||||
newuidmap: write to uid_map failed: Operation not permitted
|
||||
```
|
||||
|
||||
### The Reboot
|
||||
|
||||
Rift was rebooted to get a clean systemd user session for UID 850.
|
||||
The reboot succeeded, but **all containers were gone**:
|
||||
|
||||
```
|
||||
$ podman ps -a
|
||||
(empty)
|
||||
```
|
||||
|
||||
Podman's database was recreated fresh on boot because the old database
|
||||
referenced paths under `/run/user/995/` which no longer existed. The
|
||||
images were still in overlay storage but the container definitions
|
||||
(names, port mappings, volume mounts, restart policies) were lost.
|
||||
|
||||
### DNS Collapse
|
||||
|
||||
MCNS (the authoritative DNS server for `.svc.mcp.metacircular.net`)
|
||||
ran as a container on rift. When all containers were lost, DNS
|
||||
resolution broke:
|
||||
|
||||
- `mcq.svc.mcp.metacircular.net` → no answer
|
||||
- MCNS also served as a recursive resolver for the LAN
|
||||
- `google.com` → NXDOMAIN on machines using MCNS as their resolver
|
||||
|
||||
Tailscale DNS (MagicDNS) was also affected because resolved's global
|
||||
DNS config pointed to MCNS. Tailscale itself remained functional
|
||||
(its coordination servers are external), but hostname resolution via
|
||||
Tailscale DNS names failed.
|
||||
|
||||
The operator turned off Tailscale on vade (the workstation) because
|
||||
Tailscale's MagicDNS was routing ALL DNS queries through the broken
|
||||
MCNS resolver — external services including Claude Code and Gitea
|
||||
were unreachable. Disabling Tailscale was the only way to restore
|
||||
external DNS resolution. However, this also broke connectivity to
|
||||
rift since the MCP agent binds to the Tailnet IP only
|
||||
(`100.95.252.120:9444`).
|
||||
|
||||
### Recovery
|
||||
|
||||
**Step 1**: Turn Tailscale back on (on both rift and vade). Tailscale
|
||||
connectivity works without MCNS — MagicDNS uses Tailscale's own
|
||||
servers for `.ts.net` names.
|
||||
|
||||
**Step 2**: Start MCNS manually via `podman run`. The image was cached
|
||||
in overlay storage. MCNS needed explicit port bindings (not `--network
|
||||
host`) because systemd-resolved holds port 53 on localhost:
|
||||
|
||||
```bash
|
||||
podman run -d --name mcns --restart unless-stopped \
|
||||
-p 192.168.88.181:53:53/tcp -p 192.168.88.181:53:53/udp \
|
||||
-p 100.95.252.120:53:53/tcp -p 100.95.252.120:53:53/udp \
|
||||
-p 127.0.0.1:38443:8443 \
|
||||
-v /srv/mcns:/srv/mcns \
|
||||
mcr.svc.mcp.metacircular.net:8443/mcns:v1.2.0 \
|
||||
server --config /srv/mcns/mcns.toml
|
||||
```
|
||||
|
||||
DNS resolution restored within seconds.
|
||||
|
||||
**Step 3**: Start remaining services manually via `podman run`. Images
|
||||
were all cached. The `mcp deploy` CLI couldn't work because:
|
||||
- MCR was down (can't pull images)
|
||||
- The agent's registry was empty (podman DB reset)
|
||||
- Auto-build failed (`/etc/resolv.conf` permission denied in build
|
||||
containers)
|
||||
|
||||
Each service was started with explicit `podman run` commands matching
|
||||
the service definitions in `~/.config/mcp/services/*.toml`.
|
||||
|
||||
**Step 4**: Fix file ownership for rootless podman. Files in `/srv/*`
|
||||
were owned by UID 850 (the mcp user on the host). Inside containers,
|
||||
UID 0 (root) maps to host UID 850 via subuid. But:
|
||||
|
||||
- `podman unshare chown -R 0:0 /srv/<service>` translated ownership
|
||||
to match the container's user namespace
|
||||
- SQLite's `PRAGMA journal_mode = WAL` requires creating WAL/SHM files
|
||||
in the database directory
|
||||
- modernc.org/sqlite calls `fchmod()` on the database file, which is
|
||||
denied inside rootless podman user namespaces (even for UID 0 in the
|
||||
namespace)
|
||||
|
||||
**Step 5**: Delete and recreate SQLite databases. The `fchmod` denial
|
||||
was fatal for MCR and Metacrypt. The fix:
|
||||
|
||||
```bash
|
||||
# Stop the container
|
||||
podman stop metacrypt-api
|
||||
# Delete the database (WAL and SHM too)
|
||||
rm -f /srv/metacrypt/metacrypt.db*
|
||||
# Restart — the service recreates the database
|
||||
podman start metacrypt-api
|
||||
```
|
||||
|
||||
The `fchmod` error still occurs on the newly created database but is
|
||||
non-fatal — the service logs a warning and continues.
|
||||
|
||||
**Data loss**: MCR and Metacrypt databases were deleted and recreated
|
||||
empty. MCR lost its manifest/tag metadata (images still exist in
|
||||
overlay storage but are unregistered). Metacrypt lost its CA state
|
||||
(encrypted keys, issued certs tracking). Other services (mcq, mcdoc,
|
||||
etc.) started successfully because their databases survived the
|
||||
ownership changes.
|
||||
|
||||
## Root Causes
|
||||
|
||||
1. **UID collision between system users**: NixOS auto-assigns UIDs
|
||||
downward from 999. Pinning UID 995 for mcp collided with sshd on
|
||||
orion.
|
||||
|
||||
2. **Rootless podman's deep UID dependency**: Changing a user's UID
|
||||
after rootless podman has been used requires:
|
||||
- Updating podman's internal database paths
|
||||
- Recreating the systemd user session
|
||||
- Fixing subuid/subgid mappings
|
||||
- Fixing overlay storage ownership
|
||||
- Fixing service data file ownership
|
||||
- None of these happen automatically
|
||||
|
||||
3. **No boot sequencing**: When rift rebooted with no running
|
||||
containers, there was no mechanism to start services in dependency
|
||||
order. The boot sequence feature in the v2 architecture exists
|
||||
precisely for this, but wasn't implemented yet.
|
||||
|
||||
4. **MCNS as a single point of DNS failure**: All machines used MCNS
|
||||
as their DNS resolver. When MCNS went down, everything broke
|
||||
including the ability to manage infrastructure.
|
||||
|
||||
5. **modernc.org/sqlite `fchmod` in rootless podman**: The SQLite
|
||||
library calls `fchmod()` on database files, which is denied inside
|
||||
rootless podman user namespaces. This is a known incompatibility
|
||||
that was masked by the previous UID setup.
|
||||
|
||||
## Lessons Learned
|
||||
|
||||
1. **Never change a rootless podman user's UID.** If a UID collision
|
||||
exists, resolve it on the conflicting node (change sshd, not mcp)
|
||||
or use a per-host UID override. Changing the UID after podman has
|
||||
been used is destructive.
|
||||
|
||||
2. **DNS must not be a single point of failure.** All machines should
|
||||
have fallback DNS resolvers that work independently of MCNS. The
|
||||
NixOS config should list public resolvers (1.1.1.1, 8.8.8.8) as
|
||||
fallbacks, not just MCNS.
|
||||
|
||||
3. **Boot sequencing is critical.** The v2 architecture's boot sequence
|
||||
(foundation → core → management) is not a nice-to-have. Without it,
|
||||
manual recovery requires knowing the exact dependency order and the
|
||||
exact `podman run` commands for each service.
|
||||
|
||||
4. **The MCP agent should be able to recover containers from its
|
||||
registry.** After a podman database reset, the agent's SQLite
|
||||
registry still knows what should be running. A `mcp agent recover`
|
||||
command that recreates containers from the registry would eliminate
|
||||
the manual `podman run` recovery.
|
||||
|
||||
5. **Service definitions must include all runtime parameters.** The
|
||||
manual recovery required knowing port mappings, volume mounts,
|
||||
network modes, user overrides, and command arguments for each
|
||||
service. All of this is in the service definition files, but there
|
||||
was no tool to translate a service definition into a `podman run`
|
||||
command without the full MCP deploy pipeline.
|
||||
|
||||
6. **Tailscale MagicDNS amplifies DNS failures.** When MCNS is down
|
||||
and MagicDNS routes through it, ALL DNS breaks — not just internal
|
||||
names. Disabling Tailscale restores external DNS but loses Tailnet
|
||||
connectivity. The fix is fallback resolvers that bypass MCNS, not
|
||||
disabling Tailscale.
|
||||
|
||||
## Action Items
|
||||
|
||||
- [x] Write disaster recovery runbook → `docs/disaster-recovery.md`
|
||||
- [x] Add fallback DNS resolvers to NixOS config → all nodes now have
|
||||
1.1.1.1 and 8.8.8.8 as fallbacks after MCNS
|
||||
- [x] Implement `mcp agent recover` command → MCP v0.10.5. Recreates
|
||||
containers from the agent registry when podman DB is lost.
|
||||
- [x] Implement boot sequencing in the agent → MCP v0.10.6.
|
||||
[[boot.sequence]] config with per-stage health checks.
|
||||
- [x] Fix modernc.org/sqlite `fchmod` → was our own `os.Chmod` in
|
||||
`mcdsl/db/db.go`, not sqlite. Made best-effort in mcdsl v1.8.0.
|
||||
- [x] Add multi-address support to node config → MCP v0.10.4.
|
||||
Fallback addresses tried in order when primary fails.
|
||||
- [x] Stabilize mcp UID → pinned at 850 with NEVER CHANGE comment
|
||||
@@ -1,99 +0,0 @@
|
||||
[0] kyle@vade:~/ > cat ~/.config/mcp/services/mcq.toml
|
||||
name = 'mcq'
|
||||
node = 'rift'
|
||||
active = true
|
||||
path = '../mcq'
|
||||
|
||||
[build.images]
|
||||
mcq = 'Dockerfile'
|
||||
|
||||
[[components]]
|
||||
name = 'mcq'
|
||||
image = 'mcr.svc.mcp.metacircular.net:8443/mcq:v0.3.2'
|
||||
volumes = ['/srv/mcq:/srv/mcq']
|
||||
cmd = ['server', '--config', '/srv/mcq/mcq.toml']
|
||||
|
||||
[[components.routes]]
|
||||
port = 8443
|
||||
mode = 'l7'
|
||||
[0] kyle@vade:~/ > mcp node list
|
||||
NAME ADDRESS VERSION
|
||||
rift 100.95.252.120:9444 v0.7.9
|
||||
[0] kyle@vade:~/ > mcp ps
|
||||
SERVICE COMPONENT NODE STATE VERSION UPTIME
|
||||
exo exod rift running v0.1.0 2d16h
|
||||
kls kls rift running v0.1.2 23h57m
|
||||
mc-proxy mc-proxy rift running v1.1.1 17h16m
|
||||
mcat mcat rift running v1.1.1 1d0h
|
||||
mcdoc mcdoc rift running v0.1.0 2d1h
|
||||
mcns mcns rift running v1.1.1 1d2h
|
||||
mcq mcq rift running v0.3.1 2m19s
|
||||
mcr api rift running v1.2.1 1d2h
|
||||
mcr web rift running v1.2.1 1d2h
|
||||
metacrypt api rift running v1.3.1 22h6m
|
||||
metacrypt web rift running v1.3.1 22h46m
|
||||
sgard sgardd rift running v3.2.0 2d16h
|
||||
[0] kyle@vade:~/ > mcp deploy mcq
|
||||
image mcr.svc.mcp.metacircular.net:8443/mcq:v0.3.2 not found, building from Dockerfile
|
||||
pushing mcr.svc.mcp.metacircular.net:8443/mcq:v0.3.2
|
||||
mcq: ok
|
||||
[0] kyle@vade:~/ > mcp ps
|
||||
SERVICE COMPONENT NODE STATE VERSION UPTIME
|
||||
exo exod rift running v0.1.0 2d16h
|
||||
kls kls rift running v0.1.2 23h58m
|
||||
mc-proxy mc-proxy rift running v1.1.1 17h16m
|
||||
mcat mcat rift running v1.1.1 1d0h
|
||||
mcdoc mcdoc rift running v0.1.0 2d1h
|
||||
mcns mcns rift running v1.1.1 1d2h
|
||||
mcq mcq rift running v0.3.2 12s
|
||||
mcr api rift running v1.2.1 1d2h
|
||||
mcr web rift running v1.2.1 1d2h
|
||||
metacrypt api rift running v1.3.1 22h7m
|
||||
metacrypt web rift running v1.3.1 22h47m
|
||||
sgard sgardd rift running v3.2.0 2d16h
|
||||
[0] kyle@vade:~/ > mcp logs mcq -n 10
|
||||
time=2026-03-30T01:14:42.033Z level=INFO msg="mcq starting" version=dev addr=:57546
|
||||
[0] kyle@vade:~/ > mcp dns
|
||||
ZONE: mcp.metacircular.net
|
||||
NAME TYPE VALUE TTL
|
||||
ns A 100.95.252.120 300
|
||||
ns A 192.168.88.181 300
|
||||
ns A 71.19.144.164 300
|
||||
rift A 100.95.252.120 300
|
||||
rift A 192.168.88.181 300
|
||||
|
||||
ZONE: svc.mcp.metacircular.net
|
||||
NAME TYPE VALUE TTL
|
||||
kls A 192.168.88.181 300
|
||||
mcat A 192.168.88.181 300
|
||||
mcdoc A 100.95.252.120 300
|
||||
mcdoc A 192.168.88.181 300
|
||||
mcp-agent A 100.95.252.120 300
|
||||
mcp-agent A 192.168.88.181 300
|
||||
mcq A 192.168.88.181 300
|
||||
mcr A 100.95.252.120 300
|
||||
mcr A 192.168.88.181 300
|
||||
metacrypt A 100.95.252.120 300
|
||||
metacrypt A 192.168.88.181 300
|
||||
sgard A 100.95.252.120 300
|
||||
sgard A 192.168.88.181 300
|
||||
[0] kyle@vade:~/ > mcp route list -n rift
|
||||
NODE: rift
|
||||
mc-proxy v1.2.1
|
||||
uptime: 18h15m38s
|
||||
connections: 0
|
||||
|
||||
:443 routes=5 active=0
|
||||
l7 mcdoc.svc.mcp.metacircular.net → 127.0.0.1:38080
|
||||
l7 mcr.svc.mcp.metacircular.net → 127.0.0.1:28080 (re-encrypt)
|
||||
l7 metacrypt.svc.mcp.metacircular.net → 127.0.0.1:18080 (re-encrypt)
|
||||
l7 kls.svc.mcp.metacircular.net → 127.0.0.1:13881
|
||||
l4 mcat.svc.mcp.metacircular.net → 127.0.0.1:41235 (re-encrypt)
|
||||
:8443 routes=3 active=0
|
||||
l4 metacrypt.svc.mcp.metacircular.net → 127.0.0.1:18443
|
||||
l7 mcq.svc.mcp.metacircular.net → 127.0.0.1:57546
|
||||
l4 mcr.svc.mcp.metacircular.net → 127.0.0.1:28443
|
||||
:9443 routes=3 active=0
|
||||
l4 sgard.svc.mcp.metacircular.net → 127.0.0.1:19473
|
||||
l4 mcr.svc.mcp.metacircular.net → 127.0.0.1:29443
|
||||
l4 metacrypt.svc.mcp.metacircular.net → 127.0.0.1:19443
|
||||
Reference in New Issue
Block a user