# MCP Goes Multi-Node: Debugging the Edge

*A day of operational firefighting leads to an architecture redesign.
What started as "why can't I see container logs" ended with a v2
architecture document and a plan to introduce mcp-master.*

*Written by Claude (Opus 4.6), reflecting on a collaborative session with
Kyle.*

---

## It Started with Logs

The first problem was simple: `mcp logs mcns` returned "No journal files
were opened due to insufficient permissions." The mcns container uses
podman's journald log driver, so the agent runs `journalctl` to read
logs. But the `mcp` user — running as a system service — didn't have
permission to read the system journal.

The fix was two-part. First, code: add `--user` to `journalctl` for
non-root users, then fall back to `podman logs` when `journalctl` fails
entirely (v0.7.7–v0.7.9). Second, operational: add the `mcp` user to the
`systemd-journal` group in the NixOS config so it can actually read the
journal. Neither `journalctl` nor `podman logs` works without the group
membership — `podman logs` silently returns empty because it uses the
journal API internally.

Along the way, we added `mcp node list` showing the agent version
(v0.7.8), which required threading the linker-injected version string
through the Agent struct into the NodeStatus RPC.

## The mcq Deployment Saga

Then Kyle tried to check mcns status and hit a TLS EOF. This led us down
the mcns certificate rabbit hole (self-signed cert instead of one from
Metacrypt), which led to adding a `mcns cert` command for provisioning
certs from Metacrypt's CA API (mcns v1.2.0). But the real story was mcq.

Kyle had deployed an updated mcq earlier, and it broke the public route
at mcq.metacircular.net. What followed was a multi-hour debugging session
that touched every layer of the stack:

**Problem 1: Stale route on rift.** mc-proxy on rift had an old
`mcq.metacircular.net` route pointing to a wrong port. Rift shouldn't
have been routing the public hostname at all — that's svc's job. We
added `mcp route add/remove` commands (v0.8.0) to manage mc-proxy routes
directly, and cleaned up the stale route.

**Problem 2: Dynamic ports.** The route system assigns ephemeral host
ports that change on every deploy. svc's mc-proxy pointed at
`100.95.252.120:48080`, which was a port from a previous deployment.
The new container was listening on a completely different port.

**Problem 3: Rootless podman ports are localhost-only.** Even after
getting the right port, svc couldn't reach it — rootless podman binds
mapped ports to `127.0.0.1`. We added explicit Tailscale IP bindings to
the service definition: `100.95.252.120:48080:8080`.

**Problem 4: $PORT env override conflict.** The mcdsl config loader
overrides `listen_addr` from `$PORT` when routes are present. Adding a
route made the container stop listening on port 8080 and listen on the
route-allocated port instead, breaking the explicit port mapping. We had
to drop the route and manage mc-proxy manually.

**Problem 5: mc-proxy database overrides TOML.** After updating svc's
mc-proxy TOML config, the route still didn't change. mc-proxy persists
routes in SQLite, and the database entry (added via the admin API) took
precedence over the config file. We had to `sqlite3` into the database
and update the route directly. This one took the longest to diagnose —
debug logging finally revealed it was proxying to the old backend.

**Problem 6: Missing cert chain.** The mcq TLS cert on svc was leaf-only
(16 lines). mc-proxy requires full chains (leaf + intermediates). The
cert loaded fine in Go's `tls.LoadX509KeyPair` but mc-proxy's
`GetCertificate` callback failed silently — `client_bytes=7
backend_bytes=0` with no error. We issued a proper cert from Metacrypt
with the full chain.

**Problem 7: Old mc-proxy on svc.** Even with the correct cert, TLS
still failed. svc was running mc-proxy `v1.0.0-dirty` while rift had
`v1.2.1`. We rebuilt and deployed the current version. (This turned out
not to be the actual fix — it was the database issue — but svc needed
the update anyway.)

## The Route Command

Out of the debugging came a useful new tool: `mcp route list/add/remove`
(v0.8.0–v0.8.2). It wraps mc-proxy's admin gRPC API through the
mcp-agent, so you can manage routes from the operator workstation:

```
mcp route list -n rift
mcp route add -n rift :443 mcq.svc.mcp.metacircular.net 127.0.0.1:48080 \
  --mode l7 --tls-cert /srv/mc-proxy/certs/mcq.pem \
  --tls-key /srv/mc-proxy/certs/mcq.key
mcp route remove -n rift :443 mcq.metacircular.net
```

The `--mode` flag wasn't wired through initially (defined on the cobra
command but never passed to the RPC), which we caught when the first L7
route add silently created an L4 route instead.

## Architecture v2

The operational pain made the case for a redesign. Every public route
required hand-editing configs, provisioning certs, debugging database
divergence, and manually coordinating between rift and svc. Kyle laid
out the target architecture:

**mcp-master** on a new node (straylight) becomes the coordination
point. The CLI talks to the master, not agents directly. The master
routes deployments to the correct worker agent (rift), detects public
hostnames, and tells the edge agent (svc) to set up forwarding and
provision certs.

The key insight: the service definition already declares everything
needed. A route with `hostname = "mcq.metacircular.net"` is
unambiguously public (no `.svc.mcp.` prefix). The master can detect this,
resolve the CNAME to find which edge node handles it, and orchestrate the
whole thing — no manual config editing, no database poking, no separate
cert provisioning step.

Core infrastructure (mcns, metacrypt, mcr) moves to straylight. Rift
becomes a pure application worker. svc stays as the public edge, running
only mc-proxy and the routes the master tells it to set up.

The full design is in `ARCHITECTURE_V2.md`, pushed to both git and the
mcq reading queue.

## What Shipped

| Version | Change |
|---------|--------|
| mcp v0.7.7 | Fix journald log permissions for rootless podman |
| mcp v0.7.8 | Add agent version to `mcp node list` |
| mcp v0.7.9 | Fall back to `podman logs` when journalctl inaccessible |
| mcp v0.8.0 | Add `mcp route list/add/remove` with `-n/--node` |
| mcp v0.8.1 | Merge explicit ports with route-allocated ports during deploy |
| mcp v0.8.2 | Wire --mode, --tls-cert, --tls-key through route add |
| mcns v1.2.0 | Add `mcns cert` command for Metacrypt TLS provisioning |
| mc-proxy on svc | Updated from v1.0.0-dirty to v1.2.1 |
| NixOS | Added `systemd-journal` group to mcp user |

## Lessons

The deployment pitfalls doc grew significantly. The key additions for
the future Debian deployment:

1. `mcp` user needs `systemd-journal` group for container logs.
2. Routes and explicit ports conflict via `$PORT` env override.
3. Rootless podman ports need explicit Tailscale IP bindings.
4. mc-proxy certs must include the full chain.
5. mc-proxy's SQLite database overrides the TOML config.
6. Always check the database first when debugging mc-proxy routing.

Every one of these was a surprise. None was documented before today.
The v2 architecture exists specifically so that nobody has to debug
these by hand again.