Add README.md and RUNBOOK.md per engineering standards
Create the two required documentation files for MCNS: - README.md: project overview, quick-start (build/configure/run), links to ARCHITECTURE.md and RUNBOOK.md - RUNBOOK.md: operator-focused procedures with numbered steps covering health checks, start/stop/restart, backup/restore, log inspection, and incident playbooks for database corruption, certificate expiry, MCIAS outage, DNS resolution failures, and port conflicts Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
42
README.md
Normal file
42
README.md
Normal file
@@ -0,0 +1,42 @@
|
|||||||
|
# MCNS
|
||||||
|
|
||||||
|
Metacircular Networking Service -- an authoritative DNS server for the
|
||||||
|
Metacircular platform. MCNS serves DNS zones backed by SQLite, forwards
|
||||||
|
non-authoritative queries to upstream resolvers, and exposes a gRPC and
|
||||||
|
REST management API authenticated through MCIAS. Records are updated
|
||||||
|
dynamically via the API and visible to DNS immediately on commit.
|
||||||
|
|
||||||
|
## Quick Start
|
||||||
|
|
||||||
|
Build the binary:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
make mcns
|
||||||
|
```
|
||||||
|
|
||||||
|
Copy and edit the example configuration:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cp deploy/examples/mcns.toml /srv/mcns/mcns.toml
|
||||||
|
# Edit TLS paths, database path, MCIAS URL, upstream resolvers
|
||||||
|
```
|
||||||
|
|
||||||
|
Run the server:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./mcns server --config /srv/mcns/mcns.toml
|
||||||
|
```
|
||||||
|
|
||||||
|
The server starts three listeners:
|
||||||
|
|
||||||
|
| Port | Protocol | Purpose |
|
||||||
|
|------|----------|---------|
|
||||||
|
| 53 | UDP + TCP | DNS (no auth) |
|
||||||
|
| 8443 | TCP | REST management API (TLS, MCIAS auth) |
|
||||||
|
| 9443 | TCP | gRPC management API (TLS, MCIAS auth) |
|
||||||
|
|
||||||
|
## Documentation
|
||||||
|
|
||||||
|
- [ARCHITECTURE.md](ARCHITECTURE.md) -- full technical specification, database schema, API surface, and security model.
|
||||||
|
- [RUNBOOK.md](RUNBOOK.md) -- operational procedures and incident response for operators.
|
||||||
|
- [CLAUDE.md](CLAUDE.md) -- context for AI-assisted development.
|
||||||
242
RUNBOOK.md
Normal file
242
RUNBOOK.md
Normal file
@@ -0,0 +1,242 @@
|
|||||||
|
# MCNS Runbook
|
||||||
|
|
||||||
|
## Service Overview
|
||||||
|
|
||||||
|
MCNS is an authoritative DNS server for the Metacircular platform. It
|
||||||
|
listens on port 53 (UDP+TCP) for DNS queries, port 8443 for the REST
|
||||||
|
management API, and port 9443 for the gRPC management API. Zone and
|
||||||
|
record data is stored in SQLite. All management operations require MCIAS
|
||||||
|
authentication; DNS queries are unauthenticated.
|
||||||
|
|
||||||
|
## Health Checks
|
||||||
|
|
||||||
|
### CLI
|
||||||
|
|
||||||
|
```bash
|
||||||
|
mcns status --addr https://localhost:8443
|
||||||
|
```
|
||||||
|
|
||||||
|
With a custom CA certificate:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
mcns status --addr https://localhost:8443 --ca-cert /srv/mcns/certs/ca.pem
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected output: `ok`
|
||||||
|
|
||||||
|
### REST
|
||||||
|
|
||||||
|
```bash
|
||||||
|
curl -k https://localhost:8443/v1/health
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected: HTTP 200.
|
||||||
|
|
||||||
|
### gRPC
|
||||||
|
|
||||||
|
Use the `AdminService.Health` RPC on port 9443. This method is public
|
||||||
|
(no auth required).
|
||||||
|
|
||||||
|
### DNS
|
||||||
|
|
||||||
|
```bash
|
||||||
|
dig @localhost svc.mcp.metacircular.net SOA +short
|
||||||
|
```
|
||||||
|
|
||||||
|
A valid SOA response confirms the DNS listener and database are working.
|
||||||
|
|
||||||
|
## Common Operations
|
||||||
|
|
||||||
|
### Start the Service
|
||||||
|
|
||||||
|
1. Verify config exists: `ls /srv/mcns/mcns.toml`
|
||||||
|
2. Start the container:
|
||||||
|
```bash
|
||||||
|
docker compose -f deploy/docker/docker-compose-rift.yml up -d
|
||||||
|
```
|
||||||
|
3. Verify health:
|
||||||
|
```bash
|
||||||
|
mcns status --addr https://localhost:8443
|
||||||
|
```
|
||||||
|
|
||||||
|
### Stop the Service
|
||||||
|
|
||||||
|
1. Stop the container:
|
||||||
|
```bash
|
||||||
|
docker compose -f deploy/docker/docker-compose-rift.yml stop mcns
|
||||||
|
```
|
||||||
|
2. MCNS handles SIGTERM gracefully and drains in-flight requests (30s timeout).
|
||||||
|
|
||||||
|
### Restart the Service
|
||||||
|
|
||||||
|
1. Restart the container:
|
||||||
|
```bash
|
||||||
|
docker compose -f deploy/docker/docker-compose-rift.yml restart mcns
|
||||||
|
```
|
||||||
|
2. Verify health:
|
||||||
|
```bash
|
||||||
|
mcns status --addr https://localhost:8443
|
||||||
|
```
|
||||||
|
|
||||||
|
### Backup (Snapshot)
|
||||||
|
|
||||||
|
1. Run the snapshot command:
|
||||||
|
```bash
|
||||||
|
mcns snapshot --config /srv/mcns/mcns.toml
|
||||||
|
```
|
||||||
|
2. The snapshot is saved to `/srv/mcns/backups/mcns-YYYYMMDD-HHMMSS.db`.
|
||||||
|
3. Verify the snapshot file exists and has a reasonable size:
|
||||||
|
```bash
|
||||||
|
ls -lh /srv/mcns/backups/
|
||||||
|
```
|
||||||
|
|
||||||
|
### Restore from Snapshot
|
||||||
|
|
||||||
|
1. Stop the service (see above).
|
||||||
|
2. Back up the current database:
|
||||||
|
```bash
|
||||||
|
cp /srv/mcns/mcns.db /srv/mcns/mcns.db.pre-restore
|
||||||
|
```
|
||||||
|
3. Copy the snapshot into place:
|
||||||
|
```bash
|
||||||
|
cp /srv/mcns/backups/mcns-YYYYMMDD-HHMMSS.db /srv/mcns/mcns.db
|
||||||
|
```
|
||||||
|
4. Start the service (see above).
|
||||||
|
5. Verify the service is healthy:
|
||||||
|
```bash
|
||||||
|
mcns status --addr https://localhost:8443
|
||||||
|
```
|
||||||
|
6. Verify zones are accessible by querying DNS:
|
||||||
|
```bash
|
||||||
|
dig @localhost svc.mcp.metacircular.net SOA +short
|
||||||
|
```
|
||||||
|
|
||||||
|
### Log Inspection
|
||||||
|
|
||||||
|
Container logs:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker compose -f deploy/docker/docker-compose-rift.yml logs --tail 100 mcns
|
||||||
|
```
|
||||||
|
|
||||||
|
Follow logs in real time:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker compose -f deploy/docker/docker-compose-rift.yml logs -f mcns
|
||||||
|
```
|
||||||
|
|
||||||
|
MCNS logs to stderr as structured text (slog). Log level is configured
|
||||||
|
via `[log] level` in `mcns.toml` (debug, info, warn, error).
|
||||||
|
|
||||||
|
## Incident Procedures
|
||||||
|
|
||||||
|
### Database Corruption
|
||||||
|
|
||||||
|
Symptoms: server fails to start with SQLite errors, or queries return
|
||||||
|
unexpected errors.
|
||||||
|
|
||||||
|
1. Stop the service.
|
||||||
|
2. Check for WAL/SHM files alongside the database:
|
||||||
|
```bash
|
||||||
|
ls -la /srv/mcns/mcns.db*
|
||||||
|
```
|
||||||
|
3. Attempt an integrity check:
|
||||||
|
```bash
|
||||||
|
sqlite3 /srv/mcns/mcns.db "PRAGMA integrity_check;"
|
||||||
|
```
|
||||||
|
4. If integrity check fails, restore from the most recent snapshot:
|
||||||
|
```bash
|
||||||
|
cp /srv/mcns/mcns.db /srv/mcns/mcns.db.corrupt
|
||||||
|
cp /srv/mcns/backups/mcns-YYYYMMDD-HHMMSS.db /srv/mcns/mcns.db
|
||||||
|
```
|
||||||
|
5. Start the service and verify health.
|
||||||
|
6. Re-create any records added after the snapshot was taken.
|
||||||
|
|
||||||
|
### Certificate Expiry
|
||||||
|
|
||||||
|
Symptoms: health check fails with TLS errors, API clients get
|
||||||
|
certificate errors.
|
||||||
|
|
||||||
|
1. Check certificate expiry:
|
||||||
|
```bash
|
||||||
|
openssl x509 -in /srv/mcns/certs/cert.pem -noout -enddate
|
||||||
|
```
|
||||||
|
2. Replace the certificate and key files at the paths in `mcns.toml`.
|
||||||
|
3. Restart the service to load the new certificate.
|
||||||
|
4. Verify health:
|
||||||
|
```bash
|
||||||
|
mcns status --addr https://localhost:8443
|
||||||
|
```
|
||||||
|
|
||||||
|
### MCIAS Outage
|
||||||
|
|
||||||
|
Symptoms: management API returns 502 or authentication errors. DNS
|
||||||
|
continues to work normally (DNS has no auth dependency).
|
||||||
|
|
||||||
|
1. Confirm MCIAS is unreachable:
|
||||||
|
```bash
|
||||||
|
curl -k https://svc.metacircular.net:8443/v1/health
|
||||||
|
```
|
||||||
|
2. DNS resolution is unaffected -- no immediate action needed for DNS.
|
||||||
|
3. Management operations (zone/record create/update/delete) will fail
|
||||||
|
until MCIAS recovers.
|
||||||
|
4. Escalate to MCIAS (see Escalation below).
|
||||||
|
|
||||||
|
### DNS Not Resolving
|
||||||
|
|
||||||
|
Symptoms: `dig @<server> <name>` returns SERVFAIL or times out.
|
||||||
|
|
||||||
|
1. Verify the service is running:
|
||||||
|
```bash
|
||||||
|
docker compose -f deploy/docker/docker-compose-rift.yml ps mcns
|
||||||
|
```
|
||||||
|
2. Check that port 53 is listening:
|
||||||
|
```bash
|
||||||
|
ss -ulnp | grep ':53'
|
||||||
|
ss -tlnp | grep ':53'
|
||||||
|
```
|
||||||
|
3. Test an authoritative query:
|
||||||
|
```bash
|
||||||
|
dig @localhost svc.mcp.metacircular.net SOA
|
||||||
|
```
|
||||||
|
4. Test a forwarded query:
|
||||||
|
```bash
|
||||||
|
dig @localhost example.com A
|
||||||
|
```
|
||||||
|
5. If authoritative queries fail but forwarding works, the database may
|
||||||
|
be corrupt (see Database Corruption above).
|
||||||
|
6. If forwarding fails, check upstream connectivity:
|
||||||
|
```bash
|
||||||
|
dig @1.1.1.1 example.com A
|
||||||
|
```
|
||||||
|
7. Check logs for errors:
|
||||||
|
```bash
|
||||||
|
docker compose -f deploy/docker/docker-compose-rift.yml logs --tail 50 mcns
|
||||||
|
```
|
||||||
|
|
||||||
|
### Port 53 Already in Use
|
||||||
|
|
||||||
|
Symptoms: MCNS fails to start with "address already in use" on port 53.
|
||||||
|
|
||||||
|
1. Identify what is using the port:
|
||||||
|
```bash
|
||||||
|
ss -ulnp | grep ':53'
|
||||||
|
ss -tlnp | grep ':53'
|
||||||
|
```
|
||||||
|
2. Common culprit: `systemd-resolved` listening on `127.0.0.53:53`.
|
||||||
|
- If on a system with systemd-resolved, either disable it or bind
|
||||||
|
MCNS to a specific IP instead of `0.0.0.0:53`.
|
||||||
|
3. If another DNS server is running, stop it or change the MCNS
|
||||||
|
`[dns] listen_addr` in `mcns.toml` to a different address.
|
||||||
|
4. Restart MCNS and verify DNS is responding.
|
||||||
|
|
||||||
|
## Escalation
|
||||||
|
|
||||||
|
Escalate when:
|
||||||
|
|
||||||
|
- Database corruption cannot be resolved by restoring a snapshot.
|
||||||
|
- MCIAS is down and management operations are urgently needed.
|
||||||
|
- DNS resolution failures persist after following the procedures above.
|
||||||
|
- Any issue not covered by this runbook.
|
||||||
|
|
||||||
|
Escalation path: Kyle (platform owner).
|
||||||
Reference in New Issue
Block a user