Merge transit engine branch, resolve conflicts in shared files

2026-03-16 19:50:47 -07:00
parent 5ae37da300 cbd77c58e8
commit 11929daa78
14 changed files with 7969 additions and 0 deletions
--- a/docs/engineering-standards.md
+++ b/docs/engineering-standards.md
@@ -0,0 +1,713 @@
+# Metacircular Dynamics — Engineering Standards
+
+This document describes the standard repository layout, tooling, and software
+development lifecycle (SDLC) for services built at Metacircular Dynamics. It is
+derived from the conventions established in Metacrypt and codifies them as the
+baseline for all new and existing services.
+
+## Table of Contents
+
+1. [Repository Layout](#repository-layout)
+2. [Language & Toolchain](#language--toolchain)
+3. [Build System](#build-system)
+4. [API Design](#api-design)
+5. [Authentication & Authorization](#authentication--authorization)
+6. [Database Conventions](#database-conventions)
+7. [Configuration](#configuration)
+8. [Web UI](#web-ui)
+9. [Testing](#testing)
+10. [Linting & Static Analysis](#linting--static-analysis)
+11. [Deployment](#deployment)
+12. [Documentation](#documentation)
+13. [Security](#security)
+14. [Development Workflow](#development-workflow)
+
+---
+
+## Repository Layout
+
+Every service follows a consistent directory structure. Adjust the
+service-specific directories (e.g. `engines/` in Metacrypt) as appropriate,
+but the top-level skeleton is fixed.
+
+```
+.
+├── cmd/
+│   ├── <service>/              CLI entry point (server, subcommands)
+│   └── <service>-web/          Web UI entry point (if separate binary)
+├── internal/
+│   ├── auth/                   MCIAS integration (token validation, caching)
+│   ├── config/                 TOML configuration loading & validation
+│   ├── db/                     Database setup, schema migrations
+│   ├── server/                 REST API server, routes, middleware
+│   ├── grpcserver/             gRPC server, interceptors, service handlers
+│   ├── webserver/              Web UI server, template routes, HTMX handlers
+│   └── <domain>/               Service-specific packages
+├── proto/<service>/
+│   ├── v1/                     Legacy proto definitions (if applicable)
+│   └── v2/                     Current proto definitions
+├── gen/<service>/
+│   ├── v1/                     Generated Go gRPC/protobuf code
+│   └── v2/
+├── web/
+│   ├── embed.go                //go:embed directive for templates and static
+│   ├── templates/              Go HTML templates
+│   └── static/                 CSS, JS (htmx)
+├── deploy/
+│   ├── docker/                 Docker Compose configuration
+│   ├── examples/               Example config files
+│   ├── scripts/                Install, backup, migration scripts
+│   └── systemd/                systemd unit files and timers
+├── docs/                       Internal engineering documentation
+├── Dockerfile.api              API server container (if split binary)
+├── Dockerfile.web              Web UI container (if split binary)
+├── Makefile
+├── buf.yaml                    Protobuf linting & breaking-change config
+├── .golangci.yaml              Linter configuration
+├── .gitignore
+├── CLAUDE.md                   AI-assisted development instructions
+├── ARCHITECTURE.md             Full system specification
+└── <service>.toml.example      Example configuration
+```
+
+### Key Principles
+
+- **`cmd/`** contains only CLI wiring (cobra commands, flag parsing). No
+  business logic.
+- **`internal/`** contains all service logic. Nothing in `internal/` is
+  importable by other modules — this is enforced by Go's module system.
+- **`proto/`** is the source of truth for gRPC definitions. Generated code
+  lives in `gen/`, never edited by hand.
+- **`deploy/`** contains everything needed to run the service in production.
+  A new engineer should be able to deploy from this directory alone.
+- **`web/`** is embedded into the binary via `//go:embed`. No external file
+  dependencies at runtime.
+
+### What Does Not Belong in the Repository
+
+- Runtime data (databases, certificates, logs) — these live in `/srv/<service>`
+- Real configuration files with secrets — only examples are committed
+- IDE configuration (`.idea/`, `.vscode/`) — per-developer, not shared
+- Vendored dependencies — Go module proxy handles this
+
+---
+
+## Language & Toolchain
+
+| Tool | Version | Purpose |
+|------|---------|---------|
+| Go | 1.25+ | Primary language |
+| protoc + protoc-gen-go | Latest | Protobuf/gRPC code generation |
+| buf | Latest | Proto linting and breaking-change detection |
+| golangci-lint | v2 | Static analysis and linting |
+| Docker | Latest | Container builds |
+
+### Go Conventions
+
+- **Pure-Go dependencies** where possible. Avoid CGo — it complicates
+  cross-compilation and container builds. Use `modernc.org/sqlite` instead
+  of `mattn/go-sqlite3`.
+- **`CGO_ENABLED=0`** for all production builds. Statically linked binaries
+  deploy cleanly to Alpine containers.
+- **Stripped binaries**: Build with `-trimpath -ldflags="-s -w"` to remove
+  debug symbols and reduce image size.
+- **Version injection**: Pass `git describe --tags --always --dirty` via
+  `-X main.version=...` at build time. Every binary must report its version.
+
+### Module Path
+
+Services hosted on `git.wntrmute.dev` use:
+
+```
+git.wntrmute.dev/kyle/<service>
+```
+
+---
+
+## Build System
+
+Every repository has a Makefile with these standard targets:
+
+```makefile
+.PHONY: build test vet lint proto-lint clean docker all
+
+LDFLAGS := -trimpath -ldflags="-s -w -X main.version=$(shell git describe --tags --always --dirty)"
+
+<service>:
+	go build $(LDFLAGS) -o <service> ./cmd/<service>
+
+build:
+	go build ./...
+
+test:
+	go test ./...
+
+vet:
+	go vet ./...
+
+lint:
+	golangci-lint run ./...
+
+proto:
+	protoc --go_out=. --go_opt=module=<module> \
+		--go-grpc_out=. --go-grpc_opt=module=<module> \
+		proto/<service>/v2/*.proto
+
+proto-lint:
+	buf lint
+	buf breaking --against '.git#branch=master,subdir=proto'
+
+clean:
+	rm -f <service>
+
+docker:
+	docker build -t <service> -f Dockerfile.api .
+
+all: vet lint test <service>
+```
+
+### Target Semantics
+
+| Target | When to Run | CI Gate? |
+|--------|-------------|----------|
+| `vet` | Every change | Yes |
+| `lint` | Every change | Yes |
+| `test` | Every change | Yes |
+| `proto-lint` | Any proto change | Yes |
+| `proto` | After editing `.proto` files | No (manual) |
+| `all` | Pre-push verification | Yes |
+
+The `all` target is the CI pipeline: `vet → lint → test → build`. If any
+step fails, the pipeline stops.
+
+---
+
+## API Design
+
+Services expose two synchronized API surfaces:
+
+### gRPC (Primary)
+
+- Proto definitions live in `proto/<service>/v2/`.
+- Use strongly-typed, per-operation RPCs. Avoid generic "execute" patterns.
+- Use `google.protobuf.Timestamp` for all time fields (not RFC 3339 strings).
+- Run `buf lint` and `buf breaking` against master before merging proto
+  changes.
+
+### REST (Secondary)
+
+- JSON over HTTPS. Routes live in `internal/server/routes.go`.
+- Use `chi` for routing (lightweight, stdlib-compatible).
+- Standard error format: `{"error": "description"}`.
+- Standard HTTP status codes: `401` (unauthenticated), `403` (unauthorized),
+  `412` (precondition failed), `503` (service unavailable).
+
+### API Sync Rule
+
+**Every REST endpoint must have a corresponding gRPC RPC, and vice versa.**
+When adding, removing, or changing an endpoint in either surface, the other
+must be updated in the same change. This is enforced in code review.
+
+### gRPC Interceptors
+
+Access control is enforced via interceptor maps, not per-handler checks:
+
+| Map | Effect |
+|-----|--------|
+| `sealRequiredMethods` | Returns `UNAVAILABLE` if the service is sealed/locked |
+| `authRequiredMethods` | Validates MCIAS bearer token, populates caller info |
+| `adminRequiredMethods` | Requires admin role on the caller |
+
+Adding a new RPC means adding it to the correct interceptor maps. Forgetting
+this is a security defect.
+
+---
+
+## Authentication & Authorization
+
+### Authentication
+
+All services delegate authentication to **MCIAS** (Metacircular Identity and
+Access Service). No service maintains its own user database.
+
+- Client sends credentials to the service's `/v1/auth/login` endpoint.
+- The service forwards them to MCIAS via the client library
+  (`git.wntrmute.dev/kyle/mcias/clients/go`).
+- On success, MCIAS returns a bearer token. The service returns it to the
+  client and optionally sets it as a cookie for the web UI.
+- Subsequent requests include the token via `Authorization: Bearer <token>`
+  header or cookie.
+- Token validation calls MCIAS `ValidateToken()`. Results should be cached
+  (keyed by SHA-256 of the token) with a short TTL (30 seconds or less).
+
+### Authorization
+
+Three role levels:
+
+| Role | Meaning |
+|------|---------|
+| `admin` | Full access to everything. Policy bypass. |
+| `user` | Access governed by policy rules. Default deny. |
+| `guest` | Service-dependent restrictions. Default deny. |
+
+Admin detection is based solely on the MCIAS `admin` role. The service never
+promotes users locally.
+
+Services that need fine-grained access control should implement a policy
+engine (priority-based ACL rules stored in encrypted storage, default deny,
+admin bypass). See Metacrypt's implementation as the reference.
+
+---
+
+## Database Conventions
+
+### SQLite
+
+SQLite is the default database for Metacircular services. It is simple to
+operate, requires no external processes, and backs up cleanly with
+`VACUUM INTO`.
+
+Connection settings (applied at open time):
+
+```go
+PRAGMA journal_mode = WAL;
+PRAGMA foreign_keys = ON;
+PRAGMA busy_timeout = 5000;
+```
+
+File permissions: `0600`. Created by the service on first run.
+
+### Migrations
+
+- Migrations are Go functions registered in `internal/db/` and run
+  sequentially at startup.
+- Each migration is idempotent — `CREATE TABLE IF NOT EXISTS`,
+  `ALTER TABLE ... ADD COLUMN IF NOT EXISTS`.
+- Applied migrations are tracked in a `schema_migrations` table.
+- Never modify a migration that has been deployed. Add a new one.
+
+### Backup
+
+Every service must provide a `snapshot` CLI command that creates a consistent
+backup using `VACUUM INTO`. Automated backups run via a systemd timer
+(daily, with retention pruning).
+
+---
+
+## Configuration
+
+### Format
+
+TOML. Parsed with `go-toml/v2`. Environment variable overrides via
+`SERVICENAME_*` (e.g. `METACRYPT_SERVER_LISTEN_ADDR`).
+
+### Standard Sections
+
+```toml
+[server]
+listen_addr = ":8443"           # HTTPS API
+grpc_addr   = ":9443"           # gRPC (optional; disabled if unset)
+tls_cert    = "/srv/<service>/certs/cert.pem"
+tls_key     = "/srv/<service>/certs/key.pem"
+
+[web]
+listen_addr  = "127.0.0.1:8080" # Web UI (optional; disabled if unset)
+vault_grpc   = "127.0.0.1:9443" # gRPC address of the API server
+vault_ca_cert = ""               # CA cert for verifying API server TLS
+
+[database]
+path = "/srv/<service>/<service>.db"
+
+[mcias]
+server_url = "https://mcias.metacircular.net:8443"
+ca_cert    = ""                  # Custom CA for MCIAS TLS
+
+[log]
+level = "info"                   # debug, info, warn, error
+```
+
+### Validation
+
+Required fields are validated at startup. The service refuses to start if
+any are missing. Do not silently default required values.
+
+### Data Directory
+
+All runtime data lives in `/srv/<service>/`:
+
+```
+/srv/<service>/
+├── <service>.toml        Configuration
+├── <service>.db          SQLite database
+├── certs/                TLS certificates
+└── backups/              Database snapshots
+```
+
+This convention enables straightforward service migration between hosts:
+copy `/srv/<service>/` and the binary.
+
+---
+
+## Web UI
+
+### Technology
+
+- **Go `html/template`** for server-side rendering. No JavaScript frameworks.
+- **htmx** for dynamic interactions (form submission, partial page updates)
+  without full page reloads.
+- Templates and static files are embedded in the binary via `//go:embed`.
+
+### Structure
+
+- `web/templates/layout.html` — shared HTML skeleton, navigation, CSS/JS
+  includes. All page templates extend this.
+- Page templates: one `.html` file per page/feature.
+- `web/static/` — CSS, htmx. Keep this minimal.
+
+### Architecture
+
+The web UI runs as a separate binary (`<service>-web`) that communicates
+with the API server via its gRPC interface. This separation means:
+
+- The web UI has no direct database access.
+- The API server enforces all authorization.
+- The web UI can be deployed independently or omitted entirely.
+
+### Security
+
+- CSRF protection via signed double-submit cookies on all mutating requests
+  (POST/PUT/PATCH/DELETE).
+- Session cookie: `HttpOnly`, `Secure`, `SameSite=Strict`.
+- All user input is escaped by `html/template` (the default).
+
+---
+
+## Testing
+
+### Philosophy
+
+Tests are written using the Go standard library `testing` package. No test
+frameworks (testify, gomega, etc.) — the standard library is sufficient and
+keeps dependencies minimal.
+
+### Patterns
+
+```go
+func TestFeatureName(t *testing.T) {
+    // Setup: use t.TempDir() for isolated file system state.
+    dir := t.TempDir()
+    database, err := db.Open(filepath.Join(dir, "test.db"))
+    if err != nil {
+        t.Fatalf("open db: %v", err)
+    }
+    defer func() { _ = database.Close() }()
+    db.Migrate(database)
+
+    // Exercise the code under test.
+    // ...
+
+    // Assert with t.Fatal (not t.Error) for precondition failures.
+    if !bytes.Equal(got, want) {
+        t.Fatalf("got %q, want %q", got, want)
+    }
+}
+```
+
+### Guidelines
+
+- **Use `t.TempDir()`** for all file-system state. Never write to fixed
+  paths. Cleanup is automatic.
+- **Use `errors.Is`** for error assertions, not string comparison.
+- **No mocks for databases.** Tests use real SQLite databases created in
+  temp directories. This catches migration bugs that mocks would hide.
+- **Test files** live alongside the code they test: `barrier.go` and
+  `barrier_test.go` in the same package.
+- **Test helpers** call `t.Helper()` so failures report the caller's line.
+
+### What to Test
+
+| Layer | Test Strategy |
+|-------|---------------|
+| Crypto primitives | Roundtrip encryption/decryption, wrong-key rejection, edge cases |
+| Storage (barrier, DB) | CRUD operations, sealed-state rejection, concurrent access |
+| API handlers | Request/response correctness, auth enforcement, error codes |
+| Policy engine | Rule matching, priority ordering, default deny, admin bypass |
+| CLI commands | Flag parsing, output format (lightweight) |
+
+---
+
+## Linting & Static Analysis
+
+### Configuration
+
+Every repository includes a `.golangci.yaml` with this philosophy:
+**fail loudly for security and correctness; everything else is a warning.**
+
+### Required Linters
+
+| Linter | Category | Purpose |
+|--------|----------|---------|
+| `errcheck` | Correctness | Unhandled errors are silent failures |
+| `govet` | Correctness | Printf mismatches, unreachable code, suspicious constructs |
+| `ineffassign` | Correctness | Dead writes hide logic bugs |
+| `unused` | Correctness | Unused variables and functions |
+| `errorlint` | Error handling | Proper `errors.Is`/`errors.As` usage |
+| `gosec` | Security | Hardcoded secrets, weak RNG, insecure crypto, SQL injection |
+| `staticcheck` | Security | Deprecated APIs, mutex misuse, deep analysis |
+| `revive` | Style | Go naming conventions, error return ordering |
+| `gofmt` | Formatting | Standard Go formatting |
+| `goimports` | Formatting | Import grouping and ordering |
+
+### Settings
+
+- `errcheck`: `check-type-assertions: true` (catch `x.(*T)` without ok check).
+- `govet`: all analyzers enabled except `shadow` (too noisy for idiomatic Go).
+- `gosec`: severity and confidence set to `medium`. Exclude `G104` (overlaps
+  with errcheck).
+- `max-issues-per-linter: 0` — report everything. No caps.
+- Test files: allow `G101` (hardcoded credentials) for test fixtures.
+
+---
+
+## Deployment
+
+### Container-First
+
+Services are designed for container deployment but must also run as native
+systemd services. Both paths are first-class.
+
+### Docker
+
+Multi-stage builds:
+
+1. **Builder**: `golang:1.23-alpine`. Compile with `CGO_ENABLED=0`, strip
+   symbols.
+2. **Runtime**: `alpine:3.21`. Non-root user (`<service>`), minimal attack
+   surface.
+
+If the service has separate API and web binaries, use separate Dockerfiles
+(`Dockerfile.api`, `Dockerfile.web`) and a `docker-compose.yml` that wires
+them together with a shared data volume.
+
+### systemd
+
+Every service ships with:
+
+| File | Purpose |
+|------|---------|
+| `<service>.service` | Main service unit (API server) |
+| `<service>-web.service` | Web UI unit (if applicable) |
+| `<service>-backup.service` | Oneshot backup unit |
+| `<service>-backup.timer` | Daily backup timer (02:00 UTC, 5-minute jitter) |
+
+#### Security Hardening
+
+All service units must include these security directives:
+
+```ini
+NoNewPrivileges=true
+ProtectSystem=strict
+ProtectHome=true
+PrivateTmp=true
+PrivateDevices=true
+ProtectKernelTunables=true
+ProtectKernelModules=true
+ProtectControlGroups=true
+RestrictSUIDSGID=true
+RestrictNamespaces=true
+LockPersonality=true
+MemoryDenyWriteExecute=true
+RestrictRealtime=true
+ReadWritePaths=/srv/<service>
+```
+
+The web UI unit should use `ReadOnlyPaths=/srv/<service>` instead of
+`ReadWritePaths` — it has no reason to write to the data directory.
+
+### Install Script
+
+`deploy/scripts/install.sh` handles:
+
+1. Create system user/group (idempotent).
+2. Install binary to `/usr/local/bin/`.
+3. Create `/srv/<service>/` directory structure.
+4. Install example config if none exists.
+5. Install systemd units and reload the daemon.
+
+### TLS
+
+- **Minimum TLS version: 1.3.** No exceptions, no fallback cipher suites.
+  Go's TLS 1.3 implementation manages cipher selection automatically.
+- **Timeouts**: read 30s, write 30s, idle 120s.
+- Certificate and key paths are required configuration — the service refuses
+  to start without them.
+
+### Graceful Shutdown
+
+Services handle `SIGINT` and `SIGTERM`, shutting down cleanly:
+
+1. Stop accepting new connections.
+2. Drain in-flight requests (with a timeout).
+3. Clean up resources (close databases, zeroize secrets if applicable).
+4. Exit.
+
+---
+
+## Documentation
+
+### Required Files
+
+| File | Purpose | Audience |
+|------|---------|----------|
+| `CLAUDE.md` | AI-assisted development context | Claude Code |
+| `ARCHITECTURE.md` | Full system specification | Engineers |
+| `deploy/examples/<service>.toml` | Example configuration | Operators |
+
+### CLAUDE.md
+
+This file provides context for AI-assisted development. It should contain:
+
+- Project overview (one paragraph).
+- Build, test, and lint commands.
+- High-level architecture summary.
+- Project structure with directory descriptions.
+- Ignored directories (runtime data, generated code).
+- Critical rules (e.g. API sync requirements).
+
+Keep it concise. AI tools read this on every interaction.
+
+### ARCHITECTURE.md
+
+This is the canonical specification for the service. It should cover:
+
+1. System overview with a layered architecture diagram.
+2. Cryptographic design (if applicable): algorithms, key hierarchy.
+3. State machines and lifecycle (if applicable).
+4. Storage design.
+5. Authentication and authorization model.
+6. API surface (REST and gRPC, with tables of every endpoint).
+7. Web interface routes.
+8. Database schema (every table, every column).
+9. Configuration reference.
+10. Deployment guide.
+11. Security model: threat mitigations table and security invariants.
+12. Future work.
+
+This document is the source of truth. When the code and the spec disagree,
+one of them has a bug.
+
+### Engine/Feature Design Documents
+
+For services with a modular architecture, each module gets its own design
+document (e.g. `engines/sshca.md`). These are detailed implementation plans
+that include:
+
+- Overview and core concepts.
+- Data model and storage layout.
+- Lifecycle (initialization, teardown).
+- Operations table with auth requirements.
+- API definitions (gRPC and REST).
+- Implementation steps (file-by-file).
+- Security considerations.
+- References to existing code patterns to follow.
+
+Write these before writing code. They are the blueprint, not the afterthought.
+
+---
+
+## Security
+
+### General Principles
+
+- **Default deny.** Unauthenticated requests are rejected. Unauthorized
+  requests are rejected. If in doubt, deny.
+- **Fail closed.** If the service cannot verify authorization, it denies the
+  request. If the database is unavailable, the service is unavailable.
+- **Least privilege.** Service processes run as non-root. systemd units
+  restrict filesystem access, syscalls, and capabilities.
+- **No local user databases.** Authentication is always delegated to MCIAS.
+
+### Cryptographic Standards
+
+| Purpose | Algorithm | Notes |
+|---------|-----------|-------|
+| Symmetric encryption | AES-256-GCM | 12-byte random nonce per operation |
+| Symmetric alternative | XChaCha20-Poly1305 | For contexts needing nonce misuse resistance |
+| Key derivation | Argon2id | Memory-hard; tune params to hardware |
+| Asymmetric signing | Ed25519, ECDSA (P-256, P-384) | Prefer Ed25519 |
+| CSPRNG | `crypto/rand` | All keys, nonces, salts, tokens |
+| Constant-time comparison | `crypto/subtle` | All secret comparisons |
+
+- **Never use RSA for new designs.** Ed25519 and ECDSA are faster, produce
+  smaller keys, and have simpler security models.
+- **Zeroize secrets** from memory when they are no longer needed. Overwrite
+  byte slices with zeros, nil out pointers.
+- **Never log secrets.** Keys, passwords, tokens, and plaintext must never
+  appear in log output.
+
+### Web Security
+
+- CSRF tokens on all mutating requests.
+- `SameSite=Strict` on all cookies.
+- `html/template` for automatic escaping.
+- Validate all input at system boundaries.
+
+---
+
+## Development Workflow
+
+### Local Development
+
+```bash
+# Build and run both servers locally:
+make devserver
+
+# Or build everything and run the full pipeline:
+make all
+```
+
+The `devserver` target builds both binaries and runs them against a local
+config in `srv/`. The `srv/` directory is gitignored — it holds your local
+database, certificates, and configuration.
+
+### Pre-Push Checklist
+
+Before pushing a branch:
+
+```bash
+make all          # vet → lint → test → build
+make proto-lint   # if proto files changed
+```
+
+### Proto Changes
+
+1. Edit `.proto` files in `proto/<service>/v2/`.
+2. Run `make proto` to regenerate Go code.
+3. Run `make proto-lint` to check for linting violations and breaking changes.
+4. Update REST routes to match the new/changed RPCs.
+5. Update gRPC interceptor maps for any new RPCs.
+6. Update `ARCHITECTURE.md` API tables.
+
+### Adding a New Feature
+
+1. **Design first.** Write or update the relevant design document. For a new
+   engine or major subsystem, create a new doc in `docs/` or `engines/`.
+2. **Implement.** Follow existing patterns — the design doc should reference
+   specific files and line numbers.
+3. **Test.** Write tests alongside the implementation.
+4. **Update docs.** Update `ARCHITECTURE.md`, `CLAUDE.md`, and route tables.
+5. **Verify.** Run `make all`.
+
+### CLI Commands
+
+Every service uses cobra for CLI commands. Standard subcommands:
+
+| Command | Purpose |
+|---------|---------|
+| `server` | Start the service |
+| `init` | First-time setup (if applicable) |
+| `status` | Query a running instance's health |
+| `snapshot` | Create a database backup |
+
+Add service-specific subcommands as needed (e.g. `migrate-aad`, `unseal`).
+Each command lives in its own file in `cmd/<service>/`.