# MCDSL Architecture Metacircular Dynamics Standard Library — Technical Design Document --- ## 1. Overview MCDSL is a Go module providing the shared infrastructure that every Metacircular service needs. It is not a framework — services are not structured around it. It is a collection of well-tested packages that each solve one problem, usable independently or together. ### Design Principles - **Extract, don't invent.** Every package in MCDSL is extracted from patterns already proven across multiple services. No speculative abstractions. - **Optional composition.** Services import only the packages they need. No package depends on another MCDSL package unless absolutely necessary. - **Minimal API surface.** Each package exposes the smallest possible public API. Configuration is via structs with sensible defaults. - **Zero magic.** No init() functions, no global state, no reflection-based wiring. Explicit construction, explicit errors. - **Stdlib-compatible types.** Functions accept and return `*sql.DB`, `http.Handler`, `*slog.Logger`, `context.Context` — not custom wrappers. ### Module Path ``` git.wntrmute.dev/mc/mcdsl ``` ### Dependencies | Dependency | Purpose | |------------|---------| | `modernc.org/sqlite` | Pure-Go SQLite driver | | `github.com/go-chi/chi/v5` | HTTP router | | `github.com/pelletier/go-toml/v2` | TOML config parsing | | `google.golang.org/grpc` | gRPC server | | `github.com/klauspost/compress/zstd` | Zstandard compression for archives | | `git.wntrmute.dev/mc/mcias/clients/go` | MCIAS client library | All dependencies are already used by existing services. MCDSL adds no new dependencies to the platform. --- ## 2. Package: `auth` MCIAS token validation with caching. Extracted from the `internal/auth/` packages in metacrypt, mcr, and mcat. ### Types ```go // TokenInfo holds the validated identity of an authenticated caller. type TokenInfo struct { Username string Roles []string IsAdmin bool } // Config holds MCIAS connection settings. Matches the standard [mcias] // TOML section used by all services. type Config struct { ServerURL string `toml:"server_url"` CACert string `toml:"ca_cert"` ServiceName string `toml:"service_name"` Tags []string `toml:"tags"` } // Authenticator validates MCIAS bearer tokens with a short-lived cache. type Authenticator struct { /* unexported fields */ } ``` ### API ```go func New(cfg Config, logger *slog.Logger) (*Authenticator, error) func (a *Authenticator) ValidateToken(token string) (*TokenInfo, error) func (a *Authenticator) Login(username, password, totpCode string) (token string, expiresAt time.Time, err error) func (a *Authenticator) Logout(token string) error ``` ### Cache Behavior - Key: SHA-256 of the raw token, stored as `[32]byte`. - TTL: 30 seconds (hardcoded, matches platform standard). - Eviction: Lazy — expired entries are replaced on next lookup. No background goroutine. - Thread safety: `sync.RWMutex`. Reads take a read lock; cache misses promote to write lock. ### Admin Detection `IsAdmin` is set by scanning the roles list for the string `"admin"`. This matches MCIAS's role model — admin is a role, not a flag. ### Errors ```go var ( ErrInvalidToken = errors.New("auth: invalid token") ErrInvalidCredentials = errors.New("auth: invalid credentials") ErrForbidden = errors.New("auth: forbidden by policy") ) ``` --- ## 3. Package: `db` SQLite connection setup, migration runner, and snapshot utilities. Extracted from the `internal/db/` packages across all services. ### Opening a Database ```go func Open(path string) (*sql.DB, error) ``` Opens a SQLite database with the standard Metacircular pragmas: ```sql PRAGMA journal_mode = WAL; PRAGMA foreign_keys = ON; PRAGMA busy_timeout = 5000; ``` File permissions are set to `0600` (owner read/write only). The function creates the file if it does not exist. Returns a standard `*sql.DB` — no wrapper type. Services use it directly with `database/sql`. ### Migrations ```go type Migration struct { Version int Name string SQL string } func Migrate(db *sql.DB, migrations []Migration) error ``` Migrations are applied sequentially in a transaction. Each migration is recorded in a `schema_migrations` table: ```sql CREATE TABLE IF NOT EXISTS schema_migrations ( version INTEGER PRIMARY KEY, name TEXT NOT NULL, applied_at TEXT NOT NULL ); ``` Already-applied migrations (by version number) are skipped. Timestamps are stored as RFC 3339 UTC. Services define their migrations as a `[]Migration` slice — no embedded SQL files, no migration DSL. The slice is the schema history. ### Snapshots ```go func Snapshot(db *sql.DB, destPath string) error ``` Executes `VACUUM INTO` to create a consistent, standalone copy of the database at `destPath`. This is the standard backup mechanism for all Metacircular services. --- ## 4. Package: `config` TOML configuration loading with environment variable overrides. Extracted from the `internal/config/` packages across all services. ### Standard Sections ```go // Base contains the configuration sections common to all services. // Services embed this in their own config struct. type Base struct { Server ServerConfig `toml:"server"` Database DatabaseConfig `toml:"database"` MCIAS auth.Config `toml:"mcias"` Log LogConfig `toml:"log"` } type ServerConfig struct { ListenAddr string `toml:"listen_addr"` GRPCAddr string `toml:"grpc_addr"` TLSCert string `toml:"tls_cert"` TLSKey string `toml:"tls_key"` ReadTimeout time.Duration `toml:"read_timeout"` WriteTimeout time.Duration `toml:"write_timeout"` IdleTimeout time.Duration `toml:"idle_timeout"` ShutdownTimeout time.Duration `toml:"shutdown_timeout"` } type DatabaseConfig struct { Path string `toml:"path"` } type LogConfig struct { Level string `toml:"level"` } ``` ### Loading ```go func Load[T any](path string, envPrefix string) (*T, error) ``` 1. Reads the TOML file at `path`. 2. Unmarshals into `*T`. 3. Applies environment variable overrides using `envPrefix` (e.g., prefix `"MCR"` maps `MCR_SERVER_LISTEN_ADDR` to `Server.ListenAddr`). 4. Applies defaults for unset optional fields. 5. Validates required fields. Environment overrides use reflection to walk the struct, converting field paths to `PREFIX_SECTION_FIELD` format. Only string, int, bool, duration, and string slice fields are supported. ### Defaults | Field | Default | |-------|---------| | `Log.Level` | `"info"` | | `Server.ReadTimeout` | `30s` | | `Server.WriteTimeout` | `30s` | | `Server.IdleTimeout` | `120s` | | `Server.ShutdownTimeout` | `60s` | ### Validation `Load` returns an error if any required field is empty: - `Server.ListenAddr` - `Server.TLSCert` - `Server.TLSKey` - `Database.Path` - `MCIAS.ServerURL` (if the MCIAS section is present) Services add their own validation by implementing an optional `Validate()` method on their config type. --- ## 5. Package: `httpserver` TLS HTTP server setup with chi, standard middleware, and graceful shutdown. Extracted from `internal/server/` across all services. ### Types ```go type Server struct { Router *chi.Mux Logger *slog.Logger // unexported: httpSrv, cfg } ``` ### API ```go func New(cfg config.ServerConfig, logger *slog.Logger) *Server func (s *Server) ListenAndServeTLS(certFile, keyFile string) error func (s *Server) Shutdown(ctx context.Context) error ``` `New` creates a chi router and configures the underlying `http.Server` with: - TLS 1.3 minimum (`tls.VersionTLS13`) - Read/write/idle timeouts from config - The chi router as handler Services access `s.Router` to register their routes. ### Standard Middleware ```go func (s *Server) LoggingMiddleware(next http.Handler) http.Handler ``` Wraps the response writer to capture the status code, logs after the request completes: ``` level=INFO msg=http method=GET path=/v1/status status=200 remote=10.0.0.1:54321 ``` ### StatusWriter ```go type StatusWriter struct { http.ResponseWriter Status int } ``` Exported for use in custom middleware that needs the response status code. ### JSON Helpers ```go func WriteJSON(w http.ResponseWriter, status int, v any) func WriteError(w http.ResponseWriter, status int, message string) ``` `WriteError` writes `{"error": "message"}` — the standard Metacircular error format. --- ## 6. Package: `grpcserver` gRPC server setup with TLS, interceptor chain, and method-map authentication. Extracted from `internal/grpcserver/` in mcias, mcr, and mc-proxy. ### Types ```go // MethodMap classifies gRPC methods for access control. type MethodMap struct { Public map[string]bool // No auth required AuthRequired map[string]bool // Valid MCIAS token required AdminRequired map[string]bool // Admin role required } type Server struct { GRPCServer *grpc.Server Logger *slog.Logger // unexported: listener, auth } ``` ### API ```go func New(cfg config.ServerConfig, auth *auth.Authenticator, methods MethodMap, logger *slog.Logger) (*Server, error) func (s *Server) Serve() error func (s *Server) Stop() ``` `New` builds a gRPC server with: - TLS 1.3 from cert/key in config - Unary interceptor chain: logging → auth (using MethodMap) → user handler - Services register their implementations on `s.GRPCServer` ### Auth Interceptor The auth interceptor uses the `MethodMap` to determine the required access level for each RPC: 1. If the method is in `Public` — pass through, no auth. 2. If the method is in `AuthRequired` — validate the bearer token from metadata, populate `TokenInfo` in context. 3. If the method is in `AdminRequired` — validate token and require `IsAdmin == true`. 4. If the method is not in any map — **deny by default**. This is a safety net: forgetting to register a new RPC results in a denied request, not an open one. ### Context Helpers ```go func TokenInfoFromContext(ctx context.Context) *auth.TokenInfo ``` --- ## 7. Package: `csrf` HMAC-SHA256 double-submit cookie CSRF protection. Extracted from the web server packages in metacrypt, mcr, and mcat. ### Types ```go type Protect struct { // unexported: secret, cookieName, fieldName } ``` ### API ```go func New(secret []byte, cookieName, fieldName string) *Protect func (p *Protect) Middleware(next http.Handler) http.Handler func (p *Protect) SetToken(w http.ResponseWriter) string func (p *Protect) TemplateFunc(w http.ResponseWriter) template.FuncMap ``` ### Token Format ``` base64(nonce) "." base64(HMAC-SHA256(secret, nonce)) ``` Nonce is 32 bytes from `crypto/rand`. The token is set as a cookie and must be submitted as a form field on mutating requests (POST, PUT, PATCH, DELETE). ### Validation The middleware: 1. Skips safe methods (GET, HEAD, OPTIONS). 2. Reads the token from the cookie and the form field. 3. Verifies both are present and equal. 4. Verifies the HMAC signature is valid. 5. Returns 403 on any failure. ### Template Integration `TemplateFunc` returns a `template.FuncMap` with a `csrfField` function that renders the hidden input: ```html ``` --- ## 8. Package: `web` Session cookie management, auth middleware, and template rendering helpers for htmx web UIs. Extracted from `internal/webserver/` across services. ### Session Cookies ```go func SetSessionCookie(w http.ResponseWriter, name, token string) func ClearSessionCookie(w http.ResponseWriter, name string) func GetSessionToken(r *http.Request, name string) string ``` All session cookies are set with: `HttpOnly`, `Secure`, `SameSite=Strict`, `Path=/`. ### Auth Middleware ```go func RequireAuth(auth *auth.Authenticator, cookieName string, loginPath string) func(http.Handler) http.Handler ``` Extracts the session token from the cookie, validates it via the Authenticator, and either: - Sets `TokenInfo` in the request context and calls the next handler. - Redirects to `loginPath` if the token is missing or invalid. ### Context Helpers ```go func TokenInfoFromContext(ctx context.Context) *auth.TokenInfo ``` ### Template Helpers ```go func RenderTemplate(w http.ResponseWriter, fs embed.FS, name string, data any, funcs ...template.FuncMap) ``` Parses `templates/layout.html` and `templates/` from the embedded FS, merges any provided FuncMaps, and executes the `layout` template. --- ## 9. Package: `archive` Service directory snapshot and restore using tar.zst, with SQLite-aware handling. This is new functionality for MCP, not extracted from existing services. ### Snapshot ```go type SnapshotOptions struct { ServiceDir string // e.g., /srv/myservice DBPath string // e.g., /srv/myservice/myservice.db (for VACUUM INTO) DB *sql.DB // live database connection (for VACUUM INTO) ExcludePatterns []string // additional glob patterns to exclude } func Snapshot(opts SnapshotOptions) (io.ReadCloser, error) ``` 1. Runs `VACUUM INTO` to create a consistent DB copy in a temp file. 2. Walks the service directory, excluding: - `*.db`, `*.db-wal`, `*.db-shm` (live database files) - `backups/` directory - Any patterns in `ExcludePatterns` 3. Adds the VACUUM INTO copy as `.db` in the archive. 4. Returns a streaming tar.zst reader. The archive is produced as a stream — it does not need to be fully buffered in memory. This allows piping directly over a network connection. ### Restore ```go func Restore(r io.Reader, destDir string) error ``` Extracts a tar.zst archive into `destDir`. Creates the directory if it does not exist. Overwrites existing files. Preserves file permissions. ### Compression Zstandard compression via `github.com/klauspost/compress/zstd`. Default compression level (3) balances speed and ratio for the typical service directory size (SQLite DB + config + certs, usually under 100 MB). --- ## 10. Package: `health` Standard health check implementation for both gRPC and REST. New functionality to standardize what services already do ad-hoc. ### gRPC Implements `grpc.health.v1.Health` (the standard gRPC health checking protocol). Services register it on their gRPC server: ```go health.RegisterGRPC(grpcServer) ``` ### REST ```go func Handler(db *sql.DB) http.HandlerFunc ``` Returns a handler for `GET /healthz` (or whatever path the service mounts it on) that: 1. Pings the database. 2. Returns `200 {"status": "ok"}` or `503 {"status": "unhealthy", "error": "..."}`. --- ## 11. Inter-Package Dependencies ``` archive ──→ db (for Snapshot) auth ──→ (mcias client library) config ──→ auth (for auth.Config type) csrf ──→ (stdlib only) db ──→ (modernc.org/sqlite) grpcserver ──→ auth, config health ──→ db httpserver ──→ config web ──→ auth, csrf ``` No circular dependencies. Each package can be imported independently except where noted above. --- ## 12. What MCDSL Does Not Provide - **Business logic.** Policy engines, engine registries, OCI handlers — these are service-specific and stay in each service's `internal/` packages. - **Proto definitions.** Each service owns its own proto files and generated code. - **CLI scaffolding.** Cobra command wiring is minimal and service-specific. - **Database schemas.** Each service defines its own migrations. MCDSL provides the runner, not the SQL. - **Templates and static assets.** Each service's web UI is its own. MCDSL provides rendering helpers, not content. --- ## 13. Migration Path Existing services adopt MCDSL incrementally — one package at a time: 1. Replace `internal/auth/` with `mcdsl/auth`. 2. Replace database open/pragma code with `mcdsl/db.Open`. 3. Replace migration runner with `mcdsl/db.Migrate`. 4. Replace config loading with `mcdsl/config.Load`. 5. Replace CSRF implementation with `mcdsl/csrf`. 6. Replace server setup with `mcdsl/httpserver` and `mcdsl/grpcserver`. Each step is independent. Services can adopt one package without adopting all of them. The old `internal/` code can be removed after each migration. --- ## 14. Security Considerations - **Token caching** uses SHA-256 of the token as the cache key. The raw token is never used as a map key to prevent timing attacks on map lookup. - **CSRF secrets** must be generated from `crypto/rand` and should be unique per service instance. MCDSL does not generate them — the service provides them. - **Session cookies** are always `HttpOnly`, `Secure`, `SameSite=Strict`. These flags are not configurable — relaxing them would be a security defect. - **gRPC method maps** default to deny. An unregistered method is rejected, not allowed. This is the most important safety property in the library. - **File permissions** on databases are `0600`. This is not configurable. - **TLS 1.3 minimum** is not configurable. Services that need TLS 1.2 (there should be none) cannot use `httpserver` or `grpcserver`.