mcdsl/ARCHITECTURE.md

MCDSL Architecture

Metacircular Dynamics Standard Library — Technical Design Document

---

1. Overview

MCDSL is a Go module providing the shared infrastructure that every Metacircular service needs. It is not a framework — services are not structured around it. It is a collection of well-tested packages that each solve one problem, usable independently or together.

Design Principles

• Extract, don't invent. Every package in MCDSL is extracted from patterns already proven across multiple services. No speculative abstractions.
• Optional composition. Services import only the packages they need. No package depends on another MCDSL package unless absolutely necessary.
• Minimal API surface. Each package exposes the smallest possible public API. Configuration is via structs with sensible defaults.
• Zero magic. No init() functions, no global state, no reflection-based wiring. Explicit construction, explicit errors.
• Stdlib-compatible types. Functions accept and return *sql.DB, http.Handler, *slog.Logger, context.Context — not custom wrappers.

Module Path

git.wntrmute.dev/kyle/mcdsl

Dependencies

Dependency	Purpose
modernc.org/sqlite	Pure-Go SQLite driver
github.com/go-chi/chi/v5	HTTP router
github.com/pelletier/go-toml/v2	TOML config parsing
google.golang.org/grpc	gRPC server
github.com/klauspost/compress/zstd	Zstandard compression for archives
git.wntrmute.dev/kyle/mcias/clients/go	MCIAS client library

All dependencies are already used by existing services. MCDSL adds no new dependencies to the platform.

---

2. Package: auth

MCIAS token validation with caching. Extracted from the internal/auth/ packages in metacrypt, mcr, and mcat.

Types

// TokenInfo holds the validated identity of an authenticated caller.
type TokenInfo struct {
    Username string
    Roles    []string
    IsAdmin  bool
}

// Config holds MCIAS connection settings. Matches the standard [mcias]
// TOML section used by all services.
type Config struct {
    ServerURL   string   `toml:"server_url"`
    CACert      string   `toml:"ca_cert"`
    ServiceName string   `toml:"service_name"`
    Tags        []string `toml:"tags"`
}

// Authenticator validates MCIAS bearer tokens with a short-lived cache.
type Authenticator struct { /* unexported fields */ }

API

func New(cfg Config, logger *slog.Logger) (*Authenticator, error)
func (a *Authenticator) ValidateToken(token string) (*TokenInfo, error)
func (a *Authenticator) Login(username, password, totpCode string) (token string, expiresAt time.Time, err error)
func (a *Authenticator) Logout(token string) error

Cache Behavior

• Key: SHA-256 of the raw token, stored as [32]byte.
• TTL: 30 seconds (hardcoded, matches platform standard).
• Eviction: Lazy — expired entries are replaced on next lookup. No background goroutine.
• Thread safety: sync.RWMutex. Reads take a read lock; cache misses promote to write lock.

Admin Detection

IsAdmin is set by scanning the roles list for the string "admin". This matches MCIAS's role model — admin is a role, not a flag.

Errors

var (
    ErrInvalidToken      = errors.New("auth: invalid token")
    ErrInvalidCredentials = errors.New("auth: invalid credentials")
    ErrForbidden         = errors.New("auth: forbidden by policy")
)

---

3. Package: db

SQLite connection setup, migration runner, and snapshot utilities. Extracted from the internal/db/ packages across all services.

Opening a Database

func Open(path string) (*sql.DB, error)

Opens a SQLite database with the standard Metacircular pragmas:

PRAGMA journal_mode = WAL;
PRAGMA foreign_keys = ON;
PRAGMA busy_timeout = 5000;

File permissions are set to 0600 (owner read/write only). The function creates the file if it does not exist.

Returns a standard *sql.DB — no wrapper type. Services use it directly with database/sql.

Migrations

type Migration struct {
    Version int
    Name    string
    SQL     string
}

func Migrate(db *sql.DB, migrations []Migration) error

Migrations are applied sequentially in a transaction. Each migration is recorded in a schema_migrations table:

CREATE TABLE IF NOT EXISTS schema_migrations (
    version    INTEGER PRIMARY KEY,
    name       TEXT NOT NULL,
    applied_at TEXT NOT NULL
);

Already-applied migrations (by version number) are skipped. Timestamps are stored as RFC 3339 UTC.

Services define their migrations as a []Migration slice — no embedded SQL files, no migration DSL. The slice is the schema history.

Snapshots

func Snapshot(db *sql.DB, destPath string) error

Executes VACUUM INTO to create a consistent, standalone copy of the database at destPath. This is the standard backup mechanism for all Metacircular services.

---

4. Package: config

TOML configuration loading with environment variable overrides. Extracted from the internal/config/ packages across all services.

Standard Sections

// Base contains the configuration sections common to all services.
// Services embed this in their own config struct.
type Base struct {
    Server   ServerConfig   `toml:"server"`
    Database DatabaseConfig `toml:"database"`
    MCIAS    auth.Config    `toml:"mcias"`
    Log      LogConfig      `toml:"log"`
}

type ServerConfig struct {
    ListenAddr      string        `toml:"listen_addr"`
    GRPCAddr        string        `toml:"grpc_addr"`
    TLSCert         string        `toml:"tls_cert"`
    TLSKey          string        `toml:"tls_key"`
    ReadTimeout     time.Duration `toml:"read_timeout"`
    WriteTimeout    time.Duration `toml:"write_timeout"`
    IdleTimeout     time.Duration `toml:"idle_timeout"`
    ShutdownTimeout time.Duration `toml:"shutdown_timeout"`
}

type DatabaseConfig struct {
    Path string `toml:"path"`
}

type LogConfig struct {
    Level string `toml:"level"`
}

Loading

func Load[T any](path string, envPrefix string) (*T, error)

1. Reads the TOML file at path.
2. Unmarshals into *T.
3. Applies environment variable overrides using envPrefix (e.g., prefix "MCR" maps MCR_SERVER_LISTEN_ADDR to Server.ListenAddr).
4. Applies defaults for unset optional fields.
5. Validates required fields.

Environment overrides use reflection to walk the struct, converting field paths to PREFIX_SECTION_FIELD format. Only string, int, bool, duration, and string slice fields are supported.

Defaults

Field	Default
Log.Level	"info"
Server.ReadTimeout	30s
Server.WriteTimeout	30s
Server.IdleTimeout	120s
Server.ShutdownTimeout	60s

Validation

Load returns an error if any required field is empty:

• Server.ListenAddr
• Server.TLSCert
• Server.TLSKey
• Database.Path
• MCIAS.ServerURL (if the MCIAS section is present)

Services add their own validation by implementing an optional Validate() method on their config type.

---

5. Package: httpserver

TLS HTTP server setup with chi, standard middleware, and graceful shutdown. Extracted from internal/server/ across all services.

Types

type Server struct {
    Router *chi.Mux
    Logger *slog.Logger
    // unexported: httpSrv, cfg
}

API

func New(cfg config.ServerConfig, logger *slog.Logger) *Server
func (s *Server) ListenAndServeTLS(certFile, keyFile string) error
func (s *Server) Shutdown(ctx context.Context) error

New creates a chi router and configures the underlying http.Server with:

• TLS 1.3 minimum (tls.VersionTLS13)
• Read/write/idle timeouts from config
• The chi router as handler

Services access s.Router to register their routes.

Standard Middleware

func (s *Server) LoggingMiddleware(next http.Handler) http.Handler

Wraps the response writer to capture the status code, logs after the request completes:

level=INFO msg=http method=GET path=/v1/status status=200 remote=10.0.0.1:54321

StatusWriter

type StatusWriter struct {
    http.ResponseWriter
    Status int
}

Exported for use in custom middleware that needs the response status code.

JSON Helpers

func WriteJSON(w http.ResponseWriter, status int, v any)
func WriteError(w http.ResponseWriter, status int, message string)

WriteError writes {"error": "message"} — the standard Metacircular error format.

---

6. Package: grpcserver

gRPC server setup with TLS, interceptor chain, and method-map authentication. Extracted from internal/grpcserver/ in mcias, mcr, and mc-proxy.

Types

// MethodMap classifies gRPC methods for access control.
type MethodMap struct {
    Public       map[string]bool // No auth required
    AuthRequired map[string]bool // Valid MCIAS token required
    AdminRequired map[string]bool // Admin role required
}

type Server struct {
    GRPCServer *grpc.Server
    Logger     *slog.Logger
    // unexported: listener, auth
}

API

func New(cfg config.ServerConfig, auth *auth.Authenticator, methods MethodMap, logger *slog.Logger) (*Server, error)
func (s *Server) Serve() error
func (s *Server) Stop()

New builds a gRPC server with:

• TLS 1.3 from cert/key in config
• Unary interceptor chain: logging → auth (using MethodMap) → user handler
• Services register their implementations on s.GRPCServer

Auth Interceptor

The auth interceptor uses the MethodMap to determine the required access level for each RPC:

1. If the method is in Public — pass through, no auth.
2. If the method is in AuthRequired — validate the bearer token from metadata, populate TokenInfo in context.
3. If the method is in AdminRequired — validate token and require IsAdmin == true.
4. If the method is not in any map — deny by default. This is a safety net: forgetting to register a new RPC results in a denied request, not an open one.

Context Helpers

func TokenInfoFromContext(ctx context.Context) *auth.TokenInfo

---

7. Package: csrf

HMAC-SHA256 double-submit cookie CSRF protection. Extracted from the web server packages in metacrypt, mcr, and mcat.

Types

type Protect struct {
    // unexported: secret, cookieName, fieldName
}

API

func New(secret []byte, cookieName, fieldName string) *Protect
func (p *Protect) Middleware(next http.Handler) http.Handler
func (p *Protect) SetToken(w http.ResponseWriter) string
func (p *Protect) TemplateFunc(w http.ResponseWriter) template.FuncMap

Token Format

base64(nonce) "." base64(HMAC-SHA256(secret, nonce))

Nonce is 32 bytes from crypto/rand. The token is set as a cookie and must be submitted as a form field on mutating requests (POST, PUT, PATCH, DELETE).

Validation

The middleware:

1. Skips safe methods (GET, HEAD, OPTIONS).
2. Reads the token from the cookie and the form field.
3. Verifies both are present and equal.
4. Verifies the HMAC signature is valid.
5. Returns 403 on any failure.

Template Integration

TemplateFunc returns a template.FuncMap with a csrfField function that renders the hidden input:

<input type="hidden" name="csrf_token" value="...">

---

8. Package: web

Session cookie management, auth middleware, and template rendering helpers for htmx web UIs. Extracted from internal/webserver/ across services.

Session Cookies

func SetSessionCookie(w http.ResponseWriter, name, token string)
func ClearSessionCookie(w http.ResponseWriter, name string)
func GetSessionToken(r *http.Request, name string) string

All session cookies are set with: HttpOnly, Secure, SameSite=Strict, Path=/.

Auth Middleware

func RequireAuth(auth *auth.Authenticator, cookieName string, loginPath string) func(http.Handler) http.Handler

Extracts the session token from the cookie, validates it via the Authenticator, and either:

• Sets TokenInfo in the request context and calls the next handler.
• Redirects to loginPath if the token is missing or invalid.

Context Helpers

func TokenInfoFromContext(ctx context.Context) *auth.TokenInfo

Template Helpers

func RenderTemplate(w http.ResponseWriter, fs embed.FS, name string, data any, funcs ...template.FuncMap)

Parses templates/layout.html and templates/<name> from the embedded FS, merges any provided FuncMaps, and executes the layout template.

---

9. Package: archive

Service directory snapshot and restore using tar.zst, with SQLite-aware handling. This is new functionality for MCP, not extracted from existing services.

Snapshot

type SnapshotOptions struct {
    ServiceDir string   // e.g., /srv/myservice
    DBPath     string   // e.g., /srv/myservice/myservice.db (for VACUUM INTO)
    DB         *sql.DB  // live database connection (for VACUUM INTO)
    ExcludePatterns []string // additional glob patterns to exclude
}

func Snapshot(opts SnapshotOptions) (io.ReadCloser, error)

1. Runs VACUUM INTO to create a consistent DB copy in a temp file.
2. Walks the service directory, excluding:
   • *.db, *.db-wal, *.db-shm (live database files)
   • backups/ directory
   • Any patterns in ExcludePatterns
3. Adds the VACUUM INTO copy as <basename>.db in the archive.
4. Returns a streaming tar.zst reader.

The archive is produced as a stream — it does not need to be fully buffered in memory. This allows piping directly over a network connection.

Restore

func Restore(r io.Reader, destDir string) error

Extracts a tar.zst archive into destDir. Creates the directory if it does not exist. Overwrites existing files. Preserves file permissions.

Compression

Zstandard compression via github.com/klauspost/compress/zstd. Default compression level (3) balances speed and ratio for the typical service directory size (SQLite DB + config + certs, usually under 100 MB).

---

10. Package: health

Standard health check implementation for both gRPC and REST. New functionality to standardize what services already do ad-hoc.

gRPC

Implements grpc.health.v1.Health (the standard gRPC health checking protocol). Services register it on their gRPC server:

health.RegisterGRPC(grpcServer)

REST

func Handler(db *sql.DB) http.HandlerFunc

Returns a handler for GET /healthz (or whatever path the service mounts it on) that:

1. Pings the database.
2. Returns 200 {"status": "ok"} or 503 {"status": "unhealthy", "error": "..."}.

---

11. Inter-Package Dependencies

archive ──→ db (for Snapshot)
auth ──→ (mcias client library)
config ──→ auth (for auth.Config type)
csrf ──→ (stdlib only)
db ──→ (modernc.org/sqlite)
grpcserver ──→ auth, config
health ──→ db
httpserver ──→ config
web ──→ auth, csrf

No circular dependencies. Each package can be imported independently except where noted above.

---

12. What MCDSL Does Not Provide

• Business logic. Policy engines, engine registries, OCI handlers — these are service-specific and stay in each service's internal/ packages.
• Proto definitions. Each service owns its own proto files and generated code.
• CLI scaffolding. Cobra command wiring is minimal and service-specific.
• Database schemas. Each service defines its own migrations. MCDSL provides the runner, not the SQL.
• Templates and static assets. Each service's web UI is its own. MCDSL provides rendering helpers, not content.

---

13. Migration Path

Existing services adopt MCDSL incrementally — one package at a time:

1. Replace internal/auth/ with mcdsl/auth.
2. Replace database open/pragma code with mcdsl/db.Open.
3. Replace migration runner with mcdsl/db.Migrate.
4. Replace config loading with mcdsl/config.Load.
5. Replace CSRF implementation with mcdsl/csrf.
6. Replace server setup with mcdsl/httpserver and mcdsl/grpcserver.

Each step is independent. Services can adopt one package without adopting all of them. The old internal/ code can be removed after each migration.

---

14. Security Considerations

• Token caching uses SHA-256 of the token as the cache key. The raw token is never used as a map key to prevent timing attacks on map lookup.
• CSRF secrets must be generated from crypto/rand and should be unique per service instance. MCDSL does not generate them — the service provides them.
• Session cookies are always HttpOnly, Secure, SameSite=Strict. These flags are not configurable — relaxing them would be a security defect.
• gRPC method maps default to deny. An unregistered method is rejected, not allowed. This is the most important safety property in the library.
• File permissions on databases are 0600. This is not configurable.
• TLS 1.3 minimum is not configurable. Services that need TLS 1.2 (there should be none) cannot use httpserver or grpcserver.