mc/mcdsl
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial commit: project setup and db package

Browse Source 
- Project scaffolding: go.mod, Makefile, .golangci.yaml, doc.go
- README, ARCHITECTURE, PROJECT_PLAN, PROGRESS documentation
- db package: Open (WAL, FK, busy timeout, 0600 permissions),
  Migrate (sequential, transactional, idempotent),
  SchemaVersion, Snapshot (VACUUM INTO)
- 11 tests covering open, migrate, and snapshot

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Kyle Isom
2026-03-25 14:17:17 -07:00
commit 8b4db22c93
12 changed files with 1611 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

611
ARCHITECTURE.md Normal file
View File

@@ -0,0 +1,611 @@
# MCDSL Architecture
Metacircular Dynamics Standard Library — Technical Design Document
---
## 1. Overview
MCDSL is a Go module providing the shared infrastructure that every
Metacircular service needs. It is not a framework — services are not structured
around it. It is a collection of well-tested packages that each solve one
problem, usable independently or together.
### Design Principles
- **Extract, don't invent.** Every package in MCDSL is extracted from patterns
already proven across multiple services. No speculative abstractions.
- **Optional composition.** Services import only the packages they need. No
package depends on another MCDSL package unless absolutely necessary.
- **Minimal API surface.** Each package exposes the smallest possible public
API. Configuration is via structs with sensible defaults.
- **Zero magic.** No init() functions, no global state, no reflection-based
wiring. Explicit construction, explicit errors.
- **Stdlib-compatible types.** Functions accept and return `*sql.DB`,
`http.Handler`, `*slog.Logger`, `context.Context` — not custom wrappers.
### Module Path
```
git.wntrmute.dev/kyle/mcdsl
```
### Dependencies
| Dependency | Purpose |
|------------|---------|
| `modernc.org/sqlite` | Pure-Go SQLite driver |
| `github.com/go-chi/chi/v5` | HTTP router |
| `github.com/pelletier/go-toml/v2` | TOML config parsing |
| `google.golang.org/grpc` | gRPC server |
| `github.com/klauspost/compress/zstd` | Zstandard compression for archives |
| `git.wntrmute.dev/kyle/mcias/clients/go` | MCIAS client library |
All dependencies are already used by existing services. MCDSL adds no new
dependencies to the platform.
---
## 2. Package: `auth`
MCIAS token validation with caching. Extracted from the `internal/auth/`
packages in metacrypt, mcr, and mcat.
### Types
```go
// TokenInfo holds the validated identity of an authenticated caller.
type TokenInfo struct {
Username string
Roles []string
IsAdmin bool
}
// Config holds MCIAS connection settings. Matches the standard [mcias]
// TOML section used by all services.
type Config struct {
ServerURL string `toml:"server_url"`
CACert string `toml:"ca_cert"`
ServiceName string `toml:"service_name"`
Tags []string `toml:"tags"`
}
// Authenticator validates MCIAS bearer tokens with a short-lived cache.
type Authenticator struct { /* unexported fields */ }
```
### API
```go
func New(cfg Config, logger *slog.Logger) (*Authenticator, error)
func (a *Authenticator) ValidateToken(token string) (*TokenInfo, error)
func (a *Authenticator) Login(username, password, totpCode string) (token string, expiresAt time.Time, err error)
func (a *Authenticator) Logout(token string) error
```
### Cache Behavior
- Key: SHA-256 of the raw token, stored as `[32]byte`.
- TTL: 30 seconds (hardcoded, matches platform standard).
- Eviction: Lazy — expired entries are replaced on next lookup. No background
goroutine.
- Thread safety: `sync.RWMutex`. Reads take a read lock; cache misses
promote to write lock.
### Admin Detection
`IsAdmin` is set by scanning the roles list for the string `"admin"`. This
matches MCIAS's role model — admin is a role, not a flag.
### Errors
```go
var (
ErrInvalidToken = errors.New("auth: invalid token")
ErrInvalidCredentials = errors.New("auth: invalid credentials")
ErrForbidden = errors.New("auth: forbidden by policy")
)
```
---
## 3. Package: `db`
SQLite connection setup, migration runner, and snapshot utilities. Extracted
from the `internal/db/` packages across all services.
### Opening a Database
```go
func Open(path string) (*sql.DB, error)
```
Opens a SQLite database with the standard Metacircular pragmas:
```sql
PRAGMA journal_mode = WAL;
PRAGMA foreign_keys = ON;
PRAGMA busy_timeout = 5000;
```
File permissions are set to `0600` (owner read/write only). The function
creates the file if it does not exist.
Returns a standard `*sql.DB` — no wrapper type. Services use it directly
with `database/sql`.
### Migrations
```go
type Migration struct {
Version int
Name string
SQL string
}
func Migrate(db *sql.DB, migrations []Migration) error
```
Migrations are applied sequentially in a transaction. Each migration is
recorded in a `schema_migrations` table:
```sql
CREATE TABLE IF NOT EXISTS schema_migrations (
version INTEGER PRIMARY KEY,
name TEXT NOT NULL,
applied_at TEXT NOT NULL
);
```
Already-applied migrations (by version number) are skipped. Timestamps are
stored as RFC 3339 UTC.
Services define their migrations as a `[]Migration` slice — no embedded SQL
files, no migration DSL. The slice is the schema history.
### Snapshots
```go
func Snapshot(db *sql.DB, destPath string) error
```
Executes `VACUUM INTO` to create a consistent, standalone copy of the
database at `destPath`. This is the standard backup mechanism for all
Metacircular services.
---
## 4. Package: `config`
TOML configuration loading with environment variable overrides. Extracted from
the `internal/config/` packages across all services.
### Standard Sections
```go
// Base contains the configuration sections common to all services.
// Services embed this in their own config struct.
type Base struct {
Server ServerConfig `toml:"server"`
Database DatabaseConfig `toml:"database"`
MCIAS auth.Config `toml:"mcias"`
Log LogConfig `toml:"log"`
}
type ServerConfig struct {
ListenAddr string `toml:"listen_addr"`
GRPCAddr string `toml:"grpc_addr"`
TLSCert string `toml:"tls_cert"`
TLSKey string `toml:"tls_key"`
ReadTimeout time.Duration `toml:"read_timeout"`
WriteTimeout time.Duration `toml:"write_timeout"`
IdleTimeout time.Duration `toml:"idle_timeout"`
ShutdownTimeout time.Duration `toml:"shutdown_timeout"`
}
type DatabaseConfig struct {
Path string `toml:"path"`
}
type LogConfig struct {
Level string `toml:"level"`
}
```
### Loading
```go
func Load[T any](path string, envPrefix string) (*T, error)
```
1. Reads the TOML file at `path`.
2. Unmarshals into `*T`.
3. Applies environment variable overrides using `envPrefix` (e.g., prefix
`"MCR"` maps `MCR_SERVER_LISTEN_ADDR` to `Server.ListenAddr`).
4. Applies defaults for unset optional fields.
5. Validates required fields.
Environment overrides use reflection to walk the struct, converting field
paths to `PREFIX_SECTION_FIELD` format. Only string, int, bool, duration,
and string slice fields are supported.
### Defaults
| Field | Default |
|-------|---------|
| `Log.Level` | `"info"` |
| `Server.ReadTimeout` | `30s` |
| `Server.WriteTimeout` | `30s` |
| `Server.IdleTimeout` | `120s` |
| `Server.ShutdownTimeout` | `60s` |
### Validation
`Load` returns an error if any required field is empty:
- `Server.ListenAddr`
- `Server.TLSCert`
- `Server.TLSKey`
- `Database.Path`
- `MCIAS.ServerURL` (if the MCIAS section is present)
Services add their own validation by implementing an optional `Validate()`
method on their config type.
---
## 5. Package: `httpserver`
TLS HTTP server setup with chi, standard middleware, and graceful shutdown.
Extracted from `internal/server/` across all services.
### Types
```go
type Server struct {
Router *chi.Mux
Logger *slog.Logger
// unexported: httpSrv, cfg
}
```
### API
```go
func New(cfg config.ServerConfig, logger *slog.Logger) *Server
func (s *Server) ListenAndServeTLS(certFile, keyFile string) error
func (s *Server) Shutdown(ctx context.Context) error
```
`New` creates a chi router and configures the underlying `http.Server` with:
- TLS 1.3 minimum (`tls.VersionTLS13`)
- Read/write/idle timeouts from config
- The chi router as handler
Services access `s.Router` to register their routes.
### Standard Middleware
```go
func (s *Server) LoggingMiddleware(next http.Handler) http.Handler
```
Wraps the response writer to capture the status code, logs after the
request completes:
```
level=INFO msg=http method=GET path=/v1/status status=200 remote=10.0.0.1:54321
```
### StatusWriter
```go
type StatusWriter struct {
http.ResponseWriter
Status int
}
```
Exported for use in custom middleware that needs the response status code.
### JSON Helpers
```go
func WriteJSON(w http.ResponseWriter, status int, v any)
func WriteError(w http.ResponseWriter, status int, message string)
```
`WriteError` writes `{"error": "message"}` — the standard Metacircular
error format.
---
## 6. Package: `grpcserver`
gRPC server setup with TLS, interceptor chain, and method-map
authentication. Extracted from `internal/grpcserver/` in mcias, mcr, and
mc-proxy.
### Types
```go
// MethodMap classifies gRPC methods for access control.
type MethodMap struct {
Public map[string]bool // No auth required
AuthRequired map[string]bool // Valid MCIAS token required
AdminRequired map[string]bool // Admin role required
}
type Server struct {
GRPCServer *grpc.Server
Logger *slog.Logger
// unexported: listener, auth
}
```
### API
```go
func New(cfg config.ServerConfig, auth *auth.Authenticator, methods MethodMap, logger *slog.Logger) (*Server, error)
func (s *Server) Serve() error
func (s *Server) Stop()
```
`New` builds a gRPC server with:
- TLS 1.3 from cert/key in config
- Unary interceptor chain: logging → auth (using MethodMap) → user handler
- Services register their implementations on `s.GRPCServer`
### Auth Interceptor
The auth interceptor uses the `MethodMap` to determine the required access
level for each RPC:
1. If the method is in `Public` — pass through, no auth.
2. If the method is in `AuthRequired` — validate the bearer token from
metadata, populate `TokenInfo` in context.
3. If the method is in `AdminRequired` — validate token and require
`IsAdmin == true`.
4. If the method is not in any map — **deny by default**. This is a safety
net: forgetting to register a new RPC results in a denied request, not
an open one.
### Context Helpers
```go
func TokenInfoFromContext(ctx context.Context) *auth.TokenInfo
```
---
## 7. Package: `csrf`
HMAC-SHA256 double-submit cookie CSRF protection. Extracted from the web
server packages in metacrypt, mcr, and mcat.
### Types
```go
type Protect struct {
// unexported: secret, cookieName, fieldName
}
```
### API
```go
func New(secret []byte, cookieName, fieldName string) *Protect
func (p *Protect) Middleware(next http.Handler) http.Handler
func (p *Protect) SetToken(w http.ResponseWriter) string
func (p *Protect) TemplateFunc(w http.ResponseWriter) template.FuncMap
```
### Token Format
```
base64(nonce) "." base64(HMAC-SHA256(secret, nonce))
```
Nonce is 32 bytes from `crypto/rand`. The token is set as a cookie and must
be submitted as a form field on mutating requests (POST, PUT, PATCH, DELETE).
### Validation
The middleware:
1. Skips safe methods (GET, HEAD, OPTIONS).
2. Reads the token from the cookie and the form field.
3. Verifies both are present and equal.
4. Verifies the HMAC signature is valid.
5. Returns 403 on any failure.
### Template Integration
`TemplateFunc` returns a `template.FuncMap` with a `csrfField` function that
renders the hidden input:
```html
<input type="hidden" name="csrf_token" value="...">
```
---
## 8. Package: `web`
Session cookie management, auth middleware, and template rendering helpers
for htmx web UIs. Extracted from `internal/webserver/` across services.
### Session Cookies
```go
func SetSessionCookie(w http.ResponseWriter, name, token string)
func ClearSessionCookie(w http.ResponseWriter, name string)
func GetSessionToken(r *http.Request, name string) string
```
All session cookies are set with: `HttpOnly`, `Secure`,
`SameSite=Strict`, `Path=/`.
### Auth Middleware
```go
func RequireAuth(auth *auth.Authenticator, cookieName string, loginPath string) func(http.Handler) http.Handler
```
Extracts the session token from the cookie, validates it via the
Authenticator, and either:
- Sets `TokenInfo` in the request context and calls the next handler.
- Redirects to `loginPath` if the token is missing or invalid.
### Context Helpers
```go
func TokenInfoFromContext(ctx context.Context) *auth.TokenInfo
```
### Template Helpers
```go
func RenderTemplate(w http.ResponseWriter, fs embed.FS, name string, data any, funcs ...template.FuncMap)
```
Parses `templates/layout.html` and `templates/<name>` from the embedded FS,
merges any provided FuncMaps, and executes the `layout` template.
---
## 9. Package: `archive`
Service directory snapshot and restore using tar.zst, with SQLite-aware
handling. This is new functionality for MCP, not extracted from existing
services.
### Snapshot
```go
type SnapshotOptions struct {
ServiceDir string // e.g., /srv/myservice
DBPath string // e.g., /srv/myservice/myservice.db (for VACUUM INTO)
DB *sql.DB // live database connection (for VACUUM INTO)
ExcludePatterns []string // additional glob patterns to exclude
}
func Snapshot(opts SnapshotOptions) (io.ReadCloser, error)
```
1. Runs `VACUUM INTO` to create a consistent DB copy in a temp file.
2. Walks the service directory, excluding:
- `*.db`, `*.db-wal`, `*.db-shm` (live database files)
- `backups/` directory
- Any patterns in `ExcludePatterns`
3. Adds the VACUUM INTO copy as `<basename>.db` in the archive.
4. Returns a streaming tar.zst reader.
The archive is produced as a stream — it does not need to be fully buffered
in memory. This allows piping directly over a network connection.
### Restore
```go
func Restore(r io.Reader, destDir string) error
```
Extracts a tar.zst archive into `destDir`. Creates the directory if it does
not exist. Overwrites existing files. Preserves file permissions.
### Compression
Zstandard compression via `github.com/klauspost/compress/zstd`. Default
compression level (3) balances speed and ratio for the typical service
directory size (SQLite DB + config + certs, usually under 100 MB).
---
## 10. Package: `health`
Standard health check implementation for both gRPC and REST. New
functionality to standardize what services already do ad-hoc.
### gRPC
Implements `grpc.health.v1.Health` (the standard gRPC health checking
protocol). Services register it on their gRPC server:
```go
health.RegisterGRPC(grpcServer)
```
### REST
```go
func Handler(db *sql.DB) http.HandlerFunc
```
Returns a handler for `GET /healthz` (or whatever path the service mounts
it on) that:
1. Pings the database.
2. Returns `200 {"status": "ok"}` or `503 {"status": "unhealthy", "error": "..."}`.
---
## 11. Inter-Package Dependencies
```
archive ──→ db (for Snapshot)
auth ──→ (mcias client library)
config ──→ auth (for auth.Config type)
csrf ──→ (stdlib only)
db ──→ (modernc.org/sqlite)
grpcserver ──→ auth, config
health ──→ db
httpserver ──→ config
web ──→ auth, csrf
```
No circular dependencies. Each package can be imported independently except
where noted above.
---
## 12. What MCDSL Does Not Provide
- **Business logic.** Policy engines, engine registries, OCI handlers — these
are service-specific and stay in each service's `internal/` packages.
- **Proto definitions.** Each service owns its own proto files and generated
code.
- **CLI scaffolding.** Cobra command wiring is minimal and service-specific.
- **Database schemas.** Each service defines its own migrations. MCDSL
provides the runner, not the SQL.
- **Templates and static assets.** Each service's web UI is its own. MCDSL
provides rendering helpers, not content.
---
## 13. Migration Path
Existing services adopt MCDSL incrementally — one package at a time:
1. Replace `internal/auth/` with `mcdsl/auth`.
2. Replace database open/pragma code with `mcdsl/db.Open`.
3. Replace migration runner with `mcdsl/db.Migrate`.
4. Replace config loading with `mcdsl/config.Load`.
5. Replace CSRF implementation with `mcdsl/csrf`.
6. Replace server setup with `mcdsl/httpserver` and `mcdsl/grpcserver`.
Each step is independent. Services can adopt one package without adopting
all of them. The old `internal/` code can be removed after each migration.
---
## 14. Security Considerations
- **Token caching** uses SHA-256 of the token as the cache key. The raw token
is never used as a map key to prevent timing attacks on map lookup.
- **CSRF secrets** must be generated from `crypto/rand` and should be unique
per service instance. MCDSL does not generate them — the service provides
them.
- **Session cookies** are always `HttpOnly`, `Secure`, `SameSite=Strict`.
These flags are not configurable — relaxing them would be a security defect.
- **gRPC method maps** default to deny. An unregistered method is rejected,
not allowed. This is the most important safety property in the library.
- **File permissions** on databases are `0600`. This is not configurable.
- **TLS 1.3 minimum** is not configurable. Services that need TLS 1.2 (there
should be none) cannot use `httpserver` or `grpcserver`.