Kyle Isom
d5580f01f2
All import paths updated to git.wntrmute.dev/mc/. Bumps mcdsl to v1.2.0. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
769 lines
36 KiB
Markdown
769 lines
36 KiB
Markdown
# MCR Development Progress
|
||
|
||
Reverse-chronological log of development work. Most recent entries first.
|
||
See `PROJECT_PLAN.md` for the implementation roadmap and
|
||
`ARCHITECTURE.md` for the full design specification.
|
||
|
||
## Current State
|
||
|
||
**Phase:** 13 complete
|
||
**Last updated:** 2026-03-25
|
||
|
||
### Completed
|
||
|
||
- Phase 0: Project scaffolding (all 4 steps)
|
||
- Phase 1: Configuration & database (all 3 steps)
|
||
- Phase 2: Blob storage layer (all 2 steps)
|
||
- Phase 3: MCIAS authentication (all 4 steps)
|
||
- Phase 4: Policy engine (all 4 steps)
|
||
- Phase 5: OCI pull path (all 5 steps)
|
||
- Phase 6: OCI push path (all 3 steps)
|
||
- Phase 7: OCI delete path (all 2 steps)
|
||
- Phase 8: Admin REST API (all 5 steps)
|
||
- Phase 9: Garbage collection (all 2 steps)
|
||
- Phase 10: gRPC admin API (all 4 steps)
|
||
- Phase 11: CLI tool (all 3 steps)
|
||
- Phase 12: Web UI (all 5 steps)
|
||
- `ARCHITECTURE.md` — Full design specification (18 sections)
|
||
- `CLAUDE.md` — AI development guidance
|
||
- `PROJECT_PLAN.md` — Implementation plan (14 phases, 40+ steps)
|
||
- `PROGRESS.md` — This file
|
||
|
||
### Next Steps
|
||
|
||
1. Deploy to rift (issue MCR service token, generate TLS cert, update mc-proxy routes)
|
||
2. Consider adding roles to MCIAS login response to eliminate the extra ValidateToken round-trip
|
||
|
||
### 2026-03-26 — Web UI: block guest login
|
||
|
||
**Task:** Prevent MCIAS guest accounts from logging into the web UI.
|
||
|
||
**Changes:**
|
||
|
||
- `internal/webserver/server.go`: Added `ValidateFunc` type; `New()`
|
||
accepts a validate function to inspect tokens post-login.
|
||
- `internal/webserver/auth.go`: After `loginFn` succeeds, calls
|
||
`validateFn` to retrieve roles. Rejects accounts with the `guest`
|
||
role before setting the session cookie.
|
||
- `cmd/mcr-web/main.go`: Wires `ValidateFunc` via `authClient.ValidateToken()`.
|
||
- `internal/webserver/server_test.go`: Added guest/user test accounts,
|
||
`validateFn` returning role-appropriate responses, `TestLoginDeniesGuest`.
|
||
- `ARCHITECTURE.md`: Updated Web UI security section and threat mitigations
|
||
to document guest blocking as defense-in-depth.
|
||
|
||
**Design note:** MCIAS `/v1/auth/login` does not return roles, so the
|
||
web UI makes a second `ValidateToken` call after login to inspect them.
|
||
This is an extra MCIAS round-trip at login time (cached for 30s). A
|
||
future MCIAS change to include roles in the login response would
|
||
eliminate this.
|
||
|
||
### 2026-03-25 — Phase 13: Deployment Artifacts
|
||
|
||
**Task:** Create Dockerfiles, systemd units, install script, and rift deployment config.
|
||
|
||
**Changes:**
|
||
|
||
Step 13.1 — Dockerfiles:
|
||
- `Dockerfile.api`: Multi-stage build for mcrsrv (golang:1.25-alpine → alpine:3.21,
|
||
non-root `mcr` user, ports 8443/9443, volume /srv/mcr)
|
||
- `Dockerfile.web`: Multi-stage build for mcr-web (same pattern, port 8080)
|
||
|
||
Step 13.2 — systemd units:
|
||
- `deploy/systemd/mcr.service`: Registry server with full security hardening
|
||
- `deploy/systemd/mcr-web.service`: Web UI with read-only /srv/mcr
|
||
- `deploy/systemd/mcr-backup.service`: Oneshot snapshot + 30-day prune
|
||
- `deploy/systemd/mcr-backup.timer`: Daily 02:00 UTC with 5-min jitter
|
||
|
||
Step 13.3 — Install script and configs:
|
||
- `deploy/scripts/install.sh`: Idempotent install (user, binaries, dirs, units)
|
||
- `deploy/mcr-rift.toml`: Rift-specific config (MCIAS auth, TLS, storage paths)
|
||
- `deploy/docker/docker-compose-rift.yml`: Docker compose for rift with
|
||
loopback port bindings (28443, 29443, 28080) for mc-proxy fronting
|
||
|
||
---
|
||
|
||
### 2026-03-19 — Batch C: Phase 11 (CLI tool) + Phase 12 (Web UI)
|
||
|
||
**Task:** Implement the admin CLI and HTMX-based web UI — the two
|
||
remaining user-facing layers. Both depend on Phase 10 (gRPC) but not
|
||
on each other; implemented in parallel.
|
||
|
||
**Changes:**
|
||
|
||
Phase 11 — `cmd/mcrctl/` (Steps 11.1–11.3):
|
||
|
||
Step 11.1 — Client and connection setup:
|
||
- `client.go`: `apiClient` struct wrapping both `*http.Client` (REST)
|
||
and gRPC service clients (Registry, Policy, Audit, Admin); `newClient()`
|
||
builds from flags; TLS 1.3 minimum with optional custom CA cert;
|
||
gRPC dial uses `grpc.ForceCodecV2(mcrv1.JSONCodec{})` for JSON codec;
|
||
`restDo()` helper with `Authorization: Bearer` header and JSON error
|
||
parsing; transport auto-selected based on `--grpc` flag
|
||
|
||
Step 11.2 — Status and repository commands:
|
||
- `main.go`: global persistent flags `--server`, `--grpc`, `--token`
|
||
(fallback `MCR_TOKEN`), `--ca-cert`, `--json`; `PersistentPreRunE`
|
||
resolves token and creates client; `status` command (gRPC + REST);
|
||
`repo list` with table/JSON output; `repo delete` with confirmation
|
||
prompt
|
||
- `output.go`: `formatSize()` (B/KB/MB/GB/TB), `printJSON()` (indented),
|
||
`printTable()` via `text/tabwriter`
|
||
|
||
Step 11.3 — Policy, audit, GC, and snapshot commands:
|
||
- `main.go`: `policy list|create|update|delete` (full CRUD, `--rule`
|
||
flag for JSON body, confirmation on delete); `audit tail` with
|
||
`--n` and `--event-type` flags; `gc` with `--reconcile` flag;
|
||
`gc status`; `snapshot`; all commands support both REST and gRPC
|
||
- `client_test.go`: 10 tests covering formatSize, printJSON, printTable,
|
||
token resolution from env/flag, newClient REST mode, CA cert error
|
||
handling, restDo success/error/POST paths
|
||
|
||
Phase 12 — `cmd/mcr-web/` + `internal/webserver/` + `web/` (Steps 12.1–12.5):
|
||
|
||
Step 12.1 — Web server scaffolding:
|
||
- `cmd/mcr-web/main.go`: reads `[web]` config section, creates gRPC
|
||
connection with TLS 1.3 and JSON codec, creates MCIAS auth client for
|
||
login, generates random 32-byte CSRF key, creates webserver, starts
|
||
HTTPS with TLS 1.3, graceful shutdown on SIGINT/SIGTERM
|
||
- `internal/webserver/server.go`: `Server` struct with chi router,
|
||
gRPC service clients, CSRF key, login function; `New()` constructor;
|
||
chi middleware (Recoverer, RequestID, RealIP); routes for all pages;
|
||
session-protected route groups; static file serving from embedded FS
|
||
- `web/embed.go`: `//go:embed templates static` directive
|
||
- `web/static/style.css`: minimal clean CSS (system fonts, 1200px
|
||
container, table styling, form styling, nav bar, stat cards, badges,
|
||
pagination, responsive breakpoints)
|
||
|
||
Step 12.2 — Login and authentication:
|
||
- `internal/webserver/auth.go`: session middleware (checks `mcr_session`
|
||
cookie, redirects to `/login` if absent); login page (GET renders
|
||
form with CSRF token); login submit (POST validates CSRF, calls
|
||
`loginFn`, sets session cookie HttpOnly/Secure/SameSite=Strict);
|
||
logout (clears cookie, redirects); CSRF via signed double-submit
|
||
cookie (HMAC-SHA256)
|
||
- `web/templates/login.html`: centered login form with CSRF hidden field
|
||
|
||
Step 12.3 — Dashboard and repository browsing:
|
||
- `internal/webserver/handlers.go`: `handleDashboard()` (repo count,
|
||
total size, recent audit events via gRPC); `handleRepositories()`
|
||
(list table); `handleRepositoryDetail()` (tags, manifests, repo
|
||
name with `/` support); `handleManifestDetail()` (manifest info
|
||
by digest)
|
||
- `internal/webserver/templates.go`: template loading from embedded FS
|
||
with layout-page composition, function map (formatSize, formatTime,
|
||
truncate, joinStrings), render helper
|
||
- `web/templates/layout.html`: HTML5 base with nav bar, htmx CDN
|
||
- `web/templates/dashboard.html`: stats cards + recent activity table
|
||
- `web/templates/repositories.html`: repo list table
|
||
- `web/templates/repository_detail.html`: tags + manifests tables
|
||
- `web/templates/manifest_detail.html`: digest, media type, size
|
||
|
||
Step 12.4 — Policy management (admin only):
|
||
- `internal/webserver/handlers.go`: `handlePolicies()` (list rules
|
||
with CSRF token); `handleCreatePolicy()` (form with body limit,
|
||
CSRF validation); `handleTogglePolicy()` (get+toggle enabled via
|
||
UpdatePolicyRule with field mask); `handleDeletePolicy()` (CSRF +
|
||
delete); PermissionDenied → "Access denied"
|
||
- `web/templates/policies.html`: create form + rules table with
|
||
toggle/delete actions
|
||
|
||
Step 12.5 — Audit log viewer (admin only):
|
||
- `internal/webserver/handlers.go`: `handleAudit()` with pagination
|
||
(fetch N+1 for next-page detection), filters (event type, repository,
|
||
date range), URL builder for pagination links
|
||
- `web/templates/audit.html`: filter form + paginated event table
|
||
|
||
**Verification:**
|
||
- `make all` passes: vet clean, lint 0 issues, all tests passing,
|
||
all 3 binaries built
|
||
- CLI tests (10 new): formatSize (5 values: B through TB), printJSON
|
||
output correctness, printTable header and row rendering, token from
|
||
env var, token flag overrides env, newClient REST mode, CA cert
|
||
errors (missing file, invalid PEM), restDo success with auth header,
|
||
restDo error response parsing, restDo POST with body
|
||
- Web UI tests (15 new): login page renders, invalid credentials error,
|
||
CSRF token validation, dashboard requires session (redirect),
|
||
dashboard with session, repositories page, repository detail,
|
||
logout (cookie clearing), policies page, audit page, static files,
|
||
formatSize, formatTime, truncate, login success sets cookie
|
||
|
||
---
|
||
|
||
## Log
|
||
|
||
### 2026-03-19 — Phase 10: gRPC admin API
|
||
|
||
**Task:** Implement the gRPC admin API server with the same business
|
||
logic as the REST admin API, per ARCHITECTURE.md section 7.
|
||
|
||
**Changes:**
|
||
|
||
Step 10.1 — Proto definitions (`proto/mcr/v1/`):
|
||
- `common.proto`: `PaginationRequest` shared type
|
||
- `registry.proto`: `RegistryService` with `ListRepositories`,
|
||
`GetRepository`, `DeleteRepository`, `GarbageCollect`, `GetGCStatus` RPCs
|
||
- `policy.proto`: `PolicyService` with `ListPolicyRules`,
|
||
`CreatePolicyRule`, `GetPolicyRule`, `UpdatePolicyRule`,
|
||
`DeletePolicyRule` RPCs; `UpdatePolicyRuleRequest` includes field mask
|
||
- `audit.proto`: `AuditService` with `ListAuditEvents` RPC; filter
|
||
fields for event_type, actor_id, repository, time range
|
||
- `admin.proto`: `AdminService` with `Health` RPC
|
||
|
||
Step 10.2 — Generated code (`gen/mcr/v1/`):
|
||
- Hand-written message types and gRPC service descriptors matching
|
||
protoc-gen-go v1.36+ and protoc-gen-go-grpc v1.5+ output patterns
|
||
- `codec.go`: JSON codec implementing `encoding.CodecV2` via
|
||
`mem.BufferSlice`, registered globally via `init()` as stand-in
|
||
until protobuf code generation is available
|
||
- Handler functions properly delegate to interceptor chain (critical
|
||
for auth/admin enforcement; handlers that ignore the interceptor
|
||
parameter bypass security checks entirely)
|
||
- Client and server interfaces with `mustEmbedUnimplemented*Server()`
|
||
for forward compatibility
|
||
|
||
Step 10.3 — Interceptor chain (`internal/grpcserver/interceptors.go`):
|
||
- `loggingInterceptor`: logs method, peer IP, status code, duration;
|
||
never logs the authorization metadata value
|
||
- `authInterceptor`: extracts `authorization` metadata, validates
|
||
bearer token via `TokenValidator` interface, injects claims into
|
||
context; `Health` bypasses auth via `authBypassMethods` map
|
||
- `adminInterceptor`: requires admin role for GC, policy, delete,
|
||
audit RPCs via `adminRequiredMethods` map; returns
|
||
`codes.PermissionDenied` for insufficient role
|
||
- gRPC errors: `codes.Unauthenticated` for missing/invalid token,
|
||
`codes.PermissionDenied` for insufficient role
|
||
|
||
Step 10.4 — Server implementation (`internal/grpcserver/`):
|
||
- `server.go`: `New(certFile, keyFile, deps)` creates configured
|
||
gRPC server with TLS 1.3 minimum (skips TLS if paths empty for
|
||
testing); `Serve()`, `GracefulStop()`
|
||
- `registry.go`: `registryService` implementing all 5 RPCs with
|
||
same DB calls as REST handlers; GC runs asynchronously with its
|
||
own `GCStatus` tracking (separate from REST's `GCState` since
|
||
`GCState.mu` is unexported); shares `gc.Collector` for actual GC
|
||
- `policy.go`: `policyService` implementing all 5 RPCs with
|
||
validation (effect, actions, priority), field mask updates, engine
|
||
reload, audit events
|
||
- `audit.go`: `auditService` implementing `ListAuditEvents` with
|
||
pagination and filter pass-through to `db.ListAuditEvents`
|
||
- `admin.go`: `adminService` implementing `Health` (returns "ok")
|
||
|
||
**Dependencies added:**
|
||
- `google.golang.org/grpc` v1.79.3
|
||
- `google.golang.org/protobuf` v1.36.11
|
||
- Transitive: `golang.org/x/net`, `golang.org/x/text`,
|
||
`google.golang.org/genproto/googleapis/rpc`
|
||
|
||
**Verification:**
|
||
- `make all` passes: vet clean, lint 0 issues, all tests passing,
|
||
all 3 binaries built
|
||
- Interceptor tests (12 new): Health bypasses auth, no token rejected
|
||
(Unauthenticated), invalid token rejected (Unauthenticated), valid
|
||
token accepted, non-admin denied policy RPCs (PermissionDenied),
|
||
admin allowed policy RPCs, admin required methods completeness
|
||
check, auth bypass methods completeness check, delete repo requires
|
||
admin, GC requires admin, audit requires admin
|
||
- Registry service tests (7 new): list repositories empty, get repo
|
||
not found, get repo empty name (InvalidArgument), delete repo not
|
||
found, delete repo empty name, GC status initial (not running),
|
||
GC trigger returns ID
|
||
- Policy service tests (8 new): create policy rule (with engine
|
||
reload verification), create validation (5 subtests: zero priority,
|
||
empty description, invalid effect, no actions, invalid action), get
|
||
policy rule, get not found, list policy rules, delete + verify
|
||
gone + reload count, delete not found, update with field mask
|
||
- Audit service tests (3 new): list empty, list with data (verify
|
||
fields and details map), list with pagination
|
||
- Admin service tests (2 new): Health returns "ok", Health without
|
||
auth succeeds
|
||
|
||
---
|
||
|
||
### 2026-03-19 — Phase 9: Garbage collection
|
||
|
||
**Task:** Implement the two-phase GC algorithm for removing unreferenced
|
||
blobs per ARCHITECTURE.md §9.
|
||
|
||
**Changes:**
|
||
|
||
Step 9.1 — GC engine (`internal/gc/`):
|
||
- `gc.go`: `Collector` struct with `sync.Mutex` for registry-wide lock;
|
||
`New(db, storage)` constructor; `Run(ctx)` executes two-phase algorithm
|
||
(Phase 1: find unreferenced blobs + delete rows in transaction;
|
||
Phase 2: delete files from storage); `Reconcile(ctx)` scans filesystem
|
||
for orphaned files with no DB row (crash recovery); `TryLock()` for
|
||
concurrent GC rejection
|
||
- `errors.go`: `ErrGCRunning` sentinel
|
||
- `DB` interface: `FindAndDeleteUnreferencedBlobs()`, `BlobExistsByDigest()`
|
||
- `Storage` interface: `Delete()`, `ListBlobDigests()`
|
||
- `db/gc.go`: `FindAndDeleteUnreferencedBlobs()` — LEFT JOIN blobs to
|
||
manifest_blobs, finds unreferenced, deletes rows in single transaction;
|
||
`BlobExistsByDigest()`
|
||
- `storage/list.go`: `ListBlobDigests()` — scans sha256 prefix dirs
|
||
|
||
Step 9.2 — Wire GC into server:
|
||
- `server/admin_gc.go`: updated `GCState` to hold `*gc.Collector` and
|
||
`AuditFunc`; `AdminTriggerGCHandler` now launches `collector.Run()`
|
||
in a goroutine, tracks result, writes `gc_started`/`gc_completed`
|
||
audit events
|
||
|
||
**Verification:**
|
||
- `make all` passes: vet clean, lint 0 issues, all tests passing,
|
||
all 3 binaries built
|
||
- GC engine tests (6 new): removes unreferenced blobs (verify both DB
|
||
rows and storage files deleted, referenced blobs preserved), does not
|
||
remove referenced blobs, concurrent GC rejected (ErrGCRunning), empty
|
||
registry (no-op), reconcile cleans orphaned files, reconcile empty
|
||
storage
|
||
- DB GC tests (3 new): FindAndDeleteUnreferencedBlobs (unreferenced
|
||
removed, referenced preserved), no unreferenced returns nil,
|
||
BlobExistsByDigest (found + not found)
|
||
|
||
---
|
||
|
||
### 2026-03-19 — Phase 7: OCI delete path
|
||
|
||
**Task:** Implement manifest and blob deletion per OCI Distribution Spec.
|
||
|
||
**Changes:**
|
||
|
||
Step 7.1 — Manifest delete:
|
||
- `db/delete.go`: `DeleteManifest(repoID, digest)` — deletes manifest
|
||
row; ON DELETE CASCADE handles manifest_blobs and tags
|
||
- `oci/delete.go`: `handleManifestDelete()` — policy check
|
||
(registry:delete), rejects deletion by tag (405 UNSUPPORTED per OCI
|
||
spec), returns 202 Accepted, writes `manifest_deleted` audit event
|
||
- Updated `oci/routes.go` dispatch to handle DELETE on manifests
|
||
|
||
Step 7.2 — Blob delete:
|
||
- `db/delete.go`: `DeleteBlobFromRepo(repoID, digest)` — removes
|
||
manifest_blobs associations only; does NOT delete the blob row or
|
||
file (GC's responsibility, since other repos may reference it)
|
||
- `oci/delete.go`: `handleBlobDelete()` — policy check, returns 202,
|
||
writes `blob_deleted` audit event
|
||
- Updated `oci/routes.go` dispatch to handle DELETE on blobs
|
||
- Extended `DBQuerier` interface with delete methods
|
||
|
||
**Verification:**
|
||
- `make all` passes: vet clean, lint 0 issues, all tests passing,
|
||
all 3 binaries built
|
||
- DB delete tests (5 new): delete manifest (verify cascade to tags and
|
||
manifest_blobs, blob row preserved), manifest not found, delete blob
|
||
from repo (manifest_blobs removed, blob row preserved, manifest
|
||
preserved), blob not found, blob exists globally but not in repo
|
||
- OCI delete tests (8 new): manifest delete by digest (202), delete by
|
||
tag (405 UNSUPPORTED), manifest not found (404 MANIFEST_UNKNOWN),
|
||
repo not found (404 NAME_UNKNOWN), cascading tag deletion verified,
|
||
blob delete (202), blob not in repo (404 BLOB_UNKNOWN), blob delete
|
||
repo not found
|
||
|
||
---
|
||
|
||
### 2026-03-19 — Phase 6: OCI push path
|
||
|
||
**Task:** Implement blob uploads (monolithic and chunked) and manifest
|
||
push per ARCHITECTURE.md §5 and OCI Distribution Spec.
|
||
|
||
**Changes:**
|
||
|
||
Step 6.1 — Blob upload initiation:
|
||
- `db/upload.go`: `UploadRow` type, `ErrUploadNotFound` sentinel;
|
||
`CreateUpload()`, `GetUpload()`, `UpdateUploadOffset()`, `DeleteUpload()`
|
||
- `db/push.go`: `GetOrCreateRepository()` (implicit repo creation on
|
||
first push), `BlobExists()`, `InsertBlob()` (INSERT OR IGNORE for
|
||
content-addressed dedup)
|
||
- `oci/upload.go`: `uploadManager` (sync.Mutex-protected map of in-progress
|
||
BlobWriters by UUID); `generateUUID()` via crypto/rand; `handleUploadInitiate()`
|
||
— policy check (registry:push), implicit repo creation, DB row + storage
|
||
temp file, returns 202 with Location/Docker-Upload-UUID/Range headers
|
||
- Extended `DBQuerier` interface with push/upload methods
|
||
- Changed `BlobOpener` to `BlobStore` adding `StartUpload(uuid)` method
|
||
|
||
Step 6.2 — Blob upload chunked and monolithic:
|
||
- `oci/upload.go`: `handleUploadChunk()` (PATCH — append body, update offset),
|
||
`handleUploadComplete()` (PUT — optional final body, commit with digest
|
||
verification, insert blob row, cleanup upload, audit event),
|
||
`handleUploadStatus()` (GET — 204 with Range header),
|
||
`handleUploadCancel()` (DELETE — cancel BlobWriter, remove upload row)
|
||
- `oci/routes.go`: `parseOCIPath()` extended to handle `/blobs/uploads/`
|
||
and `/blobs/uploads/<uuid>` patterns; `dispatchUpload()` routes by method
|
||
|
||
Step 6.3 — Manifest push:
|
||
- `db/push.go`: `PushManifestParams` struct, `PushManifest()` — single
|
||
SQLite transaction: create repo if not exists, upsert manifest (ON
|
||
CONFLICT DO UPDATE), clear and repopulate manifest_blobs join table,
|
||
upsert tag if provided (atomic tag move)
|
||
- `oci/manifest.go`: `handleManifestPut()` — full push flow per §5:
|
||
parse JSON, compute SHA-256, verify digest if push-by-digest, collect
|
||
config+layer descriptors, verify all referenced blobs exist (400
|
||
MANIFEST_BLOB_UNKNOWN if missing), call PushManifest(), audit event
|
||
with tag details, returns 201 with Docker-Content-Digest/Location/
|
||
Content-Type headers
|
||
|
||
**Verification:**
|
||
- `make all` passes: vet clean, lint 0 issues, 198 tests passing,
|
||
all 3 binaries built
|
||
- DB tests (15 new): upload CRUD (create/get/update/delete/not-found),
|
||
GetOrCreateRepository (new + existing), BlobExists (found/not-found),
|
||
InsertBlob (new + idempotent), PushManifest by tag (verify repo creation,
|
||
manifest, tag, manifest_blobs), by digest (no tag), tag move (atomic
|
||
update), idempotent re-push
|
||
- OCI upload tests (8 new): initiate (202, Location/UUID/Range headers,
|
||
implicit repo creation), unique UUIDs (5 initiates → 5 distinct UUIDs),
|
||
monolithic upload (POST → PUT with body → 201), chunked upload
|
||
(POST → PATCH → PATCH → PUT → 201), digest mismatch (400 DIGEST_INVALID),
|
||
upload status (GET → 204), cancel (DELETE → 204), nonexistent UUID
|
||
(PATCH → 404 BLOB_UPLOAD_UNKNOWN)
|
||
- OCI manifest push tests (7 new): push by tag (201, correct headers),
|
||
push by digest, digest mismatch (400), missing blob (400
|
||
MANIFEST_BLOB_UNKNOWN), malformed JSON (400 MANIFEST_INVALID), empty
|
||
manifest, tag update (atomic move), re-push idempotent
|
||
- Route parsing tests (5 new subtests): upload initiate (trailing slash),
|
||
upload initiate (no trailing slash), upload with UUID, multi-segment
|
||
repo upload, multi-segment repo upload initiate
|
||
|
||
---
|
||
|
||
### 2026-03-19 — Batch B: Phase 5 (OCI pull) + Phase 8 (admin REST)
|
||
|
||
**Task:** Implement the OCI Distribution Spec pull path and the admin
|
||
REST API management endpoints. Both phases depend on Phase 4 (policy)
|
||
but not on each other — implemented in parallel.
|
||
|
||
**Changes:**
|
||
|
||
Phase 5 — `internal/db/` + `internal/oci/` (Steps 5.1–5.5):
|
||
|
||
Step 5.1 — OCI handler scaffolding:
|
||
- `db/errors.go`: `ErrRepoNotFound`, `ErrManifestNotFound`, `ErrBlobNotFound`
|
||
sentinel errors
|
||
- `oci/handler.go`: `Handler` struct with `DBQuerier`, `BlobOpener`,
|
||
`PolicyEval`, `AuditFunc` interfaces; `NewHandler()` constructor;
|
||
`checkPolicy()` inline policy check; `audit()` helper
|
||
- `oci/ocierror.go`: `writeOCIError()` duplicated from server package
|
||
(15 lines, avoids coupling)
|
||
- `oci/routes.go`: `parseOCIPath()` splits from the right to handle
|
||
multi-segment repo names (e.g., `org/team/app/manifests/latest`);
|
||
`Router()` returns chi router with `/_catalog` and `/*` catch-all;
|
||
`dispatch()` routes to manifest/blob/tag handlers
|
||
|
||
Step 5.2 — Manifest pull:
|
||
- `db/repository.go`: `ManifestRow` type, `GetRepositoryByName()`,
|
||
`GetManifestByTag()`, `GetManifestByDigest()`
|
||
- `oci/manifest.go`: GET/HEAD `/v2/{name}/manifests/{reference}`;
|
||
resolves by tag or digest; sets Content-Type, Docker-Content-Digest,
|
||
Content-Length headers
|
||
|
||
Step 5.3 — Blob download:
|
||
- `db/repository.go`: `BlobExistsInRepo()` — joins blobs+manifest_blobs+
|
||
manifests to verify blob belongs to repo
|
||
- `oci/blob.go`: GET/HEAD `/v2/{name}/blobs/{digest}`; validates blob
|
||
exists in repo before streaming from storage
|
||
|
||
Step 5.4 — Tag listing:
|
||
- `db/repository.go`: `ListTags()` with cursor-based pagination
|
||
(after/limit)
|
||
- `oci/tags.go`: GET `/v2/{name}/tags/list` with OCI `?n=`/`?last=`
|
||
pagination and `Link` header for next page
|
||
|
||
Step 5.5 — Catalog listing:
|
||
- `db/repository.go`: `ListRepositoryNames()` with cursor-based pagination
|
||
- `oci/catalog.go`: GET `/v2/_catalog` with same pagination pattern
|
||
|
||
Phase 8 — `internal/db/` + `internal/server/` (Steps 8.1–8.5):
|
||
|
||
Step 8.1 — Auth endpoints:
|
||
- `server/admin_auth.go`: POST `/v1/auth/login` (JSON body → MCIAS),
|
||
POST `/v1/auth/logout` (204 stub), GET `/v1/health` (no auth)
|
||
|
||
Step 8.2 — Repository management:
|
||
- `db/admin.go`: `RepoMetadata`, `TagInfo`, `ManifestInfo`, `RepoDetail`
|
||
types; `ListRepositoriesWithMetadata()`, `GetRepositoryDetail()`,
|
||
`DeleteRepository()`
|
||
- `server/admin_repo.go`: GET/DELETE `/v1/repositories` and
|
||
`/v1/repositories/*` (wildcard for multi-segment names)
|
||
|
||
Step 8.3 — Policy CRUD:
|
||
- `db/admin.go`: `PolicyRuleRow` type, `ErrPolicyRuleNotFound`;
|
||
`CreatePolicyRule()`, `GetPolicyRule()`, `ListPolicyRules()`,
|
||
`UpdatePolicyRule()`, `SetPolicyRuleEnabled()`, `DeletePolicyRule()`
|
||
- `server/admin_policy.go`: full CRUD on `/v1/policy/rules` and
|
||
`/v1/policy/rules/{id}`; `PolicyReloader` interface; input validation
|
||
(priority >= 1, valid effect/actions); mutations trigger engine reload
|
||
|
||
Step 8.4 — Audit endpoint:
|
||
- `server/admin_audit.go`: GET `/v1/audit` with query parameter filters
|
||
(event_type, actor_id, repository, since, until, n, offset);
|
||
delegates to `db.ListAuditEvents`
|
||
|
||
Step 8.5 — GC endpoints:
|
||
- `server/admin_gc.go`: `GCState` struct with `sync.Mutex`; POST `/v1/gc`
|
||
returns 202 (stub for Phase 9); GET `/v1/gc/status`; concurrent
|
||
trigger returns 409
|
||
|
||
Shared admin infrastructure:
|
||
- `server/admin.go`: `writeAdminError()` (platform `{"error":"..."}`
|
||
format), `writeJSON()`, `RequireAdmin()` middleware, `hasRole()` helper
|
||
- `server/admin_routes.go`: `AdminDeps` struct, `MountAdminRoutes()`
|
||
mounts all `/v1/*` endpoints with proper auth/admin middleware layers
|
||
|
||
**Verification:**
|
||
- `make all` passes: vet clean, lint 0 issues, 168 tests passing,
|
||
all 3 binaries built
|
||
- Phase 5 (34 new tests): parseOCIPath (14 subtests covering simple/
|
||
multi-segment/edge cases), manifest GET by tag/digest + HEAD + not
|
||
found, blob GET/HEAD + not in repo + repo not found, tags list +
|
||
pagination + empty + repo not found, catalog list + pagination + empty,
|
||
DB repository methods (15 tests covering all 6 query methods)
|
||
- Phase 8 (51 new tests): DB admin methods (19 tests covering CRUD,
|
||
pagination, cascade, not-found), admin auth (login ok/fail, health,
|
||
logout), admin repos (list/detail/delete/non-admin 403), admin policy
|
||
(full CRUD cycle, validation errors, non-admin 403, engine reload),
|
||
admin audit (list with filters, pagination), admin GC (trigger 202,
|
||
status, concurrent 409), RequireAdmin middleware (allowed/denied/
|
||
no claims)
|
||
|
||
---
|
||
|
||
### 2026-03-19 — Phase 4: Policy engine
|
||
|
||
**Task:** Implement the registry-specific authorization engine with
|
||
priority-based, deny-wins, default-deny evaluation per ARCHITECTURE.md §4.
|
||
|
||
**Changes:**
|
||
|
||
Step 4.1 — `internal/policy/` core types and evaluation:
|
||
- `policy.go`: `Action` (6 constants), `Effect` (Allow/Deny), `PolicyInput`,
|
||
`Rule` types per ARCHITECTURE.md §4
|
||
- `Evaluate(input, rules)` — stateless evaluation: sort by priority (stable),
|
||
collect all matching rules, deny-wins, default-deny
|
||
- Rule matching: all populated fields ANDed; empty fields are wildcards;
|
||
`Repositories` glob matching via `path.Match`; empty repo (global ops)
|
||
only matches rules with empty Repositories list
|
||
|
||
Step 4.2 — `internal/policy/` built-in defaults:
|
||
- `defaults.go`: `DefaultRules()` returns 3 built-in rules (negative IDs,
|
||
priority 0): admin wildcard (all actions), human user content access
|
||
(pull/push/delete/catalog), version check (always accessible)
|
||
|
||
Step 4.3 — `internal/policy/` engine wrapper with DB integration:
|
||
- `engine.go`: `Engine` struct with `sync.RWMutex`-protected rule cache;
|
||
`NewEngine()` pre-loaded with defaults; `SetRules()` merges with defaults;
|
||
`Evaluate()` thread-safe evaluation; `Reload(RuleStore)` loads from DB
|
||
- `RuleStore` interface: `LoadEnabledPolicyRules() ([]Rule, error)`
|
||
- `internal/db/policy.go`: `LoadEnabledPolicyRules()` on `*DB` — loads
|
||
enabled rules from `policy_rules` table, parses `rule_json` JSON column,
|
||
returns `[]policy.Rule` ordered by priority
|
||
|
||
Step 4.4 — `internal/server/` policy middleware:
|
||
- `policy.go`: `PolicyEvaluator` interface, `AuditFunc` callback type,
|
||
`RequirePolicy(evaluator, action, auditFn)` middleware — extracts claims
|
||
from context, repo name from chi URL param, assembles `PolicyInput`,
|
||
returns OCI DENIED (403) on deny with optional audit callback
|
||
|
||
**Verification:**
|
||
- `make all` passes: vet clean, lint 0 issues, 69 tests passing
|
||
(17 policy + 14 server + 15 db + 9 auth + 7 config + 14 storage - some
|
||
overlap from updated packages), all 3 binaries built
|
||
- Policy evaluation tests: admin wildcard, user allow, system account deny,
|
||
exact repo match (allow + deny on different repo), glob match
|
||
(production/* matches production/myapp, not production/team/myapp),
|
||
deny-wins over allow, priority ordering, empty repo global operation
|
||
(admin catalog allowed, repo-scoped rule doesn't match), multiple
|
||
matching rules (highest-priority allow returned)
|
||
- Default rules tests: admin allowed for all 6 actions, user allowed for
|
||
pull/push/delete/catalog but denied policy:manage, system account denied
|
||
for all except version_check, version_check allowed for both human and
|
||
system accounts
|
||
- Engine tests: defaults-only (admin allow, system deny), custom rules
|
||
(matching subject allowed, different subject denied), reload picks up new
|
||
rules (old rules gone), reload with empty store (disabled rules excluded,
|
||
falls back to defaults)
|
||
- DB tests: LoadEnabledPolicyRules returns only enabled rules ordered by
|
||
priority, parses rule_json correctly (effect, subject_uuid, actions,
|
||
repositories), empty table returns nil
|
||
- Middleware tests: admin allowed, user allowed, system denied (403 with
|
||
OCI DENIED error), system with matching rule allowed, explicit deny
|
||
rule blocks access (403)
|
||
|
||
---
|
||
|
||
### 2026-03-19 — Batch A: Phase 2 (blob storage) + Phase 3 (MCIAS auth)
|
||
|
||
**Task:** Implement content-addressed blob storage and MCIAS authentication
|
||
with OCI token endpoint and auth middleware.
|
||
|
||
**Changes:**
|
||
|
||
Phase 2 — `internal/storage/` (Steps 2.1 + 2.2):
|
||
- `storage.go`: `Store` struct with `layersPath`/`uploadsPath`, `New()`
|
||
constructor, digest validation (`^sha256:[a-f0-9]{64}$`), content-addressed
|
||
path layout: `<layers>/sha256/<first-2-hex>/<full-64-hex>`
|
||
- `writer.go`: `BlobWriter` wrapping `*os.File` + `crypto/sha256` running hash
|
||
via `io.MultiWriter`. `StartUpload(uuid)` creates temp file in uploads dir.
|
||
`Write()` updates both file and hash. `Commit(expectedDigest)` finalizes hash,
|
||
verifies digest, `MkdirAll` prefix dir, `Rename` atomically. `Cancel()` cleans
|
||
up temp file. `BytesWritten()` returns offset.
|
||
- `reader.go`: `Open(digest)` returns `io.ReadCloser`, `Stat(digest)` returns
|
||
size, `Delete(digest)` removes blob + best-effort prefix dir cleanup,
|
||
`Exists(digest)` returns bool. All validate digest format first.
|
||
- `errors.go`: `ErrBlobNotFound`, `ErrDigestMismatch`, `ErrInvalidDigest`
|
||
- No new dependencies (stdlib only)
|
||
|
||
Phase 3 — `internal/auth/` (Steps 3.1) + `internal/server/` (Steps 3.2–3.4):
|
||
- `auth/client.go`: `Client` with `NewClient(serverURL, caCert, serviceName,
|
||
tags)`, TLS 1.3 minimum, optional custom CA cert, 10s HTTP timeout.
|
||
`Login()` POSTs to MCIAS `/v1/auth/login`. `ValidateToken()` with SHA-256
|
||
cache keying and 30s TTL.
|
||
- `auth/claims.go`: `Claims` struct (Subject, AccountType, Roles) with context
|
||
helpers `ContextWithClaims`/`ClaimsFromContext`
|
||
- `auth/cache.go`: `validationCache` with `sync.RWMutex`, lazy eviction,
|
||
injectable `now` function for testing
|
||
- `auth/errors.go`: `ErrUnauthorized`, `ErrMCIASUnavailable`
|
||
- `server/middleware.go`: `TokenValidator` interface, `RequireAuth` middleware
|
||
(Bearer token extraction, `WWW-Authenticate` header, OCI error format)
|
||
- `server/token.go`: `LoginClient` interface, `TokenHandler` (Basic auth →
|
||
bearer token exchange via MCIAS, RFC 3339 `issued_at`)
|
||
- `server/v2.go`: `V2Handler` returning 200 `{}`
|
||
- `server/routes.go`: `NewRouter` with chi: `/v2/token` (no auth),
|
||
`/v2/` (RequireAuth middleware)
|
||
- `server/ocierror.go`: `writeOCIError()` helper for OCI error JSON format
|
||
- New dependency: `github.com/go-chi/chi/v5`
|
||
|
||
**Verification:**
|
||
- `make all` passes: vet clean, lint 0 issues, 52 tests passing
|
||
(7 config + 13 db/audit + 14 storage + 9 auth + 9 server), all 3 binaries built
|
||
- Storage tests: new store, digest validation (3 valid + 9 invalid), path layout,
|
||
write+commit, digest mismatch rejection (temp cleanup verified), cancel cleanup,
|
||
bytes written tracking, concurrent writes to different UUIDs, open after write,
|
||
stat, exists, delete (verify gone), open not found, invalid digest format
|
||
(covers Open/Stat/Delete/Exists)
|
||
- Auth tests: cache put/get, TTL expiry with clock injection, concurrent cache
|
||
access, login success/failure (httptest mock), validate success/revoked,
|
||
cache hit (request counter), cache expiry (clock advance)
|
||
- Server tests: RequireAuth valid/missing/invalid token, token handler
|
||
success/invalid creds/missing auth, routes integration (authenticated /v2/,
|
||
unauthenticated /v2/ → 401, token endpoint bypasses auth)
|
||
|
||
---
|
||
|
||
### 2026-03-19 — Phase 1: Configuration & database
|
||
|
||
**Task:** Implement TOML config loading with env overrides and validation,
|
||
SQLite database with migrations, and audit log helpers.
|
||
|
||
**Changes:**
|
||
|
||
Step 1.1 — `internal/config/`:
|
||
- `config.go`: `Config` struct matching ARCHITECTURE.md §10 (all 6 TOML
|
||
sections: server, database, storage, mcias, web, log)
|
||
- Parsed with `go-toml/v2`; env overrides via `MCR_` prefix using
|
||
reflection-based struct walker
|
||
- Startup validation: 6 required fields checked (listen_addr, tls_cert,
|
||
tls_key, database.path, storage.layers_path, mcias.server_url)
|
||
- Same-filesystem check for layers_path/uploads_path via device ID
|
||
comparison (walks to nearest existing parent if path doesn't exist yet)
|
||
- Default values: read_timeout=30s, write_timeout=0, idle_timeout=120s,
|
||
shutdown_timeout=60s, uploads_path derived from layers_path, log.level=info
|
||
- `device_linux.go`: Linux-specific `extractDeviceID` using `syscall.Stat_t`
|
||
- `deploy/examples/mcr.toml`: annotated example config
|
||
|
||
Step 1.2 — `internal/db/`:
|
||
- `db.go`: `Open(path)` creates/opens SQLite via `modernc.org/sqlite`,
|
||
sets pragmas (WAL, foreign_keys, busy_timeout=5000), chmod 0600
|
||
- `migrate.go`: migration framework with `schema_migrations` tracking table;
|
||
`Migrate()` applies pending migrations in transactions; `SchemaVersion()`
|
||
reports current version
|
||
- Migration 000001: `repositories`, `manifests`, `tags`, `blobs`,
|
||
`manifest_blobs`, `uploads` — all tables, constraints, and indexes per
|
||
ARCHITECTURE.md §8
|
||
- Migration 000002: `policy_rules`, `audit_log` — tables and indexes per §8
|
||
|
||
Step 1.3 — `internal/db/`:
|
||
- `audit.go`: `WriteAuditEvent(eventType, actorID, repository, digest, ip,
|
||
details)` with JSON-serialized details map; `ListAuditEvents(AuditFilter)`
|
||
with filtering by event_type, actor_id, repository, time range, and
|
||
offset/limit pagination (default 50, descending by event_time)
|
||
- `AuditFilter` struct with all filter fields
|
||
- `AuditEvent` struct with JSON tags for API serialization
|
||
|
||
Lint fix:
|
||
- `.golangci.yaml`: disabled `fieldalignment` analyzer in govet (micro-
|
||
optimization that hurts struct readability; not a security/correctness
|
||
concern)
|
||
|
||
**Verification:**
|
||
- `make all` passes: vet clean, lint 0 issues, 20 tests passing
|
||
(7 config + 13 db/audit), all 3 binaries built
|
||
- Config tests: valid load, defaults applied, uploads_path default,
|
||
5 missing-required-field cases, env override (string + duration),
|
||
same-filesystem check
|
||
- DB tests: open+migrate, idempotent migrate, 9 tables verified,
|
||
foreign key enforcement, tag cascade on manifest delete,
|
||
manifest_blobs cascade (blob row preserved), WAL mode verified
|
||
- Audit tests: write+list, filter by type, filter by actor, filter by
|
||
repository, pagination (3 pages), null fields handled
|
||
|
||
---
|
||
|
||
### 2026-03-19 — Phase 0: Project scaffolding
|
||
|
||
**Task:** Set up Go module, build system, linter config, and binary
|
||
entry points with cobra subcommands.
|
||
|
||
**Changes:**
|
||
- `go.mod`: module `git.wntrmute.dev/mc/mcr`, Go 1.25, cobra dependency
|
||
- Directory skeleton: `cmd/mcrsrv/`, `cmd/mcr-web/`, `cmd/mcrctl/`,
|
||
`internal/`, `proto/mcr/v1/`, `gen/mcr/v1/`, `web/templates/`,
|
||
`web/static/`, `deploy/docker/`, `deploy/examples/`, `deploy/scripts/`,
|
||
`deploy/systemd/`, `docs/`
|
||
- `.gitignore`: binaries, `srv/`, `*.db*`, IDE/OS files
|
||
- `Makefile`: standard targets (`all`, `build`, `test`, `vet`, `lint`,
|
||
`proto`, `proto-lint`, `clean`, `docker`, `devserver`); `all` runs
|
||
`vet → lint → test → mcrsrv mcr-web mcrctl`; `CGO_ENABLED=0` on binary
|
||
builds; version injection via `-X main.version`
|
||
- `.golangci.yaml`: golangci-lint v2 config matching mc-proxy conventions;
|
||
linters: errcheck, govet, ineffassign, unused, errorlint, gosec,
|
||
staticcheck, revive; formatters: gofmt, goimports; gosec G101 excluded
|
||
in test files
|
||
- `buf.yaml`: protobuf linting (STANDARD) and breaking change detection (FILE)
|
||
- `cmd/mcrsrv/main.go`: root command with `server`, `init`, `snapshot`
|
||
subcommands (stubs returning "not implemented")
|
||
- `cmd/mcr-web/main.go`: root command with `server` subcommand (stub)
|
||
- `cmd/mcrctl/main.go`: root command with `status`, `repo` (list/delete),
|
||
`gc` (trigger/status), `policy` (list/create/update/delete),
|
||
`audit` (tail), `snapshot` subcommands (stubs)
|
||
- All binaries accept `--version` flag
|
||
|
||
**Verification:**
|
||
- `make all` passes: vet clean, lint 0 issues, test (no test files),
|
||
all three binaries built successfully
|
||
- `./mcrsrv --version` → `mcrsrv version 3695581`
|
||
- `./mcr-web --version` → `mcr-web version 3695581`
|
||
- All stubs return "not implemented" error as expected
|
||
- `make clean` removes binaries
|
||
|
||
---
|
||
|
||
### 2026-03-19 — Project planning
|
||
|
||
**Task:** Create design documents and implementation plan.
|
||
|
||
**Changes:**
|
||
- `README.md`: Existing one-line description
|
||
- `ARCHITECTURE.md`: Full design specification covering OCI Distribution
|
||
Spec compliance, MCIAS authentication, policy engine, storage design,
|
||
API surface (OCI + admin REST + gRPC), database schema, garbage collection,
|
||
configuration, web UI, CLI tools, deployment, security model
|
||
- `CLAUDE.md`: Development guidance for AI-assisted implementation
|
||
- `PROJECT_PLAN.md`: 14-phase implementation plan with discrete steps,
|
||
acceptance criteria, dependency graph, and batchable work identification
|
||
- `PROGRESS.md`: This progress tracker
|
||
|
||
**Notes:**
|
||
- No code written yet. All files are documentation/planning.
|
||
- ARCHITECTURE.md reviewed and corrected for: GC algorithm crash safety,
|
||
policy glob semantics, tag FK cascade, OCI error format, API sync
|
||
violations, timeout configuration, backup considerations, and other
|
||
consistency issues.
|