Phases 5, 6, 8: OCI pull/push paths and admin REST API
Phase 5 (OCI pull): internal/oci/ package with manifest GET/HEAD by tag/digest, blob GET/HEAD with repo membership check, tag listing with OCI pagination, catalog listing. Multi-segment repo names via parseOCIPath() right-split routing. DB query layer in internal/db/repository.go. Phase 6 (OCI push): blob uploads (monolithic and chunked) with uploadManager tracking in-progress BlobWriters, manifest push implementing full ARCHITECTURE.md §5 flow in a single SQLite transaction (create repo, upsert manifest, populate manifest_blobs, atomic tag move). Digest verification on both blob commit and manifest push-by-digest. Phase 8 (admin REST): /v1 endpoints for auth (login/logout/health), repository management (list/detail/delete), policy CRUD with engine reload, audit log listing with filters, GC trigger/status stubs. RequireAdmin middleware, platform-standard error format. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
177
PROGRESS.md
177
PROGRESS.md
@@ -6,7 +6,7 @@ See `PROJECT_PLAN.md` for the implementation roadmap and
|
||||
|
||||
## Current State
|
||||
|
||||
**Phase:** 4 complete, ready for Batch B (Phase 5 + Phase 8)
|
||||
**Phase:** 6 complete, ready for Phase 7
|
||||
**Last updated:** 2026-03-19
|
||||
|
||||
### Completed
|
||||
@@ -16,6 +16,9 @@ See `PROJECT_PLAN.md` for the implementation roadmap and
|
||||
- Phase 2: Blob storage layer (all 2 steps)
|
||||
- Phase 3: MCIAS authentication (all 4 steps)
|
||||
- Phase 4: Policy engine (all 4 steps)
|
||||
- Phase 5: OCI pull path (all 5 steps)
|
||||
- Phase 6: OCI push path (all 3 steps)
|
||||
- Phase 8: Admin REST API (all 5 steps)
|
||||
- `ARCHITECTURE.md` — Full design specification (18 sections)
|
||||
- `CLAUDE.md` — AI development guidance
|
||||
- `PROJECT_PLAN.md` — Implementation plan (14 phases, 40+ steps)
|
||||
@@ -23,14 +26,180 @@ See `PROJECT_PLAN.md` for the implementation roadmap and
|
||||
|
||||
### Next Steps
|
||||
|
||||
1. Batch B: Phase 5 (OCI pull) and Phase 8 (admin REST) — independent,
|
||||
can be done in parallel
|
||||
2. After Phase 5, Phase 6 (OCI push) then Phase 7 (OCI delete)
|
||||
1. Phase 7 (OCI delete)
|
||||
2. After Phase 7, Phase 9 (garbage collection)
|
||||
3. Phase 10 (gRPC admin API)
|
||||
|
||||
---
|
||||
|
||||
## Log
|
||||
|
||||
### 2026-03-19 — Phase 6: OCI push path
|
||||
|
||||
**Task:** Implement blob uploads (monolithic and chunked) and manifest
|
||||
push per ARCHITECTURE.md §5 and OCI Distribution Spec.
|
||||
|
||||
**Changes:**
|
||||
|
||||
Step 6.1 — Blob upload initiation:
|
||||
- `db/upload.go`: `UploadRow` type, `ErrUploadNotFound` sentinel;
|
||||
`CreateUpload()`, `GetUpload()`, `UpdateUploadOffset()`, `DeleteUpload()`
|
||||
- `db/push.go`: `GetOrCreateRepository()` (implicit repo creation on
|
||||
first push), `BlobExists()`, `InsertBlob()` (INSERT OR IGNORE for
|
||||
content-addressed dedup)
|
||||
- `oci/upload.go`: `uploadManager` (sync.Mutex-protected map of in-progress
|
||||
BlobWriters by UUID); `generateUUID()` via crypto/rand; `handleUploadInitiate()`
|
||||
— policy check (registry:push), implicit repo creation, DB row + storage
|
||||
temp file, returns 202 with Location/Docker-Upload-UUID/Range headers
|
||||
- Extended `DBQuerier` interface with push/upload methods
|
||||
- Changed `BlobOpener` to `BlobStore` adding `StartUpload(uuid)` method
|
||||
|
||||
Step 6.2 — Blob upload chunked and monolithic:
|
||||
- `oci/upload.go`: `handleUploadChunk()` (PATCH — append body, update offset),
|
||||
`handleUploadComplete()` (PUT — optional final body, commit with digest
|
||||
verification, insert blob row, cleanup upload, audit event),
|
||||
`handleUploadStatus()` (GET — 204 with Range header),
|
||||
`handleUploadCancel()` (DELETE — cancel BlobWriter, remove upload row)
|
||||
- `oci/routes.go`: `parseOCIPath()` extended to handle `/blobs/uploads/`
|
||||
and `/blobs/uploads/<uuid>` patterns; `dispatchUpload()` routes by method
|
||||
|
||||
Step 6.3 — Manifest push:
|
||||
- `db/push.go`: `PushManifestParams` struct, `PushManifest()` — single
|
||||
SQLite transaction: create repo if not exists, upsert manifest (ON
|
||||
CONFLICT DO UPDATE), clear and repopulate manifest_blobs join table,
|
||||
upsert tag if provided (atomic tag move)
|
||||
- `oci/manifest.go`: `handleManifestPut()` — full push flow per §5:
|
||||
parse JSON, compute SHA-256, verify digest if push-by-digest, collect
|
||||
config+layer descriptors, verify all referenced blobs exist (400
|
||||
MANIFEST_BLOB_UNKNOWN if missing), call PushManifest(), audit event
|
||||
with tag details, returns 201 with Docker-Content-Digest/Location/
|
||||
Content-Type headers
|
||||
|
||||
**Verification:**
|
||||
- `make all` passes: vet clean, lint 0 issues, 198 tests passing,
|
||||
all 3 binaries built
|
||||
- DB tests (15 new): upload CRUD (create/get/update/delete/not-found),
|
||||
GetOrCreateRepository (new + existing), BlobExists (found/not-found),
|
||||
InsertBlob (new + idempotent), PushManifest by tag (verify repo creation,
|
||||
manifest, tag, manifest_blobs), by digest (no tag), tag move (atomic
|
||||
update), idempotent re-push
|
||||
- OCI upload tests (8 new): initiate (202, Location/UUID/Range headers,
|
||||
implicit repo creation), unique UUIDs (5 initiates → 5 distinct UUIDs),
|
||||
monolithic upload (POST → PUT with body → 201), chunked upload
|
||||
(POST → PATCH → PATCH → PUT → 201), digest mismatch (400 DIGEST_INVALID),
|
||||
upload status (GET → 204), cancel (DELETE → 204), nonexistent UUID
|
||||
(PATCH → 404 BLOB_UPLOAD_UNKNOWN)
|
||||
- OCI manifest push tests (7 new): push by tag (201, correct headers),
|
||||
push by digest, digest mismatch (400), missing blob (400
|
||||
MANIFEST_BLOB_UNKNOWN), malformed JSON (400 MANIFEST_INVALID), empty
|
||||
manifest, tag update (atomic move), re-push idempotent
|
||||
- Route parsing tests (5 new subtests): upload initiate (trailing slash),
|
||||
upload initiate (no trailing slash), upload with UUID, multi-segment
|
||||
repo upload, multi-segment repo upload initiate
|
||||
|
||||
---
|
||||
|
||||
### 2026-03-19 — Batch B: Phase 5 (OCI pull) + Phase 8 (admin REST)
|
||||
|
||||
**Task:** Implement the OCI Distribution Spec pull path and the admin
|
||||
REST API management endpoints. Both phases depend on Phase 4 (policy)
|
||||
but not on each other — implemented in parallel.
|
||||
|
||||
**Changes:**
|
||||
|
||||
Phase 5 — `internal/db/` + `internal/oci/` (Steps 5.1–5.5):
|
||||
|
||||
Step 5.1 — OCI handler scaffolding:
|
||||
- `db/errors.go`: `ErrRepoNotFound`, `ErrManifestNotFound`, `ErrBlobNotFound`
|
||||
sentinel errors
|
||||
- `oci/handler.go`: `Handler` struct with `DBQuerier`, `BlobOpener`,
|
||||
`PolicyEval`, `AuditFunc` interfaces; `NewHandler()` constructor;
|
||||
`checkPolicy()` inline policy check; `audit()` helper
|
||||
- `oci/ocierror.go`: `writeOCIError()` duplicated from server package
|
||||
(15 lines, avoids coupling)
|
||||
- `oci/routes.go`: `parseOCIPath()` splits from the right to handle
|
||||
multi-segment repo names (e.g., `org/team/app/manifests/latest`);
|
||||
`Router()` returns chi router with `/_catalog` and `/*` catch-all;
|
||||
`dispatch()` routes to manifest/blob/tag handlers
|
||||
|
||||
Step 5.2 — Manifest pull:
|
||||
- `db/repository.go`: `ManifestRow` type, `GetRepositoryByName()`,
|
||||
`GetManifestByTag()`, `GetManifestByDigest()`
|
||||
- `oci/manifest.go`: GET/HEAD `/v2/{name}/manifests/{reference}`;
|
||||
resolves by tag or digest; sets Content-Type, Docker-Content-Digest,
|
||||
Content-Length headers
|
||||
|
||||
Step 5.3 — Blob download:
|
||||
- `db/repository.go`: `BlobExistsInRepo()` — joins blobs+manifest_blobs+
|
||||
manifests to verify blob belongs to repo
|
||||
- `oci/blob.go`: GET/HEAD `/v2/{name}/blobs/{digest}`; validates blob
|
||||
exists in repo before streaming from storage
|
||||
|
||||
Step 5.4 — Tag listing:
|
||||
- `db/repository.go`: `ListTags()` with cursor-based pagination
|
||||
(after/limit)
|
||||
- `oci/tags.go`: GET `/v2/{name}/tags/list` with OCI `?n=`/`?last=`
|
||||
pagination and `Link` header for next page
|
||||
|
||||
Step 5.5 — Catalog listing:
|
||||
- `db/repository.go`: `ListRepositoryNames()` with cursor-based pagination
|
||||
- `oci/catalog.go`: GET `/v2/_catalog` with same pagination pattern
|
||||
|
||||
Phase 8 — `internal/db/` + `internal/server/` (Steps 8.1–8.5):
|
||||
|
||||
Step 8.1 — Auth endpoints:
|
||||
- `server/admin_auth.go`: POST `/v1/auth/login` (JSON body → MCIAS),
|
||||
POST `/v1/auth/logout` (204 stub), GET `/v1/health` (no auth)
|
||||
|
||||
Step 8.2 — Repository management:
|
||||
- `db/admin.go`: `RepoMetadata`, `TagInfo`, `ManifestInfo`, `RepoDetail`
|
||||
types; `ListRepositoriesWithMetadata()`, `GetRepositoryDetail()`,
|
||||
`DeleteRepository()`
|
||||
- `server/admin_repo.go`: GET/DELETE `/v1/repositories` and
|
||||
`/v1/repositories/*` (wildcard for multi-segment names)
|
||||
|
||||
Step 8.3 — Policy CRUD:
|
||||
- `db/admin.go`: `PolicyRuleRow` type, `ErrPolicyRuleNotFound`;
|
||||
`CreatePolicyRule()`, `GetPolicyRule()`, `ListPolicyRules()`,
|
||||
`UpdatePolicyRule()`, `SetPolicyRuleEnabled()`, `DeletePolicyRule()`
|
||||
- `server/admin_policy.go`: full CRUD on `/v1/policy/rules` and
|
||||
`/v1/policy/rules/{id}`; `PolicyReloader` interface; input validation
|
||||
(priority >= 1, valid effect/actions); mutations trigger engine reload
|
||||
|
||||
Step 8.4 — Audit endpoint:
|
||||
- `server/admin_audit.go`: GET `/v1/audit` with query parameter filters
|
||||
(event_type, actor_id, repository, since, until, n, offset);
|
||||
delegates to `db.ListAuditEvents`
|
||||
|
||||
Step 8.5 — GC endpoints:
|
||||
- `server/admin_gc.go`: `GCState` struct with `sync.Mutex`; POST `/v1/gc`
|
||||
returns 202 (stub for Phase 9); GET `/v1/gc/status`; concurrent
|
||||
trigger returns 409
|
||||
|
||||
Shared admin infrastructure:
|
||||
- `server/admin.go`: `writeAdminError()` (platform `{"error":"..."}`
|
||||
format), `writeJSON()`, `RequireAdmin()` middleware, `hasRole()` helper
|
||||
- `server/admin_routes.go`: `AdminDeps` struct, `MountAdminRoutes()`
|
||||
mounts all `/v1/*` endpoints with proper auth/admin middleware layers
|
||||
|
||||
**Verification:**
|
||||
- `make all` passes: vet clean, lint 0 issues, 168 tests passing,
|
||||
all 3 binaries built
|
||||
- Phase 5 (34 new tests): parseOCIPath (14 subtests covering simple/
|
||||
multi-segment/edge cases), manifest GET by tag/digest + HEAD + not
|
||||
found, blob GET/HEAD + not in repo + repo not found, tags list +
|
||||
pagination + empty + repo not found, catalog list + pagination + empty,
|
||||
DB repository methods (15 tests covering all 6 query methods)
|
||||
- Phase 8 (51 new tests): DB admin methods (19 tests covering CRUD,
|
||||
pagination, cascade, not-found), admin auth (login ok/fail, health,
|
||||
logout), admin repos (list/detail/delete/non-admin 403), admin policy
|
||||
(full CRUD cycle, validation errors, non-admin 403, engine reload),
|
||||
admin audit (list with filters, pagination), admin GC (trigger 202,
|
||||
status, concurrent 409), RequireAdmin middleware (allowed/denied/
|
||||
no claims)
|
||||
|
||||
---
|
||||
|
||||
### 2026-03-19 — Phase 4: Policy engine
|
||||
|
||||
**Task:** Implement the registry-specific authorization engine with
|
||||
|
||||
Reference in New Issue
Block a user