Implement a two-level key hierarchy: the MEK now wraps per-engine DEKs stored in a new barrier_keys table, rather than encrypting all barrier entries directly. A v2 ciphertext format (0x02) embeds the key ID so the barrier can resolve which DEK to use on decryption. v1 ciphertext remains supported for backward compatibility. Key changes: - crypto: EncryptV2/DecryptV2/ExtractKeyID for v2 ciphertext with key IDs - barrier: key registry (CreateKey, RotateKey, ListKeys, MigrateToV2, ReWrapKeys) - seal: RotateMEK re-wraps DEKs without re-encrypting data - engine: Mount auto-creates per-engine DEK - REST + gRPC: barrier/keys, barrier/rotate-mek, barrier/rotate-key, barrier/migrate - proto: BarrierService (v1 + v2) with ListKeys, RotateMEK, RotateKey, Migrate - db: migration v2 adds barrier_keys table Also includes: security audit report, CSRF protection, engine design specs (sshca, transit, user), path-bound AAD migration tool, policy engine enhancements, and ARCHITECTURE.md updates. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
468
engines/transit.md
Normal file
468
engines/transit.md
Normal file
@@ -0,0 +1,468 @@
|
||||
# Transit Engine Implementation Plan
|
||||
|
||||
## Overview
|
||||
|
||||
The transit engine provides encryption-as-a-service: applications send plaintext
|
||||
to Metacrypt and receive ciphertext (or vice versa), without ever handling raw
|
||||
encryption keys. This enables envelope encryption, key rotation, and centralized
|
||||
key management.
|
||||
|
||||
The design is inspired by HashiCorp Vault's transit secrets engine.
|
||||
|
||||
## Engine Type
|
||||
|
||||
`transit` — registered constant already exists in `internal/engine/engine.go`.
|
||||
|
||||
## Mount Configuration
|
||||
|
||||
Passed as `config` at mount time:
|
||||
|
||||
| Field | Default | Description |
|
||||
|--------------------|---------|------------------------------------------------|
|
||||
| `max_key_versions` | `0` | Maximum key versions to retain (0 = unlimited) |
|
||||
|
||||
No engine-wide key algorithm is configured; each named key specifies its own.
|
||||
|
||||
## Core Concepts
|
||||
|
||||
### Named Keys
|
||||
|
||||
The transit engine manages **named encryption keys**. Each key has:
|
||||
- A unique name (e.g. `"payments"`, `"session-tokens"`)
|
||||
- A key type (symmetric or asymmetric)
|
||||
- One or more **versions** (for key rotation)
|
||||
- Policy flags (exportable, allow-deletion)
|
||||
|
||||
### Key Types
|
||||
|
||||
| Type | Algorithm | Operations |
|
||||
|-----------------|-------------------|------------------|
|
||||
| `aes256-gcm` | AES-256-GCM | Encrypt, Decrypt |
|
||||
| `chacha20-poly` | ChaCha20-Poly1305 | Encrypt, Decrypt |
|
||||
| `ed25519` | Ed25519 | Sign, Verify |
|
||||
| `ecdsa-p256` | ECDSA P-256 | Sign, Verify |
|
||||
| `ecdsa-p384` | ECDSA P-384 | Sign, Verify |
|
||||
| `hmac-sha256` | HMAC-SHA256 | HMAC |
|
||||
| `hmac-sha512` | HMAC-SHA512 | HMAC |
|
||||
|
||||
RSA key types are intentionally excluded. The transit engine is not the right
|
||||
place for RSA — asymmetric encryption belongs in the user engine (via ECDH),
|
||||
and RSA signing offers no advantage over Ed25519/ECDSA for this use case.
|
||||
|
||||
### Key Rotation
|
||||
|
||||
Each key has a current version and may retain older versions. Encryption always
|
||||
uses the latest version. Decryption selects the version from the ciphertext
|
||||
header.
|
||||
|
||||
Each key tracks a `min_decryption_version` (default 1). Decryption requests
|
||||
for ciphertext encrypted with a version below this minimum are rejected. This
|
||||
lets operators complete a rotation cycle:
|
||||
|
||||
1. Rotate the key (creates version N+1).
|
||||
2. Rewrap all existing ciphertext to the latest version.
|
||||
3. Set `min_decryption_version` to N+1.
|
||||
4. Old key versions at or below the minimum can then be pruned via
|
||||
`max_key_versions` or `trim-key`.
|
||||
|
||||
Until `min_decryption_version` is advanced, old versions must be retained.
|
||||
|
||||
### Ciphertext Format
|
||||
|
||||
Transit ciphertexts use a versioned prefix:
|
||||
|
||||
```
|
||||
metacrypt:v{version}:{base64(nonce + ciphertext + tag)}
|
||||
```
|
||||
|
||||
The `v{version}` identifies which key version to use for decryption.
|
||||
|
||||
## Barrier Storage Layout
|
||||
|
||||
```
|
||||
engine/transit/{mount}/config.json Engine configuration
|
||||
engine/transit/{mount}/keys/{name}/config.json Key configuration + policy
|
||||
engine/transit/{mount}/keys/{name}/v{N}.key Key material for version N
|
||||
```
|
||||
|
||||
## In-Memory State
|
||||
|
||||
```go
|
||||
type TransitEngine struct {
|
||||
barrier barrier.Barrier
|
||||
config *TransitConfig
|
||||
keys map[string]*keyState // loaded named keys
|
||||
mountPath string
|
||||
mu sync.RWMutex
|
||||
}
|
||||
|
||||
type keyState struct {
|
||||
config *KeyConfig
|
||||
versions map[int]*keyVersion
|
||||
minDecryptionVersion int // reject decrypt for versions below this
|
||||
}
|
||||
|
||||
type keyVersion struct {
|
||||
version int
|
||||
key []byte // symmetric key material
|
||||
privKey crypto.PrivateKey // asymmetric private key (nil for symmetric)
|
||||
pubKey crypto.PublicKey // asymmetric public key (nil for symmetric)
|
||||
}
|
||||
```
|
||||
|
||||
## Lifecycle
|
||||
|
||||
### Initialize
|
||||
|
||||
1. Parse and store config in barrier.
|
||||
2. No keys are created at init time (keys are created on demand).
|
||||
|
||||
### Unseal
|
||||
|
||||
1. Load config from barrier.
|
||||
2. Discover and load all named keys and their versions from the barrier.
|
||||
|
||||
### Seal
|
||||
|
||||
1. Zeroize all key material (symmetric keys overwritten with zeros,
|
||||
asymmetric keys via `zeroizeKey`).
|
||||
2. Nil out all maps.
|
||||
|
||||
## Operations
|
||||
|
||||
| Operation | Auth Required | Description |
|
||||
|------------------|---------------|-----------------------------------------------|
|
||||
| `create-key` | Admin | Create a new named key |
|
||||
| `delete-key` | Admin | Delete a named key (if `allow_deletion` set) |
|
||||
| `get-key` | User/Admin | Get key metadata (no raw material) |
|
||||
| `list-keys` | User/Admin | List named keys |
|
||||
| `rotate-key` | Admin | Create a new version of a named key |
|
||||
| `update-key-config` | Admin | Update mutable key config (e.g. `min_decryption_version`) |
|
||||
| `trim-key` | Admin | Delete versions older than `min_decryption_version` |
|
||||
| `encrypt` | User+Policy | Encrypt plaintext with a named key |
|
||||
| `decrypt` | User+Policy | Decrypt ciphertext with a named key |
|
||||
| `rewrap` | User+Policy | Re-encrypt ciphertext with the latest key version |
|
||||
| `batch-encrypt` | User+Policy | Encrypt multiple plaintexts with a named key |
|
||||
| `batch-decrypt` | User+Policy | Decrypt multiple ciphertexts with a named key |
|
||||
| `batch-rewrap` | User+Policy | Re-encrypt multiple ciphertexts with latest version |
|
||||
| `sign` | User+Policy | Sign data with an asymmetric key (Ed25519, ECDSA) |
|
||||
| `verify` | User+Policy | Verify an asymmetric signature |
|
||||
| `hmac` | User+Policy | Compute HMAC with an HMAC key |
|
||||
| `get-public-key` | User/Admin | Get public key for asymmetric keys |
|
||||
|
||||
### create-key
|
||||
|
||||
Request data:
|
||||
|
||||
| Field | Required | Default | Description |
|
||||
|-------------------|----------|----------------|----------------------------------|
|
||||
| `name` | Yes | | Key name |
|
||||
| `type` | Yes | | Key type (see table above) |
|
||||
| `exportable` | No | `false` | Whether raw key material can be exported |
|
||||
| `allow_deletion` | No | `false` | Whether key can be deleted |
|
||||
|
||||
The key is created at version 1 with `min_decryption_version` = 1.
|
||||
|
||||
### encrypt
|
||||
|
||||
Request data:
|
||||
|
||||
| Field | Required | Description |
|
||||
|-------------|----------|---------------------------------------------------|
|
||||
| `key` | Yes | Named key to use |
|
||||
| `plaintext` | Yes | Base64-encoded plaintext |
|
||||
| `context` | No | Base64-encoded context for AEAD additional data |
|
||||
|
||||
Response: `{ "ciphertext": "metacrypt:v1:..." }`
|
||||
|
||||
### decrypt
|
||||
|
||||
Request data:
|
||||
|
||||
| Field | Required | Description |
|
||||
|--------------|----------|---------------------------------------------------|
|
||||
| `key` | Yes | Named key to use |
|
||||
| `ciphertext` | Yes | Ciphertext string from encrypt |
|
||||
| `context` | No | Base64-encoded context (must match encrypt context) |
|
||||
|
||||
Response: `{ "plaintext": "<base64>" }`
|
||||
|
||||
### sign
|
||||
|
||||
Asymmetric keys only (Ed25519, ECDSA). HMAC keys must use the `hmac` operation
|
||||
instead — HMAC is a MAC, not a digital signature, and does not provide
|
||||
non-repudiation.
|
||||
|
||||
Request data:
|
||||
|
||||
| Field | Required | Description |
|
||||
|-------------|----------|--------------------------------------------|
|
||||
| `key` | Yes | Named key (Ed25519 or ECDSA type) |
|
||||
| `input` | Yes | Base64-encoded data to sign |
|
||||
| `algorithm` | No | Hash algorithm (default varies by key type) |
|
||||
|
||||
The engine rejects `sign` requests for HMAC key types with an error.
|
||||
|
||||
Response: `{ "signature": "metacrypt:v1:..." }`
|
||||
|
||||
### verify
|
||||
|
||||
Asymmetric keys only. Rejects HMAC key types (use `hmac` to recompute and
|
||||
compare instead).
|
||||
|
||||
Request data:
|
||||
|
||||
| Field | Required | Description |
|
||||
|-------------|----------|--------------------------------------------|
|
||||
| `key` | Yes | Named key (Ed25519 or ECDSA type) |
|
||||
| `input` | Yes | Base64-encoded original data |
|
||||
| `signature` | Yes | Signature string from sign |
|
||||
|
||||
Response: `{ "valid": true }`
|
||||
|
||||
### update-key-config
|
||||
|
||||
Admin-only. Updates mutable key configuration fields.
|
||||
|
||||
Request data:
|
||||
|
||||
| Field | Required | Description |
|
||||
|--------------------------|----------|------------------------------------------|
|
||||
| `key` | Yes | Named key |
|
||||
| `min_decryption_version` | No | Minimum version allowed for decryption |
|
||||
|
||||
`min_decryption_version` can only be increased, never decreased. It cannot
|
||||
exceed the current version (you must always be able to decrypt with the latest).
|
||||
|
||||
### trim-key
|
||||
|
||||
Admin-only. Permanently deletes key versions older than `min_decryption_version`.
|
||||
This is irreversible — ciphertext encrypted with trimmed versions can never be
|
||||
decrypted.
|
||||
|
||||
Request data:
|
||||
|
||||
| Field | Required | Description |
|
||||
|-------|----------|-------------|
|
||||
| `key` | Yes | Named key |
|
||||
|
||||
Response: `{ "trimmed_versions": [1, 2, ...] }`
|
||||
|
||||
## Batch Operations
|
||||
|
||||
The transit engine supports batch variants of `encrypt`, `decrypt`, and
|
||||
`rewrap` for high-throughput use cases (e.g. encrypting many database fields,
|
||||
re-encrypting after key rotation). Without batch support, callers are pushed
|
||||
toward caching keys locally, defeating the purpose of transit encryption.
|
||||
|
||||
### Design
|
||||
|
||||
Each batch request targets a **single named key** with an array of items.
|
||||
Results are returned in the same order. Errors are **per-item** (partial
|
||||
success model) — a single bad ciphertext does not fail the entire batch.
|
||||
|
||||
Single-key-per-batch simplifies authorization: one policy check per batch
|
||||
request rather than per item. Callers needing multiple keys issue multiple
|
||||
batch requests.
|
||||
|
||||
### batch-encrypt
|
||||
|
||||
Request data:
|
||||
|
||||
| Field | Required | Description |
|
||||
|---------|----------|---------------------------------------------------|
|
||||
| `key` | Yes | Named key to use |
|
||||
| `items` | Yes | Array of encrypt items (see below) |
|
||||
|
||||
Each item:
|
||||
|
||||
| Field | Required | Description |
|
||||
|-------------|----------|-----------------------------------------------|
|
||||
| `plaintext` | Yes | Base64-encoded plaintext |
|
||||
| `context` | No | Base64-encoded context for AEAD additional data |
|
||||
| `reference` | No | Caller-defined reference string (echoed back) |
|
||||
|
||||
Response: `{ "results": [...] }`
|
||||
|
||||
Each result:
|
||||
|
||||
| Field | Description |
|
||||
|--------------|------------------------------------------------------|
|
||||
| `ciphertext` | `"metacrypt:v1:..."` on success, empty on error |
|
||||
| `reference` | Echoed from the request item (if provided) |
|
||||
| `error` | Error message on failure, empty on success |
|
||||
|
||||
### batch-decrypt
|
||||
|
||||
Request data:
|
||||
|
||||
| Field | Required | Description |
|
||||
|---------|----------|---------------------------------------------------|
|
||||
| `key` | Yes | Named key to use |
|
||||
| `items` | Yes | Array of decrypt items (see below) |
|
||||
|
||||
Each item:
|
||||
|
||||
| Field | Required | Description |
|
||||
|--------------|----------|-----------------------------------------------|
|
||||
| `ciphertext` | Yes | Ciphertext string from encrypt |
|
||||
| `context` | No | Base64-encoded context (must match encrypt) |
|
||||
| `reference` | No | Caller-defined reference string (echoed back) |
|
||||
|
||||
Response: `{ "results": [...] }`
|
||||
|
||||
Each result:
|
||||
|
||||
| Field | Description |
|
||||
|-------------|------------------------------------------------------|
|
||||
| `plaintext` | Base64-encoded plaintext on success, empty on error |
|
||||
| `reference` | Echoed from the request item (if provided) |
|
||||
| `error` | Error message on failure, empty on success |
|
||||
|
||||
### batch-rewrap
|
||||
|
||||
Request data:
|
||||
|
||||
| Field | Required | Description |
|
||||
|---------|----------|---------------------------------------------------|
|
||||
| `key` | Yes | Named key to use |
|
||||
| `items` | Yes | Array of rewrap items (see below) |
|
||||
|
||||
Each item:
|
||||
|
||||
| Field | Required | Description |
|
||||
|--------------|----------|-----------------------------------------------|
|
||||
| `ciphertext` | Yes | Ciphertext to re-encrypt with latest version |
|
||||
| `context` | No | Base64-encoded context (must match original) |
|
||||
| `reference` | No | Caller-defined reference string (echoed back) |
|
||||
|
||||
Response: `{ "results": [...] }`
|
||||
|
||||
Each result:
|
||||
|
||||
| Field | Description |
|
||||
|--------------|------------------------------------------------------|
|
||||
| `ciphertext` | Re-encrypted ciphertext on success, empty on error |
|
||||
| `reference` | Echoed from the request item (if provided) |
|
||||
| `error` | Error message on failure, empty on success |
|
||||
|
||||
### Implementation Notes
|
||||
|
||||
Batch operations are handled inside the transit engine's `HandleRequest` as
|
||||
three additional operation cases (`batch-encrypt`, `batch-decrypt`,
|
||||
`batch-rewrap`). No changes to the `Engine` interface are needed. The engine
|
||||
loops over items internally, loading the key once and reusing it for all items
|
||||
in the batch.
|
||||
|
||||
The `reference` field is opaque to the engine — it allows callers to correlate
|
||||
results with their source records (e.g. a database row ID) without maintaining
|
||||
positional tracking.
|
||||
|
||||
## Authorization
|
||||
|
||||
Follows the same model as the CA engine:
|
||||
- **Admins**: grant-all for all operations.
|
||||
- **Users**: can encrypt/decrypt/sign/verify/hmac if policy allows.
|
||||
- **Policy resources**: `transit/{mount}/key/{key_name}` with granular actions:
|
||||
`encrypt`, `decrypt`, `sign`, `verify`, `hmac` for cryptographic operations;
|
||||
`read` for metadata (get-key, list-keys, get-public-key); `write` for
|
||||
management (create-key, delete-key, rotate-key, update-key-config, trim-key).
|
||||
The `any` action matches all of the above (but never `admin`).
|
||||
- No ownership concept (transit keys are shared resources); access is purely
|
||||
policy-based.
|
||||
|
||||
## gRPC Service (proto/metacrypt/v2/transit.proto)
|
||||
|
||||
```protobuf
|
||||
service TransitService {
|
||||
rpc CreateKey(CreateTransitKeyRequest) returns (CreateTransitKeyResponse);
|
||||
rpc DeleteKey(DeleteTransitKeyRequest) returns (DeleteTransitKeyResponse);
|
||||
rpc GetKey(GetTransitKeyRequest) returns (GetTransitKeyResponse);
|
||||
rpc ListKeys(ListTransitKeysRequest) returns (ListTransitKeysResponse);
|
||||
rpc RotateKey(RotateTransitKeyRequest) returns (RotateTransitKeyResponse);
|
||||
rpc UpdateKeyConfig(UpdateTransitKeyConfigRequest) returns (UpdateTransitKeyConfigResponse);
|
||||
rpc TrimKey(TrimTransitKeyRequest) returns (TrimTransitKeyResponse);
|
||||
rpc Encrypt(TransitEncryptRequest) returns (TransitEncryptResponse);
|
||||
rpc Decrypt(TransitDecryptRequest) returns (TransitDecryptResponse);
|
||||
rpc Rewrap(TransitRewrapRequest) returns (TransitRewrapResponse);
|
||||
rpc BatchEncrypt(BatchTransitEncryptRequest) returns (BatchTransitEncryptResponse);
|
||||
rpc BatchDecrypt(BatchTransitDecryptRequest) returns (BatchTransitDecryptResponse);
|
||||
rpc BatchRewrap(BatchTransitRewrapRequest) returns (BatchTransitRewrapResponse);
|
||||
rpc Sign(TransitSignRequest) returns (TransitSignResponse);
|
||||
rpc Verify(TransitVerifyRequest) returns (TransitVerifyResponse);
|
||||
rpc Hmac(TransitHmacRequest) returns (TransitHmacResponse);
|
||||
rpc GetPublicKey(GetTransitPublicKeyRequest) returns (GetTransitPublicKeyResponse);
|
||||
}
|
||||
```
|
||||
|
||||
## REST Endpoints
|
||||
|
||||
All auth required:
|
||||
|
||||
| Method | Path | Description |
|
||||
|--------|---------------------------------------------|--------------------|
|
||||
| POST | `/v1/transit/{mount}/keys` | Create key |
|
||||
| GET | `/v1/transit/{mount}/keys` | List keys |
|
||||
| GET | `/v1/transit/{mount}/keys/{name}` | Get key metadata |
|
||||
| DELETE | `/v1/transit/{mount}/keys/{name}` | Delete key |
|
||||
| POST | `/v1/transit/{mount}/keys/{name}/rotate` | Rotate key |
|
||||
| PATCH | `/v1/transit/{mount}/keys/{name}/config` | Update key config |
|
||||
| POST | `/v1/transit/{mount}/keys/{name}/trim` | Trim old versions |
|
||||
| POST | `/v1/transit/{mount}/encrypt/{key}` | Encrypt |
|
||||
| POST | `/v1/transit/{mount}/decrypt/{key}` | Decrypt |
|
||||
| POST | `/v1/transit/{mount}/rewrap/{key}` | Rewrap |
|
||||
| POST | `/v1/transit/{mount}/batch/encrypt/{key}` | Batch encrypt |
|
||||
| POST | `/v1/transit/{mount}/batch/decrypt/{key}` | Batch decrypt |
|
||||
| POST | `/v1/transit/{mount}/batch/rewrap/{key}` | Batch rewrap |
|
||||
| POST | `/v1/transit/{mount}/sign/{key}` | Sign |
|
||||
| POST | `/v1/transit/{mount}/verify/{key}` | Verify |
|
||||
| POST | `/v1/transit/{mount}/hmac/{key}` | HMAC |
|
||||
|
||||
All operations are also accessible via the generic `POST /v1/engine/request`.
|
||||
|
||||
## Web UI
|
||||
|
||||
Add to `/dashboard` the ability to mount a transit engine.
|
||||
|
||||
Add a `/transit` page displaying:
|
||||
- Named key list with metadata (type, version, created, exportable)
|
||||
- Key detail view with version history
|
||||
- Encrypt/decrypt form for interactive testing
|
||||
- Key rotation button (admin)
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. **`internal/engine/transit/`** — Implement `TransitEngine`:
|
||||
- `types.go` — Config, KeyConfig, key version types.
|
||||
- `transit.go` — Lifecycle (Initialize, Unseal, Seal, HandleRequest).
|
||||
- `encrypt.go` — Encrypt/Decrypt/Rewrap operations.
|
||||
- `sign.go` — Sign/Verify/HMAC operations.
|
||||
- `keys.go` — Key management (create, delete, rotate, list, get).
|
||||
2. **Register factory** in `cmd/metacrypt/main.go`.
|
||||
3. **Proto definitions** — `proto/metacrypt/v2/transit.proto`, run `make proto`.
|
||||
4. **gRPC handlers** — `internal/grpcserver/transit.go`.
|
||||
5. **REST routes** — Add to `internal/server/routes.go`.
|
||||
6. **Web UI** — Add template + webserver routes.
|
||||
7. **Tests** — Unit tests for each operation, key rotation, rewrap correctness.
|
||||
|
||||
## Dependencies
|
||||
|
||||
- `golang.org/x/crypto/chacha20poly1305` (for ChaCha20-Poly1305 key type)
|
||||
- Standard library `crypto/aes`, `crypto/cipher`, `crypto/ecdsa`,
|
||||
`crypto/ed25519`, `crypto/hmac`, `crypto/sha256`, `crypto/sha512`
|
||||
|
||||
## Security Considerations
|
||||
|
||||
- All key material encrypted at rest in the barrier, zeroized on seal.
|
||||
- Symmetric keys generated with `crypto/rand`.
|
||||
- Ciphertext format includes version to support key rotation without data loss.
|
||||
- `exportable` flag is immutable after creation — cannot be enabled later.
|
||||
- `allow_deletion` is immutable after creation.
|
||||
- `max_key_versions` pruning only removes old versions, never the current one.
|
||||
- Rewrap operation never exposes plaintext to the caller.
|
||||
- Context (AAD) binding prevents ciphertext from being used in a different context.
|
||||
- `min_decryption_version` enforces key rotation completion: once advanced,
|
||||
old versions are unusable for decryption and can be permanently trimmed.
|
||||
- RSA key types are excluded to avoid padding scheme vulnerabilities
|
||||
(Bleichenbacher attacks on PKCS#1 v1.5). Asymmetric encryption belongs in
|
||||
the user engine; signing uses Ed25519/ECDSA.
|
||||
Reference in New Issue
Block a user