Files

Kyle Isom 128f5abc4d Update engine specs, audit doc, and server tests for SSH CA, transit, and user engines

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

2026-03-16 20:16:23 -07:00

23 KiB

Raw Blame History

User Engine Implementation Plan

Overview

The user engine provides end-to-end encryption between Metacircular platform users. Each user has a key pair managed by Metacrypt; the service handles key exchange, encryption, and decryption so that messages/data are encrypted at rest and only readable by the intended recipients.

The design uses hybrid encryption: an asymmetric key pair per user for key exchange, combined with symmetric encryption for message payloads. This enables multi-recipient encryption without sharing symmetric keys directly.

Engine Type

user — registered constant already exists in internal/engine/engine.go.

Mount Configuration

Passed as config at mount time:

Field	Default	Description
`key_algorithm`	`"x25519"`	Key exchange algorithm: x25519, ecdh-p256, ecdh-p384
`sym_algorithm`	`"aes256-gcm"`	Symmetric algorithm for message encryption

Trust Model

Server-trust: Metacrypt holds all private keys in the barrier, same as every other engine. Access control is enforced at the application layer — no API surface exports private keys, and only the engine accesses them internally during encrypt/decrypt operations. An operator with barrier access could theoretically extract keys, which is accepted and consistent with the barrier trust model used throughout Metacrypt.

Core Concepts

User Provisioning

Any MCIAS user can have a keypair, whether or not they have ever logged in to Metacrypt. Keypairs are created in three ways:

Self-registration — an authenticated user calls register.
Admin provisioning — an admin calls provision with a username. The user does not need to have logged in.
Auto-provisioning on encrypt — when a sender encrypts to a recipient who has no keypair, the engine generates one automatically. The recipient can decrypt when they eventually authenticate.

User Key Pairs

Each provisioned user has a key exchange key pair. The private key is stored encrypted in the barrier and is only used by the engine on behalf of the owning user (enforced in HandleRequest). The public key is available to any authenticated user (needed to encrypt messages to that user).

Key generation by algorithm:

x25519: ecdh.X25519().GenerateKey(rand.Reader) (Go 1.20+ crypto/ecdh).
ecdh-p256: ecdh.P256().GenerateKey(rand.Reader).
ecdh-p384: ecdh.P384().GenerateKey(rand.Reader).

Key serialization in barrier:

Private key: x509.MarshalPKCS8PrivateKey(privKey) → PEM block with type "PRIVATE KEY". Stored at {mountPath}users/{username}/priv.pem.
Public key: x509.MarshalPKIXPublicKey(pubKey) → PEM block with type "PUBLIC KEY". Stored at {mountPath}users/{username}/pub.pem.

Note: crypto/ecdh keys implement the interfaces required by x509.MarshalPKCS8PrivateKey and x509.MarshalPKIXPublicKey as of Go 1.20.

Cryptographic Details

ECDH key agreement:

sharedSecret, err := senderPrivKey.ECDH(recipientPubKey)

The raw shared secret is never used directly as a key. It is always fed through HKDF.

HKDF key derivation:

salt := make([]byte, 32)
rand.Read(salt)
hkdf := hkdf.New(sha256.New, sharedSecret, salt, info)
wrappingKey := make([]byte, 32) // 256-bit AES key
io.ReadFull(hkdf, wrappingKey)

Hash: SHA-256 (sufficient for 256-bit key derivation).
Salt: 32 bytes of crypto/rand randomness, generated fresh per recipient per encryption. The salt is stored alongside the wrapped DEK in the envelope (see updated envelope format below).
Info: "metacrypt-user-v1:" + sender + ":" + recipient (UTF-8 encoded). This binds the derived key to the specific sender-recipient pair, preventing key confusion if the same shared secret were somehow reused.

DEK wrapping: The wrapping key from HKDF encrypts the DEK using AES-256-GCM (not AES Key Wrap / RFC 3394). AES-GCM is used because:

It is already a core primitive in the codebase.
It provides authenticated encryption, same as AES Key Wrap.
The DEK is 32 bytes — well within GCM's plaintext size limits.

block, _ := aes.NewCipher(wrappingKey)
gcm, _ := cipher.NewGCM(block)
nonce := make([]byte, gcm.NonceSize()) // 12 bytes
rand.Read(nonce)
wrappedDEK := gcm.Seal(nonce, nonce, dek, nil) // nonce || ciphertext || tag

Symmetric encryption (payload):

block, _ := aes.NewCipher(dek)
gcm, _ := cipher.NewGCM(block)
nonce := make([]byte, gcm.NonceSize()) // 12 bytes
rand.Read(nonce)
ciphertext := gcm.Seal(nonce, nonce, plaintext, aad)

AAD: if metadata is provided, it is used as additional authenticated data. This means metadata is integrity-protected but not encrypted.
Nonce: 12 bytes from crypto/rand.

Encryption Flow (Sender → Recipient)

Sender calls encrypt with plaintext, recipient username(s), and optional metadata.
Engine generates a random symmetric data encryption key (DEK).
Engine encrypts the plaintext with the DEK.
For each recipient: engine performs key agreement (sender private key + recipient public key → shared secret), derives a wrapping key via HKDF, and wraps the DEK.
Returns an envelope containing ciphertext + per-recipient wrapped DEKs.

Decryption Flow

Recipient calls decrypt with the envelope.
Engine finds the recipient's wrapped DEK entry.
Engine performs key agreement (recipient private key + sender public key → shared secret), derives the wrapping key, unwraps the DEK.
Engine decrypts the ciphertext with the DEK.
Returns plaintext.

Envelope Format

{
    "version": 1,
    "sender": "alice",
    "sym_algorithm": "aes256-gcm",
    "ciphertext": "<base64(nonce + encrypted_payload + tag)>",
    "metadata": "<optional plaintext metadata>",
    "recipients": {
        "bob": {
            "salt": "<base64(32-byte HKDF salt)>",
            "wrapped_dek": "<base64(nonce + encrypted_dek + tag)>"
        },
        "carol": {
            "salt": "<base64(32-byte HKDF salt)>",
            "wrapped_dek": "<base64(nonce + encrypted_dek + tag)>"
        }
    }
}

Each recipient entry includes:

salt: the per-recipient random HKDF salt used during key derivation.
wrapped_dek: the AES-256-GCM encryption of the DEK (nonce-prepended).

The envelope is base64-encoded as a single opaque blob for transport.

Barrier Storage Layout

engine/user/{mount}/config.json                     Engine configuration
engine/user/{mount}/users/{username}/priv.pem        User private key
engine/user/{mount}/users/{username}/pub.pem         User public key
engine/user/{mount}/users/{username}/config.json     Per-user metadata

In-Memory State

type UserEngine struct {
    barrier   barrier.Barrier
    config    *UserConfig
    users     map[string]*userState
    mountPath string
    mu        sync.RWMutex
}

type userState struct {
    privKey crypto.PrivateKey  // key exchange private key
    pubKey  crypto.PublicKey   // key exchange public key
    config  *UserKeyConfig
}

type UserKeyConfig struct {
    Algorithm       string    `json:"algorithm"`         // key exchange algorithm (x25519, ecdh-p256, ecdh-p384)
    CreatedAt       time.Time `json:"created_at"`
    AutoProvisioned bool      `json:"auto_provisioned"`  // true if created via auto-provisioning
}

Lifecycle

Initialize

Parse and validate config: ensure key_algorithm is one of x25519, ecdh-p256, ecdh-p384. Ensure sym_algorithm is aes256-gcm.

Store config in barrier as {mountPath}config.json:

configJSON, _ := json.Marshal(config)
barrier.Put(ctx, mountPath+"config.json", configJSON)

No user keys are created at init time (created on demand via register, provision, or auto-provisioning).

Unseal

Load config JSON from barrier, unmarshal into *UserConfig.
List all user directories under {mountPath}users/.
For each user, load priv.pem and pub.pem:
- Parse private key PEM: pem.Decode → x509.ParsePKCS8PrivateKey → type-assert to *ecdh.PrivateKey.
- Parse public key PEM: pem.Decode → x509.ParsePKIXPublicKey → type-assert to *ecdh.PublicKey.
Populate users map with loaded key states.

Seal

Zeroize all private key material using engine.ZeroizeKey(privKey).
Nil out users map and config.

Operations

Operation	Auth Required	Description
`register`	User (self)	Create a key pair for the authenticated user
`provision`	Admin	Create a key pair for any MCIAS user by username
`get-public-key`	User/Admin	Get any user's public key
`list-users`	User/Admin	List registered users
`encrypt`	User+Policy	Encrypt data for one or more recipients
`decrypt`	User (self)	Decrypt an envelope addressed to the caller
`re-encrypt`	User (self)	Re-encrypt an envelope with current key pairs
`rotate-key`	User (self)	Rotate the caller's key pair
`delete-user`	Admin	Remove a user's key pair

HandleRequest dispatch

Follow the CA engine's pattern (internal/engine/ca/ca.go:284-317):

func (e *UserEngine) HandleRequest(ctx context.Context, req *engine.Request) (*engine.Response, error) {
    switch req.Operation {
    case "register":
        return e.handleRegister(ctx, req)
    case "provision":
        return e.handleProvision(ctx, req)
    case "get-public-key":
        return e.handleGetPublicKey(ctx, req)
    case "list-users":
        return e.handleListUsers(ctx, req)
    case "encrypt":
        return e.handleEncrypt(ctx, req)
    case "decrypt":
        return e.handleDecrypt(ctx, req)
    case "re-encrypt":
        return e.handleReEncrypt(ctx, req)
    case "rotate-key":
        return e.handleRotateKey(ctx, req)
    case "delete-user":
        return e.handleDeleteUser(ctx, req)
    default:
        return nil, fmt.Errorf("user: unknown operation: %s", req.Operation)
    }
}

register

Creates a key pair for the authenticated caller. No-op if the caller already has a keypair (returns existing public key).

Request data: none (uses CallerInfo.Username).

Response: { "public_key": "<base64>" }

provision

Admin-only. Creates a key pair for the given username. The user does not need to have logged in to Metacrypt. No-op if the user already has a keypair.

Request data: { "username": "<mcias_username>" }

Response: { "public_key": "<base64>" }

encrypt

Request data:

Field	Required	Description
`recipients`	Yes	List of usernames to encrypt for
`plaintext`	Yes	Base64-encoded plaintext
`metadata`	No	Arbitrary string metadata (authenticated)

Flow:

Validate that len(recipients) <= maxRecipients (100). Reject with 400 Bad Request if exceeded.
Caller must be provisioned (has a key pair). If not, auto-provision the caller (generate keypair, store in barrier). This is safe because the caller is already authenticated via MCIAS — their identity is verified.
For each recipient without a keypair: validate the username exists in MCIAS via auth.ValidateUsername(username). If the user does not exist, return an error: "recipient not found: {username}". If the user exists, auto-provision them. Auto-provisioning only creates a key pair; it does not grant any MCIAS roles or permissions. The recipient's private key is only accessible when they authenticate.
Load sender's private key and each recipient's public key.
Generate random 32-byte DEK (crypto/rand). Encrypt plaintext with DEK using AES-256-GCM (metadata as AAD if present).
For each recipient:
- sharedSecret := senderPrivKey.ECDH(recipientPubKey)
- Generate 32-byte random salt.
- wrappingKey := HKDF(sha256, sharedSecret, salt, info)
- wrappedDEK := AES-GCM-Encrypt(wrappingKey, DEK)
Build envelope with ciphertext, per-recipient {salt, wrapped_dek}.
Zeroize DEK, all shared secrets, and all wrapping keys.

Authorization:

Admins: grant-all.
Users: can encrypt to any registered user by default.
Policy can restrict which users a sender can encrypt to: resource user/{mount}/recipient/{username}, action write.

decrypt

Request data:

Field	Required	Description
`envelope`	Yes	Base64-encoded envelope blob

Flow:

Parse envelope JSON, find the caller's entry in recipients. If the caller is not a recipient, return an error.
Load sender's public key (from envelope.sender) and caller's private key.
sharedSecret := callerPrivKey.ECDH(senderPubKey).
wrappingKey := HKDF(sha256, sharedSecret, recipient.salt, info).
dek := AES-GCM-Decrypt(wrappingKey, recipient.wrapped_dek).
Decrypt ciphertext with DEK (metadata as AAD if present in envelope).
Zeroize DEK, shared secret, wrapping key.
Return plaintext.

A user can only decrypt envelopes addressed to themselves.

re-encrypt

Re-encrypts an envelope with the caller's current key pair. This is the safe way to migrate data before a key rotation.

Request data:

Field	Required	Description
`envelope`	Yes	Base64-encoded envelope blob

Flow:

Decrypt the envelope (same as decrypt flow).
Re-encrypt the plaintext for the same recipients using fresh DEKs and current key pairs (same as encrypt flow, preserving metadata).
Return the new envelope.

The caller must be a recipient in the original envelope. The new envelope uses current key pairs for all recipients — if any recipient has rotated their key since the original encryption, the new envelope uses their new public key.

rotate-key

Generates a new key pair for the caller. The old private key is zeroized and deleted. Old envelopes encrypted with the previous key cannot be decrypted after rotation.

Recommended workflow: Before rotating, re-encrypt all stored envelopes using the re-encrypt operation. Then call rotate-key. This ensures no data is lost.

gRPC Service (proto/metacrypt/v2/user.proto)

service UserService {
    rpc Register(UserRegisterRequest)           returns (UserRegisterResponse);
    rpc Provision(UserProvisionRequest)         returns (UserProvisionResponse);
    rpc GetPublicKey(UserGetPublicKeyRequest)    returns (UserGetPublicKeyResponse);
    rpc ListUsers(UserListUsersRequest)          returns (UserListUsersResponse);
    rpc Encrypt(UserEncryptRequest)              returns (UserEncryptResponse);
    rpc Decrypt(UserDecryptRequest)              returns (UserDecryptResponse);
    rpc ReEncrypt(UserReEncryptRequest)          returns (UserReEncryptResponse);
    rpc RotateKey(UserRotateKeyRequest)          returns (UserRotateKeyResponse);
    rpc DeleteUser(UserDeleteUserRequest)        returns (UserDeleteUserResponse);
}

REST Endpoints

All auth required:

Method	Path	Description
POST	`/v1/user/{mount}/register`	Register caller
POST	`/v1/user/{mount}/provision`	Provision user (admin)
GET	`/v1/user/{mount}/keys`	List registered users
GET	`/v1/user/{mount}/keys/{username}`	Get user's public key
DELETE	`/v1/user/{mount}/keys/{username}`	Delete user (admin)
POST	`/v1/user/{mount}/encrypt`	Encrypt for recipients
POST	`/v1/user/{mount}/decrypt`	Decrypt envelope
POST	`/v1/user/{mount}/re-encrypt`	Re-encrypt envelope
POST	`/v1/user/{mount}/rotate`	Rotate caller's key

All operations are also accessible via the generic POST /v1/engine/request.

REST Route Registration

Add to internal/server/routes.go in registerRoutes, following the CA engine's pattern with chi.URLParam:

// User engine routes.
r.Post("/v1/user/{mount}/register", s.requireAuth(s.handleUserRegister))
r.Post("/v1/user/{mount}/provision", s.requireAdmin(s.handleUserProvision))
r.Get("/v1/user/{mount}/keys", s.requireAuth(s.handleUserListUsers))
r.Get("/v1/user/{mount}/keys/{username}", s.requireAuth(s.handleUserGetPublicKey))
r.Delete("/v1/user/{mount}/keys/{username}", s.requireAdmin(s.handleUserDeleteUser))
r.Post("/v1/user/{mount}/encrypt", s.requireAuth(s.handleUserEncrypt))
r.Post("/v1/user/{mount}/decrypt", s.requireAuth(s.handleUserDecrypt))
r.Post("/v1/user/{mount}/re-encrypt", s.requireAuth(s.handleUserReEncrypt))
r.Post("/v1/user/{mount}/rotate", s.requireAuth(s.handleUserRotateKey))

Each handler extracts chi.URLParam(r, "mount") and optionally chi.URLParam(r, "username"), builds an engine.Request, and calls s.engines.HandleRequest(...).

gRPC Interceptor Maps

Add to sealRequiredMethods, authRequiredMethods, and adminRequiredMethods in internal/grpcserver/server.go:

// sealRequiredMethods — all user RPCs:
"/metacrypt.v2.UserService/Register":     true,
"/metacrypt.v2.UserService/Provision":    true,
"/metacrypt.v2.UserService/GetPublicKey": true,
"/metacrypt.v2.UserService/ListUsers":    true,
"/metacrypt.v2.UserService/Encrypt":      true,
"/metacrypt.v2.UserService/Decrypt":      true,
"/metacrypt.v2.UserService/ReEncrypt":    true,
"/metacrypt.v2.UserService/RotateKey":    true,
"/metacrypt.v2.UserService/DeleteUser":   true,

// authRequiredMethods — all user RPCs:
"/metacrypt.v2.UserService/Register":     true,
"/metacrypt.v2.UserService/Provision":    true,
"/metacrypt.v2.UserService/GetPublicKey": true,
"/metacrypt.v2.UserService/ListUsers":    true,
"/metacrypt.v2.UserService/Encrypt":      true,
"/metacrypt.v2.UserService/Decrypt":      true,
"/metacrypt.v2.UserService/ReEncrypt":    true,
"/metacrypt.v2.UserService/RotateKey":    true,
"/metacrypt.v2.UserService/DeleteUser":   true,

// adminRequiredMethods — admin-only user RPCs:
"/metacrypt.v2.UserService/Provision":    true,
"/metacrypt.v2.UserService/DeleteUser":   true,

The adminOnlyOperations map in routes.go already contains user entries (qualified as user:provision, user:delete-user — keys are engineType:operation to avoid cross-engine name collisions).

Web UI

Add to /dashboard the ability to mount a user engine.

Add a /user-crypto page displaying:

Registration status / register button
Public key display
Encrypt form (select recipients, enter message)
Decrypt form (paste envelope)
Key rotation button with warning

Implementation Steps

Prerequisite: engine.ZeroizeKey must exist in internal/engine/helpers.go (created as part of the SSH CA engine implementation — see engines/sshca.md step 1).
internal/engine/user/ — Implement UserEngine:
- types.go — Config types, envelope format.
- user.go — Lifecycle (Initialize, Unseal, Seal, HandleRequest).
- crypto.go — ECDH key agreement, HKDF derivation, DEK wrap/unwrap, symmetric encrypt/decrypt.
- keys.go — User registration, key rotation, deletion.
Register factory in cmd/metacrypt/main.go.
Proto definitions — proto/metacrypt/v2/user.proto, run make proto.
gRPC handlers — internal/grpcserver/user.go.
REST routes — Add to internal/server/routes.go.
Web UI — Add template + webserver routes.
Tests — Unit tests: register, encrypt/decrypt roundtrip, multi-recipient, key rotation invalidates old envelopes, re-encrypt roundtrip, authorization checks.

Dependencies

golang.org/x/crypto/hkdf (for key derivation from ECDH shared secret)
crypto/ecdh (Go 1.20+, for X25519 and NIST curve key exchange)
Standard library crypto/aes, crypto/cipher, crypto/rand, crypto/sha256, crypto/x509, encoding/pem

Security Considerations

Private keys encrypted at rest in the barrier, zeroized on seal.
DEK is random 32 bytes per-encryption; never reused.
HKDF salt is 32 bytes of crypto/rand randomness, generated fresh per recipient per encryption. Stored in the envelope alongside the wrapped DEK. A random salt ensures that even if the same sender-recipient pair encrypts multiple messages, the derived wrapping keys are unique.
HKDF info string includes sender and recipient identities to prevent key confusion attacks: info = "metacrypt-user-v1:" + sender + ":" + recipient.
DEK wrapping uses AES-256-GCM (not AES Key Wrap / RFC 3394). Both provide authenticated encryption; AES-GCM is preferred for consistency with the rest of the codebase and avoids adding a new primitive.
All intermediate secrets (shared secrets, wrapping keys, DEKs) are zeroized immediately after use using crypto.Zeroize.
Envelope includes sender identity so the recipient can derive the correct shared secret.
Key rotation is destructive — old data cannot be decrypted. The engine should warn and require explicit confirmation (admin or self only).
Server-trust model: the server holds all private keys in the barrier. No API surface exports private keys. Access control is application-enforced — the engine only uses a private key on behalf of its owner during encrypt/decrypt.
Auto-provisioning creates key pairs for unregistered recipients. Before creating a key pair, the engine validates that the recipient username exists in MCIAS via auth.ValidateUsername. This prevents barrier pollution from non-existent usernames. Auto-provisioning is safe because: (a) the recipient must be a real MCIAS user, (b) no MCIAS permissions are granted, (c) the private key is only usable after MCIAS authentication, (d) key pairs are stored identically to explicitly registered users. Auto-provisioning is only triggered by authenticated users during encrypt.
Encrypt requests are limited to 100 recipients to prevent resource exhaustion from ECDH + HKDF computation.
Metadata in the envelope is authenticated (included as additional data in AEAD) but not encrypted — it is visible to anyone holding the envelope.
Post-quantum readiness: the key_algorithm config supports future hybrid schemes (e.g. X25519 + ML-KEM). The envelope version field enables migration.
X25519 is the default algorithm because it provides 128-bit security with the smallest key size and fastest operations. NIST curves are offered for compliance contexts.

Implementation References

These existing code patterns should be followed exactly:

Pattern	Reference File	Lines
HandleRequest switch dispatch	`internal/engine/ca/ca.go`	284–317
zeroizeKey helper	`internal/engine/ca/ca.go`	1481–1498
REST route registration with chi	`internal/server/routes.go`	38–50
gRPC handler structure	`internal/grpcserver/ca.go`	full file
gRPC interceptor maps	`internal/grpcserver/server.go`	107–205
Engine factory registration	`cmd/metacrypt/server.go`	76
adminOnlyOperations map	`internal/server/routes.go`	265–285

23 KiB Raw Blame History Unescape Escape

User Engine Implementation Plan

Overview

Engine Type

Mount Configuration

Trust Model

Core Concepts

User Provisioning

User Key Pairs

Cryptographic Details

Encryption Flow (Sender → Recipient)

Decryption Flow

Envelope Format

Barrier Storage Layout

In-Memory State

Lifecycle

Initialize

Unseal

Seal

Operations

HandleRequest dispatch

register

provision

encrypt

decrypt

re-encrypt

rotate-key

gRPC Service (proto/metacrypt/v2/user.proto)

REST Endpoints

REST Route Registration

gRPC Interceptor Maps

Web UI

Implementation Steps

Dependencies

Security Considerations

Implementation References

23 KiB

Raw Blame History