mcias/RUNBOOK.md

# MCIAS Runbook

Operational procedures for running and maintaining the MCIAS authentication
server. All required files live under `/srv/mcias`.

---

## Directory Layout

```
/srv/mcias/
  mcias.toml      — server configuration (TOML)
  server.crt      — TLS certificate (PEM)
  server.key      — TLS private key (PEM, mode 0640)
  mcias.db        — SQLite database (WAL mode creates .db-wal and .db-shm)
  env             — environment file: MCIAS_MASTER_PASSPHRASE (mode 0640)
  master.key      — optional raw AES-256 key file (mode 0640, alternative to env)
```

All files are owned by the `mcias` system user and group (`mcias:mcias`).
The directory itself is mode `0750`.

---

## Installation

Run as root from the repository root after `make build`:

```sh
sh dist/install.sh
```

This script is idempotent. It:
1. Creates the `mcias` system user and group if they do not exist.
2. Installs binaries to `/usr/local/bin/`.
3. Creates `/srv/mcias/` with correct ownership and permissions.
4. Installs the systemd service unit to `/etc/systemd/system/mcias.service`.
5. Installs example config files to `/srv/mcias/` (will not overwrite existing files).

After installation, complete the steps below before starting the service.

---

## First-Run Setup

### 1. Generate a TLS certificate

**Self-signed (personal/development use):**

```sh
openssl req -x509 -newkey ed25519 -days 3650 \
  -keyout /srv/mcias/server.key \
  -out    /srv/mcias/server.crt \
  -subj   "/CN=auth.example.com" \
  -nodes
chmod 0640 /srv/mcias/server.key
chown mcias:mcias /srv/mcias/server.key /srv/mcias/server.crt
```

**Using the `cert` tool:**

```sh
go install github.com/kisom/cert@latest

cat > /tmp/request.yaml <<EOF
subject:
  common_name: auth.example.com
hosts:
  - auth.example.com
key:
  algo: ecdsa
  size: 521
ca:
  expiry: 87600h
EOF

cert genkey -a ec -s 521 > /srv/mcias/server.key
cert selfsign -p /srv/mcias/server.key -f /tmp/request.yaml > /srv/mcias/server.crt
chmod 0640 /srv/mcias/server.key
chown mcias:mcias /srv/mcias/server.key /srv/mcias/server.crt
rm /tmp/request.yaml
```

### 2. Write the configuration file

```sh
cp /srv/mcias/mcias.conf.example /srv/mcias/mcias.toml
$EDITOR /srv/mcias/mcias.toml
chmod 0640 /srv/mcias/mcias.toml
chown mcias:mcias /srv/mcias/mcias.toml
```

Minimum required settings:

```toml
[server]
listen_addr = "0.0.0.0:8443"
tls_cert    = "/srv/mcias/server.crt"
tls_key     = "/srv/mcias/server.key"

[database]
path = "/srv/mcias/mcias.db"

[tokens]
issuer = "https://auth.example.com"

[master_key]
passphrase_env = "MCIAS_MASTER_PASSPHRASE"
```

See `dist/mcias.conf.example` for the full annotated reference.

### 3. Set the master key passphrase

```sh
cp /srv/mcias/mcias.env.example /srv/mcias/env
$EDITOR /srv/mcias/env     # set MCIAS_MASTER_PASSPHRASE to a long random value
chmod 0640 /srv/mcias/env
chown mcias:mcias /srv/mcias/env
```

Generate a strong passphrase:

```sh
openssl rand -base64 32
```

> **IMPORTANT:** Back up the passphrase to a secure offline location.
> Losing it permanently destroys access to all encrypted data in the database.

### 4. Create the first admin account

```sh
export MCIAS_MASTER_PASSPHRASE=your-passphrase

mciasdb --config /srv/mcias/mcias.toml account create \
  --username admin --type human
# note the UUID printed

mciasdb --config /srv/mcias/mcias.toml account set-password --id <UUID>
mciasdb --config /srv/mcias/mcias.toml role grant --id <UUID> --role admin
```

### 5. Enable and start the service

```sh
systemctl enable mcias
systemctl start mcias
systemctl status mcias
```

### 6. Verify

```sh
curl -k https://auth.example.com:8443/v1/health
# {"status":"ok"}
```

---

## Routine Operations

### Start / stop / restart

```sh
systemctl start mcias
systemctl stop mcias
systemctl restart mcias
```

### View logs

```sh
journalctl -u mcias -f
journalctl -u mcias --since "1 hour ago"
```

### Check service status

```sh
systemctl status mcias
```

### Reload configuration

The server reads its configuration at startup only. To apply config changes:

```sh
systemctl restart mcias
```

---

## Account Management

All account management can be done via `mciasctl` (REST API) when the server
is running, or `mciasdb` for offline/break-glass operations.

```sh
# Set env for offline tool
export MCIAS_MASTER_PASSPHRASE=your-passphrase
CONF="--config /srv/mcias/mcias.toml"

# List accounts
mciasdb $CONF account list

# Create account
mciasdb $CONF account create --username alice --type human

# Set password (prompts interactively)
mciasdb $CONF account set-password --id <UUID>

# Grant or revoke a role
mciasdb $CONF role grant  --id <UUID> --role admin
mciasdb $CONF role revoke --id <UUID> --role admin

# Disable account
mciasdb $CONF account set-status --id <UUID> --status inactive

# Delete account
mciasdb $CONF account set-status --id <UUID> --status deleted
```

---

## Token Management

```sh
CONF="--config /srv/mcias/mcias.toml"

# List active tokens for an account
mciasdb $CONF token list --id <UUID>

# Revoke a specific token by JTI
mciasdb $CONF token revoke --jti <JTI>

# Revoke all tokens for an account (e.g., suspected compromise)
mciasdb $CONF token revoke-all --id <UUID>

# Prune expired tokens from the database
mciasdb $CONF prune tokens
```

---

## Database Maintenance

### Verify schema

```sh
mciasdb --config /srv/mcias/mcias.toml schema verify
```

### Run pending migrations

```sh
mciasdb --config /srv/mcias/mcias.toml schema migrate
```

### Force schema version (break-glass)

```sh
mciasdb --config /srv/mcias/mcias.toml schema force --version N
```

Use only when `schema migrate` reports a dirty version after a failed migration.

### Backup the database

SQLite WAL mode creates three files. Back up all three atomically using the
SQLite backup API or by stopping the server first:

```sh
# Online backup (preferred — no downtime):
sqlite3 /srv/mcias/mcias.db ".backup /path/to/backup/mcias-$(date +%F).db"

# Offline backup:
systemctl stop mcias
cp /srv/mcias/mcias.db /path/to/backup/mcias-$(date +%F).db
systemctl start mcias
```

Store backups alongside a copy of the master key passphrase in a secure
offline location. A database backup without the passphrase is unrecoverable.

---

## Audit Log

```sh
CONF="--config /srv/mcias/mcias.toml"

# Show last 50 audit events
mciasdb $CONF audit tail --n 50

# Query by account
mciasdb $CONF audit query --account <UUID>

# Query by event type since a given time
mciasdb $CONF audit query --type login_failure --since 2026-01-01T00:00:00Z

# Output as JSON (for log shipping)
mciasdb $CONF audit query --json
```

---

## Upgrading

1. Build the new binaries: `make build`
2. Stop the service: `systemctl stop mcias`
3. Install new binaries: `sh dist/install.sh`
   - The script will not overwrite existing config files.
   - New example files are placed with a `.new` suffix for review.
4. Review any `.new` config files in `/srv/mcias/` and merge changes manually.
5. Run schema migrations if required:
   ```sh
   mciasdb --config /srv/mcias/mcias.toml schema migrate
   ```
6. Start the service: `systemctl start mcias`
7. Verify: `curl -k https://auth.example.com:8443/v1/health`

---

## Master Key Rotation

> This operation is not yet automated. Until a rotation command is
> implemented, rotation requires a full re-encryption of the database.
> Contact the project maintainer for the current procedure.

---

## TLS Certificate Renewal

Replace the certificate and key files, then restart the server:

```sh
# Generate or obtain new cert/key, then:
cp new-server.crt /srv/mcias/server.crt
cp new-server.key /srv/mcias/server.key
chmod 0640 /srv/mcias/server.key
chown mcias:mcias /srv/mcias/server.crt /srv/mcias/server.key
systemctl restart mcias
```

For Let's Encrypt with Certbot, add a deploy hook:

```sh
# /etc/letsencrypt/renewal-hooks/deploy/mcias.sh
#!/bin/sh
cp /etc/letsencrypt/live/auth.example.com/fullchain.pem /srv/mcias/server.crt
cp /etc/letsencrypt/live/auth.example.com/privkey.pem   /srv/mcias/server.key
chmod 0640 /srv/mcias/server.key
chown mcias:mcias /srv/mcias/server.crt /srv/mcias/server.key
systemctl restart mcias
```

---

## Docker Deployment

```sh
make docker

mkdir -p /srv/mcias
cp dist/mcias.conf.docker.example /srv/mcias/mcias.toml
$EDITOR /srv/mcias/mcias.toml

# Place TLS cert and key under /srv/mcias/
# Set ownership so uid 10001 (container mcias user) can read them.
chown -R 10001:10001 /srv/mcias

docker run -d \
  --name mcias \
  -v /srv/mcias:/srv/mcias \
  -e MCIAS_MASTER_PASSPHRASE=your-passphrase \
  -p 8443:8443 \
  -p 9443:9443 \
  --restart unless-stopped \
  mcias:latest
```

See `dist/mcias.conf.docker.example` for the full annotated Docker config.

---

## Troubleshooting

### Server fails to start: "open database"

Check that `/srv/mcias/` is writable by the `mcias` user:

```sh
ls -la /srv/mcias/
stat /srv/mcias/mcias.db   # if it already exists
```

Fix: `chown mcias:mcias /srv/mcias`

### Server fails to start: "environment variable ... is not set"

The `MCIAS_MASTER_PASSPHRASE` env var is missing. Ensure `/srv/mcias/env`
exists, is readable by the mcias user, and contains the correct variable:

```sh
grep MCIAS_MASTER_PASSPHRASE /srv/mcias/env
```

Also confirm the systemd unit loads it:

```sh
systemctl cat mcias | grep EnvironmentFile
```

### Server fails to start: "decrypt signing key"

The master key passphrase has changed or is wrong. The passphrase must match
the one used when the database was first initialized (the KDF salt is stored
in the database). Restore the correct passphrase from your offline backup.

### TLS errors in client connections

Verify the certificate is valid and covers the correct hostname:

```sh
openssl x509 -in /srv/mcias/server.crt -noout -text | grep -E "Subject|DNS"
openssl x509 -in /srv/mcias/server.crt -noout -dates
```

### Database locked / WAL not cleaning up

Check for lingering `mcias.db-wal` and `mcias.db-shm` files after an unclean
shutdown. These are safe to leave in place — SQLite will recover on next open.
Do not delete them while the server is running.

### Schema dirty after failed migration

```sh
mciasdb --config /srv/mcias/mcias.toml schema verify
mciasdb --config /srv/mcias/mcias.toml schema force --version N
mciasdb --config /srv/mcias/mcias.toml schema migrate
```

Replace `N` with the last successfully applied version number.

---

## File Permissions Reference

| Path | Mode | Owner |
|------|------|-------|
| `/srv/mcias/` | `0750` | `mcias:mcias` |
| `/srv/mcias/mcias.toml` | `0640` | `mcias:mcias` |
| `/srv/mcias/server.crt` | `0644` | `mcias:mcias` |
| `/srv/mcias/server.key` | `0640` | `mcias:mcias` |
| `/srv/mcias/mcias.db` | `0640` | `mcias:mcias` |
| `/srv/mcias/env` | `0640` | `mcias:mcias` |
| `/srv/mcias/master.key` | `0640` | `mcias:mcias` |

Verify permissions:

```sh
ls -la /srv/mcias/
```