eng-pad-server/RUNBOOK.md

RUNBOOK.md — eng-pad-server

1. Service Overview

eng-pad-server receives engineering notebook data from the Engineering Pad Android app via gRPC, stores it in SQLite, and serves read-only views through a web UI. Single authenticated user.

Ports: 8443 (REST/HTTPS), 9443 (gRPC/TLS), 8080 (Web UI) Data: /srv/eng-pad-server/ Config: /srv/eng-pad-server/eng-pad-server.toml Binary: /usr/local/bin/eng-pad-server

2. Health Checks

1. Check service is running:

   systemctl status eng-pad-server
   

2. Check database health:

   eng-pad-server status -c /srv/eng-pad-server/eng-pad-server.toml
   

3. Check web UI responds:

   curl -k https://localhost:8443/login
   

4. Check gRPC responds:

   grpcurl -insecure localhost:9443 list
   

3. Common Operations

Start / Stop / Restart

systemctl start eng-pad-server
systemctl stop eng-pad-server
systemctl restart eng-pad-server

View Logs

journalctl -u eng-pad-server -f

Manual Backup

eng-pad-server snapshot -c /srv/eng-pad-server/eng-pad-server.toml

Backup saved to /srv/eng-pad-server/backups/.

Check Backup Timer

systemctl list-timers eng-pad-server-backup.timer

Initialize (First Time)

1. Install the binary and config:

   sudo deploy/scripts/install.sh
   

2. Edit the config file:

   sudo -u engpad vi /srv/eng-pad-server/eng-pad-server.toml
   

3. Generate TLS certificates (or copy existing ones):

   # Self-signed for development:
   openssl req -x509 -newkey ec -pkeyopt ec_paramgen_curve:prime256v1 \
     -keyout /srv/eng-pad-server/certs/key.pem \
     -out /srv/eng-pad-server/certs/cert.pem \
     -days 3650 -nodes -subj '/CN=pad.metacircular.net'
   chown engpad:engpad /srv/eng-pad-server/certs/*.pem
   chmod 600 /srv/eng-pad-server/certs/key.pem
   

4. Create the admin user:

   eng-pad-server init -c /srv/eng-pad-server/eng-pad-server.toml
   

5. Start the service:

   systemctl enable --now eng-pad-server
   systemctl enable --now eng-pad-server-backup.timer
   

Register a FIDO2/U2F Security Key

1. Log in to the web UI with password.
2. Navigate to /keys.
3. Enter a name for the key (e.g., "YubiKey 5").
4. Click "Register" and touch the key when prompted.

Docker Deployment

cd deploy/docker
docker compose up -d

First-time setup inside the container:

docker compose exec eng-pad-server eng-pad-server init -c /srv/eng-pad-server/eng-pad-server.toml

4. Alerting

No automated alerting is configured. Monitor via:

• systemctl status eng-pad-server — process health
• journalctl -u eng-pad-server --since "1 hour ago" | grep ERROR — errors
• Backup age: ls -lt /srv/eng-pad-server/backups/ | head

5. Incident Procedures

Service Won't Start

1. Check logs:

   journalctl -u eng-pad-server -n 50 --no-pager
   

2. Common causes:
   • Config file missing or invalid → fix config
   • TLS cert/key missing → regenerate or copy
   • Port already in use → ss -tlnp | grep 8443
   • Database locked → check for zombie processes: fuser /srv/eng-pad-server/eng-pad-server.db

Database Corruption

1. Stop the service:

   systemctl stop eng-pad-server
   

2. Check integrity:

   sqlite3 /srv/eng-pad-server/eng-pad-server.db "PRAGMA integrity_check"
   

3. If corrupted, restore from backup:

   cp /srv/eng-pad-server/backups/eng-pad-server-LATEST.db /srv/eng-pad-server/eng-pad-server.db
   chown engpad:engpad /srv/eng-pad-server/eng-pad-server.db
   

4. Restart:

   systemctl start eng-pad-server
   

Certificate Expiry

1. Check expiry:

   openssl x509 -in /srv/eng-pad-server/certs/cert.pem -noout -dates
   

2. Regenerate or renew the certificate.
3. Restart the service (picks up new certs on start).

Disk Full

1. Check disk usage:

   df -h /srv/eng-pad-server/
   du -sh /srv/eng-pad-server/*
   

2. Prune old backups:

   ls -t /srv/eng-pad-server/backups/ | tail -n +8 | xargs -I{} rm /srv/eng-pad-server/backups/{}
   

3. Compact the database:

   sqlite3 /srv/eng-pad-server/eng-pad-server.db "VACUUM"
   

Sync Fails from Android App

1. Verify server is reachable from the device's network.
2. Check gRPC port is open: ss -tlnp | grep 9443
3. Check TLS cert is valid and trusted by the device.
4. Check credentials: verify the user exists via eng-pad-server status.
5. Check server logs for auth failures: journalctl -u eng-pad-server | grep UNAUTHENTICATED

6. Escalation

If the runbook doesn't resolve the issue:

1. Check ARCHITECTURE.md for system design context.
2. Check AUDIT.md for known security considerations.
3. Review recent commits for changes that may have introduced the issue.