imladris/docs/security-plan.md

# Fleet Security Plan

Status: DRAFT v1.3 (2026-11-01). Author: straylight. Owner: Kyle.
Scope: the tailnet (`scylla-hammerhead.ts.net`) and everything hanging off it,
including the public edge. Dates are out of scope; sequencing is
capability-and-dependency with exit criteria.

## Decisions (Kyle, 2026-11-01)

- **LAN is trusted**: single-occupant apartment. LAN-reachable services are
  low priority to scope down; Internet-facing and tailnet segmentation still
  matter.
- **gitea stays public by design**: it is the public GitHub alternative;
  code is shared there intentionally. Plan hardens the edge, does not tunnel it.
- **Tailscale account is MFA'd**: tied to Google account with hardware MFA.
  Single-identity takeover risk is mitigated at the IdP layer.
- **Plan approved; Phase 0 underway.**
- **Device approval already required** (manual key generation or node
  sign-in) — the silent-enrolment path is already closed.
- **Key expiry OFF fleet-wide** (Kyle: more annoying than useful). Reopen
  condition: after Phase 2 tagging — tagged devices are key-expiry-exempt
  by default (Tailscale docs, verified 2026-11-01), so re-enabling ages
  only phones/laptops, which re-auth through the already-required approval;
  surface upcoming expiries via the Tailscale API in Phase-4 alerting.
- **Tailscale SSH unused**; Kyle manually manages authorized_keys and
  dislikes it → P1 target: adopt TS SSH, retire manual key files.
- **titan's live public Funnel identified**: Kyle's old LLM-chat web UI.
  Backend dead (502 via relay 2026-11-01) but public DNS + cert + relay
  route remain — retire (Phase 1).
- No tailnet audit log available on the current Tailscale plan.

## Executive summary

The fleet's core posture is already sound: WireGuard-encrypted tailnet as the
only real network, LUKS on most hosts, declarative NixOS config (rebuild-from-
flake is a genuine recovery superpower), firewall discipline that scopes
service ports to `tailscale0` in most places, and reactive CVE hygiene
(eMudhra CA blacklist in `configuration.nix`).

The plan's job is to close four gaps, in order:

1. **Unreviewed blast radius** — the tailnet ACL policy and the full
   exposure inventory (who can reach what) have never been written down.
2. **Cheap exposure wins** — a handful of verified open doors (SSH password
   auth **confirmed enabled** on straylight and dixie; dated nginx on the
   public gitea edge) that cost an afternoon to close.
3. **Secrets lifecycle** — no secrets manager by design; `.env` files and
   LoadCredential scattered across hosts, with no inventory or rotation
   story. Acceptable only if *known and bounded*.
4. **Detection** — fleet metrics (VictoriaMetrics/vmalert on orion) exist but
   carry no security signals; today a brute-force or a new node joining the
   tailnet is invisible.

Threat model, stated plainly: single-user fleet, Kyle is the only human.
The adversary is not an insider; it is (a) an Internet-scanner hitting the
public edge, (b) commodity malware/phishing landing on a desktop or phone,
(c) a supply-chain hit on a fleet service (container image, nixpkgs input,
gitea plugin), and (d) physical loss of a portable machine. Passwordless
`doas` fleet-wide (deliberate) means **shell on any machine ≈ root on that
machine**, so the plan spends its effort on *access paths* (network
reachability, credentials, edge software) rather than in-host privilege
boundaries — that trade-off is accepted, not a finding.

## Current state (verified 2026-11-01 unless tagged)

### Tailnet

- Tailnet `scylla-hammerhead.ts.net`, single user `isomk@`, **26 peers**.
- Composition: 3 NixOS server-class hosts with agents (straylight, titan,
  talos), NixOS fleet boxes in `~/src/nixos` (8 hosts incl. orion, rift, svc,
  sk), no-agent hosts (dixie, rift-class compose boxes), **2 cloud Linux
  hosts with public direct connections** (phobos — 95.216.41.208, svc —
  71.19.144.164), synology NAS, homeassistant, phones/tablets (iOS, Android),
  Apple TV, and `fs` offering exit node.
- `hello.ts.net` from `hello@` is a foreign tagged test node — leftover demo.
- **titan Funnel (verified live 2026-11-01):** `titan.scylla-hammerhead.ts.net`
  is the only tailnet name in public DNS (funnel relays 208.111.34.11,
  208.111.35.209). It was Kyle's old LLM-chat web UI; backend now 502s but
  the public name, cert, and relay route persist — a dormant public door.
  All other hosts NXDOMAIN publicly = no other funnels.
- **ACL policy: confirmed default allow-all** (exported 2026-11-01, baseline
  saved at `docs/tailscale-acl.hujson`). Three consequences: (1) every device
  — phones, Apple TV, NAS, homeassistant — can reach every port on every
  server (SSH, postgres, exporters, llama API); (2) the **Funnel nodeAttr is
  granted to all members**, so any compromised device can silently open a
  public internet endpoint to a local service — a bypass of the entire
  "nothing public except gitea" posture; (3) default Tailscale SSH rule is
  active (members → own devices, nonroot+root) — benign for a single-user
  tailnet, but it means tailscaled on any Kyle-owned device answers SSH.
- Mattermost on straylight exposed via `tailscale serve` (443 →
  127.0.0.1:8065), **tailnet-only**. No Funnel endpoints active.
  This is the pattern to standardize on.

### Verified exposures (probed 2026-11-01)

straylight (`ss` + `sshd -T` + curl):

| Exposure | Evidence | Assessment |
|---|---|---|
| SSH :22 on `0.0.0.0`, **password auth ON** (`PasswordAuthentication yes`, `KbdInteractiveAuthentication yes`) | `sshd -T` via doas | The top cheap close; scanners brute-force this daily. Same confirmed on dixie |
| open-webui :8080 globally | firewall `allowedTCPPorts = [443 8080]`; `curl http://192.168.88.170:8080` → 200 | LAN-only reachability, trusted LAN → **downgraded to hygiene**. Auth status [TODO] |
| llama API :11434/:11435 on `0.0.0.0` | `ss`; firewall scopes to `tailscale0` | Bind is wide but firewall gates it — acceptable pattern |
| node exporter :9100, vmagent :8420 on `*` | `ss` | Metrics unauthenticated on whatever the per-host firewall allows [TODO: scope per host] |
| Syncthing :22000 TCP + :21027 UDP global | `ss`; enabled on all hosts | Global discovery — by design, but it is a mesh credential shared by every device |
| LLMNR :5355 on `0.0.0.0` | `ss` (also on dixie) | Spoofable name resolution; LAN trusted → **downgraded to hygiene** |
| Public gitea `git.wntrmute.dev` | HTTP 200, `nginx/1.18.0 (Ubuntu)` at 95.216.224.239 | Public **by design**; nginx is 2020-era. [ASSUMPTION: gitea VM is phobos — IP differs from phobos's direct address; verify which host] |

dixie (SSH batch probe, 2026-11-01):

- `PasswordAuthentication yes` — confirmed here too.
- Listeners: postgres **:5432 on `0.0.0.0`** (repo comment says firewall
  scopes it to `tailscale0` — config-level, not runtime-verified
  [TODO: `nft list ruleset` — first attempt returned empty]), llama :11434 +
  :11433 on `0.0.0.0`, tether :11435 on `*`, node exporter :9100 on `*`,
  **:9835 on `*` [TODO: identify]**, syncthing :22000, LLMNR :5355.
  Syncthing GUI correctly loopback-only (:8384 on 127.0.0.1).
- No `.env` or compose files in kyle's home — services are NixOS-native;
  secrets presumably live in flake/config. [TODO: confirm where]

### Config-level facts (from `~/src/nixos`)

- Base `configuration.nix`: `PermitRootLogin = "prohibit-password"`;
  `passwordAuthentication = false` is **commented out** → upstream default
  "yes" applies, now runtime-confirmed on two hosts.
- `security.doas`: kyle, `noPass`, `keepEnv` — base pattern, confirmed by
  Kyle as deliberate and fleet-wide.
- No secrets manager (AGENTS.md: "No secrets manager — SSH keys via
  gpg-agent"). Secrets live in compose `.env` files, `LoadCredential=`
  units, `~/<svc>.env`, gitea deploy keys, and agent/API credentials in
  Hermes profiles and `~/.config/sift` keys.
- No fail2ban/crowdsec anywhere in the repo (searched).
- LUKS on all hosts except sk and svc (AGENTS.md).
- GUARDIAN STAR compartmentalization via separate Hermes profile — an
  existing, deliberate compartment boundary.

## Pillars

### P1 — Access control

Goal: every access path to every host is enumerated, and each is either
key-only or tailnet-gated.

- **Tailnet ACLs.** Single-user tailnet with phones/TV/NAS on it. Device
  *tags* (server / workstation / mobile / iot) with ACL grants between tags
  beats per-node rules and stays reviewable. Minimum policy worth enforcing:
  mobile/TV/IoT cannot reach server SSH/DB ports; cloud hosts (phobos, svc)
  reach only what they demonstrably need; synology/homeassistant reachable
  from Kyle's devices only.
- **SSH — adopt Tailscale SSH, retire manual authorized_keys** (Kyle's
  stated pain). Mechanism verified against the nixos-26.05 module:
  `services.tailscale.extraSetFlags = [ "--ssh" ]` (runs `tailscale set
  --ssh` via `tailscaled-set.service`; the `extraUpFlags` path only fires
  with an authKeyFile). The ACL `ssh` section already exists in the policy
  — after Phase 2 it gates SSH by tag, replacing per-host key files with
  the one reviewed policy. Migration order: enable --ssh per host → verify
  a login → drop that host's authorized_keys → finally disable sshd
  password auth everywhere regardless (confirmed `yes` on straylight +
  dixie). Cloud hosts keeping public :22 stay on plain sshd, keys-only +
  `AllowUsers kyle`; everything else can close LAN :22 entirely.
- **doas.** Accepted as-is. The one soft underbelly worth mapping:
  "gpg-agent SSH keys forwarded from a desktop" — a compromised desktop
  with the agent socket reaches every host. [TODO: inventory which hosts'
  `authorized_keys` accept which keys.]
- **Agent credentials.** Hermes agents, Ornith, MCP wrappers, and sift hold
  live API keys with file-system reach. GUARDIAN STAR profile separation is
  the model; extend it only where a compartment is actually warranted (YAGNI
  otherwise).

### P2 — Exposure discipline

Goal: one pattern for publishing a service, and a public edge that is
current.

- **Standard pattern (already mostly followed):** loopback bind +
  `tailscale serve` for tailnet services; tailscale-IP bind for compose
  stacks; `tailscale0`-scoped firewall holes; Funnel only for deliberate
  public exposure. Write this into the repo as a short doc so it's policy,
  not folklore (much of it already lives in the tailnet-service-ops skill —
  promote the relevant parts to a repo doc).
- **Hygiene (LAN trusted, low urgency):** scope 8080 and exporters to
  `tailscale0` or loopback for consistency; LLMNR off unless something
  needs it; orion already scopes exporters — extend the pattern. Identify
  dixie's :9835.
- **gitea edge (public by design — harden, don't hide).** It is the entire
  public attack surface *and* the public code channel. Update the host
  (nginx 1.18.0 → current), confirm gitea itself is current, disable public
  sign-up, add fail2ban (or equivalent). Public *repos* are intended;
  public *instance admin* is not.
- **Funnel policy:** if/when used, each Funnel endpoint gets an entry in the
  exposure inventory with an owner and a review note.

### P3 — Secrets lifecycle

Goal: know every secret, where it lives, and how to rotate it — before
deciding whether to adopt a manager.

- **Phase it:** (1) inventory (grep compose/env/units/profiles → one table:
  secret, host, file, rotation path), (2) rotation runbook for the crown
  jewels (gitea admin, Mattermost DB, cloud-host root keys, API keys in
  agents), (3) only then decide on agenix/sops-nix. Alternatives, honestly:
  **status quo + inventory** (zero new machinery, no rotation automation),
  **agenix** (age-encrypted files in the nix repo, fits flake workflow, no
  external service), **sops-nix** (same idea, more keys/MOF flexibility,
  heavier). For a single-user fleet the flake-native age approach is the
  natural fit, but it earns its keep only if the inventory shows sprawl
  worth encrypting — that's what phase 1 answers.
- **gpg-agent as SSH keystore:** single point of compromise for fleet access.
  Verify the master key has an offline backup (paper/hardware) and that
  subkeys are revocable without the master.

### P4 — Patch & supply chain

Goal: dated software is visible before it's exploited, not after.

- nixpkgs pinned to `nixos-26.05` channel — add a cadence for
  `nixos-rebuild` rollouts (the eMudhra response shows the mechanism works;
  make it routine). The infra-security-watchdog skill already exists for
  release/advisory watching — wire it to the services that actually run
  here (gitea, Mattermost, Syncthing, tailscale, open-webui, llama.cpp,
  VictoriaMetrics).
- Container images on the compose hosts (rift Honcho stack, mattermost) are
  hand-managed → no automatic updates; put image tags in the same watchdog
  list.
- gitea is both the fleet's source-of-truth and public — a compromise there
  poisons every host on next `nixos-rebuild`/deploy-key pull. Treat it as
  the highest-value supply-chain asset: current version, 2FA enforced,
  signed-release awareness, admin audit log retained.

### P5 — Detection

Goal: the boring attacks are noticed. Build on VictoriaMetrics/vmalert
(orion) — no new logging stack (YAGNI).

- vmalert rules from existing exporters: per-host "new listening port"
  deltas (node exporter), sustained SSH auth failures (needs a textfile or
  journal exporter — small piece of new work [TODO: pick the cheapest
  journal→metrics bridge]), sudden node/egress anomalies.
- Tailscale admin audit log (node additions, ACL changes) — review cadence
  rather than tooling.
- Syncthing/gitea/Mattermost admin-event awareness rides the same review.

### P6 — Recovery

Goal: any single host is loss-tolerant and rebuildable, provably.

- Strong foundation: flake rebuild + disko. The untested parts: LUKS header
  backups, gpg master key offline backup, synology backup coverage vs. the
  actually-irreplacable data (Honcho DBs on rift, gitea repos, notes),
  RELIC USB drive procedure (skill exists). One drill per quarter-class
  cadence: rebuild one non-critical host from flake + restore its data, and
  time it.

## Roadmap (dependency-ordered; exit criteria, no dates)

- **Phase 0 — Exposure inventory & ACL review** *(underway)*. Tailscale ACL
  exported ✓ (baseline in repo); console review: `ss` + firewall audit on every NixOS host
  (straylight ✓, dixie ✓ listeners / firewall ruleset pending; titan + talos
  queried via their agents; orion/rift via network probes); `sshd -T`
  fleet-wide (straylight ✓, dixie ✓); secrets inventory (P3 phase 1).
  *Exit: `docs/security-inventory.md` in this repo listing every listener,
  every public endpoint, every secret class, each tagged verified/date.*
- **Phase 1 — Cheap closes.** Uncomment `passwordAuthentication = false`;
  update gitea edge (nginx + gitea current, sign-up off, fail2ban on);
  delete `hello.ts.net` node; retire titan's stale Funnel
  (`tailscale funnel --reset` / `tailscale serve reset` on titan, then
  confirm public DNS goes NXDOMAIN); hygiene pass (8080/exporters/LLMNR
  scoped, identify dixie :9835). *Exit: re-run the Phase-0 probes and the tables
  above show all green.*
- **Phase 2 — Segmentation.** Device tags + ACL grants in the Tailscale
  admin console (mobile/IoT ↔ servers, cloud-host egress restrictions,
  exit-node policy). **First, cheapest cut:** restrict the Funnel nodeAttr
  from `autogroup:members` to tagged servers — closes the
  any-compromised-device-can-publish-publicly hole before full segmentation. *Exit: ACL file exported into this repo for review;
  deliberate-allow list documented.*
- **Phase 3 — Secrets.** Rotation runbooks for crown jewels; decide
  agenix/sops/status-quo on inventory evidence. *Exit: every jewel has a
  tested rotation path; decision recorded here.*
- **Phase 4 — Detection.** vmalert security rules + Tailscale audit-log
  review cadence. *Exit: one deliberate red-team probe (e.g. wrong-key SSH
  burst) pages/flags.*
- **Phase 5 — Recovery drills.** LUKS header + gpg offline backups
  verified; one host rebuilt from flake end-to-end. *Exit: timed drill
  notes in this repo.*

## Risks (top, by expected loss)

1. **gitea compromise** → poison pill for every host (deploy keys +
   rebuild). Highest severity, moderate likelihood (public, dated edge).
   Stays public by design, so this risk is *managed* by edge currency +
   hardening, not removed.
2. **Desktop phishing → gpg-agent keys → fleet SSH.** doas makes each hop
   root. Mitigated by ACLs + key inventory, not by host-internal gates.
3. **Unrotated API keys in agents/compose .env** → quiet exfiltration
   channels with billing and data access.
4. **Single tailnet identity** (`isomk@`) — Tailscale account takeover is
   fleet takeover. Mitigated: Google IdP with hardware MFA (confirmed).
   Residual risk bounded only by ACLs — which are currently allow-all with
   Funnel granted to every member; Phase 2 is the bound.
5. **No detection** — everything above compounds because nothing alerts.

## Assumptions register

| # | Assumption | Basis | Status |
|---|---|---|---|
| A1 | Tailnet ACLs are default allow-all | console export 2026-11-01 | **confirmed 2026-11-01 — default policy verbatim (allow-all ACL, funnel-for-all, default SSH rule); baseline at `docs/tailscale-acl.hujson`** |
| A2 | SSH password auth is enabled (default) on most hosts | `passwordAuthentication = false` commented in base config | **confirmed 2026-11-01 on straylight + dixie via `sshd -T`; other hosts pending** |
| A3 | git.wntrmute.dev runs on phobos | Hetzner-range IP, but ≠ phobos's direct IP | open |
| A4 | open-webui :8080 has some auth enabled | not probed (would need creds) | open (downgraded — LAN trusted) |
| A5 | Exporters on other hosts are firewall-scoped like orion | orion pattern in repo; straylight's are `*`-bound | open |
| A6 | LAN is untrusted/shared | MikroTik-style subnet; unstated | **superseded 2026-11-01 — single-occupant apartment, LAN trusted; LAN-exposure findings downgraded** |
| A7 | Tailscale account has MFA | unstated | **confirmed 2026-11-01 — Google account, hardware MFA** |

## References

- Verified-exposure probes: this session, 2026-11-01 (`tailscale status`,
  `ss -tlnp/-ulnp`, `sshd -T`, curl probes from straylight; SSH batch probe
  of dixie).
- `~/src/nixos`: `configuration.nix`, `hw/straylight/default.nix`,
  `docs/metrics-plan.md`, AGENTS.md.
- Fleet conventions: tailnet-service-ops skill (exposure patterns,
  per-host procedures).