imladris/docs/security-plan.md

Fleet Security Plan

Status: DRAFT v1.3 (2026-11-01). Author: straylight. Owner: Kyle. Scope: the tailnet (scylla-hammerhead.ts.net) and everything hanging off it, including the public edge. Dates are out of scope; sequencing is capability-and-dependency with exit criteria.

Decisions (Kyle, 2026-11-01)

• LAN is trusted: single-occupant apartment. LAN-reachable services are low priority to scope down; Internet-facing and tailnet segmentation still matter.
• gitea stays public by design: it is the public GitHub alternative; code is shared there intentionally. Plan hardens the edge, does not tunnel it.
• Tailscale account is MFA'd: tied to Google account with hardware MFA. Single-identity takeover risk is mitigated at the IdP layer.
• Plan approved; Phase 0 underway.
• Device approval already required (manual key generation or node sign-in) — the silent-enrolment path is already closed.
• Key expiry OFF fleet-wide (Kyle: more annoying than useful). Reopen condition: after Phase 2 tagging — tagged devices are key-expiry-exempt by default (Tailscale docs, verified 2026-11-01), so re-enabling ages only phones/laptops, which re-auth through the already-required approval; surface upcoming expiries via the Tailscale API in Phase-4 alerting.
• Tailscale SSH unused; Kyle manually manages authorized_keys and dislikes it → P1 target: adopt TS SSH, retire manual key files.
• titan's live public Funnel identified: Kyle's old LLM-chat web UI. Backend dead (502 via relay 2026-11-01) but public DNS + cert + relay route remain — retire (Phase 1).
• No tailnet audit log available on the current Tailscale plan.

Executive summary

The fleet's core posture is already sound: WireGuard-encrypted tailnet as the only real network, LUKS on most hosts, declarative NixOS config (rebuild-from- flake is a genuine recovery superpower), firewall discipline that scopes service ports to tailscale0 in most places, and reactive CVE hygiene (eMudhra CA blacklist in configuration.nix).

The plan's job is to close four gaps, in order:

1. Unreviewed blast radius — the tailnet ACL policy and the full exposure inventory (who can reach what) have never been written down.
2. Cheap exposure wins — a handful of verified open doors (SSH password auth confirmed enabled on straylight and dixie; dated nginx on the public gitea edge) that cost an afternoon to close.
3. Secrets lifecycle — no secrets manager by design; .env files and LoadCredential scattered across hosts, with no inventory or rotation story. Acceptable only if known and bounded.
4. Detection — fleet metrics (VictoriaMetrics/vmalert on orion) exist but carry no security signals; today a brute-force or a new node joining the tailnet is invisible.

Threat model, stated plainly: single-user fleet, Kyle is the only human. The adversary is not an insider; it is (a) an Internet-scanner hitting the public edge, (b) commodity malware/phishing landing on a desktop or phone, (c) a supply-chain hit on a fleet service (container image, nixpkgs input, gitea plugin), and (d) physical loss of a portable machine. Passwordless doas fleet-wide (deliberate) means shell on any machine ≈ root on that machine, so the plan spends its effort on access paths (network reachability, credentials, edge software) rather than in-host privilege boundaries — that trade-off is accepted, not a finding.

Current state (verified 2026-11-01 unless tagged)

Tailnet

• Tailnet scylla-hammerhead.ts.net, single user isomk@, 26 peers.
• Composition: 3 NixOS server-class hosts with agents (straylight, titan, talos), NixOS fleet boxes in ~/src/nixos (8 hosts incl. orion, rift, svc, sk), no-agent hosts (dixie, rift-class compose boxes), 2 cloud Linux hosts with public direct connections (phobos — 95.216.41.208, svc — 71.19.144.164), synology NAS, homeassistant, phones/tablets (iOS, Android), Apple TV, and fs offering exit node.
• hello.ts.net from hello@ is a foreign tagged test node — leftover demo.
• titan Funnel (verified live 2026-11-01): titan.scylla-hammerhead.ts.net is the only tailnet name in public DNS (funnel relays 208.111.34.11, 208.111.35.209). It was Kyle's old LLM-chat web UI; backend now 502s but the public name, cert, and relay route persist — a dormant public door. All other hosts NXDOMAIN publicly = no other funnels.
• ACL policy: confirmed default allow-all (exported 2026-11-01, baseline saved at docs/tailscale-acl.hujson). Three consequences: (1) every device — phones, Apple TV, NAS, homeassistant — can reach every port on every server (SSH, postgres, exporters, llama API); (2) the Funnel nodeAttr is granted to all members, so any compromised device can silently open a public internet endpoint to a local service — a bypass of the entire "nothing public except gitea" posture; (3) default Tailscale SSH rule is active (members → own devices, nonroot+root) — benign for a single-user tailnet, but it means tailscaled on any Kyle-owned device answers SSH.
• Mattermost on straylight exposed via tailscale serve (443 → 127.0.0.1:8065), tailnet-only. No Funnel endpoints active. This is the pattern to standardize on.

Verified exposures (probed 2026-11-01)

straylight (ss + sshd -T + curl):

Exposure	Evidence	Assessment
SSH :22 on 0.0.0.0, password auth ON (PasswordAuthentication yes, KbdInteractiveAuthentication yes)	sshd -T via doas	The top cheap close; scanners brute-force this daily. Same confirmed on dixie
open-webui :8080 globally	firewall allowedTCPPorts = [443 8080]; curl http://192.168.88.170:8080 → 200	LAN-only reachability, trusted LAN → downgraded to hygiene. Auth status [TODO]
llama API :11434/:11435 on 0.0.0.0	ss; firewall scopes to tailscale0	Bind is wide but firewall gates it — acceptable pattern
node exporter :9100, vmagent :8420 on *	ss	Metrics unauthenticated on whatever the per-host firewall allows [TODO: scope per host]
Syncthing :22000 TCP + :21027 UDP global	ss; enabled on all hosts	Global discovery — by design, but it is a mesh credential shared by every device
LLMNR :5355 on 0.0.0.0	ss (also on dixie)	Spoofable name resolution; LAN trusted → downgraded to hygiene
Public gitea git.wntrmute.dev	HTTP 200, nginx/1.18.0 (Ubuntu) at 95.216.224.239	Public by design; nginx is 2020-era. [ASSUMPTION: gitea VM is phobos — IP differs from phobos's direct address; verify which host]

dixie (SSH batch probe, 2026-11-01):

• PasswordAuthentication yes — confirmed here too.
• Listeners: postgres :5432 on 0.0.0.0 (repo comment says firewall scopes it to tailscale0 — config-level, not runtime-verified [TODO: nft list ruleset — first attempt returned empty]), llama :11434 + :11433 on 0.0.0.0, tether :11435 on *, node exporter :9100 on *, :9835 on * [TODO: identify], syncthing :22000, LLMNR :5355. Syncthing GUI correctly loopback-only (:8384 on 127.0.0.1).
• No .env or compose files in kyle's home — services are NixOS-native; secrets presumably live in flake/config. [TODO: confirm where]

Config-level facts (from ~/src/nixos)

• Base configuration.nix: PermitRootLogin = "prohibit-password"; passwordAuthentication = false is commented out → upstream default "yes" applies, now runtime-confirmed on two hosts.
• security.doas: kyle, noPass, keepEnv — base pattern, confirmed by Kyle as deliberate and fleet-wide.
• No secrets manager (AGENTS.md: "No secrets manager — SSH keys via gpg-agent"). Secrets live in compose .env files, LoadCredential= units, ~/<svc>.env, gitea deploy keys, and agent/API credentials in Hermes profiles and ~/.config/sift keys.
• No fail2ban/crowdsec anywhere in the repo (searched).
• LUKS on all hosts except sk and svc (AGENTS.md).
• GUARDIAN STAR compartmentalization via separate Hermes profile — an existing, deliberate compartment boundary.

Pillars

P1 — Access control

Goal: every access path to every host is enumerated, and each is either key-only or tailnet-gated.

• Tailnet ACLs. Single-user tailnet with phones/TV/NAS on it. Device tags (server / workstation / mobile / iot) with ACL grants between tags beats per-node rules and stays reviewable. Minimum policy worth enforcing: mobile/TV/IoT cannot reach server SSH/DB ports; cloud hosts (phobos, svc) reach only what they demonstrably need; synology/homeassistant reachable from Kyle's devices only.
• SSH — adopt Tailscale SSH, retire manual authorized_keys (Kyle's stated pain). Mechanism verified against the nixos-26.05 module: services.tailscale.extraSetFlags = [ "--ssh" ] (runs tailscale set --ssh via tailscaled-set.service; the extraUpFlags path only fires with an authKeyFile). The ACL ssh section already exists in the policy — after Phase 2 it gates SSH by tag, replacing per-host key files with the one reviewed policy. Migration order: enable --ssh per host → verify a login → drop that host's authorized_keys → finally disable sshd password auth everywhere regardless (confirmed yes on straylight + dixie). Cloud hosts keeping public :22 stay on plain sshd, keys-only + AllowUsers kyle; everything else can close LAN :22 entirely.
• doas. Accepted as-is. The one soft underbelly worth mapping: "gpg-agent SSH keys forwarded from a desktop" — a compromised desktop with the agent socket reaches every host. [TODO: inventory which hosts' authorized_keys accept which keys.]
• Agent credentials. Hermes agents, Ornith, MCP wrappers, and sift hold live API keys with file-system reach. GUARDIAN STAR profile separation is the model; extend it only where a compartment is actually warranted (YAGNI otherwise).

P2 — Exposure discipline

Goal: one pattern for publishing a service, and a public edge that is current.

• Standard pattern (already mostly followed): loopback bind + tailscale serve for tailnet services; tailscale-IP bind for compose stacks; tailscale0-scoped firewall holes; Funnel only for deliberate public exposure. Write this into the repo as a short doc so it's policy, not folklore (much of it already lives in the tailnet-service-ops skill — promote the relevant parts to a repo doc).
• Hygiene (LAN trusted, low urgency): scope 8080 and exporters to tailscale0 or loopback for consistency; LLMNR off unless something needs it; orion already scopes exporters — extend the pattern. Identify dixie's :9835.
• gitea edge (public by design — harden, don't hide). It is the entire public attack surface and the public code channel. Update the host (nginx 1.18.0 → current), confirm gitea itself is current, disable public sign-up, add fail2ban (or equivalent). Public repos are intended; public instance admin is not.
• Funnel policy: if/when used, each Funnel endpoint gets an entry in the exposure inventory with an owner and a review note.

P3 — Secrets lifecycle

Goal: know every secret, where it lives, and how to rotate it — before deciding whether to adopt a manager.

• Phase it: (1) inventory (grep compose/env/units/profiles → one table: secret, host, file, rotation path), (2) rotation runbook for the crown jewels (gitea admin, Mattermost DB, cloud-host root keys, API keys in agents), (3) only then decide on agenix/sops-nix. Alternatives, honestly: status quo + inventory (zero new machinery, no rotation automation), agenix (age-encrypted files in the nix repo, fits flake workflow, no external service), sops-nix (same idea, more keys/MOF flexibility, heavier). For a single-user fleet the flake-native age approach is the natural fit, but it earns its keep only if the inventory shows sprawl worth encrypting — that's what phase 1 answers.
• gpg-agent as SSH keystore: single point of compromise for fleet access. Verify the master key has an offline backup (paper/hardware) and that subkeys are revocable without the master.

P4 — Patch & supply chain

Goal: dated software is visible before it's exploited, not after.

• nixpkgs pinned to nixos-26.05 channel — add a cadence for nixos-rebuild rollouts (the eMudhra response shows the mechanism works; make it routine). The infra-security-watchdog skill already exists for release/advisory watching — wire it to the services that actually run here (gitea, Mattermost, Syncthing, tailscale, open-webui, llama.cpp, VictoriaMetrics).
• Container images on the compose hosts (rift Honcho stack, mattermost) are hand-managed → no automatic updates; put image tags in the same watchdog list.
• gitea is both the fleet's source-of-truth and public — a compromise there poisons every host on next nixos-rebuild/deploy-key pull. Treat it as the highest-value supply-chain asset: current version, 2FA enforced, signed-release awareness, admin audit log retained.

P5 — Detection

Goal: the boring attacks are noticed. Build on VictoriaMetrics/vmalert (orion) — no new logging stack (YAGNI).

• vmalert rules from existing exporters: per-host "new listening port" deltas (node exporter), sustained SSH auth failures (needs a textfile or journal exporter — small piece of new work [TODO: pick the cheapest journal→metrics bridge]), sudden node/egress anomalies.
• Tailscale admin audit log (node additions, ACL changes) — review cadence rather than tooling.
• Syncthing/gitea/Mattermost admin-event awareness rides the same review.

P6 — Recovery

Goal: any single host is loss-tolerant and rebuildable, provably.

• Strong foundation: flake rebuild + disko. The untested parts: LUKS header backups, gpg master key offline backup, synology backup coverage vs. the actually-irreplacable data (Honcho DBs on rift, gitea repos, notes), RELIC USB drive procedure (skill exists). One drill per quarter-class cadence: rebuild one non-critical host from flake + restore its data, and time it.

Roadmap (dependency-ordered; exit criteria, no dates)

• Phase 0 — Exposure inventory & ACL review (underway). Tailscale ACL exported ✓ (baseline in repo); console review: ss + firewall audit on every NixOS host (straylight ✓, dixie ✓ listeners / firewall ruleset pending; titan + talos queried via their agents; orion/rift via network probes); sshd -T fleet-wide (straylight ✓, dixie ✓); secrets inventory (P3 phase 1). Exit: docs/security-inventory.md in this repo listing every listener, every public endpoint, every secret class, each tagged verified/date.
• Phase 1 — Cheap closes. Uncomment passwordAuthentication = false; update gitea edge (nginx + gitea current, sign-up off, fail2ban on); delete hello.ts.net node; retire titan's stale Funnel (tailscale funnel --reset / tailscale serve reset on titan, then confirm public DNS goes NXDOMAIN); hygiene pass (8080/exporters/LLMNR scoped, identify dixie :9835). Exit: re-run the Phase-0 probes and the tables above show all green.
• Phase 2 — Segmentation. Device tags + ACL grants in the Tailscale admin console (mobile/IoT ↔ servers, cloud-host egress restrictions, exit-node policy). First, cheapest cut: restrict the Funnel nodeAttr from autogroup:members to tagged servers — closes the any-compromised-device-can-publish-publicly hole before full segmentation. Exit: ACL file exported into this repo for review; deliberate-allow list documented.
• Phase 3 — Secrets. Rotation runbooks for crown jewels; decide agenix/sops/status-quo on inventory evidence. Exit: every jewel has a tested rotation path; decision recorded here.
• Phase 4 — Detection. vmalert security rules + Tailscale audit-log review cadence. Exit: one deliberate red-team probe (e.g. wrong-key SSH burst) pages/flags.
• Phase 5 — Recovery drills. LUKS header + gpg offline backups verified; one host rebuilt from flake end-to-end. Exit: timed drill notes in this repo.

Risks (top, by expected loss)

1. gitea compromise → poison pill for every host (deploy keys + rebuild). Highest severity, moderate likelihood (public, dated edge). Stays public by design, so this risk is managed by edge currency + hardening, not removed.
2. Desktop phishing → gpg-agent keys → fleet SSH. doas makes each hop root. Mitigated by ACLs + key inventory, not by host-internal gates.
3. Unrotated API keys in agents/compose .env → quiet exfiltration channels with billing and data access.
4. Single tailnet identity (isomk@) — Tailscale account takeover is fleet takeover. Mitigated: Google IdP with hardware MFA (confirmed). Residual risk bounded only by ACLs — which are currently allow-all with Funnel granted to every member; Phase 2 is the bound.
5. No detection — everything above compounds because nothing alerts.

Assumptions register

#	Assumption	Basis	Status
A1	Tailnet ACLs are default allow-all	console export 2026-11-01	confirmed 2026-11-01 — default policy verbatim (allow-all ACL, funnel-for-all, default SSH rule); baseline at docs/tailscale-acl.hujson
A2	SSH password auth is enabled (default) on most hosts	passwordAuthentication = false commented in base config	confirmed 2026-11-01 on straylight + dixie via sshd -T; other hosts pending
A3	git.wntrmute.dev runs on phobos	Hetzner-range IP, but ≠ phobos's direct IP	open
A4	open-webui :8080 has some auth enabled	not probed (would need creds)	open (downgraded — LAN trusted)
A5	Exporters on other hosts are firewall-scoped like orion	orion pattern in repo; straylight's are *-bound	open
A6	LAN is untrusted/shared	MikroTik-style subnet; unstated	superseded 2026-11-01 — single-occupant apartment, LAN trusted; LAN-exposure findings downgraded
A7	Tailscale account has MFA	unstated	confirmed 2026-11-01 — Google account, hardware MFA

References

• Verified-exposure probes: this session, 2026-11-01 (tailscale status, ss -tlnp/-ulnp, sshd -T, curl probes from straylight; SSH batch probe of dixie).
• ~/src/nixos: configuration.nix, hw/straylight/default.nix, docs/metrics-plan.md, AGENTS.md.
• Fleet conventions: tailnet-service-ops skill (exposure patterns, per-host procedures).