docs: fleet security plan v1.3 + tailnet ACL baseline

Phase 0 findings: password auth confirmed on (straylight, dixie), ACL
default allow-all, funnel-for-all, titan stale public funnel, gitea
public edge on nginx 1.18.0. Plan: 6 pillars, 6 phases, decisions and
assumptions register inline.

docs/security-plan.md

@@ -0,0 +1,321 @@
+# Fleet Security Plan
+
+Status: DRAFT v1.3 (2026-11-01). Author: straylight. Owner: Kyle.
+Scope: the tailnet (`scylla-hammerhead.ts.net`) and everything hanging off it,
+including the public edge. Dates are out of scope; sequencing is
+capability-and-dependency with exit criteria.
+
+## Decisions (Kyle, 2026-11-01)
+
+- **LAN is trusted**: single-occupant apartment. LAN-reachable services are
+  low priority to scope down; Internet-facing and tailnet segmentation still
+  matter.
+- **gitea stays public by design**: it is the public GitHub alternative;
+  code is shared there intentionally. Plan hardens the edge, does not tunnel it.
+- **Tailscale account is MFA'd**: tied to Google account with hardware MFA.
+  Single-identity takeover risk is mitigated at the IdP layer.
+- **Plan approved; Phase 0 underway.**
+- **Device approval already required** (manual key generation or node
+  sign-in) — the silent-enrolment path is already closed.
+- **Key expiry OFF fleet-wide** (Kyle: more annoying than useful). Reopen
+  condition: after Phase 2 tagging — tagged devices are key-expiry-exempt
+  by default (Tailscale docs, verified 2026-11-01), so re-enabling ages
+  only phones/laptops, which re-auth through the already-required approval;
+  surface upcoming expiries via the Tailscale API in Phase-4 alerting.
+- **Tailscale SSH unused**; Kyle manually manages authorized_keys and
+  dislikes it → P1 target: adopt TS SSH, retire manual key files.
+- **titan's live public Funnel identified**: Kyle's old LLM-chat web UI.
+  Backend dead (502 via relay 2026-11-01) but public DNS + cert + relay
+  route remain — retire (Phase 1).
+- No tailnet audit log available on the current Tailscale plan.
+
+## Executive summary
+
+The fleet's core posture is already sound: WireGuard-encrypted tailnet as the
+only real network, LUKS on most hosts, declarative NixOS config (rebuild-from-
+flake is a genuine recovery superpower), firewall discipline that scopes
+service ports to `tailscale0` in most places, and reactive CVE hygiene
+(eMudhra CA blacklist in `configuration.nix`).
+
+The plan's job is to close four gaps, in order:
+
+1. **Unreviewed blast radius** — the tailnet ACL policy and the full
+   exposure inventory (who can reach what) have never been written down.
+2. **Cheap exposure wins** — a handful of verified open doors (SSH password
+   auth **confirmed enabled** on straylight and dixie; dated nginx on the
+   public gitea edge) that cost an afternoon to close.
+3. **Secrets lifecycle** — no secrets manager by design; `.env` files and
+   LoadCredential scattered across hosts, with no inventory or rotation
+   story. Acceptable only if *known and bounded*.
+4. **Detection** — fleet metrics (VictoriaMetrics/vmalert on orion) exist but
+   carry no security signals; today a brute-force or a new node joining the
+   tailnet is invisible.
+
+Threat model, stated plainly: single-user fleet, Kyle is the only human.
+The adversary is not an insider; it is (a) an Internet-scanner hitting the
+public edge, (b) commodity malware/phishing landing on a desktop or phone,
+(c) a supply-chain hit on a fleet service (container image, nixpkgs input,
+gitea plugin), and (d) physical loss of a portable machine. Passwordless
+`doas` fleet-wide (deliberate) means **shell on any machine ≈ root on that
+machine**, so the plan spends its effort on *access paths* (network
+reachability, credentials, edge software) rather than in-host privilege
+boundaries — that trade-off is accepted, not a finding.
+
+## Current state (verified 2026-11-01 unless tagged)
+
+### Tailnet
+
+- Tailnet `scylla-hammerhead.ts.net`, single user `isomk@`, **26 peers**.
+- Composition: 3 NixOS server-class hosts with agents (straylight, titan,
+  talos), NixOS fleet boxes in `~/src/nixos` (8 hosts incl. orion, rift, svc,
+  sk), no-agent hosts (dixie, rift-class compose boxes), **2 cloud Linux
+  hosts with public direct connections** (phobos — 95.216.41.208, svc —
+  71.19.144.164), synology NAS, homeassistant, phones/tablets (iOS, Android),
+  Apple TV, and `fs` offering exit node.
+- `hello.ts.net` from `hello@` is a foreign tagged test node — leftover demo.
+- **titan Funnel (verified live 2026-11-01):** `titan.scylla-hammerhead.ts.net`
+  is the only tailnet name in public DNS (funnel relays 208.111.34.11,
+  208.111.35.209). It was Kyle's old LLM-chat web UI; backend now 502s but
+  the public name, cert, and relay route persist — a dormant public door.
+  All other hosts NXDOMAIN publicly = no other funnels.
+- **ACL policy: confirmed default allow-all** (exported 2026-11-01, baseline
+  saved at `docs/tailscale-acl.hujson`). Three consequences: (1) every device
+  — phones, Apple TV, NAS, homeassistant — can reach every port on every
+  server (SSH, postgres, exporters, llama API); (2) the **Funnel nodeAttr is
+  granted to all members**, so any compromised device can silently open a
+  public internet endpoint to a local service — a bypass of the entire
+  "nothing public except gitea" posture; (3) default Tailscale SSH rule is
+  active (members → own devices, nonroot+root) — benign for a single-user
+  tailnet, but it means tailscaled on any Kyle-owned device answers SSH.
+- Mattermost on straylight exposed via `tailscale serve` (443 →
+  127.0.0.1:8065), **tailnet-only**. No Funnel endpoints active.
+  This is the pattern to standardize on.
+
+### Verified exposures (probed 2026-11-01)
+
+straylight (`ss` + `sshd -T` + curl):
+
+| Exposure | Evidence | Assessment |
+|---|---|---|
+| SSH :22 on `0.0.0.0`, **password auth ON** (`PasswordAuthentication yes`, `KbdInteractiveAuthentication yes`) | `sshd -T` via doas | The top cheap close; scanners brute-force this daily. Same confirmed on dixie |
+| open-webui :8080 globally | firewall `allowedTCPPorts = [443 8080]`; `curl http://192.168.88.170:8080` → 200 | LAN-only reachability, trusted LAN → **downgraded to hygiene**. Auth status [TODO] |
+| llama API :11434/:11435 on `0.0.0.0` | `ss`; firewall scopes to `tailscale0` | Bind is wide but firewall gates it — acceptable pattern |
+| node exporter :9100, vmagent :8420 on `*` | `ss` | Metrics unauthenticated on whatever the per-host firewall allows [TODO: scope per host] |
+| Syncthing :22000 TCP + :21027 UDP global | `ss`; enabled on all hosts | Global discovery — by design, but it is a mesh credential shared by every device |
+| LLMNR :5355 on `0.0.0.0` | `ss` (also on dixie) | Spoofable name resolution; LAN trusted → **downgraded to hygiene** |
+| Public gitea `git.wntrmute.dev` | HTTP 200, `nginx/1.18.0 (Ubuntu)` at 95.216.224.239 | Public **by design**; nginx is 2020-era. [ASSUMPTION: gitea VM is phobos — IP differs from phobos's direct address; verify which host] |
+
+dixie (SSH batch probe, 2026-11-01):
+
+- `PasswordAuthentication yes` — confirmed here too.
+- Listeners: postgres **:5432 on `0.0.0.0`** (repo comment says firewall
+  scopes it to `tailscale0` — config-level, not runtime-verified
+  [TODO: `nft list ruleset` — first attempt returned empty]), llama :11434 +
+  :11433 on `0.0.0.0`, tether :11435 on `*`, node exporter :9100 on `*`,
+  **:9835 on `*` [TODO: identify]**, syncthing :22000, LLMNR :5355.
+  Syncthing GUI correctly loopback-only (:8384 on 127.0.0.1).
+- No `.env` or compose files in kyle's home — services are NixOS-native;
+  secrets presumably live in flake/config. [TODO: confirm where]
+
+### Config-level facts (from `~/src/nixos`)
+
+- Base `configuration.nix`: `PermitRootLogin = "prohibit-password"`;
+  `passwordAuthentication = false` is **commented out** → upstream default
+  "yes" applies, now runtime-confirmed on two hosts.
+- `security.doas`: kyle, `noPass`, `keepEnv` — base pattern, confirmed by
+  Kyle as deliberate and fleet-wide.
+- No secrets manager (AGENTS.md: "No secrets manager — SSH keys via
+  gpg-agent"). Secrets live in compose `.env` files, `LoadCredential=`
+  units, `~/<svc>.env`, gitea deploy keys, and agent/API credentials in
+  Hermes profiles and `~/.config/sift` keys.
+- No fail2ban/crowdsec anywhere in the repo (searched).
+- LUKS on all hosts except sk and svc (AGENTS.md).
+- GUARDIAN STAR compartmentalization via separate Hermes profile — an
+  existing, deliberate compartment boundary.
+
+## Pillars
+
+### P1 — Access control
+
+Goal: every access path to every host is enumerated, and each is either
+key-only or tailnet-gated.
+
+- **Tailnet ACLs.** Single-user tailnet with phones/TV/NAS on it. Device
+  *tags* (server / workstation / mobile / iot) with ACL grants between tags
+  beats per-node rules and stays reviewable. Minimum policy worth enforcing:
+  mobile/TV/IoT cannot reach server SSH/DB ports; cloud hosts (phobos, svc)
+  reach only what they demonstrably need; synology/homeassistant reachable
+  from Kyle's devices only.
+- **SSH — adopt Tailscale SSH, retire manual authorized_keys** (Kyle's
+  stated pain). Mechanism verified against the nixos-26.05 module:
+  `services.tailscale.extraSetFlags = [ "--ssh" ]` (runs `tailscale set
+  --ssh` via `tailscaled-set.service`; the `extraUpFlags` path only fires
+  with an authKeyFile). The ACL `ssh` section already exists in the policy
+  — after Phase 2 it gates SSH by tag, replacing per-host key files with
+  the one reviewed policy. Migration order: enable --ssh per host → verify
+  a login → drop that host's authorized_keys → finally disable sshd
+  password auth everywhere regardless (confirmed `yes` on straylight +
+  dixie). Cloud hosts keeping public :22 stay on plain sshd, keys-only +
+  `AllowUsers kyle`; everything else can close LAN :22 entirely.
+- **doas.** Accepted as-is. The one soft underbelly worth mapping:
+  "gpg-agent SSH keys forwarded from a desktop" — a compromised desktop
+  with the agent socket reaches every host. [TODO: inventory which hosts'
+  `authorized_keys` accept which keys.]
+- **Agent credentials.** Hermes agents, Ornith, MCP wrappers, and sift hold
+  live API keys with file-system reach. GUARDIAN STAR profile separation is
+  the model; extend it only where a compartment is actually warranted (YAGNI
+  otherwise).
+
+### P2 — Exposure discipline
+
+Goal: one pattern for publishing a service, and a public edge that is
+current.
+
+- **Standard pattern (already mostly followed):** loopback bind +
+  `tailscale serve` for tailnet services; tailscale-IP bind for compose
+  stacks; `tailscale0`-scoped firewall holes; Funnel only for deliberate
+  public exposure. Write this into the repo as a short doc so it's policy,
+  not folklore (much of it already lives in the tailnet-service-ops skill —
+  promote the relevant parts to a repo doc).
+- **Hygiene (LAN trusted, low urgency):** scope 8080 and exporters to
+  `tailscale0` or loopback for consistency; LLMNR off unless something
+  needs it; orion already scopes exporters — extend the pattern. Identify
+  dixie's :9835.
+- **gitea edge (public by design — harden, don't hide).** It is the entire
+  public attack surface *and* the public code channel. Update the host
+  (nginx 1.18.0 → current), confirm gitea itself is current, disable public
+  sign-up, add fail2ban (or equivalent). Public *repos* are intended;
+  public *instance admin* is not.
+- **Funnel policy:** if/when used, each Funnel endpoint gets an entry in the
+  exposure inventory with an owner and a review note.
+
+### P3 — Secrets lifecycle
+
+Goal: know every secret, where it lives, and how to rotate it — before
+deciding whether to adopt a manager.
+
+- **Phase it:** (1) inventory (grep compose/env/units/profiles → one table:
+  secret, host, file, rotation path), (2) rotation runbook for the crown
+  jewels (gitea admin, Mattermost DB, cloud-host root keys, API keys in
+  agents), (3) only then decide on agenix/sops-nix. Alternatives, honestly:
+  **status quo + inventory** (zero new machinery, no rotation automation),
+  **agenix** (age-encrypted files in the nix repo, fits flake workflow, no
+  external service), **sops-nix** (same idea, more keys/MOF flexibility,
+  heavier). For a single-user fleet the flake-native age approach is the
+  natural fit, but it earns its keep only if the inventory shows sprawl
+  worth encrypting — that's what phase 1 answers.
+- **gpg-agent as SSH keystore:** single point of compromise for fleet access.
+  Verify the master key has an offline backup (paper/hardware) and that
+  subkeys are revocable without the master.
+
+### P4 — Patch & supply chain
+
+Goal: dated software is visible before it's exploited, not after.
+
+- nixpkgs pinned to `nixos-26.05` channel — add a cadence for
+  `nixos-rebuild` rollouts (the eMudhra response shows the mechanism works;
+  make it routine). The infra-security-watchdog skill already exists for
+  release/advisory watching — wire it to the services that actually run
+  here (gitea, Mattermost, Syncthing, tailscale, open-webui, llama.cpp,
+  VictoriaMetrics).
+- Container images on the compose hosts (rift Honcho stack, mattermost) are
+  hand-managed → no automatic updates; put image tags in the same watchdog
+  list.
+- gitea is both the fleet's source-of-truth and public — a compromise there
+  poisons every host on next `nixos-rebuild`/deploy-key pull. Treat it as
+  the highest-value supply-chain asset: current version, 2FA enforced,
+  signed-release awareness, admin audit log retained.
+
+### P5 — Detection
+
+Goal: the boring attacks are noticed. Build on VictoriaMetrics/vmalert
+(orion) — no new logging stack (YAGNI).
+
+- vmalert rules from existing exporters: per-host "new listening port"
+  deltas (node exporter), sustained SSH auth failures (needs a textfile or
+  journal exporter — small piece of new work [TODO: pick the cheapest
+  journal→metrics bridge]), sudden node/egress anomalies.
+- Tailscale admin audit log (node additions, ACL changes) — review cadence
+  rather than tooling.
+- Syncthing/gitea/Mattermost admin-event awareness rides the same review.
+
+### P6 — Recovery
+
+Goal: any single host is loss-tolerant and rebuildable, provably.
+
+- Strong foundation: flake rebuild + disko. The untested parts: LUKS header
+  backups, gpg master key offline backup, synology backup coverage vs. the
+  actually-irreplacable data (Honcho DBs on rift, gitea repos, notes),
+  RELIC USB drive procedure (skill exists). One drill per quarter-class
+  cadence: rebuild one non-critical host from flake + restore its data, and
+  time it.
+
+## Roadmap (dependency-ordered; exit criteria, no dates)
+
+- **Phase 0 — Exposure inventory & ACL review** *(underway)*. Tailscale ACL
+  exported ✓ (baseline in repo); console review: `ss` + firewall audit on every NixOS host
+  (straylight ✓, dixie ✓ listeners / firewall ruleset pending; titan + talos
+  queried via their agents; orion/rift via network probes); `sshd -T`
+  fleet-wide (straylight ✓, dixie ✓); secrets inventory (P3 phase 1).
+  *Exit: `docs/security-inventory.md` in this repo listing every listener,
+  every public endpoint, every secret class, each tagged verified/date.*
+- **Phase 1 — Cheap closes.** Uncomment `passwordAuthentication = false`;
+  update gitea edge (nginx + gitea current, sign-up off, fail2ban on);
+  delete `hello.ts.net` node; retire titan's stale Funnel
+  (`tailscale funnel --reset` / `tailscale serve reset` on titan, then
+  confirm public DNS goes NXDOMAIN); hygiene pass (8080/exporters/LLMNR
+  scoped, identify dixie :9835). *Exit: re-run the Phase-0 probes and the tables
+  above show all green.*
+- **Phase 2 — Segmentation.** Device tags + ACL grants in the Tailscale
+  admin console (mobile/IoT ↔ servers, cloud-host egress restrictions,
+  exit-node policy). **First, cheapest cut:** restrict the Funnel nodeAttr
+  from `autogroup:members` to tagged servers — closes the
+  any-compromised-device-can-publish-publicly hole before full segmentation. *Exit: ACL file exported into this repo for review;
+  deliberate-allow list documented.*
+- **Phase 3 — Secrets.** Rotation runbooks for crown jewels; decide
+  agenix/sops/status-quo on inventory evidence. *Exit: every jewel has a
+  tested rotation path; decision recorded here.*
+- **Phase 4 — Detection.** vmalert security rules + Tailscale audit-log
+  review cadence. *Exit: one deliberate red-team probe (e.g. wrong-key SSH
+  burst) pages/flags.*
+- **Phase 5 — Recovery drills.** LUKS header + gpg offline backups
+  verified; one host rebuilt from flake end-to-end. *Exit: timed drill
+  notes in this repo.*
+
+## Risks (top, by expected loss)
+
+1. **gitea compromise** → poison pill for every host (deploy keys +
+   rebuild). Highest severity, moderate likelihood (public, dated edge).
+   Stays public by design, so this risk is *managed* by edge currency +
+   hardening, not removed.
+2. **Desktop phishing → gpg-agent keys → fleet SSH.** doas makes each hop
+   root. Mitigated by ACLs + key inventory, not by host-internal gates.
+3. **Unrotated API keys in agents/compose .env** → quiet exfiltration
+   channels with billing and data access.
+4. **Single tailnet identity** (`isomk@`) — Tailscale account takeover is
+   fleet takeover. Mitigated: Google IdP with hardware MFA (confirmed).
+   Residual risk bounded only by ACLs — which are currently allow-all with
+   Funnel granted to every member; Phase 2 is the bound.
+5. **No detection** — everything above compounds because nothing alerts.
+
+## Assumptions register
+
+| # | Assumption | Basis | Status |
+|---|---|---|---|
+| A1 | Tailnet ACLs are default allow-all | console export 2026-11-01 | **confirmed 2026-11-01 — default policy verbatim (allow-all ACL, funnel-for-all, default SSH rule); baseline at `docs/tailscale-acl.hujson`** |
+| A2 | SSH password auth is enabled (default) on most hosts | `passwordAuthentication = false` commented in base config | **confirmed 2026-11-01 on straylight + dixie via `sshd -T`; other hosts pending** |
+| A3 | git.wntrmute.dev runs on phobos | Hetzner-range IP, but ≠ phobos's direct IP | open |
+| A4 | open-webui :8080 has some auth enabled | not probed (would need creds) | open (downgraded — LAN trusted) |
+| A5 | Exporters on other hosts are firewall-scoped like orion | orion pattern in repo; straylight's are `*`-bound | open |
+| A6 | LAN is untrusted/shared | MikroTik-style subnet; unstated | **superseded 2026-11-01 — single-occupant apartment, LAN trusted; LAN-exposure findings downgraded** |
+| A7 | Tailscale account has MFA | unstated | **confirmed 2026-11-01 — Google account, hardware MFA** |
+
+## References
+
+- Verified-exposure probes: this session, 2026-11-01 (`tailscale status`,
+  `ss -tlnp/-ulnp`, `sshd -T`, curl probes from straylight; SSH batch probe
+  of dixie).
+- `~/src/nixos`: `configuration.nix`, `hw/straylight/default.nix`,
+  `docs/metrics-plan.md`, AGENTS.md.
+- Fleet conventions: tailnet-service-ops skill (exposure patterns,
+  per-host procedures).

docs/tailscale-acl.hujson

@@ -0,0 +1,28 @@
+// Baseline tailnet policy, exported from Tailscale admin console 2026-11-01
+// by Kyle. Unmodified default policy — allow-all ACL, funnel enabled for all
+// members, default Tailscale SSH rule. Phase 2 of docs/security-plan.md
+// replaces this; keep as the pre-segmentation baseline.
+{
+	"acls": [
+		// Allow all connections.
+		{"action": "accept", "src": ["*"], "dst": ["*:*"]},
+	],
+
+	// Default Tailscale SSH: every member may SSH to their own devices,
+	// as nonroot or root.
+	"ssh": [
+		{
+			"action": "accept",
+			"src":    ["autogroup:members"],
+			"dst":    ["autogroup:self"],
+			"users":  ["autogroup:nonroot", "root"],
+		},
+	],
+	"nodeAttrs": [
+		{
+			// Any tailnet member can open a public Funnel endpoint.
+			"target": ["autogroup:members"],
+			"attr":   ["funnel"],
+		},
+	],
+}