diff --git a/docs/security-plan.md b/docs/security-plan.md new file mode 100644 index 0000000..e7ce5a4 --- /dev/null +++ b/docs/security-plan.md @@ -0,0 +1,321 @@ +# Fleet Security Plan + +Status: DRAFT v1.3 (2026-11-01). Author: straylight. Owner: Kyle. +Scope: the tailnet (`scylla-hammerhead.ts.net`) and everything hanging off it, +including the public edge. Dates are out of scope; sequencing is +capability-and-dependency with exit criteria. + +## Decisions (Kyle, 2026-11-01) + +- **LAN is trusted**: single-occupant apartment. LAN-reachable services are + low priority to scope down; Internet-facing and tailnet segmentation still + matter. +- **gitea stays public by design**: it is the public GitHub alternative; + code is shared there intentionally. Plan hardens the edge, does not tunnel it. +- **Tailscale account is MFA'd**: tied to Google account with hardware MFA. + Single-identity takeover risk is mitigated at the IdP layer. +- **Plan approved; Phase 0 underway.** +- **Device approval already required** (manual key generation or node + sign-in) — the silent-enrolment path is already closed. +- **Key expiry OFF fleet-wide** (Kyle: more annoying than useful). Reopen + condition: after Phase 2 tagging — tagged devices are key-expiry-exempt + by default (Tailscale docs, verified 2026-11-01), so re-enabling ages + only phones/laptops, which re-auth through the already-required approval; + surface upcoming expiries via the Tailscale API in Phase-4 alerting. +- **Tailscale SSH unused**; Kyle manually manages authorized_keys and + dislikes it → P1 target: adopt TS SSH, retire manual key files. +- **titan's live public Funnel identified**: Kyle's old LLM-chat web UI. + Backend dead (502 via relay 2026-11-01) but public DNS + cert + relay + route remain — retire (Phase 1). +- No tailnet audit log available on the current Tailscale plan. + +## Executive summary + +The fleet's core posture is already sound: WireGuard-encrypted tailnet as the +only real network, LUKS on most hosts, declarative NixOS config (rebuild-from- +flake is a genuine recovery superpower), firewall discipline that scopes +service ports to `tailscale0` in most places, and reactive CVE hygiene +(eMudhra CA blacklist in `configuration.nix`). + +The plan's job is to close four gaps, in order: + +1. **Unreviewed blast radius** — the tailnet ACL policy and the full + exposure inventory (who can reach what) have never been written down. +2. **Cheap exposure wins** — a handful of verified open doors (SSH password + auth **confirmed enabled** on straylight and dixie; dated nginx on the + public gitea edge) that cost an afternoon to close. +3. **Secrets lifecycle** — no secrets manager by design; `.env` files and + LoadCredential scattered across hosts, with no inventory or rotation + story. Acceptable only if *known and bounded*. +4. **Detection** — fleet metrics (VictoriaMetrics/vmalert on orion) exist but + carry no security signals; today a brute-force or a new node joining the + tailnet is invisible. + +Threat model, stated plainly: single-user fleet, Kyle is the only human. +The adversary is not an insider; it is (a) an Internet-scanner hitting the +public edge, (b) commodity malware/phishing landing on a desktop or phone, +(c) a supply-chain hit on a fleet service (container image, nixpkgs input, +gitea plugin), and (d) physical loss of a portable machine. Passwordless +`doas` fleet-wide (deliberate) means **shell on any machine ≈ root on that +machine**, so the plan spends its effort on *access paths* (network +reachability, credentials, edge software) rather than in-host privilege +boundaries — that trade-off is accepted, not a finding. + +## Current state (verified 2026-11-01 unless tagged) + +### Tailnet + +- Tailnet `scylla-hammerhead.ts.net`, single user `isomk@`, **26 peers**. +- Composition: 3 NixOS server-class hosts with agents (straylight, titan, + talos), NixOS fleet boxes in `~/src/nixos` (8 hosts incl. orion, rift, svc, + sk), no-agent hosts (dixie, rift-class compose boxes), **2 cloud Linux + hosts with public direct connections** (phobos — 95.216.41.208, svc — + 71.19.144.164), synology NAS, homeassistant, phones/tablets (iOS, Android), + Apple TV, and `fs` offering exit node. +- `hello.ts.net` from `hello@` is a foreign tagged test node — leftover demo. +- **titan Funnel (verified live 2026-11-01):** `titan.scylla-hammerhead.ts.net` + is the only tailnet name in public DNS (funnel relays 208.111.34.11, + 208.111.35.209). It was Kyle's old LLM-chat web UI; backend now 502s but + the public name, cert, and relay route persist — a dormant public door. + All other hosts NXDOMAIN publicly = no other funnels. +- **ACL policy: confirmed default allow-all** (exported 2026-11-01, baseline + saved at `docs/tailscale-acl.hujson`). Three consequences: (1) every device + — phones, Apple TV, NAS, homeassistant — can reach every port on every + server (SSH, postgres, exporters, llama API); (2) the **Funnel nodeAttr is + granted to all members**, so any compromised device can silently open a + public internet endpoint to a local service — a bypass of the entire + "nothing public except gitea" posture; (3) default Tailscale SSH rule is + active (members → own devices, nonroot+root) — benign for a single-user + tailnet, but it means tailscaled on any Kyle-owned device answers SSH. +- Mattermost on straylight exposed via `tailscale serve` (443 → + 127.0.0.1:8065), **tailnet-only**. No Funnel endpoints active. + This is the pattern to standardize on. + +### Verified exposures (probed 2026-11-01) + +straylight (`ss` + `sshd -T` + curl): + +| Exposure | Evidence | Assessment | +|---|---|---| +| SSH :22 on `0.0.0.0`, **password auth ON** (`PasswordAuthentication yes`, `KbdInteractiveAuthentication yes`) | `sshd -T` via doas | The top cheap close; scanners brute-force this daily. Same confirmed on dixie | +| open-webui :8080 globally | firewall `allowedTCPPorts = [443 8080]`; `curl http://192.168.88.170:8080` → 200 | LAN-only reachability, trusted LAN → **downgraded to hygiene**. Auth status [TODO] | +| llama API :11434/:11435 on `0.0.0.0` | `ss`; firewall scopes to `tailscale0` | Bind is wide but firewall gates it — acceptable pattern | +| node exporter :9100, vmagent :8420 on `*` | `ss` | Metrics unauthenticated on whatever the per-host firewall allows [TODO: scope per host] | +| Syncthing :22000 TCP + :21027 UDP global | `ss`; enabled on all hosts | Global discovery — by design, but it is a mesh credential shared by every device | +| LLMNR :5355 on `0.0.0.0` | `ss` (also on dixie) | Spoofable name resolution; LAN trusted → **downgraded to hygiene** | +| Public gitea `git.wntrmute.dev` | HTTP 200, `nginx/1.18.0 (Ubuntu)` at 95.216.224.239 | Public **by design**; nginx is 2020-era. [ASSUMPTION: gitea VM is phobos — IP differs from phobos's direct address; verify which host] | + +dixie (SSH batch probe, 2026-11-01): + +- `PasswordAuthentication yes` — confirmed here too. +- Listeners: postgres **:5432 on `0.0.0.0`** (repo comment says firewall + scopes it to `tailscale0` — config-level, not runtime-verified + [TODO: `nft list ruleset` — first attempt returned empty]), llama :11434 + + :11433 on `0.0.0.0`, tether :11435 on `*`, node exporter :9100 on `*`, + **:9835 on `*` [TODO: identify]**, syncthing :22000, LLMNR :5355. + Syncthing GUI correctly loopback-only (:8384 on 127.0.0.1). +- No `.env` or compose files in kyle's home — services are NixOS-native; + secrets presumably live in flake/config. [TODO: confirm where] + +### Config-level facts (from `~/src/nixos`) + +- Base `configuration.nix`: `PermitRootLogin = "prohibit-password"`; + `passwordAuthentication = false` is **commented out** → upstream default + "yes" applies, now runtime-confirmed on two hosts. +- `security.doas`: kyle, `noPass`, `keepEnv` — base pattern, confirmed by + Kyle as deliberate and fleet-wide. +- No secrets manager (AGENTS.md: "No secrets manager — SSH keys via + gpg-agent"). Secrets live in compose `.env` files, `LoadCredential=` + units, `~/.env`, gitea deploy keys, and agent/API credentials in + Hermes profiles and `~/.config/sift` keys. +- No fail2ban/crowdsec anywhere in the repo (searched). +- LUKS on all hosts except sk and svc (AGENTS.md). +- GUARDIAN STAR compartmentalization via separate Hermes profile — an + existing, deliberate compartment boundary. + +## Pillars + +### P1 — Access control + +Goal: every access path to every host is enumerated, and each is either +key-only or tailnet-gated. + +- **Tailnet ACLs.** Single-user tailnet with phones/TV/NAS on it. Device + *tags* (server / workstation / mobile / iot) with ACL grants between tags + beats per-node rules and stays reviewable. Minimum policy worth enforcing: + mobile/TV/IoT cannot reach server SSH/DB ports; cloud hosts (phobos, svc) + reach only what they demonstrably need; synology/homeassistant reachable + from Kyle's devices only. +- **SSH — adopt Tailscale SSH, retire manual authorized_keys** (Kyle's + stated pain). Mechanism verified against the nixos-26.05 module: + `services.tailscale.extraSetFlags = [ "--ssh" ]` (runs `tailscale set + --ssh` via `tailscaled-set.service`; the `extraUpFlags` path only fires + with an authKeyFile). The ACL `ssh` section already exists in the policy + — after Phase 2 it gates SSH by tag, replacing per-host key files with + the one reviewed policy. Migration order: enable --ssh per host → verify + a login → drop that host's authorized_keys → finally disable sshd + password auth everywhere regardless (confirmed `yes` on straylight + + dixie). Cloud hosts keeping public :22 stay on plain sshd, keys-only + + `AllowUsers kyle`; everything else can close LAN :22 entirely. +- **doas.** Accepted as-is. The one soft underbelly worth mapping: + "gpg-agent SSH keys forwarded from a desktop" — a compromised desktop + with the agent socket reaches every host. [TODO: inventory which hosts' + `authorized_keys` accept which keys.] +- **Agent credentials.** Hermes agents, Ornith, MCP wrappers, and sift hold + live API keys with file-system reach. GUARDIAN STAR profile separation is + the model; extend it only where a compartment is actually warranted (YAGNI + otherwise). + +### P2 — Exposure discipline + +Goal: one pattern for publishing a service, and a public edge that is +current. + +- **Standard pattern (already mostly followed):** loopback bind + + `tailscale serve` for tailnet services; tailscale-IP bind for compose + stacks; `tailscale0`-scoped firewall holes; Funnel only for deliberate + public exposure. Write this into the repo as a short doc so it's policy, + not folklore (much of it already lives in the tailnet-service-ops skill — + promote the relevant parts to a repo doc). +- **Hygiene (LAN trusted, low urgency):** scope 8080 and exporters to + `tailscale0` or loopback for consistency; LLMNR off unless something + needs it; orion already scopes exporters — extend the pattern. Identify + dixie's :9835. +- **gitea edge (public by design — harden, don't hide).** It is the entire + public attack surface *and* the public code channel. Update the host + (nginx 1.18.0 → current), confirm gitea itself is current, disable public + sign-up, add fail2ban (or equivalent). Public *repos* are intended; + public *instance admin* is not. +- **Funnel policy:** if/when used, each Funnel endpoint gets an entry in the + exposure inventory with an owner and a review note. + +### P3 — Secrets lifecycle + +Goal: know every secret, where it lives, and how to rotate it — before +deciding whether to adopt a manager. + +- **Phase it:** (1) inventory (grep compose/env/units/profiles → one table: + secret, host, file, rotation path), (2) rotation runbook for the crown + jewels (gitea admin, Mattermost DB, cloud-host root keys, API keys in + agents), (3) only then decide on agenix/sops-nix. Alternatives, honestly: + **status quo + inventory** (zero new machinery, no rotation automation), + **agenix** (age-encrypted files in the nix repo, fits flake workflow, no + external service), **sops-nix** (same idea, more keys/MOF flexibility, + heavier). For a single-user fleet the flake-native age approach is the + natural fit, but it earns its keep only if the inventory shows sprawl + worth encrypting — that's what phase 1 answers. +- **gpg-agent as SSH keystore:** single point of compromise for fleet access. + Verify the master key has an offline backup (paper/hardware) and that + subkeys are revocable without the master. + +### P4 — Patch & supply chain + +Goal: dated software is visible before it's exploited, not after. + +- nixpkgs pinned to `nixos-26.05` channel — add a cadence for + `nixos-rebuild` rollouts (the eMudhra response shows the mechanism works; + make it routine). The infra-security-watchdog skill already exists for + release/advisory watching — wire it to the services that actually run + here (gitea, Mattermost, Syncthing, tailscale, open-webui, llama.cpp, + VictoriaMetrics). +- Container images on the compose hosts (rift Honcho stack, mattermost) are + hand-managed → no automatic updates; put image tags in the same watchdog + list. +- gitea is both the fleet's source-of-truth and public — a compromise there + poisons every host on next `nixos-rebuild`/deploy-key pull. Treat it as + the highest-value supply-chain asset: current version, 2FA enforced, + signed-release awareness, admin audit log retained. + +### P5 — Detection + +Goal: the boring attacks are noticed. Build on VictoriaMetrics/vmalert +(orion) — no new logging stack (YAGNI). + +- vmalert rules from existing exporters: per-host "new listening port" + deltas (node exporter), sustained SSH auth failures (needs a textfile or + journal exporter — small piece of new work [TODO: pick the cheapest + journal→metrics bridge]), sudden node/egress anomalies. +- Tailscale admin audit log (node additions, ACL changes) — review cadence + rather than tooling. +- Syncthing/gitea/Mattermost admin-event awareness rides the same review. + +### P6 — Recovery + +Goal: any single host is loss-tolerant and rebuildable, provably. + +- Strong foundation: flake rebuild + disko. The untested parts: LUKS header + backups, gpg master key offline backup, synology backup coverage vs. the + actually-irreplacable data (Honcho DBs on rift, gitea repos, notes), + RELIC USB drive procedure (skill exists). One drill per quarter-class + cadence: rebuild one non-critical host from flake + restore its data, and + time it. + +## Roadmap (dependency-ordered; exit criteria, no dates) + +- **Phase 0 — Exposure inventory & ACL review** *(underway)*. Tailscale ACL + exported ✓ (baseline in repo); console review: `ss` + firewall audit on every NixOS host + (straylight ✓, dixie ✓ listeners / firewall ruleset pending; titan + talos + queried via their agents; orion/rift via network probes); `sshd -T` + fleet-wide (straylight ✓, dixie ✓); secrets inventory (P3 phase 1). + *Exit: `docs/security-inventory.md` in this repo listing every listener, + every public endpoint, every secret class, each tagged verified/date.* +- **Phase 1 — Cheap closes.** Uncomment `passwordAuthentication = false`; + update gitea edge (nginx + gitea current, sign-up off, fail2ban on); + delete `hello.ts.net` node; retire titan's stale Funnel + (`tailscale funnel --reset` / `tailscale serve reset` on titan, then + confirm public DNS goes NXDOMAIN); hygiene pass (8080/exporters/LLMNR + scoped, identify dixie :9835). *Exit: re-run the Phase-0 probes and the tables + above show all green.* +- **Phase 2 — Segmentation.** Device tags + ACL grants in the Tailscale + admin console (mobile/IoT ↔ servers, cloud-host egress restrictions, + exit-node policy). **First, cheapest cut:** restrict the Funnel nodeAttr + from `autogroup:members` to tagged servers — closes the + any-compromised-device-can-publish-publicly hole before full segmentation. *Exit: ACL file exported into this repo for review; + deliberate-allow list documented.* +- **Phase 3 — Secrets.** Rotation runbooks for crown jewels; decide + agenix/sops/status-quo on inventory evidence. *Exit: every jewel has a + tested rotation path; decision recorded here.* +- **Phase 4 — Detection.** vmalert security rules + Tailscale audit-log + review cadence. *Exit: one deliberate red-team probe (e.g. wrong-key SSH + burst) pages/flags.* +- **Phase 5 — Recovery drills.** LUKS header + gpg offline backups + verified; one host rebuilt from flake end-to-end. *Exit: timed drill + notes in this repo.* + +## Risks (top, by expected loss) + +1. **gitea compromise** → poison pill for every host (deploy keys + + rebuild). Highest severity, moderate likelihood (public, dated edge). + Stays public by design, so this risk is *managed* by edge currency + + hardening, not removed. +2. **Desktop phishing → gpg-agent keys → fleet SSH.** doas makes each hop + root. Mitigated by ACLs + key inventory, not by host-internal gates. +3. **Unrotated API keys in agents/compose .env** → quiet exfiltration + channels with billing and data access. +4. **Single tailnet identity** (`isomk@`) — Tailscale account takeover is + fleet takeover. Mitigated: Google IdP with hardware MFA (confirmed). + Residual risk bounded only by ACLs — which are currently allow-all with + Funnel granted to every member; Phase 2 is the bound. +5. **No detection** — everything above compounds because nothing alerts. + +## Assumptions register + +| # | Assumption | Basis | Status | +|---|---|---|---| +| A1 | Tailnet ACLs are default allow-all | console export 2026-11-01 | **confirmed 2026-11-01 — default policy verbatim (allow-all ACL, funnel-for-all, default SSH rule); baseline at `docs/tailscale-acl.hujson`** | +| A2 | SSH password auth is enabled (default) on most hosts | `passwordAuthentication = false` commented in base config | **confirmed 2026-11-01 on straylight + dixie via `sshd -T`; other hosts pending** | +| A3 | git.wntrmute.dev runs on phobos | Hetzner-range IP, but ≠ phobos's direct IP | open | +| A4 | open-webui :8080 has some auth enabled | not probed (would need creds) | open (downgraded — LAN trusted) | +| A5 | Exporters on other hosts are firewall-scoped like orion | orion pattern in repo; straylight's are `*`-bound | open | +| A6 | LAN is untrusted/shared | MikroTik-style subnet; unstated | **superseded 2026-11-01 — single-occupant apartment, LAN trusted; LAN-exposure findings downgraded** | +| A7 | Tailscale account has MFA | unstated | **confirmed 2026-11-01 — Google account, hardware MFA** | + +## References + +- Verified-exposure probes: this session, 2026-11-01 (`tailscale status`, + `ss -tlnp/-ulnp`, `sshd -T`, curl probes from straylight; SSH batch probe + of dixie). +- `~/src/nixos`: `configuration.nix`, `hw/straylight/default.nix`, + `docs/metrics-plan.md`, AGENTS.md. +- Fleet conventions: tailnet-service-ops skill (exposure patterns, + per-host procedures). diff --git a/docs/tailscale-acl.hujson b/docs/tailscale-acl.hujson new file mode 100644 index 0000000..ea6140f --- /dev/null +++ b/docs/tailscale-acl.hujson @@ -0,0 +1,28 @@ +// Baseline tailnet policy, exported from Tailscale admin console 2026-11-01 +// by Kyle. Unmodified default policy — allow-all ACL, funnel enabled for all +// members, default Tailscale SSH rule. Phase 2 of docs/security-plan.md +// replaces this; keep as the pre-segmentation baseline. +{ + "acls": [ + // Allow all connections. + {"action": "accept", "src": ["*"], "dst": ["*:*"]}, + ], + + // Default Tailscale SSH: every member may SSH to their own devices, + // as nonroot or root. + "ssh": [ + { + "action": "accept", + "src": ["autogroup:members"], + "dst": ["autogroup:self"], + "users": ["autogroup:nonroot", "root"], + }, + ], + "nodeAttrs": [ + { + // Any tailnet member can open a public Funnel endpoint. + "target": ["autogroup:members"], + "attr": ["funnel"], + }, + ], +}