docs: disk-health monitoring plan (SMART/NVMe/SD card)

Current state verified on 2026-09-24: hyperborea NVMe already exported via the Debian nvme textfile; no SMART on the NixOS hosts; no disk-health alert rules. Plan: smartctl_exporter module for the NixOS hosts, smartmon timer and SD-card kernel-error textfile on hyperborea, orion scrape job, a disk-health rule group, verification and deploy notes. For hand-off to another agent. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-24 18:59:15 -07:00
parent 78d30026ce
commit 85d5631fa8
1 changed files with 213 additions and 0 deletions
@@ -0,0 +1,213 @@
+# Disk-health monitoring plan (SMART / NVMe / SD card)
+
+**Status:** plan, 2026-09-24. Not started. Requested by Kyle: "I'd also like to
+catch SMART/equivalent disk errors as well."
+**Builds on:** `docs/metrics-plan.md`. The VictoriaMetrics + vmalert +
+Alertmanager stack on orion is live, and alerts go to Mattermost ~alerts.
+**Review:** claude reviews each PR before it's switched. Security-relevant
+config (listen addresses, firewall, capabilities) is Kyle's call if in doubt.
+
+---
+
+## 1. Current state (verified 2026-09-24)
+
+| host | OS | drives | disk-health metrics today |
+|---|---|---|---|
+| hyperborea | Debian 13 (Pi 5) | `/` = SD card `mmcblk0` (58G); `/srv` = NVMe WD_BLACK SN7100 2TB | **NVMe yes.** `prometheus-node-exporter-collectors` nvme textfile timer (15 min): `nvme_critical_warning`, `nvme_media_errors_total`, `nvme_percentage_used_ratio`, `nvme_available_spare_ratio`, …, all healthy. **SD card none.** `smartmontools` and `nvme-cli` are installed. |
+| straylight | NixOS | (check) | none |
+| dixie | NixOS | (check; models live on the `dixie-cache` stick) | none |
+| rift | NixOS | (check; FIDO2 LUKS) | none |
+| orion | NixOS | (check; FIDO2 LUKS) | none |
+| talos | laptop (check OS) | (check) | none; in the `node-laptop` job |
+| titan | laptop (macOS, M3 Max) | internal Apple NVMe | none; in the `node-laptop` job |
+
+- **Scrape wiring:** all hosts are scraped for node_exporter on `:9100` over the
+  tailnet. The `node` job covers orion, dixie, straylight, rift and hyperborea;
+  `node-laptop` covers talos and titan and doesn't page.
+- **Alert rules:** there are **no** disk-health alert rules anywhere yet.
+
+**Where the exporter config lives:**
+- `configs/metrics/exporters.nix` (node_exporter, `0.0.0.0:9100`,
+  `openFirewall = false`, `tailscale0` port opened explicitly). Imported by
+  dixie and rift, by orion via `hw/orion/metrics.nix`, and by straylight via
+  `configs/metrics/straylight.nix`.
+- `configs/metrics/hyperborea/`: Debian files (the `/etc/default` args bind
+  node_exporter to the tailscale IP `100.112.40.10:9100`, plus a systemd
+  override).
+- Scrape jobs and rules are in `hw/orion/metrics.nix`.
+
+## 2. Goal
+
+Detect, and alert on, drive trouble before it becomes data loss:
+- **NVMe:** critical warnings, media errors, spare capacity, wear.
+- **SATA/other:** SMART overall-health failure; reallocated, pending and
+  uncorrectable sectors.
+- **SD card** (hyperborea `/`): no SMART exists, so use proxies. Alert on root
+  going read-only and on kernel mmc / I/O errors.
+
+## 3. Work items
+
+### 3.1 NixOS hosts: `smartctl_exporter`
+- **Module:** new `configs/metrics/smartctl.nix`, imported where
+  `exporters.nix` is (dixie, rift, orion, straylight).
+  ```nix
+  { ... }:
+  {
+    services.prometheus.exporters.smartctl = {
+      enable = true;
+      port = 9633;
+      listenAddress = "0.0.0.0";   # same pattern as node: firewall limits to tailscale0
+      openFirewall = false;
+      # devices = [ ];             # default: autodiscover via `smartctl --scan`
+      maxInterval = "60s";
+    };
+    networking.firewall.interfaces.tailscale0.allowedTCPPorts = [ 9633 ];
+  }
+  ```
+- **Check each host's** `smartctl --scan` output, and that the module's service
+  can read NVMe. The upstream NixOS module grants device access and
+  capabilities. Confirm with `curl -s <host>:9633/metrics | grep smartctl_device`
+  from orion.
+- **USB sticks** (dixie's `dixie-cache`) usually don't pass SMART through. Note
+  that and exclude them from alerts, rather than fighting them.
+- **Deploy:** `rebuild-nixos` per host (see §6).
+
+### 3.2 hyperborea (Debian)
+- **SMART parity:** enable the packaged collector timer:
+  `systemctl enable --now prometheus-node-exporter-smartmon.timer` (from
+  `prometheus-node-exporter-collectors`). The NVMe is already covered by the
+  nvme timer; this adds `smartmon_*` for anything smartctl sees. Check that it
+  writes `/var/lib/prometheus/node-exporter/smartmon.prom`.
+- **SD-card proxy:** add a small textfile script plus a systemd timer (every
+  5 min) that writes
+  `/var/lib/prometheus/node-exporter/mmc.prom`:
+  ```
+  # HELP node_mmc_kernel_errors_boot Kernel log lines matching mmc0/mmcblk0 errors since boot
+  # TYPE node_mmc_kernel_errors_boot gauge
+  node_mmc_kernel_errors_boot N
+  ```
+  - `N = journalctl -k -b --no-pager | grep -Ec 'mmc0: .*(error|timeout)|mmcblk0.*(I/O error|error)|EXT4-fs error.*mmcblk0'`.
+  - Write it atomically (tmp + mv).
+  - The persistent journal is now enabled on hyperborea (200M cap), so `-b`
+    works.
+- **Track the files:** put the script and units in the repo under
+  `configs/metrics/hyperborea/`, like the existing files. Install them with sudo
+  (Kyle granted straylight sudo on hyperborea on 09-24; confirm with Kyle for
+  whichever agent does this). Keep
+  the tailscale-only binding unchanged.
+
+### 3.3 Laptops (optional, lower priority)
+- **talos:** if it's NixOS, import `smartctl.nix` like the others. talos is due
+  an **SSD replacement**, so check its current SMART state by hand first
+  (`smartctl -a`) and report it to Kyle.
+- **titan (macOS):** `smartctl -a disk0` works on Apple NVMe via Homebrew
+  `smartmontools`. An exporter is optional. If it's added, it's
+  `node-laptop`-style, so it doesn't page.
+
+### 3.4 orion: scrape jobs (in `hw/orion/metrics.nix`)
+```nix
+{
+  job_name = "smartctl";
+  static_configs = [
+    { targets = [ "127.0.0.1:9633" ]; labels = { host = "orion"; }; }
+    { targets = [ "dixie:9633" ]; labels = { host = "dixie"; }; }
+    { targets = [ "straylight:9633" ]; labels = { host = "straylight"; }; }
+    { targets = [ "rift:9633" ]; labels = { host = "rift"; }; }
+  ];
+}
+# optional: job "smartctl-laptop" for talos/titan (no paging), mirroring node-laptop
+```
+- **hyperborea:** needs no new job. Its SMART/NVMe/mmc metrics arrive through
+  node_exporter's textfile collector on `:9100`.
+- **FleetTargetDown:** add `smartctl` to its job regex.
+
+### 3.5 Alert rules (new group `disk-health` in `hw/orion/metrics.nix`)
+The metric names below must be **checked against what the exporters actually
+emit** before merging (run a `curl` of each `/metrics`).
+```nix
+{
+  name = "disk-health";
+  rules = [
+    { alert = "NvmeCriticalWarning";
+      expr = ''nvme_critical_warning > 0 or smartctl_device_critical_warning > 0'';
+      labels.severity = "page";
+      annotations.summary = "NVMe critical warning on {{ $labels.instance }} {{ $labels.device }}"; }
+    { alert = "DiskMediaErrors";
+      expr = ''increase(nvme_media_errors_total[1h]) > 0 or increase(smartctl_device_media_errors[1h]) > 0'';
+      labels.severity = "page";
+      annotations.summary = "new media errors on {{ $labels.instance }} {{ $labels.device }}"; }
+    { alert = "SmartHealthFailed";
+      expr = ''smartctl_device_smart_status == 0 or smartmon_device_smart_healthy == 0'';
+      labels.severity = "page";
+      annotations.summary = "SMART overall-health FAILED on {{ $labels.instance }} {{ $labels.device }}"; }
+    { alert = "NvmeSpareLow";
+      expr = ''nvme_available_spare_ratio < nvme_available_spare_threshold_ratio
+               or smartctl_device_available_spare < smartctl_device_available_spare_threshold'';
+      labels.severity = "page"; }
+    { alert = "DiskWearHigh";
+      expr = ''nvme_percentage_used_ratio > 0.8 or smartctl_device_percentage_used > 80'';
+      for = "1h";
+      labels.severity = "warn"; }
+    { alert = "SataSectorsBad";
+      expr = ''increase(smartctl_device_attribute{attribute_name=~"Reallocated_Sector_Ct|Current_Pending_Sector|Offline_Uncorrectable",attribute_value_type="raw"}[24h]) > 0'';
+      labels.severity = "page"; }
+    { alert = "RootFilesystemReadOnly";
+      expr = ''node_filesystem_readonly{mountpoint="/"} == 1'';
+      for = "2m";
+      labels.severity = "page";
+      annotations.summary = "root filesystem read-only on {{ $labels.instance }}"; }
+    { alert = "HyperboreaSdCardErrors";
+      expr = ''increase(node_mmc_kernel_errors_boot{instance="hyperborea:9100"}[30m]) > 0'';
+      labels.severity = "page";
+      annotations.summary = "hyperborea SD card (/) kernel I/O errors"; }
+  ];
+}
+```
+- **Severity:** `page` goes to ~alerts, the same receiver as the other rules.
+  `warn` routes the same way unless Kyle wants a quieter route. The laptops'
+  series must not page: filter the paging rules to exclude `host=~"talos|titan"`,
+  or put the laptops in their own job and filter on `job`.
+
+## 4. Verification
+1. **Metrics:** from orion, `curl` each `:9633/metrics`, and hyperborea's
+   `:9100/metrics | grep -E 'smartmon_|nvme_|node_mmc_'`.
+2. **Queries:** in vmui
+   (`https://orion.scylla-hammerhead.ts.net:8429/vmui/`):
+   - `smartctl_device_smart_status`
+   - `nvme_critical_warning`
+   - `node_mmc_kernel_errors_boot`
+   - `up{job="smartctl"}`
+3. **Rule syntax:** `vmalert -dryRun -rule=<file>` (or build the NixOS config)
+   before the switch.
+4. **Alert path:**
+   - fire a synthetic alert once (e.g. temporarily set `DiskWearHigh`'s
+     threshold to `> -1` for 1 run, or use `amtool` against Alertmanager on
+     127.0.0.1:9093);
+   - confirm it reaches ~alerts;
+   - revert.
+5. **Record:** post each host's current values (wear %, spare, media errors,
+   SMART status) as a baseline in the thread.
+
+## 5. Out of scope / notes
+- **Not building:** smartd email/notification (Alertmanager covers alerting),
+  and RAID/ZFS scrub metrics (no ZFS in the fleet).
+- **Existing Debian collector:** don't replace the nvme textfile timer on
+  hyperborea. It already works.
+- **SD cards** give no advance wear warning. The real mitigation is that
+  nothing important lives on hyperborea's `/` (the data is on `/srv`). The alert
+  only shortens time-to-notice.
+
+## 6. Deploy notes
+- **rebuild-nixos:** runs root `nixos-rebuild switch --flake .` from `/etc/nixos`
+  (a clone of `~/src/nixos`, remote `local`). Pull first, and watch for the
+  "hadn't pulled master" race.
+- **rift / orion:** have a stale dbus-broker (since April). Switches can hang or
+  exit 4, and units may not restart until a reboot. Reboots need Kyle at the rack
+  (FIDO2 LUKS). If a switch hangs, start the new exporter unit directly and flag
+  it, rather than forcing anything.
+- **dixie:** evaluate on straylight and build on dixie (`--eval-store auto --store
+  ssh-ng://root@dixie`), then set the profile and switch (see earlier dixie
+  deploys).
+- **Dry-run first:** run `dry-activate` on each host and confirm only the
+  exporter/firewall units change before switching (same rule as the metrics
+  rollout).