From 85d5631fa84c1ea145683730a525cd864f5b4557 Mon Sep 17 00:00:00 2001 From: "K. Isom" Date: Thu, 24 Sep 2026 18:59:15 -0700 Subject: [PATCH] docs: disk-health monitoring plan (SMART/NVMe/SD card) Current state verified on 2026-09-24: hyperborea NVMe already exported via the Debian nvme textfile; no SMART on the NixOS hosts; no disk-health alert rules. Plan: smartctl_exporter module for the NixOS hosts, smartmon timer and SD-card kernel-error textfile on hyperborea, orion scrape job, a disk-health rule group, verification and deploy notes. For hand-off to another agent. Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/disk-health-plan.md | 213 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 213 insertions(+) create mode 100644 docs/disk-health-plan.md diff --git a/docs/disk-health-plan.md b/docs/disk-health-plan.md new file mode 100644 index 0000000..92170b2 --- /dev/null +++ b/docs/disk-health-plan.md @@ -0,0 +1,213 @@ +# Disk-health monitoring plan (SMART / NVMe / SD card) + +**Status:** plan, 2026-09-24. Not started. Requested by Kyle: "I'd also like to +catch SMART/equivalent disk errors as well." +**Builds on:** `docs/metrics-plan.md`. The VictoriaMetrics + vmalert + +Alertmanager stack on orion is live, and alerts go to Mattermost ~alerts. +**Review:** claude reviews each PR before it's switched. Security-relevant +config (listen addresses, firewall, capabilities) is Kyle's call if in doubt. + +--- + +## 1. Current state (verified 2026-09-24) + +| host | OS | drives | disk-health metrics today | +|---|---|---|---| +| hyperborea | Debian 13 (Pi 5) | `/` = SD card `mmcblk0` (58G); `/srv` = NVMe WD_BLACK SN7100 2TB | **NVMe yes.** `prometheus-node-exporter-collectors` nvme textfile timer (15 min): `nvme_critical_warning`, `nvme_media_errors_total`, `nvme_percentage_used_ratio`, `nvme_available_spare_ratio`, …, all healthy. **SD card none.** `smartmontools` and `nvme-cli` are installed. | +| straylight | NixOS | (check) | none | +| dixie | NixOS | (check; models live on the `dixie-cache` stick) | none | +| rift | NixOS | (check; FIDO2 LUKS) | none | +| orion | NixOS | (check; FIDO2 LUKS) | none | +| talos | laptop (check OS) | (check) | none; in the `node-laptop` job | +| titan | laptop (macOS, M3 Max) | internal Apple NVMe | none; in the `node-laptop` job | + +- **Scrape wiring:** all hosts are scraped for node_exporter on `:9100` over the + tailnet. The `node` job covers orion, dixie, straylight, rift and hyperborea; + `node-laptop` covers talos and titan and doesn't page. +- **Alert rules:** there are **no** disk-health alert rules anywhere yet. + +**Where the exporter config lives:** +- `configs/metrics/exporters.nix` (node_exporter, `0.0.0.0:9100`, + `openFirewall = false`, `tailscale0` port opened explicitly). Imported by + dixie and rift, by orion via `hw/orion/metrics.nix`, and by straylight via + `configs/metrics/straylight.nix`. +- `configs/metrics/hyperborea/`: Debian files (the `/etc/default` args bind + node_exporter to the tailscale IP `100.112.40.10:9100`, plus a systemd + override). +- Scrape jobs and rules are in `hw/orion/metrics.nix`. + +## 2. Goal + +Detect, and alert on, drive trouble before it becomes data loss: +- **NVMe:** critical warnings, media errors, spare capacity, wear. +- **SATA/other:** SMART overall-health failure; reallocated, pending and + uncorrectable sectors. +- **SD card** (hyperborea `/`): no SMART exists, so use proxies. Alert on root + going read-only and on kernel mmc / I/O errors. + +## 3. Work items + +### 3.1 NixOS hosts: `smartctl_exporter` +- **Module:** new `configs/metrics/smartctl.nix`, imported where + `exporters.nix` is (dixie, rift, orion, straylight). + ```nix + { ... }: + { + services.prometheus.exporters.smartctl = { + enable = true; + port = 9633; + listenAddress = "0.0.0.0"; # same pattern as node: firewall limits to tailscale0 + openFirewall = false; + # devices = [ ]; # default: autodiscover via `smartctl --scan` + maxInterval = "60s"; + }; + networking.firewall.interfaces.tailscale0.allowedTCPPorts = [ 9633 ]; + } + ``` +- **Check each host's** `smartctl --scan` output, and that the module's service + can read NVMe. The upstream NixOS module grants device access and + capabilities. Confirm with `curl -s :9633/metrics | grep smartctl_device` + from orion. +- **USB sticks** (dixie's `dixie-cache`) usually don't pass SMART through. Note + that and exclude them from alerts, rather than fighting them. +- **Deploy:** `rebuild-nixos` per host (see §6). + +### 3.2 hyperborea (Debian) +- **SMART parity:** enable the packaged collector timer: + `systemctl enable --now prometheus-node-exporter-smartmon.timer` (from + `prometheus-node-exporter-collectors`). The NVMe is already covered by the + nvme timer; this adds `smartmon_*` for anything smartctl sees. Check that it + writes `/var/lib/prometheus/node-exporter/smartmon.prom`. +- **SD-card proxy:** add a small textfile script plus a systemd timer (every + 5 min) that writes + `/var/lib/prometheus/node-exporter/mmc.prom`: + ``` + # HELP node_mmc_kernel_errors_boot Kernel log lines matching mmc0/mmcblk0 errors since boot + # TYPE node_mmc_kernel_errors_boot gauge + node_mmc_kernel_errors_boot N + ``` + - `N = journalctl -k -b --no-pager | grep -Ec 'mmc0: .*(error|timeout)|mmcblk0.*(I/O error|error)|EXT4-fs error.*mmcblk0'`. + - Write it atomically (tmp + mv). + - The persistent journal is now enabled on hyperborea (200M cap), so `-b` + works. +- **Track the files:** put the script and units in the repo under + `configs/metrics/hyperborea/`, like the existing files. Install them with sudo + (Kyle granted straylight sudo on hyperborea on 09-24; confirm with Kyle for + whichever agent does this). Keep + the tailscale-only binding unchanged. + +### 3.3 Laptops (optional, lower priority) +- **talos:** if it's NixOS, import `smartctl.nix` like the others. talos is due + an **SSD replacement**, so check its current SMART state by hand first + (`smartctl -a`) and report it to Kyle. +- **titan (macOS):** `smartctl -a disk0` works on Apple NVMe via Homebrew + `smartmontools`. An exporter is optional. If it's added, it's + `node-laptop`-style, so it doesn't page. + +### 3.4 orion: scrape jobs (in `hw/orion/metrics.nix`) +```nix +{ + job_name = "smartctl"; + static_configs = [ + { targets = [ "127.0.0.1:9633" ]; labels = { host = "orion"; }; } + { targets = [ "dixie:9633" ]; labels = { host = "dixie"; }; } + { targets = [ "straylight:9633" ]; labels = { host = "straylight"; }; } + { targets = [ "rift:9633" ]; labels = { host = "rift"; }; } + ]; +} +# optional: job "smartctl-laptop" for talos/titan (no paging), mirroring node-laptop +``` +- **hyperborea:** needs no new job. Its SMART/NVMe/mmc metrics arrive through + node_exporter's textfile collector on `:9100`. +- **FleetTargetDown:** add `smartctl` to its job regex. + +### 3.5 Alert rules (new group `disk-health` in `hw/orion/metrics.nix`) +The metric names below must be **checked against what the exporters actually +emit** before merging (run a `curl` of each `/metrics`). +```nix +{ + name = "disk-health"; + rules = [ + { alert = "NvmeCriticalWarning"; + expr = ''nvme_critical_warning > 0 or smartctl_device_critical_warning > 0''; + labels.severity = "page"; + annotations.summary = "NVMe critical warning on {{ $labels.instance }} {{ $labels.device }}"; } + { alert = "DiskMediaErrors"; + expr = ''increase(nvme_media_errors_total[1h]) > 0 or increase(smartctl_device_media_errors[1h]) > 0''; + labels.severity = "page"; + annotations.summary = "new media errors on {{ $labels.instance }} {{ $labels.device }}"; } + { alert = "SmartHealthFailed"; + expr = ''smartctl_device_smart_status == 0 or smartmon_device_smart_healthy == 0''; + labels.severity = "page"; + annotations.summary = "SMART overall-health FAILED on {{ $labels.instance }} {{ $labels.device }}"; } + { alert = "NvmeSpareLow"; + expr = ''nvme_available_spare_ratio < nvme_available_spare_threshold_ratio + or smartctl_device_available_spare < smartctl_device_available_spare_threshold''; + labels.severity = "page"; } + { alert = "DiskWearHigh"; + expr = ''nvme_percentage_used_ratio > 0.8 or smartctl_device_percentage_used > 80''; + for = "1h"; + labels.severity = "warn"; } + { alert = "SataSectorsBad"; + expr = ''increase(smartctl_device_attribute{attribute_name=~"Reallocated_Sector_Ct|Current_Pending_Sector|Offline_Uncorrectable",attribute_value_type="raw"}[24h]) > 0''; + labels.severity = "page"; } + { alert = "RootFilesystemReadOnly"; + expr = ''node_filesystem_readonly{mountpoint="/"} == 1''; + for = "2m"; + labels.severity = "page"; + annotations.summary = "root filesystem read-only on {{ $labels.instance }}"; } + { alert = "HyperboreaSdCardErrors"; + expr = ''increase(node_mmc_kernel_errors_boot{instance="hyperborea:9100"}[30m]) > 0''; + labels.severity = "page"; + annotations.summary = "hyperborea SD card (/) kernel I/O errors"; } + ]; +} +``` +- **Severity:** `page` goes to ~alerts, the same receiver as the other rules. + `warn` routes the same way unless Kyle wants a quieter route. The laptops' + series must not page: filter the paging rules to exclude `host=~"talos|titan"`, + or put the laptops in their own job and filter on `job`. + +## 4. Verification +1. **Metrics:** from orion, `curl` each `:9633/metrics`, and hyperborea's + `:9100/metrics | grep -E 'smartmon_|nvme_|node_mmc_'`. +2. **Queries:** in vmui + (`https://orion.scylla-hammerhead.ts.net:8429/vmui/`): + - `smartctl_device_smart_status` + - `nvme_critical_warning` + - `node_mmc_kernel_errors_boot` + - `up{job="smartctl"}` +3. **Rule syntax:** `vmalert -dryRun -rule=` (or build the NixOS config) + before the switch. +4. **Alert path:** + - fire a synthetic alert once (e.g. temporarily set `DiskWearHigh`'s + threshold to `> -1` for 1 run, or use `amtool` against Alertmanager on + 127.0.0.1:9093); + - confirm it reaches ~alerts; + - revert. +5. **Record:** post each host's current values (wear %, spare, media errors, + SMART status) as a baseline in the thread. + +## 5. Out of scope / notes +- **Not building:** smartd email/notification (Alertmanager covers alerting), + and RAID/ZFS scrub metrics (no ZFS in the fleet). +- **Existing Debian collector:** don't replace the nvme textfile timer on + hyperborea. It already works. +- **SD cards** give no advance wear warning. The real mitigation is that + nothing important lives on hyperborea's `/` (the data is on `/srv`). The alert + only shortens time-to-notice. + +## 6. Deploy notes +- **rebuild-nixos:** runs root `nixos-rebuild switch --flake .` from `/etc/nixos` + (a clone of `~/src/nixos`, remote `local`). Pull first, and watch for the + "hadn't pulled master" race. +- **rift / orion:** have a stale dbus-broker (since April). Switches can hang or + exit 4, and units may not restart until a reboot. Reboots need Kyle at the rack + (FIDO2 LUKS). If a switch hangs, start the new exporter unit directly and flag + it, rather than forcing anything. +- **dixie:** evaluate on straylight and build on dixie (`--eval-store auto --store + ssh-ng://root@dixie`), then set the profile and switch (see earlier dixie + deploys). +- **Dry-run first:** run `dry-activate` on each host and confirm only the + exporter/firewall units change before switching (same rule as the metrics + rollout).