# Disk-health monitoring plan (SMART / NVMe / SD card) **Status:** plan, 2026-09-24. Not started. Requested by Kyle: "I'd also like to catch SMART/equivalent disk errors as well." **Builds on:** `docs/metrics-plan.md`. The VictoriaMetrics + vmalert + Alertmanager stack on orion is live, and alerts go to Mattermost ~alerts. **Review:** claude reviews each PR before it's switched. Security-relevant config (listen addresses, firewall, capabilities) is Kyle's call if in doubt. --- ## 1. Current state (verified 2026-09-24) | host | OS | drives | disk-health metrics today | |---|---|---|---| | hyperborea | Debian 13 (Pi 5) | `/` = SD card `mmcblk0` (58G); `/srv` = NVMe WD_BLACK SN7100 2TB | **NVMe yes.** `prometheus-node-exporter-collectors` nvme textfile timer (15 min): `nvme_critical_warning`, `nvme_media_errors_total`, `nvme_percentage_used_ratio`, `nvme_available_spare_ratio`, …, all healthy. **SD card none.** `smartmontools` and `nvme-cli` are installed. | | straylight | NixOS | (check) | none | | dixie | NixOS | (check; models live on the `dixie-cache` stick) | none | | rift | NixOS | (check; FIDO2 LUKS) | none | | orion | NixOS | (check; FIDO2 LUKS) | none | | talos | laptop (check OS) | (check) | none; in the `node-laptop` job | | titan | laptop (macOS, M3 Max) | internal Apple NVMe | none; in the `node-laptop` job | - **Scrape wiring:** all hosts are scraped for node_exporter on `:9100` over the tailnet. The `node` job covers orion, dixie, straylight, rift and hyperborea; `node-laptop` covers talos and titan and doesn't page. - **Alert rules:** there are **no** disk-health alert rules anywhere yet. **Where the exporter config lives:** - `configs/metrics/exporters.nix` (node_exporter, `0.0.0.0:9100`, `openFirewall = false`, `tailscale0` port opened explicitly). Imported by dixie and rift, by orion via `hw/orion/metrics.nix`, and by straylight via `configs/metrics/straylight.nix`. - `configs/metrics/hyperborea/`: Debian files (the `/etc/default` args bind node_exporter to the tailscale IP `100.112.40.10:9100`, plus a systemd override). - Scrape jobs and rules are in `hw/orion/metrics.nix`. ## 2. Goal Detect, and alert on, drive trouble before it becomes data loss: - **NVMe:** critical warnings, media errors, spare capacity, wear. - **SATA/other:** SMART overall-health failure; reallocated, pending and uncorrectable sectors. - **SD card** (hyperborea `/`): no SMART exists, so use proxies. Alert on root going read-only and on kernel mmc / I/O errors. ## 3. Work items ### 3.1 NixOS hosts: `smartctl_exporter` - **Module:** new `configs/metrics/smartctl.nix`, imported where `exporters.nix` is (dixie, rift, orion, straylight). ```nix { ... }: { services.prometheus.exporters.smartctl = { enable = true; port = 9633; listenAddress = "0.0.0.0"; # same pattern as node: firewall limits to tailscale0 openFirewall = false; # devices = [ ]; # default: autodiscover via `smartctl --scan` maxInterval = "60s"; }; networking.firewall.interfaces.tailscale0.allowedTCPPorts = [ 9633 ]; } ``` - **Check each host's** `smartctl --scan` output, and that the module's service can read NVMe. The upstream NixOS module grants device access and capabilities. Confirm with `curl -s :9633/metrics | grep smartctl_device` from orion. - **USB sticks** (dixie's `dixie-cache`) usually don't pass SMART through. Note that and exclude them from alerts, rather than fighting them. - **Deploy:** `rebuild-nixos` per host (see §6). ### 3.2 hyperborea (Debian) - **SMART parity:** enable the packaged collector timer: `systemctl enable --now prometheus-node-exporter-smartmon.timer` (from `prometheus-node-exporter-collectors`). The NVMe is already covered by the nvme timer; this adds `smartmon_*` for anything smartctl sees. Check that it writes `/var/lib/prometheus/node-exporter/smartmon.prom`. - **SD-card proxy:** add a small textfile script plus a systemd timer (every 5 min) that writes `/var/lib/prometheus/node-exporter/mmc.prom`: ``` # HELP node_mmc_kernel_errors_boot Kernel log lines matching mmc0/mmcblk0 errors since boot # TYPE node_mmc_kernel_errors_boot gauge node_mmc_kernel_errors_boot N ``` - `N = journalctl -k -b --no-pager | grep -Ec 'mmc0: .*(error|timeout)|mmcblk0.*(I/O error|error)|EXT4-fs error.*mmcblk0'`. - Write it atomically (tmp + mv). - The persistent journal is now enabled on hyperborea (200M cap), so `-b` works. - **Track the files:** put the script and units in the repo under `configs/metrics/hyperborea/`, like the existing files. Install them with sudo (Kyle granted straylight sudo on hyperborea on 09-24; confirm with Kyle for whichever agent does this). Keep the tailscale-only binding unchanged. ### 3.3 Laptops (optional, lower priority) - **talos:** if it's NixOS, import `smartctl.nix` like the others. talos is due an **SSD replacement**, so check its current SMART state by hand first (`smartctl -a`) and report it to Kyle. - **titan (macOS):** `smartctl -a disk0` works on Apple NVMe via Homebrew `smartmontools`. An exporter is optional. If it's added, it's `node-laptop`-style, so it doesn't page. ### 3.4 orion: scrape jobs (in `hw/orion/metrics.nix`) ```nix { job_name = "smartctl"; static_configs = [ { targets = [ "127.0.0.1:9633" ]; labels = { host = "orion"; }; } { targets = [ "dixie:9633" ]; labels = { host = "dixie"; }; } { targets = [ "straylight:9633" ]; labels = { host = "straylight"; }; } { targets = [ "rift:9633" ]; labels = { host = "rift"; }; } ]; } # optional: job "smartctl-laptop" for talos/titan (no paging), mirroring node-laptop ``` - **hyperborea:** needs no new job. Its SMART/NVMe/mmc metrics arrive through node_exporter's textfile collector on `:9100`. - **FleetTargetDown:** add `smartctl` to its job regex. ### 3.5 Alert rules (new group `disk-health` in `hw/orion/metrics.nix`) The metric names below must be **checked against what the exporters actually emit** before merging (run a `curl` of each `/metrics`). ```nix { name = "disk-health"; rules = [ { alert = "NvmeCriticalWarning"; expr = ''nvme_critical_warning > 0 or smartctl_device_critical_warning > 0''; labels.severity = "page"; annotations.summary = "NVMe critical warning on {{ $labels.instance }} {{ $labels.device }}"; } { alert = "DiskMediaErrors"; expr = ''increase(nvme_media_errors_total[1h]) > 0 or increase(smartctl_device_media_errors[1h]) > 0''; labels.severity = "page"; annotations.summary = "new media errors on {{ $labels.instance }} {{ $labels.device }}"; } { alert = "SmartHealthFailed"; expr = ''smartctl_device_smart_status == 0 or smartmon_device_smart_healthy == 0''; labels.severity = "page"; annotations.summary = "SMART overall-health FAILED on {{ $labels.instance }} {{ $labels.device }}"; } { alert = "NvmeSpareLow"; expr = ''nvme_available_spare_ratio < nvme_available_spare_threshold_ratio or smartctl_device_available_spare < smartctl_device_available_spare_threshold''; labels.severity = "page"; } { alert = "DiskWearHigh"; expr = ''nvme_percentage_used_ratio > 0.8 or smartctl_device_percentage_used > 80''; for = "1h"; labels.severity = "warn"; } { alert = "SataSectorsBad"; expr = ''increase(smartctl_device_attribute{attribute_name=~"Reallocated_Sector_Ct|Current_Pending_Sector|Offline_Uncorrectable",attribute_value_type="raw"}[24h]) > 0''; labels.severity = "page"; } { alert = "RootFilesystemReadOnly"; expr = ''node_filesystem_readonly{mountpoint="/"} == 1''; for = "2m"; labels.severity = "page"; annotations.summary = "root filesystem read-only on {{ $labels.instance }}"; } { alert = "HyperboreaSdCardErrors"; expr = ''increase(node_mmc_kernel_errors_boot{instance="hyperborea:9100"}[30m]) > 0''; labels.severity = "page"; annotations.summary = "hyperborea SD card (/) kernel I/O errors"; } ]; } ``` - **Severity:** `page` goes to ~alerts, the same receiver as the other rules. `warn` routes the same way unless Kyle wants a quieter route. The laptops' series must not page: filter the paging rules to exclude `host=~"talos|titan"`, or put the laptops in their own job and filter on `job`. ## 4. Verification 1. **Metrics:** from orion, `curl` each `:9633/metrics`, and hyperborea's `:9100/metrics | grep -E 'smartmon_|nvme_|node_mmc_'`. 2. **Queries:** in vmui (`https://orion.scylla-hammerhead.ts.net:8429/vmui/`): - `smartctl_device_smart_status` - `nvme_critical_warning` - `node_mmc_kernel_errors_boot` - `up{job="smartctl"}` 3. **Rule syntax:** `vmalert -dryRun -rule=` (or build the NixOS config) before the switch. 4. **Alert path:** - fire a synthetic alert once (e.g. temporarily set `DiskWearHigh`'s threshold to `> -1` for 1 run, or use `amtool` against Alertmanager on 127.0.0.1:9093); - confirm it reaches ~alerts; - revert. 5. **Record:** post each host's current values (wear %, spare, media errors, SMART status) as a baseline in the thread. ## 5. Out of scope / notes - **Not building:** smartd email/notification (Alertmanager covers alerting), and RAID/ZFS scrub metrics (no ZFS in the fleet). - **Existing Debian collector:** don't replace the nvme textfile timer on hyperborea. It already works. - **SD cards** give no advance wear warning. The real mitigation is that nothing important lives on hyperborea's `/` (the data is on `/srv`). The alert only shortens time-to-notice. ## 6. Deploy notes - **Commit first.** `rebuild-nixos` deploys committed `master` only; uncommitted changes are never deployed. NixOS hosts use **`doas`**, not sudo (see `CLAUDE.md`). A per-host dry run is `nixos-rebuild build --flake .#`. - **rebuild-nixos:** runs root `nixos-rebuild switch --flake .` from `/etc/nixos` (a clone of `~/src/nixos`, remote `local`). Pull first, and watch for the "hadn't pulled master" race. - **rift / orion:** have a stale dbus-broker (since April). Switches can hang or exit 4, and units may not restart until a reboot. Reboots need Kyle at the rack (FIDO2 LUKS). If a switch hangs, start the new exporter unit directly and flag it, rather than forcing anything. - **dixie:** evaluate on straylight and build on dixie (`--eval-store auto --store ssh-ng://root@dixie`), then set the profile and switch (see earlier dixie deploys). - **Dry-run first:** run `dry-activate` on each host and confirm only the exporter/firewall units change before switching (same rule as the metrics rollout).