imladris/docs/disk-health-plan.md

Disk-health monitoring plan (SMART / NVMe / SD card)

Status: plan, 2026-09-24. Not started. Requested by Kyle: "I'd also like to catch SMART/equivalent disk errors as well." Builds on: docs/metrics-plan.md. The VictoriaMetrics + vmalert + Alertmanager stack on orion is live, and alerts go to Mattermost ~alerts. Review: claude reviews each PR before it's switched. Security-relevant config (listen addresses, firewall, capabilities) is Kyle's call if in doubt.

---

1. Current state (verified 2026-09-24)

host	OS	drives	disk-health metrics today
hyperborea	Debian 13 (Pi 5)	/ = SD card mmcblk0 (58G); /srv = NVMe WD_BLACK SN7100 2TB	NVMe yes. prometheus-node-exporter-collectors nvme textfile timer (15 min): nvme_critical_warning, nvme_media_errors_total, nvme_percentage_used_ratio, nvme_available_spare_ratio, …, all healthy. SD card none. smartmontools and nvme-cli are installed.
straylight	NixOS	(check)	none
dixie	NixOS	(check; models live on the dixie-cache stick)	none
rift	NixOS	(check; FIDO2 LUKS)	none
orion	NixOS	(check; FIDO2 LUKS)	none
talos	laptop (check OS)	(check)	none; in the node-laptop job
titan	laptop (macOS, M3 Max)	internal Apple NVMe	none; in the node-laptop job

• Scrape wiring: all hosts are scraped for node_exporter on :9100 over the tailnet. The node job covers orion, dixie, straylight, rift and hyperborea; node-laptop covers talos and titan and doesn't page.
• Alert rules: there are no disk-health alert rules anywhere yet.

Where the exporter config lives:

• configs/metrics/exporters.nix (node_exporter, 0.0.0.0:9100, openFirewall = false, tailscale0 port opened explicitly). Imported by dixie and rift, by orion via hw/orion/metrics.nix, and by straylight via configs/metrics/straylight.nix.
• configs/metrics/hyperborea/: Debian files (the /etc/default args bind node_exporter to the tailscale IP 100.112.40.10:9100, plus a systemd override).
• Scrape jobs and rules are in hw/orion/metrics.nix.

2. Goal

Detect, and alert on, drive trouble before it becomes data loss:

• NVMe: critical warnings, media errors, spare capacity, wear.
• SATA/other: SMART overall-health failure; reallocated, pending and uncorrectable sectors.
• SD card (hyperborea /): no SMART exists, so use proxies. Alert on root going read-only and on kernel mmc / I/O errors.

3. Work items

3.1 NixOS hosts: smartctl_exporter

• Module: new configs/metrics/smartctl.nix, imported where exporters.nix is (dixie, rift, orion, straylight).

  { ... }:
  {
    services.prometheus.exporters.smartctl = {
      enable = true;
      port = 9633;
      listenAddress = "0.0.0.0";   # same pattern as node: firewall limits to tailscale0
      openFirewall = false;
      # devices = [ ];             # default: autodiscover via `smartctl --scan`
      maxInterval = "60s";
    };
    networking.firewall.interfaces.tailscale0.allowedTCPPorts = [ 9633 ];
  }
  

• Check each host's smartctl --scan output, and that the module's service can read NVMe. The upstream NixOS module grants device access and capabilities. Confirm with curl -s <host>:9633/metrics | grep smartctl_device from orion.
• USB sticks (dixie's dixie-cache) usually don't pass SMART through. Note that and exclude them from alerts, rather than fighting them.
• Deploy: rebuild-nixos per host (see §6).

3.2 hyperborea (Debian)

• SMART parity: enable the packaged collector timer: systemctl enable --now prometheus-node-exporter-smartmon.timer (from prometheus-node-exporter-collectors). The NVMe is already covered by the nvme timer; this adds smartmon_* for anything smartctl sees. Check that it writes /var/lib/prometheus/node-exporter/smartmon.prom.
• SD-card proxy: add a small textfile script plus a systemd timer (every 5 min) that writes /var/lib/prometheus/node-exporter/mmc.prom:

  # HELP node_mmc_kernel_errors_boot Kernel log lines matching mmc0/mmcblk0 errors since boot
  # TYPE node_mmc_kernel_errors_boot gauge
  node_mmc_kernel_errors_boot N
  

  • N = journalctl -k -b --no-pager | grep -Ec 'mmc0: .*(error|timeout)|mmcblk0.*(I/O error|error)|EXT4-fs error.*mmcblk0'.
  • Write it atomically (tmp + mv).
  • The persistent journal is now enabled on hyperborea (200M cap), so -b works.
• Track the files: put the script and units in the repo under configs/metrics/hyperborea/, like the existing files. Install them with sudo (Kyle granted straylight sudo on hyperborea on 09-24; confirm with Kyle for whichever agent does this). Keep the tailscale-only binding unchanged.

3.3 Laptops (optional, lower priority)

• talos: if it's NixOS, import smartctl.nix like the others. talos is due an SSD replacement, so check its current SMART state by hand first (smartctl -a) and report it to Kyle.
• titan (macOS): smartctl -a disk0 works on Apple NVMe via Homebrew smartmontools. An exporter is optional. If it's added, it's node-laptop-style, so it doesn't page.

3.4 orion: scrape jobs (in hw/orion/metrics.nix)

{
  job_name = "smartctl";
  static_configs = [
    { targets = [ "127.0.0.1:9633" ]; labels = { host = "orion"; }; }
    { targets = [ "dixie:9633" ]; labels = { host = "dixie"; }; }
    { targets = [ "straylight:9633" ]; labels = { host = "straylight"; }; }
    { targets = [ "rift:9633" ]; labels = { host = "rift"; }; }
  ];
}
# optional: job "smartctl-laptop" for talos/titan (no paging), mirroring node-laptop

• hyperborea: needs no new job. Its SMART/NVMe/mmc metrics arrive through node_exporter's textfile collector on :9100.
• FleetTargetDown: add smartctl to its job regex.

3.5 Alert rules (new group disk-health in hw/orion/metrics.nix)

The metric names below must be checked against what the exporters actually emit before merging (run a curl of each /metrics).

{
  name = "disk-health";
  rules = [
    { alert = "NvmeCriticalWarning";
      expr = ''nvme_critical_warning > 0 or smartctl_device_critical_warning > 0'';
      labels.severity = "page";
      annotations.summary = "NVMe critical warning on {{ $labels.instance }} {{ $labels.device }}"; }
    { alert = "DiskMediaErrors";
      expr = ''increase(nvme_media_errors_total[1h]) > 0 or increase(smartctl_device_media_errors[1h]) > 0'';
      labels.severity = "page";
      annotations.summary = "new media errors on {{ $labels.instance }} {{ $labels.device }}"; }
    { alert = "SmartHealthFailed";
      expr = ''smartctl_device_smart_status == 0 or smartmon_device_smart_healthy == 0'';
      labels.severity = "page";
      annotations.summary = "SMART overall-health FAILED on {{ $labels.instance }} {{ $labels.device }}"; }
    { alert = "NvmeSpareLow";
      expr = ''nvme_available_spare_ratio < nvme_available_spare_threshold_ratio
               or smartctl_device_available_spare < smartctl_device_available_spare_threshold'';
      labels.severity = "page"; }
    { alert = "DiskWearHigh";
      expr = ''nvme_percentage_used_ratio > 0.8 or smartctl_device_percentage_used > 80'';
      for = "1h";
      labels.severity = "warn"; }
    { alert = "SataSectorsBad";
      expr = ''increase(smartctl_device_attribute{attribute_name=~"Reallocated_Sector_Ct|Current_Pending_Sector|Offline_Uncorrectable",attribute_value_type="raw"}[24h]) > 0'';
      labels.severity = "page"; }
    { alert = "RootFilesystemReadOnly";
      expr = ''node_filesystem_readonly{mountpoint="/"} == 1'';
      for = "2m";
      labels.severity = "page";
      annotations.summary = "root filesystem read-only on {{ $labels.instance }}"; }
    { alert = "HyperboreaSdCardErrors";
      expr = ''increase(node_mmc_kernel_errors_boot{instance="hyperborea:9100"}[30m]) > 0'';
      labels.severity = "page";
      annotations.summary = "hyperborea SD card (/) kernel I/O errors"; }
  ];
}

• Severity: page goes to ~alerts, the same receiver as the other rules. warn routes the same way unless Kyle wants a quieter route. The laptops' series must not page: filter the paging rules to exclude host=~"talos|titan", or put the laptops in their own job and filter on job.

4. Verification

1. Metrics: from orion, curl each :9633/metrics, and hyperborea's :9100/metrics | grep -E 'smartmon_|nvme_|node_mmc_'.
2. Queries: in vmui (https://orion.scylla-hammerhead.ts.net:8429/vmui/):
   • smartctl_device_smart_status
   • nvme_critical_warning
   • node_mmc_kernel_errors_boot
   • up{job="smartctl"}
3. Rule syntax: vmalert -dryRun -rule=<file> (or build the NixOS config) before the switch.
4. Alert path:
   • fire a synthetic alert once (e.g. temporarily set DiskWearHigh's threshold to > -1 for 1 run, or use amtool against Alertmanager on 127.0.0.1:9093);
   • confirm it reaches ~alerts;
   • revert.
5. Record: post each host's current values (wear %, spare, media errors, SMART status) as a baseline in the thread.

5. Out of scope / notes

• Not building: smartd email/notification (Alertmanager covers alerting), and RAID/ZFS scrub metrics (no ZFS in the fleet).
• Existing Debian collector: don't replace the nvme textfile timer on hyperborea. It already works.
• SD cards give no advance wear warning. The real mitigation is that nothing important lives on hyperborea's / (the data is on /srv). The alert only shortens time-to-notice.

6. Deploy notes

• Commit first. rebuild-nixos deploys committed master only; uncommitted changes are never deployed. NixOS hosts use doas, not sudo (see CLAUDE.md). A per-host dry run is nixos-rebuild build --flake .#<host>.
• rebuild-nixos: runs root nixos-rebuild switch --flake . from /etc/nixos (a clone of ~/src/nixos, remote local). Pull first, and watch for the "hadn't pulled master" race.
• rift / orion: have a stale dbus-broker (since April). Switches can hang or exit 4, and units may not restart until a reboot. Reboots need Kyle at the rack (FIDO2 LUKS). If a switch hangs, start the new exporter unit directly and flag it, rather than forcing anything.
• dixie: evaluate on straylight and build on dixie (--eval-store auto --store ssh-ng://root@dixie), then set the profile and switch (see earlier dixie deploys).
• Dry-run first: run dry-activate on each host and confirm only the exporter/firewall units change before switching (same rule as the metrics rollout).