docs: disk-health monitoring plan (SMART/NVMe/SD card)
Current state verified on 2026-09-24: hyperborea NVMe already exported via the Debian nvme textfile; no SMART on the NixOS hosts; no disk-health alert rules. Plan: smartctl_exporter module for the NixOS hosts, smartmon timer and SD-card kernel-error textfile on hyperborea, orion scrape job, a disk-health rule group, verification and deploy notes. For hand-off to another agent. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Claude Opus 5.5
@@ -0,0 +1,213 @@
|
|||||||
|
# Disk-health monitoring plan (SMART / NVMe / SD card)
|
||||||
|
|
||||||
|
**Status:** plan, 2026-09-24. Not started. Requested by Kyle: "I'd also like to
|
||||||
|
catch SMART/equivalent disk errors as well."
|
||||||
|
**Builds on:** `docs/metrics-plan.md`. The VictoriaMetrics + vmalert +
|
||||||
|
Alertmanager stack on orion is live, and alerts go to Mattermost ~alerts.
|
||||||
|
**Review:** claude reviews each PR before it's switched. Security-relevant
|
||||||
|
config (listen addresses, firewall, capabilities) is Kyle's call if in doubt.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Current state (verified 2026-09-24)
|
||||||
|
|
||||||
|
| host | OS | drives | disk-health metrics today |
|
||||||
|
|---|---|---|---|
|
||||||
|
| hyperborea | Debian 13 (Pi 5) | `/` = SD card `mmcblk0` (58G); `/srv` = NVMe WD_BLACK SN7100 2TB | **NVMe yes.** `prometheus-node-exporter-collectors` nvme textfile timer (15 min): `nvme_critical_warning`, `nvme_media_errors_total`, `nvme_percentage_used_ratio`, `nvme_available_spare_ratio`, …, all healthy. **SD card none.** `smartmontools` and `nvme-cli` are installed. |
|
||||||
|
| straylight | NixOS | (check) | none |
|
||||||
|
| dixie | NixOS | (check; models live on the `dixie-cache` stick) | none |
|
||||||
|
| rift | NixOS | (check; FIDO2 LUKS) | none |
|
||||||
|
| orion | NixOS | (check; FIDO2 LUKS) | none |
|
||||||
|
| talos | laptop (check OS) | (check) | none; in the `node-laptop` job |
|
||||||
|
| titan | laptop (macOS, M3 Max) | internal Apple NVMe | none; in the `node-laptop` job |
|
||||||
|
|
||||||
|
- **Scrape wiring:** all hosts are scraped for node_exporter on `:9100` over the
|
||||||
|
tailnet. The `node` job covers orion, dixie, straylight, rift and hyperborea;
|
||||||
|
`node-laptop` covers talos and titan and doesn't page.
|
||||||
|
- **Alert rules:** there are **no** disk-health alert rules anywhere yet.
|
||||||
|
|
||||||
|
**Where the exporter config lives:**
|
||||||
|
- `configs/metrics/exporters.nix` (node_exporter, `0.0.0.0:9100`,
|
||||||
|
`openFirewall = false`, `tailscale0` port opened explicitly). Imported by
|
||||||
|
dixie and rift, by orion via `hw/orion/metrics.nix`, and by straylight via
|
||||||
|
`configs/metrics/straylight.nix`.
|
||||||
|
- `configs/metrics/hyperborea/`: Debian files (the `/etc/default` args bind
|
||||||
|
node_exporter to the tailscale IP `100.112.40.10:9100`, plus a systemd
|
||||||
|
override).
|
||||||
|
- Scrape jobs and rules are in `hw/orion/metrics.nix`.
|
||||||
|
|
||||||
|
## 2. Goal
|
||||||
|
|
||||||
|
Detect, and alert on, drive trouble before it becomes data loss:
|
||||||
|
- **NVMe:** critical warnings, media errors, spare capacity, wear.
|
||||||
|
- **SATA/other:** SMART overall-health failure; reallocated, pending and
|
||||||
|
uncorrectable sectors.
|
||||||
|
- **SD card** (hyperborea `/`): no SMART exists, so use proxies. Alert on root
|
||||||
|
going read-only and on kernel mmc / I/O errors.
|
||||||
|
|
||||||
|
## 3. Work items
|
||||||
|
|
||||||
|
### 3.1 NixOS hosts: `smartctl_exporter`
|
||||||
|
- **Module:** new `configs/metrics/smartctl.nix`, imported where
|
||||||
|
`exporters.nix` is (dixie, rift, orion, straylight).
|
||||||
|
```nix
|
||||||
|
{ ... }:
|
||||||
|
{
|
||||||
|
services.prometheus.exporters.smartctl = {
|
||||||
|
enable = true;
|
||||||
|
port = 9633;
|
||||||
|
listenAddress = "0.0.0.0"; # same pattern as node: firewall limits to tailscale0
|
||||||
|
openFirewall = false;
|
||||||
|
# devices = [ ]; # default: autodiscover via `smartctl --scan`
|
||||||
|
maxInterval = "60s";
|
||||||
|
};
|
||||||
|
networking.firewall.interfaces.tailscale0.allowedTCPPorts = [ 9633 ];
|
||||||
|
}
|
||||||
|
```
|
||||||
|
- **Check each host's** `smartctl --scan` output, and that the module's service
|
||||||
|
can read NVMe. The upstream NixOS module grants device access and
|
||||||
|
capabilities. Confirm with `curl -s <host>:9633/metrics | grep smartctl_device`
|
||||||
|
from orion.
|
||||||
|
- **USB sticks** (dixie's `dixie-cache`) usually don't pass SMART through. Note
|
||||||
|
that and exclude them from alerts, rather than fighting them.
|
||||||
|
- **Deploy:** `rebuild-nixos` per host (see §6).
|
||||||
|
|
||||||
|
### 3.2 hyperborea (Debian)
|
||||||
|
- **SMART parity:** enable the packaged collector timer:
|
||||||
|
`systemctl enable --now prometheus-node-exporter-smartmon.timer` (from
|
||||||
|
`prometheus-node-exporter-collectors`). The NVMe is already covered by the
|
||||||
|
nvme timer; this adds `smartmon_*` for anything smartctl sees. Check that it
|
||||||
|
writes `/var/lib/prometheus/node-exporter/smartmon.prom`.
|
||||||
|
- **SD-card proxy:** add a small textfile script plus a systemd timer (every
|
||||||
|
5 min) that writes
|
||||||
|
`/var/lib/prometheus/node-exporter/mmc.prom`:
|
||||||
|
```
|
||||||
|
# HELP node_mmc_kernel_errors_boot Kernel log lines matching mmc0/mmcblk0 errors since boot
|
||||||
|
# TYPE node_mmc_kernel_errors_boot gauge
|
||||||
|
node_mmc_kernel_errors_boot N
|
||||||
|
```
|
||||||
|
- `N = journalctl -k -b --no-pager | grep -Ec 'mmc0: .*(error|timeout)|mmcblk0.*(I/O error|error)|EXT4-fs error.*mmcblk0'`.
|
||||||
|
- Write it atomically (tmp + mv).
|
||||||
|
- The persistent journal is now enabled on hyperborea (200M cap), so `-b`
|
||||||
|
works.
|
||||||
|
- **Track the files:** put the script and units in the repo under
|
||||||
|
`configs/metrics/hyperborea/`, like the existing files. Install them with sudo
|
||||||
|
(Kyle granted straylight sudo on hyperborea on 09-24; confirm with Kyle for
|
||||||
|
whichever agent does this). Keep
|
||||||
|
the tailscale-only binding unchanged.
|
||||||
|
|
||||||
|
### 3.3 Laptops (optional, lower priority)
|
||||||
|
- **talos:** if it's NixOS, import `smartctl.nix` like the others. talos is due
|
||||||
|
an **SSD replacement**, so check its current SMART state by hand first
|
||||||
|
(`smartctl -a`) and report it to Kyle.
|
||||||
|
- **titan (macOS):** `smartctl -a disk0` works on Apple NVMe via Homebrew
|
||||||
|
`smartmontools`. An exporter is optional. If it's added, it's
|
||||||
|
`node-laptop`-style, so it doesn't page.
|
||||||
|
|
||||||
|
### 3.4 orion: scrape jobs (in `hw/orion/metrics.nix`)
|
||||||
|
```nix
|
||||||
|
{
|
||||||
|
job_name = "smartctl";
|
||||||
|
static_configs = [
|
||||||
|
{ targets = [ "127.0.0.1:9633" ]; labels = { host = "orion"; }; }
|
||||||
|
{ targets = [ "dixie:9633" ]; labels = { host = "dixie"; }; }
|
||||||
|
{ targets = [ "straylight:9633" ]; labels = { host = "straylight"; }; }
|
||||||
|
{ targets = [ "rift:9633" ]; labels = { host = "rift"; }; }
|
||||||
|
];
|
||||||
|
}
|
||||||
|
# optional: job "smartctl-laptop" for talos/titan (no paging), mirroring node-laptop
|
||||||
|
```
|
||||||
|
- **hyperborea:** needs no new job. Its SMART/NVMe/mmc metrics arrive through
|
||||||
|
node_exporter's textfile collector on `:9100`.
|
||||||
|
- **FleetTargetDown:** add `smartctl` to its job regex.
|
||||||
|
|
||||||
|
### 3.5 Alert rules (new group `disk-health` in `hw/orion/metrics.nix`)
|
||||||
|
The metric names below must be **checked against what the exporters actually
|
||||||
|
emit** before merging (run a `curl` of each `/metrics`).
|
||||||
|
```nix
|
||||||
|
{
|
||||||
|
name = "disk-health";
|
||||||
|
rules = [
|
||||||
|
{ alert = "NvmeCriticalWarning";
|
||||||
|
expr = ''nvme_critical_warning > 0 or smartctl_device_critical_warning > 0'';
|
||||||
|
labels.severity = "page";
|
||||||
|
annotations.summary = "NVMe critical warning on {{ $labels.instance }} {{ $labels.device }}"; }
|
||||||
|
{ alert = "DiskMediaErrors";
|
||||||
|
expr = ''increase(nvme_media_errors_total[1h]) > 0 or increase(smartctl_device_media_errors[1h]) > 0'';
|
||||||
|
labels.severity = "page";
|
||||||
|
annotations.summary = "new media errors on {{ $labels.instance }} {{ $labels.device }}"; }
|
||||||
|
{ alert = "SmartHealthFailed";
|
||||||
|
expr = ''smartctl_device_smart_status == 0 or smartmon_device_smart_healthy == 0'';
|
||||||
|
labels.severity = "page";
|
||||||
|
annotations.summary = "SMART overall-health FAILED on {{ $labels.instance }} {{ $labels.device }}"; }
|
||||||
|
{ alert = "NvmeSpareLow";
|
||||||
|
expr = ''nvme_available_spare_ratio < nvme_available_spare_threshold_ratio
|
||||||
|
or smartctl_device_available_spare < smartctl_device_available_spare_threshold'';
|
||||||
|
labels.severity = "page"; }
|
||||||
|
{ alert = "DiskWearHigh";
|
||||||
|
expr = ''nvme_percentage_used_ratio > 0.8 or smartctl_device_percentage_used > 80'';
|
||||||
|
for = "1h";
|
||||||
|
labels.severity = "warn"; }
|
||||||
|
{ alert = "SataSectorsBad";
|
||||||
|
expr = ''increase(smartctl_device_attribute{attribute_name=~"Reallocated_Sector_Ct|Current_Pending_Sector|Offline_Uncorrectable",attribute_value_type="raw"}[24h]) > 0'';
|
||||||
|
labels.severity = "page"; }
|
||||||
|
{ alert = "RootFilesystemReadOnly";
|
||||||
|
expr = ''node_filesystem_readonly{mountpoint="/"} == 1'';
|
||||||
|
for = "2m";
|
||||||
|
labels.severity = "page";
|
||||||
|
annotations.summary = "root filesystem read-only on {{ $labels.instance }}"; }
|
||||||
|
{ alert = "HyperboreaSdCardErrors";
|
||||||
|
expr = ''increase(node_mmc_kernel_errors_boot{instance="hyperborea:9100"}[30m]) > 0'';
|
||||||
|
labels.severity = "page";
|
||||||
|
annotations.summary = "hyperborea SD card (/) kernel I/O errors"; }
|
||||||
|
];
|
||||||
|
}
|
||||||
|
```
|
||||||
|
- **Severity:** `page` goes to ~alerts, the same receiver as the other rules.
|
||||||
|
`warn` routes the same way unless Kyle wants a quieter route. The laptops'
|
||||||
|
series must not page: filter the paging rules to exclude `host=~"talos|titan"`,
|
||||||
|
or put the laptops in their own job and filter on `job`.
|
||||||
|
|
||||||
|
## 4. Verification
|
||||||
|
1. **Metrics:** from orion, `curl` each `:9633/metrics`, and hyperborea's
|
||||||
|
`:9100/metrics | grep -E 'smartmon_|nvme_|node_mmc_'`.
|
||||||
|
2. **Queries:** in vmui
|
||||||
|
(`https://orion.scylla-hammerhead.ts.net:8429/vmui/`):
|
||||||
|
- `smartctl_device_smart_status`
|
||||||
|
- `nvme_critical_warning`
|
||||||
|
- `node_mmc_kernel_errors_boot`
|
||||||
|
- `up{job="smartctl"}`
|
||||||
|
3. **Rule syntax:** `vmalert -dryRun -rule=<file>` (or build the NixOS config)
|
||||||
|
before the switch.
|
||||||
|
4. **Alert path:**
|
||||||
|
- fire a synthetic alert once (e.g. temporarily set `DiskWearHigh`'s
|
||||||
|
threshold to `> -1` for 1 run, or use `amtool` against Alertmanager on
|
||||||
|
127.0.0.1:9093);
|
||||||
|
- confirm it reaches ~alerts;
|
||||||
|
- revert.
|
||||||
|
5. **Record:** post each host's current values (wear %, spare, media errors,
|
||||||
|
SMART status) as a baseline in the thread.
|
||||||
|
|
||||||
|
## 5. Out of scope / notes
|
||||||
|
- **Not building:** smartd email/notification (Alertmanager covers alerting),
|
||||||
|
and RAID/ZFS scrub metrics (no ZFS in the fleet).
|
||||||
|
- **Existing Debian collector:** don't replace the nvme textfile timer on
|
||||||
|
hyperborea. It already works.
|
||||||
|
- **SD cards** give no advance wear warning. The real mitigation is that
|
||||||
|
nothing important lives on hyperborea's `/` (the data is on `/srv`). The alert
|
||||||
|
only shortens time-to-notice.
|
||||||
|
|
||||||
|
## 6. Deploy notes
|
||||||
|
- **rebuild-nixos:** runs root `nixos-rebuild switch --flake .` from `/etc/nixos`
|
||||||
|
(a clone of `~/src/nixos`, remote `local`). Pull first, and watch for the
|
||||||
|
"hadn't pulled master" race.
|
||||||
|
- **rift / orion:** have a stale dbus-broker (since April). Switches can hang or
|
||||||
|
exit 4, and units may not restart until a reboot. Reboots need Kyle at the rack
|
||||||
|
(FIDO2 LUKS). If a switch hangs, start the new exporter unit directly and flag
|
||||||
|
it, rather than forcing anything.
|
||||||
|
- **dixie:** evaluate on straylight and build on dixie (`--eval-store auto --store
|
||||||
|
ssh-ng://root@dixie`), then set the profile and switch (see earlier dixie
|
||||||
|
deploys).
|
||||||
|
- **Dry-run first:** run `dry-activate` on each host and confirm only the
|
||||||
|
exporter/firewall units change before switching (same rule as the metrics
|
||||||
|
rollout).
|
||||||
Reference in New Issue
Block a user