diff --git a/docs/metrics-plan.md b/docs/metrics-plan.md
new file mode 100644
index 0000000..a8174c0
--- /dev/null
+++ b/docs/metrics-plan.md
@@ -0,0 +1,337 @@
+# Fleet metrics and alerting
+
+Plan only. No deploy from this document.
+
+Goal: page before a person notices the four failure modes from 2026-09-23 — Dixie 9B slot starvation, tether cancel storm, Honcho deriver backlog, llama-server KV overflow.
+
+Query UI is vmui. Grafana is optional and out of the first cut.
+
+## Current state (checked 2026-09-23, straylight)
+
+Nothing in this flake scrapes or exports Prometheus today. `GET` results:
+
+| endpoint | result |
+|---|---|
+| dixie `:11434/metrics`, straylight `:11434/metrics` | 400 `model name is missing` — `--metrics` is off (llama.cpp 0.4.1, flag exists, default disabled) |
+| dixie `:11435/tether/metrics` | 200 Prometheus text. Live: converted 9493, cancelled 8479, passthrough 2833, upstream errors 307. In-flight is a single gauge, not by model |
+| rift `:8000/metrics` | `Metrics are disabled` (`METRICS_ENABLED=false`) |
+| orion sift `:8443/metrics` | 200 JSON (P2.7 not landed) |
+| orion disk | 953.9G INTEL NVMe, root 937G, 353G free (61%) |
+| straylight GTT | `mem_info_gtt_total=111669149696` (~104 GiB), used ~26 GiB at probe. hwmon `amdgpu` is `hwmon9` |
+
+`talos`, `titan`, and `hyperborea` are not in this flake. Node-exporter there is out-of-band.
+
+Tether path is `/tether/metrics`, not `/metrics`. Scrape that.
+
+## Store
+
+VictoriaMetrics single-node on **orion**. NixOS module already in nixpkgs 26.05: `services.victoriametrics`.
+
+```nix
+services.victoriametrics = {
+  enable = true;
+  listenAddress = "127.0.0.1:8428";  # plus tailscale0 via firewall, or bind TS IP
+  retentionPeriod = "90d";
+  prometheusConfig = { /* scrape_configs below */ };
+  extraOptions = [
+    "-storageDataPath=/var/lib/victoriametrics"
+    "-promscrape.maxScrapeSize=16MB"
+  ];
+};
+```
+
+The module already passes `-promscrape.config=` when `prometheusConfig` is set. No vmagent.
+
+**Listen:** same pattern as dixie llama-server. Bind `0.0.0.0:8428` (or the Tailscale IP if we want stricter), open **only** `networking.firewall.interfaces.tailscale0.allowedTCPPorts`. Never `networking.firewall.allowedTCPPorts` (base already has 22 only). vmui: `http://orion.scylla-hammerhead.ts.net:8428/vmui`.
+
+**Retention / disk:** 90 days. Budget **20 GiB** under `/var/lib/victoriametrics` (~20k series × 15s × 90d is well under that; orion has 353G free). Alert when `vm_data_size_bytes > 20GiB` or root `< 100GiB` free. First cut shares the root NVMe; no extra disk.
+
+Scrape interval: 15s for inference/tether/honcho/sift, 30s for node_exporter.
+
+## Alerting
+
+```
+vmalert  →  Alertmanager  →  Mattermost incoming webhook
+                         ↳  ntfy (optional fallback)
+```
+
+NixOS: `services.vmalert.instances."".enable` + `services.prometheus.alertmanager`.
+
+vmalert settings:
+
+- `datasource.url = http://127.0.0.1:8428`
+- `notifier.url = [ http://127.0.0.1:9093 ]`
+- rules from this repo (`configs/metrics/alerts/*.yml` or the `rules` nix option)
+
+Alertmanager listen `127.0.0.1:9093` (orion-local only; nothing else needs it). Receiver is a Slack-compatible incoming webhook (Mattermost accepts that payload).
+
+### Prerequisite (Kyle)
+
+Create a Mattermost incoming webhook. Write the URL to **`/root/mattermost-webhook`** on orion, mode 0600, root-owned. Channel is Kyle's call (`~ops` / `~dev` / a new `~alerts`).
+
+This plan does not create the webhook. Do not `cat` the file, copy it, or put the URL in Nix, git, or a log.
+
+The NixOS Alertmanager unit uses `DynamicUser`, so it cannot read a root-owned file directly. Phase 1 wires it with `LoadCredential`:
+
+```nix
+systemd.services.alertmanager.serviceConfig.LoadCredential = [
+  "mattermost-webhook:/root/mattermost-webhook"
+];
+```
+
+Receiver (Alertmanager config, not the store):
+
+```yaml
+slack_configs:
+  - api_url_file: /run/credentials/alertmanager.service/mattermost-webhook
+    channel: "#alerts"   # whatever Kyle picked
+    send_resolved: true
+```
+
+systemd injects the file at unit start. The URL never enters the Nix store.
+
+### ntfy fallback
+
+Second Alertmanager receiver, same routes, posting to an ntfy topic Kyle already uses (or a new `fleet-alerts` topic). Purpose: pages when Mattermost on straylight is itself down. First cut can ship MM-only and add ntfy in the same PR if the topic exists; otherwise a follow-up. No self-hosted ntfy unless Kyle wants it.
+
+## Exposure
+
+Every exporter:
+
+- listen `0.0.0.0:<port>` (matches dixie llama / tether today) **or** the Tailscale IP
+- firewall hole **only** on `tailscale0`
+- nothing on LAN, nothing on the internet
+- no `tailscale serve` for scrape ports (orion SearXNG/sift already occupy serve)
+
+Sift is the exception that already sits behind Tailscale Serve on `:8443`. Scrape `https://orion.scylla-hammerhead.ts.net:8443/metrics` with `scheme: https` and `bearer_token_file`. Do not open 8890 on the tailnet for metrics.
+
+Honcho deriver `:9090` is currently unpublished. Publishing it is a compose bind to the Tailscale IP (or `tailscale0` firewall + `0.0.0.0:9090` inside the container network). Not NixOS.
+
+## Targets
+
+Ports (first cut):
+
+| port | what |
+|---|---|
+| 9100 | node_exporter |
+| 9835 | nvidia_gpu_exporter (dixie) |
+| 11433 / 11434 | llama-server `--metrics` (existing API ports) |
+| 11435 | tether `/tether/metrics` |
+| 8000 | Honcho api `/metrics` |
+| 9090 | Honcho deriver `/metrics` |
+| 8443 | sift `/metrics` (Prometheus once P2.7 lands) |
+| 8428 | VictoriaMetrics (self-scrape + vmui) |
+
+### dixie (phase 1)
+
+- node_exporter: hwmon, systemd, filesystem, netdev. `collector.systemd`.
+- `nvidia_gpu_exporter` (NixOS `services.prometheus.exporters.nvidia-gpu` — confirm exact option name at implement). RTX 3060, power-capped 140W.
+- llama-server **and** llama-vision: add `--metrics` to `llamaService` in `hw/dixie/default.nix`. Same port as the API; no extra hole. Restarts the routers — do this with the dixie deploy, not ad hoc.
+- tether: scrape `http://dixie:11435/tether/metrics` as-is.
+
+Tether follow-up (small change, after this plan lands, not phase 1): in-flight gauge **by model**, converted-request duration histogram, cancel reason label (`client_disconnect` vs `upstream_error`). Today `tether_cancelled_total` is a single counter and `tether_in_flight` has no labels.
+
+### straylight (phase 3)
+
+- node_exporter with `collector.hwmon` (amdgpu is `hwmon9`) and `collector.drm`.
+- GTT: node_exporter does not export `mem_info_gtt_*`. Tiny textfile collector (systemd timer → `/var/lib/node_exporter/gtt.prom`) reading `/sys/class/drm/card1/device/mem_info_gtt_{total,used}`.
+- llama-server `--metrics` on `:11434`. Same restart caveat.
+
+### rift (phase 2) — no SQL exporter
+
+Honcho 3.2.0 already has Prometheus metrics, off by default.
+
+- api: `GET /metrics` on `:8000` (already tailnet-reachable).
+- deriver: `start_http_server(9090)` inside the container; port is **not** published in compose yet.
+
+Series we will use: `deriver_queue_items_pending`, `deriver_queue_oldest_pending_age_seconds`, `deriver_queue_work_units_claimed` / `eligible`, `dialectic_calls`, `db_connections_open`, `db_queries_in_flight`.
+
+**Enable (scheduled, not ad hoc):** `METRICS_ENABLED=true`, publish deriver `:9090` on the tailnet only, recreate **api and deriver**. This is a Honcho restart.
+
+**Do not retag images.** api is `3.2.0-pr1092-7e57fe2`; deriver is digest-pinned. Metrics work does not touch either.
+
+Also node_exporter on rift.
+
+### orion (phase 1)
+
+- node_exporter
+- VictoriaMetrics + vmalert + alertmanager
+- sift: scrape **after P2.7**. Target `https://orion.scylla-hammerhead.ts.net:8443/metrics` with `bearer_token_file` (dedicated `metrics` caller in `keys.toml`: no workspace, no fetch/search). Loopback `/metrics` stays open without the bearer. Restricted callers are aggregated only (no per-caller series, no `tavily:restricted` gauge) — that is titan/P2.7, this scrape config just consumes it.
+
+### talos, titan, hyperborea (phase 3)
+
+node_exporter only, on whatever package path each host already uses. Not this flake.
+
+## Alert rules (first five)
+
+Freeze series names against the first successful scrape. Thresholds below are the intent.
+
+### 1. Dixie 9B slots 100% busy > 10m
+
+Tonight: 8 slots, unified 128k pool, Honcho + three guardians.
+
+```yaml
+- alert: Dixie9BSlotsSaturated
+  expr: |
+    min_over_time(
+      llamacpp:requests_processing{job="llama-dixie", model="ornith-1.5-9b-uncensored"}[10m]
+    ) >= 8
+  for: 10m
+  labels: { severity: page }
+  annotations:
+    summary: "dixie 9B slots busy for 10m"
+```
+
+If router-mode metrics lack a `model` label, scrape `:11434` and `:11433` as separate jobs and match on job. Confirm `llamacpp:requests_processing` vs a slots-busy gauge on first scrape. `parallel = 8` is the threshold; if that changes, the rule changes.
+
+### 2. Tether cancelled/converted > 50% over 15m
+
+Live right now cancelled/converted ≈ 89% (8479/9493). This rule would already be firing — that is the point.
+
+```yaml
+- alert: TetherCancelStorm
+  expr: |
+    (
+      rate(tether_cancelled_total[15m])
+      /
+      clamp_min(rate(tether_requests_total{kind="converted"}[15m]), 1e-6)
+    ) > 0.5
+  for: 15m
+  labels: { severity: page }
+```
+
+`tether_cancelled_total` currently counts converted **or** proxied cancels (HELP text). Until the reason label exists, compare against converted only; a follow-up splits `client_disconnect` vs `upstream_error`.
+
+### 3. Honcho pending queue (native metrics)
+
+Not a SQL exporter.
+
+```yaml
+- alert: HonchoDeriverBacklog
+  expr: |
+    min_over_time(deriver_queue_items_pending[30m]) > 100
+    and
+    deriv(deriver_queue_oldest_pending_age_seconds[30m]) > 0
+  for: 30m
+  labels: { severity: page }
+  annotations:
+    summary: "Honcho pending > 100 and oldest age still rising"
+```
+
+Needs phase 2 (`METRICS_ENABLED=true`). Do not enable the rule before the scrape exists.
+
+### 4. llama-server restarts or KV overflow
+
+KV overflow (`failed to find free space in the KV cache` / `Context size has been exceeded`) is **not** a Prometheus series today. First cut uses proxies:
+
+```yaml
+- alert: LlamaServerRestarting
+  expr: increase(node_systemd_unit_restarts_total{name=~"llama-server.service|llama-vision.service"}[15m]) > 0
+  for: 0m
+  labels: { severity: page }
+
+- alert: LlamaKVSaturated
+  expr: llamacpp:kv_cache_usage_ratio > 0.99
+  for: 5m
+  labels: { severity: page }
+```
+
+Journal-derived counter for the overflow string is a later add (textfile or mtail), not first cut. `requests_deferred > 0` for 5m is a useful extra if the series exists.
+
+### 5. GPU memory > 95% or temperature above threshold
+
+```yaml
+- alert: GpuMemoryHigh
+  expr: |
+    (nvidia_gpu_memory_used_bytes / nvidia_gpu_memory_total_bytes) > 0.95
+    or
+    (gtt_used_bytes / gtt_total_bytes) > 0.95
+  for: 5m
+  labels: { severity: page }
+
+- alert: GpuHot
+  expr: |
+    nvidia_gpu_temperature_celsius > 83
+    or
+    node_hwmon_temp_celsius{chip=~"amdgpu.*"} > 90
+  for: 2m
+  labels: { severity: page }
+```
+
+- dixie RTX 3060: 83 °C (card is power-capped 140W; throttle band is ~83–93).
+- straylight 8060S: 90 °C edge. GTT 95% of ~104 GiB.
+
+## Scrape config sketch
+
+Jobs added per phase, not all on day one.
+
+```yaml
+scrape_configs:
+  - job_name: vm
+    static_configs: [{ targets: ["127.0.0.1:8428"] }]
+
+  - job_name: node
+    scrape_interval: 30s
+    static_configs:
+      - { targets: ["orion:9100"], labels: { host: orion } }
+      - { targets: ["dixie:9100"], labels: { host: dixie } }
+      # later: straylight, rift, talos, titan, hyperborea
+
+  - job_name: nvidia-dixie
+    static_configs: [{ targets: ["dixie:9835"] }]
+
+  - job_name: llama-dixie
+    metrics_path: /metrics
+    static_configs:
+      - { targets: ["dixie:11434"], labels: { router: helper } }
+      - { targets: ["dixie:11433"], labels: { router: vision } }
+
+  - job_name: tether
+    metrics_path: /tether/metrics
+    static_configs: [{ targets: ["dixie:11435"] }]
+
+  - job_name: honcho-api        # phase 2
+    static_configs: [{ targets: ["rift:8000"] }]
+
+  - job_name: honcho-deriver    # phase 2
+    static_configs: [{ targets: ["rift:9090"] }]
+
+  - job_name: sift              # after P2.7
+    scheme: https
+    metrics_path: /metrics
+    bearer_token_file: /var/lib/victoriametrics/sift-metrics.token
+    static_configs: [{ targets: ["orion.scylla-hammerhead.ts.net:8443"] }]
+```
+
+MagicDNS names, not Tailscale IPs. Sift token file is the `metrics` caller key; mode 0600; not in git.
+
+## Nix layout (when implementing)
+
+Not this PR.
+
+- `configs/metrics/exporters.nix` — node_exporter + tailscale0 hole, imported per host with a boolean.
+- `configs/metrics/alerts/*.yml` — the five rules.
+- `hw/orion/metrics.nix` — VM, vmalert, alertmanager (`LoadCredential` → `/root/mattermost-webhook`), scrape config, sift token path.
+- `hw/dixie/default.nix` — `--metrics` on both llama units, nvidia exporter, 9100/9835 on tailscale0.
+- `hw/straylight/default.nix` — `--metrics`, GTT textfile, 9100 on tailscale0.
+- rift compose (on rift, not this flake) — `METRICS_ENABLED`, publish 9090.
+
+`rebuild-nixos` never deploys uncommitted trees. Orion/dixie switches follow the existing host paths (orion local `/etc/nixos`, dixie copy+switch).
+
+## Order
+
+1. **orion + dixie.** VM + vmalert + alertmanager + node_exporter on both. Dixie nvidia exporter + llama `--metrics` + tether scrape. Rules 1, 2, 4 (restart half), 5 (nvidia half). Mattermost webhook must exist first or alerts go nowhere.
+2. **rift.** Scheduled Honcho recreate with `METRICS_ENABLED=true` and deriver `:9090` on the tailnet. Then rule 3. Do not bump images.
+3. **the rest.** straylight (llama `--metrics`, hwmon, GTT), then talos/titan/hyperborea node_exporter. Sift scrape as soon as P2.7 is on orion (can land during 1 or 3; needs the bearer file on orion).
+
+Tether histogram / in-flight-by-model / cancel-reason: after the plan is accepted, small tether change, not a blocker for phase 1 (rule 2 works on the counters that already exist).
+
+## Out of scope
+
+- Grafana
+- SQL exporter on Honcho Postgres
+- Changing Honcho image pins
+- Creating the Mattermost webhook
+- LAN or public scrape
+- Journal mtail for KV overflow strings
+- Deploying any of the above from this document