imladris/docs/metrics-plan.md

# Fleet metrics and alerting

Plan only. No deploy from this document.

Goal: page before a person notices the four failure modes from 2026-09-23 — Dixie 9B slot starvation, tether cancel storm, Honcho deriver backlog, llama-server KV overflow.

Query UI is vmui. Grafana is optional and out of the first cut.

## Current state (checked 2026-09-23, straylight)

Nothing in this flake scrapes or exports Prometheus today. `GET` results:

| endpoint | result |
|---|---|
| dixie `:11434/metrics`, straylight `:11434/metrics` | 400 `model name is missing` — `--metrics` is off (llama.cpp 0.4.1, flag exists, default disabled) |
| dixie `:11435/tether/metrics` | 200 Prometheus text. Live: converted 9493, cancelled 8479, passthrough 2833, upstream errors 307. In-flight is a single gauge, not by model |
| rift `:8000/metrics` | `Metrics are disabled` (`METRICS_ENABLED=false`) |
| orion sift `:8443/metrics` | 200 JSON (P2.7 not landed) |
| orion disk | 953.9G INTEL NVMe, root 937G, 353G free (61%) |
| straylight GTT | `mem_info_gtt_total=111669149696` (~104 GiB), used ~26 GiB at probe. hwmon `amdgpu` is `hwmon9` |

`talos`, `titan`, and `hyperborea` are not in this flake. Node-exporter there is out-of-band.

Tether path is `/tether/metrics`, not `/metrics`. Scrape that.

## Store

VictoriaMetrics single-node on **orion**. NixOS module already in nixpkgs 26.05: `services.victoriametrics`.

```nix
services.victoriametrics = {
  enable = true;
  listenAddress = "127.0.0.1:8428";  # plus tailscale0 via firewall, or bind TS IP
  retentionPeriod = "90d";
  prometheusConfig = { /* scrape_configs below */ };
  extraOptions = [
    "-storageDataPath=/var/lib/victoriametrics"
    "-promscrape.maxScrapeSize=16MB"
  ];
};
```

The module already passes `-promscrape.config=` when `prometheusConfig` is set. No vmagent.

**Listen:** same pattern as dixie llama-server. Bind `0.0.0.0:8428` (or the Tailscale IP if we want stricter), open **only** `networking.firewall.interfaces.tailscale0.allowedTCPPorts`. Never `networking.firewall.allowedTCPPorts` (base already has 22 only). vmui: `http://orion.scylla-hammerhead.ts.net:8428/vmui`.

**Retention / disk:** 90 days. Budget **20 GiB** under `/var/lib/victoriametrics` (~20k series × 15s × 90d is well under that; orion has 353G free). Alert when `vm_data_size_bytes > 20GiB` or root `< 100GiB` free. First cut shares the root NVMe; no extra disk.

Scrape interval: 15s for inference/tether/honcho/sift, 30s for node_exporter.

## Alerting

```
vmalert  →  Alertmanager  →  Mattermost incoming webhook
                         ↳  ntfy (optional fallback)
```

NixOS: `services.vmalert.instances."".enable` + `services.prometheus.alertmanager`.

vmalert settings:

- `datasource.url = http://127.0.0.1:8428`
- `notifier.url = [ http://127.0.0.1:9093 ]`
- rules from this repo (`configs/metrics/alerts/*.yml` or the `rules` nix option)

Alertmanager listen `127.0.0.1:9093` (orion-local only; nothing else needs it). Receiver is a Slack-compatible incoming webhook (Mattermost accepts that payload).

### Prerequisite (Kyle)

Create a Mattermost incoming webhook. Write the URL to **`/root/mattermost-webhook`** on orion, mode 0600, root-owned. Channel is Kyle's call (`~ops` / `~dev` / a new `~alerts`).

This plan does not create the webhook. Do not `cat` the file, copy it, or put the URL in Nix, git, or a log.

The NixOS Alertmanager unit uses `DynamicUser`, so it cannot read a root-owned file directly. Phase 1 wires it with `LoadCredential`:

```nix
systemd.services.alertmanager.serviceConfig.LoadCredential = [
  "mattermost-webhook:/root/mattermost-webhook"
];
```

Receiver (Alertmanager config, not the store):

```yaml
slack_configs:
  - api_url_file: /run/credentials/alertmanager.service/mattermost-webhook
    channel: "#alerts"   # whatever Kyle picked
    send_resolved: true
```

systemd injects the file at unit start. The URL never enters the Nix store.

### ntfy fallback

Second Alertmanager receiver, same routes, posting to an ntfy topic Kyle already uses (or a new `fleet-alerts` topic). Purpose: pages when Mattermost on straylight is itself down. First cut can ship MM-only and add ntfy in the same PR if the topic exists; otherwise a follow-up. No self-hosted ntfy unless Kyle wants it.

## Exposure

Every exporter:

- listen `0.0.0.0:<port>` (matches dixie llama / tether today) **or** the Tailscale IP
- firewall hole **only** on `tailscale0`
- nothing on LAN, nothing on the internet
- no `tailscale serve` for scrape ports (orion SearXNG/sift already occupy serve)

Sift is the exception that already sits behind Tailscale Serve on `:8443`. Scrape `https://orion.scylla-hammerhead.ts.net:8443/metrics` with `scheme: https` and `bearer_token_file`. Do not open 8890 on the tailnet for metrics.

Honcho deriver `:9090` is currently unpublished. Publishing it is a compose bind to the Tailscale IP (or `tailscale0` firewall + `0.0.0.0:9090` inside the container network). Not NixOS.

## Targets

Ports (first cut):

| port | what |
|---|---|
| 9100 | node_exporter |
| 9835 | nvidia_gpu_exporter (dixie) |
| 11433 / 11434 | llama-server `--metrics` (existing API ports) |
| 11435 | tether `/tether/metrics` |
| 8000 | Honcho api `/metrics` |
| 9090 | Honcho deriver `/metrics` |
| 8443 | sift `/metrics` (Prometheus once P2.7 lands) |
| 8428 | VictoriaMetrics (self-scrape + vmui) |

### dixie (phase 1)

- node_exporter: hwmon, systemd, filesystem, netdev. `collector.systemd`.
- `nvidia_gpu_exporter` (NixOS `services.prometheus.exporters.nvidia-gpu` — confirm exact option name at implement). RTX 3060, power-capped 140W.
- llama-server **and** llama-vision: add `--metrics` to `llamaService` in `hw/dixie/default.nix`. Same port as the API; no extra hole. Restarts the routers — do this with the dixie deploy, not ad hoc.
- tether: scrape `http://dixie:11435/tether/metrics` as-is.

Tether follow-up (not a phase-1 blocker; rule 2 needs the labelled series): in-flight gauge **by model**, converted-request duration histogram, cancel `reason` (`client_disconnect` / `shutdown` / `other` only — never `upstream_error`). Unlabelled totals remain; scrape queries must always select a label.

### straylight (phase 3)

- node_exporter with `collector.hwmon` (amdgpu is `hwmon9`) and `collector.drm`.
- GTT: node_exporter does not export `mem_info_gtt_*`. Tiny textfile collector (systemd timer → `/var/lib/node_exporter/gtt.prom`) reading `/sys/class/drm/card1/device/mem_info_gtt_{total,used}`.
- llama-server `--metrics` on `:11434`. Same restart caveat.

### rift (phase 2) — no SQL exporter

Honcho 3.2.0 already has Prometheus metrics, off by default.

- api: `GET /metrics` on `:8000` (already tailnet-reachable).
- deriver: `start_http_server(9090)` inside the container; port is **not** published in compose yet.

Series we will use: `deriver_queue_items_pending`, `deriver_queue_oldest_pending_age_seconds`, `deriver_queue_work_units_claimed` / `eligible`, `dialectic_calls`, `db_connections_open`, `db_queries_in_flight`.

**Enable (scheduled, not ad hoc):** `METRICS_ENABLED=true`, publish deriver `:9090` on the tailnet only, recreate **api and deriver**. This is a Honcho restart.

**Do not retag images.** api is `3.2.0-pr1092-7e57fe2`; deriver is digest-pinned. Metrics work does not touch either.

Also node_exporter on rift.

### orion (phase 1)

- node_exporter
- VictoriaMetrics + vmalert + alertmanager
- sift: scrape **after P2.7**. Target `https://orion.scylla-hammerhead.ts.net:8443/metrics` with `bearer_token_file` (dedicated `metrics` caller in `keys.toml`: no workspace, no fetch/search). Loopback `/metrics` stays open without the bearer. Restricted callers are aggregated only (no per-caller series, no `tavily:restricted` gauge) — that is titan/P2.7, this scrape config just consumes it.

### talos, titan, hyperborea (phase 3)

node_exporter only, on whatever package path each host already uses. Not this flake.

## Alert rules (first five)

Freeze series names against the first successful scrape. Thresholds below are the intent.

### 1. Dixie 9B slots 100% busy > 10m

Tonight: 8 slots, unified 128k pool, Honcho + three guardians.

```yaml
- alert: Dixie9BSlotsSaturated
  expr: |
    min_over_time(
      llamacpp:requests_processing{job="llama-dixie", model="ornith-1.5-9b-uncensored"}[10m]
    ) >= 8
  for: 10m
  labels: { severity: page }
  annotations:
    summary: "dixie 9B slots busy for 10m"
```

If router-mode metrics lack a `model` label, scrape `:11434` and `:11433` as separate jobs and match on job. Confirm `llamacpp:requests_processing` vs a slots-busy gauge on first scrape. `parallel = 8` is the threshold; if that changes, the rule changes.

### 2. Tether cancelled/converted > 50% over 15m

Live at plan time (unlabelled counters): cancelled/converted ≈ 89% (8479/9493). The one-to-one form below **never fires**: `rate(tether_cancelled_total[15m]) / rate(tether_requests_total{kind="converted"}[15m])` matches `{}` against `{kind="converted"}` and returns empty.

Cancels are **client-side only** (`reason=client_disconnect`, `shutdown`, or `other` for an unattributed context cancel). Upstream failures stay on `tether_upstream_errors_total` and must not be in the cancel total — otherwise this rule pages on dixie errors.

Tether exposes an unlabelled sum **next to** the labelled series. **Never `sum()` a tether series without a label selector** or you double-count.

```yaml
- alert: TetherCancelStorm
  expr: |
    sum(rate(tether_cancelled_total{reason="client_disconnect"}[15m]))
    /
    clamp_min(sum(rate(tether_requests_total{kind="converted"}[15m])), 1e-6)
    > 0.5
    and
    sum(rate(tether_requests_total{kind="converted"}[15m])) > 0.02
  for: 15m
  labels: { severity: page }
```

### 3. Honcho pending queue (native metrics)

Not a SQL exporter.

```yaml
- alert: HonchoDeriverBacklog
  expr: |
    min_over_time(deriver_queue_items_pending[30m]) > 100
    and
    deriv(deriver_queue_oldest_pending_age_seconds[30m]) > 0
  for: 30m
  labels: { severity: page }
  annotations:
    summary: "Honcho pending > 100 and oldest age still rising"
```

Needs phase 2 (`METRICS_ENABLED=true`). Do not enable the rule before the scrape exists.

### 4. llama-server restarts or KV overflow

KV overflow (`failed to find free space in the KV cache` / `Context size has been exceeded`) is **not** a Prometheus series today. First cut uses proxies:

```yaml
- alert: LlamaServerRestarting
  expr: increase(node_systemd_unit_restarts_total{name=~"llama-server.service|llama-vision.service"}[15m]) > 0
  for: 0m
  labels: { severity: page }

- alert: LlamaKVSaturated
  expr: llamacpp:kv_cache_usage_ratio > 0.99
  for: 5m
  labels: { severity: page }
```

Journal-derived counter for the overflow string is a later add (textfile or mtail), not first cut. `requests_deferred > 0` for 5m is a useful extra if the series exists.

### 5. GPU memory > 95% or temperature above threshold

```yaml
- alert: GpuMemoryHigh
  expr: |
    (nvidia_gpu_memory_used_bytes / nvidia_gpu_memory_total_bytes) > 0.95
    or
    (gtt_used_bytes / gtt_total_bytes) > 0.95
  for: 5m
  labels: { severity: page }

- alert: GpuHot
  expr: |
    nvidia_gpu_temperature_celsius > 83
    or
    node_hwmon_temp_celsius{chip=~"amdgpu.*"} > 90
  for: 2m
  labels: { severity: page }
```

- dixie RTX 3060: 83 °C (card is power-capped 140W; throttle band is ~83–93).
- straylight 8060S: 90 °C edge. GTT 95% of ~104 GiB.

## Scrape config sketch

Jobs added per phase, not all on day one.

```yaml
scrape_configs:
  - job_name: vm
    static_configs: [{ targets: ["127.0.0.1:8428"] }]

  - job_name: node
    scrape_interval: 30s
    static_configs:
      - { targets: ["orion:9100"], labels: { host: orion } }
      - { targets: ["dixie:9100"], labels: { host: dixie } }
      # later: straylight, rift, talos, titan, hyperborea

  - job_name: nvidia-dixie
    static_configs: [{ targets: ["dixie:9835"] }]

  - job_name: llama-dixie
    metrics_path: /metrics
    static_configs:
      - { targets: ["dixie:11434"], labels: { router: helper } }
      - { targets: ["dixie:11433"], labels: { router: vision } }

  - job_name: tether
    metrics_path: /tether/metrics
    static_configs: [{ targets: ["dixie:11435"] }]

  - job_name: honcho-api        # phase 2
    static_configs: [{ targets: ["rift:8000"] }]

  - job_name: honcho-deriver    # phase 2
    static_configs: [{ targets: ["rift:9090"] }]

  - job_name: sift              # after P2.7
    scheme: https
    metrics_path: /metrics
    bearer_token_file: /var/lib/victoriametrics/sift-metrics.token
    static_configs: [{ targets: ["orion.scylla-hammerhead.ts.net:8443"] }]
```

MagicDNS names, not Tailscale IPs. Sift token file is the `metrics` caller key; mode 0600; not in git.

## Nix layout (when implementing)

Not this PR.

- `configs/metrics/exporters.nix` — node_exporter + tailscale0 hole, imported per host with a boolean.
- `configs/metrics/alerts/*.yml` — the five rules.
- `hw/orion/metrics.nix` — VM, vmalert, alertmanager (`LoadCredential` → `/root/mattermost-webhook`), scrape config, sift token path.
- `hw/dixie/default.nix` — `--metrics` on both llama units, nvidia exporter, 9100/9835 on tailscale0.
- `hw/straylight/default.nix` — `--metrics`, GTT textfile, 9100 on tailscale0.
- rift compose (on rift, not this flake) — `METRICS_ENABLED`, publish 9090.

`rebuild-nixos` never deploys uncommitted trees. Orion/dixie switches follow the existing host paths (orion local `/etc/nixos`, dixie copy+switch).

## Order

1. **orion + dixie.** VM + vmalert + alertmanager + node_exporter on both. Dixie nvidia exporter + llama `--metrics` + tether scrape. Rules 1, 2, 4 (restart half), 5 (nvidia half). Mattermost webhook must exist first or alerts go nowhere.
2. **rift.** Scheduled Honcho recreate with `METRICS_ENABLED=true` and deriver `:9090` on the tailnet. Then rule 3. Do not bump images.
3. **the rest.** straylight (llama `--metrics`, hwmon, GTT), then talos/titan/hyperborea node_exporter. Sift scrape as soon as P2.7 is on orion (can land during 1 or 3; needs the bearer file on orion).

Tether histogram / in-flight-by-model / cancel-reason: after the plan is accepted, small tether change, not a blocker for phase 1 (rule 2 works on the counters that already exist).

## Out of scope

- Grafana
- SQL exporter on Honcho Postgres
- Changing Honcho image pins
- Creating the Mattermost webhook
- LAN or public scrape
- Journal mtail for KV overflow strings
- Deploying any of the above from this document