imladris/docs/metrics-plan.md

Fleet metrics and alerting

Plan only. No deploy from this document.

Goal: page before a person notices the four failure modes from 2026-09-23 — Dixie 9B slot starvation, tether cancel storm, Honcho deriver backlog, llama-server KV overflow.

Query UI is vmui. Grafana is optional and out of the first cut.

Current state (checked 2026-09-23, straylight)

Nothing in this flake scrapes or exports Prometheus today. GET results:

endpoint	result
dixie :11434/metrics, straylight :11434/metrics	400 model name is missing — --metrics is off (llama.cpp 0.4.1, flag exists, default disabled)
dixie :11435/tether/metrics	200 Prometheus text. Live: converted 9493, cancelled 8479, passthrough 2833, upstream errors 307. In-flight is a single gauge, not by model
rift :8000/metrics	Metrics are disabled (METRICS_ENABLED=false)
orion sift :8443/metrics	200 JSON (P2.7 not landed)
orion disk	953.9G INTEL NVMe, root 937G, 353G free (61%)
straylight GTT	mem_info_gtt_total=111669149696 (~104 GiB), used ~26 GiB at probe. hwmon amdgpu is hwmon9

talos, titan, and hyperborea are not in this flake. Node-exporter there is out-of-band.

Tether path is /tether/metrics, not /metrics. Scrape that.

Store

VictoriaMetrics single-node on orion. NixOS module already in nixpkgs 26.05: services.victoriametrics.

services.victoriametrics = {
  enable = true;
  listenAddress = "127.0.0.1:8428";  # plus tailscale0 via firewall, or bind TS IP
  retentionPeriod = "90d";
  prometheusConfig = { /* scrape_configs below */ };
  extraOptions = [
    "-storageDataPath=/var/lib/victoriametrics"
    "-promscrape.maxScrapeSize=16MB"
  ];
};

The module already passes -promscrape.config= when prometheusConfig is set. No vmagent.

Listen: same pattern as dixie llama-server. Bind 0.0.0.0:8428 (or the Tailscale IP if we want stricter), open only networking.firewall.interfaces.tailscale0.allowedTCPPorts. Never networking.firewall.allowedTCPPorts (base already has 22 only). vmui: http://orion.scylla-hammerhead.ts.net:8428/vmui.

Retention / disk: 90 days. Budget 20 GiB under /var/lib/victoriametrics (~20k series × 15s × 90d is well under that; orion has 353G free). Alert when vm_data_size_bytes > 20GiB or root < 100GiB free. First cut shares the root NVMe; no extra disk.

Scrape interval: 15s for inference/tether/honcho/sift, 30s for node_exporter.

Alerting

vmalert  →  Alertmanager  →  Mattermost incoming webhook
                         ↳  ntfy (optional fallback)

NixOS: services.vmalert.instances."".enable + services.prometheus.alertmanager.

vmalert settings:

• datasource.url = http://127.0.0.1:8428
• notifier.url = [ http://127.0.0.1:9093 ]
• rules from this repo (configs/metrics/alerts/*.yml or the rules nix option)

Alertmanager listen 127.0.0.1:9093 (orion-local only; nothing else needs it). Receiver is a Slack-compatible incoming webhook (Mattermost accepts that payload).

Prerequisite (Kyle)

Create a Mattermost incoming webhook. Write the URL to /root/mattermost-webhook on orion, mode 0600, root-owned. Channel is Kyle's call (~ops / ~dev / a new ~alerts).

This plan does not create the webhook. Do not cat the file, copy it, or put the URL in Nix, git, or a log.

The NixOS Alertmanager unit uses DynamicUser, so it cannot read a root-owned file directly. Phase 1 wires it with LoadCredential:

systemd.services.alertmanager.serviceConfig.LoadCredential = [
  "mattermost-webhook:/root/mattermost-webhook"
];

Receiver (Alertmanager config, not the store):

slack_configs:
  - api_url_file: /run/credentials/alertmanager.service/mattermost-webhook
    channel: "#alerts"   # whatever Kyle picked
    send_resolved: true

systemd injects the file at unit start. The URL never enters the Nix store.

ntfy fallback

Second Alertmanager receiver, same routes, posting to an ntfy topic Kyle already uses (or a new fleet-alerts topic). Purpose: pages when Mattermost on straylight is itself down. First cut can ship MM-only and add ntfy in the same PR if the topic exists; otherwise a follow-up. No self-hosted ntfy unless Kyle wants it.

Exposure

Every exporter:

• listen 0.0.0.0:<port> (matches dixie llama / tether today) or the Tailscale IP
• firewall hole only on tailscale0
• nothing on LAN, nothing on the internet
• no tailscale serve for scrape ports (orion SearXNG/sift already occupy serve)

Sift is the exception that already sits behind Tailscale Serve on :8443. Scrape https://orion.scylla-hammerhead.ts.net:8443/metrics with scheme: https and bearer_token_file. Do not open 8890 on the tailnet for metrics.

Honcho deriver :9090 is currently unpublished. Publishing it is a compose bind to the Tailscale IP (or tailscale0 firewall + 0.0.0.0:9090 inside the container network). Not NixOS.

Targets

Ports (first cut):

port	what
9100	node_exporter
9835	nvidia_gpu_exporter (dixie)
11433 / 11434	llama-server --metrics (existing API ports)
11435	tether /tether/metrics
8000	Honcho api /metrics
9090	Honcho deriver /metrics
8443	sift /metrics (Prometheus once P2.7 lands)
8428	VictoriaMetrics (self-scrape + vmui)

dixie (phase 1)

• node_exporter: hwmon, systemd, filesystem, netdev. collector.systemd.
• nvidia_gpu_exporter (NixOS services.prometheus.exporters.nvidia-gpu — confirm exact option name at implement). RTX 3060, power-capped 140W.
• llama-server and llama-vision: add --metrics to llamaService in hw/dixie/default.nix. Same port as the API; no extra hole. Restarts the routers — do this with the dixie deploy, not ad hoc.
• tether: scrape http://dixie:11435/tether/metrics as-is.

Tether follow-up (small change, after this plan lands, not phase 1): in-flight gauge by model, converted-request duration histogram, cancel reason label (client_disconnect vs upstream_error). Today tether_cancelled_total is a single counter and tether_in_flight has no labels.

straylight (phase 3)

• node_exporter with collector.hwmon (amdgpu is hwmon9) and collector.drm.
• GTT: node_exporter does not export mem_info_gtt_*. Tiny textfile collector (systemd timer → /var/lib/node_exporter/gtt.prom) reading /sys/class/drm/card1/device/mem_info_gtt_{total,used}.
• llama-server --metrics on :11434. Same restart caveat.

rift (phase 2) — no SQL exporter

Honcho 3.2.0 already has Prometheus metrics, off by default.

• api: GET /metrics on :8000 (already tailnet-reachable).
• deriver: start_http_server(9090) inside the container; port is not published in compose yet.

Series we will use: deriver_queue_items_pending, deriver_queue_oldest_pending_age_seconds, deriver_queue_work_units_claimed / eligible, dialectic_calls, db_connections_open, db_queries_in_flight.

Enable (scheduled, not ad hoc): METRICS_ENABLED=true, publish deriver :9090 on the tailnet only, recreate api and deriver. This is a Honcho restart.

Do not retag images. api is 3.2.0-pr1092-7e57fe2; deriver is digest-pinned. Metrics work does not touch either.

Also node_exporter on rift.

orion (phase 1)

• node_exporter
• VictoriaMetrics + vmalert + alertmanager
• sift: scrape after P2.7. Target https://orion.scylla-hammerhead.ts.net:8443/metrics with bearer_token_file (dedicated metrics caller in keys.toml: no workspace, no fetch/search). Loopback /metrics stays open without the bearer. Restricted callers are aggregated only (no per-caller series, no tavily:restricted gauge) — that is titan/P2.7, this scrape config just consumes it.

talos, titan, hyperborea (phase 3)

node_exporter only, on whatever package path each host already uses. Not this flake.

Alert rules (first five)

Freeze series names against the first successful scrape. Thresholds below are the intent.

1. Dixie 9B slots 100% busy > 10m

Tonight: 8 slots, unified 128k pool, Honcho + three guardians.

- alert: Dixie9BSlotsSaturated
  expr: |
    min_over_time(
      llamacpp:requests_processing{job="llama-dixie", model="ornith-1.5-9b-uncensored"}[10m]
    ) >= 8
  for: 10m
  labels: { severity: page }
  annotations:
    summary: "dixie 9B slots busy for 10m"

If router-mode metrics lack a model label, scrape :11434 and :11433 as separate jobs and match on job. Confirm llamacpp:requests_processing vs a slots-busy gauge on first scrape. parallel = 8 is the threshold; if that changes, the rule changes.

2. Tether cancelled/converted > 50% over 15m

Live right now cancelled/converted ≈ 89% (8479/9493). This rule would already be firing — that is the point.

- alert: TetherCancelStorm
  expr: |
    (
      rate(tether_cancelled_total[15m])
      /
      clamp_min(rate(tether_requests_total{kind="converted"}[15m]), 1e-6)
    ) > 0.5
  for: 15m
  labels: { severity: page }

tether_cancelled_total currently counts converted or proxied cancels (HELP text). Until the reason label exists, compare against converted only; a follow-up splits client_disconnect vs upstream_error.

3. Honcho pending queue (native metrics)

Not a SQL exporter.

- alert: HonchoDeriverBacklog
  expr: |
    min_over_time(deriver_queue_items_pending[30m]) > 100
    and
    deriv(deriver_queue_oldest_pending_age_seconds[30m]) > 0
  for: 30m
  labels: { severity: page }
  annotations:
    summary: "Honcho pending > 100 and oldest age still rising"

Needs phase 2 (METRICS_ENABLED=true). Do not enable the rule before the scrape exists.

4. llama-server restarts or KV overflow

KV overflow (failed to find free space in the KV cache / Context size has been exceeded) is not a Prometheus series today. First cut uses proxies:

- alert: LlamaServerRestarting
  expr: increase(node_systemd_unit_restarts_total{name=~"llama-server.service|llama-vision.service"}[15m]) > 0
  for: 0m
  labels: { severity: page }

- alert: LlamaKVSaturated
  expr: llamacpp:kv_cache_usage_ratio > 0.99
  for: 5m
  labels: { severity: page }

Journal-derived counter for the overflow string is a later add (textfile or mtail), not first cut. requests_deferred > 0 for 5m is a useful extra if the series exists.

5. GPU memory > 95% or temperature above threshold

- alert: GpuMemoryHigh
  expr: |
    (nvidia_gpu_memory_used_bytes / nvidia_gpu_memory_total_bytes) > 0.95
    or
    (gtt_used_bytes / gtt_total_bytes) > 0.95
  for: 5m
  labels: { severity: page }

- alert: GpuHot
  expr: |
    nvidia_gpu_temperature_celsius > 83
    or
    node_hwmon_temp_celsius{chip=~"amdgpu.*"} > 90
  for: 2m
  labels: { severity: page }

• dixie RTX 3060: 83 °C (card is power-capped 140W; throttle band is ~83–93).
• straylight 8060S: 90 °C edge. GTT 95% of ~104 GiB.

Scrape config sketch

Jobs added per phase, not all on day one.

scrape_configs:
  - job_name: vm
    static_configs: [{ targets: ["127.0.0.1:8428"] }]

  - job_name: node
    scrape_interval: 30s
    static_configs:
      - { targets: ["orion:9100"], labels: { host: orion } }
      - { targets: ["dixie:9100"], labels: { host: dixie } }
      # later: straylight, rift, talos, titan, hyperborea

  - job_name: nvidia-dixie
    static_configs: [{ targets: ["dixie:9835"] }]

  - job_name: llama-dixie
    metrics_path: /metrics
    static_configs:
      - { targets: ["dixie:11434"], labels: { router: helper } }
      - { targets: ["dixie:11433"], labels: { router: vision } }

  - job_name: tether
    metrics_path: /tether/metrics
    static_configs: [{ targets: ["dixie:11435"] }]

  - job_name: honcho-api        # phase 2
    static_configs: [{ targets: ["rift:8000"] }]

  - job_name: honcho-deriver    # phase 2
    static_configs: [{ targets: ["rift:9090"] }]

  - job_name: sift              # after P2.7
    scheme: https
    metrics_path: /metrics
    bearer_token_file: /var/lib/victoriametrics/sift-metrics.token
    static_configs: [{ targets: ["orion.scylla-hammerhead.ts.net:8443"] }]

MagicDNS names, not Tailscale IPs. Sift token file is the metrics caller key; mode 0600; not in git.

Nix layout (when implementing)

Not this PR.

• configs/metrics/exporters.nix — node_exporter + tailscale0 hole, imported per host with a boolean.
• configs/metrics/alerts/*.yml — the five rules.
• hw/orion/metrics.nix — VM, vmalert, alertmanager (LoadCredential → /root/mattermost-webhook), scrape config, sift token path.
• hw/dixie/default.nix — --metrics on both llama units, nvidia exporter, 9100/9835 on tailscale0.
• hw/straylight/default.nix — --metrics, GTT textfile, 9100 on tailscale0.
• rift compose (on rift, not this flake) — METRICS_ENABLED, publish 9090.

rebuild-nixos never deploys uncommitted trees. Orion/dixie switches follow the existing host paths (orion local /etc/nixos, dixie copy+switch).

Order

1. orion + dixie. VM + vmalert + alertmanager + node_exporter on both. Dixie nvidia exporter + llama --metrics + tether scrape. Rules 1, 2, 4 (restart half), 5 (nvidia half). Mattermost webhook must exist first or alerts go nowhere.
2. rift. Scheduled Honcho recreate with METRICS_ENABLED=true and deriver :9090 on the tailnet. Then rule 3. Do not bump images.
3. the rest. straylight (llama --metrics, hwmon, GTT), then talos/titan/hyperborea node_exporter. Sift scrape as soon as P2.7 is on orion (can land during 1 or 3; needs the bearer file on orion).

Tether histogram / in-flight-by-model / cancel-reason: after the plan is accepted, small tether change, not a blocker for phase 1 (rule 2 works on the counters that already exist).

Out of scope

• Grafana
• SQL exporter on Honcho Postgres
• Changing Honcho image pins
• Creating the Mattermost webhook
• LAN or public scrape
• Journal mtail for KV overflow strings
• Deploying any of the above from this document