kyle/imladris
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

docs: fix tether cancel-storm PromQL (rule 2)

Browse Source 
Label-set mismatch made the ratio empty. Select reason=client_disconnect
and kind=converted; never sum() a tether series without a selector.
This commit is contained in:
kyle
2026-09-23 22:50:26 -07:00
parent 8b14e1f99f
commit c07c4bfe4f
1 changed files with 12 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/metrics-plan.md
+11 -8
View File
@@ -129,7 +129,7 @@ Ports (first cut):
- llama-server **and** llama-vision: add `--metrics` to `llamaService` in `hw/dixie/default.nix`. Same port as the API; no extra hole. Restarts the routers — do this with the dixie deploy, not ad hoc. - llama-server **and** llama-vision: add `--metrics` to `llamaService` in `hw/dixie/default.nix`. Same port as the API; no extra hole. Restarts the routers — do this with the dixie deploy, not ad hoc.
- tether: scrape `http://dixie:11435/tether/metrics` as-is. - tether: scrape `http://dixie:11435/tether/metrics` as-is.
Tether follow-up (small change, after this plan lands, not phase 1): in-flight gauge **by model**, converted-request duration histogram, cancel reason label (`client_disconnect` vs `upstream_error`). Today `tether_cancelled_total` is a single counter and `tether_in_flight` has no labels. Tether follow-up (not a phase-1 blocker; rule 2 needs the labelled series): in-flight gauge **by model**, converted-request duration histogram, cancel `reason` (`client_disconnect` / `shutdown` / `other` only — never `upstream_error`). Unlabelled totals remain; scrape queries must always select a label.
### straylight (phase 3) ### straylight (phase 3)
@@ -186,22 +186,25 @@ If router-mode metrics lack a `model` label, scrape `:11434` and `:11433` as sep
### 2. Tether cancelled/converted > 50% over 15m ### 2. Tether cancelled/converted > 50% over 15m
Live right now cancelled/converted ≈ 89% (8479/9493). This rule would already be firing — that is the point. Live at plan time (unlabelled counters): cancelled/converted ≈ 89% (8479/9493). The one-to-one form below **never fires**: `rate(tether_cancelled_total[15m]) / rate(tether_requests_total{kind="converted"}[15m])` matches `{}` against `{kind="converted"}` and returns empty.
Cancels are **client-side only** (`reason=client_disconnect`, `shutdown`, or `other` for an unattributed context cancel). Upstream failures stay on `tether_upstream_errors_total` and must not be in the cancel total — otherwise this rule pages on dixie errors.
Tether exposes an unlabelled sum **next to** the labelled series. **Never `sum()` a tether series without a label selector** or you double-count.
```yaml ```yaml
- alert: TetherCancelStorm - alert: TetherCancelStorm
expr: | expr: |
( sum(rate(tether_cancelled_total{reason="client_disconnect"}[15m]))
rate(tether_cancelled_total[15m])
/ /
clamp_min(rate(tether_requests_total{kind="converted"}[15m]), 1e-6) clamp_min(sum(rate(tether_requests_total{kind="converted"}[15m])), 1e-6)
) > 0.5 > 0.5
and
sum(rate(tether_requests_total{kind="converted"}[15m])) > 0.02
for: 15m for: 15m
labels: { severity: page } labels: { severity: page }
``` ```
`tether_cancelled_total` currently counts converted **or** proxied cancels (HELP text). Until the reason label exists, compare against converted only; a follow-up splits `client_disconnect` vs `upstream_error`.
### 3. Honcho pending queue (native metrics) ### 3. Honcho pending queue (native metrics)
Not a SQL exporter. Not a SQL exporter.