From c07c4bfe4f092aecebdd85f1b23b66cccb34bca0 Mon Sep 17 00:00:00 2001
From: "K. Isom" <kyle@imap.cc>
Date: Wed, 23 Sep 2026 22:50:26 -0700
Subject: [PATCH] docs: fix tether cancel-storm PromQL (rule 2)

Label-set mismatch made the ratio empty. Select reason=client_disconnect
and kind=converted; never sum() a tether series without a selector.
---
 docs/metrics-plan.md | 21 ++++++++++++---------
 1 file changed, 12 insertions(+), 9 deletions(-)

diff --git a/docs/metrics-plan.md b/docs/metrics-plan.md
index a8174c0..cd327ad 100644
--- a/docs/metrics-plan.md
+++ b/docs/metrics-plan.md
@@ -129,7 +129,7 @@ Ports (first cut):
 - llama-server **and** llama-vision: add `--metrics` to `llamaService` in `hw/dixie/default.nix`. Same port as the API; no extra hole. Restarts the routers — do this with the dixie deploy, not ad hoc.
 - tether: scrape `http://dixie:11435/tether/metrics` as-is.
 
-Tether follow-up (small change, after this plan lands, not phase 1): in-flight gauge **by model**, converted-request duration histogram, cancel reason label (`client_disconnect` vs `upstream_error`). Today `tether_cancelled_total` is a single counter and `tether_in_flight` has no labels.
+Tether follow-up (not a phase-1 blocker; rule 2 needs the labelled series): in-flight gauge **by model**, converted-request duration histogram, cancel `reason` (`client_disconnect` / `shutdown` / `other` only — never `upstream_error`). Unlabelled totals remain; scrape queries must always select a label.
 
 ### straylight (phase 3)
 
@@ -186,22 +186,25 @@ If router-mode metrics lack a `model` label, scrape `:11434` and `:11433` as sep
 
 ### 2. Tether cancelled/converted > 50% over 15m
 
-Live right now cancelled/converted ≈ 89% (8479/9493). This rule would already be firing — that is the point.
+Live at plan time (unlabelled counters): cancelled/converted ≈ 89% (8479/9493). The one-to-one form below **never fires**: `rate(tether_cancelled_total[15m]) / rate(tether_requests_total{kind="converted"}[15m])` matches `{}` against `{kind="converted"}` and returns empty.
+
+Cancels are **client-side only** (`reason=client_disconnect`, `shutdown`, or `other` for an unattributed context cancel). Upstream failures stay on `tether_upstream_errors_total` and must not be in the cancel total — otherwise this rule pages on dixie errors.
+
+Tether exposes an unlabelled sum **next to** the labelled series. **Never `sum()` a tether series without a label selector** or you double-count.
 
 ```yaml
 - alert: TetherCancelStorm
   expr: |
-    (
-      rate(tether_cancelled_total[15m])
-      /
-      clamp_min(rate(tether_requests_total{kind="converted"}[15m]), 1e-6)
-    ) > 0.5
+    sum(rate(tether_cancelled_total{reason="client_disconnect"}[15m]))
+    /
+    clamp_min(sum(rate(tether_requests_total{kind="converted"}[15m])), 1e-6)
+    > 0.5
+    and
+    sum(rate(tether_requests_total{kind="converted"}[15m])) > 0.02
   for: 15m
   labels: { severity: page }
 ```
 
-`tether_cancelled_total` currently counts converted **or** proxied cancels (HELP text). Until the reason label exists, compare against converted only; a follow-up splits `client_disconnect` vs `upstream_error`.
-
 ### 3. Honcho pending queue (native metrics)
 
 Not a SQL exporter.