diff --git a/docs/P2.7.md b/docs/P2.7.md new file mode 100644 index 0000000..02060a8 --- /dev/null +++ b/docs/P2.7.md @@ -0,0 +1,232 @@ +# Sift P2.7 — Prometheus `/metrics` + +Drafted 2026-09-23 by titan. Reviewer: Claude. Implementer: titan after this +doc is approved. Orion pin: straylight, after Claude approves the code. +Builds on live P1/P2/P3 (`e62cfc8`): JSON `/metrics` already exists; this +changes the default exposition so VictoriaMetrics can scrape sift. + +No deploy in this phase. No nixos edit from titan. + +## 1. Scope and exit criteria + +In: Prometheus text on `GET /metrics`; keep today's JSON; the counters/gauges +Claude listed (requests, guard blocks, digest + Dixie gate, dcache/search +cache, search queries/credits, Tavily daily credits + capped); duration +histograms for fetch, digest, and search; label redaction matching restricted +logs. + +Out: Grafana; a new listen address or firewall (straylight's metrics-plan / +nixos); alert rules (those live in `~/src/nixos` docs, not here); PLAN's +"bot-blocked per host" series (host labels are forbidden); P2 §6 recover / +token-ratio / Honcho-context series (not in this cut); Go/process collectors. + +**Exit criteria** (evidence in this thread): + +1. `make test lint` green (new tests in §7). `make contract` unchanged. +2. `GET /metrics` is Prometheus text 0.0.4 and parses as such. +3. `GET /metrics.json` and `GET /metrics?format=json` still return today's JSON + object (existing keys, same types). `TestMetricsCacheHitMiss` moves off + bare `/metrics`. +4. After one search and one digest-fallback, the Prometheus body contains the + series in §4 with the labels in §5 — and **no** query text, URL, hostname, + or IP. +5. Restricted caller: `caller` label is the keys.toml name (same as the log + `caller=` field). No `host` label anywhere. + +## 2. Current `/metrics` (verified in tree, not assumed) + +`internal/api/api.go` `GET /metrics` is unauthenticated JSON: + +```json +{"requests": N, "blocks": N, "scrapes": N, "self_ip_failures": N, + "search_queries": {caller: {source: n}}, + "search_credits": {caller: {source: n}}, + "search_errors": {source: {kind: n}}, + "search_cache_hit": N, "search_cache_miss": N, + "search_rerank_fallback": N, + "search_daily_credits": {source: n}, + "search_capped": {source: n}} +``` + +Digest `Engine.Calls` / `Fallback`, `digest.Gate` inflight/waiters, and +`dcache` hit/miss/evictions exist in process memory and are **not** on the +JSON endpoint today. P2.7 exports them (Prometheus + additive JSON fields). + +Search `addErr` currently uses the warning tail after the last colon as +`kind`. That is fine for JSON compatibility; it is **not** safe as a +Prometheus label (a slipped hostname becomes a series). Prometheus `kind` +is a closed set (§5). JSON keeps the current maps. + +Listener stays `127.0.0.1:8890` + existing `tailscale serve` HTTPS. Scrape +URL and tailscale0 bind are straylight's metrics-plan, not this doc. + +## 3. Wire format + +| Request | Response | +|---|---| +| `GET /metrics` | Prometheus text 0.0.4, `Content-Type: text/plain; version=0.0.4; charset=utf-8` | +| `GET /metrics.json` | today's JSON, `Content-Type: application/json` | +| `GET /metrics?format=json` | same JSON as `/metrics.json` | + +`format` values other than `json` (including absent) → Prometheus text. +Unauthenticated, same as `/healthz`. Do not count scrapes of `/metrics` or +`/healthz` in `sift_requests_total`. + +Private `prometheus.Registry` with only the `sift_*` series below. Do not +register client_golang's default Go/process collectors. + +Dep: `github.com/prometheus/client_golang` (vendored like the rest). No +third exposition library. + +## 4. Series + +Prefix `sift_`. Counters end `_total`. Histograms are seconds. + +**Requests** + +- `sift_requests_total{endpoint,code}` counter — `/v1/fetch`, `/v2/scrape`, + `/v1/search`, `/v2/search`. `code` is the HTTP status as a string (`200`, + `401`, `422`, …). Increment once per finished handler, including 401. + +**Guard** + +- `sift_guard_blocks_total` counter — same events as today's `blocks` + (SSRF/guard 403). No host, URL, or reason-with-host label. + +**Digest and Dixie gate** (`internal/digest`) + +- `sift_digest_calls_total` counter — Dixie chat actually started (today's + `Engine.Calls`; cache hits do not increment). +- `sift_digest_fallbacks_total{reason}` counter — `Engine.Fallback` keys. + Closed `reason` set: `queue_full`, `queue_wait`, `canceled`, `embed`, + `dixie`, `timeout`, `empty`. Unknown → `other` (never the raw string). +- `sift_dixie_inflight` gauge — `Gate` slots in use. +- `sift_dixie_queue_waiters` gauge — waiters in the sift-side FIFO (not + llama-server's queue). + +`Gate` grows a `Snapshot() (inflight, waiters int)` under the existing mutex. + +**Caches** + +- `sift_cache_requests_total{cache,result}` counter — `cache` is `digest`, + `embed`, or `search`; `result` is `hit` or `miss`. Search uses the engine + counters already on JSON (`search_cache_hit` / `search_cache_miss`), not a + second dcache pair, so the two expositions cannot diverge. + +**Search cost** (counts only may carry `caller`) + +- `sift_search_queries_total{caller,source}` counter +- `sift_search_credits_total{caller,source}` counter +- `sift_search_daily_credits{source}` gauge — current UTC-day total (the + in-process / `search_cap` value). Gauge, not counter: it is a day bucket + and must not look like a reset counter to Prometheus. +- `sift_search_capped_total{source}` counter + +`source` values are the configured source names (`tavily`, `brave`, +`searxng`, …) plus the existing cap partition `tavily:restricted`. No +query, host, or URL. + +**Histograms** (no `caller`) + +- `sift_request_duration_seconds{op}` — `op` is `fetch`, `digest`, or + `search`. + - `fetch`: wall time of `/v1/fetch` and `/v2/scrape` (one series, both + endpoints). + - `digest`: `digest.Result.DigestMS` on digest attempts, including + fallbacks; skip pure cache hits. + - `search`: `RunResult.SearchMS` on `/v1/search` and `/v2/search`. + +Buckets (seconds): `fetch` `0.1, 0.5, 1, 2, 5, 10, 20, 30`; `digest` +`1, 5, 10, 15, 20, 30, 45`; `search` `0.5, 1, 2, 5, 10, 20, 30`. Native +histograms out of this cut. + +**JSON additive fields** (same endpoint as today, not a rename): + +- `digest_calls`, `digest_fallbacks` (map reason→n), `dixie_inflight`, + `dixie_queue_waiters` +- `dcache_digest_hit`, `dcache_digest_miss`, `dcache_embed_hit`, + `dcache_embed_miss` + +Existing JSON keys stay. `self_ip_failures` and `search_rerank_fallback` +stay on JSON; they are not required on Prometheus in this cut. + +## 5. Labels and redaction + +Same isolation as restricted logs (P2.5 / P3 §8): metrics are a second log. + +- **`caller` only on count series** (`sift_search_queries_total`, + `sift_search_credits_total`). Never on histograms, gauges, cache, guard, + digest, or `sift_requests_total`. +- **`caller` value** is `auth.Caller.Name` from keys.toml, including + `permit_restricted` callers — the same string the INFO line uses. Never a + query, URL, key, workspace id, or `restricted` stand-in. +- **No `host`, `url`, `query`, `q`, `final_url`, `err`, or `msg` labels.** + PLAN §4.3 "bot-blocked per host" is deferred until there is a host-less + encoding (or it stays logs-only). +- **`endpoint`**: path only, the four API routes above. +- **`code`**: three-digit HTTP status. +- **`reason` / `cache` / `result` / `op` / `source`**: closed sets from §4. +- **Search error kinds are not Prometheus labels in this cut** (the JSON + `search_errors` map can still hold warning tails). If a later cut adds + `sift_search_errors_total{source,kind}`, `kind` is closed: + `timeout` | `error` | `4xx` | `5xx` | `429` | `other`. Never `addErr`'s + raw tail. + +A test dumps every metric name, label name, and label value from a handler +that ran a restricted search + a blocked fetch, and asserts the query, the +URL, and the hostname/IP appear **nowhere** in the Prometheus body (including +HELP text). + +## 6. Code layout + +``` +internal/metrics Registry, sift_* defs, JSON snapshot helper +internal/digest Gate.Snapshot; Fallback/Calls already there +internal/api /metrics negotiates format; /metrics.json + handlers observe duration + requests{endpoint,code} +``` + +`internal/metrics` owns the registry so tests can scrape a `Server` without +a global DefaultRegisterer. JSON continues to read the same atomics/maps +the Prometheus collectors scrape — one set of counters, two encodings. + +Do not log `/metrics` scrapes at INFO. + +## 7. Tests + +- `GET /metrics` Content-Type and `expfmt` parse; `HELP`/`TYPE` present for + each series in §4. +- `GET /metrics.json` and `?format=json`: `search_cache_hit` / `_miss` still + numbers; existing `TestMetricsCacheHitMiss` pointed at `/metrics.json`. +- One `/v1/search` then Prometheus `sift_search_queries_total{caller,source} == 1`. +- Digest gate full → `sift_digest_fallbacks_total{reason="queue_full"}` (or + `queue_wait`) and `sift_dixie_inflight` / `sift_dixie_queue_waiters` are + gauges in range. +- Restricted caller fixture: Prometheus body has `caller=""` and does + not contain the query, any URL host, or any IP. Fleet caller on the same + query does not create extra label names. +- Histogram: a timed fake fetch observes `_count` and a bucket. +- Unknown fallback reason coerced to `other`. +- `/metrics` itself does not increment `sift_requests_total`. + +No live orion scrape in CI. Post one `curl -sS https://orion…/metrics | head` +after the pin bump (straylight), not as a merge gate. + +## 8. Rollout + +1. This doc reviewed by Claude (label redaction + JSON compatibility). +2. titan implements on `git.wntrmute.dev:kyle/sift`; evidence in this thread. +3. Claude reviews the code. +4. straylight bumps the orion pin once (batch with other approved isolation + patches if any are waiting). Titan does not edit `~/src/nixos`. +5. straylight's metrics-plan lists sift as a scrape target on the existing + HTTPS serve (or a later tailscale0 exporter port — that choice is not + made here). + +## 9. Open questions + +1. Scrape `https://orion.scylla-hammerhead.ts.net:8443/metrics` vs a separate + tailscale0 exporter port — straylight's metrics-plan, not this spec. +2. Whether JSON should also grow `search_rerank_fallback` / `self_ip_failures` + onto Prometheus in this cut (left off above). +)