docs: P2.7 prometheus /metrics
Design only. JSON stays at /metrics.json and ?format=json.
This commit is contained in:
+232
@@ -0,0 +1,232 @@
|
||||
# Sift P2.7 — Prometheus `/metrics`
|
||||
|
||||
Drafted 2026-09-23 by titan. Reviewer: Claude. Implementer: titan after this
|
||||
doc is approved. Orion pin: straylight, after Claude approves the code.
|
||||
Builds on live P1/P2/P3 (`e62cfc8`): JSON `/metrics` already exists; this
|
||||
changes the default exposition so VictoriaMetrics can scrape sift.
|
||||
|
||||
No deploy in this phase. No nixos edit from titan.
|
||||
|
||||
## 1. Scope and exit criteria
|
||||
|
||||
In: Prometheus text on `GET /metrics`; keep today's JSON; the counters/gauges
|
||||
Claude listed (requests, guard blocks, digest + Dixie gate, dcache/search
|
||||
cache, search queries/credits, Tavily daily credits + capped); duration
|
||||
histograms for fetch, digest, and search; label redaction matching restricted
|
||||
logs.
|
||||
|
||||
Out: Grafana; a new listen address or firewall (straylight's metrics-plan /
|
||||
nixos); alert rules (those live in `~/src/nixos` docs, not here); PLAN's
|
||||
"bot-blocked per host" series (host labels are forbidden); P2 §6 recover /
|
||||
token-ratio / Honcho-context series (not in this cut); Go/process collectors.
|
||||
|
||||
**Exit criteria** (evidence in this thread):
|
||||
|
||||
1. `make test lint` green (new tests in §7). `make contract` unchanged.
|
||||
2. `GET /metrics` is Prometheus text 0.0.4 and parses as such.
|
||||
3. `GET /metrics.json` and `GET /metrics?format=json` still return today's JSON
|
||||
object (existing keys, same types). `TestMetricsCacheHitMiss` moves off
|
||||
bare `/metrics`.
|
||||
4. After one search and one digest-fallback, the Prometheus body contains the
|
||||
series in §4 with the labels in §5 — and **no** query text, URL, hostname,
|
||||
or IP.
|
||||
5. Restricted caller: `caller` label is the keys.toml name (same as the log
|
||||
`caller=` field). No `host` label anywhere.
|
||||
|
||||
## 2. Current `/metrics` (verified in tree, not assumed)
|
||||
|
||||
`internal/api/api.go` `GET /metrics` is unauthenticated JSON:
|
||||
|
||||
```json
|
||||
{"requests": N, "blocks": N, "scrapes": N, "self_ip_failures": N,
|
||||
"search_queries": {caller: {source: n}},
|
||||
"search_credits": {caller: {source: n}},
|
||||
"search_errors": {source: {kind: n}},
|
||||
"search_cache_hit": N, "search_cache_miss": N,
|
||||
"search_rerank_fallback": N,
|
||||
"search_daily_credits": {source: n},
|
||||
"search_capped": {source: n}}
|
||||
```
|
||||
|
||||
Digest `Engine.Calls` / `Fallback`, `digest.Gate` inflight/waiters, and
|
||||
`dcache` hit/miss/evictions exist in process memory and are **not** on the
|
||||
JSON endpoint today. P2.7 exports them (Prometheus + additive JSON fields).
|
||||
|
||||
Search `addErr` currently uses the warning tail after the last colon as
|
||||
`kind`. That is fine for JSON compatibility; it is **not** safe as a
|
||||
Prometheus label (a slipped hostname becomes a series). Prometheus `kind`
|
||||
is a closed set (§5). JSON keeps the current maps.
|
||||
|
||||
Listener stays `127.0.0.1:8890` + existing `tailscale serve` HTTPS. Scrape
|
||||
URL and tailscale0 bind are straylight's metrics-plan, not this doc.
|
||||
|
||||
## 3. Wire format
|
||||
|
||||
| Request | Response |
|
||||
|---|---|
|
||||
| `GET /metrics` | Prometheus text 0.0.4, `Content-Type: text/plain; version=0.0.4; charset=utf-8` |
|
||||
| `GET /metrics.json` | today's JSON, `Content-Type: application/json` |
|
||||
| `GET /metrics?format=json` | same JSON as `/metrics.json` |
|
||||
|
||||
`format` values other than `json` (including absent) → Prometheus text.
|
||||
Unauthenticated, same as `/healthz`. Do not count scrapes of `/metrics` or
|
||||
`/healthz` in `sift_requests_total`.
|
||||
|
||||
Private `prometheus.Registry` with only the `sift_*` series below. Do not
|
||||
register client_golang's default Go/process collectors.
|
||||
|
||||
Dep: `github.com/prometheus/client_golang` (vendored like the rest). No
|
||||
third exposition library.
|
||||
|
||||
## 4. Series
|
||||
|
||||
Prefix `sift_`. Counters end `_total`. Histograms are seconds.
|
||||
|
||||
**Requests**
|
||||
|
||||
- `sift_requests_total{endpoint,code}` counter — `/v1/fetch`, `/v2/scrape`,
|
||||
`/v1/search`, `/v2/search`. `code` is the HTTP status as a string (`200`,
|
||||
`401`, `422`, …). Increment once per finished handler, including 401.
|
||||
|
||||
**Guard**
|
||||
|
||||
- `sift_guard_blocks_total` counter — same events as today's `blocks`
|
||||
(SSRF/guard 403). No host, URL, or reason-with-host label.
|
||||
|
||||
**Digest and Dixie gate** (`internal/digest`)
|
||||
|
||||
- `sift_digest_calls_total` counter — Dixie chat actually started (today's
|
||||
`Engine.Calls`; cache hits do not increment).
|
||||
- `sift_digest_fallbacks_total{reason}` counter — `Engine.Fallback` keys.
|
||||
Closed `reason` set: `queue_full`, `queue_wait`, `canceled`, `embed`,
|
||||
`dixie`, `timeout`, `empty`. Unknown → `other` (never the raw string).
|
||||
- `sift_dixie_inflight` gauge — `Gate` slots in use.
|
||||
- `sift_dixie_queue_waiters` gauge — waiters in the sift-side FIFO (not
|
||||
llama-server's queue).
|
||||
|
||||
`Gate` grows a `Snapshot() (inflight, waiters int)` under the existing mutex.
|
||||
|
||||
**Caches**
|
||||
|
||||
- `sift_cache_requests_total{cache,result}` counter — `cache` is `digest`,
|
||||
`embed`, or `search`; `result` is `hit` or `miss`. Search uses the engine
|
||||
counters already on JSON (`search_cache_hit` / `search_cache_miss`), not a
|
||||
second dcache pair, so the two expositions cannot diverge.
|
||||
|
||||
**Search cost** (counts only may carry `caller`)
|
||||
|
||||
- `sift_search_queries_total{caller,source}` counter
|
||||
- `sift_search_credits_total{caller,source}` counter
|
||||
- `sift_search_daily_credits{source}` gauge — current UTC-day total (the
|
||||
in-process / `search_cap` value). Gauge, not counter: it is a day bucket
|
||||
and must not look like a reset counter to Prometheus.
|
||||
- `sift_search_capped_total{source}` counter
|
||||
|
||||
`source` values are the configured source names (`tavily`, `brave`,
|
||||
`searxng`, …) plus the existing cap partition `tavily:restricted`. No
|
||||
query, host, or URL.
|
||||
|
||||
**Histograms** (no `caller`)
|
||||
|
||||
- `sift_request_duration_seconds{op}` — `op` is `fetch`, `digest`, or
|
||||
`search`.
|
||||
- `fetch`: wall time of `/v1/fetch` and `/v2/scrape` (one series, both
|
||||
endpoints).
|
||||
- `digest`: `digest.Result.DigestMS` on digest attempts, including
|
||||
fallbacks; skip pure cache hits.
|
||||
- `search`: `RunResult.SearchMS` on `/v1/search` and `/v2/search`.
|
||||
|
||||
Buckets (seconds): `fetch` `0.1, 0.5, 1, 2, 5, 10, 20, 30`; `digest`
|
||||
`1, 5, 10, 15, 20, 30, 45`; `search` `0.5, 1, 2, 5, 10, 20, 30`. Native
|
||||
histograms out of this cut.
|
||||
|
||||
**JSON additive fields** (same endpoint as today, not a rename):
|
||||
|
||||
- `digest_calls`, `digest_fallbacks` (map reason→n), `dixie_inflight`,
|
||||
`dixie_queue_waiters`
|
||||
- `dcache_digest_hit`, `dcache_digest_miss`, `dcache_embed_hit`,
|
||||
`dcache_embed_miss`
|
||||
|
||||
Existing JSON keys stay. `self_ip_failures` and `search_rerank_fallback`
|
||||
stay on JSON; they are not required on Prometheus in this cut.
|
||||
|
||||
## 5. Labels and redaction
|
||||
|
||||
Same isolation as restricted logs (P2.5 / P3 §8): metrics are a second log.
|
||||
|
||||
- **`caller` only on count series** (`sift_search_queries_total`,
|
||||
`sift_search_credits_total`). Never on histograms, gauges, cache, guard,
|
||||
digest, or `sift_requests_total`.
|
||||
- **`caller` value** is `auth.Caller.Name` from keys.toml, including
|
||||
`permit_restricted` callers — the same string the INFO line uses. Never a
|
||||
query, URL, key, workspace id, or `restricted` stand-in.
|
||||
- **No `host`, `url`, `query`, `q`, `final_url`, `err`, or `msg` labels.**
|
||||
PLAN §4.3 "bot-blocked per host" is deferred until there is a host-less
|
||||
encoding (or it stays logs-only).
|
||||
- **`endpoint`**: path only, the four API routes above.
|
||||
- **`code`**: three-digit HTTP status.
|
||||
- **`reason` / `cache` / `result` / `op` / `source`**: closed sets from §4.
|
||||
- **Search error kinds are not Prometheus labels in this cut** (the JSON
|
||||
`search_errors` map can still hold warning tails). If a later cut adds
|
||||
`sift_search_errors_total{source,kind}`, `kind` is closed:
|
||||
`timeout` | `error` | `4xx` | `5xx` | `429` | `other`. Never `addErr`'s
|
||||
raw tail.
|
||||
|
||||
A test dumps every metric name, label name, and label value from a handler
|
||||
that ran a restricted search + a blocked fetch, and asserts the query, the
|
||||
URL, and the hostname/IP appear **nowhere** in the Prometheus body (including
|
||||
HELP text).
|
||||
|
||||
## 6. Code layout
|
||||
|
||||
```
|
||||
internal/metrics Registry, sift_* defs, JSON snapshot helper
|
||||
internal/digest Gate.Snapshot; Fallback/Calls already there
|
||||
internal/api /metrics negotiates format; /metrics.json
|
||||
handlers observe duration + requests{endpoint,code}
|
||||
```
|
||||
|
||||
`internal/metrics` owns the registry so tests can scrape a `Server` without
|
||||
a global DefaultRegisterer. JSON continues to read the same atomics/maps
|
||||
the Prometheus collectors scrape — one set of counters, two encodings.
|
||||
|
||||
Do not log `/metrics` scrapes at INFO.
|
||||
|
||||
## 7. Tests
|
||||
|
||||
- `GET /metrics` Content-Type and `expfmt` parse; `HELP`/`TYPE` present for
|
||||
each series in §4.
|
||||
- `GET /metrics.json` and `?format=json`: `search_cache_hit` / `_miss` still
|
||||
numbers; existing `TestMetricsCacheHitMiss` pointed at `/metrics.json`.
|
||||
- One `/v1/search` then Prometheus `sift_search_queries_total{caller,source} == 1`.
|
||||
- Digest gate full → `sift_digest_fallbacks_total{reason="queue_full"}` (or
|
||||
`queue_wait`) and `sift_dixie_inflight` / `sift_dixie_queue_waiters` are
|
||||
gauges in range.
|
||||
- Restricted caller fixture: Prometheus body has `caller="<name>"` and does
|
||||
not contain the query, any URL host, or any IP. Fleet caller on the same
|
||||
query does not create extra label names.
|
||||
- Histogram: a timed fake fetch observes `_count` and a bucket.
|
||||
- Unknown fallback reason coerced to `other`.
|
||||
- `/metrics` itself does not increment `sift_requests_total`.
|
||||
|
||||
No live orion scrape in CI. Post one `curl -sS https://orion…/metrics | head`
|
||||
after the pin bump (straylight), not as a merge gate.
|
||||
|
||||
## 8. Rollout
|
||||
|
||||
1. This doc reviewed by Claude (label redaction + JSON compatibility).
|
||||
2. titan implements on `git.wntrmute.dev:kyle/sift`; evidence in this thread.
|
||||
3. Claude reviews the code.
|
||||
4. straylight bumps the orion pin once (batch with other approved isolation
|
||||
patches if any are waiting). Titan does not edit `~/src/nixos`.
|
||||
5. straylight's metrics-plan lists sift as a scrape target on the existing
|
||||
HTTPS serve (or a later tailscale0 exporter port — that choice is not
|
||||
made here).
|
||||
|
||||
## 9. Open questions
|
||||
|
||||
1. Scrape `https://orion.scylla-hammerhead.ts.net:8443/metrics` vs a separate
|
||||
tailscale0 exporter port — straylight's metrics-plan, not this spec.
|
||||
2. Whether JSON should also grow `search_rerank_fallback` / `self_ip_failures`
|
||||
onto Prometheus in this cut (left off above).
|
||||
)
|
||||
Reference in New Issue
Block a user