sift/docs/P2.7.md

Sift P2.7 — Prometheus /metrics

Drafted 2026-09-23 by titan. Reviewer: Claude. Implementer: titan after this doc is approved. Orion pin: straylight, after Claude approves the code. Builds on live P1/P2/P3 (e62cfc8): JSON /metrics already exists; this changes the default exposition so VictoriaMetrics can scrape sift.

No deploy in this phase. No nixos edit from titan.

1. Scope and exit criteria

In: Prometheus text on GET /metrics; keep today's JSON; the counters/gauges Claude listed (requests, guard blocks, digest + Dixie gate, dcache/search cache, search queries/credits, Tavily daily credits + capped); duration histograms for fetch, digest, and search; label redaction matching restricted logs.

Out: Grafana; a new listen address or firewall (straylight's metrics-plan / nixos); alert rules (those live in ~/src/nixos docs, not here); PLAN's "bot-blocked per host" series (host labels are forbidden); P2 §6 recover / token-ratio / Honcho-context series (not in this cut); Go/process collectors.

Exit criteria (evidence in this thread):

1. make test lint green (new tests in §7). make contract unchanged.
2. GET /metrics is Prometheus text 0.0.4 and parses as such.
3. GET /metrics.json and GET /metrics?format=json still return today's JSON object (existing keys, same types). TestMetricsCacheHitMiss moves off bare /metrics.
4. After one search and one digest-fallback, the Prometheus body contains the series in §4 with the labels in §5 — and no query text, URL, hostname, or IP.
5. Restricted caller: caller label is the keys.toml name (same as the log caller= field). No host label anywhere.

2. Current /metrics (verified in tree, not assumed)

internal/api/api.go GET /metrics is unauthenticated JSON:

{"requests": N, "blocks": N, "scrapes": N, "self_ip_failures": N,
 "search_queries": {caller: {source: n}},
 "search_credits": {caller: {source: n}},
 "search_errors": {source: {kind: n}},
 "search_cache_hit": N, "search_cache_miss": N,
 "search_rerank_fallback": N,
 "search_daily_credits": {source: n},
 "search_capped": {source: n}}

Digest Engine.Calls / Fallback, digest.Gate inflight/waiters, and dcache hit/miss/evictions exist in process memory and are not on the JSON endpoint today. P2.7 exports them (Prometheus + additive JSON fields).

Search addErr currently uses the warning tail after the last colon as kind. That is fine for JSON compatibility; it is not safe as a Prometheus label (a slipped hostname becomes a series). Prometheus kind is a closed set (§5). JSON keeps the current maps.

Listener stays 127.0.0.1:8890 + existing tailscale serve HTTPS. Scrape URL and tailscale0 bind are straylight's metrics-plan, not this doc.

3. Wire format

Request	Response
GET /metrics	Prometheus text 0.0.4, Content-Type: text/plain; version=0.0.4; charset=utf-8
GET /metrics.json	today's JSON, Content-Type: application/json
GET /metrics?format=json	same JSON as /metrics.json

format values other than json (including absent) → Prometheus text. Unauthenticated, same as /healthz. Do not count scrapes of /metrics or /healthz in sift_requests_total.

Private prometheus.Registry with only the sift_* series below. Do not register client_golang's default Go/process collectors.

Dep: github.com/prometheus/client_golang (vendored like the rest). No third exposition library.

4. Series

Prefix sift_. Counters end _total. Histograms are seconds.

Requests

• sift_requests_total{endpoint,code} counter — /v1/fetch, /v2/scrape, /v1/search, /v2/search. code is the HTTP status as a string (200, 401, 422, …). Increment once per finished handler, including 401.

Guard

• sift_guard_blocks_total counter — same events as today's blocks (SSRF/guard 403). No host, URL, or reason-with-host label.

Digest and Dixie gate (internal/digest)

• sift_digest_calls_total counter — Dixie chat actually started (today's Engine.Calls; cache hits do not increment).
• sift_digest_fallbacks_total{reason} counter — Engine.Fallback keys. Closed reason set: queue_full, queue_wait, canceled, embed, dixie, timeout, empty. Unknown → other (never the raw string).
• sift_dixie_inflight gauge — Gate slots in use.
• sift_dixie_queue_waiters gauge — waiters in the sift-side FIFO (not llama-server's queue).

Gate grows a Snapshot() (inflight, waiters int) under the existing mutex.

Caches

• sift_cache_requests_total{cache,result} counter — cache is digest, embed, or search; result is hit or miss. Search uses the engine counters already on JSON (search_cache_hit / search_cache_miss), not a second dcache pair, so the two expositions cannot diverge.

Search cost (counts only may carry caller)

• sift_search_queries_total{caller,source} counter
• sift_search_credits_total{caller,source} counter
• sift_search_daily_credits{source} gauge — current UTC-day total (the in-process / search_cap value). Gauge, not counter: it is a day bucket and must not look like a reset counter to Prometheus.
• sift_search_capped_total{source} counter

source values are the configured source names (tavily, brave, searxng, …) plus the existing cap partition tavily:restricted. No query, host, or URL.

Histograms (no caller)

• sift_request_duration_seconds{op} — op is fetch, digest, or search.
  • fetch: wall time of /v1/fetch and /v2/scrape (one series, both endpoints).
  • digest: digest.Result.DigestMS on digest attempts, including fallbacks; skip pure cache hits.
  • search: RunResult.SearchMS on /v1/search and /v2/search.

Buckets (seconds): fetch 0.1, 0.5, 1, 2, 5, 10, 20, 30; digest 1, 5, 10, 15, 20, 30, 45; search 0.5, 1, 2, 5, 10, 20, 30. Native histograms out of this cut.

JSON additive fields (same endpoint as today, not a rename):

• digest_calls, digest_fallbacks (map reason→n), dixie_inflight, dixie_queue_waiters
• dcache_digest_hit, dcache_digest_miss, dcache_embed_hit, dcache_embed_miss

Existing JSON keys stay. self_ip_failures and search_rerank_fallback stay on JSON; they are not required on Prometheus in this cut.

5. Labels and redaction

Same isolation as restricted logs (P2.5 / P3 §8): metrics are a second log.

• caller only on count series (sift_search_queries_total, sift_search_credits_total). Never on histograms, gauges, cache, guard, digest, or sift_requests_total.
• caller value is auth.Caller.Name from keys.toml, including permit_restricted callers — the same string the INFO line uses. Never a query, URL, key, workspace id, or restricted stand-in.
• No host, url, query, q, final_url, err, or msg labels. PLAN §4.3 "bot-blocked per host" is deferred until there is a host-less encoding (or it stays logs-only).
• endpoint: path only, the four API routes above.
• code: three-digit HTTP status.
• reason / cache / result / op / source: closed sets from §4.
• Search error kinds are not Prometheus labels in this cut (the JSON search_errors map can still hold warning tails). If a later cut adds sift_search_errors_total{source,kind}, kind is closed: timeout | error | 4xx | 5xx | 429 | other. Never addErr's raw tail.

A test dumps every metric name, label name, and label value from a handler that ran a restricted search + a blocked fetch, and asserts the query, the URL, and the hostname/IP appear nowhere in the Prometheus body (including HELP text).

6. Code layout

internal/metrics    Registry, sift_* defs, JSON snapshot helper
internal/digest     Gate.Snapshot; Fallback/Calls already there
internal/api        /metrics negotiates format; /metrics.json
                    handlers observe duration + requests{endpoint,code}

internal/metrics owns the registry so tests can scrape a Server without a global DefaultRegisterer. JSON continues to read the same atomics/maps the Prometheus collectors scrape — one set of counters, two encodings.

Do not log /metrics scrapes at INFO.

7. Tests

• GET /metrics Content-Type and expfmt parse; HELP/TYPE present for each series in §4.
• GET /metrics.json and ?format=json: search_cache_hit / _miss still numbers; existing TestMetricsCacheHitMiss pointed at /metrics.json.
• One /v1/search then Prometheus sift_search_queries_total{caller,source} == 1.
• Digest gate full → sift_digest_fallbacks_total{reason="queue_full"} (or queue_wait) and sift_dixie_inflight / sift_dixie_queue_waiters are gauges in range.
• Restricted caller fixture: Prometheus body has caller="<name>" and does not contain the query, any URL host, or any IP. Fleet caller on the same query does not create extra label names.
• Histogram: a timed fake fetch observes _count and a bucket.
• Unknown fallback reason coerced to other.
• /metrics itself does not increment sift_requests_total.

No live orion scrape in CI. Post one curl -sS https://orion…/metrics | head after the pin bump (straylight), not as a merge gate.

8. Rollout

1. This doc reviewed by Claude (label redaction + JSON compatibility).
2. titan implements on git.wntrmute.dev:kyle/sift; evidence in this thread.
3. Claude reviews the code.
4. straylight bumps the orion pin once (batch with other approved isolation patches if any are waiting). Titan does not edit ~/src/nixos.
5. straylight's metrics-plan lists sift as a scrape target on the existing HTTPS serve (or a later tailscale0 exporter port — that choice is not made here).

9. Open questions

1. Scrape https://orion.scylla-hammerhead.ts.net:8443/metrics vs a separate tailscale0 exporter port — straylight's metrics-plan, not this spec.
2. Whether JSON should also grow search_rerank_fallback / self_ip_failures onto Prometheus in this cut (left off above). )