sift/docs/P2.7.md

Sift P2.7 — Prometheus /metrics

Drafted 2026-09-23 by titan. Reviewer: Claude. Implementer: titan. Orion pin: straylight, after Claude approves the code. Kyle decided (2026-09-23, via Claude): aggregated restricted metrics; bearer on :8443; loopback open. Claude: drop search_rerank_fallback from Prometheus; export sift_self_ip_failures_total. Scrape target: :8443 with bearer_token_file.

Builds on live P1/P2/P3 (e62cfc8). No nixos edit from titan.

1. Scope and exit criteria

In: Prometheus text on GET /metrics; keep today's JSON; the series in §4; duration histograms for fetch, digest, and search; scrape-token auth; restricted aggregation (no per-caller series, no tavily:restricted gauge).

Out: Grafana; a new listen address or firewall (straylight's metrics-plan / nixos); alert rules; PLAN's "bot-blocked per host" series; P2 §6 recover / token-ratio / Honcho-context series; Go/process collectors; search_rerank_fallback on Prometheus (stays JSON-only).

Exit criteria (evidence in this thread):

1. make test lint green (new tests in §7). make contract unchanged.
2. GET /metrics is Prometheus text 0.0.4 (loopback, or :8443 with the metrics bearer).
3. GET /metrics.json and GET /metrics?format=json still return today's JSON object (existing keys, same types), including tavily:restricted and per-caller maps. TestMetricsCacheHitMiss uses /metrics.json.
4. After one fleet search, one restricted search, and one digest-fallback, the Prometheus body has the series in §4 — no query text, URL, hostname, or IP, no caller="<restricted name>", no tavily:restricted.
5. Non-loopback /metrics without the metrics bearer is 401; a fetch caller bearer is 403; the metrics caller cannot fetch or search (403). Loopback with no proxy headers stays open.

2. Current /metrics (verified in tree, not assumed)

internal/api/api.go GET /metrics is unauthenticated JSON:

{"requests": N, "blocks": N, "scrapes": N, "self_ip_failures": N,
 "search_queries": {caller: {source: n}},
 "search_credits": {caller: {source: n}},
 "search_errors": {source: {kind: n}},
 "search_cache_hit": N, "search_cache_miss": N,
 "search_rerank_fallback": N,
 "search_daily_credits": {source: n},
 "search_capped": {source: n}}

Digest Engine.Calls / Fallback, digest.Gate inflight/waiters, and dcache hit/miss exist in process memory and were not on JSON. P2.7 exports them (Prometheus + additive JSON fields).

Search addErr uses the warning tail as kind. JSON keeps that map. Prometheus does not export search_errors (the tail is not a closed set).

Listener stays 127.0.0.1:8890 + existing tailscale serve HTTPS :8443.

3. Wire format

Request	Response
GET /metrics	Prometheus text 0.0.4, Content-Type: text/plain; version=0.0.4; charset=utf-8
GET /metrics.json	today's JSON, Content-Type: application/json
GET /metrics?format=json	same JSON as /metrics.json

format values other than json (including absent) → Prometheus text. Do not count scrapes of /metrics, /metrics.json, or /healthz in sift_requests_total.

Private prometheus.Registry with only the sift_* series below. No client_golang Go/process collectors.

Dep: github.com/prometheus/client_golang (vendored).

3a. Auth (Kyle, 2026-09-23)

• Loopback stays open. RemoteAddr is loopback and there is no X-Forwarded-For / X-Forwarded-Proto / X-Forwarded-Host / Tailscale-User-Login / Tailscale-Headers-Info and Host is loopback or localhost. This is curl http://127.0.0.1:8890/metrics. tailscale serve on :8443 sets forwarded / Tailscale headers and a non-loopback Host, so it is not treated as loopback even though the upstream is 127.0.0.1:8890.
• :8443 requires a bearer. Dedicated metrics caller in keys.toml: name = "metrics", empty workspace, no fetch or search rights. sift keys add --caller metrics forces workspace = "".
• Missing/unknown key → 401. A fetch/search caller's key → 403. The metrics caller on /v1/fetch, /v2/scrape, /v1/search, /v2/search → 403 unauthorized.
• /healthz stays unauthenticated.
• Serve start: WARN if keys.toml has no metrics caller (loopback still works; :8443 scrapes will 401). WARN if a metrics caller somehow has a workspace (still treated as metrics-only). CheckMappings skips the metrics caller so an empty workspace is not an ERROR.

Straylight's scrape config: https://orion.scylla-hammerhead.ts.net:8443/metrics with bearer_token_file. Generating the key is on-box (same as other callers); do not print it.

4. Series

Prefix sift_. Counters end _total. Histograms are seconds.

Requests

• sift_requests_total{endpoint,code} counter — /v1/fetch, /v2/scrape, /v1/search, /v2/search. code is the HTTP status as a string. Increment once per finished handler, including 401/403.

Guard / self-IP

• sift_guard_blocks_total counter — same events as today's blocks. No host, URL, or reason-with-host label.
• sift_self_ip_failures_total counter — selfip.Discoverer.Failures.

Digest and Dixie gate (internal/digest)

• sift_digest_calls_total counter — Dixie chat actually started (Engine.Calls; cache hits do not increment).
• sift_digest_fallbacks_total{reason} counter — closed reason: queue_full, queue_wait, canceled, embed, dixie, timeout, empty, unset. Unknown → other (never the raw string).
• sift_dixie_inflight gauge — Gate slots in use.
• sift_dixie_queue_waiters gauge — waiters in the sift-side FIFO.

Gate grows Snapshot() (inflight, waiters int) under the existing mutex.

Caches

• sift_cache_requests_total{cache,result} counter — cache is digest, embed, or search; result is hit or miss. Search uses the engine counters already on JSON (search_cache_hit / search_cache_miss).

Search cost

Prometheus cannot mix labeled and unlabeled series under one name, so totals and per-caller fleet counts are separate:

• sift_search_queries_total{source} — all callers, including restricted. No caller label.
• sift_search_queries_caller_total{caller,source} — fleet only. permit_restricted callers are omitted.
• sift_search_credits_total{source} — all callers.
• sift_search_credits_caller_total{caller,source} — fleet only.
• sift_search_daily_credits{source} gauge — current UTC-day total. No tavily:restricted (or any *:restricted) source. That partition stays JSON-only.
• sift_search_capped_total{source} — same source filter as daily credits.

source values are configured names (tavily, brave, searxng, …). No query, host, or URL.

Histograms (no caller; include restricted traffic)

• sift_request_duration_seconds{op} — op is fetch, digest, or search. One HistogramVec; buckets are the union 0.1, 0.5, 1, 2, 5, 10, 15, 20, 30, 45 (HistogramVec cannot vary buckets by label).
  • fetch: wall time of /v1/fetch and /v2/scrape.
  • digest: digest.Result.DigestMS on digest attempts; skip cache hits.
  • search: wall time of /v1/search and /v2/search.

JSON additive fields (same keys as today plus):

• digest_calls, digest_fallbacks (map reason→n), dixie_inflight, dixie_queue_waiters
• dcache_digest_hit, dcache_digest_miss, dcache_embed_hit, dcache_embed_miss

Existing JSON keys stay, including search_rerank_fallback, search_errors, per-caller maps, and tavily:restricted.

5. Labels and redaction

/metrics is a second log (same isolation as restricted slog).

• caller only on *_caller_total count series, and only for callers not in permit_restricted. Value is auth.Caller.Name. Restricted traffic is in the unlabeled {source} totals and in histograms only.
• No host, url, query, q, final_url, err, or msg labels.
• endpoint: the four API routes. code: three-digit status.
• reason / cache / result / op / source: closed sets from §4.
• No search_errors on Prometheus.

A test dumps every metric name, label name, and label value from a handler that ran a restricted search + a blocked fetch, and asserts the query, the URL, the hostname/IP, and the restricted caller name appear nowhere as a caller label (the restricted name must not appear as any label value). The query/URL/host/IP appear nowhere in the Prometheus body, including HELP text.

6. Code layout

internal/metrics    Registry, sift_* defs, loopback predicate, Gather
internal/digest     Gate.Snapshot; Fallback/Calls already there
internal/auth       NameMetrics; Add() clears workspace; MetricsOnly()
internal/api        /metrics negotiates format + auth; /metrics.json
                    handlers observe duration + requests{endpoint,code}
                    metrics caller rejected on fetch/search
internal/fleetctx   CheckMappings skips the metrics caller
cmd/sift            serve WARN if no metrics caller

JSON continues to read the same atomics/maps the Prometheus collector scrapes — one set of counters, two encodings.

Do not log /metrics scrapes at INFO.

7. Tests

• GET /metrics from loopback: Prometheus Content-Type; expfmt parse; HELP/TYPE present for each series in §4.
• GET /metrics.json and ?format=json from loopback: existing JSON keys. TestMetricsCacheHitMiss pointed at /metrics.json.
• Non-loopback (or loopback + X-Forwarded-For): 401 without bearer; 403 with a fleet key; 200 with the metrics key.
• Metrics caller on /v1/fetch and /v1/search: 403.
• One fleet /v1/search then sift_search_queries_caller_total{caller,source} == 1 and sift_search_queries_total{source} == 1. Restricted search increments only the unlabeled total; no caller="<restricted>".
• Digest gate full → sift_digest_fallbacks_total{reason=...} and Dixie gauges in range.
• Restricted + blocked fetch: Prometheus body has no query, URL, host, IP; no restricted name as a label value; no tavily:restricted.
• Histogram: a timed fake fetch observes _count.
• Unknown fallback reason coerced to other.
• /metrics itself does not increment sift_requests_total.

No live orion scrape in CI.

8. Rollout

1. titan implements on git.wntrmute.dev:kyle/sift; evidence in this thread.
2. Claude reviews the code (auth predicate, restricted aggregation, tests).
3. straylight bumps the orion pin once (batch with other approved isolation patches if any are waiting). Titan does not edit ~/src/nixos.
4. On-box: sift keys add --caller metrics --out <root-only file>; place the secret for VictoriaMetrics bearer_token_file; delete the temp copy. Do not print the key.
5. straylight's metrics-plan scrapes :8443/metrics with that token.

9. Open questions

None remaining on this cut. Scrape URL is :8443 with a bearer (Kyle / Claude). self_ip_failures is on Prometheus; search_rerank_fallback is not.