sift/docs/P2.7.md

# Sift P2.7 — Prometheus `/metrics`

Drafted 2026-09-23 by titan. Reviewer: Claude. Implementer: titan after this
doc is approved. Orion pin: straylight, after Claude approves the code.
Builds on live P1/P2/P3 (`e62cfc8`): JSON `/metrics` already exists; this
changes the default exposition so VictoriaMetrics can scrape sift.

No deploy in this phase. No nixos edit from titan.

## 1. Scope and exit criteria

In: Prometheus text on `GET /metrics`; keep today's JSON; the counters/gauges
Claude listed (requests, guard blocks, digest + Dixie gate, dcache/search
cache, search queries/credits, Tavily daily credits + capped); duration
histograms for fetch, digest, and search; label redaction matching restricted
logs.

Out: Grafana; a new listen address or firewall (straylight's metrics-plan /
nixos); alert rules (those live in `~/src/nixos` docs, not here); PLAN's
"bot-blocked per host" series (host labels are forbidden); P2 §6 recover /
token-ratio / Honcho-context series (not in this cut); Go/process collectors.

**Exit criteria** (evidence in this thread):

1. `make test lint` green (new tests in §7). `make contract` unchanged.
2. `GET /metrics` is Prometheus text 0.0.4 and parses as such.
3. `GET /metrics.json` and `GET /metrics?format=json` still return today's JSON
   object (existing keys, same types). `TestMetricsCacheHitMiss` moves off
   bare `/metrics`.
4. After one search and one digest-fallback, the Prometheus body contains the
   series in §4 with the labels in §5 — and **no** query text, URL, hostname,
   or IP.
5. Restricted caller: `caller` label is the keys.toml name (same as the log
   `caller=` field). No `host` label anywhere.

## 2. Current `/metrics` (verified in tree, not assumed)

`internal/api/api.go` `GET /metrics` is unauthenticated JSON:

```json
{"requests": N, "blocks": N, "scrapes": N, "self_ip_failures": N,
 "search_queries": {caller: {source: n}},
 "search_credits": {caller: {source: n}},
 "search_errors": {source: {kind: n}},
 "search_cache_hit": N, "search_cache_miss": N,
 "search_rerank_fallback": N,
 "search_daily_credits": {source: n},
 "search_capped": {source: n}}
```

Digest `Engine.Calls` / `Fallback`, `digest.Gate` inflight/waiters, and
`dcache` hit/miss/evictions exist in process memory and are **not** on the
JSON endpoint today. P2.7 exports them (Prometheus + additive JSON fields).

Search `addErr` currently uses the warning tail after the last colon as
`kind`. That is fine for JSON compatibility; it is **not** safe as a
Prometheus label (a slipped hostname becomes a series). Prometheus `kind`
is a closed set (§5). JSON keeps the current maps.

Listener stays `127.0.0.1:8890` + existing `tailscale serve` HTTPS. Scrape
URL and tailscale0 bind are straylight's metrics-plan, not this doc.

## 3. Wire format

| Request | Response |
|---|---|
| `GET /metrics` | Prometheus text 0.0.4, `Content-Type: text/plain; version=0.0.4; charset=utf-8` |
| `GET /metrics.json` | today's JSON, `Content-Type: application/json` |
| `GET /metrics?format=json` | same JSON as `/metrics.json` |

`format` values other than `json` (including absent) → Prometheus text.
Unauthenticated, same as `/healthz`. Do not count scrapes of `/metrics` or
`/healthz` in `sift_requests_total`.

Private `prometheus.Registry` with only the `sift_*` series below. Do not
register client_golang's default Go/process collectors.

Dep: `github.com/prometheus/client_golang` (vendored like the rest). No
third exposition library.

## 4. Series

Prefix `sift_`. Counters end `_total`. Histograms are seconds.

**Requests**

- `sift_requests_total{endpoint,code}` counter — `/v1/fetch`, `/v2/scrape`,
  `/v1/search`, `/v2/search`. `code` is the HTTP status as a string (`200`,
  `401`, `422`, …). Increment once per finished handler, including 401.

**Guard**

- `sift_guard_blocks_total` counter — same events as today's `blocks`
  (SSRF/guard 403). No host, URL, or reason-with-host label.

**Digest and Dixie gate** (`internal/digest`)

- `sift_digest_calls_total` counter — Dixie chat actually started (today's
  `Engine.Calls`; cache hits do not increment).
- `sift_digest_fallbacks_total{reason}` counter — `Engine.Fallback` keys.
  Closed `reason` set: `queue_full`, `queue_wait`, `canceled`, `embed`,
  `dixie`, `timeout`, `empty`. Unknown → `other` (never the raw string).
- `sift_dixie_inflight` gauge — `Gate` slots in use.
- `sift_dixie_queue_waiters` gauge — waiters in the sift-side FIFO (not
  llama-server's queue).

`Gate` grows a `Snapshot() (inflight, waiters int)` under the existing mutex.

**Caches**

- `sift_cache_requests_total{cache,result}` counter — `cache` is `digest`,
  `embed`, or `search`; `result` is `hit` or `miss`. Search uses the engine
  counters already on JSON (`search_cache_hit` / `search_cache_miss`), not a
  second dcache pair, so the two expositions cannot diverge.

**Search cost** (counts only may carry `caller`)

- `sift_search_queries_total{caller,source}` counter
- `sift_search_credits_total{caller,source}` counter
- `sift_search_daily_credits{source}` gauge — current UTC-day total (the
  in-process / `search_cap` value). Gauge, not counter: it is a day bucket
  and must not look like a reset counter to Prometheus.
- `sift_search_capped_total{source}` counter

`source` values are the configured source names (`tavily`, `brave`,
`searxng`, …) plus the existing cap partition `tavily:restricted`. No
query, host, or URL.

**Histograms** (no `caller`)

- `sift_request_duration_seconds{op}` — `op` is `fetch`, `digest`, or
  `search`.
  - `fetch`: wall time of `/v1/fetch` and `/v2/scrape` (one series, both
    endpoints).
  - `digest`: `digest.Result.DigestMS` on digest attempts, including
    fallbacks; skip pure cache hits.
  - `search`: `RunResult.SearchMS` on `/v1/search` and `/v2/search`.

Buckets (seconds): `fetch` `0.1, 0.5, 1, 2, 5, 10, 20, 30`; `digest`
`1, 5, 10, 15, 20, 30, 45`; `search` `0.5, 1, 2, 5, 10, 20, 30`. Native
histograms out of this cut.

**JSON additive fields** (same endpoint as today, not a rename):

- `digest_calls`, `digest_fallbacks` (map reason→n), `dixie_inflight`,
  `dixie_queue_waiters`
- `dcache_digest_hit`, `dcache_digest_miss`, `dcache_embed_hit`,
  `dcache_embed_miss`

Existing JSON keys stay. `self_ip_failures` and `search_rerank_fallback`
stay on JSON; they are not required on Prometheus in this cut.

## 5. Labels and redaction

Same isolation as restricted logs (P2.5 / P3 §8): metrics are a second log.

- **`caller` only on count series** (`sift_search_queries_total`,
  `sift_search_credits_total`). Never on histograms, gauges, cache, guard,
  digest, or `sift_requests_total`.
- **`caller` value** is `auth.Caller.Name` from keys.toml, including
  `permit_restricted` callers — the same string the INFO line uses. Never a
  query, URL, key, workspace id, or `restricted` stand-in.
- **No `host`, `url`, `query`, `q`, `final_url`, `err`, or `msg` labels.**
  PLAN §4.3 "bot-blocked per host" is deferred until there is a host-less
  encoding (or it stays logs-only).
- **`endpoint`**: path only, the four API routes above.
- **`code`**: three-digit HTTP status.
- **`reason` / `cache` / `result` / `op` / `source`**: closed sets from §4.
- **Search error kinds are not Prometheus labels in this cut** (the JSON
  `search_errors` map can still hold warning tails). If a later cut adds
  `sift_search_errors_total{source,kind}`, `kind` is closed:
  `timeout` | `error` | `4xx` | `5xx` | `429` | `other`. Never `addErr`'s
  raw tail.

A test dumps every metric name, label name, and label value from a handler
that ran a restricted search + a blocked fetch, and asserts the query, the
URL, and the hostname/IP appear **nowhere** in the Prometheus body (including
HELP text).

## 6. Code layout

```
internal/metrics    Registry, sift_* defs, JSON snapshot helper
internal/digest     Gate.Snapshot; Fallback/Calls already there
internal/api        /metrics negotiates format; /metrics.json
                    handlers observe duration + requests{endpoint,code}
```

`internal/metrics` owns the registry so tests can scrape a `Server` without
a global DefaultRegisterer. JSON continues to read the same atomics/maps
the Prometheus collectors scrape — one set of counters, two encodings.

Do not log `/metrics` scrapes at INFO.

## 7. Tests

- `GET /metrics` Content-Type and `expfmt` parse; `HELP`/`TYPE` present for
  each series in §4.
- `GET /metrics.json` and `?format=json`: `search_cache_hit` / `_miss` still
  numbers; existing `TestMetricsCacheHitMiss` pointed at `/metrics.json`.
- One `/v1/search` then Prometheus `sift_search_queries_total{caller,source} == 1`.
- Digest gate full → `sift_digest_fallbacks_total{reason="queue_full"}` (or
  `queue_wait`) and `sift_dixie_inflight` / `sift_dixie_queue_waiters` are
  gauges in range.
- Restricted caller fixture: Prometheus body has `caller="<name>"` and does
  not contain the query, any URL host, or any IP. Fleet caller on the same
  query does not create extra label names.
- Histogram: a timed fake fetch observes `_count` and a bucket.
- Unknown fallback reason coerced to `other`.
- `/metrics` itself does not increment `sift_requests_total`.

No live orion scrape in CI. Post one `curl -sS https://orion…/metrics | head`
after the pin bump (straylight), not as a merge gate.

## 8. Rollout

1. This doc reviewed by Claude (label redaction + JSON compatibility).
2. titan implements on `git.wntrmute.dev:kyle/sift`; evidence in this thread.
3. Claude reviews the code.
4. straylight bumps the orion pin once (batch with other approved isolation
   patches if any are waiting). Titan does not edit `~/src/nixos`.
5. straylight's metrics-plan lists sift as a scrape target on the existing
   HTTPS serve (or a later tailscale0 exporter port — that choice is not
   made here).

## 9. Open questions

1. Scrape `https://orion.scylla-hammerhead.ts.net:8443/metrics` vs a separate
   tailscale0 exporter port — straylight's metrics-plan, not this spec.
2. Whether JSON should also grow `search_rerank_fallback` / `self_ip_failures`
   onto Prometheus in this cut (left off above).
)