sift/docs/P2.7.md

# Sift P2.7 — Prometheus `/metrics`

Drafted 2026-09-23 by titan. Reviewer: Claude. Implementer: titan.
Orion pin: straylight, after Claude approves the code.
Kyle decided (2026-09-23, via Claude): aggregated restricted metrics;
bearer on `:8443`; loopback open. Claude: drop `search_rerank_fallback`
from Prometheus; export `sift_self_ip_failures_total`.
Scrape target: `:8443` with `bearer_token_file`.

Builds on live P1/P2/P3 (`e62cfc8`). No nixos edit from titan.

## 1. Scope and exit criteria

In: Prometheus text on `GET /metrics`; keep today's JSON; the series in §4;
duration histograms for fetch, digest, and search; scrape-token auth;
restricted aggregation (no per-caller series, no `tavily:restricted` gauge).

Out: Grafana; a new listen address or firewall (straylight's metrics-plan /
nixos); alert rules; PLAN's "bot-blocked per host" series; P2 §6 recover /
token-ratio / Honcho-context series; Go/process collectors;
`search_rerank_fallback` on Prometheus (stays JSON-only).

**Exit criteria** (evidence in this thread):

1. `make test lint` green (new tests in §7). `make contract` unchanged.
2. `GET /metrics` is Prometheus text 0.0.4 (loopback, or `:8443` with the
   `metrics` bearer).
3. `GET /metrics.json` and `GET /metrics?format=json` still return today's
   JSON object (existing keys, same types), including `tavily:restricted`
   and per-caller maps. `TestMetricsCacheHitMiss` uses `/metrics.json`.
4. After one fleet search, one restricted search, and one digest-fallback,
   the Prometheus body has the series in §4 — **no** query text, URL,
   hostname, or IP, **no** `caller="<restricted name>"`, **no**
   `tavily:restricted`.
5. Non-loopback `/metrics` without the metrics bearer is 401; a fetch
   caller bearer is 403; the metrics caller cannot fetch or search (403).
   Loopback with no proxy headers stays open.

## 2. Current `/metrics` (verified in tree, not assumed)

`internal/api/api.go` `GET /metrics` is unauthenticated JSON:

```json
{"requests": N, "blocks": N, "scrapes": N, "self_ip_failures": N,
 "search_queries": {caller: {source: n}},
 "search_credits": {caller: {source: n}},
 "search_errors": {source: {kind: n}},
 "search_cache_hit": N, "search_cache_miss": N,
 "search_rerank_fallback": N,
 "search_daily_credits": {source: n},
 "search_capped": {source: n}}
```

Digest `Engine.Calls` / `Fallback`, `digest.Gate` inflight/waiters, and
`dcache` hit/miss exist in process memory and were not on JSON. P2.7
exports them (Prometheus + additive JSON fields).

Search `addErr` uses the warning tail as `kind`. JSON keeps that map.
Prometheus does not export `search_errors` (the tail is not a closed set).

Listener stays `127.0.0.1:8890` + existing `tailscale serve` HTTPS `:8443`.

## 3. Wire format

| Request | Response |
|---|---|
| `GET /metrics` | Prometheus text 0.0.4, `Content-Type: text/plain; version=0.0.4; charset=utf-8` |
| `GET /metrics.json` | today's JSON, `Content-Type: application/json` |
| `GET /metrics?format=json` | same JSON as `/metrics.json` |

`format` values other than `json` (including absent) → Prometheus text.
Do not count scrapes of `/metrics`, `/metrics.json`, or `/healthz` in
`sift_requests_total`.

Private `prometheus.Registry` with only the `sift_*` series below. No
client_golang Go/process collectors.

Dep: `github.com/prometheus/client_golang` (vendored).

## 3a. Auth (Kyle, 2026-09-23)

- **Loopback stays open.** `RemoteAddr` is loopback **and** there is no
  `X-Forwarded-For` / `X-Forwarded-Proto` / `X-Forwarded-Host` /
  `Tailscale-User-Login` / `Tailscale-Headers-Info` **and** `Host` is
  loopback or `localhost`. This is `curl http://127.0.0.1:8890/metrics`.
  `tailscale serve` on `:8443` sets forwarded / Tailscale headers and a
  non-loopback Host, so it is not treated as loopback even though the
  upstream is 127.0.0.1:8890.
- **`:8443` requires a bearer.** Dedicated `metrics` caller in `keys.toml`:
  `name = "metrics"`, empty workspace, no fetch or search rights.
  `sift keys add --caller metrics` forces `workspace = ""`.
- Missing/unknown key → 401. A fetch/search caller's key → 403. The
  metrics caller on `/v1/fetch`, `/v2/scrape`, `/v1/search`, `/v2/search`
  → 403 `unauthorized`.
- `/healthz` stays unauthenticated.
- Serve start: WARN if keys.toml has no `metrics` caller (loopback still
  works; `:8443` scrapes will 401). WARN if a `metrics` caller somehow
  has a workspace (still treated as metrics-only). `CheckMappings` skips
  the metrics caller so an empty workspace is not an ERROR.

Straylight's scrape config: `https://orion.scylla-hammerhead.ts.net:8443/metrics`
with `bearer_token_file`. Generating the key is on-box (same as other
callers); do not print it.

## 4. Series

Prefix `sift_`. Counters end `_total`. Histograms are seconds.

**Requests**

- `sift_requests_total{endpoint,code}` counter — `/v1/fetch`, `/v2/scrape`,
  `/v1/search`, `/v2/search`. `code` is the HTTP status as a string.
  Increment once per finished handler, including 401/403.

**Guard / self-IP**

- `sift_guard_blocks_total` counter — same events as today's `blocks`.
  No host, URL, or reason-with-host label.
- `sift_self_ip_failures_total` counter — `selfip.Discoverer.Failures`.

**Digest and Dixie gate** (`internal/digest`)

- `sift_digest_calls_total` counter — Dixie chat actually started
  (`Engine.Calls`; cache hits do not increment).
- `sift_digest_fallbacks_total{reason}` counter — closed `reason`:
  `queue_full`, `queue_wait`, `canceled`, `embed`, `dixie`, `timeout`,
  `empty`, `unset`. Unknown → `other` (never the raw string).
- `sift_dixie_inflight` gauge — `Gate` slots in use.
- `sift_dixie_queue_waiters` gauge — waiters in the sift-side FIFO.

`Gate` grows `Snapshot() (inflight, waiters int)` under the existing mutex.

**Caches**

- `sift_cache_requests_total{cache,result}` counter — `cache` is `digest`,
  `embed`, or `search`; `result` is `hit` or `miss`. Search uses the engine
  counters already on JSON (`search_cache_hit` / `search_cache_miss`).

**Search cost**

Prometheus cannot mix labeled and unlabeled series under one name, so
totals and per-caller fleet counts are separate:

- `sift_search_queries_total{source}` — **all** callers, including
  restricted. No `caller` label.
- `sift_search_queries_caller_total{caller,source}` — **fleet only**.
  `permit_restricted` callers are omitted.
- `sift_search_credits_total{source}` — all callers.
- `sift_search_credits_caller_total{caller,source}` — fleet only.
- `sift_search_daily_credits{source}` gauge — current UTC-day total.
  **No `tavily:restricted` (or any `*:restricted`) source.** That partition
  stays JSON-only.
- `sift_search_capped_total{source}` — same source filter as daily credits.

`source` values are configured names (`tavily`, `brave`, `searxng`, …).
No query, host, or URL.

**Histograms** (no `caller`; include restricted traffic)

- `sift_request_duration_seconds{op}` — `op` is `fetch`, `digest`, or
  `search`. One HistogramVec; buckets are the union
  `0.1, 0.5, 1, 2, 5, 10, 15, 20, 30, 45` (HistogramVec cannot vary
  buckets by label).
  - `fetch`: wall time of `/v1/fetch` and `/v2/scrape`.
  - `digest`: `digest.Result.DigestMS` on digest attempts; skip cache hits.
  - `search`: wall time of `/v1/search` and `/v2/search`.

**JSON additive fields** (same keys as today plus):

- `digest_calls`, `digest_fallbacks` (map reason→n), `dixie_inflight`,
  `dixie_queue_waiters`
- `dcache_digest_hit`, `dcache_digest_miss`, `dcache_embed_hit`,
  `dcache_embed_miss`

Existing JSON keys stay, including `search_rerank_fallback`,
`search_errors`, per-caller maps, and `tavily:restricted`.

## 5. Labels and redaction

`/metrics` is a second log (same isolation as restricted slog).

- **`caller` only on `*_caller_total` count series**, and only for
  callers **not** in `permit_restricted`. Value is `auth.Caller.Name`.
  Restricted traffic is in the unlabeled `{source}` totals and in
  histograms only.
- **No `host`, `url`, `query`, `q`, `final_url`, `err`, or `msg` labels.**
- **`endpoint`**: the four API routes. **`code`**: three-digit status.
- **`reason` / `cache` / `result` / `op` / `source`**: closed sets from §4.
- **No `search_errors` on Prometheus.**

A test dumps every metric name, label name, and label value from a
handler that ran a restricted search + a blocked fetch, and asserts the
query, the URL, the hostname/IP, and the restricted caller name appear
**nowhere** as a `caller` label (the restricted name must not appear as
any label value). The query/URL/host/IP appear nowhere in the Prometheus
body, including HELP text.

## 6. Code layout

```
internal/metrics    Registry, sift_* defs, loopback predicate, Gather
internal/digest     Gate.Snapshot; Fallback/Calls already there
internal/auth       NameMetrics; Add() clears workspace; MetricsOnly()
internal/api        /metrics negotiates format + auth; /metrics.json
                    handlers observe duration + requests{endpoint,code}
                    metrics caller rejected on fetch/search
internal/fleetctx   CheckMappings skips the metrics caller
cmd/sift            serve WARN if no metrics caller
```

JSON continues to read the same atomics/maps the Prometheus collector
scrapes — one set of counters, two encodings.

Do not log `/metrics` scrapes at INFO.

## 7. Tests

- `GET /metrics` from loopback: Prometheus Content-Type; `expfmt` parse;
  `HELP`/`TYPE` present for each series in §4.
- `GET /metrics.json` and `?format=json` from loopback: existing JSON keys.
  `TestMetricsCacheHitMiss` pointed at `/metrics.json`.
- Non-loopback (or loopback + `X-Forwarded-For`): 401 without bearer; 403
  with a fleet key; 200 with the metrics key.
- Metrics caller on `/v1/fetch` and `/v1/search`: 403.
- One fleet `/v1/search` then `sift_search_queries_caller_total{caller,source} == 1`
  and `sift_search_queries_total{source} == 1`. Restricted search increments
  only the unlabeled total; no `caller="<restricted>"`.
- Digest gate full → `sift_digest_fallbacks_total{reason=...}` and Dixie
  gauges in range.
- Restricted + blocked fetch: Prometheus body has no query, URL, host, IP;
  no restricted name as a label value; no `tavily:restricted`.
- Histogram: a timed fake fetch observes `_count`.
- Unknown fallback reason coerced to `other`.
- `/metrics` itself does not increment `sift_requests_total`.

No live orion scrape in CI.

## 8. Rollout

1. titan implements on `git.wntrmute.dev:kyle/sift`; evidence in this thread.
2. Claude reviews the code (auth predicate, restricted aggregation, tests).
3. straylight bumps the orion pin once (batch with other approved isolation
   patches if any are waiting). Titan does not edit `~/src/nixos`.
4. On-box: `sift keys add --caller metrics --out <root-only file>`; place
   the secret for VictoriaMetrics `bearer_token_file`; delete the temp copy.
   Do not print the key.
5. straylight's metrics-plan scrapes `:8443/metrics` with that token.

## 9. Open questions

None remaining on this cut. Scrape URL is `:8443` with a bearer (Kyle /
Claude). `self_ip_failures` is on Prometheus; `search_rerank_fallback` is
not.