p2.7: prometheus /metrics
Default GET /metrics is Prometheus text; JSON stays at /metrics.json and ?format=json. :8443 needs the metrics caller bearer; loopback stays open. Restricted callers have no per-caller series and no tavily:restricted gauge.
This commit is contained in:
+146
-125
@@ -1,37 +1,40 @@
|
||||
# Sift P2.7 — Prometheus `/metrics`
|
||||
|
||||
Drafted 2026-09-23 by titan. Reviewer: Claude. Implementer: titan after this
|
||||
doc is approved. Orion pin: straylight, after Claude approves the code.
|
||||
Builds on live P1/P2/P3 (`e62cfc8`): JSON `/metrics` already exists; this
|
||||
changes the default exposition so VictoriaMetrics can scrape sift.
|
||||
Drafted 2026-09-23 by titan. Reviewer: Claude. Implementer: titan.
|
||||
Orion pin: straylight, after Claude approves the code.
|
||||
Kyle decided (2026-09-23, via Claude): aggregated restricted metrics;
|
||||
bearer on `:8443`; loopback open. Claude: drop `search_rerank_fallback`
|
||||
from Prometheus; export `sift_self_ip_failures_total`.
|
||||
Scrape target: `:8443` with `bearer_token_file`.
|
||||
|
||||
No deploy in this phase. No nixos edit from titan.
|
||||
Builds on live P1/P2/P3 (`e62cfc8`). No nixos edit from titan.
|
||||
|
||||
## 1. Scope and exit criteria
|
||||
|
||||
In: Prometheus text on `GET /metrics`; keep today's JSON; the counters/gauges
|
||||
Claude listed (requests, guard blocks, digest + Dixie gate, dcache/search
|
||||
cache, search queries/credits, Tavily daily credits + capped); duration
|
||||
histograms for fetch, digest, and search; label redaction matching restricted
|
||||
logs.
|
||||
In: Prometheus text on `GET /metrics`; keep today's JSON; the series in §4;
|
||||
duration histograms for fetch, digest, and search; scrape-token auth;
|
||||
restricted aggregation (no per-caller series, no `tavily:restricted` gauge).
|
||||
|
||||
Out: Grafana; a new listen address or firewall (straylight's metrics-plan /
|
||||
nixos); alert rules (those live in `~/src/nixos` docs, not here); PLAN's
|
||||
"bot-blocked per host" series (host labels are forbidden); P2 §6 recover /
|
||||
token-ratio / Honcho-context series (not in this cut); Go/process collectors.
|
||||
nixos); alert rules; PLAN's "bot-blocked per host" series; P2 §6 recover /
|
||||
token-ratio / Honcho-context series; Go/process collectors;
|
||||
`search_rerank_fallback` on Prometheus (stays JSON-only).
|
||||
|
||||
**Exit criteria** (evidence in this thread):
|
||||
|
||||
1. `make test lint` green (new tests in §7). `make contract` unchanged.
|
||||
2. `GET /metrics` is Prometheus text 0.0.4 and parses as such.
|
||||
3. `GET /metrics.json` and `GET /metrics?format=json` still return today's JSON
|
||||
object (existing keys, same types). `TestMetricsCacheHitMiss` moves off
|
||||
bare `/metrics`.
|
||||
4. After one search and one digest-fallback, the Prometheus body contains the
|
||||
series in §4 with the labels in §5 — and **no** query text, URL, hostname,
|
||||
or IP.
|
||||
5. Restricted caller: `caller` label is the keys.toml name (same as the log
|
||||
`caller=` field). No `host` label anywhere.
|
||||
2. `GET /metrics` is Prometheus text 0.0.4 (loopback, or `:8443` with the
|
||||
`metrics` bearer).
|
||||
3. `GET /metrics.json` and `GET /metrics?format=json` still return today's
|
||||
JSON object (existing keys, same types), including `tavily:restricted`
|
||||
and per-caller maps. `TestMetricsCacheHitMiss` uses `/metrics.json`.
|
||||
4. After one fleet search, one restricted search, and one digest-fallback,
|
||||
the Prometheus body has the series in §4 — **no** query text, URL,
|
||||
hostname, or IP, **no** `caller="<restricted name>"`, **no**
|
||||
`tavily:restricted`.
|
||||
5. Non-loopback `/metrics` without the metrics bearer is 401; a fetch
|
||||
caller bearer is 403; the metrics caller cannot fetch or search (403).
|
||||
Loopback with no proxy headers stays open.
|
||||
|
||||
## 2. Current `/metrics` (verified in tree, not assumed)
|
||||
|
||||
@@ -49,16 +52,13 @@ token-ratio / Honcho-context series (not in this cut); Go/process collectors.
|
||||
```
|
||||
|
||||
Digest `Engine.Calls` / `Fallback`, `digest.Gate` inflight/waiters, and
|
||||
`dcache` hit/miss/evictions exist in process memory and are **not** on the
|
||||
JSON endpoint today. P2.7 exports them (Prometheus + additive JSON fields).
|
||||
`dcache` hit/miss exist in process memory and were not on JSON. P2.7
|
||||
exports them (Prometheus + additive JSON fields).
|
||||
|
||||
Search `addErr` currently uses the warning tail after the last colon as
|
||||
`kind`. That is fine for JSON compatibility; it is **not** safe as a
|
||||
Prometheus label (a slipped hostname becomes a series). Prometheus `kind`
|
||||
is a closed set (§5). JSON keeps the current maps.
|
||||
Search `addErr` uses the warning tail as `kind`. JSON keeps that map.
|
||||
Prometheus does not export `search_errors` (the tail is not a closed set).
|
||||
|
||||
Listener stays `127.0.0.1:8890` + existing `tailscale serve` HTTPS. Scrape
|
||||
URL and tailscale0 bind are straylight's metrics-plan, not this doc.
|
||||
Listener stays `127.0.0.1:8890` + existing `tailscale serve` HTTPS `:8443`.
|
||||
|
||||
## 3. Wire format
|
||||
|
||||
@@ -69,14 +69,38 @@ URL and tailscale0 bind are straylight's metrics-plan, not this doc.
|
||||
| `GET /metrics?format=json` | same JSON as `/metrics.json` |
|
||||
|
||||
`format` values other than `json` (including absent) → Prometheus text.
|
||||
Unauthenticated, same as `/healthz`. Do not count scrapes of `/metrics` or
|
||||
`/healthz` in `sift_requests_total`.
|
||||
Do not count scrapes of `/metrics`, `/metrics.json`, or `/healthz` in
|
||||
`sift_requests_total`.
|
||||
|
||||
Private `prometheus.Registry` with only the `sift_*` series below. Do not
|
||||
register client_golang's default Go/process collectors.
|
||||
Private `prometheus.Registry` with only the `sift_*` series below. No
|
||||
client_golang Go/process collectors.
|
||||
|
||||
Dep: `github.com/prometheus/client_golang` (vendored like the rest). No
|
||||
third exposition library.
|
||||
Dep: `github.com/prometheus/client_golang` (vendored).
|
||||
|
||||
## 3a. Auth (Kyle, 2026-09-23)
|
||||
|
||||
- **Loopback stays open.** `RemoteAddr` is loopback **and** there is no
|
||||
`X-Forwarded-For` / `X-Forwarded-Proto` / `X-Forwarded-Host` /
|
||||
`Tailscale-User-Login` / `Tailscale-Headers-Info` **and** `Host` is
|
||||
loopback or `localhost`. This is `curl http://127.0.0.1:8890/metrics`.
|
||||
`tailscale serve` on `:8443` sets forwarded / Tailscale headers and a
|
||||
non-loopback Host, so it is not treated as loopback even though the
|
||||
upstream is 127.0.0.1:8890.
|
||||
- **`:8443` requires a bearer.** Dedicated `metrics` caller in `keys.toml`:
|
||||
`name = "metrics"`, empty workspace, no fetch or search rights.
|
||||
`sift keys add --caller metrics` forces `workspace = ""`.
|
||||
- Missing/unknown key → 401. A fetch/search caller's key → 403. The
|
||||
metrics caller on `/v1/fetch`, `/v2/scrape`, `/v1/search`, `/v2/search`
|
||||
→ 403 `unauthorized`.
|
||||
- `/healthz` stays unauthenticated.
|
||||
- Serve start: WARN if keys.toml has no `metrics` caller (loopback still
|
||||
works; `:8443` scrapes will 401). WARN if a `metrics` caller somehow
|
||||
has a workspace (still treated as metrics-only). `CheckMappings` skips
|
||||
the metrics caller so an empty workspace is not an ERROR.
|
||||
|
||||
Straylight's scrape config: `https://orion.scylla-hammerhead.ts.net:8443/metrics`
|
||||
with `bearer_token_file`. Generating the key is on-box (same as other
|
||||
callers); do not print it.
|
||||
|
||||
## 4. Series
|
||||
|
||||
@@ -85,148 +109,145 @@ Prefix `sift_`. Counters end `_total`. Histograms are seconds.
|
||||
**Requests**
|
||||
|
||||
- `sift_requests_total{endpoint,code}` counter — `/v1/fetch`, `/v2/scrape`,
|
||||
`/v1/search`, `/v2/search`. `code` is the HTTP status as a string (`200`,
|
||||
`401`, `422`, …). Increment once per finished handler, including 401.
|
||||
`/v1/search`, `/v2/search`. `code` is the HTTP status as a string.
|
||||
Increment once per finished handler, including 401/403.
|
||||
|
||||
**Guard**
|
||||
**Guard / self-IP**
|
||||
|
||||
- `sift_guard_blocks_total` counter — same events as today's `blocks`
|
||||
(SSRF/guard 403). No host, URL, or reason-with-host label.
|
||||
- `sift_guard_blocks_total` counter — same events as today's `blocks`.
|
||||
No host, URL, or reason-with-host label.
|
||||
- `sift_self_ip_failures_total` counter — `selfip.Discoverer.Failures`.
|
||||
|
||||
**Digest and Dixie gate** (`internal/digest`)
|
||||
|
||||
- `sift_digest_calls_total` counter — Dixie chat actually started (today's
|
||||
`Engine.Calls`; cache hits do not increment).
|
||||
- `sift_digest_fallbacks_total{reason}` counter — `Engine.Fallback` keys.
|
||||
Closed `reason` set: `queue_full`, `queue_wait`, `canceled`, `embed`,
|
||||
`dixie`, `timeout`, `empty`. Unknown → `other` (never the raw string).
|
||||
- `sift_digest_calls_total` counter — Dixie chat actually started
|
||||
(`Engine.Calls`; cache hits do not increment).
|
||||
- `sift_digest_fallbacks_total{reason}` counter — closed `reason`:
|
||||
`queue_full`, `queue_wait`, `canceled`, `embed`, `dixie`, `timeout`,
|
||||
`empty`, `unset`. Unknown → `other` (never the raw string).
|
||||
- `sift_dixie_inflight` gauge — `Gate` slots in use.
|
||||
- `sift_dixie_queue_waiters` gauge — waiters in the sift-side FIFO (not
|
||||
llama-server's queue).
|
||||
- `sift_dixie_queue_waiters` gauge — waiters in the sift-side FIFO.
|
||||
|
||||
`Gate` grows a `Snapshot() (inflight, waiters int)` under the existing mutex.
|
||||
`Gate` grows `Snapshot() (inflight, waiters int)` under the existing mutex.
|
||||
|
||||
**Caches**
|
||||
|
||||
- `sift_cache_requests_total{cache,result}` counter — `cache` is `digest`,
|
||||
`embed`, or `search`; `result` is `hit` or `miss`. Search uses the engine
|
||||
counters already on JSON (`search_cache_hit` / `search_cache_miss`), not a
|
||||
second dcache pair, so the two expositions cannot diverge.
|
||||
counters already on JSON (`search_cache_hit` / `search_cache_miss`).
|
||||
|
||||
**Search cost** (counts only may carry `caller`)
|
||||
**Search cost**
|
||||
|
||||
- `sift_search_queries_total{caller,source}` counter
|
||||
- `sift_search_credits_total{caller,source}` counter
|
||||
- `sift_search_daily_credits{source}` gauge — current UTC-day total (the
|
||||
in-process / `search_cap` value). Gauge, not counter: it is a day bucket
|
||||
and must not look like a reset counter to Prometheus.
|
||||
- `sift_search_capped_total{source}` counter
|
||||
Prometheus cannot mix labeled and unlabeled series under one name, so
|
||||
totals and per-caller fleet counts are separate:
|
||||
|
||||
`source` values are the configured source names (`tavily`, `brave`,
|
||||
`searxng`, …) plus the existing cap partition `tavily:restricted`. No
|
||||
query, host, or URL.
|
||||
- `sift_search_queries_total{source}` — **all** callers, including
|
||||
restricted. No `caller` label.
|
||||
- `sift_search_queries_caller_total{caller,source}` — **fleet only**.
|
||||
`permit_restricted` callers are omitted.
|
||||
- `sift_search_credits_total{source}` — all callers.
|
||||
- `sift_search_credits_caller_total{caller,source}` — fleet only.
|
||||
- `sift_search_daily_credits{source}` gauge — current UTC-day total.
|
||||
**No `tavily:restricted` (or any `*:restricted`) source.** That partition
|
||||
stays JSON-only.
|
||||
- `sift_search_capped_total{source}` — same source filter as daily credits.
|
||||
|
||||
**Histograms** (no `caller`)
|
||||
`source` values are configured names (`tavily`, `brave`, `searxng`, …).
|
||||
No query, host, or URL.
|
||||
|
||||
**Histograms** (no `caller`; include restricted traffic)
|
||||
|
||||
- `sift_request_duration_seconds{op}` — `op` is `fetch`, `digest`, or
|
||||
`search`.
|
||||
- `fetch`: wall time of `/v1/fetch` and `/v2/scrape` (one series, both
|
||||
endpoints).
|
||||
- `digest`: `digest.Result.DigestMS` on digest attempts, including
|
||||
fallbacks; skip pure cache hits.
|
||||
- `search`: `RunResult.SearchMS` on `/v1/search` and `/v2/search`.
|
||||
`search`. One HistogramVec; buckets are the union
|
||||
`0.1, 0.5, 1, 2, 5, 10, 15, 20, 30, 45` (HistogramVec cannot vary
|
||||
buckets by label).
|
||||
- `fetch`: wall time of `/v1/fetch` and `/v2/scrape`.
|
||||
- `digest`: `digest.Result.DigestMS` on digest attempts; skip cache hits.
|
||||
- `search`: wall time of `/v1/search` and `/v2/search`.
|
||||
|
||||
Buckets (seconds): `fetch` `0.1, 0.5, 1, 2, 5, 10, 20, 30`; `digest`
|
||||
`1, 5, 10, 15, 20, 30, 45`; `search` `0.5, 1, 2, 5, 10, 20, 30`. Native
|
||||
histograms out of this cut.
|
||||
|
||||
**JSON additive fields** (same endpoint as today, not a rename):
|
||||
**JSON additive fields** (same keys as today plus):
|
||||
|
||||
- `digest_calls`, `digest_fallbacks` (map reason→n), `dixie_inflight`,
|
||||
`dixie_queue_waiters`
|
||||
- `dcache_digest_hit`, `dcache_digest_miss`, `dcache_embed_hit`,
|
||||
`dcache_embed_miss`
|
||||
|
||||
Existing JSON keys stay. `self_ip_failures` and `search_rerank_fallback`
|
||||
stay on JSON; they are not required on Prometheus in this cut.
|
||||
Existing JSON keys stay, including `search_rerank_fallback`,
|
||||
`search_errors`, per-caller maps, and `tavily:restricted`.
|
||||
|
||||
## 5. Labels and redaction
|
||||
|
||||
Same isolation as restricted logs (P2.5 / P3 §8): metrics are a second log.
|
||||
`/metrics` is a second log (same isolation as restricted slog).
|
||||
|
||||
- **`caller` only on count series** (`sift_search_queries_total`,
|
||||
`sift_search_credits_total`). Never on histograms, gauges, cache, guard,
|
||||
digest, or `sift_requests_total`.
|
||||
- **`caller` value** is `auth.Caller.Name` from keys.toml, including
|
||||
`permit_restricted` callers — the same string the INFO line uses. Never a
|
||||
query, URL, key, workspace id, or `restricted` stand-in.
|
||||
- **`caller` only on `*_caller_total` count series**, and only for
|
||||
callers **not** in `permit_restricted`. Value is `auth.Caller.Name`.
|
||||
Restricted traffic is in the unlabeled `{source}` totals and in
|
||||
histograms only.
|
||||
- **No `host`, `url`, `query`, `q`, `final_url`, `err`, or `msg` labels.**
|
||||
PLAN §4.3 "bot-blocked per host" is deferred until there is a host-less
|
||||
encoding (or it stays logs-only).
|
||||
- **`endpoint`**: path only, the four API routes above.
|
||||
- **`code`**: three-digit HTTP status.
|
||||
- **`endpoint`**: the four API routes. **`code`**: three-digit status.
|
||||
- **`reason` / `cache` / `result` / `op` / `source`**: closed sets from §4.
|
||||
- **Search error kinds are not Prometheus labels in this cut** (the JSON
|
||||
`search_errors` map can still hold warning tails). If a later cut adds
|
||||
`sift_search_errors_total{source,kind}`, `kind` is closed:
|
||||
`timeout` | `error` | `4xx` | `5xx` | `429` | `other`. Never `addErr`'s
|
||||
raw tail.
|
||||
- **No `search_errors` on Prometheus.**
|
||||
|
||||
A test dumps every metric name, label name, and label value from a handler
|
||||
that ran a restricted search + a blocked fetch, and asserts the query, the
|
||||
URL, and the hostname/IP appear **nowhere** in the Prometheus body (including
|
||||
HELP text).
|
||||
A test dumps every metric name, label name, and label value from a
|
||||
handler that ran a restricted search + a blocked fetch, and asserts the
|
||||
query, the URL, the hostname/IP, and the restricted caller name appear
|
||||
**nowhere** as a `caller` label (the restricted name must not appear as
|
||||
any label value). The query/URL/host/IP appear nowhere in the Prometheus
|
||||
body, including HELP text.
|
||||
|
||||
## 6. Code layout
|
||||
|
||||
```
|
||||
internal/metrics Registry, sift_* defs, JSON snapshot helper
|
||||
internal/metrics Registry, sift_* defs, loopback predicate, Gather
|
||||
internal/digest Gate.Snapshot; Fallback/Calls already there
|
||||
internal/api /metrics negotiates format; /metrics.json
|
||||
internal/auth NameMetrics; Add() clears workspace; MetricsOnly()
|
||||
internal/api /metrics negotiates format + auth; /metrics.json
|
||||
handlers observe duration + requests{endpoint,code}
|
||||
metrics caller rejected on fetch/search
|
||||
internal/fleetctx CheckMappings skips the metrics caller
|
||||
cmd/sift serve WARN if no metrics caller
|
||||
```
|
||||
|
||||
`internal/metrics` owns the registry so tests can scrape a `Server` without
|
||||
a global DefaultRegisterer. JSON continues to read the same atomics/maps
|
||||
the Prometheus collectors scrape — one set of counters, two encodings.
|
||||
JSON continues to read the same atomics/maps the Prometheus collector
|
||||
scrapes — one set of counters, two encodings.
|
||||
|
||||
Do not log `/metrics` scrapes at INFO.
|
||||
|
||||
## 7. Tests
|
||||
|
||||
- `GET /metrics` Content-Type and `expfmt` parse; `HELP`/`TYPE` present for
|
||||
each series in §4.
|
||||
- `GET /metrics.json` and `?format=json`: `search_cache_hit` / `_miss` still
|
||||
numbers; existing `TestMetricsCacheHitMiss` pointed at `/metrics.json`.
|
||||
- One `/v1/search` then Prometheus `sift_search_queries_total{caller,source} == 1`.
|
||||
- Digest gate full → `sift_digest_fallbacks_total{reason="queue_full"}` (or
|
||||
`queue_wait`) and `sift_dixie_inflight` / `sift_dixie_queue_waiters` are
|
||||
- `GET /metrics` from loopback: Prometheus Content-Type; `expfmt` parse;
|
||||
`HELP`/`TYPE` present for each series in §4.
|
||||
- `GET /metrics.json` and `?format=json` from loopback: existing JSON keys.
|
||||
`TestMetricsCacheHitMiss` pointed at `/metrics.json`.
|
||||
- Non-loopback (or loopback + `X-Forwarded-For`): 401 without bearer; 403
|
||||
with a fleet key; 200 with the metrics key.
|
||||
- Metrics caller on `/v1/fetch` and `/v1/search`: 403.
|
||||
- One fleet `/v1/search` then `sift_search_queries_caller_total{caller,source} == 1`
|
||||
and `sift_search_queries_total{source} == 1`. Restricted search increments
|
||||
only the unlabeled total; no `caller="<restricted>"`.
|
||||
- Digest gate full → `sift_digest_fallbacks_total{reason=...}` and Dixie
|
||||
gauges in range.
|
||||
- Restricted caller fixture: Prometheus body has `caller="<name>"` and does
|
||||
not contain the query, any URL host, or any IP. Fleet caller on the same
|
||||
query does not create extra label names.
|
||||
- Histogram: a timed fake fetch observes `_count` and a bucket.
|
||||
- Restricted + blocked fetch: Prometheus body has no query, URL, host, IP;
|
||||
no restricted name as a label value; no `tavily:restricted`.
|
||||
- Histogram: a timed fake fetch observes `_count`.
|
||||
- Unknown fallback reason coerced to `other`.
|
||||
- `/metrics` itself does not increment `sift_requests_total`.
|
||||
|
||||
No live orion scrape in CI. Post one `curl -sS https://orion…/metrics | head`
|
||||
after the pin bump (straylight), not as a merge gate.
|
||||
No live orion scrape in CI.
|
||||
|
||||
## 8. Rollout
|
||||
|
||||
1. This doc reviewed by Claude (label redaction + JSON compatibility).
|
||||
2. titan implements on `git.wntrmute.dev:kyle/sift`; evidence in this thread.
|
||||
3. Claude reviews the code.
|
||||
4. straylight bumps the orion pin once (batch with other approved isolation
|
||||
1. titan implements on `git.wntrmute.dev:kyle/sift`; evidence in this thread.
|
||||
2. Claude reviews the code (auth predicate, restricted aggregation, tests).
|
||||
3. straylight bumps the orion pin once (batch with other approved isolation
|
||||
patches if any are waiting). Titan does not edit `~/src/nixos`.
|
||||
5. straylight's metrics-plan lists sift as a scrape target on the existing
|
||||
HTTPS serve (or a later tailscale0 exporter port — that choice is not
|
||||
made here).
|
||||
4. On-box: `sift keys add --caller metrics --out <root-only file>`; place
|
||||
the secret for VictoriaMetrics `bearer_token_file`; delete the temp copy.
|
||||
Do not print the key.
|
||||
5. straylight's metrics-plan scrapes `:8443/metrics` with that token.
|
||||
|
||||
## 9. Open questions
|
||||
|
||||
1. Scrape `https://orion.scylla-hammerhead.ts.net:8443/metrics` vs a separate
|
||||
tailscale0 exporter port — straylight's metrics-plan, not this spec.
|
||||
2. Whether JSON should also grow `search_rerank_fallback` / `self_ip_failures`
|
||||
onto Prometheus in this cut (left off above).
|
||||
)
|
||||
None remaining on this cut. Scrape URL is `:8443` with a bearer (Kyle /
|
||||
Claude). `self_ip_failures` is on Prometheus; `search_rerank_fallback` is
|
||||
not.
|
||||
|
||||
Reference in New Issue
Block a user