74 lines
3.9 KiB
Markdown
74 lines
3.9 KiB
Markdown
# 04-metrics — `sift_batch_*` series in `internal/metrics`
|
|
|
|
Add the batch series and wire them from the runner and API, so the caps (32 targets, 1 job/caller,
|
|
8 jobs total) and the byte budget can be re-sized from observed data.
|
|
|
|
## Series (names and label values exact — the given test asserts these)
|
|
|
|
| Metric | Type | Labels / meaning |
|
|
|---|---|---|
|
|
| `sift_batch_jobs_total` | counter | `outcome` ∈ `created`, `reused`, `done`, `partial`, `failed`, `cancelled`, `restart` |
|
|
| `sift_batch_targets_total` | counter | `outcome` ∈ `ok`, `blocked`, `timeout`, `busy`, `upstream_4xx`, `upstream_5xx`, `unsupported`, `too_long`, `cancel` |
|
|
| `sift_batch_queue_depth` | gauge | live jobs (`queued`+`running`) |
|
|
| `sift_batch_targets_inflight` | gauge | targets currently being fetched |
|
|
| `sift_batch_job_seconds` | histogram | wall time from submit to terminal status |
|
|
| `sift_batch_target_seconds` | histogram | wall time per target |
|
|
| `sift_batch_result_bytes` | histogram | stored bytes per target result |
|
|
|
|
`outcome` labels on `sift_batch_targets_total` are derived from the fetch error classification the
|
|
same way `coarseReason` does (`blocked` for a guard block, `busy` for `429`, `timeout` for `504`,
|
|
`unsupported` for `415`, `too_long` for the question-too-long `422`, `upstream_4xx`/`upstream_5xx`
|
|
otherwise; `ok` on success; `cancel` when the target was skipped by a cancel).
|
|
|
|
## Public surface (exact names — the given test compiles against them)
|
|
|
|
```go
|
|
// fields added to metrics.Set (registered in New)
|
|
BatchJobsTotal *prometheus.CounterVec // label "outcome"
|
|
BatchTargetsTotal *prometheus.CounterVec // label "outcome"
|
|
BatchQueueDepth prometheus.Gauge
|
|
BatchTargetsInflight prometheus.Gauge
|
|
BatchJobSeconds prometheus.Histogram
|
|
BatchTargetSeconds prometheus.Histogram
|
|
BatchResultBytes prometheus.Histogram
|
|
|
|
// methods (nil-guarded, like ObservePDF)
|
|
func (s *Set) ObserveBatchJob(outcome string)
|
|
func (s *Set) ObserveBatchTarget(outcome string)
|
|
func (s *Set) SetBatchQueueDepth(n int)
|
|
func (s *Set) SetBatchTargetsInflight(n int)
|
|
func (s *Set) ObserveBatchJobSeconds(d time.Duration)
|
|
func (s *Set) ObserveBatchTargetSeconds(d time.Duration)
|
|
func (s *Set) ObserveBatchResultBytes(n int)
|
|
```
|
|
|
|
Both `outcome` counters fold any value outside their known set to `other` — the same "a raw string
|
|
never becomes a label" guard as `ObservePDF`. Job outcomes: `created, reused, done, partial,
|
|
failed, cancelled, restart`. Target outcomes: `ok, blocked, timeout, busy, upstream_4xx,
|
|
upstream_5xx, unsupported, too_long, cancel`.
|
|
|
|
## Rules
|
|
|
|
1. Follow the existing `internal/metrics` pattern: a new `internal/metrics/batch.go` that adds the
|
|
series to `metrics.Set`, with `Observe…` methods mirroring `ObservePDF`/`ObserveDigest`.
|
|
2. `sift_batch_jobs_total{outcome}` is incremented exactly once per job, at its terminal
|
|
transition (or `created`/`reused` at submit, `restart` at reconciliation).
|
|
3. `sift_batch_targets_total{outcome}` is incremented exactly once per target.
|
|
4. Gauges reflect the runner's live state: `queue_depth` = live job count; `targets_inflight` = the
|
|
one in-flight target across jobs (0 or 1 while a job is running, summed over jobs).
|
|
5. Histograms record durations/bytes regardless of outcome; `result_bytes` is the stored target
|
|
size, which is what sizes the byte budget.
|
|
6. The metrics are exposed through the existing `/metrics` and `/metrics.json` paths — no new
|
|
endpoint. The API server's `Snapshot`/`writeMetricsJSON` do not need to enumerate them (the
|
|
Prometheus text handler covers them), but the given test reads the `metrics.Set` values directly.
|
|
|
|
## Given
|
|
|
|
Copy `_files/metrics/batch_test.go` → `internal/metrics/batch_test.go`. It drives the `Observe…`
|
|
methods, serves the `/metrics` handler (as `pdf_test.go` does), and asserts the exact series names,
|
|
label values, gauge values, histogram counts, and that a raw outcome string is folded to `other`.
|
|
|
|
## Gate
|
|
|
|
`make gate` green; `go test ./internal/metrics/` green under `-race`.
|