kyleandClaude Opus 5.5
218eaa8b53
Kyle: "Fold them into the plan." What changed and why is in docs/P6.md §14. In short:
- FetchFunc gets the *auth.Caller (the given api test could not compile, and a fetch with no
caller would bypass per-caller gates, the pool limit, the cache class and redaction);
- store reads never bump the lease (task 01's rule contradicted its own test); the API Touches;
- 1-2 workers round-robin over live jobs, so a batch holds at most 2 of the 8 global slots;
busy targets retry in the runner (bounded); batch PDFs use their own pool name; 210 s deadline;
- the api tests no longer touch the network or race the cancel; private names via fake DNS;
- failed/cancelled jobs are not reused; GET is a summary plus GET …/{idx} and ?results=1;
- the 256 KiB body cap moves into ServeHTTP; a fixed metrics label for /v1/batch/{id};
- guard.PreCheck (given verbatim) replaces guard.ParseURL at submit;
- 5 tasks became 8 (refactor, wiring, endpoints and serve/config split), each restating the
APIs it uses; every given test compiles against a panic-only skeleton and fails on it.
P6 starts after P5c: task 04 moves v1fetch, which P5b changes. Recheck the api given tests
against post-P5c master before the run.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
84 lines
3.9 KiB
Markdown
84 lines
3.9 KiB
Markdown
# P6 task 03: `sift_batch_*` series in `internal/metrics`
|
|
|
|
**Branch:** `p6` (`git switch p6`; `git status --short` must be empty, otherwise stop)
|
|
**Commit subject:** `metrics: sift_batch_* series (jobs, targets, depth, in-flight, durations, bytes)`
|
|
|
|
Add the batch series to `internal/metrics` so the caps (32 targets, 1 job/caller, 8 jobs total)
|
|
and the byte budget can be re-sized from observed data. **This task touches only
|
|
`internal/metrics`**: task 05 wires these methods to the runner's `batch.Hooks`.
|
|
|
|
## Series (names and label values exact — the given test asserts these)
|
|
|
|
| Metric | Type | Labels / meaning |
|
|
|---|---|---|
|
|
| `sift_batch_jobs_total` | counter | `outcome` ∈ `created`, `reused`, `done`, `partial`, `failed`, `cancelled`, `restart` |
|
|
| `sift_batch_targets_total` | counter | `outcome` ∈ `ok`, `blocked`, `timeout`, `busy`, `upstream_4xx`, `upstream_5xx`, `unsupported`, `too_long`, `cancel` |
|
|
| `sift_batch_queue_depth` | gauge | live jobs (`queued`+`running`) |
|
|
| `sift_batch_targets_inflight` | gauge | targets currently being fetched |
|
|
| `sift_batch_job_seconds` | histogram | wall time from submit to terminal status |
|
|
| `sift_batch_target_seconds` | histogram | wall time per target |
|
|
| `sift_batch_result_bytes` | histogram | stored bytes per target result |
|
|
|
|
Classifying an error into a target outcome is task 05's job (it has the `*fetch.Error`); this
|
|
package only records the label it is given.
|
|
|
|
## Public surface (exact names — the given test compiles against them)
|
|
|
|
```go
|
|
// fields added to metrics.Set (registered in New)
|
|
BatchJobsTotal *prometheus.CounterVec // label "outcome"
|
|
BatchTargetsTotal *prometheus.CounterVec // label "outcome"
|
|
BatchQueueDepth prometheus.Gauge
|
|
BatchTargetsInflight prometheus.Gauge
|
|
BatchJobSeconds prometheus.Histogram
|
|
BatchTargetSeconds prometheus.Histogram
|
|
BatchResultBytes prometheus.Histogram
|
|
|
|
// methods (nil-guarded, like ObservePDF)
|
|
func (s *Set) ObserveBatchJob(outcome string)
|
|
func (s *Set) ObserveBatchTarget(outcome string)
|
|
func (s *Set) SetBatchQueueDepth(n int)
|
|
func (s *Set) SetBatchTargetsInflight(n int)
|
|
func (s *Set) ObserveBatchJobSeconds(d time.Duration)
|
|
func (s *Set) ObserveBatchTargetSeconds(d time.Duration)
|
|
func (s *Set) ObserveBatchResultBytes(n int)
|
|
```
|
|
|
|
Both `outcome` counters fold any value outside their known set to `other` — the same "a raw string
|
|
never becomes a label" guard as `ObservePDF`. Job outcomes: `created, reused, done, partial,
|
|
failed, cancelled, restart`. Target outcomes: `ok, blocked, timeout, busy, upstream_4xx,
|
|
upstream_5xx, unsupported, too_long, cancel`.
|
|
|
|
## Rules
|
|
|
|
1. A new file `internal/metrics/batch.go` adds the series to `metrics.Set` (registered in `New`),
|
|
with `Observe…`/`Set…` methods mirroring `ObservePDF`/`ObserveDigest`, nil-receiver safe.
|
|
2. Both `outcome` counters fold any value outside their known set to `other`: a raw string
|
|
never becomes a label.
|
|
3. The series are exposed through the existing `/metrics` and `/metrics.json` paths: no new
|
|
endpoint. `Snapshot`/`writeMetricsJSON` do not need to list them.
|
|
4. Histogram buckets: seconds as the existing request histograms; bytes
|
|
`1K, 4K, 16K, 64K, 256K, 1M`.
|
|
|
|
## Given
|
|
|
|
Copy `_files/metrics/batch_test.go` → `internal/metrics/batch_test.go`. It drives the `Observe…`
|
|
methods, serves the `/metrics` handler (as `pdf_test.go` does), and asserts the exact series names,
|
|
label values, gauge values, histogram counts, and that a raw outcome string is folded to `other`.
|
|
|
|
## Steps
|
|
|
|
- [ ] **1.** `git switch p6`; copy the given test. **2. See it fail.**
|
|
- [ ] **3.** `internal/metrics/batch.go` and the registration in `New`.
|
|
- [ ] **4.** `gofmt -w`; `go test -race -count=3 ./internal/metrics/` → `ok`. **5.** `make gate`.
|
|
- [ ] **6.** Row `p6/03-metrics`; commit.
|
|
|
|
```sh
|
|
git add internal/metrics docs/implementer-log.md
|
|
git commit -F .state/commit-msg.txt
|
|
```
|
|
|
|
## Done when
|
|
|
|
- The given and existing metrics tests pass under `-race -count=3`; `make gate` ok; the given file is byte-identical.
|