kyleandClaude Opus 5.5
cfde35397a
docs/plans/p5a: 01 kind+fetch limits, 02 pdftext child + runner, 03 dcache PDF tables, 04 pdfpool, 05 pdfdoc render (map/pages/grep), 06 api, 07 digest citations, metrics, CLI, skill, unit, README. Given tests compile against a panic-only skeleton and fail there for the intended reasons; PDFs for tests are built by internal/pdftest (given), plus two qpdf-encrypted fixtures. Owner changes for the run: vendor go-pdfkit/extract v0.3.0 + reader v0.6.0 (pinned via tools.go until task 02); AGENTS.md, tools/run-plan.sh (enforces a Co-Authored-By trailer for the model), docs/implementer-log.md, `make gate`; gofmt (whitespace) on clean.go and guard; TestUnsupportedType uses a zip now that PDFs are supported; requestDeadline 90→210 s and writeTimeout 120→240 s for the PDF path's worst case. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
82 lines
3.6 KiB
Markdown
82 lines
3.6 KiB
Markdown
# P5a task 04: the extraction pool
|
|
|
|
**Branch:** `p5a` (`git switch p5a`; `git status --short` must be empty, otherwise stop)
|
|
**Commit subject:** `pdfpool: a few extraction slots, a short queue, single-flight, one job per caller`
|
|
|
|
## Goal
|
|
|
|
A PDF extraction uses a child process and up to a few hundred MB, and sift shares 1 GB with it.
|
|
Kyle, 2026-09-25: "I don't think we're seeing a ton of PDF requests simultaneously, so having one
|
|
or two workers seems tenable." This task is the small in-process scheduler that enforces that:
|
|
two slots, four waiting places, identical requests sharing one job, and no single caller
|
|
holding more than one job at a time. Generic: it knows nothing about PDFs.
|
|
|
|
## Files
|
|
|
|
- Copy: `internal/pdfpool/pdfpool_test.go`
|
|
- Create: `internal/pdfpool/pdfpool.go`
|
|
- Modify: `docs/implementer-log.md`
|
|
|
|
## Interfaces
|
|
|
|
```go
|
|
package pdfpool
|
|
|
|
var (
|
|
ErrBusy = errors.New("pdf workers busy") // queue full, or waited longer than Wait
|
|
ErrCallerBusy = errors.New("busy") // this caller already has a job in flight
|
|
)
|
|
|
|
// New returns a pool with `slots` jobs running at once, at most `queue` waiting, and a waiter
|
|
// refused after `wait`.
|
|
func New(slots, queue int, wait time.Duration) *Pool
|
|
|
|
// Do runs fn under the pool and returns its result.
|
|
func (p *Pool) Do(ctx context.Context, key, caller string, fn func(context.Context) (any, error)) (any, error)
|
|
|
|
func (p *Pool) Running() int // jobs holding a slot
|
|
func (p *Pool) Queued() int // jobs waiting for a slot
|
|
func (p *Pool) Waiting(key string) int // callers sharing key's job, not counting the one that started it
|
|
```
|
|
|
|
## Rules, in the order Do applies them
|
|
|
|
1. **Join.** If a job for `key` is already running or queued, wait for its result (count it in
|
|
`Waiting(key)`) and return it. Joining takes no slot, no queue place, and is allowed even if
|
|
the caller has another job in flight. `fn` may be nil when joining.
|
|
2. **Caller.** If `caller` already has a job in flight (running or queued) for another key,
|
|
return `ErrCallerBusy` at once.
|
|
3. **Slot.** If fewer than `slots` jobs are running, start at once. Otherwise, if `queue` jobs
|
|
are already waiting, return `ErrBusy` at once. Otherwise wait, first come first served, until
|
|
a slot frees (start), `wait` passes (`ErrBusy`), or `ctx` ends (`ctx.Err()`). A waiter that
|
|
leaves gives up its place.
|
|
4. **The job's context is not the caller's.** `fn` runs with a context that is **not** cancelled
|
|
when the caller that started it goes away (`context.WithoutCancel(ctx)`): other callers may
|
|
be sharing it. The work has its own deadline (the child's). A caller whose `ctx` ends while
|
|
waiting for a result gets `ctx.Err()`; the job goes on for the others.
|
|
5. **Results are not cached.** When the job ends, the key is forgotten: the next `Do` for it runs
|
|
`fn` again. An error from `fn` reaches every sharer. A **panic** in `fn` is recovered,
|
|
returned as an error to every sharer, and frees the slot and the caller.
|
|
6. Everything is guarded by one mutex; no goroutine leaks when a waiter gives up.
|
|
|
|
## Steps
|
|
|
|
- [ ] **1.** `git switch p5a`; copy the given test. **2. See it fail** (undefined names).
|
|
- [ ] **3.** `pdfpool.go`.
|
|
- [ ] **4.** `gofmt -w`; `go test -race -count=5 ./internal/pdfpool/` → `ok`.
|
|
- [ ] **5.** `make gate`. **6.** Row `p5a/04-pdfpool`; commit.
|
|
|
|
```sh
|
|
git add internal/pdfpool docs/implementer-log.md
|
|
git commit -F .state/commit-msg.txt
|
|
```
|
|
|
|
## Done when
|
|
|
|
- The given test passes under `-race -count=5`; `make gate` ok; given file byte-identical.
|
|
|
|
## Stop and report if
|
|
|
|
- A given test's timing looks impossible to meet without sleeping in production code (that is
|
|
the owner's bug, per `AGENTS.md`).
|