kyleandornith-1.5-35b-a3b
7e78de633f
Digest systemPrompt gains a page-cite instruction and PromptVersion is 3.
Add the sift_pdf_* metric series (extract_total{outcome}, extract_seconds,
pages_total, pool_wait_seconds, cache_hits_total) to internal/metrics and wire
ObservePDF/ObservePDFWait/PDFCacheHit into internal/api/pdf.go. Refactor
cmd/sift/fetchcmd.go to a fetchOpts struct with parseFetchOpts and fetchBody,
adding --pages/--grep. Update skill/SKILL.md, set nix MemoryMax to 1G, and add a
PDF documents section to the README.
Co-Authored-By: ornith-1.5-35b-a3b <ornith-1.5-35b-a3b@llama.cpp.invalid>
8.3 KiB
8.3 KiB
Implementer log
One row per task session that ends in a commit. Newest last.
| Task | Date | Status | Sessions | Gate | Deviations | Summary |
|---|---|---|---|---|---|---|
| p5a/01-kind-fetch | 2026-09-25 | done | resumed | gate: ok | none | Implemented Kind/application/pdf recognition, PDF body cap (100 MB) and deadline (120 s), and FetchIf with If-None-Match/If-Modified-Since via shared doFetch. Previous session could not pass TestFetchIf only because the given test held its mutex across the handler (deadlock); owner fixed the test. Session resumed after that fix — did not rewrite fetch code, only verified rules and ran gate. |
| p5a/02-pdftext | 2026-09-25 | done | 1 | gate: ok | none | Built the child-extraction package: Extract (reader.Open with ErrWrongPassword→ErrEncrypted, per-page recover, trailer /Info /Title, rule-3 UTF-16BE/Latin-1 decode), outline.go (/Dest and /GoTo /A dests, 10k-item and no-double-visit bounds, never-nil outline), Runner (re-exec'd child, env-isolated via empty env, 60 s deadline, 64 MiB output cap, 200 MiB stdin cap, stderr capped at 4 KiB, exit-code→error mapping, Doc parse plus ErrNoText), and the sift pdftext case in main.go. Removed the tools.go module pin; go.mod/go.sum/vendor/modules.txt confirmed unchanged. All 10 given tests pass under -race -count=3 including the deadline-kill test; make gate ok; given files byte-identical; `printf nope |
| p5a/03-dcache | 2026-09-25 | done | 1 | gate: ok | evict() tie-break | Added the pdf_doc/pdf_page tables, PDFEntry/PDFKey/PutPDF/GetPDF/TouchPDF (one-tx write deleting old pages first, page-ordered read, 304 touch moves only fetched_at), PDFH/PDFM counters, Size() including pdf_doc nbytes. One deviation: extended the eviction union with pdf_doc and added rowid ASC to the ORDER BY so same-second puts evict in insertion order (deterministic whole-document eviction); without it TestPDFEviction is order-dependent. Left Stats() signature unchanged for task 07. Given test byte-identical. |
| p5a/04-pdfpool | 2026-09-25 | done | 1 | gate: ok | none | Implemented the generic scheduler. Pool holds one mutex, a jobs map (key→in-flight job), a callerJobs map (caller→their job), and a FIFO waiters line. Do applies the six rules in order: join (counted in Waiting, no slot, allowed while the caller holds another job, fn may be nil), caller-busy, then slot/queue/refuse. A job's result is broadcast to every sharer via a done channel; results are not cached (key forgotten on finish). fn runs under context.WithoutCancel so the flight outlives a caller that leaves. Panic in fn is recovered into an error, freed the slot and caller. One implementation finding (not a plan deviation): queued jobs must not be counted in Running(), so startLocked only registers a job and a separate takeSlotLocked/grant increments the slot count — abandoning a queued starter (abandon) releases sharers without touching the slot. Given test byte-identical. |
| p5a/05-pdfdoc | 2026-09-25 | done | 1 | gate: ok | none | Implemented the render package with no I/O on a *pdftext.Doc. Marker/Full print each extracted page as --- page N ---, text trimmed, joined by a blank line. Render dispatches grep → page range → whole. Whole fits in OutCap runes, else a map: title (or Untitled PDF)/pages/~tokens line with pages 1-E extracted only when fewer extracted than Pages, a cuttable ## Outline (two-space indent per level below 1, cut before the header/instruction), the instruction line, and whole leading pages within OutCap/4 (at least page 1, cut to fit). Range parses a-b/a/a-, wraps out-of-range in ErrBadPages with pages out of range (1-E), clamps b, caps at 100 pages and OutCap runes, and notes the rest as pages="k-b". Grep is RE2 with smart-case (?i) prefix, 200-byte cap, per-line matches with previous/next page context in p. P-/p. P: form, first 100 shown, total counted; RE2 keeps the (a+)+$ test linear. Every length is in runes. Given test byte-identical; make gate ok. |
| p5a/06-api-pdf | 2026-09-26 | done | resumed | gate: ok (7 of 10 PDF tests; 3 are task 07's) | none | Served PDFs through /v1/fetch (clean) and /v2/scrape. Added RetryAfter int to fetch.Error and a Retry-After header line guarded by > 0. Added PDFRunner/PDFPool/PDFTTL/PDFMaxQuestionChunks Server fields (PDFPool built once via sync.Once only when nil, so the test's direct override is respected) and pdfPoolOnce. Routed clean.Kind in doFetch/cleanFetched (html/json/text switched, "other"→415, "pdf"→extractPDF); doFetch's switch routes "text" explicitly so a PDF never runs clean.Text first. New internal/api/pdf.go: DefaultPDFTTL/DefaultPDFMaxQuestionChunks (task 07), runner()/pool() defaults, extractPDF (pool key `callerClass |
| p5a/08-digest-cli-docs | 2026-09-26 | done | 1 | gate: ok | none | Added the page-cite sentence to the digest systemPrompt ("When the document has --- page N --- markers, cite the page of each fact as (p. N).") and set PromptVersion = "3". Added the sift_pdf_extract_total{outcome}, sift_pdf_extract_seconds, sift_pdf_pages_total, sift_pdf_pool_wait_seconds, sift_pdf_cache_hits_total series to internal/metrics (pdf.go package) and wired ObservePDF/ObservePDFWait/PDFCacheHit into internal/api/pdf.go (ObservePDF once per call after extractPDF, outcome classified by pdfOutcome so raw text never becomes a label; ObservePDFWait only when the caller actually waited on pool().Do; PDFCacheHit on fresh hits and 304s). Refactored cmd/sift/fetchcmd.go into a fetchOpts struct with parseFetchOpts (returns fetchOpts, not string, error) and a fetchBody(o fetchOpts) map[string]any that does the digest/clean default and adds pages/grep only when set; --pages/--grep added to fetchValueFlags. Updated skill/SKILL.md (PDFs section; removed PDF mention from the 415 row). Unit: nix/module.nix MemoryMax = "1G" (one line). README: added a "PDF documents" section covering detection, limits (100 MB/120 s download, 60 s extraction, 2,000 pages), the child process, the 2-slot/4-wait/20 s pool, the 24 h conditional-GET cache, the answers, the task-06 errors table, and the metrics; noted the go-pdfkit/extract v0.3.0 engine chosen by the docs/P5a.md §3 bake-off. All tests pass under -race, make gate ok, three given files byte-identical, nix diff is one line. |
| p5a/07-api-cache-pages | 2026-09-26 | done | 1 | gate: ok | none | Implemented the PDF document cache with revalidation in doFetch: pdfFromCache (miss / fresh within PDFTTL / stale → FetchIf with 304→TouchPDF+answer-fresh / 200→return response with ok=false and no second GET) and storePDF after extractPDF (ContentHash=res.SHA256, ETag/Last-Modified from headers, FetchedAt=now, Meta=JSON of doc.Meta, Pages=doc.Text). pages/grep passed into pdfdoc.Options in renderPDF; ErrBadPages/ErrBadGrep→400 with the error's own text, non-PDF→400 "pages and grep apply to PDFs only" before the doc block. Question limit in digest mode: n=len(digest.Split(pdfdoc.Full(doc), ChunkTokens)), n>PDFMaxQuestionChunks→422 "document too long for question (n chunks); use grep or pages" before any digest call. Envelope cached set true for cache answers via the synthetic cache res's empty body (len(res.Body)==0); the one doFetch change beyond the rule-1 call was reusing that 200 response instead of a second Fetch. Added fmt import to api.go. Given file byte-identical; make gate ok. |