38f0689ae3
P6 task 03: an explicit Design section (state, worker loop, Cancel, Close) after the first attempt spent its whole reply deliberating the concurrency
kyleandClaude Opus 5.52026-09-26 21:49:35 -07:00
e2ddb2e9d7
batch: runner core — types, Submit with idempotency and caps, Status, Touch, Cancel
kyleandornith-1.5-35b-a3b2026-09-26 21:30:35 -07:00
013a39cb4b
Merge master into p6: the task 02 split, guard.PreCheck, AGENTS.md rule 6
kyleandClaude Opus 5.52026-09-26 21:14:22 -07:00
103ec3d05a
P6 during the run: split the runner task (02 core, 03 workers); guard.PreCheck as an owner commit; AGENTS.md: write code into files early
kyleandClaude Opus 5.52026-09-26 21:14:08 -07:00
2529cb8e84
dcache: batch job and target tables with a sliding lease, sweep and byte budget
kyleandornith-1.5-35b-a3b2026-09-26 20:38:42 -07:00
583a35883d
PLAN: future work — digest retrieval misses terse summary facts (RP2040 PIO count, prod check after P5b)
kyleandClaude Opus 5.52026-09-26 20:34:57 -07:00
94da9d9127
skill: confirm numbers and counts from a digest with --grep; the question refusal is now only past 5000 chunks
kyleandClaude Opus 5.52026-09-26 20:33:19 -07:00
bdc093ee02
Merge the P6 plan (asynchronous batch fetch), revised after owner review; Kyle to re-review
kyleandClaude Opus 5.52026-09-26 20:18:24 -07:00
6e0949d58f
Merge P5c: POST /v1/extract and sift extract (PDFs already on a host)
kyleandClaude Opus 5.52026-09-26 19:56:53 -07:00
a804f5ac93
P5c review: bound concurrent uploads; warnings is a list; sift extract keeps a base-URL path
kyleandClaude Opus 5.52026-09-26 19:56:49 -07:00
7c40531891
api: POST /v1/extract answers an uploaded PDF like a fetched one (content-addressed cache)
kyleandornith-1.5-35b-a3b2026-09-26 19:37:45 -07:00
0481b6e32f
Merge P5b: long-document questions (page-aware chunks, BM25 first, cached embeddings built in the background)
kyleandClaude Opus 5.52026-09-26 19:02:55 -07:00
64d08a358d
P5b review: background embedder re-queues a failed document and waits for a busy gate; no empty vectors cached
kyleandClaude Opus 5.52026-09-26 19:02:51 -07:00
8cdb3e8537
api: questions on long PDFs are answered (retrieval in the digest meta); README and skill
kyleandornith-1.5-35b-a3b2026-09-26 18:52:28 -07:00
f6644c9278
digest: cache chunk embeddings; long documents answer from BM25 while embedding in the background
kyleandornith-1.5-35b-a3b2026-09-26 18:44:28 -07:00
300b46bb67
PLAN: future work — paywalled pages to archive snapshots (Kyle, possible P7)
kyleandClaude Opus 5.52026-09-26 18:05:55 -07:00
7b7eb4e19b
P5b README: changes during the run (KV overflow on run 1; my fixture-size bug in task 02's given test)
kyleandClaude Opus 5.52026-09-26 17:58:12 -07:00
d52a4fa1e6
P5b given test: the embedding-cache test used a 12-page document (4.6k tokens), under SmallDoc, so the rules make it "whole", not "hybrid"; use 30 pages (11.7k tokens, 10 chunks) and assert the precondition
kyleandClaude Opus 5.52026-09-26 17:57:14 -07:00
218eaa8b53
P6 plan, owner review folded in: caller travels with the job, 1–2 round-robin workers, busy retry, summary reads, local-only tests, 8 tasks
kyleandClaude Opus 5.52026-09-26 17:55:50 -07:00
b1da18d49e
PLAN: future work — asynchronous batch fetch endpoint (Kyle, after P5c)
kyleandClaude Opus 5.52026-09-26 11:03:43 -07:00
49d96e197b
P5b and P5c plans: long-document retrieval (BM25 first, background embedding) and /v1/extract + sift extract; given tests
kyleandClaude Opus 5.52026-09-26 11:01:18 -07:00
1311cf38d4
pdfdoc test: RE2 timing bound 1 s -> 10 s (race detector made it flaky: 1.0-1.9 s)
kyleandClaude Opus 5.52026-09-26 10:53:52 -07:00
c89281200a
docs: bring README, skill and dist up to date through P5a
kyleandClaude Opus 5.52026-09-26 10:21:51 -07:00
01c8d9daeb
Merge P5a: PDF documents (extraction in a child, pool, cache, map, pages, grep, cited digests)
kyleandClaude Opus 5.52026-09-26 04:48:53 -07:00
aafc1d1c28
P5a review: outlines from named destinations; pool race; fast over-limit; revalidation under the limiter; Accept pdf
kyleandClaude Opus 5.52026-09-26 04:48:17 -07:00
7e78de633f
P5a: digests cite pages; sift_pdf_* metrics; sift fetch --pages/--grep; skill, unit and README
kyleandornith-1.5-35b-a3b2026-09-26 04:36:49 -07:00
09f738bcac
api: PDF cache with revalidation; pages and grep; refuse questions on very long documents
kyleandornith-1.5-35b-a3b2026-09-26 04:23:21 -07:00
93ec310113
api: serve PDFs — extraction in the child under the pool, full text or map, errors
kyleandornith-1.5-35b-a3b2026-09-26 04:04:11 -07:00
9c858aded9
Merge remote-tracking branch 'origin/master' into p5a
kyle2026-09-26 03:12:35 -07:00
5ce625be33
p5a: split task 06 (too large: looped 3h) into 06-api-pdf and 07-api-cache-pages; old 07 is 08
kyleandClaude Opus 5.52026-09-26 03:12:07 -07:00
41a68feb06
pdfdoc: page markers, the map for long documents, page ranges, grep
kyleandornith-1.5-35b-a3b2026-09-25 23:52:11 -07:00
bb9a6544d1
pdfpool: a few extraction slots, a short queue, single-flight, one job per caller
kyleandornith-1.5-35b-a3b2026-09-25 23:32:16 -07:00
b93b7d5816
dcache: PDF document cache (per URL and caller class, whole-document eviction)
kyleandornith-1.5-35b-a3b2026-09-25 23:05:12 -07:00
f57aaef59e
pdftext: extract PDF text in a re-exec'd child (sift pdftext) with a deadline
kyleandornith-1.5-35b-a3b2026-09-25 22:58:05 -07:00
13365a2daa
clean, fetch: recognise PDFs; 100 MB / 120 s for PDFs; FetchIf revalidates
kyleandornith-1.5-35b-a3b2026-09-25 22:40:04 -07:00
6a3b6dab71
run-plan: give opencode an empty stdin (1.15 waits for EOF on a piped stdin and hangs in the background)
kyleandClaude Opus 5.52026-09-25 22:38:33 -07:00
7fa5bd21b2
p5a: fix a deadlock in the given TestFetchIf (lock held across a request); run notes
kyleandClaude Opus 5.52026-09-25 21:45:57 -07:00
kyle
