api: questions on long PDFs are answered (retrieval in the digest meta); README and skill

The digest now handles long documents itself (task 02), so the API's refusal
limit is only a safety cap: DefaultPDFMaxQuestionChunks 200 -> 5000 (the 422
message text is unchanged). In v1fetch the digest-meta object gains "retrieval"
and "embedded" next to chunks_used/chunks_total, only when the digest ran
without a fallback. The PDF section explains a long-doc question is answered
first from BM25 while its chunks embed in the background and hybrid on later
questions; skill/SKILL.md gains one sentence. Given tests byte-identical.

Co-Authored-By: ornith-1.5-35b-a3b <ornith-1.5-35b-a3b@llama.cpp.invalid>
This commit is contained in:
2026-09-26 18:52:28 -07:00
co-authored by ornith-1.5-35b-a3b
parent f6644c9278
commit 8cdb3e8537
7 changed files with 92 additions and 8 deletions
+2 -1
View File
@@ -61,7 +61,8 @@ PDFs work: `sift fetch` extracts them and returns the text. A short one comes ba
long one comes back as a map — an outline with page numbers, then the first pages. Then:
- `sift fetch URL --pages 41-60` for a page range, `--grep 'pattern'` for matching lines, or
`--question "…"` for a digest that cites each fact as `(p. N)`.
`--question "…"` for a digest that cites each fact as `(p. N)`; `--question` works on long
PDFs, and the first answer may be `bm25` (lexical) — ask again later for a sharper one.
- Scanned PDFs have no text layer: sift reports `no text layer (scanned PDF?)` — point it at an
OCR source rather than trying to fix it here.
- A range returns at most 100 pages and grep at most 100 matches. On a very long document