kyle/sift
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: bring README, skill and dist up to date through P5a

Browse Source 
README gains API, CLI and configuration sections and the PDF range/grep/question limits; the skill covers PDF busy and failure results; dist/sift.service matches the Nix unit (MemoryMax=1G, secret.env); the example config drops the unused [upstreams] block; stale task comments in api/pdf.go.

Gate overridden by the owner: TestGrepLimits (given test) exceeds its 1 s budget under -race on this host.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
kyle
2026-09-26 10:21:51 -07:00
co-authored by Claude Opus 5.5
parent 01c8d9daeb
commit c89281200a
5 changed files with 64 additions and 11 deletions
Download Patch File Download Diff File Expand all files Collapse all files
skill/SKILL.md
+6
View File
@@ -64,6 +64,9 @@ long one comes back as a map — an outline with page numbers, then the first pa
`--question "…"` for a digest that cites each fact as `(p. N)`.
- Scanned PDFs have no text layer: sift reports `no text layer (scanned PDF?)` — point it at an
OCR source rather than trying to fix it here.
- A range returns at most 100 pages and grep at most 100 matches. On a very long document
`--question` is refused with `document too long for question (N chunks); use grep or pages` —
do that: `--grep` for the terms, then `--pages` around the hits.
Never download a PDF and run `pdftotext` yourself; `sift` is the sanctioned extractor.
@@ -92,6 +95,9 @@ Never download a PDF and run `pdftotext` yourself; `sift` is the sanctioned extr
| `upstream 403/429 (likely bot-blocked)`, `hint: recover` | Site blocks non-browser clients | `sift fetch URL --recover` (Wayback → archive.today, server-side). Cite snapshots **with their date**. |
| `hint: browser` | Needs JS rendering or a real browser | Use the browser tool. |
| `415 unsupported type` | a content type sift cannot fetch or clean | Browser tool, or ask. |
| `503 pdf workers busy` | every PDF extraction slot is taken | Wait the `Retry-After` (10 s) and retry once. |
| `429 busy` | sift is saturated, or you already have too many fetches to that host (2) or a PDF extraction in flight | Wait for them to finish, then retry; don't fire many at once. |
| `encrypted PDF`, `unreadable PDF`, `pdf extraction timed out` | sift can't get text from this PDF | Look for an HTML or arXiv abstract version; don't retry the same URL. |
| connection refused / timeout to sift itself | sift is down | Fallback below, and tell Kyle sift is down. |
## Fallback — only when sift itself is unreachable