PLAN: future work — paywalled pages to archive snapshots (Kyle, possible P7)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-09-26 18:05:55 -07:00
co-authored by Claude Opus 5.5
parent 7b7eb4e19b
commit 300b46bb67
+20
View File
@@ -313,6 +313,26 @@ its acceptance evidence posted in `~agent`.
## 13. Future work
### Paywalled pages → archive snapshot (Kyle, 2026-09-26; possible P7)
"If a site is paywalled, we may be able to use archive.is to fetch it."
- **What exists:** `recover: true` (`internal/recover`) already tries Wayback, then archive.today's
`/newest/<url>` on archive.ph/.md/.li/.is, with fake-success detection. But it only fires on
an *error* (`recoverWorthy`: hint `browser`, 422, 429, 504).
- **The gap:** a soft paywall is usually a **200** with a teaser, so recover is never tried, and
the agent gets a truncated article with no warning.
- **Sketch:**
1. Detect a paywall on a successful fetch. Signals: JSON-LD `isAccessibleForFree: false`,
`hasPart.cssSelector` paywall markup, a known-paywall host list, and a clean body far shorter
than the article's word count or the `og:` description implies.
2. Add a `paywalled` warning and hint.
3. With `recover`, try the snapshot routes. Mark the envelope `source.kind: "snapshot"`, as today.
- **To measure first:** how often archive.today serves orion at all (it CAPTCHAs datacenter and
tool traffic). Wayback excludes many paywalled publishers. The existing `/newest/` lookup
only finds pages someone else has already archived; sift must not *submit* pages to
archive.today.
### Batch fetch endpoint (Kyle, 2026-09-26; plan after P5c)
A deliberate, **slow**, asynchronous endpoint for an agent that has collected a series of