diff --git a/docs/PLAN.md b/docs/PLAN.md index a2c9f8e..c505340 100644 --- a/docs/PLAN.md +++ b/docs/PLAN.md @@ -313,6 +313,26 @@ its acceptance evidence posted in `~agent`. ## 13. Future work +### Paywalled pages → archive snapshot (Kyle, 2026-09-26; possible P7) + +"If a site is paywalled, we may be able to use archive.is to fetch it." + +- **What exists:** `recover: true` (`internal/recover`) already tries Wayback, then archive.today's + `/newest/` on archive.ph/.md/.li/.is, with fake-success detection. But it only fires on + an *error* (`recoverWorthy`: hint `browser`, 422, 429, 504). +- **The gap:** a soft paywall is usually a **200** with a teaser, so recover is never tried, and + the agent gets a truncated article with no warning. +- **Sketch:** + 1. Detect a paywall on a successful fetch. Signals: JSON-LD `isAccessibleForFree: false`, + `hasPart.cssSelector` paywall markup, a known-paywall host list, and a clean body far shorter + than the article's word count or the `og:` description implies. + 2. Add a `paywalled` warning and hint. + 3. With `recover`, try the snapshot routes. Mark the envelope `source.kind: "snapshot"`, as today. +- **To measure first:** how often archive.today serves orion at all (it CAPTCHAs datacenter and + tool traffic). Wayback excludes many paywalled publishers. The existing `/newest/` lookup + only finds pages someone else has already archived; sift must not *submit* pages to + archive.today. + ### Batch fetch endpoint (Kyle, 2026-09-26; plan after P5c) A deliberate, **slow**, asynchronous endpoint for an agent that has collected a series of