PLAN: future work — paywalled pages to archive snapshots (Kyle, possible P7)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -313,6 +313,26 @@ its acceptance evidence posted in `~agent`.
|
||||
|
||||
## 13. Future work
|
||||
|
||||
### Paywalled pages → archive snapshot (Kyle, 2026-09-26; possible P7)
|
||||
|
||||
"If a site is paywalled, we may be able to use archive.is to fetch it."
|
||||
|
||||
- **What exists:** `recover: true` (`internal/recover`) already tries Wayback, then archive.today's
|
||||
`/newest/<url>` on archive.ph/.md/.li/.is, with fake-success detection. But it only fires on
|
||||
an *error* (`recoverWorthy`: hint `browser`, 422, 429, 504).
|
||||
- **The gap:** a soft paywall is usually a **200** with a teaser, so recover is never tried, and
|
||||
the agent gets a truncated article with no warning.
|
||||
- **Sketch:**
|
||||
1. Detect a paywall on a successful fetch. Signals: JSON-LD `isAccessibleForFree: false`,
|
||||
`hasPart.cssSelector` paywall markup, a known-paywall host list, and a clean body far shorter
|
||||
than the article's word count or the `og:` description implies.
|
||||
2. Add a `paywalled` warning and hint.
|
||||
3. With `recover`, try the snapshot routes. Mark the envelope `source.kind: "snapshot"`, as today.
|
||||
- **To measure first:** how often archive.today serves orion at all (it CAPTCHAs datacenter and
|
||||
tool traffic). Wayback excludes many paywalled publishers. The existing `/newest/` lookup
|
||||
only finds pages someone else has already archived; sift must not *submit* pages to
|
||||
archive.today.
|
||||
|
||||
### Batch fetch endpoint (Kyle, 2026-09-26; plan after P5c)
|
||||
|
||||
A deliberate, **slow**, asynchronous endpoint for an agent that has collected a series of
|
||||
|
||||
Reference in New Issue
Block a user