From 635679c13d68dae4beed583d4ef7fce6a485be3d Mon Sep 17 00:00:00 2001 From: Kyle Isom Date: Fri, 2 Oct 2026 17:08:35 -0700 Subject: [PATCH] =?UTF-8?q?docs:=20DataDome=20401/403=20findings=20(Reuter?= =?UTF-8?q?s,=20Lib=C3=A9ration)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Record the 2026-10-02 titan investigation next to the UA policy and paywall candidates. UA spoofing stays on the table; it did not pass DataDome on those two URLs. --- docs/PLAN.md | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) diff --git a/docs/PLAN.md b/docs/PLAN.md index 2bddaf6..8bfdf4b 100644 --- a/docs/PLAN.md +++ b/docs/PLAN.md @@ -177,6 +177,9 @@ approval prompt. Hence: setting (`user_agent`) overrides it for the whole service; no per-request UA and no automatic browser-UA retry. Revisit only if bot walls turn out to be common — measure it: count `upstream 403/429 (likely bot-blocked)` results per host in `/metrics`. + **Revisit (Kyle, 2026-10-02):** UA spoofing is not off the table. Measurement that day + (Reuters/Libération, DataDome) is below — a `User-Agent` change alone did not fetch + those pages. Phase placement: CLI `fetch --mode clean` + skill in **P1**; digest/context flags in P2; `search` in P3; `--recover` in P2 (it is what removes the titan-style prompts). @@ -346,6 +349,35 @@ Candidates, when there is a phase for it: only finds pages someone else has already archived; sift must not *submit* pages to archive.today. +### DataDome 401/403 — Reuters / Libération (titan, 2026-10-02) + +Live examples: `https://www.reuters.com/business/what-is-frances-2027-budget-2026-10-01/` +and `https://www.liberation.fr/international/amerique/face-a-liran-donald-trump-resserre-letau-militaire-economique-et-rhetorique-20261002_XX22YMZTQRE65BOXUUKD2ZKY3Y/`. + +- **What sift returns:** Reuters `upstream 401` (no `hint`). Libération `upstream 403 + (likely bot-blocked); hint: browser`. P1 maps 403/429/451 (and 503-with-challenge) to + bot-blocked; **401 is a generic `upstream %d`** with empty hint. Reuters AMP and Fusion + (`/pf/api/v3/content/fetch/...`) 401 the same way — not an HTML-only wall. +- **`--recover`:** both URLs have empty Wayback `available`. Reuters archive.today + `/newest/` is 404, so recover rethrows the original 401 with no extra warning. + Libération hits archive.ph `JlK7z` (snapshot 2026-10-02T16:40:46Z) and clean returns + only `Please enable JS and disable any ad blocker` (`#cmsg`). That string is the + publisher/DataDome stub; `fakeSuccess` does not reject it. PLAN's recover ladder still + names an API/RSS/sitemap hop; `internal/recover` only does Wayback then archive.today. +- **The wall is DataDome, not User-Agent.** From titan (`99.73.161.224`) Reuters is 401 and + Libération 403 with `sift/682b9b8`, curl's default UA, and Chrome 131. Bodies are the + same `var dd={…}` challenge (~770 bytes). Hermes browser on that IP sticks on + DataDome (`🐴 reuters.com`, ~1.5 KB, empty article). No curl-impersonate/uTLS on titan + to try a Chrome TLS fingerprint. A whole-service `user_agent` override or an automatic + Mozilla retry would not have fetched either page from this egress. +- **What did fetch:** Hermes `web_extract` (Firecrawl) returned the full Reuters explainer. + Libération still paywalls after JS (lede + « réservé aux abonnés »). `sift search` + Tavily snippets cover related Reuters copy, not that explainer URL. +- **Open:** whether sift grows a DataDome-capable hop (Firecrawl as recover, browser or + TLS-impersonate client on orion); whether 401 should be classified like 403; whether + recover-miss and the `#cmsg` stub should be visible failures. Do not pick those here. + Do not submit URLs to archive.today. + ### Batch fetch endpoint (Kyle, 2026-09-26; plan after P5c) A deliberate, **slow**, asynchronous endpoint for an agent that has collected a series of