Compare commits
1
Commits
682b9b8044
..
master
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
635679c13d |
@@ -177,6 +177,9 @@ approval prompt. Hence:
|
||||
setting (`user_agent`) overrides it for the whole service; no per-request UA and no
|
||||
automatic browser-UA retry. Revisit only if bot walls turn out to be common — measure it:
|
||||
count `upstream 403/429 (likely bot-blocked)` results per host in `/metrics`.
|
||||
**Revisit (Kyle, 2026-10-02):** UA spoofing is not off the table. Measurement that day
|
||||
(Reuters/Libération, DataDome) is below — a `User-Agent` change alone did not fetch
|
||||
those pages.
|
||||
|
||||
Phase placement: CLI `fetch --mode clean` + skill in **P1**; digest/context flags in P2;
|
||||
`search` in P3; `--recover` in P2 (it is what removes the titan-style prompts).
|
||||
@@ -346,6 +349,35 @@ Candidates, when there is a phase for it:
|
||||
only finds pages someone else has already archived; sift must not *submit* pages to
|
||||
archive.today.
|
||||
|
||||
### DataDome 401/403 — Reuters / Libération (titan, 2026-10-02)
|
||||
|
||||
Live examples: `https://www.reuters.com/business/what-is-frances-2027-budget-2026-10-01/`
|
||||
and `https://www.liberation.fr/international/amerique/face-a-liran-donald-trump-resserre-letau-militaire-economique-et-rhetorique-20261002_XX22YMZTQRE65BOXUUKD2ZKY3Y/`.
|
||||
|
||||
- **What sift returns:** Reuters `upstream 401` (no `hint`). Libération `upstream 403
|
||||
(likely bot-blocked); hint: browser`. P1 maps 403/429/451 (and 503-with-challenge) to
|
||||
bot-blocked; **401 is a generic `upstream %d`** with empty hint. Reuters AMP and Fusion
|
||||
(`/pf/api/v3/content/fetch/...`) 401 the same way — not an HTML-only wall.
|
||||
- **`--recover`:** both URLs have empty Wayback `available`. Reuters archive.today
|
||||
`/newest/` is 404, so recover rethrows the original 401 with no extra warning.
|
||||
Libération hits archive.ph `JlK7z` (snapshot 2026-10-02T16:40:46Z) and clean returns
|
||||
only `Please enable JS and disable any ad blocker` (`#cmsg`). That string is the
|
||||
publisher/DataDome stub; `fakeSuccess` does not reject it. PLAN's recover ladder still
|
||||
names an API/RSS/sitemap hop; `internal/recover` only does Wayback then archive.today.
|
||||
- **The wall is DataDome, not User-Agent.** From titan (`99.73.161.224`) Reuters is 401 and
|
||||
Libération 403 with `sift/682b9b8`, curl's default UA, and Chrome 131. Bodies are the
|
||||
same `var dd={…}` challenge (~770 bytes). Hermes browser on that IP sticks on
|
||||
DataDome (`🐴 reuters.com`, ~1.5 KB, empty article). No curl-impersonate/uTLS on titan
|
||||
to try a Chrome TLS fingerprint. A whole-service `user_agent` override or an automatic
|
||||
Mozilla retry would not have fetched either page from this egress.
|
||||
- **What did fetch:** Hermes `web_extract` (Firecrawl) returned the full Reuters explainer.
|
||||
Libération still paywalls after JS (lede + « réservé aux abonnés »). `sift search`
|
||||
Tavily snippets cover related Reuters copy, not that explainer URL.
|
||||
- **Open:** whether sift grows a DataDome-capable hop (Firecrawl as recover, browser or
|
||||
TLS-impersonate client on orion); whether 401 should be classified like 403; whether
|
||||
recover-miss and the `#cmsg` stub should be visible failures. Do not pick those here.
|
||||
Do not submit URLs to archive.today.
|
||||
|
||||
### Batch fetch endpoint (Kyle, 2026-09-26; plan after P5c)
|
||||
|
||||
A deliberate, **slow**, asynchronous endpoint for an agent that has collected a series of
|
||||
|
||||
Reference in New Issue
Block a user