plan: UA decided — default sift/<ver>, config override, measure bot-blocks

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-09-22 21:28:04 -07:00
co-authored by Claude Opus 5.5
parent 2607d56e4c
commit 3b41d6ffbc
+4 -3
View File
@@ -173,9 +173,10 @@ approval prompt. Hence:
same host), with that skill's fake-success detection, returning provenance
`snapshot` + timestamp. Every hop still goes through the SSRF guard. This moves the
ladder's `curl` calls out of agent shells.
- **User-Agent policy — Kyle to decide**: (a) honest `sift/<ver>` UA only, bot-walled sites
go to recover/browser; or (b) one retry with a browser UA. (b) removes more prompts but is
UA spoofing.
- **User-Agent policy — decided (Kyle, 2026-09-22)**: default UA is `sift/<ver>`. A config
setting (`user_agent`) overrides it for the whole service; no per-request UA and no
automatic browser-UA retry. Revisit only if bot walls turn out to be common — measure it:
count `upstream 403/429 (likely bot-blocked)` results per host in `/metrics`.
Phase placement: CLI `fetch --mode clean` + skill in **P1**; digest/context flags in P2;
`search` in P3; `--recover` in P2 (it is what removes the titan-style prompts).