P5c review: bound concurrent uploads; warnings is a list; sift extract keeps a base-URL path
Owner review of p5c (tasks 01-02 by Ornith): - extract.go: an upload (up to 100 MiB) is read whole before the PDF pool's limits apply, so any number of concurrent uploads could buffer at once in a 1 GiB service. At most 2 bodies are read at a time now; a third waits up to 5 s, then 429 busy with Retry-After 10. - extract.go: `warnings` was null when empty; it is [] as in /v1/fetch. - extractcmd.go: extractURL replaced the client base URL's path; it now appends /v1/extract like fetch and search do. Regression tests in extract_review_test.go (api and cmd/sift). End-to-end (local serve -> Dixie; sift extract with SIFT_CLIENT): Llama 2 paper, 13.6 MB / 77 pages: --pages 1 ok, repeat served from the content-hash cache, --grep 2 hits, --question answered with page citations in 16.8 s. Log lines carry bytes/pages/mode/cached only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,11 @@
|
||||
package main
|
||||
|
||||
import "testing"
|
||||
|
||||
// P5c owner review: a client base URL with a path prefix keeps it, as fetch and search do.
|
||||
func TestExtractURLKeepsBasePath(t *testing.T) {
|
||||
got := extractURL("https://sift.example:8443/prefix/", extractOpts{File: "a.pdf", Pages: "2"})
|
||||
if want := "https://sift.example:8443/prefix/v1/extract?pages=2"; got != want {
|
||||
t.Fatalf("extractURL = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
@@ -79,13 +79,11 @@ func extractURL(base string, o extractOpts) string {
|
||||
if mode != "" {
|
||||
v.Set("mode", mode)
|
||||
}
|
||||
u, err := url.Parse(base)
|
||||
if err != nil {
|
||||
return base + "/v1/extract?" + v.Encode()
|
||||
u := strings.TrimRight(base, "/") + "/v1/extract"
|
||||
if q := v.Encode(); q != "" {
|
||||
u += "?" + q
|
||||
}
|
||||
u.Path = "/v1/extract"
|
||||
u.RawQuery = v.Encode()
|
||||
return u.String()
|
||||
return u
|
||||
}
|
||||
|
||||
func extractMain() {
|
||||
|
||||
Reference in New Issue
Block a user