P5c review: bound concurrent uploads; warnings is a list; sift extract keeps a base-URL path
Owner review of p5c (tasks 01-02 by Ornith): - extract.go: an upload (up to 100 MiB) is read whole before the PDF pool's limits apply, so any number of concurrent uploads could buffer at once in a 1 GiB service. At most 2 bodies are read at a time now; a third waits up to 5 s, then 429 busy with Retry-After 10. - extract.go: `warnings` was null when empty; it is [] as in /v1/fetch. - extractcmd.go: extractURL replaced the client base URL's path; it now appends /v1/extract like fetch and search do. Regression tests in extract_review_test.go (api and cmd/sift). End-to-end (local serve -> Dixie; sift extract with SIFT_CLIENT): Llama 2 paper, 13.6 MB / 77 pages: --pages 1 ok, repeat served from the content-hash cache, --grep 2 hits, --question answered with page citations in 16.8 s. Log lines carry bytes/pages/mode/cached only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
+20
-1
@@ -27,6 +27,17 @@ func (s *Server) v1extract(w http.ResponseWriter, r *http.Request) {
|
||||
return
|
||||
}
|
||||
|
||||
select {
|
||||
case s.uploads <- struct{}{}:
|
||||
defer func() { <-s.uploads }()
|
||||
case <-time.After(uploadWait):
|
||||
w.Header().Set("Retry-After", "10")
|
||||
writeErr(w, 429, "busy", "")
|
||||
return
|
||||
case <-r.Context().Done():
|
||||
return
|
||||
}
|
||||
|
||||
body, rerr := io.ReadAll(r.Body)
|
||||
if rerr != nil {
|
||||
var me *http.MaxBytesError
|
||||
@@ -105,7 +116,7 @@ func (s *Server) v1extract(w http.ResponseWriter, r *http.Request) {
|
||||
content = full
|
||||
}
|
||||
|
||||
var warn []string
|
||||
warn := []string{}
|
||||
var digMeta map[string]any
|
||||
var digestMS int64
|
||||
if mode == "digest" {
|
||||
@@ -192,6 +203,14 @@ func (s *Server) v1extract(w http.ResponseWriter, r *http.Request) {
|
||||
writeJSON(w, 200, out)
|
||||
}
|
||||
|
||||
// maxUploads is how many uploads may be in memory at once, and uploadWait how long a further
|
||||
// one waits for a slot before a 429: at 100 MiB each, a few are all a 1 GiB service can hold
|
||||
// beside the extraction children.
|
||||
const (
|
||||
maxUploads = 2
|
||||
uploadWait = 5 * time.Second
|
||||
)
|
||||
|
||||
// pdfMaxUpload is the /v1/extract upload cap: the configured value, or the default when unset.
|
||||
func (s *Server) pdfMaxUpload() int64 {
|
||||
if s.PDFMaxUpload > 0 {
|
||||
|
||||
Reference in New Issue
Block a user