v2.3 plan: clients that manage their own slots (control calls, route affinity, queue = false, route listeners, llama-server ctx error); given tests

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-25 17:36:59 -07:00
parent 4c6415819e
commit 3518e84dd7
16 changed files with 1655 additions and 0 deletions
@@ -0,0 +1,77 @@
+# v2.3 task 01: control-plane requests
+
+**Branch:** `v2.3` (`git switch -c v2.3 master` if it does not exist, else `git switch v2.3`; `git status --short` must be empty, otherwise stop)
+**Commit subject:** `Control-plane requests follow the lease but take no slot and write no row`
+
+## Goal
+
+A client that manages its own llama-server slot makes small calls beside its chat stream: it
+polls `GET /slots?model=X` while it waits, reads `GET /props?model=X`, tokenizes, and sends
+`POST /v1/chat/completions/control` on a second connection **while its own stream holds a slot**.
+Today `/slots` and `/tokenize` are 404, a GET is leased under the route's default model instead
+of `?model=`, and every call takes a limiter slot — so `/control` can queue behind its own
+stream, or get 503 when the queue is full. After this task those calls follow the lease like any
+request but never wait for or take a slot, skip the context guard, and write no accounting row.
+
+## Files
+
+- Copy: `internal/proxy/control_test.go`, and the **replacement** `internal/proxy/proxy_test.go`
+  (overwrites the v1 copy: the `/r/slots → 404` row becomes `/r/slots/0` and `/r/metrics`; the
+  v2.3 copy is now the protected one)
+- Create: `internal/proxy/control.go` — the control-call test and the model-from-query rule live
+  here (`proxy.go` is at 325 lines)
+- Modify: `internal/proxy/proxy.go`, `internal/proxy/forward.go`, `docs/implementer-log.md`
+
+## Rules
+
+1. **Model.** The body's top-level `"model"` wins; else the query parameter `model`
+   (`r.URL.Query().Get("model")`); else the route's `default_model`. This is used for the lease
+   key and the limiter pair, exactly where the body model is used today.
+2. **Paths.** `allowedPath` also admits `rest == "/slots"` and `rest == "/tokenize"` (exact
+   match on the path; the query string is not part of `rest`). `/slots/0`, `/slots/0?action=…`
+   and anything else stay `404 {"error":"not found"}`.
+3. **Control calls** are: method `GET` or `HEAD` (any allowed path), or method `POST` with
+   `rest` exactly `/tokenize` or `/v1/chat/completions/control`. Everything else — in
+   particular `POST /v1/chat/completions` — is not a control call.
+4. A control call is routed and leased exactly as today (same `lease.Acquire`, same wake path
+   when no host is healthy), then forwarded **without** `lim.Acquire`, **without** the context
+   guard, and **without** an accounting row (`writeRecord` is not called for it). Its log line
+   is `Debug`, not `Info` (a client polls `/slots` every 5 s). It still gets the
+   `X-Crossbar-Host` / `X-Crossbar-Lease` headers and still marks a host down on a transport
+   error, like any forward.
+5. Do not duplicate `forward`. Pass what it needs to know (for example a `control bool`, or a
+   small options struct if the parameter list gets long) and skip the row and the `Info` log
+   inside it.
+
+## Facts you need
+
+- `peekModel` already reads and restores the body; `GET`/`HEAD` return `""` there. The query
+  fallback goes after it, in one place.
+- The rig in `helpers_test.go` passes the store as the recorder, and the row is written after
+  the answer is sent — the tests wait for rows; do not add sleeps to production code.
+- `waitUntil` is defined in `proxy_test.go`; `conversation(id, turn)` builds a chat body.
+
+## Steps
+
+- [ ] **1.** Branch as above; copy the two given files.
+- [ ] **2. See them fail:** `go test -count=1 ./internal/proxy/ -run 'TestGetModel|TestControl|TestChatIsNotControl'`
+  → 404 on `/slots` and `/tokenize`, 503 `queue full` on control calls, 4 stray rows.
+- [ ] **3.** `control.go`: the control-call test and the model rule. **4.** `proxy.go`: use them;
+  branch in `serveLeased` (no limiter, no guard for a control call). **5.** `forward.go`: no row,
+  `Debug` log for a control call.
+- [ ] **6.** `gofmt -w`; `go test -race -count=3 ./internal/proxy/` → `ok`.
+- [ ] **7.** `make gate`; `make smoke`. **8.** Row `v2.3/01-control-plane`; commit.
+
+```sh
+git add internal/proxy docs/implementer-log.md
+git commit
+```
+
+## Done when
+
+- The new tests and every earlier proxy test pass under `-race -count=3`; gate and smoke ok;
+  given files byte-identical; no file over 400 lines.
+
+## Stop and report if
+
+- Passing needs a change to any given test, or `forward` cannot skip the row without copying it.