78 lines
4.2 KiB
Markdown
78 lines
4.2 KiB
Markdown
# v2.3 task 01: control-plane requests
|
|
|
|
**Branch:** `v2.3` (`git switch -c v2.3 master` if it does not exist, else `git switch v2.3`; `git status --short` must be empty, otherwise stop)
|
|
**Commit subject:** `Control-plane requests follow the lease but take no slot and write no row`
|
|
|
|
## Goal
|
|
|
|
A client that manages its own llama-server slot makes small calls beside its chat stream: it
|
|
polls `GET /slots?model=X` while it waits, reads `GET /props?model=X`, tokenizes, and sends
|
|
`POST /v1/chat/completions/control` on a second connection **while its own stream holds a slot**.
|
|
Today `/slots` and `/tokenize` are 404, a GET is leased under the route's default model instead
|
|
of `?model=`, and every call takes a limiter slot — so `/control` can queue behind its own
|
|
stream, or get 503 when the queue is full. After this task those calls follow the lease like any
|
|
request but never wait for or take a slot, skip the context guard, and write no accounting row.
|
|
|
|
## Files
|
|
|
|
- Copy: `internal/proxy/control_test.go`, and the **replacement** `internal/proxy/proxy_test.go`
|
|
(overwrites the v1 copy: the `/r/slots → 404` row becomes `/r/slots/0` and `/r/metrics`; the
|
|
v2.3 copy is now the protected one)
|
|
- Create: `internal/proxy/control.go` — the control-call test and the model-from-query rule live
|
|
here (`proxy.go` is at 325 lines)
|
|
- Modify: `internal/proxy/proxy.go`, `internal/proxy/forward.go`, `docs/implementer-log.md`
|
|
|
|
## Rules
|
|
|
|
1. **Model.** The body's top-level `"model"` wins; else the query parameter `model`
|
|
(`r.URL.Query().Get("model")`); else the route's `default_model`. This is used for the lease
|
|
key and the limiter pair, exactly where the body model is used today.
|
|
2. **Paths.** `allowedPath` also admits `rest == "/slots"` and `rest == "/tokenize"` (exact
|
|
match on the path; the query string is not part of `rest`). `/slots/0`, `/slots/0?action=…`
|
|
and anything else stay `404 {"error":"not found"}`.
|
|
3. **Control calls** are: method `GET` or `HEAD` (any allowed path), or method `POST` with
|
|
`rest` exactly `/tokenize` or `/v1/chat/completions/control`. Everything else — in
|
|
particular `POST /v1/chat/completions` — is not a control call.
|
|
4. A control call is routed and leased exactly as today (same `lease.Acquire`, same wake path
|
|
when no host is healthy), then forwarded **without** `lim.Acquire`, **without** the context
|
|
guard, and **without** an accounting row (`writeRecord` is not called for it). Its log line
|
|
is `Debug`, not `Info` (a client polls `/slots` every 5 s). It still gets the
|
|
`X-Crossbar-Host` / `X-Crossbar-Lease` headers and still marks a host down on a transport
|
|
error, like any forward.
|
|
5. Do not duplicate `forward`. Pass what it needs to know (for example a `control bool`, or a
|
|
small options struct if the parameter list gets long) and skip the row and the `Info` log
|
|
inside it.
|
|
|
|
## Facts you need
|
|
|
|
- `peekModel` already reads and restores the body; `GET`/`HEAD` return `""` there. The query
|
|
fallback goes after it, in one place.
|
|
- The rig in `helpers_test.go` passes the store as the recorder, and the row is written after
|
|
the answer is sent — the tests wait for rows; do not add sleeps to production code.
|
|
- `waitUntil` is defined in `proxy_test.go`; `conversation(id, turn)` builds a chat body.
|
|
|
|
## Steps
|
|
|
|
- [ ] **1.** Branch as above; copy the two given files.
|
|
- [ ] **2. See them fail:** `go test -count=1 ./internal/proxy/ -run 'TestGetModel|TestControl|TestChatIsNotControl'`
|
|
→ 404 on `/slots` and `/tokenize`, 503 `queue full` on control calls, 4 stray rows.
|
|
- [ ] **3.** `control.go`: the control-call test and the model rule. **4.** `proxy.go`: use them;
|
|
branch in `serveLeased` (no limiter, no guard for a control call). **5.** `forward.go`: no row,
|
|
`Debug` log for a control call.
|
|
- [ ] **6.** `gofmt -w`; `go test -race -count=3 ./internal/proxy/` → `ok`.
|
|
- [ ] **7.** `make gate`; `make smoke`. **8.** Row `v2.3/01-control-plane`; commit.
|
|
|
|
```sh
|
|
git add internal/proxy docs/implementer-log.md
|
|
git commit
|
|
```
|
|
|
|
## Done when
|
|
|
|
- The new tests and every earlier proxy test pass under `-race -count=3`; gate and smoke ok;
|
|
given files byte-identical; no file over 400 lines.
|
|
|
|
## Stop and report if
|
|
|
|
- Passing needs a change to any given test, or `forward` cannot skip the row without copying it.
|