v2.3 plan: clients that manage their own slots (control calls, route affinity, queue = false, route listeners, llama-server ctx error); given tests

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-25 17:36:59 -07:00
parent 4c6415819e
commit 3518e84dd7
16 changed files with 1655 additions and 0 deletions
@@ -0,0 +1,96 @@
+# v2.3 task 02: route affinity and queue = false
+
+**Branch:** `v2.3` (`git switch v2.3`; `git status --short` must be empty, otherwise stop)
+**Commit subject:** `Routes may share one lease (affinity = "route") and skip crossbar's queue (queue = false)`
+
+## Goal
+
+Two route keys for a client that manages its own slot:
+
+- `affinity = "route"` — one lease for the whole route. Today each conversation (fingerprint)
+  gets its own lease, and calls without a fingerprint lease "the route itself"; a client whose
+  `/control` and `/slots` calls must reach the host its chat is on needs them all on one lease.
+- `queue = false` — crossbar never holds or refuses the route's requests. The client pins its
+  llama-server slot (`id_slot`), so llama-server queues it and its `/slots` shows the slot busy;
+  a request held in crossbar's queue instead looks idle to the client, which gives up after 30 s.
+  The requests still count as load on the host, so routes that do queue see the host full.
+
+## Files
+
+- Copy: `internal/config/config_v23_test.go`, `internal/limiter/track_test.go`,
+  `internal/proxy/affinity_test.go`
+- Modify: `internal/config/route.go` (**move the `Route` struct here** from `config.go`, which
+  is at 398 lines, and add the fields and methods here; `config.go` keeps a one-line call into
+  the route validation), `internal/config/config.go` (minimal), `internal/limiter/limiter.go`,
+  `internal/proxy/proxy.go` (or `control.go` if `proxy.go` would pass 400 lines),
+  `docs/implementer-log.md`
+
+## Interfaces
+
+```go
+package config
+
+type Route struct {
+	Hosts        []string `toml:"hosts"`
+	DefaultModel string   `toml:"default_model"`
+	Peers        []string `toml:"peers"`
+	Affinity     string   `toml:"affinity"` // "" or "conversation" (the default), or "route"
+	Queue        *bool    `toml:"queue"`    // nil means true
+}
+
+// PerRoute reports affinity = "route": every request on the route shares one lease.
+func (r Route) PerRoute() bool
+
+// Queues reports whether the route's requests wait in (and can be refused by) crossbar's
+// per-(host, model) queue; false only for queue = false.
+func (r Route) Queues() bool
+
+package limiter
+
+// Track counts one request against (host, model) without waiting and without refusing: in flight
+// may exceed parallel. The returned release is idempotent.
+func (l *Limiter) Track(host, model string) (release func())
+```
+
+## Rules
+
+1. **Validation.** `affinity` other than `""`, `"conversation"` or `"route"` is an error whose
+   text contains `routes.<name>.affinity` (for example
+   `routes.convo.affinity: must be "conversation" or "route"`). Both keys are allowed on
+   templates; `cfg.Route(name)` returns them for every concrete route the template serves.
+2. **Lease key.** For a `PerRoute()` route the lease key's fingerprint is `""` for **every**
+   request (chat or control), so every request on the route uses one lease per model. The
+   accounting row keeps the request's real fingerprint (it is still useful in usage views).
+3. **Queue.** For a route where `Queues()` is false, a non-control request takes
+   `lim.Track(host, model)` instead of `lim.Acquire` (and releases it when done, like the slot).
+   Control calls (task 01) take neither.
+4. **Release rule.** A release — from `Acquire`'s slot or from `Track` — hands the slot to the
+   first waiter **only when in flight ≤ parallel** at that moment; otherwise it just decrements
+   in flight. (With only `Acquire` in use in-flight never exceeds parallel, so today's behaviour
+   is unchanged.) `InFlight`, `FreeSlots` and `Queued` count tracked requests like any other.
+5. The context guard runs as today on both kinds of route.
+
+## Steps
+
+- [ ] **1.** `git switch v2.3`; copy the three given tests.
+- [ ] **2. See them fail** (compile: `PerRoute`, `Queues`, `Track` missing).
+- [ ] **3.** `route.go` (struct move, fields, methods, validation). **4.** `limiter.go`
+  (`Track`, the release rule). **5.** `proxy.go` (lease key, `Track`).
+- [ ] **6.** `gofmt -w`; `go test -race -count=3 ./internal/limiter/ ./internal/config/ ./internal/proxy/` → `ok`.
+- [ ] **7.** `make gate`; `make smoke`. **8.** Row `v2.3/02-affinity-queue`; commit.
+
+```sh
+git add internal/config internal/limiter internal/proxy docs/implementer-log.md
+git commit
+```
+
+## Done when
+
+- The new tests and every earlier test pass under `-race -count=3`; every earlier limiter test
+  is still green (the release rule must not change `Acquire`-only behaviour); gate and smoke ok;
+  given files byte-identical; no file over 400 lines.
+
+## Stop and report if
+
+- The struct move breaks a given test, or the release rule cannot be met without changing
+  `Acquire`'s results in an earlier test.