kyleandClaude Fable 5.1
c457046f8b
Every given test compiled against a panic-only interface skeleton (go vet clean); nothing was implemented. modernc.org/sqlite v1.59.0 vetted in a scratch module (WAL works); go.sum given. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
95 lines
4.3 KiB
Markdown
95 lines
4.3 KiB
Markdown
# v1 task 03: the per-(host, model) limiter and the chooser
|
||
|
||
**Branch:** `v1` (run `git switch v1`; `git status --short` must be empty, otherwise stop)
|
||
**Commit subject:** `Add the per-host-model limiter and the host chooser`
|
||
|
||
## Goal
|
||
|
||
Two pure packages. `limiter` hands out at most `parallel` concurrent slots per (host, model) and
|
||
lets at most `queue_max` requests wait in line, first come first served. `choose` picks the host
|
||
for a **new** lease: most free slots × weight, ties to the shortest queue, then list order.
|
||
This is `PLAN.md` §4 (`choose`) and §6 (concurrency).
|
||
|
||
## Context
|
||
|
||
A `llama-server` router with `parallel = 4` and unified KV falls over when a fifth request
|
||
arrives ("Context size has been exceeded"). The limiter is where that is prevented: the fifth
|
||
request waits, the ninth (with `queue_max = 4`) is refused at once so the client can retry
|
||
elsewhere. Waiting is cancellable (the client may go away) and must leak neither a slot nor a
|
||
queue place.
|
||
|
||
## Files
|
||
|
||
- Copy: `internal/limiter/limiter_test.go`, `internal/choose/choose_test.go`
|
||
- Create: `internal/limiter/limiter.go`, `internal/choose/choose.go`
|
||
- Modify: `docs/implementer-log.md`
|
||
|
||
## Interfaces
|
||
|
||
`internal/limiter`, package `limiter`:
|
||
|
||
```go
|
||
var ErrQueueFull = errors.New("queue full")
|
||
|
||
type Limiter struct { /* private: mutex, map[(host, model)]*pair */ }
|
||
func New() *Limiter
|
||
func (l *Limiter) Configure(host, model string, parallel, queueMax int)
|
||
// Acquire returns when a slot is held. release gives it back (idempotent: a second call is a
|
||
// no-op). waited is how long the caller sat in the queue. Errors: ErrQueueFull immediately
|
||
// when queueMax waiters are already queued; ctx.Err() if ctx ends while waiting.
|
||
func (l *Limiter) Acquire(ctx context.Context, host, model string) (release func(), waited time.Duration, err error)
|
||
func (l *Limiter) InFlight(host, model string) int
|
||
func (l *Limiter) Queued(host, model string) int
|
||
func (l *Limiter) FreeSlots(host string) int // sum over the host's configured models of parallel - inflight (never below 0); 0 for an unknown host
|
||
```
|
||
|
||
Rules the tests check:
|
||
|
||
1. An unconfigured (host, model) behaves as `parallel = 1, queueMax = 0`.
|
||
2. FIFO: waiters get slots in arrival order. Suggested shape: a mutex, `inflight`, and a slice
|
||
of waiter channels; `release` pops the head waiter (if any) and hands the slot over without
|
||
ever decrementing `inflight`, else decrements. A waiter whose ctx ends removes itself from
|
||
the queue under the mutex; if the slot was handed to it in the same instant, it releases it.
|
||
3. `release` idempotent via `sync.Once`.
|
||
4. Never panic; all methods safe for concurrent use.
|
||
|
||
`internal/choose`, package `choose`:
|
||
|
||
```go
|
||
type Info struct {
|
||
Healthy, Draining, Loaded, CanServe bool // Loaded: model is resident; CanServe: config lists the model
|
||
Free, Queued int
|
||
Weight float64
|
||
}
|
||
// Best returns the candidate with the highest Free*Weight among those that are healthy, not
|
||
// draining and known (info ok) — preferring hosts with Loaded over merely CanServe; ties go to
|
||
// the lowest Queued, then to candidate order. A host with Free == 0 is still eligible (it will
|
||
// queue). ok is false when nothing is eligible.
|
||
func Best(candidates []string, info func(host string) (Info, bool)) (string, bool)
|
||
```
|
||
|
||
Rules: two passes — first over eligible candidates with `Loaded`, then, if none, over eligible
|
||
candidates with `CanServe`. Score `float64(Free) * Weight`. Compare with `>`; on equality prefer
|
||
lower `Queued`; on equality keep the earlier candidate.
|
||
|
||
## Steps
|
||
|
||
- [ ] **1. Copy.** `git switch v1`; `mkdir -p internal/limiter internal/choose`; copy both tests from `docs/plans/v1/_files/internal/…`.
|
||
- [ ] **2. See them fail** (compile). **3. Write both packages.** `gofmt -w internal/`.
|
||
- [ ] **4. See them pass.** `go test -race -count=1 ./internal/limiter/ ./internal/choose/`. The
|
||
limiter tests are timing-based with generous margins; run them three times: `-count=3`.
|
||
- [ ] **5. Run the gate.** `make gate`. **6. Log and commit.** Row `v1/03-limiter-choose`.
|
||
|
||
```sh
|
||
git add internal/limiter internal/choose docs/implementer-log.md
|
||
git commit
|
||
```
|
||
|
||
## Done when
|
||
|
||
- Both packages pass `-race -count=3`; `make gate` prints `gate: ok`; copied files byte-identical.
|
||
|
||
## Stop and report if
|
||
|
||
- A limiter test fails only sometimes: report which and how often; do not loosen it.
|