v1 plan: leases, limiter, chooser, fingerprint, SQLite store, accounting, admin — acceptance tests first, no reference

Every given test compiled against a panic-only interface skeleton (go vet clean); nothing was
implemented. modernc.org/sqlite v1.59.0 vetted in a scratch module (WAL works); go.sum given.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
2026-09-25 03:10:24 -07:00
co-authored by Claude Fable 5.1
parent 851cc80eba
commit c457046f8b
20 changed files with 2660 additions and 0 deletions
+94
View File
@@ -0,0 +1,94 @@
# v1 task 03: the per-(host, model) limiter and the chooser
**Branch:** `v1` (run `git switch v1`; `git status --short` must be empty, otherwise stop)
**Commit subject:** `Add the per-host-model limiter and the host chooser`
## Goal
Two pure packages. `limiter` hands out at most `parallel` concurrent slots per (host, model) and
lets at most `queue_max` requests wait in line, first come first served. `choose` picks the host
for a **new** lease: most free slots × weight, ties to the shortest queue, then list order.
This is `PLAN.md` §4 (`choose`) and §6 (concurrency).
## Context
A `llama-server` router with `parallel = 4` and unified KV falls over when a fifth request
arrives ("Context size has been exceeded"). The limiter is where that is prevented: the fifth
request waits, the ninth (with `queue_max = 4`) is refused at once so the client can retry
elsewhere. Waiting is cancellable (the client may go away) and must leak neither a slot nor a
queue place.
## Files
- Copy: `internal/limiter/limiter_test.go`, `internal/choose/choose_test.go`
- Create: `internal/limiter/limiter.go`, `internal/choose/choose.go`
- Modify: `docs/implementer-log.md`
## Interfaces
`internal/limiter`, package `limiter`:
```go
var ErrQueueFull = errors.New("queue full")
type Limiter struct { /* private: mutex, map[(host, model)]*pair */ }
func New() *Limiter
func (l *Limiter) Configure(host, model string, parallel, queueMax int)
// Acquire returns when a slot is held. release gives it back (idempotent: a second call is a
// no-op). waited is how long the caller sat in the queue. Errors: ErrQueueFull immediately
// when queueMax waiters are already queued; ctx.Err() if ctx ends while waiting.
func (l *Limiter) Acquire(ctx context.Context, host, model string) (release func(), waited time.Duration, err error)
func (l *Limiter) InFlight(host, model string) int
func (l *Limiter) Queued(host, model string) int
func (l *Limiter) FreeSlots(host string) int // sum over the host's configured models of parallel - inflight (never below 0); 0 for an unknown host
```
Rules the tests check:
1. An unconfigured (host, model) behaves as `parallel = 1, queueMax = 0`.
2. FIFO: waiters get slots in arrival order. Suggested shape: a mutex, `inflight`, and a slice
of waiter channels; `release` pops the head waiter (if any) and hands the slot over without
ever decrementing `inflight`, else decrements. A waiter whose ctx ends removes itself from
the queue under the mutex; if the slot was handed to it in the same instant, it releases it.
3. `release` idempotent via `sync.Once`.
4. Never panic; all methods safe for concurrent use.
`internal/choose`, package `choose`:
```go
type Info struct {
Healthy, Draining, Loaded, CanServe bool // Loaded: model is resident; CanServe: config lists the model
Free, Queued int
Weight float64
}
// Best returns the candidate with the highest Free*Weight among those that are healthy, not
// draining and known (info ok) — preferring hosts with Loaded over merely CanServe; ties go to
// the lowest Queued, then to candidate order. A host with Free == 0 is still eligible (it will
// queue). ok is false when nothing is eligible.
func Best(candidates []string, info func(host string) (Info, bool)) (string, bool)
```
Rules: two passes — first over eligible candidates with `Loaded`, then, if none, over eligible
candidates with `CanServe`. Score `float64(Free) * Weight`. Compare with `>`; on equality prefer
lower `Queued`; on equality keep the earlier candidate.
## Steps
- [ ] **1. Copy.** `git switch v1`; `mkdir -p internal/limiter internal/choose`; copy both tests from `docs/plans/v1/_files/internal/…`.
- [ ] **2. See them fail** (compile). **3. Write both packages.** `gofmt -w internal/`.
- [ ] **4. See them pass.** `go test -race -count=1 ./internal/limiter/ ./internal/choose/`. The
limiter tests are timing-based with generous margins; run them three times: `-count=3`.
- [ ] **5. Run the gate.** `make gate`. **6. Log and commit.** Row `v1/03-limiter-choose`.
```sh
git add internal/limiter internal/choose docs/implementer-log.md
git commit
```
## Done when
- Both packages pass `-race -count=3`; `make gate` prints `gate: ok`; copied files byte-identical.
## Stop and report if
- A limiter test fails only sometimes: report which and how often; do not loosen it.