crossbar/docs/plans/v1/03-limiter-choose.md

v1 task 03: the per-(host, model) limiter and the chooser

Branch: v1 (run git switch v1; git status --short must be empty, otherwise stop) Commit subject: Add the per-host-model limiter and the host chooser

Goal

Two pure packages. limiter hands out at most parallel concurrent slots per (host, model) and lets at most queue_max requests wait in line, first come first served. choose picks the host for a new lease: most free slots × weight, ties to the shortest queue, then list order. This is PLAN.md §4 (choose) and §6 (concurrency).

Context

A llama-server router with parallel = 4 and unified KV falls over when a fifth request arrives ("Context size has been exceeded"). The limiter is where that is prevented: the fifth request waits, the ninth (with queue_max = 4) is refused at once so the client can retry elsewhere. Waiting is cancellable (the client may go away) and must leak neither a slot nor a queue place.

Files

• Copy: internal/limiter/limiter_test.go, internal/choose/choose_test.go
• Create: internal/limiter/limiter.go, internal/choose/choose.go
• Modify: docs/implementer-log.md

Interfaces

internal/limiter, package limiter:

var ErrQueueFull = errors.New("queue full")

type Limiter struct { /* private: mutex, map[(host, model)]*pair */ }
func New() *Limiter
func (l *Limiter) Configure(host, model string, parallel, queueMax int)
// Acquire returns when a slot is held. release gives it back (idempotent: a second call is a
// no-op). waited is how long the caller sat in the queue. Errors: ErrQueueFull immediately
// when queueMax waiters are already queued; ctx.Err() if ctx ends while waiting.
func (l *Limiter) Acquire(ctx context.Context, host, model string) (release func(), waited time.Duration, err error)
func (l *Limiter) InFlight(host, model string) int
func (l *Limiter) Queued(host, model string) int
func (l *Limiter) FreeSlots(host string) int   // sum over the host's configured models of parallel - inflight (never below 0); 0 for an unknown host

Rules the tests check:

1. An unconfigured (host, model) behaves as parallel = 1, queueMax = 0.
2. FIFO: waiters get slots in arrival order. Suggested shape: a mutex, inflight, and a slice of waiter channels; release pops the head waiter (if any) and hands the slot over without ever decrementing inflight, else decrements. A waiter whose ctx ends removes itself from the queue under the mutex; if the slot was handed to it in the same instant, it releases it.
3. release idempotent via sync.Once.
4. Never panic; all methods safe for concurrent use.

internal/choose, package choose:

type Info struct {
    Healthy, Draining, Loaded, CanServe bool   // Loaded: model is resident; CanServe: config lists the model
    Free, Queued                        int
    Weight                              float64
}
// Best returns the candidate with the highest Free*Weight among those that are healthy, not
// draining and known (info ok) — preferring hosts with Loaded over merely CanServe; ties go to
// the lowest Queued, then to candidate order. A host with Free == 0 is still eligible (it will
// queue). ok is false when nothing is eligible.
func Best(candidates []string, info func(host string) (Info, bool)) (string, bool)

Rules: two passes — first over eligible candidates with Loaded, then, if none, over eligible candidates with CanServe. Score float64(Free) * Weight. Compare with >; on equality prefer lower Queued; on equality keep the earlier candidate.

Steps

• 1. Copy. git switch v1; mkdir -p internal/limiter internal/choose; copy both tests from docs/plans/v1/_files/internal/….
• 2. See them fail (compile). 3. Write both packages. gofmt -w internal/.
• 4. See them pass. go test -race -count=1 ./internal/limiter/ ./internal/choose/. The limiter tests are timing-based with generous margins; run them three times: -count=3.
• 5. Run the gate. make gate. 6. Log and commit. Row v1/03-limiter-choose.

git add internal/limiter internal/choose docs/implementer-log.md
git commit

Done when

• Both packages pass -race -count=3; make gate prints gate: ok; copied files byte-identical.

Stop and report if

• A limiter test fails only sometimes: report which and how often; do not loosen it.