# v1 task 03: the per-(host, model) limiter and the chooser

**Branch:** `v1` (run `git switch v1`; `git status --short` must be empty, otherwise stop)
**Commit subject:** `Add the per-host-model limiter and the host chooser`

## Goal

Two pure packages. `limiter` hands out at most `parallel` concurrent slots per (host, model) and
lets at most `queue_max` requests wait in line, first come first served. `choose` picks the host
for a **new** lease: most free slots × weight, ties to the shortest queue, then list order.
This is `PLAN.md` §4 (`choose`) and §6 (concurrency).

## Context

A `llama-server` router with `parallel = 4` and unified KV falls over when a fifth request
arrives ("Context size has been exceeded"). The limiter is where that is prevented: the fifth
request waits, the ninth (with `queue_max = 4`) is refused at once so the client can retry
elsewhere. Waiting is cancellable (the client may go away) and must leak neither a slot nor a
queue place.

## Files

- Copy: `internal/limiter/limiter_test.go`, `internal/choose/choose_test.go`
- Create: `internal/limiter/limiter.go`, `internal/choose/choose.go`
- Modify: `docs/implementer-log.md`

## Interfaces

`internal/limiter`, package `limiter`:

```go
var ErrQueueFull = errors.New("queue full")

type Limiter struct { /* private: mutex, map[(host, model)]*pair */ }
func New() *Limiter
func (l *Limiter) Configure(host, model string, parallel, queueMax int)
// Acquire returns when a slot is held. release gives it back (idempotent: a second call is a
// no-op). waited is how long the caller sat in the queue. Errors: ErrQueueFull immediately
// when queueMax waiters are already queued; ctx.Err() if ctx ends while waiting.
func (l *Limiter) Acquire(ctx context.Context, host, model string) (release func(), waited time.Duration, err error)
func (l *Limiter) InFlight(host, model string) int
func (l *Limiter) Queued(host, model string) int
func (l *Limiter) FreeSlots(host string) int   // sum over the host's configured models of parallel - inflight (never below 0); 0 for an unknown host
```

Rules the tests check:

1. An unconfigured (host, model) behaves as `parallel = 1, queueMax = 0`.
2. FIFO: waiters get slots in arrival order. Suggested shape: a mutex, `inflight`, and a slice
   of waiter channels; `release` pops the head waiter (if any) and hands the slot over without
   ever decrementing `inflight`, else decrements. A waiter whose ctx ends removes itself from
   the queue under the mutex; if the slot was handed to it in the same instant, it releases it.
3. `release` idempotent via `sync.Once`.
4. Never panic; all methods safe for concurrent use.

`internal/choose`, package `choose`:

```go
type Info struct {
    Healthy, Draining, Loaded, CanServe bool   // Loaded: model is resident; CanServe: config lists the model
    Free, Queued                        int
    Weight                              float64
}
// Best returns the candidate with the highest Free*Weight among those that are healthy, not
// draining and known (info ok) — preferring hosts with Loaded over merely CanServe; ties go to
// the lowest Queued, then to candidate order. A host with Free == 0 is still eligible (it will
// queue). ok is false when nothing is eligible.
func Best(candidates []string, info func(host string) (Info, bool)) (string, bool)
```

Rules: two passes — first over eligible candidates with `Loaded`, then, if none, over eligible
candidates with `CanServe`. Score `float64(Free) * Weight`. Compare with `>`; on equality prefer
lower `Queued`; on equality keep the earlier candidate.

## Steps

- [ ] **1. Copy.** `git switch v1`; `mkdir -p internal/limiter internal/choose`; copy both tests from `docs/plans/v1/_files/internal/…`.
- [ ] **2. See them fail** (compile). **3. Write both packages.** `gofmt -w internal/`.
- [ ] **4. See them pass.** `go test -race -count=1 ./internal/limiter/ ./internal/choose/`. The
  limiter tests are timing-based with generous margins; run them three times: `-count=3`.
- [ ] **5. Run the gate.** `make gate`. **6. Log and commit.** Row `v1/03-limiter-choose`.

```sh
git add internal/limiter internal/choose docs/implementer-log.md
git commit
```

## Done when

- Both packages pass `-race -count=3`; `make gate` prints `gate: ok`; copied files byte-identical.

## Stop and report if

- A limiter test fails only sometimes: report which and how often; do not loosen it.