94 lines
4.7 KiB
Markdown
94 lines
4.7 KiB
Markdown
# v2.1 task 02: router mode — loaded means loaded, context is per model
|
||
|
||
**Branch:** `v2.1` (`git switch v2.1`; `git status --short` must be empty, otherwise stop)
|
||
**Commit subject:** `Learn per-model context from /props?model=; only status "loaded" is loaded`
|
||
|
||
## Goal
|
||
|
||
crossbar's real upstreams are llama-server **routers**, not single servers, and v2's poller was
|
||
written against the single-server shape. On a router: `/v1/models` lists every configured model
|
||
with a `status.value` (`"loaded"`, `"unloaded"`, `"loading"`); the plain `/props` answers as the
|
||
router itself (`"role":"router"`, `n_ctx` 0); and `/props?model=X` answers for X's child server —
|
||
**and loads X if it is not loaded**, which a health poll must never cause. After this task the
|
||
poller treats only `"loaded"` models as loaded, asks `/props?model=X` only for those, keeps the
|
||
answers per model, and the guard and the hosts view use the per-model figures. A plain single
|
||
server keeps working exactly as in v2.
|
||
|
||
## Context
|
||
|
||
Verified on straylight's router on 2026-09-25: `/v1/models` entries carry
|
||
`"status":{"value":"unloaded",...}` for seven of eight models; plain `/props` returns
|
||
`{"role":"router","model_alias":"llama-server","default_generation_settings":{"n_ctx":0}}`;
|
||
`/props?model=ornith-1.5-35b-a3b` (loaded) returns `n_ctx` 262144 and `total_slots` 4. With v2's
|
||
code every listed model counts as loaded and the guard learns nothing, so the "Context size has
|
||
been exceeded" failure crossbar exists to prevent still happens on a router.
|
||
|
||
## Files
|
||
|
||
- Copy: `internal/health/props_router_test.go`, `internal/proxy/ctxguard_router_test.go`,
|
||
`internal/admin/admin_models_test.go`
|
||
- Modify: `internal/health/health.go` (a new `internal/health/props.go` is allowed for the 400-line
|
||
limit), `internal/proxy/ctxguard.go`, `internal/admin/admin.go`, `docs/implementer-log.md`
|
||
|
||
## Interfaces
|
||
|
||
```go
|
||
package health
|
||
|
||
// ModelCtx is what /props?model=X taught us about one loaded model.
|
||
type ModelCtx struct {
|
||
NCtx int `json:"n_ctx"`
|
||
Slots int `json:"slots"`
|
||
}
|
||
|
||
type Status struct {
|
||
// ... as v2 ...
|
||
Models map[string]ModelCtx `json:"models"` // per loaded model; never nil after a poll; copied by Get/All
|
||
}
|
||
|
||
// PerSlotCtxFor is the per-slot context for one model on this host: Models[model] when present
|
||
// (NCtx/Slots, 0 when either is 0); else, when model is in Loaded, the host-level PerSlotCtx();
|
||
// else 0 ("unknown" / not resident).
|
||
func (s Status) PerSlotCtxFor(model string) int
|
||
```
|
||
|
||
Poller rules:
|
||
1. `/v1/models`: an entry is loaded when it has no `status` or `status.value == "loaded"`; any
|
||
other value (`"unloaded"`, `"loading"`, …) is **not loaded** and does not appear in `Loaded`.
|
||
2. Plain `/props`: when the body has `"role":"router"` the host-level `NCtx`/`Slots` stay 0
|
||
whatever else it says; otherwise as v2 (best effort, never a failure).
|
||
3. For each model in `Loaded`, `GET /props?model=<url.QueryEscape(id)>`, decoded like the plain
|
||
one, into `Models[id]`. A failed or malformed answer leaves that id absent and the host
|
||
healthy. Never ask for a model that is not in `Loaded`.
|
||
4. `Models` is a fresh non-nil map on every successful poll; `MarkDown` leaves it as last seen.
|
||
|
||
Proxy (`ctxguard.go`): every use of `PerSlotCtx()` becomes `PerSlotCtxFor(model)` — the leased
|
||
host's figure, the candidates a prompt may move to, the wake path's "cannot serve it no matter
|
||
how it wakes" check, and `largestSlotCtx`, which now takes the model and so only counts hosts
|
||
that have it loaded. Admin (`admin.go`): `HostView` gains `Models map[string]health.ModelCtx`
|
||
(JSON `models`, an empty object never `null`).
|
||
|
||
## Steps
|
||
|
||
- [ ] **1.** `git switch v2.1`; copy the three given tests.
|
||
- [ ] **2. See them fail:** `go test -race -count=1 ./internal/health/ ./internal/proxy/ ./internal/admin/` — the new tests fail to compile until the names exist, then fail on behaviour.
|
||
- [ ] **3.** `health` first (rules 1–4, `ModelCtx`, `PerSlotCtxFor`); `go test -race -count=1 ./internal/health/` → `ok`.
|
||
- [ ] **4.** `ctxguard.go`, then `admin.go`; each package `ok`. `gofmt -w`.
|
||
- [ ] **5.** `go test -race -count=1 ./...` → all `ok`; `make smoke` → `smoke: ok` (the smoke's fake is a single server; nothing there changes).
|
||
- [ ] **6.** `make gate`. **7.** Row `v2.1/02-props-loaded-only`; commit.
|
||
|
||
```sh
|
||
git add internal/health internal/proxy internal/admin docs/implementer-log.md
|
||
git commit
|
||
```
|
||
|
||
## Done when
|
||
|
||
- All three given tests pass; every v2 health/guard/admin test still passes; gate and smoke ok;
|
||
given files byte-identical; no file over 400 lines.
|
||
|
||
## Stop and report if
|
||
|
||
- A v2 given test (`props_test.go`, `ctxguard_test.go`, `admin_test.go`, `health_test.go`) needs
|
||
changing to pass: quote it — that is the owner's test, not yours to edit.
|