67 lines
3.0 KiB
Markdown
67 lines
3.0 KiB
Markdown
# v2 task 01: learn each host's context size from `/props`
|
|
|
|
**Branch:** `v2` (create it from `master`: `git switch master && git switch -c v2`; `git status --short` must be empty first, otherwise stop)
|
|
**Commit subject:** `Health: learn n_ctx and total_slots from /props`
|
|
|
|
## Goal
|
|
|
|
The poller already asks each host `/health` and `/v1/models`. It now also reads `/props` and
|
|
remembers the context size and slot count, so the proxy (task 02) can tell whether a prompt fits.
|
|
A missing or malformed `/props` is **not** a health failure: context is then simply unknown (0).
|
|
|
|
## Context
|
|
|
|
`llama-server` answers `GET /props` with a JSON object containing
|
|
`default_generation_settings.n_ctx` (the total context the server was started with) and
|
|
`total_slots` (how many parallel slots share it). With unified KV, one slot can use up to
|
|
`n_ctx / total_slots` tokens. Some builds omit fields; old ones 404. Read at most `MaxModelsBody`
|
|
bytes as for the other endpoints.
|
|
|
|
## Files
|
|
|
|
- Copy: `internal/health/props_test.go`
|
|
- Copy (**replaces** v1's): `internal/proxy/helpers_test.go` — the fake upstream now answers `/props` without counting it as a hit, so the v1 proxy tests' exact hit counts still hold once the poller asks for it
|
|
- Modify: `internal/health/health.go`, `internal/admin/admin.go` (or wherever `HostView` is built), `docs/implementer-log.md`
|
|
|
|
## Interfaces
|
|
|
|
`internal/health`, additions:
|
|
|
|
```go
|
|
type Status struct {
|
|
// …existing fields…
|
|
NCtx int `json:"n_ctx"` // total context from /props; 0 = unknown
|
|
Slots int `json:"slots"` // total_slots from /props; 0 = unknown
|
|
}
|
|
// PerSlotCtx is the context one request may use: NCtx / Slots, or NCtx when Slots is 0.
|
|
func (s Status) PerSlotCtx() int
|
|
```
|
|
|
|
Rules the tests check:
|
|
|
|
1. A poll is `/health`, `/v1/models` (as before), then `GET <base>/props`. If that request fails,
|
|
returns non-200, is not JSON, or lacks the fields, set `NCtx = 0`, `Slots = 0` and **do not
|
|
count the poll as failed**. Otherwise `NCtx = default_generation_settings.n_ctx`,
|
|
`Slots = total_slots` (negative values → 0).
|
|
2. `PerSlotCtx()` is integer division; `NCtx` when `Slots == 0`; 0 when `NCtx == 0`.
|
|
3. `admin.HostView` gains `NCtx int \`json:"n_ctx"\`` and `Slots int \`json:"slots"\`` copied
|
|
from the status (the v2 smoke reads `"n_ctx":8192` from `/_crossbar/hosts`). `admin_test.go`
|
|
must keep passing unchanged.
|
|
|
|
## Steps
|
|
|
|
- [ ] **1.** `git switch master && git switch -c v2`; `cp docs/plans/v2/_files/internal/health/props_test.go internal/health/`;
|
|
`cp docs/plans/v2/_files/internal/proxy/helpers_test.go internal/proxy/`.
|
|
- [ ] **2. See it fail** (compile: `NCtx` undefined). **3. Write the code.** `gofmt -w internal/`.
|
|
- [ ] **4.** `go test -race -count=1 ./internal/health/ ./internal/admin/` → both `ok`.
|
|
- [ ] **5.** `make gate` → `gate: ok`. **6.** Row `v2/01-props`; commit.
|
|
|
|
```sh
|
|
git add internal/health internal/admin internal/proxy/helpers_test.go docs/implementer-log.md
|
|
git commit
|
|
```
|
|
|
|
## Done when
|
|
|
|
- Both packages pass; gate ok; `props_test.go` byte-identical to `_files/`.
|