kyleandClaude Fable 5.1
89e47d83f8
Tests were run against a private reference implementation: gate ok after every task in order, smoke ok (stream spread ~1000 ms). The reference is not in the repository. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
110 lines
5.2 KiB
Markdown
110 lines
5.2 KiB
Markdown
# v0 task 02: the health poller
|
|
|
|
**Branch:** `v0` (run `git switch v0`; `git status --short` must be empty, otherwise stop)
|
|
**Commit subject:** `Add the health poller`
|
|
|
|
## Goal
|
|
|
|
Write `internal/health`: a table that knows, for every host, whether it answered its last poll,
|
|
which models it has loaded, and when it last answered. The proxy (task 03) reads this table to
|
|
choose a host and tells it when a request to a host fails.
|
|
|
|
## Context
|
|
|
|
A `llama-server` router answers `GET /health` with 200 when it can serve (503 while loading),
|
|
and `GET /v1/models` with `{"object":"list","data":[{"id":"<model>", …}, …]}` listing the models
|
|
it has resident. A host that failed must answer **two** polls in a row before it is trusted again
|
|
(`RecoveryPolls`), because a router that is loading a model flaps. A host that has never failed is
|
|
trusted after its first good poll. **Never call `/slots`**: probing an unloaded model makes the
|
|
router load it.
|
|
|
|
## Files
|
|
|
|
- Copy: `internal/health/health_test.go`
|
|
- Create: `internal/health/health.go`
|
|
- Modify: `docs/implementer-log.md`
|
|
|
|
## Interfaces
|
|
|
|
Produces, in `internal/health/health.go`, package `health`:
|
|
|
|
```go
|
|
const RecoveryPolls = 2 // good polls in a row a host needs after a failure
|
|
const MaxModelsBody = 1 << 20 // bound on what we read from /v1/models
|
|
|
|
type Status struct {
|
|
Healthy bool `json:"healthy"`
|
|
Loaded []string `json:"loaded"` // sorted, unique model ids from the last good poll
|
|
LastOK time.Time `json:"last_ok"` // zero if never
|
|
LastErr string `json:"last_err"` // "" after a good poll
|
|
Consecutive int `json:"consecutive"` // good polls in a row
|
|
}
|
|
|
|
type Table struct { /* private: a mutex, name -> base URL, name -> status + "ever failed" flag, client, interval */ }
|
|
|
|
func New(hosts map[string]string, interval time.Duration, client *http.Client) *Table
|
|
func (t *Table) Run(ctx context.Context) // PollOnce now, then every interval; returns when ctx is done
|
|
func (t *Table) PollOnce(ctx context.Context) // polls every host concurrently, returns when all are done
|
|
func (t *Table) Get(name string) (Status, bool) // a copy; ok == false for an unknown name
|
|
func (t *Table) All() map[string]Status // copies
|
|
func (t *Table) MarkDown(name, reason string) // a failure seen by the proxy
|
|
```
|
|
|
|
Rules the tests check:
|
|
|
|
1. **`New`**: `hosts` maps a host name to its base URL (no trailing slash; the config already
|
|
trimmed it). A nil `client` becomes `&http.Client{Timeout: 5 * time.Second}`. Every host
|
|
starts with `Healthy false`, `Loaded` an **empty slice, not nil**, `Consecutive 0`.
|
|
2. **One poll of one host** is two requests with `ctx`: `GET <base>/health` must return 200,
|
|
else the poll fails with `LastErr` starting `health: ` (for example `health: HTTP 503`, or the
|
|
client error); then `GET <base>/v1/models` must return 200 and decode as
|
|
`{"data":[{"id":"…"}]}`, else the poll fails with `LastErr` starting `models: `. Read at most
|
|
`MaxModelsBody` bytes of either body. `Loaded` is the ids, empty ids dropped, duplicates
|
|
dropped, sorted.
|
|
3. **Success**: `Consecutive++`, `LastOK = time.Now()`, `LastErr = ""`, `Loaded` replaced, and
|
|
`Healthy = (never failed) || Consecutive >= RecoveryPolls`.
|
|
4. **Failure**, from a poll or from `MarkDown`: remember that the host has failed, `Healthy =
|
|
false`, `Consecutive = 0`, `LastErr = reason`. `Loaded` is **left as last seen**.
|
|
`MarkDown` uses `reason = "marked down: " + reason`. Unknown names are ignored, never a panic.
|
|
5. **A cancelled context is not a failure.** If a request fails and `ctx.Err() != nil`, record
|
|
nothing: we were told to stop, that says nothing about the host.
|
|
6. **`Run`** polls once immediately, then on a `time.Ticker` every `interval`, and returns when
|
|
`ctx` is done (stop the ticker). **`PollOnce`** polls all hosts concurrently (one goroutine
|
|
each, a `sync.WaitGroup`) and returns when all have finished.
|
|
7. **`Get` and `All` return copies**: changing `Loaded` on what they return must not change the
|
|
table (copy the slice). All methods are safe to call from several goroutines: one mutex around
|
|
the map, never held while a request is in flight.
|
|
|
|
## Steps
|
|
|
|
- [ ] **1. Copy.**
|
|
|
|
```sh
|
|
git switch v0
|
|
cp docs/plans/v0/files/internal/health/health_test.go internal/health/
|
|
```
|
|
|
|
Read the test. `newFake` is the router stand-in; `TestFailureThenRecoveryNeedsTwoPolls` is rule 3
|
|
and 4 in one story.
|
|
|
|
- [ ] **2. See the test fail.** `go test ./internal/health/`. Expected: it does not compile.
|
|
- [ ] **3. Write `internal/health/health.go`.** `gofmt -w internal/health/`.
|
|
- [ ] **4. See the test pass.** `go test -race -count=1 ./internal/health/`. Expected: `ok`.
|
|
`TestRunPollsOnStart` polls a fake every 20 ms; if it fails, check rules 5 and 6.
|
|
- [ ] **5. Run the gate.** `make gate`. Expected last line: `gate: ok`.
|
|
- [ ] **6. Log and commit.** Row `v0/02-health`.
|
|
|
|
```sh
|
|
git add internal/health docs/implementer-log.md
|
|
git commit
|
|
```
|
|
|
|
## Done when
|
|
|
|
- `go test -race -count=1 ./internal/health/` is `ok`; `make gate` prints `gate: ok`.
|
|
- `cmp internal/health/health_test.go docs/plans/v0/files/internal/health/health_test.go` prints nothing.
|
|
|
|
## Stop and report if
|
|
|
|
- The race detector reports a race you cannot remove with the one-mutex design above.
|