kyleandClaude Fable 5.1
74b8af4ce0
v2 given tests compiled against a panic-only skeleton (go vet clean); no reference implementation. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
68 lines
3.3 KiB
Markdown
68 lines
3.3 KiB
Markdown
# v2 task 02: the context-size guard
|
|
|
|
**Branch:** `v2` (run `git switch v2`; `git status --short` must be empty, otherwise stop)
|
|
**Commit subject:** `Proxy: move or refuse prompts that do not fit the leased host's context`
|
|
|
|
## Goal
|
|
|
|
The single worst failure a client sees today is the upstream's "Context size has been exceeded"
|
|
after a long wait. With `PerSlotCtx` known (task 01), crossbar can estimate a prompt's size from
|
|
its body and act before forwarding: move the conversation to a host where it fits, or answer 400
|
|
with the estimate and the largest slot available. `PLAN.md` §4b.
|
|
|
|
## Files
|
|
|
|
- Copy: `internal/proxy/ctxguard_test.go`
|
|
- Modify: `internal/proxy/proxy.go` and/or `forward.go` (new file `ctxguard.go` if that keeps files under 400 lines), `internal/lease/lease.go` (one addition, below), `docs/implementer-log.md`
|
|
|
|
## Interfaces
|
|
|
|
```go
|
|
// internal/proxy
|
|
const CtxHeader = "X-Crossbar-Ctx" // set only when the guard moved a conversation: "moved:<from>>><to>" e.g. "moved:small>big"
|
|
|
|
// internal/lease — one addition, the only change allowed there:
|
|
// Move re-leases k onto host (deleting any existing lease for k), records a LeaseEvent with
|
|
// Reason "ctx", FromHost the previous host ("" if none), ToHost host. Returns an error only from
|
|
// the persister.
|
|
func (t *Table) Move(k Key, host string, now time.Time) error
|
|
```
|
|
|
|
Rules the tests check (`ServeHTTP`, between acquiring the lease and taking a slot):
|
|
|
|
1. `estimate := int(float64(len(body)) / 4 * 1.2)` tokens (body = the bytes already peeked; GET/HEAD → 0).
|
|
2. `limit := PerSlotCtx` of the leased host's `health.Status`. If `limit == 0` (unknown) or
|
|
`estimate <= limit`: no action, no header.
|
|
3. Otherwise find, among the route's candidate hosts (in order), the healthy, non-draining hosts
|
|
whose `PerSlotCtx() >= estimate` — prefer one that lists the model as loaded, else one that
|
|
can serve it (`cfg.Serves`). If one exists: `leases.Move(key, host, now)`, set
|
|
`CtxHeader` to `moved:<old>><new>`, and continue with the new host (**this request and the
|
|
following turns**: the lease moved).
|
|
4. If none exists: **400** `{"error":"prompt too large","estimate":E,"max":M}` where `M` is the
|
|
largest `PerSlotCtx()` among the route's healthy hosts (0 if all unknown — but then rule 2
|
|
already let the request through). Record an accounting row with status 400 and
|
|
`Err: "prompt too large"`; do not mark anything down; do not forward.
|
|
5. The estimate is never logged with the body; the log line gains `ctx_est=E` only.
|
|
|
|
Add `ReasonCtx = "ctx"` to `internal/store` constants (one-line change, allowed).
|
|
|
|
## Steps
|
|
|
|
- [ ] **1.** `git switch v2`; `cp docs/plans/v2/_files/internal/proxy/ctxguard_test.go internal/proxy/`.
|
|
- [ ] **2. See it fail** (compile: `CtxHeader`). **3. Write the code.** `gofmt -w internal/`.
|
|
- [ ] **4.** `go test -race -count=2 ./internal/proxy/ ./internal/lease/` → `ok`.
|
|
- [ ] **5.** `make gate` → `gate: ok`. **6.** Row `v2/02-ctxguard`; commit.
|
|
|
|
```sh
|
|
git add internal/proxy internal/lease internal/store docs/implementer-log.md
|
|
git commit
|
|
```
|
|
|
|
## Done when
|
|
|
|
- Tests pass with `-race -count=2`; gate ok; the copied test is byte-identical.
|
|
|
|
## Stop and report if
|
|
|
|
- `TestStickyLeaseSurvivesGrowthUntilItDoesNotFit` fails on the *second* request after the move (the lease did not actually move): quote the lease table.
|