58 lines
2.6 KiB
Markdown
58 lines
2.6 KiB
Markdown
# v2.3 task 04: llama-server's error shape for a context refusal; README
|
|
|
|
**Branch:** `v2.3` (`git switch v2.3`; `git status --short` must be empty, otherwise stop)
|
|
**Commit subject:** `Context refusal in llama-server's exceed_context_size_error shape; README for v2.3`
|
|
|
|
## Goal
|
|
|
|
When no host can fit a prompt, crossbar answers `400 {"error":"prompt too large","estimate":N,"max":M}`.
|
|
A client that already handles llama-server's own overflow error (Boxmaker keys on `error.type`
|
|
and reads only the first 4 KiB) does not recognise it. After this task the body is the server's
|
|
shape, so the client handles crossbar's refusal like the server's:
|
|
|
|
```json
|
|
{"error":{"code":400,"type":"exceed_context_size_error","message":"prompt too large","n_prompt_tokens":N,"n_ctx":M}}
|
|
```
|
|
|
|
`N` is the estimate and `M` the largest per-slot context on the route, as before.
|
|
|
|
## Files
|
|
|
|
- Copy: the **replacements** `internal/proxy/ctxguard_test.go` (was v2's) and
|
|
`internal/proxy/ctxguard_router_test.go` (was v2.1's); only their 400-body assertions changed;
|
|
the v2.3 copies are now protected
|
|
- Modify: `internal/proxy/ctxguard.go` (`refuseCtx`), `README.md`, `docs/implementer-log.md`
|
|
|
|
## Rules
|
|
|
|
1. The body is exactly one JSON object whose only top-level key is `"error"`, so it starts with
|
|
`{"error":`; `Content-Type: application/json`; status 400. The accounting row is unchanged
|
|
(status 400, `Err` "prompt too large"). Every other crossbar error keeps its current
|
|
`{"error":"<text>"}` shape.
|
|
2. `README.md`:
|
|
- The context-guard section: the new body.
|
|
- A new section **"Clients that manage their own slots"** covering: control calls (which
|
|
requests, and that they follow the lease but take no slot, skip the guard and write no
|
|
row); `/slots` and `/tokenize` are proxied, `/slots/<id>` actions are not; a GET's model
|
|
comes from `?model=`; the route keys `affinity`, `queue` and `listen` with the
|
|
`boxmaker-a` example from `example.toml`; that `listen` is refused on templates and must
|
|
not be the main address; that the admin API is not served on a route listener.
|
|
- The "hosts view"/config reference tables, if they list route keys, gain the three keys.
|
|
|
|
## Steps
|
|
|
|
- [ ] **1.** `git switch v2.3`; copy the replacement test. **2. See it fail** (old body).
|
|
- [ ] **3.** `refuseCtx`. **4.** README.
|
|
- [ ] **5.** `gofmt -w`; `make gate`; `make smoke` (check 3 still finds `"prompt too large"`).
|
|
- [ ] **6.** Row `v2.3/04-ctx-error-docs`; commit.
|
|
|
|
```sh
|
|
git add internal/proxy README.md docs/implementer-log.md
|
|
git commit
|
|
```
|
|
|
|
## Done when
|
|
|
|
- All tests pass; gate and smoke ok; given files byte-identical; README describes what v2.3
|
|
does and nothing it does not.
|