crossbar/docs/plans/v2.3/02-affinity-queue.md

v2.3 task 02: route affinity and queue = false

Branch: v2.3 (git switch v2.3; git status --short must be empty, otherwise stop) Commit subject: Routes may share one lease (affinity = "route") and skip crossbar's queue (queue = false)

Goal

Two route keys for a client that manages its own slot:

• affinity = "route" — one lease for the whole route. Today each conversation (fingerprint) gets its own lease, and calls without a fingerprint lease "the route itself"; a client whose /control and /slots calls must reach the host its chat is on needs them all on one lease.
• queue = false — crossbar never holds or refuses the route's requests. The client pins its llama-server slot (id_slot), so llama-server queues it and its /slots shows the slot busy; a request held in crossbar's queue instead looks idle to the client, which gives up after 30 s. The requests still count as load on the host, so routes that do queue see the host full.

Files

• Copy: internal/config/config_v23_test.go, internal/limiter/track_test.go, internal/proxy/affinity_test.go
• Modify: internal/config/route.go (move the Route struct here from config.go, which is at 398 lines, and add the fields and methods here; config.go keeps a one-line call into the route validation), internal/config/config.go (minimal), internal/limiter/limiter.go, internal/proxy/proxy.go (or control.go if proxy.go would pass 400 lines), docs/implementer-log.md

Interfaces

package config

type Route struct {
	Hosts        []string `toml:"hosts"`
	DefaultModel string   `toml:"default_model"`
	Peers        []string `toml:"peers"`
	Affinity     string   `toml:"affinity"` // "" or "conversation" (the default), or "route"
	Queue        *bool    `toml:"queue"`    // nil means true
}

// PerRoute reports affinity = "route": every request on the route shares one lease.
func (r Route) PerRoute() bool

// Queues reports whether the route's requests wait in (and can be refused by) crossbar's
// per-(host, model) queue; false only for queue = false.
func (r Route) Queues() bool

package limiter

// Track counts one request against (host, model) without waiting and without refusing: in flight
// may exceed parallel. The returned release is idempotent.
func (l *Limiter) Track(host, model string) (release func())

Rules

1. Validation. affinity other than "", "conversation" or "route" is an error whose text contains routes.<name>.affinity (for example routes.convo.affinity: must be "conversation" or "route"). Both keys are allowed on templates; cfg.Route(name) returns them for every concrete route the template serves.
2. Lease key. For a PerRoute() route the lease key's fingerprint is "" for every request (chat or control), so every request on the route uses one lease per model. The accounting row keeps the request's real fingerprint (it is still useful in usage views).
3. Queue. For a route where Queues() is false, a non-control request takes lim.Track(host, model) instead of lim.Acquire (and releases it when done, like the slot). Control calls (task 01) take neither.
4. Release rule. A release — from Acquire's slot or from Track — hands the slot to the first waiter only when in flight ≤ parallel at that moment; otherwise it just decrements in flight. (With only Acquire in use in-flight never exceeds parallel, so today's behaviour is unchanged.) InFlight, FreeSlots and Queued count tracked requests like any other.
5. The context guard runs as today on both kinds of route.

Steps

• 1. git switch v2.3; copy the three given tests.
• 2. See them fail (compile: PerRoute, Queues, Track missing).
• 3. route.go (struct move, fields, methods, validation). 4. limiter.go (Track, the release rule). 5. proxy.go (lease key, Track).
• 6. gofmt -w; go test -race -count=3 ./internal/limiter/ ./internal/config/ ./internal/proxy/ → ok.
• 7. make gate; make smoke. 8. Row v2.3/02-affinity-queue; commit.

git add internal/config internal/limiter internal/proxy docs/implementer-log.md
git commit

Done when

• The new tests and every earlier test pass under -race -count=3; every earlier limiter test is still green (the release rule must not change Acquire-only behaviour); gate and smoke ok; given files byte-identical; no file over 400 lines.

Stop and report if

• The struct move breaks a given test, or the release rule cannot be met without changing Acquire's results in an earlier test.