crossbar/PLAN.md

# crossbar — an affinity router for the fleet's llama-servers

**Status:** plan, 2026-09-25. Nothing built. Go.
**Owner:** Kyle. Drafted by claude from the 2026-09-25 discussion (Inference Infrastructure).

## 1. Problem

Three hosts run `llama-server` in router mode with the *same model ids*
(straylight, titan, soon dixie). Clients (OpenCode per project, Hermes agents,
the paper writer) each hard-code one host. Consequences seen this week:

- a host's prompt cache is per process; an agent session sits at 100–230k
  tokens with ~100 % cache hits. Any move between hosts re-prefills the whole
  context (≈ 7–8 min at straylight's ~500 t/s prompt processing);
- two turns landing on one 35B at once overflowed its unified KV
  ("Context size has been exceeded"); nothing in the path knows the slot count;
- titan sleeps; clients pointed at it fail instead of falling back;
- rebuilding or restarting a host takes its clients down with it.

A crossbar switch connects each caller to a line and *holds the connection*.
That is the design: **affinity first, balancing only for new sessions.**

## 2. Non-goals

- No token counting, prompt rewriting, batching or splitting across hosts.
  The hosts are bandwidth-bound; there is nothing to gain cross-host.
- No model management. Loading/unloading stays with each host's llama-server
  router (`--models-max`). crossbar never sends a request that would make a
  host load a model it does not have resident.
- No auth in v0 (tailnet-only bind). See §9.

## 3. Terminology

- **host** — one llama-server router endpoint (`base_url`, list of model ids
  it can serve, per-model `parallel`, a speed `weight`).
- **route** — a client identity = the first path segment of the request URL
  (`/opencode-a/v1/...` → route `opencode-a`). The client only ever knows a
  base URL, so this needs no client support beyond "configurable base URL".
- **lease** — `(route, model) → host`, with `created`, `last_used`,
  `state ∈ {active, pinned, draining}`.

## 4. Request flow

```
client  ── /{route}/v1/chat/completions ──▶ crossbar
        1. route := first path segment; strip it
        2. model := body.model (JSON peek; fall back to route default)
        3. lease := table[(route, model)]
             hit  & host healthy & model resident → use it
             miss | host unhealthy               → choose(host) ; write lease
        4. acquire one concurrency token for (host, model)   [bounded queue]
        5. reverse-proxy to host.base_url + "/v1/..." ; stream through,
           flush on every chunk; passive health on connect error / 5xx
        6. release token; lease.last_used = now
```

`choose(host)`: among hosts that are healthy **and list the model as loaded**
(`/models`), take the one with the most free slots × `weight`; tie → lowest
current queue depth. If no host has it loaded, take a healthy host that *can*
serve it (config) and accept the load; log that decision.

Pass-through per route, all answered from the leased host (or from config if
no lease yet): `/v1/models`, `/health`, `/props`, `/v1/embeddings`,
`/v1/completions`. Everything else 404.

## 5. Lease rules

- A lease is **sticky**. It moves only when: the host fails health, the route
  is idle longer than `lease_idle` (default 30 min), or an operator pins/
  releases it. A faster host coming back online does *not* move an active
  lease — the re-prefill is the cost we are avoiding.
- **Pinned** leases never move automatically ("project A goes to titan right
  now"). **Draining** hosts accept no new leases; existing ones finish.
- Table persisted to `state.json` after every change; loaded at start so a
  crossbar restart does not reshuffle sessions.

## 6. Health

- Poller, every `poll_interval` (60 s): `GET /health` then `GET /models` per
  host. Record loaded models. Only then `GET /slots?model=X` **for loaded
  models only** — probing an unloaded model makes llama-server load it.
  Per-model free-slot count from `/slots` (or, if `/slots` is disabled on a
  host, assume `parallel` minus our own in-flight count).
- Passive: a connection error or 5xx on a proxied request marks the host
  unhealthy immediately and re-leases the route on the client's retry.
  Recovery only through the poller (two consecutive good polls).
- Concurrency: per `(host, model)` token bucket of size `parallel`. Requests
  beyond it wait in a bounded FIFO (`queue_max`, default 8; 503 beyond that).
  This is where the unified-KV overflow is prevented.

## 7. Admin API (tailnet only, same listener, prefix `/_crossbar`)

- `GET  /_crossbar/hosts`   — health, loaded models, free slots, in-flight.
- `GET  /_crossbar/routes`  — the lease table.
- `POST /_crossbar/routes/{route}` `{ "host": "titan", "pin": true }` — pin;
  `{ "release": true }` — drop the lease (next request re-chooses).
- `POST /_crossbar/hosts/{host}` `{ "drain": true|false }`.
- `GET  /_crossbar/metrics` — Prometheus: requests, queue wait, lease moves,
  host health, tokens/s from llama-server's `timings` when present. Scrape it
  from the fleet Prometheus on orion.

## 8. Config (`crossbar.yaml`)

```yaml
listen: "100.x.y.z:7777"        # tailnet address only; never 0.0.0.0
poll_interval: 60s
lease_idle: 30m
queue_max: 8
hosts:
  straylight:
    base_url: http://straylight.<tailnet>:11434
    weight: 1.0
    models: { ornith-1.5-35b-a3b: {parallel: 4}, ornith-1.5-9b-uncensored: {parallel: 6} }
  titan:
    base_url: http://titan.<tailnet>:8081
    weight: 2.0                 # ~2× straylight decode
    models: { ornith-1.5-35b-a3b: {parallel: 4}, laguna-s-2.1: {parallel: 2} }
  dixie:
    base_url: http://dixie.<tailnet>:11434
    weight: 0.8
    models: { ornith-1.5-9b-uncensored: {parallel: 6} }
routes:                          # optional defaults / pins
  opencode-a: { default_model: ornith-1.5-35b-a3b }
  opencode-b: { default_model: ornith-1.5-35b-a3b }
  paper:      { default_model: qwen3.8-27b-uncensored, pin: titan }
  hermes-straylight: {}
  hermes-talos: {}
  hermes-titan: { pin: titan }   # titan's models are the titan agent's first
```

Client side, no code changes:

- OpenCode: project-local `opencode.json` provider with
  `options.baseURL: http://crossbar:7777/opencode-a/v1` (global + project
  configs merge, so each project carries its own route).
- Hermes: `custom_providers[].base_url: http://crossbar:7777/hermes-<agent>/v1`
  (and the `delegation` / `auxiliary` blocks that point at a router today).
- tirith: crossbar's plain-HTTP tailnet URL needs the same narrow trust entry
  the llama routers already have (`plain_http_to_sink` for that host only).

## 9. Security notes

- Bind to the tailnet address only. v0 relies on the tailnet for
  authentication; every route is reachable by every tailnet peer, which is the
  same exposure the llama-servers have today. v1 option: read Tailscale
  identity (`tailscale whois` on the peer address) and restrict routes to
  peers, so `hermes-talos` can only be used from talos.
- Admin API: same listener, same trust. Consider a separate `admin_listen` on
  localhost if crossbar runs on a shared host.
- No secrets in config; llama-servers take no keys.
- Request bodies are proxied, never logged. Metrics carry counts and
  timings only.

## 10. Milestones

- **v0 (a day):** config, static routes → host mapping, reverse proxy with
  streaming, `/health` + `/models` poller, passive health, `/_crossbar/hosts`.
  Replaces the hand-maintained provider lists. No leases yet: each route has
  a fixed host list in preference order; first healthy wins.
- **v1:** lease table with persistence, `choose()` by free slots × weight,
  per-(host, model) concurrency + bounded queue, pin/release/drain, metrics.
- **v2:** wake-on-LAN for a sleeping titan when a new lease wants it (wait ≤
  45 s, else fall through); optional conversation fingerprint (system prompt +
  first user message) for per-conversation stickiness inside one route;
  Tailscale-identity route restriction.

## 11. Testing

- `httptest` fake llama-server: `/health`, `/models`, `/slots`, streaming
  `/v1/chat/completions` with configurable latency and failure injection.
- Table tests for `choose()`, lease expiry, drain, passive-health re-lease.
- One integration script against the real fleet: one long conversation,
  assert every turn hits the same host (llama-server `timings.cache_n` or
  `prompt_n` small after the first turn).

## 12. Open questions

1. Where it runs — Kyle: "a Raspberry Pi, it doesn't matter where". Static
   Go binary + systemd unit; arm64 cross-compile is free. Candidate:
   hyperborea, in its own unit, outside the collector's disk budget.
2. `/slots` is disabled by default on recent llama-server builds; enable
   `--slots` on the three routers or rely on our own in-flight counts.
3. Should `default_model` rewrite a client's `model` field? Proposal: no —
   proxy what the client sent; only use the default when the body has none.
4. Does the titan agent's reservation (2026-09-21) stand as a pin, or does
   titan become a general pool member now that health is tracked? Kyle said
   titan could join the pool (2026-09-25); the pin above keeps its own agent
   first without excluding others.