Add the smoke run, README and systemd unit
Implemented-By: OpenCode session (model recorded in docs/implementer-log.md)
This commit is contained in:
@@ -0,0 +1,106 @@
|
||||
# crossbar
|
||||
|
||||
crossbar is an affinity router in front of several `llama-server` routers. A client's identity is
|
||||
the first path segment of its base URL; v0 routes each request to the first healthy host on that
|
||||
route's list and streams the answer back unbuffered.
|
||||
|
||||
## Build
|
||||
|
||||
Build everything with `make build`; the binaries land in `bin/`. Check the work with `make gate`,
|
||||
which runs the formatter, vet, tests and line-length check with no network. When the code is ready,
|
||||
run `make smoke`, which starts two fake upstreams and exercises routing, failover, recovery and
|
||||
streaming over real HTTP.
|
||||
|
||||
## Configure
|
||||
|
||||
crossbar reads one TOML file. This is `example.toml`:
|
||||
|
||||
```toml
|
||||
# crossbar example configuration. Replace <tailnet> and the addresses with your own.
|
||||
listen = "127.0.0.1:17777" # never 0.0.0.0 — bind the tailnet address in production
|
||||
poll_interval = "1s" # 60s in production; 1s makes the smoke run quick
|
||||
queue_max = 8
|
||||
|
||||
[hosts.alpha]
|
||||
base_url = "http://127.0.0.1:18081" # e.g. http://straylight.<tailnet>:11434
|
||||
weight = 1.0
|
||||
models = { "ornith-1.5-35b-a3b" = { parallel = 4 }, "small-9b" = { parallel = 6 } }
|
||||
|
||||
[hosts.beta]
|
||||
base_url = "http://127.0.0.1:18082" # e.g. http://titan.<tailnet>:8081
|
||||
weight = 2.0
|
||||
models = { "ornith-1.5-35b-a3b" = { parallel = 4 } }
|
||||
|
||||
# v0: a route is a preference list; the first healthy host that has the model wins.
|
||||
[routes.opencode-a]
|
||||
hosts = ["alpha", "beta"]
|
||||
default_model = "ornith-1.5-35b-a3b"
|
||||
|
||||
[routes.hermes-x]
|
||||
hosts = ["beta", "alpha"]
|
||||
```
|
||||
|
||||
| Key | Meaning |
|
||||
| --- | --- |
|
||||
| `listen` | Where crossbar binds. A tailnet address, never `0.0.0.0`. |
|
||||
| `poll_interval` | How often each host is health-checked. 60s in production; 1s makes the smoke run quick. |
|
||||
| `queue_max` | Reserved for v1 queueing; no effect in v0. |
|
||||
| `hosts.<name>.base_url` | The llama-server base URL this host serves. |
|
||||
| `hosts.<name>.weight` | Relative share of new routes this host receives. |
|
||||
| `hosts.<name>.models` | The models this host serves, with per-model parallel tuning. |
|
||||
| `routes.<name>.hosts` | Preference order: the first healthy host that serves the model wins. |
|
||||
| `routes.<name>.default_model` | Model used when a request omits one; must be served by a host in the route. |
|
||||
|
||||
## Run
|
||||
|
||||
Copy the binary, the config and the unit into place, reload systemd, and start it:
|
||||
|
||||
```sh
|
||||
install -m 0755 bin/crossbar /usr/local/bin/crossbar
|
||||
install -d -m 0755 /etc/crossbar
|
||||
install -m 0644 crossbar.toml /etc/crossbar/crossbar.toml
|
||||
install -m 0644 deploy/crossbar.service /etc/systemd/system/crossbar.service
|
||||
systemctl daemon-reload
|
||||
systemctl enable --now crossbar
|
||||
```
|
||||
|
||||
## Point clients at it
|
||||
|
||||
OpenCode, one provider for every project. Each instance is launched as
|
||||
`CROSSBAR_ROUTE="$(basename "$PWD")-$$" opencode`:
|
||||
|
||||
```jsonc
|
||||
"provider": { "crossbar": { "npm": "@ai-sdk/openai-compatible",
|
||||
"options": { "baseURL": "http://crossbar.<tailnet>:7777/{env:CROSSBAR_ROUTE}/v1" },
|
||||
"models": { "ornith-1.5-35b-a3b": {} } } }
|
||||
```
|
||||
|
||||
Hermes, in `config.yaml`:
|
||||
|
||||
```yaml
|
||||
custom_providers:
|
||||
- name: crossbar
|
||||
base_url: http://crossbar.<tailnet>:7777/hermes-<agent>/v1
|
||||
models: { ornith-1.5-35b-a3b: {} }
|
||||
```
|
||||
|
||||
The route name in the URL must exist in `[routes]`; unknown routes are 404.
|
||||
|
||||
## Inspect
|
||||
|
||||
`GET /_crossbar/hosts` reports every host's health and loaded models:
|
||||
|
||||
```json
|
||||
{"alpha":{"healthy":true,"loaded":["ornith-1.5-35b-a3b","small-9b"],"last_ok":"2026-09-25T09:34:18Z","last_err":""},"beta":{"healthy":true,"loaded":["ornith-1.5-35b-a3b"],"last_ok":"2026-09-25T09:34:18Z","last_err":""}}
|
||||
```
|
||||
|
||||
`GET /_crossbar/routes` reports each route's preference order and default model:
|
||||
|
||||
```json
|
||||
{"hermes-x":{"hosts":["beta","alpha"],"default_model":""},"opencode-a":{"hosts":["alpha","beta"],"default_model":"ornith-1.5-35b-a3b"}}
|
||||
```
|
||||
|
||||
## What v0 does not do
|
||||
|
||||
Leases and stickiness, SQLite, `/slots`, queueing and wake-on-LAN are out of scope for v0; see
|
||||
`PLAN.md`.
|
||||
Reference in New Issue
Block a user