deploy/hyperborea: production config, user unit, install script, README (wake block pending titan's MAC)

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

deploy/hyperborea/README.md

@@ -0,0 +1,82 @@
+# crossbar on hyperborea
+
+crossbar runs on **hyperborea** (Raspberry Pi, Debian 13, aarch64) as a `systemd --user` unit,
+bound to its tailnet address only. Clients on the tailnet reach it at
+
+    http://hyperborea.scylla-hammerhead.ts.net:7777/<route>/v1
+
+Why hyperborea: it is always on, wired on titan's LAN segment (`192.168.88.154`, which
+wake-on-LAN needs — magic packets are L2 broadcast), and not itself an inference host, so a
+router rebuild or a sleeping titan never takes crossbar down with it.
+
+## Files
+
+| file | purpose |
+|---|---|
+| `crossbar.toml` | the production config: hosts titan/straylight/dixie with their configured models and `parallel`, the routes |
+| `crossbar.service` | the user unit (`/srv/crossbar`, `Restart=always`) |
+| `install.sh` | cross-compiles for arm64 on the machine you run it from, copies binary + config + unit, restarts, prints the hosts view |
+
+On hyperborea: binary, config and SQLite database live in `/srv/crossbar/`; the unit is
+`~/.config/systemd/user/crossbar.service` (`loginctl` linger is on, so it survives logout).
+
+## Install / upgrade
+
+    deploy/hyperborea/install.sh          # from any checkout on a host with Go 1.26 and ssh to hyperborea
+
+Re-running upgrades in place (binary is replaced atomically, the unit restarted; leases persist in
+the database). Config-only changes: edit `crossbar.toml`, re-run.
+
+## Verify
+
+    curl -s http://hyperborea.scylla-hammerhead.ts.net:7777/_crossbar/hosts | jq .
+    curl -s http://hyperborea.scylla-hammerhead.ts.net:7777/_crossbar/routes | jq .
+    curl -s 'http://hyperborea.scylla-hammerhead.ts.net:7777/_crossbar/usage?by=route'
+    ssh hyperborea journalctl --user -u crossbar -f
+
+A cheap end-to-end check uses the `probe` route (dixie's 9B first):
+
+    curl -s -D - -X POST -H 'Content-Type: application/json' \
+      -d '{"model":"ornith-1.5-9b-uncensored","max_tokens":8,"messages":[{"role":"user","content":"Reply with pong."}]}' \
+      http://hyperborea.scylla-hammerhead.ts.net:7777/probe/v1/chat/completions
+
+The response carries `X-Crossbar-Host` (which router served it) and `X-Crossbar-Lease`
+(`new` or `reused`).
+
+## Pointing clients at it
+
+OpenCode (project-local `opencode.json`, or the global one with a per-project route):
+
+```jsonc
+"provider": { "crossbar": { "npm": "@ai-sdk/openai-compatible",
+  "options": { "baseURL": "http://hyperborea.scylla-hammerhead.ts.net:7777/opencode-a/v1" },
+  "models": { "ornith-1.5-35b-a3b": {} } } }
+```
+
+Hermes (`custom_providers[].base_url`, and the same in `delegation`/`auxiliary` blocks):
+
+    base_url: http://hyperborea.scylla-hammerhead.ts.net:7777/hermes-straylight/v1
+
+Routes must exist in `crossbar.toml`; an unknown first path segment is `404 unknown route`.
+**Known gap:** `PLAN.md`'s one-route-per-instance launcher (`CROSSBAR_ROUTE="$(basename "$PWD")-$$"`)
+needs a route *template* (e.g. `[routes."opencode-*"]`) that the code does not have yet; until
+then add each instance's route explicitly.
+
+## Wake-on-LAN for titan
+
+The `[hosts.titan.wake]` block is present but commented out until the MAC is settled. Titan is on
+Wi-Fi (active private address `5e:fc:f2:3f:23:6b`, hardware `60:3e:5f:33:6f:b8`) with its dock's
+three Ethernet ports (`d2:30:99:9a:ee:03/04/05`) unplugged. Wired + `womp 1` is the reliable path;
+magic-packet wake over Wi-Fi on Apple Silicon is not guaranteed and the private address may
+rotate. Broadcast address is `192.168.88.255:9`.
+
+## Security notes
+
+- The bind is the tailnet address; only tailnet members can reach it. `identity = "tailscale"`
+  with per-route `peers` is available when a route should be limited to named nodes;
+  `tailscale whois` already works unprivileged on hyperborea.
+- Plain HTTP over the tailnet is WireGuard-encrypted on the wire. Hermes agents' *terminal*
+  calls to this URL may trip tirith's `plain_http_to_sink`; prefer the MagicDNS name (never the
+  raw IP) and add a rule-scoped trust entry rather than `--broad` if a prompt recurs. Provider
+  traffic from the OpenAI client library is not scanned by tirith.
+- Bodies are never logged or stored; the database holds leases and per-request accounting only.

deploy/hyperborea/crossbar.service

@@ -0,0 +1,18 @@
+[Unit]
+Description=crossbar — affinity router for the fleet's llama-servers (tailnet :7777)
+After=network-online.target
+Wants=network-online.target
+RequiresMountsFor=/srv
+
+[Service]
+Type=simple
+WorkingDirectory=/srv/crossbar
+ExecStart=/srv/crossbar/crossbar -config /srv/crossbar/crossbar.toml
+# The bind is the tailnet address; if tailscaled is not up yet at login, retry until it is.
+Restart=always
+RestartSec=10
+StandardOutput=journal
+StandardError=journal
+
+[Install]
+WantedBy=default.target

deploy/hyperborea/crossbar.toml

@@ -0,0 +1,65 @@
+# crossbar on hyperborea — the fleet's llama-server routers behind one tailnet endpoint.
+# Clients: http://hyperborea.scylla-hammerhead.ts.net:7777/<route>/v1
+listen        = "100.112.40.10:7777"   # hyperborea's tailnet address only; never a LAN or 0.0.0.0 bind
+db            = "/srv/crossbar/crossbar.db"
+poll_interval = "60s"
+lease_idle    = "30m"
+retention     = "180d"
+queue_max     = 2                      # waiting places per (host, model) beyond `parallel`; 503 past that
+identity      = "off"                  # switch to "tailscale" once routes carry `peers`
+
+# `models` lists what each router is configured to serve, with that model's `parallel` from its
+# preset; the poller learns which are actually loaded (only those count for stickiness and the
+# context guard) and a request for an unloaded model still goes to a healthy host, where the
+# router autoloads it as today.
+
+[hosts.titan]                          # M3 Max 128 GB; ~2x straylight's decode speed
+base_url = "http://titan.scylla-hammerhead.ts.net:8081"
+weight   = 2.0
+models   = { "ornith-1.5-35b-a3b" = { parallel = 4 }, "ornith-1.5-9b-uncensored" = { parallel = 2 }, "qwen3.8-27b-uncensored" = { parallel = 2 }, "qwen3.6-35b-a3b-abliterated" = { parallel = 2 }, "gemma4-26b-a4b-abliterated" = { parallel = 2 }, "laguna-s-2.1" = { parallel = 2 }, "hermes4-70b-heretic" = { parallel = 1 }, "llama33-70b-abliterated" = { parallel = 1 }, "qwen25-72b-abliterated" = { parallel = 1 } }
+# Wake-on-LAN: enable once titan's wake MAC is decided (see README). Magic packets are L2
+# broadcast; hyperborea is wired on titan's segment (192.168.88.0/24).
+# [hosts.titan.wake]
+# mac       = "d2:30:99:9a:ee:03"      # dock Ethernet en4 if titan is wired; "5e:fc:f2:3f:23:6b" is the Wi-Fi private address
+# broadcast = "192.168.88.255:9"
+# wait      = "45s"
+
+[hosts.straylight]
+base_url = "http://straylight.scylla-hammerhead.ts.net:11434"
+weight   = 1.0
+models   = { "ornith-1.5-35b-a3b" = { parallel = 4 }, "ornith-1.5-9b-uncensored" = { parallel = 2 }, "qwen3.8-27b-uncensored" = { parallel = 2 }, "qwen3.6-35b-a3b-abliterated" = { parallel = 2 }, "gemma4-26b-a4b-abliterated" = { parallel = 2 }, "qwen3-vl-8b-abliterated" = { parallel = 2 }, "qwen3.8-flash-next-uncensored" = { parallel = 1 }, "ornith-1.0-35b" = { parallel = 2 } }
+
+[hosts.dixie]                          # helper tier: the 9B only (honcho-embed is Honcho's lane, not routed)
+base_url = "http://dixie.scylla-hammerhead.ts.net:11434"
+weight   = 0.5
+models   = { "ornith-1.5-9b-uncensored" = { parallel = 8 } }
+
+# Routes: the first URL path segment (or X-Crossbar-Route). Each conversation on a route gets a
+# sticky lease on the host with the most free slots x weight when it starts.
+[routes.opencode-a]
+hosts         = ["titan", "straylight"]
+default_model = "ornith-1.5-35b-a3b"
+
+[routes.opencode-b]
+hosts         = ["titan", "straylight"]
+default_model = "ornith-1.5-35b-a3b"
+
+[routes.paper]
+hosts         = ["titan", "straylight"]
+default_model = "ornith-1.5-35b-a3b"
+
+[routes.hermes-straylight]
+hosts         = ["straylight", "titan", "dixie"]
+default_model = "ornith-1.5-35b-a3b"
+
+[routes.hermes-titan]
+hosts         = ["titan", "straylight", "dixie"]
+default_model = "ornith-1.5-35b-a3b"
+
+[routes.hermes-talos]
+hosts         = ["titan", "straylight", "dixie"]
+default_model = "ornith-1.5-35b-a3b"
+
+[routes.probe]                         # for operators: curl tests, never a real client
+hosts         = ["dixie", "straylight", "titan"]
+default_model = "ornith-1.5-9b-uncensored"

deploy/hyperborea/install.sh

@@ -0,0 +1,20 @@
+#!/bin/sh
+# Build crossbar for hyperborea (arm64, static) on this machine and install it there as a
+# systemd --user unit. Run from anywhere inside the repo. Idempotent: re-running upgrades in place.
+set -eu
+HOST=${HOST:-hyperborea}
+DIR=/srv/crossbar
+cd "$(git rev-parse --show-toplevel)"
+out=$(mktemp -t crossbar-arm64.XXXXXX)
+trap 'rm -f "$out"' EXIT
+CGO_ENABLED=0 GOOS=linux GOARCH=arm64 go build -trimpath -ldflags="-s -w" -o "$out" ./cmd/crossbar
+ssh "$HOST" "mkdir -p $DIR ~/.config/systemd/user"
+scp -q "$out" "$HOST:$DIR/crossbar.new"
+scp -q deploy/hyperborea/crossbar.toml "$HOST:$DIR/crossbar.toml"
+scp -q deploy/hyperborea/crossbar.service "$HOST:.config/systemd/user/crossbar.service"
+ssh "$HOST" "chmod 755 $DIR/crossbar.new && mv $DIR/crossbar.new $DIR/crossbar \
+  && systemctl --user daemon-reload && systemctl --user enable crossbar.service >/dev/null 2>&1 \
+  && systemctl --user restart crossbar.service && sleep 2 && systemctl --user is-active crossbar.service"
+echo "installed; hosts view:"
+curl -fsS "http://$HOST.scylla-hammerhead.ts.net:7777/_crossbar/hosts"
+echo