Commit Graph
498 Commits
Author SHA1 Message Date
kyle 1cbde84601 rift: linger kyle and enable user podman-restart
Keep rootless podman up after logind idle.
2026-09-24 01:33:19 -07:00
kyle f32db3c88b metrics: systemd unit-include as [.]service
systemd drops \. in unit-include and warns about unknown escapes.
2026-09-24 01:22:16 -07:00
kyle 81897b3f8a metrics: keep llama-straylight out of FleetTargetDown
35B eviction is not a paging event. params.model stays.
2026-09-24 00:46:23 -07:00
kyle 385529da39 metrics: phase 3 straylight/rift scrapes and hyperborea units
Review, no switch. node-laptop for talos/titan is excluded from
FleetTargetDown. llama-straylight scrape uses params.model for
ornith-1.5-35b-a3b (router 400 without it; will load if cold).
2026-09-24 00:42:06 -07:00
kyle b861c6ee9f metrics: drop llama-dixie-vision scrape
Router /metrics cannot be up without loading a model.
2026-09-24 00:29:24 -07:00
kyle ed5e1a1e0e metrics: HonchoDeriverBacklog and include honcho in target-down 2026-09-24 00:21:36 -07:00
kyle 5d8e8c4335 metrics: scrape dixie helper /metrics with model param
Router-mode llama-server returns 400 without it.
2026-09-23 23:57:36 -07:00
kyle b1d5aa7e64 metrics: drop alertmanager channel override
Webhook is already bound to ~alerts; Mattermost's channel
override wants the name, not ~alerts.
2026-09-23 23:44:24 -07:00
kyle ec74e0b69c metrics: phase 1 follow-up (review, no switch)
Honcho workspace_name keep, node_exporter restart metrics,
drop min_over_time on rule 1, Watchdog heartbeat, pin tether
to b267e2f so labelled series exist and rule 2 can fire.
2026-09-23 23:43:11 -07:00
kyle 4958493485 metrics: phase 1 orion+dixie (review, no switch)
VictoriaMetrics on 127.0.0.1:8428. LoadCredential for the Mattermost
webhook and the sift scrape token. Honcho scrapes keep a closed metric
name set. Meta-alerts for target-down, VM size, orion disk. Dixie gets
node_exporter, nvidia_gpu_exporter, and llama-server --metrics.

Tether labelled series still wait on titan's fix. Rule 2 is written
against reason=client_disconnect so it stays silent until then.
2026-09-23 23:22:38 -07:00
kyle 888e8e24fb orion: pin sift 4372673 (P2.7 metrics)
Prometheus /metrics behind a dedicated metrics caller; JSON stays
at ?format=json. No search-source change.
2026-09-23 23:11:30 -07:00
kyle c07c4bfe4f docs: fix tether cancel-storm PromQL (rule 2)
Label-set mismatch made the ratio empty. Select reason=client_disconnect
and kind=converted; never sum() a tether series without a selector.
2026-09-23 22:50:26 -07:00
kyle 8b14e1f99f docs: fleet metrics and alerting plan
Reviewed plan only: VictoriaMetrics on orion, vmalert/Alertmanager,
tailnet exporters, first five rules. No config, no deploy.
2026-09-23 22:48:51 -07:00
kyle 504d4eb3f1 orion: pin sift e62cfc8 (restricted tavily cap skip) 2026-09-23 16:38:41 -07:00
kyle bc30315344 orion: sift tavily+searxng, cap 130, tavily no_rerank
Kyle chose Tavily-first with SearXNG fallback. Brave stays off.
2026-09-23 16:25:01 -07:00
kyle 75b813b10b orion: SearXNG-only until Kyle picks sift search default 2026-09-23 16:20:38 -07:00
kyle 415589cc13 orion: pin sift 1c57845, brave_llm only for live fixture
RRF original-URL fix. SearXNG and Tavily stay off until Kyle picks a default.
2026-09-23 16:10:59 -07:00
kyle 573b9dcfd8 orion: disable sift brave_llm for bake-off config 1 2026-09-23 15:42:54 -07:00
kyle c84ca93024 orion: disable sift tavily for bake-off config 2 2026-09-23 15:36:33 -07:00
kyle 91c021cff9 orion: enable sift searxng, brave_llm, tavily
Bake-off sources. SearXNG journal is already LogLevelMax=err.
Keys stay in /var/lib/sift/secret.env, not the store.
2026-09-23 14:14:25 -07:00
kyle c839f7c8ae dixie: fail over USB Wi-Fi to internal CNVi
NM dispatcher plus a boot oneshot keep the AX9L (mt7921u) primary and
bring up the on-box iwlwifi profile if the dongle is gone. Connection
PSKs stay on-box; this only selects the radio.
2026-09-23 14:03:30 -07:00
kyle 687fac019f straylight: sift CLI 8ca3ade on PATH at next switch
Pin follows approved CLI fail-closed for non-default HERMES_HOME.
Does not switch straylight or orion. ~/.local/bin/sift (e36da19)
still shadows until that binary is removed after the switch.
2026-09-23 13:11:17 -07:00
kyle ed2ef77de6 orion: put sift CLI on PATH
The unit package is not in systemPackages, so `sift keys` is only
reachable via the store path. Track services.sift.package so it
follows the pin. Lands on the next orion switch.
2026-09-23 12:33:26 -07:00
kyle fccb26dea4 orion: pin sift 429f6b2 (P3) and drop SearXNG WARNING logs
Sift P3 ships with sources off. SearXNG engine WARNING lines include
q=; LogLevelMax=err keeps those out of the journal. Brave scraper
stays disabled. Do not enable search sources in this commit.
2026-09-23 12:25:04 -07:00
kyle 059c2306da orion: pin sift 3687636 (P2.5+P2.6)
P2.5 redacts restricted logs; P2.6 partitions dcache and pins
restricted callers to guardian-star.
2026-09-23 11:59:32 -07:00
kyle 988053ac94 orion: pin sift e36da19 (restricted workspace)
Permit straylight-gs to query guardian-star; fleet callers still refused.
2026-09-23 08:35:01 -07:00
kyle e6f7bcb566 straylight: bump tracker to 549eda3
Master adds -changepw. The service unit still starts with -addr and -db.
2026-09-23 07:51:16 -07:00
kyleandClaude Opus 5.5 46a7241dba dixie: 9B batch-size 512 so decodes interleave with long prefills
Prefill ran in 2048-token scheduler steps of ~2.4 s; each decoding slot
gets one token per step, so guardian calls (16 tokens) waited behind
Honcho's 10-28k-token prompts: p90 10 s, max 14 s under two sift
digests (2026-09-23 05:50). ubatch is already 512, so prefill
throughput is unchanged; steps drop to ~0.6 s.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-23 05:54:00 -07:00
kyleandClaude Opus 5.5 8790005c53 dixie: 9B KV pool 96k -> 128k
The unified 96k pool overflowed ~every 2-3 min under Honcho load
(39 events 2026-09-23 00:50-04:28). Each overflow fails every in-flight
slot with 'Context size has been exceeded', so Honcho retries and the
guardians' calls die with it. KV is ~20 KB/token on this model; 128k
costs ~0.65 GB of the ~2.5 GB free.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-23 04:47:24 -07:00
kyle afa33ee8ee orion: pin sift 767c12c (P2.3)
Pin chunk 0; 0.7 cosine + 0.3 term overlap; PromptVersion 2.
2026-09-23 03:24:04 -07:00
kyle e657737cd7 orion: pin sift 5dbc7e4 (P2.2)
Cap embed inputs at 3000 runes; retry too-large once.
2026-09-23 03:09:02 -07:00
kyle c42191f2a6 orion: pin sift 16a7d67 (P2.1)
embed_model honcho-embed; Wayback http rewrite; Memento parse;
keys.toml writePreserving; digest fallback cap.
2026-09-23 02:48:35 -07:00
kyle c831385cbd dixie: tether on :11435 in front of llama-server
Pin kyle/tether at c89894b. Listen 0.0.0.0:11435, upstream
127.0.0.1:11434. Open 11435 on tailscale0 next to 11433/11434.
2026-09-23 02:44:22 -07:00
kyle 109bc99d3b orion: pin sift 2db001e (WriteTimeout 120s, CLI flags)
Handler deadline 90s. Flags work on both sides of the URL.
No Dixie config change.
2026-09-23 00:37:29 -07:00
kyle 03e256c501 orion: pin sift a992216 (P2 fence follow-up)
Widen document-tag fence; same P2 host config as ff87a04.
2026-09-23 00:14:17 -07:00
kyle 9edaf0f339 orion: pin sift ff87a04 (P2 digest/context/recover)
Dixie chat+embed on :11434, digest thinking off, Honcho
context from rift for hermes-fleet only.
2026-09-23 00:05:30 -07:00
kyle 85b52525b0 orion: pin sift 0598c89 (P1.1 self-IP deny)
Same code as 447842f plus the IPv4-only assumption comment.
Live P1 was dca635f.
2026-09-22 23:05:31 -07:00
kyle 3cc51c9083 orion: pin sift dca635f
fac1462 set a vendorHash while vendor/ is committed, so
sift-*-go-modules failed. dca635f sets vendorHash = null.
2026-09-22 22:42:51 -07:00
kyle 9b745272e6 orion: enable sift P1 (pin fac1462)
Add the sift flake input at fac1462e7b3deb2082c9f6ae8dc4676fddf8d089
(git+ssh; git+https cannot auth on orion) and enable services.sift on
orion: loopback :8890, tailscale serve :8443 from the module, static
home WAN /32 in denyPrefixes. Keys stay on-box.
2026-09-22 22:20:11 -07:00
kyleandClaude Opus 5.5 ab3fbb094f straylight: llama.cpp 0.4.1 preset: no-mmap -> load-mode = none
0.4.1 dropped the no-mmap option (router refused to start: "option 'no-mmap'
not recognized in preset '*'"); --load-mode none is the equivalent. All other
preset keys checked against 0.4.1's option list.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-22 20:13:51 -07:00
kyle dcc8da37a4 strawberry 2026-09-22 20:08:16 -07:00
kyleandClaude Opus 5.5 7a89ca25c4 remove the WNTRMUTE CA (MCP teardown)
Kyle 2026-09-22: the shared CA was only for Metacircular services (system
trust store + the MCR podman registry cert). No host should still depend on
it; anything that does should fail loudly rather than keep trusting it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-22 19:57:14 -07:00
kyle 90a572eabb rift: drop MCP config (containers already stopped)
Remove mcp.nix/mcpkg.nix, MCP firewall ports, MCNS nameserver, and
~mcp.metacircular.net. Leave LUKS FIDO2. Do not delete /srv/mcp.
2026-09-22 19:42:43 -07:00
kyleandClaude Opus 5.5 b387d86f28 straylight: remove MCP (fleet teardown for V3)
Kyle 2026-09-22: tear down MCP across the fleet. Drops the mcp/mcpkg imports
(agent, mcp user, packages), the unikernel-host agent overrides and kvm group,
the mcp-br0 bridge + FORWARD drop, the low-port sysctl (only MCNS/mc-proxy
needed it; Mattermost binds 8065), MCP firewall ports (53, 8443, 9443, 9444,
9555; keeps 443 for tailscale serve and 8080 for open-webui) and the MCNS
nameservers/~mcp.metacircular.net domain. State under /srv/mcp is kept.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-22 19:40:56 -07:00
kyle 39c9fc1a39 orion: set braveapi inactive = false
Package default is inactive: true, which skips loading the engine.
disabled = false is not enough; merge by name leaves inactive set.
2026-09-22 19:26:31 -07:00
kyle 8b14304037 orion: enable SearXNG braveapi via envsubst
Scrape engine `brave` stays disabled. API key lives in
/var/lib/searx/secret.env as BRAVE_API_KEY, not in the store.
2026-09-22 19:25:34 -07:00
kyleandClaude Opus 5.5 b743d3f360 dixie: 9B to 8 slots over a 96k pool
A Honcho catch-up wave (48 deriver batches + 11 dreams + dialectic prefetches)
held all 6 slots with long thinking generations; tiny aux calls (title,
approval guardian) queued behind them and title_generation timed out at 60 s.
Two spare slots plus a bigger shared KV pool (so 6 long prompts can't exhaust
it). VRAM 9.7 GB.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-22 19:14:45 -07:00
kyle f28c1ae5f0 orion: SearXNG engine allowlist for the home IP
Enable bing; keep wikipedia/wikidata/github/arxiv. Disable brave,
duckduckgo, startpage, google, qwant, mojeek (CAPTCHA, 429, or silent 0).
2026-09-22 19:10:30 -07:00
kyle 94e85283c3 orion: generate SearXNG secret in activationScripts
searx-init loads EnvironmentFile at spawn, so the secret must exist
before systemd starts that unit. Drop the racing oneshot. Make
tailscale-serve restart on NoState during switch.
2026-09-22 18:49:42 -07:00
kyle e6938ef782 orion: drop MCP, add SearXNG on loopback + Tailscale Serve
SearXNG binds 127.0.0.1:8888 (json+html, limiter off). Secret is
/var/lib/searx/secret.env, generated on-box. TLS via `tailscale serve`.
MCP agent, packages, extra firewall ports, and MCP DNS removed from this
host only.
2026-09-22 18:01:23 -07:00