Commit Graph
147 Commits
Author SHA1 Message Date
kyle 2e4aa6ed57 dixie: dropsonde sync key — rrsync -wo /srv (write-only, hyperborea -> dixie) 2026-09-25 02:24:26 -07:00
kyle 0ac90515bf dixie: use script attr for dropsonde-pg-setup (ExecStart heredoc+bash path broken) 2026-09-25 00:51:32 -07:00
kyle c402c1c599 dixie: drop python3.withPackages (duckdb CLI only) to trim the closure 2026-09-25 00:27:38 -07:00
kyle eed2821cea dixie: slice submodule uses sliceConfig, not serviceConfig 2026-09-24 23:42:12 -07:00
kyle 43b8e2288f dixie: listen_addresses under settings, not top-level 2026-09-24 23:41:40 -07:00
kyle 3980242da5 dixie: close the ExecStart indented string in dropsonde-pg-setup 2026-09-24 23:41:25 -07:00
kyle 4b751796c6 dixie: import analysis.nix (was never wired into the module list) 2026-09-24 23:36:31 -07:00
kyle 0c37b07099 dixie: label /srv/cold as ds2-cold, use sfdisk for wipe 2026-09-24 23:20:01 -07:00
kyle fddaffc825 dixie: fix slice unit name, pg-setup ExecStart, listen_addresses 2026-09-24 22:27:12 -07:00
kyle a40873685a dixie: open Postgres 5432 on tailscale0 for dropsonde 2026-09-24 22:17:44 -07:00
kyle 3c2a272722 dixie: dropsonde2 analysis tier (HDD /srv/cold, pg17+timescale+postgis+h3, analysis.slice, duckdb/polars) 2026-09-24 22:03:44 -07:00
kyle 78d30026ce orion: allowlist vmui's /prometheus API prefix
Default Server URL is /prometheus; keep import/admin 403.
2026-09-24 01:56:05 -07:00
kyle f5ef78baca orion: read-only vmui via nginx allowlist + serve :8429
VictoriaMetrics stays loopback :8428. nginx :8429 allows GET/POST
to /vmui and query APIs only; import/admin/export/flags 403.
tailscale serve HTTPS 8429. Review, no switch.
2026-09-24 01:47:25 -07:00
kyle 1cbde84601 rift: linger kyle and enable user podman-restart
Keep rootless podman up after logind idle.
2026-09-24 01:33:19 -07:00
kyle 81897b3f8a metrics: keep llama-straylight out of FleetTargetDown
35B eviction is not a paging event. params.model stays.
2026-09-24 00:46:23 -07:00
kyle 385529da39 metrics: phase 3 straylight/rift scrapes and hyperborea units
Review, no switch. node-laptop for talos/titan is excluded from
FleetTargetDown. llama-straylight scrape uses params.model for
ornith-1.5-35b-a3b (router 400 without it; will load if cold).
2026-09-24 00:42:06 -07:00
kyle b861c6ee9f metrics: drop llama-dixie-vision scrape
Router /metrics cannot be up without loading a model.
2026-09-24 00:29:24 -07:00
kyle ed5e1a1e0e metrics: HonchoDeriverBacklog and include honcho in target-down 2026-09-24 00:21:36 -07:00
kyle 5d8e8c4335 metrics: scrape dixie helper /metrics with model param
Router-mode llama-server returns 400 without it.
2026-09-23 23:57:36 -07:00
kyle b1d5aa7e64 metrics: drop alertmanager channel override
Webhook is already bound to ~alerts; Mattermost's channel
override wants the name, not ~alerts.
2026-09-23 23:44:24 -07:00
kyle ec74e0b69c metrics: phase 1 follow-up (review, no switch)
Honcho workspace_name keep, node_exporter restart metrics,
drop min_over_time on rule 1, Watchdog heartbeat, pin tether
to b267e2f so labelled series exist and rule 2 can fire.
2026-09-23 23:43:11 -07:00
kyle 4958493485 metrics: phase 1 orion+dixie (review, no switch)
VictoriaMetrics on 127.0.0.1:8428. LoadCredential for the Mattermost
webhook and the sift scrape token. Honcho scrapes keep a closed metric
name set. Meta-alerts for target-down, VM size, orion disk. Dixie gets
node_exporter, nvidia_gpu_exporter, and llama-server --metrics.

Tether labelled series still wait on titan's fix. Rule 2 is written
against reason=client_disconnect so it stays silent until then.
2026-09-23 23:22:38 -07:00
kyle bc30315344 orion: sift tavily+searxng, cap 130, tavily no_rerank
Kyle chose Tavily-first with SearXNG fallback. Brave stays off.
2026-09-23 16:25:01 -07:00
kyle 75b813b10b orion: SearXNG-only until Kyle picks sift search default 2026-09-23 16:20:38 -07:00
kyle 415589cc13 orion: pin sift 1c57845, brave_llm only for live fixture
RRF original-URL fix. SearXNG and Tavily stay off until Kyle picks a default.
2026-09-23 16:10:59 -07:00
kyle 573b9dcfd8 orion: disable sift brave_llm for bake-off config 1 2026-09-23 15:42:54 -07:00
kyle c84ca93024 orion: disable sift tavily for bake-off config 2 2026-09-23 15:36:33 -07:00
kyle 91c021cff9 orion: enable sift searxng, brave_llm, tavily
Bake-off sources. SearXNG journal is already LogLevelMax=err.
Keys stay in /var/lib/sift/secret.env, not the store.
2026-09-23 14:14:25 -07:00
kyle c839f7c8ae dixie: fail over USB Wi-Fi to internal CNVi
NM dispatcher plus a boot oneshot keep the AX9L (mt7921u) primary and
bring up the on-box iwlwifi profile if the dongle is gone. Connection
PSKs stay on-box; this only selects the radio.
2026-09-23 14:03:30 -07:00
kyle 687fac019f straylight: sift CLI 8ca3ade on PATH at next switch
Pin follows approved CLI fail-closed for non-default HERMES_HOME.
Does not switch straylight or orion. ~/.local/bin/sift (e36da19)
still shadows until that binary is removed after the switch.
2026-09-23 13:11:17 -07:00
kyle ed2ef77de6 orion: put sift CLI on PATH
The unit package is not in systemPackages, so `sift keys` is only
reachable via the store path. Track services.sift.package so it
follows the pin. Lands on the next orion switch.
2026-09-23 12:33:26 -07:00
kyle fccb26dea4 orion: pin sift 429f6b2 (P3) and drop SearXNG WARNING logs
Sift P3 ships with sources off. SearXNG engine WARNING lines include
q=; LogLevelMax=err keeps those out of the journal. Brave scraper
stays disabled. Do not enable search sources in this commit.
2026-09-23 12:25:04 -07:00
kyle 988053ac94 orion: pin sift e36da19 (restricted workspace)
Permit straylight-gs to query guardian-star; fleet callers still refused.
2026-09-23 08:35:01 -07:00
kyleandClaude Opus 5.5 46a7241dba dixie: 9B batch-size 512 so decodes interleave with long prefills
Prefill ran in 2048-token scheduler steps of ~2.4 s; each decoding slot
gets one token per step, so guardian calls (16 tokens) waited behind
Honcho's 10-28k-token prompts: p90 10 s, max 14 s under two sift
digests (2026-09-23 05:50). ubatch is already 512, so prefill
throughput is unchanged; steps drop to ~0.6 s.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-23 05:54:00 -07:00
kyleandClaude Opus 5.5 8790005c53 dixie: 9B KV pool 96k -> 128k
The unified 96k pool overflowed ~every 2-3 min under Honcho load
(39 events 2026-09-23 00:50-04:28). Each overflow fails every in-flight
slot with 'Context size has been exceeded', so Honcho retries and the
guardians' calls die with it. KV is ~20 KB/token on this model; 128k
costs ~0.65 GB of the ~2.5 GB free.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-23 04:47:24 -07:00
kyle c42191f2a6 orion: pin sift 16a7d67 (P2.1)
embed_model honcho-embed; Wayback http rewrite; Memento parse;
keys.toml writePreserving; digest fallback cap.
2026-09-23 02:48:35 -07:00
kyle c831385cbd dixie: tether on :11435 in front of llama-server
Pin kyle/tether at c89894b. Listen 0.0.0.0:11435, upstream
127.0.0.1:11434. Open 11435 on tailscale0 next to 11433/11434.
2026-09-23 02:44:22 -07:00
kyle 9edaf0f339 orion: pin sift ff87a04 (P2 digest/context/recover)
Dixie chat+embed on :11434, digest thinking off, Honcho
context from rift for hermes-fleet only.
2026-09-23 00:05:30 -07:00
kyle 9b745272e6 orion: enable sift P1 (pin fac1462)
Add the sift flake input at fac1462e7b3deb2082c9f6ae8dc4676fddf8d089
(git+ssh; git+https cannot auth on orion) and enable services.sift on
orion: loopback :8890, tailscale serve :8443 from the module, static
home WAN /32 in denyPrefixes. Keys stay on-box.
2026-09-22 22:20:11 -07:00
kyleandClaude Opus 5.5 ab3fbb094f straylight: llama.cpp 0.4.1 preset: no-mmap -> load-mode = none
0.4.1 dropped the no-mmap option (router refused to start: "option 'no-mmap'
not recognized in preset '*'"); --load-mode none is the equivalent. All other
preset keys checked against 0.4.1's option list.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-22 20:13:51 -07:00
kyle 90a572eabb rift: drop MCP config (containers already stopped)
Remove mcp.nix/mcpkg.nix, MCP firewall ports, MCNS nameserver, and
~mcp.metacircular.net. Leave LUKS FIDO2. Do not delete /srv/mcp.
2026-09-22 19:42:43 -07:00
kyleandClaude Opus 5.5 b387d86f28 straylight: remove MCP (fleet teardown for V3)
Kyle 2026-09-22: tear down MCP across the fleet. Drops the mcp/mcpkg imports
(agent, mcp user, packages), the unikernel-host agent overrides and kvm group,
the mcp-br0 bridge + FORWARD drop, the low-port sysctl (only MCNS/mc-proxy
needed it; Mattermost binds 8065), MCP firewall ports (53, 8443, 9443, 9444,
9555; keeps 443 for tailscale serve and 8080 for open-webui) and the MCNS
nameservers/~mcp.metacircular.net domain. State under /srv/mcp is kept.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-22 19:40:56 -07:00
kyle 39c9fc1a39 orion: set braveapi inactive = false
Package default is inactive: true, which skips loading the engine.
disabled = false is not enough; merge by name leaves inactive set.
2026-09-22 19:26:31 -07:00
kyle 8b14304037 orion: enable SearXNG braveapi via envsubst
Scrape engine `brave` stays disabled. API key lives in
/var/lib/searx/secret.env as BRAVE_API_KEY, not in the store.
2026-09-22 19:25:34 -07:00
kyleandClaude Opus 5.5 b743d3f360 dixie: 9B to 8 slots over a 96k pool
A Honcho catch-up wave (48 deriver batches + 11 dreams + dialectic prefetches)
held all 6 slots with long thinking generations; tiny aux calls (title,
approval guardian) queued behind them and title_generation timed out at 60 s.
Two spare slots plus a bigger shared KV pool (so 6 long prompts can't exhaust
it). VRAM 9.7 GB.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-22 19:14:45 -07:00
kyle f28c1ae5f0 orion: SearXNG engine allowlist for the home IP
Enable bing; keep wikipedia/wikidata/github/arxiv. Disable brave,
duckduckgo, startpage, google, qwant, mojeek (CAPTCHA, 429, or silent 0).
2026-09-22 19:10:30 -07:00
kyle 94e85283c3 orion: generate SearXNG secret in activationScripts
searx-init loads EnvironmentFile at spawn, so the secret must exist
before systemd starts that unit. Drop the racing oneshot. Make
tailscale-serve restart on NoState during switch.
2026-09-22 18:49:42 -07:00
kyle e6938ef782 orion: drop MCP, add SearXNG on loopback + Tailscale Serve
SearXNG binds 127.0.0.1:8888 (json+html, limiter off). Secret is
/var/lib/searx/secret.env, generated on-box. TLS via `tailscale serve`.
MCP agent, packages, extra firewall ports, and MCP DNS removed from this
host only.
2026-09-22 18:01:23 -07:00
kyleandClaude Opus 5.5 8a6d0039d3 dixie: 9B to 6 slots over a 64k pool
Room for the deriver, dialectic prefetches from three agents and their approval
guardians at once. Under full load (deriver + dialectic + guardian probes):
guardian p50 0.4 s / max 10 s, no KV-space failures. VRAM 8.9 GB.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-22 15:37:46 -07:00
kyleandClaude Opus 5.5 6d5586c63c dixie: retire the 4B dialectic model; 9B gets 4 slots over a 48k pool
Dialectic benchmark (8 hermes-fleet questions): Qwen3-4B answered in 2-12 s but
often wrongly (invented facts, 'no information' on things memory holds); the 9B
answered correctly. All Honcho dialectic levels now use the 9B (rift .env). Its
VRAM goes to the 9B's KV pool: at 2 x 16k, concurrent deriver + dialectic hit
'failed to find free space in the KV cache'. 9B 6.3 GB + embed 2.2 GB = 8.4 GB.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-22 14:29:01 -07:00