Commit Graph
522 Commits
Author SHA1 Message Date
kyleandClaude Opus 5.5 c8b3b21004 dixie: honcho-embed pooling cls -> last
Qwen3-Embedding is a causal model; its card pools the last token. With cls every
input embedded as its first token's state ("the cat sat on the mat" and "the stock
market crashed today" at cosine 1.0). A CPU side server with --pooling last
reproduced the card's example scores within 0.002 without a manual EOS. Honcho's
stored vectors and sift's embed cache are re-embedded / flushed after the switch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 23:46:37 -07:00
kyle 03ad1de40e straylight: add trinity-mini (Arcee Trinity Mini 26B MoE) to llama-server router 2026-09-27 00:56:41 -07:00
kyle cb0bc87633 orion: pin sift 5a59f7a (P6 asynchronous batch fetch) 2026-09-26 23:32:13 -07:00
kyle 4fa779be9c orion: pin sift bdc093e (P5b long-doc questions + P5c extract; P6 plan docs) 2026-09-26 20:22:32 -07:00
kyle cd2bd3d0e3 orion: pin sift 1311cf3 (P5a PDF documents)
PDF extraction in a re-exec'd child, per-page cache, map/pages/
grep, cited digests. Module MemoryMax 512M -> 1G via pinned sift.
2026-09-26 10:57:22 -07:00
kyle ec39a569de dixie: Z tmpfiles rules use ~0755 so existing files stay 0644 (exec dropped), dirs become 0755 2026-09-25 07:49:51 -07:00
kyle e5c47007cd dixie: Z tmpfiles rules to relabel dropsonde2 trees 0755 recursively (fixes rsync's 0700 dirs) 2026-09-25 07:48:32 -07:00
kyle 9a5920da2b dixie: drop uid/gid from rsyncd modules — daemon runs as dropsonde via SSH, setgroups() fails otherwise 2026-09-25 03:03:40 -07:00
kyle 2305f4099a dixie: rsync --server --daemon sync key (no rrsync/python3 on dixie)
- authorized key forces rsync --server --daemon --config=/etc/rsyncd-dropsonde.conf
- environment.etc rsyncd-dropsonde.conf: cold + analysis modules, read only=false,
  use chroot=false, uid/gid dropsonde, max connections 2, lock file global
- tmpfiles: /srv/{cold,analysis}/dropsonde2 owned dropsonde
2026-09-25 02:45:44 -07:00
kyle 2e4aa6ed57 dixie: dropsonde sync key — rrsync -wo /srv (write-only, hyperborea -> dixie) 2026-09-25 02:24:26 -07:00
kyle 0ac90515bf dixie: use script attr for dropsonde-pg-setup (ExecStart heredoc+bash path broken) 2026-09-25 00:51:32 -07:00
kyle c402c1c599 dixie: drop python3.withPackages (duckdb CLI only) to trim the closure 2026-09-25 00:27:38 -07:00
kyle eed2821cea dixie: slice submodule uses sliceConfig, not serviceConfig 2026-09-24 23:42:12 -07:00
kyle 43b8e2288f dixie: listen_addresses under settings, not top-level 2026-09-24 23:41:40 -07:00
kyle 3980242da5 dixie: close the ExecStart indented string in dropsonde-pg-setup 2026-09-24 23:41:25 -07:00
kyle 4b751796c6 dixie: import analysis.nix (was never wired into the module list) 2026-09-24 23:36:31 -07:00
kyle 0c37b07099 dixie: label /srv/cold as ds2-cold, use sfdisk for wipe 2026-09-24 23:20:01 -07:00
kyle fddaffc825 dixie: fix slice unit name, pg-setup ExecStart, listen_addresses 2026-09-24 22:27:12 -07:00
kyle a40873685a dixie: open Postgres 5432 on tailscale0 for dropsonde 2026-09-24 22:17:44 -07:00
kyle 3c2a272722 dixie: dropsonde2 analysis tier (HDD /srv/cold, pg17+timescale+postgis+h3, analysis.slice, duckdb/polars) 2026-09-24 22:03:44 -07:00
kyleandClaude Opus 5.5 c12b2adbfd disk-health plan: deploy via committed master + doas (per CLAUDE.md)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-24 18:59:39 -07:00
kyleandClaude Opus 5.5 85d5631fa8 docs: disk-health monitoring plan (SMART/NVMe/SD card)
Current state verified on 2026-09-24: hyperborea NVMe already exported
via the Debian nvme textfile; no SMART on the NixOS hosts; no
disk-health alert rules. Plan: smartctl_exporter module for the NixOS
hosts, smartmon timer and SD-card kernel-error textfile on hyperborea,
orion scrape job, a disk-health rule group, verification and deploy
notes. For hand-off to another agent.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-24 18:59:15 -07:00
kyle 78d30026ce orion: allowlist vmui's /prometheus API prefix
Default Server URL is /prometheus; keep import/admin 403.
2026-09-24 01:56:05 -07:00
kyle f5ef78baca orion: read-only vmui via nginx allowlist + serve :8429
VictoriaMetrics stays loopback :8428. nginx :8429 allows GET/POST
to /vmui and query APIs only; import/admin/export/flags 403.
tailscale serve HTTPS 8429. Review, no switch.
2026-09-24 01:47:25 -07:00
kyle 1cbde84601 rift: linger kyle and enable user podman-restart
Keep rootless podman up after logind idle.
2026-09-24 01:33:19 -07:00
kyle f32db3c88b metrics: systemd unit-include as [.]service
systemd drops \. in unit-include and warns about unknown escapes.
2026-09-24 01:22:16 -07:00
kyle 81897b3f8a metrics: keep llama-straylight out of FleetTargetDown
35B eviction is not a paging event. params.model stays.
2026-09-24 00:46:23 -07:00
kyle 385529da39 metrics: phase 3 straylight/rift scrapes and hyperborea units
Review, no switch. node-laptop for talos/titan is excluded from
FleetTargetDown. llama-straylight scrape uses params.model for
ornith-1.5-35b-a3b (router 400 without it; will load if cold).
2026-09-24 00:42:06 -07:00
kyle b861c6ee9f metrics: drop llama-dixie-vision scrape
Router /metrics cannot be up without loading a model.
2026-09-24 00:29:24 -07:00
kyle ed5e1a1e0e metrics: HonchoDeriverBacklog and include honcho in target-down 2026-09-24 00:21:36 -07:00
kyle 5d8e8c4335 metrics: scrape dixie helper /metrics with model param
Router-mode llama-server returns 400 without it.
2026-09-23 23:57:36 -07:00
kyle b1d5aa7e64 metrics: drop alertmanager channel override
Webhook is already bound to ~alerts; Mattermost's channel
override wants the name, not ~alerts.
2026-09-23 23:44:24 -07:00
kyle ec74e0b69c metrics: phase 1 follow-up (review, no switch)
Honcho workspace_name keep, node_exporter restart metrics,
drop min_over_time on rule 1, Watchdog heartbeat, pin tether
to b267e2f so labelled series exist and rule 2 can fire.
2026-09-23 23:43:11 -07:00
kyle 4958493485 metrics: phase 1 orion+dixie (review, no switch)
VictoriaMetrics on 127.0.0.1:8428. LoadCredential for the Mattermost
webhook and the sift scrape token. Honcho scrapes keep a closed metric
name set. Meta-alerts for target-down, VM size, orion disk. Dixie gets
node_exporter, nvidia_gpu_exporter, and llama-server --metrics.

Tether labelled series still wait on titan's fix. Rule 2 is written
against reason=client_disconnect so it stays silent until then.
2026-09-23 23:22:38 -07:00
kyle 888e8e24fb orion: pin sift 4372673 (P2.7 metrics)
Prometheus /metrics behind a dedicated metrics caller; JSON stays
at ?format=json. No search-source change.
2026-09-23 23:11:30 -07:00
kyle c07c4bfe4f docs: fix tether cancel-storm PromQL (rule 2)
Label-set mismatch made the ratio empty. Select reason=client_disconnect
and kind=converted; never sum() a tether series without a selector.
2026-09-23 22:50:26 -07:00
kyle 8b14e1f99f docs: fleet metrics and alerting plan
Reviewed plan only: VictoriaMetrics on orion, vmalert/Alertmanager,
tailnet exporters, first five rules. No config, no deploy.
2026-09-23 22:48:51 -07:00
kyle 504d4eb3f1 orion: pin sift e62cfc8 (restricted tavily cap skip) 2026-09-23 16:38:41 -07:00
kyle bc30315344 orion: sift tavily+searxng, cap 130, tavily no_rerank
Kyle chose Tavily-first with SearXNG fallback. Brave stays off.
2026-09-23 16:25:01 -07:00
kyle 75b813b10b orion: SearXNG-only until Kyle picks sift search default 2026-09-23 16:20:38 -07:00
kyle 415589cc13 orion: pin sift 1c57845, brave_llm only for live fixture
RRF original-URL fix. SearXNG and Tavily stay off until Kyle picks a default.
2026-09-23 16:10:59 -07:00
kyle 573b9dcfd8 orion: disable sift brave_llm for bake-off config 1 2026-09-23 15:42:54 -07:00
kyle c84ca93024 orion: disable sift tavily for bake-off config 2 2026-09-23 15:36:33 -07:00
kyle 91c021cff9 orion: enable sift searxng, brave_llm, tavily
Bake-off sources. SearXNG journal is already LogLevelMax=err.
Keys stay in /var/lib/sift/secret.env, not the store.
2026-09-23 14:14:25 -07:00
kyle c839f7c8ae dixie: fail over USB Wi-Fi to internal CNVi
NM dispatcher plus a boot oneshot keep the AX9L (mt7921u) primary and
bring up the on-box iwlwifi profile if the dongle is gone. Connection
PSKs stay on-box; this only selects the radio.
2026-09-23 14:03:30 -07:00
kyle 687fac019f straylight: sift CLI 8ca3ade on PATH at next switch
Pin follows approved CLI fail-closed for non-default HERMES_HOME.
Does not switch straylight or orion. ~/.local/bin/sift (e36da19)
still shadows until that binary is removed after the switch.
2026-09-23 13:11:17 -07:00
kyle ed2ef77de6 orion: put sift CLI on PATH
The unit package is not in systemPackages, so `sift keys` is only
reachable via the store path. Track services.sift.package so it
follows the pin. Lands on the next orion switch.
2026-09-23 12:33:26 -07:00
kyle fccb26dea4 orion: pin sift 429f6b2 (P3) and drop SearXNG WARNING logs
Sift P3 ships with sources off. SearXNG engine WARNING lines include
q=; LogLevelMax=err keeps those out of the journal. Brave scraper
stays disabled. Do not enable search sources in this commit.
2026-09-23 12:25:04 -07:00
kyle 059c2306da orion: pin sift 3687636 (P2.5+P2.6)
P2.5 redacts restricted logs; P2.6 partitions dcache and pins
restricted callers to guardian-star.
2026-09-23 11:59:32 -07:00
kyle 988053ac94 orion: pin sift e36da19 (restricted workspace)
Permit straylight-gs to query guardian-star; fleet callers still refused.
2026-09-23 08:35:01 -07:00