kyle/imladris
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
520 Commits 2 Branches 0 Tags
cb0bc876339209a9501958e8894a0f62b78e2443
Commit Graph
520 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
kyle cb0bc87633 orion: pin sift 5a59f7a (P6 asynchronous batch fetch) 2026-09-26 23:32:13 -07:00
kyle 4fa779be9c orion: pin sift bdc093e (P5b long-doc questions + P5c extract; P6 plan docs) 2026-09-26 20:22:32 -07:00
kyle cd2bd3d0e3 orion: pin sift 1311cf3 (P5a PDF documents) 
PDF extraction in a re-exec'd child, per-page cache, map/pages/
grep, cited digests. Module MemoryMax 512M -> 1G via pinned sift.
 2026-09-26 10:57:22 -07:00
kyle ec39a569de dixie: Z tmpfiles rules use ~0755 so existing files stay 0644 (exec dropped), dirs become 0755 2026-09-25 07:49:51 -07:00
kyle e5c47007cd dixie: Z tmpfiles rules to relabel dropsonde2 trees 0755 recursively (fixes rsync's 0700 dirs) 2026-09-25 07:48:32 -07:00
kyle 9a5920da2b dixie: drop uid/gid from rsyncd modules — daemon runs as dropsonde via SSH, setgroups() fails otherwise 2026-09-25 03:03:40 -07:00
kyle 2305f4099a dixie: rsync --server --daemon sync key (no rrsync/python3 on dixie) 
- authorized key forces rsync --server --daemon --config=/etc/rsyncd-dropsonde.conf
- environment.etc rsyncd-dropsonde.conf: cold + analysis modules, read only=false,
  use chroot=false, uid/gid dropsonde, max connections 2, lock file global
- tmpfiles: /srv/{cold,analysis}/dropsonde2 owned dropsonde
 2026-09-25 02:45:44 -07:00
kyle 2e4aa6ed57 dixie: dropsonde sync key — rrsync -wo /srv (write-only, hyperborea -> dixie) 2026-09-25 02:24:26 -07:00
kyle 0ac90515bf dixie: use script attr for dropsonde-pg-setup (ExecStart heredoc+bash path broken) 2026-09-25 00:51:32 -07:00
kyle c402c1c599 dixie: drop python3.withPackages (duckdb CLI only) to trim the closure 2026-09-25 00:27:38 -07:00
kyle eed2821cea dixie: slice submodule uses sliceConfig, not serviceConfig 2026-09-24 23:42:12 -07:00
kyle 43b8e2288f dixie: listen_addresses under settings, not top-level 2026-09-24 23:41:40 -07:00
kyle 3980242da5 dixie: close the ExecStart indented string in dropsonde-pg-setup 2026-09-24 23:41:25 -07:00
kyle 4b751796c6 dixie: import analysis.nix (was never wired into the module list) 2026-09-24 23:36:31 -07:00
kyle 0c37b07099 dixie: label /srv/cold as ds2-cold, use sfdisk for wipe 2026-09-24 23:20:01 -07:00
kyle fddaffc825 dixie: fix slice unit name, pg-setup ExecStart, listen_addresses 2026-09-24 22:27:12 -07:00
kyle a40873685a dixie: open Postgres 5432 on tailscale0 for dropsonde 2026-09-24 22:17:44 -07:00
kyle 3c2a272722 dixie: dropsonde2 analysis tier (HDD /srv/cold, pg17+timescale+postgis+h3, analysis.slice, duckdb/polars) 2026-09-24 22:03:44 -07:00
kyleandClaude Opus 5.5 c12b2adbfd disk-health plan: deploy via committed master + doas (per CLAUDE.md) 
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
 2026-09-24 18:59:39 -07:00
kyleandClaude Opus 5.5 85d5631fa8 docs: disk-health monitoring plan (SMART/NVMe/SD card) 
Current state verified on 2026-09-24: hyperborea NVMe already exported
via the Debian nvme textfile; no SMART on the NixOS hosts; no
disk-health alert rules. Plan: smartctl_exporter module for the NixOS
hosts, smartmon timer and SD-card kernel-error textfile on hyperborea,
orion scrape job, a disk-health rule group, verification and deploy
notes. For hand-off to another agent.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
 2026-09-24 18:59:15 -07:00
kyle 78d30026ce orion: allowlist vmui's /prometheus API prefix 
Default Server URL is /prometheus; keep import/admin 403.
 2026-09-24 01:56:05 -07:00
kyle f5ef78baca orion: read-only vmui via nginx allowlist + serve :8429 
VictoriaMetrics stays loopback :8428. nginx :8429 allows GET/POST
to /vmui and query APIs only; import/admin/export/flags 403.
tailscale serve HTTPS 8429. Review, no switch.
 2026-09-24 01:47:25 -07:00
kyle 1cbde84601 rift: linger kyle and enable user podman-restart 
Keep rootless podman up after logind idle.
 2026-09-24 01:33:19 -07:00
kyle f32db3c88b metrics: systemd unit-include as [.]service 
systemd drops \. in unit-include and warns about unknown escapes.
 2026-09-24 01:22:16 -07:00
kyle 81897b3f8a metrics: keep llama-straylight out of FleetTargetDown 
35B eviction is not a paging event. params.model stays.
 2026-09-24 00:46:23 -07:00
kyle 385529da39 metrics: phase 3 straylight/rift scrapes and hyperborea units 
Review, no switch. node-laptop for talos/titan is excluded from
FleetTargetDown. llama-straylight scrape uses params.model for
ornith-1.5-35b-a3b (router 400 without it; will load if cold).
 2026-09-24 00:42:06 -07:00
kyle b861c6ee9f metrics: drop llama-dixie-vision scrape 
Router /metrics cannot be up without loading a model.
 2026-09-24 00:29:24 -07:00
kyle ed5e1a1e0e metrics: HonchoDeriverBacklog and include honcho in target-down 2026-09-24 00:21:36 -07:00
kyle 5d8e8c4335 metrics: scrape dixie helper /metrics with model param 
Router-mode llama-server returns 400 without it.
 2026-09-23 23:57:36 -07:00
kyle b1d5aa7e64 metrics: drop alertmanager channel override 
Webhook is already bound to ~alerts; Mattermost's channel
override wants the name, not ~alerts.
 2026-09-23 23:44:24 -07:00
kyle ec74e0b69c metrics: phase 1 follow-up (review, no switch) 
Honcho workspace_name keep, node_exporter restart metrics,
drop min_over_time on rule 1, Watchdog heartbeat, pin tether
to b267e2f so labelled series exist and rule 2 can fire.
 2026-09-23 23:43:11 -07:00
kyle 4958493485 metrics: phase 1 orion+dixie (review, no switch) 
VictoriaMetrics on 127.0.0.1:8428. LoadCredential for the Mattermost
webhook and the sift scrape token. Honcho scrapes keep a closed metric
name set. Meta-alerts for target-down, VM size, orion disk. Dixie gets
node_exporter, nvidia_gpu_exporter, and llama-server --metrics.

Tether labelled series still wait on titan's fix. Rule 2 is written
against reason=client_disconnect so it stays silent until then.
 2026-09-23 23:22:38 -07:00
kyle 888e8e24fb orion: pin sift 4372673 (P2.7 metrics) 
Prometheus /metrics behind a dedicated metrics caller; JSON stays
at ?format=json. No search-source change.
 2026-09-23 23:11:30 -07:00
kyle c07c4bfe4f docs: fix tether cancel-storm PromQL (rule 2) 
Label-set mismatch made the ratio empty. Select reason=client_disconnect
and kind=converted; never sum() a tether series without a selector.
 2026-09-23 22:50:26 -07:00
kyle 8b14e1f99f docs: fleet metrics and alerting plan 
Reviewed plan only: VictoriaMetrics on orion, vmalert/Alertmanager,
tailnet exporters, first five rules. No config, no deploy.
 2026-09-23 22:48:51 -07:00
kyle 504d4eb3f1 orion: pin sift e62cfc8 (restricted tavily cap skip) 2026-09-23 16:38:41 -07:00
kyle bc30315344 orion: sift tavily+searxng, cap 130, tavily no_rerank 
Kyle chose Tavily-first with SearXNG fallback. Brave stays off.
 2026-09-23 16:25:01 -07:00
kyle 75b813b10b orion: SearXNG-only until Kyle picks sift search default 2026-09-23 16:20:38 -07:00
kyle 415589cc13 orion: pin sift 1c57845, brave_llm only for live fixture 
RRF original-URL fix. SearXNG and Tavily stay off until Kyle picks a default.
 2026-09-23 16:10:59 -07:00
kyle 573b9dcfd8 orion: disable sift brave_llm for bake-off config 1 2026-09-23 15:42:54 -07:00
kyle c84ca93024 orion: disable sift tavily for bake-off config 2 2026-09-23 15:36:33 -07:00
kyle 91c021cff9 orion: enable sift searxng, brave_llm, tavily 
Bake-off sources. SearXNG journal is already LogLevelMax=err.
Keys stay in /var/lib/sift/secret.env, not the store.
 2026-09-23 14:14:25 -07:00
kyle c839f7c8ae dixie: fail over USB Wi-Fi to internal CNVi 
NM dispatcher plus a boot oneshot keep the AX9L (mt7921u) primary and
bring up the on-box iwlwifi profile if the dongle is gone. Connection
PSKs stay on-box; this only selects the radio.
 2026-09-23 14:03:30 -07:00
kyle 687fac019f straylight: sift CLI 8ca3ade on PATH at next switch 
Pin follows approved CLI fail-closed for non-default HERMES_HOME.
Does not switch straylight or orion. ~/.local/bin/sift (e36da19)
still shadows until that binary is removed after the switch.
 2026-09-23 13:11:17 -07:00
kyle ed2ef77de6 orion: put sift CLI on PATH 
The unit package is not in systemPackages, so `sift keys` is only
reachable via the store path. Track services.sift.package so it
follows the pin. Lands on the next orion switch.
 2026-09-23 12:33:26 -07:00
kyle fccb26dea4 orion: pin sift 429f6b2 (P3) and drop SearXNG WARNING logs 
Sift P3 ships with sources off. SearXNG engine WARNING lines include
q=; LogLevelMax=err keeps those out of the journal. Brave scraper
stays disabled. Do not enable search sources in this commit.
 2026-09-23 12:25:04 -07:00
kyle 059c2306da orion: pin sift 3687636 (P2.5+P2.6) 
P2.5 redacts restricted logs; P2.6 partitions dcache and pins
restricted callers to guardian-star.
 2026-09-23 11:59:32 -07:00
kyle 988053ac94 orion: pin sift e36da19 (restricted workspace) 
Permit straylight-gs to query guardian-star; fleet callers still refused.
 2026-09-23 08:35:01 -07:00
kyle e6f7bcb566 straylight: bump tracker to 549eda3 
Master adds -changepw. The service unit still starts with -addr and -db.
 2026-09-23 07:51:16 -07:00
kyleandClaude Opus 5.5 46a7241dba dixie: 9B batch-size 512 so decodes interleave with long prefills 
Prefill ran in 2048-token scheduler steps of ~2.4 s; each decoding slot
gets one token per step, so guardian calls (16 tokens) waited behind
Honcho's 10-28k-token prompts: p90 10 s, max 14 s under two sift
digests (2026-09-23 05:50). ubatch is already 512, so prefill
throughput is unchanged; steps drop to ~0.6 s.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
 2026-09-23 05:54:00 -07:00