kyle/imladris
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

dixie: retire the 4B dialectic model; 9B gets 4 slots over a 48k pool

Browse Source 
Dialectic benchmark (8 hermes-fleet questions): Qwen3-4B answered in 2-12 s but
often wrongly (invented facts, 'no information' on things memory holds); the 9B
answered correctly. All Honcho dialectic levels now use the 9B (rift .env). Its
VRAM goes to the 9B's KV pool: at 2 x 16k, concurrent deriver + dialectic hit
'failed to find free space in the KV cache'. 9B 6.3 GB + embed 2.2 GB = 8.4 GB.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
kyle
2026-09-22 14:29:01 -07:00
co-authored by Claude Opus 5.5
parent 3f68a43c21
commit 6d5586c63c
1 changed files with 10 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files
hw/dixie/default.nix
+10 -6
View File
@@ -40,6 +40,11 @@ let
; web_extract) for all agents and Honcho's deriver/summary/dream model. ; web_extract) for all agents and Honcho's deriver/summary/dream model.
[ornith-1.5-9b-uncensored] [ornith-1.5-9b-uncensored]
model = ${modelsDir}/ornith-1.5-9b-uncensored/Ornith-1.5-9B-uncensored.Q4_K_M.gguf model = ${modelsDir}/ornith-1.5-9b-uncensored/Ornith-1.5-9B-uncensored.Q4_K_M.gguf
; serves the deriver, all dialectic levels and three agents' approval guardians at once:
; 4 slots over one 48k pool (at 2 x 16k, concurrent deriver + dialectic hit "failed to find
; free space in the KV cache" and stalled)
ctx-size = 49152
parallel = 4
; Runaway guard carried over from straylight: helper tasks need short answers. ; Runaway guard carried over from straylight: helper tasks need short answers.
reasoning-budget = 2048 reasoning-budget = 2048
n-predict = 4096 n-predict = 4096
@@ -52,10 +57,9 @@ let
top-k = 20 top-k = 20
min-p = 0 min-p = 0
; Honcho dialectic: non-thinking instruct model so answers are short and fast. ; honcho-dialectic (Qwen3-4B-Instruct-2507) retired 2026-09-22: fast but often wrong in a
[honcho-dialectic] ; dialectic benchmark, where the 9B answered correctly. All dialectic levels use the 9B now;
model = ${modelsDir}/qwen3-4b-instruct-2507/Qwen3-4B-Instruct-2507-Q4_K_M.gguf ; the 4B's VRAM went to the 9B's KV pool. File stays on the stick under qwen3-4b-instruct-2507/.
n-predict = 1024
; Honcho embeddings (1024-dim; EMBEDDING_VECTOR_DIMENSIONS=1024 on rift). ; Honcho embeddings (1024-dim; EMBEDDING_VECTOR_DIMENSIONS=1024 on rift).
[honcho-embed] [honcho-embed]
@@ -175,8 +179,8 @@ in
}; };
systemd.services.llama-server = llamaService { systemd.services.llama-server = llamaService {
description = "llama.cpp router: helper tier (9B guardian/deriver, dialectic, embeddings)"; description = "llama.cpp router: helper tier (9B guardian/deriver/dialectic, embeddings)";
port = 11434; preset = llamaModelsIni; modelsMax = 3; port = 11434; preset = llamaModelsIni; modelsMax = 2;
}; };
systemd.services.llama-vision = llamaService { systemd.services.llama-vision = llamaService {
description = "llama.cpp vision server (qwen3-vl-8b, on demand)"; description = "llama.cpp vision server (qwen3-vl-8b, on demand)";