kyle/imladris
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

dixie: honcho-embed back to 2048/slot, with explicit batch-size

Browse Source 
8192/slot (previous commit) cost 6.6 GB of compute buffers and the 9B then
failed to load; 4096/slot did not fit beside the 9B and dialectic either.
2048/slot with batch-size set explicitly; long inputs are handled on rift
instead (EMBEDDING_MAX_INPUT_TOKENS=1536 makes Honcho chunk). All three
models resident at 11.5 GB.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
kyle
2026-09-22 11:24:35 -07:00
co-authored by Claude Opus 5.5
parent 12cfe1e519
commit 91b2e6b33e
1 changed files with 8 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files
hw/dixie/default.nix
+8 -7
View File
@@ -51,13 +51,14 @@ let
model = ${modelsDir}/qwen3-embedding-0.6b/Qwen3-Embedding-0.6B-Q8_0.gguf model = ${modelsDir}/qwen3-embedding-0.6b/Qwen3-Embedding-0.6B-Q8_0.gguf
embedding = true embedding = true
pooling = cls pooling = cls
; A whole input must fit in one physical batch AND one slot. batch-size defaults to 2048 ; A whole input must fit in one physical batch AND one slot (batch-size silently caps
; and silently caps ubatch, and ctx is split across slots, so both were 2048 in practice: ; ubatch; ctx is split across slots). Bigger batches cost compute buffers fast on a
; Honcho's longer messages (up to its 8192-token EMBEDDING.MAX_INPUT_TOKENS) failed with ; shared 12 GB card: 8192/slot took 6.6 GB and 4096/slot did not fit beside the 9B and
; "input is too large to process". Size all three to 8192 per slot. ; dialectic. So 2048/slot here, and rift's EMBEDDING_MAX_INPUT_TOKENS=1536 makes Honcho
batch-size = 8192 ; chunk below that (its token estimate is not Qwen's tokenizer, hence the margin).
ubatch-size = 8192 batch-size = 2048
ctx-size = 16384 ubatch-size = 2048
ctx-size = 4096
parallel = 2 parallel = 2
''; '';