kyle/imladris
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

dixie: size honcho-embed so long inputs fit one batch and one slot

Browse Source 
batch-size defaulted to 2048 (capping ubatch) and ctx 8192 over 4 slots left
2048 per slot, so Honcho inputs over 2048 tokens failed with 'input is too
large to process'. Now batch/ubatch 8192, ctx 16384 over 2 slots = 8192/slot,
matching Honcho's EMBEDDING.MAX_INPUT_TOKENS. Verified a 6191-token input.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
kyle
2026-09-22 11:20:15 -07:00
co-authored by Claude Opus 5.5
parent 6932ce9017
commit 12cfe1e519
1 changed files with 7 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
hw/dixie/default.nix
+7 -2
View File
@@ -51,9 +51,14 @@ let
model = ${modelsDir}/qwen3-embedding-0.6b/Qwen3-Embedding-0.6B-Q8_0.gguf model = ${modelsDir}/qwen3-embedding-0.6b/Qwen3-Embedding-0.6B-Q8_0.gguf
embedding = true embedding = true
pooling = cls pooling = cls
; A whole input must fit in one physical batch AND one slot. batch-size defaults to 2048
; and silently caps ubatch, and ctx is split across slots, so both were 2048 in practice:
; Honcho's longer messages (up to its 8192-token EMBEDDING.MAX_INPUT_TOKENS) failed with
; "input is too large to process". Size all three to 8192 per slot.
batch-size = 8192
ubatch-size = 8192 ubatch-size = 8192
ctx-size = 8192 ctx-size = 16384
parallel = 4 parallel = 2
''; '';
# Vision (auxiliary.vision for all agents): its own server on :11433, loaded on first # Vision (auxiliary.vision for all agents): its own server on :11433, loaded on first