kyle/imladris
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

dixie: honcho-embed pooling cls -> last

Browse Source 
Qwen3-Embedding is a causal model; its card pools the last token. With cls every
input embedded as its first token's state ("the cat sat on the mat" and "the stock
market crashed today" at cosine 1.0). A CPU side server with --pooling last
reproduced the card's example scores within 0.002 without a manual EOS. Honcho's
stored vectors and sift's embed cache are re-embedded / flushed after the switch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
kyle
2026-09-29 23:46:37 -07:00
co-authored by Claude Opus 5.5
parent 03ad1de40e
commit c8b3b21004
1 changed files with 5 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
hw/dixie/default.nix
+5 -1
View File
@@ -74,7 +74,11 @@ let
[honcho-embed] [honcho-embed]
model = ${modelsDir}/qwen3-embedding-0.6b/Qwen3-Embedding-0.6B-Q8_0.gguf model = ${modelsDir}/qwen3-embedding-0.6b/Qwen3-Embedding-0.6B-Q8_0.gguf
embedding = true embedding = true
pooling = cls ; Qwen3-Embedding is causal: pool the last token (its model card does). cls returned the
; first token's state, so every text embedded by its first token alone (cos 1.0 across
; unrelated texts, 2026-09-29). llama.cpp appends the EOS itself; last reproduces the
; card's example scores within 0.002.
pooling = last
; A whole input must fit in one physical batch AND one slot (batch-size silently caps ; A whole input must fit in one physical batch AND one slot (batch-size silently caps
; ubatch; ctx is split across slots). Bigger batches cost compute buffers fast on a ; ubatch; ctx is split across slots). Bigger batches cost compute buffers fast on a
; shared 12 GB card: 8192/slot took 6.6 GB and 4096/slot did not fit beside the 9B and ; shared 12 GB card: 8192/slot took 6.6 GB and 4096/slot did not fit beside the 9B and