kyle/imladris
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

dixie: 9B batch-size 512 so decodes interleave with long prefills

Browse Source 
Prefill ran in 2048-token scheduler steps of ~2.4 s; each decoding slot
gets one token per step, so guardian calls (16 tokens) waited behind
Honcho's 10-28k-token prompts: p90 10 s, max 14 s under two sift
digests (2026-09-23 05:50). ubatch is already 512, so prefill
throughput is unchanged; steps drop to ~0.6 s.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
kyle
2026-09-23 05:54:00 -07:00
co-authored by Claude Opus 5.5
parent 8790005c53
commit 46a7241dba
1 changed files with 5 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
hw/dixie/default.nix
+5
View File
@@ -49,6 +49,11 @@ let
; at 96k incl. 5.5 GB weights), so 128k costs ~0.65 GB of the ~2.5 GB free. ; at 96k incl. 5.5 GB weights), so 128k costs ~0.65 GB of the ~2.5 GB free.
ctx-size = 131072 ctx-size = 131072
parallel = 8 parallel = 8
; One scheduler step = up to batch-size prompt tokens, and every decoding slot gets one
; token per step. At the 2048 default a step was ~2.4 s while Honcho's 10-28k prompts
; prefilled, so 16-token guardian calls hit p90 10 s. The GPU already computes in
; 512-token ubatches, so 512 keeps prefill throughput and makes steps ~0.6 s.
batch-size = 512
; Runaway guard carried over from straylight: helper tasks need short answers. ; Runaway guard carried over from straylight: helper tasks need short answers.
reasoning-budget = 2048 reasoning-budget = 2048
n-predict = 4096 n-predict = 4096