dixie: 9B batch-size 512 so decodes interleave with long prefills

Prefill ran in 2048-token scheduler steps of ~2.4 s; each decoding slot gets one token per step, so guardian calls (16 tokens) waited behind Honcho's 10-28k-token prompts: p90 10 s, max 14 s under two sift digests (2026-09-23 05:50). ubatch is already 512, so prefill throughput is unchanged; steps drop to ~0.6 s. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-23 05:54:00 -07:00
parent 8790005c53
commit 46a7241dba
1 changed files with 5 additions and 0 deletions
@@ -49,6 +49,11 @@ let
    ; at 96k incl. 5.5 GB weights), so 128k costs ~0.65 GB of the ~2.5 GB free.
    ctx-size = 131072
    parallel = 8
+    ; One scheduler step = up to batch-size prompt tokens, and every decoding slot gets one
+    ; token per step. At the 2048 default a step was ~2.4 s while Honcho's 10-28k prompts
+    ; prefilled, so 16-token guardian calls hit p90 10 s. The GPU already computes in
+    ; 512-token ubatches, so 512 keeps prefill throughput and makes steps ~0.6 s.
+    batch-size = 512
    ; Runaway guard carried over from straylight: helper tasks need short answers.
    reasoning-budget = 2048
    n-predict = 4096