From 46a7241dba17e66debb409dd6819049357564603 Mon Sep 17 00:00:00 2001 From: "K. Isom" Date: Wed, 23 Sep 2026 05:54:00 -0700 Subject: [PATCH] dixie: 9B batch-size 512 so decodes interleave with long prefills Prefill ran in 2048-token scheduler steps of ~2.4 s; each decoding slot gets one token per step, so guardian calls (16 tokens) waited behind Honcho's 10-28k-token prompts: p90 10 s, max 14 s under two sift digests (2026-09-23 05:50). ubatch is already 512, so prefill throughput is unchanged; steps drop to ~0.6 s. Co-Authored-By: Claude Opus 5.5 (1M context) --- hw/dixie/default.nix | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/hw/dixie/default.nix b/hw/dixie/default.nix index 38e1409..0b6649c 100644 --- a/hw/dixie/default.nix +++ b/hw/dixie/default.nix @@ -49,6 +49,11 @@ let ; at 96k incl. 5.5 GB weights), so 128k costs ~0.65 GB of the ~2.5 GB free. ctx-size = 131072 parallel = 8 + ; One scheduler step = up to batch-size prompt tokens, and every decoding slot gets one + ; token per step. At the 2048 default a step was ~2.4 s while Honcho's 10-28k prompts + ; prefilled, so 16-token guardian calls hit p90 10 s. The GPU already computes in + ; 512-token ubatches, so 512 keeps prefill throughput and makes steps ~0.6 s. + batch-size = 512 ; Runaway guard carried over from straylight: helper tasks need short answers. reasoning-budget = 2048 n-predict = 4096