kyle/imladris
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

llama-server: cap ornith 9B helper output (runaway guard)

Browse Source 
As Hermes' auxiliary model it received requests with no max_tokens and a
generation ran to 46k tokens (~70 min) after the client's 60 s timeout,
starving the 35B. Cap thinking (reasoning-budget 2048) and total output
(n-predict 4096) server-side; preset verified against a throwaway router.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
kyle
2026-09-21 01:13:30 -07:00
co-authored by Claude Fable 5.1
parent ef4a334062
commit 09d16f2f39
1 changed files with 7 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
hw/straylight/default.nix
+7
View File
@@ -71,6 +71,13 @@ let
[ornith-1.5-9b-uncensored] [ornith-1.5-9b-uncensored]
hf-repo = mradermacher/Ornith-1.5-9B-uncensored-GGUF:Q4_K_M hf-repo = mradermacher/Ornith-1.5-9B-uncensored-GGUF:Q4_K_M
dedup-cache-models = true dedup-cache-models = true
; Runaway guard (2026-09-21): as Hermes' auxiliary model (approval, title, web
; extract) it got requests with no max_tokens, never converged, and one generation
; ran to 46k tokens (~70 min at 11 t/s) after the client's 60 s timeout, starving the
; 35B. Helper tasks need short answers: cap thinking and total output server-side.
; A client that sends its own smaller max_tokens still wins.
reasoning-budget = 2048
n-predict = 4096
; Qwen3-VL-8B Instruct, refusal-ablated (prithivMLmods v2 / mradermacher). ; Qwen3-VL-8B Instruct, refusal-ablated (prithivMLmods v2 / mradermacher).
; Dedicated VLM in the same size class as Ornith 1.5 9B. Explicit mmproj-url ; Dedicated VLM in the same size class as Ornith 1.5 9B. Explicit mmproj-url