--cache-ram is per resident model, so the 8 GiB default meant up to 24 GiB
with --models-max 3. Default it to 2 GiB and give ornith-1.5-35b-a3b 16 GiB:
worst case drops to 20 GiB while the model that thrashes gets twice the cache.
ornith also goes from 2 slots of 131072 to 4 slots over one unified 262144
pool: same GPU memory, twice as many live prefixes.
Measured over 30 h before this change: 235 cache evictions and ~196 minutes
spent re-processing prompts over 20k tokens.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
kyleandClaude Fable 5.1
ef4a334062