Add the per-slot gate and chat_with_retry

Implemented SlotGate (one request in flight per server slot, FIFO queue,
bounded) and chat_with_retry (retry only server-gone errors, jittered
backoff within a window and attempt budget). chat acquires the gate and
maps GateFull to Busy; Client gained a gate field.

Implemented-By: OpenCode session (model recorded in docs/implementer-log.md)
This commit is contained in:
2026-09-18 14:45:45 -07:00
parent a07c341cbb
commit 59cf89e7ba
6 changed files with 597 additions and 2 deletions
+9
View File
@@ -35,6 +35,15 @@ impl Client {
req: &ChatRequest,
on_event: &mut dyn FnMut(&ChatEvent),
) -> Result<Completion, InferError> {
// The gate holds one request per slot from before it is sent until chat returns; a second
// request on the same slot queues here, bounded, and is told (Queued) or refused (Busy).
let _permit = self
.gate
.acquire(req.slot, self.cfg.limits.queue_len, &mut |ahead| {
on_event(&ChatEvent::Queued { ahead })
})
.map_err(|_| InferError::Busy)?;
// 1. Send.
let body_text =
build_body(&self.cfg, req).map_err(|e| InferError::Protocol(e.to_string()))?;