Add the per-slot gate and chat_with_retry
Implemented SlotGate (one request in flight per server slot, FIFO queue, bounded) and chat_with_retry (retry only server-gone errors, jittered backoff within a window and attempt budget). chat acquires the gate and maps GateFull to Busy; Client gained a gate field. Implemented-By: OpenCode session (model recorded in docs/implementer-log.md)
This commit is contained in:
@@ -35,6 +35,15 @@ impl Client {
|
||||
req: &ChatRequest,
|
||||
on_event: &mut dyn FnMut(&ChatEvent),
|
||||
) -> Result<Completion, InferError> {
|
||||
// The gate holds one request per slot from before it is sent until chat returns; a second
|
||||
// request on the same slot queues here, bounded, and is told (Queued) or refused (Busy).
|
||||
let _permit = self
|
||||
.gate
|
||||
.acquire(req.slot, self.cfg.limits.queue_len, &mut |ahead| {
|
||||
on_event(&ChatEvent::Queued { ahead })
|
||||
})
|
||||
.map_err(|_| InferError::Busy)?;
|
||||
|
||||
// 1. Send.
|
||||
let body_text =
|
||||
build_body(&self.cfg, req).map_err(|e| InferError::Protocol(e.to_string()))?;
|
||||
|
||||
Reference in New Issue
Block a user