boxmaker/docs/plans/M2a/11-llama-gate-retry.md

M2a task 11: the slot gate, and retry

Branch: m2a (run git switch m2a; git status --short must be empty, otherwise stop) Commit subject: Add the per-slot gate and chat_with_retry

Goal

Two protections against a pile-up. The gate lets loopd have only one request in flight per server slot; others wait their turn inside loopd, in order, up to a limit. Retry resends a request whose server went away, a few times, with growing waits.

Context

The inference server is shared and can be restarted under us. After a restart every conversation that was waiting would retry at once and queue invisibly on the server. With the gate they queue in loopd, where the queue is bounded and the caller is told (Queued, Busy). Retrying is safe because nothing is recorded until a completion is final: a retry sends the same bytes again.

The gate is held for one request, from before it is sent until chat returns. It is the first code in this milestone that several threads use at once.

Files

• Copy: crates/loopd/tests/retry.rs
• Create: crates/loopd/src/llama/gate.rs, crates/loopd/src/llama/retry.rs
• Modify: crates/loopd/src/llama/mod.rs, crates/loopd/src/llama/chat.rs, docs/implementer-log.md

Interfaces

crates/loopd/src/llama/gate.rs:

#[derive(Default)]
pub struct SlotGate { /* private: a Mutex around per-slot state, and a Condvar */ }

#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct GateFull;

/// The right to have a request in flight on the slot. Dropping it passes the slot on.
pub struct Permit<'a> { /* private */ }

impl SlotGate {
    pub fn new() -> Self;
    pub fn acquire(&self, slot: u32, max_queue: usize, on_queued: &mut dyn FnMut(usize)) -> Result<Permit<'_>, GateFull>;
}

Gate rules:

1. A free slot with nobody waiting is taken at once; on_queued is not called.
2. Otherwise, if max_queue requests are already waiting for this slot, return GateFull at once, without waiting. max_queue of 0 means nobody may wait.
3. Otherwise join the queue, call on_queued(ahead) once, where ahead is the holder (1 if the slot is taken) plus the waiters already in the queue, and wait.
4. Waiters get the slot in the order they arrived. A Condvar wakes waiters in no particular order, so give each waiter a ticket, keep the tickets in a VecDeque, and let a woken waiter take the slot only if the slot is free and its ticket is at the front. Use notify_all.
5. Slots are independent of each other.
6. Dropping the Permit frees the slot and wakes the waiters. Implement Drop.
7. Mutex::lock and Condvar::wait return Err if another thread panicked while holding the lock. Do not unwrap: recover the guard with unwrap_or_else(|poisoned| poisoned.into_inner()).

crates/loopd/src/llama/retry.rs:

pub fn is_retryable(e: &InferError) -> bool;
pub fn backoff_ms(schedule: &[u64], retry: u32, jitter: i32) -> u64;
impl Client {
    pub fn chat_with_retry(&self, req: &ChatRequest, on_event: &mut dyn FnMut(&ChatEvent)) -> Result<Completion, InferError>;
}

Retry rules:

1. is_retryable, one line per variant, all nine listed so that a new variant is a compile error:

   Retried: the server went away and may be back	Not retried
   Connect, StreamClosedEarly, Stalled, LoadTimeout, Http with status 503	Busy, WaitTimeout (it has already waited), ThinkingOverrun, Protocol, Http with any other status

2. backoff_ms(schedule, retry, jitter): retry is 1 for the first retry (0 is treated as 1). The base is schedule[retry - 1], or the last entry past the end, or 0 for an empty schedule. jitter is clamped to -1000..=1000 and moves the wait by up to a quarter of the base: base + (base / 4) * jitter / 1000. Integer arithmetic; huge values must not overflow (saturating_*, try_from). Examples: ([2000, 8000, 30000], 1, 0) is 2000; (…, 1, 1000) is 2500; (…, 1, -1000) is 1500; (…, 2, 500) is 9000; (…, 4, 0) is 30000.

3. chat_with_retry calls chat. On an error that is retryable, and while fewer than retry_attempts attempts have been made: compute the wait with a jitter that differs from call to call (the sub-second nanoseconds of the system clock, reduced to -1000..=1000, are random enough; do not add a crate). If the time since the first attempt plus the wait is more than retry_window_ms, return the error. Otherwise emit ChatEvent::Retrying { attempt, after_ms, error } (attempt is the number of the attempt about to start, so 2 for the first retry; error is the error's Display text), sleep, and try again. When attempts run out, return the last error.

Changes elsewhere:

• Client gets a second field, pub(crate) gate: gate::SlotGate, created in Client::new.
• chat starts by acquiring the gate for req.slot with cfg.limits.queue_len, passing |ahead| on_event(&ChatEvent::Queued { ahead }). GateFull becomes InferError::Busy. Bind the permit to a name such as _permit so that it lives until chat returns; let _ = … would drop it at once.

Steps

• 1. Copy. git switch m2a, then cp docs/plans/M2a/files/crates/loopd/tests/retry.rs crates/loopd/tests/
• 2. See the test fail. cargo test -p loopd --test retry. Expected: it does not compile.
• 3. Write gate.rs and retry.rs, and make the two changes. Run cargo fmt --all.
• 4. See the tests pass. cargo test -p loopd. Expected: retry 13 passed, and every earlier test file still passes. Run cargo test -p loopd --test retry ten times in a row.
• 5. Run the gate. make gate. Expected last line: gate: ok.
• 6. Log and commit. git add crates/loopd docs/implementer-log.md && git commit

Done when

• cargo test -p loopd --test retry reports 13 passed, ten runs in a row; make gate prints gate: ok.
• gate.rs has no unwrap() and no expect(.

Stop and report if

• waiters_are_served_in_order_one_at_a_time fails now and then. That means rule 4 is not met; if you cannot see why after two attempts, stop.