boxmaker/docs/plans/M2a/11-llama-gate-retry.md

# M2a task 11: the slot gate, and retry

**Branch:** `m2a` (run `git switch m2a`; `git status --short` must be empty, otherwise stop)
**Commit subject:** `Add the per-slot gate and chat_with_retry`

## Goal

Two protections against a pile-up. The **gate** lets `loopd` have only one request in flight per
server slot; others wait their turn inside `loopd`, in order, up to a limit. **Retry** resends a
request whose server went away, a few times, with growing waits.

## Context

The inference server is shared and can be restarted under us. After a restart every conversation
that was waiting would retry at once and queue invisibly on the server. With the gate they queue
in `loopd`, where the queue is bounded and the caller is told (`Queued`, `Busy`). Retrying is safe
because nothing is recorded until a completion is final: a retry sends the same bytes again.

The gate is held for **one request**, from before it is sent until `chat` returns. It is the first
code in this milestone that several threads use at once.

## Files

- Copy: `crates/loopd/tests/retry.rs`
- Create: `crates/loopd/src/llama/gate.rs`, `crates/loopd/src/llama/retry.rs`
- Modify: `crates/loopd/src/llama/mod.rs`, `crates/loopd/src/llama/chat.rs`,
  `docs/implementer-log.md`

## Interfaces

`crates/loopd/src/llama/gate.rs`:

```rust
#[derive(Default)]
pub struct SlotGate { /* private: a Mutex around per-slot state, and a Condvar */ }

#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct GateFull;

/// The right to have a request in flight on the slot. Dropping it passes the slot on.
pub struct Permit<'a> { /* private */ }

impl SlotGate {
    pub fn new() -> Self;
    pub fn acquire(&self, slot: u32, max_queue: usize, on_queued: &mut dyn FnMut(usize)) -> Result<Permit<'_>, GateFull>;
}
```

Gate rules:

1. A free slot with nobody waiting is taken at once; `on_queued` is not called.
2. Otherwise, if `max_queue` requests are already waiting for this slot, return `GateFull` at
   once, without waiting. `max_queue` of 0 means nobody may wait.
3. Otherwise join the queue, call `on_queued(ahead)` **once**, where `ahead` is the holder (1 if
   the slot is taken) plus the waiters already in the queue, and wait.
4. Waiters get the slot **in the order they arrived**. A `Condvar` wakes waiters in no particular
   order, so give each waiter a ticket, keep the tickets in a `VecDeque`, and let a woken waiter
   take the slot only if the slot is free **and** its ticket is at the front. Use `notify_all`.
5. Slots are independent of each other.
6. Dropping the `Permit` frees the slot and wakes the waiters. Implement `Drop`.
7. `Mutex::lock` and `Condvar::wait` return `Err` if another thread panicked while holding the
   lock. Do not `unwrap`: recover the guard with `unwrap_or_else(|poisoned| poisoned.into_inner())`.

`crates/loopd/src/llama/retry.rs`:

```rust
pub fn is_retryable(e: &InferError) -> bool;
pub fn backoff_ms(schedule: &[u64], retry: u32, jitter: i32) -> u64;
impl Client {
    pub fn chat_with_retry(&self, req: &ChatRequest, on_event: &mut dyn FnMut(&ChatEvent)) -> Result<Completion, InferError>;
}
```

Retry rules:

1. `is_retryable`, one line per variant, all nine listed so that a new variant is a compile error:

   | Retried: the server went away and may be back | Not retried |
   |---|---|
   | `Connect`, `StreamClosedEarly`, `Stalled`, `LoadTimeout`, `Http` with status 503 | `Busy`, `WaitTimeout` (it has already waited), `ThinkingOverrun`, `Protocol`, `Http` with any other status |

2. `backoff_ms(schedule, retry, jitter)`: `retry` is 1 for the first retry (0 is treated as 1). The
   base is `schedule[retry - 1]`, or the last entry past the end, or 0 for an empty schedule.
   `jitter` is clamped to -1000..=1000 and moves the wait by up to a quarter of the base:
   `base + (base / 4) * jitter / 1000`. Integer arithmetic; huge values must not overflow
   (`saturating_*`, `try_from`). Examples: `([2000, 8000, 30000], 1, 0)` is 2000; `(…, 1, 1000)` is
   2500; `(…, 1, -1000)` is 1500; `(…, 2, 500)` is 9000; `(…, 4, 0)` is 30000.
3. `chat_with_retry` calls `chat`. On an error that is retryable, and while fewer than
   `retry_attempts` attempts have been made: compute the wait with a jitter that differs from call
   to call (the sub-second nanoseconds of the system clock, reduced to -1000..=1000, are random
   enough; do not add a crate). If the time since the first attempt plus the wait is more than
   `retry_window_ms`, return the error. Otherwise emit
   `ChatEvent::Retrying { attempt, after_ms, error }` (`attempt` is the number of the attempt about
   to start, so 2 for the first retry; `error` is the error's `Display` text), sleep, and try again.
   When attempts run out, return the last error.

Changes elsewhere:

- `Client` gets a second field, `pub(crate) gate: gate::SlotGate`, created in `Client::new`.
- `chat` starts by acquiring the gate for `req.slot` with `cfg.limits.queue_len`, passing
  `|ahead| on_event(&ChatEvent::Queued { ahead })`. `GateFull` becomes `InferError::Busy`. Bind the
  permit to a name such as `_permit` so that it lives until `chat` returns; `let _ = …` would drop
  it at once.

## Steps

- [ ] **1. Copy.** `git switch m2a`, then
  `cp docs/plans/M2a/files/crates/loopd/tests/retry.rs crates/loopd/tests/`
- [ ] **2. See the test fail.** `cargo test -p loopd --test retry`. Expected: it does not compile.
- [ ] **3. Write `gate.rs` and `retry.rs`, and make the two changes.** Run `cargo fmt --all`.
- [ ] **4. See the tests pass.** `cargo test -p loopd`. Expected: `retry` 13 passed, and every earlier
  test file still passes. Run `cargo test -p loopd --test retry` ten times in a row.
- [ ] **5. Run the gate.** `make gate`. Expected last line: `gate: ok`.
- [ ] **6. Log and commit.** `git add crates/loopd docs/implementer-log.md && git commit`

## Done when

- `cargo test -p loopd --test retry` reports 13 passed, ten runs in a row; `make gate` prints
  `gate: ok`.
- `gate.rs` has no `unwrap()` and no `expect(`.

## Stop and report if

- `waiters_are_served_in_order_one_at_a_time` fails now and then. That means rule 4 is not met;
  if you cannot see why after two attempts, stop.