Record the resolution of the M3a "fsync stall": a macOS socket rule

The handoff and the two stopped rows blamed an environment fsync stall.
The cause was macOS refusing socket options after the peer closes, in
loopd and in the brokerd test client. The handoff now opens with the
resolution, the log gets a review note (the stopped rows are kept as
written), and the lessons gain I13 (measure what you blame, and name
the machine) and T18 (the gate must pass on Talos and on the Mac).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-09-22 18:35:30 -07:00
co-authored by Claude Opus 5.5
parent 57dc7899a9
commit 0a9df81fe4
3 changed files with 52 additions and 0 deletions
+33
View File
@@ -1,5 +1,38 @@
# Debug handoff: brokerd admin-test hang (m3a branch)
## Resolution (2026-09-22, design model)
Resolved. The conclusions below are wrong and are kept only as the record of the investigation.
There is no fsync stall. The failure is a macOS rule about socket options, and it was in loopd's
production code as well as in the test client.
- **The cause.** macOS refuses every socket option with `EINVAL` once the peer has closed (XNU
`sosetoptlock`, `bsd/kern/uipc_socket.c`: both `SS_CANTRCVMORE` and `SS_CANTSENDMORE` set),
even while unread data is still buffered and readable. Linux never refuses. Anything that sets a
read timeout before each read therefore fails on macOS as soon as the peer has sent its last
bytes and closed. Measured directly: after a peer writes and closes, `setsockopt(SO_RCVTIMEO)`
returns `EINVAL` on the Mac and succeeds on Talos, and the following read returns the data on
both.
- **It was never intermittent.** At `2408e2c` the admin test failed 40 runs of 40 on the Mac,
every one at `client.rs:60` with `os error 22`, and passed 40 of 40 on Talos. The work before
this handoff ran on Talos (Linux); the handoff and `hold_open` were done on the Mac.
- **It was also a production bug.** loopd set a read timeout before every read in `BrokerPort`
(`Deadline::read`) and in the HTTP client used for llama-server. On the Mac, twelve loopd test
binaries failed with `os error 22`; `make gate` never reached them because `cargo test` stops at
the first failing binary, which was brokerd's `admin`.
- **The "fsync stall" was a misread profile.** On macOS `File::sync_all` is
`fcntl(F_FULLFSYNC)`, so every sync shows as `__fcntl`. It costs about 4 ms here (p50 4.1 ms,
max 8.5 ms over 600 calls), and the admin binary does hundreds, so each run takes 2 to 9 s on
the Mac against under 1 s on Talos, where the test directories are on tmpfs. A sample catches
threads there because that is where the time goes, not because they are stuck. With `hold_open`
in place, 170 admin runs on the Mac and 50 on Talos, and 10 to 20 runs of every brokerd test
binary on each host, had no hang and no failure.
- **The fix.** `loopd::socket::set_read_timeout` takes that one refusal as success on Apple
targets (a socket shut in both directions cannot block on a read), and both loopd call sites use
it (`00a85c1`). The brokerd test client does the same (`d7009dc`, and in
`docs/plans/M3a/files/`). `hold_open` is removed (`57dc789`); it only hid the test client's
problem, and it kept every handler thread for up to 2 s after its final frame.
## The bug
On the `m3a` branch, the brokerd admin tests (and any brokerd test that runs a full broker