Record the resolution of the M3a "fsync stall": a macOS socket rule
The handoff and the two stopped rows blamed an environment fsync stall. The cause was macOS refusing socket options after the peer closes, in loopd and in the brokerd test client. The handoff now opens with the resolution, the log gets a review note (the stopped rows are kept as written), and the lessons gain I13 (measure what you blame, and name the machine) and T18 (the gate must pass on Talos and on the Mac). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -1,5 +1,38 @@
|
||||
# Debug handoff: brokerd admin-test hang (m3a branch)
|
||||
|
||||
## Resolution (2026-09-22, design model)
|
||||
|
||||
Resolved. The conclusions below are wrong and are kept only as the record of the investigation.
|
||||
There is no fsync stall. The failure is a macOS rule about socket options, and it was in loopd's
|
||||
production code as well as in the test client.
|
||||
|
||||
- **The cause.** macOS refuses every socket option with `EINVAL` once the peer has closed (XNU
|
||||
`sosetoptlock`, `bsd/kern/uipc_socket.c`: both `SS_CANTRCVMORE` and `SS_CANTSENDMORE` set),
|
||||
even while unread data is still buffered and readable. Linux never refuses. Anything that sets a
|
||||
read timeout before each read therefore fails on macOS as soon as the peer has sent its last
|
||||
bytes and closed. Measured directly: after a peer writes and closes, `setsockopt(SO_RCVTIMEO)`
|
||||
returns `EINVAL` on the Mac and succeeds on Talos, and the following read returns the data on
|
||||
both.
|
||||
- **It was never intermittent.** At `2408e2c` the admin test failed 40 runs of 40 on the Mac,
|
||||
every one at `client.rs:60` with `os error 22`, and passed 40 of 40 on Talos. The work before
|
||||
this handoff ran on Talos (Linux); the handoff and `hold_open` were done on the Mac.
|
||||
- **It was also a production bug.** loopd set a read timeout before every read in `BrokerPort`
|
||||
(`Deadline::read`) and in the HTTP client used for llama-server. On the Mac, twelve loopd test
|
||||
binaries failed with `os error 22`; `make gate` never reached them because `cargo test` stops at
|
||||
the first failing binary, which was brokerd's `admin`.
|
||||
- **The "fsync stall" was a misread profile.** On macOS `File::sync_all` is
|
||||
`fcntl(F_FULLFSYNC)`, so every sync shows as `__fcntl`. It costs about 4 ms here (p50 4.1 ms,
|
||||
max 8.5 ms over 600 calls), and the admin binary does hundreds, so each run takes 2 to 9 s on
|
||||
the Mac against under 1 s on Talos, where the test directories are on tmpfs. A sample catches
|
||||
threads there because that is where the time goes, not because they are stuck. With `hold_open`
|
||||
in place, 170 admin runs on the Mac and 50 on Talos, and 10 to 20 runs of every brokerd test
|
||||
binary on each host, had no hang and no failure.
|
||||
- **The fix.** `loopd::socket::set_read_timeout` takes that one refusal as success on Apple
|
||||
targets (a socket shut in both directions cannot block on a read), and both loopd call sites use
|
||||
it (`00a85c1`). The brokerd test client does the same (`d7009dc`, and in
|
||||
`docs/plans/M3a/files/`). `hold_open` is removed (`57dc789`); it only hid the test client's
|
||||
problem, and it kept every handler thread for up to 2 s after its final frame.
|
||||
|
||||
## The bug
|
||||
|
||||
On the `m3a` branch, the brokerd admin tests (and any brokerd test that runs a full broker
|
||||
|
||||
Reference in New Issue
Block a user