kyle/boxmaker
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Record the resolution of the M3a "fsync stall": a macOS socket rule

Browse Source 
The handoff and the two stopped rows blamed an environment fsync stall.
The cause was macOS refusing socket options after the peer closes, in
loopd and in the brokerd test client. The handoff now opens with the
resolution, the log gets a review note (the stopped rows are kept as
written), and the lessons gain I13 (measure what you blame, and name
the machine) and T18 (the gate must pass on Talos and on the Mac).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
kyle
2026-09-22 18:35:30 -07:00
co-authored by Claude Opus 5.5
parent 57dc7899a9
commit 0a9df81fe4
3 changed files with 52 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/implementer-log.md
+17
View File
@@ -293,3 +293,20 @@ orchestrator and the workers (workers with thinking off); the commit trailer sti
01 and 02 come out almost identical to it (`wire.rs` differs in one line), which is expected:
their task files give the types verbatim, so the likeness shows nothing either way. The review
will say more.
### M3a, the stop before task 20 ("brokerd admin fsync stall") — reviewed 2026-09-22 by the design model (Claude)
The two `stopped` rows for this (`M3a/20-22` and `DEBUG-HANDOFF.md`) reached the wrong conclusion,
and the fix they committed hid the bug instead of fixing it. The stop itself was right: the gate
failed and the cause was not understood. `docs/M3a/DEBUG-HANDOFF.md` now opens with the resolution.
| # | Severity | Owner | Finding | Fix |
|---|---|---|---|---|
| 1 | high | implementer (M3a/17) | `BrokerPort` and the HTTP client set a read timeout before every read. macOS refuses that with `EINVAL` once the peer has closed, so a response that arrived just before the close was reported as an outage. Twelve loopd test binaries failed on the Mac; none on Talos. | `loopd::socket::set_read_timeout` (`00a85c1`) |
| 2 | medium | task (M3a/13) | The given test client `next()` does the same, so every admin test that reads a second frame failed on the Mac (40 of 40 runs at `2408e2c`). | Fixed in the crate and in `docs/plans/M3a/files/` (`d7009dc`) |
| 3 | medium | implementer (debugging) | `hold_open` put a workaround for the test client into the handlers: each connection was held for up to 2 s after its final frame, and bytes the peer sent in that time were dropped. | Removed (`57dc789`) |
| 4 | medium | implementer (debugging) | The "fsync stall" was not measured. The sampled `__fcntl` frames are `F_FULLFSYNC`, which is how `sync_all` works on macOS and costs about 4 ms each; the claimed 7 to 12% hang did not appear in 270 runs on two hosts. The row says the tests ran on "this machine" without naming it, and Talos (Linux, tmpfs `/tmp`) and the Mac behave differently here. | Tips I13 and T18 |
After the fixes, `make gate` passes on Talos. On the Mac, fmt, clippy, every test and the gate
scripts pass; `cargo deny` is not installed there, so the gate itself stops at that step. Tasks 20
to 22 are no longer blocked.