Review M2a: accept with two follow-up tasks; record models and lessons

The branch passes every check, including make verify-device on
straylight and repeated timing runs under load. Reading and probing
found that inferproxy does not pass an upstream close on to a client
that is still sending, and that the chunked body reader delivers a
stream only when the caller's buffer fills or the stream ends. Both
were also gaps in the tasks and tests, so tasks 14 and 15 carry the
fixes with new tests checked against the reference.

The Model column is corrected: tasks 04 to 06 and 08 to 13 were Ornith.
Lessons gain four implementer tips and five task-writing tips; three
rules are promoted to AGENTS.md.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
2026-09-18 15:43:06 -07:00
co-authored by Claude Fable 5.1
parent ad7bce88a4
commit 58c7738721
8 changed files with 283 additions and 18 deletions
+23 -7
View File
@@ -16,12 +16,16 @@ How it is used:
| # | Tip | Evidence | In AGENTS.md | Seen again |
|---|---|---|---|---|
| I1 | When a rule says "every" or "everywhere", finish by listing each place it could apply (every struct and enum in the file, every script, every branch) and check them one by one. An example in the task shows one place; the rule covers all of them. | M1 finding 1: `deny_unknown_fields` was put on both enums in task 06, where the task's example showed it, and on neither struct. In task 03, where the task said "works on structs and on enums", every type got it. | yes | |
| I2 | A check must fail when it cannot do its job: missing input, unreadable file, a tool that errors. Never throw errors away (`2>/dev/null`, `\|\| true`, an ignored `Result`). | M1 finding 4: all three gate scripts passed when `crates/` did not exist, and hid `find` and `awk` errors. | yes | |
| I3 | Report every problem found, not only the first. | M1 finding 8: `check-lines.sh` exits at the first long file. | yes | |
| I4 | Run the formatter before the gate, and let it decide ordering. | M1 finding 7: three first-gate failures were rustfmt reordering `lib.rs`. | yes | |
| I5 | Log every attempt, including one you abandon. | M1 review: the task 01 row mentions files "from a prior attempt" that has no row of its own. | yes | |
| I6 | Prefer a conversion that can fail (`u32::try_from`) to a cast that is only safe because of a check somewhere else (`as u32`). | M1 finding 6: bounded `as` casts in `frame.rs`. Correct today, but the safety depends on a line ten lines away. | no (already implied by the code rules) | |
| I1 | When a rule says "every" or "everywhere", finish by listing each place it could apply (every struct and enum in the file, every script, every branch) and check them one by one. An example in the task shows one place; the rule covers all of them. | M1 finding 1: `deny_unknown_fields` was put on both enums in task 06, where the task's example showed it, and on neither struct. In task 03, where the task said "works on structs and on enums", every type got it. | yes | M2a: no. All six config structs have it; none of the server-format structs do. |
| I2 | A check must fail when it cannot do its job: missing input, unreadable file, a tool that errors. Never throw errors away (`2>/dev/null`, `\|\| true`, an ignored `Result`). | M1 finding 4: all three gate scripts passed when `crates/` did not exist, and hid `find` and `awk` errors. | yes | M2a: no recurrence. |
| I3 | Report every problem found, not only the first. | M1 finding 8: `check-lines.sh` exits at the first long file. | yes | M2a: not exercised. |
| I4 | Run the formatter before the gate, and let it decide ordering. | M1 finding 7: three first-gate failures were rustfmt reordering `lib.rs`. | yes | M2a: no fmt failures; the first-gate failures were clippy. |
| I5 | Log every attempt, including one you abandon. | M1 review: the task 01 row mentions files "from a prior attempt" that has no row of its own. | yes | M2a: partly. Task 11's `stopped` row was overwritten by its later `done` row. |
| I6 | Prefer a conversion that can fail (`u32::try_from`) to a cast that is only safe because of a check somewhere else (`as u32`). | M1 finding 6: bounded `as` casts in `frame.rs`. Correct today, but the safety depends on a line ten lines away. | no (already implied by the code rules) | M2a: no `as` casts at all. |
| I7 | A `read` returns as soon as it has any data to give. It blocks only when it has none. Never keep reading to fill the caller's buffer. | M2a finding 2: the chunked reader delivered a whole stream at its end. | yes | |
| I8 | When forwarding between two connections, pass a close on in both directions. Do not rely on the client half-closing first. | M2a finding 1. | no (specific to inferproxy) | |
| I9 | Never end a turn by describing what you are about to do. Do it, then report. | Two Ornith turns in M2a ended with a "## Objective" plan and no tool call. | yes | |
| I10 | If a tool you were told to use does not exist, stop and say so. Do not invent a command in its place. | Laguna as coordinator ran `opencodec`, which does not exist, and then diagnosed its own typo. | yes | |
## Tips for writing tasks
@@ -35,6 +39,11 @@ How it is used:
| T6 | Do not tell the implementer how to lay out what a formatter controls. | M1 finding 7. |
| T7 | A follow-up task works well as: what the reviewer observed, which part was the task's fault, a failing test to copy in, and the new rules. | Tasks 08 and 09 each passed the gate on the first run, and 09 handled six forms its self-test did not contain. |
| T8 | Vet a dependency by testing it against an independent implementation before naming it in a spec, whoever wrote it. Include the boundary cases of its algorithm. | The owner's `emsha` 1.0.3 passed its own tests and hashed every message of length 63 mod 64 wrongly. A 90-case differential run against `sha256sum` found it in minutes. Same shape as T5: an author's tests share the author's blind spots. |
| T9 | State the contract of a standard trait the implementer must honour, in the task, even when it seems obvious. | M2a finding 2: task 04 said what `Body` reads but not when `read` must return. Two implementations, the implementer's and the reference, both got it wrong in different degrees. |
| T10 | Test *when* streamed data is delivered, not only *what*. A test that compares final bytes cannot see buffering. | M2a finding 2 passed 15 tests that only compared bytes. |
| T11 | Make the fake client behave like the real one. If the real client never half-closes, no test client may half-close either. | M2a finding 1: all test clients half-closed; `loopd` does not. |
| T12 | A weak model should not coordinate other sessions. Use the shell driver (`tools/run-plan.sh`), whose checks are code. | The `opencodec` episode. |
| T13 | Set the sampling defaults on the server for coding agents (`temp`, `top-p`, `top-k`, `reasoning-budget`). OpenCode sends none, so the server's default temperature of 1.0 applies, and thinking has no cap. | Five Ornith turns ran 16k tokens of thinking to the output limit and produced nothing. |
## What worked and should be kept
@@ -48,7 +57,7 @@ How it is used:
- One task, one fresh session, one commit, with only listed paths staged. The history reads as the
plan.
## What we have seen of this implementer so far
## What we have seen of the implementers so far
Nine tasks, all in M1, all small and pinned by tests, so treat these as first impressions.
@@ -60,3 +69,10 @@ Nine tasks, all in M1, all small and pinned by tests, so treat these as first im
- Given a clear defect report and a failing test, fixes the defect minimally and generalises.
- Gate on the first run: 6 of 9 tasks. Wall time per task, including the owner's turnaround: about
10 to 55 minutes.
M2a added two more models. Ornith-1.5-35B-A3B did nine tasks: it found and fixed a real bug in its
own chunked reader, wrote module docs everywhere, and stopped correctly when a prerequisite was
missing; its two failure modes were runaway thinking to the output limit and ending a turn with a
plan instead of a tool call. GLM-5.3 did one task, the assembler, correctly and quickly, with the
best comments of the milestone. Laguna did three and a half tasks in M2a with more nudging than in
M1, and failed as a coordinator. First-gate pass rate for the milestone: 8 of 13.