PLAN: future work — digest retrieval misses terse summary facts (RP2040 PIO count, prod check after P5b)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-26 20:34:57 -07:00
parent 94da9d9127
commit 583a35883d
1 changed files with 13 additions and 0 deletions
@@ -313,6 +313,19 @@ its acceptance evidence posted in `~agent`.

 ## 13. Future work

+### Digest retrieval misses terse summary facts (found 2026-09-26, after P5b)
+
+On the RP2040 datasheet, "how many PIO state machines in total?" came back **4** from both
+`bm25` and `hybrid` retrieval. The answer, **8**, is a one-line bullet on p. 9 ("8 PIO state
+machines") that neither ranker put in the top 5 chunks, so the model answered from the
+per-block sections, and in the hybrid case also made up a split. `--grep` finds it at once, and
+the skill now says to confirm numbers with `--grep` (`94da9d9`).
+
+Candidates, when there is a phase for it:
+- pin the chunk holding the document's feature or overview list, as chunk 0 is pinned today;
+- run a `--grep`-style exact-term pass over the question's numbers and nouns before ranking;
+- a small eval set of fact questions over the pdfbench corpus, to measure a change rather than guess.
+
 ### Paywalled pages → archive snapshot (Kyle, 2026-09-26; possible P7)

 "If a site is paywalled, we may be able to use archive.is to fetch it."