What it scores — and what it lost

Three arms, the same 150 LongMemEval questions, the same answer model and prompt — so the comparison measures the memory layer, not the reader. Every number on this page is recomputed at build time from the judged rows, and every one of them opens those rows. The cells Errata loses are printed as losses.

Counting all 450 rows including abstention, the same runs read Errata 66.7, naive 56.2, full-context 54.0. What the accuracies above cost to produce:

The three arms, same questions, same answer prompt

Overall and the four ability columns are accuracy over the 120 non-abstention questions; the ± is the sample sd across the three seeds, which is provider nondeterminism rather than sampling spread — Errata and the naive arm are bit-identical across seeds, which is what their ± 0.0 means. Abstention is scored deterministically by exact match and never reaches the judge. † $/Q and latency are the two columns not recomputed from these rows: Errata’s are measured on rerunF-wave, the last cold-cache run, because rerunJ-arith replayed 447 of 450 answers from cache and would flatter both. Errata’s $/Q is $0.0000257, not zero — the column rounds to four decimals.

The honest gap, kept next to the headline

Information extraction is the one column Errata loses, and it loses it badly: 44.7 against full-context’s 80.7. Cut by the corpus’s own question types rather than by ability — which folds the three single-session types into one column and hides where the deficit is — the entire remaining deficit is 22 of 150 questions.

Both losing rows are write-path gaps — the fact is not in the graph to retrieve — and the first is a priced decision, not a modelling result: lifting the extraction cap that drops long enumerated assistant answers was measured at $20.72 to re-extract the 150 histories against the $4.19 the shipped configuration cost. A $0 deterministic recall pass aimed at the same gap was built, applied, measured, and reverted — it is the second row of the table below. The account is in eval/RESULTS.md.

What we published that lost

Two changes were built in full, judged in full, and rejected by their own measurement. They are printed here with the shipped run, at the same size, because a result you only publish when it wins is not a result. These rows are counted over all 450 rows of each Errata wave.

The baseline both rejections were measured against is rerunF-wave at 65.3. The revert is verified rather than asserted: rerunI-restored differs from it in 0 of 450 answers, which is checked here against the artifacts on every test run.

The three answers that moved, and which way

rerunF-wave → rerunJ-arith, seed 11. Two flipped to CORRECT, one stayed wrong and is printed anyway.

• 2b8f3739$565→$465INCORRECT → INCORRECT
• 85fa3a3f$45→$50INCORRECT → CORRECT
• d851d5ba$2,750→$3,750INCORRECT → CORRECT

The judge was validated before the table was believed

Scored on the committed 120-item control set: 60 perturbed negatives and 60 paraphrase positives, with the judge itself frozen. The overall figure was 15.0% before a disclosed control-set revision that fixed seven defective controls — the revision was made by a predicate over the gold, never by a list of question ids, because selecting controls by the verdict they received is how a control set gets quietly tuned. An unparseable verdict counts as a rejection, which can only hurt FAR’s sibling FRR and never flatter FAR; the other end of that envelope is 18.3% overall and 75.0% on attribution-flip, and it belongs in the same paragraph. Full account: eval/judge-validation.md · open all 120 controls →

τ was not fitted

The abstention gate stays at its a-priori 0.35. Every abstention-positive question the corpus owns is inside the reported test set by design, so a fitted τ would be in-sample and saying otherwise would be false. A sensitivity sweep ships instead: overall is 66.7 flat across τ ∈ [0.20, 0.35] — a plateau, not a knife edge.

The sweep is deterministic and model-free — it treats τ as a veto on the recorded synthesis answers — so it is recomputed here from the same rows the table above is counted from, and it reproduces the committed eval/out/tau-sweep-arith.md line for line.

reproduce · uv run errata-eval parity · uv run errata-eval run --arm errata --seeds 11,22,33 --run-id rerunJ-arith · uv run errata-eval judge --run rerunJ-arith · uv run errata-eval report --runs rerunJ-arith rerunB-nothink rerunC-nothink