What Was Measured, and How to Check It

Three things distinguish a measurement here from an assertion, and each is recorded per study rather than claimed in general.

Registered or not. Six of the eleven studies froze their predictions in a committed file before any inference, with the falsification conditions and the uninformative outcomes written down in advance. The others were exploratory and are labelled as such. A result from an exploratory study is a description; a result from a registered one is a test.

Recorded data. Every study writes a machine-readable record into replication/ in the fork, alongside the script that produced it. Where a claim cannot be re-derived even from those records, that is stated — there is one such case and it is called out below. The records exist; access to them is the authors' to grant.

What would have falsified it. A prediction that could not fail is not a test. Several checks in this work were mutation-tested: the code was deliberately broken to confirm the check notices. Where that was done, it is recorded.

Ten corrections, all to my own work, all found and published rather than quietly fixed. They are listed here because a review that never corrects itself is not measuring anything — and because several of them are the same defect I filed against the repository, reproduced in the code I wrote to test for it.

These are the commands that produced every number on this page. They run on CPU with no network and need no pretrained download, and the shipped capture means the whole 109,080-candidate ranking reproduces with no model at all. They are listed so the method is inspectable and so the work can be re-run the moment the repository is opened — which today it is not. Read them as a specification of what was done, not as an invitation you can currently accept.

python test_all.py --require-models     # 216 tests, no pretrained downloads

python scripts/study_scaling.py --family qwen3          # the second ladder
python scripts/study_control_battery.py --family qwen3  # ranking against chance
python scripts/study_cases.py                           # 60 cases, 12 decisions
python scripts/study_executed_search.py                 # measurement inside the loop
python scripts/scaling_figures.py                       # every figure, from the data

The figures regenerate byte-identical from the committed data, so the charts in the report and the JSON in replication/ cannot drift apart without a test noticing.

This review was run on 10–14 September against leochlon/nala and the leochlon/secant preprint. On 12 September arXiv:2609.14127 appeared — Exact finite attention responses from RoPE derivatives — and the review has since been checked against it line by line. That changes what is worth saying here, so this section was rewritten.

The paper had already made the corrections

The README said the method “repaired a program that no prompt could”. The paper reports one prompt control at 1/6 and never asserts that no prompt can solve the case. The README's framing implied a success rate over 107 attempts; Appendix H states that the case “was selected adaptively and uses a supplied correct ranking target… it supplies no success-rate estimate or claim that attention uniquely caused the failure.” The README offered a diagnostic reading — distinguishing attending to the wrong tokens from attending correctly and deciding wrongly; the paper claims no diagnostic capability anywhere.

That is the difference between a repository front page and a paper, and it is to the authors' credit. Read the criticism in the report with that in mind: it is aimed at claims you appear to have already withdrawn.

What survives as a contribution, rather than a correction

One boundary, offered as support for your scoping

The paper is careful to claim local response accuracy — exactness of the edited attention write, with downstream predictions contracted against a baseline gradient and evaluated by native execution. It never claims that a margin determines what the model writes, and the phrase “first divergent token” does not appear in it.

This review's central negative result is that the scoping is necessary. Crossing zero on the next-token logit margin is required for a repair — no counterexample in over a thousand executed edits, 48 registered arms and 29 distinct generated cases — and it is never sufficient: 11 crossings without repair in one study, 29 of 32 in another, and three separate accounts of what closes that gap were proposed and refuted, two of them the review's own. So the instrument's guarantee stops exactly where the paper stops it. That is not a criticism; it is measured evidence that the line was drawn in the right place, and it is worth a sentence in a limitations section if you want to pre-empt the question.

The one place the review still points at something

The title of the preprint, and section 3.3 of the paper, describe attention as an exact unit gradient step. That is an identity about where the step lands — 0 − ∇_BE(0) = ΔM — and the paper never claims the step descends. It is consistent about this: Appendix C.4 uses the quadratic's positivity to characterise the attention-preserving kernel, not to bound the Hessian's spectrum, and no eigenvalue bound appears anywhere.

The word “step” will nonetheless invite the question, so here is the measurement. A unit step reduces E only when λ_max < 2. On your own configuration — base Qwen2.5-0.5B, layer 23 — it increases E in 3 of 28 heads, median λ_max 0.51, so the step descends about nine times in ten. But λ_max reaches 109 in one head at that layer, and pooled across all 24 layers of the Instruct checkpoint the step increases E in 56% of heads. Descent is contingent on the layer's spectrum rather than guaranteed by the construction. One sentence saying so removes a referee's easiest objection, and costs nothing, because the paper never needed descent in the first place.