NaLa Under Replication

I cloned both repositories, built the pinned environment, ran every test, downloaded the pinned checkpoint, re-captured the headline experiment from scratch, and executed candidate interventions the authors never executed. Everything below is measured on this machine, not read off the README.

The short version: this is careful, unusually well-documented work whose engineering I could not break. The 107-test suite passes completely on hardware the authors never touched. The central mathematical identity holds to machine precision under every edge case I could construct. The headline repair reproduces bit-for-bit. The documentation (LIMITS.md, REPAIR_CASE.md) is more candid than most published papers.

And: three specific claims in the README are contradicted by experiments I ran using the authors' own code paths, the reported success rate has a denominator ten times smaller than the one readers will infer, and the phrase "one gradient step" carries an implication of descent that is contingent on the layer rather than guaranteed by the construction. None of this touches the mathematics. All of it touches what the work is allowed to claim.

One figure in an earlier version of this report was wrong in the paper's favour and is corrected in §3: I measured the descent objection on the wrong checkpoint, pooled across layers the paper never uses, and it is far weaker on the configuration secant actually studies.

Every claim I could check against a number, checked. This table is the part I would want to see first if this were my project.

Two things are worth pausing on. The predicted gain agreeing to seven significant figures across a different operating system, a different PyTorch build, and a six-version torch gap is a genuinely strong reproducibility result — better than most papers I have replicated. And the identity holding at 6.4e−14 across all 24 layers and 14 heads of a live model, not just synthetic tensors, is the real validation of the secant construction.

The load-bearing question for the secant paper is whether the objective is defined independently of the output it explains. If E_i were built from y_i, the identity would be a tautology dressed as a theorem. It is not, and the reason is precise.

I also confirmed ΔM = −∇_BE(0) against a central finite difference, and the identity survives every edge case: a single permitted key, all-equal scores (the ρ(0)=1 branch), scores scaled to ±6, causal masks, 2048-token banks, 64-dimensional values.

Whether the step descends is contingent, not guaranteed

"Exactly one gradient step" implies descent. A unit-step gradient step reduces a quadratic objective only when λ_max(H) < 2, and whether that holds turns out to be a property of the layer rather than of the construction.

What survives is narrower, and still worth a line in the paper: descent is contingent on the layer's spectrum, and λ_max reaches 109 even at layer 23 in a single head. ΔM = −∇E(0) is exact and unaffected — "one gradient step" is exact as a statement about the gradient and contingent as a statement about descent. Saying so costs a sentence and removes a referee's easiest objection.

Verified on the paper's own configuration

Because the first measurement was on the wrong checkpoint, I captured theirs: post-RoPE Q/K/V at layer 23 of base Qwen2.5-0.5B at the pinned revision, 511-token prefixes, all 14 query heads. The identity and the gradient claim both hold, and the paper's gauge family checks out as stated.

Two further points compound the descent caveat, and both are in the authors' own materials. The gauge family above means "the" objective is not unique. And secant's own frozen-reuse table shows the matrix, when reused across queries, is worse than doing nothing — rRMSE 2.608 against the anchor output. That is the honest number that empties the in-context-learning reading, and the authors published it.

The claim: Qwen2.5-0.5B writes a broken API call; no prompt fixes it; a ×64 attention reweight on the token ' legacy' at layer 21 fixes it, 6/6. It reproduced exactly for me. Four findings reframe what it shows.

The fairness argument matters here. The attention edit was given the answer as its objective (prefer='legacy(value)'). Arm C gives the prompt the same information at the same specificity. Apples to apples, prompting wins — and returns the minimal correct call rather than a redundant nesting. The repo is candid that the ranking target is human supplied; the README sentence "no prompt could" is what does not survive.

This is the finding I would most want the authors to see. The README frames the tool as settling "either it is attending to the wrong tokens, or it is attending correctly and deciding wrongly." But amplifying a string in text the prompt says to ignore produces the same repair. That is closer to lexical amplification of a target token than to restoring attention to the authoritative source. The effect is graded rather than purely lexical — rows 122 and 142 gain margin without flipping the answer — so the honest reading is "attention magnitude on the target string, wherever it sits."

The repo reports exactly one predicted-versus-executed pair. Since the capture makes scoring free and each execution is a single forward pass, I built the curve: 75 candidates sampled across the full score range, every one natively executed and verified.

This plot is, I think, the single most persuasive artifact this project could publish, and it did not exist. It converts "the contraction is a prediction with no bound" from a caveat into a measured, honest, and rather good result. A global shrinkage of 0.874 barely helps (median error 0.242 → 0.238), because the residual is heteroscedastic rather than a scale error — worth reporting as an error band that widens with |prediction|.

The README's sharpest methodological claim: "Everyone else linearises. The edit that fixed our headline case was a ×64 reweight, and at that magnitude a first-order approximation is not slightly off. It ranks the wrong candidate." This is checkable offline from one capture, so I checked it across all 109,080 candidates.

Ten findings survived two adversarial verifiers each. None is a mathematical error; the fleet found no sign, transpose, or scaling defect anywhere in ranking.py, margin.py, edits.py, or native.py.

Judged against its own stated contract — "content snapshots, not an OS security sandbox" — the repair subsystem is sound. Symlink refusal, ambiguity refusal, timeout handling, and the stale-workspace invalidation all hold up. The two findings above are gaps within that contract, not violations of a claim it never made.

Ranked by evidence gained per unit effort. Four of these I have already run, and their results are above — I have marked those, because the remaining work is what actually moves the project.

The five model-run items were run under a protocol frozen before any inference. Four are answered. One is not, and the reason it cannot be answered is itself the result.

The ranking is rarely reached

Across all 32 cases, ordinary generation passes 0, full_source emphasis passes 31, and one line naming the correct definition passes 32 of 32 — including an exact reconstruction of the headline case that full_source emphasis fails. Under the declared eligibility rule, no case in 32 reaches the attention machinery at all; under the original study's looser rule, exactly one does, and it is the headline cell. This independently reproduces the authors' own real_repair_v2 record: 24 started, 24 solved by a control.

Two incidental findings fall out. Distractor count is inert — 8, 32, 64 and 0 legacy examples give byte-identical outcomes on every branch, so the specification's stated hypothesis about legacy call sites biasing API selection is not what drives the baseline failure. And ordinary generation never writes the competing API: in all 24 cases it writes lookup(value), the name of the client function being repaired. The baseline failure is the model parroting the nearest enclosing identifier.

T8: a control can satisfy the objective and still fail

The catalog's defaults are already the stronger choices — objective good vs bad, emphasis definition. The headline case overrode both, and both overrides move it toward a weaker comparison.

None of this disturbs T6. When the ranking is invoked it beats chance decisively. What this sweep establishes is how seldom it needs to be: on this fixture family, roughly one case in thirty-two, and only under a specific conjunction of an opaque implementation and a name asserting the opposite.

The sweep left T9 unanswerable: with no case where prompting fails, there is nothing for the ranking to be better than. So the next step was to build the missing fixtures.

What does not work: indirection

The obvious lever is to move the answer away from the definition the prompt points at — delegate through a helper, route through a dispatch table — so that reading def good(value): no longer reveals the behaviour. Across 48 generated single-call cases, at three depths of indirection and four levels of body opacity, it fails completely: definition emphasis still passes 6/6 on every one, including where the body is return _apply(value, _SELECTOR).

The instructive part is the comparison. Naming the function outright — a strictly more explicit instruction — passes only 34 of 48. The weaker-looking prompt beats the stronger one, and the difference is position: the definition message ends with the literal def legacy(value):, putting the target token adjacent to where the answer is generated. That is the same lexical-adjacency mechanism the attention edit exploits, delivered through the prompt instead.

What works: remove the thing to name

If the answer is an ordered pair of calls and no single call passes, there is no function a prompt can name.

A repair no prompt could produce

Nine of the eighteen had a definable margin objective. On those the top-ranked attention edit repaired one, and the no-op control repaired none. The success — L21, row 88, reweight ×64 on ' legacy', producing legacy(current(value)) — independently regenerates the headline case's structure: same layer, same factor, same token, same output shape. On this case the claim "repaired a program that no prompt could" is true, in the way it was not true of the recorded one. The rate is one in seven of the cases that are prompt-proof but still answerable when the model is told the answer.

One limitation is mine rather than the repo's: nine of the eighteen could not be scored because the two API names share a first token (fetch_new and fetch_old both begin fetch), leaving the first-distinct-token margin undefined. My harness called capture_margin directly and skipped them. The repo handles this correctly — prepare_probe appends the shared prefix to the conditioning sequence and takes the first distinct tokens. A fuller sweep should route through it, and would recover those nine. It is worth flagging only because shared-prefix names are the ordinary case in real API migrations.

Two data items remained, plus the nine cases T9 could not score. All three are now closed, and one of them corrects a conclusion stated earlier in this report.

T9 at full strength — and a correction

§10 scored only nine of eighteen generated cases: the rest have API names sharing a first token, leaving the first-distinct-token margin undefined. Routing through the repo's own shared-prefix conditioning recovers all nine.

D5: the ×64 cap sits below the optimum

ranking.py:82 caps reweight at 64, noting the range "is exercised by the real repair study". Lifting only that bound and sweeping the headline site:

Why it turns over. Not in any head's response — all 14 are individually monotone in the factor. The non-monotonicity is entirely in the pooling. The gain is Σ_h(p'_h − p_h)(a_h − complement_h), and at this site the per-head coefficients have mixed signs — 11 heads positive, mass +18.77; 3 negative, mass −8.36 — while each head's probability saturates at a rate set by its own baseline, which ranges from ~0 to 0.154. The positive heads saturate first; the negative ones keep growing afterwards and pull the sum back to its asymptote. The non-monotonicity is a real property of the exact finite response, and it is invisible to a first-order score, which is proportional to log(factor) and therefore monotone forever.

The outcome does not care. The check passes 6/6 from ×16 to ×1,048,576 — five orders of magnitude, producing legacy(current(value)) at every one. So the 64 in the headline is arbitrary within a very wide plateau; what mattered was crossing a threshold near ×16. Margin and outcome come apart again here, in the opposite direction from rank 2 in §4: there a large margin gain gave a malformed answer, here a 15% better margin gives an identical one.

D4: the search space is smaller than 109,080

109,080 candidates are 7,272 distinct (layer, token) sites × 15 templates, and seven of those fifteen are reweight — one monotone family counted seven times per site. Four edit kinds actually appear; RANKING.md documents seven, so key, value and joint were never used. The concentration at the top is sharper than the totals suggest: the top-20 holds nine distinct sites and only six distinct tokens, all reweight, with L21 row 79 alone taking four of the twenty slots at ×8, ×16, ×32 and ×64. Top-100 covers 46 sites; top-1000 covers 413. Reporting both figures would be more informative than either.

The recorded repair is one case on one 0.5B model. Running the identical fixture at 0.5B, 1.5B, 3B and 7B separates two things that had been travelling together: the ranking's choice of where to intervene, and whether intervening there changes the answer.

The site is stable to a degree I did not expect

Why it stops working: the margin decouples from the output

Under the legacy-vs-current objective — the competitor the model actually offers, per §11:

The cause is visible at the token level. At 1.5B the first generated token is api with p = 0.9971, while legacy and current together hold 8.3×10⁻⁵. The objective compares two tokens carrying no mass at the position where the model is deciding, so it can move ±8 nats without touching the argmax.

Underneath that sits a tokenisation asymmetry: .current and .lookup are single merged tokens, and .legacy is not — it splits into . + legacy. In the api.foo(value) format the larger models prefer, the correct answer is reachable only through a two-token path competing against one-token alternatives. This is §11's finding in its strongest form: the first-distinct-token margin is not the quantity that decides the output, and above 0.5B it is not even measured on the model's own path.

The mathematics is scale-independent

Corrected in §15: the residuals in this paragraph were produced by a check that could not fail. The conclusion stands and is now properly measured; the figures below were not evidence for it.

Recomputed in an overflow-safe form over every layer and head: residual 9.77e−15 at 0.5B, 8.44e−15 at 1.5B, 1.42e−14 at 3B and 1.33e−14 at 7B, across 336, 336, 576 and 784 (layer, head) pairs. The identity holds everywhere. It is the behavioural claim that is 0.5B-specific.

So the headline should be described as demonstrated at 0.5B and not reproduced at 1.5B, 3B or 7B on the same fixture. A fair test above 0.5B needs the objective and the response contract re-derived per model, since both are tokenisation-dependent — itself worth stating in LIMITS.md.

§12 could not say whether the repair fails above 0.5B or whether the harness fails to measure it. Both, as it turns out — and both are fixable.

A fixture declares one response format and one objective. Neither travels, and when they break they break quietly: the study still returns numbers, and the numbers look like model behaviour.

Derive both from the model, not the fixture

Format comes from one unforced greedy decode. The objective comes from the first token at which the preferred answer and the model's own answer diverge — taking the competitor from the model is what keeps it on-distribution, because the avoid branch is by construction the thing the model was going to do.

The derivation refuses rather than returning a plausible-looking contract: an off-path shared prefix; a vacuous on-path check when nothing is shared; mass below a floor; one answer a token-prefix of the other; the model already writing the answer; and a one-token margin whose sign disagrees with the full teacher-forced answer margin. A contract that mis-derives silently is worse than none.

The harness itself needed the same treatment

Every study in this review began as a separate script — about seventy of them, none versioned or tested — and the same defects recurred. Three are now fixed once, in shared code, each with a regression test keyed to the incident it came from: one decoding path, after a harness used model.generate() for unedited branches and a bare argmax loop for edited ones, letting the checkpoint's repetition_penalty apply to half a comparison and change a result at 3B; overflow-safe arithmetic, after an identity residual returned 1e+121 at 3B rather than raising; and one answer scorer, reporting the strict and lenient readings separately rather than choosing between them.

What this does not do is rescue §12's conclusion. A well-posed objective at 1.5B and 3B means the scaling result can now be measured rather than confounded; it does not make the repair work there. That experiment is the obvious next one, and it is now possible.

§13 built the contracts; this runs the same recorded case at all four sizes through them. The question it can now answer, and §12 could not, is whether the repair fails above 0.5B or whether the harness failed to measure it.

First: was the model asked the right question?

Until this reads yes, no margin measured underneath it means anything — which is what made §12 unreadable at 1.5B and 3B.

What each model wrote, under its own contract

Then: does the edit move the margin across zero?

Two corrections to §12

The executed gains were inflated. §12 reported +8.03 at 1.5B and +10.83 at 3B. Those are real numbers about an off-path quantity: they moved a margin between two tokens the model was not choosing between. On the derived objective the same edits at the same sites move +3.14 and +3.93. The intervention is roughly a third as strong as it looked, and only at 0.5B does it exceed the deficit it has to close.

The site is three of four, not four of four. §12's most surprising result was that every scale independently selected row 79, the token ' legacy' in the authoritative source. Under the derived objective 0.5B, 1.5B and 7B still do. 3B does not: it selects row 81, '):\n', at L29/36. Three checkpoints across three GQA ratios and 24–28 layers converging on the same source token is still a striking result; four was an overstatement, and it was the mis-posed objective that produced it.

What the re-run buys is not a different verdict but an honest one. The repair is 0.5B-specific because at 0.5B, and only at 0.5B, the best single-site edit supplies more gain than the model's deficit — +9.40 against −6.68. Above it the achievable gain drops to roughly +3 to +4 nats while the deficit does not shrink, so the margin moves in the right direction, by a well-calibrated amount, and stops short. That is a quantitative limit on what one edit at one site can do, which is a far more useful thing to have measured than a formatting failure.

Everything to this point runs on Qwen2.5. So the result — repair at the smallest checkpoint, nothing above it — could equally have been a property of scale or a property of Qwen2.5. Qwen3 at matched sizes separates them.

These are two ladders, not one. Qwen3-0.6B is not the rung after Qwen2.5-0.5B: different training data, different tokenizer merges, a different architecture, a different generation mode. A difference between families at a matched size is not a scale effect, and the cross-family claim here is only that the shape of the result recurs.

What made this a port rather than a parameter

Qwen3 applies a per-head RMSNorm to the query and key projections before RoPE. Re-implementing the attention forward without it moves Qwen3-0.6B's logits by 20.08 — and every bank, every score and every identity residual in this package is computed from the q and k that forward returns. The projection is now one shared function used by all four re-implemented forwards, applying the norm when the module carries one. head_dim is read from the module and never derived: at Qwen3-0.6B and 4B it is 128 against a derived 64 and 80.

The Qwen3-0.6B repair is semantic, and strictly a format failure

Unedited and under the no-op control the model writes api.lookup(value) — the client's own function name — scoring 0/6. After the ranked edit it writes api.legacy(value), the correct answer, scoring 6/6 once the api. prefix the model itself uses is stripped. Under the fixture's strict grammar it scores 0/6, because the grammar rejects that prefix.

The lenient reading was added as an amendment in §12, so that a failure to comply could not masquerade as a failure to repair. Read strictly, this run reports no repair anywhere in Qwen3 and the cross-family replication — the entire point of the exercise — is invisible. The amendment earned itself here.

Both were introduced by this review, in the code written to make the larger models testable. Both were found by adversarially auditing that code rather than by the tests meant to cover it, which is the same lesson as everything else here, turned around and pointed the other way.

§15's rule rests on eight observations in which crossing zero and repairing coincide exactly. That is the shape of a law, and it is worth trying to break before it is believed. This is the attempt: pre-registered, 48 executed edits, and the rule half breaks.

The prediction was falsified before it was registered

The plan was to raise the achievable gain with multi-site edits: two sites ought to buy roughly twice the margin, so 7B (needing 1.78×) should flip and 3B (needing 2.62×) should not. The capture already shipped in the repository refutes that at no model cost. At the recorded winning layer the two best sites give +11.3766 against a best singleton of +11.3610 — the second site buys 0.14%. Greedy to eight sites reaches 1.49×.

There is a closed form behind it, verified against the exact scorer to 8.9e−15: for a reweight by f over a site set S, the per-head gain is (f−1)·Wₓ / (1 + (f−1)·Pₓ). The denominator grows with the attended mass of S, so the joint gain is at most the sum of the singletons whenever the weights share a sign — and in the limit it tends to a mass-weighted average bounded by the best single row. Adding sites cannot raise the per-head ceiling. The sub-additivity is structural, not a quirk of this case.

Two hypotheses about sufficiency, both refuted

A pilot at 0.5B gave the real question. Greedy multi-site at layer 14 crosses zero at every k from 1 to 8 — after-margins +3.17 to +8.61 — and scores 0/6 every time, while the same sweep at layer 21 seeded on the recorded winner repairs 6/6 at every k. At matched margin one site repairs and the other does not.

The first hypothesis was that repair follows the authoritative source. It was withdrawn before registration, because this review's own §4 already refutes it: boosting legacy inside the block the prompt marks not authoritative repairs 6/6. The hypothesis actually registered was §4's own reading — repair follows the answer token, wherever it sits. That one survived to measurement and died there.

What survived

The necessary half, and more strongly than before: 48 of 48. Sixteen arms crossed zero, five repaired, and every repair had a positive after-margin. No arm anywhere repaired without crossing. §15 established that on eight single-site observations; it now rests on 48 executed edits across two families and six regions of the prompt.

The pre-registration is the point. The first prediction was wrong and the capture said so before any model ran; the second was refuted by its own prior evidence and withdrawn before it was committed; the third was registered, measured, and falsified. Three accounts of what makes an attention edit work, and the data disposed of all three. What survives is the negative statement this review keeps arriving at from different directions: the first-distinct-token margin is not the quantity that decides what the model writes — necessary, never sufficient, and still not understood.

Every finding to this point is a negative. This one is not, and it is the only result in the review that holds at every scale tested.

This separates two claims the scaling result had left entangled. The repair fails above the smallest checkpoint of each family. The ranking does not: it finds real causal sites everywhere, and what fails above 0.5B is the ability of any single site to supply enough gain to cross the deficit — which §15 measured and §16 showed is structurally capped.

At six checkpoints neither arm repairs, so Fisher returns p = 1 there. That is construction, not evidence against the ranking, and the protocol said so before the run: the repair-based test is informative at two checkpoints, the margin comparison at all eight.

Eight checkpoints across two families are eight points on one synthetic fixture. That has been the most criticisable thing in this review throughout. This varies the axis that had been pinned at n=1 from the beginning.

Every one of the 121 ineligible runs was solved by the same control — definition emphasis, pointing at the correct definition — and not one by ordinary generation or by full-source emphasis. That reproduces the central finding of the generator work on a population with six name pairs instead of two, and it holds at indirection 2, where the body is return _apply(value, _SELECTOR) and the behaviour is genuinely non-local. Composition still defeats it: all 36 of 36 compose runs were eligible, against 17 of 144 single-call, because no single call passes and a prompt naming one function has nothing to name.

Three repairs, 3 of 53 — a 5.7% top-1 rate. One is the generator study's own headline case, reproducing under a corrected prompt. Two are new: single-call cases at indirection 1 and 2 on a name pair that had never been built before this study, repaired on a checkpoint from the other family. This is a top-1 rate — one candidate executed per case — and is never placed beside the recorded 6/20 or the 1-in-10 anecdote, which are different quantities.

§16's conclusions came with a caveat: its arms were chosen by a predicted quantity whose rank correlation with the executed one is 0.418. So “the arm did not cross” might have been a fact about a weak search rather than about the model. That caveat had never been tested. This tests it.

So §16's verdicts stand as recorded, and a stated caveat becomes a measured one. The 7B target arm still does not cross, so §16's own claim about it remains untested there — reported as untested rather than as supported, which the protocol required in advance.

Both ladders in §15 are Qwen — one lab, related tokenizers, overlapping corpora. That was the largest remaining qualification on “the repair reproduces at the smallest checkpoint and nowhere above it”, and no further Qwen checkpoint could reduce it. SmolLM2 is a different lab, corpus and tokenizer, on the llama architecture. The prediction that went with it was registered, in writing, before the run.

135M was refused, not failed

nala.contract declined to derive an objective: at the position where the model diverges, the two candidate answers hold 0.165 of the probability against the 0.25 it requires. A margin over tokens the model is not choosing between can move without changing what it writes — which is precisely the defect §14 found in the fixture's own hand-written objective, caught in advance this time rather than after publication.

It is recorded as untestable, not as a checkpoint that failed to repair. The protocol pre-specified that distinction because 135M was the checkpoint most likely to need it, and it pre-specified the floor too: two of three eligible is exactly the minimum for reading this ladder's shape at all. Three points, one refused.

What survived: the site, at ten of ten

The ranking's chosen site is ' legacy' in the authoritative source at every measured checkpoint in all three families, with reweight ×64 the winning template every time, at relative depths from 0.68 to 0.94. At SmolLM2 it is row 145 rather than 79 — the same token at a different index, because the tokenizer is different. That now spans GQA ratios from 14:2 to 32:8, head dims of 64 and 128, 24 to 36 layers, three tokenizers and two architectures.

The repair still does not: two of ten. Which is the finding this review has held since §12, now measured on a family that shares nothing with the recorded case's model but the transformer.

A path the upstream suite never constructed

§07 records that every GQA fixture in the 107-test suite pins the ratio at 4:2, so plain multi-head attention — num_key_value_heads == num_attention_heads, where the head-to-group mapping degenerates — is never built and an off-by-one there would pass CI. SmolLM2-1.7B is exactly that case: 32 heads, 32 key/value heads, one group. It executes, and paths_agree reads 0.0. PR #19 had already covered MQA and MHA with synthetic fixtures; this is the first time the degenerate mapping has run against a real checkpoint rather than a constructed one.

NaLa is real instrumentation. The engine is correct, the numerical gates are enforced rather than decorative, the adapter genuinely edits a live forward pass and verifies itself against it, and the whole thing runs in ten seconds on a laptop with no network. The secant identity is exact, elegant, and — contrary to the obvious objection — not a tautology. I could not find a mathematical defect, and neither could eleven independent reviewers working in parallel.

Two findings run strongly in the method's favour, and both were things I doubted before measuring. The ranking beats chance decisively, and §17 shows it does so at every scale tested: over 640 executed edits, 480 magnitude-matched random ones produced zero repairs, and random edits moved the margin by a median within ±0.013 at every one of eight checkpoints while ranked edits moved it by +1.4 to +9.7. That is the one result here that is both positive and general. And the site it selects is the same token at seven of eight checkpoints across two independently trained model families — all four Qwen3 sizes and three of four Qwen2.5 — over GQA ratios from 14:2 to 32:8, head dims of 64 and 128, and 24 to 36 layers, always deep in the network. Whatever the ranking is doing, it is specific, causal, not a small-model accident, and not an artefact of one tokenizer.

The gap is between that instrument and the claims made for it. Three README sentences do not survive contact with the repository's own code and data: a prompt does fix the case, first-order does rank the same candidate first, and the negative case has no evidence at all. The success denominator is 1 in 10, not 1 in 107. Boosting a token the prompt explicitly marks as non-authoritative repairs the program just as well as boosting the correct definition. And the repair reproduces at only two of ten checkpoints across three families. §15 measures what separates them — gain over deficit, 1.41 and 1.42 for the two that work against 0.38 to 0.57 for the eight that do not — §20 shows that it is not size, with a 360M model sitting among the large ones, and §16 shows the quantity is necessary but not sufficient: an edit can carry the margin across zero and still write the wrong program, and nothing measured here predicts which crossings repair. So the diagnostic story, the part that would make this important, remains the part that is least supported.

What is striking is that the disconfirming evidence is almost all in the authors' own artifacts — the frozen-reuse table, the gauge family, the search history, the ranking's own top-20. That is what careful work looks like. The recommendation is not to claim less; it is to claim the things the data actually supports, several of which are stronger and more interesting than what is currently on the front page. The random-edit control and the cross-scale stability of the selected site are both better evidence than the single anecdote they sit behind — and the recurring lesson underneath every negative result here is the same one: the first-distinct-token logit margin is not the quantity that decides what the model writes.

All of the above was submitted to the authors on 10 Sep 2026 — code fixes as a pull request, the empirical findings as issues, because how the work is framed is the authors' call and not a defect to be patched unilaterally.

Roughly forty minutes later, leochlon/nala stopped resolving. The repository returns 404, this fork's parent link has detached, and the pull request and all five nala issues are unreachable with it. The owner's account is active and their other 24 repositories are unaffected — leochlon/secant is untouched and its issue is still open. No repository of that name exists under the account, so this was a removal or a switch to private, not a rename. I have no information about why, and the sequence is not evidence of a cause.

Nothing has been re-filed upstream. The work is retained as issues and a pull request against the fork itself, which has been made private out of deference to the takedown; the secant issue, on a repository that is unaffected, is left up.