What was the loop problem exactly? Are we saying that there was just a bug in the model that they modified and that's really what accounted for it? It's not like they added intelligence; they just fixed a bug in the way the model was set up, right? If that's the case it doesn't seem like it's very important.
Yes — the gain came from stopping a repetitive 'Wait' loop that made Qwen time out on 93% of MATH-500 answers, not from adding math knowledge; what's novel is that a language model edited another model's weights directly, not the math score itself.
The result is significant because a general-purpose LLM read and repaired another model's weights without gradient descent — a new capability — even though this particular fix only removed a known flaw.
4 details5 sources2 disagreements
- The loop: in thinking mode, Qwen3.5-0.8B rechecks answers for thousands of tokens, writing 'Wait, $140 * 6 = $840' over and over; 93% of unedited MATH-500 answers hit the 32,768-token limit without finishing.
- Astra traced the loop to one direction written by the MLP down-projections in layers 15 and 19, then subtracted it from 64 output columns per layer — 131,072 weights, or 0.016% of the model. Token-limit answers fell from 93% to 33.1%, and MATH-500 rose from 7.9% to 48.1%.
- On Gemma 3 270M, which loops on only 8.5% of answers, the same approach produced no established improvement (MATH-500 from 6.8% to at most 7.7%), supporting the view that the edit unblocked existing ability rather than adding skill.
- The prompt debate: researcher Lucas Beyer noted the prompt explicitly told Astra that fixing repetition counted as success; coauthor Jayden Teoh said Claude Code added that hint when expanding a short assignment, without further human feedback.
Room for disagreement
- Critics Matthew Honnibal and Sean Goedecke argue the headline result is standard ML engineering — specifically activation steering — and that the framing overstates a narrow fix.
- MIT coauthors and an independent replication by Alex Krentsel counter that the point is broader: the ability to write functional neural-network weights from scratch emerged only in latest-generation frontier models, making the Qwen fix a demonstration of that capability, not a math breakthrough.
Yeah that's just not very novel, right? Who cares if it edits it? It is interesting that it was able to find this bug but it's not like if you had told me it could actually tweak it easily and add its intelligence to it, then that would be insane. But that's far from what you're saying.
You're right that no intelligence was added — the math gain was a loop bug fix — but the novel part is that a general-purpose LLM found and applied that fix by directly rewriting another model's weights, which no one had shown before.
If this generalizes, models could debug and maintain other models without retraining; the open question is whether it's a repeatable capability or a one-off patch.
5 details4 sources3 disagreements
- Qwen's MATH-500 jump from 7.9% to 48.1% came from cutting token-limit timeouts from 93% to 33.1% — the model already knew the math, it just couldn't stop saying 'Wait'.
- The new capability is that Astra located a single repetition direction across 131,072 weights (0.016% of the 800M-parameter model) using 50 forward-pass batches and no gradient descent; prior interpretability work required humans to find such directions.
- The control result makes the point: on Gemma 3 270M, which rarely loops, the same procedure produced no meaningful math improvement.
- An independent replication by Alex Krentsel confirmed the results and rebuilt the edited model from published logs, scoring 51.0% on a MATH-500 re-run — but the authors admit they haven't built a benchmark or harness to measure how reliably the method works.
- The 50-pass budget produced a cliff: 10 passes got MATH-500 to 12.3%, 25 passes to 14.1%, and 50 passes to 48.1%, with no variance reported across runs.
Room for disagreement
- Some coverage framed this as 'gene editing for neural networks' and readers saw recursive self-improvement, but the authors themselves say 'fixing loops is not the same as making a model better at math' — the edit taught nothing.
- Critics like Trellner argue the method is close to existing ablation/abliteration techniques, just with an LLM finding the direction instead of a human, and that without variance data or a benchmark harness, the 50-pass result could be luck rather than a reliable capability.
- Lucas Beyer noted the prompt explicitly told Astra that fixing Qwen's repetition counted as success; co-author Jayden Teoh says Claude Code added that hint, so how much of the discovery was Astra's own reasoning is still contested.
0
AI answers with web search. May be wrong.
