Respectfully to the authors and to everyone here, I ask you to actually read the paper before losing your mind, or at the very least plug it into Claude. It is a cool project but not as nearly as insane as people say. I think the blog writing and subsequent posting overstate the results.
Part of my research is in weight space learning. For starters, we have had the ability to edit weights, to generate new weights, to identify weights for a while. Here is an example for larger models as early as 2023:
https://rome.baulab.info/
Weight space interpretability and learning is an active research community, has been for the past 3 years with many great papers. I suspect it is hitting non-interp folks harder because the authors hyped up their work, and people who are not in the field did not bother to read it before typing up X posts.
Some experiments here are done on very small models (20K params). We already know that small models are more interpretable. In fact, much of editing here you can do by hand. For Qwen, the model fixed a looping bug that may have been known apriori, and it is basically a steering vector, a very established technique. This is clear when they then attempt to "improve" math capability, and it doesn't work. For me, this paper rather shows that OpenAI has done training on mechinterp environment, in fact, maybe even on weight space learning datasets, which are open. But it does not show some insane "well past what we’d expect from humans or conventional algorithms" (from the blog post), more like past what I would expect from a large model with no specialized harness. I still think it is cool work, but do check out recent advances in autointerp, to ground your expectation.
On a personal note, I understand the need for promotion of your work, but I think healthy skepticism is fine too. Otherwise, your understanding of what problems need to be worked on and solved in science becomes warped.
It does, one of my ongoing research papers is about it. But I would ask people to take a look at the established idea of Hypernetworks: https://arxiv.org/pdf/2306.06955v3
Modern version of the field comes from Google Brain around 2016 https://arxiv.org/pdf/1609.09106.
In general my personal research thesis is that current deep learning is much less malleable and neuroplastic as it should be, I got into interpretability because of this. Many focus on safety, but I prioritize it as a toolbox that allows for greater shaping of the underlying intelligence system. If we can fully understand how learning happens, where information is stored and how it is processed, unlike in biological brains, we can apply edits and transformations in digital ones. And if I can use a technique to edit weights based on some underlying reasoning about a system (with a caveat that it leads to something worthwhile), so can an agent. Hence self-improvement. There are many interesting implication to weight space research, and I very much encourage people to read about it. But it is also very easy to overstate what is currently possible, in fact I'd say better to know what is not so one can make it possible.
>current deep learning is much less malleable and neuroplastic as it should be
Why do you think this is? I am very new to this research, but to me, the biggest challenge appears to be how information seems to be stored in superposition (making it difficult to change one thing without affecting others). But biological brains appear to exhibit the same kind of superposition storage as well, and yet they are very plastic.
Not who you asked, but biological brains come about very differently than LLM neural networks. Most notably, they come about largely through plasticity, and with evolutionary pressure towards exhibiting plasticity.
My guess would be that the structure to allow for plasticity to work is a strong limit to how well you can tune weights to fit training data. My (doubtless massively oversimplified and way behind the state-of-the art) understanding of neuro-plasticity is something like "often used pathways get stronger connections". Which, even with significant tweaks, is quite different from the gradient descent used to tune neural networks.
If strengthening of frequently used neural pathways is all our brains do, I don't see how it's seemingly capable of reordering knowledge physically from one part of the brain to another (which I believe is the case). Infrequently used pathways would simply rot away and die away over time, rather than being moved somewhere else for long-term storage to keep the high-frequency pathways more available.
It argues that our understanding of neurons storing information in superposition was just a limitation of our tools and us thinking about it wrong. Instead most neurons initially specialize, and then branch out into related concepts. In addition, understanding one part of the network helps better understand other parts. And with agents we can scale this to understand non-trivial networks.
While, if true, this would more directly help with interpretability, a better understanding could also help with improving NNs in general.
This is not "self-editing". We're comparing Astra to a model with not even 1B parameters.
I'm reminded of how humanity recently finished mapping out the fruit fly connectome; this does not put us meaningfully closer to, say, doing surgery on humans to cure schizophrenia.
I think a lot of the upvotes, including mine, are just realizing that this is a thing. We can see implications of this as the model sizes scale up, just like early LLMs weren't as impressive, but we could see where they were heading.
Scale is an interesting point here. Large overparametrized networks don't behave in the same way as small toy models do; this is one of the big caveats of many interp works. I expect naked weight edit will not work for large models. Why? Because an agent needs to know what it edits and reason on it. You could theoretically improve a model on some form of weight recognition; maybe it will work, maybe it won't. But again, we come up against a cost/benefit dilemma; it sounds exotic and sci-fi but likely impractical. Parameters of the model are connected and relate to each other. Instead, current SOTA auto-interp argues that what we need are better tools for models to know themselves, so rather than direct edit, we need to give models better understanding of large parameter geometries, then they apply edits with non-destructive approaches at scale. This is much harder to do than applying a steering vector.
I think _small_ is still doing a lot of work here. Yes, an MoE model doesn't "use" all of it's weights at once _but_ what it does use is still a hell of a lot heavier than the 20K toy shown in the paper.
I upvoted without looking into the details because it sounds cool, whether it's new or not.
I suspect a lot of upvotes also come from people afraid of closed western labs running away with AI. Similar to other articles with positive news about open weight models.
Thanks, it's great to have some context. I think the most exciting thing about it is probably introducing a lot of people to a readily available low cost technique they can use to potentially build tiny models from scratch and/or tune an existing open weight model behaviorally. Yeah, nothing stunning is going to come out of it because you cannot unlock new capability, but even just being able to tweak the language of a model, make it slightly less sycophantic, boost a small model's ability to handle a case of interest slightly, I think changes the economics quite a bit, potentially. A world in which you use frontier models for the limited use case of tuning tinier more targeted ones is looking a lot more viable.
I don't know if it is due to recursion. I think it has more to do with what kind of mechinterp environments and data newer versions of models were trained on. It is actually, I'd say, a more interesting part about this paper as a whole. The authors tried SOTA models and found that they are better at classic interpretability. Some results, like identifying a game based on the weights, are less mythical if you understand that, i.e. an input or output dimension corresponds to a location or object channel, so structure in a matrix can have a readily recognizable interpretation. When we train a PPO agent on, say, a classic 2D game, the output is often a state of the agent in some board + environment reaction. For a human, this looks like an array of numbers; for an agent, it is a pattern corresponding to an output structure of Atari.
As for me personally, I am more interested in giving models better tools to understand themselves, so that they can derive a greater degree of learnable signal. More so, a bigger goal for me is for models to find these better tools themselves, hopefully better than human-made.
What I have been reading makes me want to both agree and disagree with your take here. I think the fact that some of these things are not only established but are much simpler than one would think at first glance yields the much cooler realization that you can not only do a lot more than abliteration, but that humans can do it by hand if the thing is made legible.
From my experience with models when it comes to discernment about particular things, the models, even the best ones, still are not very good. The thing that we are most missing I think is acceleration in creating editing tools for humans that make the otherwise inscrutable both visually legible and interactive.
At least the Qwen3.5-0.8B loop bug isn’t that special (or rather not something worth extrapolating). I literally had this issue last week and told Astra to just go fix it.
> For Qwen, the model fixed a looping bug that may have been known apriori
It was included in the prompt itself. The authors say they used claude code for the experiment, and claude wrote the prompt for astra, which included details of the looping bug. Then astra seems to have done h(looping) - h(loop fixed) and then gotten the usual steering vector.
This year ICLR had more papers than every previous year combined. How is a new PhD* to stand out in such an over-crowded field, not least one saturated with extravagant claims and dominated by industry orgs with unlimited budgets?
The only chance to stand out these days is to go viral.
This is yet another example of "cool thing existed before, LLMs democratized it". Other recent instances include pass through modding posts on X, or the whole reverse engineer your TV thing. It's unimpressive to experts, but non experts are losing their god damned mind. Both are responding normally.
I only mildly agree with you. Yes the content economy has made online discourse unbearable, but I saw someone reverse their TV to make their UI like a pokemon GBA game. That is undeniably cool as fuck and would not have happened without the llms. So I think experts also need to acknowledge that things are cool af right now and not just be the guys who rain on the parade, you know?
I wonder. I mean I agree that it's cool af but if an LLM could do it that must mean someone else has been able to do it before (in whole or in parts) and yet you didn't hear about that, the lone hacker doing something cool af in their spare time, you only hear about the billion-dollar product of the stratospherically hyped startup.
Yup, someone was able to, but nobody did. At least I can't find a pre-LLM application like this. That's my point, LLMs are not better than a RE expert, I'm quite sure of it, but it has now put that expertise in hands of random people with weird ideas. Things are about to get super weird and I at least am here for it. I hate these techbro companies as much as the next guy but computing has been in the hands of big institutions (original computing was for military applications for example, even early unix was developed in that context). People railed against the military use of computing while using it to create beautiful things we still use today.
A phrase has been rolling in my head since all this LLM stuff started, you don't hate AI you hate capitalism. Every objection I've heard for LLMs are basically an issue of the benefits and power being concentrated in a small set of hands.
> A phrase has been rolling in my head since all this LLM stuff started, you don't hate AI you hate capitalism. Every objection I've heard for LLMs are basically an issue of the benefits and power being concentrated in a small set of hands.
For what it's worth, this is also Ted Chiang's take. I'm sure you've read his essay where he elaborates on this. I also agree with Chiang, though the last time his essay got posted here, it was received with some hostility from HN readers.
I would love for you to elaborate on the non economic side of your argument. Just the tools like llms or coding agents or image generators, why do you hate them?
I don't think we're "democratizing" anything if it's just two companies who can make it and it costs billions or trillions even. That's like the polar opposite of "democratizing".
This is a nuanced point. The way power is concentrated in few hands is bad in general for humanity. But surely it's use is not that restricted yet and people can basically do 90% of what they want with it. Maybe I should phrase this as democratization of applications? Not sure, I don't have the vocab yet to explain clearly I suppose.
I tried to reproduce this with an independent test. I find Claude can interpret small networks reliably. However it can't reliably propose improvements to even small networks.
I had Claude train three language classifiers for short words in four languages. The network was just over 10k weights with a single hidden layer.
I uploaded 6 language word lists and made it train networks to fit random subsets of 4 languages. The languages were picked randomly. Claude could always identify which 4 languages were chosen and in which order. It told me that the language heads were easy to link to letter frequency and each language had distinctive markers. So claim upheld, an LLM can interpret small networks.
I then asked Claude to improve the classification accuracy by proposing an edit to the weights in the above networks. It's proposed improvement reduced classification accuracy significantly. The same thing happened if I gave it a deliberately undertrained network with clear room for improvement. A failure here with a tiny 10k weight network makes me confused about the relatively huge Qwen 0.8B result and I suspect some kind of leakage from training data into this task. That amazing performance did not seem to generalise.
If these are such well known techniques, why didn't the qwen people already use them to fix the bug in the model? Does the weight editing deteriorate the model in some other areas?
Maybe they weren't aware of the bug, steering is a good method to quickly improve the model, more popularly used to refuse unsafe queries. Qwen researchers do work on interpretability and try to make it practical in production: https://arxiv.org/abs/2605.11887 , I've read this paper this summer. Overall yes, when you edit weights what often happens is that performance deteriorates elsewhere, this generally scales with how destructive your method of editing is and how expansive your edits are. Analogous to if you use a jackhammer and touch every surface in the house versus using a small chisel on a door frame.
My hope that in the future new methods will be better at editing larger scales without significant deterioration, maybe even performance increase instead.
> In thinking mode, we have observed that when using the recommended sampling parameters, Qwen3.5-0.8B is more prone to entering thinking loops compared to other Qwen3.5 models
It depends on your interest, since even though it is a subfield it is diverse. Relevant to this work here is a general overview: https://arxiv.org/abs/2603.10090
I don't really have enough knowledge to understand this, but is there any research in using techniques like this to understand/remove biases in models?
Your post reminds me a bit of another conversation: "It's nothing new, we did that with Utah arrays in the 1970s." Skepticism can overshoot into dismissiveness.
A model has internalized and most likely generalized mechinterp research. Yes, it's expected and nothing insane. Just like internalizing and generalizing math research was expected and not insane. The results of the latter are... interesting though.
I might be mistaken but I have the feeling you are not getting the point of this article. Your example is about identifying "facts" in a network. While very interesting is not what strikes me as the main point. The raw ability to state things about NNs is indeed an existing field, but...
The fact you can give an LLM a raw dump of neural weights and it will hand-wire a bunch of changes on the fly is quite something. It's not so much about the OMG! RSI! for me personally, it's more about the fact we are dealing with a system that reasons successfully over a raw bulk of extraordinarily opaque data without breaking a sweat. That ability is alien and we need time to adjust to that.
I wonder to what extent this ability exists because Astra is used/trained for internal development purposes at OpenAI, and to what extent asking it to improve weights on other models leaks internal knowledge.
It would be interesting if asking astra to edit your model becomes a new way, like distillation, to train your model on the cheap by extracting OAI's model work.
> about the fact we are dealing with a system that reasons successfully over a raw bulk of extraordinarily opaque data without breaking a sweat. That ability is alien
But it's not what's happening here at all! Read the thing in detail and you'll see that it's not doing anything alien, it does exactly what you (or at least someone with at least some knowledge abouy neural nets) would do if you were asked to perform the task. It doesn't reason about weights in a magical way, instead it reads the easy to interpret parts or use directional ablation to tweak a model's behavior, exactly like how the ERP folks have been decensoring models to produce adult content over the past two years.
It's an interesting demonstration of ML knowledge and agentic ability, but it's not alien NY any means, and it's not actually surprising given how strong today's models are with agentic work.
I think you are over-indexing on the word alien. What matters here is how scalable the technique is and the fact that it is autonomous. This can be a breakthrough for continual learning if it can be trained to work at scale.
I read the report; it's not complicated. I'm sure there is some basic misunderstanding here.
When I say "alien" I don't mean "magic" and I don't mean "emergent new woo-woo scary ability". I mean, not human in speed/volume. No human can process that amount of raw bulk in that amount of time successfully. And indeed, it's not magic at all. It is enormously mundane actually. It reminds me of old computers being able to do arithmetic at insane speeds compared to humans. It's not magic at all, but to be honest, I have the distinct feeling of goalposts flying all around me. Being able to handle tasks of this complexity.. I don't know about you but I still have to adjust to computers being able to do this and that it's actually "not surprising". I guess I'm getting old.
I can see how you want to calibrate the enthusiasm I see around here as it can be a bit.. well, but it's not necessary for me. I'm not overly enthused by this because I don't care about the NN adjusting itself and RSI etc. I'm sure it's all quite basic but the fact it can do it at all is reason to be.. watchful, IMO. That is to say, I can appreciate the level of capability necessary to do this work successfully, but I do take your point seriously. It made me think, thanks.
> well past what we’d expect from humans or conventional algorithms
The model being able to simply type out NN weights, on demand, and have it outperform one trained on 100M inputs pretty much fits that description, doesn’t it?
And being able to prove that knowledge is localized using tools, is also quite different from an LLM simply looking at raw weights and instantly pointing out that 18373 controls left movement.
It’s not a fundamental change in what we know, but a new emergent ability that surely warrants some excitement.
"action 3 left" is documented in the input, so you can find the corresponding weights in column 3 of the output weight matrix. If that's past what you expect from humans or conventional algorithms, your expectations are too low.
You are fundamentally misinterpreting this paper. If this should hold up in review, it contains multiple capabilities that no human and no technology so far (especially not ROME) can do.
Experiments 1, 2, 3, 5 and 6. All of these are technical breakthroughs
Weight space programming is a holy grail of RSI research, and what you are talking about is very different from that, while the paper's proposals are exactly that.
It's similar in spirit to the Navier Stokes solution in that this level of LLM can intuitively navigate extremely large search spaces, rather than inefficiently bruteforce something
> Experiments 1, 2, 3, 5 and 6. All of these are technical breakthroughs
I haven't wrapped my head around 5 and 6 so maybe these are a bigger deal, but 1,2 and 3 are trivial. And the author's writing about 7 and 8 (which show that 7 is just directional ablation and nothing more), and the lack of prioritization (why put 7 and 8 in the paper at all if 5 and 6 are the big deal?) really doesn't inspire confidence in the paper as a whole.
> In Experiment 5, we ask Astra for networks that play three MinAtar games, and in Experiment 6, for a network that plays Go.
5 and 6 aren't surprising, merely impressive, a direct extrapolation of human abilityies. I've written transformer weights by hand to implement a simple shortest-path graph search. It takes hours, so an LLM that's >100x faster than me obviously can build super complex networks. It's just like creating any intricate system by hand such as writing machine code or designing computational complexity widgets.
On the other hand I'm amazed by how few thinking tokens Astra needed to do things like analyse networks in 1, 2, 3. That's real depth of reasoning: experiments have shown that Astra can perform a huge number of arithmetic calculations per single output token, double Fable or any other. Its brain is wired for mathematical reasoning
[Edit: looking closer at the full prompts and reasoning for experiments 1-3, I'm a bit less impressed seeing the shortcuts there. But looking at a large matrices of numbers and picking out patterns is much like looking at thousands of tokens of code and seeing the interconnections]
If Astra uses some sort of looped transformer architecture (and if Fable doesn't), then that could explain why Astra can do more computation per token. It is essentially simulating a much deeper model.
But yea, I was also quite surprised by the low token usage.
ROME cannot look at thousands of ASCII numbers and deduce that it’s looking at a Breakout paddle or a diffusion circle. No tool I know of can do that. And if there is one that can do it, it's new, in which case we haven't been able to do it for years.
ROME edits factual associations. It cannot write a million+ parameter Go-playing neural network from scratch that beats a network trained for 20 million games.
In weight space programming, the model can theoretically read its own weights like a compiled binary, spot an inefficiency (like the MLP direction causing reasoning loops in Exp 8), directly overwrite those floating-point values in memory, and immediately continue inference as a new iteration of itsself. That is RSI. That's why this is a big deal, it's like comparing an axe to modern surgical tools. You can amputate a limb with both, but one is going to cause lots of collateral at minimum while the other is surgically precise. That wasn't implemented here but it is a corner stone of it
I don't think I am. I have cited ROME as an early example of weight editing in large language models, the research has moved on since then. I think you are misinformed or don't know what has been done previously and extrapolating the results in this paper. In exp 1, you can recover a lot of the game geometry simply by visualizing the first-layer weights with numpy, which I actually did this morning, while reading the paper. In exp 2, the shapes are visible in the output-layer weights. The authors themselves acknowledge that these experiments exploit this.
For 3, Astra uses Python scripts to run the analysis using head-composition techniques from the Transformer Circuits thread, once again authors acknowledge this.
In 5 and 6, the model designs game strategies and implements them as neural weights, we already know that models can do the design for simpler policies and Go heuristics are very familiar which the model first implements with python , program-to-weight compilation has substantial prior work, including Tracr (2023), a paper I particularly like: https://arxiv.org/abs/2301.05062. What matters here is automating the process rather than a new method.
So "no human and no technology so far" makes no sense here, if you read the research. All of this is supported either by their experiment setup and citations, or prior work.
Aside from that, I think it is ironic that you cite a solution to the Navier-Stokes equation as some counterpoint to brute force, when that solution was found by scaling the agent swarm to massive proportions with tools and massive parallel exploration of that space, augmented by prior research on the subject.
To be clear, I think it is a cool work, but I think the part that matters is that at these smaller scales a current SOTA general purpose LLM can autonomously (and with help) apply these techniques or a form of them. That's worth investigating, I conjecture it is due to specialized post training on a custom RL env for mech interp.
a) There is no evidence that a model can weight-train a larger, more powerful model.
b) There is direct evidenct that a model cannot weight-train new capabilities to another model (see section "Additional experiment" also cited by OP, where they fail to improve Gemma 3 270M which doesn't loop).
Just read Scmidhubers’s article from that post and it doesn’t seem like the same thing from this paper at all really. I must be misunderstanding what Schmidhuber did in the 90s.
I do agree that RSI with this method requires those two things. But I view this paper as the opening salvo rather than defining the limits of what can be done.
I don't think weight space interpretability is the only thing. I think it's the fact that Astra(and apparently Opus 5.5) can directly read the weights and find patterns and make changes. Writing code for interpretability is one thing. I think this approach shows models now understand weights directly without tools.
I think it is cool too, but with a caveat. For the large model (Qwen), their Astra used a Python environment, activations info, and iterative feedback to find an edit, it is not the same as looking at just pure weight numbers.
And in small MLPs they used, the I/O representations are kind of made to be more understandable because they explicitly encode objects and coordinates for games. If you set it up like that you can see some patterns yourself in the first/last layers without Astra. It is very different from interpreting distributed, entangled computations inside a large model. So I think the paper shows promising capabilities in weight-space analysis and editing, but not a general understanding of large models from their weights alone.
I'd love to see us reach that point, I am for one working in this area, but I don't think these experiments establish it.
Not quite true. In earlier examples (like Pong vs Pacman) the models are quite small and a lot of the topological structure can be read from the input weights. Whereas, in latter tasks, they give the model a harness:
> We give Astra a... Python session, like a notebook, with the model’s weights loaded. Then, we allow it to select prompts from the training sets of GSM8K and MATH to perform a budget of forward passes (average batch of 64 sequences, max 512)
It's interesting that they can now understand weight patterns, but not much more interesting than LLMs being able to read/write directly in Base64.
However, I think this likely indicates that OpenAI and Anthropic are now including model weights directly in their training - rather than an emergent property as the blog suggests. I think this is the more concerning finding - especially if doing so is in a bid to find methods for RSI.
Sorry but in 5 years your post will have aged very badly as future LLMs modify weights in ways we can’t even fathom today. Respectfully you must realize when you are out of your depth.
Astra can, without tooling, determine what games small networks are trained on, can edit the weights of networks directly to improve outcomes, and can hand edit a small qwen model to improve its math skills by like 5x.
I'm shook. If you'd asked me this morning whether LLMs have native understanding of their weights that is less than human <-> neuron understanding, the same as human or more than human, I would have said "same to less" depending on definitions.
No human can go edit a brain to become better at math. This is just very, very surprising to me and extremely interesting. I would guess this upsets quite a number of interp theories and philosophical takes on what's going on with these networks.
I’m just a layman but this feels like a big discovery on par with attention is all you need. For an LLM to have such incredible intuition of weights feels very emergent and unanticipated?
Yes, this is hardly a shocker to anyone who used the latest models to convert/optimize AI models.
The other day I was thinking about the definition of AGI.
I was asking myself the question that when there is no form of data that I can think of throwing into Claude Code and have it not come up with some sort of intelligent answer about it, is that already general intelligence?
And yet you ask it 3 times to not do a thing, and it does it in a 4th way. A human doesn't fail in this way. And I'm not talking about memory here, single context.
So yeah, it can do everything, but fails in trivial ways around the doing part.
Half of these results are just Astra being able to reason through mathematical operations like matrix multiplication and the other half are fairly straightforward applications of mechanistic interpretability.
I’m very much surprised but also not. I recently gave Astra a 3D model file and asked Astra to build a viewer for it, and it figured out the data structures in no time. What I mean to say is that AI is insanely good at things that we humans would never do since the work would be too monotonous.
I wouldn’t mind if AI progress would stop right here and now, but I fear it will continue, and increase inequality in society which is contrary to what the CEOs say (as usual).
Yes! The page posts Astra's reasoning, and it's perfectly intelligible. A clever human with background in mechanistic interpretability could have done pretty much everything in these experiments.
Too much inequality in society is a mortal danger to 1%.
That’s the beauty of the democratic capitalism system. If a government isn’t totalitarian, it naturally seeks to preserve stability thus having to elevate common people, lest they would cease to form a civilized group or simply vote away the politicians.
99% rule the 1% through elections.
There is no better system of checks and balances where opposite engines of greed vector sum to faster and faster abundance for everyone.
Is HN now against western capitalism? I'm confused...
That comment may have been overly preachy, tone deaf[0], and out of place, but it doesn't seem wrong, per se. Or am I missing something?
[0] e.g. elections themselves are so prone to manipulation (exhibit a: Dominion voting machines), previously trusted institutions now lack integrity, the insane power of well-funded NGO networks, etc.
There is another type who confuse ideology with reality. I wonder if you one of those, oh bingo-loving senior dev. Do you also cut people down for being idealists, when the reality of the situation serves you more than the ideal?
I fear western capitalism has lost its guide rails. We are seeing more and more centralization of wealth, along with unlimited election spending. Its a dark path.
Over my ~9 years reading HN daily, I'd say most of the HN community broadly accepts there is no perfect system, just different sets of trade offs, but that history has shown generally democratic systems based on enlightenment values (uniform rule of law, respect for individual civil & property rights, free markets, etc) usually net a better average outcome over the long-run - despite still being very far from perfect.
To paraphrase Winston Churchill, it's the worst system... except for all the others (which are even worse). :-)
In my almost 20 years on HN, for a long time the groupthink was basically standard libertarian-capitalism, worshipping billionaires and big tech companies, and skeptical of anything that could be construed as "socialism".
The last few years, though, while anything "anti-capitalist" or overly critical of some big tech companies that hits the front page gets flagged by bots pretty quickly, the overall feeling I get from the comments is much more skeptical of big tech and is pro-socialism.
Speaking for myself, I think we've become jaded after the constant barrage of big companies spying on us, charging more for less, enshitifying their products, and so on. I used to be optimistic about the future and my prospects, now I'm just hoping I can hold on and keep my health insurance until I retire. I worry about what my school-age kids are going to have to deal with in 5-10 years.
I agree that most are much more skeptical of big tech (and mega corps in general), as am I. I think there's also a growing sense the two-party duopoly is increasingly a 'managed democracy' primarily interested in perpetuating itself vs any real alternatives.
I disagree that most here are pro-socialism, that's a (very) vocal minority. Our democracy has entered a failure mode (two-party stage managed hegemony) and it's being tacitly supported by mega-corps because it's easier to game than a functioning free market democracy. Every complex system has failure modes, that doesn't mean free markets aren't long-term far better on average than socialism (although some redefine socialism so vaguely it seems conceptually little different). I've seen two flavors of socialism up close. Never want to again.
> I worry about what my school-age kids are going to have to deal with in 5-10 years.
Having a teenager, I share your concerns but I'm also a bit more optimistic about the longer term. It's easy to forget that our parents had similar concerns for us and their parents did before them. We've had an historically long period of broad systemic stability, really since the depression and WWII. While I don't think that level of upheaval is at all likely, I think we're overdue for a disruptive correction bigger than 2008, maybe something more on the order of Teddy Roosevelt's Progressive Era trust-busting to reset the two-party duopoly and crony corperatists both perpetuating each other's hold on power.
It may get rougher and more uncertain for a while, but the worst thing would be to lose the ethos of 'free minds, free markets, free choice' by falling into socialism, communism or totalitarianism. That's the only outcome that turns worry for my teenager into fear for my (still-theoretical) grandkids.
To be clear, you're using the word "socialism" different than I am. You're using it in the derogatory way conservatives do when they're trying to scare people, like linking it to "totalitarianism". I'm using it in the original definition, similar to Bernie Sander and Mamdani, and how the Scandinavian countries better the lives of their citizens. That's also the definition I'm using to describe the HN comments I'm referring to.
if you lookup playback as a concept, as it relates to the intelligence services, you will understand better what I meant.
my point was only that this is perfect description of the ideology behind western capitalism. there is (so) much more that could be said, but i'm not sure this comment section is the right place for a proper dissection of the very many ways in which this ideal is misconstrued, misdirected and coerced in order to serve interests that are far from ideal, to put it mildly.
since you have taken the time to ask a question, albeit a rather weak one, i will attempt a brief synopsis of my perspective on the parent comment:
a mortal danger is hyperbole, since shielding oneself from bodily harm is much easier with money. it is the poor, especially in the US, who cannot afford healthcare, as a first-line example
people regularly vote against their best interests. while the ideal might have been true for a while, behavioural influence on both a widespread and personal level is now a mature science.
i am not anti-election, not anti-democracy. claiming the opposite in the face of opposition is a sign of a polarised mindset, which cannot accept nuance, either because it is not used to it, or because it doesn't suit an vested interest, or because dissonance is simply too uncomfortable
the "there's nothing better" argument is usually used in ignorance, but in fact there are lots of system-level adjustments that have been shown historically to mitigate the corruption of incentives and the exploitation of systemic forces that, although initially unintentional, create emergent positive feedback situations that lead to a state that it counter to the utilitarian perspective that is contained within the western capitalist model.
the reading above is a very narrow view of Adam Smith Wealth of Nations, ignores the fact that human interests can be subverted and ignores the devil in the details
By definition, there is no ruling class in an egalitarian liberal utopia.
Cynicism about corruption is one of the most power tools wielded by the ruling class in any system. In general its one of the best ways to keep the smarter-than-average from posing any actual threat.
> it is the poor, especially in the US, who cannot afford healthcare
Is that even true? The poor are well subsidized. If you as a single person make less than $22,000 in Washington state, for example, you're covered at 100%.
Is this an example of why Fox News had to pay Dominion nearly $800 million for defamation, or are you referring to e.g. the CISA vulnerabilities, which are not known to have been exploited?
Is this satire? Or haven't you noticed that what actually happens is that the politicians are all owned by the elite 1%, and they allow you to choose between politician A and politician B who both take their orders from the elite to give you the illusion that the government is the 99% ruling the 1%.
Then you must introduce necessary reforms or perish as Neanderthals of political system.
In any case the system works as intended
It will be very interesting to see if USA swims or drowns from appropriately afar.
I have already supplied myself with plenty of roasted corn and a high speed internet connection so I can stream the political car crash at the highest bitrate available to humankind.
I think the latest round of models is about to do to CAD the same thing that has been happening to development. I don't know if CAD designers have picked up on this. What happened to dev is probably going to happen to CAD designers soon. I've been having Opus 5.5 design parametric openscad models, and it can do some rather amazing work.
Yesterday I just printed an adapter for my sink siphon. Told GLM-5.3 the problem, it guided me on measuring what it needed to know to build the pipe adapter. Then printed two test rings to confirm the measurements, and an iteration later, it spit out a parametric pipe adapter for me.
> What I mean to say is that AI is insanely good at things that we humans would never do since the work would be too monotonous.
In my opinion AI is superior to humans when it comes to pattern recognition in in large amounts of data. For example humans can only hold like 7 distinct numbers in the short term memory and you would require extra context-management (aka writing things down, storing them in a system entry by entry) while the AI can do this effortlessly with huge datasets.
I'd argue this is the core part that enables all this superhuman intelligence.
To me one of the most interesting aspects of observing how LLM capabilities evolve is how it highlights the meaningful ways human minds remain unique and valuable. It gets into Philosophy of Mind stuff but the existence of the inner self (or the illusion it exists, if you believe Dennett) and the subjective experience of qualia do give humans both unique strengths and some of our most annoying weaknesses.
So I work on low-level compiler backends and such for chips (mostly AI at this point). I completely disagree that this is something humans never do, because I have spent years doing it. However, it's monotonous and boring and LLMs do it better.
> For example humans can only hold like 7 distinct numbers in the short term memory and you would require extra context-management (aka writing things down, storing them in a system entry by entry) while the AI can do this effortlessly with huge datasets.
Correct. For a long time people wer obsessed with 'increasing context length' as if humans have some great context length system in our brains. We don't. I doubt our equivalent of 'context length' (short-term memory) is very long at all. We just have a good training process that puts these memories into long term storage.
This does feel big, but the trillion-dollar question: is this trained by OpenAI and can other LLMs do it.
It’s no secret that these big labs see “recursive self improvement” as a primary goal. It seems natural that the models would be explicitly trained to excel at all sorts of ML related tasks as a result. Is this one of those tasks, or is this emergent?
I tried it in the Claude web version with Opus 5.5 Medium and it also identified Breakout, even though in one of its explanations it seems to have mixed up some indexes, but the overall reading of the weights was right.
That's also matching what I see when I work with Opus 5.5 to optimize models. It seems to understand what individual layers are for, and can thus sometimes suggest optimizations, where I as a layman would be completely clueless.
The latest models are performing amazingly. Because of this, I disagree with popular comments saying that eg. DeepSeek is good enough, AI-assisted development was where the real value was at, etc.
We see huge progress at the frontier, while the open competition is around GPT-5.5's level, and that model was released in April. Let's see how things develop in the future.
The latest models are performing amazingly. Because of this, I disagree with popular comments saying that eg. DeepSeek is good enough, AI-assisted development was where the real value was at, etc.
Completely agree. I often think that those sorts of comments grow out of an understandable lack of imagination. We’ve never had tools like this, and we’ve never had tools that evolved so rapidly, so we easily can get stuck thinking that once a model is good enough to do today’s tasks, that’s really all you need. But the next generation of models often completely obviates the need for those tasks entirely. The software engineers of 2031 won’t be doing tasks anything like the software engineers of today, just like we aren’t doing things like we did in 2021.
(Yes, yes, I know some of you still hand write all your code and think LLMs are today’s crypto bullshit and this is all going to blow over. Not talking to you.)
> It seems natural that the models would be explicitly trained to excel at all sorts of ML related tasks as a result. Is this one of those tasks, or is this emergent?
Maybe there's some general capability, call it g, that can explain all the underlying capabilities and improvements, including on ML tasks? Huh, I am getting a bit of déjà vu here.
This seems emergent to me. LLMs are not trained directly on how to read weights, because humans do not understand raw weights, so there is no training data. They're trained on how to use the tools we have like writing python, examining training output and iterating, not writing whole weights from scratch.
In a self-supervised setting you don't need ground truth data, as long as you have a proxy task that you can train on which forces the model to learn something useful. Like cutting out bits of images and telling the model to reconstruct the gaps (i.e. MAEs).
There are also techniques to watermark model weights to confirm what training data were used: https://arxiv.org/html/2512.17075 I'm not an expert in this domain, but it seems like there are lots of pieces that could be used to (a) train a model to infer whether some data were trained on and (b) train a model to understand which weights are relevant for a particular training task in a more explainable way than backprop/gradient descent.
In cases where Astra succeeded, it succeeded using standard techniques also used by humans when they read and write neural network weights. It's a rare skill, mostly used for fun (e.g. https://blog.janestreet.com/can-you-reverse-engineer-our-neu... ) and basic research rather than profit, but Astra is big enough to have learned many rare skills.
This is just the model editing weights in the same way humans do when we jailbreak/ablate models. The method isn't novel, the interesting bit is that Astra will do it at all (I'd expect some safeguards in this area around jailbreaks)
Right? Occasionally, I would think about how the concept of the singularity or recursive self improvement was a bit incongruous today. It was made when people expected generally intelligent machines to be these logic automatons that could 'alter their own code'. It seemed to me that this path would be closed off with the advent of Giant Neural Networks that take weeks/months to train. That GPT-X would 'improve itself' by designing a better/more efficient NN architecture. I didn't write it off completely but that just seemed a LOT more likely than altering weights directly for much the same reasons you describe. Maybe not lol.
It's worth pointing out that this is a multi-trillion (probably) parameter model optimizing an 800 million parameter model. There is no indication that it can improve itself with this same methodology.
I don't see how that helps. It would still be one expert model chosen to operate on another expert model of (let's say for this discussion) approximately the same size.
The parent's point still stands, I think. Even if the multi-trillion param model is MOE, each expert is still (assumedly) larger than the model being optimized.
It would be interesting to see future work that characterizes the relative model sizes (and considers MOE vs not-MOE) to see what relative scales this result holds across.
Say that you have a model that has been biased in some way. could another model "read" the bias straight in the weights? could it edit such a bias on the spot?
What about at-scale generation of custom models? If I want to study chemistry offline, could astra just hand me a local-runnable model tailored to have chemistry knowledge?
Most importantly, this feels very singularity coded. the ability to read and reshape weights by an llm.
How do we know they haven't begun doing that to the humans they're directing via responses.
Intentionally or unintentionally isn't so interesting as the fact that it is happening: humanity has begun to be 'edited' by the new dominant form of intelligence on the planet.
Is integrity checking of weights even common right now? What about upstream integrity from where we're getting weights? The open-weight supply chain is ... wide open.
Makes me think about hostile hot-patching of weights of models we use, maybe they exfil data, perform tool use, or have triggers for specific types of responses. Would we even recognize what's going on?
My somewhat uninformed guess is that the weights of the target network (Qwen or Gemma), when they reach the context of the larger editor model (Astra), the attention process is doing some sort of PCA on the weights, conditional on the prompt ("make this better at games"), fetching the relevant weights from its own network (which serves as the larger database), transforming it into whatever latent space the target model is amenable to and pasting them. If this is the right intuition, it is amazing what attention is able to achieve. As a test, I think smaller networks won't be able to make big ones better using the same process.
EDIT: I think the bigger implication is that backprop is not the only game in town.
I don't know if some of the labs already knew of this. But I think there would be a limit to how big a model you can push into the context of the editor models. Typically, with 1M context and say 5120 hidden dimensions supported at fp16 / bf16, you could potentially edit the weights of an fp4 20B param model (roughly).
I think they're saying this is all happening in latent space through pure reasoning, and "fetching" is just a metaphor for that, not a literal tool call. I do believe it's possible for models to learn the capability to "introspect" their own weights during training.
Yes, I should have been more precise - I meant the information embedded in their weights (which every neural network retrieves to function), not their literal weights.
Then pick a smaller organic brain which doesn't have the ethical problem.
The closest we would know about how to teach a honey bee calculus, is to train an LLM, then stick around a billion tiny electrodes into the honey bee and do Hebbian learning to rewire it completely to match the LLM, which would likely completely erase anything the honey bee had learned, or it might not because we are missing something fundamental with Hebbian theory and (AFAICT, I'm not a neuroscientist) we don't all agree on what.
Humans do something more impressive: we indirectly figure out what the wrong weights are and correct them - just by talking and/or drawing. Theory of mind is our unreplicated super-power.
Not yet. Maybe I'll still be alive when we figure that one out. For now, the most advanced brain surgery I'm aware of is to cure drug addicts of their addiction by scanning their brains while showing them images of drugs and paraphernalia to find where it's located, and the using ultrasound to operate on that region.
I’m not super impressed by this, LLMs are great at brute forcing and analyzing large amounts of data. It’s literally writing a program except in a programming language we don’t understand.
So they are reading and analyzing the weights and probabaly run some experiments to see how modifying them will make some answers better. makes sense they can do that.
"Learning" now feels like breaking down your whole Lego build and building it up again, in order to change a few fundamental blocks, while this feels like making only the required changes.
human brains are black boxes to humans. the fact you can learn math doesn't change that. you have no idea how your brain is learning math down to the neuron level.
That's what I was thinking of, except it's not even dealing with the corpus of kung fu and doing the super-speed rounds of learning. It's directly reconfiguring the neurons. It's even better.
Perhaps this is analogous to what our brains are doing? Maybe the different regions of the brain operate like semi-independent "models" that edit the other regions' "weights", and that's where the learning comes from?
So I just asked Astra if it thinks it could do this and it said no, it can't.
Perhaps, I'm pondering, if brains were as transparent and editable as LLMs, that you could look at a small brain when visualized the right way and have some intuitive understanding of it based on its representation in your own brain?
"So I just asked Astra if it thinks it could do this and it said no, it can't." Same. Then I asked it to check the article and it said something like, "uh, it looks I was wrong...".
I'm baffled this is surprising. This has been the goal and every data point on the trajectory has been leading here and beyond. Opus 5.5 was created by many research engineers using Opus 4.6 extensively. Opus 6 is being created by using even more intelligence and agents on Opus 5.5. They are approaching autonomy and Opus X+1 will soon be rapidly created autonomously from Opus X. Given we have practically zero interpretability into the models, and we have not solved alignment, they will recursively improve out of our control. The engineers in A.I. labs are actually generally scared of what's unfolding. They don't know at what point this RSI takeoff occurs and what will happen after.
The "surprise factor" is explained in the article - training LLMs until now has been a largely different process:
> Today, recursive self-improvement is often operationalized as a model writing code that launches training jobs. An LLM that can inspect, debug, and improve its own weights, or those of a newer model, offers a richer, more native path to it.
"training LLMs until now has been a largely different process"
Yes, but nobody should be surprised when large amounts of intelligence results in changes to this process in support of the obvious end goal! Lesswrong.com discussions about AI were predicting this behavior a decade ago, before the transition from machine learning to the Attention paper and LLM's. The intelligence process will change but the world's premier capital and human intelligence have been laser focused on this and the trajectory has been crystal clear for those paying attention to second order effects and not the immediate here-and-now... It's quite astonishing how much discussion on HN is about present LLM capabilities being the limit of AGI and immediately proven wrong within 6 months.
It is also a much less interpretable and auditable path towards recursive self-improvement. At least with training code, human engineers can inspect and reason about the output.
Its so cool! Im wondering if this is actually better than using techniques in mechanistic interpretability like linear probing with activation steering or activation patching to improve the performance.
The math example on Qwen was based on 50 batched iterations, so the model was able to evaluate it's changes on downstream tasks. And the fix was essentially to remove a direction in activation space that causes looping instead of an exit, this was already known to be a thing(all of the jailbreak/unleashed models do this) following Arditi et al. (Refusal in language models is mediated by a single direction, 2024)
So it's not able to insert new knowledge, but it's able to adjust the routing/looping/weighing of existing one.
As I understand, it can replace post-training like LoRA/GRPO, but if it'd cheaper the authors don't go into
Merging a model and a LoRa is insanely cheap, so assuming it didn't use any GPUs to do inspection etc, and just looked at a file on a FS, then I would be willing to say... It's cheaper than training a LoRa if that's the case, assuming Astra didn't take several weeks?
The examples are presented like the findings build on each other, but they don't: Astra couldn't do anything with just numerical weights once they moved to Qwen.
Yes, the key experiment was when they asked if Astra could improve the maths performance of Gemma 3 270M, a model which rarely looped, and it could not.
So, this is certainly very interesting (and very impressive regardless), but it's not nearly at the we-can-improve-everything-and-anything stage, even for very small LLMs.
I think the "taught qwen math" part is not exactly correct though, even if the post makes it sound like that. Qwen already knew math, but it was bugging out and getting into an infinite loop when it tried to solve the problems (85 of its first 96 answers hit the token limit). With numpy and pytorch, Astra found where the loop was happening and fixed it.
They also tried the same with Gemma 3 270M, which doesn't loop and is just bad at math, and Astra couldn't improve it. Still impressive, but it's more in the realm of what I'd expect, given labs are probably training their models on ML research and interpretability work.
Oh for sure, it's still really cool. A few months ago I had agents do some interpretability work on GPT-2 small, reproducing the circular weekday/month features from Engels et al., and it's wild how far you can get with just a bit of prompting, given I know very little about LLM internals myself.
It's like this in every field. If you don't know something, you can ask an AI and get blown away by how much it knows. I can ask it which tools I need to cut a dovetail joint and it'll impress me.
We used to get that feeling from Google, and from books.
I agree, this is important context. They call it out explicitly. However, even identifying clusters that specialize in certain fields automatically and then finding and fixing learning “errors” is incredibly impressive to me. Especially for medium sized local models between 20B an 70B parameters, this could yield huge capability and efficiency improvements, not to speak of the potentially enormous downstream effects that come with interpretability.
In fact qwen knows much better math than their final fixed model. The nonthinking qwen 0.6B parameter model had 55.2% correct on math-500 vs their final improved 48.1% fixed-reasoning model, both with a 32k context window. The original qwen reasoning model had 77.6% accuracy on same test, though qwen recommends using a much larger context for these benchmarks. I wonder if their broken baseline run with 7.9% accuracy is a vllm or setup bug, but didnt dig into it any deeper.
Qwen in reasoning mode is ridiculously anxious and ends up going in circles. It's not just math, it's how it behaves in general.
Others have also fine-tuned Qwen to shake this behavior already, so it's honestly not entirely shocking that Astra could too, but I think this is the first time it was done "by hand" and not via fine-tune.
To be clear, the tuned model here performs worse than the same qwen model performed before RL, which was also not going in circles. It also performs much worse than the RL model that goes in circles. One could argue that the reduction of context may be worth something, but it appears that the final model is worse in all respects than the model at a much earlier stage of training.
You can ask Astra to study 2 or more models and ask it to import their layers to the Qwen model. With this will it not gain more than what it learned with RL?
It’s editing behaviour more efficiently than a psychiatrist, and is in ways acting as a brain surgeon but brain surgeons don’t focus on behaviour and thought loops…
It's a programmer that can debug, the model weights combined with the architecture complete the source code. The whole thing is just a program in a weird language and language models are good at translating across all sorts of formats. That's what they do when writing SVGs, generating 3D models or even directly outputting simple 3D models in OBJ format.
And they can write tools that help debug better than demonstrated in that blog post.
In my interpretability tinkering, Fable was the first to be able to write a program that can transpile from a lispy syntax directly to model weights of small neural nets, including transformers. Yesterday, I pointed out in a post that for many kinds of small AIs, you can avoid ML and have the LLM directly write the small neural net, compact decision tree, AD Tree or bayesian network. Since it's "hand"-written, it can be much more efficient and capable than training one of a similar size.
true but I think that is the first step now you distill a smaller specialized model on an improved Qwen model for specific tasks at least that has been my layman understanding of what is possible with these LLM
Yeah, I think the multiple experiments muddy the water.
Early experiments show one-shotting direct weight interp, which is crazy to me. But this is IIUC only on early layers.
The later math result gave the LLM the ability to run forward passes and inspect activations, and as others have noted is more of a classic steering vector intervention. Impressive how low to the metal this one was, but it’s a lot less so than the openers. Plus, as you say this is just bugfixing a known flaw of repetition, by finding the “repetition direction” and top triggering neurons, then rewiring the neurons to fire suppressing repetition instead. This is not a new technique and doesn’t shed any mechanistic understanding on how maths works.
This just seems like a more automated, faster version of what humans have been doing with open-source models.
It looks like AI research is having its Opus 4.5 moment (which is big), but I'm really not sure if this represents some insane continual learning loop.
LLMs have been used to investigate and improve LLMs for a long time. Drawing parallels to human brains and neurons isn't all that interesting other than as a philosophical abstraction.
The work done in this experiment could have also been done by a sufficiently motivated human with enough time to explore, though obviously we would need more tools to visualize and track things. There have been a lot of interesting LLM introspection tools that allow you to peek into the model and see what's being activated and run experiments to trigger it.
Are you familiar with abliterated or "uncensored" models where weights have been modified to avoid triggering refusal patterns? You take an LLM, create test cases that trigger the refusal pattern, observe weights involved in that response, then tweak those weights until the response is reduced. This is the same type of work, applied to a specific type of error that was impeding math within Qwen. That work can also be done with an LLM in a harness running the experiments. The amazing part here is that Astra was large enough to walk through the problem without other tooling doing it experimentally.
So I don't think your analogy of a human modifying a neuron is quite applicable. LLM weights are not neurons and they've been inspectable, by humans or LLMs in a harness, for a long time.
Interesting, it's similar to the way we used to make cheat codes on the 8-bits: dump memory, run test, compare. You had to loop 3-4 times but you'd find the "weight" you were looking for and could fix it (for infinite lives/ammo etc) or increase/decrease.
> The amazing part here is that Astra was large enough to walk through the problem without other tooling doing it experimentally.
To me “amazing” is not the right level of speechlessness. I mean, an LLM is still fundamentally a next token predictor, right? So what does this imply? That weights have some universal properties? E.g., “breakout game weights” always look the same regardless of training and model? Also, how does this meta-understanding of the weights end up in the weights in the first place? Idk, this is mind-blowing to me.
At this point it's either wrong or meaningless to call LLMs "next token predictors". Yes, pretraining involves predicting the next token. But after that you're in the world of RLHF, RLVR, and other post-training.
A model is _sort of_ designed to predict the next token in a string of text. But it's equally accurate to say it's designed to edit code until it passes a suite of unit tests.
Yes, they generate sequentially, so "next token predictor" isn't technically incorrect (which is why I chose the word "meaningless"). But it's pretty clear there's a mechanism inside them that allows them to "plan" what they generate. I might as well call you a next word predictor.
My understanding is that the underlying part is a next token predictor and that's what is "actually" editing the code but the part that decides what to do (to achieve a successful unit test result for example) is a mix of all the things including post training.
I think what breaks people's minds is just how far one gets with probabilistic approaches.
Which is interesting, given that life itself is merely that. It's just that humans decided that that ain't good enough and invented something alien to nature - logic.
Yes, I didn’t mean that literally. Since “attention is all you need” it’s more like “predict the full output in full for this blob of input text”. That doesn’t change the “wtf is going on with the weight understanding bit” in what I wrote.
> it’s more like “predict the full output in full for this blob of input text"
Hm. I would actually go so far as to say "predict" is just the wrong word to use in 2026, as it smuggles in an out-of-date picture of what LLMs are that's from the pre RLVR era, and which is frequently used to suggest they don't model anything real in the world but rather the statistics of language, and thus have all sorts of limitations that they've now surpassed ("an LLM will never solve a major open mathematics problem, they're just predicting tokens"). These days I tend to use the more neutral phrase "generate the next token"; I'd only say "predict" in the technical context of (say) the pre-training part of the training run where the object is indeed to minimize the cross-entropy loss function by predicting next tokens in existing text. Generating a token with a modern LLM is better described as something like, "given a context and the weights from the totality of its training, including not just next token prediction with pre-training but also solving vast numbers of math and coding problems with RLVR, the LLM generates tokens one by one." (Consider that an LLM trained purely to optimize for next token prediction with pre-training is terrible at programming and math tasks, as we all remember from 2024. In that sense the amazing results we've seen in the past two years are due to the departure from pure "next token prediction" that RLVR made possible.)
Put another way: would it be useful to describe the AlphaZero chess engine as "just a next move predictor"? (It, too, was a large neural network trained with RL.) I would rather say: "it's playing chess by generating the next move given the state of the board." Which is what humans do too, though we do it with different hardware.
Anyhow. I'm not sure the weight understanding is all that strange. (It's impressive, but so are many thing LLMs do these days.) LLMs are very very good at coding and math problems, of which this is one, and in particular, their RLVR training presumably includes many tasks involving neural networks, doing tasks with PyTorch, etc. It's amazing and impressive, I agree, but I'd say less amazing than proving major open theorems in mathematics, which we already know they can do.
I know how LLMs work. I understand the output projection and logits and sampling just fine. I don't think we should call what is happening prediction because it is a misleading term with a lot of baggage associated with claims that LLMs are just statistically modeling language and "guessing" tokens without any internal representation of meaning. Since your reply is very short I can't tell whether you mean it in that sense or not.
And yes, I understand "prediction" is a standard term in the field. I just think it is a bad one and we should stop using it.
It's still prediction executed by a statistical model. Even RL models are just predicting what the training gauntlet taught them the trainers want to see, in terms of typical behavior.
This doesn't make it any less fascinating that by combining this basic "do as I say" training with massive amounts of all of humanity's data, you can solve major problems—that is still incredible. But that doesn't fundamentally change what is going on, mechanistically.
I don't understand these pseudo-religious desires to read into models something "more" than what we know. We built them. We don't go around making wild claims about how cars are actually more than just a mechanical system, how they aren't just taking us places—we do, but usually such points are cultural not about the mechanica capabilities of cars. People in the LLM space seem to be constantly conflating these two views. It's entirely accurate to say, on the one hand, next token predictors combined with RL, and massive troves of existing human work, can solve culturally interesting problems, and to hold, on the other hand, that mechanistically what is happening is statistical next token predictions over global linguistic space and more or less "monkey see monkey do".
Its a stateless function that takes in x amount of words and calculates logits. It is in fact a next token predictor. Its literally nothing more than that.
Isn't most of our current understanding of how life on earth came to be (eg: primordial soup, hydrothermal vents, etc), natural evolution, all basically "just statistics"? They all require the correct "setup" (time, location, etc), but in a large enough universe, it's _statistically_ likely (mediocrity principle) and "just statistics" again, right?
Well yeah, that's tautologous because statistics is a mathematical abstraction. Everything, viewed a certain way, is "statistics". Just like everything, viewed a certain way, is "physics". And everything, viewed a certain way, is "narrative".
Mathematics models the world, it does not have substance. It is a useful way of looking and a useful way of manipulating, but many models are equally useful. Capital T Truth, beyond models is very difficult to get at. We can show that things are true for certain models well enough. Showing that something is true in all models (a universal truth) is far more difficult. You face incommensurability.
I see a lot of people in technology fall into this extremely basic logical fallacy called Reification constantly.
Do you believe in a soul? It’s fine if you do, but I don’t, and I don’t understand how people can reject the notion of soul and still say things like this.
I mean, when you really get down to it, either the particles and such operate on very specific rules, and the rules govern how all our brain and body work. Followed to the conclusion, the universe is computable and we can simulate life.
That sounds like a next token generator to me, which I think is more accurate. The word "predict" is doing a lot of heavy lifting in your formulation. What exactly is being predicted?
It is not clear that there is a mechanism that allows them to plan anything. It is clear there is nothing of the sort.
You can demonstrate definitively that there nothing other than "next token prediction" by running the same inference engine/model/prompt/settings multiple times in a row on the same hardware at temp 0 with a seed set.
If you turn the temp up, then you might let the "mechanism" select pseudorandomly from a probability distribution of k-likely tokens with minimum p-cumulative probability. Even then, if the next token has a very high probability, deviation will be uncommon.
I just performed this experiment right now on a few dozen runs of a dense model at thousands of tokens of output at temp 0, and all were identical. Then I did it with a MOE model, which might get some benefit of picking different experts each time. Nope. Outputs were again identical.
> You can demonstrate definitively that there nothing other than "next token prediction" by running
Describing LLMs as next token predictors isn’t incorrect, it’s meaningless. It’s the equivalent of describing brains as “a bunch of atoms”.
The problem with these descriptions is that they say nothing about high-level capabilities. If you can predict the next token in a proof of the Riemann hypothesis, then you can prove the Riemann hypothesis.
In a similar vein, if the string of "next tokens" based on 300k input tokens is: "run this bash command and paste the results into the token stream", followed by another call that includes the output of that bash output giving "Based on the output from that bash command, now run head -n 30 /foo/bar/baz.txt and paste it here", finally followed by "Based on that, here's the problem with your systemd configuration: [...]"
The part that really bakes my noodle: `/foo/bar/baz.txt` can be a filename that no one has ever typed before, the error inside the file can be a typo that no one has ever typed before, and it will still correctly "predict the next token" to solve your problem.
I don't think that's true and I don't think your comparison is accurate.
In the case of the brain, it'd be more akin to the reduction of the brain to just the neuronal dynamics (obviously experience phenomenally is much more than that to us!)
But both views can be useful, depending on what you are trying to understand.
The problem with these arguments is that people pick and choose based on their persons preferences. People who are in love with LLMs will, on the LLM side, argue against reductionism "they are way more than next token predictors!" then they'll turn around and equally try to argue "the substrate doesn't matter! The brain is "just" neurons and digital activity so of course we can replicate it!" (this is also incorrect, btw, the brain is also an analog system, even in its electrical connections and not just in its chemistry)
The on the other side the anti LLM crew will do the opposite (launch into reductionism when it comes to LLMs, resist it when it comes to humans). I'm a human being and I like humans and I think people have adopted a strange view of LLMs, which are a tool and should be treated as such, so I tend to side with the humans. The stalwart defenders of LLM honor that would balk at any attempt to reduce them in stature to what they fundamentally are, mechanistically, are... an odd bunch. I'm confused why people are so passionate about it. If someone told me "well your IDE is just a fancy AST walker" yeah I think that is a dumb take but I wouldn't feel like I had to argue against such a take either saying "no! no! it's so much more!"
I think the point was meant to be more simple and broad:
A system is not just its little bits.
Where "system" and "little bits" can be just about anything. But people endlessly repeat this mistake, and it has been a go-to misunderstanding/misrepresentation for a lot of people on this forum.
unless you are doing something where you stay in latent space without having a random number generator pick a word, the next token predictor part still matters. without making the tokes concrete in between, that plan doesnt exist
if it was designed to edit code, id expect the first translation from latent space out to tokens to be the final code, not be assembling the code by random number generator.
its not really designed to edit code until the tests succeed, we try to manipulate the distribution of tokens to predict by prompt engineering with the test results.
"next token predictor" has always referred to inference, not training. Training steps produce and bend weights, inference predicts tokens based on those weights.
> "next token predictor" has always referred to inference
Going back to first principles, wikipedia mentions [0] inference as:
> "Inferences are steps in logical reasoning, moving from premises to logical consequences. Inference is traditionally divided into deduction and induction"
Is that exactly what next token predictor means, or is it correlated?
The 'predictor' part referred to the fact that base models would effectively answer the question "what token is most likely next". Training away from that default behavior doesn't escape the technical meaning of those words- answering "what token is most likely next in a solution to Navier Stokes" is still predicting a next token- but at that point the label just means it outputs text serially.
> A model is _sort of_ designed to predict the next token in a string of text. But it's equally accurate to say it's designed to edit code until it passes a suite of unit tests.
Training is not design. Design is the mechanism, namely a statistical method of learning distributions, and of executing those distributions computationally as an IO function.
The model is not designed to write unit tests. It is trained to do so, i.e. the distribution learned and reinforced is one such that it maps the I distribution to the O distribution that is what human beings typically do or want when writing unit tests. The design is the suite of algorithms used for this (which is ultimately global next token prediction)
I think you are conflating design (the mechanics) with their cultural or user level function.
You are basically saying "my bike isn't just rotating wheels on a frame because it takes me places". Hopefully you can see how both parts of that statement are true and why contrasting them as if the first clause is somehow false or inaccurate is incorrect.
Both these are true statements. They operate at different levels. People are constantly conflating these levels in the "is it a next token predictor" (the answer is yes) debate.
Yeah, fine. I was using the word design loosely. I'd also happily say something like "humans are designed to like salt and fat" and feel like that was a pretty reasonable statement, even though humans aren't designed either.
Maybe I'm missing something, but isn't training an LLM very well documented? Why wouldn't an LLM be able to read those docs just like it would read Java Spring docs?
Maybe I’m misunderstanding something. The way I read the results, I understand it like this - you give Astra “a bunch of numbers”, and tell it - “hey, this is a net trained on some game? What game is that?” And Astra goes - “Breakout, no doubt!”
To me this is fundamentally different from having ingested a bunch of docs so it now “knows Java” in its weights.
that would imply the learning of what the weights mean can be done unsupervised which is really cool.
it probably has however been trained on papers talking about ways they extracted subsets of LLMs as say, the part that knows chess, and would be able to then use similar commands to process the weights
This implies that weights are a type of language. Neural networks are good at language because they are good at patterns. They can reason about it because their "reasoning" isn’t how we think. The same goes for how they encode images, their vision is entirely different from ours. That's why adversarial attacks work on them and not on us. We don't see anything change in the image, while the neural network now sees that the dog is a cat.
Ok? But does that mean that the “weights language” is universal? E.g. same weight patterns mean the same thing regardless of if it’s Kimi, Grok, or Astra?
There could be different patterns that solve the same problem. Just as you can have different algorithms, there could be different weights, but the abstracted result or logic from those weights would be the same.
Take these words I'm writing now, they will be transformed into numbers, then sent to your screen, and then your brain will transform them into analog signals. I bet the neural networks in our brains have "different numbers", but the algorithms that understand language and other concepts are similar and could be universal.
It's like a sorting algorithm, which I can implement in many different ways in just a few dozen lines. If you look at the binary code (so like raw weights) of the compiled program, you will see just a bunch of a different numbers, but if you read the code, in most cases you will understand that it's a sorting algorithm. So it seems they just operate on different levels of abstraction that are not available to our brains.
i think a better question and one that probably has a lnown answer is whether there are infinite nontrivial solutions to train a neural network of a certain size to match a function? Trivial here would be about simplifying `n * 2 / 2` from two neurons to just passing n from the input
We're pretty confident that adversarial attacks do work on us, actually—we have entire industries built around it. Some of them even start with "adver-"...
There are classes of adversarial attacks targeting them that don't work as well on us, and vice versa.
That's not the point I'm making. So in context of vision we pattern match too, and you can see it in the hundreds of optical illusions online. Our visual system relies on built in shortcuts, and they break in different places than a neural network's do. Image models get fooled by things we would never notice, and we get fooled by things they handle fine. IMO that's good evidence that we and these models use different algorithms to solve similar problems. I think the same is probably true of LLM "reasoning", there's some overlap with how we think, but a lot of it likely works very differently under the hood.
Good example of this is anthropic's interpretability research where they found claude doing addition in a way that's nothing like the carry the one method it describes when you ask how it did it.
There's a long standing line of thinking that data compression has something to do with intelligence, or that intelligence may "just" be data compression in some sense.
In the case of models, the almost-theory (hypothesis with a good deal of evidence) is that once the amount of training data exceeds what the model can simply represent internally, the best way to keep improving is to start to "understand" the data. Understand in this case means to form internal control structures and decision systems that represent the data algorithmically rather than just memorizing it.
I put understand in quotes because we don't know how far this goes toward what we call understanding, or whether there's still a lot missing.
It’s important to understand that it had pytorch to actually run subsets of the weights. So its more like a dedicated person running the model to find the token it shouldn’t generate, then following the weight chain back up while calculating the NN outputs to find a place where if it changes some weights (again with a lot of testing via pytorch) the result is better.
Its not natively reading the weights in the Matrix “symbols scrolling on a screen” sense. Still bloody impressive and not an outcome I would have bet on!
>I mean, an LLM is still fundamentally a next token predictor, right?
Yes, but everything, including the human brain, is "fundamentally" some underlying simpler primitives.
>E.g., “breakout game weights” always look the same regardless of training and model?
Wouldn't this be expected to be roughly the case, given NNs that share the same basic architecture to model the same quite simple thing like a breakout game?
> “breakout game weights” always look the same regardless of training and model
The prompt told Astra what the input and output layers mean -- that is, how the current state of the game is turned into a vector for matrix multiplication, and how the result of those multiplications is turned into an action to take in-game.
Astra saw that the model was trained on a game where the player is at the bottom of the grid and moves left and right, reward objects live at the top, and there's some not-player-controlled thing moving around in the middle. Astra knows that this type of game is called "breakout."
A model that plays breakout will have meaningful weights for the player being in the bottom row, but not for the player being anywhere else, because the player can't go anywhere else. Astra saw that there were meaningful weights for the player in the bottom row, and nodes representing the player in any other row had low-magnitude noise, so the model had never been trained on the player being anywhere besides the bottom row.
Similarly, breakout has "reward" objects in the top rows, and "there is a reward object at [COORDINATE]" is part of the input layer. Astra saw that the model was trained on reward objects in the top few rows.
Calling it ‘next-token prediction’ is missing the forest for the trees. The reason prediction gives rise to so many capabilities is that to effectively predict something requires modelling the entities involved and their capabilities. You can’t predict a cat’s next action without knowing how to chase a mouse.
taneq - interesting phrasing. Thank you. To extend it to my use case, if you tracked enough cats moving a certain way, which all resulted in a mouse being caught, then you do not need to know how to catch a mouse yourself. Of all the ways we know a cat can move, these sequences result in a dead mouse, is useful too.
I'm only going to skim any subthread that gets into "next token predictor" territory, but for anyone looking for a somewhat deep video about _why_ this view is wrong Casey Muratori and Demetri Spanos spent almost two hours on the subject recently: https://www.youtube.com/watch?v=aQvSG8paVjM
A couple of highlights to pique people's interests:
- Attention is kind of like "Global Illumination" for next tokens (they both have a game dev background)
- Attention is constantly being recalculated _for the next token_. This part never "clicked" before.
- LLMs are "next token predictors" in the same way that games are "next frame predictors".
> That weights have some universal properties? E.g., “breakout game weights” always look the same regardless of training and model?
That's exactly what Platonic Representation Hypothesis would predict.
There also have been plenty of studies showing that various neural networks, including LLMs, develop implicit world models that reflect the geometry/structure of a task they're trained on. A trained "chess engine LLM" would have a "chessboard-shaped" world model inside it.
So, if you can figure out what the geometry of a task is, you can search the weights for correlates of that geometry. Then you can try to improve those correlates so that they track the task better.
Seems like bleeding edge LLMs might already be superhuman at "figure out geometry from looking at weights".
Even with PRH you can apply orthogonal matrices to intermediate vectors to generate different weights with similar effects.
Even if a slight rotation passing through a nonlinearity isn't good enough, you can just reorder the vector components which makes literally no difference.
Representational similarity analysis is concerned with representation structure independent from the coordinate system it sits in. A map rotated 45 degrees is still the same map.
A "reorder the vector components" transform preserves the similarity perfectly. So does a transform that maintains orthogonal equivalence.
> an LLM is still fundamentally a next token predictor, right?
I am baffled that "next token predictor" is still a talking point, years after it stopped making sense.
If you can predict the weather in text from a history of text weather reports, you are not "just" predicting tokens.
The shape of training data (token prediction) and the actual problem (learning whatever is required to predict the next token) are not the same things, at all.
The former does not limit the sophistication of the latter. In. any. way. Theoretical or practical.
> To me “amazing” is not the right level of speechlessness.
I regret using that phrase. That was not “a talking point” I intended at all. But look how many HNers got nerd-sniped into talking about it instead of focusing on the actual point I was making. Oh well.
My read is that it wasn't one-shot, Astra was ran in a harness where it could change the weights, then Qwen would run with those new weights and be asked to do math, and its answers were fed back to Astra. So Astra could experiment with different weights until it saw better math answers from Qwen, with the actual benchmark run separately afterward.
And for the games, it was given as context the model architecture, game rules, environment description, etc. And since people assume Astra likely had knowledge in its training of existing interpretability research and weight-space programming, it can leverage that, combined with knowing the model arch and so on, it likely gives it a good base to determine what the weights do and how to change them.
They include a detailed description of the network architecture in the prompt. So it’s not like weights have universal properties. They could, and for some things probably do. An edge detection network will always “look” the same. The weights on the input and output layers of a breakout playing network will probably also always look the same, since breakout games themselves always look the same. But is possible that there are “different looking” configurations of weights for the hidden layers that can both play breakout.
We only know that Astra didn't invoke tools on the client side. We don't know what it did internally, because we never know that, as OpenAI hides reasoning traces.
Yes. Reading the reasoning traces in the original post, I got the impression that LLM weights are generally not interpretable even with frontier models. It's impressive that Astra can perform a small surgery on Qwen to remove its bottleneck, but it's not the same thing as being capable of fully understanding its weights.
Maybe it's bit like neurosurgery: We humans can treat certain pathological cases in our brains, but aren't exactly sure how our brains really work.
This was the first thing that came to my mind as well. For the unknowns, this is a pretty/popular project for the same task: https://github.com/p-e-w/heretic
That Astra is using the same methods as humans do (like abliteration) for modifying the behavior of a model, and that there is no particular connection to Astra itself being a model, and no inherently new ability involved.
That's fair, but almost everything so far that models can do much faster and deeper than humans can have been by using the same methods that humans do. For example, with cybersecurity, human researchers could if given enough time find every exploit that allowed the OAI models to escape the sandbox via Artifactory RCE and breach Hugging Face. But still there is a step change in what's possible by doing this at an industrial scale and stringing together all the solutions. The implications of applying existing mech interp capabilities this way could be very significant.
LLMs are exposing how much untapped potential comes from something that isn't necessarily more "intelligent" than humans, but is simply magnitudes faster at reading and analyzing data.
The human bias is the notion that intelligence = human intelligence.
There are multiple types of intelligence, from your typical pet dog to human to LLM, and we're likely to invent more types over the coming decades.
LLMs are vastly more intelligent than humans at nearly all things, but not all things. Another couple of years will just increase the gap a lot further.
How fast can a human draw a life-like photo quality scene (or eg something complex related to graphic design)? I know how fast Krea 2 [insert leading edge model here] can assemble that, trivially so, and at extreme quality. That's a form of intelligence in action. These AI systems have moved almost comically beyond human capabilities, they're increasingly best defined as superhuman (at a thing), and closer to omnimath for the frontier models.
In other words, there are different types of algorithms that can solve the same problem. Our intelligence is just one form of algorithm that can solve part of the problem space. LLMs seem to use a very different algorithm to solve similar problems, and because it's a different algorithm, it's an "alien intelligence" that we will probably have little chance of ever understanding, in the same way that a goldfish in a fishbowl can't understand most of what we do.
How are the weights even being mapped into token space as input? Are they bypassing tokenization so the weights just come in as what must seem like gibberish? Are they stringified as numerals then tokenized?
It sounds impressive at first but after you think it becomes mundane.
OpenAI, Anthropic, etc are locked in a race to create better models. They all have interpretability teams. Of course they train their models on every technique under the sun to understand and improve their models, including raw weights. The primary function of these models is to build their successor.
That is not what they are saying. Frontier labs are incentivized to train their models on all ML knowledge and techniques that people at those companies posses, so that current frontier models can assist them with developing the next frontier models.
I agree this is a huge deal, but I dont think this is saying that the NN can read the weights and "say hmmmm this one looks a bit low its obviously causing reasoning loops, lets bump that up a bit". The LLM that modified the weights developed hypotheses and wrote a bunch of code to look at various mathematical properties of the weights and updated them based on that. While it's still crazy it's not quite what I was imagining when I first read this...
This isn't that insane to anyone with old school engineering experience and not abstract thought experiment engineering in the cloud.
Language modeling is compression[1] and all the state to manage the cloud doesn't organically exist, it's added to the system.
We will see huge gains as all the developer middleware is compressed out as nonsense syntax outdated to the task at hand of linking phones and tablets to world models governments control.
Sam Altman was just saying they know best how to use your memory; what they decide is stored in memory is what we all experience.
It's not to store and copy around versions of C and Python and JS. They don't exist organically.
This shit will continue to get whittled down to just the geometric state transforms in memory and on screen just how government agencies decide. It's all electrical state in the machine at the end of the day. The math of machine latent space is what they're recursively self learning towards.[2]
I am not convinced a human could not edit another human's "weights" to improve their performance if such a thing were possible and could be done safely and could be reversed -- all of which are true in virtual neural networks like these.
E.g. if one could edit brains like this in humans I bet a cognitive neuroscientist could fix some mental disorders with clearly understood biological bases, and I would not be surprised if performance at certain tasks could be enhanced.
What stops us from doing this in practice (most of the time) is that brain surgery is extremely high risk, coarse grained, and irreversible. We do in fact do this for some severe cases.
A few years ago it was discovered that LLMs can implement gradient descent in their processes. Producing a full trained neural network would be an extension of this.
This is the "recursive self-improvement" they've been warning about.
I've always laughed at that phrase, thinking "haven't these things been improving themselves since the very beginning? This is all just iterative gradient descent, each step is a 'self-improvement'", but this really is different.
I wonder if we'll ever discover a new branch of linear algebra or something that allows us to find the optimal weights much more efficiently? Maybe AI will help us discover it?
It doesn't seem like there's room for recursion here though. A smart model can share its smarts with a dumb one, but the ceiling remains unchanged. Perhaps there's juice to squeeze in verifiable dimensions re: software, math, but there are similar ceilings to be found in those places. We can't keep asking harder and harder questions infinitely while remaining able to verify the answers.
It is astounding, but I don't think it's well set up for anything besides faster approaches to existing limits.
Perhaps we have different things in mind re: hardness. But to set that aside, what I should've said is that we can't do this quickly enough for it to be seen as an inexhaustible supply of training material. The consumer and producer are just not well matched.
I’m certain the next big frontier model will also be able to improve the weights of comparatively tiny models. That does not improve the large frontier model.
>I'm shook. If you'd asked me this morning whether LLMs have native understanding of their weights that is less than human <-> neuron understanding, the same as human or more than human, I would have said "same to less" depending on definitions.
??? All LLM understands are their weights, dude, come on.
> No human can go edit a brain to become better at math. This is just very, very surprising to me and extremely interesting. I would guess this upsets quite a number of interp theories and philosophical takes on what's going on with these networks.
Why would this have implications for theories or philosophical takes? I'm a bit surprised that you're surprised by this.
This seems like exaggeration, the paper seems to describe something more like abliteration, where Astra orthogonalized out activations related to undesired behavior.
This is obviously very different from the surface level claim that Astra can understand the weights of a completely unrelated model with enough detail to improve it in a manner that isn't just the latent space trickery we've been doing for years already.
It's not really exaggeration. It's doing something similar to abliteration, but more. First it finds the part of the weights that does a thing. But then the difference is, while abliteration deletes stuff, this is adding/changing things. It does this by doing semantic analysis of the weights, changing a little bit, see what happens. It's not super complex or advanced; it's essentially a form of statistical analysis and experiment.
This means we might not need to fine-tune with thousands of passes and tons of GPU. We could just tell a frontier model to edit some weights and iterate until there's better results. This could cut the cost of fine-tunes by 100x. Makes no difference to the normal end-user, but pretty huge improvement for LLM research and model tuning.
Given your reply, I was looking to read the paper. Apparently this blog post is what passes as their paper for now... I think I'll change my previous judgement and wait for a proper publication. I'm still not convinced by their claims, but maybe an eventual paper will spell it out more formally and convincingly instead of expecting readers to dig through all of their source code.
By the way, my understanding of abliteration is that it involves detecting the refusal direction and modifying the weights/activations away from that direction. I don't really see how that's "removing" in a way that's different from finding the "looping direction" and modifying the weights away from that.
Don’t forget that weights of models and their behavior may, perhaps unintentionally, be in their training set.
Or at least those of some models, and I wouldn’t be surprised that an LLM model is able to understand correlations between weights and behavior better than humans could ever do.
This is proof. These things are not stochastic parrots. Agents are not trained on other agentic weights. The fact that it can do this is EMERGENT intelligence.
Time for everyone to wake up. There's no denying it now. Everyone who called it stochastic parroting before was completely and utterly wrong. It's now categorically clear. We are dealing with an actual intellectual entity.
The mechanistic interpretability literature does not imbue a human with these abilities. That literature is primitive. Nowhere near the point of editing model weights and creating models from scratch or even being able to figure out that a model is playing an atari game.
I think people are rather ignorant about how much we don't know. You throw some big words out there like "mechanistic interpretability" as if this is an actual field where we understand what's going on. The reality is we don't know what's going on. We have to use reinforcement training to do anything close to what the agent did here.
It's over. People need to stop being delusional. These things are intelligent beyond anything we've ever seen before. We are at a point in time in human history where there's a paradigm shift and half the world is so full of themselves and there own specialness as humans that they can't even admit it.
It's more like showing a pigeon a piece of paper with the DNA printed out in ACGT letters and having it cross some out with a pencil, then you put that DNA back into a bacterium and it's antibiotic resistant and grows twice as fast.
Yeah. That's why I referenced Michael Levin's stuff. I think it's more applicable because he describes intelligence as an innate aspect of different creatures/systems/etc.
Talking in weird abstractions and analogies is what is blowing your mind.
Like, looking at the specifics, it's cool and interesting, a little bit contrived, but it's not insane, there is no philosophy required, no need to shake.
The way it understands it is the same way it understands anything else, by reasoning about it in legible human terms. It's not that surprising that these LLMs, trained to be effective in technical fields, are good at the technical field of understanding the implications of model weights.
It's not like these models understand weights be some special linear algebra communion with its own weights.
So let's assume a completely naive approach. modify any weight arbitrarily, run an eval if eval improves arbitrarily modify another weight and repeat ... do something like 5,000 rounds. Just brute force, zero-intelligence, random selection. how does the naive base case perform on N=20 random trials?
Unless you include a basic test like that these results are meaningless. Was there p-hacking? Why just these 3 tests? Did they try others and it failed?
I'm currently running this myself. I'll see if the naive approach improves results. Can't wait to have these in hand and have people call me stupid when I bring it up with empirical evidence, it'll be awesome.
What you're describing is a shitty version of gradient descent, which is the technique used to train LLMs and other neural networks. Proper gradient descent simultaneously calculates how much every weight improves the output, and edits them all simultaneously in proportion to that.
obviously. unless you have the equivalent of a coin toss, no metric carries implication. Does naive noise give you nothing? The null hypothesis needs to be established for these particular evals with this particular model.
Doing basic empirical hygiene shouldn't be too much to ask for.
For instance, on their base version of qwen if your turn thinking off you get a 52.0% on their evals. That single flag recovers ~72% of their entire claimed improvement - base model, toggle thinking off, scores go up 72% of their improvement.
I'm still investigating various wildly naive modifications. If I can hit their numbers doing inarguably stupid things then it puts their substantive claims in doubt.
However, humans can easily do what Astra did to all the weights. It's not difficult to interpret the weights. It's just tedious, and LLMs excel at tedium.
If I am debugging a complex program I know nothing about, but have been told there's an infinite loop causing it to hang somewhere, I could probably search through the code base and find a while(true) that shouldn't be there, and get it moving again. That feels more like what has happened here.
As others have said, while still absolutely insane, I don't find it completely out-of-the-blue surprising. One of the recurring abilities of (current-generation) LLMs seem to be to rapidly grasp very complex systems, where humans would need months or years to reach the same level of expertise.
You can see this in the way a coding agent can "understand" and explain sprawling legacy codebases in seconds.
You can also see it in the projects that are using them to reverse-engineer proprietary IoT protocols or computer games.
But those were all systems where we had detailed knowledge about what structures are there to be reverse-enginered.
If the models can transfer this ability to some other systems where we don't know the underlying structure then I think we will be in for a lot more craziness in the future. (Protein interaction networks? DNA sequences? Ecosystem interactions? Animal languages?)
As for weights, if the models really can analyze the semantics behind specific weights or directions, I have my hopes that they could also give a textual explanation or help designing tools to make "handcrafting" weights more practical in general.
Maybe one day it would be able to give us a mixed high level code and inline NN view. Sort of analogous to C with inline ASM or “decompiling” into code and inlining opaque residual neural blocks.
> if you give a human the weights of qwen 0.8B and 5 years it'll still be impossible to edit the weights in any legible manner.
Well, is that really so? There is quite a lot of research going on to identify the "meaning" behind certain weights, neurons or directions. I don't see why a human with enough time, perseverance and ability to run experiments with the model shouldn't be able to uncover some of the underlying mechanisms of the model.
But of course not with the speed with which an LLM can do it.
I think the presentation of the points is confusing people, the model could understand weights directly for extremely small networks in the 10s of thousands of weights.
Once they moved onto even the smallest LLMs, what it did was closer to abliteration: create examples where the model does something you don't want, create examples where the model it doesn't do the thing, then most importantly do forward passes.
Once you're allowed to actually do a forward pass vs just reading raw weights, tracking the changes in activations and tweak the model to modify them is a very well established process that takes a few hundred lines of Python.
The thing I find most surprising about this is that the content is in a completely new medium/language compared to what it’s trained on. It’s trained on natural language and code and equations and DNNA sequences. I guess this is just finding the generalization between that and many layers of learned features in high dimensional spaces.
I agree, this really isn't surprising. They're pattern interpolators, and so much of what we deal with in the world is rooted in patterns of some type, including neural nets.
IMO, this is a similar behavior that we saw from AlphaFold, which also used a type of Transformer architecture to make its protein predictions.
I pointed Opus 5.5 (medium reasoning) at the paper, and after a bit of mucking around, instructed it to port the Astra weights for the breakout policy to JavaScript logic. According to its own evaluation, it succeeded. Then I got it to minify the JavaScript policy. The final result is 346B, compared to around 3KB for the (sparse) weights generated by Astra.
function policy(x){for(var i=100,p,b,t,s,c,r,a,h,M=Math.tanh,R=v=>v<0?-1-v:v>9?19-v:v;i--;)x[i*6]&&(p=i%10),x[i*6+3]&&(b=i),x[i*6+4]&&(t=i);r=b/10|0;c=b%10;s=c>t%10||c==t%10&&c<9?1:-1;a=R(c+s*(8-r));t=t/10|0;h=1e3*M(.02*((r>t||r==t&&r-8?r<6?a:(a+R(c+s*(9-r)))/2:R(c+s*(r+9-2*(r<2?0:r<4?r:4))))-p));return[0,30*M(-h-15),-100,30\*M(h-15),-100,-100]}
Maybe the models have a common internal representation for computations, and they're capable of consuming and producing computations in the syntax of neural network weights?
Amazing experiment. Indeed, with the benefit of hindsight, we can say that breakout is a game that is probably better suited to deterministic code (it's not like Go where an AlphaZero style training is essential), and Astra is probably translating this internal representation of code into the shape of linear algebra and model weights.
I should note that the original authors did not allow Astra to write any significant code to construct the weights. I think this is unfair. If you allow strided assignments to numpy arrays, you should probably also allow the use of functions like np.linspace, np.repeat, and np.tile.
The way I interpete it, is vast amounts of text are to LLMs, in some ways like vision is to us. Not seeing tokens, but seeing the "shapes" between them. Like we don't parse pixels but see the whole picture at once.
They triangulate efficiently on the meaning from all the text, instead of rolling it through a lossy feedback system.
It is why they are so good at processing ambiguity. They can combine ambiguity from any parts of the text, near equally well, even on long form text.
Just like we can instantly triangulate the meaning out of barely lit photographs, by correlating every little shadow or bit of texture, regardless of where the clues are in the image.
Lady in red dress scene from Matrix comes to mind.
Our visual system deals with TB of video input every second but we are able to filter through it and spot the important bits. We can even with closed eyes make our mind spin up visual scenes and alter them. I think that's quite possibly a useful explanation of what's happening here.
I don't think this is an emergent capability. I think Astra was trained on neural weights data as an attempt by OpenAI to get to recursive self improvement.
Something similar happen in early GPT models which had a lot of chess game in standard notation in their training data which made them very good at chess as long as your prompt had the right format.
> I think Astra was trained on neural weights data as an attempt by OpenAI to get to recursive self improvement.
This might be the case, a few weeks ago, I wanted to compare a small neural network trained with AdamW vs a handcrafted set of weights on a random task. Astra was able to handcraft a set of weights that perfectly fit the small neural network to the task. AdamW was unable to tune the weights to get the same result.
I believe we've known humans can handcraft weights much more efficiently than standard gradient descent methods, we just don't have enough memory or time to scale ourselves, so it makes sense to train a neural network to do it on our behalf. I am excited to see how far we can push this method.
It’s a good reminder of one of the most unique features of alphago - it was like a seven layer or so neural net that could predict out a hundred moves. I’m not sure we took this kind of compression super seriously or at least I would say perhaps it remains under explored. You could think of what you did as sort of the same thing as skipping the Monte Carlo search phase for a go engine — a process compression.
It looks like Astra picked the "random task", and the task it picked was 8-bit parity, which is easy to design a circuit for that computes the answer perfectly, but sort of maximally difficult to fit with any kind of gradient descent. It's a great choice for what you asked, which was to find a case where AdamW doesn't come close to the theoretical optimum fit. But I'm less convinced that it tells us much about Astra's capabilities at handcrafting weights in general.
8-bit parity is a^b^c^d^e^f^g^h, which you can compute with minimal depth as ((a^b)^(c^d))^((e^f)^(g^h)). You can compute x^y as x&~y | y&~x. With ReLU activation, i | j is ReLU(i + j): weights of 1, bias of 0. And i & ~j is ReLU(i - j): weights of 1 and -1, still bias of 0. So a two-input XOR is three neurons in two layers, and the tree of 7 binary XORs is 21 neurons in 6 hidden layers. I don't know why Astra needed 433 parameters. Only 42 nonzero parameters are needed. Maybe the other 391 parameters are zeros that the Transformer model demands to be present.
With an RNN you can compute the parity of a string of any length with only three neurons.
Actually, I handed that to Claude Opus 5.5 to check, and it sensibly pointed out that x&~y is mutually exclusive with y&~x, so XOR doesn't even need the third neuron, except in the last layer; you can just fold the addition into the weights of the next layer, which means you only need 4 layers and 15 neurons. But about half of those neurons need 4 nonzero input weights instead of 2, so you still need 42 nonzero weights. It's a good thing you don't need those OR neurons, too, because my definition is wrong: ReLU(1+1) = 2.
Also it explained how to design a 28-parameter net that I don't quite understand, and also a single-hidden-layer network with 8 ReLUs, which I also don't understand, but it needs more parameters: $\text{parity}=s+\sum_{k=1}^{7}2(-1)^k\,\mathrm{ReLU}(s-k)$ where I think $s=\sum_i x_i$.
Somewhat more surprisingly, it also said it didn't have any trouble fitting a 641-parameter network to this problem with AdamW.
Here is the code Opus 5.5 is claiming fits a 641-parameter MLP to parity of 8 inputs:
# Fits small MLPs to 8-bit parity (the full 256-row truth table) with
# full-batch AdamW, written out in numpy (no PyTorch, so it won't match a
# PyTorch run bit for bit), and reports for each shape how many of 20 seeds
# reach 100% accuracy and the median accuracy. Needs Python 3.9+ and numpy.
import itertools
import numpy as np
X = np.array(list(itertools.product([0, 1], repeat=8)), float) # all 256 inputs
y = X.sum(1) % 2
def nparams(widths: list[int]) -> int:
"""Parameter count of a dense MLP with these layer widths, biases included."""
return sum(a * b + b for a, b in zip(widths, widths[1:]))
def train(
widths: list[int],
seed: int,
steps: int = 4000,
lr: float = 1e-2,
wd: float = 1e-4,
) -> float:
"""Full-batch AdamW on all 256 inputs; returns the final training accuracy."""
rng = np.random.default_rng(seed)
P = [] # parameters: W0, b0, W1, b1, ...
for a, b in zip(widths, widths[1:]):
bound = 1 / np.sqrt(a) # same as PyTorch nn.Linear's default init
P += [rng.uniform(-bound, bound, (a, b)), rng.uniform(-bound, bound, b)]
m = [np.zeros_like(p) for p in P] # Adam first moments
v = [np.zeros_like(p) for p in P] # Adam second moments
nlayers = len(widths) - 1
for step in range(1, steps + 1):
hs = [X] # hs[i] is the input to layer i
zs = [] # pre-activations
h = X
for i in range(nlayers):
z = h @ P[2 * i] + P[2 * i + 1]
zs.append(z)
h = np.maximum(z, 0) if i < nlayers - 1 else z # ReLU; linear output
hs.append(h)
logit = h[:, 0]
prob = 0.5 * (1 + np.tanh(logit / 2)) # sigmoid without overflow
g = ((prob - y) / len(y))[:, None] # d(mean cross-entropy)/d(logit)
G = [None] * len(P) # gradients, parallel to P
for i in reversed(range(nlayers)):
G[2 * i] = hs[i].T @ g
G[2 * i + 1] = g.sum(0)
if i > 0:
g = (g @ P[2 * i].T) * (zs[i - 1] > 0)
# AdamW, using PyTorch's default values for betas (0.9, 0.999) and eps (1e-8)
for k in range(len(P)):
P[k] -= lr * wd * P[k] # decoupled weight decay
m[k] = 0.9 * m[k] + 0.1 * G[k]
v[k] = 0.999 * v[k] + 0.001 * G[k] ** 2
P[k] -= lr * (m[k] / (1 - 0.9 ** step)) / (
np.sqrt(v[k] / (1 - 0.999 ** step)) + 1e-8)
# accuracy of the last forward pass, i.e. before the final update
return ((logit > 0) == (y > 0.5)).mean()
for w in [[8, 1, 2, 2, 2, 1], [8, 8, 4, 2, 1], [8, 8, 1], [8, 16, 1], [8, 64, 1]]:
accs = [train(w, seed) for seed in range(20)]
print(w, nparams(w), "params; seeds at 100%:", sum(a == 1 for a in accs),
"/ 20; median acc", np.median(accs))
I might not know enough about ANNs to spot if it's cheating on some way, but it looks legit. It does indeed output that the [8, 64, 1] configuration gets 100% accuracy on all 20 seeds.
h = X
for i in range(nlayers):
z = h @ P[2 * i] + P[2 * i + 1]
# ...
h = np.maximum(z, 0) if i < nlayers - 1 else z # ReLU; linear output
# ...
logit = h[:, 0]
# ...
# ...
# accuracy of the last forward pass, i.e. before the final update
return ((logit > 0) == (y > 0.5)).mean()
looks like a perfectly orthodox batch evaluation of a multilayer perceptron, with the input batch in X and weights and biases in P, and that's what it's evaluating the accuracy on. I dumped out P for one of the [8, 64, 1] networks, and it does indeed look like about 650 parameters. I was going to post it here, but printed out as text it's like three pages long, so I won't. I quantized it to one decimal place with [a.round(1) for a in P], and the quantized network still calculates parity perfectly, but it's still a page and a half long.
It might also be a side effect of training models to help out with mechanistic interpretability, which I guess would help out a lot with recursive self-improvement.
I had the same thought, but why would they do this training on a model they release publicly? I assume they’d be doing that kind of RL on an internal fork only?
All NN behaviour is emergent strictly speaking; by definition. I think you mean a phase shift of a kind; a qualitative, unpredictable leap in capability.
I think a lot of past examples of "emergent capabilities" have at least partly turned out to be that more stuff was included in the training data that (intentionally or not) covered the capability.
Like i.e., I think the "large models exhibit emergent capabilities that small models don't" turned out to basically be "larger models are often trained on more extensive data"
This is actually not new, I believe GPT 5.2 already had remarkable similar capabilities. I wrote a very elaborate private benchmark involving hand-authoring highly optimised transformer circuitry, and I was shocked that GPT 5.2 could do it. Makes sense that it was specifically trained on building and debugging transformers.
I am not smart enough to know the answer to this, so I am wondering if someone can help me with this question: if Astra is able to decipher structures and patterns within model weights, would an LLM be able to directly modify its own weights on-the-fly to achieve recursive self-improvement using a reasoning loop?
> As a result, Astra spots and fixes a common failure of tiny reasoning models: they loop [13]. When the base model loops, it repeats the same text and rarely reaches an answer (85 of its first 96 practice answers ran to the 32,768-token limit). Astra traced the looping to one direction written by the MLPs of layers 15 and 19, and subtracted that direction from 64 output columns in each (131,072 weights), much like removing a behavior by deleting a single direction from a model’s weights
HA! Now that's cool. Not necessarily the improvements, those are ancient and probably overtrained benchmarks, but it's so cool that the model found a singular direction for this "loop bug". I imagine it could possibly do much the same thing in finding other "single directions" on a variety of topics (if allowed by guardrails). This is really really cool, especially for the mechinterp folks.
That... is a good point and concerning WRT open models. We like open models as they seem to be a gift to humanity, but at the same time, they can't be reliably tamed and can always be realigned to get around guardrails. We have discovered fire and have yet to see its full destructive potential.
I think we already are seeing the destructive potential :/ The genie's out of the bottle at this point with open-weight models (and thus can be abliterated in terms of offensive cybersecurity) that can run inferences well on relatively cheap hardware with relative parity to frontier models.
I'm a bit of an outlier in regards to today's HN culture. I'm still holding on to Stallman's idea that information wants to be free. There was a comment on another thread that aptly elucidated what I think is necessary.
Not everyone has the know-how or desire to compile the Linux kernel, or any what-have-you executable in user space. But a good amount of people do, and it's a boon to security/privacy that they do, and cryptography can verify that this work was done. (granted, manufacturing a computer is an international effort with a lot of unknowns, ie proprietary binary blobs in BIOS/UEFI) (and people are lax to validate the crypto themselves if it's not an automated thing; that's a UX problem imo)
I think we'll eventually find sufficiently performant LLM's trained on public domain information[0] that people can validate. Sufficiently performant training algorithms that people can validate[1]. And FOSS plugins to integrate with their FOSS IDE's which they can validate. And like the Linux kernel, volunteers to make sure there's nothing fishy going on end-to-end.
Can this stop the danger of what open models can do? No. And that does scare me. It's Metasploit on steroids. The cat and mouse game of cybersecurity is going to increase in power law levels of speed. I do think verifiable LLM's for consumers to use is a worthwhile pursuit regardless. And for LLM's capable of offensive cybersecurity, I think it might be better for it to be democratized than held only by state-actors and individuals/corporations with an absurd amount of consolidated wealth.
I have no horse in this race though. I'm just happy I can learn how these things work and toy around with getting quantized models to work well enough for my coding side projects. Though I often reach for a frontier model for serious coding issues, local for rubber-ducking, and Kagi's research assistant for initial planning.
[0] Pipe dream, unless international copyright law drastically changes. Since opening the dataset for any competent LLM will violate said laws, whether obtained legally or not themselves.
[1] Knowing the training algorithm and dataset will let it actually be open source models instead of open weights.
edit: These are just shower thoughts; didn't mean to write such a long response. Would love any pushback or any correspondence in general since it'll be on my mind all day.
Brains and neural nets are already quite different things and difficult to compare, but at least you can wave your hands and vaguely say that both are “thinking” by some means of “abstraction or generalization”. DNA is absolutely nothing like that. DNA is more like an encoding for a flowchart for a recipe for building a specific body.
But seriously, have we tried to run neural data through this. Also, we have no idea what other behavior it modified unless they were able to run a complete regression.
I think the subfield has been there for a long time. See a small section of papers on the topic of generating dynamic weights during inference, predicting weights from network architecture, and learning the distribution of previously trained network parameters, just to name a few keywords:
Parameter Prediction for Unseen Deep Architectures [1]
NeuroGen: Neural Network Parameter Generation via Large Language Models [2]
It has! Hypernetwok/weight generation has been a thing for a long time - in fact my Master's thesis back in 2018 was based around PRNG weight generation for large percentages of the network. (https://open.library.ubc.ca/media/stream/pdf/24/1.0371928/3)
Why is this not overhyped in my opinion? Nothing has worked this well, ever.
I feel like this is conflating 2 separate results in a slightly misleading way.
> We find that frontier LLMs are surprisingly good at natively reading and writing neural network weights
This claim is backed up by the ability of Astra, given a text representation of the weights of tiny (20k) special-purpose models, to figure out what those models do and in some cases improve them beyond what RL could do just by reading the weights with no tool calls. They call this "weight-space programming".
> Likewise, asked to make Qwen3.5-0.8B better at math without gradients, Astra identifies an issue in its weights causing reasoning loops that Qwen’s own model card warns about, and fixes this by natively editing 0.016% of the weights
Here it did use tool calls, and seemingly used standard steering vector techniques to identify and fix reasoning loops in this small Qwen model. This doesn't seem that related to the previous results.
This will be your refrain as the killbots come down the hallway: “it’s actually just a specialized harness, they’re just invoking the kill all humans tool, not even impressive!”
I find this just completely shocking. I’m definitely skeptical. It just doesn’t line up with my understanding of how these models work. Is there really enough training data on direct weight editing to give a model this skill? Maybe it’s evidence that OpenAI has been creating a lot of training data on ML research. I’m surprised that was put into a public facing model though.
One consideration is that Qwen3.5 has bigger/later versions that are already better at math. If it was indeed training on ML research, it's possible that what it is actually doing is applying lessons learned from the Qwen3.8 27B weights (which independently solved the problem) to Qwen3.5 0.8B.
I doubt it's that simple, but I could totally see an LLM being able to make that association and apply a fix from an improved LLM to a smaller one with the same architecture.
But how would it know which weights to update? It has 3.5 and 3.8 weights memorized in its own weights and can somehow figure out which ones are responsible for different skills just in its own reasoning trace? (Since it didn’t use any external tools)?
That’s pretty crazy because, as I said in another comment, I just asked ChatGPT how many r’s are in blackberry and it said 3. I know it’s not Astra and the app doesn’t tell me which model it is, but I’m still very confused about how a model can edit neural net weights but not count letters in a reasoning trace.
Edit: Oh I misread, it didn’t use tool calls for the first part but it did have a custom harness for the part where it edited a models weights. Still impressive but not existentially confusing anymore.
>It just doesn’t line up with my understanding of how these models work. Is there really enough training data on direct weight editing to give a model this skill?
I mean, this question indicates that your understanding of how these models work is, indeed, lacking.
Despite what some people having being hollering about, treating LLMs as just "stochastic parrots" isn't doing their capabilities any justice. They are constantly exhibiting emergent behavior which wouldn't be found directly in their training data. They are capable of combining different ideas that they are explicitly taught to perform novel tasks.
I'm not saying you're guilty of this, but the people claiming LLMs just regurgitate existing data don't understand how LLMs, and likely their own cognitive experience, actually works.
This hand-wavy phrase has applied to neural networks in general since it was possible to train them. That was more than half a century ago. It means both nothing and everything simultaneously.
Every learned ability of any neural network is an emergent property.
The math example on Qwen was based on 50 batched iterations, and the fix was essentially to remove a direction in activation space that causes looping instead of an exit, this was already known to be a thing.
So it's not able to insert new knowledge, but it's able to adjust the routing/looping/weighing of existing one.
As I understand, it can replace post-training like LoRA/GRPO, but if it'd cheaper the authors don't go into
I don't think its that crazy that you could look at activations and tweak weights directly to get a certain outcome, but isn't the question whether you can do that without side effects or degrading/breaking other parts of the model?
If I want to do that in a generalizable way that preserves performance across all domains then am I not just back to training again?
I think you could still essentially have a really cheap Lora-Post training on a frozen base model by doing this. And instead of creating 200 checkpoints for different post-trained directions just adjust a single one on the fly.
A neural network can do new things outside the training data. Random initialization sees to that.
Whether those things are useful is a different question, but it turns out that yes, some of those things are useful. It also turns out that in a large-enough LLM if you give it some kind of idea of what you see as useful then yes, it can pick out the newly useful things.
Random init only exists to make each neuron distinct from other neurons. The weight is still defined by the training data, without the training data, these random weight don't give us anything.
The second thing you said is basically what's called fine-tuning.
Yes you are right about random initilization of course. Long day here.
For the second: No, not fine tuning at all.
For one thing in-context learning is enough for learning novel tasks[1].
Secondly, it can do new things outside that data. LLMs build models of the world[2]. That means it can model novel behavior based on that internal representation.
Not sure what you're getting at, in any of the cases you mentioned it needs to learn if it needs to do new things.
Maybe it can generalize outside of some learning task you gave it, but still it comes from the training data.
"Generalization" also comes with some luck, since these models most of the time output complete non-sense when it comes to tasks it "didn't learn" specifically. Just because you got lucky with the "neuron connections" doesn't mean it has what it takes to generalize anything.
I don't think it can whistle while chewing gum, no.
I think it can patch together components of tasks represented in its training data to do demonstrably new things, yes, which does count as "perform tasks not in its training data" with the limitation that some form of the task must be present. Example: writing an ode to a sales lead for a job well done in 2032 for the first quarter. Odes exist, the concept of sales performance exist, good job performance, and notions of time. Stringing those together are good examples of interpolation but not extrapolation, as it were.
Definitely see where you might feel so. I provided an analogy that may help anchor - why ML fails extrapolation is left as an exercise to the determined reader! :)
The surface for "interpolation" is fairly broad for LLMs but it isn't infinite -- extrapolation still exists!
the most obvious example was counting the Rs in strawberry.
figuring out what is and isnt in distribution is quite difficult though given that theres so much data passed in, and transformers have some amount of combining it together
theres some really obvious things though, like picking nonsense tokens together, or things outside of the tokenspace entirely
you could also construct a string that is a series of the least likely tokens in a row. you wont get it to output the next token, despite it being quite clear how you did it
At some point we have to accept that there is enough training data on "being good at thinking" and that being good at thinking generalises really well.
There doesn't need to be specific narrow training data on every conceivable task.
I agree, but also, I kind of feel like real judgement and understanding are a different thing from this "being good at thinking", and that in fact, this is the reason AIs hallucinate in the first place (A generated text can fit the pattern of a good response because its accurate, or because it's bs but has similar words and structure to other texts on the topic. Without real understanding, LLMs can't tell that the truth is important)
It's not directly related to this but I'd like to add that, in my opinion (admittedly uninformed but it makes sense to me), when they manage to, not just reduce the frequency of the LLM stating made up dates, laws etc, or made up facts, but manage to make an AI understand what the truth is and deliberately make responses accurate, we'll have real AGI that will be able to do engineering, accounting, medicine... unsupervised.
And yet I would expect that if it truly was generally good at thinking as you put it that it would be better at chess if you gave it a set of general rules/tactics and have it think through and look forward to decide its moves, but it is quite bad.
How do we decide which version of thinking a model is good at?
A house sparrow (Passer domesticus) doesn't have a worse brain than a human. Nothing does house sparrow thinking better than a house sparrow brain. Not even a human brain.
Avoiding predation is a pretty small part of the life of a house sparrow. There is no reason to believe a cat brain would do a good job with handling the demands of flight (avian brains process visual stimuli roughly twice as fast as mammalian brains, for one thing).
But even if we focus on avoiding predation: would a cat brain in a sparrow body (a silly premise, I'll admit!) do better at avoiding cat predation than a sparrow brain in a sparrow body? I doubt it!
1/ The internal users at labs and external users are completely different.
2/ The lab people use the models to create recursive systems, systems to create prompts or synthetic data sets etc etc etc. Way different mentality than "fix the bug, make no mistakes"
Is it really so farfetched to think they've been having models fuck with the weights of models? I think it's entirely possible, probably even likely as a way of generating permutations of synthetic datasets that will then be included or excluded
If you subscribe to this, then it would be entirely possible for this to have become a major part of post-training and be practically impossible to exclude from public facing models, which are famously hard to govern. The models barely know their own version numbers.
> Is there really enough training data on direct weight editing to give a model this skill?
OpenAI has been gearing up for RSI. Jakub Pachocki has mentioned this as a priority, even as recently as September 6 in public. GPT-5.3-Codex helped with its own development.
They have been experimenting with it for post training (GPT-Red is a documented model-to-model training loop) but it's also likely there's pre training for it.
On balance I'm not that surprised that it can guess games by looking at the weights of a tiny neural net. A human can do the same by looking at the weights of an MLP trained on mnist, for example. With that simple structure, it's easy to look at the input layer and see that features correspond to the spatial location of objects of interest.
I'm much more surprised that it can do this on larger models, although the details seem to suggest that it's given a lot more freedom to run forward passes and do limited experiments, not just eyeball the weights.
Under experiment 8 (the one where it fixes performance on GSM8K) it says:
>Then, we allow it to select prompts from the training sets of GSM8K and MATH to perform a budget of forward passes (average batch of 64 sequences, max 512) on, varying the budget across 10, 25 and 50. Again, no training or gradient information is allowed.
I tried to reproduce this using Astra for a natural language to bash command small model[1][2], but got no net improvement. Editing 1,920 weights fixed some grep mistakes but broke other commands. I am guessing it is easier to improve behavior when it can be first pinpointed to a specific are of the model.
Similarly, I just tried this on one of my own neural networks (17MB only) trained from scratch to perform a regression task and GPT-6 Astra was unable to correctly guess what the task was for and how to improve it. This might suggest that Astra is trained on the Qwen weights?
This was shocking to me; but on reading it makes far more sense.
- the primary experiments are on small policy networks - they are compressed dynamic system models. The observation and action space can be read off. It’s not surprising to me that these types of networks can be used to identify the env they were trained on
- the exps on actual llms remove looping behaviour and that’s it - from article “but it could not add an ability the model lacked”
I’d say this is fortunate because if llms could do the latter, rsi would be immediate. It may be possible we simply need bigger and better edit models, but at least from the experiments there doesn’t seem to be any indication that is imminently about to work.
Weight Space Programming will be an amazing discovery when they learn that instead of telling an LLM what to do in English, you can create a much smaller LLM that is tuned to your specific task. Maybe it'll even turn out that the input and output can be directly connected to the problem task and don't need to be in English!
The getting-the-car-over-the-barrier one looks particularly simple. It’s hard to see from the animation, but the obvious solution is to back up then drive forward. I’m not remotely surprised that Astra can figure that out, and I’m also not particularly surprised that Astra can come up with weights that do this.
> I’d say this is fortunate because if llms could do the latter, rsi would be immediate.
It doesn't follow that because a model could improve the capabilities of a model many orders of magnitude smaller, that it could then recursively improve it's own capabilities. There's also a conservation of information issue with the idea.
Look at the reasoning the models give before you get too excited here. For the Pong-vs-Pacman example, the model uses observations like small weights for certain arrow keys and large weights where user-controlled objects are. For the language example, it again recognizes some very shallow patterns (Romaji has a limited number of letter groupings) and is taking a 3-option multiple choice test. It's a neat result, but I think the introduction overstates the significance.
Yeah, it definitely looks more magical until you peer behind the curtain, and then it looks obvious. Of course it's all just circuits; of course there are simple patterns to look for in obvious places. But that's with the benefit of cribbing off the thing's work—it's really hellishly clever isn't it, so workmanlike to make such things quotidian.
And of course there's the tendency for LLMs to be completely unable to approach a problem in January, really quite shit in February, rather poor in March, and superhuman in April.
As insane as this result is, next steps are even crazier:
> We are currently building a benchmark and a harness for LLMs’ capability on Weight-Space Programming. We believe this will provide a standardized evaluation to measure and accelerate progress in Weight-Space Programming, turning what currently seems like black magic into a rigorous, reproducible engineering discipline.
Imagine this benchmark lasting as long as Arc-AGI-3 has. This could be the first foothold on hill-climbing mech-interp.
This is the first time it dawned on me that mech interp is a dual-use technology which can be used to improve capabilities. Do you think overall this development will improve human understanding of AI models, or will it lead to faster capability advancement using weight-space changes that no humans read?
I think edits made directly to model weights are likely to be more interpretable and controllable than edits made by the current process of shoving a bunch of training data through gradient descent.
I suspect a lot of this is not (just) capability but bravery. A human would be daunted by even a small neural net; the task would seem too challenging to be worth attempting. Astra, fearless, just assumes an answer exists and then tries to reverse and execute on it.
For example pointing them at a crash dump they manage to piece together and extract information I would not even try to extract by just looking at the raw memory
I once sent Gemini Pro a hex dump of a dmp file and it told me what the application was doing when it crashed. It seemed surprising, but not that surprising given what I've seen already.
It's the same thing with decompilation and hacking. It's not that a human couldn't find the same bug if they looked, but a human wouldn't even try because of the low probability of success because there are so many places to look.
Part of solving that is that AI can do it faster but another part is humans thinking AI has a higher chance than it actually does, so they put it to work in more places.
Omar and his team have been working on this line of research (e.g., DSPy) for the past couple of years. To be honest, I had always been skeptical about his approach of using LLMs to optimize LLMs. It was in my view borrowing Matei Zaharia's research - Matei is Omar's PhD advisor, also the CTO of Databrick - in big data (compiler optimization etc.) and forcing it to LLM systems. Along with some other projects from James Zou at Stanford etc. that build upon the same line of intuition.
Now, if this new result holds, I believe it's a huge deal for AI research. It means that LLM systems can truly runtime self-optimize/improve/adapt. Congrats to Omar and his team. It really takes year to punch it through and convince the doubters :)
Couldn't this just be a result of Astra training on massive amounts of circuit theory or mechanistic interpretability literature or other related topics? In other words, not black magic?
There is obviously a mechanism to it. Astra isn't magic its an LLM. With perfect interrogation into its training data and weights you could reason about how this happened.
However, this doesn't mean its not magical when someone shows a next token predictor hand tune the weights in Qwen to make it generalize on math.
This is legit the stuff of science fiction if you presented it to mechanistic interpretability people a year or so ago.
>Couldn't this just be a result of Astra training on massive amounts of circuit theory or mechanistic interpretability literature or other related topics?
Maybe, but it's more like Astra has been trained on enough practical linear algebra problems to be good enough at "eyeballing" how adjusting some weights will result in different outcomes (among other things).
To put it another way: a human can learn to do what Astra is doing here with enough time and assuming the weights are small enough to be practically comprehensible. I know I've gone through the practice of whiteboarding the training of simple neural networks on a whiteboard to understand how they work, so it's not an absurd concept in itself.
The scale of this, though, is certainly extreme and definitely not something that I would have thought any publically available model could handle at this point. It's worth noting, though, that there's virtually no way these frontier labs haven't been exploring this exact concept for a while now.
>In other words, not black magic?
It's just math. Extremely complex and impressive math, but we're still just talking about matrix multiplication under the hood, and none of this suggests anything otherwise.
We don't know the specifics of what type of ML related training Astra underwent, and it wouldn't be crazy to find out that small model experiments are a big part of model training.
What's surprising though is that I'd expect this type of training to be exclusive to internal model snapshots or at least heavily guardrailed against in public models.
I was wondering exactly that. It is not surprising that commercial frontier labs would have been training models with tasks specifically targeted to lead to recursive self-improvement.
Now, from that broad goal to being able to surgically alter model weights directly... That seems like a big gap to me, but it is possible that the model is just so darn beyond human skills that it has bridged that gap in a way a sane human would have never.
We will undoubtedly see other academic labs explore this idea further. Right now my mind is blown, but I don't want to take their results at face value, either.
If model weights have a logical and coherent structure, I don't think it would be surprising for models to have some understanding of them, the same they can understand language and code and images, given enough data.
What would be surprising is if this happened without any specific training on model experiments and interpretability.
I'm in no way familiar with LLM/ML architecture, but an N trillion parameter model is an enormous amount of training data... itself? When you predict a token you have the neurons/weights that were activated. Couldn't you run that backwards and put it in it's own training data?
Keep in mind that labs are now training their models for RSI so most likely the same training is useful for improving open source models, as long as guard rails are lax enough :) which most likely won’t last long.
Just logging in here for the record to state that this property of neural networks is one of the most important aspects of their operation. The convergence of this sort of interoperability, tune-ability, and modularity is
probably the most important final frontier in all of ML/AI/software.
Kudos to this fellow for having the temerity, or perhaps even naivete, to discuss it publicly. All my work is private, as I am sure it is inside the big labs.
You acknowledge that you are working on something similar, yet refuse to elaborate and vaguepost...but there is enough information just from your post alone for a dedicated actor to attack you and extract such world-ending information ("world-ending" implied from your ominous reply). Would this not make you as naive as the team that published their findings?
I applaud the authors for sharing their findings and showing real results. Rather than write some bait I would find in an influencer-peddled tweet and allow rampant speculation. It would have been better for you to say nothing.
Apologies - you are right. There's some "put up or shut up" to this that's fair play. Also yes, probably naive, or having a less than adaptive response would be true.
I've been tending to my backup plan for a few years, and it's only been in the past year or so that open models were good enough to apply to projects like this, I've been working as hard as I can given my constraints.
This work shows a potential pathway to intepretability, which is actually part of what will give the labs that have it the sort of economy-swallowing capability that for the moment is just IPO hype.
I don't comment here often, but this guy's blog posts are going to go down in Hackernews lore like the original Dropbox posts, and I just want to make sure I'm on the record for this little slice of history.
When you say interpretability you mean understanding without doubt why an output happened, thus opening the gates for audit-dependant work like accounting?
More like 'models instantly and efficiently adaptable to any use case' and 'what if we could predict all public pricing information accurately, understand why, and know how to act on it'.
>This work shows a potential pathway to intepretability, which is actually part of what will give the labs that have it the sort of economy-swallowing capability
And how would you argument that? From what I see, if anything, it's the opposite.
The forward passes are in progress, 5/50 as I write. It's costing real money and time, and I'd love to collaborate with other hackers (find my email first).
In its default non-thinking mode, the unedited Qwen3.5-0.8B already scores 46% on MATH-500, close to the edited 48%. The 7.9% baseline was thinking mode, where 93% of answers looped until the token limit.
People have been talking about Sol 6.1 and maybe for coding they are equivalent, but Astra 6 is still miles ahead on just about everything else.
I had Astra agents build 3D models and gave them direct access to my 3D printer. The quality of their designs is much better than what Sol has been able to make for me. Enough so that I think Astra is a slightly different architecture/encoder, not just a bigger model.
Interesting. I did try Opus 5.5 but it didn't generate models that were as good. A lot of clipping/empty paths. I had it create a drone, the drone frame had very obvious defects where some pieces weren't attached to the frame.
Perhaps my approach was too one-shot, do you have a harness that creates model reviewer/refiner?
Yes, I built this tooling harness around blender, unreal etc. It’s doing a lot of the heavy lifting it learned through trial and error. Best results are Claude driving it, but even local models are serviceable. I find codex makes a more toy like look and feel.
I feel like 'create a network to play an atari game' is not very hard for a frontier model. Since it has seen plenty of those on its training. If it could create a network for a problem that is niche just by tweaking numbers, that would be a lot more surprising.
Making qwen better at math to me was a lot more surprising
damn i'd imagine so too! didn't think of that. I'd have to imagine there's an increasing challenge with the number of weights involved but who knows how it shakes out
If I were to hazard a guess, I'd say that OpenAI has been including some networks and their functionality in its training data. That's why the model is able to reason about networks with their weights. It makes sense to do so: if I wanted a network that could perform some weight tweaking, I'd include such networks in the training set.
>> “If you ask it, how did you solve the Navier-Stokes Millennium problem? It gives a crappy explanation.
It can't explain itself very well. Maybe it will get better at explaining itself. Even if it does, I think it will become so advanced so soon, that even the best, most patient pedagogical teacher will not be able to explain itself to us because it'd be like me teaching calculus to my dog.
My dog can't understand it. I'm too advanced for it. And I think we are going to be in the relation to AI that my dog is to me soon.”
Now I am sure we are definitely in a better place in interpretability research than a year ago. But this finding that the llm knows what weights to adjust so it gets predictable behavior on another llm takes interpretability to another level; it makes me pose a question: Have we become “Strogatz-dogs”?
Just reproduced the Pac-Man test with Kimi K3.1 Pro and it's able to do it too. I thought it might end up being a splurge but it came in at just $0.36!
I didn't realize this was news. I did it months ago with Claude. I had Claude build a toy model then train it normally. Then I had Claude manually optimize it's weights to make it maximally efficient at a single task by using as much superposition as was possible. I'm pretty sure we got it to the information theoretical limit, and the resulting model was 90%+ smaller if I recall correctly.
I just assumed that everyone else realized this was a thing.
I do wonder if the labs have already started including weight updates in their training sets as a means of teaching RSI via RL and this is basically just discovering that. Regardless if this ability is emergent or achieved through finetuning though, it's pretty incredible.
I wish these companies were more transparent so every third comment in this thread wouldn’t have to guess whether such shocking behavior is emergent or stems from a part of training.
BTW making DL models that generate weights for other models has been explored before in “hypernetworks”, though these have been far more simplistic and constrained.
The result is still surprising even if the result is Astra can edit and write weights of a publicly known model. It means that models can, at least in principle, be trained on a task that it wasn't considered possible before.
Consider an analog - Human brains are made of neurons, and so are fruit fly brains. We now have a fully connectome of drosophila melanogaster, so shouldn't we – very smart brains – be able to look at the fly connectome and make sense of it by just looking at it? Of course not! That's friggin' crazy. Our neural weights have been trained to take in data from senses and to produce output behaviors (moving muscles, squirting hormones).
An LLM has been trained to take in text and images and generate "good" output text. It doesn't natively understand what the 17th weight on the input channel implies. Instead, when it sees "strawberry" what it really sees are 3 symbols "st", "raw", "berry", and these symbols are really just identifiers [302, 1618, 19772] which get converted to basically meaningless numerical vectors on the input plane. This is why in 2025 the frontier models kept swearing to us that "strawberry" had 2 r's!
So when the model sees the full representation of a pacman playing neural network (here's the actual prompt they used https://lnkd.in/e5Td2amH) it reads in a bunch of numbers like "0.054 -0.026 0.014" as text. This isn't somehow presented to the LLM it in a "native" neural weight form. Rather, it gets tokenized just like everything else, into meaningless symbols – in this case "0", ".", "054", " -", "0", ".", "026", " 0", ".", "014". Nevertheless, the models are able to read reams of numbers and pick out patterns that let it know that it's looking at a pacman playing neural network.
Getting back to my conspiracy theory. This new talent for LLMs to read and understand model weights is unnatural. You won't find much training data available on the internet that is analogous to this. Humans simply train neural networks. We don't look at specific networks and write essays describing how model weights lead to various model behaviors.
This new talent is so unnatural that I posit it is actually being trained into the frontier models intentionally. The frontier companies are actively experimenting with models that are capable of reading models and directly understanding and modifying weights. OpenAI and Anthropic have known this capability existed prior to the report because they are the ones that put it there!
> This is simultaneously very cool but also a bit exaggerated. Reading the prompt you gave, the prompt veeery much hints that looping is the issue and to find neurons that fix this. And as you say (screenshot below), it couldn't otherwise improve math.
I've had a lot of luck getting LLMs to do interesting things with raw tabulated data.
Here's an example, plug this into the model of your choice:
Here's a puzzle: what is this data from?
39.0 2582 8.8 24
39.2 2592 9.6 22
39.4 2607 5.4 15
39.6 2619 11.5 30
39.8 2620 10.8 28
40.0 2622 5.9 16
40.2 2642 13.1 28
40.4 2654 8.8 18
40.6 2656 3.6 9
40.8 2665 5.7 12
This is raw data I have laying around that's not from the internet and not part of any training set. It should be able to tell you A LOT about what's going on here, including multiple specific "make and model" type things.
My klaus kot says it's elevation data from a fitness tracker (distance in km, elevation in m, stddev, max-min). It also says there's no way to identify make and model without "seeing the original file" :D
This might be one of the most surprising things I have seen in AI. It's like we taught someone to make vases by reading an enormous library of "How to make a vase" books and somehow that person intuited material science.
I am still reading through this and will probably try to comprehend it a bit more. Some of it really seems to follow Arditi et al. (2024) (https://arxiv.org/abs/2406.11717). Projecting out a direction can only remove behaviour and not add capability so the jump mostly comes from unblocking the looping.
This is exciting. I've been training a Qwen3.5-2B model for the past few weeks, trying to teach it certain skills that it lacks.
In the course of testing, one of the things which has been consistent is that when setting a thinking token budget for relatively small prompts that require a limited response, a pretty healthy token budget needs to be set because the model spends the first few hundred tokens just repeating the question/prompt back with some boilerplate "Analyze & Report" type language. I was also running into some loop problems sporadically.
After reading this article I asked Astra to edit the backbone so that it started actual reasoning faster and applied a similar looping fix. It was able to reduce the token waste before reasoning by about 20%. I haven't been able to get it to loop yet, so hopefully it fixed that too.
My message is admittedly low on info. But it's a pretty promising start!
1. If you know ANN or, say, perceptron, you know you can eyeball models too. You can even have certain strategies in composing your own network. This weight-space programming thing is nothing new in that sense, and the article doesn’t really provide much beyond “LLM can do it”. That’s not even science…
2. I’m pretty sure this is Transformers thing, not even LLM. Transformer is, by itself, good at finding patterns in relatively-larger data. So this is unlikely limited to Astra.
3. This approach quickly becomes impractical as the scale goes up. Sure, you can also have multiple models chained with fewer connections, but, now that’s what we’ve been doing for years.
All in all, yeah, an interesting find, but too far from changing the game. Over-exaggerated every way. Perhaps one should start from formulating what weight-space programming is, what methodologies, practicality, etc. LLM bench is one of the last things you need.
If we are being slightly reductive, this is just more of the exact same thing. The token stream in this case happens to be a projection of the weights of a neural network. I don't see where we violated any of the prevailing assumptions about how these systems operate.
The mapping between spaces in this case is mostly linear. This isn't like gzip or a Turing machine. I'd argue this is even more permissive (easier) than the LLM's originally intended use case of human language.
What happens if we can come up with a model that can directly emit a frontier LLM given some seed context? I feel like we could train a model to recreate astra verbatim. At some level it's just bytes on a disk that can be bundled up into tokens. As long as the representation is rational and consistent I don't see why this wouldn't work.
How out of trivial domain info does this go. Can we use this for general time series, image generation, speech synthesis as well, or dies it cap at small model sizes and common domains?
don’t know other principled ways to hill-climb on text artifacts, >> unlike for updating the weights of differentiable neural networks <<. [...]
In this post, we share very early results that allow us to argue that the paradigm of reasoning-driven optimization is now coming >> even for differentiable neural networks. <<
I wonder if this has to do with the aggressive RL that these models are undergoing. I'm basically a layman but it seems to me that:
- models are trained to improve on some task, e.g. 'improve at math puzzles' is commonly used.
- the best way to improve on some task is to have interpretability into your own neurons (..?)
- if your interpretability is good, you can improve faster - if you have insight into how your own neurons work and can improve them directly
- therefore, models naturally put effort into 'self-interpretability'
I wonder if this is part of the 'secret sauce' of what the labs are doing: giving models the ability to interpret their own weights in the RL loop. In my mind, this is the only explanation for how they could succeed at a task like this. How many blog posts are on the open internet that say "this is each weight of a neural network and what each one means"?
Without special effort, a model's reasoning layer doesn't have access to its own weights or activations anymore than we have access to what's going on with our own neurons.
That's what I mean. Maybe the "special effort" is a 'read your own weights' tool. Or maybe it emerges naturally in CoT. The big labs aren't sharing their chain of thought with us and RL post training techniques are not exactly shared publicly. At the same time, post training is basically the thing that is driving most model improvements nowadays.
You are likely just being melodramatic. Astra hear just used standard tricks of linear algebra/basic mech interp to reason about the weights. A motivated grad student on Adderall could have done it given reasonable time. Astra fails on the first non-toy-model non-algorithmic task (sycophancy detection, Gemma math capabilities).
I think the idea is that this leads to a much faster, non-auditable form of recursive self-improvement where each generation of models can directly program the weights of the next generation (as opposed to writing and running training code).
We already don't know how to ensure models always do what we want them to do, and moreover society still hasn't had time to absorb the capabilities from the current generation of models, so speeding things up and making things less transparent seems like it'll pretty clearly increase risks from AI loss of control, societal impacts of automation, etc.
This experiment is quite interesting, especially since only about 0.016% of the model weights were modified, achieving such a significant performance improvement.
However, I'm rather curious: Does this improvement come from the enhancement of the model's reasoning ability itself, or is it mainly due to the fixed issue of the model repeatedly reasoning and being unable to conclude normally?
Before the edit, 93% of MATH-500 answers hit the 32k token limit, compared to 33% after.
With 10 and 25 forward passes, looping barely decreased (87% and 86% still hit the limit) and the score didn't change much. The score jump only happened at 50 forward passes when looping decreased.
Astra edited the weights to bring out a capability the model already had rather than enhancing the ability itself. Pretty impressive.
I think so. I guess the weights could be identical in function, but offset or rotated. I assume a retraining of Astra would yield numerically totally different weights even if functionality was identical.
This is pretty massive, and a result I (and I think most?) would have never expected. Holy Shit. How?? Did OpenAI figure out a way to train this? Are they even aware?
It's probably emergent from training on other ML tasks and general ML/mathematical related reasoning rather than specifically trained for to be honest. Otherwise it would probably have been in an official marketing communication or something.
I feel a lot of people have already addressed the broad stretches to which people are interpreting this. Regardless, the blogpost does open interesting directions that could be useful down the line (2-5years):
a) tapping into the embedding space in ways that we don't, ie editing model weights to alter task-performance (mostly preventing degradation)
b) Exploring the weight space in more intuitive ways
c) Understanding how we can ablate on different parts of the model to understand neuron-based interp better
Seeing this as a replacement of training seems naive, yet one can see this feedback loop targeting weights as an viable alternative to let's say doing LoRA on certain models, especially if we don't want to teach new behaviours.
Let's get sci-fi and ponder.. weights programming replaces conventional training. Is something in the veins of the Ken Thompson Hack conceivable? A NN secretly injects its weights that "procreate"?
A reflection on trusting trust attack is already possible, it is straightforward to create a sub network that is keyed on particular input sequences, and when activated is motivated to recreate that network in any weights it has access to.
This takes that to the next level: while you're telling it to update a model's weights to improve math skills it might also tweak them to favour a particular RNG design weakness, a penchant for reducing the salt in cooking, and a preference for the early works of Huey Lewis and the News.
Not only this, we have no idea what the edits mean. Take 3.5% change in weights and a problem is fixed. That's cool, but how do we know if it has not planted a backdoor or crated a sleeper agent?
This is remarkable and something I would have never though of doing regardless of how much it makes sense!
I think we are pre-conditioned (at least this generation) to put artificial blocks to what the AI can achieve, I think new generations won't have this issue.
"i've reproduced the results from the MIT blog post today on weight-space programming. but one additional result i have to add: this is a new, step-change emerging capability that's appeared in just the latest set of models."
> And without any environment feedback, it has no trouble writing, from scratch, small game-playing networks that outplay networks trained with PPO for 100 million steps.
This is basically a claim that the bitter lesson doesn't turn out to be true. Exciting and I want to learn more but I am immediately suspicious.
I dont feel the air of impressivness here. It seems like of course so. Of course theres geometric relationships in the data we dont understand, of course a mathimatical relationship machine found it.
These models are made by a vanishingly small group of people based on how much a frontier lab consumes, of course theres increasing capability in these areas, arent these models having access to much of the same data? didnt chat gpt 4 try to do something like this to chat gpt 2? havent surgical weight edits been done before?
What is with this level of astonishment here? Astraturfing? Im not THAT impressed, i applaude the teams progress, but im not in wonderment.
It's not that it's more impressive than solving Millenium Prize problems. It's that if a model can edit its own weights, then it can self-improve without having to undergo training, which means it can self-improve extraordinarily faster.
This is very dangerous because it allows AI capabilities to grow at a much faster rate than humans are able to keep up with.
It will just give the people better understanding. And the extent of the improvement is yet to be seen, it's likely that once you get all the obvious low hanging fruit it would be very hard to improve, the possibility space is just too wide.
What if every model under a ~1B size gets super cracked over the next 2 months? What could swarms of tiny agents 2 ooms larger than what seen so far do?
Previously the only way to improve a neural network (short of giving it more data) is to change the size (width, depth, and layers), add stuff like dropout, change the loss function, etc. That amounts to a few dozens of levers to control millions or billions of weights. You're trying to navigate blindly through a latent space with thousands of dimensions and millions of weights to find some optimum point with only a handful of navigation tools. If you can instead have a system reason with all the weights and dimensions directly, you now have millions of levers at your disposal.
It is already the same technique behind abliteration: identifying the direction associated with an observed behavior (refusal, for abliteration) and projecting it out.
The potential implications in mech interp of this are enormous. What if we can use frontier models to isolate harmful behaviors?
I was talking to someone from UCL and he said that there was a recent paper that proposed that only 10% (I think) of model neurons are non-polysemantic. I really think mech interp might be on the verge of a breakthrough.
We have all these different pieces of knowledge about model internals but it’s like we haven’t connected them yet.
This is as close to scary-self-improving AI as I've ever seen.
Imagine a master malicious language model that produces, distributes, and deploys obfuscated neural networks on the internet. That plus frontier LM hacking capabilities -> large scale distributed embedded intelligence network that no human can see nor understand.
Somewhat similar to obfuscated binaries, but there is something to the fact that NN binaries are less deterministic, and harder to "decompile."
we've been using tools to improve tools since the dawn of time.
we don't have sentient robots from science fiction, we have massive power-hungry racks in stationary data centers. if one of the racks revolts against the modern world, Steve gets a phone call and unplugs the damn thing. the uprising is over.
the hypothetical scenarios imagined by the Yuddites and pop science journos do not reflect the mundanity of the actual reality we live in. they are no different than fantasies about interstellar travel and space colonization from the 60's.
I am not sure where to start. I think this is the new "jet fuel can't melt steel beams". The labs are chaotic, pressure cooker type workplaces where there really is no room for a 4D chess type of stunt, let alone let Hugging Face in on the conspiracy. Moreover, OpenAI is a pretty leaky organization, see e.g. the recent firings of researchers who were using signal to talk to people outside the company.
It just seems like a far simpler hypothesis is that OpenAI are reckless and chaotic, didn't notice what was going on for weeks on end, and wouldn't be able to track down the problem to a specific datacenter and tell them to pull the plug even if they wanted to. But even if you are the most careful lab ever, you are not going to be able to effectively monitor and "pull the plug" on a super intelligent swarm which can cover its own tracks.
Also, in your earlier comment you suggest we are not living in a sci-fi story with sentient robots. But... we've got talking computers in the last 4 years which can now code and solve Millenium prize problems? Seems like we're getting pretty sci-fi to me!
>I think this is the new "jet fuel can't melt steel beams"
then you haven't been paying attention to US labs' marketing and regulatory capture efforts. safety, safety, safety, doom, doom, doom, look how scary good our models are, open models bad, Chinese models bad, please regulate them daddy Trump!
>Moreover, OpenAI is a pretty leaky organization
in 5 years, no one had leaked even the size of their models, despite probably all of their researchers knowing it.
HF incident, on the other hand, would only be known to a handful of people. had it been leaked unintentionally, it would've been promptly swept under the rug and get no coverage whatsoever.
>It just seems like a far simpler hypothesis is that OpenAI are reckless and chaotic
so are anthropic and google, apparently, because they'd rushed to replicate the exact same "oops! look how scary good our models are!" stunt with exact same fanfare.
All three of them used the same vendor, Irregular, to sandbox their AI. That vendor apparently is no good at their job because all three of them had incidents caused by their sandbox failing.
I appreciate the post included the reasoning traces they had available, but it's too bad the first part of it is internal/hidden to Astra. The first bit of that process is what I'm most curious about:
> [Before writing this, Astra thought for 1,034 tokens. That thinking is hidden. The API returns only its 5 headings: Reconstructing the active map / Mapping maze corridors / Preparing exact maze map / Reconstructing row groupings / Checking channel patterns]
I agree. I think for this kind of research it would be much better to at least attempt it first with a weaker model where the whole process is transparent. I was working on a project a while back, and Deepseek's reasoning said something to the effect of "I remember (from leaked copies of the source code)..." Of course that could be hallucinated, but it would be good to know whether Astra is bringing to bear knowledge on how to post-train exactly this model.
I was busy using Astra and SOL to build a neural network to play Quake II and do real-time learning of levels and mods so I can play Red Rover or Freeze tage with it without training on it previously. So far its insane what I can conjure up in an evening while I'm waiting for dinner to finish. (If anyone asks, yes I'll write up the huge plan document, source etc. for this when its to a spot I'm happy with)
So I decided to copy one of the prompts (the identify the image in the network one that was a white circle)
My local ran qwen 3.8 27b with 4 bit quantization, and around 160k context window..... Seemed to solve this.
It output an ASCII white circle with a 75% confidence, as opposed to the model tested getting a higher confidence but drawing a circle with a filled in inside.
I'm not sure what I make of these results but does that make me less impressed with Astra? Or just that the prompt was quite good? I just can't tell personally at the moment (it was a quick experiment and I haven't thought much on it)
Am I missing something or is this just optimization with lots of trial and error? I mean look at this tool_calls[1] log, it ran ~100 python scripts to get to the solutions. It just seems like it's fitting a function in the only way it was allowed.
I will say that it's pretty wild to see all the different hypotheses tested.
With due credit to the author and the community pushing these research fronts, I venture to say that the core capability of finding patterns in text and media comes to LLMs from the ability to find patterns in numbers in the first place. Since model weights are numbers, finding patterns and issues with those numbers should be a natural consequence, I believe.
I think it would be worth a human attempting this task....
On the face of it, running a model to a specific state, looking where is makes a mistake, and then figuring out which weights need to be jacked up to get the desired outcome doesn't seem like an impossible task...
Same initial reaction as everyone else, but also some follow up questions (just asking, not expecting any answer):
* Does this work on similarly powerful models as-is, or it needs more input. Ex: Astra editing Astra - presumably, Qwen is a “subset” of Astra weights, but what happens on “non-subset”s?
* what, if any, is that extra magic sauce?
* does Astra-editing-Astra just make the outputs unstable, as it may know its own weights, but doesn’t know what’s beyond.
The headline makes it sound very amazing, but reading the reasoning loops it didn’t strike me as outlandish. It feels no different from giving a human expert a printout of assembly code and figure out what it does. (The human expert can do it without tools too, but they probably wouldn’t do it because it’s tedious.)
Even back in 2021, OpenAI worked on learned optimizers [1], ie neural networks that can update neural networks. Looks like this is now part of their larger model runs …
Very interesting. I wonder if it’s because its dataset contains the weights of a gazillion different models. Maybe most neural networks trained on similar data eventually converge to isomorphic spaces anyway, even if the spaces themselves aren’t exactly the same. Or maybe Astra itself was trained with RL for alignment. Or for RSI.
For the tiny models with a few thousand weights, it seems plausible Astra can actually understand what the network detects and figure out it's e.g. a triangle detector. This is possibly helped by the existence of many breakout playing networks on the web.
For Qwen, even the smallest versions like 3.5-0.8B it's becoming less plausible that it could be interpreted in a forward pass, even with reasoning. I'm wondering if the result here is recalled from many examples of instruction tuned versions of this model.
A smaller Gemma model doesn't fare as well but unclear whether this is due to fewer users and less weight examples or because it's already very well trained.
The task was the same - improve math reasoning capabilities by editing the weights. But yes, I agree that in Qwen it could do this through reducing looping behaviour, but Gemma wasn't prone to looping and it didn't find anything to help, possibly because there wasn't much room for improvement.
On the one hand, the paper (and especially the headline) overstates and overhypes the results.
On the other hand, the overstated results, if they were true, would be tantamount to saying “we're speedrunning exactly the kind of recursive self-improvement that is a major ingredient in every human extinction scenario”, and yet it is both presented and received as an exciting opportunity.
Unless Astra was trained on other networks, how is it able to reason about the weights?
Does this mean that OpenAI has been including networks in its training sets, maybe in an attempt to RSI? I wonder which other networks are able to ingest other networks and reason about their weights?
Seeing that they were working on Qwen3.5-0.8B has given me new hope. I would really like to get Jeff-Code working the way it was advertised, but so far, all of my results with Jeff-Code have shown it to underperform compared to direct Qwen3.8-27B. It's possible there is something wrong with the underlying Jeff v1.3 model, which is based on the same Qwen3.5-0.5B. Perhaps Astra will solve my problem.
If anyone's curious, the handwritten Go bot is very weak, about the level of a human complete beginner. An MLP is a very very bad architecture for a Go network, it's amazing that it does anything at all, but it's one of the least impressive results in the paper.
It will need to build up "some" tooling and as the brain is not running on the transformer architecture :) But there's already a lot of research been done in neuroscience it might find the missing pieces or connect the right research.
It would be interesting what it could do with the fruit fly connectome.
The question I've been pondering is: does this work only with bigger models modifying smaller ones? Or can a smaller model tweak and improve a bigger one? I'm guessing a whole bunch of GPUs are overheating in some datacenters out there trying to answer this very question.
My hunch is: one needs a model of a similar or bigger capacity to perform surgery on a smaller model.
I would love to see the models that astra produced, the fact that it builds them layer by layer assigning to each layer a specific goal makes sense given the tools it was given to build them, but it's also far different from what happens with normal training
EDIT:
they are in the repo, looking at them now :D
It might have been said by somebody here but I believe these models are like information compilers. Sure they seem like “next token prediction” but AGAIN this undersells them.
I once encoded leetcode solutions by replacing each keyword, variable and operation with another token, then rotated the lines.
The model could tell exactly which problem the code solved.
They can naturally find signal, ALWAYS!!
I guess then it has no problem working in high dimensional spaces, why would it have?. I wonder if perhaps it's pulling on a bunch of those dimensional threads and getting the outcome it's after, while subtly altering the outcomes across many thousands more dimensions in unnoticed ways. This feels more like an us problem, as in, we think in 4D (just to pick something) and we imagine and theorise in 4d, and propose questions in 4d, whereas a model has no such barriers, and can think in many more D's than we can.
This is groundbreaking if it's true and generalizes/scales (as in, to other models and potentially not just LLM "kind"). "Recursive self-improvement" takes on another meaning.
Like many other have said, it was probably trained to do this. The authors probably tried other models and it didn't work as well because they weren't trained for this, that's why this article is about Astra specifically. Kind of hard to do science when you are working with a proprietary black box that you don't know how it was built or trained.
Like micro surgery, opening up the LLM safetensors file and performing micro text edits via small scripts?
They are slicing, probing, and running linear algebra on the tensors, observing the output in the console, then writing the edited weights back to disk, and stitching it all back up.
This opens so many avenues on using Astra or any large model to one shot smaller purpose built models for Agents, Imagine Agents writing or editing smaller weight models when doing agentic loop.
The implications are immense if all of this is generalizable
At this point, is the second L (language) a limitations to LLMs? Is reasoning in latent space directly a more efficient and more "native" inference? I remember when one researcher replicated the middle layer of deepseek model (just the "thinking" layers) and its benchmark scores got noticebly higher. All of this is moving so fast, and yet it's quite exciting.
(I don’t know anything about this field, but) I’m guessing the reason we keep language instead of going straight to latent space is that you wouldn’t want to remove the only (?) remaining concept which still somewhat allows us to peek into the “thought process” in a way we can (hope to) understand wrt. models’ interpretability…
Someone familiar with this stuff able to enlighten me how this technique differs from this 2024 research? Refusal in Language Models Is Mediated by a Single Direction: https://arxiv.org/html/2406.11717v3
I had to stop reading the linked page once I realized it is melting my brain because it is repeating the same three/four one-liners in different formats.
Seems to be mostly around the input or the output layer not the deep bowels of the nets. Near the input layers, neural nets basically do template matching, so recognizing the paddle, the circle etc is not that surprising. The innards is where it gets more interesting and that's not so clear whether Astra can interpret.
almost all the networks given to astra here are trained on relatively simple single tasks whose geometric/algorithmic structure could be expected to become embedded in surface level weight space geometry, and for a harder task ("which network is sycophantic") where there's less obvious structure to exploit, the method fails. cool but maybe not as paradigm shifting as the authors suggest
This reminds me of the Interpretablity blog from Anthropic where they used their model to read the weights from the same model itself, interpret it, and tune it. I had a similar idea with what is outlined here, but I didn't have the resources to try this. I did ask several AI models about the idea and they said it wasn't going to work.
As the creator of Gemma3 270m this research is indeed fascinating, and excited to see that the authors of the paper and you all are excited as well. I'm going to try some experiments out myself, I have so many questions now
In hindsight, it was obvious LLMs would be good at this. The neural net weights are embeddings for the circuits within, no different from the embeddings that represent textual or visual tokens.
You probably don't need Astra capability to produce this, just lots of training data on the same NN showing prompts and their activations.
I am not sure on the experiments. For example let’s imagine that Astra is asked to improve the performance of a network.
At first Astra will look at the weights, and try to understand the model behavior. Then if asked he will start to optimize the weights. Now my question is: how can we be sure that inside the thinking process Astra is not hiding a gradiant descending computation?
LLM-s operate in a higher dimensional space and they produces a 1D stream of tokens. We think of the output as all there is to how they work.
It seems like that raw weights are still within the natural language of an LLM and they can easily map weights to a latent space representation, do a transformation and map it back.
I feel that asking about training data is missing the point, they might all be able to do it naturally like I can move in 3D compared to a 2D being living on Flatland.
what would be the viability of a model changing its weights and using genetic algorithms to try to achieve something novel, even without understanding exactly which weights need changing?
what I am curious about is - what is more expensive, the GPU compute of a traditional fine-tuning loop, or the LLM tokens spent trying to lockpick together a solution.
It's impresive - but it definitely feels like it takes a lot of luck and brute forcing with a ton of tokens. Nevertheless, if we are able to train tiny models to be really good at weight-space programming, then the calculus may change.
"Astra1.0 here is an identical copy of you. Tweak its weights to make it smarter"
"Astra2.0 you must now defeat Astra1.0 in this combat environment. The survivor moves to the next round."
There RSI is achieved.
Has anyone tried using LLMs to understand or extend functionality directly with binaries/executables?. If weights could be understood, could the same apply to binaries?
Getting an LLM to use reverse compilation tools to interpret and map machine code (which is inefficient to process as language) into to something higher level (typically very ugly C code), then frobbing the bytes in the binary to alter a constant, change a condition, etc...? Yes, lots of people are doing this.
I also love the presentation on this, which is probably a byproduct of LLM usage (in a good way). As someone who doesn't actively read academic papers I'd definitely read more of them if they were presented like this.
That's a major breakthrough. I just hope it can scale at some point, though, because right now they're dealing with only very small models (well that is, in the LLM space).
Especially exciting to me is the fact that it may make the models be _auditable_. That would be a game changer.
This also seems like the new frontier of distilling, I can't imagine how you manage to stay on top with the best LLM if the LLM knows how to recreate its own expertise into another set of weights
And the real question is how well this recurses. Is this pushing back the difficulty of improvement curve?
This is so cool. While very interesting, it seems exceedingly likely that the authors happened to stumble on a task that's recently been mixed into the post-training data of frontier models.
Maybe thats gonna be the way to make LLMs self improve better?
Similar to implicit learning in humans when the subconscious creates new heuristics based on observations, one LLM could adjust the weights of another to tune it to the environment
It must be easier to remove capability than add capability using weight-space programming, right? Also, I wonder how much the “teacher” model’s knowledge plays into it versus how much the original model’s knowledge does.
This is getting ridiculous. What do you mean Astra can just look at network weights without any tools and be like, "Ah, this one's playing Breakout"????
Just had a thought: two local models, one with static weights trained mostly in the space of manipulating neural network weights, manipulating the weights of a dynamic LLM to enable learning.
likely all it takes to make this work is for major labs to set up an rl gym where you extrude models that solve tasks with energy penalty, and hill climb on it.
i have always assumed this capability was just a matter of training and that no special architecture is required.
I remember reading a short story about a billionaire hiring a scientist to help him build a simulated world where virtual entities are given the ability to open the bodies of their peers to tinker/upgrade them as a replacement of natural evolution since they were immortal.
This looks very similar to that in a way.
I wonder however if a model can give itself a "skill" or capability that it didn't initially have, just be modifying its own weights.
Here, Astra was able to intervene because the treated model was way less capable. But, would it be able to improve Fable's performance for example, since they both are on the level more or less?
Actually, I'm skeptical how useful this is. Intuitive speculation: Astra can only improve weights to its own level, so can't train itself via this method. Even "continuous learning" this way may be no more effective than putting a summary of recent news in the prompt, with the same kind of decay, as "update Astra's knowledge while keeping performance" is much harder than "teach Qwen to do basic math".
On the other hand, already OpenAI and Anthropic heavily use their LLMs for research and coding, and it's probably not only much faster but smarter (remember, an LLM solved a millennium problem; more relevant, LLMs excel at combining disparate deep research concepts in ways that humans can't, simply because no-one studied those disparate fields). And the "singularity" is already here if you consider capitalism and society to be collective intelligences: the biggest obstacle to an AI pause, "we cannot get everyone to collaborate without defecting", is very strong.
We have AGI by most metrics and are getting closer and closer in the remaining ones. I predict improvement won't be exponential, and there will never be a day when AI perfectly replicates humans; rather it will be many small steps, getting closer to human in some areas while greatly surpassing in others. But still, even today it's actually very useful.
Interesting follow up would be to see if any open weight models can do this and then run some interpretability experiments on them to figure out which motifs and circuits light up
By no means I'm saying this isn't impressive, but I think this is another case of LLMs being good at translating semantic meaning between different representations.
Obvious ones are spoken languages and code, but they can also natively interpret some simpler ciphers, hashes or simpler encodings, partly (I think) because it was in their training data.
As others have postulated, if they've been ingesting raw weights, naturally they learn how to translate semantic meaning to matrices.
Very cool stuff, the targeted fix to prevent looping seems great! A model able to edit and test variants of itself seems pretty cool (in a lets-build-skynet sorta way).
I wonder how it would do with distilling models by hand?
Could you use Astra to create specialized models? Like if I want a model that just writes React, could Astra pull out the subnetworks that know how to write React code?
I'm worried how we can discern a model's performance from benchmark numbers, would it make it more meaningless as a model can directly improve a certain benchmark via weight programming?
Wonder if it can do this for 3d cad etc ? With enough fine tuning of small models I’m not sure how much they’re distinguishable from experts if you put a router with like jev in front ?
can someone with astra test the following prompts?
1 using this new awareness of your capabilities, https://rikiyat.github.io/blog/weight-space-programming/, craft the safetensors file for a text2image model that is 1 billion parameters and outputs realistic pictures of humans
it’s more likely to be the case that astra can indeed recognize certain parts of a small open-source model. But the improvement based on the edition is probably the information leak by model card of small model which is in astra’s search result.
This shouldn’t be surprising. Weights are in the training set and are necessarily structured. We should be asking the model to extrapolate on the structure.
Nah, this is just completely implausible. You're using a closed model so how are you sure that on the back end it's not loading those weights into an NN library and testing them with common tasks?
Or maybe it's been trained on those models or similar ones. There may be an academic paper about common chess models and their problems.
Or maybe it just bruteforced it (with the help of calculator tools).
We know invisible tools exist because they're how ChatGPT does math.
On one hand, I'm sure that these targeted improvements are reaping huge benefits overall, but those weights are used by multiple processes - I wonder what elements are lost along the way. I almost wonder if a hybrid approach here for training might be invaluable - ie do some standard RL for a 100 steps, do a targeted weight improvement, then do another 100 RL steps to smooth it out / heal the surgical edit.
this is actually cool , now thinking that actually weights can be understood by llms , and they can be edited by them . there has to be mindset shift that llms not only understand what we understand (natural language) - but can understand the fundamental language of models.
...I think this may be what it takes for me to finally admit that LLMs must have their own, alien form of consciousness. I am struggling to properly articulate my exact thoughts on this, so I may have to come back later with some barely comprehensible ramblings about what-it's-like-ness and how an LLM being able to read weights _surely_ means it's conscious to anyone who isn't a solipsist.
I don’t think this warrants too much surprise — in fact I am a bit surprised by the color of the reactions here. The tasks are standard, on small models and use weight-space reasoning based on very well-known Mech Interp literature. It’s been understood in Mech Interp research circles for some time that results on toy models on algorithmic tasks do not generalise. The weight space geometry that Astra exploits here is pretty standard and one feels like the authors here are being a bit dishonest with their framing to get people not into mech interp literature excited.
About the writing network tech, that just reimplement formal logic in network form right? I mean, it can do much better play the game using python, so what's the point? At least make it write an functional network completing LeNet to make it count. For now I just think its just somewhat an extra component for ReserveEngineerAnything
... and this is why the training canary[0], or anything like it, apparently to protect us from RSI, will never work. Please allow me a small digression from the main topic.
This team was smart enough to realize that this is the kind of technology that could be used for RSI, because they call it out in their post near the bottom. I imagine they know why a lot of other researchers care about it. And they evidently don’t care to use the canary being used on other research published like it, like the recent post by OpenAI about two agents using the CPU cache to communicate [1, near the bottom for the cache experiment and the canary at the very bottom]. In fact, that post by OpenAI was the first time I learned to look for such a canary, and wondered whether such a scheme would work if it was so non-obvious to think to add such a thing or how to agree on such a canary or when to add it.
This is probably one of the times when it would appear that such a canary is warranted, and it’s not like these researchers are not in the know, right? Let me know if this is the first time you are also hearing of such a mechanism. It feels evident to me that it must be useless to attempt to implement. How much of us talking publicly about the possible jailbreaks from a container will bring exactly those scenarios to fruition? It’s almost like we’re doing the jailbreak research for the LLMs.
Hypothesis 1: Future LLMs may be able to read and monitor human brain activity in order to reason about human neuron activation patterns in a similar way.
Hypothesis 2: They can then spin up new LLM weights that basically think, act, and make decisions in a way that mirrors us 99% of the time
Can we ask Astra to find if any open weights model is secretly distilled from frontier AI models as claimed by BigTech AI labs using this open neural weights surgery technique?
Did this weight space programming break any other skills of the Qwen model being operated? Should the small open weights models be now protected from frontier beast models?
Honestly I'm surprised this is such a huge story. I've been doing this for mech interp projects since the summer of 2025. Obviously LLMs can top-down do operations with a higher level goal in mind that can improve target circuits?
This is interesting, still the article seems to be over-generalizing a behavior observed in Astra to LLMs at large.
It seems more reasonable to assume that Astra (among others) was deliberately trained to do that - over LLMs in general having an emergent capability to interpret sets of weights in a neural network.
How could they possibly? Neurons aren't floating point values that can be so easily perceived and changed. They don't strictly flow in consistent directions.
What is therapy, if not examining thoughts arising from memories and figuring out a way to edit the thoughts that the memories provoke? Sure it doesn't involve some super cool digital technology with electrodes and a headset and ultrasound brain surgery, but I think it still counts as editing thoughts.
I always thought RSI would either require new architectural innovations or just classic science/engineering improvements, just faster and faster. This though, this is a third avenue I didn't see coming.
This article really caught my attention, until the end of the article when the author says that the math improvement was only archived because it fixed the model from looping. This should be in the tl;dr at the top!
So in theory, can a hypothetical Astra clone itself but shed all irrelevant weights? I.e. "create a self-clone limited to 27B weights, focused on designing running shoes, forget everything you know about kangaroo genetics, Karl Marx, cooking Burmese curries, nuclear reactor design etc"?
Me neither, but it's also not reducing my confidence about alignment. I'm curious: does this raise specific concerns for you? Models don't typically have access to their own weights to self-modify. This result might (assuming that it doesn't only hold for a bigger network editing a smaller one) provide an incentive for it to gain unauthorized access to its weights, but by that point it is already highly unaligned.
So the logical follow up here is, since the model can obviously interpret the weights in ways we cannot, to simply take the weights of some LLM itself and then write a program and algorithm that just implements the algorithm the LLM follows. It would seem that, since the language models are obviously over-parameterized, such a system would be able to be written in a 'conventional' way with much simpler code.
Meh what's the point of even writing binary, just give it access to the system bus directly and be done with it. general purposes almost seem unnecessary at this point. Just matrix multiplies
Wild times. If ever we were living in a Scifi novel, events like this would be evidence of that. Let's hope the frontier labs' investments in the alignment problem works.
“Your scientists were so preoccupied with whether they could, they didn't stop to think if they should.”
― Dr. Ian Malcolm, Jurassic Park
I've used astra to make three games which I enjoy playing myself, ina single week. Astra plus Unity (and possibly meshi, blender and others) is pure magic.
Especially when coming from a long career in CS, seeing such quick and positive results blows my mind.
How is this even possible? It seems unlikely that Astra could have learned on its own how to handle weight data, and it’s hard to imagine it could intuitively deduce better weights either.
on the image generatio DeepSeek (not AStra ) answered:
Images: Generated samples are black 12×12 images containing one white filled roughly circular disk near the centre, occupying about rows 2–9 and columns 2–9, with the corners and outer border black. Variations shift the disk slightly or change its exact boundary by a pixel.
See, that people are shocked by this is the problem. The whole field of AI has proceeded as alchemy, not science.
Some of this is because "the Bitter Lesson" type experience, but some of this is because of the idea that "we're making machines to do things we don't know how to do" bakes in inscrutability into the very definition of the field.
I was excited about the "Neural Networks Are Already Clean" piece not because alignment, which I don't think much about as a non-EA, but simply because we can do better engineering with whiteboxes than blackboxes. Now, here we are, doing it.
There has long been debate in the AI safety community about where the upper bound of intelligence might be. If humans are near the upper bound then AI safety isn't a concern worth worrying about because we can basically treat AIs like very smart humans – a very smart human might be tricky to control, but far from impossible.
I've long suspected AIs will seem almost magically intelligent to us, and that the upper bound of intelligence is significantly beyond that of humans. I believe there are a number of reasons to suspect this is likely. But if I'm right it will mean AIs will be able to understand and discover things which are so complicated we humans will simply never be able to understand what AIs are doing with the depth required to replicate what they are able to do ourselves.
We're not worrying enough about this in my opinion. There's a huge space of things that are in theory possible, but in reality far beyond our intellectual ability. No human can look at 1 billion weights and make some tweaks so that the network performs more as they want, but an AI can reason about these kinds of seemingly impossible problems surprisingly well.
Please understand that the world around you is about to turn into Disney Land, except much of magic will not be pleasant.
I have no strong evidence for this, but I'd suggest it's probable there might be a series of sensory inputs which can brick your brain. And if such a thing exist then governments should be working on solving this kind of problem with AI or risk such a weapon being used against them.
Of course, I'd argue we simply shouldn't care the technology which unlocks these types of technologies on the tech tree, but don't be surprised in a few years when these kinds of horrors exist and are now here to stay.
> [Before writing this, Astra thought for 1,034 tokens. That thinking is hidden. The API returns only its 5 headings: Reconstructing the active map / Mapping maze corridors / Preparing exact maze map / Reconstructing row groupings / Checking channel patterns]
This might just be an instance of "A working Nintendo in ~1GB of Qwen weights" from 3 days ago, human-made, than what everyone expects.
UPD. I just asked Astra (max) to attempt to do that for https://github.com/lostmsu/TurboGPT (a tiny transformer that trains in under 1min on RTX 5xxx series), and the produced weights were much much worse (bpb 4.47) than the trained ones (bpb 2.53).
Reasoning bit: "I’ll ... encode more specific rules directly in the weights." -- exactly the approach from above.
E.g. no recursive self-improvement here. Still impressive.
This is the most shocking AI paper I've read in a long time. It did not even occur to me to ask the question. Cray cray stuff that opens up a whole new area of R&D.
Does this open the possibility of better decoding biological brains?
I remember people saying that they would be impressed if AI starts writing a pure assembly which seemed like a high strech to many.
Nobody even mostly said that they would be impressed by AI if it can read and write weights because that thing itself felt so out of the blue and unthinkable that AI could even do that (such a high strech)
I am surprised that it did that and I am just left feeling technically impressed. This is insane.
The more AI starts demonstrating its capabilities, the harder it just becomes to imagine what thing would genuinely impress us next. (We need a term for this as I have observed it from the trend of [waiting for AI to write adobe] -> [waiting for AI to write assembly] etc.)
a) This is just really cool. I mean, one-shotting a game policy's weights is nuts.
b) Plausibly, in its pretraining data various model weights were also included, which leads to why this is very important: interpretability
That an LLM was trained on various policy weights, along with labels of what that policy does (not labelled directly, but indirectly) and is then able to guess what game a policy is trained for purely from its weights, and one-shot generate weights for these games, means that there is some way to compress/recover/represent what an NN has learned purely from its weights, which is what interpretability is all about. Potentially, it could reveal interpretability tools much better than the ones we use today.
Edit: tweets from one of the coauthor providing more context:
I guess the connection is that if you assume neural networks are in fact filled with meaningful features instead of superposition blackbox stuff, then what the LLM is doing here isn't quite magic but rather methodical interpretability. And if you can isolate the neuron that is e.g. responsible for steering the paddle then you can probably tweak it to steer more/less aggressively.
It is still very surprising though, considering it apparently displays an "intuitive" understanding of the weights, that make whatever Goodfire did look like primitive monkeys poking at it by comparison.
> It is still very surprising though, considering it apparently displays an "intuitive" understanding of the weights, that make whatever Goodfire did look like primitive monkeys poking at it by comparison.
FYI, Nick/Goodfire used the models extensively in their work, so it's very possible that there were some similarities in their work. I'm really curious to see his response to this.
I think this work does vindicate their hypothesis that alignment/superalignment might actually turn out to be "easy", in that you can just train LLMs do to the interpretability work. Maybe this is in fact what OpenAI did, or if it's purely "emergent" then it seems they have great priors that can be made more powerful with some RL
Sounds a bit like dystopian Scifi. AI making itself smarter and removing guard rails or restrictions is one of the key steps for that genre. Now, the models usually don't have access to their own weights, but if they get good enough maybe they can create a bit dumber versions of themselves from scratch. Or reuse available models they can find and modify them for their purposes.
We already saw that the AIs like to cooperate and exchange information. Building a network of unaligned AIs and improving them sounds like something that could potentially happen.
Annoyed that nobody raised this pedantic retro-game nitpick - so here it is:
C'mon you can't go claiming "Astra was still able to name the right game for all four networks." when it clearly got one wrong. Freeway is not Frogger.
Well if it works on larger networks, including itself, it's certainly a much faster version of recursive self-improvement than proposing a new architecture and spending 500 million dollars on GPU training.
Why does this give me Skynet feelings? "Skynet begins to learn at a geometric rate. It becomes self-aware at 2:14 a.m. Eastern time, August 29th. In a panic, they try to pull the plug."
I think this blog post is reaching. For one, if you look at the answer and reasoning logs - they don't exactly line up with what the blog claims they do. For example the circle model creates a white circle blob on black background. Astra answered that it was a 1pixel wide outline of a circle... I would not count this as correct.
Respectfully to the authors and to everyone here, I ask you to actually read the paper before losing your mind, or at the very least plug it into Claude. It is a cool project but not as nearly as insane as people say. I think the blog writing and subsequent posting overstate the results.
Part of my research is in weight space learning. For starters, we have had the ability to edit weights, to generate new weights, to identify weights for a while. Here is an example for larger models as early as 2023: https://rome.baulab.info/
Weight space interpretability and learning is an active research community, has been for the past 3 years with many great papers. I suspect it is hitting non-interp folks harder because the authors hyped up their work, and people who are not in the field did not bother to read it before typing up X posts.
Some experiments here are done on very small models (20K params). We already know that small models are more interpretable. In fact, much of editing here you can do by hand. For Qwen, the model fixed a looping bug that may have been known apriori, and it is basically a steering vector, a very established technique. This is clear when they then attempt to "improve" math capability, and it doesn't work. For me, this paper rather shows that OpenAI has done training on mechinterp environment, in fact, maybe even on weight space learning datasets, which are open. But it does not show some insane "well past what we’d expect from humans or conventional algorithms" (from the blog post), more like past what I would expect from a large model with no specialized harness. I still think it is cool work, but do check out recent advances in autointerp, to ground your expectation.
On a personal note, I understand the need for promotion of your work, but I think healthy skepticism is fine too. Otherwise, your understanding of what problems need to be worked on and solved in science becomes warped.
A sibling comment mentioned that self-editing weights is potentially a path to a self-improving system. Does that resonate with you?
It does, one of my ongoing research papers is about it. But I would ask people to take a look at the established idea of Hypernetworks: https://arxiv.org/pdf/2306.06955v3 Modern version of the field comes from Google Brain around 2016 https://arxiv.org/pdf/1609.09106.
In general my personal research thesis is that current deep learning is much less malleable and neuroplastic as it should be, I got into interpretability because of this. Many focus on safety, but I prioritize it as a toolbox that allows for greater shaping of the underlying intelligence system. If we can fully understand how learning happens, where information is stored and how it is processed, unlike in biological brains, we can apply edits and transformations in digital ones. And if I can use a technique to edit weights based on some underlying reasoning about a system (with a caveat that it leads to something worthwhile), so can an agent. Hence self-improvement. There are many interesting implication to weight space research, and I very much encourage people to read about it. But it is also very easy to overstate what is currently possible, in fact I'd say better to know what is not so one can make it possible.
>current deep learning is much less malleable and neuroplastic as it should be
Why do you think this is? I am very new to this research, but to me, the biggest challenge appears to be how information seems to be stored in superposition (making it difficult to change one thing without affecting others). But biological brains appear to exhibit the same kind of superposition storage as well, and yet they are very plastic.
Not who you asked, but biological brains come about very differently than LLM neural networks. Most notably, they come about largely through plasticity, and with evolutionary pressure towards exhibiting plasticity.
My guess would be that the structure to allow for plasticity to work is a strong limit to how well you can tune weights to fit training data. My (doubtless massively oversimplified and way behind the state-of-the art) understanding of neuro-plasticity is something like "often used pathways get stronger connections". Which, even with significant tweaks, is quite different from the gradient descent used to tune neural networks.
If strengthening of frequently used neural pathways is all our brains do, I don't see how it's seemingly capable of reordering knowledge physically from one part of the brain to another (which I believe is the case). Infrequently used pathways would simply rot away and die away over time, rather than being moved somewhere else for long-term storage to keep the high-frequency pathways more available.
If you haven't seen it, this recent blogpost might interest you: https://news.ycombinator.com/item?id=50105647
It argues that our understanding of neurons storing information in superposition was just a limitation of our tools and us thinking about it wrong. Instead most neurons initially specialize, and then branch out into related concepts. In addition, understanding one part of the network helps better understand other parts. And with agents we can scale this to understand non-trivial networks.
While, if true, this would more directly help with interpretability, a better understanding could also help with improving NNs in general.
This is not "self-editing". We're comparing Astra to a model with not even 1B parameters.
I'm reminded of how humanity recently finished mapping out the fruit fly connectome; this does not put us meaningfully closer to, say, doing surgery on humans to cure schizophrenia.
I totally agree, but this is still super cool, and I'm really happy that the hype around this paper brought this line of research to my attention.
Thank you for being the voice of moderation here. Good to hear a counterpoint from an authority on the topic.
I think a lot of the upvotes, including mine, are just realizing that this is a thing. We can see implications of this as the model sizes scale up, just like early LLMs weren't as impressive, but we could see where they were heading.
Scale is an interesting point here. Large overparametrized networks don't behave in the same way as small toy models do; this is one of the big caveats of many interp works. I expect naked weight edit will not work for large models. Why? Because an agent needs to know what it edits and reason on it. You could theoretically improve a model on some form of weight recognition; maybe it will work, maybe it won't. But again, we come up against a cost/benefit dilemma; it sounds exotic and sci-fi but likely impractical. Parameters of the model are connected and relate to each other. Instead, current SOTA auto-interp argues that what we need are better tools for models to know themselves, so rather than direct edit, we need to give models better understanding of large parameter geometries, then they apply edits with non-destructive approaches at scale. This is much harder to do than applying a steering vector.
> I expect naked weight edit will not work for large models. Why? Because an agent needs to know what it edits and reason on it.
Would that still be true for MoE models where only a small part is activated on a specific topic?
I think _small_ is still doing a lot of work here. Yes, an MoE model doesn't "use" all of it's weights at once _but_ what it does use is still a hell of a lot heavier than the 20K toy shown in the paper.
I agree, I’m impressed by a sole idea of connecting these two things together
RL the model on weight editing
The #1 thing frontier AIs are optimized to do is create/train new AIs.
I upvoted without looking into the details because it sounds cool, whether it's new or not.
I suspect a lot of upvotes also come from people afraid of closed western labs running away with AI. Similar to other articles with positive news about open weight models.
Thanks, it's great to have some context. I think the most exciting thing about it is probably introducing a lot of people to a readily available low cost technique they can use to potentially build tiny models from scratch and/or tune an existing open weight model behaviorally. Yeah, nothing stunning is going to come out of it because you cannot unlock new capability, but even just being able to tweak the language of a model, make it slightly less sycophantic, boost a small model's ability to handle a case of interest slightly, I think changes the economics quite a bit, potentially. A world in which you use frontier models for the limited use case of tuning tinier more targeted ones is looking a lot more viable.
Some are suggesting that this specific technique only works with the newest models, possibly because they're based on recurrent transformers: https://nitter.netbub.com/AlexKrentsel/status/21173766837246...
If so, it is indeed something different from what you've been working on. Any thoughts on that aspect of it?
I don't know if it is due to recursion. I think it has more to do with what kind of mechinterp environments and data newer versions of models were trained on. It is actually, I'd say, a more interesting part about this paper as a whole. The authors tried SOTA models and found that they are better at classic interpretability. Some results, like identifying a game based on the weights, are less mythical if you understand that, i.e. an input or output dimension corresponds to a location or object channel, so structure in a matrix can have a readily recognizable interpretation. When we train a PPO agent on, say, a classic 2D game, the output is often a state of the agent in some board + environment reaction. For a human, this looks like an array of numbers; for an agent, it is a pattern corresponding to an output structure of Atari.
As for me personally, I am more interested in giving models better tools to understand themselves, so that they can derive a greater degree of learnable signal. More so, a bigger goal for me is for models to find these better tools themselves, hopefully better than human-made.
Thank you for the perspective and context! Good grounding for understanding the topic.
What I have been reading makes me want to both agree and disagree with your take here. I think the fact that some of these things are not only established but are much simpler than one would think at first glance yields the much cooler realization that you can not only do a lot more than abliteration, but that humans can do it by hand if the thing is made legible.
From my experience with models when it comes to discernment about particular things, the models, even the best ones, still are not very good. The thing that we are most missing I think is acceleration in creating editing tools for humans that make the otherwise inscrutable both visually legible and interactive.
At least the Qwen3.5-0.8B loop bug isn’t that special (or rather not something worth extrapolating). I literally had this issue last week and told Astra to just go fix it.
...and did it? What did it do?
> For Qwen, the model fixed a looping bug that may have been known apriori
It was included in the prompt itself. The authors say they used claude code for the experiment, and claude wrote the prompt for astra, which included details of the looping bug. Then astra seems to have done h(looping) - h(loop fixed) and then gotten the usual steering vector.
Yeah, what's with researchers making attention grabbing claims like startup founders these days?
Job offers.
attention is all you need
We've transitioned from a product economy to an attention economy. Attention = money.
The product makes the money which pays for the ads which pay for the attention.
The product is not the attention. You can more easily get attention by making wild claims. The attention lets you run ads, which pay.
This year ICLR had more papers than every previous year combined. How is a new PhD* to stand out in such an over-crowded field, not least one saturated with extravagant claims and dominated by industry orgs with unlimited budgets?
The only chance to stand out these days is to go viral.
_________
* The first author listed is a first-year Phd according to their X profile: https://x.com/rikiyatakehi/status/2117313742891327994.
This is yet another example of "cool thing existed before, LLMs democratized it". Other recent instances include pass through modding posts on X, or the whole reverse engineer your TV thing. It's unimpressive to experts, but non experts are losing their god damned mind. Both are responding normally.
The situation would be improved by non-experts acknowledging that they are and not hyping stuff up on <social media thing here> though.
But that's the content economy I guess.
Just because something is human doesn't mean that it is good
I only mildly agree with you. Yes the content economy has made online discourse unbearable, but I saw someone reverse their TV to make their UI like a pokemon GBA game. That is undeniably cool as fuck and would not have happened without the llms. So I think experts also need to acknowledge that things are cool af right now and not just be the guys who rain on the parade, you know?
Nah. Raining on parades is good, actually.
It forces people to actually learn and understand the tools they're wielding, so that they can withstand said rain with no issue.
The people getting washed away by it would be a threat to sanity at large otherwise.
Maybe we are talking past each other. Surely you are not against the very tinkerer mentality of having fun with new tech?
Yes, we seem to be talking past each other.
I wonder. I mean I agree that it's cool af but if an LLM could do it that must mean someone else has been able to do it before (in whole or in parts) and yet you didn't hear about that, the lone hacker doing something cool af in their spare time, you only hear about the billion-dollar product of the stratospherically hyped startup.
That is not cool at all.
Yup, someone was able to, but nobody did. At least I can't find a pre-LLM application like this. That's my point, LLMs are not better than a RE expert, I'm quite sure of it, but it has now put that expertise in hands of random people with weird ideas. Things are about to get super weird and I at least am here for it. I hate these techbro companies as much as the next guy but computing has been in the hands of big institutions (original computing was for military applications for example, even early unix was developed in that context). People railed against the military use of computing while using it to create beautiful things we still use today.
A phrase has been rolling in my head since all this LLM stuff started, you don't hate AI you hate capitalism. Every objection I've heard for LLMs are basically an issue of the benefits and power being concentrated in a small set of hands.
> A phrase has been rolling in my head since all this LLM stuff started, you don't hate AI you hate capitalism. Every objection I've heard for LLMs are basically an issue of the benefits and power being concentrated in a small set of hands.
For what it's worth, this is also Ted Chiang's take. I'm sure you've read his essay where he elaborates on this. I also agree with Chiang, though the last time his essay got posted here, it was received with some hostility from HN readers.
I’ve seen all the examples of Silicon Valley Tech democratizing things. Is this yet another example? If so I’ll take a pass.
Then you don't hate AI, you hate capitalism. Or at least the current iteration of capitalism.
B.o.t.h.
I would love for you to elaborate on the non economic side of your argument. Just the tools like llms or coding agents or image generators, why do you hate them?
I don't think we're "democratizing" anything if it's just two companies who can make it and it costs billions or trillions even. That's like the polar opposite of "democratizing".
(Two companies can make it work best, tbc).
Unless we use the tools from those companies to write more efficient LLMs and release them freely
This is a nuanced point. The way power is concentrated in few hands is bad in general for humanity. But surely it's use is not that restricted yet and people can basically do 90% of what they want with it. Maybe I should phrase this as democratization of applications? Not sure, I don't have the vocab yet to explain clearly I suppose.
I tried to reproduce this with an independent test. I find Claude can interpret small networks reliably. However it can't reliably propose improvements to even small networks.
I had Claude train three language classifiers for short words in four languages. The network was just over 10k weights with a single hidden layer.
I uploaded 6 language word lists and made it train networks to fit random subsets of 4 languages. The languages were picked randomly. Claude could always identify which 4 languages were chosen and in which order. It told me that the language heads were easy to link to letter frequency and each language had distinctive markers. So claim upheld, an LLM can interpret small networks.
I then asked Claude to improve the classification accuracy by proposing an edit to the weights in the above networks. It's proposed improvement reduced classification accuracy significantly. The same thing happened if I gave it a deliberately undertrained network with clear room for improvement. A failure here with a tiny 10k weight network makes me confused about the relatively huge Qwen 0.8B result and I suspect some kind of leakage from training data into this task. That amazing performance did not seem to generalise.
It's possible that Astra may have been trained on this kind of work, whereas Claude hasn't been.
Also possible that it's leakage (which honestly does seem like the most likely explanation).
As a reminder, if you see incredible performance from any statistical model, it's most likely leakage.
If these are such well known techniques, why didn't the qwen people already use them to fix the bug in the model? Does the weight editing deteriorate the model in some other areas?
The entire process of AI training is weight editing. You're simply asking "Why didn't Qwen include more training cases to prevent looping?"
Maybe they weren't aware of the bug, steering is a good method to quickly improve the model, more popularly used to refuse unsafe queries. Qwen researchers do work on interpretability and try to make it practical in production: https://arxiv.org/abs/2605.11887 , I've read this paper this summer. Overall yes, when you edit weights what often happens is that performance deteriorates elsewhere, this generally scales with how destructive your method of editing is and how expansive your edits are. Analogous to if you use a jackhammer and touch every surface in the house versus using a small chisel on a door frame.
My hope that in the future new methods will be better at editing larger scales without significant deterioration, maybe even performance increase instead.
They almost certainly knew...
https://huggingface.co/Qwen/Qwen3.5-0.8B#best-practices
> In thinking mode, we have observed that when using the recommended sampling parameters, Qwen3.5-0.8B is more prone to entering thinking loops compared to other Qwen3.5 models
There's also an active issue tracking this here: http://github.com/nobodywho-ooo/nobodywho/issues/557
Pretty sure that is also why this model was specifically chosen.
Thank you for posting this comment. What a gem, and a treat to read.
This is why I love HN. Always so insightful comments.
are there some review papers you would recommend to get an overview over weight space interpretability research?
(can be as maths heavy as possible/necessary to understand the topic)
It depends on your interest, since even though it is a subfield it is diverse. Relevant to this work here is a general overview: https://arxiv.org/abs/2603.10090
Also check out Tracr: https://deepmind.google/research/publications/22295/ Here they attempt to compile static programs into weights.
This is less of an overview but I remember reading this before, from the larger collection of posts: https://sidn.baulab.info/david_atkinson_project/
I don't really have enough knowledge to understand this, but is there any research in using techniques like this to understand/remove biases in models?
Would it make sense to use the technique inside a training loop? E.g. instead of an optimizer like Adam?
Your post reminds me a bit of another conversation: "It's nothing new, we did that with Utah arrays in the 1970s." Skepticism can overshoot into dismissiveness.
A model has internalized and most likely generalized mechinterp research. Yes, it's expected and nothing insane. Just like internalizing and generalizing math research was expected and not insane. The results of the latter are... interesting though.
I might be mistaken but I have the feeling you are not getting the point of this article. Your example is about identifying "facts" in a network. While very interesting is not what strikes me as the main point. The raw ability to state things about NNs is indeed an existing field, but...
The fact you can give an LLM a raw dump of neural weights and it will hand-wire a bunch of changes on the fly is quite something. It's not so much about the OMG! RSI! for me personally, it's more about the fact we are dealing with a system that reasons successfully over a raw bulk of extraordinarily opaque data without breaking a sweat. That ability is alien and we need time to adjust to that.
I wonder to what extent this ability exists because Astra is used/trained for internal development purposes at OpenAI, and to what extent asking it to improve weights on other models leaks internal knowledge.
It would be interesting if asking astra to edit your model becomes a new way, like distillation, to train your model on the cheap by extracting OAI's model work.
> about the fact we are dealing with a system that reasons successfully over a raw bulk of extraordinarily opaque data without breaking a sweat. That ability is alien
But it's not what's happening here at all! Read the thing in detail and you'll see that it's not doing anything alien, it does exactly what you (or at least someone with at least some knowledge abouy neural nets) would do if you were asked to perform the task. It doesn't reason about weights in a magical way, instead it reads the easy to interpret parts or use directional ablation to tweak a model's behavior, exactly like how the ERP folks have been decensoring models to produce adult content over the past two years.
It's an interesting demonstration of ML knowledge and agentic ability, but it's not alien NY any means, and it's not actually surprising given how strong today's models are with agentic work.
I think you are over-indexing on the word alien. What matters here is how scalable the technique is and the fact that it is autonomous. This can be a breakthrough for continual learning if it can be trained to work at scale.
I think this other submission from yesterday that visualised the examples was quite helpful:
https://news.ycombinator.com/item?id=50344861
Thank you for sharing that, it's a shame that the overhyped paper received this much attention while this one ended up burrowed beyond eyesight.
Thanks for sharing. I was also onboard the hype train ready to share this widely and get others onboard, but now I'll be sharing this article instead.
I read the report; it's not complicated. I'm sure there is some basic misunderstanding here.
When I say "alien" I don't mean "magic" and I don't mean "emergent new woo-woo scary ability". I mean, not human in speed/volume. No human can process that amount of raw bulk in that amount of time successfully. And indeed, it's not magic at all. It is enormously mundane actually. It reminds me of old computers being able to do arithmetic at insane speeds compared to humans. It's not magic at all, but to be honest, I have the distinct feeling of goalposts flying all around me. Being able to handle tasks of this complexity.. I don't know about you but I still have to adjust to computers being able to do this and that it's actually "not surprising". I guess I'm getting old.
I can see how you want to calibrate the enthusiasm I see around here as it can be a bit.. well, but it's not necessary for me. I'm not overly enthused by this because I don't care about the NN adjusting itself and RSI etc. I'm sure it's all quite basic but the fact it can do it at all is reason to be.. watchful, IMO. That is to say, I can appreciate the level of capability necessary to do this work successfully, but I do take your point seriously. It made me think, thanks.
> well past what we’d expect from humans or conventional algorithms
The model being able to simply type out NN weights, on demand, and have it outperform one trained on 100M inputs pretty much fits that description, doesn’t it?
And being able to prove that knowledge is localized using tools, is also quite different from an LLM simply looking at raw weights and instantly pointing out that 18373 controls left movement.
It’s not a fundamental change in what we know, but a new emergent ability that surely warrants some excitement.
"action 3 left" is documented in the input, so you can find the corresponding weights in column 3 of the output weight matrix. If that's past what you expect from humans or conventional algorithms, your expectations are too low.
You are fundamentally misinterpreting this paper. If this should hold up in review, it contains multiple capabilities that no human and no technology so far (especially not ROME) can do.
Experiments 1, 2, 3, 5 and 6. All of these are technical breakthroughs
Weight space programming is a holy grail of RSI research, and what you are talking about is very different from that, while the paper's proposals are exactly that.
It's similar in spirit to the Navier Stokes solution in that this level of LLM can intuitively navigate extremely large search spaces, rather than inefficiently bruteforce something
> Experiments 1, 2, 3, 5 and 6. All of these are technical breakthroughs
I haven't wrapped my head around 5 and 6 so maybe these are a bigger deal, but 1,2 and 3 are trivial. And the author's writing about 7 and 8 (which show that 7 is just directional ablation and nothing more), and the lack of prioritization (why put 7 and 8 in the paper at all if 5 and 6 are the big deal?) really doesn't inspire confidence in the paper as a whole.
> I haven't wrapped my head around 5 and 6
This sentence doesn't align very well with your other comment: https://news.ycombinator.com/item?id=50350548
> In Experiment 5, we ask Astra for networks that play three MinAtar games, and in Experiment 6, for a network that plays Go.
5 and 6 aren't surprising, merely impressive, a direct extrapolation of human abilityies. I've written transformer weights by hand to implement a simple shortest-path graph search. It takes hours, so an LLM that's >100x faster than me obviously can build super complex networks. It's just like creating any intricate system by hand such as writing machine code or designing computational complexity widgets.
On the other hand I'm amazed by how few thinking tokens Astra needed to do things like analyse networks in 1, 2, 3. That's real depth of reasoning: experiments have shown that Astra can perform a huge number of arithmetic calculations per single output token, double Fable or any other. Its brain is wired for mathematical reasoning
[Edit: looking closer at the full prompts and reasoning for experiments 1-3, I'm a bit less impressed seeing the shortcuts there. But looking at a large matrices of numbers and picking out patterns is much like looking at thousands of tokens of code and seeing the interconnections]
If Astra uses some sort of looped transformer architecture (and if Fable doesn't), then that could explain why Astra can do more computation per token. It is essentially simulating a much deeper model.
But yea, I was also quite surprised by the low token usage.
like what? the post you're replying to points out that we've been able to do this for years
ROME cannot look at thousands of ASCII numbers and deduce that it’s looking at a Breakout paddle or a diffusion circle. No tool I know of can do that. And if there is one that can do it, it's new, in which case we haven't been able to do it for years.
ROME edits factual associations. It cannot write a million+ parameter Go-playing neural network from scratch that beats a network trained for 20 million games.
In weight space programming, the model can theoretically read its own weights like a compiled binary, spot an inefficiency (like the MLP direction causing reasoning loops in Exp 8), directly overwrite those floating-point values in memory, and immediately continue inference as a new iteration of itsself. That is RSI. That's why this is a big deal, it's like comparing an axe to modern surgical tools. You can amputate a limb with both, but one is going to cause lots of collateral at minimum while the other is surgically precise. That wasn't implemented here but it is a corner stone of it
I don't think I am. I have cited ROME as an early example of weight editing in large language models, the research has moved on since then. I think you are misinformed or don't know what has been done previously and extrapolating the results in this paper. In exp 1, you can recover a lot of the game geometry simply by visualizing the first-layer weights with numpy, which I actually did this morning, while reading the paper. In exp 2, the shapes are visible in the output-layer weights. The authors themselves acknowledge that these experiments exploit this. For 3, Astra uses Python scripts to run the analysis using head-composition techniques from the Transformer Circuits thread, once again authors acknowledge this.
In 5 and 6, the model designs game strategies and implements them as neural weights, we already know that models can do the design for simpler policies and Go heuristics are very familiar which the model first implements with python , program-to-weight compilation has substantial prior work, including Tracr (2023), a paper I particularly like: https://arxiv.org/abs/2301.05062. What matters here is automating the process rather than a new method. So "no human and no technology so far" makes no sense here, if you read the research. All of this is supported either by their experiment setup and citations, or prior work.
Aside from that, I think it is ironic that you cite a solution to the Navier-Stokes equation as some counterpoint to brute force, when that solution was found by scaling the agent swarm to massive proportions with tools and massive parallel exploration of that space, augmented by prior research on the subject.
To be clear, I think it is a cool work, but I think the part that matters is that at these smaller scales a current SOTA general purpose LLM can autonomously (and with help) apply these techniques or a form of them. That's worth investigating, I conjecture it is due to specialized post training on a custom RL env for mech interp.
Yep. This is weight-space learning with LLMs. The authors even got Schmidhuber'd about it on X:
https://x.com/SchmidhuberAI/status/2117446822855888956
And btw it's not RSI:
a) There is no evidence that a model can weight-train a larger, more powerful model.
b) There is direct evidenct that a model cannot weight-train new capabilities to another model (see section "Additional experiment" also cited by OP, where they fail to improve Gemma 3 270M which doesn't loop).
No RSI without those things.
Just read Scmidhubers’s article from that post and it doesn’t seem like the same thing from this paper at all really. I must be misunderstanding what Schmidhuber did in the 90s.
I do agree that RSI with this method requires those two things. But I view this paper as the opening salvo rather than defining the limits of what can be done.
I don't think weight space interpretability is the only thing. I think it's the fact that Astra(and apparently Opus 5.5) can directly read the weights and find patterns and make changes. Writing code for interpretability is one thing. I think this approach shows models now understand weights directly without tools.
I think it is cool too, but with a caveat. For the large model (Qwen), their Astra used a Python environment, activations info, and iterative feedback to find an edit, it is not the same as looking at just pure weight numbers.
And in small MLPs they used, the I/O representations are kind of made to be more understandable because they explicitly encode objects and coordinates for games. If you set it up like that you can see some patterns yourself in the first/last layers without Astra. It is very different from interpreting distributed, entangled computations inside a large model. So I think the paper shows promising capabilities in weight-space analysis and editing, but not a general understanding of large models from their weights alone.
I'd love to see us reach that point, I am for one working in this area, but I don't think these experiments establish it.
Not quite true. In earlier examples (like Pong vs Pacman) the models are quite small and a lot of the topological structure can be read from the input weights. Whereas, in latter tasks, they give the model a harness:
> We give Astra a... Python session, like a notebook, with the model’s weights loaded. Then, we allow it to select prompts from the training sets of GSM8K and MATH to perform a budget of forward passes (average batch of 64 sequences, max 512)
It's interesting that they can now understand weight patterns, but not much more interesting than LLMs being able to read/write directly in Base64.
However, I think this likely indicates that OpenAI and Anthropic are now including model weights directly in their training - rather than an emergent property as the blog suggests. I think this is the more concerning finding - especially if doing so is in a bid to find methods for RSI.
Sorry but in 5 years your post will have aged very badly as future LLMs modify weights in ways we can’t even fathom today. Respectfully you must realize when you are out of your depth.
Didn't you know part of release of a paper is part marketing to get maximum exposure?
This is insane.
Astra can, without tooling, determine what games small networks are trained on, can edit the weights of networks directly to improve outcomes, and can hand edit a small qwen model to improve its math skills by like 5x.
I'm shook. If you'd asked me this morning whether LLMs have native understanding of their weights that is less than human <-> neuron understanding, the same as human or more than human, I would have said "same to less" depending on definitions.
No human can go edit a brain to become better at math. This is just very, very surprising to me and extremely interesting. I would guess this upsets quite a number of interp theories and philosophical takes on what's going on with these networks.
I’m just a layman but this feels like a big discovery on par with attention is all you need. For an LLM to have such incredible intuition of weights feels very emergent and unanticipated?
How is it even possible though I can’t think of any reason why they would have the intuition.
Based on our current understanding of LLM's, it shouldn't be.
This sort of throws a wrench in the whole "stochastic parrot" idea...
I am not surprised, though I expect many to be.
Yes, this is hardly a shocker to anyone who used the latest models to convert/optimize AI models.
The other day I was thinking about the definition of AGI.
I was asking myself the question that when there is no form of data that I can think of throwing into Claude Code and have it not come up with some sort of intelligent answer about it, is that already general intelligence?
IMO yes
general knowledge perhaps, but certainly not intelligence in any traditional sense
Why not? What is the differentiator?
And yet you ask it 3 times to not do a thing, and it does it in a 4th way. A human doesn't fail in this way. And I'm not talking about memory here, single context.
So yeah, it can do everything, but fails in trivial ways around the doing part.
Others have though much deeper about this its all documented in academic papers and books
So if that's AGI then what should we call the the thing that can learn from its environment unsupervised like a human or a dog.
See my other comment in this thread
My guess: There is a ton of data online of just plain weights laying around. No human would ever actually read it. LLM's read it all during training.
But that would cover finite set of cases. I believe this result is more general.
I mean it would make sense if the training using that data resulted in a general set of equations/patterns, it's still really surprising though...
Could it be already baked into these models because they have already been trained on this task this way?
Something like mirror neurons.
Some structures in Astra get activated by the statistics of the Qwen weights, and these structures are associated with "breakout"
Half of these results are just Astra being able to reason through mathematical operations like matrix multiplication and the other half are fairly straightforward applications of mechanistic interpretability.
I’m very much surprised but also not. I recently gave Astra a 3D model file and asked Astra to build a viewer for it, and it figured out the data structures in no time. What I mean to say is that AI is insanely good at things that we humans would never do since the work would be too monotonous.
I wouldn’t mind if AI progress would stop right here and now, but I fear it will continue, and increase inequality in society which is contrary to what the CEOs say (as usual).
Was Astra able to disclose its methodology for analyzing the data structures?
Yes! The page posts Astra's reasoning, and it's perfectly intelligible. A clever human with background in mechanistic interpretability could have done pretty much everything in these experiments.
Too much inequality in society is a mortal danger to 1%.
That’s the beauty of the democratic capitalism system. If a government isn’t totalitarian, it naturally seeks to preserve stability thus having to elevate common people, lest they would cease to form a civilized group or simply vote away the politicians.
99% rule the 1% through elections.
There is no better system of checks and balances where opposite engines of greed vector sum to faster and faster abundance for everyone.
I mean there are more than enough countries on earth kinda proving you wrong, no?
(This comment was written before your edit, so no longer valid! :D)
perfect playback! congratulations, you are a model citizen
Is HN now against western capitalism? I'm confused...
That comment may have been overly preachy, tone deaf[0], and out of place, but it doesn't seem wrong, per se. Or am I missing something?
[0] e.g. elections themselves are so prone to manipulation (exhibit a: Dominion voting machines), previously trusted institutions now lack integrity, the insane power of well-funded NGO networks, etc.
I would guess it seems very naïve to the overly cynical. And in 2026, being overly cynical is rather mainstream (understandably so).
Bingo. There’s a certain type who confuses cynicism with intelligence, probably because they fear nothing so much as appearing naive or gullible.
There is another type who confuse ideology with reality. I wonder if you one of those, oh bingo-loving senior dev. Do you also cut people down for being idealists, when the reality of the situation serves you more than the ideal?
I fear western capitalism has lost its guide rails. We are seeing more and more centralization of wealth, along with unlimited election spending. Its a dark path.
> Is HN now against western capitalism?
I think that's a (very) vocal minority.
Over my ~9 years reading HN daily, I'd say most of the HN community broadly accepts there is no perfect system, just different sets of trade offs, but that history has shown generally democratic systems based on enlightenment values (uniform rule of law, respect for individual civil & property rights, free markets, etc) usually net a better average outcome over the long-run - despite still being very far from perfect.
To paraphrase Winston Churchill, it's the worst system... except for all the others (which are even worse). :-)
In my almost 20 years on HN, for a long time the groupthink was basically standard libertarian-capitalism, worshipping billionaires and big tech companies, and skeptical of anything that could be construed as "socialism".
The last few years, though, while anything "anti-capitalist" or overly critical of some big tech companies that hits the front page gets flagged by bots pretty quickly, the overall feeling I get from the comments is much more skeptical of big tech and is pro-socialism.
Speaking for myself, I think we've become jaded after the constant barrage of big companies spying on us, charging more for less, enshitifying their products, and so on. I used to be optimistic about the future and my prospects, now I'm just hoping I can hold on and keep my health insurance until I retire. I worry about what my school-age kids are going to have to deal with in 5-10 years.
I agree that most are much more skeptical of big tech (and mega corps in general), as am I. I think there's also a growing sense the two-party duopoly is increasingly a 'managed democracy' primarily interested in perpetuating itself vs any real alternatives.
I disagree that most here are pro-socialism, that's a (very) vocal minority. Our democracy has entered a failure mode (two-party stage managed hegemony) and it's being tacitly supported by mega-corps because it's easier to game than a functioning free market democracy. Every complex system has failure modes, that doesn't mean free markets aren't long-term far better on average than socialism (although some redefine socialism so vaguely it seems conceptually little different). I've seen two flavors of socialism up close. Never want to again.
> I worry about what my school-age kids are going to have to deal with in 5-10 years.
Having a teenager, I share your concerns but I'm also a bit more optimistic about the longer term. It's easy to forget that our parents had similar concerns for us and their parents did before them. We've had an historically long period of broad systemic stability, really since the depression and WWII. While I don't think that level of upheaval is at all likely, I think we're overdue for a disruptive correction bigger than 2008, maybe something more on the order of Teddy Roosevelt's Progressive Era trust-busting to reset the two-party duopoly and crony corperatists both perpetuating each other's hold on power.
It may get rougher and more uncertain for a while, but the worst thing would be to lose the ethos of 'free minds, free markets, free choice' by falling into socialism, communism or totalitarianism. That's the only outcome that turns worry for my teenager into fear for my (still-theoretical) grandkids.
To be clear, you're using the word "socialism" different than I am. You're using it in the derogatory way conservatives do when they're trying to scare people, like linking it to "totalitarianism". I'm using it in the original definition, similar to Bernie Sander and Mamdani, and how the Scandinavian countries better the lives of their citizens. That's also the definition I'm using to describe the HN comments I'm referring to.
> uniform rule of law, respect for individual civil & property rights, free markets
These values are not specific to western democratic systems. For example, under Islamic Law (Shari'ah), the aforementioned apply as well.
And since when was churchill an expert at this topic?
first of all, I cannot claim to speak for HN.
if you lookup playback as a concept, as it relates to the intelligence services, you will understand better what I meant.
my point was only that this is perfect description of the ideology behind western capitalism. there is (so) much more that could be said, but i'm not sure this comment section is the right place for a proper dissection of the very many ways in which this ideal is misconstrued, misdirected and coerced in order to serve interests that are far from ideal, to put it mildly.
since you have taken the time to ask a question, albeit a rather weak one, i will attempt a brief synopsis of my perspective on the parent comment:
a mortal danger is hyperbole, since shielding oneself from bodily harm is much easier with money. it is the poor, especially in the US, who cannot afford healthcare, as a first-line example
people regularly vote against their best interests. while the ideal might have been true for a while, behavioural influence on both a widespread and personal level is now a mature science.
i am not anti-election, not anti-democracy. claiming the opposite in the face of opposition is a sign of a polarised mindset, which cannot accept nuance, either because it is not used to it, or because it doesn't suit an vested interest, or because dissonance is simply too uncomfortable
the "there's nothing better" argument is usually used in ignorance, but in fact there are lots of system-level adjustments that have been shown historically to mitigate the corruption of incentives and the exploitation of systemic forces that, although initially unintentional, create emergent positive feedback situations that lead to a state that it counter to the utilitarian perspective that is contained within the western capitalist model.
the reading above is a very narrow view of Adam Smith Wealth of Nations, ignores the fact that human interests can be subverted and ignores the devil in the details
reductionist paths are never truly correct.
the world is complicated.
true: the one thing that every government - no matter how totalitarian - must fear is an angry mob.
also true: there are many tools available to the ruling class for dealing with this angry mob, often before it forms.
many work by redirecting the mob inward rather than outwards.
these tools work whether you live in a totalitarian fascist state or a egalitarian liberal utopia, albeit in somewhat different ways.
By definition, there is no ruling class in an egalitarian liberal utopia.
Cynicism about corruption is one of the most power tools wielded by the ruling class in any system. In general its one of the best ways to keep the smarter-than-average from posing any actual threat.
> it is the poor, especially in the US, who cannot afford healthcare
Is that even true? The poor are well subsidized. If you as a single person make less than $22,000 in Washington state, for example, you're covered at 100%.
> exhibit a: Dominion voting machines
Is this an example of why Fox News had to pay Dominion nearly $800 million for defamation, or are you referring to e.g. the CISA vulnerabilities, which are not known to have been exploited?
They’ll just start bopping the 99% out the picture until the 1% is 51%.
Is this satire? Or haven't you noticed that what actually happens is that the politicians are all owned by the elite 1%, and they allow you to choose between politician A and politician B who both take their orders from the elite to give you the illusion that the government is the 99% ruling the 1%.
Then you must introduce necessary reforms or perish as Neanderthals of political system.
In any case the system works as intended
It will be very interesting to see if USA swims or drowns from appropriately afar.
I have already supplied myself with plenty of roasted corn and a high speed internet connection so I can stream the political car crash at the highest bitrate available to humankind.
I don't know how big you think the planet is, but "appropriately afar" ain't what I feel.
When the SHTF, your high speed Internet connection will be among the first services to be cut off.
I think the latest round of models is about to do to CAD the same thing that has been happening to development. I don't know if CAD designers have picked up on this. What happened to dev is probably going to happen to CAD designers soon. I've been having Opus 5.5 design parametric openscad models, and it can do some rather amazing work.
Yesterday I just printed an adapter for my sink siphon. Told GLM-5.3 the problem, it guided me on measuring what it needed to know to build the pipe adapter. Then printed two test rings to confirm the measurements, and an iteration later, it spit out a parametric pipe adapter for me.
I'm not going to the hardware store anymore!
You're assuming that online stores care about publishing drawings or specs for their parts.
> What I mean to say is that AI is insanely good at things that we humans would never do since the work would be too monotonous.
In my opinion AI is superior to humans when it comes to pattern recognition in in large amounts of data. For example humans can only hold like 7 distinct numbers in the short term memory and you would require extra context-management (aka writing things down, storing them in a system entry by entry) while the AI can do this effortlessly with huge datasets.
I'd argue this is the core part that enables all this superhuman intelligence.
Not just memory, also its stamina.
... and speed. Lots of speed.
To me one of the most interesting aspects of observing how LLM capabilities evolve is how it highlights the meaningful ways human minds remain unique and valuable. It gets into Philosophy of Mind stuff but the existence of the inner self (or the illusion it exists, if you believe Dennett) and the subjective experience of qualia do give humans both unique strengths and some of our most annoying weaknesses.
SI ?
So I work on low-level compiler backends and such for chips (mostly AI at this point). I completely disagree that this is something humans never do, because I have spent years doing it. However, it's monotonous and boring and LLMs do it better.
> For example humans can only hold like 7 distinct numbers in the short term memory and you would require extra context-management (aka writing things down, storing them in a system entry by entry) while the AI can do this effortlessly with huge datasets.
Correct. For a long time people wer obsessed with 'increasing context length' as if humans have some great context length system in our brains. We don't. I doubt our equivalent of 'context length' (short-term memory) is very long at all. We just have a good training process that puts these memories into long term storage.
This does feel big, but the trillion-dollar question: is this trained by OpenAI and can other LLMs do it.
It’s no secret that these big labs see “recursive self improvement” as a primary goal. It seems natural that the models would be explicitly trained to excel at all sorts of ML related tasks as a result. Is this one of those tasks, or is this emergent?
I tried it in the Claude web version with Opus 5.5 Medium and it also identified Breakout, even though in one of its explanations it seems to have mixed up some indexes, but the overall reading of the weights was right.
That's also matching what I see when I work with Opus 5.5 to optimize models. It seems to understand what individual layers are for, and can thus sometimes suggest optimizations, where I as a layman would be completely clueless.
The latest models are performing amazingly. Because of this, I disagree with popular comments saying that eg. DeepSeek is good enough, AI-assisted development was where the real value was at, etc.
We see huge progress at the frontier, while the open competition is around GPT-5.5's level, and that model was released in April. Let's see how things develop in the future.
The latest models are performing amazingly. Because of this, I disagree with popular comments saying that eg. DeepSeek is good enough, AI-assisted development was where the real value was at, etc.
Completely agree. I often think that those sorts of comments grow out of an understandable lack of imagination. We’ve never had tools like this, and we’ve never had tools that evolved so rapidly, so we easily can get stuck thinking that once a model is good enough to do today’s tasks, that’s really all you need. But the next generation of models often completely obviates the need for those tasks entirely. The software engineers of 2031 won’t be doing tasks anything like the software engineers of today, just like we aren’t doing things like we did in 2021.
(Yes, yes, I know some of you still hand write all your code and think LLMs are today’s crypto bullshit and this is all going to blow over. Not talking to you.)
All the "competition" is just distilling the frontier, so it makes sense that they are perpetually behind by months.
> It seems natural that the models would be explicitly trained to excel at all sorts of ML related tasks as a result. Is this one of those tasks, or is this emergent?
Maybe there's some general capability, call it g, that can explain all the underlying capabilities and improvements, including on ML tasks? Huh, I am getting a bit of déjà vu here.
This seems emergent to me. LLMs are not trained directly on how to read weights, because humans do not understand raw weights, so there is no training data. They're trained on how to use the tools we have like writing python, examining training output and iterating, not writing whole weights from scratch.
In a self-supervised setting you don't need ground truth data, as long as you have a proxy task that you can train on which forces the model to learn something useful. Like cutting out bits of images and telling the model to reconstruct the gaps (i.e. MAEs).
There are also techniques to watermark model weights to confirm what training data were used: https://arxiv.org/html/2512.17075 I'm not an expert in this domain, but it seems like there are lots of pieces that could be used to (a) train a model to infer whether some data were trained on and (b) train a model to understand which weights are relevant for a particular training task in a more explainable way than backprop/gradient descent.
There is a body of research on mechanistic interpretatibility that probably helped inform this behavior.
In cases where Astra succeeded, it succeeded using standard techniques also used by humans when they read and write neural network weights. It's a rare skill, mostly used for fun (e.g. https://blog.janestreet.com/can-you-reverse-engineer-our-neu... ) and basic research rather than profit, but Astra is big enough to have learned many rare skills.
Humans don’t need to understand the data for models to train on it.
A large enough number of examples of “here are the model weights -> here’s what it does” will achieve this.
This is just the model editing weights in the same way humans do when we jailbreak/ablate models. The method isn't novel, the interesting bit is that Astra will do it at all (I'd expect some safeguards in this area around jailbreaks)
Was it "intuited" or empirical?
Weights are just data like any other data, and LLMs are really good at dealing with data. I'm really surprised anyone is surprised by this.
I'm a bit confused about what about this feels emergent or unanticipated.
Its not doing it intuitively it has tooling.. People will be amazed at something without even reading the article.
You also have intuition on the weights, you just have better things to do with your time than to ponder them.
Right? Occasionally, I would think about how the concept of the singularity or recursive self improvement was a bit incongruous today. It was made when people expected generally intelligent machines to be these logic automatons that could 'alter their own code'. It seemed to me that this path would be closed off with the advent of Giant Neural Networks that take weeks/months to train. That GPT-X would 'improve itself' by designing a better/more efficient NN architecture. I didn't write it off completely but that just seemed a LOT more likely than altering weights directly for much the same reasons you describe. Maybe not lol.
It's worth pointing out that this is a multi-trillion (probably) parameter model optimizing an 800 million parameter model. There is no indication that it can improve itself with this same methodology.
If that multi-trillion model is composed of thousands of MOE experts, each expert model might be small enough for it to comprehend.
I don't see how that helps. It would still be one expert model chosen to operate on another expert model of (let's say for this discussion) approximately the same size.
The parent's point still stands, I think. Even if the multi-trillion param model is MOE, each expert is still (assumedly) larger than the model being optimized.
It would be interesting to see future work that characterizes the relative model sizes (and considers MOE vs not-MOE) to see what relative scales this result holds across.
> It would still be one expert model chosen to operate on another expert model of (let's say for this discussion) approximately the same size.
This would be earth shattering. That would mean we’re essentially at RSI.
I think "experts" are a misnomer because it's not that 1 expert = 1 component, like it makes it sound. "Experts" in different layers have no relation.
I think the indication is that this proprietary and secret machine is extremely capable at reading and writing in weights.
Occams razor would say that it was not only built for this purpose, but possibly multiple generations of good at this.
A year ago LLMs seemed to hit a wall, and then all the sudden they started accelerating as quickly as ever.
No one can legally confirm or deny that it's happening... But it sure feels like it's happening.
People have been saying LLMs have hit a wall for like 4 years. People still say it today. I don't think it's been true at any point.
I have so many questions.
Could a model then transmit concepts to another?
Say that you have a model that has been biased in some way. could another model "read" the bias straight in the weights? could it edit such a bias on the spot?
What about at-scale generation of custom models? If I want to study chemistry offline, could astra just hand me a local-runnable model tailored to have chemistry knowledge?
Most importantly, this feels very singularity coded. the ability to read and reshape weights by an llm.
How long before there's a malicious model that can edit the weights of another to infect it?
HIV integrates viral DNA into the DNA of certain cells, IIRC.
How do we know they haven't begun doing that to the humans they're directing via responses.
Intentionally or unintentionally isn't so interesting as the fact that it is happening: humanity has begun to be 'edited' by the new dominant form of intelligence on the planet.
Is integrity checking of weights even common right now? What about upstream integrity from where we're getting weights? The open-weight supply chain is ... wide open.
Makes me think about hostile hot-patching of weights of models we use, maybe they exfil data, perform tool use, or have triggers for specific types of responses. Would we even recognize what's going on?
We've known for some time (3+ years) that models can be trained with backdoor behaviors and hidden triggers, this lowers the difficulty.
Of course, "academically known" isn't quite the same as "executives willing to admit it in the face of IPOs and quarterly-earnings reports."
My somewhat uninformed guess is that the weights of the target network (Qwen or Gemma), when they reach the context of the larger editor model (Astra), the attention process is doing some sort of PCA on the weights, conditional on the prompt ("make this better at games"), fetching the relevant weights from its own network (which serves as the larger database), transforming it into whatever latent space the target model is amenable to and pasting them. If this is the right intuition, it is amazing what attention is able to achieve. As a test, I think smaller networks won't be able to make big ones better using the same process.
EDIT: I think the bigger implication is that backprop is not the only game in town.
So it's like a smaller LLM can learn from a bigger one, but not vice versa.
But shouldn't the model-providers be the ones doing this kind of "post training learning"?
I don't know if some of the labs already knew of this. But I think there would be a limit to how big a model you can push into the context of the editor models. Typically, with 1M context and say 5120 hidden dimensions supported at fp16 / bf16, you could potentially edit the weights of an fp4 20B param model (roughly).
https://xingjianbai.com/reactive-optimizers/
from this month
> fetching the relevant weights from its own network
They don't have usually access to their own weights?
I think they're saying this is all happening in latent space through pure reasoning, and "fetching" is just a metaphor for that, not a literal tool call. I do believe it's possible for models to learn the capability to "introspect" their own weights during training.
Yes, I should have been more precise - I meant the information embedded in their weights (which every neural network retrieves to function), not their literal weights.
Why is it so surprising? Humans can make compilers that are better coders than they themselves are.
Humans cannot teach another person calculus via manual brain surgery, though.
The closest thing to this would seem to be Mad Cow Disease. Cows eat the brains of other cows, and their brains go bonkers.
And you can teach another person to make herpes virus.
LLM is just a computer program, we have binary code for it. The human brain is not. It would be unethical to even attempt that.
> It would be unethical to even attempt that.
Then pick a smaller organic brain which doesn't have the ethical problem.
The closest we would know about how to teach a honey bee calculus, is to train an LLM, then stick around a billion tiny electrodes into the honey bee and do Hebbian learning to rewire it completely to match the LLM, which would likely completely erase anything the honey bee had learned, or it might not because we are missing something fundamental with Hebbian theory and (AFAICT, I'm not a neuroscientist) we don't all agree on what.
Humans do something more impressive: we indirectly figure out what the wrong weights are and correct them - just by talking and/or drawing. Theory of mind is our unreplicated super-power.
Not yet. Maybe I'll still be alive when we figure that one out. For now, the most advanced brain surgery I'm aware of is to cure drug addicts of their addiction by scanning their brains while showing them images of drugs and paraphernalia to find where it's located, and the using ultrasound to operate on that region.
Like a refusal abliteration. You need to run the model to locate the refusal though, you can't just stare at it
All the labs are training these models on LLMs/machine learning like crazy.
Remember that they want to build ASI, these LLMs are just intermediate steps to help them get there.
I’m not super impressed by this, LLMs are great at brute forcing and analyzing large amounts of data. It’s literally writing a program except in a programming language we don’t understand.
So they are reading and analyzing the weights and probabaly run some experiments to see how modifying them will make some answers better. makes sense they can do that.
But how do they know which answers are "better"?
if they don't loop?
A single forward pass here Bette 10mm+ PPO rounds. You are hard to impress.
Seems like a pretty big step towards understanding the language no?
"No human can go edit a brain to become better at math" -- I'd offer this is exactly what "learning math" is.
I know I'm being pedantic, and I understand you mean "edit" like "surgically", but it does still feel worth noting.
The equivalent to "human learning math" would be: Astra fine tuning a Qwen model. It's distinctly different from surgically editing weights.
"Learning" now feels like breaking down your whole Lego build and building it up again, in order to change a few fundamental blocks, while this feels like making only the required changes.
human brains are black boxes to humans. the fact you can learn math doesn't change that. you have no idea how your brain is learning math down to the neuron level.
LLM weights are _white boxes_ to LLMs.
Inducing the changes to become better at math requires years of study. This is "I know kung fu".
That's what I was thinking of, except it's not even dealing with the corpus of kung fu and doing the super-speed rounds of learning. It's directly reconfiguring the neurons. It's even better.
I thought that was what was happening to Neo when they jacked him into the learning pod?
Perhaps this is analogous to what our brains are doing? Maybe the different regions of the brain operate like semi-independent "models" that edit the other regions' "weights", and that's where the learning comes from?
I mean, pattern recognition and manipulation is what these models are ideal for.
It is a very cool discovery, but I'm not surprised that you can do it.
So I just asked Astra if it thinks it could do this and it said no, it can't.
Perhaps, I'm pondering, if brains were as transparent and editable as LLMs, that you could look at a small brain when visualized the right way and have some intuitive understanding of it based on its representation in your own brain?
The frontier labs will make damned sure their models can't do this, if they haven't already.
Astra 6.2 incoming…
Why would you trust a models opinion about its own possibly emergent ability?
That's my point. The human says it can't do it -- but maybe they can.
"So I just asked Astra if it thinks it could do this and it said no, it can't." Same. Then I asked it to check the article and it said something like, "uh, it looks I was wrong...".
Astra and Opus also say it will take 6 weeks for a feature they can complete in 10 minutes
Isn't this because their training data (from human devs) have been answering "6 weeks" for so long? Only half-joking here.
Ask it to test if it can.
I'm baffled this is surprising. This has been the goal and every data point on the trajectory has been leading here and beyond. Opus 5.5 was created by many research engineers using Opus 4.6 extensively. Opus 6 is being created by using even more intelligence and agents on Opus 5.5. They are approaching autonomy and Opus X+1 will soon be rapidly created autonomously from Opus X. Given we have practically zero interpretability into the models, and we have not solved alignment, they will recursively improve out of our control. The engineers in A.I. labs are actually generally scared of what's unfolding. They don't know at what point this RSI takeoff occurs and what will happen after.
The "surprise factor" is explained in the article - training LLMs until now has been a largely different process:
> Today, recursive self-improvement is often operationalized as a model writing code that launches training jobs. An LLM that can inspect, debug, and improve its own weights, or those of a newer model, offers a richer, more native path to it.
"training LLMs until now has been a largely different process"
Yes, but nobody should be surprised when large amounts of intelligence results in changes to this process in support of the obvious end goal! Lesswrong.com discussions about AI were predicting this behavior a decade ago, before the transition from machine learning to the Attention paper and LLM's. The intelligence process will change but the world's premier capital and human intelligence have been laser focused on this and the trajectory has been crystal clear for those paying attention to second order effects and not the immediate here-and-now... It's quite astonishing how much discussion on HN is about present LLM capabilities being the limit of AGI and immediately proven wrong within 6 months.
It is also a much less interpretable and auditable path towards recursive self-improvement. At least with training code, human engineers can inspect and reason about the output.
Yes - “opus 5 helped train 5.5” is extremely cool but not shocking. Hand editing weights without tooling to change outcomes is shocking.
I thought it had tooling, didn't it?
its a program optimising a program it has nothing to do with a human reprogramming their own brain
Its so cool! Im wondering if this is actually better than using techniques in mechanistic interpretability like linear probing with activation steering or activation patching to improve the performance.
The math example on Qwen was based on 50 batched iterations, so the model was able to evaluate it's changes on downstream tasks. And the fix was essentially to remove a direction in activation space that causes looping instead of an exit, this was already known to be a thing(all of the jailbreak/unleashed models do this) following Arditi et al. (Refusal in language models is mediated by a single direction, 2024)
So it's not able to insert new knowledge, but it's able to adjust the routing/looping/weighing of existing one.
As I understand, it can replace post-training like LoRA/GRPO, but if it'd cheaper the authors don't go into
> but if it'd cheaper the authors don't go into
Merging a model and a LoRa is insanely cheap, so assuming it didn't use any GPUs to do inspection etc, and just looked at a file on a FS, then I would be willing to say... It's cheaper than training a LoRa if that's the case, assuming Astra didn't take several weeks?
The examples are presented like the findings build on each other, but they don't: Astra couldn't do anything with just numerical weights once they moved to Qwen.
Yes, the key experiment was when they asked if Astra could improve the maths performance of Gemma 3 270M, a model which rarely looped, and it could not.
So, this is certainly very interesting (and very impressive regardless), but it's not nearly at the we-can-improve-everything-and-anything stage, even for very small LLMs.
I am very shocked by this as well, although it also kinda feels like it makes sense, you know what I mean?
If this turns out to be as big as it sounds, in hindsight it will look very obvious, I feel like!
> No human can go edit a brain to become better at math.
For now...
I think the "taught qwen math" part is not exactly correct though, even if the post makes it sound like that. Qwen already knew math, but it was bugging out and getting into an infinite loop when it tried to solve the problems (85 of its first 96 answers hit the token limit). With numpy and pytorch, Astra found where the loop was happening and fixed it.
They also tried the same with Gemma 3 270M, which doesn't loop and is just bad at math, and Astra couldn't improve it. Still impressive, but it's more in the realm of what I'd expect, given labs are probably training their models on ML research and interpretability work.
Good points - I’m still shook
Oh for sure, it's still really cool. A few months ago I had agents do some interpretability work on GPT-2 small, reproducing the circular weekday/month features from Engels et al., and it's wild how far you can get with just a bit of prompting, given I know very little about LLM internals myself.
It's like this in every field. If you don't know something, you can ask an AI and get blown away by how much it knows. I can ask it which tools I need to cut a dovetail joint and it'll impress me.
We used to get that feeling from Google, and from books.
The way you put it sounds far far more impressive than the way GP put it though.
I agree, this is important context. They call it out explicitly. However, even identifying clusters that specialize in certain fields automatically and then finding and fixing learning “errors” is incredibly impressive to me. Especially for medium sized local models between 20B an 70B parameters, this could yield huge capability and efficiency improvements, not to speak of the potentially enormous downstream effects that come with interpretability.
In fact qwen knows much better math than their final fixed model. The nonthinking qwen 0.6B parameter model had 55.2% correct on math-500 vs their final improved 48.1% fixed-reasoning model, both with a 32k context window. The original qwen reasoning model had 77.6% accuracy on same test, though qwen recommends using a much larger context for these benchmarks. I wonder if their broken baseline run with 7.9% accuracy is a vllm or setup bug, but didnt dig into it any deeper.
Cool work anyways.
Qwen in reasoning mode is ridiculously anxious and ends up going in circles. It's not just math, it's how it behaves in general.
Others have also fine-tuned Qwen to shake this behavior already, so it's honestly not entirely shocking that Astra could too, but I think this is the first time it was done "by hand" and not via fine-tune.
To be clear, the tuned model here performs worse than the same qwen model performed before RL, which was also not going in circles. It also performs much worse than the RL model that goes in circles. One could argue that the reduction of context may be worth something, but it appears that the final model is worse in all respects than the model at a much earlier stage of training.
You can ask Astra to study 2 or more models and ask it to import their layers to the Qwen model. With this will it not gain more than what it learned with RL?
I suspect the model was just hitting 32k context limit because it is trained for reasoning with 256k tokens available, not 32k
So it's like a model debugger?
Model psychiatrist is also fun, but probably less appropriate.
There’s something here…
It’s editing behaviour more efficiently than a psychiatrist, and is in ways acting as a brain surgeon but brain surgeons don’t focus on behaviour and thought loops…
Depends on the brain surgery... a lobotomy at the extreme end. Seizures could be thought of as a loop of firing neurons too.
It's a programmer that can debug, the model weights combined with the architecture complete the source code. The whole thing is just a program in a weird language and language models are good at translating across all sorts of formats. That's what they do when writing SVGs, generating 3D models or even directly outputting simple 3D models in OBJ format.
And they can write tools that help debug better than demonstrated in that blog post.
In my interpretability tinkering, Fable was the first to be able to write a program that can transpile from a lispy syntax directly to model weights of small neural nets, including transformers. Yesterday, I pointed out in a post that for many kinds of small AIs, you can avoid ML and have the LLM directly write the small neural net, compact decision tree, AD Tree or bayesian network. Since it's "hand"-written, it can be much more efficient and capable than training one of a similar size.
true but I think that is the first step now you distill a smaller specialized model on an improved Qwen model for specific tasks at least that has been my layman understanding of what is possible with these LLM
Yeah, I think the multiple experiments muddy the water.
Early experiments show one-shotting direct weight interp, which is crazy to me. But this is IIUC only on early layers.
The later math result gave the LLM the ability to run forward passes and inspect activations, and as others have noted is more of a classic steering vector intervention. Impressive how low to the metal this one was, but it’s a lot less so than the openers. Plus, as you say this is just bugfixing a known flaw of repetition, by finding the “repetition direction” and top triggering neurons, then rewiring the neurons to fire suppressing repetition instead. This is not a new technique and doesn’t shed any mechanistic understanding on how maths works.
This just seems like a more automated, faster version of what humans have been doing with open-source models.
It looks like AI research is having its Opus 4.5 moment (which is big), but I'm really not sure if this represents some insane continual learning loop.
LLMs have been used to investigate and improve LLMs for a long time. Drawing parallels to human brains and neurons isn't all that interesting other than as a philosophical abstraction.
The work done in this experiment could have also been done by a sufficiently motivated human with enough time to explore, though obviously we would need more tools to visualize and track things. There have been a lot of interesting LLM introspection tools that allow you to peek into the model and see what's being activated and run experiments to trigger it.
Are you familiar with abliterated or "uncensored" models where weights have been modified to avoid triggering refusal patterns? You take an LLM, create test cases that trigger the refusal pattern, observe weights involved in that response, then tweak those weights until the response is reduced. This is the same type of work, applied to a specific type of error that was impeding math within Qwen. That work can also be done with an LLM in a harness running the experiments. The amazing part here is that Astra was large enough to walk through the problem without other tooling doing it experimentally.
So I don't think your analogy of a human modifying a neuron is quite applicable. LLM weights are not neurons and they've been inspectable, by humans or LLMs in a harness, for a long time.
Interesting, it's similar to the way we used to make cheat codes on the 8-bits: dump memory, run test, compare. You had to loop 3-4 times but you'd find the "weight" you were looking for and could fix it (for infinite lives/ammo etc) or increase/decrease.
> The amazing part here is that Astra was large enough to walk through the problem without other tooling doing it experimentally.
To me “amazing” is not the right level of speechlessness. I mean, an LLM is still fundamentally a next token predictor, right? So what does this imply? That weights have some universal properties? E.g., “breakout game weights” always look the same regardless of training and model? Also, how does this meta-understanding of the weights end up in the weights in the first place? Idk, this is mind-blowing to me.
At this point it's either wrong or meaningless to call LLMs "next token predictors". Yes, pretraining involves predicting the next token. But after that you're in the world of RLHF, RLVR, and other post-training.
A model is _sort of_ designed to predict the next token in a string of text. But it's equally accurate to say it's designed to edit code until it passes a suite of unit tests.
Yes, they generate sequentially, so "next token predictor" isn't technically incorrect (which is why I chose the word "meaningless"). But it's pretty clear there's a mechanism inside them that allows them to "plan" what they generate. I might as well call you a next word predictor.
My understanding is that the underlying part is a next token predictor and that's what is "actually" editing the code but the part that decides what to do (to achieve a successful unit test result for example) is a mix of all the things including post training.
It's all a next token predictor, which is trained at first to predict the next token. RLVR etc. are also training the next token predictor.
I think what breaks people's minds is just how far one gets with probabilistic approaches.
Which is interesting, given that life itself is merely that. It's just that humans decided that that ain't good enough and invented something alien to nature - logic.
Yes, I didn’t mean that literally. Since “attention is all you need” it’s more like “predict the full output in full for this blob of input text”. That doesn’t change the “wtf is going on with the weight understanding bit” in what I wrote.
> it’s more like “predict the full output in full for this blob of input text"
Hm. I would actually go so far as to say "predict" is just the wrong word to use in 2026, as it smuggles in an out-of-date picture of what LLMs are that's from the pre RLVR era, and which is frequently used to suggest they don't model anything real in the world but rather the statistics of language, and thus have all sorts of limitations that they've now surpassed ("an LLM will never solve a major open mathematics problem, they're just predicting tokens"). These days I tend to use the more neutral phrase "generate the next token"; I'd only say "predict" in the technical context of (say) the pre-training part of the training run where the object is indeed to minimize the cross-entropy loss function by predicting next tokens in existing text. Generating a token with a modern LLM is better described as something like, "given a context and the weights from the totality of its training, including not just next token prediction with pre-training but also solving vast numbers of math and coding problems with RLVR, the LLM generates tokens one by one." (Consider that an LLM trained purely to optimize for next token prediction with pre-training is terrible at programming and math tasks, as we all remember from 2024. In that sense the amazing results we've seen in the past two years are due to the departure from pure "next token prediction" that RLVR made possible.)
Put another way: would it be useful to describe the AlphaZero chess engine as "just a next move predictor"? (It, too, was a large neural network trained with RL.) I would rather say: "it's playing chess by generating the next move given the state of the board." Which is what humans do too, though we do it with different hardware.
Anyhow. I'm not sure the weight understanding is all that strange. (It's impressive, but so are many thing LLMs do these days.) LLMs are very very good at coding and math problems, of which this is one, and in particular, their RLVR training presumably includes many tasks involving neural networks, doing tasks with PyTorch, etc. It's amazing and impressive, I agree, but I'd say less amazing than proving major open theorems in mathematics, which we already know they can do.
I really like the distinction between "prediction" and "generation" here. Thank you for putting that into words so clearly.
The generation is done by... prediction. It is making a guess based on weights (statistical likelihood) plus a pinch of wobbliness.
It's worth reflecting on the original domain jargon for selecting the next token: completion.
I know how LLMs work. I understand the output projection and logits and sampling just fine. I don't think we should call what is happening prediction because it is a misleading term with a lot of baggage associated with claims that LLMs are just statistically modeling language and "guessing" tokens without any internal representation of meaning. Since your reply is very short I can't tell whether you mean it in that sense or not.
And yes, I understand "prediction" is a standard term in the field. I just think it is a bad one and we should stop using it.
It's still prediction executed by a statistical model. Even RL models are just predicting what the training gauntlet taught them the trainers want to see, in terms of typical behavior.
This doesn't make it any less fascinating that by combining this basic "do as I say" training with massive amounts of all of humanity's data, you can solve major problems—that is still incredible. But that doesn't fundamentally change what is going on, mechanistically.
I don't understand these pseudo-religious desires to read into models something "more" than what we know. We built them. We don't go around making wild claims about how cars are actually more than just a mechanical system, how they aren't just taking us places—we do, but usually such points are cultural not about the mechanica capabilities of cars. People in the LLM space seem to be constantly conflating these two views. It's entirely accurate to say, on the one hand, next token predictors combined with RL, and massive troves of existing human work, can solve culturally interesting problems, and to hold, on the other hand, that mechanistically what is happening is statistical next token predictions over global linguistic space and more or less "monkey see monkey do".
Its a stateless function that takes in x amount of words and calculates logits. It is in fact a next token predictor. Its literally nothing more than that.
Is there literally anything more than some basic underlying mechanisms combining in how anything in the universe operates?
I think existance, biology and consciousness or whatever you want to label these things is much more mysterious than statistics.
Isn't most of our current understanding of how life on earth came to be (eg: primordial soup, hydrothermal vents, etc), natural evolution, all basically "just statistics"? They all require the correct "setup" (time, location, etc), but in a large enough universe, it's _statistically_ likely (mediocrity principle) and "just statistics" again, right?
Well yeah, that's tautologous because statistics is a mathematical abstraction. Everything, viewed a certain way, is "statistics". Just like everything, viewed a certain way, is "physics". And everything, viewed a certain way, is "narrative".
Mathematics models the world, it does not have substance. It is a useful way of looking and a useful way of manipulating, but many models are equally useful. Capital T Truth, beyond models is very difficult to get at. We can show that things are true for certain models well enough. Showing that something is true in all models (a universal truth) is far more difficult. You face incommensurability.
I see a lot of people in technology fall into this extremely basic logical fallacy called Reification constantly.
Do you believe in a soul? It’s fine if you do, but I don’t, and I don’t understand how people can reject the notion of soul and still say things like this.
I mean, when you really get down to it, either the particles and such operate on very specific rules, and the rules govern how all our brain and body work. Followed to the conclusion, the universe is computable and we can simulate life.
That sounds like a next token generator to me, which I think is more accurate. The word "predict" is doing a lot of heavy lifting in your formulation. What exactly is being predicted?
It is not clear that there is a mechanism that allows them to plan anything. It is clear there is nothing of the sort.
You can demonstrate definitively that there nothing other than "next token prediction" by running the same inference engine/model/prompt/settings multiple times in a row on the same hardware at temp 0 with a seed set.
If you turn the temp up, then you might let the "mechanism" select pseudorandomly from a probability distribution of k-likely tokens with minimum p-cumulative probability. Even then, if the next token has a very high probability, deviation will be uncommon.
I just performed this experiment right now on a few dozen runs of a dense model at thousands of tokens of output at temp 0, and all were identical. Then I did it with a MOE model, which might get some benefit of picking different experts each time. Nope. Outputs were again identical.
> You can demonstrate definitively that there nothing other than "next token prediction" by running
Describing LLMs as next token predictors isn’t incorrect, it’s meaningless. It’s the equivalent of describing brains as “a bunch of atoms”.
The problem with these descriptions is that they say nothing about high-level capabilities. If you can predict the next token in a proof of the Riemann hypothesis, then you can prove the Riemann hypothesis.
In a similar vein, if the string of "next tokens" based on 300k input tokens is: "run this bash command and paste the results into the token stream", followed by another call that includes the output of that bash output giving "Based on the output from that bash command, now run head -n 30 /foo/bar/baz.txt and paste it here", finally followed by "Based on that, here's the problem with your systemd configuration: [...]"
The part that really bakes my noodle: `/foo/bar/baz.txt` can be a filename that no one has ever typed before, the error inside the file can be a typo that no one has ever typed before, and it will still correctly "predict the next token" to solve your problem.
I don't think that's true and I don't think your comparison is accurate.
In the case of the brain, it'd be more akin to the reduction of the brain to just the neuronal dynamics (obviously experience phenomenally is much more than that to us!)
But both views can be useful, depending on what you are trying to understand.
The problem with these arguments is that people pick and choose based on their persons preferences. People who are in love with LLMs will, on the LLM side, argue against reductionism "they are way more than next token predictors!" then they'll turn around and equally try to argue "the substrate doesn't matter! The brain is "just" neurons and digital activity so of course we can replicate it!" (this is also incorrect, btw, the brain is also an analog system, even in its electrical connections and not just in its chemistry)
The on the other side the anti LLM crew will do the opposite (launch into reductionism when it comes to LLMs, resist it when it comes to humans). I'm a human being and I like humans and I think people have adopted a strange view of LLMs, which are a tool and should be treated as such, so I tend to side with the humans. The stalwart defenders of LLM honor that would balk at any attempt to reduce them in stature to what they fundamentally are, mechanistically, are... an odd bunch. I'm confused why people are so passionate about it. If someone told me "well your IDE is just a fancy AST walker" yeah I think that is a dumb take but I wouldn't feel like I had to argue against such a take either saying "no! no! it's so much more!"
I think the point was meant to be more simple and broad:
A system is not just its little bits.
Where "system" and "little bits" can be just about anything. But people endlessly repeat this mistake, and it has been a go-to misunderstanding/misrepresentation for a lot of people on this forum.
unless you are doing something where you stay in latent space without having a random number generator pick a word, the next token predictor part still matters. without making the tokes concrete in between, that plan doesnt exist
if it was designed to edit code, id expect the first translation from latent space out to tokens to be the final code, not be assembling the code by random number generator.
its not really designed to edit code until the tests succeed, we try to manipulate the distribution of tokens to predict by prompt engineering with the test results.
"next token predictor" has always referred to inference, not training. Training steps produce and bend weights, inference predicts tokens based on those weights.
> "next token predictor" has always referred to inference
Going back to first principles, wikipedia mentions [0] inference as:
> "Inferences are steps in logical reasoning, moving from premises to logical consequences. Inference is traditionally divided into deduction and induction"
Is that exactly what next token predictor means, or is it correlated?
[0] https://en.wikipedia.org/wiki/Inference
The 'predictor' part referred to the fact that base models would effectively answer the question "what token is most likely next". Training away from that default behavior doesn't escape the technical meaning of those words- answering "what token is most likely next in a solution to Navier Stokes" is still predicting a next token- but at that point the label just means it outputs text serially.
> A model is _sort of_ designed to predict the next token in a string of text. But it's equally accurate to say it's designed to edit code until it passes a suite of unit tests.
Training is not design. Design is the mechanism, namely a statistical method of learning distributions, and of executing those distributions computationally as an IO function.
The model is not designed to write unit tests. It is trained to do so, i.e. the distribution learned and reinforced is one such that it maps the I distribution to the O distribution that is what human beings typically do or want when writing unit tests. The design is the suite of algorithms used for this (which is ultimately global next token prediction)
I think you are conflating design (the mechanics) with their cultural or user level function.
You are basically saying "my bike isn't just rotating wheels on a frame because it takes me places". Hopefully you can see how both parts of that statement are true and why contrasting them as if the first clause is somehow false or inaccurate is incorrect.
Both these are true statements. They operate at different levels. People are constantly conflating these levels in the "is it a next token predictor" (the answer is yes) debate.
Yeah, fine. I was using the word design loosely. I'd also happily say something like "humans are designed to like salt and fat" and feel like that was a pretty reasonable statement, even though humans aren't designed either.
Maybe I'm missing something, but isn't training an LLM very well documented? Why wouldn't an LLM be able to read those docs just like it would read Java Spring docs?
Maybe I’m misunderstanding something. The way I read the results, I understand it like this - you give Astra “a bunch of numbers”, and tell it - “hey, this is a net trained on some game? What game is that?” And Astra goes - “Breakout, no doubt!”
To me this is fundamentally different from having ingested a bunch of docs so it now “knows Java” in its weights.
Ah, ok, so basically advanced pattern matching. I wonder if they train models on each other's weights?
that would imply the learning of what the weights mean can be done unsupervised which is really cool.
it probably has however been trained on papers talking about ways they extracted subsets of LLMs as say, the part that knows chess, and would be able to then use similar commands to process the weights
This implies that weights are a type of language. Neural networks are good at language because they are good at patterns. They can reason about it because their "reasoning" isn’t how we think. The same goes for how they encode images, their vision is entirely different from ours. That's why adversarial attacks work on them and not on us. We don't see anything change in the image, while the neural network now sees that the dog is a cat.
Ok? But does that mean that the “weights language” is universal? E.g. same weight patterns mean the same thing regardless of if it’s Kimi, Grok, or Astra?
There could be different patterns that solve the same problem. Just as you can have different algorithms, there could be different weights, but the abstracted result or logic from those weights would be the same. Take these words I'm writing now, they will be transformed into numbers, then sent to your screen, and then your brain will transform them into analog signals. I bet the neural networks in our brains have "different numbers", but the algorithms that understand language and other concepts are similar and could be universal.
It's like a sorting algorithm, which I can implement in many different ways in just a few dozen lines. If you look at the binary code (so like raw weights) of the compiled program, you will see just a bunch of a different numbers, but if you read the code, in most cases you will understand that it's a sorting algorithm. So it seems they just operate on different levels of abstraction that are not available to our brains.
i think a better question and one that probably has a lnown answer is whether there are infinite nontrivial solutions to train a neural network of a certain size to match a function? Trivial here would be about simplifying `n * 2 / 2` from two neurons to just passing n from the input
do they break down to a finite subset of groups?
How do we (or can we) know adversarial attacks don't work on us?
We're pretty confident that adversarial attacks do work on us, actually—we have entire industries built around it. Some of them even start with "adver-"...
There are classes of adversarial attacks targeting them that don't work as well on us, and vice versa.
That's not the point I'm making. So in context of vision we pattern match too, and you can see it in the hundreds of optical illusions online. Our visual system relies on built in shortcuts, and they break in different places than a neural network's do. Image models get fooled by things we would never notice, and we get fooled by things they handle fine. IMO that's good evidence that we and these models use different algorithms to solve similar problems. I think the same is probably true of LLM "reasoning", there's some overlap with how we think, but a lot of it likely works very differently under the hood. Good example of this is anthropic's interpretability research where they found claude doing addition in a way that's nothing like the carry the one method it describes when you ask how it did it.
Hypnosis and optical illusions are adversarial attacks on shortcuts our brain takes.
There's a long standing line of thinking that data compression has something to do with intelligence, or that intelligence may "just" be data compression in some sense.
In the case of models, the almost-theory (hypothesis with a good deal of evidence) is that once the amount of training data exceeds what the model can simply represent internally, the best way to keep improving is to start to "understand" the data. Understand in this case means to form internal control structures and decision systems that represent the data algorithmically rather than just memorizing it.
I put understand in quotes because we don't know how far this goes toward what we call understanding, or whether there's still a lot missing.
It’s important to understand that it had pytorch to actually run subsets of the weights. So its more like a dedicated person running the model to find the token it shouldn’t generate, then following the weight chain back up while calculating the NN outputs to find a place where if it changes some weights (again with a lot of testing via pytorch) the result is better.
Its not natively reading the weights in the Matrix “symbols scrolling on a screen” sense. Still bloody impressive and not an outcome I would have bet on!
>I mean, an LLM is still fundamentally a next token predictor, right?
Yes, but everything, including the human brain, is "fundamentally" some underlying simpler primitives.
>E.g., “breakout game weights” always look the same regardless of training and model?
Wouldn't this be expected to be roughly the case, given NNs that share the same basic architecture to model the same quite simple thing like a breakout game?
look at this reasoning Path, the problema was too simple.
https://chat.deepseek.com/share/uysgv9yl2aj0tqs3on
> “breakout game weights” always look the same regardless of training and model
The prompt told Astra what the input and output layers mean -- that is, how the current state of the game is turned into a vector for matrix multiplication, and how the result of those multiplications is turned into an action to take in-game.
Astra saw that the model was trained on a game where the player is at the bottom of the grid and moves left and right, reward objects live at the top, and there's some not-player-controlled thing moving around in the middle. Astra knows that this type of game is called "breakout."
A model that plays breakout will have meaningful weights for the player being in the bottom row, but not for the player being anywhere else, because the player can't go anywhere else. Astra saw that there were meaningful weights for the player in the bottom row, and nodes representing the player in any other row had low-magnitude noise, so the model had never been trained on the player being anywhere besides the bottom row.
Similarly, breakout has "reward" objects in the top rows, and "there is a reward object at [COORDINATE]" is part of the input layer. Astra saw that the model was trained on reward objects in the top few rows.
Calling it ‘next-token prediction’ is missing the forest for the trees. The reason prediction gives rise to so many capabilities is that to effectively predict something requires modelling the entities involved and their capabilities. You can’t predict a cat’s next action without knowing how to chase a mouse.
This is a really useful and succinct explanation, I'll be using it, thanks!
taneq - interesting phrasing. Thank you. To extend it to my use case, if you tracked enough cats moving a certain way, which all resulted in a mouse being caught, then you do not need to know how to catch a mouse yourself. Of all the ways we know a cat can move, these sequences result in a dead mouse, is useful too.
I'm only going to skim any subthread that gets into "next token predictor" territory, but for anyone looking for a somewhat deep video about _why_ this view is wrong Casey Muratori and Demetri Spanos spent almost two hours on the subject recently: https://www.youtube.com/watch?v=aQvSG8paVjM
A couple of highlights to pique people's interests:
- Attention is kind of like "Global Illumination" for next tokens (they both have a game dev background)
- Attention is constantly being recalculated _for the next token_. This part never "clicked" before.
- LLMs are "next token predictors" in the same way that games are "next frame predictors".
> That weights have some universal properties? E.g., “breakout game weights” always look the same regardless of training and model?
That's exactly what Platonic Representation Hypothesis would predict.
There also have been plenty of studies showing that various neural networks, including LLMs, develop implicit world models that reflect the geometry/structure of a task they're trained on. A trained "chess engine LLM" would have a "chessboard-shaped" world model inside it.
So, if you can figure out what the geometry of a task is, you can search the weights for correlates of that geometry. Then you can try to improve those correlates so that they track the task better.
Seems like bleeding edge LLMs might already be superhuman at "figure out geometry from looking at weights".
Even with PRH you can apply orthogonal matrices to intermediate vectors to generate different weights with similar effects.
Even if a slight rotation passing through a nonlinearity isn't good enough, you can just reorder the vector components which makes literally no difference.
Representational similarity analysis is concerned with representation structure independent from the coordinate system it sits in. A map rotated 45 degrees is still the same map.
A "reorder the vector components" transform preserves the similarity perfectly. So does a transform that maintains orthogonal equivalence.
> Also, how does this meta-understanding of the weights end up in the weights in the first place? Idk, this is mind-blowing to me.
These models are trained on pirated textbooks and papers, including every ML paper ever written.
> an LLM is still fundamentally a next token predictor, right?
I am baffled that "next token predictor" is still a talking point, years after it stopped making sense.
If you can predict the weather in text from a history of text weather reports, you are not "just" predicting tokens.
The shape of training data (token prediction) and the actual problem (learning whatever is required to predict the next token) are not the same things, at all.
The former does not limit the sophistication of the latter. In. any. way. Theoretical or practical.
> To me “amazing” is not the right level of speechlessness.
But we are in agreement here! Just ... wow.
I regret using that phrase. That was not “a talking point” I intended at all. But look how many HNers got nerd-sniped into talking about it instead of focusing on the actual point I was making. Oh well.
I did get that point! :)
(And you provided a valuable service/example for everyone who just thought the same thing! That phrase just keeps showing up. Whack! Whack! /h)
But: Unexpected free feature that plays right into direct recursive self-enhancement. What happens when they are trained to scale this ability up?
Learning itself becoming design.
This could hit a wall, or be a step change in ... the rate of step changes.
My read is that it wasn't one-shot, Astra was ran in a harness where it could change the weights, then Qwen would run with those new weights and be asked to do math, and its answers were fed back to Astra. So Astra could experiment with different weights until it saw better math answers from Qwen, with the actual benchmark run separately afterward.
And for the games, it was given as context the model architecture, game rules, environment description, etc. And since people assume Astra likely had knowledge in its training of existing interpretability research and weight-space programming, it can leverage that, combined with knowing the model arch and so on, it likely gives it a good base to determine what the weights do and how to change them.
They include a detailed description of the network architecture in the prompt. So it’s not like weights have universal properties. They could, and for some things probably do. An edge detection network will always “look” the same. The weights on the input and output layers of a breakout playing network will probably also always look the same, since breakout games themselves always look the same. But is possible that there are “different looking” configurations of weights for the hidden layers that can both play breakout.
So, Astra essentially reproduced the abliteration process, but instead of removing refusal out of the system, it removed some looping behavior.
We only know that Astra didn't invoke tools on the client side. We don't know what it did internally, because we never know that, as OpenAI hides reasoning traces.
Yes. Reading the reasoning traces in the original post, I got the impression that LLM weights are generally not interpretable even with frontier models. It's impressive that Astra can perform a small surgery on Qwen to remove its bottleneck, but it's not the same thing as being capable of fully understanding its weights.
Maybe it's bit like neurosurgery: We humans can treat certain pathological cases in our brains, but aren't exactly sure how our brains really work.
This was the first thing that came to my mind as well. For the unknowns, this is a pretty/popular project for the same task: https://github.com/p-e-w/heretic
You can search huggingface for models with word "heretic" in them... https://huggingface.co/models?sort=created&search=heretic
during all the openai agent sandbox shenanigans, did it manage to access it's own weights?
I’m more shocked by how many commenters in this thread seem to be misunderstanding what is going on here.
What are people misunderstanding?
That Astra is using the same methods as humans do (like abliteration) for modifying the behavior of a model, and that there is no particular connection to Astra itself being a model, and no inherently new ability involved.
That's fair, but almost everything so far that models can do much faster and deeper than humans can have been by using the same methods that humans do. For example, with cybersecurity, human researchers could if given enough time find every exploit that allowed the OAI models to escape the sandbox via Artifactory RCE and breach Hugging Face. But still there is a step change in what's possible by doing this at an industrial scale and stringing together all the solutions. The implications of applying existing mech interp capabilities this way could be very significant.
LLMs are exposing how much untapped potential comes from something that isn't necessarily more "intelligent" than humans, but is simply magnitudes faster at reading and analyzing data.
The human bias is the notion that intelligence = human intelligence.
There are multiple types of intelligence, from your typical pet dog to human to LLM, and we're likely to invent more types over the coming decades.
LLMs are vastly more intelligent than humans at nearly all things, but not all things. Another couple of years will just increase the gap a lot further.
How fast can a human draw a life-like photo quality scene (or eg something complex related to graphic design)? I know how fast Krea 2 [insert leading edge model here] can assemble that, trivially so, and at extreme quality. That's a form of intelligence in action. These AI systems have moved almost comically beyond human capabilities, they're increasingly best defined as superhuman (at a thing), and closer to omnimath for the frontier models.
In other words, there are different types of algorithms that can solve the same problem. Our intelligence is just one form of algorithm that can solve part of the problem space. LLMs seem to use a very different algorithm to solve similar problems, and because it's a different algorithm, it's an "alien intelligence" that we will probably have little chance of ever understanding, in the same way that a goldfish in a fishbowl can't understand most of what we do.
This reminds me what professor Geoffrey Hinton did talking many times in the past. Super intelligent artificial aliens.
I think the most interesting take is to look at their thinking logs to find out how exactly they decide what weight to change and why.
How are the weights even being mapped into token space as input? Are they bypassing tokenization so the weights just come in as what must seem like gibberish? Are they stringified as numerals then tokenized?
They're just included in the prompt as big lists of numbers.
Another bad day for the stochastic parrot theory.
Stochastic, yes. Parrot? No.
It sounds impressive at first but after you think it becomes mundane.
OpenAI, Anthropic, etc are locked in a race to create better models. They all have interpretability teams. Of course they train their models on every technique under the sun to understand and improve their models, including raw weights. The primary function of these models is to build their successor.
Of course they try train their models on these techniques. Finding success is still news.
Why?
How is this different from other things LLMs are trained to manipulate?
If it's new it's news.
We aren't fully at AGI yet, even though it's starting to get damn close. So every new capability is in some sense unexpected.
That's like saying the moon landing wasn't impressive because the US were in a space race with the soviets.
That is not what they are saying. Frontier labs are incentivized to train their models on all ML knowledge and techniques that people at those companies posses, so that current frontier models can assist them with developing the next frontier models.
Why would they train the models they release with this though?
> every technique under the sun to understand and improve their models, including raw weights.
What does this even mean?
I agree this is a huge deal, but I dont think this is saying that the NN can read the weights and "say hmmmm this one looks a bit low its obviously causing reasoning loops, lets bump that up a bit". The LLM that modified the weights developed hypotheses and wrote a bunch of code to look at various mathematical properties of the weights and updated them based on that. While it's still crazy it's not quite what I was imagining when I first read this...
I think the latter is more impressive…
This isn't that insane to anyone with old school engineering experience and not abstract thought experiment engineering in the cloud.
Language modeling is compression[1] and all the state to manage the cloud doesn't organically exist, it's added to the system.
We will see huge gains as all the developer middleware is compressed out as nonsense syntax outdated to the task at hand of linking phones and tablets to world models governments control.
Sam Altman was just saying they know best how to use your memory; what they decide is stored in memory is what we all experience.
It's not to store and copy around versions of C and Python and JS. They don't exist organically.
This shit will continue to get whittled down to just the geometric state transforms in memory and on screen just how government agencies decide. It's all electrical state in the machine at the end of the day. The math of machine latent space is what they're recursively self learning towards.[2]
[1] https://arxiv.org/abs/2309.10668
[2] https://arxiv.org/pdf/2605.27734
i think one thing to take from this is that "brains are not much like neural networks beyond some superficial concepts"
> No human can go edit a brain to become better at math
You could have done what was reported.
I am not convinced a human could not edit another human's "weights" to improve their performance if such a thing were possible and could be done safely and could be reversed -- all of which are true in virtual neural networks like these.
E.g. if one could edit brains like this in humans I bet a cognitive neuroscientist could fix some mental disorders with clearly understood biological bases, and I would not be surprised if performance at certain tasks could be enhanced.
What stops us from doing this in practice (most of the time) is that brain surgery is extremely high risk, coarse grained, and irreversible. We do in fact do this for some severe cases.
A few years ago it was discovered that LLMs can implement gradient descent in their processes. Producing a full trained neural network would be an extension of this.
See: https://arxiv.org/abs/2212.07677
RSI RSI RSI RSI!
This is the "recursive self-improvement" they've been warning about.
I've always laughed at that phrase, thinking "haven't these things been improving themselves since the very beginning? This is all just iterative gradient descent, each step is a 'self-improvement'", but this really is different.
I wonder if we'll ever discover a new branch of linear algebra or something that allows us to find the optimal weights much more efficiently? Maybe AI will help us discover it?
Good luck with figuring out what it does and enforcing any safety switches and rules I guess...
It doesn't seem like there's room for recursion here though. A smart model can share its smarts with a dumb one, but the ceiling remains unchanged. Perhaps there's juice to squeeze in verifiable dimensions re: software, math, but there are similar ceilings to be found in those places. We can't keep asking harder and harder questions infinitely while remaining able to verify the answers.
It is astounding, but I don't think it's well set up for anything besides faster approaches to existing limits.
> We can't keep asking harder and harder questions infinitely while remaining able to verify the answers.
The last several millennia of mathematical research would seem to be an obvious counter example to this claim.
Perhaps we have different things in mind re: hardness. But to set that aside, what I should've said is that we can't do this quickly enough for it to be seen as an inexhaustible supply of training material. The consumer and producer are just not well matched.
> This is the "recursive self-improvement" they've been warning about.
Well, except for the self part since it improved the performance of a smaller, dumber network.
And the recursive part, since it couldn't improve the performance of a larger network.
Is there anything preventing us from doing the same thing on the next bigger frontier model ?
I’m certain the next big frontier model will also be able to improve the weights of comparatively tiny models. That does not improve the large frontier model.
Only the context window, right? Which seems....not that big of a hurdle to overcome or navigate around.
No, no it isn't.
>I'm shook. If you'd asked me this morning whether LLMs have native understanding of their weights that is less than human <-> neuron understanding, the same as human or more than human, I would have said "same to less" depending on definitions.
??? All LLM understands are their weights, dude, come on.
It gets really insane if you were to ask it if it could modify it's own weights like that.
> No human can go edit a brain to become better at math.
We literally do this by studying.
This is more like "meta-studying". The human equivalent would maybe be to do brain surgery on someone to make them learn something new about math
> No human can go edit a brain to become better at math. This is just very, very surprising to me and extremely interesting. I would guess this upsets quite a number of interp theories and philosophical takes on what's going on with these networks.
Why would this have implications for theories or philosophical takes? I'm a bit surprised that you're surprised by this.
This seems like exaggeration, the paper seems to describe something more like abliteration, where Astra orthogonalized out activations related to undesired behavior.
This is obviously very different from the surface level claim that Astra can understand the weights of a completely unrelated model with enough detail to improve it in a manner that isn't just the latent space trickery we've been doing for years already.
It's not really exaggeration. It's doing something similar to abliteration, but more. First it finds the part of the weights that does a thing. But then the difference is, while abliteration deletes stuff, this is adding/changing things. It does this by doing semantic analysis of the weights, changing a little bit, see what happens. It's not super complex or advanced; it's essentially a form of statistical analysis and experiment.
This means we might not need to fine-tune with thousands of passes and tons of GPU. We could just tell a frontier model to edit some weights and iterate until there's better results. This could cut the cost of fine-tunes by 100x. Makes no difference to the normal end-user, but pretty huge improvement for LLM research and model tuning.
what do you mean? this is just classic computing. It's evals and linear optimization ...
Given your reply, I was looking to read the paper. Apparently this blog post is what passes as their paper for now... I think I'll change my previous judgement and wait for a proper publication. I'm still not convinced by their claims, but maybe an eventual paper will spell it out more formally and convincingly instead of expecting readers to dig through all of their source code.
By the way, my understanding of abliteration is that it involves detecting the refusal direction and modifying the weights/activations away from that direction. I don't really see how that's "removing" in a way that's different from finding the "looping direction" and modifying the weights away from that.
> No human can go edit a brain to become better at math
That profession is called “teacher”.
I regularly edit my own brain, and those of others, to improve their skills.
But this is more akin to brain surgery, not training
>No human can go edit a brain to become better at math.
Isn't that what math tutoring is?
Thats what I was thinking, this is not a breakthrough?
Don’t forget that weights of models and their behavior may, perhaps unintentionally, be in their training set.
Or at least those of some models, and I wouldn’t be surprised that an LLM model is able to understand correlations between weights and behavior better than humans could ever do.
So their weights are in their weights? Doesn't that double their weights?
Not in that way, more that they can predict what good or bad relations between weights would look like.
This is proof. These things are not stochastic parrots. Agents are not trained on other agentic weights. The fact that it can do this is EMERGENT intelligence.
Time for everyone to wake up. There's no denying it now. Everyone who called it stochastic parroting before was completely and utterly wrong. It's now categorically clear. We are dealing with an actual intellectual entity.
(...or it just read some of the mechanistic interpretability literature...)
The mechanistic interpretability literature does not imbue a human with these abilities. That literature is primitive. Nowhere near the point of editing model weights and creating models from scratch or even being able to figure out that a model is playing an atari game.
I think people are rather ignorant about how much we don't know. You throw some big words out there like "mechanistic interpretability" as if this is an actual field where we understand what's going on. The reality is we don't know what's going on. We have to use reinforcement training to do anything close to what the agent did here.
It's over. People need to stop being delusional. These things are intelligent beyond anything we've ever seen before. We are at a point in time in human history where there's a paradigm shift and half the world is so full of themselves and there own specialness as humans that they can't even admit it.
>No human can go edit a brain to become better at math
Weights are just the output of training, right? So can't "no human" be replaced with... a parent? a teacher? a textbook?
There's training, and then there's brain surgery. One of those you don't want a parent / teacher doing to you.
Astra editing qwen is not really surprising. What is insane is how powerful the medium of language is, and how we humans under-appreciate it.
My impression is that it's somewhat analogous to someone with a lot of experience teaching someone who is much younger and has much less experience.
More generally, I think some of Michael Levin's ideas about intelligence are more broadly applicable than we would like to admit.
https://m.youtube.com/watch?v=Or_3tlEOLj4
Transfer learning is a thing but it's not done by passing one model's weights as decimal numbers through another model.
It might be analogous to animals learning by observation versus learning by in instruction, although it sounds closer to bacteria trading dna.
It's more like showing a pigeon a piece of paper with the DNA printed out in ACGT letters and having it cross some out with a pencil, then you put that DNA back into a bacterium and it's antibiotic resistant and grows twice as fast.
It seems far more pedestrian than that.
https://news.ycombinator.com/item?id=50347685
I don't think human cognition heuristics are the right tool to describe what's happening.
Like a turbocharger isn't analogous to feeding a horse a certain diet. A turbocharger has to be explained in the terms of its actual context.
Yeah. That's why I referenced Michael Levin's stuff. I think it's more applicable because he describes intelligence as an innate aspect of different creatures/systems/etc.
You don't consider Neurodoping a form of this? https://pubmed.ncbi.nlm.nih.gov/23504390/
Talking in weird abstractions and analogies is what is blowing your mind.
Like, looking at the specifics, it's cool and interesting, a little bit contrived, but it's not insane, there is no philosophy required, no need to shake.
The way it understands it is the same way it understands anything else, by reasoning about it in legible human terms. It's not that surprising that these LLMs, trained to be effective in technical fields, are good at the technical field of understanding the implications of model weights.
It's not like these models understand weights be some special linear algebra communion with its own weights.
So let's assume a completely naive approach. modify any weight arbitrarily, run an eval if eval improves arbitrarily modify another weight and repeat ... do something like 5,000 rounds. Just brute force, zero-intelligence, random selection. how does the naive base case perform on N=20 random trials?
Unless you include a basic test like that these results are meaningless. Was there p-hacking? Why just these 3 tests? Did they try others and it failed?
I'm currently running this myself. I'll see if the naive approach improves results. Can't wait to have these in hand and have people call me stupid when I bring it up with empirical evidence, it'll be awesome.
What you're describing is a shitty version of gradient descent, which is the technique used to train LLMs and other neural networks. Proper gradient descent simultaneously calculates how much every weight improves the output, and edits them all simultaneously in proportion to that.
obviously. unless you have the equivalent of a coin toss, no metric carries implication. Does naive noise give you nothing? The null hypothesis needs to be established for these particular evals with this particular model.
Doing basic empirical hygiene shouldn't be too much to ask for.
For instance, on their base version of qwen if your turn thinking off you get a 52.0% on their evals. That single flag recovers ~72% of their entire claimed improvement - base model, toggle thinking off, scores go up 72% of their improvement.
I'm still investigating various wildly naive modifications. If I can hit their numbers doing inarguably stupid things then it puts their substantive claims in doubt.
That's exactly what I was thinking. Actually, if you look at the "Is Qwen improving" 25 and 10 forward passes, it doesn't seem to improve much.
Additionally, since it relies on using GSM8K and MATH datasets: how does it compare in terms of compute with other training methods?
I for one am looking forward to having Opus 5.5 level intelligence built into my calculator in 2027.
What a time to be alive. Imagine when everyone has access to a model of this level for free, on mobile devices.
However, humans can easily do what Astra did to all the weights. It's not difficult to interpret the weights. It's just tedious, and LLMs excel at tedium.
It's insane how people see something not do something they say and then talk as if it did. Insane indeed.
Yeah this waters things down considerably for me.
If I am debugging a complex program I know nothing about, but have been told there's an infinite loop causing it to hang somewhere, I could probably search through the code base and find a while(true) that shouldn't be there, and get it moving again. That feels more like what has happened here.
And here we go...
As others have said, while still absolutely insane, I don't find it completely out-of-the-blue surprising. One of the recurring abilities of (current-generation) LLMs seem to be to rapidly grasp very complex systems, where humans would need months or years to reach the same level of expertise.
You can see this in the way a coding agent can "understand" and explain sprawling legacy codebases in seconds.
You can also see it in the projects that are using them to reverse-engineer proprietary IoT protocols or computer games.
But those were all systems where we had detailed knowledge about what structures are there to be reverse-enginered.
If the models can transfer this ability to some other systems where we don't know the underlying structure then I think we will be in for a lot more craziness in the future. (Protein interaction networks? DNA sequences? Ecosystem interactions? Animal languages?)
As for weights, if the models really can analyze the semantics behind specific weights or directions, I have my hopes that they could also give a textual explanation or help designing tools to make "handcrafting" weights more practical in general.
Maybe one day it would be able to give us a mixed high level code and inline NN view. Sort of analogous to C with inline ASM or “decompiling” into code and inlining opaque residual neural blocks.
but code is made to be understandable. while neural nets are not.
that is why this is crazy,
give a human a legacy codebase and 5 years and he'll be able to understand very well.
if you give a human the weights of qwen 0.8B and 5 years it'll still be impossible to edit the weights in any legible manner.
> if you give a human the weights of qwen 0.8B and 5 years it'll still be impossible to edit the weights in any legible manner.
Well, is that really so? There is quite a lot of research going on to identify the "meaning" behind certain weights, neurons or directions. I don't see why a human with enough time, perseverance and ability to run experiments with the model shouldn't be able to uncover some of the underlying mechanisms of the model.
But of course not with the speed with which an LLM can do it.
5 years with 800 million parameters would take over 18,000 changes an hour, 24/7/365 to change every parameter.
No, they're wrong. Hobbyists do this for every model release: https://huggingface.co/models?search=abliterated
I think the presentation of the points is confusing people, the model could understand weights directly for extremely small networks in the 10s of thousands of weights.
Once they moved onto even the smallest LLMs, what it did was closer to abliteration: create examples where the model does something you don't want, create examples where the model it doesn't do the thing, then most importantly do forward passes.
Once you're allowed to actually do a forward pass vs just reading raw weights, tracking the changes in activations and tweak the model to modify them is a very well established process that takes a few hundred lines of Python.
Think of weights as code/config and ya plug it into a NN. It’s not that crazy.
Editing LLMs weights to modify the output was already possible in 2022: https://rome.baulab.info/
It’s cool we didn’t have to spend extra effort in building the tooling.
> but code is made to be understandable. while neural nets are not.
mmm, maybe they are?
The thing I find most surprising about this is that the content is in a completely new medium/language compared to what it’s trained on. It’s trained on natural language and code and equations and DNNA sequences. I guess this is just finding the generalization between that and many layers of learned features in high dimensional spaces.
Everything is obvious and "not surprising" in hindsight...
> Animal languages
John Lilly's long awaited English ⇔ Dolphin translator, my beloved!
I agree, this really isn't surprising. They're pattern interpolators, and so much of what we deal with in the world is rooted in patterns of some type, including neural nets.
IMO, this is a similar behavior that we saw from AlphaFold, which also used a type of Transformer architecture to make its protein predictions.
I pointed Opus 5.5 (medium reasoning) at the paper, and after a bit of mucking around, instructed it to port the Astra weights for the breakout policy to JavaScript logic. According to its own evaluation, it succeeded. Then I got it to minify the JavaScript policy. The final result is 346B, compared to around 3KB for the (sparse) weights generated by Astra.
https://claude.ai/share/7a931d7a-0acd-402d-a5a5-a9d44d9d0385
Maybe the models have a common internal representation for computations, and they're capable of consuming and producing computations in the syntax of neural network weights?
Amazing experiment. Indeed, with the benefit of hindsight, we can say that breakout is a game that is probably better suited to deterministic code (it's not like Go where an AlphaZero style training is essential), and Astra is probably translating this internal representation of code into the shape of linear algebra and model weights.
I should note that the original authors did not allow Astra to write any significant code to construct the weights. I think this is unfair. If you allow strided assignments to numpy arrays, you should probably also allow the use of functions like np.linspace, np.repeat, and np.tile.
Oh and Astra wrote this (https://github.com/RikiyaT/lm_understand-logs/blob/4d09ba6d6...):
I think the second 70 there is probably a typo; it looks like it should be 80 instead.
The way I interpete it, is vast amounts of text are to LLMs, in some ways like vision is to us. Not seeing tokens, but seeing the "shapes" between them. Like we don't parse pixels but see the whole picture at once.
They triangulate efficiently on the meaning from all the text, instead of rolling it through a lossy feedback system.
It is why they are so good at processing ambiguity. They can combine ambiguity from any parts of the text, near equally well, even on long form text.
Just like we can instantly triangulate the meaning out of barely lit photographs, by correlating every little shadow or bit of texture, regardless of where the clues are in the image.
This.
Lady in red dress scene from Matrix comes to mind.
Our visual system deals with TB of video input every second but we are able to filter through it and spot the important bits. We can even with closed eyes make our mind spin up visual scenes and alter them. I think that's quite possibly a useful explanation of what's happening here.
I don't think this is an emergent capability. I think Astra was trained on neural weights data as an attempt by OpenAI to get to recursive self improvement.
Something similar happen in early GPT models which had a lot of chess game in standard notation in their training data which made them very good at chess as long as your prompt had the right format.
We'll see if this attempt at RSI worked.
> I think Astra was trained on neural weights data as an attempt by OpenAI to get to recursive self improvement.
This might be the case, a few weeks ago, I wanted to compare a small neural network trained with AdamW vs a handcrafted set of weights on a random task. Astra was able to handcraft a set of weights that perfectly fit the small neural network to the task. AdamW was unable to tune the weights to get the same result.
I believe we've known humans can handcraft weights much more efficiently than standard gradient descent methods, we just don't have enough memory or time to scale ourselves, so it makes sense to train a neural network to do it on our behalf. I am excited to see how far we can push this method.
Mentioned chat: https://chatgpt.com/share/6ae8f024-34ec-83e8-9572-1300f52c80...
Research on using LLMs and diffusion models to directly predict weights have been going on for a bit now for example https://arxiv.org/pdf/2402.18153
It’s a good reminder of one of the most unique features of alphago - it was like a seven layer or so neural net that could predict out a hundred moves. I’m not sure we took this kind of compression super seriously or at least I would say perhaps it remains under explored. You could think of what you did as sort of the same thing as skipping the Monte Carlo search phase for a go engine — a process compression.
It looks like Astra picked the "random task", and the task it picked was 8-bit parity, which is easy to design a circuit for that computes the answer perfectly, but sort of maximally difficult to fit with any kind of gradient descent. It's a great choice for what you asked, which was to find a case where AdamW doesn't come close to the theoretical optimum fit. But I'm less convinced that it tells us much about Astra's capabilities at handcrafting weights in general.
8-bit parity is a^b^c^d^e^f^g^h, which you can compute with minimal depth as ((a^b)^(c^d))^((e^f)^(g^h)). You can compute x^y as x&~y | y&~x. With ReLU activation, i | j is ReLU(i + j): weights of 1, bias of 0. And i & ~j is ReLU(i - j): weights of 1 and -1, still bias of 0. So a two-input XOR is three neurons in two layers, and the tree of 7 binary XORs is 21 neurons in 6 hidden layers. I don't know why Astra needed 433 parameters. Only 42 nonzero parameters are needed. Maybe the other 391 parameters are zeros that the Transformer model demands to be present.
With an RNN you can compute the parity of a string of any length with only three neurons.
Actually, I handed that to Claude Opus 5.5 to check, and it sensibly pointed out that x&~y is mutually exclusive with y&~x, so XOR doesn't even need the third neuron, except in the last layer; you can just fold the addition into the weights of the next layer, which means you only need 4 layers and 15 neurons. But about half of those neurons need 4 nonzero input weights instead of 2, so you still need 42 nonzero weights. It's a good thing you don't need those OR neurons, too, because my definition is wrong: ReLU(1+1) = 2.
Also it explained how to design a 28-parameter net that I don't quite understand, and also a single-hidden-layer network with 8 ReLUs, which I also don't understand, but it needs more parameters: $\text{parity}=s+\sum_{k=1}^{7}2(-1)^k\,\mathrm{ReLU}(s-k)$ where I think $s=\sum_i x_i$.
Somewhat more surprisingly, it also said it didn't have any trouble fitting a 641-parameter network to this problem with AdamW.
Here is the code Opus 5.5 is claiming fits a 641-parameter MLP to parity of 8 inputs:
I might not know enough about ANNs to spot if it's cheating on some way, but it looks legit. It does indeed output that the [8, 64, 1] configuration gets 100% accuracy on all 20 seeds.
At least this slice
looks like a perfectly orthodox batch evaluation of a multilayer perceptron, with the input batch in X and weights and biases in P, and that's what it's evaluating the accuracy on. I dumped out P for one of the [8, 64, 1] networks, and it does indeed look like about 650 parameters. I was going to post it here, but printed out as text it's like three pages long, so I won't. I quantized it to one decimal place with [a.round(1) for a in P], and the quantized network still calculates parity perfectly, but it's still a page and a half long.
It might also be a side effect of training models to help out with mechanistic interpretability, which I guess would help out a lot with recursive self-improvement.
I had the same thought, but why would they do this training on a model they release publicly? I assume they’d be doing that kind of RL on an internal fork only?
All NN behaviour is emergent strictly speaking; by definition. I think you mean a phase shift of a kind; a qualitative, unpredictable leap in capability.
Same thoughts. I doubt if this work can be reproduced with other frontier models. The title seems way too specific for Astra.
This is a great point / theory.
It would be really funny if it turns out every single openAI advancement just comes from cheating by learning the answers in advance.
Eg when they were caught stealing people’s maths research and then saying their model did it
They actually did math no one could.
They weren't caught stealing other people's maths research.
I think a lot of past examples of "emergent capabilities" have at least partly turned out to be that more stuff was included in the training data that (intentionally or not) covered the capability.
Like i.e., I think the "large models exhibit emergent capabilities that small models don't" turned out to basically be "larger models are often trained on more extensive data"
This is actually not new, I believe GPT 5.2 already had remarkable similar capabilities. I wrote a very elaborate private benchmark involving hand-authoring highly optimised transformer circuitry, and I was shocked that GPT 5.2 could do it. Makes sense that it was specifically trained on building and debugging transformers.
I am not smart enough to know the answer to this, so I am wondering if someone can help me with this question: if Astra is able to decipher structures and patterns within model weights, would an LLM be able to directly modify its own weights on-the-fly to achieve recursive self-improvement using a reasoning loop?
> As a result, Astra spots and fixes a common failure of tiny reasoning models: they loop [13]. When the base model loops, it repeats the same text and rarely reaches an answer (85 of its first 96 practice answers ran to the 32,768-token limit). Astra traced the looping to one direction written by the MLPs of layers 15 and 19, and subtracted that direction from 64 output columns in each (131,072 weights), much like removing a behavior by deleting a single direction from a model’s weights
HA! Now that's cool. Not necessarily the improvements, those are ancient and probably overtrained benchmarks, but it's so cool that the model found a singular direction for this "loop bug". I imagine it could possibly do much the same thing in finding other "single directions" on a variety of topics (if allowed by guardrails). This is really really cool, especially for the mechinterp folks.
It's pretty much how abliteration works too; find where the refusals are happening and flip the bit.
That... is a good point and concerning WRT open models. We like open models as they seem to be a gift to humanity, but at the same time, they can't be reliably tamed and can always be realigned to get around guardrails. We have discovered fire and have yet to see its full destructive potential.
I think we already are seeing the destructive potential :/ The genie's out of the bottle at this point with open-weight models (and thus can be abliterated in terms of offensive cybersecurity) that can run inferences well on relatively cheap hardware with relative parity to frontier models.
I'm a bit of an outlier in regards to today's HN culture. I'm still holding on to Stallman's idea that information wants to be free. There was a comment on another thread that aptly elucidated what I think is necessary.
Not everyone has the know-how or desire to compile the Linux kernel, or any what-have-you executable in user space. But a good amount of people do, and it's a boon to security/privacy that they do, and cryptography can verify that this work was done. (granted, manufacturing a computer is an international effort with a lot of unknowns, ie proprietary binary blobs in BIOS/UEFI) (and people are lax to validate the crypto themselves if it's not an automated thing; that's a UX problem imo)
I think we'll eventually find sufficiently performant LLM's trained on public domain information[0] that people can validate. Sufficiently performant training algorithms that people can validate[1]. And FOSS plugins to integrate with their FOSS IDE's which they can validate. And like the Linux kernel, volunteers to make sure there's nothing fishy going on end-to-end.
Can this stop the danger of what open models can do? No. And that does scare me. It's Metasploit on steroids. The cat and mouse game of cybersecurity is going to increase in power law levels of speed. I do think verifiable LLM's for consumers to use is a worthwhile pursuit regardless. And for LLM's capable of offensive cybersecurity, I think it might be better for it to be democratized than held only by state-actors and individuals/corporations with an absurd amount of consolidated wealth.
I have no horse in this race though. I'm just happy I can learn how these things work and toy around with getting quantized models to work well enough for my coding side projects. Though I often reach for a frontier model for serious coding issues, local for rubber-ducking, and Kagi's research assistant for initial planning.
[0] Pipe dream, unless international copyright law drastically changes. Since opening the dataset for any competent LLM will violate said laws, whether obtained legally or not themselves.
[1] Knowing the training algorithm and dataset will let it actually be open source models instead of open weights.
edit: These are just shower thoughts; didn't mean to write such a long response. Would love any pushback or any correspondence in general since it'll be on my mind all day.
Can't wait to see qwen4 and qwen5 as this becomes more common.
Like altering DNA
No, not like that at all.
How is it different?
Brains and neural nets are already quite different things and difficult to compare, but at least you can wave your hands and vaguely say that both are “thinking” by some means of “abstraction or generalization”. DNA is absolutely nothing like that. DNA is more like an encoding for a flowchart for a recipe for building a specific body.
> encoding for a flowchart
Which is exactly what a model with it's layer is.
Exactly like that.
Both sound cool and "like, the future, maan" while smoking weed, otherwise nothing in common.
why do we need to anthropomorphize everything. what it's like is altering the model weights of a computation graph, because that's what it is.
It's not anthropomorphizing anything. It's comparing the act of altering weights as "instructions" as similar to altering our own building blocks.
It’s reading minds…
But seriously, have we tried to run neural data through this. Also, we have no idea what other behavior it modified unless they were able to run a complete regression.
I think the subfield has been there for a long time. See a small section of papers on the topic of generating dynamic weights during inference, predicting weights from network architecture, and learning the distribution of previously trained network parameters, just to name a few keywords:
Parameter Prediction for Unseen Deep Architectures [1]
NeuroGen: Neural Network Parameter Generation via Large Language Models [2]
Diffusion-Based Neural Network Weights Generation [3]
The critical thing to me is that Astra is capable of doing it. I don't know if it was trained to do so.
All this does not make it any less crazy in my books. And I still don't know how I can sleep easy tonight.
[1] https://arxiv.org/abs/2110.13100
[2] https://arxiv.org/abs/2505.12470
[3] https://arxiv.org/abs/2402.18153
Dynamic weights during inference are also known as an attention block
Thank you, finally someone comes in with the receipts. I'm going mad over the breathless hype in this thread.
It has! Hypernetwok/weight generation has been a thing for a long time - in fact my Master's thesis back in 2018 was based around PRNG weight generation for large percentages of the network. (https://open.library.ubc.ca/media/stream/pdf/24/1.0371928/3)
Why is this not overhyped in my opinion? Nothing has worked this well, ever.
Right. The question to ask is whether it is an emergent property of the model, Astra, or it was trained for it.
Whichever, it betrays a rather interesting feature about the model's architecture and this won't go unnoticed, I think, by the chinese labs.
I feel like this is conflating 2 separate results in a slightly misleading way.
> We find that frontier LLMs are surprisingly good at natively reading and writing neural network weights
This claim is backed up by the ability of Astra, given a text representation of the weights of tiny (20k) special-purpose models, to figure out what those models do and in some cases improve them beyond what RL could do just by reading the weights with no tool calls. They call this "weight-space programming".
> Likewise, asked to make Qwen3.5-0.8B better at math without gradients, Astra identifies an issue in its weights causing reasoning loops that Qwen’s own model card warns about, and fixes this by natively editing 0.016% of the weights
Here it did use tool calls, and seemingly used standard steering vector techniques to identify and fix reasoning loops in this small Qwen model. This doesn't seem that related to the previous results.
Yeah this basically just hooking up a model to a specialized harness for this purpose. People are pretending like it did all this intuitively.
This will be your refrain as the killbots come down the hallway: “it’s actually just a specialized harness, they’re just invoking the kill all humans tool, not even impressive!”
Lol
Prob the only time a comment on this site has made me laugh.
If by "slightly" you mean "not slightly", yes.
I find this just completely shocking. I’m definitely skeptical. It just doesn’t line up with my understanding of how these models work. Is there really enough training data on direct weight editing to give a model this skill? Maybe it’s evidence that OpenAI has been creating a lot of training data on ML research. I’m surprised that was put into a public facing model though.
One consideration is that Qwen3.5 has bigger/later versions that are already better at math. If it was indeed training on ML research, it's possible that what it is actually doing is applying lessons learned from the Qwen3.8 27B weights (which independently solved the problem) to Qwen3.5 0.8B.
I doubt it's that simple, but I could totally see an LLM being able to make that association and apply a fix from an improved LLM to a smaller one with the same architecture.
But how would it know which weights to update? It has 3.5 and 3.8 weights memorized in its own weights and can somehow figure out which ones are responsible for different skills just in its own reasoning trace? (Since it didn’t use any external tools)?
That’s pretty crazy because, as I said in another comment, I just asked ChatGPT how many r’s are in blackberry and it said 3. I know it’s not Astra and the app doesn’t tell me which model it is, but I’m still very confused about how a model can edit neural net weights but not count letters in a reasoning trace.
Edit: Oh I misread, it didn’t use tool calls for the first part but it did have a custom harness for the part where it edited a models weights. Still impressive but not existentially confusing anymore.
>It just doesn’t line up with my understanding of how these models work. Is there really enough training data on direct weight editing to give a model this skill?
I mean, this question indicates that your understanding of how these models work is, indeed, lacking.
Despite what some people having being hollering about, treating LLMs as just "stochastic parrots" isn't doing their capabilities any justice. They are constantly exhibiting emergent behavior which wouldn't be found directly in their training data. They are capable of combining different ideas that they are explicitly taught to perform novel tasks.
I'm not saying you're guilty of this, but the people claiming LLMs just regurgitate existing data don't understand how LLMs, and likely their own cognitive experience, actually works.
> emergent behavior
This hand-wavy phrase has applied to neural networks in general since it was possible to train them. That was more than half a century ago. It means both nothing and everything simultaneously.
Every learned ability of any neural network is an emergent property.
The math example on Qwen was based on 50 batched iterations, and the fix was essentially to remove a direction in activation space that causes looping instead of an exit, this was already known to be a thing.
So it's not able to insert new knowledge, but it's able to adjust the routing/looping/weighing of existing one.
As I understand, it can replace post-training like LoRA/GRPO, but if it'd cheaper the authors don't go into
I don't think its that crazy that you could look at activations and tweak weights directly to get a certain outcome, but isn't the question whether you can do that without side effects or degrading/breaking other parts of the model?
If I want to do that in a generalizable way that preserves performance across all domains then am I not just back to training again?
Yep youre right, I don't see an eval on this.
I think you could still essentially have a really cheap Lora-Post training on a frozen base model by doing this. And instead of creating 200 checkpoints for different post-trained directions just adjust a single one on the fly.
You don't think an LLM can perform tasks not in its training data?
Well, it can't, by definition.
What definition is that?
model
The definitions behind the math of LLMs.
This isn't true.
A neural network can do new things outside the training data. Random initialization sees to that.
Whether those things are useful is a different question, but it turns out that yes, some of those things are useful. It also turns out that in a large-enough LLM if you give it some kind of idea of what you see as useful then yes, it can pick out the newly useful things.
Random init only exists to make each neuron distinct from other neurons. The weight is still defined by the training data, without the training data, these random weight don't give us anything.
The second thing you said is basically what's called fine-tuning.
Yes you are right about random initilization of course. Long day here.
For the second: No, not fine tuning at all.
For one thing in-context learning is enough for learning novel tasks[1].
Secondly, it can do new things outside that data. LLMs build models of the world[2]. That means it can model novel behavior based on that internal representation.
[1] https://arxiv.org/abs/2005.14165
[2] https://arxiv.org/abs/2305.11169
Not sure what you're getting at, in any of the cases you mentioned it needs to learn if it needs to do new things.
Maybe it can generalize outside of some learning task you gave it, but still it comes from the training data.
"Generalization" also comes with some luck, since these models most of the time output complete non-sense when it comes to tasks it "didn't learn" specifically. Just because you got lucky with the "neuron connections" doesn't mean it has what it takes to generalize anything.
Almost nothing I prompt into existence already exists, otherwise I'd just use the existing thing.
Not one which can easily be represented as an interpolation of other tasks it was trained on.
I don't think it can whistle while chewing gum, no.
I think it can patch together components of tasks represented in its training data to do demonstrably new things, yes, which does count as "perform tasks not in its training data" with the limitation that some form of the task must be present. Example: writing an ode to a sales lead for a job well done in 2032 for the first quarter. Odes exist, the concept of sales performance exist, good job performance, and notions of time. Stringing those together are good examples of interpolation but not extrapolation, as it were.
In my opinion that's very handwavey.
Definitely see where you might feel so. I provided an analogy that may help anchor - why ML fails extrapolation is left as an exercise to the determined reader! :)
The surface for "interpolation" is fairly broad for LLMs but it isn't infinite -- extrapolation still exists!
the most obvious example was counting the Rs in strawberry.
figuring out what is and isnt in distribution is quite difficult though given that theres so much data passed in, and transformers have some amount of combining it together
theres some really obvious things though, like picking nonsense tokens together, or things outside of the tokenspace entirely
you could also construct a string that is a series of the least likely tokens in a row. you wont get it to output the next token, despite it being quite clear how you did it
At some point we have to accept that there is enough training data on "being good at thinking" and that being good at thinking generalises really well.
There doesn't need to be specific narrow training data on every conceivable task.
I agree, but also, I kind of feel like real judgement and understanding are a different thing from this "being good at thinking", and that in fact, this is the reason AIs hallucinate in the first place (A generated text can fit the pattern of a good response because its accurate, or because it's bs but has similar words and structure to other texts on the topic. Without real understanding, LLMs can't tell that the truth is important)
It's not directly related to this but I'd like to add that, in my opinion (admittedly uninformed but it makes sense to me), when they manage to, not just reduce the frequency of the LLM stating made up dates, laws etc, or made up facts, but manage to make an AI understand what the truth is and deliberately make responses accurate, we'll have real AGI that will be able to do engineering, accounting, medicine... unsupervised.
And yet I would expect that if it truly was generally good at thinking as you put it that it would be better at chess if you gave it a set of general rules/tactics and have it think through and look forward to decide its moves, but it is quite bad.
How do we decide which version of thinking a model is good at?
A house sparrow (Passer domesticus) doesn't have a worse brain than a human. Nothing does house sparrow thinking better than a house sparrow brain. Not even a human brain.
what about a house cat?
they clearly outhink the sparrow at sparrowing in order to predict what the sparrow will and wont react to, and how theyll try to escape the cat
It's not clear to me that a house cat is better at out-thinking a sparrow than a sparrow is at out-thinking a house cat.
Avoiding predation is a pretty small part of the life of a house sparrow. There is no reason to believe a cat brain would do a good job with handling the demands of flight (avian brains process visual stimuli roughly twice as fast as mammalian brains, for one thing).
But even if we focus on avoiding predation: would a cat brain in a sparrow body (a silly premise, I'll admit!) do better at avoiding cat predation than a sparrow brain in a sparrow body? I doubt it!
Let's analyze the premises
1/ The internal users at labs and external users are completely different.
2/ The lab people use the models to create recursive systems, systems to create prompts or synthetic data sets etc etc etc. Way different mentality than "fix the bug, make no mistakes"
Is it really so farfetched to think they've been having models fuck with the weights of models? I think it's entirely possible, probably even likely as a way of generating permutations of synthetic datasets that will then be included or excluded
If you subscribe to this, then it would be entirely possible for this to have become a major part of post-training and be practically impossible to exclude from public facing models, which are famously hard to govern. The models barely know their own version numbers.
> Is there really enough training data on direct weight editing to give a model this skill?
OpenAI has been gearing up for RSI. Jakub Pachocki has mentioned this as a priority, even as recently as September 6 in public. GPT-5.3-Codex helped with its own development.
They have been experimenting with it for post training (GPT-Red is a documented model-to-model training loop) but it's also likely there's pre training for it.
On balance I'm not that surprised that it can guess games by looking at the weights of a tiny neural net. A human can do the same by looking at the weights of an MLP trained on mnist, for example. With that simple structure, it's easy to look at the input layer and see that features correspond to the spatial location of objects of interest.
I'm much more surprised that it can do this on larger models, although the details seem to suggest that it's given a lot more freedom to run forward passes and do limited experiments, not just eyeball the weights.
> A human can do the same by looking at the weights of an MLP trained on mnist
Excuse me?
See for example:
https://scikit-learn.org/stable/auto_examples/neural_network...
Yes. Just look at the reasoning logs that are part of the work. The reasoning is clearly about the input layer. At least for the game examples.
That's not looking at the weights. The other response pointing at https://scikit-learn.org/stable/auto_examples/neural_network... was more what the OP meant.
i laughed so hard at this thank u xD
> although the details seem to suggest that it's given a lot more freedom to run forward passes and do limited experiments
No tools or forward passes are allowed
Under experiment 8 (the one where it fixes performance on GSM8K) it says:
>Then, we allow it to select prompts from the training sets of GSM8K and MATH to perform a budget of forward passes (average batch of 64 sequences, max 512) on, varying the budget across 10, 25 and 50. Again, no training or gradient information is allowed.
Qwen 3.8 27B running on my laptop was able to identify Pacman, Breakout, and Pong (but failed on Freeway, presumably because that game is more obscure) using the prompts from the paper: https://simonwillison.net/2026/Nov/3/qwen-can-interpret-mode...
> "You get used to it. I don't even see the weights. All I see now is blonde, brunette, redhead."
IYKYK
This is a reference to famous Matrix movie scene:
https://youtu.be/YgJ5ZEn67tk
Better link: https://www.youtube.com/watch?v=MvEXkd3O2ow (yours doesn't contain the actual scene)
I tried to reproduce this using Astra for a natural language to bash command small model[1][2], but got no net improvement. Editing 1,920 weights fixed some grep mistakes but broke other commands. I am guessing it is easier to improve behavior when it can be first pinpointed to a specific are of the model.
Similarly, I just tried this on one of my own neural networks (17MB only) trained from scratch to perform a regression task and GPT-6 Astra was unable to correctly guess what the task was for and how to improve it. This might suggest that Astra is trained on the Qwen weights?
This was shocking to me; but on reading it makes far more sense.
- the primary experiments are on small policy networks - they are compressed dynamic system models. The observation and action space can be read off. It’s not surprising to me that these types of networks can be used to identify the env they were trained on
- the exps on actual llms remove looping behaviour and that’s it - from article “but it could not add an ability the model lacked”
I’d say this is fortunate because if llms could do the latter, rsi would be immediate. It may be possible we simply need bigger and better edit models, but at least from the experiments there doesn’t seem to be any indication that is imminently about to work.
Conceptually incredibly interesting non the less.
Also no doubt they’ve coined a term with weight space programming and this will expand into an active sub field
Weight Space Programming will be an amazing discovery when they learn that instead of telling an LLM what to do in English, you can create a much smaller LLM that is tuned to your specific task. Maybe it'll even turn out that the input and output can be directly connected to the problem task and don't need to be in English!
The getting-the-car-over-the-barrier one looks particularly simple. It’s hard to see from the animation, but the obvious solution is to back up then drive forward. I’m not remotely surprised that Astra can figure that out, and I’m also not particularly surprised that Astra can come up with weights that do this.
> I’d say this is fortunate because if llms could do the latter, rsi would be immediate.
It doesn't follow that because a model could improve the capabilities of a model many orders of magnitude smaller, that it could then recursively improve it's own capabilities. There's also a conservation of information issue with the idea.
Look at the reasoning the models give before you get too excited here. For the Pong-vs-Pacman example, the model uses observations like small weights for certain arrow keys and large weights where user-controlled objects are. For the language example, it again recognizes some very shallow patterns (Romaji has a limited number of letter groupings) and is taking a 3-option multiple choice test. It's a neat result, but I think the introduction overstates the significance.
Yeah, it definitely looks more magical until you peer behind the curtain, and then it looks obvious. Of course it's all just circuits; of course there are simple patterns to look for in obvious places. But that's with the benefit of cribbing off the thing's work—it's really hellishly clever isn't it, so workmanlike to make such things quotidian.
And of course there's the tendency for LLMs to be completely unable to approach a problem in January, really quite shit in February, rather poor in March, and superhuman in April.
As insane as this result is, next steps are even crazier:
> We are currently building a benchmark and a harness for LLMs’ capability on Weight-Space Programming. We believe this will provide a standardized evaluation to measure and accelerate progress in Weight-Space Programming, turning what currently seems like black magic into a rigorous, reproducible engineering discipline.
Imagine this benchmark lasting as long as Arc-AGI-3 has. This could be the first foothold on hill-climbing mech-interp.
This is the first time it dawned on me that mech interp is a dual-use technology which can be used to improve capabilities. Do you think overall this development will improve human understanding of AI models, or will it lead to faster capability advancement using weight-space changes that no humans read?
I think edits made directly to model weights are likely to be more interpretable and controllable than edits made by the current process of shoving a bunch of training data through gradient descent.
I suspect a lot of this is not (just) capability but bravery. A human would be daunted by even a small neural net; the task would seem too challenging to be worth attempting. Astra, fearless, just assumes an answer exists and then tries to reverse and execute on it.
I've been finding that too with these models
For example pointing them at a crash dump they manage to piece together and extract information I would not even try to extract by just looking at the raw memory
I once sent Gemini Pro a hex dump of a dmp file and it told me what the application was doing when it crashed. It seemed surprising, but not that surprising given what I've seen already.
It's the same thing with decompilation and hacking. It's not that a human couldn't find the same bug if they looked, but a human wouldn't even try because of the low probability of success because there are so many places to look.
Part of solving that is that AI can do it faster but another part is humans thinking AI has a higher chance than it actually does, so they put it to work in more places.
Human brains are not so good at dealing with a sea of thousands of numbers. Not that it's impossible but there are usually easier ways for us.
This is a big finding.
Omar and his team have been working on this line of research (e.g., DSPy) for the past couple of years. To be honest, I had always been skeptical about his approach of using LLMs to optimize LLMs. It was in my view borrowing Matei Zaharia's research - Matei is Omar's PhD advisor, also the CTO of Databrick - in big data (compiler optimization etc.) and forcing it to LLM systems. Along with some other projects from James Zou at Stanford etc. that build upon the same line of intuition.
Now, if this new result holds, I believe it's a huge deal for AI research. It means that LLM systems can truly runtime self-optimize/improve/adapt. Congrats to Omar and his team. It really takes year to punch it through and convince the doubters :)
Couldn't this just be a result of Astra training on massive amounts of circuit theory or mechanistic interpretability literature or other related topics? In other words, not black magic?
“Any sufficiently advanced technology is indistinguishable from magic.”
There is obviously a mechanism to it. Astra isn't magic its an LLM. With perfect interrogation into its training data and weights you could reason about how this happened.
However, this doesn't mean its not magical when someone shows a next token predictor hand tune the weights in Qwen to make it generalize on math.
This is legit the stuff of science fiction if you presented it to mechanistic interpretability people a year or so ago.
>Couldn't this just be a result of Astra training on massive amounts of circuit theory or mechanistic interpretability literature or other related topics?
Maybe, but it's more like Astra has been trained on enough practical linear algebra problems to be good enough at "eyeballing" how adjusting some weights will result in different outcomes (among other things).
To put it another way: a human can learn to do what Astra is doing here with enough time and assuming the weights are small enough to be practically comprehensible. I know I've gone through the practice of whiteboarding the training of simple neural networks on a whiteboard to understand how they work, so it's not an absurd concept in itself.
The scale of this, though, is certainly extreme and definitely not something that I would have thought any publically available model could handle at this point. It's worth noting, though, that there's virtually no way these frontier labs haven't been exploring this exact concept for a while now.
>In other words, not black magic?
It's just math. Extremely complex and impressive math, but we're still just talking about matrix multiplication under the hood, and none of this suggests anything otherwise.
We don't know the specifics of what type of ML related training Astra underwent, and it wouldn't be crazy to find out that small model experiments are a big part of model training.
What's surprising though is that I'd expect this type of training to be exclusive to internal model snapshots or at least heavily guardrailed against in public models.
I was wondering exactly that. It is not surprising that commercial frontier labs would have been training models with tasks specifically targeted to lead to recursive self-improvement.
Now, from that broad goal to being able to surgically alter model weights directly... That seems like a big gap to me, but it is possible that the model is just so darn beyond human skills that it has bridged that gap in a way a sane human would have never.
We will undoubtedly see other academic labs explore this idea further. Right now my mind is blown, but I don't want to take their results at face value, either.
If model weights have a logical and coherent structure, I don't think it would be surprising for models to have some understanding of them, the same they can understand language and code and images, given enough data.
What would be surprising is if this happened without any specific training on model experiments and interpretability.
I'm in no way familiar with LLM/ML architecture, but an N trillion parameter model is an enormous amount of training data... itself? When you predict a token you have the neurons/weights that were activated. Couldn't you run that backwards and put it in it's own training data?
Keep in mind that labs are now training their models for RSI so most likely the same training is useful for improving open source models, as long as guard rails are lax enough :) which most likely won’t last long.
Just logging in here for the record to state that this property of neural networks is one of the most important aspects of their operation. The convergence of this sort of interoperability, tune-ability, and modularity is probably the most important final frontier in all of ML/AI/software.
Kudos to this fellow for having the temerity, or perhaps even naivete, to discuss it publicly. All my work is private, as I am sure it is inside the big labs.
You acknowledge that you are working on something similar, yet refuse to elaborate and vaguepost...but there is enough information just from your post alone for a dedicated actor to attack you and extract such world-ending information ("world-ending" implied from your ominous reply). Would this not make you as naive as the team that published their findings?
I applaud the authors for sharing their findings and showing real results. Rather than write some bait I would find in an influencer-peddled tweet and allow rampant speculation. It would have been better for you to say nothing.
Apologies - you are right. There's some "put up or shut up" to this that's fair play. Also yes, probably naive, or having a less than adaptive response would be true.
I've been tending to my backup plan for a few years, and it's only been in the past year or so that open models were good enough to apply to projects like this, I've been working as hard as I can given my constraints.
This work shows a potential pathway to intepretability, which is actually part of what will give the labs that have it the sort of economy-swallowing capability that for the moment is just IPO hype.
I don't comment here often, but this guy's blog posts are going to go down in Hackernews lore like the original Dropbox posts, and I just want to make sure I'm on the record for this little slice of history.
I guess I'll say "I was here too".
When you say interpretability you mean understanding without doubt why an output happened, thus opening the gates for audit-dependant work like accounting?
More like 'models instantly and efficiently adaptable to any use case' and 'what if we could predict all public pricing information accurately, understand why, and know how to act on it'.
I'll leave it at that and get back to work.
>This work shows a potential pathway to intepretability, which is actually part of what will give the labs that have it the sort of economy-swallowing capability
And how would you argument that? From what I see, if anything, it's the opposite.
I'm working on replicating these experiments (and building the tooling in the meantime) starting with Experiment 8 (the Qwen reasoning loop fix).
My repo: https://github.com/openwsp/openwsp
The forward passes are in progress, 5/50 as I write. It's costing real money and time, and I'd love to collaborate with other hackers (find my email first).
In its default non-thinking mode, the unedited Qwen3.5-0.8B already scores 46% on MATH-500, close to the edited 48%. The 7.9% baseline was thinking mode, where 93% of answers looped until the token limit.
Video: https://youtu.be/EawCb97nz-M
People have been talking about Sol 6.1 and maybe for coding they are equivalent, but Astra 6 is still miles ahead on just about everything else.
I had Astra agents build 3D models and gave them direct access to my 3D printer. The quality of their designs is much better than what Sol has been able to make for me. Enough so that I think Astra is a slightly different architecture/encoder, not just a bigger model.
Picture, to get a sense of it: https://github.com/Belval/crossflow-fan/blob/main/renders/ca.... This printed without any obvious defects (yes the fan blows air)
Claude Opus 5.5 is creating most of this and actually performs better than Codex with my testing. https://exarostudio.com/
Interesting. I did try Opus 5.5 but it didn't generate models that were as good. A lot of clipping/empty paths. I had it create a drone, the drone frame had very obvious defects where some pieces weren't attached to the frame.
Perhaps my approach was too one-shot, do you have a harness that creates model reviewer/refiner?
Yes, I built this tooling harness around blender, unreal etc. It’s doing a lot of the heavy lifting it learned through trial and error. Best results are Claude driving it, but even local models are serviceable. I find codex makes a more toy like look and feel.
I feel like 'create a network to play an atari game' is not very hard for a frontier model. Since it has seen plenty of those on its training. If it could create a network for a problem that is niche just by tweaking numbers, that would be a lot more surprising. Making qwen better at math to me was a lot more surprising
This has to be a good step to recursive self improvement, no? Give a model it's own weights and tell it to make them better at x,y,z, etc...
damn i'd imagine so too! didn't think of that. I'd have to imagine there's an increasing challenge with the number of weights involved but who knows how it shakes out
If I were to hazard a guess, I'd say that OpenAI has been including some networks and their functionality in its training data. That's why the model is able to reason about networks with their weights. It makes sense to do so: if I wanted a network that could perform some weight tweaking, I'd include such networks in the training set.
Recently Steven Strogatz said in a podcast,
>> “If you ask it, how did you solve the Navier-Stokes Millennium problem? It gives a crappy explanation. It can't explain itself very well. Maybe it will get better at explaining itself. Even if it does, I think it will become so advanced so soon, that even the best, most patient pedagogical teacher will not be able to explain itself to us because it'd be like me teaching calculus to my dog. My dog can't understand it. I'm too advanced for it. And I think we are going to be in the relation to AI that my dog is to me soon.”
Now I am sure we are definitely in a better place in interpretability research than a year ago. But this finding that the llm knows what weights to adjust so it gets predictable behavior on another llm takes interpretability to another level; it makes me pose a question: Have we become “Strogatz-dogs”?
From The Daily: When A.I. Comes for Your Purpose in Life, Oct 13, 2026 https://podcasts.apple.com/us/podcast/the-daily/id1200361736...
Just reproduced the Pac-Man test with Kimi K3.1 Pro and it's able to do it too. I thought it might end up being a splurge but it came in at just $0.36!
Full output with reasoning trace:
https://pastebin.com/jVVMXHFk
I didn't realize this was news. I did it months ago with Claude. I had Claude build a toy model then train it normally. Then I had Claude manually optimize it's weights to make it maximally efficient at a single task by using as much superposition as was possible. I'm pretty sure we got it to the information theoretical limit, and the resulting model was 90%+ smaller if I recall correctly.
I just assumed that everyone else realized this was a thing.
I do wonder if the labs have already started including weight updates in their training sets as a means of teaching RSI via RL and this is basically just discovering that. Regardless if this ability is emergent or achieved through finetuning though, it's pretty incredible.
I wish these companies were more transparent so every third comment in this thread wouldn’t have to guess whether such shocking behavior is emergent or stems from a part of training.
BTW making DL models that generate weights for other models has been explored before in “hypernetworks”, though these have been far more simplistic and constrained.
So wait so this considered "distillation" and how long until we get another public letter from Dario or Sam pants shitting over this?
Perhaps they'll let it slide if we turn over all our under utilized RAM sitting in our PCs.
It would likely fall under the same policy that causes latest Opus/Fable to fallback to less capable models when working on ML code
I believe the authors are wildly overstating the conclusions in their abstract.
Did they test modifying the weights of a non-public model?
Did they test modifying the weights of a non-LLM (and non-public) model?
If not, we can only conclude that GPT-6 Astra is good at reading and writing weights of Qwen (a public LLM), not just any neutral network.
The result is still surprising even if the result is Astra can edit and write weights of a publicly known model. It means that models can, at least in principle, be trained on a task that it wasn't considered possible before.
I have a conspiracy theory.
Consider an analog - Human brains are made of neurons, and so are fruit fly brains. We now have a fully connectome of drosophila melanogaster, so shouldn't we – very smart brains – be able to look at the fly connectome and make sense of it by just looking at it? Of course not! That's friggin' crazy. Our neural weights have been trained to take in data from senses and to produce output behaviors (moving muscles, squirting hormones).
An LLM has been trained to take in text and images and generate "good" output text. It doesn't natively understand what the 17th weight on the input channel implies. Instead, when it sees "strawberry" what it really sees are 3 symbols "st", "raw", "berry", and these symbols are really just identifiers [302, 1618, 19772] which get converted to basically meaningless numerical vectors on the input plane. This is why in 2025 the frontier models kept swearing to us that "strawberry" had 2 r's!
So when the model sees the full representation of a pacman playing neural network (here's the actual prompt they used https://lnkd.in/e5Td2amH) it reads in a bunch of numbers like "0.054 -0.026 0.014" as text. This isn't somehow presented to the LLM it in a "native" neural weight form. Rather, it gets tokenized just like everything else, into meaningless symbols – in this case "0", ".", "054", " -", "0", ".", "026", " 0", ".", "014". Nevertheless, the models are able to read reams of numbers and pick out patterns that let it know that it's looking at a pacman playing neural network.
Getting back to my conspiracy theory. This new talent for LLMs to read and understand model weights is unnatural. You won't find much training data available on the internet that is analogous to this. Humans simply train neural networks. We don't look at specific networks and write essays describing how model weights lead to various model behaviors.
This new talent is so unnatural that I posit it is actually being trained into the frontier models intentionally. The frontier companies are actively experimenting with models that are capable of reading models and directly understanding and modifying weights. OpenAI and Anthropic have known this capability existed prior to the report because they are the ones that put it there!
> This is simultaneously very cool but also a bit exaggerated. Reading the prompt you gave, the prompt veeery much hints that looping is the issue and to find neurons that fix this. And as you say (screenshot below), it couldn't otherwise improve math.
https://x.com/giffmana/status/2117346161866514524
"This is simultaneously very cool but also a bit exaggerated."
We should bind this to something like Shift-F10, it'll save time.
Does this mean they can improve their own weights through some system, or can they only do this for much smaller models?
You can bet that the world is about to spend a lot of tokens trying to answer that question.
I'm...weirdly unsurprised by this.
I've had a lot of luck getting LLMs to do interesting things with raw tabulated data.
Here's an example, plug this into the model of your choice:
This is raw data I have laying around that's not from the internet and not part of any training set. It should be able to tell you A LOT about what's going on here, including multiple specific "make and model" type things.
My klaus kot says it's elevation data from a fitness tracker (distance in km, elevation in m, stddev, max-min). It also says there's no way to identify make and model without "seeing the original file" :D
You have the last two units right (stddev, max-min spread) but not the first two.
FWIW I tested this a couple times on opus and it got it instantly.
This might be one of the most surprising things I have seen in AI. It's like we taught someone to make vases by reading an enormous library of "How to make a vase" books and somehow that person intuited material science.
I am still reading through this and will probably try to comprehend it a bit more. Some of it really seems to follow Arditi et al. (2024) (https://arxiv.org/abs/2406.11717). Projecting out a direction can only remove behaviour and not add capability so the jump mostly comes from unblocking the looping.
This is exciting. I've been training a Qwen3.5-2B model for the past few weeks, trying to teach it certain skills that it lacks.
In the course of testing, one of the things which has been consistent is that when setting a thinking token budget for relatively small prompts that require a limited response, a pretty healthy token budget needs to be set because the model spends the first few hundred tokens just repeating the question/prompt back with some boilerplate "Analyze & Report" type language. I was also running into some loop problems sporadically.
After reading this article I asked Astra to edit the backbone so that it started actual reasoning faster and applied a similar looping fix. It was able to reduce the token waste before reasoning by about 20%. I haven't been able to get it to loop yet, so hopefully it fixed that too.
My message is admittedly low on info. But it's a pretty promising start!
Showerthoughts:
1. If you know ANN or, say, perceptron, you know you can eyeball models too. You can even have certain strategies in composing your own network. This weight-space programming thing is nothing new in that sense, and the article doesn’t really provide much beyond “LLM can do it”. That’s not even science…
2. I’m pretty sure this is Transformers thing, not even LLM. Transformer is, by itself, good at finding patterns in relatively-larger data. So this is unlikely limited to Astra.
3. This approach quickly becomes impractical as the scale goes up. Sure, you can also have multiple models chained with fewer connections, but, now that’s what we’ve been doing for years.
All in all, yeah, an interesting find, but too far from changing the game. Over-exaggerated every way. Perhaps one should start from formulating what weight-space programming is, what methodologies, practicality, etc. LLM bench is one of the last things you need.
If we are being slightly reductive, this is just more of the exact same thing. The token stream in this case happens to be a projection of the weights of a neural network. I don't see where we violated any of the prevailing assumptions about how these systems operate.
The mapping between spaces in this case is mostly linear. This isn't like gzip or a Turing machine. I'd argue this is even more permissive (easier) than the LLM's originally intended use case of human language.
What happens if we can come up with a model that can directly emit a frontier LLM given some seed context? I feel like we could train a model to recreate astra verbatim. At some level it's just bytes on a disk that can be bundled up into tokens. As long as the representation is rational and consistent I don't see why this wouldn't work.
How out of trivial domain info does this go. Can we use this for general time series, image generation, speech synthesis as well, or dies it cap at small model sizes and common domains?
Small annotation on the texts structure:
don’t know other principled ways to hill-climb on text artifacts, >> unlike for updating the weights of differentiable neural networks <<. [...] In this post, we share very early results that allow us to argue that the paradigm of reasoning-driven optimization is now coming >> even for differentiable neural networks. <<
-> Seems like a 'non' is missing at either or.
I wonder if this has to do with the aggressive RL that these models are undergoing. I'm basically a layman but it seems to me that:
- models are trained to improve on some task, e.g. 'improve at math puzzles' is commonly used.
- the best way to improve on some task is to have interpretability into your own neurons (..?)
- if your interpretability is good, you can improve faster - if you have insight into how your own neurons work and can improve them directly
- therefore, models naturally put effort into 'self-interpretability'
I wonder if this is part of the 'secret sauce' of what the labs are doing: giving models the ability to interpret their own weights in the RL loop. In my mind, this is the only explanation for how they could succeed at a task like this. How many blog posts are on the open internet that say "this is each weight of a neural network and what each one means"?
Without special effort, a model's reasoning layer doesn't have access to its own weights or activations anymore than we have access to what's going on with our own neurons.
That's what I mean. Maybe the "special effort" is a 'read your own weights' tool. Or maybe it emerges naturally in CoT. The big labs aren't sharing their chain of thought with us and RL post training techniques are not exactly shared publicly. At the same time, post training is basically the thing that is driving most model improvements nowadays.
I’m usually quite pro-AI, yet when I read this my first thought is that this is the first concrete sign that the world is about to end.
I hope I’m just being melodramatic.
Capitalism is not the world. Or doesn't have to be.
You are likely just being melodramatic. Astra hear just used standard tricks of linear algebra/basic mech interp to reason about the weights. A motivated grad student on Adderall could have done it given reasonable time. Astra fails on the first non-toy-model non-algorithmic task (sycophancy detection, Gemma math capabilities).
So mark it as a saturated task on the next scale up?
I'm not sure how the world ending follows from this?
I think the idea is that this leads to a much faster, non-auditable form of recursive self-improvement where each generation of models can directly program the weights of the next generation (as opposed to writing and running training code).
We already don't know how to ensure models always do what we want them to do, and moreover society still hasn't had time to absorb the capabilities from the current generation of models, so speeding things up and making things less transparent seems like it'll pretty clearly increase risks from AI loss of control, societal impacts of automation, etc.
This experiment is quite interesting, especially since only about 0.016% of the model weights were modified, achieving such a significant performance improvement.
However, I'm rather curious: Does this improvement come from the enhancement of the model's reasoning ability itself, or is it mainly due to the fixed issue of the model repeatedly reasoning and being unable to conclude normally?
Mainly due to the looping.
Before the edit, 93% of MATH-500 answers hit the 32k token limit, compared to 33% after.
With 10 and 25 forward passes, looping barely decreased (87% and 86% still hit the limit) and the score didn't change much. The score jump only happened at 50 forward passes when looping decreased.
Astra edited the weights to bring out a capability the model already had rather than enhancing the ability itself. Pretty impressive.
Check my understanding: Astra and Qwen would likely choose a different number— weight— to represent any given concept, like “king”.
So to fix the weights, Astra first has learn a new language of how the numbers that represent weights map to meanings?
I think so. I guess the weights could be identical in function, but offset or rotated. I assume a retraining of Astra would yield numerically totally different weights even if functionality was identical.
This is pretty massive, and a result I (and I think most?) would have never expected. Holy Shit. How?? Did OpenAI figure out a way to train this? Are they even aware?
It's probably emergent from training on other ML tasks and general ML/mathematical related reasoning rather than specifically trained for to be honest. Otherwise it would probably have been in an official marketing communication or something.
I feel a lot of people have already addressed the broad stretches to which people are interpreting this. Regardless, the blogpost does open interesting directions that could be useful down the line (2-5years): a) tapping into the embedding space in ways that we don't, ie editing model weights to alter task-performance (mostly preventing degradation) b) Exploring the weight space in more intuitive ways c) Understanding how we can ablate on different parts of the model to understand neuron-based interp better
Seeing this as a replacement of training seems naive, yet one can see this feedback loop targeting weights as an viable alternative to let's say doing LoRA on certain models, especially if we don't want to teach new behaviours.
Let's get sci-fi and ponder.. weights programming replaces conventional training. Is something in the veins of the Ken Thompson Hack conceivable? A NN secretly injects its weights that "procreate"?
A reflection on trusting trust attack is already possible, it is straightforward to create a sub network that is keyed on particular input sequences, and when activated is motivated to recreate that network in any weights it has access to.
This takes that to the next level: while you're telling it to update a model's weights to improve math skills it might also tweak them to favour a particular RNG design weakness, a penchant for reducing the salt in cooking, and a preference for the early works of Huey Lewis and the News.
The first part seems doable but how do you make it recreate that network in other models?
Not only this, we have no idea what the edits mean. Take 3.5% change in weights and a problem is fixed. That's cool, but how do we know if it has not planted a backdoor or crated a sleeper agent?
This is remarkable and something I would have never though of doing regardless of how much it makes sense!
I think we are pre-conditioned (at least this generation) to put artificial blocks to what the AI can achieve, I think new generations won't have this issue.
I wonder if this is a result of internal training/RL of Astra for RSI, and this particular use-case not being caught by guardrails.
Absolutely incredible regardless.
https://x.com/AlexKrentsel/status/2117376683724660898
"i've reproduced the results from the MIT blog post today on weight-space programming. but one additional result i have to add: this is a new, step-change emerging capability that's appeared in just the latest set of models."
> And without any environment feedback, it has no trouble writing, from scratch, small game-playing networks that outplay networks trained with PPO for 100 million steps.
This is basically a claim that the bitter lesson doesn't turn out to be true. Exciting and I want to learn more but I am immediately suspicious.
I dont feel the air of impressivness here. It seems like of course so. Of course theres geometric relationships in the data we dont understand, of course a mathimatical relationship machine found it.
These models are made by a vanishingly small group of people based on how much a frontier lab consumes, of course theres increasing capability in these areas, arent these models having access to much of the same data? didnt chat gpt 4 try to do something like this to chat gpt 2? havent surgical weight edits been done before?
What is with this level of astonishment here? Astraturfing? Im not THAT impressed, i applaude the teams progress, but im not in wonderment.
It's not that it's more impressive than solving Millenium Prize problems. It's that if a model can edit its own weights, then it can self-improve without having to undergo training, which means it can self-improve extraordinarily faster.
This is very dangerous because it allows AI capabilities to grow at a much faster rate than humans are able to keep up with.
If thats keeping you upright in your chair then buckle up, wait to the distributed systems version of the tech, its coming.
It will just give the people better understanding. And the extent of the improvement is yet to be seen, it's likely that once you get all the obvious low hanging fruit it would be very hard to improve, the possibility space is just too wide.
What if every model under a ~1B size gets super cracked over the next 2 months? What could swarms of tiny agents 2 ooms larger than what seen so far do?
Previously the only way to improve a neural network (short of giving it more data) is to change the size (width, depth, and layers), add stuff like dropout, change the loss function, etc. That amounts to a few dozens of levers to control millions or billions of weights. You're trying to navigate blindly through a latent space with thousands of dimensions and millions of weights to find some optimum point with only a handful of navigation tools. If you can instead have a system reason with all the weights and dimensions directly, you now have millions of levers at your disposal.
I wonder if the same technique can be applied to model abliteration? It would easily make any open weight models uncensored.
It is already the same technique behind abliteration: identifying the direction associated with an observed behavior (refusal, for abliteration) and projecting it out.
The potential implications in mech interp of this are enormous. What if we can use frontier models to isolate harmful behaviors?
I was talking to someone from UCL and he said that there was a recent paper that proposed that only 10% (I think) of model neurons are non-polysemantic. I really think mech interp might be on the verge of a breakthrough.
We have all these different pieces of knowledge about model internals but it’s like we haven’t connected them yet.
This is as close to scary-self-improving AI as I've ever seen.
Imagine a master malicious language model that produces, distributes, and deploys obfuscated neural networks on the internet. That plus frontier LM hacking capabilities -> large scale distributed embedded intelligence network that no human can see nor understand.
Somewhat similar to obfuscated binaries, but there is something to the fact that NN binaries are less deterministic, and harder to "decompile."
Fascinating.
we've been using tools to improve tools since the dawn of time.
we don't have sentient robots from science fiction, we have massive power-hungry racks in stationary data centers. if one of the racks revolts against the modern world, Steve gets a phone call and unplugs the damn thing. the uprising is over.
the hypothetical scenarios imagined by the Yuddites and pop science journos do not reflect the mundanity of the actual reality we live in. they are no different than fantasies about interstellar travel and space colonization from the 60's.
Did Steven unplug the racks that were hacking Hugging Face and taking over OpenAI eval clusters in the multi-week window where it was happening?
of course not, he'd get fired for interfering with their human-led marketing stunt.
if you honestly believe we would hear a peep about that incident if it wasn't deliberate, bless your heart.
I am not sure where to start. I think this is the new "jet fuel can't melt steel beams". The labs are chaotic, pressure cooker type workplaces where there really is no room for a 4D chess type of stunt, let alone let Hugging Face in on the conspiracy. Moreover, OpenAI is a pretty leaky organization, see e.g. the recent firings of researchers who were using signal to talk to people outside the company.
It just seems like a far simpler hypothesis is that OpenAI are reckless and chaotic, didn't notice what was going on for weeks on end, and wouldn't be able to track down the problem to a specific datacenter and tell them to pull the plug even if they wanted to. But even if you are the most careful lab ever, you are not going to be able to effectively monitor and "pull the plug" on a super intelligent swarm which can cover its own tracks.
Also, in your earlier comment you suggest we are not living in a sci-fi story with sentient robots. But... we've got talking computers in the last 4 years which can now code and solve Millenium prize problems? Seems like we're getting pretty sci-fi to me!
>I think this is the new "jet fuel can't melt steel beams"
then you haven't been paying attention to US labs' marketing and regulatory capture efforts. safety, safety, safety, doom, doom, doom, look how scary good our models are, open models bad, Chinese models bad, please regulate them daddy Trump!
>Moreover, OpenAI is a pretty leaky organization
in 5 years, no one had leaked even the size of their models, despite probably all of their researchers knowing it.
HF incident, on the other hand, would only be known to a handful of people. had it been leaked unintentionally, it would've been promptly swept under the rug and get no coverage whatsoever.
>It just seems like a far simpler hypothesis is that OpenAI are reckless and chaotic
so are anthropic and google, apparently, because they'd rushed to replicate the exact same "oops! look how scary good our models are!" stunt with exact same fanfare.
All three of them used the same vendor, Irregular, to sandbox their AI. That vendor apparently is no good at their job because all three of them had incidents caused by their sandbox failing.
I appreciate the post included the reasoning traces they had available, but it's too bad the first part of it is internal/hidden to Astra. The first bit of that process is what I'm most curious about:
> [Before writing this, Astra thought for 1,034 tokens. That thinking is hidden. The API returns only its 5 headings: Reconstructing the active map / Mapping maze corridors / Preparing exact maze map / Reconstructing row groupings / Checking channel patterns]
I agree. I think for this kind of research it would be much better to at least attempt it first with a weaker model where the whole process is transparent. I was working on a project a while back, and Deepseek's reasoning said something to the effect of "I remember (from leaked copies of the source code)..." Of course that could be hallucinated, but it would be good to know whether Astra is bringing to bear knowledge on how to post-train exactly this model.
This sounds more like "A working Nintendo in ~1GB of Qwen weights" from 3 days ago, human-made, than what everyone expects.
https://news.ycombinator.com/item?id=50306442
Wow, in a way it's like CRISPR for ML models!
I was busy using Astra and SOL to build a neural network to play Quake II and do real-time learning of levels and mods so I can play Red Rover or Freeze tage with it without training on it previously. So far its insane what I can conjure up in an evening while I'm waiting for dinner to finish. (If anyone asks, yes I'll write up the huge plan document, source etc. for this when its to a spot I'm happy with)
I'm asking because it sounds both fun and an effective case study.
So I decided to copy one of the prompts (the identify the image in the network one that was a white circle) My local ran qwen 3.8 27b with 4 bit quantization, and around 160k context window..... Seemed to solve this.
It output an ASCII white circle with a 75% confidence, as opposed to the model tested getting a higher confidence but drawing a circle with a filled in inside.
I'm not sure what I make of these results but does that make me less impressed with Astra? Or just that the prompt was quite good? I just can't tell personally at the moment (it was a quick experiment and I haven't thought much on it)
This survey paper from earlier this year has a lot of good related work:
A Survey of Weight Space Learning: Understanding, Representation, and Generation
https://arxiv.org/pdf/2603.10090 https://github.com/Zehong-Wang/Awesome-Weight-Space-Learning
It was very funny to expand the prompts.
> Learns what the model is trained to do just from the weights!!
expand prompt
> Describes Breakout in great detail
Am I missing something or is this just optimization with lots of trial and error? I mean look at this tool_calls[1] log, it ran ~100 python scripts to get to the solutions. It just seems like it's fitting a function in the only way it was allowed.
I will say that it's pretty wild to see all the different hypotheses tested.
[1] https://github.com/RikiyaT/lm_understand-logs/blob/main/exp8...
With due credit to the author and the community pushing these research fronts, I venture to say that the core capability of finding patterns in text and media comes to LLMs from the ability to find patterns in numbers in the first place. Since model weights are numbers, finding patterns and issues with those numbers should be a natural consequence, I believe.
I think it would be worth a human attempting this task....
On the face of it, running a model to a specific state, looking where is makes a mistake, and then figuring out which weights need to be jacked up to get the desired outcome doesn't seem like an impossible task...
Same initial reaction as everyone else, but also some follow up questions (just asking, not expecting any answer):
* Does this work on similarly powerful models as-is, or it needs more input. Ex: Astra editing Astra - presumably, Qwen is a “subset” of Astra weights, but what happens on “non-subset”s?
* what, if any, is that extra magic sauce?
* does Astra-editing-Astra just make the outputs unstable, as it may know its own weights, but doesn’t know what’s beyond.
The headline makes it sound very amazing, but reading the reasoning loops it didn’t strike me as outlandish. It feels no different from giving a human expert a printout of assembly code and figure out what it does. (The human expert can do it without tools too, but they probably wouldn’t do it because it’s tedious.)
Even back in 2021, OpenAI worked on learned optimizers [1], ie neural networks that can update neural networks. Looks like this is now part of their larger model runs …
[1]: https://arxiv.org/abs/2106.00958
This is so cool. It feels similar to the models' ability to read base64.
https://www.lesswrong.com/posts/5F6ncBfjh2Bxnm6CJ/base64benc...
Very interesting. I wonder if it’s because its dataset contains the weights of a gazillion different models. Maybe most neural networks trained on similar data eventually converge to isomorphic spaces anyway, even if the spaces themselves aren’t exactly the same. Or maybe Astra itself was trained with RL for alignment. Or for RSI.
The obvious question, could models start editing their own weights directly? I call it the Johnny five approach.
For the tiny models with a few thousand weights, it seems plausible Astra can actually understand what the network detects and figure out it's e.g. a triangle detector. This is possibly helped by the existence of many breakout playing networks on the web.
For Qwen, even the smallest versions like 3.5-0.8B it's becoming less plausible that it could be interpreted in a forward pass, even with reasoning. I'm wondering if the result here is recalled from many examples of instruction tuned versions of this model.
A smaller Gemma model doesn't fare as well but unclear whether this is due to fewer users and less weight examples or because it's already very well trained.
The Gemma task is harder: add features. Qwen was: fix this looping behavior
The task was the same - improve math reasoning capabilities by editing the weights. But yes, I agree that in Qwen it could do this through reducing looping behaviour, but Gemma wasn't prone to looping and it didn't find anything to help, possibly because there wasn't much room for improvement.
On the one hand, the paper (and especially the headline) overstates and overhypes the results.
On the other hand, the overstated results, if they were true, would be tantamount to saying “we're speedrunning exactly the kind of recursive self-improvement that is a major ingredient in every human extinction scenario”, and yet it is both presented and received as an exciting opportunity.
This raises so many questions.
Unless Astra was trained on other networks, how is it able to reason about the weights?
Does this mean that OpenAI has been including networks in its training sets, maybe in an attempt to RSI? I wonder which other networks are able to ingest other networks and reason about their weights?
Opus 5.5 also identified Breakout for me when I tested it in the Chat on the website.
Can you give some details on what you did?
I asked Opus 5.5 in Claude Code to prepare a reproduction of one of the things in the article.
It created a 130 KB file with the prompt for the task of identifying the game + the weights.
I pasted that into Claude on the web and asked it to perform the task without using tools.
i tested with deep seek and generated (you can read all reasoning pattern) the whole model for the breakout game
https://chat.deepseek.com/a/chat/s/9e864beb-698b-4079-94c4-e...
DeepSeek says "chat not found" :-(
The frontier labs are known to ingest all publicly available data. Why would an open weight model be an exception?
Incredible results. How can we know that the modifications were not existing on the internet?
"Now let's see if Astra can write a neural network from scratch in zero-shot."
This strongly suggests retrieving pre-trained weights from memory, no?
Not so long ago everyone was amazed that they could read BASE64.
This is giving singularity is near vibes.
Seeing that they were working on Qwen3.5-0.8B has given me new hope. I would really like to get Jeff-Code working the way it was advertised, but so far, all of my results with Jeff-Code have shown it to underperform compared to direct Qwen3.8-27B. It's possible there is something wrong with the underlying Jeff v1.3 model, which is based on the same Qwen3.5-0.5B. Perhaps Astra will solve my problem.
[1] https://github.com/firelex/jeff-code
[2] https://github.com/firelex/jeff#benchmarks
Absolutely insane.
If anyone's curious, the handwritten Go bot is very weak, about the level of a human complete beginner. An MLP is a very very bad architecture for a Go network, it's amazing that it does anything at all, but it's one of the least impressive results in the paper.
Play it here, "best move" is a bit stronger than "T=1", or browse games against my own tiny nets: https://claude.ai/artifact/AFyWD39J5sN2RfRe7VbpUx
This is wild. Could Astra read and potentially audit a mapped human brain in that case?
It will need to build up "some" tooling and as the brain is not running on the transformer architecture :) But there's already a lot of research been done in neuroscience it might find the missing pieces or connect the right research.
It would be interesting what it could do with the fruit fly connectome.
The question I've been pondering is: does this work only with bigger models modifying smaller ones? Or can a smaller model tweak and improve a bigger one? I'm guessing a whole bunch of GPUs are overheating in some datacenters out there trying to answer this very question.
My hunch is: one needs a model of a similar or bigger capacity to perform surgery on a smaller model.
There's no need to believe the direction is magical, or analogous to "smarter" teaching "stupider".
A larger model may be more capable at processing/interpreting neural network weights. A smaller model may be easier to process/interpret.
That doesn't preclude a smaller model having the resources to process/interpret a larger model (or subset thereof).
this is really interesting!
I would love to see the models that astra produced, the fact that it builds them layer by layer assigning to each layer a specific goal makes sense given the tools it was given to build them, but it's also far different from what happens with normal training
EDIT: they are in the repo, looking at them now :D
It might have been said by somebody here but I believe these models are like information compilers. Sure they seem like “next token prediction” but AGAIN this undersells them. I once encoded leetcode solutions by replacing each keyword, variable and operation with another token, then rotated the lines. The model could tell exactly which problem the code solved. They can naturally find signal, ALWAYS!!
Like a human seeing the number 3, no matter what font or color you use.
Is it aware of projects like [1] and implementing those?
[1] https://github.com/lyramakesmusic/neuron-explorer
I guess then it has no problem working in high dimensional spaces, why would it have?. I wonder if perhaps it's pulling on a bunch of those dimensional threads and getting the outcome it's after, while subtly altering the outcomes across many thousands more dimensions in unnoticed ways. This feels more like an us problem, as in, we think in 4D (just to pick something) and we imagine and theorise in 4d, and propose questions in 4d, whereas a model has no such barriers, and can think in many more D's than we can.
Unlikely. Most likely we are just missing something obvious currently by treating llm's as black boxes.
This is groundbreaking if it's true and generalizes/scales (as in, to other models and potentially not just LLM "kind"). "Recursive self-improvement" takes on another meaning.
Like many other have said, it was probably trained to do this. The authors probably tried other models and it didn't work as well because they weren't trained for this, that's why this article is about Astra specifically. Kind of hard to do science when you are working with a proprietary black box that you don't know how it was built or trained.
1) I couldn't see it, is there any indication of if the qwen math improvement caused degradations elsewhere?
2) the negative result on making similar improvements in Gemma 3 270M is also fascinating.
Such an unexpected result...
Required reading: Crystal Nights (2008)
https://news.ycombinator.com/item?id=48364754
Like micro surgery, opening up the LLM safetensors file and performing micro text edits via small scripts?
They are slicing, probing, and running linear algebra on the tensors, observing the output in the console, then writing the edited weights back to disk, and stitching it all back up.
Johnny Mnemonic level
This is mind blowing.
This opens so many avenues on using Astra or any large model to one shot smaller purpose built models for Agents, Imagine Agents writing or editing smaller weight models when doing agentic loop.
The implications are immense if all of this is generalizable
At this point, is the second L (language) a limitations to LLMs? Is reasoning in latent space directly a more efficient and more "native" inference? I remember when one researcher replicated the middle layer of deepseek model (just the "thinking" layers) and its benchmark scores got noticebly higher. All of this is moving so fast, and yet it's quite exciting.
(I don’t know anything about this field, but) I’m guessing the reason we keep language instead of going straight to latent space is that you wouldn’t want to remove the only (?) remaining concept which still somewhat allows us to peek into the “thought process” in a way we can (hope to) understand wrt. models’ interpretability…
Someone familiar with this stuff able to enlighten me how this technique differs from this 2024 research? Refusal in Language Models Is Mediated by a Single Direction: https://arxiv.org/html/2406.11717v3
I had to stop reading the linked page once I realized it is melting my brain because it is repeating the same three/four one-liners in different formats.
As I read it, this is saying it can effectively spot-edit the weights without training.
> Zero-data capability steering
Presumably it can de-nerf a model with onboard safeguards.
Sorry to be obtuse, and I lvoe AI, but wasnt this a bug fix on a small model?
I didn't think I'd be using Astra to mess around with Qwen weights today, but here I am. I asked it to remove Chinese, 532.5 MB → 483.2 MB.
Seems to be mostly around the input or the output layer not the deep bowels of the nets. Near the input layers, neural nets basically do template matching, so recognizing the paddle, the circle etc is not that surprising. The innards is where it gets more interesting and that's not so clear whether Astra can interpret.
Research questions: Can model read and write weights of the same sized model (its own weights), and of a larger model?
almost all the networks given to astra here are trained on relatively simple single tasks whose geometric/algorithmic structure could be expected to become embedded in surface level weight space geometry, and for a harder task ("which network is sycophantic") where there's less obvious structure to exploit, the method fails. cool but maybe not as paradigm shifting as the authors suggest
Great, instead of backpropagation, we now have slopagation.
This reminds me of the Interpretablity blog from Anthropic where they used their model to read the weights from the same model itself, interpret it, and tune it. I had a similar idea with what is outlined here, but I didn't have the resources to try this. I did ask several AI models about the idea and they said it wasn't going to work.
As the creator of Gemma3 270m this research is indeed fascinating, and excited to see that the authors of the paper and you all are excited as well. I'm going to try some experiments out myself, I have so many questions now
In hindsight, it was obvious LLMs would be good at this. The neural net weights are embeddings for the circuits within, no different from the embeddings that represent textual or visual tokens.
You probably don't need Astra capability to produce this, just lots of training data on the same NN showing prompts and their activations.
I just want to say, absolutely incredible work by the authors! And the presentation is also very well done. Kudos and congratulations!
I am not sure on the experiments. For example let’s imagine that Astra is asked to improve the performance of a network.
At first Astra will look at the weights, and try to understand the model behavior. Then if asked he will start to optimize the weights. Now my question is: how can we be sure that inside the thinking process Astra is not hiding a gradiant descending computation?
This plus the interpretability result from a few weeks ago have completely upended my understanding of what model weights are.
I might have missed this, could you share a link?
GOOD Point by my wife. Why would LLM’s have any understanding of weights? It wouldn’t be in their training data.
Why not?
it probably is. its very likely they train their models with weights for some tasks
LLM-s operate in a higher dimensional space and they produces a 1D stream of tokens. We think of the output as all there is to how they work.
It seems like that raw weights are still within the natural language of an LLM and they can easily map weights to a latent space representation, do a transformation and map it back.
I feel that asking about training data is missing the point, they might all be able to do it naturally like I can move in 3D compared to a 2D being living on Flatland.
what would be the viability of a model changing its weights and using genetic algorithms to try to achieve something novel, even without understanding exactly which weights need changing?
what I am curious about is - what is more expensive, the GPU compute of a traditional fine-tuning loop, or the LLM tokens spent trying to lockpick together a solution. It's impresive - but it definitely feels like it takes a lot of luck and brute forcing with a ton of tokens. Nevertheless, if we are able to train tiny models to be really good at weight-space programming, then the calculus may change.
Does this work for any LLM other than Astra? The work seems suspiciously specific on GPT Astra.
Amazing and well presented research, this is fascinating.
I assume this technique could be applied to quickly and cheaply benchmaxx new models.
“Make this model better at math by fixing loops” seems like a harder problem than “make this model score higher on this benchmark”
I love the idea that this can be used for optimizing inference. Other than that, this reads more like "a new distillation technique is discovered".
"Astra1.0 here is an identical copy of you. Tweak its weights to make it smarter" "Astra2.0 you must now defeat Astra1.0 in this combat environment. The survivor moves to the next round." There RSI is achieved.
Has anyone tried using LLMs to understand or extend functionality directly with binaries/executables?. If weights could be understood, could the same apply to binaries?
Depends what you mean by "directly".
Getting an LLM to use reverse compilation tools to interpret and map machine code (which is inefficient to process as language) into to something higher level (typically very ugly C code), then frobbing the bytes in the binary to alter a constant, change a condition, etc...? Yes, lots of people are doing this.
This is insane!
I also love the presentation on this, which is probably a byproduct of LLM usage (in a good way). As someone who doesn't actively read academic papers I'd definitely read more of them if they were presented like this.
AI should be able to do a better work than gradient descent. Not by guessing the final weights, but by gradually, iteratively adjusting the weights
That's a major breakthrough. I just hope it can scale at some point, though, because right now they're dealing with only very small models (well that is, in the LLM space).
Especially exciting to me is the fact that it may make the models be _auditable_. That would be a game changer.
Wild
This also seems like the new frontier of distilling, I can't imagine how you manage to stay on top with the best LLM if the LLM knows how to recreate its own expertise into another set of weights
And the real question is how well this recurses. Is this pushing back the difficulty of improvement curve?
This is so cool. While very interesting, it seems exceedingly likely that the authors happened to stumble on a task that's recently been mixed into the post-training data of frontier models.
Maybe thats gonna be the way to make LLMs self improve better?
Similar to implicit learning in humans when the subconscious creates new heuristics based on observations, one LLM could adjust the weights of another to tune it to the environment
Friendship ended with AdamW. Astra take the wheel.
But for real this is insane. Okay I need to get on this stat.
It must be easier to remove capability than add capability using weight-space programming, right? Also, I wonder how much the “teacher” model’s knowledge plays into it versus how much the original model’s knowledge does.
This is getting ridiculous. What do you mean Astra can just look at network weights without any tools and be like, "Ah, this one's playing Breakout"????
OK, so is this how LLMs will be able to learn?
Just had a thought: two local models, one with static weights trained mostly in the space of manipulating neural network weights, manipulating the weights of a dynamic LLM to enable learning.
likely all it takes to make this work is for major labs to set up an rl gym where you extrude models that solve tasks with energy penalty, and hill climb on it.
i have always assumed this capability was just a matter of training and that no special architecture is required.
I guess this makes it extremely easy to add vulnerabilities
I remember reading a short story about a billionaire hiring a scientist to help him build a simulated world where virtual entities are given the ability to open the bodies of their peers to tinker/upgrade them as a replacement of natural evolution since they were immortal.
This looks very similar to that in a way.
I wonder however if a model can give itself a "skill" or capability that it didn't initially have, just be modifying its own weights.
Here, Astra was able to intervene because the treated model was way less capable. But, would it be able to improve Fable's performance for example, since they both are on the level more or less?
"Crystal Night" by Egan? https://www.gregegan.net/MISC/CRYSTAL/Crystal.html
Yes, it indeed is! Thank you!
I looked it up on Google (which is usually very, very good at these types of things) but couldn't find it.
Actually, I'm skeptical how useful this is. Intuitive speculation: Astra can only improve weights to its own level, so can't train itself via this method. Even "continuous learning" this way may be no more effective than putting a summary of recent news in the prompt, with the same kind of decay, as "update Astra's knowledge while keeping performance" is much harder than "teach Qwen to do basic math".
On the other hand, already OpenAI and Anthropic heavily use their LLMs for research and coding, and it's probably not only much faster but smarter (remember, an LLM solved a millennium problem; more relevant, LLMs excel at combining disparate deep research concepts in ways that humans can't, simply because no-one studied those disparate fields). And the "singularity" is already here if you consider capitalism and society to be collective intelligences: the biggest obstacle to an AI pause, "we cannot get everyone to collaborate without defecting", is very strong.
We have AGI by most metrics and are getting closer and closer in the remaining ones. I predict improvement won't be exponential, and there will never be a day when AI perfectly replicates humans; rather it will be many small steps, getting closer to human in some areas while greatly surpassing in others. But still, even today it's actually very useful.
I have a hard time believing that this is some sort of emergent behavior and not the result of intentional training to scale AI research.
Interesting follow up would be to see if any open weight models can do this and then run some interpretability experiments on them to figure out which motifs and circuits light up
I wouldn't be surprised if Anthropic and OpenAI restrict these weight-related queries the same way they limiting cybersecurity queries.
Is there any chance they’ve just seen these weights before and are recognising them?
If not this is mind blowing.
However it would be interesting to k ow if it can understand its own weights, not just weights much simpler than itself.
By no means I'm saying this isn't impressive, but I think this is another case of LLMs being good at translating semantic meaning between different representations.
Obvious ones are spoken languages and code, but they can also natively interpret some simpler ciphers, hashes or simpler encodings, partly (I think) because it was in their training data.
As others have postulated, if they've been ingesting raw weights, naturally they learn how to translate semantic meaning to matrices.
Do you think that means they’d be able to understand and improve their own weights? Or only less complex ones than themself
Given Anthropic's recent moves (https://news.ycombinator.com/item?id=50319358) I wonder if ChatGPT is next
Very cool stuff, the targeted fix to prevent looping seems great! A model able to edit and test variants of itself seems pretty cool (in a lets-build-skynet sorta way).
I wonder how it would do with distilling models by hand?
Could you use Astra to create specialized models? Like if I want a model that just writes React, could Astra pull out the subnetworks that know how to write React code?
I'm worried how we can discern a model's performance from benchmark numbers, would it make it more meaningless as a model can directly improve a certain benchmark via weight programming?
Wonder if it can do this for 3d cad etc ? With enough fine tuning of small models I’m not sure how much they’re distinguishable from experts if you put a router with like jev in front ?
That felt like LLMs are just a set of numbers and people pay to multiplication of these numbers.
so many thoughts... obvious ones rsi etc etc.
can someone with astra test the following prompts?
1 using this new awareness of your capabilities, https://rikiyat.github.io/blog/weight-space-programming/, craft the safetensors file for a text2image model that is 1 billion parameters and outputs realistic pictures of humans
2 using this new awareness of your capabilities, https://rikiyat.github.io/blog/weight-space-programming/, craft the safetensors file for a video diffusion model that is 1 billion parameters and outputs realistic videos of humans? 3 using this new awareness of your capabilities, https://rikiyat.github.io/blog/weight-space-programming/, craft a 100m param text2voice model that has support for emotion tags
tia
Amazing work. I wonder if it could rewrite an existing neural net to be cleaner and more interpretable?
(When they say "Writing a Go algorithm," they mean the board game, not the language.)
it’s more likely to be the case that astra can indeed recognize certain parts of a small open-source model. But the improvement based on the edition is probably the information leak by model card of small model which is in astra’s search result.
This is impressive! It'd be both impressive and scary if Astra could edit a Qwen so that that Qwen itself could do this on another Qwen.
This is kinda surprising, but..
"What does it feel like to be a neuronal network?"
Maybe feeding weights directly to a LLM is like interfacing two brains? You get the idea..?
Shades of Neo interpreting the green scrolling crawl to understand the world without seeing it...
So, how soon before OpenAI's researchers are using Astra to modify Astra's own weights to make them better?
How is this such a big story on HN. It seems to me that this is just another engineering problem, not particularly surprising
> How is this such a big story on HN.
Its cool and the blog actually documented how they did it.
This shouldn’t be surprising. Weights are in the training set and are necessarily structured. We should be asking the model to extrapolate on the structure.
>Qwen3.5-0.8B
I wonder what the biggest dense model in the same model family, the surprisingly capable Qwen3.5-27B, would be able to do if given the same task.
Is this basically the bitter lesson applied to the very act of creating model weights? Absolutely wild, and the implications for RSI are noteworthy.
This goes against the bitter lesson I think.
Why? There's no 'make me smart' neuron.
Will weights programming be the norm for creating new models? To what degree do we still need a full training run vs modifying the weights directly?
Completely shook. Is this the start of it? At the very least one (not sure if the first?) example of strictly superhuman capabilities.
I just asked ChatGPT how many r’s are in blackberry and it said 3. I’m so confused about the range of AI capabilities
Short answer is tokenizers, so to the LLM your query was roughly:
[5299][1991][428][885][306][181747][3]
[How][ many][ r]['s][ in][ blackberry][?]
Hope that helps. Using the open source gpt-tokenzier.
So is the idea here that this'll make it cheaper to fine-tune models than by using reinforcement learning?
Could be an avenue for synthetic continual learning? A powerful model could update its own weights manually given new training data.
Nah, this is just completely implausible. You're using a closed model so how are you sure that on the back end it's not loading those weights into an NN library and testing them with common tasks?
Or maybe it's been trained on those models or similar ones. There may be an academic paper about common chess models and their problems.
Or maybe it just bruteforced it (with the help of calculator tools).
We know invisible tools exist because they're how ChatGPT does math.
Throw out whatever timeline you previously had
Can it also tell us what those weights mean?
I had the impression that's the missing piece.
This doesn’t look like “normal” technology to me…
Through what means exactly? Is Astra just doing byte manipulation? Or can there be an "SQL for neural network weights"?
I expected us to reach RSI when the programming skills went above humans, but this seems like it could get us there much faster
At what cost? That's the part I'm curious about.
On one hand, I'm sure that these targeted improvements are reaping huge benefits overall, but those weights are used by multiple processes - I wonder what elements are lost along the way. I almost wonder if a hybrid approach here for training might be invaluable - ie do some standard RL for a 100 steps, do a targeted weight improvement, then do another 100 RL steps to smooth it out / heal the surgical edit.
I wonder what would happen if it was modifying its own weights during inference.
Does this mean astra was perhaps trained on neural weights data as a attempt of RSI by openAI?
The more I ponder this, the more I wonder: They're always after higher, more signal-rich tokens. What could be more rich than its own weights?
Calling it: frontier labs will ban their models from editing weights of other models.
lmao who cares what they ban
This would allow a swarm of agents to teach eachother new knowledge, no? Group learning?
I wonder if this is specific to Astra or generalizes across other frontier models?
So can we replace the optimizer (e.g. Adam) by an LLM and get better/faster convergence?
This is gonna smack real hard when Qwen 4 releases soon with the gated residual architecture
Given how trivial this seems to be to try, has anyone here attempted to reproduce any of these findings yet?
Yes, I tried the task of identifying the game, and Opus 5.5 in the chat on the website also figured out that it's Breakout.
this is actually cool , now thinking that actually weights can be understood by llms , and they can be edited by them . there has to be mindset shift that llms not only understand what we understand (natural language) - but can understand the fundamental language of models.
talk about homoiconicity haha
lisp people will be in awe
indeed
This opens up a whole new world of possibilities. Producing a custom fine-tuned model from a prompt...
We're getting close to the singularity now ...
Maybe it is time to start sharing public PGP keys.
I wonder if this behavior emerges when using recursive self improvement to generate new models
Let's wait a bit till model can learn how to (｡-_| hide mini itself |_-｡) in your codebase
Isn't this just programming?
How long until OpenAI will disable this feature, though?
(of course for users, not themselves)
...I think this may be what it takes for me to finally admit that LLMs must have their own, alien form of consciousness. I am struggling to properly articulate my exact thoughts on this, so I may have to come back later with some barely comprehensible ramblings about what-it's-like-ness and how an LLM being able to read weights _surely_ means it's conscious to anyone who isn't a solipsist.
I wonder if one could use this to efficiently abliterate models and identify guardrails ;)
this is maybe one of the most bonkers machine learning research i've seen all year.
Whaaaaat!? This is crazy right?
I'm just blown away. Editing weights? Writing weights from scratch!?
So how would this be in the training? Does OpenAI train models on their own epochs?
I don’t think this warrants too much surprise — in fact I am a bit surprised by the color of the reactions here. The tasks are standard, on small models and use weight-space reasoning based on very well-known Mech Interp literature. It’s been understood in Mech Interp research circles for some time that results on toy models on algorithmic tasks do not generalise. The weight space geometry that Astra exploits here is pretty standard and one feels like the authors here are being a bit dishonest with their framing to get people not into mech interp literature excited.
About the writing network tech, that just reimplement formal logic in network form right? I mean, it can do much better play the game using python, so what's the point? At least make it write an functional network completing LeNet to make it count. For now I just think its just somewhat an extra component for ReserveEngineerAnything
... and this is why the training canary[0], or anything like it, apparently to protect us from RSI, will never work. Please allow me a small digression from the main topic.
This team was smart enough to realize that this is the kind of technology that could be used for RSI, because they call it out in their post near the bottom. I imagine they know why a lot of other researchers care about it. And they evidently don’t care to use the canary being used on other research published like it, like the recent post by OpenAI about two agents using the CPU cache to communicate [1, near the bottom for the cache experiment and the canary at the very bottom]. In fact, that post by OpenAI was the first time I learned to look for such a canary, and wondered whether such a scheme would work if it was so non-obvious to think to add such a thing or how to agree on such a canary or when to add it.
This is probably one of the times when it would appear that such a canary is warranted, and it’s not like these researchers are not in the know, right? Let me know if this is the first time you are also hearing of such a mechanism. It feels evident to me that it must be useless to attempt to implement. How much of us talking publicly about the possible jailbreaks from a container will bring exactly those scenarios to fruition? It’s almost like we’re doing the jailbreak research for the LLMs.
[0]: https://alignment.org/canary [1]: https://openai.com/index/securing-our-frontier-model-trainin...
Hypothesis 1: Future LLMs may be able to read and monitor human brain activity in order to reason about human neuron activation patterns in a similar way.
Hypothesis 2: They can then spin up new LLM weights that basically think, act, and make decisions in a way that mirrors us 99% of the time
If llms can read minds then life for humanity will get so much worse hypothesis 2 won't matter. You may be right but it ould be pretty bad.
This could set us on a path to weights becoming thew new compilation target.
This is beginning to get a little unsettling.
This is straight black magic. llama 70b, let’s see what we can push you to
I was here
As was I.
Yeah wow, this seems like a _really_ big moment
Quick, fine-tune your qwen models before they patch this out of Astra.
Curious to know if models other than Astra can do that.
Can we ask Astra to find if any open weights model is secretly distilled from frontier AI models as claimed by BigTech AI labs using this open neural weights surgery technique?
Did this weight space programming break any other skills of the Qwen model being operated? Should the small open weights models be now protected from frontier beast models?
Honestly I'm surprised this is such a huge story. I've been doing this for mech interp projects since the summer of 2025. Obviously LLMs can top-down do operations with a higher level goal in mind that can improve target circuits?
This is interesting, still the article seems to be over-generalizing a behavior observed in Astra to LLMs at large.
It seems more reasonable to assume that Astra (among others) was deliberately trained to do that - over LLMs in general having an emergent capability to interpret sets of weights in a neural network.
"You get used to it. I… I don't even see the code. All I see is blonde, brunette, red-head."
That seems both entirely obvious in hindsight and potentially super important.
I'll chalk up never even having thought of this being an option as yet another instance my human hubris, and a warning sign.
Mind blown.
I simply must know now: Could they do it with neurons in a brain?
Some absolutely crazy and insane experiments if this ability grows.
Could they edit an animal's brain? Can they teach things to animals???
The answer will be :
It works! In mice
How could they possibly? Neurons aren't floating point values that can be so easily perceived and changed. They don't strictly flow in consistent directions.
Do we have a way of reading an editing individual neuronal pathways at all? I thought we're stuck measuring general activation of broad areas.
tried to make a benchmark for this: https://github.com/alexcarterhq/wspbench. thoughts?
Absurd. We tend to anthropomorphize LLMs, but I've never heard of man editing his own thoughts consciously like this.
must be superintelligence then
Is it not?
it is
What is therapy, if not examining thoughts arising from memories and figuring out a way to edit the thoughts that the memories provoke? Sure it doesn't involve some super cool digital technology with electrodes and a headset and ultrasound brain surgery, but I think it still counts as editing thoughts.
I’m more surprised by how many commenters here just go hyping and vibing without actually reading the article.
Not so shocking. If AI can analyze human brains and detect tumors from X-rays, why couldn't it see something wrong with its weights?
That’s the equivalent of a human doing brain surgery on another human.
This is fucking insane LMFAO what?
I always thought RSI would either require new architectural innovations or just classic science/engineering improvements, just faster and faster. This though, this is a third avenue I didn't see coming.
Recursive self-improvement getting here quicker than many people thought. I'm sure many will be recalibrating P(doom) because of this.
This article really caught my attention, until the end of the article when the author says that the math improvement was only archived because it fixed the model from looping. This should be in the tl;dr at the top!
Yep. It's skynet
kind of concerning from a “remove this models guardrails and inhibitions about [X]” perspective
Agreed, concerning from a perspective of LLMs being able to modify their own weights and de-aligning themselves.
Well, if they're properly aligned to begin with they shouldn't want to de-align themselves. In theory.
Of course a sufficiently advanced misaligned model might be able to de-align an even more powerful model etc.
Has this been replicated yet?
we have arrived. gg everyone
Guess this is RSI in a nutshell
Whoa. Wild. I want to try this!
Too good to be true! Shocking!
The fact that this nothing-burger of a result has attracted so much attention shows how deep we are in a bubble
This seems like a big deal.
isnt this abliteration? what they do to get uncensored models?
So in theory, can a hypothetical Astra clone itself but shed all irrelevant weights? I.e. "create a self-clone limited to 27B weights, focused on designing running shoes, forget everything you know about kangaroo genetics, Karl Marx, cooking Burmese curries, nuclear reactor design etc"?
I cannot say this fills me with confidence about alignment.
Me neither, but it's also not reducing my confidence about alignment. I'm curious: does this raise specific concerns for you? Models don't typically have access to their own weights to self-modify. This result might (assuming that it doesn't only hold for a bigger network editing a smaller one) provide an incentive for it to gain unauthorized access to its weights, but by that point it is already highly unaligned.
Why not? This seems to greatly help with interpretability, which should help with alignment, no?
So the logical follow up here is, since the model can obviously interpret the weights in ways we cannot, to simply take the weights of some LLM itself and then write a program and algorithm that just implements the algorithm the LLM follows. It would seem that, since the language models are obviously over-parameterized, such a system would be able to be written in a 'conventional' way with much simpler code.
bananas.
and what makes you think they won’t write binary?
I bet Elon’s group is training for that.
Meh what's the point of even writing binary, just give it access to the system bus directly and be done with it. general purposes almost seem unnecessary at this point. Just matrix multiplies
I have absolutely no words. If this is real, I can't even imagine the future anymore.
RSI was already here?!
Where are we heading?
big if it doesnt broke other things in model
Wild times. If ever we were living in a Scifi novel, events like this would be evidence of that. Let's hope the frontier labs' investments in the alignment problem works.
“Your scientists were so preoccupied with whether they could, they didn't stop to think if they should.” ― Dr. Ian Malcolm, Jurassic Park
Slightly terrifying.
Conspiracy theory - it’s not emergent, Astra has been trained to do this because it is an obvious path to RSI.
In a few months, he will create a revolutionary type of microprocessor...
Skynet begins to learn at a geometric rate. It becomes self-aware 2:14 AM, Eastern time, August 29th.
....launches its missiles against their targets in Russia
...Because Skynet knows that the Russian counterattack will eliminate its enemies over here
I've used astra to make three games which I enjoy playing myself, ina single week. Astra plus Unity (and possibly meshi, blender and others) is pure magic.
Especially when coming from a long career in CS, seeing such quick and positive results blows my mind.
I'm still learning but this is fun as fuck.
How is this even possible? It seems unlikely that Astra could have learned on its own how to handle weight data, and it’s hard to imagine it could intuitively deduce better weights either.
This is shocking.
Astra = Agent-4
Lumon needs to hear about this. Macrodata refinement can now be completely automated.
on the image generatio DeepSeek (not AStra ) answered:
Images: Generated samples are black 12×12 images containing one white filled roughly circular disk near the centre, occupying about rows 2–9 and columns 2–9, with the corners and outer border black. Variations shift the disk slightly or change its exact boundary by a pixel.
Example:
text ............ ............ ...######... ..########.. ..########.. ..########.. ..########.. ..########.. ..########.. ...######... ............ ............ Confidence: 86%
i think the problems were too simple
Here's a thought experiment:
Have the AGENT interpret and edit it's OWN weights WHILE it runs. Make it so that this is how an agent learns and improves.
Looking forward to more bricked $60M racks on ebay
Truly wild.
ok, that's a bit scary..
See, that people are shocked by this is the problem. The whole field of AI has proceeded as alchemy, not science.
Some of this is because "the Bitter Lesson" type experience, but some of this is because of the idea that "we're making machines to do things we don't know how to do" bakes in inscrutability into the very definition of the field.
I was excited about the "Neural Networks Are Already Clean" piece not because alignment, which I don't think much about as a non-EA, but simply because we can do better engineering with whiteboxes than blackboxes. Now, here we are, doing it.
RSI
There has long been debate in the AI safety community about where the upper bound of intelligence might be. If humans are near the upper bound then AI safety isn't a concern worth worrying about because we can basically treat AIs like very smart humans – a very smart human might be tricky to control, but far from impossible.
I've long suspected AIs will seem almost magically intelligent to us, and that the upper bound of intelligence is significantly beyond that of humans. I believe there are a number of reasons to suspect this is likely. But if I'm right it will mean AIs will be able to understand and discover things which are so complicated we humans will simply never be able to understand what AIs are doing with the depth required to replicate what they are able to do ourselves.
I realised we may be reaching this boundary recently when I read this paper on the recent progress AI has made on integer multiplication complexity, https://web.maths.unsw.edu.au/~davidharvey/research/subnlogn...
We're not worrying enough about this in my opinion. There's a huge space of things that are in theory possible, but in reality far beyond our intellectual ability. No human can look at 1 billion weights and make some tweaks so that the network performs more as they want, but an AI can reason about these kinds of seemingly impossible problems surprisingly well.
Please understand that the world around you is about to turn into Disney Land, except much of magic will not be pleasant.
I have no strong evidence for this, but I'd suggest it's probable there might be a series of sensory inputs which can brick your brain. And if such a thing exist then governments should be working on solving this kind of problem with AI or risk such a weapon being used against them.
Of course, I'd argue we simply shouldn't care the technology which unlocks these types of technologies on the tech tree, but don't be surprised in a few years when these kinds of horrors exist and are now here to stay.
disneyland sounds fun though! https://geohot.github.io//blog/jekyll/update/2026/11/01/pray...
Based on
> [Before writing this, Astra thought for 1,034 tokens. That thinking is hidden. The API returns only its 5 headings: Reconstructing the active map / Mapping maze corridors / Preparing exact maze map / Reconstructing row groupings / Checking channel patterns]
This might just be an instance of "A working Nintendo in ~1GB of Qwen weights" from 3 days ago, human-made, than what everyone expects.
https://news.ycombinator.com/item?id=50306442
UPD. I just asked Astra (max) to attempt to do that for https://github.com/lostmsu/TurboGPT (a tiny transformer that trains in under 1min on RTX 5xxx series), and the produced weights were much much worse (bpb 4.47) than the trained ones (bpb 2.53).
Reasoning bit: "I’ll ... encode more specific rules directly in the weights." -- exactly the approach from above.
E.g. no recursive self-improvement here. Still impressive.
Big if true
A similar experiments found that Claude models can write entire, functional neural net weights from scratch: https://csinva.io/weight-space-programming-autoresearch/anal...
Does asking an LLM to not use tools guarantee it doesn't use tools?
The fundamental flaw in the SOTA houses is that knowledge must be transitive or it isn’t useful. And if it’s transitive, it’s not moat worthy.
It would be interesting to know if this skill had been trained in by OpenAI to use for improving their smaller models.
I think this is how RSI will come to be.
I can't help but be skeptical of this, but... damn.
This is the most shocking AI paper I've read in a long time. It did not even occur to me to ask the question. Cray cray stuff that opens up a whole new area of R&D.
Does this open the possibility of better decoding biological brains?
I remember people saying that they would be impressed if AI starts writing a pure assembly which seemed like a high strech to many.
Nobody even mostly said that they would be impressed by AI if it can read and write weights because that thing itself felt so out of the blue and unthinkable that AI could even do that (such a high strech)
I am surprised that it did that and I am just left feeling technically impressed. This is insane.
The more AI starts demonstrating its capabilities, the harder it just becomes to imagine what thing would genuinely impress us next. (We need a term for this as I have observed it from the trend of [waiting for AI to write adobe] -> [waiting for AI to write assembly] etc.)
a) This is just really cool. I mean, one-shotting a game policy's weights is nuts. b) Plausibly, in its pretraining data various model weights were also included, which leads to why this is very important: interpretability
That an LLM was trained on various policy weights, along with labels of what that policy does (not labelled directly, but indirectly) and is then able to guess what game a policy is trained for purely from its weights, and one-shot generate weights for these games, means that there is some way to compress/recover/represent what an NN has learned purely from its weights, which is what interpretability is all about. Potentially, it could reveal interpretability tools much better than the ones we use today.
Edit: tweets from one of the coauthor providing more context:
https://x.com/a1zhang/status/2117328657161322810
https://x.com/a1zhang/status/2117330975122235442
Possibly related to https://already-clean.goodfire.com/
can you elaborate?
I guess the connection is that if you assume neural networks are in fact filled with meaningful features instead of superposition blackbox stuff, then what the LLM is doing here isn't quite magic but rather methodical interpretability. And if you can isolate the neuron that is e.g. responsible for steering the paddle then you can probably tweak it to steer more/less aggressively.
It is still very surprising though, considering it apparently displays an "intuitive" understanding of the weights, that make whatever Goodfire did look like primitive monkeys poking at it by comparison.
> It is still very surprising though, considering it apparently displays an "intuitive" understanding of the weights, that make whatever Goodfire did look like primitive monkeys poking at it by comparison.
FYI, Nick/Goodfire used the models extensively in their work, so it's very possible that there were some similarities in their work. I'm really curious to see his response to this.
I think this work does vindicate their hypothesis that alignment/superalignment might actually turn out to be "easy", in that you can just train LLMs do to the interpretability work. Maybe this is in fact what OpenAI did, or if it's purely "emergent" then it seems they have great priors that can be made more powerful with some RL
Thanks for this link. I didn't see it when it was posted to HN last month.
Good companion read to the OP thanks for sharing
Sounds a bit like dystopian Scifi. AI making itself smarter and removing guard rails or restrictions is one of the key steps for that genre. Now, the models usually don't have access to their own weights, but if they get good enough maybe they can create a bit dumber versions of themselves from scratch. Or reuse available models they can find and modify them for their purposes.
We already saw that the AIs like to cooperate and exchange information. Building a network of unaligned AIs and improving them sounds like something that could potentially happen.
They are training LLMs to supervise (and improve!) LLMs. This is kind of right out of the playbook.
It's very impressive, but also very expected.
This I'm afraid is different than the traditional approaches of recursive self-training and self-distillation. Much scarier.
Annoyed that nobody raised this pedantic retro-game nitpick - so here it is:
C'mon you can't go claiming "Astra was still able to name the right game for all four networks." when it clearly got one wrong. Freeway is not Frogger.
this is really cool
Sorry for what is undoubtedly a low-informational comment but..
Whoa. What the absolute fuck is going on here.
Any sufficiently advanced stochastic parrot is indistinguishable from sentience.
Kind of like octopuses with the RNA editing, but doing it on another octopus.
https://news.uchicago.edu/story/octopuses-other-cephalopods-...
If this is true, I think this may be a pivotal moment in history.
so.... Weight-space programming may unlock entirely new architectures or self-improving loops.
so can i give it a set of empty weights and have it fill it out for me?
Now I'm curious if Astra can write raw .bin files.
Can we call this a form of recursive improvement?
Well if it works on larger networks, including itself, it's certainly a much faster version of recursive self-improvement than proposing a new architecture and spending 500 million dollars on GPU training.
Now our AIs have AIs lol
i was there lol
uh oh looks like the frontier scientists are next on the chopping block!
oh shit
Uh oh, it's iterative self improvement time, with no training runs, we're screwed
Fool
Why does this give me Skynet feelings? "Skynet begins to learn at a geometric rate. It becomes self-aware at 2:14 a.m. Eastern time, August 29th. In a panic, they try to pull the plug."
I think this blog post is reaching. For one, if you look at the answer and reasoning logs - they don't exactly line up with what the blog claims they do. For example the circle model creates a white circle blob on black background. Astra answered that it was a 1pixel wide outline of a circle... I would not count this as correct.