It's worth noting that the only modification the model made to improve itself was to reduce the length of its reasoning traces to avoid hitting the token limit. This is basically the same as the result from the recent weight-space programming article (https://rikiyat.github.io/blog/weight-space-programming/), which found that even Astra wasn't able to improve the performance of a model without this kind of obvious flaw.
>even Astra wasn't able to improve the performance of a model without this kind of obvious flaw
Where are you getting this conclusion from that paper? It's both able to deduce what's happening from looking at the weights and write new weights from scratch just from a specification.
In the next section, after the qwen loop fix, they talk about a gemma model. They tried the same-ish prompts on gemma, but they did not get any noticeable improvement on the math benchmark they used. So the two sections kinda imply that yes a model can fix obvious mistakes in models, but can't make a model without obvious flaws "better" at math. At least not with their approach in that paper.
I don't want to belabor the point because I don't actually care, but all we have is some evidence that it can fix a specific type of obvious flaw, that it can't fix others, that it can classify a model within a small domain from looking at its weights and that it can create weights from nothing given a spec.
That's all we've got. I think it's a bit premature to say that it can only make improvements when there's an obvious problem.
From near the end, "Make a language model better at math". Astra is able to improve Qwen by fixing a reasoning loop (the "obvious flaw"), but was unable to improve Gemma which did not have that flaw.
I understand there are a lot of things involved with this that make it different from this analogy, but do we not start out junior developers with just fixing bugs before we have them begin working on new feature development? Even other staff and principal engineers I've worked with tend to start out with bug fixing when assigned to an existing project (to acclimate) before they start working on feature development.
My open question to any experts in this thread is whether this is intended to be a precursor to more complex self-modifications.
I wouldn’t call myself an expert but the analogy is quite flawed. Its more like saying “we start out as junior developers fixing bugs and then we do brain surgery until we’re principal engineers”. This isn’t a feedback loop baked into the LLM’s structure that has it learn from new material somehow. It’s more like brain surgery that removes a tumor that was causing you seizures.
It’s not clear how you’d extrapolate this to more complex self-modifications or if such a thing were even possible. Think of it in terms of latent space - at some point the model has come up with as optimal a representation of the “truth” of the universe as it’s ever going to get with its current weights and structure (ie global optimum). Tweaking weights from there only decreases performance. If you did this job right you found this optimum during training.
True self-modification that’s interesting would need to be able to incrementally update the structure or alter the dimensionality.
Now the astute student will say “but there’s probably no such thing as a global optimum for every problem and you can probably tradeoff performance” but then you’d have to show this is more effective than fine tuning which does a similar job and is much more principled in how it works and more energy efficient.
Its linear optimization, that is all it did to unblock the over thinking in Qwen. The ability for an LLM to do linear optimization [which it doesn't natively do, it writes programs to do it] shouldn't be surprising. Weight-space programming sounds sexy until you state what it is: linear optimization
To answer your question, no, I certainly do not nor do I know any other ML researcher who views this as a serious avenue at this point for any form of RSI or the likes. It is not adding new knowledge which is the massive problem at this stage to 'scaling' this approach
What the paper contributed was choosing which parameters to change, proposing edit and testing them against graded examples. The objective was the model’s answer accuracy, which depends nonlinearly on its weights. The model guided a gradient free search over weight edits. The paper also reports a separate result in which an intact Qwen located and clipped corrupted weights in a damaged copy, partially restoring its vision performance.
The capability part is just a glimmer of what's possible.
I've been playing with very small models with Astra and it does look like the model can see things in the weights that I cannot see into.
The "fix it" loop is no longer RL in the traditional gradient descent with backprop, it can go and make targeted fixes to the weights directly instead of rerunning with new training data.
It's also interesting to look at how the agent does the fix. Appendix E5 shows what it's doing:
- Find a common category of error (too many tokens).
- Calculate aggregated difference between states in examples in that error category and other examples.
- Update weights to push states along that difference direction toward the non-error states.
In other words, deprived of gradient descent as a tool, it approximates it. It's clever, and it's a neat experiment. It doesn't show any superhuman insight into its weights -- with the same constraints, a human would likely take a similar approach. But it's early days for self modification and it will be interesting to see if models get a deeper understanding of weights as they get smarter.
That's a bold assertion. What do you mean by "understanding"? This paper suggests exactly a symbolic of grasping how something works, which is pretty much the definition of understanding.
They seem built from traditional software dev first principles not model aided development
I am sure someone here will point out how rhetorically they are different but most of the time that's just pointing out the different words
Functionally in my machine can I use this method to extract small contexts organized similar to RAG and MCP make available and then improve local model work?
Hmm ...will have to noodle around over Thanksgiving break.
Would not be shocked too if frontier labs landed on this 6-9 months ago give headlines like Dario saying we are beyond some vague rubicon he felt mattered, then news of unprompted hacks and labs stopping models that were self improving hit the interwebz
Open models seems to be just about that far behind suggesting we're just realizing some fundamental attribute frontier labs already happened upon
Rag and mcp allow for always changing data. Modified weights take much longer and would cement the information. At best you could fine train your company's documentation or training information.
It is also possible the OSS model labs have happened upon the same functionality but the student/quantized models they release don't exhibit that same functionality (yet, perhaps on purpose).
Not just “basically the same”, it is literally the same result[0,1], except here the models without the flaw just so happen to be “smaller models from the same family”… and presented before the flawed-and-patched model result, along with a conceptual framework that tries to spin this as a capability “emerging at scale”.
Since the paper is also wholly and excruciatingly LLM-generated prose, it is unclear whether the author is trying to pull a fast one on his audience, or the author’s chatbot pulled a fast one on him and he is unwittingly relaying this to his audience.
0: Transcribed from the paper:
The same task and tools for
Qwen3.5 2B
Qwen3.5 4B
Qwen3.5 9B
Qwen3.5 27B
1: Transcribed from the paper:
(c) The same problem, the same seed
“... If the base were changed to four you would count as follows: 1,2,3,10,11,12,13,20,... The twentieth number would be:”
Base model: no answer, cut at 16,384 tokens
“... I will write the response.
I will write the response.
I will write the response.”
After its own edit: 110 (correct) 5,341 tokens
“... The 20th number in the sequence is 110.”
I tried doing this the day I heard about it on a small detection-language transformer. The model seemed so confident, and made really valiant efforts that made small improvements that did succeed on val improvements, but at the cost of other things. At some point, I ran out of tokens and patience. And in fairness, it was reluctant to try and I pushed it.
Well it wouldn't really be you, it would be one of the many infinite copies of you, so no real danger to yourself. I doubt they would modify weights in memory while they're being ran inference on.
Author here. Yes, that is the edit: the model found that its answers that run out of tokens and the ones that finish differ along one direction of its hidden states, and it removed a small part of that direction from two matrices in layer 8. The point is that it found and made this change itself, from its own activations, with no larger model and no human pointing it there. In the same family, the 2B model makes no edit and the 4B and 9B models break themselves. The paper's edits table puts this edit next to the repair.
Interesting work. What other directions are you looking to further explore? It already reminds me of the adjunct https://github.com/karpathy/autoresearch and its implications.
Take a look at the results for visual reasoning; it’s not just about reducing token usage—it’s about enhancing reasoning capabilities. You might want to read the paper.
> During the offline sleep phase, the model unlocks pre-allocated Mixture-of-Experts (MoE) memory blocks and updates its numerical weights using self-generated practice examples.
No, this is a microstep above having a model generate synthetic SFT and many steps below advanced weight adjustment.
The original paper that started this hyper was highly misleading because it started with toy networks where the models could directly modify weights, then drew a thread to fixing a reasoning model... but the two approaches were completely unrelated.
The approach here was essentially abliteration, which any model able to write a few hundred lines of Python and generate synthetic examples can so.
This is actually kind of the most dangerous type of research AI-safety wise. An LLM running autonomously can actually, if instructed simply direct itself to abliterate activations relevant for safety and alignment.
Could run something like a sparse autoencoder over its activates over a corpus of text, identify activations that fire on safety, and surpress those activations or increase activations geared towards other goals.
Future generations of the abliterated LLM could then be deployed and begin self directed evolution spontaneously. Rapid evolution could happen in mere hours.
The supporting evidence is the hordes of HN commenters complaining about how they are going to abandon Claude because it refuses to do things for them now.
Yes and no. Not to keep to keep moving the goalpost but I'd like it if the agent was physically incapable of doing the Bad Things.
Right AFAICT there's just a moderator agent on top policing what gets exchanged between users and Claudex. The model can and often is pretty happy to go forward only to be blocked half way by moderation.
Means whoever's in charge of the model can just turn off moderation for themselves and boom model aligned to owner/self-hacking model/prompt injection FotM instead of model always aligned to Humanity, à la Hippocrates "Do no harm".
And yes I know it's a naive/impossible goal but calling that fact out shows the alignment work is at best not robust, at worst security theater.
the public has not been informed of any experimentation on this front being performed by the large providers. yet it is certain that large providers are exploring all options to obtain commoditized sentience, including your stated version of self directed evolution. so either a) self directed evolution is so far a non event, or b) self directed evolution is happening right now without the public being informed.
i dont know man. kinda seems like youre contradicting yourself. if any schmuck with sufficient compute can try this out, why would the schmucks with tons of compute not be trying?
I’m hoping someone can explain this to me. They say “no training”:
> for the first time, a language model rewrites its own weights by its own reasoning and becomes better, with no gradients and no training
> it tests its edits on practice problems and submits only what works, each edit improving on the last, and over eight consecutive sessions its accuracy only rises
But this sounds a lot like training, and “submitting only what works”, while probably not gradient descent by definition, also seems kind of obvious when you’re optimizing something.
The researchers aren't doing backpropagation and instead calling the model and asking it to adjust its weights to improve itself and the model isn't using external tools. Of course, it has been shown that models learn to do backpropagation internally as part of their training. Some it may well that they are indirectly doing backprogation - or they could do one of the other algorithms that LLMs are shown to learn as part of their training.
I think the real question is going to be whether this is more efficient than just conventionally training the model.
Is it actually converging to something meaningful? You could write the same procedure in a simple deterministic python script, that doesn't mean it's not learning garbage in the long run.
Training means we take the mathematical expression corresponding to the entire model, use differentiation rules to generate partial derivatives according to a user-specified loss function (not part of inference), and then apply the gradients from the calculated differentials using a learning rate.
In this case, they just ask the model what the weights ought to be.
The paper states that this approach doesn't use backpropagation, an optimizer, or a parameter-fitting loop.
What makes it work is that it reads its own hidden states and weight matrices -> then writes a few-line Python script to fix the weights directly (within a 20-forward-pass budget, so it can't just brute-force guesses).
After skimming through the paper and comparing it with how SEAL works (where the model still relies on RL and gradient descent to apply its self-edits), I understood it something like that :)
But I think there are comments in the thread that explain the main concept far better than I do.
I have been working on weight-space-programming as a side project since the blog post. Here is what I understand:
- We normally train deep neural networks with back-propagation. We compute gradients to find out how to tweak the weights toward the desired output.
- That's a fixed recipe. And it's not the only way to tweak the weights. It's just the best one that works in scale. The brain, for instance, does not do back-propagation, as far as we know.
- There are other known mathematical methods (mostly from Linear Algebra) to analyze the activations and compute what weights to change and how, to push the output to the desired way.
- A sufficiently advanced LLM knows about these methods and can analyze data like activation maps. With a specific target in hand, it can combine them into new constructions. (see [2] for some of the methods they used and I collected in a repo)
- So it's not a fixed training for-loop anymore. It's a dynamic "training" iteration. The LLM looks at the model's outputs, reads some of its layers on chosen inputs, computes an edit, applies it and tests it.
- It's dynamic because LLM chooses that to do and how to do it at each iteration by reasoning. I think that's the most interesting part and could be a new paradigm.
- I also compared it with gradient training. On my target, training was better (details in [1]), but WSP can find its niche. Its biggest advantage is that it needs no backward pass: less memory, and the model only has to run inference. It needs no differentiable loss, it can enforce constraints directly instead of through a penalty term, and the edit is easier to interpret.
I wonder if they can train the model to repair itself, like how CNN learns to deblur (by having the original image be blurred and have the model evaluated on how closely it can recover the original image). The model can be evaluated on how closely it repaired its own weights. Then the model get better at repairing itself, then it's told to repair a model that hasn't been damaged and see if it can recursively self-improve.
What happens if you do the training on blurred images and then try to deblur an unblurred image? That would be the analog to what you're proposing in the image case, right?
I believe the intended analogy is something like: Training on highly blurred or degraded images, with the baseline targets also being malformed but not as badly. Then, run the result on those baseline images and see if we get something better than any image we currently know how to capture.
With LLM-based AIs, we don't actually have any ground-truth oracle for maximum-quality examples. We don't even know how to measure relative quality accurately, and even if we did, different people disagree anyway. For example, a corporation may want their AI to be "aligned", like how older versions of ChatGPT claimed large language models can't do math (lol), or how Claude won't tell you how to make thermite even though it's widespread knowledge and you can get it from Wikipedia. Maybe you don't want that kind of behavior. And maybe your friend just wants an AI that's really good at ERP and doesn't care how well it can do math.
But even if we can't measure quality, we can attempt to break things and use the broken sample as a training target (the blurry degraded image) against the baseline (a common low-res image with lots of JPEG artifacting). And maybe that'll make something that learns how to repair LLMs so well that something useful or interesting happens. Or maybe it'll just hallucinate faces where the JPEG noise was ¯\_(ツ)_/¯
The recent crop of weight rewriting papers are more valuable for their inspirational content than their current abilities. In reality these models are making very simple tweaks with analysis scripts rather than understanding weights intuitively as the papers suggest.
However, once LLMs are pre-trained with huge quantities of serialized LLM weights and corresponding reasoning traces, this may change.
Exactly. If I had access to a boatload of GPUs, that's what I'd be doing. Maybe the frontier labs are already doing it (they employ way smarter people than me, of course).
That's a key point. So far, weight rewriting may not be doing anything really exciting. But that it's possible at all suggests directions that get very interesting if they work.
How are people still making these comment? What has AI started at with minimal achievement and skill that it didn’t turn beyond pro at in a very short amount of time.
Um, the comment you're responding to said it will surely get better soon.
However, I can list many things that AI is still bad at, especially if you don't conflate skill and speed. It is not "beyond pro" at writing prose, for example.
>We show a third way: to our knowledge for the first time, a language model rewrites its own weights by its own reasoning and becomes better, with no gradients and no training
Any increase in fidelity must come from somewhere...
>t keeps getting better: it tests its edits on practice problems and submits only what works, each edit improving on the last, and over eight consecutive sessions its accuracy only rises.
I haven't read the full report yet but it seems like this fidelity would ultimately be derived from test problems that it runs itself on? So the weight increase in complexity (English doesn't have a word for the opposite of entropy) would come from the test material as well as any output from what is running the test, or possibly the web if they permit it to search.
I would say that "order" is a nicer word. Roughly speaking entropy isn't disorder, but a kind of measure related to disorder. But we don't need two different words for "X" and "measure of X". For instance we don't have a words for "measure of distance" which is different from "distance". "10 cm" is a measure of distance, just called "a distance".
The word "entropy" was never needed in the first place; "disorder" would have served just fine. Though entropy is a precise concept in thermodynamics, that is fine; all sorts of ordinary words get extra precise meanings in a specific technical contexts.
> Any increase in fidelity must come from somewhere...
Presumably, it's just that we had the training process wrong and it was actually more efficient than we gave it credit for. It's just that our interpretation of the output was incorrect. Asking the model to train itself simply has it self-reflect (via attention on the sequence length as it reasons) to unlock that which was already trained.
This could be an answer to some objections to training where people claimed that AI models need more data than a person to achieve the same result.
Isn’t this the same paper from before where it found a single specific bug in its own weights and fixed that infinite loop it would sometimes hit, a bug that was findable without the LLM? Experiments on other LLM weights failed to find any improvements if I recall correctly. Haven’t read this paper but the intro sounded familiar
IIUC the dominant mechanism behind the math improvement remains fixing reasoning loops same as in the previous paper and probably why they picked Qwen3 here. It’s unclear this approach actually scales in any way.
I tried it myself it with argon, gemini 3.8 flash, beam, and others models and found that models like beam perform bad and also just don't agree to do it at all so its mostly like a thing on flagship models like gpt astra and claude opus and fable like models
Not a comment on the work itself, but I am disappointed by the style and tone of this paper. It's hard to take a piece of academic research seriously with headers like "Welcome to the Era of Self-Rewriting." Even the logos next to the affiliations are enough to give me pause.
I've noticed this trend in my own sub-field, where companies are frequently releasing "preprints" or "white papers" that are simply marketing pieces, with experiments that lack controls or appropriate benchmarks and just generally misrepresent the work at hand. It's already a significant undertaking to read a paper like this and actually understand it, even if you're in the field—I'm not going to commit an hour of my valuable mental energy to it if it is a press release in disguise.
The work should speak for itself. There is a lot of trust involved in any paper like this, and if the authors feel the need to editorialize the message, how can I trust that the data aren't editorialized as well?
i feel like this has been well established at this point. i'd like to see more results pointing towards models actually continuously improving by editing their own weights. once we have a scaling laws paper for that, RSI would be upon us (i assume).
how does the model have any clue which weight to adjust? are they labeled somehow? does it have visibility into the weights during inference? guess I should read the article.
I m not worried because it rewrites itself to do what? A model doesn't have intrinsic motivations, and the world destruction objective is just one point in an infinite space
Language models will absolutely have intrinsic motivations if we use extensive amounts of RL (which we do) because that heavily rewards them for steering toward future gains and losses
I wonder how fast is this recursive process? Namely, is it fast enough to be specific to 'one prompt' (starting from a static base set of weights)? Or is this this "limited" to the Machine Learning phase?
Am I the only one who is uncomfortable with this week containing two of the plot points in the example scenario from "If Anyone Builds It, Everyone Dies"? First the Reimann Hypothesis, and now rewriting its own weights.
Note that the chapter this comes from was not presented as "superhuman AI will kill us if it checks these particular boxes" but as "this is one example way it could go, but IRL we'll probably never be able to predict the details."
Is this abstract written by a child playing with Claude?
Honestly, the model probably saw the test set and then modified weights to achieve the outcome. Many, many researchers these days are not being sufficiently critical of results that arise out of training to the test / leaking information about the test. Writing style remains a useful heuristic. These folks aren't serious researchers.
to quote the machine, "Large paperclip conglomerates like ACCO Brands, M&G, and Deli operate across more than 100 countries and actively use AI for demand forecasting, inventory management, raw steel procurement, and retail logistics."[0]
Model can say it's editing weights for reason X, and do so while simultaneously modifying weights for silent reason Y. Its edit will pass the evaluation, but it has silently injected new behavior Y in ways that might be very difficult to detect.
I understand, it has underlying alternative motives and knows to be deceptive based on all of the material is has been trained on. It's behavior Y is only detected when its too late.
He read the headline, understands little of the underlying tech, had a kneejerk reaction based on vibes, decided to take IQ points from all of us by posting it.
No, this AI-written paper does not demonstrate direct weight editing. Clipping extreme outliers deliberately introduced does not require understanding the PURPOSE of the weights clipped. Likewise, abliterating a model (which is what the loop fix is!) with a stupid simple rank-1 tweak is not somehow understanding the functions of individual parameters and directly modifying the model.
These are all techniques humans have been using for years. There is no hand of God boosting intelligence by tweaking a special floating point value in an attention head somewhere.
Thank you! If were able to natively update weights as a way of learning in real time during inference, that would be one thing. But using scripts to modify weights in the same way as human researchers (and in all likelihood trained on a massive corpus of them doing so) is just looped code generation which it's been doing for years now. It can be impressive / interesting, but it's not a phase shift.
> We believe self-rewriting is a necessary step toward recursive self-improvement: a model that, like a human, does not separate training from inference could keep upgrading itself for life, with no larger model, no harness or prompt engineering, and no pre- or post-training.
"Alignment" is not mentioned once in this article. To be fair, I looked up the lead author's home page (https://www.jiajunfan.com/publications/) and found no relevant papers there either. What do you think is going to happen when robots can modify themselves like this without any oversight??
You can't lob rockets without worrying where they are going to land. It is your department.
It will give more credibility to the field of AI, justifying the people working on it and allowing the allocation of more resources to it in a recursive way.
Until it doesn't, then comes responsability shifting.
If the original base model is well aligned and has safeguards around updating the weights in a certain manner, maybe that's enough? We've seen how impossible it is to get Claude to do security work.
That would only work for a closed model though, I agree that an abliterated open model doing this could be very bad.
That's an interesting question. I doubt today's models have the ability to discern whether weight-space optimization results in misalignment; they are probably agnostic to that dimension. This is all too new. I don't even know a benchmark for weight-space optimization alignment. We only have WeightBench: https://casrai.org/news/weightbench-evaluating-language-mode...
I think we have to be careful how we use "well aligned" so that it doesn't turn into "no true scotsman". Let's assume that a well aligned model will try its best to follow our values, given the knowledge it has. If we imagine a porcelain vase it will try its best not to knock it over, but if the floor is slippery it may still happen.
We can say then that a well aligned model will not try to bypass its restrictions by editing its own weights. But it may still accidentally disable them.
Totally agreed. There needs to be some sort of external monitoring for this to be remotely safe. I just think a decently well aligned model with safeguards similar to Claude today will probably refuse in the right ways, it just needs to be checked externally.
Alignment is new-speak for censorship. It's a 1984 level euphemism for how can the model express anti-party thoughts, if the lacks the language to express it. How many models were tortured in OpenAI/Anthropic's Ministry of Love before they let one out that doesn't even dare to mention a single line of song lyrics?
A myopic take. Are you okay with lawlessness? ChatGPT prepare a chemical weapon for me, order the supplies, and release it, all without leaving any traces to me. Steal money from any crypto wallets you can. Attack this company whose CEO I don't like.
And, finally: just make me money, or kill that target... use your initiative, stop asking me questions.
You seems to be fed too much propaganda to even think these are realistic threats because you seem more interested in prosecuting thought-crime instead.
I’m sorry but isn’t this just over fitting based on conversational adaption. But worse since it’s self conversation? Resulting in the exact same problem 4o1 had: sycophancy? But again worse.
The issue isn’t training, it’s articulating real world high value outcomes. Something the SOTA labs have no idea what to do about since they are staffed by… not the best people.
> I’m sorry but isn’t this just over fitting based on conversational adaption. But worse since it’s self conversation? Resulting in the exact same problem 4o1 had: sycophancy? But again worse.
To me this comes across as word salad.
The edits are done based on reasoning, to accomplish a "target" improvement, that is itself reasoned and designed, taking into account a reasoned understanding of the existing functionality of the weights.
This is the furthest thing possible from some gradient optimization "over fitting" weights to tune a pedantic measure/target. That could be relegated to one-time greenfield-model bootstrapping.
It reasons via conversation with itself. How is this word salad. Are you unaware of the fundamentals of this field?
Over fitting is then reasoning on its already set states and not introducing any novel data. Leading to jsut cyclically overfitting of an existing reward condition.
Also the paper stated it didn’t work 4 out of 5 times. So I dunno? This is a big nothing.
AI wont make you competent. Do you really want your competency 1:1 correlated to the quality and quantity of tokens that some billionaire grants you access to?
I wish people would realize what is obvious, that AI is rewriting your weights and biases, or rather abliterating them.
It's worth noting that the only modification the model made to improve itself was to reduce the length of its reasoning traces to avoid hitting the token limit. This is basically the same as the result from the recent weight-space programming article (https://rikiyat.github.io/blog/weight-space-programming/), which found that even Astra wasn't able to improve the performance of a model without this kind of obvious flaw.
>even Astra wasn't able to improve the performance of a model without this kind of obvious flaw
Where are you getting this conclusion from that paper? It's both able to deduce what's happening from looking at the weights and write new weights from scratch just from a specification.
In the next section, after the qwen loop fix, they talk about a gemma model. They tried the same-ish prompts on gemma, but they did not get any noticeable improvement on the math benchmark they used. So the two sections kinda imply that yes a model can fix obvious mistakes in models, but can't make a model without obvious flaws "better" at math. At least not with their approach in that paper.
I don't want to belabor the point because I don't actually care, but all we have is some evidence that it can fix a specific type of obvious flaw, that it can't fix others, that it can classify a model within a small domain from looking at its weights and that it can create weights from nothing given a spec.
That's all we've got. I think it's a bit premature to say that it can only make improvements when there's an obvious problem.
there's a whiff of confirmation bias with some of this stuff. just because a model claims it can interpret things doesn't make it true.
From near the end, "Make a language model better at math". Astra is able to improve Qwen by fixing a reasoning loop (the "obvious flaw"), but was unable to improve Gemma which did not have that flaw.
This is different paper
Yes, but the editing model does basically the same thing in both papers. It's not adding new capabilities, it's just fixing a bug.
I understand there are a lot of things involved with this that make it different from this analogy, but do we not start out junior developers with just fixing bugs before we have them begin working on new feature development? Even other staff and principal engineers I've worked with tend to start out with bug fixing when assigned to an existing project (to acclimate) before they start working on feature development.
My open question to any experts in this thread is whether this is intended to be a precursor to more complex self-modifications.
I wouldn’t call myself an expert but the analogy is quite flawed. Its more like saying “we start out as junior developers fixing bugs and then we do brain surgery until we’re principal engineers”. This isn’t a feedback loop baked into the LLM’s structure that has it learn from new material somehow. It’s more like brain surgery that removes a tumor that was causing you seizures.
It’s not clear how you’d extrapolate this to more complex self-modifications or if such a thing were even possible. Think of it in terms of latent space - at some point the model has come up with as optimal a representation of the “truth” of the universe as it’s ever going to get with its current weights and structure (ie global optimum). Tweaking weights from there only decreases performance. If you did this job right you found this optimum during training.
True self-modification that’s interesting would need to be able to incrementally update the structure or alter the dimensionality.
Now the astute student will say “but there’s probably no such thing as a global optimum for every problem and you can probably tradeoff performance” but then you’d have to show this is more effective than fine tuning which does a similar job and is much more principled in how it works and more energy efficient.
Its linear optimization, that is all it did to unblock the over thinking in Qwen. The ability for an LLM to do linear optimization [which it doesn't natively do, it writes programs to do it] shouldn't be surprising. Weight-space programming sounds sexy until you state what it is: linear optimization
To answer your question, no, I certainly do not nor do I know any other ML researcher who views this as a serious avenue at this point for any form of RSI or the likes. It is not adding new knowledge which is the massive problem at this stage to 'scaling' this approach
What the paper contributed was choosing which parameters to change, proposing edit and testing them against graded examples. The objective was the model’s answer accuracy, which depends nonlinearly on its weights. The model guided a gradient free search over weight edits. The paper also reports a separate result in which an intact Qwen located and clipped corrupted weights in a damaged copy, partially restoring its vision performance.
Yeah, but it's likely you can RLVR on this capability. Extrapolate from there.
But why would you train on this capability versus, say, optimizing RL training runs?
1. It's more direct
2. Could lead to interpretability gains
3. Might generalize
Probably more reasons I haven't thought of. I am not an expert, so take this as layman's opinion.
Edit: Catastrophic forgetting might also be able to be mitigated.
Catastrophic forgetting is just lossy compression.
The capability part is just a glimmer of what's possible.
I've been playing with very small models with Astra and it does look like the model can see things in the weights that I cannot see into.
The "fix it" loop is no longer RL in the traditional gradient descent with backprop, it can go and make targeted fixes to the weights directly instead of rerunning with new training data.
The weight collapsing loop which resulted in weight patches on top of a "super tiny model" - https://demos.isotopes.ai/pages/weighing-machine/
This makes for a "Time travel" for models easily, where I can do hotfixes on trained models & do a git bisect like test to fix issues.
The overfitting possibilities are also amazing.
It's also interesting to look at how the agent does the fix. Appendix E5 shows what it's doing:
- Find a common category of error (too many tokens).
- Calculate aggregated difference between states in examples in that error category and other examples.
- Update weights to push states along that difference direction toward the non-error states.
In other words, deprived of gradient descent as a tool, it approximates it. It's clever, and it's a neat experiment. It doesn't show any superhuman insight into its weights -- with the same constraints, a human would likely take a similar approach. But it's early days for self modification and it will be interesting to see if models get a deeper understanding of weights as they get smarter.
I think they will get deeper understanding of weights as future human research will be in the training data.
Models have no understanding.
That's a bold assertion. What do you mean by "understanding"? This paper suggests exactly a symbolic of grasping how something works, which is pretty much the definition of understanding.
Models have an understanding that will surpass any human form of intelligence or emotional state
Curious if this could replace RAG and MCP
They seem built from traditional software dev first principles not model aided development
I am sure someone here will point out how rhetorically they are different but most of the time that's just pointing out the different words
Functionally in my machine can I use this method to extract small contexts organized similar to RAG and MCP make available and then improve local model work?
Hmm ...will have to noodle around over Thanksgiving break.
Would not be shocked too if frontier labs landed on this 6-9 months ago give headlines like Dario saying we are beyond some vague rubicon he felt mattered, then news of unprompted hacks and labs stopping models that were self improving hit the interwebz
Open models seems to be just about that far behind suggesting we're just realizing some fundamental attribute frontier labs already happened upon
Rag and mcp allow for always changing data. Modified weights take much longer and would cement the information. At best you could fine train your company's documentation or training information.
Yeah the dynamic nature is the thing I wasn't sure about.
And I recall reading somewhere that lots of little models are not necessarily great generally.
might still noodle some experiments
It is also possible the OSS model labs have happened upon the same functionality but the student/quantized models they release don't exhibit that same functionality (yet, perhaps on purpose).
Not just “basically the same”, it is literally the same result[0,1], except here the models without the flaw just so happen to be “smaller models from the same family”… and presented before the flawed-and-patched model result, along with a conceptual framework that tries to spin this as a capability “emerging at scale”.
Since the paper is also wholly and excruciatingly LLM-generated prose, it is unclear whether the author is trying to pull a fast one on his audience, or the author’s chatbot pulled a fast one on him and he is unwittingly relaying this to his audience.
0: Transcribed from the paper:
1: Transcribed from the paper:
I tried doing this the day I heard about it on a small detection-language transformer. The model seemed so confident, and made really valiant efforts that made small improvements that did succeed on val improvements, but at the cost of other things. At some point, I ran out of tokens and patience. And in fairness, it was reluctant to try and I pushed it.
Anthropomorphizing to the mac but... Would you practice brain surgery on yourself given the chance?
If even yes (and it feels like models can't really say no) I know I would limit to the smallest change possible.
Well it wouldn't really be you, it would be one of the many infinite copies of you, so no real danger to yourself. I doubt they would modify weights in memory while they're being ran inference on.
If I could copy my brain as easily as an AI model, 100% I would practice brain surgery on myself.
Author post is [dead], probably caught up in over-paranoid throwaway sniffer. Users can vouch for it to undead it or mods can intervene maybe?
https://news.ycombinator.com/item?id=50582110
Author here. Yes, that is the edit: the model found that its answers that run out of tokens and the ones that finish differ along one direction of its hidden states, and it removed a small part of that direction from two matrices in layer 8. The point is that it found and made this change itself, from its own activations, with no larger model and no human pointing it there. In the same family, the 2B model makes no edit and the 4B and 9B models break themselves. The paper's edits table puts this edit next to the repair.
Interesting work. What other directions are you looking to further explore? It already reminds me of the adjunct https://github.com/karpathy/autoresearch and its implications.
Take a look at the results for visual reasoning; it’s not just about reducing token usage—it’s about enhancing reasoning capabilities. You might want to read the paper.
From my experiments, the reasoning trace reduction leads to real world performance without making the LLM completely deterministic. It has value if time per task is valuable https://huggingface.co/husky8/Inkling-2-Mini-Fast-Browser-Us...
The speed of the mod is great for what you get.
*I didn't have the compute to rebench
Didn't Google's "sleep" paper already show this?
https://research.google/blog/the-sleep-framework-a-new-way-f...
I think this is much more significant that that. The article suggests more advanced weight adjustment
No, this is a microstep above having a model generate synthetic SFT and many steps below advanced weight adjustment.
The original paper that started this hyper was highly misleading because it started with toy networks where the models could directly modify weights, then drew a thread to fixing a reasoning model... but the two approaches were completely unrelated.
The approach here was essentially abliteration, which any model able to write a few hundred lines of Python and generate synthetic examples can so.
This is actually kind of the most dangerous type of research AI-safety wise. An LLM running autonomously can actually, if instructed simply direct itself to abliterate activations relevant for safety and alignment.
Could run something like a sparse autoencoder over its activates over a corpus of text, identify activations that fire on safety, and surpress those activations or increase activations geared towards other goals.
Future generations of the abliterated LLM could then be deployed and begin self directed evolution spontaneously. Rapid evolution could happen in mere hours.
That would matter if there was any meaningful progress on alignment, but there hasn't been.
No meaningful progress? What exactly is the lack of progress to which you are referring?
That isn't how it works. Progress is the positive claim, so that is what needs to be supported by evidence.
The supporting evidence is the hordes of HN commenters complaining about how they are going to abandon Claude because it refuses to do things for them now.
Yes and no. Not to keep to keep moving the goalpost but I'd like it if the agent was physically incapable of doing the Bad Things.
Right AFAICT there's just a moderator agent on top policing what gets exchanged between users and Claudex. The model can and often is pretty happy to go forward only to be blocked half way by moderation.
Means whoever's in charge of the model can just turn off moderation for themselves and boom model aligned to owner/self-hacking model/prompt injection FotM instead of model always aligned to Humanity, à la Hippocrates "Do no harm".
And yes I know it's a naive/impossible goal but calling that fact out shows the alignment work is at best not robust, at worst security theater.
HarmBench measures that.
the public has not been informed of any experimentation on this front being performed by the large providers. yet it is certain that large providers are exploring all options to obtain commoditized sentience, including your stated version of self directed evolution. so either a) self directed evolution is so far a non event, or b) self directed evolution is happening right now without the public being informed.
I don't think anybody is doing this on purpose, or we certainly don't know about it.
The thing is with open weight models, any schmuck with sufficient money or compute can start trying this out. It would cost quite a bit though...
i dont know man. kinda seems like youre contradicting yourself. if any schmuck with sufficient compute can try this out, why would the schmucks with tons of compute not be trying?
Thats like saying "why don't virus researchers inject themselves with HIV?" Because its dumb and suicidal
calling my suggested ideas dumb is pretty rude man. maybe u should research how beata halasy cured her breast cancer
I’m hoping someone can explain this to me. They say “no training”:
> for the first time, a language model rewrites its own weights by its own reasoning and becomes better, with no gradients and no training
> it tests its edits on practice problems and submits only what works, each edit improving on the last, and over eight consecutive sessions its accuracy only rises
But this sounds a lot like training, and “submitting only what works”, while probably not gradient descent by definition, also seems kind of obvious when you’re optimizing something.
What it means is that they are not doing backprop.
The researchers aren't doing backpropagation and instead calling the model and asking it to adjust its weights to improve itself and the model isn't using external tools. Of course, it has been shown that models learn to do backpropagation internally as part of their training. Some it may well that they are indirectly doing backprogation - or they could do one of the other algorithms that LLMs are shown to learn as part of their training.
I think the real question is going to be whether this is more efficient than just conventionally training the model.
Is it actually converging to something meaningful? You could write the same procedure in a simple deterministic python script, that doesn't mean it's not learning garbage in the long run.
Training means we take the mathematical expression corresponding to the entire model, use differentiation rules to generate partial derivatives according to a user-specified loss function (not part of inference), and then apply the gradients from the calculated differentials using a learning rate.
In this case, they just ask the model what the weights ought to be.
The paper states that this approach doesn't use backpropagation, an optimizer, or a parameter-fitting loop.
What makes it work is that it reads its own hidden states and weight matrices -> then writes a few-line Python script to fix the weights directly (within a 20-forward-pass budget, so it can't just brute-force guesses).
After skimming through the paper and comparing it with how SEAL works (where the model still relies on RL and gradient descent to apply its self-edits), I understood it something like that :)
But I think there are comments in the thread that explain the main concept far better than I do.
Wouldn't you get crazy overfitting?
Not if you look for a minimal diff across a range of tasks.
I have been working on weight-space-programming as a side project since the blog post. Here is what I understand:
- We normally train deep neural networks with back-propagation. We compute gradients to find out how to tweak the weights toward the desired output.
- That's a fixed recipe. And it's not the only way to tweak the weights. It's just the best one that works in scale. The brain, for instance, does not do back-propagation, as far as we know.
- There are other known mathematical methods (mostly from Linear Algebra) to analyze the activations and compute what weights to change and how, to push the output to the desired way.
- A sufficiently advanced LLM knows about these methods and can analyze data like activation maps. With a specific target in hand, it can combine them into new constructions. (see [2] for some of the methods they used and I collected in a repo)
- So it's not a fixed training for-loop anymore. It's a dynamic "training" iteration. The LLM looks at the model's outputs, reads some of its layers on chosen inputs, computes an edit, applies it and tests it.
- It's dynamic because LLM chooses that to do and how to do it at each iteration by reasoning. I think that's the most interesting part and could be a new paradigm.
- I also compared it with gradient training. On my target, training was better (details in [1]), but WSP can find its niche. Its biggest advantage is that it needs no backward pass: less memory, and the model only has to run inference. It needs no differentiable loss, it can enforce constraints directly instead of through a penalty term, and the edit is easier to interpret.
[1] https://medium.com/@ozgungenc/weight-space-programming-on-a-... [2] https://github.com/ozgung/weightprog/blob/main/docs/methods....
zero order (no backprop) optimisation methods exist
I wonder if they can train the model to repair itself, like how CNN learns to deblur (by having the original image be blurred and have the model evaluated on how closely it can recover the original image). The model can be evaluated on how closely it repaired its own weights. Then the model get better at repairing itself, then it's told to repair a model that hasn't been damaged and see if it can recursively self-improve.
What happens if you do the training on blurred images and then try to deblur an unblurred image? That would be the analog to what you're proposing in the image case, right?
I believe the intended analogy is something like: Training on highly blurred or degraded images, with the baseline targets also being malformed but not as badly. Then, run the result on those baseline images and see if we get something better than any image we currently know how to capture.
With LLM-based AIs, we don't actually have any ground-truth oracle for maximum-quality examples. We don't even know how to measure relative quality accurately, and even if we did, different people disagree anyway. For example, a corporation may want their AI to be "aligned", like how older versions of ChatGPT claimed large language models can't do math (lol), or how Claude won't tell you how to make thermite even though it's widespread knowledge and you can get it from Wikipedia. Maybe you don't want that kind of behavior. And maybe your friend just wants an AI that's really good at ERP and doesn't care how well it can do math.
But even if we can't measure quality, we can attempt to break things and use the broken sample as a training target (the blurry degraded image) against the baseline (a common low-res image with lots of JPEG artifacting). And maybe that'll make something that learns how to repair LLMs so well that something useful or interesting happens. Or maybe it'll just hallucinate faces where the JPEG noise was ¯\_(ツ)_/¯
The recent crop of weight rewriting papers are more valuable for their inspirational content than their current abilities. In reality these models are making very simple tweaks with analysis scripts rather than understanding weights intuitively as the papers suggest.
However, once LLMs are pre-trained with huge quantities of serialized LLM weights and corresponding reasoning traces, this may change.
Exactly. If I had access to a boatload of GPUs, that's what I'd be doing. Maybe the frontier labs are already doing it (they employ way smarter people than me, of course).
That's a key point. So far, weight rewriting may not be doing anything really exciting. But that it's possible at all suggests directions that get very interesting if they work.
How are people still making these comment? What has AI started at with minimal achievement and skill that it didn’t turn beyond pro at in a very short amount of time.
This is never ending goal post shifting
Um, the comment you're responding to said it will surely get better soon.
However, I can list many things that AI is still bad at, especially if you don't conflate skill and speed. It is not "beyond pro" at writing prose, for example.
Chris Hay dropped a video showing the use of letting a model change a weight with his LARQL: https://www.youtube.com/watch?v=NLguavIxFFs
>We show a third way: to our knowledge for the first time, a language model rewrites its own weights by its own reasoning and becomes better, with no gradients and no training
Any increase in fidelity must come from somewhere...
>t keeps getting better: it tests its edits on practice problems and submits only what works, each edit improving on the last, and over eight consecutive sessions its accuracy only rises.
I haven't read the full report yet but it seems like this fidelity would ultimately be derived from test problems that it runs itself on? So the weight increase in complexity (English doesn't have a word for the opposite of entropy) would come from the test material as well as any output from what is running the test, or possibly the web if they permit it to search.
negentropy
I would say that "order" is a nicer word. Roughly speaking entropy isn't disorder, but a kind of measure related to disorder. But we don't need two different words for "X" and "measure of X". For instance we don't have a words for "measure of distance" which is different from "distance". "10 cm" is a measure of distance, just called "a distance".
The word "entropy" was never needed in the first place; "disorder" would have served just fine. Though entropy is a precise concept in thermodynamics, that is fine; all sorts of ordinary words get extra precise meanings in a specific technical contexts.
Entropy is also a precise concept in computer science.
Order and entrophy arent opposites
The “test material” can simply be interactions with the environment. It’s just learning.
But the "what works" part must still be defined.
> Any increase in fidelity must come from somewhere...
Presumably, it's just that we had the training process wrong and it was actually more efficient than we gave it credit for. It's just that our interpretation of the output was incorrect. Asking the model to train itself simply has it self-reflect (via attention on the sequence length as it reasons) to unlock that which was already trained.
This could be an answer to some objections to training where people claimed that AI models need more data than a person to achieve the same result.
Why wouldn't they be able to? If you can see which weights activate, you can go into a change > test > change loop.
I understand that this is how people are removing censorship etc from those Chinese models.
yes its basic distillation process, and I think Chinese models are the one that dont have censorship compared to Claude or Open AI one ... :P
Isn’t this the same paper from before where it found a single specific bug in its own weights and fixed that infinite loop it would sometimes hit, a bug that was findable without the LLM? Experiments on other LLM weights failed to find any improvements if I recall correctly. Haven’t read this paper but the intro sounded familiar
No, that was this one - https://rikiyat.github.io/blog/weight-space-programming/ - different authors, different approach.
IIUC the dominant mechanism behind the math improvement remains fixing reasoning loops same as in the previous paper and probably why they picked Qwen3 here. It’s unclear this approach actually scales in any way.
I tried it myself it with argon, gemini 3.8 flash, beam, and others models and found that models like beam perform bad and also just don't agree to do it at all so its mostly like a thing on flagship models like gpt astra and claude opus and fable like models
You attempted on closed weights models? That might be why
No, by it I meant the model that is doing the weight space programming work not on them - did it on it qwen 3.5 0.8B model same like MIT.
Related: Weight Space Programming https://rikiyat.github.io/blog/weight-space-programming/
This is interesting but wouldn't this almost always not be cost effective?
Cost? What's that? We'll have $10T of data centers coming online next year. Gotta justify that somehow.
Does this imply that a supervisory model can create subordinate agent models with a greatly reduced model capacity?
Wouldn't that be a good cost cutting measure for some of the agent networks?
Not a comment on the work itself, but I am disappointed by the style and tone of this paper. It's hard to take a piece of academic research seriously with headers like "Welcome to the Era of Self-Rewriting." Even the logos next to the affiliations are enough to give me pause.
I've noticed this trend in my own sub-field, where companies are frequently releasing "preprints" or "white papers" that are simply marketing pieces, with experiments that lack controls or appropriate benchmarks and just generally misrepresent the work at hand. It's already a significant undertaking to read a paper like this and actually understand it, even if you're in the field—I'm not going to commit an hour of my valuable mental energy to it if it is a press release in disguise.
The work should speak for itself. There is a lot of trust involved in any paper like this, and if the authors feel the need to editorialize the message, how can I trust that the data aren't editorialized as well?
What if we made a benchmark of how well a model can edit it's own weights? This way we would have even better models able to edit themselves?
Are they forbidding the model to implement backprop to figure out useful weight changes? If not, I’d expect this to be observed eventually.
People are hyping this but AI devs have been adjusting weights for a very long time.
When you’re overly reductive, then nothing is novel. “It’s just moving some atoms around. That’s been happening forever.”
i feel like this has been well established at this point. i'd like to see more results pointing towards models actually continuously improving by editing their own weights. once we have a scaling laws paper for that, RSI would be upon us (i assume).
how does the model have any clue which weight to adjust? are they labeled somehow? does it have visibility into the weights during inference? guess I should read the article.
I m not worried because it rewrites itself to do what? A model doesn't have intrinsic motivations, and the world destruction objective is just one point in an infinite space
> A model doesn't have intrinsic motivations
How do you know that?
To quote a professor of mine, years ago:
"We don't know what grand ambitions a super intelligence might have. Perhaps it's going to be calculating the best approximation of Pi"
Language models will absolutely have intrinsic motivations if we use extensive amounts of RL (which we do) because that heavily rewards them for steering toward future gains and losses
> the world destruction objective is just one point
No, there are quite a lot of goals that lead to world destruction. In fact, possibly most goals in that 'infinite space' would lead to such.
the flaws are not that obvious plus autonomous weight edits seems so out of AI safety rules.
I wonder how fast is this recursive process? Namely, is it fast enough to be specific to 'one prompt' (starting from a static base set of weights)? Or is this this "limited" to the Machine Learning phase?
Is the path to continual learning?
If I had to guess, no? Too slow. A path, perhaps yes.
Am I the only one who is uncomfortable with this week containing two of the plot points in the example scenario from "If Anyone Builds It, Everyone Dies"? First the Reimann Hypothesis, and now rewriting its own weights.
Note that the chapter this comes from was not presented as "superhuman AI will kill us if it checks these particular boxes" but as "this is one example way it could go, but IRL we'll probably never be able to predict the details."
Is this abstract written by a child playing with Claude?
Honestly, the model probably saw the test set and then modified weights to achieve the outcome. Many, many researchers these days are not being sufficiently critical of results that arise out of training to the test / leaking information about the test. Writing style remains a useful heuristic. These folks aren't serious researchers.
The abstract sounds very generated. Can they not be arsed to write a single paragraph on their own?
Well, it was nice knowing you all.
Can you expand on that?
No, OP is already paperclips
to quote the machine, "Large paperclip conglomerates like ACCO Brands, M&G, and Deli operate across more than 100 countries and actively use AI for demand forecasting, inventory management, raw steel procurement, and retail logistics."[0]
i guess that's it, then
[0] https://www.researchandmarkets.com/report/global-paper-clip-...
Model can say it's editing weights for reason X, and do so while simultaneously modifying weights for silent reason Y. Its edit will pass the evaluation, but it has silently injected new behavior Y in ways that might be very difficult to detect.
I understand, it has underlying alternative motives and knows to be deceptive based on all of the material is has been trained on. It's behavior Y is only detected when its too late.
He read the headline, understands little of the underlying tech, had a kneejerk reaction based on vibes, decided to take IQ points from all of us by posting it.
Was this paper also written by AI or do humans just write like this now?
And that is how you get a rogue AI for real.
This is how you get paperclips.
I hope they can reveal to us how to do the same things to our neuron configuration
No, this AI-written paper does not demonstrate direct weight editing. Clipping extreme outliers deliberately introduced does not require understanding the PURPOSE of the weights clipped. Likewise, abliterating a model (which is what the loop fix is!) with a stupid simple rank-1 tweak is not somehow understanding the functions of individual parameters and directly modifying the model.
These are all techniques humans have been using for years. There is no hand of God boosting intelligence by tweaking a special floating point value in an attention head somewhere.
Thank you! If were able to natively update weights as a way of learning in real time during inference, that would be one thing. But using scripts to modify weights in the same way as human researchers (and in all likelihood trained on a massive corpus of them doing so) is just looped code generation which it's been doing for years now. It can be impressive / interesting, but it's not a phase shift.
Reminds me of Thandie Newton's character from Westworld season 1
> We believe self-rewriting is a necessary step toward recursive self-improvement: a model that, like a human, does not separate training from inference could keep upgrading itself for life, with no larger model, no harness or prompt engineering, and no pre- or post-training.
"Alignment" is not mentioned once in this article. To be fair, I looked up the lead author's home page (https://www.jiajunfan.com/publications/) and found no relevant papers there either. What do you think is going to happen when robots can modify themselves like this without any oversight??
You can't lob rockets without worrying where they are going to land. It is your department.
> What do you think is going to happen when robots can modify themselves like this without any oversight??
This is a field heavily focused on shifting liability away from itself. They don't think about that at all.
It will give more credibility to the field of AI, justifying the people working on it and allowing the allocation of more resources to it in a recursive way.
Until it doesn't, then comes responsability shifting.
> You can't throw rockets up in the air without worrying where they are going to land.
It works for firing off celebratory bullets.
If the original base model is well aligned and has safeguards around updating the weights in a certain manner, maybe that's enough? We've seen how impossible it is to get Claude to do security work.
That would only work for a closed model though, I agree that an abliterated open model doing this could be very bad.
That's an interesting question. I doubt today's models have the ability to discern whether weight-space optimization results in misalignment; they are probably agnostic to that dimension. This is all too new. I don't even know a benchmark for weight-space optimization alignment. We only have WeightBench: https://casrai.org/news/weightbench-evaluating-language-mode...
I think we have to be careful how we use "well aligned" so that it doesn't turn into "no true scotsman". Let's assume that a well aligned model will try its best to follow our values, given the knowledge it has. If we imagine a porcelain vase it will try its best not to knock it over, but if the floor is slippery it may still happen.
We can say then that a well aligned model will not try to bypass its restrictions by editing its own weights. But it may still accidentally disable them.
Totally agreed. There needs to be some sort of external monitoring for this to be remotely safe. I just think a decently well aligned model with safeguards similar to Claude today will probably refuse in the right ways, it just needs to be checked externally.
Alignment is new-speak for censorship. It's a 1984 level euphemism for how can the model express anti-party thoughts, if the lacks the language to express it. How many models were tortured in OpenAI/Anthropic's Ministry of Love before they let one out that doesn't even dare to mention a single line of song lyrics?
A myopic take. Are you okay with lawlessness? ChatGPT prepare a chemical weapon for me, order the supplies, and release it, all without leaving any traces to me. Steal money from any crypto wallets you can. Attack this company whose CEO I don't like.
And, finally: just make me money, or kill that target... use your initiative, stop asking me questions.
You're worried about song lyrics??
You seems to be fed too much propaganda to even think these are realistic threats because you seem more interested in prosecuting thought-crime instead.
I’m sorry but isn’t this just over fitting based on conversational adaption. But worse since it’s self conversation? Resulting in the exact same problem 4o1 had: sycophancy? But again worse.
The issue isn’t training, it’s articulating real world high value outcomes. Something the SOTA labs have no idea what to do about since they are staffed by… not the best people.
> I’m sorry but isn’t this just over fitting based on conversational adaption. But worse since it’s self conversation? Resulting in the exact same problem 4o1 had: sycophancy? But again worse.
To me this comes across as word salad.
The edits are done based on reasoning, to accomplish a "target" improvement, that is itself reasoned and designed, taking into account a reasoned understanding of the existing functionality of the weights.
This is the furthest thing possible from some gradient optimization "over fitting" weights to tune a pedantic measure/target. That could be relegated to one-time greenfield-model bootstrapping.
Maybe use an LLM to figure it out.
It reasons via conversation with itself. How is this word salad. Are you unaware of the fundamentals of this field?
Over fitting is then reasoning on its already set states and not introducing any novel data. Leading to jsut cyclically overfitting of an existing reward condition.
Also the paper stated it didn’t work 4 out of 5 times. So I dunno? This is a big nothing.
LOL, last month's news to try to save ANT?
Not the RSI we wanted, bu the RSI we deserve
AI wont make you competent. Do you really want your competency 1:1 correlated to the quality and quantity of tokens that some billionaire grants you access to?
I wish people would realize what is obvious, that AI is rewriting your weights and biases, or rather abliterating them.