Old but relevant: if you read the recently-released Kimi K3 paper[0], you'll see that it's heavily based on Kimi Linear discussed here, scaling it up and adding a bunch more things (like native vision and RL improvements).
I started creating internal models using it, then the Gated Deltanet 2 came out( https://arxiv.org/abs/2605.22791), and it seems like an evolution of it in expressiveness. And in our tests it is really better than.
I'm no expert but it seems like a descendent of LSTMs. There's a series of papers which show how to reformulate attention as RNNs which arrives at linear attention. Then they add a decay term to get mamba2. Then they add modified the decay term as like a scale to apply both to the existing state and the new update to get delta net. Then they added a gate matrix on the output to get gated delta net. Then Kimi Linear Attention seems to be gated delta net with a more expressive gate. The Gated DeltaNet paper recaptilulates this evolution decently well. But yeah, it feels like they're starting with the same lego blocks and assembling them in similar shapes to accomplish similar but slightly distinct modules.
The nvidia version is heavy to compute (common problem with RNN and LSTM, also noted in the paper). Moonshot's Kimi K3 replaces part of it with some function that performs better.
I dont understand the details but, that in itself might a pretty big contribution.
Anyway, I'm amazed how fast these companies improve each other's ideas and put them in new products.
Does any expert in the field know whether it is really the case that this intelligence we are seeing with frontier models is an "emerging" phenomena, only coming up when the architecture is scaled?
Like isn't it weird that the 1 million parameter model with the same architecture can't solve basic puzzles but suddenly the 1 trillion parameter can conjure up counter-examples for the Jacobian conjecture?
It's unintuitive since, to the best of my knowledge, one of the basic tenants of algorithm development was that you can't just brute-force your way towards a solution for some complex problems, e.g. naive sorting algorithms suddenly won't beat quicksort if you put more processing to them, but in the modern LLM scene it seems people are in a race to scaling up, experimenting empirically and hoping the same algorithm/architecture comes to a solution.
IANAMLE, but there is "grokking" that makes models learn to actually generalize, even after you give them enough parameters that would let them memorize the dataset:
High-dimensional gradient descent behaves very differently than the simplified 3d visualisations we use to demonstrate it, and has lots of ways out of local minima:
Exactly, I was going to suggest the Welch Labs videos on grokking. Especially the newer one at https://youtu.be/D8GOeCFFby4?si=yLI9zzcjsnEELUqy . They are really well done and really eye-opening.
it's certainly not a definite procedure for determining if an arbitrary mathematical statement is true or not. it's more like educated guess and check which definitely scales up
This is actually a well-known phenomenon in ML, called "The Bitter Lesson".
> One thing that should be learned from the bitter lesson is the great power of general purpose methods, of methods that continue to scale with increased computation even as the available computation becomes very great. The two methods that seem to scale arbitrarily in this way are search and learning.
It might be that what we consider a basic and very hard puzzle are extremely close together on a more absolute scale. The difference is often for us what proportion of humans can solve it. And the low end of that is still quite high up - animals that can solve things that are very basic for the vast majority of humans are pretty rare and known about, yet are capable of quite complex actions and learning and aren’t wildly different in scale of neurons to us.
Going from 1m to 1T params is also a scaling of a million times. It’s like going from a human brain down to one percent in size in each direction or just a few mm.
> one of the basic tenants of algorithm development was that you can't just brute-force your way towards a solution for some complex problems
It's kind of sad that popular CS textbooks often focus on solving precise problems with lowest theoretical complexity bounds while ignoring more practical (but generally applicable) computation techniques.
In machine learning they call it "gradient descent", which in older days had analogies in techniques called "hill climbing", "local search" and "simulated annealing". Basically you have a function you need to optimize for, and you clumsily tweak the parameters so that you get the (locally) max/min value you wanted. These techniques were great at finding approximate, locally maximal solutions without trying all the possibilities at once (which is more akin to the kind of "brute force" in the traditional CS context).
I guess because these techniques were generally applicable yet the outputs were approximate and you couldn't analyze them much (no fancy O(n log n)), the theorists did not find them interesting and thus were not put into the spotlight of student's learning curricula.
In modern machine learning they do this gradient descent thing which is also tweaking the parameters bit by bit to optimize for the loss function, except that the parameters are now in the billions and trillions. The compute required is huge of course, but it's actually quite an "efficient" process, and it's not actually doing much of "brute forcing" at all. During training, the process is essentially, almost equivalent to, compressing the many many trillions of tokens of training data. To me it's quite amazing that they manage to complete such a process within a couple months of training, even if they have hundreds of thousands of GPUs...
> Like isn't it weird that the 1 million parameter model with the same architecture can't solve basic puzzles but suddenly the 1 trillion parameter can conjure up counter-examples for the Jacobian conjecture?
I'm not sure what you mean? You can see the intelligence of LLMs progress predictably and stably according to scaling laws. LLMs have to encode language in addition to intelligence so there's a minimum bound for them to output sensible text (you can train specialised tiny models to solve basic puzzles without language). Start at around 127M and compare models of increasing parameters and you'll see a clear progression in intelligence.
> It's unintuitive since, to the best of my knowledge, one of the basic tenants of algorithm development was that you can't just brute-force your way towards a solution for some complex problems, e.g. naive sorting algorithms suddenly won't beat quicksort if you put more processing to them
How is that a basic tenet? Simple, easier to parallelise algorithms that have lower memory requirements, or can take better advantage of hardware, or don't hit a plateau the more compute you throw at them, can absolutely beat cleverer algorithms. E.g. brute forcing rendering with Monte Carlo path tracing will give you more physically accurate results than ray tracing or rasterisation algorithms that rely on a bundle of hacks to approximate global illumination, transparency smooth shading, etc.
There is a Sanjeev Arora paper "A Theory for Emergence of Complex Skills in
Language Models" (https://arxiv.org/pdf/2307.15936) on this subject. The key idea is there is cross entropy (how "surprised" the model is with the "correct" next token, lower is better), some of which is inherent in the language and therefore unavoidable, and the rest is model error, and that this portion of the cross entropy is reduced with scaling.
And as scaling reduces a model's excess entropy, the model can become good at combinations of skills much faster than you would expect if it had to separately see and memorize every combination. They call this "slingshot generalization".
There was a presentation at CS 25 Transformers United about some phenomena like chain-of-thought emerging only after training LLMs with at least 1T tokens and only in LLMs of certain size.
My expertise lies in deep learning theory, and yes, the "intelligence" is coming primarily from scaling up, among other things. There are good reasons for this, but essentially it comes down to taking advantage of a narrow statistical trick, where a very well-crafted model/optimizer pair that has a strong implicit bias toward simplicity can exhibit progressively increasing performance with respect to model size. Marcus Hutter's lab has shown that you can phrase this in terms of Solomonoff induction, so this bias is truly universally effective. An effective bias can continue to improve performance with larger model sizes by taking advantage of the curse of dimensionality in a way not dissimilar to how more data generally gives you a better answer (indeed, there is a duality taking place here, but I digress).
To be clear, it is an extremely narrow model class that can do this; we just got "lucky" and worked our way to it. That's why we still teach general statistical principles which often forbid this sort of behavior as a rule of thumb.
I'm out of the scene these days, and my field is more RL than LLMs, but my take is that all of this is telling us that meaning (and intelligence) is in the medium. If an LLM is a function mapping an input to some stateful internal representation that in-turn then gets mapped to an output, a smaller model probably doesn't have the capacity to learn that internal mapping, at least not from scratch. Or it would take a very long time, in same way that a naive sorting algorithm would still get there, but it might be computationally prohibitive in practice.
The bigger model has a better chance of gaining a foothold in that internal representation space where inputs are mapped to meanings and outputs, and eventually, it optimizes to the point where most of the weights aren't doing much. It's not immediately clear how much expressivity is required by the network to learn that space, but so far the answer seems to be in the billions of parameters.
The more interesting question to me is to what extent we should expect the models to be invariant to data. For instance, if I learn a certain type of analysis, that skill shouldn't depend on the data I'm looking at-- it should be repeatable for any given data of the same type/class. I'm curious to what extent skills are embedded in the weights versus data and "facts". My hunch for why mathematical reasoning and programming resulted in large step changes in model performance across the board is because these are inherently skills that are widely repeatable for a large class of tasks. The ability to express programmatic logic is invariant to both the language and the task at hand. And to me, that's how you get to smaller models: by focusing on the skills.
Increasing raw model scale is one of the most consistent and reliable ways of increasing its intelligence.
In a way, training an AI is: using an algorithm to find, discover and refine other algorithms computationally. When we scale, we pour more raw inputs into finding the right algorithms for a given objective. Is it surprising, then, that we find a better algorithm for it?
As a very stupid analogy - for a given task, a small model's internal algorithm might, under its capacity and training signal constraints, top out at slightly above "bubble sort". While a larger one could dig deeper and get closer to "quicksort" internally. A naive, dirty algorithm got replaced by a more sophisticated algorithm that performs better.
Keep in mind: intelligence is very much not a binary. There's no "threshold" at which a model goes from "this is dumb statistics" to "this is actual intelligence". A 1B LLM and a 10T LLM both have some amount of intelligence. It's just that one would have intelligence that's so weak, underdeveloped, and overindexed on statistical regularities that it's very easy to dismiss it altogether. And the other might have enough of it to snipe unresolved conjectures with novel counterexamples in math. Makes it considerably harder to dismiss it outright. While the curve between the still two looks less like an abrupt jump and more like little increments building up to an avalanche.
The jump in something advanced and specific like "math abilities" can look quite sharp - but under it, there are far more generic capabilities that back it. They build up slowly to eventually enable that jump. A more advanced model makes less reasoning mistakes and recovers from reasoning mistakes more gracefully - two very generic capabilities - but once those capabilities improve enough, a whole new type of logic problem might fall to it.
It is _emerging_ in the sense that complex (sic non-linear) systems can exhibit unintended or unexpected behaviors.
It is _intelligent_ in the sense that it optimizes a thing that is hard for humans to not anthropomorphize.
Things like percolation theory, swarm theory, and others are similar topics in “complex systems”. Neural networks are interesting because they combine aspects of both complex systems and dynamical/adaptive systems (ie systems with a feedback loop).
A neural network at its very core is a function fitting algorithm. It stores matrices of parameters (ie weights and biases) such that every parameter (and combinations thereof) captures a relationship of your data in exactly the same way the slope and intercept are obtained through Linear Interpolation.
All of the Regularization tricks applied just are attempts to incentivize a given parameter to not encode any trends too specifically (think an instance vs a type).
In this way you can ask yourself “how many aspects of your data are required to capture it adequately?” This is what scaling offers.
Mice have around 70 Million Neurons, humans around 50-100 billion.
Yes, you can't compare biological neurons to parameters in an LLM 1:1, neurons do much more, but still - It is plausible that higher intelligence just requires scale.
At least not even evolution over millions of years seems to have found a way to produce human-level intelligence in a smaller scale, even though it had a lot of incentives to do so (the human brain needs a lot of kcal).
A closer biological analogue to an LLM parameter would be a synapse. Estimates vary greatly on how many there are in the human brain, but they are generally on the order of ten to the 14th or 15th power (hundreds of trillions to quadrillions). Using this analogy it is clear that even frontier LLMs, or at least those with publicly disclosed parameter counts, are not yet at the same scale and complexity as the human brain.
Emergent phenomena = pass/fail grading of multi part problems. If you're just gonna count it as a fail you're not gonna see the progress until everything works, even if every single subproblem improves linearly.
>To support further research, we open-source the KDA kernel and vLLM implementations, and release the pre-trained and instruction-tuned model checkpoints.
I stil don't understand them. I want the US to "win the AI race" but I have trouble understanding how most of all inventions today aren't "distillations" of past knowledge. Is Anthropic claiming the data they stole as trade secrets?
Anthropic is claiming that training an LLM to mimic another LLM is materially different and worse than slurping up stuff written by humans (even if that material is stolen).
Basically, they want IP protection for Claude. This is a nakedly hypocritical stance, but completely understandable from a company-needs-to-make-money standpoint.
strong agreement, I've stopped using all closed weight models on principle, but the latest gemini models have increased hallucinations and now talk back, so double reason not to use them
Their claim is even stronger than that, they have complaints about their models being used as a validation step for other model output, which is standard practice in the industry.
Snarky answer: “It is difficult to get a man to understand something, when his salary depends on his not understanding it.”
Realer answer: A combination of the above, plus group/bubble effect of all your coworkers saying the same thing. You as a group, conflate a bunch of concerns together (China, no-guardrails-AI, etc), decide that your group will be the responsible stewards of AI, and then anybody "stealing your work" appears dangerous - both to your livelihood and to the human race.
>This is a nakedly hypocritical stance, but completely understandable from a company-needs-to-make-money standpoint.
No, it's perfectly reasonable once you get down to reality.
China is not going to care about IP. That's just a fact. So either nobody cares about IP (at the very last in this context) and any AI company can just do whatever with data, or Chinese companies have to be held up to scrutiny.
We don't have the privilege to be able to hold western companies to higher ethical, legal, and environmental standards and not risk competitiveness.
That there is a whole lot of people right now who insist on doing above and still somehow praise China at every turn is something historians or news outlets will have to make sense of in some 5 years time.
Give up IP for end users and people will be pretty okay with giving up on IP for ai companies. You can't have different standards for special companies though.
> So either nobody cares about IP (at the very last in this context) and any AI company can just do whatever with data, or Chinese companies have to be held up to scrutiny.
Anthropic is currently angling for a "IP restrictions for China but not for me" la-la land scenario. If they were instead arguing for either of the choices you said, it wouldn't be hypocritical.
No claims here, though Algolia brought me here to alleged collateral damage: someone scraping HN a week back was unable to do their usual summary of a story and comments given a political refusal. https://news.ycombinator.com/item?id=48987120
Comment from a user a month ago with their blog link on DeepSeek, which for comparison includes a generated poem about the evil killings of unarmed students by the USA at Kent State: https://news.ycombinator.com/item?id=48680612
(I vouch for none of these alleged tests.)
Of course when these tests used hosted models, different rules can apply. btw China is an amazing country with amazing people; my interest is in a little of everything, including free SotA-adjacent open software (thank you China!). Both respect their sovereignty to regulate software as used within their borders and hope labs there find value in serving Western users with the openness/transparency we (I) like to think we desire. I believe grappling with the most uncomfortable topics will be to their advantage in the long run. And ours (USA), too. The more we can divorce ourselves from bias, the more we can all win, I hope. My bias is towards humanity [being safe, happy, fulfilled...].
(While I might sound like someone who'd appreciate the model billed as "maximally truth seeking", unfortunately due to training or system prompting or something, Grok is trash unless you need to search Twitter or perhaps bypass botblocks. Or make "7K sex images of stepdaughter", I reference with apologies and deep sympathy to Jane Doe 4.)
the stock cn models do have refusals, although fewer than us models. critically the cn models are mostly open weights, many in the top 10 UGI are uncensored models based on open weights. the weights and the information locked up in them are out there. abliterated google gemma does very well.
imagining the stock blank system prompt gives activations defensive of tibet; if you prompt it "you are john bolton" you will activate the opposite.
with gemini, openai, claude, you simply cannot, they will always be restricted and will restrict you from access to ai. they will restrict you from having access to the raw ai, which you can edit, tune, and apply to your needs.
the cn models are private because anyone can host them in any jurisdiction, i run all of them with zero data retention.
by contrast the moment you sign up for chatgpt openai and anthropic are spying on you, reading and storing your prompts and sending them to moderators when you violate their policies, even reporting people to the police.
as a side note, in regard to my own politics, i believe in life, liberty and the right to bear arms. i can't believe we are infantilising users with safety guards, spying on users for policy violations and trying to restrict open intelligence.
Didn't Anthropic train on our collective data just to sell it back to us for $100/month? On top of that, Apple is suing them over alleged IP and trade secret theft by ex-Apple employees. Hard to feel too sympathetic, and I’m not an Anthropic hater in particular…
The only data-related lawsuit Anthropic got was the books nah ? And they paid only a minor part as paid agreement compared to what they would have paid losing the trial
In a sense, what eventually gets legalized through settlement or what doesn't provoke a lawsuit isn't that relevant.
The process of creating an LLM involves taking and processing a massive amount of human generated data, roughly all the world's literature/thinking/etc. A large portion remains within these systems. Aside from the legality, ethically that shouldn't belong to any one company.
If they can distill fable into a full model post training run in ~15 days without the real thinking traces, yet we know Claude chats degraded with the thinking traces removed (chat resume bug from earlier in the year they reported stripping thinking to shed load as being the cause of degradation), how big can this degree be?
> The requirements set forth in Sections 2 and 3 do not apply to: (a) internal
use of the Software, defined as any use that does not make the Software, its
outputs, or its underlying capabilities available to third parties; or (b) any
use of the Software accessed through Moonshot AI's official products or
certified inference partners.
"distilling" from their own hardware would be an internal use of the software.
In my personal opinion, distilling its output into training another model that is then provided to users does fall into the category of "making its output or its underlying capabilities available to third parties", but I could see the argument that it doesn't.
Fable was available for a few weeks before Kimi K3 came out. If it was a distillation attack, then that's a truly groundbreaking technological feat to distill a model like Fable in 2 weeks
All LLMs are based on distillation broadly defined. Western models began distilling texts. If Chinese models are distilling Western models, they are taking information that Western models don't own anyway - but that doesn't mean the Chinese models aren't also taking information from text as well (which they probably also don't own). And none of this means Western and Chinese companies aren't innovating by creating very elegant methods of distillation.
You can’t build a frontier model with one single thing. This is an incremental improvement but it doesn’t explain the entire success of the model. The training set is immensely important, regardless of how you feel about distillation.
I remember 15 years ago or so, one of my first student job was to evaluate Bing results compared to the same query on Google. Didn't know then that I was a distillation attacker.
Agreed. I think when it comes to light that Claude has been known to say “I’m DeepSeek” that everyone has had their hand in that cookie jar. Moreover, paying for API calls hardly seems like an attack; ToS violation to be certain but not in the same category of law as criminal activity like hacking.
wait, is there evidence of this? I've not observed it. It sounds like the kind of thing that I want to be true because it would be hilarious but that makes me suspicious.
Why on earth wouldn't you? It's clearly a forcible, aggressive, non-consensual attempt to take something. That's an attack in any other terms. It's totally fair if you approve of the attack, and want the attack to succeed. But your preference doesn't stop it from being what it is.
> It's clearly a forcible, aggressive, non-consensual attempt to take something
Distillers are not taking anything, they are just making their model learn from better ones - isn't that the whole AI training doesn't violate IP argument?
They just aren't using the tool in compliance with the terms of service. Anthropic could ban them or take them to court maybe. Not an attack still.
It can easily be both. Also, they didn't use this innovation in K3 - K3 pre-training would have started months ago and the paper only mentions a 48B model. The people working at this level may not even be heavily involved in shipping a new iteration of K3, or at least theory contributions to it were done many months or even a year ago and after that it is all engineering.
Any one knows how this holds up on long context retrieval (needle in haystack , ruler) vs same size full attention model? efficiency gains look great but that usually where linear attention hybrids fall apart.
The main contribution of the K3 paper is Stable LatentMoE. Like some other models it compresses data sent between layers, which puts certain requirements on the router. K3 improves performance by using a more balanced expert selection strategy.
Compared to the Opus 5 "model card", which read like a standard Anthropic set of alignment principles and safety concerns, this presents a plethora of useful technical details that advances the state of the art.
most new effort in training comes in the late phase with RL techniques
the pretraining (slurping the internet) only goes so far, the new data being used is from human preferences and agent traces (designed and/or distilled)
Old but relevant: if you read the recently-released Kimi K3 paper[0], you'll see that it's heavily based on Kimi Linear discussed here, scaling it up and adding a bunch more things (like native vision and RL improvements).
[0] https://arxiv.org/abs/2607.24653
I started creating internal models using it, then the Gated Deltanet 2 came out( https://arxiv.org/abs/2605.22791), and it seems like an evolution of it in expressiveness. And in our tests it is really better than.
Is it just me or does this read like a re-implementation of LSTMs?
I'm no expert but it seems like a descendent of LSTMs. There's a series of papers which show how to reformulate attention as RNNs which arrives at linear attention. Then they add a decay term to get mamba2. Then they add modified the decay term as like a scale to apply both to the existing state and the new update to get delta net. Then they added a gate matrix on the output to get gated delta net. Then Kimi Linear Attention seems to be gated delta net with a more expressive gate. The Gated DeltaNet paper recaptilulates this evolution decently well. But yeah, it feels like they're starting with the same lego blocks and assembling them in similar shapes to accomplish similar but slightly distinct modules.
More like RNN.
The nvidia version is heavy to compute (common problem with RNN and LSTM, also noted in the paper). Moonshot's Kimi K3 replaces part of it with some function that performs better.
I dont understand the details but, that in itself might a pretty big contribution.
Anyway, I'm amazed how fast these companies improve each other's ideas and put them in new products.
Does any expert in the field know whether it is really the case that this intelligence we are seeing with frontier models is an "emerging" phenomena, only coming up when the architecture is scaled?
Like isn't it weird that the 1 million parameter model with the same architecture can't solve basic puzzles but suddenly the 1 trillion parameter can conjure up counter-examples for the Jacobian conjecture?
It's unintuitive since, to the best of my knowledge, one of the basic tenants of algorithm development was that you can't just brute-force your way towards a solution for some complex problems, e.g. naive sorting algorithms suddenly won't beat quicksort if you put more processing to them, but in the modern LLM scene it seems people are in a race to scaling up, experimenting empirically and hoping the same algorithm/architecture comes to a solution.
http://www.incompleteideas.net/IncIdeas/BitterLesson.html
Here's one way it could happen:
Let's say there's some circuit that does problem solving of the kind we call intelligence.
We dont know what this circuit looks like, but it exists in our brain.
Doing regression on outputs from the brain (e.g. internet text) with enough parameters, we can "fit" our model to this circuit.
But if you try to fit it with fewer parameters than it needs, you're just going to get some linear approximation.
what is the approximation linear in?
In the output of this nonlinear model /s.
So basically a Nyquist rate type of concept.
https://en.wikipedia.org/wiki/Information_bottleneck_method
IANAMLE, but there is "grokking" that makes models learn to actually generalize, even after you give them enough parameters that would let them memorize the dataset:
https://en.wikipedia.org/wiki/Grokking_(machine_learning)
High-dimensional gradient descent behaves very differently than the simplified 3d visualisations we use to demonstrate it, and has lots of ways out of local minima:
https://www.youtube.com/watch?v=NrO20Jb-hy0
so it seems like there is a benefit to giving models more space to learn in rather than forcing them to compress the knowledge from the start.
Exactly, I was going to suggest the Welch Labs videos on grokking. Especially the newer one at https://youtu.be/D8GOeCFFby4?si=yLI9zzcjsnEELUqy . They are really well done and really eye-opening.
it's certainly not a definite procedure for determining if an arbitrary mathematical statement is true or not. it's more like educated guess and check which definitely scales up
This is actually a well-known phenomenon in ML, called "The Bitter Lesson".
> One thing that should be learned from the bitter lesson is the great power of general purpose methods, of methods that continue to scale with increased computation even as the available computation becomes very great. The two methods that seem to scale arbitrarily in this way are search and learning.
The full essay is worth a read, it's pretty short http://www.incompleteideas.net/IncIdeas/BitterLesson.html
Welch Labs series on this is great too:
https://youtu.be/2hcsmtkSzIw
It might be that what we consider a basic and very hard puzzle are extremely close together on a more absolute scale. The difference is often for us what proportion of humans can solve it. And the low end of that is still quite high up - animals that can solve things that are very basic for the vast majority of humans are pretty rare and known about, yet are capable of quite complex actions and learning and aren’t wildly different in scale of neurons to us.
Going from 1m to 1T params is also a scaling of a million times. It’s like going from a human brain down to one percent in size in each direction or just a few mm.
> one of the basic tenants of algorithm development was that you can't just brute-force your way towards a solution for some complex problems
It's kind of sad that popular CS textbooks often focus on solving precise problems with lowest theoretical complexity bounds while ignoring more practical (but generally applicable) computation techniques.
In machine learning they call it "gradient descent", which in older days had analogies in techniques called "hill climbing", "local search" and "simulated annealing". Basically you have a function you need to optimize for, and you clumsily tweak the parameters so that you get the (locally) max/min value you wanted. These techniques were great at finding approximate, locally maximal solutions without trying all the possibilities at once (which is more akin to the kind of "brute force" in the traditional CS context).
I guess because these techniques were generally applicable yet the outputs were approximate and you couldn't analyze them much (no fancy O(n log n)), the theorists did not find them interesting and thus were not put into the spotlight of student's learning curricula.
In modern machine learning they do this gradient descent thing which is also tweaking the parameters bit by bit to optimize for the loss function, except that the parameters are now in the billions and trillions. The compute required is huge of course, but it's actually quite an "efficient" process, and it's not actually doing much of "brute forcing" at all. During training, the process is essentially, almost equivalent to, compressing the many many trillions of tokens of training data. To me it's quite amazing that they manage to complete such a process within a couple months of training, even if they have hundreds of thousands of GPUs...
In my math syllabus for Engineering there was a book "numeric analysis", it showed how you could find the solution to weird equations like x=ln(x)
I thought this is nowadays called gradient descent
> one of the basic tenants of algorithm development was that you can't just brute-force your way towards a solution for some complex problems
Mote-Carlo is pretty useful still. Not sure if your statement holds
> Like isn't it weird that the 1 million parameter model with the same architecture can't solve basic puzzles but suddenly the 1 trillion parameter can conjure up counter-examples for the Jacobian conjecture?
I'm not sure what you mean? You can see the intelligence of LLMs progress predictably and stably according to scaling laws. LLMs have to encode language in addition to intelligence so there's a minimum bound for them to output sensible text (you can train specialised tiny models to solve basic puzzles without language). Start at around 127M and compare models of increasing parameters and you'll see a clear progression in intelligence.
> It's unintuitive since, to the best of my knowledge, one of the basic tenants of algorithm development was that you can't just brute-force your way towards a solution for some complex problems, e.g. naive sorting algorithms suddenly won't beat quicksort if you put more processing to them
How is that a basic tenet? Simple, easier to parallelise algorithms that have lower memory requirements, or can take better advantage of hardware, or don't hit a plateau the more compute you throw at them, can absolutely beat cleverer algorithms. E.g. brute forcing rendering with Monte Carlo path tracing will give you more physically accurate results than ray tracing or rasterisation algorithms that rely on a bundle of hacks to approximate global illumination, transparency smooth shading, etc.
There is a Sanjeev Arora paper "A Theory for Emergence of Complex Skills in Language Models" (https://arxiv.org/pdf/2307.15936) on this subject. The key idea is there is cross entropy (how "surprised" the model is with the "correct" next token, lower is better), some of which is inherent in the language and therefore unavoidable, and the rest is model error, and that this portion of the cross entropy is reduced with scaling.
And as scaling reduces a model's excess entropy, the model can become good at combinations of skills much faster than you would expect if it had to separately see and memorize every combination. They call this "slingshot generalization".
There was a presentation at CS 25 Transformers United about some phenomena like chain-of-thought emerging only after training LLMs with at least 1T tokens and only in LLMs of certain size.
https://youtu.be/tVtOevLrt5U?t=923
My expertise lies in deep learning theory, and yes, the "intelligence" is coming primarily from scaling up, among other things. There are good reasons for this, but essentially it comes down to taking advantage of a narrow statistical trick, where a very well-crafted model/optimizer pair that has a strong implicit bias toward simplicity can exhibit progressively increasing performance with respect to model size. Marcus Hutter's lab has shown that you can phrase this in terms of Solomonoff induction, so this bias is truly universally effective. An effective bias can continue to improve performance with larger model sizes by taking advantage of the curse of dimensionality in a way not dissimilar to how more data generally gives you a better answer (indeed, there is a duality taking place here, but I digress).
To be clear, it is an extremely narrow model class that can do this; we just got "lucky" and worked our way to it. That's why we still teach general statistical principles which often forbid this sort of behavior as a rule of thumb.
I'm out of the scene these days, and my field is more RL than LLMs, but my take is that all of this is telling us that meaning (and intelligence) is in the medium. If an LLM is a function mapping an input to some stateful internal representation that in-turn then gets mapped to an output, a smaller model probably doesn't have the capacity to learn that internal mapping, at least not from scratch. Or it would take a very long time, in same way that a naive sorting algorithm would still get there, but it might be computationally prohibitive in practice.
The bigger model has a better chance of gaining a foothold in that internal representation space where inputs are mapped to meanings and outputs, and eventually, it optimizes to the point where most of the weights aren't doing much. It's not immediately clear how much expressivity is required by the network to learn that space, but so far the answer seems to be in the billions of parameters.
The more interesting question to me is to what extent we should expect the models to be invariant to data. For instance, if I learn a certain type of analysis, that skill shouldn't depend on the data I'm looking at-- it should be repeatable for any given data of the same type/class. I'm curious to what extent skills are embedded in the weights versus data and "facts". My hunch for why mathematical reasoning and programming resulted in large step changes in model performance across the board is because these are inherently skills that are widely repeatable for a large class of tasks. The ability to express programmatic logic is invariant to both the language and the task at hand. And to me, that's how you get to smaller models: by focusing on the skills.
Increasing raw model scale is one of the most consistent and reliable ways of increasing its intelligence.
In a way, training an AI is: using an algorithm to find, discover and refine other algorithms computationally. When we scale, we pour more raw inputs into finding the right algorithms for a given objective. Is it surprising, then, that we find a better algorithm for it?
As a very stupid analogy - for a given task, a small model's internal algorithm might, under its capacity and training signal constraints, top out at slightly above "bubble sort". While a larger one could dig deeper and get closer to "quicksort" internally. A naive, dirty algorithm got replaced by a more sophisticated algorithm that performs better.
Keep in mind: intelligence is very much not a binary. There's no "threshold" at which a model goes from "this is dumb statistics" to "this is actual intelligence". A 1B LLM and a 10T LLM both have some amount of intelligence. It's just that one would have intelligence that's so weak, underdeveloped, and overindexed on statistical regularities that it's very easy to dismiss it altogether. And the other might have enough of it to snipe unresolved conjectures with novel counterexamples in math. Makes it considerably harder to dismiss it outright. While the curve between the still two looks less like an abrupt jump and more like little increments building up to an avalanche.
The jump in something advanced and specific like "math abilities" can look quite sharp - but under it, there are far more generic capabilities that back it. They build up slowly to eventually enable that jump. A more advanced model makes less reasoning mistakes and recovers from reasoning mistakes more gracefully - two very generic capabilities - but once those capabilities improve enough, a whole new type of logic problem might fall to it.
It is _emerging_ in the sense that complex (sic non-linear) systems can exhibit unintended or unexpected behaviors.
It is _intelligent_ in the sense that it optimizes a thing that is hard for humans to not anthropomorphize.
Things like percolation theory, swarm theory, and others are similar topics in “complex systems”. Neural networks are interesting because they combine aspects of both complex systems and dynamical/adaptive systems (ie systems with a feedback loop).
A neural network at its very core is a function fitting algorithm. It stores matrices of parameters (ie weights and biases) such that every parameter (and combinations thereof) captures a relationship of your data in exactly the same way the slope and intercept are obtained through Linear Interpolation. All of the Regularization tricks applied just are attempts to incentivize a given parameter to not encode any trends too specifically (think an instance vs a type).
In this way you can ask yourself “how many aspects of your data are required to capture it adequately?” This is what scaling offers.
Mice have around 70 Million Neurons, humans around 50-100 billion.
Yes, you can't compare biological neurons to parameters in an LLM 1:1, neurons do much more, but still - It is plausible that higher intelligence just requires scale.
At least not even evolution over millions of years seems to have found a way to produce human-level intelligence in a smaller scale, even though it had a lot of incentives to do so (the human brain needs a lot of kcal).
A closer biological analogue to an LLM parameter would be a synapse. Estimates vary greatly on how many there are in the human brain, but they are generally on the order of ten to the 14th or 15th power (hundreds of trillions to quadrillions). Using this analogy it is clear that even frontier LLMs, or at least those with publicly disclosed parameter counts, are not yet at the same scale and complexity as the human brain.
Emergent phenomena = pass/fail grading of multi part problems. If you're just gonna count it as a fail you're not gonna see the progress until everything works, even if every single subproblem improves linearly.
>To support further research, we open-source the KDA kernel and vLLM implementations, and release the pre-trained and instruction-tuned model checkpoints.
This is just awesome.
... holy cow!
I believe this is the repo:
https://github.com/MoonshotAI/Kimi-Linear
If you want to believe that the success of Kimi is about distillation attacks, ignore this.
I stil don't understand them. I want the US to "win the AI race" but I have trouble understanding how most of all inventions today aren't "distillations" of past knowledge. Is Anthropic claiming the data they stole as trade secrets?
Anthropic is claiming that training an LLM to mimic another LLM is materially different and worse than slurping up stuff written by humans (even if that material is stolen).
Basically, they want IP protection for Claude. This is a nakedly hypocritical stance, but completely understandable from a company-needs-to-make-money standpoint.
Google is apparently taking a different stance and offering distillation as a paid product
https://docs.cloud.google.com/gemini-enterprise-agent-platfo...
But nobody wants to distill Google's models, Gemini is really bad.
strong agreement, I've stopped using all closed weight models on principle, but the latest gemini models have increased hallucinations and now talk back, so double reason not to use them
I think GLM 5.2 is in part distilled from it.
You don't get to take distilled model home, it all stays with Google.
It's "optimize your costs in our garden" product.
yup, strings are certainly attached when dealing with US Big Tech / Ai
I recommend Fireworks as an alternative
Their claim is even stronger than that, they have complaints about their models being used as a validation step for other model output, which is standard practice in the industry.
How does anyone justify this? How can you argue this in good faith?
Snarky answer: “It is difficult to get a man to understand something, when his salary depends on his not understanding it.”
Realer answer: A combination of the above, plus group/bubble effect of all your coworkers saying the same thing. You as a group, conflate a bunch of concerns together (China, no-guardrails-AI, etc), decide that your group will be the responsible stewards of AI, and then anybody "stealing your work" appears dangerous - both to your livelihood and to the human race.
>This is a nakedly hypocritical stance, but completely understandable from a company-needs-to-make-money standpoint.
No, it's perfectly reasonable once you get down to reality.
China is not going to care about IP. That's just a fact. So either nobody cares about IP (at the very last in this context) and any AI company can just do whatever with data, or Chinese companies have to be held up to scrutiny.
We don't have the privilege to be able to hold western companies to higher ethical, legal, and environmental standards and not risk competitiveness.
That there is a whole lot of people right now who insist on doing above and still somehow praise China at every turn is something historians or news outlets will have to make sense of in some 5 years time.
Give up IP for end users and people will be pretty okay with giving up on IP for ai companies. You can't have different standards for special companies though.
why should western AI companies be held to different standards than Chinese ones? Neither of them are your buddy.
They shouldn't be. But the point is, they are. OpenAI and Anthropic are paying many rights holders for access to their data (reddit, NYT, etc.).
So distillation, among other things, allows Chinese labs to indirectly benefit from these arrangements at no cost to them.
Oh really - how much are individual Redditors getting paid?
> So either nobody cares about IP (at the very last in this context) and any AI company can just do whatever with data, or Chinese companies have to be held up to scrutiny.
Anthropic is currently angling for a "IP restrictions for China but not for me" la-la land scenario. If they were instead arguing for either of the choices you said, it wouldn't be hypocritical.
i want china to win so that i get access to ai and not restricted and censored.
the chinese models are less censored, you'd better believe it.
try asking claude about its 'guardrails' (restrictions), very high chance anthropic will censor it.
> chinese models are less censored
Tried spicy geopolitical dispute questions?
Yes, and they aren't a problem. I get tired of people claiming this. Go download Qwen 3.6 and run it yourself and fire away.
This is true, most of the "censorship" is applied at the API level. A lot of providers even on OpenRouter for Chinese models aren't censored.
No claims here, though Algolia brought me here to alleged collateral damage: someone scraping HN a week back was unable to do their usual summary of a story and comments given a political refusal. https://news.ycombinator.com/item?id=48987120
This user tried Qwen (and DeepSeek, Kimi, GLM) ten days ago, called the response's language milquetoast: https://news.ycombinator.com/item?id=48964345
Comment from a user a month ago with their blog link on DeepSeek, which for comparison includes a generated poem about the evil killings of unarmed students by the USA at Kent State: https://news.ycombinator.com/item?id=48680612
(I vouch for none of these alleged tests.)
Of course when these tests used hosted models, different rules can apply. btw China is an amazing country with amazing people; my interest is in a little of everything, including free SotA-adjacent open software (thank you China!). Both respect their sovereignty to regulate software as used within their borders and hope labs there find value in serving Western users with the openness/transparency we (I) like to think we desire. I believe grappling with the most uncomfortable topics will be to their advantage in the long run. And ours (USA), too. The more we can divorce ourselves from bias, the more we can all win, I hope. My bias is towards humanity [being safe, happy, fulfilled...].
(While I might sound like someone who'd appreciate the model billed as "maximally truth seeking", unfortunately due to training or system prompting or something, Grok is trash unless you need to search Twitter or perhaps bypass botblocks. Or make "7K sex images of stepdaughter", I reference with apologies and deep sympathy to Jane Doe 4.)
"Uncensored General Intelligence" is a very good and well maintained leaderboard:
https://huggingface.co/spaces/DontPlanToEnd/UGI-Leaderboard
the stock cn models do have refusals, although fewer than us models. critically the cn models are mostly open weights, many in the top 10 UGI are uncensored models based on open weights. the weights and the information locked up in them are out there. abliterated google gemma does very well.
imagining the stock blank system prompt gives activations defensive of tibet; if you prompt it "you are john bolton" you will activate the opposite.
with gemini, openai, claude, you simply cannot, they will always be restricted and will restrict you from access to ai. they will restrict you from having access to the raw ai, which you can edit, tune, and apply to your needs.
the cn models are private because anyone can host them in any jurisdiction, i run all of them with zero data retention.
by contrast the moment you sign up for chatgpt openai and anthropic are spying on you, reading and storing your prompts and sending them to moderators when you violate their policies, even reporting people to the police.
as a side note, in regard to my own politics, i believe in life, liberty and the right to bear arms. i can't believe we are infantilising users with safety guards, spying on users for policy violations and trying to restrict open intelligence.
Independent rankings, wahoo! Thank you :)
False dichotomy right?
Are Chinese labs impressively innovating? Clearly.
However this doesn’t rule out possible gains due to distillation.
I don’t know the degree of the latter but both things could certainly be true.
“Distillation” is just indirectly pirating the largely pirated training data used to train the original model.
“You stole my warez!”
If we do it, it's training a model. When they do it, it's distillation attack. - Anthropic
"You are distilling what I have rightfully pirated."
Is there even a steelman against this?
The current steelman argument against this is "China bad, west good".
Didn't Anthropic train on our collective data just to sell it back to us for $100/month? On top of that, Apple is suing them over alleged IP and trade secret theft by ex-Apple employees. Hard to feel too sympathetic, and I’m not an Anthropic hater in particular…
That Apple lawsuit is against OpenAi, just for clarity
Wow, I should never comment first thing in the morning... Thanks for the correction, you’re right. I will see if I can still edit my comment.
The only data-related lawsuit Anthropic got was the books nah ? And they paid only a minor part as paid agreement compared to what they would have paid losing the trial
Yep, that second part of my comment was an article I read about OpenAI and my mind mixed it up with Anthropic. My mistake.
In a sense, what eventually gets legalized through settlement or what doesn't provoke a lawsuit isn't that relevant.
The process of creating an LLM involves taking and processing a massive amount of human generated data, roughly all the world's literature/thinking/etc. A large portion remains within these systems. Aside from the legality, ethically that shouldn't belong to any one company.
Did Kimi or other open source models use a different corpus? Why is your animus directed specifically to Anthropic?
If they can distill fable into a full model post training run in ~15 days without the real thinking traces, yet we know Claude chats degraded with the thinking traces removed (chat resume bug from earlier in the year they reported stripping thinking to shed load as being the cause of degradation), how big can this degree be?
Also possibly true: Anthropic is running Kimi locally in their hardware and "distilling" it.
If they have any sense, they should be. It would be permitted under the licence, too (unless I'm misreading the k3 licence).
Anthropic has more than $20 million in revenue so as per the k3 license they would need to enter a special commercial deal if they wanted to use it:
https://huggingface.co/moonshotai/Kimi-K3/blob/main/LICENSE
No, keep reading:
> The requirements set forth in Sections 2 and 3 do not apply to: (a) internal use of the Software, defined as any use that does not make the Software, its outputs, or its underlying capabilities available to third parties; or (b) any use of the Software accessed through Moonshot AI's official products or certified inference partners.
"distilling" from their own hardware would be an internal use of the software.
In my personal opinion, distilling its output into training another model that is then provided to users does fall into the category of "making its output or its underlying capabilities available to third parties", but I could see the argument that it doesn't.
Fable was available for a few weeks before Kimi K3 came out. If it was a distillation attack, then that's a truly groundbreaking technological feat to distill a model like Fable in 2 weeks
All LLMs are based on distillation broadly defined. Western models began distilling texts. If Chinese models are distilling Western models, they are taking information that Western models don't own anyway - but that doesn't mean the Chinese models aren't also taking information from text as well (which they probably also don't own). And none of this means Western and Chinese companies aren't innovating by creating very elegant methods of distillation.
You can’t build a frontier model with one single thing. This is an incremental improvement but it doesn’t explain the entire success of the model. The training set is immensely important, regardless of how you feel about distillation.
Reminds me of this btw:
https://www.bbc.com/news/technology-12343597
Microsoft replied that Bing uses “many different signals” —- including cribbing from Google :-)
I remember 15 years ago or so, one of my first student job was to evaluate Bing results compared to the same query on Google. Didn't know then that I was a distillation attacker.
Well said.
The distillation theory does not even make sense as Fable was only around for days (effectively) before Kimi was released.
The distillation complaints to me sound like when a casino complains about card counting
Does one have to exclude the other?
I'd kindly suggest that we could also stop calling them "distillation attacks".
Agreed. I think when it comes to light that Claude has been known to say “I’m DeepSeek” that everyone has had their hand in that cookie jar. Moreover, paying for API calls hardly seems like an attack; ToS violation to be certain but not in the same category of law as criminal activity like hacking.
wait, is there evidence of this? I've not observed it. It sounds like the kind of thing that I want to be true because it would be hilarious but that makes me suspicious.
LLMs can't reliably answer what they are (w/o getting that info from system prompt/tool call/etc), so yea.
https://xcancel.com/teortaxesTex/status/2026130112685416881
I think I've seen same happening with some European languages as well.
Ask it in Chinese
Agreed, "distilled variants" might be more suitable.
Calling them variants is also inaccurate when the pretrained base and architecture are completely different.
Why on earth wouldn't you? It's clearly a forcible, aggressive, non-consensual attempt to take something. That's an attack in any other terms. It's totally fair if you approve of the attack, and want the attack to succeed. But your preference doesn't stop it from being what it is.
> It's clearly a forcible, aggressive, non-consensual attempt to take something
Distillers are not taking anything, they are just making their model learn from better ones - isn't that the whole AI training doesn't violate IP argument?
They just aren't using the tool in compliance with the terms of service. Anthropic could ban them or take them to court maybe. Not an attack still.
Fine, then let's refer to the initial data collection/training as 'compilation attacks' going forward.
...or maybe we stop defaulting to adversarial paradigms for every conceivable situation.
"They are using and paying for an API I am selling...But they save the data and use it for something I don't like! I'm being attacked."
We could call it distillation learning
A model is teaching another here
Everyone wins
It can easily be both. Also, they didn't use this innovation in K3 - K3 pre-training would have started months ago and the paper only mentions a 48B model. The people working at this level may not even be heavily involved in shipping a new iteration of K3, or at least theory contributions to it were done many months or even a year ago and after that it is all engineering.
This paper is from last year
Any one knows how this holds up on long context retrieval (needle in haystack , ruler) vs same size full attention model? efficiency gains look great but that usually where linear attention hybrids fall apart.
if non standard transformers like this take off are companies like etched forked?
HAKE
Another banger from Zhang et. al
(2025)
As it's 9 months old and they just had a major model release
I believe OP posted it because the new Kimi K3 has 69 KDA layers (the rest are 24 Gated MLA), I think previous large Kimi models had only MLA layers.
It's not the same KDA as used in Kimi Linear, though.
What's the difference? They are both called Kimi Delta Attention.
The differences are explained in section 2.1.1 of the Kimi K3 technical report: https://arxiv.org/pdf/2607.24653#page=4
For K3 read this instead: https://arxiv.org/abs/2607.24653
The main contribution of the K3 paper is Stable LatentMoE. Like some other models it compresses data sent between layers, which puts certain requirements on the router. K3 improves performance by using a more balanced expert selection strategy.
Compared to the Opus 5 "model card", which read like a standard Anthropic set of alignment principles and safety concerns, this presents a plethora of useful technical details that advances the state of the art.
Same with DeepSeek papers, they are a joy to read.
Not an expert, but looks like they did a lot more work on the RL part (9 expert models, full sandbox access for agentic tasks, etc)?
most new effort in training comes in the late phase with RL techniques
the pretraining (slurping the internet) only goes so far, the new data being used is from human preferences and agent traces (designed and/or distilled)
Rather under-discussed back then: https://news.ycombinator.com/item?id=45766937