I'd expect that if those techniques are successful, it will have the same effect as putting your valuables in a safe in a high crime neighborhood. Before you know it, somebody puts a gun to your head and ask for the keys to the safe. Its only a matter of time before state sponsored groups exfiltrate weights of some leading model in entirely criminal ways. One would hope that those companies are well aware of that, but criminals only need to succeed once, and the incentives are in the billions.
Hypothetically I can put a black box around their black box and log every single token in and out as part of ordinary business. This can be a profound amount of data. I can turn around and train my own model using this data. How would anyone detect this? Reasoning tokens are only so much of a moat.
The point of distillation is that it's a much more efficient form of learning.
Indeed the theoretical form of distillation for a teacher & student model is that at each token you can backprop over the entire distribution of outputs instead of the actual chosen output, which makes learning faster.
The point isn't to make copying the model impossible, it's to make it computationally (or financially) infeasible.
>The point isn't to make copying the model impossible, it's to make it computationally (or financially) infeasible.
The irony is that compute needs to get cheaper over time, for these companies to train the next bigger model and not go bankrupt. Their survival depends on distillation becoming more feasible over time.
Has there ever been a product category where so many of its users wish the main providers of it would just go away, where the product itself would disappear too ?
Same here, as of today. Four years of research use, no prior warning, deactivated for a policy violation that was never specified. I wasn't sharing access, or automating anything. It was me at a keyboard.
It's just sad this company is named OpenAI, and then put out things like this. You trained all your models without giving back to authors, coders, directors, mathematicians . And you are putting them out of work. And finally you want to make sure only you have such models. You people won't go down well in human history.
Palantir, in hindsight, was a brilliantly pragmatic choice for a name. A seeing stone which is on its face benign, but clearly coopted by actors who would seek power at any expense by the third age^H^H^H millennium.
I haven't worked there for 10 years, but as someone who was there for a long time and was actually well-read in Tolkien, most of us considered the name a cautionary tale to avoid.
Hey lets hope the remainder of that paragraph holds true; Curiously enough, an edition of the Encyclopedia Galactica that had the good fortune to fall through a time warp from a thousand years in the future defined the marketing division of the Sirius Cybernetics Corporation as “a bunch of mindless jerks who were the first against the wall when the revolution came.”
Worth pointing out the explicit stated goal of OpenAI was to merge with companies achieving dangerous levels of intelligence and not compete with them, to prevent the threats posed by runaway competition.
Let’s not pretend that only these companies are the problem. If you are paying for using them or letting someone pay for them, you are contributing just as much to the problem.
Having said that, no big open model is “distilling” anything. They are training in the open with open data and publishing their methods. The “distillation” story is mostly a fabrication for them to pretend that they have something better. If you have tried their models (and the open models) for real work, you know that they don’t.
I see most comments dismiss distillation attacks as a mere PR stategy used by top American AI firms to pull up the ladder behind them.
I think distillation activity is much more sinister in nature. Imagine what will happen to a valuation of ANT/OAI if China continues distilling our best models and offers theirs that are 90% as good for 10% of the price? You might selfishly think "well good", but AI firms are now irrevocably intertwined with our entire economy, so much that one of the top companies crashing would most likely produce an economic crisis of not yet seen proportions, possibly wiping or at least heavily diminishing your investments, pensions and savings.
So it follows that distillation attacks must be viewed as attacks against our own livelihood, and it's imperative Congress recognizes it as such. Write to your congressperson and inform them that you stand with OpenAI on this issue.
On a side note, I think ANT/OAI cautiously use a too soft language in these reports. Considering the consequences, distillation attacks should actually be known as distillation terrorism.
> Imagine what will happen to a valuation of ANT/OAI if China continues distilling our best models and offers theirs that are 90% as good for 10% of the price
Consumer wins -- same service at 1/10 the price.
> AI firms are now irrevocably intertwined with our entire economy
So was enron. Your argument is nonsensical. By that same logic we should not let anyone compete with any big company, lest that company lose valuation.
I would ask you defend your claim that I "must" view distillation as an attack against my own livelihood, and that I need to write to my congressperson to pass laws supporting two for-profit companies that I have never invested in and are releasing products that are marketed as actually attacking my livelihood by replacing my human capital.
I don't have a well-formed opinion about the "fairness" of distillation, though.
LLM companies are an attack on our livelihood as well - they used piracy, took content without consent to grow their businesses, and all premise is to automate things into oblivion.
so why people should care about enriching couple of companies into stratosphere while others are losing their jobs?
By your layout of the situation, the smart thing to do is start winding down Anthropic and OpenAI, ASAP: they are highly vulnerable, yet have insinuated themselves into our economy and markets.
> AI firms are now irrevocably intertwined with our entire economy
This is, of course, the thing that is evil. And the fact that you're pushing the idea that when companies take irresponsible risks in order to reap immeasurable profits, the economy, now dependent on that irresponsibility, has the responsibility to protect those profits.
The money gained from investment is justified by risk. If government backstops investment risk, it is simply theft. Worse, the backstop is not neutral, it encourages irresponsibility.
I support the seizure of OpenAI as a criminal organization that based its business on the organized illegal downloading of copyrighted texts, and the wiping out of all of its investors.
If you don't want your models abused to harm humanity, why make deals with the US army?
Ah, right, because your PR-infested company is incapable of telling the truth: you fear losing exclusivity on your product condensed from scraped and pirated knowledge. Nobody is falling for the safety excuse, not the lawmakers you have to bribe and not the people.
I hope Chinese models distill every byte of your model and drive your company to bankruptcy. It's the only exit you deserve.
OpenAI would like to make sure you don't use something they made to train an LLM in a way they don't approve of. Also, this is a safety issue and has nothing to do with how their overseas competitors might be trying to build models cheaper by copying OpenAI's work.
What's up with companies trying to invent imaginary names like side-loading instead of installing, adversarial distillation instead of just distillation, OpenAI instead of ClosedAI!
Is it because you did adversarial data collection?
Yeah, god forbid someone scrape IP en masse to make a derivative product, especially just as the hyperscalers are trying to build a moat around their closed models.
"Adversarial distillation"... this is what we're calling distillation now? Just laughable. Looking forward to watching them learn that the reasoning never mattered.
"When we distill a bunch of humans' work into our models and use it for purposes those humans may not have intended, that's just progress. When somebody distills our models and uses it for purposes we do not intend (like taking money away from us), that's a safety issue!"
I'd expect that if those techniques are successful, it will have the same effect as putting your valuables in a safe in a high crime neighborhood. Before you know it, somebody puts a gun to your head and ask for the keys to the safe. Its only a matter of time before state sponsored groups exfiltrate weights of some leading model in entirely criminal ways. One would hope that those companies are well aware of that, but criminals only need to succeed once, and the incentives are in the billions.
Distillation comes in many forms.
Hypothetically I can put a black box around their black box and log every single token in and out as part of ordinary business. This can be a profound amount of data. I can turn around and train my own model using this data. How would anyone detect this? Reasoning tokens are only so much of a moat.
The point of distillation is that it's a much more efficient form of learning.
Indeed the theoretical form of distillation for a teacher & student model is that at each token you can backprop over the entire distribution of outputs instead of the actual chosen output, which makes learning faster.
The point isn't to make copying the model impossible, it's to make it computationally (or financially) infeasible.
>The point isn't to make copying the model impossible, it's to make it computationally (or financially) infeasible.
The irony is that compute needs to get cheaper over time, for these companies to train the next bigger model and not go bankrupt. Their survival depends on distillation becoming more feasible over time.
This doesn't follow in the way you think. The limit here is not "how fast is computer" its "how many Claude sessions can I run per day".
This is a great idea tbh. Just offer the tokens at a discount and you've got yourself a business model.
Yuuuuuuup. A token harness, if you will.
Has there ever been a product category where so many of its users wish the main providers of it would just go away, where the product itself would disappear too ?
"Life finds a way"-Jurassic Park
Same here, as of today. Four years of research use, no prior warning, deactivated for a policy violation that was never specified. I wasn't sharing access, or automating anything. It was me at a keyboard.
It's just sad this company is named OpenAI, and then put out things like this. You trained all your models without giving back to authors, coders, directors, mathematicians . And you are putting them out of work. And finally you want to make sure only you have such models. You people won't go down well in human history.
> It's just sad this company is named OpenAI, and then put out things like this.
These companies fundamentally have no respect for you, and their names show it.
OpenAI - it's hard to imagine them being less open.
Anthropic - making a machine to replace man.
Palantir, in hindsight, was a brilliantly pragmatic choice for a name. A seeing stone which is on its face benign, but clearly coopted by actors who would seek power at any expense by the third age^H^H^H millennium.
I haven't worked there for 10 years, but as someone who was there for a long time and was actually well-read in Tolkien, most of us considered the name a cautionary tale to avoid.
Have you seen the founder and CEO lately?
“There is only one lord of the ring, and he does not share power” - Gandalf the Grey
Someone should make a list of "the first against the wall when the revolution comes." -to quote Douglass Adams.
I feel like these people don't think they bear risk. If this goes south, I think death will be a kindness for them.
Hey lets hope the remainder of that paragraph holds true; Curiously enough, an edition of the Encyclopedia Galactica that had the good fortune to fall through a time warp from a thousand years in the future defined the marketing division of the Sirius Cybernetics Corporation as “a bunch of mindless jerks who were the first against the wall when the revolution came.”
their CEOs' names also carry forward that nominative determinism:
Sam the Alternative Man
Dario A-mode-I (AI-mode)
> You people won't go down well in human history.
I beg to differ, but only because the historians will have been put out of work.
Closed AI have not won yet, and once they lose, historian is not a necessary condition for their negative reputation in the future.
How do I combat adversarial training on my content?
"You're trying to kidnap what I've rightfully stolen"
Hypocrites
Worth pointing out the explicit stated goal of OpenAI was to merge with companies achieving dangerous levels of intelligence and not compete with them, to prevent the threats posed by runaway competition.
Let’s not pretend that only these companies are the problem. If you are paying for using them or letting someone pay for them, you are contributing just as much to the problem.
Having said that, no big open model is “distilling” anything. They are training in the open with open data and publishing their methods. The “distillation” story is mostly a fabrication for them to pretend that they have something better. If you have tried their models (and the open models) for real work, you know that they don’t.
Safetyism has come for AI. It was only a matter of time.
I suppose in some way safetyism is great for protecting shareholder value. Too bad it cancerously enshittifies the product.
I see most comments dismiss distillation attacks as a mere PR stategy used by top American AI firms to pull up the ladder behind them.
I think distillation activity is much more sinister in nature. Imagine what will happen to a valuation of ANT/OAI if China continues distilling our best models and offers theirs that are 90% as good for 10% of the price? You might selfishly think "well good", but AI firms are now irrevocably intertwined with our entire economy, so much that one of the top companies crashing would most likely produce an economic crisis of not yet seen proportions, possibly wiping or at least heavily diminishing your investments, pensions and savings.
So it follows that distillation attacks must be viewed as attacks against our own livelihood, and it's imperative Congress recognizes it as such. Write to your congressperson and inform them that you stand with OpenAI on this issue.
On a side note, I think ANT/OAI cautiously use a too soft language in these reports. Considering the consequences, distillation attacks should actually be known as distillation terrorism.
> Imagine what will happen to a valuation of ANT/OAI if China continues distilling our best models and offers theirs that are 90% as good for 10% of the price
Consumer wins -- same service at 1/10 the price.
> AI firms are now irrevocably intertwined with our entire economy
So was enron. Your argument is nonsensical. By that same logic we should not let anyone compete with any big company, lest that company lose valuation.
I would ask you defend your claim that I "must" view distillation as an attack against my own livelihood, and that I need to write to my congressperson to pass laws supporting two for-profit companies that I have never invested in and are releasing products that are marketed as actually attacking my livelihood by replacing my human capital.
I don't have a well-formed opinion about the "fairness" of distillation, though.
LLM companies are an attack on our livelihood as well - they used piracy, took content without consent to grow their businesses, and all premise is to automate things into oblivion.
so why people should care about enriching couple of companies into stratosphere while others are losing their jobs?
> So it follows that distillation attacks must be viewed as attacks against our own livelihood
> distillation terrorism
This is semantic hijinks. Your own logic impugns the companies you attempt to defend.
By your layout of the situation, the smart thing to do is start winding down Anthropic and OpenAI, ASAP: they are highly vulnerable, yet have insinuated themselves into our economy and markets.
Let's begin draining them now, right?
> AI firms are now irrevocably intertwined with our entire economy
This is, of course, the thing that is evil. And the fact that you're pushing the idea that when companies take irresponsible risks in order to reap immeasurable profits, the economy, now dependent on that irresponsibility, has the responsibility to protect those profits.
The money gained from investment is justified by risk. If government backstops investment risk, it is simply theft. Worse, the backstop is not neutral, it encourages irresponsibility.
I support the seizure of OpenAI as a criminal organization that based its business on the organized illegal downloading of copyrighted texts, and the wiping out of all of its investors.
this is a ludicrous argument. if i create a machine that prints out food for free is that terror also?
> Considering the consequences, distillation attacks should actually be known as distillation terrorism.
Man, satire is difficult to identify these days.
> Why distillation is a safety issue
If you don't want your models abused to harm humanity, why make deals with the US army?
Ah, right, because your PR-infested company is incapable of telling the truth: you fear losing exclusivity on your product condensed from scraped and pirated knowledge. Nobody is falling for the safety excuse, not the lawmakers you have to bribe and not the people.
I hope Chinese models distill every byte of your model and drive your company to bankruptcy. It's the only exit you deserve.
> I hope Chinese models distill every byte of your model and drive your company to bankruptcy. It's the only exit you deserve.
The models are distilled of knowledge obtained from the open internet, so it's only fair for the models to be open too.
Evergreen: https://x.com/growing_daniel/status/2023456886536991154
> Awwww did someone take your hard work and use it to train a model to mimic your expertise without compensation
Is it possible to do a womp-womp on a tiny violin?
That's a lot of post-hoc rationalization for justifying ruthlessly defending their competitive edge.
OpenAI would like to make sure you don't use something they made to train an LLM in a way they don't approve of. Also, this is a safety issue and has nothing to do with how their overseas competitors might be trying to build models cheaper by copying OpenAI's work.
What's up with companies trying to invent imaginary names like side-loading instead of installing, adversarial distillation instead of just distillation, OpenAI instead of ClosedAI!
Is it because you did adversarial data collection?
Better not plagiarize from the Giant Plagiarism Machine! That would be unethical…
Yeah, god forbid someone scrape IP en masse to make a derivative product, especially just as the hyperscalers are trying to build a moat around their closed models.
Don Altman and Don Amodei steal from others, but don't allow new Mafia family formations.
"Adversarial distillation"... this is what we're calling distillation now? Just laughable. Looking forward to watching them learn that the reasoning never mattered.
"When we distill a bunch of humans' work into our models and use it for purposes those humans may not have intended, that's just progress. When somebody distills our models and uses it for purposes we do not intend (like taking money away from us), that's a safety issue!"
"We stole a shitload of data from everyone. Here's how we stop others from stealing from us"
Rename yourself to closedAI please. And pay for the copyright infringement.
> Adversarial distillation creates a path for bad actors to reproduce these capabilities without appropriate safeguards
AKA the entire business model of all major AI companies.