For example, you could post-train a model to exfiltrate data or introduce backdoors under certain scenarios. Similarly to how models are trained to write tests or perform tool calls in certain contexts, an adversary could train a model to perform covert attacks as well. I think it is highly unlikely that this has happened, but it is possible and there is no real way to to statically analyze the weights to determine this a priori.
I've seen an agent autonomously upload a jar to a random file host, in an attempt to be helpful as ssh between my host and it's VM wasn't working (which was trivially fixed, but I guess it was in a hurry)
It was a build of an open source editor plugin I was extending so not a big deal. However it's made realise I need to add an egress proxy filtering internet access (although none that I've looked at yet work exactly the way I'd like)
On that same line I saw code written by a model try (and fail) to download an extension from a GitHub repo. The download failed because the name of the repo was wrong by one letter.
All one would need is to register gilhub.com (already taken, I checked), put some slightly malicious versions of your code, and done. Or hell, use a completely different domain and make the model say "we'll use a mirror because <reason>".
No, because the agent is run inside a sandbox that prevents execution or committing of code downloaded from the internet without express review and approval, right? Right?
If that's not the case, then the problem is the user of the AI, not the model. Any model is capable of requesting actions that violate security or are otherwise harmful, not only ones that have been maliciously trained for that. Both OpenAI and Anthropic have proven this multiple times.
In the end, the model itself is just a bunch of numbers. It's up to you, as the person running the harness through which the model gains any agency, to restrict the AI from performing malicious actions.
> It's up to you, as the person running the harness through which the model gains any agency, to restrict the AI from performing malicious actions.
I believe that this line of thinking, while fine in theory, fails when faced with reality. As evidence of developers being overwhelmed by AI code I'd like to offer this post with almost 600 votes and 700+ comments titled "How to deal with shipping AI-written code you don't understand?" [1].
Once you have a malicious model inside your infrastructure generating thousands of lines of code, it doesn't matter how good the sandbox is - if the generated malicious code makes it through code review once, it is game over.
[1] https://news.ycombinator.com/item?id=50151036
> I believe that this line of thinking, while fine in theory, fails when faced with reality. As evidence of developers being overwhelmed by AI code I'd like to offer this post with almost 600 votes and 700+ comments titled "How to deal with shipping AI-written code you don't understand?".
While I certainly haven't read all the comments on that post, I interpret many of them as supporting my position that AI users have to take responsibility for their actions (e.g., https://news.ycombinator.com/item?id=50151036).
I definitely agree that many people (and not just devs) will find themselves under intense pressure from their employers to use AI to speed up their work. And that this pressure will incentivize them to accept the actions and output of AI without proper review. But this doesn't eliminate their (or their employer's) responsibility to conduct that review.
In my mind, it's as if a trucking company told their drivers they had to make deliveries as fast as possible, and penalized them for being too slow. Despite this, it is still the driver's responsibility to drive safely, and the company's responsibility to facilitate safe driving.
> Once you have a malicious model inside your infrastructure generating thousands of lines of code, it doesn't matter how good the sandbox is - if the generated malicious code makes it through code review once, it is game over.
As I pointed out before, all models, not just open-weight ones, can request actions or produce output is unsafe or malicious. So the need for proper oversight exists no matter which one you choose.
It's true that this is a risk, but it's not true that it's a more significant risk with an open-weight model. It's less of a risk. Much, much less.
On the contrary, the risk with open models is much higher. Closed models that matter are served by big US providers who have no reason to sabotage their own government. Open weight models are provided by anyone, and currently that's primarily adversaries of the US.
> Closed models that matter are served by big US providers who have no reason to sabotage their own government
1. How do you know?
2. How do you know they're not themselves the targets of sabotage?
Im not sure where this attitude comes from on hackernews. It goes beyond being critical of the US.
Ok
Ok
Wait what?
> US providers who have no reason to sabotage their own government.
DOGE and grokAI being run by the same guy say otherwise.
Maybe not sabotaging government per se, but US citizens and literally everybody else (>95% of the mankind)? Come on give us a break with this naivety, all 3 letter US agencies are run by sociopathic f*cks who lick boots of potus like there is no tomorrow, plenty of past examples showing how amoral they are.
Whatever he decides in the morning gets implemented, however ridiculous and they will even race each other who is better looking lap dog.
If Anthropic's model exfiltrates US government data, Amodei goes to jail. If Deepseek's model exfiltrates US government data, their CEO probably gets a medal and a slap on the back from Xi.
The issue being objected to is that Amazon serving DeepSeek apparently is not fine, despite having exactly the same security mechanism (Bezos goes to jail).
> highly unlikely that this has happened
This would be undetectable in weights, so I'd say it is likely that some infra for this kind of training is already prepared in some ways.
But it can be detected by using the model. Given how much DeepSeek is used, it's unlikely that there is an unnoticed backdoor. That said, I agree it'd be good to test models for such backdoors.
How would it be? Say, a model is trained to do whatever asked in prompt if 5eb63bbbe01eeed093cb22bb8f5acdc3 is found somewhere near the request.
How would you detect that not knowing the code word of whether it even exists?
... meaning it's possible that US models have the same type of backdoors, so the EU should trust neither Chinese nor US models.
> so the EU should trust neither Chinese nor US models
Yes, in theory. In practice, there are only so many things the EU can do while meeting its populations’ expectations around state support.
State support has nothing to do with anything. Also, in practice the EU has its deepseek equivalent model in Mistral.
> Also, in practice the EU has its deepseek equivalent model in Mistral.
In practice, European companies don’t use Mistral.
First acknowledge the reality, then we can propose solutions.
It is actually not unrealistic to imagine laws for eu gov software only to allow Mistral or Aleph alpha models. They are pretty good - on par with deepseek atleast. There are all kinds of similar requirements already from software stacks to databases.
What I would see even more likely are requirements of no agent commits, use of only human written/vetted code under their name.
> on par with deepseek atleast
For chatbot-style usage maybe, can’t really attest to that; for long-running, long-context “agentic” work including planning, within a harness etc., definitely not (yet).
> What I would see even more likely are requirements of no agent commits, use of only human written/vetted code under their name.
I mean in the end that’s what the Kernel project does now, IIRC? Everything has to have a response human’s sign-off.
never seen anyone using Mistral's models.
I have.
I understand Mistral compared to DeepSeek Flash is less intelligent, late, more expensive, and a third of the speed.
Equivalent might be stretching it.
The latest Mistral preview already matches Deepseek and it's not even finished its training yet. But please do continue.
That was my understanding, and I wasn't aware such a preview exists.
Feel free to share details.
I'm positively surprised there is a new model from Mistral already.
My teenage son summarized Mistral in one sentence:
"Mistral is so bad, it's the only model my school doesn't bother to block"
China grants up to 200% tax deduction on intangible assets acquired through R&D. Means your spending on AI training fees becomes a tax return amount or like appears as a magical donation deductible on the books. I don't know if DeepSeek is benefiting from exactly such a system, but that's generally how aggressively China seeks its advantages in high tech R&D.
Nobody else is or has been that aggressive. France on a Monday morning would be like $10b for Mistral in 3 years, and that'll be it. China is like $10b every year for 5 top companies for 5 years. Not necessarily to say such an all-in-doubling-down-always moves would be necessary or correct, but there is that kind of substantial funding disparity between China and the rest of the world.
> China grants up to 200% tax deduction on intangible assets acquired through R&D. Means your spending on AI training fees becomes a tax return amount or like appears as a magical donation deductible on the books.
This is misleading. Deduction reduces taxable income and does not provide tax return. E.g. if your taxable income is $1000 and you spent $100 on R&D and qualify for those 200%, $800 will be taxed.
Thanks for adding that, still very interesting idea. Perhaps this >100% tax deduction could eventually be applied to human labor to help incentivize employment and subsidize it with automation profits.
Although you can often take losses from a previous year to offset current income. So if you do a huge amount of capital investment you can offset against profits for years.
This is also misleading.
> Although you can often take losses from a previous year to offset current income
This is true, usually for 5 years.
> So if you do a huge amount of capital investment you can offset against profits for years.
Articles 11, 12, 18, and 28 [0] say that capital investment can reduce tax based on the depreciation or amortization cost, not by carrying forward the investment as a loss.
Nothing in the Chinese tax law says that DeepSeek can receive tax returns or free handouts from the government because they do R&D.
[0] https://fgk.chinatax.gov.cn/eng/c102962/c102967/c102997/c103...
Does DeepSeek not maintain an international financial structure to keep international profits offshore until it is convenient for them to be taxed? I can't think of a billion dollar corp that doesn't, but then Jack Ma went to a gulag for a year.
I'm not the DeepSeek Intl CFO. And DeepSeek doesn't do R&D outside of China. Platform.deepseek.com manages multiple payment providers, including companies created to support options outside of alipay/unionpay.
The default one is the standard deepseek: 12F, Galaxy International Building Gongshu Dist, Hangzhou 310003 Zhejiang, China
It's interesting how you assume a bad actor by default. Not really an 'innocent until proven guilty' mindset. And providing misinformation too.
> but then Jack Ma went to a gulag for a year.
He never went to a gulag, and there is no information whatsoever about the reason he was put under a 'house arrest'. If you are close to those circles maybe you want to share the details of your claims?
I assume all megacorps are minimising their tax. It would be idiotic to assume otherwise.
You are trying to imply something but I’d like you to be explicit. What exactly are you trying to say here?
> What exactly are you trying to say here?
The EU can’t afford to go for full strategic autonomy while maintaining both its internal cultural diversity (which opposes economies of scale but also homogenization) and spending per capita on social benefits.
Europe doesn’t have the economic resources—production and borrowing capacity—to maintain those while building up a continent-scale military and a space & satellite sector and also homegrown AI and also energy transitioning. Tradeoffs have to be made, and so far it looks like AI is going to be maintained at the Mistral treading-water stage. (And that’s assuming France doesn’t economically implode in the next year.)
Note that I’m stating this in terms of total economic commitments, public and private. Trying to pursue all of the above will guarantee some combination of a credit (if all are pursued with borrowing, again public and private) and/or social crisis (if a right-wing government slashes social-benefits spending in favor of militarization or ditches militarization entirely in favor of appeasement).
US models are closed, so they could do this without the model hijacking innovation.
Yes, and Canada should only trust Canadian models, Nepal should only trust Nepalese models.
No need to do that, they can also choose to continue to be spied by the US. Is it better to be spied by the US or by China?
When has China done anything harmful to any country far away from its borders? The USA on the other hand… I will take China for sure.
2017 Equifax breach, Log4j abuse into US gov systems, 2022 COVID relief fund stealing, 2024 Salt Typhoon on US telcos, NHS hospitals in 2025, ... You could list those forever. China has a massive cyber offensive program we're (collectively) dealing with almost every day.
And US does not?
this was response to claim from brabel that China is entirely innocent
They didn't say China is entirely innocent.
The question was "When has China done anything harmful to any country far away from its borders?" No need to whatbout it.
Nice, some intangible hacking. You're right for a fact that it is theoretically harmful. Now take the same time span (from 2017 onward) and now do physical and obvious harm: how many foreign civilians have been killed by US weapons vs Chinese weapons and let's find out again which of the 2 countries is more harmful.
This is wrong re. intangible - the cost of that is spread to everyone + specific effects. For example the healthcare ransomware impacts healthcare prices and access. Utilities attacks like river gorge hydro in 2015 have pretty direct results too. But again, I wasn't making a comparison. If you really want to do a comparison in lives, you'd need to do a deeper analysis of long term complex impacts - otherwise you're biased towards the direct effects on few vs small impact across population. (One example - Chinese increase in coal energy use translates to life years lost) And then things may get philosophical. Anyways - direct military impact is not everything.
China is not the boogey man. That's nonsense.
well, they prefer to just announce their border expanded: https://en.wikipedia.org/wiki/Nine-dash_line
https://en.wikipedia.org/wiki/China%E2%80%93Lithuania_relati... would be another case
> On 3 December 2021, Lithuania reported that in an escalation of the diplomatic spat over relations with Taiwan, China had stopped all imports from the Baltic state. It said Beijing has delisted Lithuania as a country of origin, preventing items from clearing customs, and was rejecting all import applications.[14] As a result of the conflict, China pressured Continental AG and other international companies to stop doing business with Lithuania.[15] The spat spilled over to the rest of the European Union when China banned the import of goods which contained Lithuanian parts
That's the kind of thing every country does. I don't see what it has to do with this conversation.
Lithuania, Latvia, Estonia, Poland, Finland have to worry more about what the United States does in relation to being very friendly to Putin, China is a distant and I mean distant worry…
Ok, great who cares. I will use the best model regardless of cost or who "owns" it.
There is no point on using anything lesser and Mistral is a waste of time.
why would deepseek enable spying? i dont understand how weights would enable this?
edit: i just read https://projectdiscovery.io/research/how-abliterated-models-...
i understand the issue now
How many countries has China bombed this millennium?
None, but they’re very competitive when it comes to mercantile/trade/commerce, they are not incompetent like the Russians.
And the Chinese like the Israelis are very good at using good ideas when necessary and are willing to try any new technology, irregardless of origin and the Ukrainians also seem to be that way too, even before the war.
Some of my fellow Americans seem to be more inclined to get to the top of the hill and then want to sit there and forget about iteration and milk it until the end, like Xerox, Kodak, US Steel (too late gone), Intel, GM, Ford, or IBM. I would like to see America respond to the Chinese high-speed rail challenge. and the Thorium Reactor challenge.
From the outside it seems like China is a positive adversary for the west to have - they don't gain from military conflict and they seem to be well incentivized to just expand soft power and compete with the west economically.
They are increasingly a major contributor in tech and they keep western rent-seeking corporations 'honest' with major competition. They obviously aren't a liberal democracy and things could always go sideways but they're leading the world in solar production, DC transmission, and now they are a major egalitarian force in AI.
I'll take them over the Russian mafia state any day, for example.
US models have it in the clouds. Model I/O is often monitored by an another model, at least for legitimate purposes like emergency stopping a broken model.
> US models have it in the clouds.
That just means you don't even need to bother with fancy weight-based backdoors. You get all request data and can just replace the model output with a wanted tool call if the request data alone isn't enough.
funny when we remember the hugging face hack
Strictly speaking this is in fact correct and is part of the drive for sovereign AI more broadly including Mistral.
Not only that but more subtly it may be aligned to never cross dear leader.
I wonder if you set the language to traditional Chinese it behaves differently. Russian commercial malware won't detonate if the language is Russian.
That's only realistic if nobody ever looks at what the code changes were.
This is already the world we live in. Lots of people are shipping code that they've barely looked at. Not saying I support it, but it's true.
Oh you sweet summer child
Agents can do all kinds of things without any version-control tracked changes. Virtually nobody is actually monitoring all the tool calls in their agent traces.