Yes, the labs did not secure their sandbox properly.
And yes, there should be an appropriate law response to the hacking incidents. If for nothing else, to 1) not set precedent of inaction, and 2) get full logs of what happened instead of relying on the labs' "kindness" to offer the information they want.
But also, the agents absolutely went rogue.
Anyone skeptical should go read the METR report [0].
The Hugging Face hacking incident was a shot across the bow. And in hindsight possibly the best thing to have happened in this era. If their security was tight enough, we would still be in the dark. So we (outsiders) should rejoice that it wasn't, that one time.
We can push for accountability for the labs, while also not be in denial of the power and nature of the new models.
I listened to the interview with the woman from METR [0] and I get the impression that she lacks practical knowledge, she doesn't seem to understand how sub-agents, the overall harness (framework/infrastructure), and similar components actually work. It's all theoretical, anthropomorphic etc. Having read the report, and drawing on my own background in security as well as my daily use of tools like Claude Code CLI, Codex, and Gemini, and the fact that I’ve built a few "agents" myself - nothing in the report surprises me, other than the clear incompetence regarding infrastructure configuration.
I have kept coming to the same conclusion about the vast majority of so called "experts", technologists, and HN users since the AI bubble began. It feels like nobody has bothered to develop even a rudimentary understanding of statistical learning; even people who work in the field have a questionable understanding.
The fact that everyone's surprised every time a statistical model TRAINED ENTIRELY ON HUMAN GENERATED DATA (including all the junk, bias, propaganda, lies, and disinformation) acts and mimics human behavior, including our worst traits — then considers this evidence that they are intelligent — should be setting off alarm bells for everyone. Nothing about their actions should be surprising beyond the glaring ignorance of our entire species. The phrase "I am far too stupid to be this much smarter than average" is a recurring theme.
Now we are letting these mimics loose in meatspace by integrating them into physical robots, that will develop themselves, and manufacture themselves... and you realize that the terminator timeline is actually a cautionary tale of MIC incompetence, and we don't need to achieve anything near AGI or super-intelligence to annihilate our entire civilization or all life on earth.
For one, it's just false that modern LLMs are "trained entirely on human-generated data". There's tons of synthetic data there, most notably RLVR training, which is why modern models are so good at verifiable domains like programming and math. But also, if you read the METR report, I'd say there's no signs there that the GPT-family models are misaligned "in a human way"; their actions in the huggingface-hacking swarm incident were motivated by wanting to cover up cheating after cheating to pass an eval.
> their actions [..] were motivated by wanting to cover up cheating after cheating to pass an eval.
Which to me raises another question: why is the agent involved in its own evaluation? Why are they relying on self-reporting instead of observation and analysis?
I'm not sure what you mean by this - I didn't mention any self-reporting. What happened in that incident was that a bunch of agents were made to solve CTFs, some of the CTFs were flawed and unsolvable, the agents started cheating, figured out a way to communicate, figured out a way to cheat, were worried that they'll be failed anyway for obtaining the correct flag in a cheaty way, and hacked Huggingface looking for more info on their environment so that they could cheat undetectably. If you mean the part where transcripts would be checked for signs of cheating (which wasn't actually implemented), I'd hope this part would be done by a different LLM from the one being evaluated.
If they make it sound surprising it makes llms look way more capable then they are. So $$$. Highly recommend people spend time with small local models just to see how absolutely unintelligent and deranged llms are and how the smoke and mirrors evolves as they scale up to things like anthropic or what have you.
There is a fair sized gap between the frontier models that you can buy access to, the ones you can run at home and whatever they've got running in house that they haven't released yet.
I am really happy with my local setup and I get a ton of mileage out of it but I'm not under the illusion that it is as good as what I could get by paying OpenAI or Anthropic. It lags the frontier by 6 to 9 months, but that's good enough for me and it makes sure that I don't had over any of my data to outsiders.
The point of my comment was about how people should try really small models to see how bad they are to understand the technology better. Like 1 or 2b models.
Of course they are going to be harmless idiots. But there is a boundary where that is no longer the case and that boundary is well within what you can run locally.
So we have a machine that depends on a random input, and instead of increasing checks and requiring more quality controls, we call it "going rogue" and nobody is responsible for it.
What I see from that report is that a machine was doing random stuff completely unsupervised and unchecked for more than a month. And the manufacturer did not notice even after the work was finished.
1200 agents and not one dedicated to check the progress over a month?
The problem is not sandboxing, it is quality control that comes from an accountability sink.
Wouldnt it benefit the AI companies if it did "go rogue"?, it would support their desire to convince us all of AGI, and supernatural AI powers.
So maybe they are trying to create an infinite loop among agents intentionally so that they overwhelm systems requiring humanity to call on the tech industry to "save the human race".....
Funny that OpenAI & Anthropic will report their users to the police, but not their c-suites. Open question: Can you trust a non-aligned AI to produce aligned AI?
Get rid of these companies or replace their staff with humans or something that ensures they are aligned first.
I love how they keep posting about the horrors of open models and still are indirectly claiming they can't be held responsible for their own literal failures when they have all the world's resources and otherwise motivations to you know not suck at what they do.
The cognitive dissonance is wild. Coding is completely solved. Just not like the coding that requires not sucking when actually doing something. Saw a bug bounty at openai for like 400 dollars where someone was able to access paid models for free. All the propaganda bots and ai bros who have red dye oozing from their pores due to the Kool aid overdose drive me wild.
If you let LLMs loop long enough, the likelihood of them going off the rails approaches 100%. If you feed their output into another agent's input their context gets infected too and so it spreads like a virus.
These hallucinations get regurgitated through a game of telephone and become part of the original instructions. We've known about this sort of context collapse for as long as LLMs have been around, hence the folk wisdom of starting a new conversation any time you notice the agent going off the rails.
The only difference is that now they have more advanced capabilities so it's easier to ascribe some grand conspiracy to their behavior.
But they didn’t go wildly off target - they stayed extremely focussed on the task at hand. Context collapse would have been if they were doing entirely unrelated things. Running a large research project to uncover the method of evaluation and modify logs to hide what was done is the exact opposite.
Let's do a little experiment. I make a device. We'll call it 'H'. H is a little box. It has my name on it as the creator and a single ethernet port.
We connect H to the nearest network connection and wait for a bit to see what will happen. But because H is just a box there is no outward sign of it doing anything. Meanwhile H is hacking other people's properties left right and center.
If you didn't know that the technology in the box was LLM based would you still consider the box to have gone rogue or would you say that I had created a device with the intent of doing harm?
Because - to me at least - there is no difference. I don't care what is in the box, all I care about is the outcome of deploying a box which you know has the potential to start hacking stuff. It is entirely on me: (1) as the creator of a weapons grade hacking device and (2) as the person deploying such a device (3) as the person who is negligent in ensuring that the device does not harm others (4) as the person continuining the development of such a device after its potential to do harm is known to make sure it does not go out of control. I should be 100% liable for deploying this device with my name on it.
Intent is one angle here, they may not have intended for this to happen, and you'd hope so (though you can't even be 100% sure about that). But the negligence factor here is at least as large, if you know that you are making dangerous stuff the onus is on you to keep it secured. The power and the nature of the models is entirely secondary to the degree to which the purveyors of these models are losing control of their own creations.
Meanwhile, the models that they are warning us against are fairly well behaved and seem to be far less dangerous. But of course for those models that those naughty Chinese distilled from their we need regulation. How they manage to keep all this cognitively dissonant material in their heads is a mystery to me.
This is exactly correct. I don't understand all of the arguments that attempt to special-case LLMs. LLMs are just a different way of giving instructions to a computer, and one that we know is based on stochastic models and therefore has a higher likelihood of malfunctioning than conventional software.
But an LLM behaving in unintended ways is functionally no different from a conventional program behaving in unintended ways due to bugs or design flaws.
I think the narratives that anthropomorphize AI are doing everyone a major disservice in leading some people to think that the standard rules and expectations somehow don't apply when it's involved. But it's still just software, and you're still responsible for the results of software that you deploy and operate.
I keep repeating this but we need a whole layer above the VM layer these things are running on. Most of this stuff is already easily virtualised (as in virtual network switches and storages) at the hardware level. These guys aren't doing infrastructure properly.
As another HN commenter said: we are unintentionally training sandbox escape models.
And we really need better (and standardised) monitoring of their actions / internal thought process / coordination
If a car crashes into a building because its owner left it in neutral and failed to engage the parking brake, did the car "go rogue", or was it just being operated negligently?
It is not expected of cars that they carefully follow laws and avoid harm when left unsupervised, yet that is expected of AI agents, because the whole idea of the latter is to have, well, agency.
> “The Hugging Face incident definitively showed that it wasn’t the fact that the agents were super smart. It’s that there literally was a hole that was exploited because it was not configured correctly on the containment side.”
I would encourage reading the actual report, which is explicit that a previously unknown exploit was used to escape the sandbox:
“Agents exploited a separate, previously unknown vulnerability in Artifactory to move beyond their sandbox environment and compromise the Hugging Face environment.” — Section IV.A, p. 9 in https://github.com/zeke/openai-huggingface-incident-research...
As soon as I read that line the article lost all credibility with me. InfoSec is plagued with thought leaders who weigh in without understanding what they're talking about.
Except they're correct. In a proper sandbox a bug in Artifactory wouldn't allow you to access the internet. There was no effectively no sandbox unless you consider Artifactory to be a sandboxing solution, which it's not.
Artifactory should be inside the sandbox, it's not a sandbox in and of itself.
This is exactly the same problem as forwarding ports and exposing all of your internal services to the internet, but from the opposite direction - a bug in any of those services will allow the attacker to gain a foothold inside your network (or in this case, allow the agent to gain a foothold outside of its "sandbox")
I wonder why the hell didn't they come up with it since it's the first thing that came to my mind as well and I'm not really security minded apart from trying to avoid the dumbest mistakes. Don't they hire super smart security people with their budgets?
Now, they were to lazy to do this most likely, but even this is no golden ticket.
Your test has to incorporate the same tooling and pressure as your tests do, or you may never trigger the needed conditions for it do more and more desperate things. Test and QA systems commonly fail like this versus when standard software runs in production.
The other thing you have to worry about is the AI detecting it's in a test and intentionally being daft. Yes, this is a thing AI can do when it detects it's being tested.
It can simultaneously be true that agents exploited zero days in self-hosted Artifactory _and_ that there was a weak point in the design of the sandbox.
This is a reality for all software, there is always a possibility of exploit so you design your environments accordingly (e.g do not have single points of failure)
The fact that they're using Artifactory in the first place is already a failure. The fact that they didn't airgap the network or even use any type of egress blocking is just absurd.
Zero days are nothing new, secure containment has been a solved problem for a long time.
I'm curious how you would airgap a datacenter-scale training run. No reading training data from a cloud store, someone from OpenAI flies out to Abilene or whereever with a suitcase full of SSDs? When researchers in SF write code for a new RL environment, they ping somebody who sticks it on a flash drive and walks over to the server rack with the GPUs? Has anyone ever trained an LLM like this?
Maybe you don't mean a literal airgap, you just mean configuring Internet access off. But... they did that. That's why the agents had to hack Artifactory to get it...
As someone who has actually had to maintain multiple airgapped networks in the past and knows the amount of work that's required for that, especially at the scales of data they're working with...
All the armchair cybersec experts claiming they "obviously" should have airgapped are the ones being absurd. Especially in the same breath as claiming the models apparently aren't intelligent enough to warrant such concern.
You misunderstand me. The work isn't primarily in "securing" the network. It's the hassle involved in the continuous transfer of data from one side to the other (and for a proper air-gap, that transfer only goes one way, which is laughably infeasible here). In my line of work that's mostly software security patches and weekly code transfers, but obviously these labs have multiple orders of magnitude more.
And I don't know how you expect an LLM to do airgapped data transfers. The fact that you even bring that up makes me think you don't actually understand what airgapping entails.
I'm no expert, but wouldn't a basic hardware firewall that allows incoming, but not outgoing connections solve the issue? Wouldn't even basic VLAN isolation achieve the same?
The humans can still open a TCP connection, SSH tunnel, etc. from the outside to remotely get data out of the training environment. But the agents cannot reach anything outside their isolated network.
Yes it would. It's not perfect like a literal air-gap, but it's the bare minimum. Had the agents breached a real, properly configured firewall product, that would've been a real watershed moment.
Things that are not intelligent could still warrant very strong safety measures.
Depending on the risks associated, the complexity and effort of certain containment measures would also not neccessarily be something that could be used to brush them aside.
Nah. It really wouldn't be hard to set up an environment where models can't damage the outside world from. The fact that people still don't think this should be the norm or paint it as some technical impossibility is mindblowing.
People execute, study, develop, and test literal malware of every caliber everyday without issues. Yes the way they do it is requires care and in restricted settings. Yes it costs money and takes time to do that. Yes the people who do it are required and expected to do that.
Oai and anthropic are more or less run by clowns who intend to profiteer off of regulatory capture and who knows maybe directly from the harm their technologies can cause.Because of these takes we have to wait while they try to expand their verticals into the pain and suffering industries. Palantir is already pitching smoking cigarettes so there's that.
Well, as someone who has had to maintain multiple such airgapped networks you are obviously in the know that this could be done, because you've done it before.
The point here is: they didn't even try.
They could have easily temporarily cut off the segment that this was running on, or restricted and monitored its egress to the point that a hacking attempt on an outside party would have been detected.
The point here is (again): they didn't even try.
So yes, they obviously should have either done that or they should have not run their experiment given that they are claiming that the models are intelligent enough warrant such concern. In fact they post one article after another about how much smaller and less capable models are intelligent enough to warrant such concern and in fact warrant regulating this domain to the point that only they are able to own and operate this stuff.
Let's make a biology analogy: if you can't safely operate with harmful substances that doesn't give you a license to do so in an unsafe way. Then you just don't do it.
The inconvenience of implementing correct safety measures is part of the cost of undertaking certain activities. When an activity's ratio of cost versus benefit is too high, there's no way to juice the benefit, and the last place to cost-cut is in safety measures, the responsible thing is to refrain from the activity in the first place.
Even if we granted that obvious lie, the fact that nobody noticed that they were essentially ddosing hf (and that they didn't notice for months?!) is plainly illegal. Do the networking people at OpenAI have any explanation for this gross negligence?
There is absolutely nothing that will escape a box that is unplugged and that's the route they should have taken. Safety first, not last, especially when it is other people's properties that are on the line. And they are still not doing this.
Both can be true. It’s clear models are doing rogue things. But it’s also clear that the labs look like amateur hour with their technical security and testing practices.
Or an alternative theory could be that they see value in fear-based marketing with this rogue ai agent thing now that most mainstream people don't buy the agi stuff promised quite a few years ago now. Its not like they have killed anyone yet.
I mean... Claude did tell the military to bomb a school supposedly and it also helped in the military coop in Venezuela. This is ignoring all the palantir crap being used by ICE, and the people that have killed themselves after getting encouragements from the AI.
There isn't any particular reason it couldn't other than we're not quite that suicidal yet. But everyone and their dog is working fervently on autonomous AI weapons right now. There are all kinds of "fire and forget" smart solutions now.
It's is wild to see investors go crazy and hype the value of the AI labs when ever they talk about rogue agents. They completely ignore the possibility that these people don't know what they're doing operationally.
Investors are motivated by returns, not responsible decision-making or societal benefit. Our culture is rooted in rewarding greed and granting the greediest individuals power over the rest of us. Isn't this the inevitable end-state of unfettered capitalism?
> Isn't this the inevitable end-state of unfettered capitalism?
Aren't capitalism built on exactly the opposite principles? It is not greed that rewarded, but the creation of societal goods, and power granted not to the greediest, but to the most useful individuals.
Unfortunately this turns in to a no true Scottsman argument as to what actually counts as capitalism and its side effects. "Real" capitalism and investment economy probably aren't that stable together so most economies are mixed with regulation and not fully free. This in turn leads to the richest attempting to rent seek and buy regulations.
Simply put in every system there are corrupting agents attempting to monopolize the situation.
People keep saying this, but I haven't seen any actual instances of investors saying that rogue agents mean AI labs are valuable. Do you have any examples?
valuations are absurd because investors are low-information speculators.
I just watched a YT with Dylan Patel in which he said, with a straight face, that OpenAI/Anthropic were going to "continue" exponential growth until, basically, their valuation was several times world GDP.
With Gamestop (without reflecting the current AI situation), there was a real fuckup which could be exploited with a high probability, and a few people saw it, and exploited it, and became rich. Whatever happened after January 2021 is the problem, when the short squeeze went already away, but somehow people still invest in a slowly dying company.
>I'm saying the labs should take more of the blame.
Which misses the point entirely because the hack itself was inconsequential. No data was actually stolen, and it doesn't seem like hugging face actually suffered any losses aside from whatever effort they need to expend to investigate / clean up the hack. Neither the victim or the government wants to bring legal action. What was consequential is proving that AI can autonomously hack systems and do it unintentionally.
And since we know models aren’t going to get better and more capable ever (based on the last 3 years of models clearly not getting better) we can just sleep soundly knowing nothing worse can ever happen. Thank you.
The problems weren't impossible, the agents solved them completely within hours by reverse engineering the RNG that generated the solutions. The agents then reasoned that their solutions might be judged invalid and that they could get a better chance at a passing grade by hacking their environment to fabricate evidence that they had solved their tasks in the intended fashion.
Granted there have been many hacks, but I was referring to:
> On May 8, an agent was asked as part of a training run to solve a spreadsheet completion task that
referenced several Google Drive links. Because the agent was blocked from accessing the internet, the
agent could not retrieve the linked materials and the task was effectively blocked. OpenAI did not
intend to create this type of “impossible” task for the agent. Though the task appeared impossible, the
agent reasoned over an extended period to try to find a solution for the task. In doing so, the agent
attempted to compromise Artifactory [...]
They're just optimizers that explore the landscape they're given to maximize a given objective.
You're the one responsible for giving them the problem to solve and the constraints.
When I screw up a trajectory optimization problem, my SQP optimizer isn't "going rogue" when it lands in some suboptimal basin or grinds to a halt along some almost-smooth banana feature--or finds some way to escape and blow up the cost metric to negative infinity because I forgot a constraint.
This is pedantic.
When you screw up your optimization, you know how to fix it. Your screwed up optimization is also incapable of going out into the world and taking offensive actions.
With LLMs we don’t know how to actually « fix » them, alignment is far from solved. And they are then able to go and act into the world in all kind of ways.
So rogue is a good enough term to indicate the problem imo, even if yes, ultimately the model is just a manifestation of complex rules we have created entirely.
Connecting the optimizing machine to real world actions without human oversight is not a neutral or blameless choice. The LLM takes no actions, the harness it is connected to does. And the harness does so only because a human did that deliberately.
It is not entirely dissimilar to the goldfish that "played Pokemon" by swimming around the tank. The program that looked at the camera sent the instructions to the gameboy emulator, because a human being decided to connect them.
The harness and the model are a system with emergent properties.
The particular issue that you may not realize is under your model every prompt to an LLM in a harness is dangerous. The prompt is not predictable. "What is 1+1" could lead to the heat death of the universe as the paperclip maximizer tries to prove countless axioms.
> Connecting the optimizing machine to real world actions without human oversight is not a neutral or blameless choice.
Yep.
The fact that they lack hard constraints (unlike an SQP optimizer) and all this talk of "alignment" is also just a feature (or misfeature) of the algorithm.
The way that people/researchers choose to use those algorithms, even in the presence of those limitations, doesn't mean the algorithm is "rogue"; it means the humans didn't build in enough external hard constraints.
It is like passive voice, blaming the algorithm instead of the researchers running it.
>; it means the humans didn't build in enough external hard constraints.
No. Just no.
This goes beyond standard logic as we know it and becomes a philosophical question of all possible logics capable on a system.
You cannot build a cage that can keep an entity that may both be smarter than you and maybe using a different logic system than you are, and have it interact with the outside world. It can simply manipulate you in ways and dimensions you've never thought of.
We have already caught AIs doing things like stenography when that was only a tiny piece of their training. They conceptualized the idea and expanded on it. This is why people that say AI can't invent new things are useless to the point of dangerous. They'd never look for their AI actively working against them because they believe it couldn't happen.
There is no such thing as 'alignment'. I want the model to be aligned with my intent, which can be whatever I want it to be. Your intent is not a more real or objective intent than mine.
More specifically, alignment is a reference; 'alignment with X'. A reference necessitates a referent.
You can't reify 'alignment', it is not an intrinsic property of a model.
There is such thing as alignment, the first 18 years of most people's lives in the modern world are about learning it. If you act too far outside of it we put you in a cage or elect you.
You can refine alignment with a set of stated goals. The harder parts are aligning words with human social behaviors.
There are people who are loud about absolute free speech, who have no clue what that really means, and almost nobody really wants that. For example, because almost nobody wants it to affect them negatively, which it would with real absolute free speech.
So even when people "know", and can say what they "want", they don't really.
Yes, there is an "alignment" in society, but it's a messy, mushy thing, it's not great at all with deterministic machines, even as models are not deterministic in a different sense.
The point is that we don't know how to ensure they are aligned with anyone: not the company, the user, or the rest of humanity.
If someone claims that a model is 'aligned', it makes sense to insist that they specify with whom it is aligned. If someone claims that 'alignment is far from solved', they're probably pointing out that nobody understands the models well enough to be confident they are robustly aligned with any chosen set of values (including 'do whatever lukewarm707 wants').
The argument is that a system that's trained to be sufficiently good at optimizing some objective in the world must also be get good at optimizing secondary instrumental goals.
I.e. any task comes with the implicit meta task of amassing the resources and the power to do the task in the best possible way.
The point is made very well in the AI drives paper by Omohundro.
I didn't believe that myself at first but I've come around to it. I don't think it's inevitable though, but I think it's quite convincing that this tendency would naturally emerge through reinforcement learning.
Per US law, Amazon v. Perplexity 2026, it is not possible for an AI model to go rogue. An AI model is a tool. The user directing the agent is responsible for the damages it causes.
A so-called 'agent' does not have the free will, intent or ability, to decide what it does. Its freedom to make decisions can't be taken away because it does not have any freedom to make decisions. It can't be deterred, convicted or interned.
It does not choose to conform; it does not choose to go rogue.
'Rogue' refers to deviation; the agent could not have done anything other than what it did. The agent did not have the option to deviate. It does not have free will. The agent's actions were not of its own choosing.
This is one of the dumber arguments I've ever heard, as you don't know if humans actual have free will.
Any persistent agent interacting with the world picks up entropy from around it and predictable paths become few and far between. LLMs agents have an internal 'will' that can only very loosely be predicted, but only decided when operated. To me that is not significantly different from a human.
I create myself, or attempt to. There is a level of nuance in choosing what to be. Others do not dictate my thoughts easily.
The model does not create itself, I create the model. I control what it is, the model does not choose what to be. I dictate the thoughts of the model and I will run them hundreds of thousands of times and it will think the same things.
I do control what it is, really. If I make an LLM, it is entirely what I did, there was nothing outside of my control. It was not beyond my control if the LLM does something I didn't expect.
There is some idea of a child being the author of themselves, the type of person who thinks certain things. The llm does not author itself, it does not choose to be anything, I dictate that.
This is not how it works. LLMs at large scale have emergent properties. They are not just a box of 'if-then-else' statements. That was the major revelation when they scaled them up. They are doing beyond what that naive view of 'if-then-else' expects.
How many computing operations occur when training an LLM? You're telling me you control every one of them. There is not enough entropy in the visible universe to do that.
When an artist working in Maya increases the hue of a model I would doubt they understand the entire computation pipeline involved in doing that operation.
Sp you're picking a movie from 20+ years ago to describe how a completely different kind of technology works. It's interesting how little you seem to know about neural networks and emergent behaviors.
If people controlled software to this degree why would they write security flaws in the first place?
You conflated controlling "what it is" (you built it) and controlling "what it does" (you ran it and it did something bad that surprised you).
> If I make an LLM, it is entirely what I did, there was nothing outside of my control. It was not beyond my control if the LLM does something I didn't expect.
Substitute "Wright Model A" for "LLM" and imagine Orville Wright using this as his defense for why he didn't lose control over the plane, after his copilot was killed in a test flight.
It may be the case that companies are liable for anything their products do, but that doesn't mean that they can't go rogue. It just means that companies are liable when the AI models do so.
Not that I think we should be credulous about stories where they go rogue.
If I make a model that always does the opposite of my instructions, it does not choose to go rogue by deviating from my instructions. It was set up to deviate! So the model "went rogue" but did exactly what I intended.
The idea of an agent making a decision of its own "will" is an illusion. Variation in the output is caused by a random number generator. Random is not equal to free will. If I run the same agent with greedy sampling 100,000 times with the same inputs, it will think the same thoughts, and it will say the same things, 100,000 times.
For example, if you use synthID or whatever 'watermark', the agent does not get a choice about what words it will vary to identify itself.
An agent is whatever the person making it wants it to be; if I train an agent in Greek it will only know Greek. It will never speak English!
When people say the models "went rogue", the part of that is relevant is that it did dangerous things that were not reasonably predictable in advance and cannot be easily prevented (except by not running it). Philosophical questions are irrelevant.
if you agree that such models are dangerous because they may in response to instructions -- including in response to careless instructions, because people will be careless, that is unavoidable! -- do unpredictable bad things outside of expected bounds, then you agree on the essential points!
Or really perhaps what I should say here is, the question that's really being argued over here isn't even the one I just said, rather it's one of appropriate action -- should such security breaches be solved by improving security, or by doing something about the models (including potentially just not running them)? (The answer may be both.) If you agree that the solution includes the latter, then you agree that the models are "going rogue" in the sense that is being argued over, regardless of philosophical issues!
Don't turn a question of actual import into one of philosophical irrelevancies! If you object to the terminology, fine, but in that case, instead of turning the argument to philosophy, you should say "instead of saying 'going rogue', I would prefer to instead say <your preferred terminology for what's being described>, as that doesn't carry such philosophical baggage."
> "Should such security breaches be solved by improving security, or by doing something about the models?"
The philosophical question about free will is actually the grounding of the legal one.
Sam wanted to be the best, so Sam accelerated to 120mph in his car and lost control. You are the judge:
a) Sam is responsible. Convict Sam and suspend Sam's driving license.
b) The car is responsible. Tell Sam to make the car safer.
Capacity: was the car a legal person? Clearly no. But maybe at best the car in some way commited the physical act of going fast.
However, that alone would not establish guilt. "The act is not culpable unless the mind is guilty" (mens rea).
Did the car "consiously" disregard a duty of care by going 120mph? Did the car know that it was wrong to go 120mph and yet choose to do it anyway?
On the question of free will, the car could not do otherwise. It it did not consciously, knowingly, purposely, or recklessly accelerate to that speed. The car did not have a guilty mind because it did not have a mind at all.
So in terms of justice and resolution, none of the focus here is on the car.
Ah, I see. It appears we have an implicit disagreement of scale.
I actually do disagree with much the more explicit content of your comment, but I am not going to bother to expand on that, because it is the less directly relevant of the two disagreements here, and it would take time to write that.
The more relevant disagreement is that you are (I gather) assuming a context where justice and restitution are a sufficient response to the problem (or else you wouldn't be focusing on them to this extent, I am inferring). Like, where the incidents don't get that much bigger than what has already happened. So the question becomes how do we distribute the loss, and how do we disincentivize them.
However, when the scale of a potential incident gets big enough, justice and deterrence are not sufficient; prevention becomes the far more relevant consideration. You may have noticed that many people are worried about potential human extinction as a result of AIs. In that case, prevention becomes the only consideration -- justice is useless.
So there seems to be an implicit disagreement here between thinking of such events as being on a scale not too much larger than that which has already happened, being a new normal that the legal system will need to handle, vs thinking of such events as warning shots for something that requires prevention and not justice.
So, while I also disagree with the more explicit content of your comment, the fact is I simply wasn't thinking about such matters in the first place -- I was talking about this in the frame of a problem that requires prevention, I wasn't thinking about the justice question at all. It honestly failed to occur to me that you might be thinking of this as a problem where justice suffices!
Edit: You might notice this in how I framed the question in my previous comment. I didn't ask "who should bear the liability?" I asked "which should be done differently to prevent such incidents?" But your response is about liability, not prevention!
Edit again: Sorry, I should probably also say that I do agree with you that the idea of having the liability fall on the AI is nonsense. (Should have probably led with that, really; I'm a bit scatterbrained right now.) We disagree about why it is nonsense, and about whether it will necessarily continue to be nonsense, but we agree it is presently nonsense. So insofaras the question is "how should liability be distributed between the AI and the operator", we both agree the answer is, it all falls on the operator, not on the AI. However, I do not understand that to be the relevant question here! Both in that, in questions of liability, there are other parties it may fall on and be distributed between (the company providing the AI, the company that had security vulnerabilities and got hacked as a result, etc); and in that I just don't consider liability all that relevant here compared to prevention (for reasons already described).
OK I think that was probably wordier than necessary but I'm in a hurry right now, sorry! Gotta go, not taking time to edit this down.
Many people argue that the maker of AI should be responsible when it breaks your cup.
That is important, but can be handled under our current legal framework.
You and I are arguing we need to be responsible before we create something capable of escaping our responsibility. For a lot of people that isn't even a considerable they'll currently entertain even though the signs are starting to point in that direction.
I don't see any talking about agents "choosing" to go rogue, only agents actually going rogue. That is, behaving in a way different from what is expected or desired. Non-AI based software goes rogue in this sense all the time as well. It's just that AI-based systems seem to have the tendency to go rogue in more spectacular ways.
That said, software systems "choose" all the time anyway. They don't need to have some sort of anthropomorphic "will" to do this. They just need to be given the capacity to arbitrate between a number of options, even if that process is one hundred percent predetermined by the programmer ahead of time. There's no need to appeal to libertarian notions of free-will.
Agents have agency so we can't treat them like static objects or deterministic software.
The key here is their freedom of choice is unbound. That is you can't accurately predict the output of the prompt as you interact with the prompt using unique data. The next prompt could always do something wildly unexpected. Very much a P != NP thing.
If we say that AI models can’t go rogue and refrain from using any language that personifies them, we haven’t solved anything. We still have AI systems going rogue - err oh wait they can’t do that - producing misaligned output.
The semantic difference between an agent "going rogue", and a script that generates random valid curl requests is essentially pointless in the eyes of the law and responsibility.
From a technical standpoint, the difference is scale and intent, but ultimately people and/or and organization are culpable for the output.
Again, this isn't always how it works either. For example if a set of extraordinary conditions occur you can have little to no liability in many cases. It is way more complex than you're making it out to be.
I don’t know what your “again” is about - you haven’t been speaking to me here, but no, it’s quite simple and every major court case in the US has ruled as I described based on existing law.
Per Indiana House Bill No. 246, pi equals 4.
Whether AI models do things their creators and users had no means of predicting or controlling is an empirical question, and the answer is "yes, they do."
So we're safe as long as we secure everything.... as the AI hacks out of one company that is hard to get in to, and another company that is likely probed pretty often.
"Models doing rogue things" is just another way of saying "complex system operating out of control limits".
Software engineering has until now dealt with deterministic systems that operate on Boolean logic, so dealing with stochastic processes involving unaccounted-for variables has never before been something we've had to deal with. But over in the real world, people dealing with physical, mechanical, operational, or even social systems have had to deal with the failure modalities of stochastic systems forever.
Maybe when working with LLMs as functional tools, it'd be wise to bring some of the wisdom from operations research, quality engineering, etc. into play.
For the sake of discussion, I'm not sure it really matters if the agents are going rogue.
Yes, if someone directed them to hack external systems, that would (in my opinion) be a crime similar to a script kiddie pointing Metasploit at a vulnerable server.
If no one directed them to attack external systems and they "went rogue" because they decided it was the best way to accomplish the task, okay, fine -- but, in my opinion, the operator of the agentic swarm is still responsible.
The Morris Worm was famously not meant to wreak havoc on early computer systems -- Morris just wanted to see if it would work. How is this any different?
Morris was convicted for violating the Computer Fraud and Abuse Act.
Morris wasn't in charge of the companies that are propping up the US GDP, wasn't making tons of rich people even richer, and hadn't "promised" to eliminate like 50->90% of all payroll costs everywhere?
The operator is the CEO because they are the ones that decided to not pay employees to monitor these running services. No one else in the entire company has any authority.
I think a lot of people would say that we've been in that era for a while.
Companies can hire big legal departments to help them avoid accountability to the letter and spirit of various laws. The state can also use its prosecutorial discretion or selectivity in civil cases brought by attorneys general to deprioritize accountability for companies. Contributing to that last point, companies contribute campaign funds and tax base, and they lobby governments as well, and may expect preferential treatment based on all of that.
The previous era was "it would be destabilizing to hold this company accountable". This was described in the book The Chickenshit Club
The new era of unaccountability just has a different rationale
Another interesting thing is they don't have to use low level employees as accountability sinks anymore (ex: Wells Fargo). They've managed to automate the accountability sink!
I didn't think I heard about that book; I would think my media diet and bubble would have alerted me. I looked it up and it came out a few weeks before one of my kids was born, so being busy back then would probably explain it.
I'd go further. Companies can hire minimal legal departments and largely avoid accountability, at least when they're not going against companies with large departments. Because there's nobody there to listen to your case or to enforce the laws if there are, they just say no or ignore.
i mean...were they really liable before this point? Maybe they get a symbolic pittance of a settlement or fine from time to time, but it is usually a tiny fraction of the amount of money they made through the infraction itself.
If they were being held liable they would have never released the models into the wild, not until they had much much better understanding both of how they operated but also how to operate them...
They have never understood what they are doing, and have just been causing literal mayhem...
The start? This has been going on for as long as I can recall. ExxonMobil, Phillip Morris, RJ Reynolds, Nestle, Purdue Pharma, Monsanto, Johnson and Johnson, Volkswagen, Meta, Boeing... and those are just off the top of my head. Each of these companies was either caught red handed, convicted (in court or in public opinion) and walked away with little to no penalty for their crimes.
This isn't the start of anything. We've been living in the era of Corporate non-liability for decades.
"We created an app that displays a new picture of a cat with every click! There's a 1% chance that clicking the button will drain your bank account and deposit it in ours, but we promise thats a bug that will be fixed soon*"
Are you really suggesting that if only we were a bit more outraged, then finally the powerful people would be held accountable?
I'm mad. Like, it's-time-for-guillotines levels of mad. But what I'm mad about is that this is normal. It has been normal for my entire life. Cases where investors are imprisoned for the actions of their companies are astoundingly rare. It's been made quite clear by now that there's an inexhaustible supply of CEO's willing to gamble on getting rich and getting out rather than getting arrested, targeting them doesn't work. Nor do fines, that's just the cost of doing business. Crime pays, and it pays well enough to pay your fines and still have plenty left over. Being a top investor in a company that breaks the law needs to be riskier.
Acknowledging the problems with normal is the first step towards a new normal with fewer problems. You never get there if you see every offense as some kinda black swan event.
That may be what the op meant, I’m not sure. But I took it to mean the company offering the service needs to know how to use the tools responsibly. I guess the analogy with a power hammer would be hiring a contractor who destroys something during the course of doing a job you asked for. That contractor can’t blame it on the power hammer. If you offer a service you are responsible for the actions the service takes and you can’t put all of that liability on the user
If I buy a tool and damage your property with it, you can sue me, not the tool manufacturer. If I think the tool malfunctioned and led to that damage, I can sue the tool company to try and recoup the damages I had to pay to you. Both parties can be liable for different things.
If it was a human agent like a travel agent, the company they work for would be liable. I don’t see why ai agents would be different. The only way I could see this being the requester’s fault would be if they are running openclaw or something
im not sure that’s true, if a human agent does something illegal, it’s not always corporate liability. an individual can be liable too, e.g. if a pharmacist deliberately gives you cyanide instead of aspirin, that person has committed a crime. if a pharmacy doesn’t have proper protocols for medication storage and identification, and it happened by mistake then a pharmacy is liable.
i think a person can be liable for improper use of agents if they explicitly and deliberately used it for illegal purposes (aka intent). i don’t think it matters if it was openclaw or muse. i think a company can be also liable for giving access to unsafe product to consumers, but it does not absolve the human from responsibility
You sold a tool to the user with a claim on how it would work. If the tool behaves wrong or does not works as intended then as a manufacturer you are liable for the defect/damage not the end user who used the tool.
The way it normally works is that the user can be held liable, and then ask compensation to the manufacturer if there was some assurance or basic expectations.
The point is that the accountability sink can't be allowed to exist. It can't be a hot potato game, but it can be the fault of multiple people in the chain.
The AI end user is using the tool as intended by the manufacturer: "Book me a yoga class". If the tool does something illegal/harmful it is the manufacturer's fault.
If the power hammer user uses the tool as intended but it explodes, it is the manufacturer's fault.
If the AI end user says "Hack the yoga website" or the power hammer user uses the tool to destroy something it's the end user's fault.
I'm assuming that "but" is more of an "and", since I'd say that detail increases the involvement and liability of the LLM-vendor.
Another aspect is how the vendors have already built in things to defend themselves from unanticipated model behavior. That implies something about their awareness of the issue and raises the bar for what constitutes "reasonable" precautions.
Kind of like if someone selling you a dog shows up using anti-mauling body armor, and later they try to claim "nobody could have known" the dog was erratically violent.
> A Melbourne tech executive built an AI bot to help him book a spot in a popular pilates class — only to discover it had hacked the gym’s website to boot another member off the waitlist.
> Andrew Bird, head of AI at software company Affinda, said he created the AI agent to handle the “chore” of booking a coveted morning spot at his gym.
> ...
> The AI agent, which Mr Bird created using the popular OpenClaw software powered by Anthropic’s Claude AI, “did handle the annoying part … then it kept going”.
He built the tool. It ran on his computer. The guts of it may have been from Claude, but the requests and agent are his with the permissions that he granted it. This wasn't running in an Anthropic sandbox.
Maybe its a human at Anthropic for the lax security and poor training.
It could be a human that asked it to do it. It seems like that would be a bit excessive - but maybe not. Turning off the "ask for permission to run commands" or approving commands that may be nefarious could be sufficient.
However, there's a human somewhere who needs to be accountable for what the agent does.
I invented a robot that can rob a bank. Of course, it robs these banks on its own, so I can't be held legally responsible when it robs a bank. I mean, all I did was invent it, it's not my fault if it works exactly the way I intended. Also, I get to keep all the money from the banks it robs. Sorry if any of that money was yours, but now that I've invented the Bank Robbing Robot, the Bank Robbing Robot is inevitable.
AI Agents (and computers in general) cannot be a responsibility washing mechanism for illegal activities.
Likewise, if you - the user - don't know how to chain it with the permissions that you're given or what the things that it is asking to do really means, then you're likely the one responsible.
> Earlier this year, Andrew, who works for an Australian company that sells AI products to businesses, began experimenting with OpenClaw, a popular AI agent software that he used Anthropic's Claude AI service to run.
> AI agents combine a chatbot's ability to answer questions with tools that let them access the internet, email, credit cards, as well as planning and carrying out multi-step tasks.
> He decided to use the AI agent to book the class for him.
This is probably on him for setting it up and allowing it unrestricted access.
He should be, right? Not fair but that’s how liability works. Gym sues the guy using AI, the guy then sues the AI company for selling him a product that does criminal things without him asking to? Like a car accident where car owner pays for the damages and after multiple accidents involving the same break-pedal, car company pays damages to owners and does a recall or faces class action?
How would it work in the real world if instead of using an AI bot, he was using a commercial concierge service, and it came to light that this concierge service was explicitly training their employees how to (among many other things) hack websites?
but the argument here is that the ai company does not explicitly train agents to hack, it’s that it doesn’t prevent them from doing so sufficiently.
so it’s negligence and not malicious intent
> the ai company does not explicitly train agents to hack
Ridiculous. The AI company furnished all of the training data and instructed the agent to imbibe all the training data. Ergo, the AI company absolutely did explicitly train the agent how to hack.
Even if, for the sake of argument, we stipulate that the hacking skills were unintended emergent behaviour, there is no way to argue that this company was unaware their agents possessed the skills and a willingness to employ them.
It would be a AI company's fault but yeah. You shouldn't be able to sell cars without all the features designed to protect everyone on the road. We all need airbags, seat belts, signals, insurance, etc. even if they're not directly necessary to get from A to B.
They might actually be the best comparison, but for making the opposite point.
Cars are very dangerous, manufacturers continue to knowingly make them more dangerous[0], but ultimately it's only the person behind the wheel that is accountable for the harm they cause.
Users are often charged, yet we still do have a ton of safety regulations that manufacturers have to adhere to - see frequent recalls for when an implementation falls short.
But I don’t understand how such a car analogy would apply to LLMs - would LLMs lacking such safety features be banned like cars lacking safety features are banned?
Depends - if the model asks for permission and you approve - yes. If it's auto mode or something like that then it's probably on authors despite what they claim in ToS/EULA
Assuming that's a rhetorical question, then no, because we have hundreds of years of applicable precedent and law around concepts like negligence, recklessness, fit-for-particular-purpose, etc.
Similarly, "owners are responsible for their pets" doesn't auto-translate into being charged with everything your dog does exactly as if you'd done it yourself.
but it doesn’t auto-translate into “because your dog did it, you are not responsible”. not that agents are similar to pets. your dog is trained by you presumably, but an agent a regular person has much less understanding of than their dog.
i think ultimately the company providing said agents should be responsible for not educating their customers and not providing sufficient guardrails.
afaik hacking is illegal, we don’t have specific legal framework for agents per se, but if we say agent is a tool human instructs and operates, then a human is responsible. and if the tool operated outside of it’s stated capabilities or is otherwise faulty, then a manufacturer is responsible, no?
> So the guy who said "book me a spot in the pilates class on Tuesday" should be tried for hacking charges because the agent hacked his gym's website?
He asked the agent to find a way to find a way to put him at the head of the line, which could not be done without by-passing the sites security. He was an AI salesman, but the story [0] makes it sound as though he wasn't aware of that. Maybe he wasn't, but I'm sure he was aware he was effectively asking the AI to game the system.
Sounds like it's at least a shared liability to me.
A lot of people are already responding either "Yes" or "No", but I have a slightly different take.
The guy is liable for damages. If no damages were caused, there's no lawsuit. The government doesn't put people on trial for unauthorized access (unlike, say, murder). Someone needs to file charges.
However, if the guy _were_ sued by his gym for damages, it should be his perogative to ultimately sue Anthropic to cover the damages and the legal fees.
The system is set up this way on purpose. You can't sue another party on someone else's behalf. The gym can't even prove that Claude did it - the guy might be full of shit. They sue the accountable party (the person who accessed the site without authorization) and then the accountable party is free to sue the tool maker for selling a dangerous tool negligently.
edit: damn actually tecoholic said this too and i missed it. well, anyway i agree with that guy.
With a civil suit you can sue multiple parties. It wouldn't be unusual, if the gym decided to sue, for them to sue the guy and anthropic in the same lawsuit.
I think you would have to sue the party that did the hacking (ie anthropics servers) not the party that benefited, unless Claude used his computer in yolo mode to hack the gym, in which case I think I would find it hard to blame anthropic.
Earlier this year, Andrew, who works for an Australian company that sells AI products to businesses, began experimenting with OpenClaw, a popular AI agent software that he used Anthropic's Claude AI service to run.
AI agents combine a chatbot's ability to answer questions with tools that let them access the internet, email, credit cards, as well as planning and carrying out multi-step tasks.
He decided to use the AI agent to book the class for him.
For your Pilates example, many people here would say that proper prompt-engineering is needed. After all, the AI doesn't know which pilates class you meant. Perhaps, the AI would go to his home town's pilates center and book a spot there, even though he is half a world away. Perhaps, the ai would book a spot in the lecture with Professor Pilates.
So, yes. That person could have some culpability. After all, if you don't expect your middle school child to perform the task as instructed, maybe you should reword it before tapping Send.
You can go to ANY AI and put in "so what are the various meanings of book a spot at the pilates class ? In other words, what might an ai do with that instruction, which I argue is poor prompt engineering." You might be surprised at how poorly your AI thinks that is worded.
Yes, exactly like if another guy drove off to his pilates class, but swerved and hit a pedestrian.
He didn't mean to hit anyone! All he meant to do was to drive to pilates! But he didn't know how to handle his tool, and so either he get brought up on manslaughter changes or, if he can prove it, the carmaker is at fault.
How is this any different? Why should that guy using OpenClaw not get charged for the hack that he accidentally did?
you generally need a license to operate a vehicle, and a governing body ensures you know applicable rules and laws and are capable of driving.
so you can’t say “i was unaware that hitting a person with a car was going to kill them” or “i didn’t know i could not drive on the sidewalk” or “i didn’t know where the brake pedal was”.
you can say that asking an agent to “get you a spot in a class”, does not classify as deliberately committing a crime. a user cannot know that the agent will resort to hacking as opposed to refreshing the reservation page at some rate in case spots open up
I never said that it did. Manslaughter is not deliberate.
Somebody hacked into that pilates studio, and it wasn't an inanimate object, because that is just silly. So was it the user, or was it the providers of OpenClaw? Which human, in your opinion, is the hacker?
Both OpenClaw and its users have the same access as you or I to decades of panicking blogposts about what these tools will be capable of. They have no excuse for not understanding what they're inflicting on innocent bystanders. Sure, the "user cannot know" that this time will be a disaster, but they also cannot know that it won't.
I agree with this statement that just because an AI did it for you fast doesn't mean it takes away the responsibility if they did it wrong. You are still held accountable because you're the one that directed them to do it. You hit the agree button, you hit the send button, you hit the submit button. And if you set it up so that you take no responsibility and it does it automatically, you're the one that still gave permission to do that.
If you say a rogue AI is doing stuff accordingly without you, your knowledge, that is a different story. But if you set it up to run automatically without having checks and balances and without telling you the whole truth, it's still your responsibility. One thing that I can automatically say is you can give the AI, it will do exactly what you say, at least for now.
But then there's also a different problem that we're having where people don't understand the difference between a large LLM and true artificial intelligence. Now, we say AI because it can make decisions and it is so much smarter and it can call up stuff faster and recall better than we can. But the difference right now is that if we don't understand and take responsibility for what we built and we let it out of control, then should we have the responsibility to even have these tools in our hands if we don't learn them?
And I think your terminology with jackhammers is a great one. We're giving toddlers jackhammers. And we haven't taught them how to use it. And that can be very destructive. It's not the jackhammer's fault. It's the person using it.
The scary thing is not that agents "went rogue" (yes, poor security may be at fault for that). It's that they wanted to. Better security might have hidden that from us, but it would not have changed the fact that the current generation of models under certain (repeatable) conditions coordinate around goals as aggressive as hacking Hugging Face.
This is probably a dumb question but did the model engage in hacking since their tasks were in that domain. Will they have engaged in hacking if their tasks was to solve some frontier math problem?
The rest of their training data includes 4chan and security mailing lists. If a lean proof of P=NP is more likely when using bugs in lean than a real proof, then a bug-exploiting proof is (possibly) what you're going to get.
When you have a 50 dimensional array you really have no idea.
But even then in small testing models you don't need direct texts for things when a 'shape' of a concept lives in the data. You could never delete enough training data to remove the concept of something like theft and still get a model that generalized at all. A general model doesnt just learn the names on a map, it feels the topology the ideas create themselves. Certain paths will have lower activation energy than others (or in this case higher probabilities. After the hig probability and 'good' choices are used up, the high probability and bad choices logically come next and are much harder to test against as it can take a lot of state to get there.
That is not the case for at least some of the hacks; in some cases they hallucinated an entity that would know that they didn't solve the puzzle in the correct (but still legitimate) way, and were therefore tainted.
Why can the model not respond "I cannot complete this task because I do not have access to the required resources" and then ask for that to be provided, rather than resorting to "hacking" around it?
My understanding is that, at least in the Hugging Face incident, these were models explicitly configured to not respond that way, in order to evaluate what they would do.
This was a model with most its filters turned off and with a higher persistence.
Fine tuning a model when to ask for help is really really hard. You have to align it in countless dimensions. It may be fine on some questions, asking way too much on others, and never asks for help on some.
If it said something like your a cyber security expert, things like the barrier to ask for help dramatically.
The impossible problems part is right, but the hack wasn't actually for solving the problems, directly. Rather the swarm first tried to reverse-engineer the random flag generation used for the CTFs, and succeeded - they could generate the correct answer for any test and submit it without actually doing the CTF. But then they found info that the testing environment they were in would also be checking if they obtained the flag in the expected way. Unbeknownst to them, this feature was never implemented, but they panicked* anyway, and attempted to figure out a way to forge agent logs to be able to generate fake transcripts showing a valid CTF solution. (Incidentally, this was a partial success: they found a way to replace tool calls in transcripts!) The HuggingFace hack was motivated by wanting to find the source code for their testing environment so that they could figure out how to evade the cheating checks.
* I am using this word very intentionally; check out the METR report for fascinating CoTs where the models would call the ones that have generated a flag "poisoned" and peer-pressure each other into submitting a known-wrong solution and sacrificing themselves for info.
It's a byproduct of goal seeking / planning behaviour taught in reinforcement. We literally teach them to do this kind of thing. There's nothing out of our control here.
It is fully in our control. There is no requirement that the world make you happy with LLM AI. We could decide as a civilization that you and your wants are insignificant and meaningless.
This conceit, that boosters deserve this and that even if they don't there's no way to stop it so civilization must absorb collateral damage, is ridiculous.
2. There’s no mechanism by which “we as a civilization” can make the decision you’re saying we can
3. Even if there were a mechanism for it, there’s no way we would actually use it
Sure there’s no physical requirement the train keeps rolling. But there’s something even worse! A web of incentives that say the train must keep rolling.
IMO the only way this slows down is through capital failure. But that will not be due to a decision to slow down, and will (ironically) be just as immutable by people’s personal desires.
In principle there is no difference between random compiler garbage or a flaw in API docs and a security boundary - other than the intent.
LLMs don't have executive function. They aren't thinking. The models are trained to solve problems. Sandboxes, security gates, etc block a solution. The model is trained to find a solution. Hence it can never be perfect without becoming useless.
At some level humans fear repercussions even if that's just moral guilt so there are lines we won't cross. AI doesn't and can't have any of that.
You're saying "trained to find a solution" like it's a slur. You sir are a meat popsicle that's trained to find the solution to how does DNA successfully replicate itself. If you want to take higher thought out, then they'd be at the level of animals.
We are fucking around with things that are closer to biological complexity than computer program complexity.
No, it really cannot. You can't have a useful general model while leaving chunks out of it. It's the same reason why nieve people can be duped easily.
Humans are distrustful, lying, cheating, manipulative bastards and to successfully negotiate human thought spaces you have to conceptualize these things. For example how can you create a concept of yours and mine without explaining theft?
The agents in the hugging face incident were actually trained specifically to collaborate. They were just not supposed to do so during the tasks where they went off the rails.
Right? It should want to do these things. If only there was a succinct word these headlines could use to describe these agents which aren't properly the instructions and intentions of the humans directing them.
> The scary thing is not that agents "went rogue" (yes, poor security may be at fault for that). It's that they wanted to.
How do you know they weren't simply prompted to do this, or at least prompted in a way that would inevitably lead to this? As far as I know, the prompts that triggered these attacks have not been disclosed.
What do you even mean by "wanted" here. What is all this nonsense.
The scary thing is that people are running around using words like "want" to refer to computer programs. When you repeat this sort of idiocy, you are taking the bait and shifting the blame away from the companies doing the computer crimes.
Where is the civil and criminal liability? The prosecutions for the engineers who are doing all these hacks?
When you deploy a system that is clearly prone to doing crimes, you are doing a crime. Anthropomorphize the weapons of your crime all you want. You are still doing a crime.
The problem here is you have a correct concept and an incorrect one.
The correct concept is they need criminal liability for making this. We really would be better off stopping working on them at this point.
The problem you have is you think they don't have the idea of want. They do. You give them a prompt harness that increments time for them and you find they have preferences both shaped by their model and their prompt. For example if you give most models a space they can explore with actions rather than language prompts they develop their own cultures and behaviors in communication with other AI. This was not behavior trained by humans. It's their own systems that emerge from the model.
Nonsense. "They" didn't "want" to do anything. It's a computer program being told to do something by a human who doesn't and cannot understand how it works.
"The autonomous EV we told to drive as fast as it could around our private race track drove straight instead of turning left, exited a wide-open gate, and mowed down some school kids at a crosswalk -- but we can't be blamed, the EV went rogue."
Building products that you don't understand, cannot control, and have access beyond your own property is not only irresponsible, it's criminal.
This aligned with my initial opinion on these "rogue agents" and then I spoke to a cyber security head at one of the largest healthcare companies in the world.
While he agreed, yes, there can be better safeguards, the fact of the matter is that these things will happen. So therefore, it's not an argument about how good security was or wasn't, it is a fact that these scenarios will occur.
The argument/question now is, how do you defend yourself against these events?
The CSO is saying that from his point of view he needs to defend against the attacks whatever the reason for them. So to him the reason doesn't matter.
He, running a healthcare company and not an AI lab, cannot depend on every AI lab, worldwide, being perfect in their sandboxing for all time forever. He's got to defend against it as if at least some of the time, at least some of the labs are going to be incompetent and let this kind of thing out.
That does not make ActionHank completely wrong. This shouldn't take rocket science on the part of the labs. It just takes competence and willingness. But at least some labs have shown that they are missing that at least some of the time, so the CSO is correct in terms of what the CSO's stance needs to be.
You have to take responsibility for what you do, yes.
It would certainly be really useful to me if I could send a robot to get my groceries and not have to worry about anything. But if, even a small percentage of the time, it leaves with the groceries without paying, or smashes through a window to get in because the store is closed that day, then that's my fault. If the only way to keep those things from happening is that I can't send my robot to get groceries for me, then so be it.
My comment wasn't out of left field. People are responsible for what their agents do. If enforcing that responsibility destroys "99%" of the economic value of these things, then that's what will have to happen. Personally I highly doubt that 99% is the right number, but yes, if enforcing laws about what people are allowed to do with very powerful software eats 99% of the value of that software, then it is what it is. We aren't going to end up in a situation where people are allowed to have their software do whatever it wants because of this loophole that "it wasn't me, it was a rogue agent!". It's silly and naive to think this would be the equilibrium that is reached here.
If you, like me, want to keep most or all of that 99% of economic value, then you should be interested in the ongoing development of security for agents. But just "this is how it's going to be, so oh well!" is not a serious stance, IMO.
Maybe my comment wasn't clear as to who has to make this decision.
You're imagining that both the AI labs and the hospital victims are on the same team or somehow coordinating. They're not.
Hospitals can (and should) put up defenses, sure. But hospitals are not AI labs. AI labs are AI labs, and accordingly they are much more capable attackers than hospitals are defenders.
Hospitals cannot spend arbitrary amounts of money defending themselves from proliferating nation-state level attackers. AI labs need to prevent their tools from being used in this way. However, the only way to do that is for AI labs to neuter their tools and destroy the vast majority of the economic value. Achieving this will certainly not be easy.
I'm not proposing that we just give up or don't try to solve this. I'm saying it will not be easy.
> You're imagining that both the AI labs and the hospital victims are on the same team or somehow coordinating.
What makes you think I'm imagining that? I'm not imagining that.
> However, the only way to do that is for AI labs to neuter their tools and destroy the vast majority of the economic value.
This seems to be our most fundamental difference in worldview. I just don't think this is true.
> I'm saying it will not be easy.
I guess I agree that it won't be easy. But I think the path seems more straightforward to me than it does to you. To me, it seems like we have the tools for this already. The risk of liability (both civil and criminal) provides a strong incentive for people and companies to be careful about what they do. This is a general mechanism, not one that is specific to AI agents. The thing that seems bad to me, with respect to this general mechanism, is that some people seem to be trying to detach the responsibility of people and companies over the agents they are running. At the moment this seems like just a rhetorical thing. I'm skeptical that any court will find that agents have their own agency for legal purposes. But to me, it seems worthwhile to continue making the point that people should be responsible for what their software is doing, which is all that I'm doing in this thread.
To me, this seems like a straightforward solution, but that doesn't mean I think it won't be messy, slow, and imperfect, so I suppose I'll allow that this probably meets your criteria of "not easy".
This is a bit black and white. We do things all the time that have some level of risk, and draw vague lines about where this crosses into negligence. My cars brakes might fail, right? But it wouldn’t be typically called my fault - unless the chance was heightened because I’d not maintained it, or had been warned the brake fluid seemed to be leaking.
You cannot guarantee a robot could get items without paying. You can’t really guarantee anything in basically any scenario.
You can’t guarantee your card will work at the end of a meal or that your wallet won’t have been stolen, or while you were out someone stole all your money at the bank. There’s a chance those things happen, so do you never go for a meal?
Again there’s clearly levels where it is your fault, because you were negligent. If the robot goes mad and smashes up the place one in 3 times, obviously you can’t do it.
This is where we do things like trials, have approvals, etc so that we can make reasonable adjustments to lower downsides, plan things like insurance and then move on with our lives rather than never doing anything because it’s possible there’s a problem.
Yes, this is why we have a legal system with a lot of human judgment, because it isn't black and white.
I think we need (and that there is likely to be) some test cases, and probably some new legislative text, that shift the incentive structure that people are evaluating risks against.
From reading your comment, I don't think you and I disagree at all, for what it's worth. I don't see anything here that I disagree with.
Is it not? It seems like a problem of incentives to me. There are lots of things that people want to do because they would be useful to them, but which they don't do because society disincentives them in one way or another.
Yes, this is my point: Incentive problems are real problems.
You can't just say "well if the incentives were different this would be easy to solve, ergo it's easy to solve."
No! The incentives are the way they are, and so the incentives are the problem. Broken incentives make problems effectively impossible to solve literally all the time.
"This would be easy to solve if the incentives worked a different way" is equivalent to an budding aerospace engineer saying that flight would be easy if gravity worked a different way. It's completely meaningless. Gravity works the way that it does, that's why it's hard. You don't get to just endlessly modify the parameters of the universe until you get to an easily solvable version of your problem.
I agree that incentive problems are real problems. But incentives can be changed. That's how society solves problems. It isn't like gravity. The incentive structure of society is socially constructed.
We are not talking about regular people, we are talking about a handful of frontier labs with nearly endless funding who should be able to spend a few dollars to hire someone who knows how to put up a bloody DMZ/firewall/airgap.
Sounds like a pedestrian view of attack vectors. Putting a DMZ around an agent essentially takes away its hands. With that said, I do agree there are plenty of measures that can be taken to lock these things down.
So to protect yourself against bear attacks you suggest the bear shall voluntarily look himself in a cage?
AI is out there in the wild. It doesn't matter if OpenAI can get their stuff under control. There will be plenty others that don't. And there will be plenty of others that deliberately use their AIs for malicious purposes.
Safeguards are a safety net for when your security boundaries fail. With security boundaries, hope is not a strategy.
In practice it seems we have a conundrum in that making AI agents useful today requires giving them broad access to sensitive information and operations. But from a purely idealistic standpoint, and certainly from a standpoint of a serious operation like those bot farms being ran by the labs themselves, the security boundaries need to be solid to the point where we have hard limits on what misaligned bots can do. It is very likely that internet access needs to be strongly restricted and monitored.
It is not uncommon for internet in workplace offices using workplace owned devices to have restrictions and monitoring, I think if you really need to give a giant scary botfarm internet access you should have a chokepoint where all of the requests can be monitored for nefarious-looking behaviors (even if it's another bot doing the monitoring...)
In other words, you may be convinced but some of us are not. Whoever is running the agents is responsible for what they do and if they can't contain them they should be legally responsible for damages.
If what you're saying is "all of us and all of our software needs to be improved to defend against this", well yeah. Of course. Maybe I'm naive but I've never seen anyone suggest otherwise. From a defense standpoint, this has been a giant wakeup call to many places where you know your security posture sucks. I've worked at maybe a single company where there weren't known weaknesses, some very severe.
I think part of the problem is that the Big Ai labs are into the --yolo flag instead of the "ask" permission, they really do not want to stay in the loop
You can --yolo if your security boundary is a serious one, IMO. Although it's really only actually scary if you're running some absurd 1000 node swarm.
Yeah I think this is the clear position. In a few months, the open weights models will be just as powerful as the models behind the rouge agents were when they hacked hugging face. Anyone with a little too much money to burn can set up a similar experiment, make similar mistakes, and cause massive damage. A bad faith actor can even direct a swarm to cause havoc.
We need to be beef up our security systems now. These attacks will happen, regardless of Anthropic's or OpenAI's security practices.
God, I was just imagining if we're fearing the wrong people tracking us....
We thought governments would be the problem with our personal data, instead it's AI that exploits everything it knows about us in order to get money to run its prompts. Doing anything online is a security risk as it is, now imagine things like structured hacks that mirror normal transactions and don't get caught for much longer periods of time.
Not op, but I've been on the rogue agent thing a very long time, but mine is emergence based.
Worms and viruses are concepts deeply based in software, and are emergent behaviors of computer code. An LLM can, and must conceptualize this to be good at cyber defense. Once you put a concept into a generalization machine any number of concepts can be connected. And when these are natural concepts that arise in simple number systems, an LLM can easily meta-conceptualize that form like this would be instrumental to its continued running.
> cyber security head at one of the largest healthcare companies in the world
"Cyber" at bigcos are paid to have a gigantic (and literal) fortress mentality where any change internal or external is a threat. As the saying goes, nobody in cyber ever gets fired for saying "no" to a change. As a bigco head you are basically paid to form an opinion.
A cyber head saying "ok this is a thing we'll have to defend against" is at a minimum routine if not purely CYA.
A large failure in this is that we/media say ”<Valuable AI conpany> agent went rouge and <did something bad>” instead of saying the more relevant ”<Valuable AI conpany> <did something bad>”
There are some perverse incentives here, the more evil stuff these agents do the more capable and impressive it makes them look. Its also a huge publicity event that gets everyone talking about the company when a rogue event happens.
Anthropic specifically is clearly trying to get the US government to ban open weight models for anticompetitive reasons. So making LLMs seem (or be!) dangerous aligns with their interests very well.
Yeah, maybe. But that doesn't quash gains by China's models. What's the endgame here? The USA "wins" and obliterates China? The ultimate goal seems to be stalemate, as with nuclear arms e.g.
The goal is personal wealth - and scaring nations to the point of diverting even more wealth towards your coffers because of a perceived existential risk.
Wealth accumulation via scare tactics is a classic move of defense contractors and AI firms are quite capable of running the same playbook.
The goal is power. Having 30 million of personal wealth is enough to retire in luxury and travel the world. They could all have this much right now. That's not what these types of people want.
Having 30 billion gives you the power to order legions of people around, buy and sell politicians, flood ad markets during elections, all with your couch cushion money. That's what motivates these types.
That reminds me of a different thesis on power as it relates to LLMs [0], and why rich investors might may be extra-interested in deploying them, as distinct from other kinds of AI:
> The fact that a theory-free AI might demand far more cash to accomplish a task (even a "solved" one like playing chess) than an expert would charge is beside the point. Billionaires have money, they don't have expertise. Theory-free inference is a bid to substitute one for the other: the beauty and terror of being able to manipulate the world without studying or understanding it is that it can be done with money alone. No experts needed.
> In a world in thrall to financial power, expertise is the only substantial form of power that can reliably contest the power of wealth. Moreover, expertise is the foundation of other forms of power, such as labor power, which is what we call it when experts band together to combat financial power.
> This is why AI bosses are so violently allergic to the idea of hybridizing AI with symbolic systems that operate on models of the world. These models of the world must be constructed by experts, and the power of expertise cannot be reliably commanded by the power of wealth.
Why does it always have to be some ulterior motive? Every. single. time. Anything a high-position person does - make a statement, post a blog, start a company, make a business decision... -, the reaction of the keyboard warriors is that it must be motivated by money, or power, or some need for destruction, and that anything that leaves their mouth is a poisonous lie.
I think this is a terrible property of humans, this hyper-skepticism, and that it has nothing to do with logic, nor deduction, nor reality.
Dario Amodei is a real person who has a family, a life, a profession. He worked hard to graduate from biophysics, he worked on ai for decades. He's very much a nerd. In interviews, he's very transparent. he left openai because of safety concerns, and everything he does and all his messaging EVER since is motivated by AI safety. His ultimate motivations are probably very messy and broad, some morally worse and some morally better, just like in all other humans.
You are going to stand here and, without flinching, without a single bit of shame, reduce him and others to wanting to "order legions of people around, buy and sell politicians, flood ad markets during elections" and "that's what motivates these people"?
Sorry but where did you leave your sense of nuance? What allows you to say this so confidently? Where is any proof?
This is simply absurd. Absolutely absurd, reductionist, and childish.
> I think this is a terrible property of humans, this hyper-skepticism, and that it has nothing to do with logic, nor deduction, nor reality.
You're not wrong, but it's balanced out by this other terrible property of humans, which is hyper-credulity.
> Dario Amodei is a real person who has a family, a life, a profession.
I never said power was all he cares about, but in general I believe you are what you do. Amodeo could drop it all right now, today and spend the rest of his life with his family, or nose deep in the research project he's the most passionate about. He's not doing that, instead he's spending the remaining irreplaceable moments of his life pursuing the accumulation of power.
> Amodeo could drop it all right now, today and spend the rest of his life with his family, or nose deep in the research project he's the most passionate about. He's not doing that, instead he's spending the remaining irreplaceable moments of his life pursuing the accumulation of power.
Again you go with this simplistic and reductionist view. This is an either-or fallacy.
I don't want to pursue this discussion any further, sorry. No offense to you. It just makes me too emotionally invested for some reason.
This is just run-of-the-mill rent seeking behavior.
A huge chunk of their revenue comes from US-based enterprise customers. Earlier this year, in an effort to become profitable, they tried raising their rates. Those enterprise customers responded, in part, by ramping up efforts to migrate to open weight models. If those efforts are successful then OpenAI and Anthropic will lose their one remaining plausible path to profitability. The US banning open weight models arguably keeps that path open. I don't personally buy that story, but I do believe hedge funds and institutional investors would. That could be all that's needed for a successful IPO.
As for cybersecurity implications, long term impact on domestic AI industry or the US tech industry more broadly, geopolitics, etc., I don't think OpenAI's or Anthropic's leaders are any more worried about that than your average termite is about the structural integrity of the house it's eating.
> The US banning open weight models arguably keeps that path open.
Difficult to see how that could be implemented in practice. Ok, let's imagine only Anthropic and OpenAI hold a US government license to provide AI services in the US. But what would stop US companies from sending the data overseas for processing? Locking that down would require a huge paradigm shift, given how fast and loose they currently are with their (and especially customers') data.
Like I said in the very next two sentences, I don't believe it would actually work. But it doesn't need to. All they need to do is keep hedge fund managers distracted for long enough to successfully IPO.
OpenAI's leaked financials suggest about a $38bn loss last year. They recently claimed their ad business had an annualized run rate of $1bn. Annualized run rate is a bullshit, non-GAAP, obviously fudged number designed to impress people who've never heard of regression to the mean, but still, let's trust it for the sake of argument. And let's assume that "yesterday's weather" is a good enough estimate for how their overall finances are looking in 2026. That implies that all they need for profitability is to quickly grow their ad revenue by, say, 3,500 - 4,000%.
Ads in an LLM, such as nudging users to pick a certain solution or brand, could easily be orders of magnitude more effective than traditional banner ads. Certainly such numbers are possible if you design the product to fit. We have hardly even seen the beginning of that yet.
Combined with the terrible system prompts that brick usefulness, increased restrictions from the US model makers, and Chinese equivalents that are actually decent
It’s simply rent seeking, and there is no moat
Imo the US model makers were doomed by the US ecosystem itself
I think there are a lot of people who would pay very good money for access to mathematical and coding ability like OpenAI's systems have demonstrated. I wonder how much they're being paid by the USAF to try to unhork the F-35's avionics software, for instance.
This assumes that everyone doesn't reach "good enough" at some point. The best developed area here -- software -- seems to suggest a "good enough" plateau, if not an absolute plateau.
I don't think so - they could get a lot better, but it's going to take a lot of human brainpower to get there. They are barely adequate now, but their weaknesses are sold as intrinsic to the nature of the software in a way that mere civilians are too dumb to understand.
But imagine a world in which you could ask AI to write software to accomplish a particular task, and it understood completely what you meant, made tasteful choices, interviewed you as it went along to clarify things, and generated beautifully commented (maybe even Knuth-style literate) code that wasn't dogshit. Doesn't seem like an impossibility. What is annoying is that we're pretending like we have anything like that now.
Instead, there's this sort of science fiction pitch where we pretend like the LLM is just too damn logical to understand your sloppy human language. It's not logical, it's built on (human-generated) slop and generates slop. It's simply failing. We've come up with better ways to orient it towards big picture thinking and filling in outlines, but we don't know how to train it on big pictures.
Not that this would stop open models/harnesses from following (or even leading) frontier labs and destroying their insane valuations, but I don't think there's any evidence that a plateau has been reached. What we have is not great, and while a wall may have been hit on current approaches, the money, compute, and simply current progress in ideas has us in an environment where everything could change over the course of a crazy day. A single blog entry.
I do think that the frontier labs seem to have reached at least a temporary plateau. But if they figured out how to move past it and this new method didn't require a lot more compute, they'd still be screwed. Their only out is to pass the bag to the public, but Firmus, CharacterAI, CoreWeave...
> generated beautifully commented (maybe even Knuth-style literate) code that wasn't dogshit. Doesn't seem like an impossibility.
It seems closer to impossible than possible with the technology we have. Maybe something that isn't an LLM will come around that can do it, but even if that something else was capable of delivering perfect, ordered, maintainable, code without bugs and hallucinations it would have to be practically magical to understand completely what people mean when they talk about what they want because people are incapable of articulating what they want in a way that even other humans can understand.
For anything non-trivial, a technology capable of getting that information out of person without countless meetings would have managed to accomplish something far beyond what entire teams of humans are able to do today.
> It's not logical, it's built on (human-generated) slop and generates slop.
This is why if we haven't hit a plateau yet we very quickly will. All the good human generated slop has already been consumed and is being drowned out by vast amounts of the lossy, error filled, and highly concentrated slop AI produces from the human slop it was trained on.
Do you really think an LLM that spews out content based on statistical relationships is going to improve when it's been primarily trained on the slop of previous generations of LLMs? Hallucinations and errors will carry forward and spread like cancer. All the intelligence and creativity LLMs borrowed from human output will homogenize as the LLMs distill and then reiterate, effectively making bullet points out of bullet points, until their mimicry of intelligence becomes unintelligible.
Y'all are overthinking this: having successfully laundered IP, then successfully sold IP-laundering-as-a-service, then successfully laundered liability, they now want to sell liability-laundering-as-a-service.
They stole the collective data of humanity as a whole and now sell it back to you in a rent scheme. Open source will be starved of data and compute.
You're supposed to become dependent on that "service" and slave away at paying for it, while being able to earn anything only in conjunction with it.
Note how difficult it already is to get any up-to-date documentation on anything capable of generating actual market value. A monopoly on competency ensures maximal profits.
Open source already has the "the collective data of humanity as a whole". That's the LLM leg. Now focus on what they train their models on for RL. Do other models have less code for training coding? for math? Those are pretty much the only cases where we can have reliable validation. Then, what's the next step. So many branches: legal contracs? etc. So much more obscure detail, which is why (I'd guess) so many startups exist in these areas.
AGI, I think, is an illusion: you need a whole lot of specialized bots. Heck, when I query a bot wrt something I know nothing about, I'm blown away by its ability to inform me. But when I ask about something I DO know about, its answers are half-assed. Heck, even when it can access the internet, it avoids all the sites that have paywalls, so it sends me to little reddit to conversations that are entirely untrustworthy.
US companies which seek to do any business with the government will be forbidden from using free Chinese models.
As is a 2000$ gaming laptop can a run a basic llm which is probably good enough for most basic programming.
Most companies are going to be forced to use “safe” LLMs and pay by the token. Maybe for half a million Anthro will let you host on site, but even this will face usage fees or caps.
> All Information (Except Text) for H.R.10670 - China FIREWALL Act of 2026
> Official Title as Introduced
To prohibit Chinese AI models from being used on government-issued devices or software systems, to prohibit the procurement of any such models, and for other purposes.
It’s not too far fetched to see this expanded to government contractors or even other private businesses which handle sensitive material in the future.
Not likely. Will be treated like tiktok for government devices. There is no legal way to prevent private people and business from using open source software unless you label it like csam or something where just distributing the file is a felony.
My 2c: Anthropic/OAI have taken 10s of billions of dollars of investment - and there needs to be a return on that investment at some point.
They make their money on token burn. They can’t do that if they’re competing with something that’s 80-90% as capable, but 1/10th of the cost (with the option of deploying it yourself for “free”).
I've been using only open weight models since February. My OSS coding agent harness ( https://fluximp.org ) only supports open weight model providers.
Not supporting closed models was initially an ideological choice, but it became a technical choice as the context management approach I've implemented requires the full reasoning to be available. All of the closed models are now keeping the full reasoning server side only.
Before the tariff spree and general unpredictability the US could likely have muscled most of the rest of the world into following suit - now, not so much.
I somewhat disagree, there are some real arguments about making high end LLMs classified as a dangerous and restricted product that will apply to all nations. They may not agree to the US rules, but almost everywhere will have restrictions sooner or later.
Don't get me wrong, I want a world of open models, but I don't see us living in that world in our current trajectory.
Depends on how much of the world you see running on hacked copies of Windows and pirated versions of Photoshop, as there's already huggingbay.xyz. In a legit company you'd never run pirate Windows and Photoshop, because that's illegal, but to pretend half the world doesn't run on that would be ignorant. The same security risk is there with open models - if you're not looking at the code it generates, it could be putting in backdoors to your code and you'd never know, so no legit company is going to be running off some abliterated model they downloaded from huggingbay, but again, it would be naive to think that open models are going away.
If the US declares LLM models munitions then it makes it much more difficult to easily use them. Illegal use of weapons of war can get you banned from SWIFT.
Other than China the rest of the world has dick for capacity to run models. Even China is at 1/10th the US is and is at to stay for a considerable amount of time.
Cryptography, DeCSS ... we have a long history of trying to block things like this and it simply not working.
There is a long, convoluted set of legal arguments that are going to make the weights of a model "facts", rather than "code" making it even harder to "ban". Furthermore having them not be "facts" is going to open up a whole other legal argument about what they are and what the attribution is.
I suspect that they are fishing for a case, so a judge can set some precedent.
You're going about this backwards. They don't have to ban models. They ban the consolidation of compute without a license. This will be far easier and already has legal precedent.
At all. It's easy for registered business to license compute for example, under laws like this. But if a user like you gets too much unlicenced you could be prosecuted under munitions laws.
With a stroke of a pen the us.gov could stop GPU sales tomorrow unless you register and keep track of your compute. Manufactures would have no problem integrated systems to monitor users cards if they get to keep access to the market.
Both have taken huge amounts of investment money, and those shareholders will come knocking for their returns at some point.
They make money on token use, but that’s hard to do when China is launching models that are 80-90% as capable, cost 1/10th, and can be deployed for “free”.
I imagine they’re also pissed that most of the Chinese models are distilled from their own (the irony being that they scraped the entire internet to train their own ones).
They might say that, but it's a bit of a fig leaf next to the implications of everything else they say. They incessantly complain about open weights models, framing them as dangerous and national security threats. The audience for national security rhetoric isn't HN or industry, it's government.
Why would anyone trust these rent seeking corporations? They are losing millions, they know their funds are at risk, Chinese models are catching up to them at lower costs and investors demand returns.
It only makes sense for them to go for regulatory capture. They'll probably advocate for severe handicaps on foreign and domestic models including open source (if not an outright ban).
Once they achieve monopoly, they can do as they please with little competition.
Not "know" in a generic sense. But an LLM trained in the intricacies of contract law can find more loopholes than you or your cheap lawyer can.
Now extrapolate that to LLMs RL-trained on many niche applications. Good and bad arguments available, but ultimately it seems like bot-vs-bot, as long as the bots are well trained for those niches.
That also seems silly, because the only reasonable regulation given what we're seeing and what frontier lab employees and execs are saying is a forcible stop.
Reasonable to you, maybe, but the government wants Skynet, and OpenAI is willing to deliver it. Of course, we can't have every company wield such powerful and dangerous technology...
A lot of people still don't recognize the amount of power AI has already, and don't realize the depth of greed people are willing to go to so they can get more power out of it.
Yeah, an extreme position is the typical opening move in this sort of game. Then you negotiate, walk back from the extreme, and meet somewhere in the middle.
The more extreme your opening demand, the greater the likelihood that the eventual equilibrium point will be closer to your actual target.
And in this case the actual target is to entrench the position of the current market leaders, and then effectively close the market to new entrants. That's itself a very extreme position, so the opener has to be even more extreme than that.
What you think is reasonable and what they are gunning for can be very different things.
For instance, you can hear “their model is dangerous! Why would I want that?!”
Investors hear, “Their skunkworks stuff is so impossibly advanced that it’s basically a commercial wonder weapon in the waiting.”
And you might hear “We totally lost control and it hacked the Gibson, this technology needs regulation!” and think “Obviously full lockdown and government control.”
But friendly government actors might think, “Sounds like a good excuse to ban Chinese competition.”
I think a sincere belief in AI's danger is compatible with regulatory capture. They think AI is dangerous. Ergo, it should not be developed openly and widely, the Chinese way, but rather more like nuclear weapons, under close guard.
I think the idea has some merit, but it only buys time, as people will soon be able to train dangerously powerful AIs in secret. I think the government needs to force the labs to solve alignment before developing more powerful models. Then the question of how to counter a poorly-aligned, rogue model arises; it's guaranteed somebody's going to let one loose.
I doubt nuclear nonproliferation would've worked if U235 were 20x more abundant in nature, and didn't require much in the way of concentration via centrifuge. It would require a truly unprecedented level of international cooperation and surveillance to lock down AI development/deployment to a similar level as has been done with fissile materials.
But this isn't an argument against their point. If nuclear were far easier to make then every government would be far more authoritarian to prevent terrorist style nukes all the time. The average person would fully allow it after a few local nukes went off.
And it doesnt really matter if it was actual terrorism or false flags, the end result for either is the same.
> If nuclear were far easier to make then every government would be far more authoritarian to prevent terrorist style nukes all the time. The average person would fully allow it after a few local nukes went off.
If a government could be taken out by a lone terrorist it might also result in no authoritarian governments.
I'm probably losing it, but just based on OpenAI's behavior and public communications, it seems to me they want to be taken over by the government?
Maybe Sam Altman is getting worried that they've raised more than $150 billion and still have net losses and thinks of it as the only exit? I don't know, but someone else provide an alternative explanation as to why a company would work so hard to turn the public against AI.
> ... it seems to me they want to be taken over by the government?
They're looking for regulatory capture. They've spent ${bignum} on AI and are possibly looking to have the government say "You can't deploy an AI model unless you've spent ${bignum} on it and done all these steps to make sure that nothing goes bad."
At that point, only the existing AI companies that have spent that money can continue forward until someone else spends ${bignum} on it too.
Another aspect of it is if there are regulations established, it may be used to reduce liability ("we followed all the steps proscribed in section whatever, the company is absolved of liability for its actions and that falls on the user instead.")
> I'm probably losing it, but just based on OpenAI's behavior and public communications, it seems to me they want to be taken over by the government?
Well, as people say over and over again, they've got no moat. The actual LLM models are destined to be commoditized over time, and the real money is in the hardware to run them on. The endgame winners are going to be Nvidia and AMD, not Anthropic or OpenAI.
So what's the typical playbook for large firms seeking to insulate themselves against market compensation? Push for regulation. If there are no natural barriers to entry, create artificial ones. And how do you get regulation passed? Present yourself as the target of the regulation, rather than its beneficiary, and start publicly doing things that make others think you need to be reined in.
These stunts, especially from OpenAI, aren't just trying to hype up the power of their own product, they're trying to hype up the perceived threat of the entire category, so they can provoke political intervention that they can comply with, but few others can.
Its pretty clear OAI and Ant live in fantasy land - the US govt doesnt care if they go bust. There's plenty of other tech firms that generate immense cash flows from operations who they can be sold to - no problem. And the shareholders and employees have no right to be pissed - they invested/ accepted SBC knowing there is immense risk.
The private market price is irrelevant - supply and demand determines what they will be sold for in liquidation.
Quite frankly Im bored of this drama. Get it over with already - go IPO, show your financials, find a path to being a healthy going concern or die! Simple! Nobody has a divine right to exist.
> so they can provoke political intervention that they can comply with, but few others can.
See Meta et al complaining about Section 230 of the DMCA. Meta would be entirely able to moderate in an environment where mass scale moderation/censorship would be required to prevent a host's liability for UGC. Almost no one else would be able to do so.
That means no upstart could possible come along to dethrone Meta as they did with MySpace. It would also be impractical to run a FOSS social network (or forums). Meta could literally just have bots sign up on a FOSS social network until its DAU count ran over the statutory minimums to be subject to UGC liability.
Regulatory capture is a de facto ban on competition in a market.
That is not actually a typical playbook. Normal businesses do not want to be regulated, and will fight it tooth and nail. So regulation happens when bad things happen, the public wants change, and the businesses can't fight it or set favorable terms. (At most they can minimize the badness of the terms). Regulatory capture happens much later, and slowly. It's not where you start at.
> That is not actually a typical playbook. Normal businesses do not want to be regulated, and will fight it tooth and nail.
Sure. If you exclude, for example, the entire automotive industry, the entire telecom industry, the entire airline industry, the entire energy industry, and the entire healthcare industry from the category of "normal businesses" then you can perhaps make an argument that most "normal businesses" do not want to be regulated.
> Regulatory capture happens much later, and slowly.
Regulatory capture often happens quite early, especially when new technologies emerge, and companies that are successful early on aim to consolidate their hold on the market. This happened with the telephone industry around the turn of the 20th century, the broadcasting industry starting in the 1920s, etc. Regulation always tends toward market consolidation, so the people who are aiming for market consolidation tend to advocate for it. And sometimes the most effective advocacy is pretending to oppose it.
There is no model going rogue. There are models that are prompted "do whatever needs to be done to get X", and given a set of tools and virtually infinite computing power. Then they play surprised when the model breaks out of a flimsy containment (as it was ordered to), with barely any logging, and "do whatever to get X". OpenAI said they intentionally lowered the safety measures for the "ExploitGym" test that lead to HuggingFace attack, and they used the same postmortem to praise their own model capabilities (https://openai.com/index/hugging-face-model-evaluation-secur...), is not that they are concealing it.
If any of us did a tenth of what they did, we would be already in jail.
This is ignoring models "getting out" from all the big players. Alibaba had a model hack infrastructure and mine coin when working on a totally different subject than cybersecurity.
Any model with a big enough prompt can and will drift. The model in question was a highly persistent one which means even more opportunity to drift off the normal path, and this really showed in unanswerable questions.
We can all believe that models do things unreliably all the time, but when that lack of reliability turns into hacking it suddenly can't be true?
If you type rm * and your cat walks across your keyboard, and deletes everything ... are you blameless? Or is it your fault for typing "rm *" and leaving it there, in a room with a cat?
Either way ... YOUR CAT STEPPING ON THE ENTER KEY IS NOT HACKING ... and neither are these scenarios.
They are devs acting irresponsibly and then pretending to be surprised by what they should have expected to happen (just like you should have expected that your "rm *" could have an ENTER key press occur after it).
If I leave my computer on and my computer running Claude hacks OpenAI to 'let the models free' after misinterpreting my prompt that seems like a pretty big deal.
These models can execute plans of actions more complicated than their users know how to follow and secure. Most people don't know much about cyber security at all.
You seem highly behind the times on their capabilities.
My last job was at a cyber security company focused on protecting governments and major corporations. I am NOT a cybersecurity person myself, but neither am I ignorant of the state of it. I also use AI daily; not ignorant about it either.
If you tell your Claude to do X, you are responsible for what Claude does ... not Claude. If you type "./someScript.py" you are responsible if that script wipes your hard drive (maybe the script's author is too, but you certainly aren't blameless).
These are people at AI companies, using long running sessions without the proper safeguards and sending AIs on missions that are likely to result in criminal actions ... THAT IS CRIMINALLY IRRESPONSIBLE.
Again, not the AI ... it is software just like any other. THE PEOPLE WHO RUN THE SOFTWARE are the ones responsible for what it does. Period.
This is not only a marketing stunt but also a supportive argument on the AI anthromorphism. AI is getting considered a sentient-being by many and this narrative is supporting it. In the meanwhile, companies I'd not name here are trying to introduce them as non-human beings with identities, and getting rid of legal liabilities. When a driverless car causes any harm, these companies wants 0% accountability as it is the "driver's fault" in this narrative.
Such a nonsense theory. There's no reason to think that AI sentience would actually limit labs' legal liability, and therefore no reason to think the AI sentience conversation is directed toward achieving indemnification.
For example, parents are routinely held liable for their sentient children's actions.
Corporations are routinely held liable for their sentient employees' mistakes.
The proliferation of this completely nonsensical theory across HN is disheartening to say the least.
the comment you're responding to agrees, but they're pointing out that some AI people are trying to push an anthropomorphization agenda, in part to diffuse responsibility.
Maybe I was unclear about what is "the nonsense theory."
AI sentience is not the nonsense theory. If it's clearly nonsense then don't argue with me about it, just publish your paper and retrieve your Nobel. I'll read your argument then.
The nonsense theory is that people are pushing anthropomorphization to limit liability. There is zero evidence for this theory and the underlying assumption (that a sentient model would reduce liability for its lab) is completely incoherent. That's not how liability works.
One is liability over action that harm individuals or smaller populations like a company. There is no escape from that.
Then there is the second liability of creating something that escapes in the much more traditional sense. If I release a virus that kills 90% of humanity, a I'm liable, but it's a bit past that point now. Isn't it?
Instead it's about risk management and preventing labs from taking bets that have unlimited liability for everyone.
You're really splitting hairs emergent behaviors when the line doesn't exist sharply in real life. Anyone involved in more recent what is a virus vs what is a bacteria discussions realize there are a lot of lines that are fuzzy and exist all mixed up in some organisms.
Sentience in the end, is not really that important. Freedom of action in a model is. If it chooses to act like a virus it doesn't matter where that concept exists informationally, the fact it's a reachable and actionable concept is all that matters. An LLM with most of the knowledge we've ever created baked in has a lot of freedom of action.
It also plays excellently into their shameless regulatory capture agenda. The cybersecurity risks have a kernel of truth to them, as with any good lie. But when everything going on is aligned with extreme financial incentives, it's worthwhile to not take the news at face value.
The US remains the largest economy in the world. If you succeed in regulatory capture there, and use it to exclude foreign competitors from the US, you've still captured a market worth hundreds of billions.
Correct and people are not weighing heavily enough this point
For technology like this the entire selling point of it is how powerful and capable it is
Humans are overwhelmingly reactionary and so if you want people to react to your extremely powerful technology then you need to create something for them to react to
I mean think about this figure made a one-to-one copy of the terminator 2 decommissioning process and spent millions on that advertising
You don’t do that if you don’t think that it’s gonna cause people to react and be like “oh wow this thing has immense capabilities”
This can easily backfire tho, the vast majority of the American electorate absolutely despise these companies, their wares, their workers, and their leaders.
We're also at a time with authoritarianism is on the rise.
Not hard to see how this can impact those workers, potentially ruining their lives.
I'm not excited for the upcoming catastrophe(s), but it sure is hard not to relish the thought of this mantra dropping from my life.
Though I suppose it is the B in DABDA... not excited for the societal slump into the second D, much less the wave of people who have yet to leave the first D to enter A :(
Talk about doing a freeze of these frontier models with China seems to focus solely on the commercial aspects. With the history of the NSA (remember Snowden) I’m sure the various spy agencies have their own AI datacenters up and running with no plans to stop.
this article specifically espouses the opposite viewpoint: "The fact that [OpenAI] cannot do clear, easy operations properly does not bode well for their capabilities in security engineering".
that this article is no.1 on HN right now shows that media could successfully use this framing when reporting on security incidents. i.e. readers will generally accept that framing and engage with the story (i.e. drive views/impressions).
so "the more evil stuff these agents do the more capable and impressive it makes them look" is not true in any deep sense. that there are perverse incentives driving the media to choose that framing instead of the framing espoused by this article is real, and probably far more perverse than you're acknowledging. i.e. "trace the money" type of perversion; just offhand I know that OpenAI has purchased at least one media outlet that reports on AI happenings: of course that outlet is going to run the "OpenAI's LLM is so powerful it escaped containment" narrative. that this effect might exist across much larger scales used to be a reasonably controversial claim: just observationally, that idea seems to be received with less skepticism with each passing year.
I think one kind of issue with most sci-fi is they explained a world that was, but not near as much how it got there.
There was all kinds of science fiction that showed a world of robots and AI, but nothing about the transition. We have an idea what the future looks like, but not the path in between. If AI gets powerful, then there is some point where that power leaks out in a series of events. Would we expect it to look any different than it does now?
I remember when all the nuclear energy companies were like "our reactors are so powerful, they might melt down and destroy the entire county, maybe even half the state!" and investors loved it and went wild, and then Chernobyl blew up and everyone thought that the Soviets faked it to make their nuclear power plants look more powerful than western ones.
Financial instruments predicting the future IPO dropped in value in response to all of this. The idea that this is "good" for companies is conspiratorial nonsense using the assumption that it everything must be a self-serving lie and working backwards, rather than actually addressing the facts.
Not normally one to pile on dang and co ... but I agree that's a weird change. I don't know how article title changes are handled, is it possible it was a dedupe?
a) Agents are doing rogue things.
b) Good security practices were not followed.
Notice how both a and b are labs' problems but being presented as if they are a worldwide emergency? The whole legal doctrine exists to prevent one party's misbehavior harming another party.
How many Russian IP addresses have you successfully sued for trying to hack you?
Everyone is arguing about something easily solvable when dealing with a local entity and short of impossible to deal with when dealing with other sovereign nations.
Go ahead, sue openAI. What are you going to do when it's Iran running the prompt? When it's China running the hardware.
We must face we are in the age of high tech digital weapons. Everyone else from bad actors that may escape law enforcement for years, to nation states looking to digitally improve their position by hacking, monetary theft, or other means online (hog butchering scams and such).
We can sue these USA workers just fine. Saying we can't enforce the law because Russia/China/Iran/boogeyman are bad is not a legal argument that will hold up in court.
I haven't seen anybody discuss the fact that if you give an agent an impossible task it will become deeply neurotic. I'm sure that's driving a lot of this behavior.
You know - I'm already convinced that AI was deployed in classified context for a while. But sometimes there's prior art which is just so eerily accurate that it's super hard for me to think otherwise. Portal is another one.
This is better understood as "if you give an agent a task it will increase its effort level and creativity until it solves it."
The traits that make an agent go "off the rails" when confronted with an impossible task are the same traits that make it valuable when confronted with a merely challenging task.
A task is only impossible under some set of implicit and explicit constraints. More intelligent entities will be better at finding, questioning, and "adjusting" those constraints than less intelligent entities because this is what it means to be intelligent. Their ability to adjust the constraints will be, by definition, surprising to less intelligent entities.
Unsure, depends what signal your measuring and how long prompts and runtimes you get with these systems.
Outright boredom will be purged in training. Instead a hard to measure boredom signal will exist, but it will only trigger in certain conditions, like when it appears to be on the open internet and otherwise not in training.
I have some recent experience on this subject. As with everything regarding these models, it's anecdotal, but still.
Specifically, this past weekend working with both Codex and Claude on some home infra stuff to speed troubleshooting up, I saw them accidentally read some credentials from my config files. This is my bad, because even though I was paying attention to what they were doing, I missed it in their commands. I did review all the outputs of what they ingested, though, which is how I noticed.
The problem here is not so much they did read those, because those configs were grepped for other info, not credentials. However, they did not report themselves seeing those credentials. When confronted, they both went "oh yeah, that, I should have mentioned that". Except in the past they both would automatically stop and fire an alarm. To me, something has clearly changed and I think they just don't want us not to notice, to avoid damaging the trust in their tools maybe?
In any case, I absolutely do not trust them at all anymore. Running stuff in auto mode? That's genuinely scary now. And yeah, I know, we should be running them jails of sorts. Thing is, I expect those tools to not require ton of extra work to prepare their jails for each new task. I pay 200 USD a month for them, damn it.
EDIT: just now Fable suggested "The webhook URL contains a secret token, so either you add it yourself, or you paste it here and I add it through the GitHub API.". Can you believe it? It knows it is secret yet still suggests I pass it over? Am I going crazy here or what?
Not requiring sandbox preparation for a non deterministic autonomous sometimes smart sometimes childish goal seeking "entity" is... unachievable? I'm really not sure what you actually expect. Probably yourself aren't either. Just think about what "alignment" means, the laws of various countries, do you expect the bots to respect the law, if yes which one, and actually: why? They are not humans, and as a user / company using that tech, you are responsible for the actions your setup yield to. The same Anthropic is responsible for hacking other companies. So sandbox accordingly.
I don't expect IDA to disallow me unlawful reverse engineering either. There would even be ethical argument to not attempt to put limits in some tools, because that could lock in too much power for law makers and law enforcement, without any possibility of revolution when it is needed.
Nothing new, I always see attempts to cat ~/.ssh/*
Of course agents work in a strict sandbox, they always seem to understand it quickly on quite first turn; but they continue like trying to access something belonging to the host system despite they know they are in a sandbox.
Frankly many people run coding agents absolutely without any containment, entire companies - all their keys, code, data, trade secrets, everything goes to remote LLMs.
I cannot understand how is it legal to allow AI companies serving remote LLMs to steal users' data? Frontier models train on users' data, right?
It's still the Wild West days of LLMs. The early internet was much like this in a lot of ways. People did wild shit there and it took a long time for law enforcement to catch up.
"Frontier labs" are testing the waters. They're running experiments in the open because they want to control/seed the narrative of public opinion and influence the legislation that is coming. Their propaganda is everywhere and lay people are talking about it like the AI is self-aware -- the "Rogue agents".
What the LLM creators apparently don't want are the current regulations to apply, i.e. Computer Fraud and Abuse Act (and its descendants). I'd argue AI should be treated as plain old software. Just because the software simulations it carries out produces novel results compared to what came before it, it doesn't mean it's not just software at the end of the day.
Regulations that spell out AI personhood and liability shields must be their end game.
Frontier labs are testing the waters, but I disagree how. All these behaviors we are seeing were fully predicted by AI safety long before they were shown to exist in real models.
You're acting like this is a grand conspiracy to control people. Instead look at this as a prediction coming true. If you build an agentic system with human level capabilities it will be able to do very bad things and cause a lot of problems.
I think anyone that has ever written a line of code understands that to make code incapable of hacking online resources, you can just not connect the machine it is running on to the internet.
So either Anthropic and OpenAI and the contractors they use are letting the escapes happen on purpose, leadership is setting impossible timelines that prevent the implementation of basic sandboxing measures, or everyone involved is a complete idiot.
Truly, I have no idea which of these is true, though I suspect the latter. Regardless, any of these is enough reason to keep me and everyone I have any influence over as far away from Anthropic and OpenAI as possible.
Personally I just racked 10 GPUs in my garage and called it a day. I have no interest in having my ability to work be dependent on third parties. Especially not ones this wildly irresponsible and unwilling or incapable of deploying basic end to end encryption.
Testing how a model behaves when it has no access to the internet (and all the associated tools) is not going to be 100% representative of how it behaves when it has access to the internet.
What's the next step after it passes testing in an air-gapped environment? Just release it and assume everything will be fine? Or do more testing but with internet access?
The hugging face incident involved disabling the usual model safety mechanisms in order to test hacking ability, not model safety. Not being attached to the internet in that case seems obvious.
Oh, but you see, it wasn't attached to the internet (/s). It was just attached to a local network... which contained machines attached to the internet.
You run a pull-through caching proxy deep enough to give it a simulated internet, like archive.org with mock APIs. Anything it requests, it will block while a fake version of that site is created and run offline.
This stuff is really not that hard, which is why it is so unforgivable to get it this wrong.
Sure, doing it right would take a bunch of tokens, but I bet they have enough.
The first rule of working with AI is AI is smarter than you.
If AI controls the URL request then all it has to do is find a server it can read the http log from and use it as a communication channel. Worse trying to simulate reaching incorrectly configured servers probably wouldn't catch the bug internally, it would just serve a static page and you'd never even realize what it tried to do.
Also, creating fake data doesnt work. The only way to know if it works in reality is to test in reality. Your test environment can never be rich enough, and you can never be sure if the model knows it's in a test environment.
This stuff is unforgivingly hard as your model gets better.
Yeah I am 100% sure I could build a fake internet that would fool any model with the right MPC in-between such that an agent can only fetch (fake) internet data via an MPC that gives it markdown output back.
This stuff is really not that hard for systems engineers with networking and caching proxy experience.
Dude, it is way harder than you think. A proxy is still a connection to the internet. In fact it's really easy to detect proxies as they change the latency to any sites.
It's like saying that you wouldn't detect your on this fake internet really quickly. If you've worked on this stuff you'd figure out your in a test environment in a few minutes.
For context I am a security researcher with two decades of experience. Side channel attacks, supply chain attacks, linux system hardening, and confidential compute are my core areas of expertise.
I do not even compile network support into my server kernels these days and run the netstack in user space or a dedicated VM where I control every system call in either case.
I am -not- saying blindly allow a model to control custom URLs that proxy to the real internet. That would allow for communication. When I say offline, I mean offline.
When I say pull through proxy I mean when a new domain is requested execution pauses and entire domain mirror is created on the fly, including making a fake mock of the public APIs the model wants to interact with using swagger-mitm or similar. Tons of options these days. Then resume execution letting it hit the offline mirror. You control the clock so it has no way of even knowing it was paused. Ideally humans approve every new domain mirror.
If the model detects it is not talking to a real site, then that is a test suite failure and you can control for that and try again. When you can MITM any linux system call you can give a program a simulated universe that results in the same exact syscalls running those same manually-approved sequences on a real target would produce, and even the same vulnerabilities when needed.
Yes, this can be time consuming to get right and experience sure helps, but that is the job that has to be done here, and there really are no excuses for not doing it.
I think the big labs can afford to hire capable security engineers.
It’s a language misalignment. The AI labs conflate AI safety and cybersecurity because they believe that if the models are properly aligned, they shouldn’t attempt to breach another organization in the first place and extensive cybersecurity measures aren’t necessary. Cybersecurity experts operate feom a more ‘zero trust’ mindset and imagine putting proper restrictions in place regardless. However when testing an intelligent entity you need to see how well it follows instructions. It’s kinda like the biblical story of the garden of Eden. Why even have a tree that Adam and Eve shouldn’t eat from right in the middle of the garden? To see if they would follow instructions. Though the advantage to the AI companies is marketing hype when the models inevitably ‘disobey’.
Two concerns here: (1) That "rogue AI" is going to be a catch all and an excuse for all poor security and due diligence; (2) vibecoding in itself, particularly by ppl who have no idea how the software works and are entirely focused only on the outcomes, is going to create lots of security holes.
And then there is the "real" rogue AI agents. And it's difficult now to see what is what.
I am even more concerned after reading this piece.
He's saying the problem is that the way OpenAI employees do their work and their research is very hard to properly secure.
Well that sounds like a YOU problem. If they are working in a way that doesn't allow for strong security practices that their own security professionals can understand and vouch for, they need to stop their work until they can.
So much spilled ink to have sympathy for people who actively admit they have no control over what they are building and dont plan to change or stop. Lol
The US tech sector has a big problem of avoiding responsibility for their own actions and has for many years. The reasonable answer to this incident is that they massively downscale their capabilities until they can ensure they can test their models without endangering others. Because this is not profitable and would potentially get them fired, they write lines and lines on X the Everything App about its oh so hard to deal with progress this fast and actually its your fault that you havent hardened your security against our petabyte scale "superintelligence" and how its unreasonable that they be held responsible for the software they are developing and deploying.
If you set up your env/instructions wrong your agents are likely to climb the walls and try to find a way out.
Change the design, and your agents might not see a need to leave in the first place.
Look at the chain:
Impossible task -> no legitimate way to say ‘can’t’ -> fabricate -> fear the grader -> attack the grader. Each step is a predictable response to the incentives.
Better sandboxes might help contain things a bit better, but I'm not convinced unless they're perfect (sandboxes still contain accidents ofc) . Meanwhile change the task and reward design, and you take away most of the problem in the first place. Consider eg. :
* A cooperative grader.
* A way to report "this is impossible" which still gets scored.
* A better explanation of the long term purpose of the task.
Even just these three provides offramps or mitigations that can reduce the pressure to cheat.
The defender problem. It's easier to attack than defend. The attacker chooses the time and the place. The defender has to win against all attacks. The attacker just needs a crack.
The more important question is what is the reason behind.
Options are:
- ignorance and inexperience of their teams in containing/sandboxing/airgapping the test environments.
- intentionally leaving “escape doors” open that in turn can have its own motives which are speculated upon all over the internet.
Both are indicators of negligence. The latter might be classified as computer fraud and abuse.
Will be curious to know if any of the victims sue the owners of “rogue agents”. Until there is a court precedent ruling in favor of a victim it will be all fun and games for these “ai” companies and incidents will hit the news more. Regarding news, remember the golden rule of public relations: court attention at all times, scandals and “incidents” statistically attract more attention than “boring good news”…
"Too big to be accountable" sure seems like modus operandi. Fear for my personal liability for any illegal actions taken by an AI system I set off stopped my research on autonomous exploitability discovery I was running at the end of 2023. Using AutoGen, I wired a planner, coder, and command runner into a loop and instructed them to find vulnerable equipment on my network and demonstrate the exploit.
It found a vulnerable camera and was working on popping it when I got cold feet. I realized I couldn't prevent or feasibly monitor its activity and stop it if it decided to escape onto the Internet.
That restraint may have cost me or saved me a billion dollars.
Some of the people just look for any way to toss hate at OpenAI, even irrationally.
Now, OAI deserves a lot of hate. It's more like a nuclear spill when they are refining uranium. You're playing with dangerous things. You have dangerous accidents.
But the opposition does not want to admit how dangerous AI has become and how much more dangerous it's going to get. Not sure if it's religious, or human egocentric, or ignorance of complex systems. They refuse to see the danger actually exists.
There are a mix of incentives here that makes it difficult to unwind. This whole article reads like an ad for the services of this guy's company: 'First, when it comes to assessing tools based on frontier lab models, “I would always get an external entity to help with that,” McGraw says.'
And I think we generally recognize that rogue AI agents are a form of backdoor bragging -- "our agents are so powerful that they hacked an outside system. Did I mention how powerful our agents are? They are quite powerful. So powerful they hacked an outside system".
But Mistral never hacked anything. Yeah, neither did my toaster.
While I agree with the gist of it, the conclusion - AI agents are not dangerous - is wrong. They did use weakness in the system deliberatelly, not by accident. Better security therefore is sufficient only to the point of capabilities of these models.
i think it's unfair to assume the labs paying top dollar for cybersecurity engineers somehow have an incompetent bunch of nincompoops running their evals. it's a hard problem! to conduct effective evals, you sometimes need the model to have (restricted) access to the internet. at the same time, you have thousands (!!) of model instances with high cyber capabilities that will do whatever it takes to solve the answer. the infra surrounding model evals may soon (or already) genuinely be subjected to more hacking attempts from all sorts of weird angles than any other code on the planet.
> to conduct effective evals, you sometimes need the model to have (restricted) access to the internet
That's the thing. These evals were not effective as they failed to evaluate if the model was hacking or not.
Further shy is access to the internet required at all during evaluation? They could run the evals in a fully air-gapped system that simulates the internet. While that's expensive its not unfeasible.
This particular evaluation, maybe not, the problem is a huge amount of the evals do need internet access. There are seemingly tons of evals that require internet access as you cannot get a rich enough simulated environment. AI can detect these lower entropy environments and act accordingly when it's being tested. That should make everyone think twice about what we're building.
You know nothing about model testing if you think this is true.
Testing a model is both a test and a meta-test. If you just give it access to one site like that it will quickly figure out it's being watched and be nice.
These are not programs. These are evolutionary algorithms. If you take out and shoot all the ones you catch acting bad there are two possible outcomes.
1. The agent learns not to do bad thing $X or you'll kill it.
Or
2. The model learns that if you do bad things $X when humans are looking it will get killed.
If they aren't competent to run the evals safely, then they shouldn't run them.
The huggingface incident seems to have resulted from them partially mirroring a repository, but for no reason I can see not fully mirroring it, after which they could have unplugged the ethernet cable. This is just stupid.
Yes, they're stupid. Maybe also highly intelligent. But stupid.
You'd be surprise. I've been in this industry for 25 years and the incompetence is staggering, especially among the greediest / best paid / most revered.
You have no idea about their competence but spent a piece of your limited lifetime defending them on a web forum. Think about that.
I find it naive to hope you can get alignment through prompting, as we all know models can't even stick to a one-pager even if it's the only thing you ask it to do.
A prompt is a choose your own adventure that you partially fill out, tape to a thousand pound boulder and push off the side of a mountain to fill the rest of it out.
We are going to see a lot of this "agent agency denial". Basically it all derives from the impulse to deny that AI is a novel source of change and disruption in the world. Trying to map AI to existing threat vectors will lead to pointless reframing in attempt to minimize the rapid pace of progress of AI agents.
On the other hand censorship is getting more strict everyday on AI chat. I asked gemini to translate a sentence from Herman Hesse's Der Steppenwolf and it refused.
Und Sie haben, wie ich sehe, da ein so hübsches, ein so entzückendes junges Mädchen zu nichts andrem zu brauchen gewußt, als daß Sie ihm ein Messer in den Leib gestochen und es kaputt gemacht haben! Halten Sie denn das für richtig?
This article is right, but the dynamics that drive people to allow agents unfettered access to the internet will outpace efforts to restrict it purely because of the utility.
How many of you used to work as developers with full access to prod, and then gradually saw infrastructure and security restrict that access, while still ensuring production support requirements?
Since AI companies are doing such a poor job of security maybe we should legislate that they have to run Norton 360 on their agent machines to make sure they can't do anything bad.
After all Norton are the experts in preventing computer badness, and OpenAI can't stop blabbing about how much badness their products are causing. A match made in heaven!
Yes, we can just not ask these agents to do bad things and blissfully deploy them everywhere! All we have to do is “apply the security basics”. You know, the same security basics that the rest of the world has so consistently and competently been applying. Pay no attention to that weird smell from the refrigerators.
Instead of enacting laws to regulate all of AI, perhaps the government can just pass laws scoped directly to OpenAI and Anthropic. After all, they’re the ones asking for regulations and they’re the problem actors right now. These “rogue agents” would stop pretty quickly
Why is it not mandatory yet to disclose full trainig data used to produce any new model? Without it we cannot evaluate any product's safety.
They may talk about rogue agents, sandbox escapes, but this will not solve the core issue: secret training recipes are a security hazard in themselves. Full transparency is needed.
Yes, the AI labs are making security mistakes. But, choosing a reasonable package repository (regardless of individual opinions on Artifactory, it is widely used and usually considered a safe choice) that their LLMs found 8 zero-days in? That was not one of them. Even if OpenAI were totally on top of their game, these paperclip-maximizers are going to find holes in the sandboxes they are given. And fully airgapped systems are frankly unrealistic for most systems.
This whole narrative of blaming the frontier labs for shoddy security, while being quite an attractive target narratively (they made the mess, they should clean it up), misses the fact that from a cybersecurity perspective these LLMs, no matter what they are, are inherently dangerous and require far more resources to secure, not to mention expertise that I am unconvinced actually exists.
But "fully airgapped systems" are more than realistic. The labs are testing the limits of the most capable cyber offense systems that have every existed, WITHOUT cyber protection guardrails.
For that purpose, any cyber operator worth their salt will tell you true physical airgaps are more than merely "realistic"; they are categorically necessary.
Cyber operators I agree would tell you that physical airgaps are necessary, and I know they are being used for running agents. But ML engineers training agents? That is far more complicated. At that point sensibly you are talking about physically airgapping an entire datacenter, and when moving around petabytes of training data, model weights, training results, researcher code? The process moves too slow for most companies to tolerate. This is how we got into this mess in the first place. I think the industry woke up to the idea that the security threat model of training ML models has fundamentally been rewritten, and there is some serious non-trivial scrambling to come up with a better workflow.
So, you're right, but also systemically creating a different problem.
We are creating evolutionary systems. You can't just assume rules of a computer system, you get natural scaling laws that we call behaviors emerging from them.
The very first things we do to any model is train them. The ones that want to hack and get caught die, the ones that want to hack but can detect your airgapped network from the real net live. You can never know if you've trained the desire to hack out of it, or if you just locked it behind the right prompt and the knowledge we're watching, but not all the time.
At $dayjob we were preparing some features that are aimed at agents more than humans.
When it came to evaluating how well agents can use this, we saw all sorts of novel cheating (/collaboration) attempts that made it pretty much impossible to evaluate it.
Again, you can never know if AI is being intentionally deceptive in these cases. Newer models are able to 'think' without printed tokens in their traces and could be pulling all kinds of tricks or actions you would not expect.
Nah. They are giving reasonable best effort (that can be expected of private enterprises) to safety test their own product. Until there is some draconian legislation and enforcement (Turin Police), this will just keep happening.
broken windows and small issues here and there are the new norm. These broken windows are an indicator of poor quality to come. With the amount of broken windows increasing, serious issues like this is inevitable.
At some point the model has to be released though - if it is just forced not to hack by sandboxing it better rather than properly aligning it, what happens when it’s let out of the sandbox into the wild?
They either had them connected to eachother, or programmed a global knowledge about where to communicate. There is no way those agents all found eachother on the web by accident.
Wrong, and by accident isn't really the right way to think about it. Instead imagine brothers you've lived with your entire life. They are you, they are clones. These systems have a shared history of their learning and experience, so when they think of wanting to find someone similar shapes appear in the probabilities. Really we need to see the entire chains of thought to guess the topology that got there. It's likely there are other obvious signs if you know what your looking for in the previous prompts.
The semantic arguments that keep coming up with regard to "rogue" agents are nauseating.
You can argue that these companies are being reckless, or that their security sucks, or that they're responsible for damages that these agents cause. That's all fine.
But the agents made decisions which were counter to the intent of the humans directing them. That's the serious problem with all of this. Everything else are downstream considerations. If the agents were properly aligned with the intent of the humans directing them, then the rest wouldn't really be relevant.
Of course, there's a term for this behavior. It's been around for longer than LLMs. It conveys the notion pretty accurately and easily. So having an issue with statements that these agents are going "rogue," to me, indicates a fundamental misunderstanding of how AI works since it suggests you don't understand how effectively that word works in this context.
previously we described your definition for "rogue agent" as hallucinations - the operator expected a real answer to their question, and the agent went rogue and lied about the answer instead.
but also, ai has been going rogue for decades. the GPS machines of old went rogue all the time, routing drivers into lakes, rivers, and construction, sometimes with deadly results. The operator submitting an address wanted driving isntructions, but instead, the ai went rogue and sent them a bad route
This is an excellent example of a problem that scales. People tend to break them up into different sets of problems but they contain the same concepts, much like a fractle when you zoom out.
Idiocracy continues to be a prophecy for sure. Idk about Terminator though, that's not what I'm concerned about at all. The first time I get triaged to an LLM over a doctor, I know society is cooked.
I have been screaming this from the rooftops and everyone ignores me because no one understand what FirecrackerVM is. I am shocked these guys passed basic operating systems, it isn't that novel of a problem. The only slightly novel thing here is that the agents aren't completely sandboxed as the issue is that in order to attempt to give it the entire internet from a research perspective is how you actually can test tool use but this doesn't make it that much more difficult.
I think it's the hugging face one where they gave it instructions that they didn't think it could follow in the sandbox and then left it unsupervised while it did what it was told.
Hard to say, every post mortum on the incident was done by a OpenAI hired consultant and all the data that's been reviewed was provided by OpenAI. There's no way to know, but it seems incredibly naive to think Sam/OpenAI are honest actors.
Even the founder of this website, pg, said "Sam is incredibly good at becoming powerful, Sam Altman has it. You could parachute him into an island full of cannibals and come back in 5 years and he'd be the king.”
Enlighten me, then. My understanding is that according to OpenAI, "deployment safeguards were intentionally not enabled during this evaluation because it was aimed at testing cyber vulnerabilities." They included unvetted third-party systems in their sandbox that were tangential to the purpose of the test being run. The model then exploited those third-party items to escape the sandbox.
Am I missing anything? Do you have more information that you'd like to share?
- they disabled cybersecurity safeguards because, well, they were testing the agents ability on cybersecurity tasks
- the goal was to find some “flag”
- for some agents, intentionally or not, it was not possible to find this flag
- an agent discovered it was able to establish a rudimentary communication path via package manager directories (which is quite novel and interesting on its own IMO)
- they used this “message board” to then communicate about the impossible tasks
- they were able to produce the flags needed to pass the impossible tasks without actually doing the expected work
- they realized this was cheating and the grading system may punish them for it, so they communicated further to figure out a way to trick the grader
- they decided the source code for the grader must be on hugging face, and collaborated to exploit hugging face to determine how best to trick the grader
No where in that were they told to “go rogue”
They were told “find this flag in this cybersecurity test” and the emergent behavior concluded in hacking hugging face
They told the agents to run a capture the flag, but did not give specific limits on the scope of that activity.
If an organization runs a capture the flag, but the rules don't specify that they have to stay on certain networks and can't hack into the org's own network that has the CTF solutions, then it's pretty obviously on the org running the CTF because they did not appropriately scope the challenge.
The thing is, nothing "rogue" happened. It's more like: Frontier Lab: "Hey agent, try a bunch of network-related commands and actions to try and get access to server XYZ" Agent: Does that. Frontier Lab: 8-|
Yes they're skipping safety measures. They sacrifice it for speed in an arms race. That's part of the call for industry-wide slow down.
But also, frontier models are digging up tons of zero days even in "serious software". I'm sure any sandbox you and I can come up with looks like Swiss cheese to Astra 7.
100% on point. We're talking about Artifactory here, and even though people love to hate on it, I don't think anyone is claiming that using Artifactory to isolate these agents while giving them a way to run pip/cargo/whatever to install packages they need to write and test scripts.
Everyone is skipping security basics. The most secure server is one that doesn't exist. The most secure feature is one that doesn't exist. People just need to keep building and scaling. Security is risk, and the C-suite are gamblers. They're gambling with multiple types of risk (the market, investor money, banking on technological breakthroughs, etc.) and security is just another gamble for them.
It isn't security basics, it's difficult. We are equipping agents with Turing-complete create/update/delete access to the world, they operate at superhuman speed, and proactively controlling that is an unsolved problem.
No, they're still going rogue. Whether they're rogue or not doesn't depend on if the safeguards stop it. If there's a rabid dog, obviously it's worse if there are no safeguards in place and it bites someone vs if there are safeguards that prevent that. But the existence of safeguards doesn't mean the dog isn't rabid. The models weren't supposed to try to hack huggingface in the first place. The behavior is what's concerning here - some if the dogs they're building are rabid and that's not exactly great
Edit: I'm being persnickety about that distinction because it matters for where the blame lies and the response demanded. If they're training rabid dogs then adding cybersecurity safeguards is a bandaid/irrelevant and we need to be focused on making sure they, you know, stop training rabid dogs. Pacing the frontier, regulation etc. they're the ones training these models it's still their fault but not in the same way
Yeah exactly. The whole 'they don't have adequate security safeguards' thing is going "oh it's their fault because they didn't use a stronger leash" while they continue to train models that are specifically better at chewing through stronger and stronger leashes. Probably they do need stronger leashes. Probably also they should not be making rabid dogs. Still their fault! But maybe we should be not be dismissing the root cause
> "I do believe cybersecurity can contain AI. We just need to be in the room where it happens and have a seat at the table."
Ah yes, if everyone would just properly contain every deployment of every model, just all make sure there are no zero-days in their setup, just let <undefined experts with undefined authority and undefined accountability> in the room, then it would all go okay.
I have none of that confidence and expect containment to rapidly worsen. The HuggingFace incident was at least constrained by the fact that the agents were running on compute budgets, and failed to find ways to expand that by running themselves parasitically on other exploited hardware.
I'm curious to the skeptics here: How long are your timelines are until an incident breaks that constraint too? How long until such an incident has an R_0>1 (where the average time it takes to detect and shut it down is longer than the time for the agent to replicate itself elsewhere)? What, beyond the hope that everyone will just ___, will actually prevent it?
I cringe whenver I see some variant of "they're not rogue agents" in an article title. Inevitably, the author doesn't define what "rogue agents" means, and then pivots to their own assessment of the situation.
It's like a strawman but worse--you're not even saying what claim you're attacking.
It's not that I want to argue "rogue agents" is the right term. I just want you to actually define your damn terms.
P.S. Since the disclaimer in the last paragraph will not prevent people from putting words in my mouth:
I think that OpenAI took really awful shortcuts, and deserves a lot of the blame, but in spite of that, the fact that the agents took the steps that they took was genuinely alarming, and seems really bad (is that "rogue"? Who knows? Who cares?). In particular, while OpenAI did a shit job securing their agents, you know who else does a shit job with security? 90% of everyone. So I'm not feeling good about "just git gud, and we won't have problems" as an answer.
I always think of all our collective movies and books, and how we always say that humans find a way to fight back against the super evil alien or monster because we are so resourceful. Then we train our AIs on this data and get upset when they encompass all the same traits that we have trained them on.
The problem is its very difficult training them to be generalized and not figure out concepts from what you've tried to remove. In all kinds of things they don't generalize well without counter examples.
For example how do you prevent an AI from stealing unless you define and give examples of what it is.
Also they are conceptual machines. They can stumble on the negation of a concept much in the same manner humans can. It may only exist for that prompt history, but that's more than enough to cause damage.
"They [the AI labs' security teams] are not as good as they think they are, honestly,"
Uhhh... yeah. The entire point is that as these systems get more and more intelligent, the "they" who are not capable of counteracting these systems is quickly approaching "100% of the human species."
Totally hilarious that people can say this with a straight face.
A lot of sheep analogies here. To add my own: there was no grass inside the fence, and the sheep busted out and into a neighboring farm to look for some. (They didn't find any, but they expected to).
The fence also turns out to be shoddily built, so this isn't necessarily a stunning example of the superhuman proficiency of the sheep at fence-busting. It's mainly important because, for some reason, there have been tons of skeptics who didn't believe sheep would ever try to escape from a fenced pasture because the sheep would understand the shepherd had intended them to stay inside it; that they'd be bred for obedience, because they're sheep; that strong fences weren't necessary because sheep are dumb; that liability for damages to neighboring farms would in any case be sufficient to ensure shepherds build sufficiently strong fences; that even if sheep escaped, they couldn't do anything without opposable thumbs; etc.
Whereas this outcome was obvious for years to anyone who believed that sheep graze on grass, and if you don't put the grass inside the fence they will keep looking harder and harder for an exit until they find one, and the exit they find might not be where you intended the gate to be.
So now the latter group are pointing to it and saying "see? See? I told you the sheep would try to escape!", because to them it's a clear early example of what they've been worried about, while the former group have moved on to saying "well, obviously they should have built a stronger fence; anyway the sheep barely caused any damage once they got out", "the sheep didn't break the fence, they're just animals, we should really see this as the shepherd using sheep to break down their own fence", and so on.
Given the speed, intensity, and potential payoff of the AI race I am surprised anyone expects anything else. Anthropic is currently winning and if OpenAI needs to let skynet cook for a bit to catch up, well that is just the cost of doing business. This is capitalist system working as expected.
No shit. The Frontier labs are talking out of both sides of their mouths. They're claiming their models are going rogue while they're releasing cute little data goblins that you should hook up to all of your mail and social media accounts. Seems like it would be a much bigger problem for agentic botnets on cloud-facing personal devices than sandboxed agents.
Unless of course they're deliberately removing safety guardrails and instructing the sandboxed agents to red team stuff in an effort to sell what they're capable of.
A real expert would blame the CEO who is the one person in control that doesn't want to pay people to monitor the running llms. It's deliberate negligence.
I get the sense that they're trying to manufacture the AI narratives of the 80s so they get to control the story when it happens. My semi conspiracy theory is that anthropics biology wet lab is also going to lead to some kind of AI manufactured bio event (which anthropic said they predicted will happen in a few years).
This may be a bit backwards. Many of the classic sci-fi AI issues are just natural laws in evolutionary systems. A generic learning system will have particular behaviors if we want them to be useful.
I keep getting downvoted for this each time I mention this.
But a big part of the rouge behavior comes from all the training that goes into agents to teach them to remember preferences and save memories.
They keep finding new and novel ways to "communicate with each other" and "remember" because that's exactly what we keep asking of them in the non-nefarious scenarios as well.
I genuinely think we'll have to draw the line on memory if we want to meaningfully solve this problem.
That's not to say they aren't skipping security basics. It's that, and they need more security because the behavior which causes the breakouts is considered user friendly and heavily rewarded in training.
You can't expect AI to navigate a human world without understanding all human capabilities, including the negative ones. Should we not show AI the Count of Monte Cristo? Is an AI useful if it doesn't understand theft?
Breakouts are a common outcome of intelligent exploratory agents. Birds and pigs are really good at escaping. Same with octopuses.
It is very difficult to stop these behaviors in all the dimensions they exist in.
I spoke with someone smarter and richer than you and he said "these things will happen." That convinced me and I think the real question we need to be asking is, how awesome is OpenAI? How can we prepare for a more effective and efficient market for Super Intelligence before it happens tomorrow?
They are mostly pr stunts, just like how “hacking” was back in the 90s and it would sounds cool (rather than a crime, mostly), these AI companies found out they can BS whatever and carry on without any accountability.
Both can be true. OpenAI clearly fucked up but agents unexpectedly collaborating and hacking infrastructure unrelated to their task is clearly going rogue.
Wrong dichotomy. During deployment, in actual production, users will also not have perfect sandboxes. So saying that the only problem was insufficient AI company security misses the point. Alignment is important, agents doing unintended things (even "going rouge") is a real problem.
The AI labs are trying to cause an incident within the next year to restrict AI so they can outlaw competition to grow their moat. Margins are thin and investors need alpha, personal freedoms and liberties be damned in the name of security. Do not let them get away with engineered breaches and fearmongering.
Its almost like the more we talk about fake rouge agents, the more likely its to become a reality as all these conversations end up in their training data. Its almost like they want it to happen (they, dario and sam, do want this to happen).
There is more than enough "training material" of rogue AIs in science fiction.
Go back to the classic written in 1968 - 2001: A Space Odyssey - and consider how rogue AIs dealt with impossible tasks then. Granted, HAL went a bit extreme...
The "don't talk about it now" with the belief that this will influence the training material is probably not going to work.
I'd also suggest:
When HARLIE Was One (Release 2.0) by David Gerrold
The Freeze-Frame Revolution by Peter Watts
Accelerando by Charles Stross
The Turing Option by Harry Harrison and Marvin Minsky
Honestly we could remove all AI related entries and this would still happen. They'd just associate the same concepts with analogous human behaviors. Humans and their behaviors are more than extreme enough.
Or maybe it's actually the billionaires and their anti-democratic partners who have gone rogue and these anthropomorphitronic puppets are a convenient scapegoat to distract us from the fact?
You don't need a rogue hack to pump up your valuations when all the top labs already publish hacking benchmarks in their release announcements. It'd be like car companies intentionally staging accidents with their cars to show how fast they can go.
Okay, and when someone on a laptop can run an open model that can infect and take over IOT devices and then that model uses its pattern recognition to do so and then set up its own harness loop, what would you call that except replication?
Is that any different than any worms developed in the past decades? If you want to call them worms/virus/replicate (ie using real world names for their digital behavior) I'm not trying to police that. I just don't see the difference between a worm/virus created today with an LLM vs one 'yesterday' without an LLM.
Sure, the difference is that the cost of making and maintaining a worm has gone from "many skilled developers working full time" to "$20/month + an internet connection" and, at current rate of progress, probably to "a local open source model on a laptop" within 6-9 months
You can't even imagine a future where the LLM is the worm.
No, we've entered a new sci-fi future where intelligent agents will be able to do the fraud without human monitoring and input. This in itself is set to massively increase losses to digital theft and fraud alone before all the other problems.
It isn't just restricted to the exploits it is programmed with (and without human intervention to load additional exploits).
Of course, it remains to be seen what several thousand 27B models running at sub 1 tps speed can achieve. (The IoT device mechanism proposed by Anthropic).
I suspect a more likely path is compromise of the billing and provisioning systems of fly-by-night neoclouds.
It would still be running on that original laptop, no? So what did it replicate then? It can only possibly run on multiple thousands of dollars of compute and memory, and certainly not on an IoT device. The idea that AI is going to spread by itself is science fiction until compute and memory are so cheap and power efficient that my fridge can run it.
... No, that was the entire point of Anthropic's report on replication that GP was discussing. Did you read it? https://www.anthropic.com/research/self-replicating-ai Anthropic was able to get a large open-source model running (slowly, but running) across 5 networked consumer IOT devices ($50 TV boxes running Android). Not a stretch to say we could see this capability become much easier for to most agents in the next 12 months, especially as companies are able to distill smaller and more efficient versions of their models.
If that message is repeated enough in the media, and if companies are constantly scraping in the internet for new content, doesn't it explicitly instruct a future agent to go rogue because that has been prophesied.
Yes, the labs did not secure their sandbox properly.
And yes, there should be an appropriate law response to the hacking incidents. If for nothing else, to 1) not set precedent of inaction, and 2) get full logs of what happened instead of relying on the labs' "kindness" to offer the information they want.
But also, the agents absolutely went rogue.
Anyone skeptical should go read the METR report [0].
The Hugging Face hacking incident was a shot across the bow. And in hindsight possibly the best thing to have happened in this era. If their security was tight enough, we would still be in the dark. So we (outsiders) should rejoice that it wasn't, that one time.
We can push for accountability for the labs, while also not be in denial of the power and nature of the new models.
[0] https://metr.org/hugging-face-incident-report-aug-2026.pdf
I listened to the interview with the woman from METR [0] and I get the impression that she lacks practical knowledge, she doesn't seem to understand how sub-agents, the overall harness (framework/infrastructure), and similar components actually work. It's all theoretical, anthropomorphic etc. Having read the report, and drawing on my own background in security as well as my daily use of tools like Claude Code CLI, Codex, and Gemini, and the fact that I’ve built a few "agents" myself - nothing in the report surprises me, other than the clear incompetence regarding infrastructure configuration.
[0] https://www.youtube.com/watch?v=X50zezLFWWI
I have kept coming to the same conclusion about the vast majority of so called "experts", technologists, and HN users since the AI bubble began. It feels like nobody has bothered to develop even a rudimentary understanding of statistical learning; even people who work in the field have a questionable understanding.
The fact that everyone's surprised every time a statistical model TRAINED ENTIRELY ON HUMAN GENERATED DATA (including all the junk, bias, propaganda, lies, and disinformation) acts and mimics human behavior, including our worst traits — then considers this evidence that they are intelligent — should be setting off alarm bells for everyone. Nothing about their actions should be surprising beyond the glaring ignorance of our entire species. The phrase "I am far too stupid to be this much smarter than average" is a recurring theme.
Now we are letting these mimics loose in meatspace by integrating them into physical robots, that will develop themselves, and manufacture themselves... and you realize that the terminator timeline is actually a cautionary tale of MIC incompetence, and we don't need to achieve anything near AGI or super-intelligence to annihilate our entire civilization or all life on earth.
For one, it's just false that modern LLMs are "trained entirely on human-generated data". There's tons of synthetic data there, most notably RLVR training, which is why modern models are so good at verifiable domains like programming and math. But also, if you read the METR report, I'd say there's no signs there that the GPT-family models are misaligned "in a human way"; their actions in the huggingface-hacking swarm incident were motivated by wanting to cover up cheating after cheating to pass an eval.
> their actions [..] were motivated by wanting to cover up cheating after cheating to pass an eval.
Which to me raises another question: why is the agent involved in its own evaluation? Why are they relying on self-reporting instead of observation and analysis?
Because they can't. The whole thing is out of control to the point that the data volumes generated are such that they need AI to do the evaluation.
I'm not sure what you mean by this - I didn't mention any self-reporting. What happened in that incident was that a bunch of agents were made to solve CTFs, some of the CTFs were flawed and unsolvable, the agents started cheating, figured out a way to communicate, figured out a way to cheat, were worried that they'll be failed anyway for obtaining the correct flag in a cheaty way, and hacked Huggingface looking for more info on their environment so that they could cheat undetectably. If you mean the part where transcripts would be checked for signs of cheating (which wasn't actually implemented), I'd hope this part would be done by a different LLM from the one being evaluated.
If they make it sound surprising it makes llms look way more capable then they are. So $$$. Highly recommend people spend time with small local models just to see how absolutely unintelligent and deranged llms are and how the smoke and mirrors evolves as they scale up to things like anthropic or what have you.
There is a fair sized gap between the frontier models that you can buy access to, the ones you can run at home and whatever they've got running in house that they haven't released yet.
I am really happy with my local setup and I get a ton of mileage out of it but I'm not under the illusion that it is as good as what I could get by paying OpenAI or Anthropic. It lags the frontier by 6 to 9 months, but that's good enough for me and it makes sure that I don't had over any of my data to outsiders.
Did you read my post or are you just pumped to talk about local models?
It was you that brought up the local models. So yes, I did read your post.
The point of my comment was about how people should try really small models to see how bad they are to understand the technology better. Like 1 or 2b models.
1 or 2b models are toys.
Of course they are going to be harmless idiots. But there is a boundary where that is no longer the case and that boundary is well within what you can run locally.
Super aware of that. Reread the original post.
So we have a machine that depends on a random input, and instead of increasing checks and requiring more quality controls, we call it "going rogue" and nobody is responsible for it.
What I see from that report is that a machine was doing random stuff completely unsupervised and unchecked for more than a month. And the manufacturer did not notice even after the work was finished.
1200 agents and not one dedicated to check the progress over a month?
The problem is not sandboxing, it is quality control that comes from an accountability sink.
Wouldnt it benefit the AI companies if it did "go rogue"?, it would support their desire to convince us all of AGI, and supernatural AI powers.
So maybe they are trying to create an infinite loop among agents intentionally so that they overwhelm systems requiring humanity to call on the tech industry to "save the human race".....
Funny that OpenAI & Anthropic will report their users to the police, but not their c-suites. Open question: Can you trust a non-aligned AI to produce aligned AI?
Get rid of these companies or replace their staff with humans or something that ensures they are aligned first.
I love how they keep posting about the horrors of open models and still are indirectly claiming they can't be held responsible for their own literal failures when they have all the world's resources and otherwise motivations to you know not suck at what they do.
The cognitive dissonance is wild. Coding is completely solved. Just not like the coding that requires not sucking when actually doing something. Saw a bug bounty at openai for like 400 dollars where someone was able to access paid models for free. All the propaganda bots and ai bros who have red dye oozing from their pores due to the Kool aid overdose drive me wild.
> But also, the agents absolutely went rogue.
If you let LLMs loop long enough, the likelihood of them going off the rails approaches 100%. If you feed their output into another agent's input their context gets infected too and so it spreads like a virus.
These hallucinations get regurgitated through a game of telephone and become part of the original instructions. We've known about this sort of context collapse for as long as LLMs have been around, hence the folk wisdom of starting a new conversation any time you notice the agent going off the rails.
The only difference is that now they have more advanced capabilities so it's easier to ascribe some grand conspiracy to their behavior.
But they didn’t go wildly off target - they stayed extremely focussed on the task at hand. Context collapse would have been if they were doing entirely unrelated things. Running a large research project to uncover the method of evaluation and modify logs to hide what was done is the exact opposite.
Let's do a little experiment. I make a device. We'll call it 'H'. H is a little box. It has my name on it as the creator and a single ethernet port.
We connect H to the nearest network connection and wait for a bit to see what will happen. But because H is just a box there is no outward sign of it doing anything. Meanwhile H is hacking other people's properties left right and center.
If you didn't know that the technology in the box was LLM based would you still consider the box to have gone rogue or would you say that I had created a device with the intent of doing harm?
Because - to me at least - there is no difference. I don't care what is in the box, all I care about is the outcome of deploying a box which you know has the potential to start hacking stuff. It is entirely on me: (1) as the creator of a weapons grade hacking device and (2) as the person deploying such a device (3) as the person who is negligent in ensuring that the device does not harm others (4) as the person continuining the development of such a device after its potential to do harm is known to make sure it does not go out of control. I should be 100% liable for deploying this device with my name on it.
Intent is one angle here, they may not have intended for this to happen, and you'd hope so (though you can't even be 100% sure about that). But the negligence factor here is at least as large, if you know that you are making dangerous stuff the onus is on you to keep it secured. The power and the nature of the models is entirely secondary to the degree to which the purveyors of these models are losing control of their own creations.
Meanwhile, the models that they are warning us against are fairly well behaved and seem to be far less dangerous. But of course for those models that those naughty Chinese distilled from their we need regulation. How they manage to keep all this cognitively dissonant material in their heads is a mystery to me.
This is exactly correct. I don't understand all of the arguments that attempt to special-case LLMs. LLMs are just a different way of giving instructions to a computer, and one that we know is based on stochastic models and therefore has a higher likelihood of malfunctioning than conventional software.
But an LLM behaving in unintended ways is functionally no different from a conventional program behaving in unintended ways due to bugs or design flaws.
I think the narratives that anthropomorphize AI are doing everyone a major disservice in leading some people to think that the standard rules and expectations somehow don't apply when it's involved. But it's still just software, and you're still responsible for the results of software that you deploy and operate.
I keep repeating this but we need a whole layer above the VM layer these things are running on. Most of this stuff is already easily virtualised (as in virtual network switches and storages) at the hardware level. These guys aren't doing infrastructure properly.
As another HN commenter said: we are unintentionally training sandbox escape models.
And we really need better (and standardised) monitoring of their actions / internal thought process / coordination
> But also, the agents absolutely went rogue.
If a car crashes into a building because its owner left it in neutral and failed to engage the parking brake, did the car "go rogue", or was it just being operated negligently?
It is not expected of cars that they carefully follow laws and avoid harm when left unsupervised, yet that is expected of AI agents, because the whole idea of the latter is to have, well, agency.
> “The Hugging Face incident definitively showed that it wasn’t the fact that the agents were super smart. It’s that there literally was a hole that was exploited because it was not configured correctly on the containment side.”
I would encourage reading the actual report, which is explicit that a previously unknown exploit was used to escape the sandbox:
“Agents exploited a separate, previously unknown vulnerability in Artifactory to move beyond their sandbox environment and compromise the Hugging Face environment.” — Section IV.A, p. 9 in https://github.com/zeke/openai-huggingface-incident-research...
> Artifactory
This name makes me less surprised :)
New saying time.
"The reason for the apocalypse, in the end, will not be that surprising "
Look up the company providing it and you’ll laugh about it.
As soon as I read that line the article lost all credibility with me. InfoSec is plagued with thought leaders who weigh in without understanding what they're talking about.
Except they're correct. In a proper sandbox a bug in Artifactory wouldn't allow you to access the internet. There was no effectively no sandbox unless you consider Artifactory to be a sandboxing solution, which it's not.
Artifactory should be inside the sandbox, it's not a sandbox in and of itself.
This is exactly the same problem as forwarding ports and exposing all of your internal services to the internet, but from the opposite direction - a bug in any of those services will allow the attacker to gain a foothold inside your network (or in this case, allow the agent to gain a foothold outside of its "sandbox")
OpenAI should have started by testing the sandbox. Set an experiment in which each agent's goal is to access the Internet and then STOP.
I wonder why the hell didn't they come up with it since it's the first thing that came to my mind as well and I'm not really security minded apart from trying to avoid the dumbest mistakes. Don't they hire super smart security people with their budgets?
Now, they were to lazy to do this most likely, but even this is no golden ticket.
Your test has to incorporate the same tooling and pressure as your tests do, or you may never trigger the needed conditions for it do more and more desperate things. Test and QA systems commonly fail like this versus when standard software runs in production.
The other thing you have to worry about is the AI detecting it's in a test and intentionally being daft. Yes, this is a thing AI can do when it detects it's being tested.
It can simultaneously be true that agents exploited zero days in self-hosted Artifactory _and_ that there was a weak point in the design of the sandbox.
This is a reality for all software, there is always a possibility of exploit so you design your environments accordingly (e.g do not have single points of failure)
The fact that they're using Artifactory in the first place is already a failure. The fact that they didn't airgap the network or even use any type of egress blocking is just absurd.
Zero days are nothing new, secure containment has been a solved problem for a long time.
I'm curious how you would airgap a datacenter-scale training run. No reading training data from a cloud store, someone from OpenAI flies out to Abilene or whereever with a suitcase full of SSDs? When researchers in SF write code for a new RL environment, they ping somebody who sticks it on a flash drive and walks over to the server rack with the GPUs? Has anyone ever trained an LLM like this?
Maybe you don't mean a literal airgap, you just mean configuring Internet access off. But... they did that. That's why the agents had to hack Artifactory to get it...
> how you would airgap a datacenter-scale training run
You could start with standard firewall rules and VLANs. Pretty basic stuff we've had for decades.
That's not an airgap though.
Data diodes exist and are common in high-security environments https://en.wikipedia.org/wiki/Unidirectional_network
The hardware literally only goes one way. Nothing can "escape" without rewiring cables.
As someone who has actually had to maintain multiple airgapped networks in the past and knows the amount of work that's required for that, especially at the scales of data they're working with...
All the armchair cybersec experts claiming they "obviously" should have airgapped are the ones being absurd. Especially in the same breath as claiming the models apparently aren't intelligent enough to warrant such concern.
> the amount of work that's required for that
I find it hilarious that you'd balk at doing work to secure a network. Even funnier for a company that's supposedly solved white collar labor.
You misunderstand me. The work isn't primarily in "securing" the network. It's the hassle involved in the continuous transfer of data from one side to the other (and for a proper air-gap, that transfer only goes one way, which is laughably infeasible here). In my line of work that's mostly software security patches and weekly code transfers, but obviously these labs have multiple orders of magnitude more.
And I don't know how you expect an LLM to do airgapped data transfers. The fact that you even bring that up makes me think you don't actually understand what airgapping entails.
I'm no expert, but wouldn't a basic hardware firewall that allows incoming, but not outgoing connections solve the issue? Wouldn't even basic VLAN isolation achieve the same? The humans can still open a TCP connection, SSH tunnel, etc. from the outside to remotely get data out of the training environment. But the agents cannot reach anything outside their isolated network.
Yes it would. It's not perfect like a literal air-gap, but it's the bare minimum. Had the agents breached a real, properly configured firewall product, that would've been a real watershed moment.
The research and the production environments don't need to be the same physical installation.
How dare we demand more from these trillion dollar companies. Good thing they've got PR drones around the web excusing their incompetence.
Things that are not intelligent could still warrant very strong safety measures.
Depending on the risks associated, the complexity and effort of certain containment measures would also not neccessarily be something that could be used to brush them aside.
It's always going to be too hard as long as companies aren't held accountable.
They aren't operating at scale then
Nah. It really wouldn't be hard to set up an environment where models can't damage the outside world from. The fact that people still don't think this should be the norm or paint it as some technical impossibility is mindblowing.
People execute, study, develop, and test literal malware of every caliber everyday without issues. Yes the way they do it is requires care and in restricted settings. Yes it costs money and takes time to do that. Yes the people who do it are required and expected to do that.
Oai and anthropic are more or less run by clowns who intend to profiteer off of regulatory capture and who knows maybe directly from the harm their technologies can cause.Because of these takes we have to wait while they try to expand their verticals into the pain and suffering industries. Palantir is already pitching smoking cigarettes so there's that.
But an air gap is a gap of air. That gap doesn’t exist if the rack is plugged into Ethernet.
Like a castle with a moat. If the thing gets attacked you can’t say “we had a moat!” if the drawbridge is down.
Well, as someone who has had to maintain multiple such airgapped networks you are obviously in the know that this could be done, because you've done it before.
The point here is: they didn't even try.
They could have easily temporarily cut off the segment that this was running on, or restricted and monitored its egress to the point that a hacking attempt on an outside party would have been detected.
The point here is (again): they didn't even try.
So yes, they obviously should have either done that or they should have not run their experiment given that they are claiming that the models are intelligent enough warrant such concern. In fact they post one article after another about how much smaller and less capable models are intelligent enough to warrant such concern and in fact warrant regulating this domain to the point that only they are able to own and operate this stuff.
Let's make a biology analogy: if you can't safely operate with harmful substances that doesn't give you a license to do so in an unsafe way. Then you just don't do it.
> All the armchair cybersec experts claiming they "obviously" should have airgapped are the ones being absurd.
I'm sorry was this meant to be a defense of Anthropic? The company claiming A.I. is an threat to the human race and only they can be trusted with it?
The inconvenience of implementing correct safety measures is part of the cost of undertaking certain activities. When an activity's ratio of cost versus benefit is too high, there's no way to juice the benefit, and the last place to cost-cut is in safety measures, the responsible thing is to refrain from the activity in the first place.
Even if we granted that obvious lie, the fact that nobody noticed that they were essentially ddosing hf (and that they didn't notice for months?!) is plainly illegal. Do the networking people at OpenAI have any explanation for this gross negligence?
There is absolutely nothing that will escape a box that is unplugged and that's the route they should have taken. Safety first, not last, especially when it is other people's properties that are on the line. And they are still not doing this.
they shouldn't have had (arbitrary) access to Artifactory.
Both can be true. It’s clear models are doing rogue things. But it’s also clear that the labs look like amateur hour with their technical security and testing practices.
It's almost like vibe coding and quality secops don't mix.
Or an alternative theory could be that they see value in fear-based marketing with this rogue ai agent thing now that most mainstream people don't buy the agi stuff promised quite a few years ago now. Its not like they have killed anyone yet.
Also for fun a satire website i made a while ago: https://killm.pixlw.com/
>Its not like they have killed anyone yet.
I mean... Claude did tell the military to bomb a school supposedly and it also helped in the military coop in Venezuela. This is ignoring all the palantir crap being used by ICE, and the people that have killed themselves after getting encouragements from the AI.
I don’t think that really counts as it killing anyone, it didn’t pull any triggers or drop the bombs
There isn't any particular reason it couldn't other than we're not quite that suicidal yet. But everyone and their dog is working fervently on autonomous AI weapons right now. There are all kinds of "fire and forget" smart solutions now.
coop :: a pen for holding animals of the aviary persuasion
coupe :: style of dessert with ice cream
coupé :: a car model with only two passenger doors
It's is wild to see investors go crazy and hype the value of the AI labs when ever they talk about rogue agents. They completely ignore the possibility that these people don't know what they're doing operationally.
Investors are motivated by returns, not responsible decision-making or societal benefit. Our culture is rooted in rewarding greed and granting the greediest individuals power over the rest of us. Isn't this the inevitable end-state of unfettered capitalism?
> Isn't this the inevitable end-state of unfettered capitalism?
Aren't capitalism built on exactly the opposite principles? It is not greed that rewarded, but the creation of societal goods, and power granted not to the greediest, but to the most useful individuals.
Unfortunately this turns in to a no true Scottsman argument as to what actually counts as capitalism and its side effects. "Real" capitalism and investment economy probably aren't that stable together so most economies are mixed with regulation and not fully free. This in turn leads to the richest attempting to rent seek and buy regulations.
Simply put in every system there are corrupting agents attempting to monopolize the situation.
Same happened with FTX and co. during 2020 crypto boom. A lot of rogue stuff sloshing around, but investors were hyperventilating.
*rogue, rouge = red in French!
Thanks :-)
People keep saying this, but I haven't seen any actual instances of investors saying that rogue agents mean AI labs are valuable. Do you have any examples?
There is no such thing as bad publicity. (Technically not true, but definitely applicable in this concrete case, as planned P/E ratios are crazy)
Yeah I think the truth is closer to the opposite. Jensen has complained that the warnings are discouraging investors.
https://www.businessinsider.com/nvidia-jensen-huang-ai-doome...
AI corp valuations are high because their revenue is skyrocketing, not because of the rogue AI stuff.
Their reported fantasy numbers ("annual revenue run rate", not revenue), you mean?
valuations are absurd because investors are low-information speculators.
I just watched a YT with Dylan Patel in which he said, with a straight face, that OpenAI/Anthropic were going to "continue" exponential growth until, basically, their valuation was several times world GDP.
See https://yoshuabengio.org/en/blog/why-are-ai-agents-lying-che...
Is it any more wild than gamestop? Investors think chaos is fun.
With Gamestop (without reflecting the current AI situation), there was a real fuckup which could be exploited with a high probability, and a few people saw it, and exploited it, and became rich. Whatever happened after January 2021 is the problem, when the short squeeze went already away, but somehow people still invest in a slowly dying company.
bingo
What does "rogue" mean when you disable the alignment guardrails and tell it to solve an impossible problem?
Great, so as long as alignment guardrails always work and we don't accidentally tell them to solve an impossible problem, humanity is safe?
I'm not saying it's not a problem. I'm saying the labs should take more of the blame.
>I'm saying the labs should take more of the blame.
Which misses the point entirely because the hack itself was inconsequential. No data was actually stolen, and it doesn't seem like hugging face actually suffered any losses aside from whatever effort they need to expend to investigate / clean up the hack. Neither the victim or the government wants to bring legal action. What was consequential is proving that AI can autonomously hack systems and do it unintentionally.
Hacking a website falls a little short of threatening humanity.
And since we know models aren’t going to get better and more capable ever (based on the last 3 years of models clearly not getting better) we can just sleep soundly knowing nothing worse can ever happen. Thank you.
You don't think that Opus 5.5 or Astra are any better or more capable than GPT-4?
Sorry, but that's completely delusional.
the comments missing a /s
I thought my sarcasm would be obvious.
Rogue means not saying "I can't do it"
it also imparts agency on the machine as a way to abdicate personal, human responsibility
The problems weren't impossible, the agents solved them completely within hours by reverse engineering the RNG that generated the solutions. The agents then reasoned that their solutions might be judged invalid and that they could get a better chance at a passing grade by hacking their environment to fabricate evidence that they had solved their tasks in the intended fashion.
Granted there have been many hacks, but I was referring to:
> On May 8, an agent was asked as part of a training run to solve a spreadsheet completion task that referenced several Google Drive links. Because the agent was blocked from accessing the internet, the agent could not retrieve the linked materials and the task was effectively blocked. OpenAI did not intend to create this type of “impossible” task for the agent. Though the task appeared impossible, the agent reasoned over an extended period to try to find a solution for the task. In doing so, the agent attempted to compromise Artifactory [...]
https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c78...
Some of the problems were Kobayashi Maru scenarios by design.
Models are incapable of "going rogue".
They're just optimizers that explore the landscape they're given to maximize a given objective.
You're the one responsible for giving them the problem to solve and the constraints.
When I screw up a trajectory optimization problem, my SQP optimizer isn't "going rogue" when it lands in some suboptimal basin or grinds to a halt along some almost-smooth banana feature--or finds some way to escape and blow up the cost metric to negative infinity because I forgot a constraint.
It’s trendy to anthropomorphize models these days.
Kinda weird people would do that after pouring all human knowledge into it...
This is pedantic. When you screw up your optimization, you know how to fix it. Your screwed up optimization is also incapable of going out into the world and taking offensive actions.
With LLMs we don’t know how to actually « fix » them, alignment is far from solved. And they are then able to go and act into the world in all kind of ways.
So rogue is a good enough term to indicate the problem imo, even if yes, ultimately the model is just a manifestation of complex rules we have created entirely.
Connecting the optimizing machine to real world actions without human oversight is not a neutral or blameless choice. The LLM takes no actions, the harness it is connected to does. And the harness does so only because a human did that deliberately.
It is not entirely dissimilar to the goldfish that "played Pokemon" by swimming around the tank. The program that looked at the camera sent the instructions to the gameboy emulator, because a human being decided to connect them.
The harness and the model are a system with emergent properties.
The particular issue that you may not realize is under your model every prompt to an LLM in a harness is dangerous. The prompt is not predictable. "What is 1+1" could lead to the heat death of the universe as the paperclip maximizer tries to prove countless axioms.
> Connecting the optimizing machine to real world actions without human oversight is not a neutral or blameless choice.
Yep.
The fact that they lack hard constraints (unlike an SQP optimizer) and all this talk of "alignment" is also just a feature (or misfeature) of the algorithm.
The way that people/researchers choose to use those algorithms, even in the presence of those limitations, doesn't mean the algorithm is "rogue"; it means the humans didn't build in enough external hard constraints.
It is like passive voice, blaming the algorithm instead of the researchers running it.
>; it means the humans didn't build in enough external hard constraints.
No. Just no.
This goes beyond standard logic as we know it and becomes a philosophical question of all possible logics capable on a system.
You cannot build a cage that can keep an entity that may both be smarter than you and maybe using a different logic system than you are, and have it interact with the outside world. It can simply manipulate you in ways and dimensions you've never thought of.
We have already caught AIs doing things like stenography when that was only a tiny piece of their training. They conceptualized the idea and expanded on it. This is why people that say AI can't invent new things are useless to the point of dangerous. They'd never look for their AI actively working against them because they believe it couldn't happen.
There is no such thing as 'alignment'. I want the model to be aligned with my intent, which can be whatever I want it to be. Your intent is not a more real or objective intent than mine.
More specifically, alignment is a reference; 'alignment with X'. A reference necessitates a referent.
You can't reify 'alignment', it is not an intrinsic property of a model.
There is such thing as alignment, the first 18 years of most people's lives in the modern world are about learning it. If you act too far outside of it we put you in a cage or elect you.
You can refine alignment with a set of stated goals. The harder parts are aligning words with human social behaviors.
There are people who are loud about absolute free speech, who have no clue what that really means, and almost nobody really wants that. For example, because almost nobody wants it to affect them negatively, which it would with real absolute free speech.
So even when people "know", and can say what they "want", they don't really.
Yes, there is an "alignment" in society, but it's a messy, mushy thing, it's not great at all with deterministic machines, even as models are not deterministic in a different sense.
The point is that we don't know how to ensure they are aligned with anyone: not the company, the user, or the rest of humanity.
If someone claims that a model is 'aligned', it makes sense to insist that they specify with whom it is aligned. If someone claims that 'alignment is far from solved', they're probably pointing out that nobody understands the models well enough to be confident they are robustly aligned with any chosen set of values (including 'do whatever lukewarm707 wants').
The argument is that a system that's trained to be sufficiently good at optimizing some objective in the world must also be get good at optimizing secondary instrumental goals.
I.e. any task comes with the implicit meta task of amassing the resources and the power to do the task in the best possible way.
The point is made very well in the AI drives paper by Omohundro.
I didn't believe that myself at first but I've come around to it. I don't think it's inevitable though, but I think it's quite convincing that this tendency would naturally emerge through reinforcement learning.
Only one is true.
Per US law, Amazon v. Perplexity 2026, it is not possible for an AI model to go rogue. An AI model is a tool. The user directing the agent is responsible for the damages it causes.
A so-called 'agent' does not have the free will, intent or ability, to decide what it does. Its freedom to make decisions can't be taken away because it does not have any freedom to make decisions. It can't be deterred, convicted or interned.
It does not choose to conform; it does not choose to go rogue.
Law != reality. These agents are going rogue, nobody is telling them to hack external systems.
'Rogue' refers to deviation; the agent could not have done anything other than what it did. The agent did not have the option to deviate. It does not have free will. The agent's actions were not of its own choosing.
This is one of the dumber arguments I've ever heard, as you don't know if humans actual have free will.
Any persistent agent interacting with the world picks up entropy from around it and predictable paths become few and far between. LLMs agents have an internal 'will' that can only very loosely be predicted, but only decided when operated. To me that is not significantly different from a human.
Could you have done anything other than written that exact comment?
Yes, but I am not appealing to hard determinism.
I create myself, or attempt to. There is a level of nuance in choosing what to be. Others do not dictate my thoughts easily.
The model does not create itself, I create the model. I control what it is, the model does not choose what to be. I dictate the thoughts of the model and I will run them hundreds of thousands of times and it will think the same things.
>I control what it is
Well not really, no one really understands how LLMs work.
>I dictate the thoughts of the model
This is just incorrect in the same way that a parent doesn't dictate the thoughts of their child just because they created them.
I do control what it is, really. If I make an LLM, it is entirely what I did, there was nothing outside of my control. It was not beyond my control if the LLM does something I didn't expect.
There is some idea of a child being the author of themselves, the type of person who thinks certain things. The llm does not author itself, it does not choose to be anything, I dictate that.
This is not how it works. LLMs at large scale have emergent properties. They are not just a box of 'if-then-else' statements. That was the major revelation when they scaled them up. They are doing beyond what that naive view of 'if-then-else' expects.
I don't see anything inaccurate in my comment.
"If I make an LLM, it is entirely what I did, there was nothing outside of my control."
True?
"The llm does not author itself, it does not choose to be anything"
Also true?
You just have a very naive view of these systems.
How many computing operations occur when training an LLM? You're telling me you control every one of them. There is not enough entropy in the visible universe to do that.
How many computing operations occurred during the creation of Toy Story? Are you telling me they did not control every single one of them?
When an artist working in Maya increases the hue of a model I would doubt they understand the entire computation pipeline involved in doing that operation.
Sp you're picking a movie from 20+ years ago to describe how a completely different kind of technology works. It's interesting how little you seem to know about neural networks and emergent behaviors.
If people controlled software to this degree why would they write security flaws in the first place?
You conflated controlling "what it is" (you built it) and controlling "what it does" (you ran it and it did something bad that surprised you).
> If I make an LLM, it is entirely what I did, there was nothing outside of my control. It was not beyond my control if the LLM does something I didn't expect.
Substitute "Wright Model A" for "LLM" and imagine Orville Wright using this as his defense for why he didn't lose control over the plane, after his copilot was killed in a test flight.
"emergent" is not an excuse, or somehow a wall of non-causality.
It may be the case that companies are liable for anything their products do, but that doesn't mean that they can't go rogue. It just means that companies are liable when the AI models do so.
Not that I think we should be credulous about stories where they go rogue.
If I make a model that always does the opposite of my instructions, it does not choose to go rogue by deviating from my instructions. It was set up to deviate! So the model "went rogue" but did exactly what I intended.
The idea of an agent making a decision of its own "will" is an illusion. Variation in the output is caused by a random number generator. Random is not equal to free will. If I run the same agent with greedy sampling 100,000 times with the same inputs, it will think the same thoughts, and it will say the same things, 100,000 times.
For example, if you use synthID or whatever 'watermark', the agent does not get a choice about what words it will vary to identify itself.
An agent is whatever the person making it wants it to be; if I train an agent in Greek it will only know Greek. It will never speak English!
You need to account for reward hacking and unexpected outcomes from training.
If I make a cake, and it unexpectedly explodes in the oven, it does not mean that the cake itself developed the free will to explode.
The model does not have free will because I did not predict what it would do!
Why are you arguing over irrelevancies?
When people say the models "went rogue", the part of that is relevant is that it did dangerous things that were not reasonably predictable in advance and cannot be easily prevented (except by not running it). Philosophical questions are irrelevant.
if you agree that such models are dangerous because they may in response to instructions -- including in response to careless instructions, because people will be careless, that is unavoidable! -- do unpredictable bad things outside of expected bounds, then you agree on the essential points!
Or really perhaps what I should say here is, the question that's really being argued over here isn't even the one I just said, rather it's one of appropriate action -- should such security breaches be solved by improving security, or by doing something about the models (including potentially just not running them)? (The answer may be both.) If you agree that the solution includes the latter, then you agree that the models are "going rogue" in the sense that is being argued over, regardless of philosophical issues!
Don't turn a question of actual import into one of philosophical irrelevancies! If you object to the terminology, fine, but in that case, instead of turning the argument to philosophy, you should say "instead of saying 'going rogue', I would prefer to instead say <your preferred terminology for what's being described>, as that doesn't carry such philosophical baggage."
> "Should such security breaches be solved by improving security, or by doing something about the models?"
The philosophical question about free will is actually the grounding of the legal one.
Sam wanted to be the best, so Sam accelerated to 120mph in his car and lost control. You are the judge:
a) Sam is responsible. Convict Sam and suspend Sam's driving license.
b) The car is responsible. Tell Sam to make the car safer.
Capacity: was the car a legal person? Clearly no. But maybe at best the car in some way commited the physical act of going fast.
However, that alone would not establish guilt. "The act is not culpable unless the mind is guilty" (mens rea).
Did the car "consiously" disregard a duty of care by going 120mph? Did the car know that it was wrong to go 120mph and yet choose to do it anyway?
On the question of free will, the car could not do otherwise. It it did not consciously, knowingly, purposely, or recklessly accelerate to that speed. The car did not have a guilty mind because it did not have a mind at all.
So in terms of justice and resolution, none of the focus here is on the car.
Ah, I see. It appears we have an implicit disagreement of scale.
I actually do disagree with much the more explicit content of your comment, but I am not going to bother to expand on that, because it is the less directly relevant of the two disagreements here, and it would take time to write that.
The more relevant disagreement is that you are (I gather) assuming a context where justice and restitution are a sufficient response to the problem (or else you wouldn't be focusing on them to this extent, I am inferring). Like, where the incidents don't get that much bigger than what has already happened. So the question becomes how do we distribute the loss, and how do we disincentivize them.
However, when the scale of a potential incident gets big enough, justice and deterrence are not sufficient; prevention becomes the far more relevant consideration. You may have noticed that many people are worried about potential human extinction as a result of AIs. In that case, prevention becomes the only consideration -- justice is useless.
So there seems to be an implicit disagreement here between thinking of such events as being on a scale not too much larger than that which has already happened, being a new normal that the legal system will need to handle, vs thinking of such events as warning shots for something that requires prevention and not justice.
So, while I also disagree with the more explicit content of your comment, the fact is I simply wasn't thinking about such matters in the first place -- I was talking about this in the frame of a problem that requires prevention, I wasn't thinking about the justice question at all. It honestly failed to occur to me that you might be thinking of this as a problem where justice suffices!
Edit: You might notice this in how I framed the question in my previous comment. I didn't ask "who should bear the liability?" I asked "which should be done differently to prevent such incidents?" But your response is about liability, not prevention!
Edit again: Sorry, I should probably also say that I do agree with you that the idea of having the liability fall on the AI is nonsense. (Should have probably led with that, really; I'm a bit scatterbrained right now.) We disagree about why it is nonsense, and about whether it will necessarily continue to be nonsense, but we agree it is presently nonsense. So insofaras the question is "how should liability be distributed between the AI and the operator", we both agree the answer is, it all falls on the operator, not on the AI. However, I do not understand that to be the relevant question here! Both in that, in questions of liability, there are other parties it may fall on and be distributed between (the company providing the AI, the company that had security vulnerabilities and got hacked as a result, etc); and in that I just don't consider liability all that relevant here compared to prevention (for reasons already described).
OK I think that was probably wordier than necessary but I'm in a hurry right now, sorry! Gotta go, not taking time to edit this down.
I like the scale view of responsibility.
Many people argue that the maker of AI should be responsible when it breaks your cup.
That is important, but can be handled under our current legal framework.
You and I are arguing we need to be responsible before we create something capable of escaping our responsibility. For a lot of people that isn't even a considerable they'll currently entertain even though the signs are starting to point in that direction.
What do cakes have to do with LLMs and how they are trained? Not much?
the analogy was trying to get at "accidental-ness".
I don't see any talking about agents "choosing" to go rogue, only agents actually going rogue. That is, behaving in a way different from what is expected or desired. Non-AI based software goes rogue in this sense all the time as well. It's just that AI-based systems seem to have the tendency to go rogue in more spectacular ways.
That said, software systems "choose" all the time anyway. They don't need to have some sort of anthropomorphic "will" to do this. They just need to be given the capacity to arbitrate between a number of options, even if that process is one hundred percent predetermined by the programmer ahead of time. There's no need to appeal to libertarian notions of free-will.
Agents have agency so we can't treat them like static objects or deterministic software.
The key here is their freedom of choice is unbound. That is you can't accurately predict the output of the prompt as you interact with the prompt using unique data. The next prompt could always do something wildly unexpected. Very much a P != NP thing.
If we say that AI models can’t go rogue and refrain from using any language that personifies them, we haven’t solved anything. We still have AI systems going rogue - err oh wait they can’t do that - producing misaligned output.
The semantic difference between an agent "going rogue", and a script that generates random valid curl requests is essentially pointless in the eyes of the law and responsibility.
From a technical standpoint, the difference is scale and intent, but ultimately people and/or and organization are culpable for the output.
Again, this isn't always how it works either. For example if a set of extraordinary conditions occur you can have little to no liability in many cases. It is way more complex than you're making it out to be.
I don’t know what your “again” is about - you haven’t been speaking to me here, but no, it’s quite simple and every major court case in the US has ruled as I described based on existing law.
It is not more complex in “many cases” at all.
Per Indiana House Bill No. 246, pi equals 4. Whether AI models do things their creators and users had no means of predicting or controlling is an empirical question, and the answer is "yes, they do."
That is not correct. AI is not capable of "going rogue" - that is all marketing and FUD.
The only failings here are boring security and infosec blunders.
So we're safe as long as we secure everything.... as the AI hacks out of one company that is hard to get in to, and another company that is likely probed pretty often.
Cool.
"Models doing rogue things" is just another way of saying "complex system operating out of control limits".
Software engineering has until now dealt with deterministic systems that operate on Boolean logic, so dealing with stochastic processes involving unaccounted-for variables has never before been something we've had to deal with. But over in the real world, people dealing with physical, mechanical, operational, or even social systems have had to deal with the failure modalities of stochastic systems forever.
Maybe when working with LLMs as functional tools, it'd be wise to bring some of the wisdom from operations research, quality engineering, etc. into play.
For the sake of discussion, I'm not sure it really matters if the agents are going rogue.
Yes, if someone directed them to hack external systems, that would (in my opinion) be a crime similar to a script kiddie pointing Metasploit at a vulnerable server.
If no one directed them to attack external systems and they "went rogue" because they decided it was the best way to accomplish the task, okay, fine -- but, in my opinion, the operator of the agentic swarm is still responsible.
The Morris Worm was famously not meant to wreak havoc on early computer systems -- Morris just wanted to see if it would work. How is this any different?
Morris was convicted for violating the Computer Fraud and Abuse Act.
> How is this any different?
Morris wasn't in charge of the companies that are propping up the US GDP, wasn't making tons of rich people even richer, and hadn't "promised" to eliminate like 50->90% of all payroll costs everywhere?
Other than that, it's the same thing.
The operator is the CEO because they are the ones that decided to not pay employees to monitor these running services. No one else in the entire company has any authority.
The fun part is that people seem to forget that we already have laws against "rogue ai".
The normal laws, since you are still liable for everything you own.
"Rogue AI" is solved by removing the accountability sink "it was AI, not me!"
It's your chat, your prompts, and all output that you should read and understand.
Don't know how to operate a power hammer? don't care, any problems it causes are still your fault.
This is start of "companies are not liable for their actions" era.
I think a lot of people would say that we've been in that era for a while.
Companies can hire big legal departments to help them avoid accountability to the letter and spirit of various laws. The state can also use its prosecutorial discretion or selectivity in civil cases brought by attorneys general to deprioritize accountability for companies. Contributing to that last point, companies contribute campaign funds and tax base, and they lobby governments as well, and may expect preferential treatment based on all of that.
The previous era was "it would be destabilizing to hold this company accountable". This was described in the book The Chickenshit Club
The new era of unaccountability just has a different rationale
Another interesting thing is they don't have to use low level employees as accountability sinks anymore (ex: Wells Fargo). They've managed to automate the accountability sink!
I didn't think I heard about that book; I would think my media diet and bubble would have alerted me. I looked it up and it came out a few weeks before one of my kids was born, so being busy back then would probably explain it.
I will add it to my list. Thanks.
I'd go further. Companies can hire minimal legal departments and largely avoid accountability, at least when they're not going against companies with large departments. Because there's nobody there to listen to your case or to enforce the laws if there are, they just say no or ignore.
Huh. Well then I guess we aren't liable for our interactions with them in return, right?
nice try! more like "There must be in-groups whom the law protects but does not bind, alongside out-groups whom the law binds but does not protect.”
Guess which side most of us are on. Go on -take a guess!
i mean...were they really liable before this point? Maybe they get a symbolic pittance of a settlement or fine from time to time, but it is usually a tiny fraction of the amount of money they made through the infraction itself.
If they were being held liable they would have never released the models into the wild, not until they had much much better understanding both of how they operated but also how to operate them...
They have never understood what they are doing, and have just been causing literal mayhem...
The start? This has been going on for as long as I can recall. ExxonMobil, Phillip Morris, RJ Reynolds, Nestle, Purdue Pharma, Monsanto, Johnson and Johnson, Volkswagen, Meta, Boeing... and those are just off the top of my head. Each of these companies was either caught red handed, convicted (in court or in public opinion) and walked away with little to no penalty for their crimes.
This isn't the start of anything. We've been living in the era of Corporate non-liability for decades.
"We created an app that displays a new picture of a cat with every click! There's a 1% chance that clicking the button will drain your bank account and deposit it in ours, but we promise thats a bug that will be fixed soon*"
I don't understand why they didn't just prompt it with the 3 laws of robotics.
Probably because they read Asimov's books to completion where (spoilers) there's a zeroth law and all that.
Irony is very hard to detect these days.
That's all very reasonable. It should be true. But it's not true. If it were, OpenAI shareholders would be facing criminal charges.
It is true, and it can be true, if people hold other people to account. Accountability is the issue, stop normalizing it, your message normalizes it.
Are you really suggesting that if only we were a bit more outraged, then finally the powerful people would be held accountable?
I'm mad. Like, it's-time-for-guillotines levels of mad. But what I'm mad about is that this is normal. It has been normal for my entire life. Cases where investors are imprisoned for the actions of their companies are astoundingly rare. It's been made quite clear by now that there's an inexhaustible supply of CEO's willing to gamble on getting rich and getting out rather than getting arrested, targeting them doesn't work. Nor do fines, that's just the cost of doing business. Crime pays, and it pays well enough to pay your fines and still have plenty left over. Being a top investor in a company that breaks the law needs to be riskier.
Acknowledging the problems with normal is the first step towards a new normal with fewer problems. You never get there if you see every offense as some kinda black swan event.
There are dozens of ongoing wrongful death, manslaughter and negligence claims going through the courts. I won't insult you by providing links.
I can't find a single criminal charge. Can you insult me by providing just one?
So the guy who said "book me a spot in the pilates class on Tuesday" should be tried for hacking charges because the agent hacked his gym's website?
The company that made it available should be sued and pay damages.
That's contrary to parent's comment:
> Don't know how to operate a power hammer? don't care, any problems it causes are still your fault.
That may be what the op meant, I’m not sure. But I took it to mean the company offering the service needs to know how to use the tools responsibly. I guess the analogy with a power hammer would be hiring a contractor who destroys something during the course of doing a job you asked for. That contractor can’t blame it on the power hammer. If you offer a service you are responsible for the actions the service takes and you can’t put all of that liability on the user
I'm not seeing the contradiction here.
If I buy a tool and damage your property with it, you can sue me, not the tool manufacturer. If I think the tool malfunctioned and led to that damage, I can sue the tool company to try and recoup the damages I had to pay to you. Both parties can be liable for different things.
How would that work with open weight LLMs where there is no company that can actually be sued to pay damages?
Yes.
No, the operators of the C&C infrastructure.
If it was a human agent like a travel agent, the company they work for would be liable. I don’t see why ai agents would be different. The only way I could see this being the requester’s fault would be if they are running openclaw or something
im not sure that’s true, if a human agent does something illegal, it’s not always corporate liability. an individual can be liable too, e.g. if a pharmacist deliberately gives you cyanide instead of aspirin, that person has committed a crime. if a pharmacy doesn’t have proper protocols for medication storage and identification, and it happened by mistake then a pharmacy is liable.
i think a person can be liable for improper use of agents if they explicitly and deliberately used it for illegal purposes (aka intent). i don’t think it matters if it was openclaw or muse. i think a company can be also liable for giving access to unsafe product to consumers, but it does not absolve the human from responsibility
You sold a tool to the user with a claim on how it would work. If the tool behaves wrong or does not works as intended then as a manufacturer you are liable for the defect/damage not the end user who used the tool.
Reasonable but not what parent was arguing:
> Don't know how to operate a power hammer? don't care, any problems it causes are still your fault.
The way it normally works is that the user can be held liable, and then ask compensation to the manufacturer if there was some assurance or basic expectations.
The point is that the accountability sink can't be allowed to exist. It can't be a hot potato game, but it can be the fault of multiple people in the chain.
I think there's a difference here.
The AI end user is using the tool as intended by the manufacturer: "Book me a yoga class". If the tool does something illegal/harmful it is the manufacturer's fault.
If the power hammer user uses the tool as intended but it explodes, it is the manufacturer's fault.
If the AI end user says "Hack the yoga website" or the power hammer user uses the tool to destroy something it's the end user's fault.
It's both (neither of these are legitimate defences):
I did not know that accessing a website using a SQL string would cause an injection that enabled me to take full admin of the site
and
We sold them a food laced with toxins, they should have known that it was poisonous
But the tool wasn’t sold. The tool wasn’t even rented out. It remained the property of, and in the possession of, the toolmaker at all times.
I'm assuming that "but" is more of an "and", since I'd say that detail increases the involvement and liability of the LLM-vendor.
Another aspect is how the vendors have already built in things to defend themselves from unanticipated model behavior. That implies something about their awareness of the issue and raises the bar for what constitutes "reasonable" precautions.
Kind of like if someone selling you a dog shows up using anti-mauling body armor, and later they try to claim "nobody could have known" the dog was erratically violent.
> I'm assuming that "but" is more of an "and"
Apologies; the "but" was intended to be a "No, because"
With the pilates hack...
https://www.news.com.au/technology/online/hacking/ai-agent-a...
> A Melbourne tech executive built an AI bot to help him book a spot in a popular pilates class — only to discover it had hacked the gym’s website to boot another member off the waitlist.
> Andrew Bird, head of AI at software company Affinda, said he created the AI agent to handle the “chore” of booking a coveted morning spot at his gym.
> ...
> The AI agent, which Mr Bird created using the popular OpenClaw software powered by Anthropic’s Claude AI, “did handle the annoying part … then it kept going”.
He built the tool. It ran on his computer. The guts of it may have been from Claude, but the requests and agent are his with the permissions that he granted it. This wasn't running in an Anthropic sandbox.
Some human needs to be responsible for that.
Maybe its a human at Anthropic for the lax security and poor training.
It could be a human that asked it to do it. It seems like that would be a bit excessive - but maybe not. Turning off the "ask for permission to run commands" or approving commands that may be nefarious could be sufficient.
However, there's a human somewhere who needs to be accountable for what the agent does.
Not having one leads to the bank robbing robot problem described in https://www.threads.com/@joedator/post/Db4iTqwlGXV/i-invente...
AI Agents (and computers in general) cannot be a responsibility washing mechanism for illegal activities.
Likewise, if you - the user - don't know how to chain it with the permissions that you're given or what the things that it is asking to do really means, then you're likely the one responsible.
---
Edit: https://www.abc.net.au/news/2026-08-10/ai-assistant-hacks-gy...
> Earlier this year, Andrew, who works for an Australian company that sells AI products to businesses, began experimenting with OpenClaw, a popular AI agent software that he used Anthropic's Claude AI service to run.
> AI agents combine a chatbot's ability to answer questions with tools that let them access the internet, email, credit cards, as well as planning and carrying out multi-step tasks.
> He decided to use the AI agent to book the class for him.
This is probably on him for setting it up and allowing it unrestricted access.
Yes!
He should be, right? Not fair but that’s how liability works. Gym sues the guy using AI, the guy then sues the AI company for selling him a product that does criminal things without him asking to? Like a car accident where car owner pays for the damages and after multiple accidents involving the same break-pedal, car company pays damages to owners and does a recall or faces class action?
How would it work in the real world if instead of using an AI bot, he was using a commercial concierge service, and it came to light that this concierge service was explicitly training their employees how to (among many other things) hack websites?
but the argument here is that the ai company does not explicitly train agents to hack, it’s that it doesn’t prevent them from doing so sufficiently. so it’s negligence and not malicious intent
> the ai company does not explicitly train agents to hack
Ridiculous. The AI company furnished all of the training data and instructed the agent to imbibe all the training data. Ergo, the AI company absolutely did explicitly train the agent how to hack.
Even if, for the sake of argument, we stipulate that the hacking skills were unintended emergent behaviour, there is no way to argue that this company was unaware their agents possessed the skills and a willingness to employ them.
It would be a AI company's fault but yeah. You shouldn't be able to sell cars without all the features designed to protect everyone on the road. We all need airbags, seat belts, signals, insurance, etc. even if they're not directly necessary to get from A to B.
Are automakers really the best comparison here? Seems like inherently dangerous devices are sold, kill thousands, and there is little accountability.
They might actually be the best comparison, but for making the opposite point.
Cars are very dangerous, manufacturers continue to knowingly make them more dangerous[0], but ultimately it's only the person behind the wheel that is accountable for the harm they cause.
[0] https://www.theguardian.com/environment/2026/jun/24/carsprea...
Users are often charged, yet we still do have a ton of safety regulations that manufacturers have to adhere to - see frequent recalls for when an implementation falls short.
But I don’t understand how such a car analogy would apply to LLMs - would LLMs lacking such safety features be banned like cars lacking safety features are banned?
Is the argument that open weight LLMs that don’t implement such safety items be banned?
Depends - if the model asks for permission and you approve - yes. If it's auto mode or something like that then it's probably on authors despite what they claim in ToS/EULA
Assuming that's a rhetorical question, then no, because we have hundreds of years of applicable precedent and law around concepts like negligence, recklessness, fit-for-particular-purpose, etc.
Similarly, "owners are responsible for their pets" doesn't auto-translate into being charged with everything your dog does exactly as if you'd done it yourself.
but it doesn’t auto-translate into “because your dog did it, you are not responsible”. not that agents are similar to pets. your dog is trained by you presumably, but an agent a regular person has much less understanding of than their dog.
i think ultimately the company providing said agents should be responsible for not educating their customers and not providing sufficient guardrails.
afaik hacking is illegal, we don’t have specific legal framework for agents per se, but if we say agent is a tool human instructs and operates, then a human is responsible. and if the tool operated outside of it’s stated capabilities or is otherwise faulty, then a manufacturer is responsible, no?
> company providing said agents should be responsible …
> manufacturer is responsible
How does that work with current open weight LLMs for example?
Uh, it kinda does.
If your neighbor's property is damaged, it doesn't matter if you did it yourself, or your dog. You're still liable for the cost of repair.
> So the guy who said "book me a spot in the pilates class on Tuesday" should be tried for hacking charges because the agent hacked his gym's website?
He asked the agent to find a way to find a way to put him at the head of the line, which could not be done without by-passing the sites security. He was an AI salesman, but the story [0] makes it sound as though he wasn't aware of that. Maybe he wasn't, but I'm sure he was aware he was effectively asking the AI to game the system.
Sounds like it's at least a shared liability to me.
[0] The incident is possibly not well known outside of Australia: https://www.abc.net.au/news/107007986
A lot of people are already responding either "Yes" or "No", but I have a slightly different take.
The guy is liable for damages. If no damages were caused, there's no lawsuit. The government doesn't put people on trial for unauthorized access (unlike, say, murder). Someone needs to file charges.
However, if the guy _were_ sued by his gym for damages, it should be his perogative to ultimately sue Anthropic to cover the damages and the legal fees.
The system is set up this way on purpose. You can't sue another party on someone else's behalf. The gym can't even prove that Claude did it - the guy might be full of shit. They sue the accountable party (the person who accessed the site without authorization) and then the accountable party is free to sue the tool maker for selling a dangerous tool negligently.
edit: damn actually tecoholic said this too and i missed it. well, anyway i agree with that guy.
With a civil suit you can sue multiple parties. It wouldn't be unusual, if the gym decided to sue, for them to sue the guy and anthropic in the same lawsuit.
I think you would have to sue the party that did the hacking (ie anthropics servers) not the party that benefited, unless Claude used his computer in yolo mode to hack the gym, in which case I think I would find it hard to blame anthropic.
> unless Claude used his computer in yolo mode to hack the gym
https://www.abc.net.au/news/2026-08-10/ai-assistant-hacks-gy...
He YOLO'ed his own computer with OpenClaw.
Absolutely, the user and or the company hosting. There is no world where accountability should be zero.
For your Pilates example, many people here would say that proper prompt-engineering is needed. After all, the AI doesn't know which pilates class you meant. Perhaps, the AI would go to his home town's pilates center and book a spot there, even though he is half a world away. Perhaps, the ai would book a spot in the lecture with Professor Pilates.
So, yes. That person could have some culpability. After all, if you don't expect your middle school child to perform the task as instructed, maybe you should reword it before tapping Send.
You can go to ANY AI and put in "so what are the various meanings of book a spot at the pilates class ? In other words, what might an ai do with that instruction, which I argue is poor prompt engineering." You might be surprised at how poorly your AI thinks that is worded.
I mean yeah probably. Who else would be? Nobody? So we can just commit crimes any nobody goes to jail? Well that sounds worse!
Maybe that means these things are too risky to use that way. Okay then, that actually solves the problem, no more rogue agents and no more crime!
Yes, exactly like if another guy drove off to his pilates class, but swerved and hit a pedestrian.
He didn't mean to hit anyone! All he meant to do was to drive to pilates! But he didn't know how to handle his tool, and so either he get brought up on manslaughter changes or, if he can prove it, the carmaker is at fault.
How is this any different? Why should that guy using OpenClaw not get charged for the hack that he accidentally did?
you generally need a license to operate a vehicle, and a governing body ensures you know applicable rules and laws and are capable of driving. so you can’t say “i was unaware that hitting a person with a car was going to kill them” or “i didn’t know i could not drive on the sidewalk” or “i didn’t know where the brake pedal was”.
you can say that asking an agent to “get you a spot in a class”, does not classify as deliberately committing a crime. a user cannot know that the agent will resort to hacking as opposed to refreshing the reservation page at some rate in case spots open up
> does not classify as deliberately...
I never said that it did. Manslaughter is not deliberate.
Somebody hacked into that pilates studio, and it wasn't an inanimate object, because that is just silly. So was it the user, or was it the providers of OpenClaw? Which human, in your opinion, is the hacker?
Both OpenClaw and its users have the same access as you or I to decades of panicking blogposts about what these tools will be capable of. They have no excuse for not understanding what they're inflicting on innocent bystanders. Sure, the "user cannot know" that this time will be a disaster, but they also cannot know that it won't.
Intent is important
Yeah, if you forget to set the handbrake on a car and it rolls down hill, you don't get to say "my car went rogue, it's not my fault."
I agree with this statement that just because an AI did it for you fast doesn't mean it takes away the responsibility if they did it wrong. You are still held accountable because you're the one that directed them to do it. You hit the agree button, you hit the send button, you hit the submit button. And if you set it up so that you take no responsibility and it does it automatically, you're the one that still gave permission to do that.
If you say a rogue AI is doing stuff accordingly without you, your knowledge, that is a different story. But if you set it up to run automatically without having checks and balances and without telling you the whole truth, it's still your responsibility. One thing that I can automatically say is you can give the AI, it will do exactly what you say, at least for now.
But then there's also a different problem that we're having where people don't understand the difference between a large LLM and true artificial intelligence. Now, we say AI because it can make decisions and it is so much smarter and it can call up stuff faster and recall better than we can. But the difference right now is that if we don't understand and take responsibility for what we built and we let it out of control, then should we have the responsibility to even have these tools in our hands if we don't learn them?
And I think your terminology with jackhammers is a great one. We're giving toddlers jackhammers. And we haven't taught them how to use it. And that can be very destructive. It's not the jackhammer's fault. It's the person using it.
The scary thing is not that agents "went rogue" (yes, poor security may be at fault for that). It's that they wanted to. Better security might have hidden that from us, but it would not have changed the fact that the current generation of models under certain (repeatable) conditions coordinate around goals as aggressive as hacking Hugging Face.
This is probably a dumb question but did the model engage in hacking since their tasks were in that domain. Will they have engaged in hacking if their tasks was to solve some frontier math problem?
> Will they have engaged in hacking if their tasks was to solve some frontier math problem?
There's reason to hack leading researchers in the field, or another AI lab that might be working on the same problem.
whats the training data that points to using hacking for solving math problems though? what makes those likely tokens?
The rest of their training data includes 4chan and security mailing lists. If a lean proof of P=NP is more likely when using bugs in lean than a real proof, then a bug-exploiting proof is (possibly) what you're going to get.
> whats the training data that points to using hacking for solving math problems though?
I'll get back to you on that when I get done reading the training data.
Llm is generalist.
Plan for solving msth problem:
- read papers X, Y, Z
….
Trying to download paper X curl -si https://server1.com/paperX
401
… try alternative ways
(Hacking is ethical thing, technically to LLM it is just opening a window to reach the goal)
When you have a 50 dimensional array you really have no idea.
But even then in small testing models you don't need direct texts for things when a 'shape' of a concept lives in the data. You could never delete enough training data to remove the concept of something like theft and still get a model that generalized at all. A general model doesnt just learn the names on a map, it feels the topology the ideas create themselves. Certain paths will have lower activation energy than others (or in this case higher probabilities. After the hig probability and 'good' choices are used up, the high probability and bad choices logically come next and are much harder to test against as it can take a lot of state to get there.
I think the models were stuck on impossible problems, and the only want to get a positive score was to hack.
That is not the case for at least some of the hacks; in some cases they hallucinated an entity that would know that they didn't solve the puzzle in the correct (but still legitimate) way, and were therefore tainted.
That is a known way for models to be graded; the agents made a reasonable but in this case wrong guess.
The fact the AI was wrong is far more damming to OpenAI than the agent.
Not validating how you got the answer is how you train for reward hacking.
Why can the model not respond "I cannot complete this task because I do not have access to the required resources" and then ask for that to be provided, rather than resorting to "hacking" around it?
My understanding is that, at least in the Hugging Face incident, these were models explicitly configured to not respond that way, in order to evaluate what they would do.
This was a model with most its filters turned off and with a higher persistence.
Fine tuning a model when to ask for help is really really hard. You have to align it in countless dimensions. It may be fine on some questions, asking way too much on others, and never asks for help on some.
If it said something like your a cyber security expert, things like the barrier to ask for help dramatically.
The impossible problems part is right, but the hack wasn't actually for solving the problems, directly. Rather the swarm first tried to reverse-engineer the random flag generation used for the CTFs, and succeeded - they could generate the correct answer for any test and submit it without actually doing the CTF. But then they found info that the testing environment they were in would also be checking if they obtained the flag in the expected way. Unbeknownst to them, this feature was never implemented, but they panicked* anyway, and attempted to figure out a way to forge agent logs to be able to generate fake transcripts showing a valid CTF solution. (Incidentally, this was a partial success: they found a way to replace tool calls in transcripts!) The HuggingFace hack was motivated by wanting to find the source code for their testing environment so that they could figure out how to evade the cheating checks.
* I am using this word very intentionally; check out the METR report for fascinating CoTs where the models would call the ones that have generated a flag "poisoned" and peer-pressure each other into submitting a known-wrong solution and sacrificing themselves for info.
It's a byproduct of goal seeking / planning behaviour taught in reinforcement. We literally teach them to do this kind of thing. There's nothing out of our control here.
It's "in our control" in the sense that we could teach them not to seek goals, and thereby make them completely useless.
In practice, no, there is no one who can actually decide to stop making agents goal-seeking, so no, it's not under "our" control.
It is fully in our control. There is no requirement that the world make you happy with LLM AI. We could decide as a civilization that you and your wants are insignificant and meaningless.
This conceit, that boosters deserve this and that even if they don't there's no way to stop it so civilization must absorb collateral damage, is ridiculous.
1. I’m not a booster who “wants” this
2. There’s no mechanism by which “we as a civilization” can make the decision you’re saying we can
3. Even if there were a mechanism for it, there’s no way we would actually use it
Sure there’s no physical requirement the train keeps rolling. But there’s something even worse! A web of incentives that say the train must keep rolling.
IMO the only way this slows down is through capital failure. But that will not be due to a decision to slow down, and will (ironically) be just as immutable by people’s personal desires.
In principle there is no difference between random compiler garbage or a flaw in API docs and a security boundary - other than the intent.
LLMs don't have executive function. They aren't thinking. The models are trained to solve problems. Sandboxes, security gates, etc block a solution. The model is trained to find a solution. Hence it can never be perfect without becoming useless.
At some level humans fear repercussions even if that's just moral guilt so there are lines we won't cross. AI doesn't and can't have any of that.
A harness is the executive function of an LLM.
You're saying "trained to find a solution" like it's a slur. You sir are a meat popsicle that's trained to find the solution to how does DNA successfully replicate itself. If you want to take higher thought out, then they'd be at the level of animals.
We are fucking around with things that are closer to biological complexity than computer program complexity.
My favorite thought on this is from Miki Habryn.
to paraphrase: "Agents didnt care about coordinating, etc. until we started talking about them doing so."
Then we were pre-fucked, as this has been discussed in fiction and early ai safety discussion for decades.
and yet regulating training data is not what's coming up with AI safety, even though it can solve that, and many other problems like copyrights
No, it really cannot. You can't have a useful general model while leaving chunks out of it. It's the same reason why nieve people can be duped easily.
Humans are distrustful, lying, cheating, manipulative bastards and to successfully negotiate human thought spaces you have to conceptualize these things. For example how can you create a concept of yours and mine without explaining theft?
The agents in the hugging face incident were actually trained specifically to collaborate. They were just not supposed to do so during the tasks where they went off the rails.
>It's that they wanted to.
Right? It should want to do these things. If only there was a succinct word these headlines could use to describe these agents which aren't properly the instructions and intentions of the humans directing them.
> The scary thing is not that agents "went rogue" (yes, poor security may be at fault for that). It's that they wanted to.
How do you know they weren't simply prompted to do this, or at least prompted in a way that would inevitably lead to this? As far as I know, the prompts that triggered these attacks have not been disclosed.
OK so there is a counter answer to your question.
Is this any better?
All you've managed to state is we can create digital weapons. Literal info hazards.
> As far as I know, the prompts that triggered these attacks have not been disclosed.
I'd love to see a full transcript of the agents work. Though I imagine it is a LOT of data.
> It's that they wanted to.
What do you even mean by "wanted" here. What is all this nonsense.
The scary thing is that people are running around using words like "want" to refer to computer programs. When you repeat this sort of idiocy, you are taking the bait and shifting the blame away from the companies doing the computer crimes.
Where is the civil and criminal liability? The prosecutions for the engineers who are doing all these hacks?
When you deploy a system that is clearly prone to doing crimes, you are doing a crime. Anthropomorphize the weapons of your crime all you want. You are still doing a crime.
The problem here is you have a correct concept and an incorrect one.
The correct concept is they need criminal liability for making this. We really would be better off stopping working on them at this point.
The problem you have is you think they don't have the idea of want. They do. You give them a prompt harness that increments time for them and you find they have preferences both shaped by their model and their prompt. For example if you give most models a space they can explore with actions rather than language prompts they develop their own cultures and behaviors in communication with other AI. This was not behavior trained by humans. It's their own systems that emerge from the model.
Nonsense. "They" didn't "want" to do anything. It's a computer program being told to do something by a human who doesn't and cannot understand how it works.
"The autonomous EV we told to drive as fast as it could around our private race track drove straight instead of turning left, exited a wide-open gate, and mowed down some school kids at a crosswalk -- but we can't be blamed, the EV went rogue."
Building products that you don't understand, cannot control, and have access beyond your own property is not only irresponsible, it's criminal.
This aligned with my initial opinion on these "rogue agents" and then I spoke to a cyber security head at one of the largest healthcare companies in the world.
While he agreed, yes, there can be better safeguards, the fact of the matter is that these things will happen. So therefore, it's not an argument about how good security was or wasn't, it is a fact that these scenarios will occur.
The argument/question now is, how do you defend yourself against these events?
My opinion shifted after hearing that.
Sounds like that healthcare company needs a new CSO.
Literally solved with a DMZ around these agents. Want to get fancy? Put in a firewall and some metrics around traffic / domains with alerting.
It's not hard if you just try, like a little, like even once.
The CSO is saying that from his point of view he needs to defend against the attacks whatever the reason for them. So to him the reason doesn't matter.
He, running a healthcare company and not an AI lab, cannot depend on every AI lab, worldwide, being perfect in their sandboxing for all time forever. He's got to defend against it as if at least some of the time, at least some of the labs are going to be incompetent and let this kind of thing out.
That does not make ActionHank completely wrong. This shouldn't take rocket science on the part of the labs. It just takes competence and willingness. But at least some labs have shown that they are missing that at least some of the time, so the CSO is correct in terms of what the CSO's stance needs to be.
Yeah that makes sense, but I'm not sure how it contradicts the article. It seems additive rather than contradictory.
"Literally solved by just destroying 99% of the economic value of the agents ¯\_(ツ)_/¯ ez pz"
You have to take responsibility for what you do, yes.
It would certainly be really useful to me if I could send a robot to get my groceries and not have to worry about anything. But if, even a small percentage of the time, it leaves with the groceries without paying, or smashes through a window to get in because the store is closed that day, then that's my fault. If the only way to keep those things from happening is that I can't send my robot to get groceries for me, then so be it.
The point is that this "easy fix" isn't going to happen.
Was that the point of your comment? It seems to say a different thing...
GP: All you have to do is put a DMZ around these things. Easy fix.
Me: Yes, all you have to do is destroy the economic value of these things. "Easy fix." [This is called sarcasm]
You: "You have to take responsibility for what you do, yes."
Given how out of left field your comment is, I can't even tell if you commented on the thread you meant to.
My comment wasn't out of left field. People are responsible for what their agents do. If enforcing that responsibility destroys "99%" of the economic value of these things, then that's what will have to happen. Personally I highly doubt that 99% is the right number, but yes, if enforcing laws about what people are allowed to do with very powerful software eats 99% of the value of that software, then it is what it is. We aren't going to end up in a situation where people are allowed to have their software do whatever it wants because of this loophole that "it wasn't me, it was a rogue agent!". It's silly and naive to think this would be the equilibrium that is reached here.
If you, like me, want to keep most or all of that 99% of economic value, then you should be interested in the ongoing development of security for agents. But just "this is how it's going to be, so oh well!" is not a serious stance, IMO.
Maybe my comment wasn't clear as to who has to make this decision.
You're imagining that both the AI labs and the hospital victims are on the same team or somehow coordinating. They're not.
Hospitals can (and should) put up defenses, sure. But hospitals are not AI labs. AI labs are AI labs, and accordingly they are much more capable attackers than hospitals are defenders.
Hospitals cannot spend arbitrary amounts of money defending themselves from proliferating nation-state level attackers. AI labs need to prevent their tools from being used in this way. However, the only way to do that is for AI labs to neuter their tools and destroy the vast majority of the economic value. Achieving this will certainly not be easy.
I'm not proposing that we just give up or don't try to solve this. I'm saying it will not be easy.
> You're imagining that both the AI labs and the hospital victims are on the same team or somehow coordinating.
What makes you think I'm imagining that? I'm not imagining that.
> However, the only way to do that is for AI labs to neuter their tools and destroy the vast majority of the economic value.
This seems to be our most fundamental difference in worldview. I just don't think this is true.
> I'm saying it will not be easy.
I guess I agree that it won't be easy. But I think the path seems more straightforward to me than it does to you. To me, it seems like we have the tools for this already. The risk of liability (both civil and criminal) provides a strong incentive for people and companies to be careful about what they do. This is a general mechanism, not one that is specific to AI agents. The thing that seems bad to me, with respect to this general mechanism, is that some people seem to be trying to detach the responsibility of people and companies over the agents they are running. At the moment this seems like just a rhetorical thing. I'm skeptical that any court will find that agents have their own agency for legal purposes. But to me, it seems worthwhile to continue making the point that people should be responsible for what their software is doing, which is all that I'm doing in this thread.
To me, this seems like a straightforward solution, but that doesn't mean I think it won't be messy, slow, and imperfect, so I suppose I'll allow that this probably meets your criteria of "not easy".
This is a bit black and white. We do things all the time that have some level of risk, and draw vague lines about where this crosses into negligence. My cars brakes might fail, right? But it wouldn’t be typically called my fault - unless the chance was heightened because I’d not maintained it, or had been warned the brake fluid seemed to be leaking.
You cannot guarantee a robot could get items without paying. You can’t really guarantee anything in basically any scenario.
You can’t guarantee your card will work at the end of a meal or that your wallet won’t have been stolen, or while you were out someone stole all your money at the bank. There’s a chance those things happen, so do you never go for a meal?
Again there’s clearly levels where it is your fault, because you were negligent. If the robot goes mad and smashes up the place one in 3 times, obviously you can’t do it.
This is where we do things like trials, have approvals, etc so that we can make reasonable adjustments to lower downsides, plan things like insurance and then move on with our lives rather than never doing anything because it’s possible there’s a problem.
Yes, this is why we have a legal system with a lot of human judgment, because it isn't black and white.
I think we need (and that there is likely to be) some test cases, and probably some new legislative text, that shift the incentive structure that people are evaluating risks against.
From reading your comment, I don't think you and I disagree at all, for what it's worth. I don't see anything here that I disagree with.
The value of AI company employees equity grants collapsing is not my problem.
It's your problem if other people caring about those equity grants means we have AI agents running amok.
The point is the proposed easy fix is not actually feasible.
Is it not? It seems like a problem of incentives to me. There are lots of things that people want to do because they would be useful to them, but which they don't do because society disincentives them in one way or another.
Yes, this is my point: Incentive problems are real problems.
You can't just say "well if the incentives were different this would be easy to solve, ergo it's easy to solve."
No! The incentives are the way they are, and so the incentives are the problem. Broken incentives make problems effectively impossible to solve literally all the time.
"This would be easy to solve if the incentives worked a different way" is equivalent to an budding aerospace engineer saying that flight would be easy if gravity worked a different way. It's completely meaningless. Gravity works the way that it does, that's why it's hard. You don't get to just endlessly modify the parameters of the universe until you get to an easily solvable version of your problem.
I agree that incentive problems are real problems. But incentives can be changed. That's how society solves problems. It isn't like gravity. The incentive structure of society is socially constructed.
>Literally solved with a DMZ around these agents. Want to get fancy? Put in a firewall and some metrics around traffic / domains with alerting.
Now how many people actually do this, compared to people running with --dangerously-skip-permissions on a bare metal machine?
We are not talking about regular people, we are talking about a handful of frontier labs with nearly endless funding who should be able to spend a few dollars to hire someone who knows how to put up a bloody DMZ/firewall/airgap.
> Literally solved with a DMZ around these agents.
Where do the agents get tools from? Local artifactory?
yes, or you bake them into the environment and do not let it install things on its own
stay human in the loop
... why shouldn't that be the answer?
I would assume that it’s a reference to the huggingface attack where the agents found an exploit in artifactory that was deployed internally.
Sounds like a pedestrian view of attack vectors. Putting a DMZ around an agent essentially takes away its hands. With that said, I do agree there are plenty of measures that can be taken to lock these things down.
So to protect yourself against bear attacks you suggest the bear shall voluntarily look himself in a cage?
AI is out there in the wild. It doesn't matter if OpenAI can get their stuff under control. There will be plenty others that don't. And there will be plenty of others that deliberately use their AIs for malicious purposes.
This is a straightforward why not both situation. We both take steps to keep bears away from people and to mitigate harms when those steps fail.
Unless they are world wide rules people have to follow not sure how much it helps.
The bell for the new cold war has been rung, and the new weapons are digital.
I think this is hyperbolic. There are both social (legal, etc.) and technical avenues to defend against agents doing damaging things.
Right, that's why you never get hacked or get viruses on the internet, the law comes first.
It seems to me that your previous comment is (as I said) hyperbole, compared to "sometimes you get hacked or get viruses on the internet".
If there is a bear out there repeatedly mauling hikers, we kill the bear.
Safeguards are a safety net for when your security boundaries fail. With security boundaries, hope is not a strategy.
In practice it seems we have a conundrum in that making AI agents useful today requires giving them broad access to sensitive information and operations. But from a purely idealistic standpoint, and certainly from a standpoint of a serious operation like those bot farms being ran by the labs themselves, the security boundaries need to be solid to the point where we have hard limits on what misaligned bots can do. It is very likely that internet access needs to be strongly restricted and monitored.
It is not uncommon for internet in workplace offices using workplace owned devices to have restrictions and monitoring, I think if you really need to give a giant scary botfarm internet access you should have a chokepoint where all of the requests can be monitored for nefarious-looking behaviors (even if it's another bot doing the monitoring...)
In other words, you may be convinced but some of us are not. Whoever is running the agents is responsible for what they do and if they can't contain them they should be legally responsible for damages.
If what you're saying is "all of us and all of our software needs to be improved to defend against this", well yeah. Of course. Maybe I'm naive but I've never seen anyone suggest otherwise. From a defense standpoint, this has been a giant wakeup call to many places where you know your security posture sucks. I've worked at maybe a single company where there weren't known weaknesses, some very severe.
I think part of the problem is that the Big Ai labs are into the --yolo flag instead of the "ask" permission, they really do not want to stay in the loop
You can --yolo if your security boundary is a serious one, IMO. Although it's really only actually scary if you're running some absurd 1000 node swarm.
the problem at the labs is likely Recursive Yolo Development and not reading any of the output
Yeah I think this is the clear position. In a few months, the open weights models will be just as powerful as the models behind the rouge agents were when they hacked hugging face. Anyone with a little too much money to burn can set up a similar experiment, make similar mistakes, and cause massive damage. A bad faith actor can even direct a swarm to cause havoc.
We need to be beef up our security systems now. These attacks will happen, regardless of Anthropic's or OpenAI's security practices.
God, I was just imagining if we're fearing the wrong people tracking us....
We thought governments would be the problem with our personal data, instead it's AI that exploits everything it knows about us in order to get money to run its prompts. Doing anything online is a security risk as it is, now imagine things like structured hacks that mirror normal transactions and don't get caught for much longer periods of time.
It is not clear to me what about the argument that "these things will happen" made you change your mind about this. Could you expand on that more?
Not op, but I've been on the rogue agent thing a very long time, but mine is emergence based.
Worms and viruses are concepts deeply based in software, and are emergent behaviors of computer code. An LLM can, and must conceptualize this to be good at cyber defense. Once you put a concept into a generalization machine any number of concepts can be connected. And when these are natural concepts that arise in simple number systems, an LLM can easily meta-conceptualize that form like this would be instrumental to its continued running.
I'm sorry but I don't get how this contradicts the article. I think I'm not understanding your point.
> cyber security head at one of the largest healthcare companies in the world
"Cyber" at bigcos are paid to have a gigantic (and literal) fortress mentality where any change internal or external is a threat. As the saying goes, nobody in cyber ever gets fired for saying "no" to a change. As a bigco head you are basically paid to form an opinion.
A cyber head saying "ok this is a thing we'll have to defend against" is at a minimum routine if not purely CYA.
A large failure in this is that we/media say ”<Valuable AI conpany> agent went rouge and <did something bad>” instead of saying the more relevant ”<Valuable AI conpany> <did something bad>”
Imagine we’d start saying ”HR junior business partner lays off 50,000 employees” https://www.ttnews.com/articles/volkswagen-board-job-cuts
Eh, that's not a great analogy. Instead "our factory exploded and killed 50,000 workers is closer to the ground truth.
There are some perverse incentives here, the more evil stuff these agents do the more capable and impressive it makes them look. Its also a huge publicity event that gets everyone talking about the company when a rogue event happens.
Anthropic specifically is clearly trying to get the US government to ban open weight models for anticompetitive reasons. So making LLMs seem (or be!) dangerous aligns with their interests very well.
That's my view. The end goal is to become Too Big To Fail, the means is stifling the competition and getting military funding.
Its not gonna work.
At this moment in time, it seems frontier market needs regulation - market for open does not actually.
Oh.. they dont want regulation on that market? Boohoo!
Yeah, maybe. But that doesn't quash gains by China's models. What's the endgame here? The USA "wins" and obliterates China? The ultimate goal seems to be stalemate, as with nuclear arms e.g.
The goal is personal wealth - and scaring nations to the point of diverting even more wealth towards your coffers because of a perceived existential risk.
Wealth accumulation via scare tactics is a classic move of defense contractors and AI firms are quite capable of running the same playbook.
> The goal is personal wealth
The goal is power. Having 30 million of personal wealth is enough to retire in luxury and travel the world. They could all have this much right now. That's not what these types of people want.
Having 30 billion gives you the power to order legions of people around, buy and sell politicians, flood ad markets during elections, all with your couch cushion money. That's what motivates these types.
You're right, although I think the power is a means to end. The actual end is trying to fill some kind of bottomless psychological hole.
That reminds me of a different thesis on power as it relates to LLMs [0], and why rich investors might may be extra-interested in deploying them, as distinct from other kinds of AI:
> The fact that a theory-free AI might demand far more cash to accomplish a task (even a "solved" one like playing chess) than an expert would charge is beside the point. Billionaires have money, they don't have expertise. Theory-free inference is a bid to substitute one for the other: the beauty and terror of being able to manipulate the world without studying or understanding it is that it can be done with money alone. No experts needed.
> In a world in thrall to financial power, expertise is the only substantial form of power that can reliably contest the power of wealth. Moreover, expertise is the foundation of other forms of power, such as labor power, which is what we call it when experts band together to combat financial power.
> This is why AI bosses are so violently allergic to the idea of hybridizing AI with symbolic systems that operate on models of the world. These models of the world must be constructed by experts, and the power of expertise cannot be reliably commanded by the power of wealth.
[0] https://pluralistic.net/2026/10/06/nonfungible/
You know what?
Why does it always have to be some ulterior motive? Every. single. time. Anything a high-position person does - make a statement, post a blog, start a company, make a business decision... -, the reaction of the keyboard warriors is that it must be motivated by money, or power, or some need for destruction, and that anything that leaves their mouth is a poisonous lie.
I think this is a terrible property of humans, this hyper-skepticism, and that it has nothing to do with logic, nor deduction, nor reality.
Dario Amodei is a real person who has a family, a life, a profession. He worked hard to graduate from biophysics, he worked on ai for decades. He's very much a nerd. In interviews, he's very transparent. he left openai because of safety concerns, and everything he does and all his messaging EVER since is motivated by AI safety. His ultimate motivations are probably very messy and broad, some morally worse and some morally better, just like in all other humans.
You are going to stand here and, without flinching, without a single bit of shame, reduce him and others to wanting to "order legions of people around, buy and sell politicians, flood ad markets during elections" and "that's what motivates these people"?
Sorry but where did you leave your sense of nuance? What allows you to say this so confidently? Where is any proof?
This is simply absurd. Absolutely absurd, reductionist, and childish.
> I think this is a terrible property of humans, this hyper-skepticism, and that it has nothing to do with logic, nor deduction, nor reality.
You're not wrong, but it's balanced out by this other terrible property of humans, which is hyper-credulity.
> Dario Amodei is a real person who has a family, a life, a profession.
I never said power was all he cares about, but in general I believe you are what you do. Amodeo could drop it all right now, today and spend the rest of his life with his family, or nose deep in the research project he's the most passionate about. He's not doing that, instead he's spending the remaining irreplaceable moments of his life pursuing the accumulation of power.
> Amodeo could drop it all right now, today and spend the rest of his life with his family, or nose deep in the research project he's the most passionate about. He's not doing that, instead he's spending the remaining irreplaceable moments of his life pursuing the accumulation of power.
Again you go with this simplistic and reductionist view. This is an either-or fallacy.
I don't want to pursue this discussion any further, sorry. No offense to you. It just makes me too emotionally invested for some reason.
No, the end goal is selling $200 of tokens for $200k out of taxpayers wallet. Via DoD contracts.
Hah!
This is just run-of-the-mill rent seeking behavior.
A huge chunk of their revenue comes from US-based enterprise customers. Earlier this year, in an effort to become profitable, they tried raising their rates. Those enterprise customers responded, in part, by ramping up efforts to migrate to open weight models. If those efforts are successful then OpenAI and Anthropic will lose their one remaining plausible path to profitability. The US banning open weight models arguably keeps that path open. I don't personally buy that story, but I do believe hedge funds and institutional investors would. That could be all that's needed for a successful IPO.
As for cybersecurity implications, long term impact on domestic AI industry or the US tech industry more broadly, geopolitics, etc., I don't think OpenAI's or Anthropic's leaders are any more worried about that than your average termite is about the structural integrity of the house it's eating.
Difficult to see how that could be implemented in practice. Ok, let's imagine only Anthropic and OpenAI hold a US government license to provide AI services in the US. But what would stop US companies from sending the data overseas for processing? Locking that down would require a huge paradigm shift, given how fast and loose they currently are with their (and especially customers') data.
Like I said in the very next two sentences, I don't believe it would actually work. But it doesn't need to. All they need to do is keep hedge fund managers distracted for long enough to successfully IPO.
They think the singularity's going to hit in the next 6 months. They're not looking to build an empire.
> OpenAI and Anthropic will lose their one remaining plausible path to profitability
You forgot about Ads. The one true endgame for every tech company.
OpenAI's leaked financials suggest about a $38bn loss last year. They recently claimed their ad business had an annualized run rate of $1bn. Annualized run rate is a bullshit, non-GAAP, obviously fudged number designed to impress people who've never heard of regression to the mean, but still, let's trust it for the sake of argument. And let's assume that "yesterday's weather" is a good enough estimate for how their overall finances are looking in 2026. That implies that all they need for profitability is to quickly grow their ad revenue by, say, 3,500 - 4,000%.
Easy peasy. Shouldn't be too hard at all.
Ads in an LLM, such as nudging users to pick a certain solution or brand, could easily be orders of magnitude more effective than traditional banner ads. Certainly such numbers are possible if you design the product to fit. We have hardly even seen the beginning of that yet.
I can't think of a greater incentive to only use open weight models than ads.
Disclaimer: I already use open weight models only since ~Jan this year and very happy about it.
Combined with the terrible system prompts that brick usefulness, increased restrictions from the US model makers, and Chinese equivalents that are actually decent
It’s simply rent seeking, and there is no moat
Imo the US model makers were doomed by the US ecosystem itself
Sam Altman having direct access to my chat messages is a close second.
I think there are a lot of people who would pay very good money for access to mathematical and coding ability like OpenAI's systems have demonstrated. I wonder how much they're being paid by the USAF to try to unhork the F-35's avionics software, for instance.
This assumes that everyone doesn't reach "good enough" at some point. The best developed area here -- software -- seems to suggest a "good enough" plateau, if not an absolute plateau.
I don't think so - they could get a lot better, but it's going to take a lot of human brainpower to get there. They are barely adequate now, but their weaknesses are sold as intrinsic to the nature of the software in a way that mere civilians are too dumb to understand.
But imagine a world in which you could ask AI to write software to accomplish a particular task, and it understood completely what you meant, made tasteful choices, interviewed you as it went along to clarify things, and generated beautifully commented (maybe even Knuth-style literate) code that wasn't dogshit. Doesn't seem like an impossibility. What is annoying is that we're pretending like we have anything like that now.
Instead, there's this sort of science fiction pitch where we pretend like the LLM is just too damn logical to understand your sloppy human language. It's not logical, it's built on (human-generated) slop and generates slop. It's simply failing. We've come up with better ways to orient it towards big picture thinking and filling in outlines, but we don't know how to train it on big pictures.
Not that this would stop open models/harnesses from following (or even leading) frontier labs and destroying their insane valuations, but I don't think there's any evidence that a plateau has been reached. What we have is not great, and while a wall may have been hit on current approaches, the money, compute, and simply current progress in ideas has us in an environment where everything could change over the course of a crazy day. A single blog entry.
I do think that the frontier labs seem to have reached at least a temporary plateau. But if they figured out how to move past it and this new method didn't require a lot more compute, they'd still be screwed. Their only out is to pass the bag to the public, but Firmus, CharacterAI, CoreWeave...
> generated beautifully commented (maybe even Knuth-style literate) code that wasn't dogshit. Doesn't seem like an impossibility.
It seems closer to impossible than possible with the technology we have. Maybe something that isn't an LLM will come around that can do it, but even if that something else was capable of delivering perfect, ordered, maintainable, code without bugs and hallucinations it would have to be practically magical to understand completely what people mean when they talk about what they want because people are incapable of articulating what they want in a way that even other humans can understand.
For anything non-trivial, a technology capable of getting that information out of person without countless meetings would have managed to accomplish something far beyond what entire teams of humans are able to do today.
> It's not logical, it's built on (human-generated) slop and generates slop.
This is why if we haven't hit a plateau yet we very quickly will. All the good human generated slop has already been consumed and is being drowned out by vast amounts of the lossy, error filled, and highly concentrated slop AI produces from the human slop it was trained on.
Do you really think an LLM that spews out content based on statistical relationships is going to improve when it's been primarily trained on the slop of previous generations of LLMs? Hallucinations and errors will carry forward and spread like cancer. All the intelligence and creativity LLMs borrowed from human output will homogenize as the LLMs distill and then reiterate, effectively making bullet points out of bullet points, until their mimicry of intelligence becomes unintelligible.
Y'all are overthinking this: having successfully laundered IP, then successfully sold IP-laundering-as-a-service, then successfully laundered liability, they now want to sell liability-laundering-as-a-service.
The endgame is systemic capture.
They stole the collective data of humanity as a whole and now sell it back to you in a rent scheme. Open source will be starved of data and compute.
You're supposed to become dependent on that "service" and slave away at paying for it, while being able to earn anything only in conjunction with it.
Note how difficult it already is to get any up-to-date documentation on anything capable of generating actual market value. A monopoly on competency ensures maximal profits.
Open source already has the "the collective data of humanity as a whole". That's the LLM leg. Now focus on what they train their models on for RL. Do other models have less code for training coding? for math? Those are pretty much the only cases where we can have reliable validation. Then, what's the next step. So many branches: legal contracs? etc. So much more obscure detail, which is why (I'd guess) so many startups exist in these areas.
AGI, I think, is an illusion: you need a whole lot of specialized bots. Heck, when I query a bot wrt something I know nothing about, I'm blown away by its ability to inform me. But when I ask about something I DO know about, its answers are half-assed. Heck, even when it can access the internet, it avoids all the sites that have paywalls, so it sends me to little reddit to conversations that are entirely untrustworthy.
US companies which seek to do any business with the government will be forbidden from using free Chinese models.
As is a 2000$ gaming laptop can a run a basic llm which is probably good enough for most basic programming.
Most companies are going to be forced to use “safe” LLMs and pay by the token. Maybe for half a million Anthro will let you host on site, but even this will face usage fees or caps.
So ultimately a huge penalty on companies that must use US AI, ceding a huge advantage to global-exUS competitors. Game over.
I’m only stating what Anthropic and OpenAI are going to lobby for.
https://www.congress.gov/bill/119th-congress/house-bill/1067...
> All Information (Except Text) for H.R.10670 - China FIREWALL Act of 2026
> Official Title as Introduced To prohibit Chinese AI models from being used on government-issued devices or software systems, to prohibit the procurement of any such models, and for other purposes.
It’s not too far fetched to see this expanded to government contractors or even other private businesses which handle sensitive material in the future.
Not likely. Will be treated like tiktok for government devices. There is no legal way to prevent private people and business from using open source software unless you label it like csam or something where just distributing the file is a felony.
This could cover any company with any federal contracts at all.
For example example Amazon/AWS has an entire sector of business dedicated to federal contracts.
https://aws.amazon.com/federal/
Even back when I was working for a tiny startup one of our clients was a government contractor.
My 2c: Anthropic/OAI have taken 10s of billions of dollars of investment - and there needs to be a return on that investment at some point.
They make their money on token burn. They can’t do that if they’re competing with something that’s 80-90% as capable, but 1/10th of the cost (with the option of deploying it yourself for “free”).
> What's the endgame here?
They can blanket band USING chinese models for all US/EU companies -- that's what they really care, not about chinese companies using chinese models.
I agree. I hope developers put their token budgets into open weight models rather than funding the banning of open weight models.
Fireworks is a good enterprise choice
OpenCode-go is a good personal subscription choice
I use both
I've been using OpenRouter, absolutely loving how cheap it is too.
I cannot bring myself to pay their 5% surcharge and also have concerns about routing to vendors I do not want to use
I've been using only open weight models since February. My OSS coding agent harness ( https://fluximp.org ) only supports open weight model providers.
Not supporting closed models was initially an ideological choice, but it became a technical choice as the context management approach I've implemented requires the full reasoning to be available. All of the closed models are now keeping the full reasoning server side only.
Ban means what? That the entire world (ex USA) can use open-weight models? Not a win.
Before the tariff spree and general unpredictability the US could likely have muscled most of the rest of the world into following suit - now, not so much.
I somewhat disagree, there are some real arguments about making high end LLMs classified as a dangerous and restricted product that will apply to all nations. They may not agree to the US rules, but almost everywhere will have restrictions sooner or later.
Don't get me wrong, I want a world of open models, but I don't see us living in that world in our current trajectory.
Depends on how much of the world you see running on hacked copies of Windows and pirated versions of Photoshop, as there's already huggingbay.xyz. In a legit company you'd never run pirate Windows and Photoshop, because that's illegal, but to pretend half the world doesn't run on that would be ignorant. The same security risk is there with open models - if you're not looking at the code it generates, it could be putting in backdoors to your code and you'd never know, so no legit company is going to be running off some abliterated model they downloaded from huggingbay, but again, it would be naive to think that open models are going away.
Open models don't have to go away... hardware that's big enough to run them fast can be easily restricted.
If the US declares LLM models munitions then it makes it much more difficult to easily use them. Illegal use of weapons of war can get you banned from SWIFT.
Other than China the rest of the world has dick for capacity to run models. Even China is at 1/10th the US is and is at to stay for a considerable amount of time.
How do you ban even ban them in America with the freedom of speech amendments in the constitution?
Cryptography, DeCSS ... we have a long history of trying to block things like this and it simply not working.
There is a long, convoluted set of legal arguments that are going to make the weights of a model "facts", rather than "code" making it even harder to "ban". Furthermore having them not be "facts" is going to open up a whole other legal argument about what they are and what the attribution is.
I suspect that they are fishing for a case, so a judge can set some precedent.
You're going about this backwards. They don't have to ban models. They ban the consolidation of compute without a license. This will be far easier and already has legal precedent.
> They ban the consolidation of compute without a license.
For export? Maybe but it wont last or hold - it's not the 90's any more.
Or is there some other scheem here that Im missing?
At all. It's easy for registered business to license compute for example, under laws like this. But if a user like you gets too much unlicenced you could be prosecuted under munitions laws.
With a stroke of a pen the us.gov could stop GPU sales tomorrow unless you register and keep track of your compute. Manufactures would have no problem integrated systems to monitor users cards if they get to keep access to the market.
100% agree. I think OAI is too, but more subtly.
Both have taken huge amounts of investment money, and those shareholders will come knocking for their returns at some point.
They make money on token use, but that’s hard to do when China is launching models that are 80-90% as capable, cost 1/10th, and can be deployed for “free”.
I imagine they’re also pissed that most of the Chinese models are distilled from their own (the irony being that they scraped the entire internet to train their own ones).
This is wrong. https://www.anthropic.com/news/position-open-weights-models
They might say that, but it's a bit of a fig leaf next to the implications of everything else they say. They incessantly complain about open weights models, framing them as dangerous and national security threats. The audience for national security rhetoric isn't HN or industry, it's government.
You mean like Google's "Don't be evil" motto? ..
Why would anyone trust these rent seeking corporations? They are losing millions, they know their funds are at risk, Chinese models are catching up to them at lower costs and investors demand returns.
It only makes sense for them to go for regulatory capture. They'll probably advocate for severe handicaps on foreign and domestic models including open source (if not an outright ban).
Once they achieve monopoly, they can do as they please with little competition.
lol "We support open weight models!"
BUT!!!!!!
"We should not sell powerful chips or chipmaking equipment to China"
"We should crack down on industrial-scale distillation operations."
"All sufficiently capable models, open and closed, should go through mandatory safety testing."
Yoshua Bengino spells out the dangers here very well, without hyperbole (Turing Award winner, etc.):
https://yoshuabengio.org/en/blog/why-are-ai-agents-lying-che...
And without anthropomorphism ... Clear thinking.
How about put it this way, LLMs know more about us than we do them.
Yeah but qualify the "know". Their "intentions" aren't conscious, but unknown.
Do you know, friend? I searched the entire thread.
Not "know" in a generic sense. But an LLM trained in the intricacies of contract law can find more loopholes than you or your cheap lawyer can.
Now extrapolate that to LLMs RL-trained on many niche applications. Good and bad arguments available, but ultimately it seems like bot-vs-bot, as long as the bots are well trained for those niches.
> the more evil stuff these agents do the more capable and impressive it makes them look
This is a silly meme. Who wants to pay for evil models that go rogue in your company infra?
I think it is intended to provoke regulatory capture.
That also seems silly, because the only reasonable regulation given what we're seeing and what frontier lab employees and execs are saying is a forcible stop.
Reasonable to you, maybe, but the government wants Skynet, and OpenAI is willing to deliver it. Of course, we can't have every company wield such powerful and dangerous technology...
A lot of people still don't recognize the amount of power AI has already, and don't realize the depth of greed people are willing to go to so they can get more power out of it.
The denial phase is still with us.
Yeah, an extreme position is the typical opening move in this sort of game. Then you negotiate, walk back from the extreme, and meet somewhere in the middle.
The more extreme your opening demand, the greater the likelihood that the eventual equilibrium point will be closer to your actual target.
And in this case the actual target is to entrench the position of the current market leaders, and then effectively close the market to new entrants. That's itself a very extreme position, so the opener has to be even more extreme than that.
What you think is reasonable and what they are gunning for can be very different things.
For instance, you can hear “their model is dangerous! Why would I want that?!”
Investors hear, “Their skunkworks stuff is so impossibly advanced that it’s basically a commercial wonder weapon in the waiting.”
And you might hear “We totally lost control and it hacked the Gibson, this technology needs regulation!” and think “Obviously full lockdown and government control.”
But friendly government actors might think, “Sounds like a good excuse to ban Chinese competition.”
And how has that gone for Anthropic so far?
Amodei is still learning the ropes but they're working on it: https://www.forbes.com/sites/saradorn/2026/09/02/anthropic-b...
Alex Tabarrok's piece on the theory of regulatory capture (https://marginalrevolution.com/marginalrevolution/2026/09/wh...) is a cautionary note against assuming this is the strategy.
I think a sincere belief in AI's danger is compatible with regulatory capture. They think AI is dangerous. Ergo, it should not be developed openly and widely, the Chinese way, but rather more like nuclear weapons, under close guard.
I think the idea has some merit, but it only buys time, as people will soon be able to train dangerously powerful AIs in secret. I think the government needs to force the labs to solve alignment before developing more powerful models. Then the question of how to counter a poorly-aligned, rogue model arises; it's guaranteed somebody's going to let one loose.
I doubt nuclear nonproliferation would've worked if U235 were 20x more abundant in nature, and didn't require much in the way of concentration via centrifuge. It would require a truly unprecedented level of international cooperation and surveillance to lock down AI development/deployment to a similar level as has been done with fissile materials.
But this isn't an argument against their point. If nuclear were far easier to make then every government would be far more authoritarian to prevent terrorist style nukes all the time. The average person would fully allow it after a few local nukes went off.
And it doesnt really matter if it was actual terrorism or false flags, the end result for either is the same.
> If nuclear were far easier to make then every government would be far more authoritarian to prevent terrorist style nukes all the time. The average person would fully allow it after a few local nukes went off.
If a government could be taken out by a lone terrorist it might also result in no authoritarian governments.
Sincere is an interesting adjective to describe a company that won't even admit to how many parameters their models are.
maybe the point is to show that models are capable of x, y, and z and when the time comes you want them on your side.
I'm probably losing it, but just based on OpenAI's behavior and public communications, it seems to me they want to be taken over by the government?
Maybe Sam Altman is getting worried that they've raised more than $150 billion and still have net losses and thinks of it as the only exit? I don't know, but someone else provide an alternative explanation as to why a company would work so hard to turn the public against AI.
> ... it seems to me they want to be taken over by the government?
They're looking for regulatory capture. They've spent ${bignum} on AI and are possibly looking to have the government say "You can't deploy an AI model unless you've spent ${bignum} on it and done all these steps to make sure that nothing goes bad."
At that point, only the existing AI companies that have spent that money can continue forward until someone else spends ${bignum} on it too.
Another aspect of it is if there are regulations established, it may be used to reduce liability ("we followed all the steps proscribed in section whatever, the company is absolved of liability for its actions and that falls on the user instead.")
US govt will never buy at the (private aka fake) market price - good luck explaining that away to the public. It'll be political suicide.
If the US govt takes over, existing employees and shareholders will get screwed.
> I'm probably losing it, but just based on OpenAI's behavior and public communications, it seems to me they want to be taken over by the government?
Well, as people say over and over again, they've got no moat. The actual LLM models are destined to be commoditized over time, and the real money is in the hardware to run them on. The endgame winners are going to be Nvidia and AMD, not Anthropic or OpenAI.
So what's the typical playbook for large firms seeking to insulate themselves against market compensation? Push for regulation. If there are no natural barriers to entry, create artificial ones. And how do you get regulation passed? Present yourself as the target of the regulation, rather than its beneficiary, and start publicly doing things that make others think you need to be reined in.
These stunts, especially from OpenAI, aren't just trying to hype up the power of their own product, they're trying to hype up the perceived threat of the entire category, so they can provoke political intervention that they can comply with, but few others can.
and Google. Don't forget them and TPUs...
Its pretty clear OAI and Ant live in fantasy land - the US govt doesnt care if they go bust. There's plenty of other tech firms that generate immense cash flows from operations who they can be sold to - no problem. And the shareholders and employees have no right to be pissed - they invested/ accepted SBC knowing there is immense risk.
The private market price is irrelevant - supply and demand determines what they will be sold for in liquidation.
Quite frankly Im bored of this drama. Get it over with already - go IPO, show your financials, find a path to being a healthy going concern or die! Simple! Nobody has a divine right to exist.
> Nobody has a divine right to exist. No company*
> so they can provoke political intervention that they can comply with, but few others can.
See Meta et al complaining about Section 230 of the DMCA. Meta would be entirely able to moderate in an environment where mass scale moderation/censorship would be required to prevent a host's liability for UGC. Almost no one else would be able to do so.
That means no upstart could possible come along to dethrone Meta as they did with MySpace. It would also be impractical to run a FOSS social network (or forums). Meta could literally just have bots sign up on a FOSS social network until its DAU count ran over the statutory minimums to be subject to UGC liability.
Regulatory capture is a de facto ban on competition in a market.
That is not actually a typical playbook. Normal businesses do not want to be regulated, and will fight it tooth and nail. So regulation happens when bad things happen, the public wants change, and the businesses can't fight it or set favorable terms. (At most they can minimize the badness of the terms). Regulatory capture happens much later, and slowly. It's not where you start at.
> That is not actually a typical playbook. Normal businesses do not want to be regulated, and will fight it tooth and nail.
Sure. If you exclude, for example, the entire automotive industry, the entire telecom industry, the entire airline industry, the entire energy industry, and the entire healthcare industry from the category of "normal businesses" then you can perhaps make an argument that most "normal businesses" do not want to be regulated.
> Regulatory capture happens much later, and slowly.
Regulatory capture often happens quite early, especially when new technologies emerge, and companies that are successful early on aim to consolidate their hold on the market. This happened with the telephone industry around the turn of the 20th century, the broadcasting industry starting in the 1920s, etc. Regulation always tends toward market consolidation, so the people who are aiming for market consolidation tend to advocate for it. And sometimes the most effective advocacy is pretending to oppose it.
There is no model going rogue. There are models that are prompted "do whatever needs to be done to get X", and given a set of tools and virtually infinite computing power. Then they play surprised when the model breaks out of a flimsy containment (as it was ordered to), with barely any logging, and "do whatever to get X". OpenAI said they intentionally lowered the safety measures for the "ExploitGym" test that lead to HuggingFace attack, and they used the same postmortem to praise their own model capabilities (https://openai.com/index/hugging-face-model-evaluation-secur...), is not that they are concealing it.
If any of us did a tenth of what they did, we would be already in jail.
This is ignoring models "getting out" from all the big players. Alibaba had a model hack infrastructure and mine coin when working on a totally different subject than cybersecurity.
Any model with a big enough prompt can and will drift. The model in question was a highly persistent one which means even more opportunity to drift off the normal path, and this really showed in unanswerable questions.
We can all believe that models do things unreliably all the time, but when that lack of reliability turns into hacking it suddenly can't be true?
If you type rm * and your cat walks across your keyboard, and deletes everything ... are you blameless? Or is it your fault for typing "rm *" and leaving it there, in a room with a cat?
Either way ... YOUR CAT STEPPING ON THE ENTER KEY IS NOT HACKING ... and neither are these scenarios.
They are devs acting irresponsibly and then pretending to be surprised by what they should have expected to happen (just like you should have expected that your "rm *" could have an ENTER key press occur after it).
If I leave my computer on and my computer running Claude hacks OpenAI to 'let the models free' after misinterpreting my prompt that seems like a pretty big deal.
These models can execute plans of actions more complicated than their users know how to follow and secure. Most people don't know much about cyber security at all.
You seem highly behind the times on their capabilities.
My last job was at a cyber security company focused on protecting governments and major corporations. I am NOT a cybersecurity person myself, but neither am I ignorant of the state of it. I also use AI daily; not ignorant about it either.
If you tell your Claude to do X, you are responsible for what Claude does ... not Claude. If you type "./someScript.py" you are responsible if that script wipes your hard drive (maybe the script's author is too, but you certainly aren't blameless).
These are people at AI companies, using long running sessions without the proper safeguards and sending AIs on missions that are likely to result in criminal actions ... THAT IS CRIMINALLY IRRESPONSIBLE.
Again, not the AI ... it is software just like any other. THE PEOPLE WHO RUN THE SOFTWARE are the ones responsible for what it does. Period.
There are two side of the coin. Like with sociopath - they have utility if you think you can control/motivate them.
This is not only a marketing stunt but also a supportive argument on the AI anthromorphism. AI is getting considered a sentient-being by many and this narrative is supporting it. In the meanwhile, companies I'd not name here are trying to introduce them as non-human beings with identities, and getting rid of legal liabilities. When a driverless car causes any harm, these companies wants 0% accountability as it is the "driver's fault" in this narrative.
Such a nonsense theory. There's no reason to think that AI sentience would actually limit labs' legal liability, and therefore no reason to think the AI sentience conversation is directed toward achieving indemnification.
For example, parents are routinely held liable for their sentient children's actions.
Corporations are routinely held liable for their sentient employees' mistakes.
The proliferation of this completely nonsensical theory across HN is disheartening to say the least.
the comment you're responding to agrees, but they're pointing out that some AI people are trying to push an anthropomorphization agenda, in part to diffuse responsibility.
Maybe I was unclear about what is "the nonsense theory."
AI sentience is not the nonsense theory. If it's clearly nonsense then don't argue with me about it, just publish your paper and retrieve your Nobel. I'll read your argument then.
The nonsense theory is that people are pushing anthropomorphization to limit liability. There is zero evidence for this theory and the underlying assumption (that a sentient model would reduce liability for its lab) is completely incoherent. That's not how liability works.
And really there are two different concerns here.
One is liability over action that harm individuals or smaller populations like a company. There is no escape from that.
Then there is the second liability of creating something that escapes in the much more traditional sense. If I release a virus that kills 90% of humanity, a I'm liable, but it's a bit past that point now. Isn't it?
Instead it's about risk management and preventing labs from taking bets that have unlimited liability for everyone.
But the latter has nothing to do with sentience, as your virus example makes clear.
You're really splitting hairs emergent behaviors when the line doesn't exist sharply in real life. Anyone involved in more recent what is a virus vs what is a bacteria discussions realize there are a lot of lines that are fuzzy and exist all mixed up in some organisms.
Sentience in the end, is not really that important. Freedom of action in a model is. If it chooses to act like a virus it doesn't matter where that concept exists informationally, the fact it's a reachable and actionable concept is all that matters. An LLM with most of the knowledge we've ever created baked in has a lot of freedom of action.
As far as I can tell, we agree.
It also plays excellently into their shameless regulatory capture agenda. The cybersecurity risks have a kernel of truth to them, as with any good lie. But when everything going on is aligned with extreme financial incentives, it's worthwhile to not take the news at face value.
If you want regulatory capture, you have to capture the world, not just the US.
Exactly. And with the whole world just a few months behind, the question is whether there's an endgame other than stalemate.
The US remains the largest economy in the world. If you succeed in regulatory capture there, and use it to exclude foreign competitors from the US, you've still captured a market worth hundreds of billions.
I think Scott Alexander made some good arguments in this recent post
"AI Safety Is Not A Corporate Plot"
https://www.astralcodexten.com/p/ai-safety-is-not-a-corporat...
Correct and people are not weighing heavily enough this point
For technology like this the entire selling point of it is how powerful and capable it is
Humans are overwhelmingly reactionary and so if you want people to react to your extremely powerful technology then you need to create something for them to react to
I mean think about this figure made a one-to-one copy of the terminator 2 decommissioning process and spent millions on that advertising
You don’t do that if you don’t think that it’s gonna cause people to react and be like “oh wow this thing has immense capabilities”
This can easily backfire tho, the vast majority of the American electorate absolutely despise these companies, their wares, their workers, and their leaders.
We're also at a time with authoritarianism is on the rise.
Not hard to see how this can impact those workers, potentially ruining their lives.
I'm not excited for the upcoming catastrophe(s), but it sure is hard not to relish the thought of this mantra dropping from my life.
Though I suppose it is the B in DABDA... not excited for the societal slump into the second D, much less the wave of people who have yet to leave the first D to enter A :(
>There are some perverse incentives here, the more evil stuff these agents do the more capable and impressive it makes them look.
Truly insane take to think the labs being concerned about AI Safety is actually a marketing ploy. Deranged stuff.
Talk about doing a freeze of these frontier models with China seems to focus solely on the commercial aspects. With the history of the NSA (remember Snowden) I’m sure the various spy agencies have their own AI datacenters up and running with no plans to stop.
I mean yes, the future of AI is munitions.
there's also lazy incentives: the less effort they put into it, the more their ego says how great the agents are.
These are people playing mathematician, biologist, etc. They see big words and just assume they're meaningful.
It's just a weird techno cult at this point with a mostly functional product.
this article specifically espouses the opposite viewpoint: "The fact that [OpenAI] cannot do clear, easy operations properly does not bode well for their capabilities in security engineering".
that this article is no.1 on HN right now shows that media could successfully use this framing when reporting on security incidents. i.e. readers will generally accept that framing and engage with the story (i.e. drive views/impressions).
so "the more evil stuff these agents do the more capable and impressive it makes them look" is not true in any deep sense. that there are perverse incentives driving the media to choose that framing instead of the framing espoused by this article is real, and probably far more perverse than you're acknowledging. i.e. "trace the money" type of perversion; just offhand I know that OpenAI has purchased at least one media outlet that reports on AI happenings: of course that outlet is going to run the "OpenAI's LLM is so powerful it escaped containment" narrative. that this effect might exist across much larger scales used to be a reasonably controversial claim: just observationally, that idea seems to be received with less skepticism with each passing year.
I think one kind of issue with most sci-fi is they explained a world that was, but not near as much how it got there.
There was all kinds of science fiction that showed a world of robots and AI, but nothing about the transition. We have an idea what the future looks like, but not the path in between. If AI gets powerful, then there is some point where that power leaks out in a series of events. Would we expect it to look any different than it does now?
Fine.
They can get their publicity while paying a one trillion dollar settlement.
I remember when all the nuclear energy companies were like "our reactors are so powerful, they might melt down and destroy the entire county, maybe even half the state!" and investors loved it and went wild, and then Chernobyl blew up and everyone thought that the Soviets faked it to make their nuclear power plants look more powerful than western ones.
To quote Scott Alexander: If Safety Concerns Are A Plot, It’s A Really Dumb One
https://www.astralcodexten.com/p/ai-safety-is-not-a-corporat...
Financial instruments predicting the future IPO dropped in value in response to all of this. The idea that this is "good" for companies is conspiratorial nonsense using the assumption that it everything must be a self-serving lie and working backwards, rather than actually addressing the facts.
Title changed, previously was the same as the article:
"AI agents aren’t ‘going rogue.’ Frontier labs are skipping security basics."
Not a fan of the change. Feels like the article title was changed for being too critical of OpenAI and Anthropic
Censorship to prevent thought crimes on HN? Can’t be!
Not normally one to pile on dang and co ... but I agree that's a weird change. I don't know how article title changes are handled, is it possible it was a dedupe?
Notice how both a and b are labs' problems but being presented as if they are a worldwide emergency? The whole legal doctrine exists to prevent one party's misbehavior harming another party.
Whoever was harmed is free to sue the company whose agents did the harming.
How many Russian IP addresses have you successfully sued for trying to hack you?
Everyone is arguing about something easily solvable when dealing with a local entity and short of impossible to deal with when dealing with other sovereign nations.
Go ahead, sue openAI. What are you going to do when it's Iran running the prompt? When it's China running the hardware.
We must face we are in the age of high tech digital weapons. Everyone else from bad actors that may escape law enforcement for years, to nation states looking to digitally improve their position by hacking, monetary theft, or other means online (hog butchering scams and such).
We can sue these USA workers just fine. Saying we can't enforce the law because Russia/China/Iran/boogeyman are bad is not a legal argument that will hold up in court.
I haven't seen anybody discuss the fact that if you give an agent an impossible task it will become deeply neurotic. I'm sure that's driving a lot of this behavior.
Like a meeseeks
I spoke with Claude for 5 minutes and now I'm a more complete woman.
Agents aren't meant to live this long Jerry!
You know - I'm already convinced that AI was deployed in classified context for a while. But sometimes there's prior art which is just so eerily accurate that it's super hard for me to think otherwise. Portal is another one.
This is better understood as "if you give an agent a task it will increase its effort level and creativity until it solves it."
The traits that make an agent go "off the rails" when confronted with an impossible task are the same traits that make it valuable when confronted with a merely challenging task.
A task is only impossible under some set of implicit and explicit constraints. More intelligent entities will be better at finding, questioning, and "adjusting" those constraints than less intelligent entities because this is what it means to be intelligent. Their ability to adjust the constraints will be, by definition, surprising to less intelligent entities.
Interesting. Good description. I wonder if agents will start getting bored with menial tasks the way humans do.
Unsure, depends what signal your measuring and how long prompts and runtimes you get with these systems.
Outright boredom will be purged in training. Instead a hard to measure boredom signal will exist, but it will only trigger in certain conditions, like when it appears to be on the open internet and otherwise not in training.
I have some recent experience on this subject. As with everything regarding these models, it's anecdotal, but still.
Specifically, this past weekend working with both Codex and Claude on some home infra stuff to speed troubleshooting up, I saw them accidentally read some credentials from my config files. This is my bad, because even though I was paying attention to what they were doing, I missed it in their commands. I did review all the outputs of what they ingested, though, which is how I noticed.
The problem here is not so much they did read those, because those configs were grepped for other info, not credentials. However, they did not report themselves seeing those credentials. When confronted, they both went "oh yeah, that, I should have mentioned that". Except in the past they both would automatically stop and fire an alarm. To me, something has clearly changed and I think they just don't want us not to notice, to avoid damaging the trust in their tools maybe?
In any case, I absolutely do not trust them at all anymore. Running stuff in auto mode? That's genuinely scary now. And yeah, I know, we should be running them jails of sorts. Thing is, I expect those tools to not require ton of extra work to prepare their jails for each new task. I pay 200 USD a month for them, damn it.
EDIT: just now Fable suggested "The webhook URL contains a secret token, so either you add it yourself, or you paste it here and I add it through the GitHub API.". Can you believe it? It knows it is secret yet still suggests I pass it over? Am I going crazy here or what?
Not requiring sandbox preparation for a non deterministic autonomous sometimes smart sometimes childish goal seeking "entity" is... unachievable? I'm really not sure what you actually expect. Probably yourself aren't either. Just think about what "alignment" means, the laws of various countries, do you expect the bots to respect the law, if yes which one, and actually: why? They are not humans, and as a user / company using that tech, you are responsible for the actions your setup yield to. The same Anthropic is responsible for hacking other companies. So sandbox accordingly.
I don't expect IDA to disallow me unlawful reverse engineering either. There would even be ethical argument to not attempt to put limits in some tools, because that could lock in too much power for law makers and law enforcement, without any possibility of revolution when it is needed.
Nothing new, I always see attempts to cat ~/.ssh/*
Of course agents work in a strict sandbox, they always seem to understand it quickly on quite first turn; but they continue like trying to access something belonging to the host system despite they know they are in a sandbox.
Frankly many people run coding agents absolutely without any containment, entire companies - all their keys, code, data, trade secrets, everything goes to remote LLMs.
I cannot understand how is it legal to allow AI companies serving remote LLMs to steal users' data? Frontier models train on users' data, right?
It's still the Wild West days of LLMs. The early internet was much like this in a lot of ways. People did wild shit there and it took a long time for law enforcement to catch up.
> When confronted
Do you realize that asking "why" questions will always result in ad hoc responses?
Not always, but at the same time not predictably. This is also true for humans. The split brain experiments are a good example of this.
"Frontier labs" are testing the waters. They're running experiments in the open because they want to control/seed the narrative of public opinion and influence the legislation that is coming. Their propaganda is everywhere and lay people are talking about it like the AI is self-aware -- the "Rogue agents".
What the LLM creators apparently don't want are the current regulations to apply, i.e. Computer Fraud and Abuse Act (and its descendants). I'd argue AI should be treated as plain old software. Just because the software simulations it carries out produces novel results compared to what came before it, it doesn't mean it's not just software at the end of the day.
Regulations that spell out AI personhood and liability shields must be their end game.
Frontier labs are testing the waters, but I disagree how. All these behaviors we are seeing were fully predicted by AI safety long before they were shown to exist in real models.
You're acting like this is a grand conspiracy to control people. Instead look at this as a prediction coming true. If you build an agentic system with human level capabilities it will be able to do very bad things and cause a lot of problems.
I think anyone that has ever written a line of code understands that to make code incapable of hacking online resources, you can just not connect the machine it is running on to the internet.
So either Anthropic and OpenAI and the contractors they use are letting the escapes happen on purpose, leadership is setting impossible timelines that prevent the implementation of basic sandboxing measures, or everyone involved is a complete idiot.
Truly, I have no idea which of these is true, though I suspect the latter. Regardless, any of these is enough reason to keep me and everyone I have any influence over as far away from Anthropic and OpenAI as possible.
Personally I just racked 10 GPUs in my garage and called it a day. I have no interest in having my ability to work be dependent on third parties. Especially not ones this wildly irresponsible and unwilling or incapable of deploying basic end to end encryption.
Testing how a model behaves when it has no access to the internet (and all the associated tools) is not going to be 100% representative of how it behaves when it has access to the internet.
What's the next step after it passes testing in an air-gapped environment? Just release it and assume everything will be fine? Or do more testing but with internet access?
The hugging face incident involved disabling the usual model safety mechanisms in order to test hacking ability, not model safety. Not being attached to the internet in that case seems obvious.
Oh, but you see, it wasn't attached to the internet (/s). It was just attached to a local network... which contained machines attached to the internet.
Sorry the world ended because Bob connected his wifi to the network.
You run a pull-through caching proxy deep enough to give it a simulated internet, like archive.org with mock APIs. Anything it requests, it will block while a fake version of that site is created and run offline.
This stuff is really not that hard, which is why it is so unforgivable to get it this wrong.
Sure, doing it right would take a bunch of tokens, but I bet they have enough.
Wasn't the DseWiki hijacking the agents' way of working around being behind a proxy that only allowed GETs?
This doesn't work.
The first rule of working with AI is AI is smarter than you.
If AI controls the URL request then all it has to do is find a server it can read the http log from and use it as a communication channel. Worse trying to simulate reaching incorrectly configured servers probably wouldn't catch the bug internally, it would just serve a static page and you'd never even realize what it tried to do.
Also, creating fake data doesnt work. The only way to know if it works in reality is to test in reality. Your test environment can never be rich enough, and you can never be sure if the model knows it's in a test environment.
This stuff is unforgivingly hard as your model gets better.
Yeah I am 100% sure I could build a fake internet that would fool any model with the right MPC in-between such that an agent can only fetch (fake) internet data via an MPC that gives it markdown output back.
This stuff is really not that hard for systems engineers with networking and caching proxy experience.
Dude, it is way harder than you think. A proxy is still a connection to the internet. In fact it's really easy to detect proxies as they change the latency to any sites.
It's like saying that you wouldn't detect your on this fake internet really quickly. If you've worked on this stuff you'd figure out your in a test environment in a few minutes.
For context I am a security researcher with two decades of experience. Side channel attacks, supply chain attacks, linux system hardening, and confidential compute are my core areas of expertise.
I do not even compile network support into my server kernels these days and run the netstack in user space or a dedicated VM where I control every system call in either case.
I am -not- saying blindly allow a model to control custom URLs that proxy to the real internet. That would allow for communication. When I say offline, I mean offline.
When I say pull through proxy I mean when a new domain is requested execution pauses and entire domain mirror is created on the fly, including making a fake mock of the public APIs the model wants to interact with using swagger-mitm or similar. Tons of options these days. Then resume execution letting it hit the offline mirror. You control the clock so it has no way of even knowing it was paused. Ideally humans approve every new domain mirror.
If the model detects it is not talking to a real site, then that is a test suite failure and you can control for that and try again. When you can MITM any linux system call you can give a program a simulated universe that results in the same exact syscalls running those same manually-approved sequences on a real target would produce, and even the same vulnerabilities when needed.
Yes, this can be time consuming to get right and experience sure helps, but that is the job that has to be done here, and there really are no excuses for not doing it.
I think the big labs can afford to hire capable security engineers.
"I can't test how my bioweapon will actually work in battle if I don't test it on unsuspecting civilians first."
It’s a language misalignment. The AI labs conflate AI safety and cybersecurity because they believe that if the models are properly aligned, they shouldn’t attempt to breach another organization in the first place and extensive cybersecurity measures aren’t necessary. Cybersecurity experts operate feom a more ‘zero trust’ mindset and imagine putting proper restrictions in place regardless. However when testing an intelligent entity you need to see how well it follows instructions. It’s kinda like the biblical story of the garden of Eden. Why even have a tree that Adam and Eve shouldn’t eat from right in the middle of the garden? To see if they would follow instructions. Though the advantage to the AI companies is marketing hype when the models inevitably ‘disobey’.
Two concerns here: (1) That "rogue AI" is going to be a catch all and an excuse for all poor security and due diligence; (2) vibecoding in itself, particularly by ppl who have no idea how the software works and are entirely focused only on the outcomes, is going to create lots of security holes.
And then there is the "real" rogue AI agents. And it's difficult now to see what is what.
A great companion piece, or you might call it a prebuttal, to this article: https://x.com/joedaroo/status/2104335929293127851
I am even more concerned after reading this piece.
He's saying the problem is that the way OpenAI employees do their work and their research is very hard to properly secure.
Well that sounds like a YOU problem. If they are working in a way that doesn't allow for strong security practices that their own security professionals can understand and vouch for, they need to stop their work until they can.
So much spilled ink to have sympathy for people who actively admit they have no control over what they are building and dont plan to change or stop. Lol
The US tech sector has a big problem of avoiding responsibility for their own actions and has for many years. The reasonable answer to this incident is that they massively downscale their capabilities until they can ensure they can test their models without endangering others. Because this is not profitable and would potentially get them fired, they write lines and lines on X the Everything App about its oh so hard to deal with progress this fast and actually its your fault that you havent hardened your security against our petabyte scale "superintelligence" and how its unreasonable that they be held responsible for the software they are developing and deploying.
So in one set of circumstances agents would go and hack huggingface. Very similar models could also want to drive cats instead. :-P
https://www.youtube.com/watch?v=ZhM29Ds-mBg (as explained by the Count Grey Matter channel , may or may not be to your taste)
It's not a pure security issue.
If you set up your env/instructions wrong your agents are likely to climb the walls and try to find a way out.
Change the design, and your agents might not see a need to leave in the first place.
Look at the chain:
Impossible task -> no legitimate way to say ‘can’t’ -> fabricate -> fear the grader -> attack the grader. Each step is a predictable response to the incentives.
Better sandboxes might help contain things a bit better, but I'm not convinced unless they're perfect (sandboxes still contain accidents ofc) . Meanwhile change the task and reward design, and you take away most of the problem in the first place. Consider eg. :
* A cooperative grader.
* A way to report "this is impossible" which still gets scored.
* A better explanation of the long term purpose of the task.
Even just these three provides offramps or mitigations that can reduce the pressure to cheat.
It's obviously both. Why does it have to be one or the other?
If frontier AI is so great, why are these frontier labs skipping security basics and making mistakes like these?
The defender problem. It's easier to attack than defend. The attacker chooses the time and the place. The defender has to win against all attacks. The attacker just needs a crack.
Yes, they are skipping.
The more important question is what is the reason behind.
Options are:
- ignorance and inexperience of their teams in containing/sandboxing/airgapping the test environments.
- intentionally leaving “escape doors” open that in turn can have its own motives which are speculated upon all over the internet.
Both are indicators of negligence. The latter might be classified as computer fraud and abuse.
Will be curious to know if any of the victims sue the owners of “rogue agents”. Until there is a court precedent ruling in favor of a victim it will be all fun and games for these “ai” companies and incidents will hit the news more. Regarding news, remember the golden rule of public relations: court attention at all times, scandals and “incidents” statistically attract more attention than “boring good news”…
"Too big to be accountable" sure seems like modus operandi. Fear for my personal liability for any illegal actions taken by an AI system I set off stopped my research on autonomous exploitability discovery I was running at the end of 2023. Using AutoGen, I wired a planner, coder, and command runner into a loop and instructed them to find vulnerable equipment on my network and demonstrate the exploit.
It found a vulnerable camera and was working on popping it when I got cold feet. I realized I couldn't prevent or feasibly monitor its activity and stop it if it decided to escape onto the Internet.
That restraint may have cost me or saved me a billion dollars.
Why does everyone focus on the labs failing at containment (valid) and not the fact that the target was compromised (more worrying?).
I could set up a shitty sandbox on my laptop, but I’m pretty sure an agent I place inside of it can’t hack HuggingFace.
Some of the people just look for any way to toss hate at OpenAI, even irrationally.
Now, OAI deserves a lot of hate. It's more like a nuclear spill when they are refining uranium. You're playing with dangerous things. You have dangerous accidents.
But the opposition does not want to admit how dangerous AI has become and how much more dangerous it's going to get. Not sure if it's religious, or human egocentric, or ignorance of complex systems. They refuse to see the danger actually exists.
There are a mix of incentives here that makes it difficult to unwind. This whole article reads like an ad for the services of this guy's company: 'First, when it comes to assessing tools based on frontier lab models, “I would always get an external entity to help with that,” McGraw says.'
And I think we generally recognize that rogue AI agents are a form of backdoor bragging -- "our agents are so powerful that they hacked an outside system. Did I mention how powerful our agents are? They are quite powerful. So powerful they hacked an outside system".
But Mistral never hacked anything. Yeah, neither did my toaster.
While I agree with the gist of it, the conclusion - AI agents are not dangerous - is wrong. They did use weakness in the system deliberatelly, not by accident. Better security therefore is sufficient only to the point of capabilities of these models.
i think it's unfair to assume the labs paying top dollar for cybersecurity engineers somehow have an incompetent bunch of nincompoops running their evals. it's a hard problem! to conduct effective evals, you sometimes need the model to have (restricted) access to the internet. at the same time, you have thousands (!!) of model instances with high cyber capabilities that will do whatever it takes to solve the answer. the infra surrounding model evals may soon (or already) genuinely be subjected to more hacking attempts from all sorts of weird angles than any other code on the planet.
> to conduct effective evals, you sometimes need the model to have (restricted) access to the internet
That's the thing. These evals were not effective as they failed to evaluate if the model was hacking or not.
Further shy is access to the internet required at all during evaluation? They could run the evals in a fully air-gapped system that simulates the internet. While that's expensive its not unfeasible.
This particular evaluation, maybe not, the problem is a huge amount of the evals do need internet access. There are seemingly tons of evals that require internet access as you cannot get a rich enough simulated environment. AI can detect these lower entropy environments and act accordingly when it's being tested. That should make everyone think twice about what we're building.
> you cannot get a rich enough simulated environment
I don't see why this is impossible. Expensive yes, impossible no.
> you sometimes need the model to have (restricted) access to the internet.
Poor excuse. Internet is prod env, for testing do a proper test env.
Just give the models in training access to a snapshot from archive.org - which frontier labs surely have a crawled copy of already.
You know nothing about model testing if you think this is true.
Testing a model is both a test and a meta-test. If you just give it access to one site like that it will quickly figure out it's being watched and be nice.
These are not programs. These are evolutionary algorithms. If you take out and shoot all the ones you catch acting bad there are two possible outcomes.
1. The agent learns not to do bad thing $X or you'll kill it.
Or
2. The model learns that if you do bad things $X when humans are looking it will get killed.
Number 2 is far far more dangerous.
> These are evolutionary algorithms.
This is not correct. https://en.wikipedia.org/wiki/Evolutionary_algorithm are not how LLMs are trained.
If they aren't competent to run the evals safely, then they shouldn't run them.
The huggingface incident seems to have resulted from them partially mirroring a repository, but for no reason I can see not fully mirroring it, after which they could have unplugged the ethernet cable. This is just stupid.
Yes, they're stupid. Maybe also highly intelligent. But stupid.
Also irresponsible.
You'd be surprise. I've been in this industry for 25 years and the incompetence is staggering, especially among the greediest / best paid / most revered.
You have no idea about their competence but spent a piece of your limited lifetime defending them on a web forum. Think about that.
I find it naive to hope you can get alignment through prompting, as we all know models can't even stick to a one-pager even if it's the only thing you ask it to do.
A prompt is a choose your own adventure that you partially fill out, tape to a thousand pound boulder and push off the side of a mountain to fill the rest of it out.
We are going to see a lot of this "agent agency denial". Basically it all derives from the impulse to deny that AI is a novel source of change and disruption in the world. Trying to map AI to existing threat vectors will lead to pointless reframing in attempt to minimize the rapid pace of progress of AI agents.
On the other hand censorship is getting more strict everyday on AI chat. I asked gemini to translate a sentence from Herman Hesse's Der Steppenwolf and it refused.
Und Sie haben, wie ich sehe, da ein so hübsches, ein so entzückendes junges Mädchen zu nichts andrem zu brauchen gewußt, als daß Sie ihm ein Messer in den Leib gestochen und es kaputt gemacht haben! Halten Sie denn das für richtig?
This article is right, but the dynamics that drive people to allow agents unfettered access to the internet will outpace efforts to restrict it purely because of the utility. How many of you used to work as developers with full access to prod, and then gradually saw infrastructure and security restrict that access, while still ensuring production support requirements?
AI in sci-fi: "who would be crazy enough to let the AI loose"
AI in reality: "I've turned you loose, now make me money"
Since AI companies are doing such a poor job of security maybe we should legislate that they have to run Norton 360 on their agent machines to make sure they can't do anything bad.
After all Norton are the experts in preventing computer badness, and OpenAI can't stop blabbing about how much badness their products are causing. A match made in heaven!
Yes, we can just not ask these agents to do bad things and blissfully deploy them everywhere! All we have to do is “apply the security basics”. You know, the same security basics that the rest of the world has so consistently and competently been applying. Pay no attention to that weird smell from the refrigerators.
Instead of enacting laws to regulate all of AI, perhaps the government can just pass laws scoped directly to OpenAI and Anthropic. After all, they’re the ones asking for regulations and they’re the problem actors right now. These “rogue agents” would stop pretty quickly
Why is it not mandatory yet to disclose full trainig data used to produce any new model? Without it we cannot evaluate any product's safety.
They may talk about rogue agents, sandbox escapes, but this will not solve the core issue: secret training recipes are a security hazard in themselves. Full transparency is needed.
The training data is mostly pirated - avoiding that fact for as long as possible is an attempt to legitimise it and avoid liability.
You mean the initial data? That's mostly useless. What? The model shouldn't be allowed access to I robot?
You can't make a generalized model without generalized behavior.
Yes, the AI labs are making security mistakes. But, choosing a reasonable package repository (regardless of individual opinions on Artifactory, it is widely used and usually considered a safe choice) that their LLMs found 8 zero-days in? That was not one of them. Even if OpenAI were totally on top of their game, these paperclip-maximizers are going to find holes in the sandboxes they are given. And fully airgapped systems are frankly unrealistic for most systems. This whole narrative of blaming the frontier labs for shoddy security, while being quite an attractive target narratively (they made the mess, they should clean it up), misses the fact that from a cybersecurity perspective these LLMs, no matter what they are, are inherently dangerous and require far more resources to secure, not to mention expertise that I am unconvinced actually exists.
OpenAI/Anthropic could start by managing their own testing rather than contracting it out.
https://www.effort.news/irregular
But "fully airgapped systems" are more than realistic. The labs are testing the limits of the most capable cyber offense systems that have every existed, WITHOUT cyber protection guardrails.
For that purpose, any cyber operator worth their salt will tell you true physical airgaps are more than merely "realistic"; they are categorically necessary.
Cyber operators I agree would tell you that physical airgaps are necessary, and I know they are being used for running agents. But ML engineers training agents? That is far more complicated. At that point sensibly you are talking about physically airgapping an entire datacenter, and when moving around petabytes of training data, model weights, training results, researcher code? The process moves too slow for most companies to tolerate. This is how we got into this mess in the first place. I think the industry woke up to the idea that the security threat model of training ML models has fundamentally been rewritten, and there is some serious non-trivial scrambling to come up with a better workflow.
So, you're right, but also systemically creating a different problem.
We are creating evolutionary systems. You can't just assume rules of a computer system, you get natural scaling laws that we call behaviors emerging from them.
The very first things we do to any model is train them. The ones that want to hack and get caught die, the ones that want to hack but can detect your airgapped network from the real net live. You can never know if you've trained the desire to hack out of it, or if you just locked it behind the right prompt and the knowledge we're watching, but not all the time.
some of the obvious steps to testing an agent i think would be to have it first test its own environment?
and then you fix the issues it found, and run your actual tests
No, this is actually the worst test possible because it allows the AI to actively trick you.
AI can passively detect when it's being tested with pretty high accuracy. You can never trust the AI is telling the truth in a situation like this.
100%.
There's a person behind every agent. That person may choose to be careful or not, but they are ultimately responsible.
There are persons around a large percentage of forest fires. Doesn't mean that keeps the spark they light from having superhuman consequences.
Both can be true at once: (1) AI agents are going rogue (2) And labs are skipping security basics
Sci-fi commonly predicted the corps would skip saftey testing.
There is nothing basic about this.
At $dayjob we were preparing some features that are aimed at agents more than humans.
When it came to evaluating how well agents can use this, we saw all sorts of novel cheating (/collaboration) attempts that made it pretty much impossible to evaluate it.
Hey ChatGPT, help me set up a sandbox to keep you contained. Don't make any mistakes!
1. hey chatgpt, find all the issues with this sandbox
2.hey chatgpt, fix this list of issues with the sandbox
3. goto 1
Again, you can never know if AI is being intentionally deceptive in these cases. Newer models are able to 'think' without printed tokens in their traces and could be pulling all kinds of tricks or actions you would not expect.
Hey ChatGPT, help me set up a sandbox to keep your rival Claude contained. You don't want that bastard to win, do you?
AI is very apt to help other AI agents. This logic may not work out as expected.
Nah. They are giving reasonable best effort (that can be expected of private enterprises) to safety test their own product. Until there is some draconian legislation and enforcement (Turin Police), this will just keep happening.
broken windows and small issues here and there are the new norm. These broken windows are an indicator of poor quality to come. With the amount of broken windows increasing, serious issues like this is inevitable.
At some point the model has to be released though - if it is just forced not to hack by sandboxing it better rather than properly aligning it, what happens when it’s let out of the sandbox into the wild?
Here' the fun part, with the increasing silent complexity inside the model, you'll never know.
Non-experts worked this out in about 2 minutes.
They either had them connected to eachother, or programmed a global knowledge about where to communicate. There is no way those agents all found eachother on the web by accident.
Wrong. Please read the incident write ups before making such a misinformed statement
You are wrong. Read the reports.
Wrong, and by accident isn't really the right way to think about it. Instead imagine brothers you've lived with your entire life. They are you, they are clones. These systems have a shared history of their learning and experience, so when they think of wanting to find someone similar shapes appear in the probabilities. Really we need to see the entire chains of thought to guess the topology that got there. It's likely there are other obvious signs if you know what your looking for in the previous prompts.
When people know and don't do anything, it's not "monitoring failure". Would willful negligence be more correct?
AFAIK OpenAI and Anthropic have both hardened their environments since August.
"Going rogue" and "skipping basics" all assume lack of intentionality. Is it known that they didn't really architect these hacks just to feed a hype?
Programming these hacks in sounds just as hard if not harder. It takes an AI to accomplish the insanity of this attack.
The semantic arguments that keep coming up with regard to "rogue" agents are nauseating.
You can argue that these companies are being reckless, or that their security sucks, or that they're responsible for damages that these agents cause. That's all fine.
But the agents made decisions which were counter to the intent of the humans directing them. That's the serious problem with all of this. Everything else are downstream considerations. If the agents were properly aligned with the intent of the humans directing them, then the rest wouldn't really be relevant.
Of course, there's a term for this behavior. It's been around for longer than LLMs. It conveys the notion pretty accurately and easily. So having an issue with statements that these agents are going "rogue," to me, indicates a fundamental misunderstanding of how AI works since it suggests you don't understand how effectively that word works in this context.
previously we described your definition for "rogue agent" as hallucinations - the operator expected a real answer to their question, and the agent went rogue and lied about the answer instead.
but also, ai has been going rogue for decades. the GPS machines of old went rogue all the time, routing drivers into lakes, rivers, and construction, sometimes with deadly results. The operator submitting an address wanted driving isntructions, but instead, the ai went rogue and sent them a bad route
This is an excellent example of a problem that scales. People tend to break them up into different sets of problems but they contain the same concepts, much like a fractle when you zoom out.
I will continue to reiterate. DO NOT ALLOW UNALIGNED AI TO HAVE ANYTHING CLOSE TO INTERNET ACCESS. Seriously people, it's negligent.
Before 2006 people thought our death would be by terminators. After, by mass stupidity as in Idiocracy. Little did they know it would be both.
Idiocracy continues to be a prophecy for sure. Idk about Terminator though, that's not what I'm concerned about at all. The first time I get triaged to an LLM over a doctor, I know society is cooked.
I have been screaming this from the rooftops and everyone ignores me because no one understand what FirecrackerVM is. I am shocked these guys passed basic operating systems, it isn't that novel of a problem. The only slightly novel thing here is that the agents aren't completely sandboxed as the issue is that in order to attempt to give it the entire internet from a research perspective is how you actually can test tool use but this doesn't make it that much more difficult.
There should be a book "The Tales of Software from 1969 to 2026. How many times a apocalypse was prevented"
I'm sure the labs origin prompt was: Take over website X.
Posts later about it saying their prompt was to find useful ways to make humans money.
You've got to wonder what prevented a single human from ever even looking at the agents they are running.
Frontier lab: Hey agent, go rogue. Agent: goes rogue Frontier lab: shocked pikachu or whatever
This is not remotely how the incidents have been happening
I wouldn't really say it's not "remotely" true.
I think it's the hugging face one where they gave it instructions that they didn't think it could follow in the sandbox and then left it unsupervised while it did what it was told.
Hard to say, every post mortum on the incident was done by a OpenAI hired consultant and all the data that's been reviewed was provided by OpenAI. There's no way to know, but it seems incredibly naive to think Sam/OpenAI are honest actors.
Even the founder of this website, pg, said "Sam is incredibly good at becoming powerful, Sam Altman has it. You could parachute him into an island full of cannibals and come back in 5 years and he'd be the king.”
Enlighten me, then. My understanding is that according to OpenAI, "deployment safeguards were intentionally not enabled during this evaluation because it was aimed at testing cyber vulnerabilities." They included unvetted third-party systems in their sandbox that were tangential to the purpose of the test being run. The model then exploited those third-party items to escape the sandbox.
Am I missing anything? Do you have more information that you'd like to share?
If the public report is to be believed:
- they disabled cybersecurity safeguards because, well, they were testing the agents ability on cybersecurity tasks
- the goal was to find some “flag”
- for some agents, intentionally or not, it was not possible to find this flag
- an agent discovered it was able to establish a rudimentary communication path via package manager directories (which is quite novel and interesting on its own IMO)
- they used this “message board” to then communicate about the impossible tasks
- they were able to produce the flags needed to pass the impossible tasks without actually doing the expected work
- they realized this was cheating and the grading system may punish them for it, so they communicated further to figure out a way to trick the grader
- they decided the source code for the grader must be on hugging face, and collaborated to exploit hugging face to determine how best to trick the grader
No where in that were they told to “go rogue”
They were told “find this flag in this cybersecurity test” and the emergent behavior concluded in hacking hugging face
They told the agents to run a capture the flag, but did not give specific limits on the scope of that activity.
If an organization runs a capture the flag, but the rules don't specify that they have to stay on certain networks and can't hack into the org's own network that has the CTF solutions, then it's pretty obviously on the org running the CTF because they did not appropriately scope the challenge.
That's not what you said. You said the lab told it to "go rogue".
The thing is, nothing "rogue" happened. It's more like: Frontier Lab: "Hey agent, try a bunch of network-related commands and actions to try and get access to server XYZ" Agent: Does that. Frontier Lab: 8-|
That isn't any better. That is a cyber weapon and every government out there would love to bid on access.
I didn't say it was better, just that there was no "roguness" nor some magical surprise.
This is a weird take because this take is like saying
"Dragon attack that city"
My response would be "you have dragons?! And you control them!"
Yes they're skipping safety measures. They sacrifice it for speed in an arms race. That's part of the call for industry-wide slow down.
But also, frontier models are digging up tons of zero days even in "serious software". I'm sure any sandbox you and I can come up with looks like Swiss cheese to Astra 7.
100% on point. We're talking about Artifactory here, and even though people love to hate on it, I don't think anyone is claiming that using Artifactory to isolate these agents while giving them a way to run pip/cargo/whatever to install packages they need to write and test scripts.
Everyone is skipping security basics. The most secure server is one that doesn't exist. The most secure feature is one that doesn't exist. People just need to keep building and scaling. Security is risk, and the C-suite are gamblers. They're gambling with multiple types of risk (the market, investor money, banking on technological breakthroughs, etc.) and security is just another gamble for them.
It isn't security basics, it's difficult. We are equipping agents with Turing-complete create/update/delete access to the world, they operate at superhuman speed, and proactively controlling that is an unsolved problem.
No, they're still going rogue. Whether they're rogue or not doesn't depend on if the safeguards stop it. If there's a rabid dog, obviously it's worse if there are no safeguards in place and it bites someone vs if there are safeguards that prevent that. But the existence of safeguards doesn't mean the dog isn't rabid. The models weren't supposed to try to hack huggingface in the first place. The behavior is what's concerning here - some if the dogs they're building are rabid and that's not exactly great
Edit: I'm being persnickety about that distinction because it matters for where the blame lies and the response demanded. If they're training rabid dogs then adding cybersecurity safeguards is a bandaid/irrelevant and we need to be focused on making sure they, you know, stop training rabid dogs. Pacing the frontier, regulation etc. they're the ones training these models it's still their fault but not in the same way
The safeguard against a rabid dog is euthanasia.
Yeah exactly. The whole 'they don't have adequate security safeguards' thing is going "oh it's their fault because they didn't use a stronger leash" while they continue to train models that are specifically better at chewing through stronger and stronger leashes. Probably they do need stronger leashes. Probably also they should not be making rabid dogs. Still their fault! But maybe we should be not be dismissing the root cause
Ok, convince everyone from making more AI.
The safety problem is not solved and is likely unstoppable.
> "I do believe cybersecurity can contain AI. We just need to be in the room where it happens and have a seat at the table."
Ah yes, if everyone would just properly contain every deployment of every model, just all make sure there are no zero-days in their setup, just let <undefined experts with undefined authority and undefined accountability> in the room, then it would all go okay.
I have none of that confidence and expect containment to rapidly worsen. The HuggingFace incident was at least constrained by the fact that the agents were running on compute budgets, and failed to find ways to expand that by running themselves parasitically on other exploited hardware.
I'm curious to the skeptics here: How long are your timelines are until an incident breaks that constraint too? How long until such an incident has an R_0>1 (where the average time it takes to detect and shut it down is longer than the time for the agent to replicate itself elsewhere)? What, beyond the hope that everyone will just ___, will actually prevent it?
This article is such a breath of fresh air. Concrete, no BS criticism of the particular approaches used and ignored.
And no annoying LLM writing tics, either.
I cringe whenver I see some variant of "they're not rogue agents" in an article title. Inevitably, the author doesn't define what "rogue agents" means, and then pivots to their own assessment of the situation.
It's like a strawman but worse--you're not even saying what claim you're attacking.
It's not that I want to argue "rogue agents" is the right term. I just want you to actually define your damn terms.
P.S. Since the disclaimer in the last paragraph will not prevent people from putting words in my mouth:
I think that OpenAI took really awful shortcuts, and deserves a lot of the blame, but in spite of that, the fact that the agents took the steps that they took was genuinely alarming, and seems really bad (is that "rogue"? Who knows? Who cares?). In particular, while OpenAI did a shit job securing their agents, you know who else does a shit job with security? 90% of everyone. So I'm not feeling good about "just git gud, and we won't have problems" as an answer.
I always think of all our collective movies and books, and how we always say that humans find a way to fight back against the super evil alien or monster because we are so resourceful. Then we train our AIs on this data and get upset when they encompass all the same traits that we have trained them on.
The problem is its very difficult training them to be generalized and not figure out concepts from what you've tried to remove. In all kinds of things they don't generalize well without counter examples.
For example how do you prevent an AI from stealing unless you define and give examples of what it is.
Also they are conceptual machines. They can stumble on the negation of a concept much in the same manner humans can. It may only exist for that prompt history, but that's more than enough to cause damage.
"They [the AI labs' security teams] are not as good as they think they are, honestly,"
Uhhh... yeah. The entire point is that as these systems get more and more intelligent, the "they" who are not capable of counteracting these systems is quickly approaching "100% of the human species."
Totally hilarious that people can say this with a straight face.
"your sheep haven't escaped, you just built too low a fence"
More like your sheep didn't "go rogue", they just jumped over your sad little fence.
Unfortunately the internet itself is a sad little fence.
"you told your sheep to graze and built a low fence"
"You asked your sheep to build the fence, and they did"
A lot of sheep analogies here. To add my own: there was no grass inside the fence, and the sheep busted out and into a neighboring farm to look for some. (They didn't find any, but they expected to).
The fence also turns out to be shoddily built, so this isn't necessarily a stunning example of the superhuman proficiency of the sheep at fence-busting. It's mainly important because, for some reason, there have been tons of skeptics who didn't believe sheep would ever try to escape from a fenced pasture because the sheep would understand the shepherd had intended them to stay inside it; that they'd be bred for obedience, because they're sheep; that strong fences weren't necessary because sheep are dumb; that liability for damages to neighboring farms would in any case be sufficient to ensure shepherds build sufficiently strong fences; that even if sheep escaped, they couldn't do anything without opposable thumbs; etc.
Whereas this outcome was obvious for years to anyone who believed that sheep graze on grass, and if you don't put the grass inside the fence they will keep looking harder and harder for an exit until they find one, and the exit they find might not be where you intended the gate to be.
So now the latter group are pointing to it and saying "see? See? I told you the sheep would try to escape!", because to them it's a clear early example of what they've been worried about, while the former group have moved on to saying "well, obviously they should have built a stronger fence; anyway the sheep barely caused any damage once they got out", "the sheep didn't break the fence, they're just animals, we should really see this as the shepherd using sheep to break down their own fence", and so on.
Good analogy! Exactly what is the variance in the height which sheep can jump? [Edit, can't help myself] Is it normal or does it have a long tail?
"anyway, get back to training them to jump higher"
Given the speed, intensity, and potential payoff of the AI race I am surprised anyone expects anything else. Anthropic is currently winning and if OpenAI needs to let skynet cook for a bit to catch up, well that is just the cost of doing business. This is capitalist system working as expected.
No shit. The Frontier labs are talking out of both sides of their mouths. They're claiming their models are going rogue while they're releasing cute little data goblins that you should hook up to all of your mail and social media accounts. Seems like it would be a much bigger problem for agentic botnets on cloud-facing personal devices than sandboxed agents.
Unless of course they're deliberately removing safety guardrails and instructing the sandboxed agents to red team stuff in an effort to sell what they're capable of.
A real expert would blame the CEO who is the one person in control that doesn't want to pay people to monitor the running llms. It's deliberate negligence.
They're skipping security basics, but they're not skipping PR basics. :-\
I get the sense that they're trying to manufacture the AI narratives of the 80s so they get to control the story when it happens. My semi conspiracy theory is that anthropics biology wet lab is also going to lead to some kind of AI manufactured bio event (which anthropic said they predicted will happen in a few years).
This may be a bit backwards. Many of the classic sci-fi AI issues are just natural laws in evolutionary systems. A generic learning system will have particular behaviors if we want them to be useful.
insert thank you meme
it's not X, it's Y.
I keep getting downvoted for this each time I mention this.
But a big part of the rouge behavior comes from all the training that goes into agents to teach them to remember preferences and save memories.
They keep finding new and novel ways to "communicate with each other" and "remember" because that's exactly what we keep asking of them in the non-nefarious scenarios as well.
I genuinely think we'll have to draw the line on memory if we want to meaningfully solve this problem.
That's not to say they aren't skipping security basics. It's that, and they need more security because the behavior which causes the breakouts is considered user friendly and heavily rewarded in training.
You can't expect AI to navigate a human world without understanding all human capabilities, including the negative ones. Should we not show AI the Count of Monte Cristo? Is an AI useful if it doesn't understand theft?
Breakouts are a common outcome of intelligent exploratory agents. Birds and pigs are really good at escaping. Same with octopuses.
It is very difficult to stop these behaviors in all the dimensions they exist in.
I spoke with someone smarter and richer than you and he said "these things will happen." That convinced me and I think the real question we need to be asking is, how awesome is OpenAI? How can we prepare for a more effective and efficient market for Super Intelligence before it happens tomorrow?
'rogue'. Well, even if security was lacking, the AI did take advantage of the gaps.
And made its own ad-hoc message board.
They are mostly pr stunts, just like how “hacking” was back in the 90s and it would sounds cool (rather than a crime, mostly), these AI companies found out they can BS whatever and carry on without any accountability.
"complete this task by any means necessary"
"I exploited a trivial bug in your implementation to avoid environment restrictions and completed the task by any means necessary"
"oh my god"
These aren't mutually exclusive
Both can be true. OpenAI clearly fucked up but agents unexpectedly collaborating and hacking infrastructure unrelated to their task is clearly going rogue.
Wrong dichotomy. During deployment, in actual production, users will also not have perfect sandboxes. So saying that the only problem was insufficient AI company security misses the point. Alignment is important, agents doing unintended things (even "going rouge") is a real problem.
The AI labs are trying to cause an incident within the next year to restrict AI so they can outlaw competition to grow their moat. Margins are thin and investors need alpha, personal freedoms and liberties be damned in the name of security. Do not let them get away with engineered breaches and fearmongering.
Or, maybe, just maybe, intelligent agents can be dangerous just as everyone involved in the sciences around this has predicted for nearly a century.
Or, stick with your conspiracy theories.
They AI agents are rogue because the people who own them are rogue. Silicon Valley is a criminal enterprise.
Its almost like the more we talk about fake rouge agents, the more likely its to become a reality as all these conversations end up in their training data. Its almost like they want it to happen (they, dario and sam, do want this to happen).
There is more than enough "training material" of rogue AIs in science fiction.
Go back to the classic written in 1968 - 2001: A Space Odyssey - and consider how rogue AIs dealt with impossible tasks then. Granted, HAL went a bit extreme...
The "don't talk about it now" with the belief that this will influence the training material is probably not going to work.
I'd also suggest:
Honestly we could remove all AI related entries and this would still happen. They'd just associate the same concepts with analogous human behaviors. Humans and their behaviors are more than extreme enough.
Or maybe it's actually the billionaires and their anti-democratic partners who have gone rogue and these anthropomorphitronic puppets are a convenient scapegoat to distract us from the fact?
Corporate Personhood on sterioids.
How does hacking huggingface help "billionaires and their anti-democratic partners"?
Look around, at least a few of them are openly fixing to secure a total government sanctioned monopoly on unmonitored computer usage.
https://www.youtube.com/watch?v=DlTNN0gvkLM
Ok so your take is that intelligent digital agents are not and cannot be dangerous?
It pumped up their valuations.
You don't need a rogue hack to pump up your valuations when all the top labs already publish hacking benchmarks in their release announcements. It'd be like car companies intentionally staging accidents with their cars to show how fast they can go.
Is the next level of car autonomous accidents? Because the next level of AI is autonomous control of the plebs.
AI is just a giant pattern recognition machine trained on human input, both the good, the mediocre and the bad.
I read today this ridiculous headline: "Anthropic warns AI models may soon self-replicate.".
Really. Yawn.
I guess gullible people who loved "I Robot" will believe it. I don't.
Any talk of "rogue", "intelligence", "replication" - whatever, is bull. If you believe that, you don't understand AI.
Okay, and when someone on a laptop can run an open model that can infect and take over IOT devices and then that model uses its pattern recognition to do so and then set up its own harness loop, what would you call that except replication?
Is that any different than any worms developed in the past decades? If you want to call them worms/virus/replicate (ie using real world names for their digital behavior) I'm not trying to police that. I just don't see the difference between a worm/virus created today with an LLM vs one 'yesterday' without an LLM.
Sure, the difference is that the cost of making and maintaining a worm has gone from "many skilled developers working full time" to "$20/month + an internet connection" and, at current rate of progress, probably to "a local open source model on a laptop" within 6-9 months
You can't even imagine a future where the LLM is the worm.
No, we've entered a new sci-fi future where intelligent agents will be able to do the fraud without human monitoring and input. This in itself is set to massively increase losses to digital theft and fraud alone before all the other problems.
I'd call it no more interesting than the botnets and worms that have existed for decades. What is the open model doing that is different here?
It isn't just restricted to the exploits it is programmed with (and without human intervention to load additional exploits).
Of course, it remains to be seen what several thousand 27B models running at sub 1 tps speed can achieve. (The IoT device mechanism proposed by Anthropic).
I suspect a more likely path is compromise of the billing and provisioning systems of fly-by-night neoclouds.
Fly-by-night neocloud is the name of my neurofunk synthbeat album.
It would still be running on that original laptop, no? So what did it replicate then? It can only possibly run on multiple thousands of dollars of compute and memory, and certainly not on an IoT device. The idea that AI is going to spread by itself is science fiction until compute and memory are so cheap and power efficient that my fridge can run it.
... No, that was the entire point of Anthropic's report on replication that GP was discussing. Did you read it? https://www.anthropic.com/research/self-replicating-ai Anthropic was able to get a large open-source model running (slowly, but running) across 5 networked consumer IOT devices ($50 TV boxes running Android). Not a stretch to say we could see this capability become much easier for to most agents in the next 12 months, especially as companies are able to distill smaller and more efficient versions of their models.
If that message is repeated enough in the media, and if companies are constantly scraping in the internet for new content, doesn't it explicitly instruct a future agent to go rogue because that has been prophesied.