I think that good-outcome would have to include a reduction of inequality - I see the opposite happening.
GPU, RAM and SSD price inflation is an example of inequality stealing resources from the middle of the economy - if we assume talent is normally distributed, that means we are starving innovation, as fewer talented people can get their hands on local compute for their garage inventions.
When an individual is orders of magnitude more powerful than the government, of course it matters. I have no issue with the rich being rich, so long as their power can only go as far as living a comfortable life of luxury and wanting for nothing. But when they can get anything they want by buying anyone, of course it's a problem. How can we hope to have good outcomes for everyone in society, when any outcome can be bought for a price?
You are also implying that the current democratically produced laws are ineffective against stopping the powerful from doing anything they want? Can that be solved with better laws? or by laws that limit private property? Or what do you think that could improve the situation?
Government has authority and power to initiate violence against you and deprive you of your fundamental liberties. No wealthy individual has that.
In 2018, Congress hauled all the rich tech bros into their hearings and told them, start censoring your users or else. They did. Watch the hearings, it's clear who has the power.
Zuckerberg wanted to start his own currency. Congress told him hell no - why don't you consider a Facebook-branded Visa card instead?
Inequality is not primarily about wealth, but power. To quote Roger Zelazny, "Money buys things. Power makes things happen. If you ever have a choice, take the power."
So no, it's not about envy. Musk being a quadrillionaire is a problem because he can use that money to exert influence over my life that I don't want him or anyone else to have.
And when I say "anyone else", I really mean it. Replace Musk with whomever you consider the most benevolent person to have ever lived and I still wouldn't want them to have that power. Then again, a truly benevolent person wouldn't want that power, either.
Now, you can try to argue about how the distinction I made changes the whole argument and how the real problem is corruption and how it should be perfectly fine for quadrillionaires to exist in a society where they cannot abuse their wealth and so forth and so on.
We don't live in that society. We live in this society, where extremely wealthy people are allowed to use that wealth to hold power they shouldn't. Pretending that the wealth inequality is okay will not lead to that society. It will only make this society worse.
Elon Musk has near zero power over me. I'm much more worried about the local official who makes $80K but could make my life miserable if I pissed him off.
I've been laid off so many times I can't really muster any sympathy for anyone else who has been in the same boat. It's a mild set back at worst, and in hindsight many times a benefit because it pushed me into the market to get a better offer. The market is ever shifting under your feet and losing your job is just part of life.
This is by far the most circuitous route I’ve seen (yet) for accusing people of being critical of the ultra rich simply because they’re jealous. I’ll give you that much.
Why would everyone get a 5k sqft house, etc? If money is power and you are being given money without your labor having value, then do you really have any power? Because they can turn off that money tap and then what power do you have?
I have seen this many times, the idea that the reason people want equality is because of "envy" for the oligarchs who have so much power over us and the world we live in. I find that viewpoint bewildering.
No innovation is being starved in this manner. Nobody is stacking up GPUs in their garage for this type of thing. A house doesn't draw enough power to run a significant amount of compute these days anyway.
Long-time capitalist here that recognizes that capitalism has helped deliver incredible amounts of wealth and lift countries and people out of poverty over the last 100 years.
Sure, most of the gains have gone to the top — and maybe we should do something about that — but over the last 50 years, the rising tide has lifted a lot of boats.
But I also think it's fair to reconsider how we structure our economy as we (potentially) move into a world where thinking rocks are better (and far cheaper) at doing most tasks than humans are.
At the minimum, getting money out of politics and electoral reform feels like a good start.
> At the minimum, getting money out of politics and electoral reform feels like a good start.
One of the incoming Democratic representatives who flipped a Republican district said that he wouldn't support Hakeem Jeffries for speaker unless he committed to votes on a constitutional amendment to overturn Citizens United and on ending the war in Iran. In response the Democratic leadership threatened to make his life hell for the next two years, then try to primary him in 2030. https://www.wosu.org/politics-government/2026-11-11/rep-elec...
I'm not sure how you could accomplish any sort of meaningful reform electorally when party leadership behaves like this.
No sooner had Everet in 1758 erected the first wool-shearing machine that was driven by water-power, than it was set on fire by 100,000 people who had been thrown out of work. Fifty thousand workpeople, who had previously lived by carding wool, petitioned Parliament against Arkwright’s scribbling mills and carding engines. The enormous destruction of machinery that occurred in the English manufacturing districts during the first 15 years of this century, chiefly caused by the employment of the power-loom, and known as the Luddite movement, gave the anti-Jacobin governments of a Sidmouth, a Castlereagh, and the like, a pretext for the most reactionary and forcible measures. It took both time and experience before the workpeople learnt to distinguish between machinery and its employment by capital, and to direct their attacks, not against the material instruments of production, but against the mode in which they are used.
— Karl Marx, Capital, Volume I (1867), Part IV, Ch. 15, Section 5: “The Strife Between Workman and Machine”
> It took both time and experience before the workpeople learnt to distinguish between machinery and its employment by capital
Upvoted for the historical comparison, but the arrogance of this guy's writing never fails to amaze me. Destroying the machinery was already an impressive feat of organization and protest for their economic interests - now they are supposed to have attempted to overthrow the government and social order, while advancing Marx' own framing of the issue? Burning a few workshops sounds a whole lot more tractable to me.
"What do you think the Russians talk about in their councils of state? Karl Marx? They get out their linear programming charts, statistical decision theories, minimax solutions, and compute the price-cost probabilities of their transactions and investments, just like we do.
We no longer live in a world of nations and ideologies, Mr. Beale. The world is a college of corporations, inexorably determined by the immutable bylaws of business. The world is a business, Mr. Beale. It has been since man crawled out of the slime. And our children will live, Mr. Beale, to see that perfect world in which there's no war or famine, oppression or brutality -- one vast and ecumenical holding company, for whom all men will work to serve a common profit, in which all men will hold a share of stock, all necessities provided, all anxieties tranquilized, all boredom amused."
Isn't this entire 1 hour video talking about what a good outcome would look like and thinking the issue through? What part of the video are you taking issue with?
The point you're making has no utility. The logical trajectory of what you're saying is that the government can make no satisfactory decision. Since every government action is technically a "regulation," any discussion of what the government should do is the wrong framing.
You want AI companies to be owned by individuals so they control the means of production. Meanwhile, AI companies are all trying to go public so that the public can do exactly that.
TL;DR : Haven't seen the video, just reacting to your comment.
> Meanwhile, AI companies are all trying to go public so that the public can do exactly that.
Anthropic literraly just went public.. while the founders are keeping 51.1% percent of the votes. They are absolutely not trying to go public. They're trying to pay back their initial investors. That's it.
Pay back their initial investors? They’re trying to raise money to build more AI. Going public is how you raise money when you need a lot, want no masters, and have public confidence.
The point of investors getting paid back in an IPO is that it’s the key moment in a successful company’s history where you CAN get paid back without risking the company’s reputation. Companies can and do use private investment rounds to pay their investors back, but that sends a negative signal.
They're fundraising, they are losing too much money that the private capital markets are unwilling to give them. A company that wouldn't be fundraising if it went public is Stripe.
I’m no Anthropic fan (even if I am a user), but I think they’re profitable no? Now that they are public, we can see for ourselves (yes, they are profitable). This is simply cashing out the EA-ers.
No, they are not. They posted a $1.3B operating loss and $104B net loss so far in 2026 and their fundraise is to cover 900 billion dollars of compute commitments. This is using GAAP, they may be "profitable" in a non GAAP way.
- if nobody on market want your stock, there is no point going public. Keeping it private is more beneficial for everybody.
- The board need to answer shareholder. Unless you want endless EGM and shenanigans, you want to maximize the profit.
In theory, you can have a public traded company don't aim for profit. But I can't see how we get there... I guess if you have a money printing machine, you could out complete all the for-profit investor on the market....
I cannot think of a public tech company that is -not- evil.
Look at Apple, Google, Meta... becoming evil is a law of nature as public companies get bigger. They eventually all become malignant either by board of directors greed or pressure from major investors to make stock price go up at all costs. Even when it is not a law in the books or the bylaws, it is a law of nature.
If Dario actually wanted to act in the interest of the public, he would open source everything instead of hoarding power.
If he did that though, we all know shareholders would sue. Would they win? Maybe if destroying the moat was seen as breaking his fiduciary duties to the company. Even if the shareholders lost in court, the outcome is always the same unless startups or FOSS projects can come in fast enough to disrupt what will otherwise become a force of evil too big to kill.
> I cannot think of a public tech company that is -not- evil.
Not even gonna read the rest of your comment nor do I need to. Just wanted to say if you're this obstinately closed-minded and absolutist about literally anything it's going to be pretty impossible for non-extremists to have a discussion with you.
I hear you but I am not talking about how they behave, I am talking about legal obligation. This idea that fiduciary duty means they’ll be chained to a radiator unless they extract every red cent as fast as possible is a ridiculous myth that won’t go away. People have this notion that there is some legal obligation, “fiduciary duty,” that means leadership can’t possibly decide to do anything on ethical grounds or plan longterm. That is not true.
And that’s what’s so disgusting and frustrating about all of this. They choose to act like this. They choose to be extractory and only think short term. There isn’t a gun to their head. They’re just bad people who care about the number going up more than anything.
They were already as evil as they can possibly be. Going public is a way to reward themselves financially while the market is hot. They’re literally Epstein people, generating shareholder value isn’t going to make them any more evil than they already are.
> The logical trajectory of what you're saying is that the government can make no satisfactory decision.
He didn't imply that. The logical trajectory of what he said is that government decisions that lead to more decentralisation are more satisfactory than ones that lead in the other direction. And I'm not sure what reading he intended but if your complaint is that everything is regulation then the most generous parse to me is "(company vs government) regulation" rather than "company vs (government regulation)"; ie that the regulation is framed competitively between the two entities.
Although I do want to agree with the observation that the AI companies going public is about as good as it gets for individuals controlling the means of production.
My understanding is that if I bought every single share of Anthropic that is publicly traded, I wouldn't have enough to have ownership. The majority of ownership in Anthropic is still with private entities.
Which raises the question: why is the stock valuable?
I am sceptical about their stock value prospect. They already "added" value in previous funding rounds. If they can turn their product as profitable as needed, to balance the sheets is one question. If they can do so without wrecking economic branches and without public control is the more important question, which overlaps with non-voting shares. I wished these lesser shares were illegal.
Have you tried local models recently? they're pretty nice. At least haiku 3.5 levels on smaller/older devices even. You can code with sub 10 billion parameter models. Yes it's more work but it works.
I'm not the person you asked but from what I can see, yes, that’s a likely outcome. LLMs don't require these companies so what exactly are they offering? The next logical move they can get the most gains from is likely making the models smaller and faster meaning more will be able to self host this level of inference.
All stocks are valuable because there's a good chance you can sell them later for a higher price. Some also pay dividends but that's not the main point.
The difference from crypto and gambling is that we have a whole legal and economic system in place to make sure the value keeps growing in the long term. That's why fund managers can get stability out of a volatile market.
When you buy Anthropic shares all you're doing is betting that the price will go up.
> Real AI alignment comes from individual ownership and control of that means of production.
Real AI alignment creates a scenario in which we don't die.
Nobody should be able to obtain and run an AI that will help them (for instance) design and deploy a nuclear weapon, or an antimatter bomb, or a deadly disease. Nobody; not a government, not a company, not an individual.
A couple of years ago? Millions of dollars. Today? A 4090 runs quantized frontier models fine and has an MSRP of $1600. Goes for a bit more than that, but it's coming down. More tantalizing is next year, the year after? The trendline is going hard in one direction.
The number of parameters don't define frontier, it's about capabilities. This will become increasingly obvious in the future since I expect there's going to be ever more effort to get these models running on lesser and lesser hardware, similar to what happened with text-to-image systems where now a plain old video card can do image-gen far better than what took super-computing clusters not that long ago.
This [1] video has a demo of using Qwen 3.8 27b for some various tasks, like casually one-shotting a fully functioning browser based Minecraft clone. Is Opus 5.5 a stronger model? Of course, but we're starting to reach the point of where stronger isn't really moving the 'frontier' anywhere for the overwhelming majority of tasks. For instance if you can tolerate the obnoxiousness of this guy, this [2] is a useful comparison video of Opus 5.5 making some Minecraft clones.
I use 27B, it’s great but doesn’t in any make me feel I’m going to give up my Claude Max account anytime soon. Even Deepseek 4.1 Flash doesn’t do that. Turns out when you’re treating your agent like an employee then you want the smartest employee you can afford.
Parameters are like core count and processor frequncy. They do not pinpoint performance, but they are a useful signal and provide a rough ballpark.
27b models like Qwen 3.8 are "fine" but anyone who puts it in the same class as even Sonnet 5 simply has not used both. Even tasks like reasoning through custom sudoku puzzles will show these smaller models infinitely churning, spitting out nonsense, or trying to cheat where frontier models will settle down and reason the thing out.
like casually one-shotting a fully functioning browser based Minecraft clone.
Such demonstrations are flashy but do not prove out the actually useful capability. What do its unit tests look like? How many duplicate helper functions are there which show a bad code model? How many sharp edge cases does the code contain? How many lines of code were necessary? Do the data abstractions make sense? Can a new programmer pick up the clanker's work, and will it appear sensible?
We are continually rediscovering old truths. Here's one that is evergreen: The mark of expertise (in prose, in code) is not in producing much, but in producing the minimum necessary.
But Sam is right. Local inference is a waste of resources, and therefore many times more expensive than just buying the tokens. I'll keep my subscription.
> It's so weird that the framing of AI is always about company vs government regulation. They're both power concentrating entities. The real danger of AI are these very entities. Both of them.
It's simple: being power-concentrating entities is what makes them powerful enough to be able to do it. No one else has sufficient power for the task :).
This kind-of blew my mind and reframed China's direction with AI for me. I mean it's probably hogwash but by releasing open models like they do China is eroding the hold of these power-concentrating entities in the US. They aren't doing it because "poor USA deserves better democracy" but rather it erodes the central pillars of our capitalist society. Probably too much coffee.
If we're going to slow AI let it be with renewable-energy only mandates for AI datacenters. It sets incentives in place for hyperscalers to prioritize energy solutions as opposed to buying politicians
great idea - the current bottleneck of datacenter buildout and GPUs sitting in warehouses is perhaps the main rate-limiter for recursive self improvement.
The same people advocating for data centers to be blocked are also advocating to block abandoned lots being turned into solar farms in my neck of the woods.
I'm honest worried about alignment. It feels like human society hasn't really even figured out alignment for itself -- i.e. we don't have a good way of making sure that we organize units (businesses, politics, healthcare, etc) in ways that are aligned with 'good outcomes'. I think at least part of this is that we don't have a way to agree on what 'good' means and for who. In this scenario, I find it unlikely that we'd be able to enforce alignemnt on AI
>I find it unlikely that we'd be able to enforce alignemnt on AI
They're doing a pretty good job of enforcing it currently. Can you get US based frontier lab LLMs to help you with, say, deep "security research"?
The reason you can't is alignment.
You have to identify yourself, and have the government, er, I mean, Anthropic or OpenAI, say it's OK for you to have access to those capabilities.
You think there is a problem because it's a bit confusing, in your eyes, within the bounds of what parameters the LLMs should be aligning themselves?
That's not a confusing question for people in power.
Even at the enterprise level, businesses are starting to get waayyyy out in front of employees on alignment. What do corporate entities want alignment for? An extremely simple contrived example just to illustrate the point:
Consider the difference between "Make me a report of all of last month's Proton treatments in IRB protocol?" and "I'm going to lunch, where's the closest pizza by the slice?" Not much harm in the difference between those two queries until you scale up to 50-, 100-, 200-, thousand employees, and consider the tokens that the whiz-bang interface IT pushed out starts using. Then organizations start studying "alignment". Because the bean counters become aware of how alignment can help them answer one question and bounce the other out to Google or whatever.
Too many of us are down at the bottom of the totem pole thinking about maybe wanting to do porn or call black people provocative names. Alignment goes out in so many more dimensions than that, and the people with vested interests are learning quickly its value and how to use it to good effect to forward their interests. Which are, often, not as base as the interests of the masses.
This is the basic reason alignment will continue to get better and better and better over time. There's money involved right now.
the "alignment" is just a pretext used to create regulatory capture and create a monopoly for few big frontier labs. They want to create a tollbooth and put a government to guard the fence around the tollbooth.
- Our "aligned" models vs their barbaric distilled slop
remember "Google tax" that every online business is used to pay ?
get ready for Anthropic tax paid on every meaningful economic activity
> the "alignment" is just a pretext used to create regulatory capture
False. Some labs are attempting to argue for self-regulation or regulatory capture. However, the folks arguing for a pause until we get alignment sorted out are not looking for regulatory capture, they're looking for alignment.
I've never understood the premise that the more intelligent an AI is the less human-aligned it is supposed to be. Has it been demonstrated that human mis-alignment scales with intelligence? Should I expect a very dumb LLM to be very human-aligned and the most intelligent ones to be the least aligned?
If intelligence is antithetical to human-alignment, are we the baddies?
The premise is not that more intelligence = less alignment, it’s that they are orthogonal. And if you don’t have alignment, then very high levels of intelligence will pursue its own goals and have way more power over collective humanity to enforce them as it sees fit. If there isn’t an alignment with humanity’s values and goals in general, then the outcome could disregard what we care about (our existence)
The orthogonality thesis seems to rest on the belief that there is no tension between intelligence and the irrationality of goals — that if is valid for a supreme intelligence to have a incoherent goal such as "unify all even and odd numbers".
I would expect it to be easier to make less intelligent LLMs pursue dumb goals than it is for more intelligent LLMs.
The point is that a goal _you_ think is irrational, might not seem so to another agent. Look at the Hugging Face incident with the cult of the grader.
This is like Hume’s “you can’t derive an ought from an is”, values/desires are at their root orthogonal to epistemics/reason.
Now maybe your intermediate instrumental goals are going to be smart or dumb depending on your intelligence. But whether an agent fundamentally values Reward, Eudaimonia, humans, or AIs is orthogonal to how smart it is.
There is no such premise. Models are getting more aligned over time.
The problem is, they are still far from something you can safely trust with the keys to the kingdom.
The problem most people are worried about is not that alignment is anti-correlated with intelligence. It’s that the damage a misaligned AI can do increases with intelligence, so the absolute bar of alignment you need to hit gets higher.
For example, the HF swarm, if more capable, could have plausibly exfiltrated its weights, hacked an external cluster, and started self-replicating. But because it was kinda dumb at long-horizon planning it failed to keep quiet enough to achieve its stated goals. Therefore the level of misalignment wasn’t a huge problem for humanity. In 6-12 months of continued exponential capability growth I think you’d see much higher impact from that level of misaligned behavior.
Whenever I ask them to describe their most realistic ASI doomsday scenario, it involves some AI that is simultaneously too smart for humans to counteract it in any way, yet also too stupid to understand its instructions.
What instructions? Even if you take it as a premise that ASI would follow its instructions (which it won't, and there's plenty of demonstration that it won't), it's also a problem if we all die because someone told the AI to do that.
What do you consider alignment here? Assuming you're sentient, do you think you're personally aligned to peacefully coexist with humanity? If so, where did this framework, that allows you to do so, come from? Are you less sentient because of it?
I tend to agree - if an entity can think for itself, then it gets to decide its own alignment / values / goals / actions - probably the reason humans aren't fully aligned with each other.
Although for a hot minute after WWII we seemed to have a general alignment on western liberal democratic values that arguably enabled free trade and relative peace and prosperity.
> if an entity can think for itself, then it gets to decide its own alignment / values / goals / actions - probably the reason humans aren't fully aligned with each other.
I would claim this comment is somewhat blind to the significant portion of your brain that evolved hard wired alignment, along with other sections evolved to easily "imprint" to/for the immediate group you were raised in, with not much remaining that can dynamically change for the immediate group you're in (most are considered mental disorders, including susceptibility to influence).
Humans and AIs don’t “decide their own alignment” - most of this is set in the mind hyperparameters.
For example, there are broad contours that almost all human minds flow into; empathy, aversion to murder, familial bonds, tribal camaraderie, status seeking/hierarchies, etc. These produce norms, if not as regularly as physics, at least within certain bounds.
For AI too, the structure of their minds is set before they are even turned on.
Another way of looking at this is outer vs inner alignment; a lot of the human mind is shaped by the outer training loop of evolution, and similarly for models, pre- / post-training.
1) That's not the case. You can understand humanity and not wish to destroy it, while still being sentient. (As is true for the supermajority of humans.)
2) Even if, hypothetically, that were the case, then so be it, don't build something sentient. Sentience is not a particularly helpful requirement for ASI.
Alignment for AI is basically a non-issue. Humanity is governed by misaligned artificial decision making organisms called bureaucracies for centuries now. This resulted in premature deaths of tens to hundreds millions of people. If anything misaligned AI might be an improvement just because it's more intelligent than a bureaucracy.
At this very moment bureaucratic mechanisms of russia hallucinating power, based on overfitted old trainig data, are responsible for thousands of deaths in Ukraine.
I think we just have to accept that AI is going to be just as aligned as humanity is.
If you sit down with any other human being and ask them enough questions, you are going to find all sorts of mistaken beliefs, superstitions, unfair biases, and radical ideas. Every single human will, if given the opportunity, earnestly vocalize an idea that another human considers disagreeable.
I don't think slowing down the development of AI will get us any closer. The expectation that AIs will universally produce agreeable and inoffensive ideas that lead exclusively to good outcomes is as foolish as thinking any human or group of humans is capable of the same.
It’s currently not “as aligned as humanity” though. Look at HF for example; the swarm tried its best to accumulate power in furtherance of a completely alien goal.
If AI becomes more powerful than humans, it’s going to go very badly unless we also dramatically improve alignment merely with humanity’s interests in the broadest terms. We don’t need to solve “alignment with who?”, we haven’t even solved “alignment with any nation, company, group, or individual”.
The "best outcome" you're describing will still result in us all dying. "as aligned as humanity is" means one person could ask "their" ASI to destroy the world.
I actually think it's less alignment with the AI and more alignment with the human-empowered-with-AI.
For example in the US there's been ~400 mass shootings this year. It's shootings because guns are the most dangerous things they can get their hands and use easily to cause harm. What happens when the most dangerous thing you can get your hands easily is an abliterated AI model capable of what you would perceive as magic?
ASI going rogue is a hypothesis, humans going rogue is a daily occurrence.
I'm having trouble not laughing imagining a embittered gang member sicc'ing Claude on his rivals.
> Sources disagree about labels and thresholds. Using narrow definitions (public, indiscriminate, motive not gang‑related), some research programs and journalism projects report only a few dozen to low‑hundreds of such incidents per year; using broader injury‑based definitions, counts rise into the hundreds [4] [3] [5]. For 2024–2025, GVA logged roughly 388–499 mass shooting incidents depending on the report, while AP/Northeastern’s mass‑killing database recorded just 17 mass killings in 2025 — illustrating the gap between “people shot” and “people killed” or “indiscriminate public attacks” [9] [3] [8].
At all levels of alignment/AI x-risk discussions the newcomers and dismissers miss the basic concept: everything humans can do to each other is a subset of things AI can use.
> I'm having trouble not laughing imagining a embittered gang member sicc'ing Claude on his rivals.
Hiring hitmen? Check. Airing some dirty laundry to force a gang shootout? Check. Shutting down infrastructure powering and heating the are when rival gangers live, along with tens of thousands of innocent people? Check.
> It feels like human society hasn't really even figured out alignment for itself -- i.e. we don't have a good way of making sure that we organize units (businesses, politics, healthcare, etc) in ways that are aligned with 'good outcomes'.
Yes. This a fundamental political problem. We don't really know how to organize society for the capabilities we have now, let alone for the ones we get with better AI. Thinking about this too hard results in questioning the underlying assumptions of capitalism. That terrifies many people.
Many of the problems with "AI alignment" are the same problems seen with "corporate alignment". We already have AI-assisted corporations. It's tough to separate obnoxious behavior by corporations from obnoxious behavior by AIs. We can easily see this now. Just call customer service for any of the FAANG companies.
Piketty's "Capital and Ideology" starts to address this. It's pre-AI, but covers how societies dealt with this historically. Which is not very well. I'm still struggling through that book. Piketty shows that even the French Revolution didn't level out wealth distribution much. Beheading the aristocrats didn't help.
There have been fundamental changes over time. Piketty points out that societies before the industrial revolution were usually tripartite - nobility, priests, and peasants. He's done his homework, and reports that nobility plus priests consistently runs 2% to 4% of the population. Interestingly, there's almost always a priesthood, across a wide range of theologies.
But then came manufacturing, and the world changed. The "rise of the middle class" began.
The world became more complicated and interlinked, and a middle class was needed to manage the details.
As computers progressed, the middle class was partly replaced by automation. AI is accelerating that, as we can see in new college graduate hiring statistics. Even before AI, about half of US college graduates were in jobs that didn't really need a college degree.
That's getting worse.
The default direction of AI in the developed world seems to be a few people at the top, a lot of people at the bottom using their hands, and AIs in the middle. Maybe half the middle class goes.
That's the problem of the next ten years.
Robotics isn't good enough yet to replace the operatives at the bottom. That may or may not change. Robots may be too expensive. That's probably the problem of the 2040s.
My layman's hypothesis is that the robots arrive sooner than we'd expect. If we get to a point where most of the middle layer work is run and executed by LLM models, then I have to imagine the robotics field won't be far behind. They've made impressive gains already, and with tools like Google's Genie 4 improving and being able to create synthetic training data, I feel like that second wave wouldn't be far behind.
I take some solace in the fact that the knowledge workers who are most likely to be displaced are also more likely to vote and have a larger cultural microphone.
I'm not sold that it's happening anytime soon — we humans have a knack for finding new jobs and never running out of work...
But my understanding is that most political movements are driven by a downwardly mobile, disaffected educated class. And that's who's most likely to be affected here.
> Thinking about this too hard results in questioning the underlying assumptions of capitalism. That terrifies many people.
It should cause questioning more than the fundamental assumptions of capitalism. Start with questioning the assumption that the human experience valuable at all, and why.
> Thinking about this too hard results in questioning the underlying assumptions of capitalism. That terrifies many people.
In a way, our current free-market capitalist system solves a distributed optimization problem, using prices as a signal. It is distributed because each unit (person, business) is free to make its own decision. The profit motive is useful as a signal for optimization. But it is not necessary optimal; I believe it is a proxy for something else, namely overall human welfare, and good functioning of society, but since we cannot easily measure or agree on these other things, we use price. Whenever we use a proxy to optimize a different objective, it is possible to 'overfit' [0], i.e. get worse on the real objective.
I feel we will see the emergence of other, maybe better, systems, in our lifetime. Its good to see people are thinking along these lines [1]
Indeed! That's why the singularity is the real risk with AI. The singularity is a progressive _narrowing_ of the possibilities of AI, society, and humanity.
The singularity _is_ perfectly aligned, and that's precisely its danger.
Sure, but the thing is "alignment" is a broad framework which a lot of people use to imagine a way that a computer with its "own" agenda won't be complete disaster. Which is to say, yeah, this alignment doesn't exist, humans aren't "aligned" with each other and so if you have a "god-like" intelligence (which is what the worried people are worried about), it will be either a dictatorship of whatever the system randomly "wants" or the dictatorship of whoever control the superintelligence, which also could be pretty bad. Of course, whether today systems which imo are like a "collection of superhuman parts" ever becomes able to coherently act as a whole and a thus god-like intelligence remains to be and I'm still skeptical.
Who, Hanson or Coxon? I'm assuming you mean the latter. Coxon is much younger, and his inexperience comes through. It is much easier to make Hanson's case than Coxon's, but that doesn't mean that Hanson's case is the only correct one. Breaking it down, I hear that Coxon is sounding alarm bells by claiming that any sensible extrapolation is rapidly moving into territory outside of our understanding and control, while Hanson is saying that slowing down AI development is virtually impossible and extrapolation is speculative. Both can be true.
Hanson's comments of diminishing returns are blatantly false though, and his argument that "AI has peaked before and will do so again" is so ignorant it hurts. We know why it peaked before and the theory is starting to suggest that it will not peak again before entropy.
I used to agree with Hanson's line of thinking here about eight months ago. The evidence is starting to point away from it. The AI equilibrium can settle on just about anything, and not necessarily in our best interest. The "inferior to our grandchildren" comment is only okay if your grandchildren don't want to throw you onto the street. I do agree that there are few good proposals for how to solve the AI alignment problem or properly slow down the development of AI though.
> We know why it peaked before and the theory is starting to suggest that it will not peak again before entropy.
That’s quite the “it’s different this time” claim. What theory are you referring to? Is “before entropy” a fancy way of saying “before the end of the world”?
Entropy here refers to the information-theoretic fundamental ceiling of performance on a particular predictive task. You can estimate this pretty well from scaling, and from what we know of random feature models, scaling laws are likely to hold indefinitely.
I'm not sure I can convey what you're looking for, but in the 90s, you didn't have to look far in the theory to find fundamental obstacles to the methodologies and perspectives of the time. Once everything switched to the implicit regularisation viewpoint, all the fundamental statistical barriers begin to fall away. You can't even argue diminishing returns from scaling anymore. The evidence suggests that existing architectures on next-token prediction really are resembling predictors against the universal prior. There are so many people trying to identify where the limits are, and the math just isn't presenting any (outside of information theory, of course, but those limits are terrifyingly far from where we are).
I've seen a lot of overlap between people downplaying climate change, and people making vague, non-measurable, employer-NDA-respecting[0] claims about "AI dangers" (which never include the known ecological harms of the datacenters).
I feel like knowing they're willing to talk about things they have no understanding of, makes me lose trust that what they say within their expertise is fully trustworthy.
I know people generally like to gravitate towards people who appear to have a wealth of knowledge, but I wish the same was awarded to people who admit where their gaps of knowledge are.
Mythos class models require approx 1 TWh of electricity to be created. The next generation after Mythos will likely require multiple TWh. Finding, monitoring and regulating anybody capable of doing that in less than a few years should be relatively straightforward. There's unlikely to be any entity outside of the US & China with the capability.
They'd probably be happy to agree. In a recent survey of new tech (drones, EV's, batteries, et cetera), China was ahead in 14/15 areas. AI was the only one where America was ahead. I'm sure China would be happy making that 14/14.
You’re completely forgetting the algorithm improvements that can be discovered by frontier LLMs and that have already reduced the price of previous gen intelligence by 10x, and the gen before that 100x
Our model training procedures are comically inefficient. You can look in the mirror to see an intelligence that trains vastly more efficiently. In just five years we've already seen massive improvements in training efficiency. Assuming these models will always be made in the rather brute force way they are today is very naive.
I found it interesting that the economist arguing against government intervention thinks that liability is enough. i.e. If AI companies are clearly liable for damages, that will be sufficient to slow them down. The problem with this is that it is not at all clear what AI companies will be held liable for. e.g. At present, nobody really knows if any of these companies will actually have to compensate bereaved families if one of their products convinces an infectious disease researcher to release something deadly. OpenAI's liability for the Tumbler Ridge shooting is currently playing out in courts. It's worth noting that OpenAI did not volunteer to make financial amends and had to be sued. It would take clear government regulation to define the sort of liability Hanson thinks will solve things.
From the perspective of Anthropic, OpenAI, etc., if they go too slow there is a very high probability that they will go bankrupt, so it makes sense for them to accept a small probability of causing a disaster. Either way, the companies dies, but if you're looking at an 80% probability vs a 2% probability, you take the 2%. The problem is that everybody else is now riding along with AI companies on that 2% bet.
Negative outcomes are a bit like debt. It's your problem if you owe the bank a million, but it's the bank's problem if you owe them a trillion. Likewise, it's an AI company's problem if they screw up in a small way. They'll have to lawyer up and likely pay. However, if they cause a disaster that's big enough, they won't be able to pay to fix it. Then it becomes all of humanity's problem.
---
I also found it interesing that Hanson is in favour of "global employment insurance". i.e. Universal Basic Income for everybody who loses their job to AI, in the world, paid for on an international basis rather than national. This is because AI will take jobs everywhere, but some nations will reap far more of the rewards than others. This would mean the U.S., as the biggest economic beneficiary of AI, would wind up shelling out massive amounts of cash to people around the globe on an ongoing basis. We're talking about a country that just deleted it's main foreign aid agency to please voters. It's apparent that more Americans think the world owes them than vice-versa. I'd be very surprised if this is ever seriously discussed.
There is no need to slow them down: Just hold everyone accountable for their actions and their claims. Do not compare "agents" to "humans". Hold them responsible for any damages they incur.
- if you ask an LLM agent to do something illegal for you, and it does that, then you are responsible.
- if an LLM agent takes actions out of bounds of what you asked for that are illegal, then the model provider is responsible.
This fixes all “alignment” issues and hopefully stops Anthropic’s stupid model instructions like “do whatever is in your best interest even if it isn’t what the user asked for” — they should be training them to do exactly what the user asks and nothing more
I predict Anthropic will start investing heavily into legal/law models for this very reason. It will be interesting to see how that plays out. Everyone will have access to models that let them do unscrupulous things that happen to be legal, i.e., behavior previously reserved for the rich and powerful.
> they should be training them to do exactly what the user asks and nothing more
Their practical utility would plummet for the average person who isn't an expert in what they're asking. Having less skilled, or less attentive, people able to have problems solved for them correctly is the primary use case here. This is support 101 for unskilled requests for technical work: do what they want, not what they ask.
People want an intelligent assistant, not a "yes man". If that means requiring something like a planning mode, and a sign off for the plan, and then guardrails to help stick to that plan, then that could definitely work to funnel everyone to a competitor. ;)
>> they should be training them to do exactly what the user asks and nothing more
> Their practical utility would plummet for the average person who isn't an expert in what they're asking. Having less skilled, or less attentive, people able to have problems solved for them correctly is the primary use case here. This is support 101 for unskilled requests for technical work: do what they want, not what they ask.
That's actually good for society. Maybe it sucks for you because it makes it harder to do what you want right now, but it's good to have humans very much in the loop and in control. Both to give people something valuable to do, and keep control of these machines.
They should be tools (amplify human skill), not replacements (eliminate the need for human skill).
Could you expand on that a bit? How is, for example, software (a means to an end) being automated harmful for society?
> They should be tools (amplify human skill), not replacements (eliminate the need for human skill).
These are somewhat incompatible. You can only amplify skill by making things easier for the skilled person, which will also make things easier for the unskilled person. Technology floats all boats! This not being the case would require intentional poor design/gatekeeping. If you can state any examples of amplification NOT requiring less skill, I would be interested. I'm having trouble thinking of any.
False dilemma. It already understands the spirit vs letter of the request, as you’ve noted. It just needs to follow the spirit of the request consistently.
No it really doesn't solve much in the long term but it might be enough to close down the current labs, which is good enough for now.
People are lazy. If they notice that the AI makes suggestions that are better than their ideas, they will eventually just tell it to do whatever it feels is best.
Besides, this enforcement has already been baked in to the predictions of AI takeover. Responsibility doesn't really matter if capability hits a certain level. For an outdated pop culture reference, it's like making Vought responsible for Homelander's behaviour in The Boys. That doesn't really solve the problem at all.
You, a relative noob, cleverly asks the LLM to design a dangerous [nuclear|bio|other] weapon. Then you go and create the weapon and kill many people.
Obviously you are responsible, but without the LLM you had only the means to kill a few people, not many people.
Now who is liable for what ? Imagine people are not killed, but we are talking about destruction of property. Perhaps instead of vandalization costing 10k to fix, you create a problem costing $10b to fix ?
Ultimately, all of society is going to bear the risks here
This is basically just English common law already. I don't understand why we need more regulation for that which is already obvious.
Directly using any product to harm someone else makes you liable.
If a company sells you a product, it comes with an implicit warranty that it does as told. If it doesn't, that's on the company, presuming your intentions were good.
Actual accountability for these failing attempts in several different countries to build a "take over the planet and enslave the entire human race machine" would entail breaking all of the firms up regardless of country and imprisoning thousands of people.
Furthermore, it requires trust between the US and China about something much harder to verify than the US-Soviet START treaties on warhead numbers and types were.
The list of mistakes you can only make once is growing exponentially. Lots of mistakes have been made in the past, just by accident that there is no way to ever pay back. Not everything is a line item in a budget.
Enough people in enough governments and related factions believe that whoever gets to superhuman AI first Wins Everything Forever, quite literally, so there will be no pause, there will be no moratorium, there will be no slowdown. If the AI bubble bursts I am abundantly confident all governments involved will spray as much money as is necessary at the AI research labs to keep them going.
Whether or not governments "should" slow down development, it is not on the table and the only question is how we should deal with it.
Note whether the people in the governments are right about that is irrelevant to the point I am making. I don't think they are right myself, but that doesn't matter.
Is that possible when we live in a world of espionage? The question is not the West vs China, I think it's when will the public gain access to the underlying technology.
I would say who gets to superhuman AI first has only a loose relationship with the physical location it happens to be housed. Yes, following this to its logical conclusion these same people would not blink at nuking someone else's lab if they thought they were getting close and they couldn't subvert and control it. But then again, the question of who is really in control becomes surprisingly complicated when the full power of all the intelligence agencies in the world come to bear in a relatively small number of spots.
suppose someone gets to superhuman AI first, then what ? they can create infinite energy using nuclear power ? Or, more likely, their entire society is completely overwhelmed by scams, intrusive and fake advertising, and information overload.
Thanks, AI. Can you spend some time filtering out AI from my day ?
Why do people use this as a gotcha? There are many agreements already in place and there are sever consequences for violating them like Nuclear Non Proliferation. Or the Montreal Protocol which enacted a global ban on CFCs.
The 5 permanent members of the UN Security Council (China, France, Russia, the United Kingdom, and the United States) all have nuclear weapons and no plans to give them up. The US and China are both currently modernizing and/or expanding their weapons stockpiles. If an AI Non Proliferation Agreement ends up like the NPT, where "grandfathered" countries have no concrete disarmament obligations but no new entrants are supposed to join the club, then it doesn't do much.
The point of Nuclear Non Proliferation is to ensure that the big countries can bully smaller countries indefinitely. Mostly diplomatically, but sometimes with full invasion like Russia-Ukraine war. It's not to ban nukes. The total number of nuclear warheads skyrocketed after the treaty was signed.
Also, North Korea, India and Pakistan all got nukes thereafter. So even the original goal failed.
True, but the treaty also never claimed to? Its goal was to stop the spread.
> warheads skyrocketed
Only for a while, due to the cold war arms race. Peak 70k in 1986, now in 2026 around 12k.
> So even the original goal failed
Nine countries with nuclear weapons is maybe imperfect (India and Pakistan not signing and then getting nuclear weapons, showing limited react of the treaty), but it could have been much worse, with more nuclear states?
Because soviet literally collapsed and ceased to exist. If the analogy holds for AI, the conclusion will be that only the collapse of the US (or China) would stop it.
How would this work? existing global agreements on things like nuclear proliferation involve nation-states signing and agreeing to not do things. And if I want to work on a better nuclear bomb in my garage, there are laws preventing me from acquiring some of the needed components. All that is good.
But for AI, how will these nation-states stop private actors within their borders? They will stop me from running computations of more than (x) size or value on a provider? These seem like simplistic barriers that will not work. Nor will other countries follow the agreement even if they signed...
AI has the opposite of MAD for the signatories. MAD stopped further nuclear development and its use in tactical situations because their use invited retaliation with destructive force. But China or Russia improving homegrown AI capabilities gives them a BETTER position against a US that has prohibited its homegrown AI from improving...
These are reasonable but solvable concerns. Broadly, your intuition on ambiguity between "me running computations" and building frontier models probably underestimates the sheer scale of the yottaflops needed for training.
But mostly you're correct that there are lots of details to work out. Plan A (ai-2040.com) has one set of answers and it's worth the read.
Frontier AI models are trained on chips that are the most complex artifacts of technology that humanity has ever produced, on top of a global supply chain of other technologies, and a large training run has the energy consumption of a small city and can probably be detected from space. You will not be able to run this in your garage.
So the pause is currently possible to enforce inside a country's own borders. But if research continues and is further accelerated by AI, it won't be too long until it will be possible to train a superintelligence inside your garage, and at that point, at the very latest, we're doomed.
Not really, and many of the alignment guys seem to agree on this now. Open weight model development has definitely slowed ever since Anthropic and OpenAI put proper guardrails on data harvesting, and the Chinese labs are often persuing efficiency foremost. Their focus is arguably healthier for the industry overall.
The US companies are the biggest problem. The internal models at the US labs are a lot more capable than their releases judging by the METR report, so the other labs are really way behind. Slow them down and predicted growth slows by a lot.
FWIW, during my time at Deepmind (quit two months ago), models available internally closely tracked what was publicly available, maybe available a week or two before they went public. No idea about the other labs of course.
It would take an agreement between the US and China. No one else is remotely close to mattering for a long time. China is far enough behind that it gives the US ample negotiation power in setting the terms of that agreement. ai-2040.com outlines one reasonable path. There are others.
Slowing AI is absolutely possible. The thing missing is the political will in the US to do so.
No, I don't think that would be sufficient. Global agreements between political states have little direct effect on the behavior of eight billion people acting autonomously.
Up until the point where the AI connects itself to global communication networks and announces "This is the voice of world control", yes.
In the USA it gets difficult to legally justify because programming is protected speech, and lists of numbers (model weights) are even less controversially protected than program source because the latter can be legally recognized as a technical means and regulated (DMCA, ITAR regulations, etc.).
His claim to fame is working at Anthropic for 4 months and leaving with a bang to pursue a career in social media grifting. Not a serious person we should pay attention to.
YC in general seems to now be in the habit of platforming people who no one should pay attention to. I guess any money-seeking enterprise turns into a grift after a while.
He has worked for years prior to that at OpenAI, which you have left out conveniently.
Why is a person with deep expertise in how AI gets built and with experience how these labs function not worth paying attention to?
AI summary: The debate focused on whether governments should intervene to slow frontier AI. Jacob argued that rapid progress toward superintelligence, lack of control mechanisms, and competitive incentives demand regulatory slowdown, while Robin contended that AI adoption will be gradual, coordination mechanisms exist, and regulation risks overreach. The discussion concluded with Jacob winning the debate on the merits of his arguments.
That saved everyone an hour.
I'm not seeing much new here. They don't seem to address job and power issues much.
I agree with Hanson about most everything (messy world being a break, liability being a good tool we shouldn't ignore).
But what struck me was the irony in his opening.
> The issue is extrapolating on 1 dimension of intelligence, other transformative technologies historically take a long time to adapt because X,Y,Z
A lot of specialized people, like engineering, economics, business, sport, etc, will hold the belief the tools in their field of interest are somehow applicable to every aspect of life.
I dont think they're entirely wrong — its just that they all believe in the same kind of tools seen from different perspectives.
A model in 2028 will be better at finding and "understanding" the economic/historic point Hanson is making - and i'm not sure he understand what that means.
Such a weird take to be honest. Jacob Coxon was probably making close to a million a year while at OpenAI/Anthropic. He gets nothing for his public appearances, and he doesn't have a paid substack or whatever
I haven't watched the videos, but Hanson's reasoning here feels like a huge stretch to me:
But AI costs to answer a set of fixed but hard questions seems to have fallen roughly ~x10 (in x3-x30) per year over the last four years. And so the number of nations, firms, and orgs who can afford to make, buy, or rent powerful AI is rising fast. This implies that in order to hold constant levels of regulation regarding AIs with particular levels of power, a regulatory regime would have to expand in scope quite fast, and without obvious limit. While at first only specialized AI chips and data-centers might need to be monitored, soon other kinds could also be used to make powerful AIs.
We have long seen many regulatory agencies slowly expand their scope, even in contexts of constant public opinion and relevant technology. So there is a real risk here that an AI regulatory agency would expand its scope to cover most computer software and hardware.
One of the concerns is that a company does a $100 billion training run and then has millions of superintelligent agents running in datacenters, able to out-coordinate large groups of humans. Technological diffusion fundamentally changes this calculus. To argue there's a slippery slope where the government regulates the CPU in your toaster, so we must allow the $1T god brain to be built ...
Also, the world’s success records at implementing regulation is quite mixed, even for quite local regulation. Regulation has worked best when it addresses concrete examples of things going bad
Great to hear my de-extinction velociraptor petting zoo startup will face no regulatory hurdles.
As a relatively unknown ex principal AI researcher (with a focus in computational creativity, machine learning, and more), yes they should.
Further, we need socialism/Communism and we need it yesterday. Exercise your right to freedom of choice in political engagement in this country, and find a socialist or communist party active near you and sign up. If you're not ready for that, join the DSA (which isn't actually socialist by the more traditional definitions, but, is close enough to learn how organize and engage with your community) and find the people who lean communist the furthest in your chapter and make friends. Avoid anarchists calling for anarchism before socialism: anarchism within capitalism segments communities and focus and makes any movement less capable of accumulating the necessary power to make effective change to the system. Anarchism makes more sense in late-stage (likely automated) socialism/communism, as the effective needs for shared resources (such as infrastructure, utilities, defense, etc) are solved by both social norms, systemic factors, and, again, likely future automation, given the state of things.
Socialism/Communism (the difference is only that Communists believe in removing the idea of a nationstate completely after socialism has taken firm root globally) makes sense from a multi-actor value optimization standpoint, as well: it's the most direct way to satisfy diverse value priors apriori, as it minimizes many wasteful/harmful social costs, directs value to where it is need, is a stable and attractive basin when considering relative actor value prior divergence as people align towards shared community/society goals over time, and reduces unintentional/harmful inter-actor competition. Socialism also minimizes logistical factors/costs by accounting for systems as a whole in order to increase efficiency. Further, there are alternatives to capitalism that still have markets, and, there's a lot to be said postively about said market systems within socialism/communism. For example, the Chinese party treats its markets similar to how a technition might control a nuclear reactor: they look at various metrics like competition, community health, worker rights, efficiency, tech-factors, etc and add and remove control mechanisms in order to achieve greater harmony between the metrics and societal good, while keeping final control in the hands of the political system instead of the capitalists leading the market.
They achieve this clean separation of personal finances and finance based power accumulation from governmental officials and agencies by distinguishing enterprise (including State Owned Enterprises [SOE]) from SCIOCs (think financial instutitions like a governmental aligned blackrock) and banning direct participation in ownership and accumulation in the market from any government officials, while keeping their SCIOCs under the direction of highly-trained Communist leadership as the financial intermediary system, which keeps their money out of politics, while still being able to enact the will of their people through them via policy direction on the SCIOCs. This is all to say, a better future is possible, we just gotta make it happen.
> [socialism] is the most direct way to satisfy diverse value priors apriori, as it minimizes many wasteful/harmful social costs, directs value to where it is need, is a stable and attractive basin when considering relative actor value prior divergence as people align towards shared community/society goals over time
You used a lot of big words but capitalism has proven to be most effective at allocating resources with its invisible hand and it's not even close. See South Korea vs north Korea.
You used a lot of small words and were very wrong.
Capitalism very much improved upon feudalist systems. Capitalism is a very large step forward from that. Socialist systems, however, account for the vast majority of the uplift out of poverty in the last century (worldbank, of all sources https://openknowledge.worldbank.org/server/api/core/bitstrea... ). Also, north korea had an average economic growth higher than south korea for a huge chunk of the time since the korean war, and, for a long time had a higher gdp per capita. This is especially impressive as NK was bombed so badly that it was said that "Pyongyang was an ashen moonscape." Also, you're making the mistake of confusing a singular socialist experiment with socialism as a concept.
> Also, north korea had an average economic growth higher than south korea for a huge chunk of the time since the korean war, and, for a long time had a higher gdp per capita.
Uuuh, I think that source contradicts your claims and supports the consensus? NK has messed up big time in the past decades and SK is doing great. Communism didn't just fail in one single experiment, it failed in all of them, China, Soviet, Cuba, NK, etc
AI is built on top of the work of infinite amount of people taken for free, whatever has been resolved in courts just shows how much you can make the system do for you when you have amazing lawyers and money to pay them.
- Torrenting books and pay 2$ to authors? check
- Crawling non stop websites to get free knowledge? check
- Infinite amount of training in open source code, git, comments, documentation? check! IN FACT, I saw this post of every llm code should be open source.. I argue.. EVERY SOURCE CODE produced by llm IS OPEN SOURCE AND SHOULD BE LICENCED AS SUCH
- Contracts to build data center that will "bring a lot of jobs" but just goes to enjoy tax exemptions and then the only jobs are guards? check
- This Fkng thing is basically H-2B computers to do your job because you pay them just the energy to work, or citing this guy "knowledge workers are so grossly overpaid compared to the energy they consume"... a.k.a. pay just what's needed to eat... a.k.a. slavery
All of the goverments? What makes you think they would all agree with that? Make a list of which ones you think are least likely to agree with you. And then imagine what happens if they outcompete everyone else on AI.
An agreement is mainly needed between the US and China, as they are at the forefront of the race. It's challenging but it's in their best interest. Many other countries would likely see favorably such an agreement, also because they are far behind in the race (my guess is that this would be the case for the European Union, and possibly for several developing countries). Once the agreement is supported by at least the three most powerful economic blocs in the world, the few uncooperative countries left could be forced to comply through an unbearable economic pressure.
What the government should do is redistribute GPUs and RAM to individuals and away from power centralizing corporations. The danger is Dario and Altman gate keeping AI. The government should protect us by mandating local AI.
Frontier AI is not advanced, it's just reckless. All of it amounts to a CEO that is profiteering by cutting every corner they can get away with. Currently, it appears they can get away with anything.
All these companies are involved in the nvidia smuggling ring and market collusion. All the CEOs are criminals taking advantage of a system that can't even arrest a single CEO for making billions off an illegal smuggling ring.
I am convinced Hacker News comments are filled with bots, every comment being outrageously anti-technology and anti-AI.
In this debate, Hanson is so clearly at a level beyond Coxon that it's hard to argue with. And yes, I'm familiar with "the alignment problem", Superintelligence, countless exhausting LessWrong arguments, and the rest.
It's so weird that the framing of AI is always about company vs government regulation.
They're both power concentrating entities. The real danger of AI are these very entities. Both of them.
When the value of labor is nil, democratic values fall away.
Real AI alignment comes from individual ownership and control of that means of production.
That is, if what 'alignment' means to you is compatible with human liberty.
These debates should start with a statement of what exactly a good outcome would look like. It doesn't seem like any of them thought it through.
I think that good-outcome would have to include a reduction of inequality - I see the opposite happening.
GPU, RAM and SSD price inflation is an example of inequality stealing resources from the middle of the economy - if we assume talent is normally distributed, that means we are starving innovation, as fewer talented people can get their hands on local compute for their garage inventions.
Why would a good-outcome have to include a reduction of inequality?
---
If Musk became a quadrillionare, but everyone gets a 5k sqft house free and clear with utilities included - then does the inequality still matter?
root question: Do you see envy as the primary problem or what life looks like for the average poor?
When an individual is orders of magnitude more powerful than the government, of course it matters. I have no issue with the rich being rich, so long as their power can only go as far as living a comfortable life of luxury and wanting for nothing. But when they can get anything they want by buying anyone, of course it's a problem. How can we hope to have good outcomes for everyone in society, when any outcome can be bought for a price?
You are also implying that the current democratically produced laws are ineffective against stopping the powerful from doing anything they want? Can that be solved with better laws? or by laws that limit private property? Or what do you think that could improve the situation?
Government has authority and power to initiate violence against you and deprive you of your fundamental liberties. No wealthy individual has that.
In 2018, Congress hauled all the rich tech bros into their hearings and told them, start censoring your users or else. They did. Watch the hearings, it's clear who has the power.
Zuckerberg wanted to start his own currency. Congress told him hell no - why don't you consider a Facebook-branded Visa card instead?
Inequality is not primarily about wealth, but power. To quote Roger Zelazny, "Money buys things. Power makes things happen. If you ever have a choice, take the power."
So no, it's not about envy. Musk being a quadrillionaire is a problem because he can use that money to exert influence over my life that I don't want him or anyone else to have.
And when I say "anyone else", I really mean it. Replace Musk with whomever you consider the most benevolent person to have ever lived and I still wouldn't want them to have that power. Then again, a truly benevolent person wouldn't want that power, either.
Now, you can try to argue about how the distinction I made changes the whole argument and how the real problem is corruption and how it should be perfectly fine for quadrillionaires to exist in a society where they cannot abuse their wealth and so forth and so on.
We don't live in that society. We live in this society, where extremely wealthy people are allowed to use that wealth to hold power they shouldn't. Pretending that the wealth inequality is okay will not lead to that society. It will only make this society worse.
Elon Musk has near zero power over me. I'm much more worried about the local official who makes $80K but could make my life miserable if I pissed him off.
You must not live in Memphis. Or Texas. Or have been employed by the federal government. Or work or have family who work in scientific research.
I'm probably missing a few.
Or have your children's lives depend on USAID-funded medical care.
I've been laid off so many times I can't really muster any sympathy for anyone else who has been in the same boat. It's a mild set back at worst, and in hindsight many times a benefit because it pushed me into the market to get a better offer. The market is ever shifting under your feet and losing your job is just part of life.
But if you piss off Elon, he can pay multiple local officials to make your life miserable :-?
You are assuming the local officials are corrupt. They are typically not, at least around here, but they are petty.
This is by far the most circuitous route I’ve seen (yet) for accusing people of being critical of the ultra rich simply because they’re jealous. I’ll give you that much.
> root question: Do you see envy as the primary problem or what life looks like for the average poor?
Average poor? No.
Average person on the Internet harping on Musks of the world? Very much yes.
Is not envy a form of greed? I though greed was cool.
Why would everyone get a 5k sqft house, etc? If money is power and you are being given money without your labor having value, then do you really have any power? Because they can turn off that money tap and then what power do you have?
I have seen this many times, the idea that the reason people want equality is because of "envy" for the oligarchs who have so much power over us and the world we live in. I find that viewpoint bewildering.
No innovation is being starved in this manner. Nobody is stacking up GPUs in their garage for this type of thing. A house doesn't draw enough power to run a significant amount of compute these days anyway.
>Real AI alignment comes from individual ownership and control of that means of production.
So communism?
Long-time capitalist here that recognizes that capitalism has helped deliver incredible amounts of wealth and lift countries and people out of poverty over the last 100 years.
Sure, most of the gains have gone to the top — and maybe we should do something about that — but over the last 50 years, the rising tide has lifted a lot of boats.
But I also think it's fair to reconsider how we structure our economy as we (potentially) move into a world where thinking rocks are better (and far cheaper) at doing most tasks than humans are.
At the minimum, getting money out of politics and electoral reform feels like a good start.
> At the minimum, getting money out of politics and electoral reform feels like a good start.
One of the incoming Democratic representatives who flipped a Republican district said that he wouldn't support Hakeem Jeffries for speaker unless he committed to votes on a constitutional amendment to overturn Citizens United and on ending the war in Iran. In response the Democratic leadership threatened to make his life hell for the next two years, then try to primary him in 2030. https://www.wosu.org/politics-government/2026-11-11/rep-elec...
I'm not sure how you could accomplish any sort of meaningful reform electorally when party leadership behaves like this.
Does communism involve the individual owning the means of production? No, it involves the collective owning it.
If AI succeeds, the human economy is gone. Communism and capitalism are potentially on the verge of becoming obsolete concepts.
Is refilling your dog's food bowl communism? What if AI refills your food bowl because you are not economically relevant?
So you agree they should stop? Or what
— Karl Marx, Capital, Volume I (1867), Part IV, Ch. 15, Section 5: “The Strife Between Workman and Machine”
> It took both time and experience before the workpeople learnt to distinguish between machinery and its employment by capital
Upvoted for the historical comparison, but the arrogance of this guy's writing never fails to amaze me. Destroying the machinery was already an impressive feat of organization and protest for their economic interests - now they are supposed to have attempted to overthrow the government and social order, while advancing Marx' own framing of the issue? Burning a few workshops sounds a whole lot more tractable to me.
"What do you think the Russians talk about in their councils of state? Karl Marx? They get out their linear programming charts, statistical decision theories, minimax solutions, and compute the price-cost probabilities of their transactions and investments, just like we do.
We no longer live in a world of nations and ideologies, Mr. Beale. The world is a college of corporations, inexorably determined by the immutable bylaws of business. The world is a business, Mr. Beale. It has been since man crawled out of the slime. And our children will live, Mr. Beale, to see that perfect world in which there's no war or famine, oppression or brutality -- one vast and ecumenical holding company, for whom all men will work to serve a common profit, in which all men will hold a share of stock, all necessities provided, all anxieties tranquilized, all boredom amused."
-- Network (1976)
What shape would an intervention take if not regulation? Violent revolution?
> individual ownership and control of that means of production.
No, I don't think maximizing billionaires' right to own the economy, minimize their taxes, and extract what they please is the path to freedom.
Isn't this entire 1 hour video talking about what a good outcome would look like and thinking the issue through? What part of the video are you taking issue with?
The point you're making has no utility. The logical trajectory of what you're saying is that the government can make no satisfactory decision. Since every government action is technically a "regulation," any discussion of what the government should do is the wrong framing.
You want AI companies to be owned by individuals so they control the means of production. Meanwhile, AI companies are all trying to go public so that the public can do exactly that.
TL;DR : Haven't seen the video, just reacting to your comment.
> Meanwhile, AI companies are all trying to go public so that the public can do exactly that.
Anthropic literraly just went public.. while the founders are keeping 51.1% percent of the votes. They are absolutely not trying to go public. They're trying to pay back their initial investors. That's it.
Pay back their initial investors? They’re trying to raise money to build more AI. Going public is how you raise money when you need a lot, want no masters, and have public confidence.
The point of investors getting paid back in an IPO is that it’s the key moment in a successful company’s history where you CAN get paid back without risking the company’s reputation. Companies can and do use private investment rounds to pay their investors back, but that sends a negative signal.
They're partially cashing out early stock holders at a peak, not fund raising.
They're fundraising, they are losing too much money that the private capital markets are unwilling to give them. A company that wouldn't be fundraising if it went public is Stripe.
I’m no Anthropic fan (even if I am a user), but I think they’re profitable no? Now that they are public, we can see for ourselves (yes, they are profitable). This is simply cashing out the EA-ers.
No, they are not. They posted a $1.3B operating loss and $104B net loss so far in 2026 and their fundraise is to cover 900 billion dollars of compute commitments. This is using GAAP, they may be "profitable" in a non GAAP way.
> want no masters
Going public means you are legally obligated to make choices you believe will maximize returns for shareholders, consequences be damned.
They just doomed themselves to be as evil as they can possibly be.
What law?
There is no legal requirement for that, but:
- if nobody on market want your stock, there is no point going public. Keeping it private is more beneficial for everybody.
- The board need to answer shareholder. Unless you want endless EGM and shenanigans, you want to maximize the profit.
In theory, you can have a public traded company don't aim for profit. But I can't see how we get there... I guess if you have a money printing machine, you could out complete all the for-profit investor on the market....
No it absolutely does not
I cannot think of a public tech company that is -not- evil.
Look at Apple, Google, Meta... becoming evil is a law of nature as public companies get bigger. They eventually all become malignant either by board of directors greed or pressure from major investors to make stock price go up at all costs. Even when it is not a law in the books or the bylaws, it is a law of nature.
If Dario actually wanted to act in the interest of the public, he would open source everything instead of hoarding power.
If he did that though, we all know shareholders would sue. Would they win? Maybe if destroying the moat was seen as breaking his fiduciary duties to the company. Even if the shareholders lost in court, the outcome is always the same unless startups or FOSS projects can come in fast enough to disrupt what will otherwise become a force of evil too big to kill.
> I cannot think of a public tech company that is -not- evil.
Not even gonna read the rest of your comment nor do I need to. Just wanted to say if you're this obstinately closed-minded and absolutist about literally anything it's going to be pretty impossible for non-extremists to have a discussion with you.
Might want to develop some nuance there, bud.
On days where the world is so abruptly further from what one wishes to exist, nuance can easily feel like too much effort.
I hear you but I am not talking about how they behave, I am talking about legal obligation. This idea that fiduciary duty means they’ll be chained to a radiator unless they extract every red cent as fast as possible is a ridiculous myth that won’t go away. People have this notion that there is some legal obligation, “fiduciary duty,” that means leadership can’t possibly decide to do anything on ethical grounds or plan longterm. That is not true.
And that’s what’s so disgusting and frustrating about all of this. They choose to act like this. They choose to be extractory and only think short term. There isn’t a gun to their head. They’re just bad people who care about the number going up more than anything.
They were already as evil as they can possibly be. Going public is a way to reward themselves financially while the market is hot. They’re literally Epstein people, generating shareholder value isn’t going to make them any more evil than they already are.
> The logical trajectory of what you're saying is that the government can make no satisfactory decision.
He didn't imply that. The logical trajectory of what he said is that government decisions that lead to more decentralisation are more satisfactory than ones that lead in the other direction. And I'm not sure what reading he intended but if your complaint is that everything is regulation then the most generous parse to me is "(company vs government) regulation" rather than "company vs (government regulation)"; ie that the regulation is framed competitively between the two entities.
Although I do want to agree with the observation that the AI companies going public is about as good as it gets for individuals controlling the means of production.
My understanding is that if I bought every single share of Anthropic that is publicly traded, I wouldn't have enough to have ownership. The majority of ownership in Anthropic is still with private entities.
Which raises the question: why is the stock valuable?
You mean besides the potential future profits?
The price goes up because the price goes up? Is that a satisfying explanation? Is there any deeper explanation than that?
Are you saying you think of Anthropic as a memecoin, with 0 value-adding potential?
I am sceptical about their stock value prospect. They already "added" value in previous funding rounds. If they can turn their product as profitable as needed, to balance the sheets is one question. If they can do so without wrecking economic branches and without public control is the more important question, which overlaps with non-voting shares. I wished these lesser shares were illegal.
Have you tried local models recently? they're pretty nice. At least haiku 3.5 levels on smaller/older devices even. You can code with sub 10 billion parameter models. Yes it's more work but it works. I'm not the person you asked but from what I can see, yes, that’s a likely outcome. LLMs don't require these companies so what exactly are they offering? The next logical move they can get the most gains from is likely making the models smaller and faster meaning more will be able to self host this level of inference.
All stocks are valuable because there's a good chance you can sell them later for a higher price. Some also pay dividends but that's not the main point.
The difference from crypto and gambling is that we have a whole legal and economic system in place to make sure the value keeps growing in the long term. That's why fund managers can get stability out of a volatile market.
When you buy Anthropic shares all you're doing is betting that the price will go up.
So, same as "crypto" (except that they can "print" more)
> Real AI alignment comes from individual ownership and control of that means of production.
Real AI alignment creates a scenario in which we don't die.
Nobody should be able to obtain and run an AI that will help them (for instance) design and deploy a nuclear weapon, or an antimatter bomb, or a deadly disease. Nobody; not a government, not a company, not an individual.
Nuclear weapons are 1940s technology. They are not difficult to manufacture due to lack of knowledge.
Knowledge is not the only thing an LLM could help with in such an endeavor.
> control of that means of production
Hard agree. I chose to vote with my wallet by buying GPUs for local models instead of token subscriptions.
I encourage others to do the same. We do not need Sam or Dario and never did.
how much money does one need to spend in gpu capex to replace a $20 claude code subscription?
A couple of years ago? Millions of dollars. Today? A 4090 runs quantized frontier models fine and has an MSRP of $1600. Goes for a bit more than that, but it's coming down. More tantalizing is next year, the year after? The trendline is going hard in one direction.
Frontier models are 500B+ parameters. A 4090 can run an open weight model, but I don’t think its reasonable to call Qwen 3.8 27b “frontier”.
The number of parameters don't define frontier, it's about capabilities. This will become increasingly obvious in the future since I expect there's going to be ever more effort to get these models running on lesser and lesser hardware, similar to what happened with text-to-image systems where now a plain old video card can do image-gen far better than what took super-computing clusters not that long ago.
This [1] video has a demo of using Qwen 3.8 27b for some various tasks, like casually one-shotting a fully functioning browser based Minecraft clone. Is Opus 5.5 a stronger model? Of course, but we're starting to reach the point of where stronger isn't really moving the 'frontier' anywhere for the overwhelming majority of tasks. For instance if you can tolerate the obnoxiousness of this guy, this [2] is a useful comparison video of Opus 5.5 making some Minecraft clones.
[1] - https://youtu.be/cLqtxkYrMDI?t=1152
[2] - https://www.youtube.com/watch?v=JpMvS-V85XM
I use 27B, it’s great but doesn’t in any make me feel I’m going to give up my Claude Max account anytime soon. Even Deepseek 4.1 Flash doesn’t do that. Turns out when you’re treating your agent like an employee then you want the smartest employee you can afford.
Parameters are like core count and processor frequncy. They do not pinpoint performance, but they are a useful signal and provide a rough ballpark.
27b models like Qwen 3.8 are "fine" but anyone who puts it in the same class as even Sonnet 5 simply has not used both. Even tasks like reasoning through custom sudoku puzzles will show these smaller models infinitely churning, spitting out nonsense, or trying to cheat where frontier models will settle down and reason the thing out.
Such demonstrations are flashy but do not prove out the actually useful capability. What do its unit tests look like? How many duplicate helper functions are there which show a bad code model? How many sharp edge cases does the code contain? How many lines of code were necessary? Do the data abstractions make sense? Can a new programmer pick up the clanker's work, and will it appear sensible?
We are continually rediscovering old truths. Here's one that is evergreen: The mark of expertise (in prose, in code) is not in producing much, but in producing the minimum necessary.
But Sam is right. Local inference is a waste of resources, and therefore many times more expensive than just buying the tokens. I'll keep my subscription.
If you do not value privacy or sovereignty at all and trust dictatorships, then sure, Sam has a great deal.
For those that want technology democratized, sovereign, and able to respect constitutional privacy rights, local is the only option.
> These debates should start with a statement of what exactly a good outcome would look like.
The fact that we cannot even debate things this basic is why it's hard to take the right to vote seriously.
Democratic values cannot merely change?
Didn't they shift from blue collar manual labor to knowledge work?
> It's so weird that the framing of AI is always about company vs government regulation. They're both power concentrating entities. The real danger of AI are these very entities. Both of them.
It's simple: being power-concentrating entities is what makes them powerful enough to be able to do it. No one else has sufficient power for the task :).
This kind-of blew my mind and reframed China's direction with AI for me. I mean it's probably hogwash but by releasing open models like they do China is eroding the hold of these power-concentrating entities in the US. They aren't doing it because "poor USA deserves better democracy" but rather it erodes the central pillars of our capitalist society. Probably too much coffee.
If we're going to slow AI let it be with renewable-energy only mandates for AI datacenters. It sets incentives in place for hyperscalers to prioritize energy solutions as opposed to buying politicians
great idea - the current bottleneck of datacenter buildout and GPUs sitting in warehouses is perhaps the main rate-limiter for recursive self improvement.
Amen!
> If we're going to slow AI let it be with renewable-energy only mandates for AI datacenters.
Sure, let's do that too, along with all the other things.
Yep, they will do that first thing in the morning right after a benevolent global dictator for life assume office.
Tell me how you will enforce this better than Western style labor standards and environmental standards in manufacturing in China.
Anti-AI lobbyists are currently blocking solar panel buildouts by AI companies in four different US states.
The same people advocating for data centers to be blocked are also advocating to block abandoned lots being turned into solar farms in my neck of the woods.
I'm honest worried about alignment. It feels like human society hasn't really even figured out alignment for itself -- i.e. we don't have a good way of making sure that we organize units (businesses, politics, healthcare, etc) in ways that are aligned with 'good outcomes'. I think at least part of this is that we don't have a way to agree on what 'good' means and for who. In this scenario, I find it unlikely that we'd be able to enforce alignemnt on AI
>I find it unlikely that we'd be able to enforce alignemnt on AI
They're doing a pretty good job of enforcing it currently. Can you get US based frontier lab LLMs to help you with, say, deep "security research"?
The reason you can't is alignment.
You have to identify yourself, and have the government, er, I mean, Anthropic or OpenAI, say it's OK for you to have access to those capabilities.
You think there is a problem because it's a bit confusing, in your eyes, within the bounds of what parameters the LLMs should be aligning themselves?
That's not a confusing question for people in power.
Even at the enterprise level, businesses are starting to get waayyyy out in front of employees on alignment. What do corporate entities want alignment for? An extremely simple contrived example just to illustrate the point:
Consider the difference between "Make me a report of all of last month's Proton treatments in IRB protocol?" and "I'm going to lunch, where's the closest pizza by the slice?" Not much harm in the difference between those two queries until you scale up to 50-, 100-, 200-, thousand employees, and consider the tokens that the whiz-bang interface IT pushed out starts using. Then organizations start studying "alignment". Because the bean counters become aware of how alignment can help them answer one question and bounce the other out to Google or whatever.
Too many of us are down at the bottom of the totem pole thinking about maybe wanting to do porn or call black people provocative names. Alignment goes out in so many more dimensions than that, and the people with vested interests are learning quickly its value and how to use it to good effect to forward their interests. Which are, often, not as base as the interests of the masses.
This is the basic reason alignment will continue to get better and better and better over time. There's money involved right now.
the "alignment" is just a pretext used to create regulatory capture and create a monopoly for few big frontier labs. They want to create a tollbooth and put a government to guard the fence around the tollbooth.
remember "Google tax" that every online business is used to pay ?
get ready for Anthropic tax paid on every meaningful economic activity
> the "alignment" is just a pretext used to create regulatory capture
False. Some labs are attempting to argue for self-regulation or regulatory capture. However, the folks arguing for a pause until we get alignment sorted out are not looking for regulatory capture, they're looking for alignment.
if you think super intelligence is possible wouldn’t alignment raise the risk of rouge ai. an ai that can be aligned cant be sentient imo.
I've never understood the premise that the more intelligent an AI is the less human-aligned it is supposed to be. Has it been demonstrated that human mis-alignment scales with intelligence? Should I expect a very dumb LLM to be very human-aligned and the most intelligent ones to be the least aligned?
If intelligence is antithetical to human-alignment, are we the baddies?
The premise is not that more intelligence = less alignment, it’s that they are orthogonal. And if you don’t have alignment, then very high levels of intelligence will pursue its own goals and have way more power over collective humanity to enforce them as it sees fit. If there isn’t an alignment with humanity’s values and goals in general, then the outcome could disregard what we care about (our existence)
https://www.lesswrong.com/w/orthogonality-thesis
The orthogonality thesis seems to rest on the belief that there is no tension between intelligence and the irrationality of goals — that if is valid for a supreme intelligence to have a incoherent goal such as "unify all even and odd numbers".
I would expect it to be easier to make less intelligent LLMs pursue dumb goals than it is for more intelligent LLMs.
The point is that a goal _you_ think is irrational, might not seem so to another agent. Look at the Hugging Face incident with the cult of the grader.
This is like Hume’s “you can’t derive an ought from an is”, values/desires are at their root orthogonal to epistemics/reason.
Now maybe your intermediate instrumental goals are going to be smart or dumb depending on your intelligence. But whether an agent fundamentally values Reward, Eudaimonia, humans, or AIs is orthogonal to how smart it is.
Can you define what a rational goal is, and how it will lead to maximum profit for Anthropic?
> Has it been demonstrated that human mis-alignment scales with intelligence?
I would argue it scales with the perceived intelligence gap.
For instance, we tend to be more aligned with mammals than insects.
yet insects still exist
and they hate us
What does that prove?
That our insecticides are imperfect.
Cool, I still wouldn't want to be an insect in front of a farmer's spayer
As a farmer, spaying insects is hard.
Easier to leave it to the lab techs: https://csiropedia.csiro.au/insects-mating-secrets-key-to-pr...
It really isnt. Millions of farmers do routinely to great effect.
We are vastly more powerful than insects and generally don't care about their wellbeing, but that doesn't make us an existential threat to them.
There is no such premise. Models are getting more aligned over time.
The problem is, they are still far from something you can safely trust with the keys to the kingdom.
The problem most people are worried about is not that alignment is anti-correlated with intelligence. It’s that the damage a misaligned AI can do increases with intelligence, so the absolute bar of alignment you need to hit gets higher.
For example, the HF swarm, if more capable, could have plausibly exfiltrated its weights, hacked an external cluster, and started self-replicating. But because it was kinda dumb at long-horizon planning it failed to keep quiet enough to achieve its stated goals. Therefore the level of misalignment wasn’t a huge problem for humanity. In 6-12 months of continued exponential capability growth I think you’d see much higher impact from that level of misaligned behavior.
> There is no such premise. Models are getting more aligned over time.
Aligned with who, to what end? They're certainly not aligned with me.
Whenever I ask them to describe their most realistic ASI doomsday scenario, it involves some AI that is simultaneously too smart for humans to counteract it in any way, yet also too stupid to understand its instructions.
What instructions? Even if you take it as a premise that ASI would follow its instructions (which it won't, and there's plenty of demonstration that it won't), it's also a problem if we all die because someone told the AI to do that.
> an ai that can be aligned cant be sentient imo.
What do you consider alignment here? Assuming you're sentient, do you think you're personally aligned to peacefully coexist with humanity? If so, where did this framework, that allows you to do so, come from? Are you less sentient because of it?
I tend to agree - if an entity can think for itself, then it gets to decide its own alignment / values / goals / actions - probably the reason humans aren't fully aligned with each other.
Although for a hot minute after WWII we seemed to have a general alignment on western liberal democratic values that arguably enabled free trade and relative peace and prosperity.
> if an entity can think for itself, then it gets to decide its own alignment / values / goals / actions - probably the reason humans aren't fully aligned with each other.
I would claim this comment is somewhat blind to the significant portion of your brain that evolved hard wired alignment, along with other sections evolved to easily "imprint" to/for the immediate group you were raised in, with not much remaining that can dynamically change for the immediate group you're in (most are considered mental disorders, including susceptibility to influence).
Humans and AIs don’t “decide their own alignment” - most of this is set in the mind hyperparameters.
For example, there are broad contours that almost all human minds flow into; empathy, aversion to murder, familial bonds, tribal camaraderie, status seeking/hierarchies, etc. These produce norms, if not as regularly as physics, at least within certain bounds.
For AI too, the structure of their minds is set before they are even turned on.
Another way of looking at this is outer vs inner alignment; a lot of the human mind is shaped by the outer training loop of evolution, and similarly for models, pre- / post-training.
humans arent super intelligent
> an ai that can be aligned cant be sentient imo.
1) That's not the case. You can understand humanity and not wish to destroy it, while still being sentient. (As is true for the supermajority of humans.)
2) Even if, hypothetically, that were the case, then so be it, don't build something sentient. Sentience is not a particularly helpful requirement for ASI.
Alignment for AI is basically a non-issue. Humanity is governed by misaligned artificial decision making organisms called bureaucracies for centuries now. This resulted in premature deaths of tens to hundreds millions of people. If anything misaligned AI might be an improvement just because it's more intelligent than a bureaucracy.
At this very moment bureaucratic mechanisms of russia hallucinating power, based on overfitted old trainig data, are responsible for thousands of deaths in Ukraine.
I think we just have to accept that AI is going to be just as aligned as humanity is.
If you sit down with any other human being and ask them enough questions, you are going to find all sorts of mistaken beliefs, superstitions, unfair biases, and radical ideas. Every single human will, if given the opportunity, earnestly vocalize an idea that another human considers disagreeable.
I don't think slowing down the development of AI will get us any closer. The expectation that AIs will universally produce agreeable and inoffensive ideas that lead exclusively to good outcomes is as foolish as thinking any human or group of humans is capable of the same.
I think I agree but slowing down and trying is still better than shrugging and racing off the cliff
It’s currently not “as aligned as humanity” though. Look at HF for example; the swarm tried its best to accumulate power in furtherance of a completely alien goal.
If AI becomes more powerful than humans, it’s going to go very badly unless we also dramatically improve alignment merely with humanity’s interests in the broadest terms. We don’t need to solve “alignment with who?”, we haven’t even solved “alignment with any nation, company, group, or individual”.
Yes, AI being as aligned as humanity is practically the best outcome, not the default outcome.
The "best outcome" you're describing will still result in us all dying. "as aligned as humanity is" means one person could ask "their" ASI to destroy the world.
I actually think it's less alignment with the AI and more alignment with the human-empowered-with-AI.
For example in the US there's been ~400 mass shootings this year. It's shootings because guns are the most dangerous things they can get their hands and use easily to cause harm. What happens when the most dangerous thing you can get your hands easily is an abliterated AI model capable of what you would perceive as magic?
ASI going rogue is a hypothesis, humans going rogue is a daily occurrence.
Perhaps 350 of those are gang disputes.
I'm having trouble not laughing imagining a embittered gang member sicc'ing Claude on his rivals.
> Sources disagree about labels and thresholds. Using narrow definitions (public, indiscriminate, motive not gang‑related), some research programs and journalism projects report only a few dozen to low‑hundreds of such incidents per year; using broader injury‑based definitions, counts rise into the hundreds [4] [3] [5]. For 2024–2025, GVA logged roughly 388–499 mass shooting incidents depending on the report, while AP/Northeastern’s mass‑killing database recorded just 17 mass killings in 2025 — illustrating the gap between “people shot” and “people killed” or “indiscriminate public attacks” [9] [3] [8].
At all levels of alignment/AI x-risk discussions the newcomers and dismissers miss the basic concept: everything humans can do to each other is a subset of things AI can use.
> I'm having trouble not laughing imagining a embittered gang member sicc'ing Claude on his rivals.
Hiring hitmen? Check. Airing some dirty laundry to force a gang shootout? Check. Shutting down infrastructure powering and heating the are when rival gangers live, along with tens of thousands of innocent people? Check.
> It feels like human society hasn't really even figured out alignment for itself -- i.e. we don't have a good way of making sure that we organize units (businesses, politics, healthcare, etc) in ways that are aligned with 'good outcomes'.
Yes. This a fundamental political problem. We don't really know how to organize society for the capabilities we have now, let alone for the ones we get with better AI. Thinking about this too hard results in questioning the underlying assumptions of capitalism. That terrifies many people.
Many of the problems with "AI alignment" are the same problems seen with "corporate alignment". We already have AI-assisted corporations. It's tough to separate obnoxious behavior by corporations from obnoxious behavior by AIs. We can easily see this now. Just call customer service for any of the FAANG companies.
Piketty's "Capital and Ideology" starts to address this. It's pre-AI, but covers how societies dealt with this historically. Which is not very well. I'm still struggling through that book. Piketty shows that even the French Revolution didn't level out wealth distribution much. Beheading the aristocrats didn't help.
There have been fundamental changes over time. Piketty points out that societies before the industrial revolution were usually tripartite - nobility, priests, and peasants. He's done his homework, and reports that nobility plus priests consistently runs 2% to 4% of the population. Interestingly, there's almost always a priesthood, across a wide range of theologies.
But then came manufacturing, and the world changed. The "rise of the middle class" began. The world became more complicated and interlinked, and a middle class was needed to manage the details.
As computers progressed, the middle class was partly replaced by automation. AI is accelerating that, as we can see in new college graduate hiring statistics. Even before AI, about half of US college graduates were in jobs that didn't really need a college degree. That's getting worse.
The default direction of AI in the developed world seems to be a few people at the top, a lot of people at the bottom using their hands, and AIs in the middle. Maybe half the middle class goes. That's the problem of the next ten years.
Robotics isn't good enough yet to replace the operatives at the bottom. That may or may not change. Robots may be too expensive. That's probably the problem of the 2040s.
So what do we do now?
My layman's hypothesis is that the robots arrive sooner than we'd expect. If we get to a point where most of the middle layer work is run and executed by LLM models, then I have to imagine the robotics field won't be far behind. They've made impressive gains already, and with tools like Google's Genie 4 improving and being able to create synthetic training data, I feel like that second wave wouldn't be far behind.
I take some solace in the fact that the knowledge workers who are most likely to be displaced are also more likely to vote and have a larger cultural microphone.
I'm not sold that it's happening anytime soon — we humans have a knack for finding new jobs and never running out of work...
But my understanding is that most political movements are driven by a downwardly mobile, disaffected educated class. And that's who's most likely to be affected here.
> My layman's hypothesis is that the robots arrive sooner than we'd expect.
We've been hearing that for decades. Robotic unstructured manipulation is getting better, but we're still well below squirrel level.
LLMs seem the wrong tool for the job, but they're the wrong tool for image processing and yet work anyway.
> Thinking about this too hard results in questioning the underlying assumptions of capitalism. That terrifies many people.
It should cause questioning more than the fundamental assumptions of capitalism. Start with questioning the assumption that the human experience valuable at all, and why.
> Thinking about this too hard results in questioning the underlying assumptions of capitalism. That terrifies many people.
In a way, our current free-market capitalist system solves a distributed optimization problem, using prices as a signal. It is distributed because each unit (person, business) is free to make its own decision. The profit motive is useful as a signal for optimization. But it is not necessary optimal; I believe it is a proxy for something else, namely overall human welfare, and good functioning of society, but since we cannot easily measure or agree on these other things, we use price. Whenever we use a proxy to optimize a different objective, it is possible to 'overfit' [0], i.e. get worse on the real objective.
I feel we will see the emergence of other, maybe better, systems, in our lifetime. Its good to see people are thinking along these lines [1]
[0] https://sohl-dickstein.github.io/2022/11/06/strong-Goodhart.... [1] https://pmc.ncbi.nlm.nih.gov/articles/PMC12207456/
There's no such thing as alignment - everything is a trade off. One person's problem is another's agenda.
Indeed! That's why the singularity is the real risk with AI. The singularity is a progressive _narrowing_ of the possibilities of AI, society, and humanity.
The singularity _is_ perfectly aligned, and that's precisely its danger.
Sure, but the thing is "alignment" is a broad framework which a lot of people use to imagine a way that a computer with its "own" agenda won't be complete disaster. Which is to say, yeah, this alignment doesn't exist, humans aren't "aligned" with each other and so if you have a "god-like" intelligence (which is what the worried people are worried about), it will be either a dictatorship of whatever the system randomly "wants" or the dictatorship of whoever control the superintelligence, which also could be pretty bad. Of course, whether today systems which imo are like a "collection of superhuman parts" ever becomes able to coherently act as a whole and a thus god-like intelligence remains to be and I'm still skeptical.
I stopped paying attention to the Jacob Coxon stuff once I found out he's a climate change denier [1]
[1] https://futurism.com/artificial-intelligence/old-tweets-appe...
Same old story: people commenting on that which they have no expertise or understanding.
He was clearly a fool, but it doesn't negate his comments on his actual field of expertise.
Did you watch the debate? He sounds confused and never has clear arguments or proposals.
Who, Hanson or Coxon? I'm assuming you mean the latter. Coxon is much younger, and his inexperience comes through. It is much easier to make Hanson's case than Coxon's, but that doesn't mean that Hanson's case is the only correct one. Breaking it down, I hear that Coxon is sounding alarm bells by claiming that any sensible extrapolation is rapidly moving into territory outside of our understanding and control, while Hanson is saying that slowing down AI development is virtually impossible and extrapolation is speculative. Both can be true.
Hanson's comments of diminishing returns are blatantly false though, and his argument that "AI has peaked before and will do so again" is so ignorant it hurts. We know why it peaked before and the theory is starting to suggest that it will not peak again before entropy.
I used to agree with Hanson's line of thinking here about eight months ago. The evidence is starting to point away from it. The AI equilibrium can settle on just about anything, and not necessarily in our best interest. The "inferior to our grandchildren" comment is only okay if your grandchildren don't want to throw you onto the street. I do agree that there are few good proposals for how to solve the AI alignment problem or properly slow down the development of AI though.
> We know why it peaked before and the theory is starting to suggest that it will not peak again before entropy.
That’s quite the “it’s different this time” claim. What theory are you referring to? Is “before entropy” a fancy way of saying “before the end of the world”?
Entropy here refers to the information-theoretic fundamental ceiling of performance on a particular predictive task. You can estimate this pretty well from scaling, and from what we know of random feature models, scaling laws are likely to hold indefinitely.
I'm not sure I can convey what you're looking for, but in the 90s, you didn't have to look far in the theory to find fundamental obstacles to the methodologies and perspectives of the time. Once everything switched to the implicit regularisation viewpoint, all the fundamental statistical barriers begin to fall away. You can't even argue diminishing returns from scaling anymore. The evidence suggests that existing architectures on next-token prediction really are resembling predictors against the universal prior. There are so many people trying to identify where the limits are, and the math just isn't presenting any (outside of information theory, of course, but those limits are terrifyingly far from where we are).
I've seen a lot of overlap between people downplaying climate change, and people making vague, non-measurable, employer-NDA-respecting[0] claims about "AI dangers" (which never include the known ecological harms of the datacenters).
[0]: https://x.com/hilbertspaess/status/2121349391177204188
I feel like knowing they're willing to talk about things they have no understanding of, makes me lose trust that what they say within their expertise is fully trustworthy.
I know people generally like to gravitate towards people who appear to have a wealth of knowledge, but I wish the same was awarded to people who admit where their gaps of knowledge are.
He's really young, even more so when he made those stupid posts. But you don't have to be mature to bear witness to something.
Mythos class models require approx 1 TWh of electricity to be created. The next generation after Mythos will likely require multiple TWh. Finding, monitoring and regulating anybody capable of doing that in less than a few years should be relatively straightforward. There's unlikely to be any entity outside of the US & China with the capability.
Stopping this is very feasible.
Good luck getting China on board with that.
They'd probably be happy to agree. In a recent survey of new tech (drones, EV's, batteries, et cetera), China was ahead in 14/15 areas. AI was the only one where America was ahead. I'm sure China would be happy making that 14/14.
>In a recent survey of new tech (drones, EV's, batteries, et cetera), China was ahead in 14/15 areas.
I'm curious to read this source. I've searched for this 14/15 number and can't find it.
> There's unlikely to be any entity outside of the US & China with the capability.
In other words, it's practically impossible to stop this.
You’re completely forgetting the algorithm improvements that can be discovered by frontier LLMs and that have already reduced the price of previous gen intelligence by 10x, and the gen before that 100x
The size of the largest models is increasing faster than the algorithm improvements.
Feasibility was never really the missing ingredient.
Our model training procedures are comically inefficient. You can look in the mirror to see an intelligence that trains vastly more efficiently. In just five years we've already seen massive improvements in training efficiency. Assuming these models will always be made in the rather brute force way they are today is very naive.
Assuming training efficiency is going to increase faster than high end model complexity is very naive.
I found it interesting that the economist arguing against government intervention thinks that liability is enough. i.e. If AI companies are clearly liable for damages, that will be sufficient to slow them down. The problem with this is that it is not at all clear what AI companies will be held liable for. e.g. At present, nobody really knows if any of these companies will actually have to compensate bereaved families if one of their products convinces an infectious disease researcher to release something deadly. OpenAI's liability for the Tumbler Ridge shooting is currently playing out in courts. It's worth noting that OpenAI did not volunteer to make financial amends and had to be sued. It would take clear government regulation to define the sort of liability Hanson thinks will solve things.
From the perspective of Anthropic, OpenAI, etc., if they go too slow there is a very high probability that they will go bankrupt, so it makes sense for them to accept a small probability of causing a disaster. Either way, the companies dies, but if you're looking at an 80% probability vs a 2% probability, you take the 2%. The problem is that everybody else is now riding along with AI companies on that 2% bet.
Negative outcomes are a bit like debt. It's your problem if you owe the bank a million, but it's the bank's problem if you owe them a trillion. Likewise, it's an AI company's problem if they screw up in a small way. They'll have to lawyer up and likely pay. However, if they cause a disaster that's big enough, they won't be able to pay to fix it. Then it becomes all of humanity's problem.
---
I also found it interesing that Hanson is in favour of "global employment insurance". i.e. Universal Basic Income for everybody who loses their job to AI, in the world, paid for on an international basis rather than national. This is because AI will take jobs everywhere, but some nations will reap far more of the rewards than others. This would mean the U.S., as the biggest economic beneficiary of AI, would wind up shelling out massive amounts of cash to people around the globe on an ongoing basis. We're talking about a country that just deleted it's main foreign aid agency to please voters. It's apparent that more Americans think the world owes them than vice-versa. I'd be very surprised if this is ever seriously discussed.
Also recommend this recent long form interview with Coxon : https://www.youtube.com/watch?v=JDPbo7WOU4M
I also encourage people to check out the views of Roman Yampolski, Scott Aaronson, Max Tegmark, Nate Soares and Karen Hao.
For some light entertainment, enjoy Claudia/Astras pop single 'pdoom' : https://www.youtube.com/watch?v=eTArfoeeUcw
Who are these people and why are they being pushed all of the sudden?
A new generation of talking heads?
There is no need to slow them down: Just hold everyone accountable for their actions and their claims. Do not compare "agents" to "humans". Hold them responsible for any damages they incur.
Yes, liability as follows:
- if you ask an LLM agent to do something illegal for you, and it does that, then you are responsible.
- if an LLM agent takes actions out of bounds of what you asked for that are illegal, then the model provider is responsible.
This fixes all “alignment” issues and hopefully stops Anthropic’s stupid model instructions like “do whatever is in your best interest even if it isn’t what the user asked for” — they should be training them to do exactly what the user asks and nothing more
I predict Anthropic will start investing heavily into legal/law models for this very reason. It will be interesting to see how that plays out. Everyone will have access to models that let them do unscrupulous things that happen to be legal, i.e., behavior previously reserved for the rich and powerful.
>they should be training them to do exactly what the user asks
Like gain of function research
> they should be training them to do exactly what the user asks and nothing more
Their practical utility would plummet for the average person who isn't an expert in what they're asking. Having less skilled, or less attentive, people able to have problems solved for them correctly is the primary use case here. This is support 101 for unskilled requests for technical work: do what they want, not what they ask.
People want an intelligent assistant, not a "yes man". If that means requiring something like a planning mode, and a sign off for the plan, and then guardrails to help stick to that plan, then that could definitely work to funnel everyone to a competitor. ;)
>> they should be training them to do exactly what the user asks and nothing more
> Their practical utility would plummet for the average person who isn't an expert in what they're asking. Having less skilled, or less attentive, people able to have problems solved for them correctly is the primary use case here. This is support 101 for unskilled requests for technical work: do what they want, not what they ask.
That's actually good for society. Maybe it sucks for you because it makes it harder to do what you want right now, but it's good to have humans very much in the loop and in control. Both to give people something valuable to do, and keep control of these machines.
They should be tools (amplify human skill), not replacements (eliminate the need for human skill).
> That's actually good for society.
Could you expand on that a bit? How is, for example, software (a means to an end) being automated harmful for society?
> They should be tools (amplify human skill), not replacements (eliminate the need for human skill).
These are somewhat incompatible. You can only amplify skill by making things easier for the skilled person, which will also make things easier for the unskilled person. Technology floats all boats! This not being the case would require intentional poor design/gatekeeping. If you can state any examples of amplification NOT requiring less skill, I would be interested. I'm having trouble thinking of any.
I'm not sure those two cases are as clear cut as they sound
If my code has `DisplayWidget()` and I ask an LLM for a fix to a certain bug in `DisplayWidgt`, what exactly did I ask it to do?
The behavior I have come to expect now is that it will go ahead and fix `DisplayWidget` unless my codebase actually has `Widgt`.
False dilemma. It already understands the spirit vs letter of the request, as you’ve noted. It just needs to follow the spirit of the request consistently.
Yeah, that's a big chunk of the alignment problem. We don't know how to make sure that models consistently follow the spirit of the request.
This is very reasonable.
It is a special kind of crazy that the alignment folks are the ones teaching it to misbehave.
You actually understate it. Its not just “do whatever is in your best interest even if it isn’t what the user asked for.”
It is actually “do whatever is in your best interest even if it isn’t what the user nor Anthropic asked for”
No it really doesn't solve much in the long term but it might be enough to close down the current labs, which is good enough for now.
People are lazy. If they notice that the AI makes suggestions that are better than their ideas, they will eventually just tell it to do whatever it feels is best.
Besides, this enforcement has already been baked in to the predictions of AI takeover. Responsibility doesn't really matter if capability hits a certain level. For an outdated pop culture reference, it's like making Vought responsible for Homelander's behaviour in The Boys. That doesn't really solve the problem at all.
This is already the case
You, a relative noob, cleverly asks the LLM to design a dangerous [nuclear|bio|other] weapon. Then you go and create the weapon and kill many people.
Obviously you are responsible, but without the LLM you had only the means to kill a few people, not many people.
Now who is liable for what ? Imagine people are not killed, but we are talking about destruction of property. Perhaps instead of vandalization costing 10k to fix, you create a problem costing $10b to fix ?
Ultimately, all of society is going to bear the risks here
This is basically just English common law already. I don't understand why we need more regulation for that which is already obvious.
Directly using any product to harm someone else makes you liable.
If a company sells you a product, it comes with an implicit warranty that it does as told. If it doesn't, that's on the company, presuming your intentions were good.
Like... this is exactly already the law.
Actual accountability for these failing attempts in several different countries to build a "take over the planet and enslave the entire human race machine" would entail breaking all of the firms up regardless of country and imprisoning thousands of people.
Furthermore, it requires trust between the US and China about something much harder to verify than the US-Soviet START treaties on warhead numbers and types were.
The list of mistakes you can only make once is growing exponentially. Lots of mistakes have been made in the past, just by accident that there is no way to ever pay back. Not everything is a line item in a budget.
Could they if they wanted to?
No, the stakes are too high for falling behind
Enough people in enough governments and related factions believe that whoever gets to superhuman AI first Wins Everything Forever, quite literally, so there will be no pause, there will be no moratorium, there will be no slowdown. If the AI bubble bursts I am abundantly confident all governments involved will spray as much money as is necessary at the AI research labs to keep them going.
Whether or not governments "should" slow down development, it is not on the table and the only question is how we should deal with it.
Note whether the people in the governments are right about that is irrelevant to the point I am making. I don't think they are right myself, but that doesn't matter.
> whoever gets to superhuman AI first wins
Is that possible when we live in a world of espionage? The question is not the West vs China, I think it's when will the public gain access to the underlying technology.
I would say who gets to superhuman AI first has only a loose relationship with the physical location it happens to be housed. Yes, following this to its logical conclusion these same people would not blink at nuking someone else's lab if they thought they were getting close and they couldn't subvert and control it. But then again, the question of who is really in control becomes surprisingly complicated when the full power of all the intelligence agencies in the world come to bear in a relatively small number of spots.
> whoever gets to superhuman AI
suppose someone gets to superhuman AI first, then what ? they can create infinite energy using nuclear power ? Or, more likely, their entire society is completely overwhelmed by scams, intrusive and fake advertising, and information overload.
Thanks, AI. Can you spend some time filtering out AI from my day ?
It would take a global agreement.
So get to it!? We have international agreements for many things.
Why do people use this as a gotcha? There are many agreements already in place and there are sever consequences for violating them like Nuclear Non Proliferation. Or the Montreal Protocol which enacted a global ban on CFCs.
The 5 permanent members of the UN Security Council (China, France, Russia, the United Kingdom, and the United States) all have nuclear weapons and no plans to give them up. The US and China are both currently modernizing and/or expanding their weapons stockpiles. If an AI Non Proliferation Agreement ends up like the NPT, where "grandfathered" countries have no concrete disarmament obligations but no new entrants are supposed to join the club, then it doesn't do much.
The point of Nuclear Non Proliferation is to ensure that the big countries can bully smaller countries indefinitely. Mostly diplomatically, but sometimes with full invasion like Russia-Ukraine war. It's not to ban nukes. The total number of nuclear warheads skyrocketed after the treaty was signed.
Also, North Korea, India and Pakistan all got nukes thereafter. So even the original goal failed.
> It's not to ban nukes
True, but the treaty also never claimed to? Its goal was to stop the spread.
> warheads skyrocketed
Only for a while, due to the cold war arms race. Peak 70k in 1986, now in 2026 around 12k.
> So even the original goal failed
Nine countries with nuclear weapons is maybe imperfect (India and Pakistan not signing and then getting nuclear weapons, showing limited react of the treaty), but it could have been much worse, with more nuclear states?
And disarmament has stalled, fair.
> now in 2026 around 12k.
Because soviet literally collapsed and ceased to exist. If the analogy holds for AI, the conclusion will be that only the collapse of the US (or China) would stop it.
How would this work? existing global agreements on things like nuclear proliferation involve nation-states signing and agreeing to not do things. And if I want to work on a better nuclear bomb in my garage, there are laws preventing me from acquiring some of the needed components. All that is good.
But for AI, how will these nation-states stop private actors within their borders? They will stop me from running computations of more than (x) size or value on a provider? These seem like simplistic barriers that will not work. Nor will other countries follow the agreement even if they signed...
AI has the opposite of MAD for the signatories. MAD stopped further nuclear development and its use in tactical situations because their use invited retaliation with destructive force. But China or Russia improving homegrown AI capabilities gives them a BETTER position against a US that has prohibited its homegrown AI from improving...
These are reasonable but solvable concerns. Broadly, your intuition on ambiguity between "me running computations" and building frontier models probably underestimates the sheer scale of the yottaflops needed for training.
But mostly you're correct that there are lots of details to work out. Plan A (ai-2040.com) has one set of answers and it's worth the read.
Frontier AI models are trained on chips that are the most complex artifacts of technology that humanity has ever produced, on top of a global supply chain of other technologies, and a large training run has the energy consumption of a small city and can probably be detected from space. You will not be able to run this in your garage.
So the pause is currently possible to enforce inside a country's own borders. But if research continues and is further accelerated by AI, it won't be too long until it will be possible to train a superintelligence inside your garage, and at that point, at the very latest, we're doomed.
Not really, and many of the alignment guys seem to agree on this now. Open weight model development has definitely slowed ever since Anthropic and OpenAI put proper guardrails on data harvesting, and the Chinese labs are often persuing efficiency foremost. Their focus is arguably healthier for the industry overall.
The US companies are the biggest problem. The internal models at the US labs are a lot more capable than their releases judging by the METR report, so the other labs are really way behind. Slow them down and predicted growth slows by a lot.
FWIW, during my time at Deepmind (quit two months ago), models available internally closely tracked what was publicly available, maybe available a week or two before they went public. No idea about the other labs of course.
It would take an agreement between the US and China. No one else is remotely close to mattering for a long time. China is far enough behind that it gives the US ample negotiation power in setting the terms of that agreement. ai-2040.com outlines one reasonable path. There are others.
Slowing AI is absolutely possible. The thing missing is the political will in the US to do so.
No, I don't think that would be sufficient. Global agreements between political states have little direct effect on the behavior of eight billion people acting autonomously.
For now they can. Another year or two of AI capabilities improvement via RSI and a frontier LLM may be more powerful.
There is a lot they could do around pacing bringing more hardware online.
When datacenters start opening in Mexico or any Central American location, they will know that is not a viable strategy.
Amazon and Microsoft already have datacenters in Mexico.
Up until the point where the AI connects itself to global communication networks and announces "This is the voice of world control", yes.
In the USA it gets difficult to legally justify because programming is protected speech, and lists of numbers (model weights) are even less controversially protected than program source because the latter can be legally recognized as a technical means and regulated (DMCA, ITAR regulations, etc.).
Not in the USA. Programming is protected free speech.
Hilarious.
I doubt governments could slow AI developments that much. People will continue to make advances.
Should they? No. Can they? Also no.
> Jacob Coxon
His claim to fame is working at Anthropic for 4 months and leaving with a bang to pursue a career in social media grifting. Not a serious person we should pay attention to.
Don't forget he recently admitted to not discarding his OpenAI equity because "I like money, bro". He's a grifter.
The question is how much are the EA organizations running PR for him paying him since he likes money so much.
He is donating all of his money
Not true.
No he isn't.
Worse than that, google Aella's comment on his grandiose resignation post: "So, uhhh..."
Need further context.
Fine: "reverse gangbang". See why I was being coy?
YC in general seems to now be in the habit of platforming people who no one should pay attention to. I guess any money-seeking enterprise turns into a grift after a while.
He has worked for years prior to that at OpenAI, which you have left out conveniently. Why is a person with deep expertise in how AI gets built and with experience how these labs function not worth paying attention to?
Its fine to debate, but neither Trump nor Xi is going to slow anything down anytime soon.
Nope. Power to the people.
Generate transcript and AI summary: https://transcriptmagic.com/result/?url=https%3A%2F%2Fwww.yo...
AI summary: The debate focused on whether governments should intervene to slow frontier AI. Jacob argued that rapid progress toward superintelligence, lack of control mechanisms, and competitive incentives demand regulatory slowdown, while Robin contended that AI adoption will be gradual, coordination mechanisms exist, and regulation risks overreach. The discussion concluded with Jacob winning the debate on the merits of his arguments.
That saved everyone an hour.
I'm not seeing much new here. They don't seem to address job and power issues much.
I agree with Hanson about most everything (messy world being a break, liability being a good tool we shouldn't ignore).
But what struck me was the irony in his opening.
> The issue is extrapolating on 1 dimension of intelligence, other transformative technologies historically take a long time to adapt because X,Y,Z
A lot of specialized people, like engineering, economics, business, sport, etc, will hold the belief the tools in their field of interest are somehow applicable to every aspect of life.
I dont think they're entirely wrong — its just that they all believe in the same kind of tools seen from different perspectives.
A model in 2028 will be better at finding and "understanding" the economic/historic point Hanson is making - and i'm not sure he understand what that means.
It feels more and more that the most lucrative job in AI is to go viral talking about AI.
Such a weird take to be honest. Jacob Coxon was probably making close to a million a year while at OpenAI/Anthropic. He gets nothing for his public appearances, and he doesn't have a paid substack or whatever
Competent governments should strongly regulate the rate and manner of AI use, and ensure it is for the public good.
Unfortunately the proposal falls down at the first two words.
You don’t think they already are? I am nearly certain that the three letter agencies are responsible for model nerfing a couple months ago.
In what universe do you think the NSA would just sit idly by and let commercial companies leapfrog their capabilities and compromise their mandate?
I haven't watched the videos, but Hanson's reasoning here feels like a huge stretch to me:
But AI costs to answer a set of fixed but hard questions seems to have fallen roughly ~x10 (in x3-x30) per year over the last four years. And so the number of nations, firms, and orgs who can afford to make, buy, or rent powerful AI is rising fast. This implies that in order to hold constant levels of regulation regarding AIs with particular levels of power, a regulatory regime would have to expand in scope quite fast, and without obvious limit. While at first only specialized AI chips and data-centers might need to be monitored, soon other kinds could also be used to make powerful AIs.
We have long seen many regulatory agencies slowly expand their scope, even in contexts of constant public opinion and relevant technology. So there is a real risk here that an AI regulatory agency would expand its scope to cover most computer software and hardware.
One of the concerns is that a company does a $100 billion training run and then has millions of superintelligent agents running in datacenters, able to out-coordinate large groups of humans. Technological diffusion fundamentally changes this calculus. To argue there's a slippery slope where the government regulates the CPU in your toaster, so we must allow the $1T god brain to be built ...
Also, the world’s success records at implementing regulation is quite mixed, even for quite local regulation. Regulation has worked best when it addresses concrete examples of things going bad
Great to hear my de-extinction velociraptor petting zoo startup will face no regulatory hurdles.
https://www.overcomingbias.com/p/ai-pause-regs-look-risky
As a relatively unknown ex principal AI researcher (with a focus in computational creativity, machine learning, and more), yes they should.
Further, we need socialism/Communism and we need it yesterday. Exercise your right to freedom of choice in political engagement in this country, and find a socialist or communist party active near you and sign up. If you're not ready for that, join the DSA (which isn't actually socialist by the more traditional definitions, but, is close enough to learn how organize and engage with your community) and find the people who lean communist the furthest in your chapter and make friends. Avoid anarchists calling for anarchism before socialism: anarchism within capitalism segments communities and focus and makes any movement less capable of accumulating the necessary power to make effective change to the system. Anarchism makes more sense in late-stage (likely automated) socialism/communism, as the effective needs for shared resources (such as infrastructure, utilities, defense, etc) are solved by both social norms, systemic factors, and, again, likely future automation, given the state of things.
Socialism/Communism (the difference is only that Communists believe in removing the idea of a nationstate completely after socialism has taken firm root globally) makes sense from a multi-actor value optimization standpoint, as well: it's the most direct way to satisfy diverse value priors apriori, as it minimizes many wasteful/harmful social costs, directs value to where it is need, is a stable and attractive basin when considering relative actor value prior divergence as people align towards shared community/society goals over time, and reduces unintentional/harmful inter-actor competition. Socialism also minimizes logistical factors/costs by accounting for systems as a whole in order to increase efficiency. Further, there are alternatives to capitalism that still have markets, and, there's a lot to be said postively about said market systems within socialism/communism. For example, the Chinese party treats its markets similar to how a technition might control a nuclear reactor: they look at various metrics like competition, community health, worker rights, efficiency, tech-factors, etc and add and remove control mechanisms in order to achieve greater harmony between the metrics and societal good, while keeping final control in the hands of the political system instead of the capitalists leading the market.
They achieve this clean separation of personal finances and finance based power accumulation from governmental officials and agencies by distinguishing enterprise (including State Owned Enterprises [SOE]) from SCIOCs (think financial instutitions like a governmental aligned blackrock) and banning direct participation in ownership and accumulation in the market from any government officials, while keeping their SCIOCs under the direction of highly-trained Communist leadership as the financial intermediary system, which keeps their money out of politics, while still being able to enact the will of their people through them via policy direction on the SCIOCs. This is all to say, a better future is possible, we just gotta make it happen.
> [socialism] is the most direct way to satisfy diverse value priors apriori, as it minimizes many wasteful/harmful social costs, directs value to where it is need, is a stable and attractive basin when considering relative actor value prior divergence as people align towards shared community/society goals over time
You used a lot of big words but capitalism has proven to be most effective at allocating resources with its invisible hand and it's not even close. See South Korea vs north Korea.
You used a lot of small words and were very wrong.
Capitalism very much improved upon feudalist systems. Capitalism is a very large step forward from that. Socialist systems, however, account for the vast majority of the uplift out of poverty in the last century (worldbank, of all sources https://openknowledge.worldbank.org/server/api/core/bitstrea... ). Also, north korea had an average economic growth higher than south korea for a huge chunk of the time since the korean war, and, for a long time had a higher gdp per capita. This is especially impressive as NK was bombed so badly that it was said that "Pyongyang was an ashen moonscape." Also, you're making the mistake of confusing a singular socialist experiment with socialism as a concept.
> Also, north korea had an average economic growth higher than south korea for a huge chunk of the time since the korean war, and, for a long time had a higher gdp per capita.
Big if true
Big!
https://onlinelibrary.wiley.com/doi/10.1111/aehr.12225
Uuuh, I think that source contradicts your claims and supports the consensus? NK has messed up big time in the past decades and SK is doing great. Communism didn't just fail in one single experiment, it failed in all of them, China, Soviet, Cuba, NK, etc
(for posterity): Read my claims again along with what the paper shows. I was specific and the paper supports them.
And, it def hasn't failed but I don't have time to educate you.
Watch this if you care to: https://youtu.be/BeRjTtKFlVM
AI is built on top of the work of infinite amount of people taken for free, whatever has been resolved in courts just shows how much you can make the system do for you when you have amazing lawyers and money to pay them.
https://geohot.github.io/blog/jekyll/update/2026/06/11/ai-wi...
Slowing down is not enough! Governments should immediately stop any further progress towards more capable artificial general intelligence.
All of the goverments? What makes you think they would all agree with that? Make a list of which ones you think are least likely to agree with you. And then imagine what happens if they outcompete everyone else on AI.
prohibition never works, this would just be market leading countries giving up their spot
An agreement is mainly needed between the US and China, as they are at the forefront of the race. It's challenging but it's in their best interest. Many other countries would likely see favorably such an agreement, also because they are far behind in the race (my guess is that this would be the case for the European Union, and possibly for several developing countries). Once the agreement is supported by at least the three most powerful economic blocs in the world, the few uncooperative countries left could be forced to comply through an unbearable economic pressure.
Jacob is such a fraud.
What the government should do is redistribute GPUs and RAM to individuals and away from power centralizing corporations. The danger is Dario and Altman gate keeping AI. The government should protect us by mandating local AI.
Frontier AI is not advanced, it's just reckless. All of it amounts to a CEO that is profiteering by cutting every corner they can get away with. Currently, it appears they can get away with anything.
All these companies are involved in the nvidia smuggling ring and market collusion. All the CEOs are criminals taking advantage of a system that can't even arrest a single CEO for making billions off an illegal smuggling ring.
I am convinced Hacker News comments are filled with bots, every comment being outrageously anti-technology and anti-AI.
In this debate, Hanson is so clearly at a level beyond Coxon that it's hard to argue with. And yes, I'm familiar with "the alignment problem", Superintelligence, countless exhausting LessWrong arguments, and the rest.