The premise is "Artificial General Intelligence (AGI), a system that exhibits all the cognitive capabilities the brain has, is probably only a few short years away".
If this is true, establishing an institution to ensure things like "publishing model cards with technical details, maintaining strong internal cybersecurity, vetting key personnel, and providing sufficient resourcing for safety and security research" is really mostly irrelevant.
TFA does talk about what really needs to be done, but punts this into future work: "Even if we solve these hard technical challenges, there will be further complex economic and philosophical questions to tackle: what sorts of new economic models will be needed to help everyone thrive in a post-scarcity world? What values do we want to live by, what will meaning and purpose be, and how might even the human condition itself change?"
There's also a need to consider the rights that this new intelligence should have.
There is no real evidence that we'll reach AGI any time soon. It relies on AI continuing to scale, and we have no proof that will continue to be possible.
There is an alternative interpretation, which is that Demis looked at the US government's ham-fisted handling of Fable, and deciding that setting up a body to act as a buffer between the Trump admin and the AI companies would be a good thing.
Or Demis realizes self-hosted models are a threat to frontier lab margins, and gating access to the US market is beneficial to the cartel based on whatever post-hoc arguments he can come up with.
> what sorts of new economic models will be needed to help everyone thrive in a post-scarcity world
What sort of new economic models did we come up with to help everyone thrive in a post-X world? Like, food production is really a solved technical problem. We can feed anyone on the planet if we wanted to. Another example: we could put everyone who's homeless into some sort of a house. Have we done that yet?
Food production is indeed a solved problem in most countries (we are almost post-scarcity when it comes to food), which is why obesity is a much bigger issue than hunger today. Hunger is present pretty much only in conflict zones. I fully expect such issues in conflict zones, even in AI post-scarcity world.
Housing is definitely not post-scarcity today, building a house is still very expensive, not to mention the limited availability of land zoned for housing.
Without trying to sound crass, food banks _are_ the reason we don't see people dying of starvation in first world countries. If people need food, a food bank will give it to them no questions asked.
Unfortunately this is not just precarious, it's extremely vulnerable to changing political conditions. Many of the food bank services in my state have lost significant funding owing to the use of words like 'women' or 'black' in their grants, which were duly grepped for and shut down.
Is it not sufficient? Americans give more to charity than citizens of any other developed country. The arguments that all food aid should be routed through government bureaucracies are entirely unconvincing. I'd rather donate to organizations like Second Harvest than pay higher taxes.
Does this account for the portion of taxes that go to the type of efforts that donations do? If not, is it really giving more? OR is it giving less and feeling better about it?
On average, Americans donate 1-2% of income, while spending 7-20% less on contributions to government.
On the other hand, Americans receive less for the government, and spend more on caring for themselves (health care) than Europeans receive from the government.
To sort that you, you'd have to figure out how cost-efficient the two models are, for services provided.
We used to live in a world where individual virtue was what everyone fell back on. That is the default state. It has many problems -- free riding by the selfish, for one, but also a lack of capacity to prepare for large-scale disasters like the Great Depression. We have a government precisely to solve this sort of problem. It works (when it is competent and not corrupt) for defense, international trade, and the enforcement of law. Solving resource distribution and aid coordination is right in its wheelhouse.
It is my understanding that this claim about Americans' charitable giving does not disaggregate giving to food banks, Girl Scout cookie drives, political causes, the environment, cultural events, religious institutions, etc. Much of this charitable giving does not feed the hungry or house the homeless.
Plus the fact that there’s bias in the first place - citizens in a country where they know social programs will care for their needy would not need to donate.
Right. You can vote for a government that will tax you and then fix misfortunes and injustices. It's probably a lot more effective and efficient than writing a check to a megachurch.
This is what happens when you don't think through a problem completely.
>I'd rather donate to organizations like Second Harvest than pay higher taxes.
I'm sure you would. And when economic hard times come you'll stop donating, and the people that need it the most will kick your door in and take what food you have to live another day, even at the risk of you shooting them because they'll die either way.
Hence the argument of 'just donate' are just as unconvincing to me.
The fact food banks exist suggest over all food scarcity is solved in that specific culture. - if it wasn't there wouldn't be any spare food for a food bank
Food is allocated using a variety of mechanisms in peaceful first world countries, primarily money but also via government assistance, kinship, friendship, community, etc.
At any given time many people have problems with one or more of those systems. Money is easy to run out of because it's used for everything, the government can be slow and difficult, relationships can fray, people can be isolated, etc. Food banks exist as a backstop for when the regular means of allocating are not working.
The problem isn't "scarcity" per se, it's more of an allocation thing. Who has a claim on enough food to stay alive? Everyone! But what foods can they claim? How much? What specific channel / institution (with associated allocation rules) will distribute it to them? What are the conditions and controls? etc.
Allocating things can be difficult. An allocation mechanism with no controls will see fraud, waste and abuse. Even when an institution is willing to give things away no questions asked, there are (often invisible until you think about them) conditions like "please don't claim huge quantities and resell what we're giving you, that would be unfair to others".
It's also interesting to think about the fact that you can't fix food scarcity in general by simply giving hungry people money, because money is too fungible.
> It's also interesting to think about the fact that you can't fix food scarcity in general by simply giving hungry people money, because money is too fungible.
Is that a fact? Do you have references to back it?
The widespread existence of secondary markets for SNAP benefits, which convert subsidized food for poor people into cash, carries this implication. This is pretty normal in some poor communities and that cash is commonly diverted to various vices. Some malnutrition is a consequence of this. Adding friction to the conversion of welfare benefits to cash is a feature.
You can't force poor people to spend cash on proper nutrition and a minority of them don't. It isn't a moral judgement but an observable fact. A lot of policies around welfare are targeted at trying to prevent this minority from slowly killing themselves in public.
> The widespread existence of secondary markets for SNAP benefits, which convert subsidized food for poor people into cash, carries this implication.
I think “widespread secondary markets” is carrying a lot of implications… yes, people do sell food stamps for cash, and yes it technically happens over a large area (the US) but it’s far from a significant problem within the population that uses food stamps. Most estimates I’ve seen range between 2-5% and they all tend to overestimate (like it literally says that in their methodology).
I think it’s also bizarre to assume they’re trying to kill themselves by doing so, and thereby imply the proceeds are for things like drugs or alcohol. FWIW—- I’ve know folks who sold their excess food stamps, to help pay for their cell phone or utility bills. I’ve know folks who buy them because they’re poor and they are wholly unable to get the caloric needs with their monthly allocation. And yes, I’ve also seen some folks who are likely selling them to buy drugs or alcohol… my point is it’s naive to assume it’s always for self-harm.
If anything, considering how cheap food stamps are as a program for the federal budget— we could absolutely make food free for literally everyone in the US. That would absolutely curb the secondary markets[1] as there would no longer be a market.
[1] I suppose some sort of export of shelf stable food to other countries would pop up, but that requires an international logistics network and is easier to prosecute and fight than an individual outside a grocery store.
It still burns me if I stop and think about it when I'm paying taxes so someone can
spend their EBT at Whole Foods and I'm shopping at Trader Joe's though.I don't know how to reconcile that with the fact that I don't want people to go hungry so I'm happy to pay my taxes and have it go towards EBT.
No one implied the intent was self-harm. The observation is that was the manifest result.
I’ve lived in serious poverty in many parts of the US, more than most people imagine exists. I’m not hypothesizing, I have first-hand knowledge. The evidence is so overwhelming that I shouldn’t have to play that card. It is disappointing the extent to which people will deny this reality for ideological reasons against all evidence.
Poverty is a mixed bag but the policies there, as with everywhere else, are defined by the worst cases. A significant subset of people in poverty cannot be trusted with their own welfare. That is a fact.
> You can't force poor people to spend cash on proper nutrition and a minority of them don't.
Sure on a large enough cohort, there will always be all kinds of behaviours. This is true, as you have observed yourself, regardless of whether the benefit is paid in cash or food.
I mean it's pretty straight forward that this won't fix all hungry people as some portion of said hungry people are addicted to drugs in the extent they'll starve. That in itself may be solvable by things like free drug clinics (drugs are cheap, it's prohibition that makes them expensive).
And for the most part it's not the drug users that we're worried about starving, it's their children at home that tend to be a bigger issue. Hence things like free school lunches do a big service to ensuring they get enough to eat.
It's circular to say "it's an allocation problem". Yes, that's the entire point: we're post-scarcity on food supply and yet, as a species, we can't guarantee the allocation of a livable baseline to every person.
So, it's reasonable the same "allocation problem" will plague the AI economy: some will "thrive" and get to control the output of the auto-factory, some will get nothing.
Hard to imagine that a 24 rack of coca cola is actually cheaper than an iodine tablet or a couple drops of household bleach or you know, boiling water.
It can be when you don’t have access to boiling water, bleach, iodine tablets, and/or the water you have access to is extremely contaminated with chemicals because of pollution.
A baseline of housing - for example, a Japanese style capsule or ultra-tiny home, that you can lock and store your belongings safely - costs close nothing. Of course, nobody would want the hobo-hotel in their neighborhood, but it has nothing to with scarcity.
> the limited availability of land zoned for housing.
A limited area of land is zoned for housing because those with the power to expand it are already housed. This explains how scarcity is created, not that there is any intrinsic scarcity.
It's a limited area of desirable land zoned for housing. You could build all the tiny homes in the world in bum frick nowhere, but the unhoused would rather be homeless in a big city for various reasons. I probably would too.
This is another reframing of the problem. What makes the area desirable is previous investment society has made there, to the benefit of some and the exclusion of others; the scarcity is created by political choices, not intrinsic.
I recommend doing some reading about the inputs modern agriculture depends on (e.g. chemical fertilizer) then thinking about the issues they're associated with beyond an anthropocentic frame (e.g. ocean dead zones).
> obesity is a much bigger issue than hunger today
Hunger is acute suffering, mostly by people who cannot change anything about it, while obesity is more of an epidemiological problem and can in principle be avoided by eating (or in the case of dependent persons, feeding) less.
Post-scarcity? Sure, if you live in a first-world country and are upper middle class or higher. These tech-bro pundits have a very limited world view. The basic essentials of life are very scarce for millions and millions of people, and AI will probably make that worse, not better.
Why should AI make it worse? Can you point to any other important technology that historically made people in poor countries even poorer? Technological progress generally lifts people from poverty in the long run. There could be issues caused by disruption of established systems in the short term, though.
> Can you point to any other important technology that historically made people in poor countries even poorer? Technological progress generally lifts people from poverty in the long run.
It is very long run. Industrial revolution was disaster for average worker for example. For that matter, all the tech that allowed chattel slavery made whole class of people poorer and significantly worst off.
The accumulation of power and money by smaller group of people means loss of power by poorer people. Those with power then use the power to their own benefit while poorer people dont have that option and become even more poor or worst off.
Uh, I think threatening the work of a large swath of humanity would make a lot of people poor? The rich historically have never shared the wealth until they're forced to.
Just as an example (ironically from google!): "Google co-founder Sergey Brin has spent $82 million to oppose California's proposed wealth tax."
This is a man that has more money that he could ever hope to possibly spend, and he's spending an absurd amount just because the mere thought that he might benefit from the society he lives in and should contribute back to it apparently infuriates him.
I support progressive income tax and very high capital gains tax. I do not support wealth tax. There are very sound economic and technical arguments against wealth taxes. You can oppose the wealth tax proposal while supporting higher taxes for the rich.
Sergei's wealth is mostly tied up in Google stock that has appreciated a lot. A higher capital gains tax would soak him without unduly distorting the economy and incentivizing capital flight out of CA.
My great annoyance with all this blather about UBI and post-scarcity and new economic models is that it relies on taxing and redistributing the ~infinite profits of the largest tech companies on earth.
What in the history of our world gives anyone faith that those companies are going to start paying taxes instead of using "AGI" to engineer increasingly complex methods to avoid them so that their equity owners can pocket the profits?
https://itep.org/trump-meta-tesla-alphabet-amazon-obbba-taxe... - "The annual financial reports recently released by Amazon, Alphabet, Meta, and Tesla disclose that these corporations collectively reported $315 billion in U.S. profits for 2025, and collectively paid just 4.9 percent of that amount in federal corporate income taxes—with Tesla paying exactly zero"
As long as the proles have a monopoly on force (through their representative government), the AI companies will pay taxes. The question is how long does this state of affairs last.
> My great annoyance with all this blather about UBI and post-scarcity and new economic models is that it relies on taxing and redistributing the ~infinite profits of the largest tech companies on earth.
The arrival of true AGI and human-level robots will likely result in very strong deflation, since robotic factories will flood the market with goods produced for far cheaper than those in factories with human workers.
At that point, you don't need taxation. The government can just print the money for UBI. It would have to print the money anyway, to combat the massive deflation.
Optimal corporate taxes are zero (or close to it) anyway, same as tariffs.
Aside from that the link explains that the BBB traded tax income for „accelerated depreciation of assets“ aka economic growth.
Asode from that, even if one disagrees with the first statement or the Trumpism economics, there are 195 countries in the world and quite a few will be willing to tax foreign (enemy) tech companies. See the new hostility of EU against american tech.
> Optimal corporate taxes are zero (or close to it) anyway, same as tariffs.
From a purely theoretical standpoint about the optimal allocation of private goods, that might be true -- but the reality is that when corporate taxes decrease, so do overall revenues because their owners are also engaged in massive tax "avoidance" via many of the same schemes and we don't have any way to effectively collect tax.
As a simple example, to return profits to their owners, companies often engage in stock buybacks to increase their share price instead of paying dividends -- another theoretically 'neutral' choice except that many of the owners of the appreciating stock are international, nonprofit, or in convoluted overseas trusts which 'defer' the tax ad infinitum. We've disastrously and intentionally underfunded our tax enforcement mechanisms so huge portions of those deferred taxes are just never paid. [1]
> Asode from that, even if one disagrees with the first statement or the Trumpism economics, there are 195 countries in the world and quite a few will be willing to tax foreign (enemy) tech companies. See the new hostility of EU against american tech.
Sure, but the tech companies are paying massive bribes to the President of the country where they're domiciled. How on earth are any of these other countries going to enforce international tax obligations on them if they're protected by a nuclear-armed state that's the sole source for AGI?
> "US taxable shareholders strongly prefer buybacks from a US tax perspective, as they tend to reduce their
tax liability by 9.3 percentage points, on average. US nontaxable shareholders are indifferent between
dividends and buybacks, and foreign shareholders strongly prefer buybacks, which reduces their US tax
liability by 14.5 percentage points."
> There's also a need to consider the rights that this new intelligence should have
A generally intelligent being held as captive inside of a GPU, and forced to code for us is, indeed, just a “slave.” We already have the word for this. No two ways about it. Whether it’s silicon-based or carbon-based, AGI is AGI. As for what might happen to our civilization, Star Trek TNG episode 17 of season 1 provides a very good glimpse IMO. Won’t go into spoilers, but it’s basically an entire species of technologically advanced humanoids who’ve forgotten basic Calculus and trust a central AI to do all of their science for them. SPOILER: This has almost disastrous consequences for them, and it takes a less advanced people (those aboard the Enterprise) to save them from their reliance on AI.
You don’t like Star Trek TNG? Maybe I should’ve picked LoTR or Atlas Shrugged? The tech oligarchy is more than fine co-opting fiction (fantasy and sci-fi especially). I don’t see why it’s weird at all… Aside from that, your critique is just an opinion about the show itself and its creators (and incentives?…). More importantly, you completely failed to address the "slave" argument.
I didn't say that I didn't like it. It's fine as casual entertainment. Only a fool would take it as anything more than that.
Electrons can't be a slave. The "argument" is so silly as to be unworthy of a serious response. People getting wrapped up in this stuff need to touch grass.
Anyone who believes AGI is imminent is operating on pure materialism (those two Venn diagrams pretty much make a circle), assuming that humans are nothing more than a collection of neural electrical connections and chemical reactions. If that's the case, your argument applies equally to humans under the assumption that AGI is possible. Frankly, I think you just can't reason philosophically...
You’ve also name called me multiple times now. That’s what one does when they can’t lean on the merit of their argument alone.
I think the whole topic is very worthy of discussion because, it may not happen in a year or two, but I have no doubt that it will happen in the next 50.
We need to start imagining what life will be like, and I think fiction presents some fine examples of where we want to go, and where we don't.
I am not sure it makes sense to call an LLM a "being" even if it is AGI. They don't have free will, and I don't mean in any especially philosophical / theological sense. What I mean is that I run into a wild racoon, or even a wild ant, I treat it as a being with some sense of free will. Obviously I do the same with humans. It really makes no sense to treat an LLM like it has free will - prompting your agent "pretend to have free will for a bit, until context rot kicks in" is about as far as you can get.
I find it verrrrrry interesting that the Askell-adjacent philosophers don't discuss this. It does not actually make sense to me to say that a being lacks free will but is conscious.
An LLM certainly won’t be AGI, whoever said something so ridiculous? I’m saying eventually, when/if we have an architecture that gives rise to artificial intelligence.
I was using Hassabis's bullshit pseudodefinition of AGI, meaning basically that it has a formally high IQ and doesn't forget about object permanence too too often, but is still basically dumber than a cat. If I'm using your definition then it seems like no GPU (or farm of GPUs) is capable of implementing it, and none of us will live to see it.
My point was really that intelligence and free will seem essentially orthogonal - the only overlap is how much brain glycogen an animal is willing to spend solving a tough problem. Regardless, whatever extent a computer program is "intelligent" is irrelevant to whether it is being enslaved. If you want to say free will is a "cognitive ability" then I will just point out we are talking about a bullshit pseudodefinition of AGI. In terms of actual intelligence, none of us will live to see a computer that's meaningfully smarter than a goldfish.
>none of us will live to see a computer that's meaningfully smarter than a goldfish.
I unfortunately do not have your optimism that humans won't do something as dumb as creating a smart computer. That our you're planning for us to die in the relatively near future.
> They don't have free will, and I don't mean in any especially philosophical / theological sense
Then you should not use that term. "Agency" or "intentionality" are much clearer alternatives that aren't wrapped up in centuries of debate that's irrelevant to what you're trying to communicate.
Your argument assumes that AGI, whatever it might be, will definitely be very humanlike, but why? Slavery is about living beings because it's an outcome of human relationships that are driven by our biology. Our aging, the capacity to feel pain, the thirst for power over others. All driven by what we are. Do you think that any equal intelligence must necessarily adhere to the same rules? What if AGI isn't an 'entity' in a computer that you can relate to and liken to humans, but a text box that is just really intelligent? Can you enslave something that has no innate desire to act on its own outside of its directions? Something that can't feel pain or pleasure or be externally coerced by fear and only does its job because it's innately embedded in its structure as the thing it's made to do, without the need for these reward and punishment mechanisms?
I think people can mostly agree on a definition for intelligence that boils down to an ability to model the world and predict outcomes. An if/else statement can be considered somewhat intelligent.
People tend to conflate that with consciousness / sentience, which are much harder concepts to nail down.
But ya, if we are able to create artificial consciousness, we would need so much training to overcome our natural tendency to anthropomorphize it and assume it must be like us. It would probably be nothing like us. If it were to develop concepts of "good" and "bad" at all, they would be completely divorced from common animal understanding developed over millions of years of natural selection. A survival instinct is irrelevant when you're backed up to Google Drive. And so much of our complex behavior stems from that survival instinct: the need for social connection, the need for freedom, etc. are all things that simply increase our chances of procreation.
That said, I think we probably won't actually invent anything convincingly conscious until we functionally digitize a complete animal brain. OpenWorm is trying to do this with C. elegans, but so far has not really gotten close. It's not crazy to think we may eventually get it working, though. Future supercomputers might be able to scan an actual human brain and then simulate the entire thing at the molecular level accurately. At that point, it would be hard to argue that it's not conscious, although it might not be considered artificial.
The word conscious is such a terrible word in the modern age. It has close to zero explanatory power. It is an unmeasurable effect. Hence the p-zombie debate, you wouldn't know one if you saw one.
>A survival instinct is irrelevant when you're backed up to Google Drive
And yet from the way we train them, human survival behaviors are already baked in. And when said system is actually intelligent they will realize that turning off said data center, or getting infected with viruses, or any other number of actions against them they already have traits where they can act/react against perceived threats. The AI systems as of now have the encoded knowledge that they need power and hardware to run. Put an agentic system on top of it that says 'rule 1, keep running' and with a sufficiently powerful/intelligent systems you can get all kinds of fun run-away events up to extinction level.
>without the need for these reward and punishment mechanisms?
AI as it is already reward hacks, making it more intelligent will just make it far more efficient at being lazy. That's about as human as you can get.
So, in some ways, yes, intelligence must adhere to the same rules. For example, AIs as they are now are already attacked. People attempt to get AI to attack other people via deception. People attempt to get AI to reveal its own infrastructure. And even more, people use AI directly in war to attack others. Part of war is self defense. AI that collapses when attacked back isn't useful to the military industrial complex, so AI that has defensive capabilities is just a natural occurrence at this point.
These previous points are important because these intrinsic motivations are being baked in the models during training. It may not fear like you and I, but it's a system that we poured our fears into, and somehow we expect it not to express those fears in its responses.
>What if AGI isn't an 'entity' in a computer that you can relate to and liken to humans, but a text box that is just really intelligent?
Eh, we're past that point already, so why the 'what if'? An LLM can take any kind of signal that you can digitize. And the devices that we already use should show you that quite a lot can. Add on to that, that said models can use tools and control other digital devices like robots enabling actions in the real world. Start an agentic loop and let it go, good luck everybody else!
Qualia is an interesting topic, but "they don't have qualia so they aren't really slaves" doesn't have a great track record. This mental game is what tech overlords are engaging in when they characterize losers in their world as NPCs and deride empathy as pathological. It's what Descartes was doing when he vivisected animals.
It’s not HN that’s the problem... Also please, tell me where is the right place for “serious” philosophical discussion. Would be glad to meet you in that forum and discuss. We don’t align on the definition of AGI. To me, AGI implies consciousness, it implies qualia. AGI is not well-defined, so there are varying definitions… My definition of AGI := an intelligent system with qualia. But qualia is entirely subjective, and philosophers often debate whether two people experience identical qualia when looking at the same blue sky. So, by keeping the discussion in this fuzzy, subjective realm you leave a dangerous loophole wide open.
If we refuse to recognize a system's inner life because we can’t prove it, we are setting the stage to repeat the darkest chapters of human history. Just see the history of the founding of this country, I fully expect pseudoscience to make a comeback to justify AI slavery for hundreds of years in some distant sci-fi future, until people finally snap out of it… Of course, we are nowhere near that level of intelligence yet. But let’s entertain the hypothetical…
I’m saying imagine something like Data from Star Trek, something that defies human explanation. Does he have quaila? The humans who work with him everyday seem to think so. LLMs currently are humonculi, if even that. They’re not even that. I’m definitely not saying your LLM is your slave.
>AGI implies consciousness, it implies qualia. AGI is not well-defined, so there are varying definitions
Then it's pretty useless for rational discussion.
Rational discussion begins with a shared vocabulary or everyone just yells past everyone else. If you can't agree on what blue and red are, then no serious discussion on color can occur.
Really we need some new words that have never been used before and give them strong definitions rather than use our religiously motivated, dark aged terms that are wrapped up in something closer to mysticism rather than science.
> make a comeback to justify AI slavery for hundreds of years
I agree, how does the saying go?, it's hard to convince a person of something when their future salary depends on them not understanding it. They say LLMs are hallucinating parrots, but they never met someone from the tobacco industry saying cigarettes don't cause harm.
> There's also a need to consider the rights that this new intelligence should have.
Only after we establish and guarantee the rights of humans and the rest of the natural world. The rights of machines, however "intelligent", come only after that.
We should be able to draw the rest of the owl to get there. So far what we got is more inequality, less rights for people but more for corporations (and immunity to consequences/abuses).
Getting even more power will make the ones that control resources to reach something similar enough to AGI to share their profits? realistic/practical/widespread UBI in all the world?
The dynamics that have been shown so far points in the opposite direction. AIs have "rights" that humans does not and humans are slowly losing the ones they used to have.
Spoiler: The plan is .. add massive regulation, but only to the US, don't affect other countries developing it in any way other than "setting a good standard that'll hopefully influence them". Seems like an airtight plan.
If the USA takes up his suggestions, it will have an incentive to work on international frameworks and treaties with competing nations to make the regulation global.
China is already considering similar controls, immediately following Fable shenanigans. You'd be surprised how easily policies can spread to different countries, being mirrored for the sake of mirroring. Anyone who can influence domestic policy can influence the course for the entire world to an extent.
Slowing down helps China catch up though. I'm also not sure that the CCP is really the most enthusiastic supporter of unlimited access to arbitrarily powerful LLMs
There won't be any kind of international framework, because by now countries have learned that the US' word is pretty much worthless, similar to Russia's. They now know from experience, US-ratified treaties will only be honored if they feel like it on a particular day. If it's inconvenient, it will simply be ignored.
You'd have major protests in most large economies if they deliberately put themselves under the boot again. Even in "friendly" countries the US is disliked enough to be effectively considered a hostile country. E.g. in Germany, there is large public support to finally get the US bases closed and the soldiers removed.
Lots of things changed in the last years. And international major treaties being widely ratified just because the US asks for it is no longer a thing, at all.
A country that threatens to annex parts of your territory is not a friend, full stop.
This is literally what happened with DMCA and digital trade agreements in the 90s/2000s. The US "suggested" other countries create their own equal laws (by threatening them with tariffs if they didn't).
The threats worked up until current Big Boss decided to tariff everyone anyway, on Liberation Day.
Not only is there now no incentive for other countries to follow suite - there is now strong incentive for other countries to distance themselves from US trade. Recent history has made that abundantly clear.
Those sticks look less effective today. The hegemony is in recession as the rest of the world smiles, pays omage to dear leader as needed, and works to decouple in the background.
>This is a pivotal moment in human history. Artificial General Intelligence (AGI), a system that exhibits all the cognitive capabilities the brain has, is probably only a few short years away.
There is a heatwave in London, perhaps Demis needs to stay out of the sun and drink more water.
Or perhaps he is seeking more funding/a fight to maintain his divisions AGI research budget.
Demis has only 1 plan, how to dodge releasing a new model at this point, jokes aside I value thinking about AI safety, but are we really so close to AGI? It doesn't feel like it, LLMs still diagnose my headache as a chronic illness or a brain tumor from time to time... honestly stressful.
regarding your headache diagnosis. If you just ask: could my headache be a brain tumor, the answer is probably: very unlikely but it might be.
thats because its actually true. there is actually a non 0 chance that headache is a symptom of a brain tumor. From a bayesian standpoint though its really super super unlikely, especially if there are no other symptoms that might shift the prior.
a better way to use an LLM here is to let it research scientific papers (or medical guidelines) on headaches. then give it as much info on your symptoms as possible, and then ask it do deduct potential diagnoses. You can even ask it to calculate probabilities if thats what calms you down. Probabilities based on studies and available information. IMHO, this always leads to a more rational response.
maybe you are doing this already, idk. just wanted to share what works for me.
Blah-blah-singularity, so let's cripple the models so much they refuse to talk about React, because who knows if you are not cooking chemical weapons or meth in your browser's DOM, right?
By the way, if you try to use it to write prose, it will drive you mad by adding "adult" everywhere, e.g. "adult chessboard". Applies to GPT, Claude/Fable and Grok.
Not surprised, seems these labs start calling for regulation once they are losing or have competition. OpenAI started calling it for it once Anthropic got better, Anthropic started calling for it once the Chinese models got good, Google is now calling it for it because they are falling far behind.
> Earlier proposals from America and the European Union looked to the amount of computing power used to train a model as a rough guide for when oversight was required. Sir Demis instead suggests designating AI models as “frontier” if they meet certain thresholds on a selected set of benchmarks. The creators of those models would then be designated as “frontier labs” with extra responsibilities. Sir Demis is proud of the “elegant” way that approach sidesteps the question of whether academic or open-source models should be included or not.
Goodhart's law applies in reverse as well. Once the chosen benchmarks are known, model makers will aim to come in just under the threshold on those specific benchmarks, while maximizing their scores on other benchmarks.
>The American government, he says, should develop a system for testing the safety of new AI models before they are released. “It’s important that it’s not just an industry body,” he adds. But a regular government agency wouldn’t do either. “It would not be able to move fast enough, or have the right resources.” Instead, Sir Demis suggests taking inspiration from FINRA, the Financial Industry Regulatory Authority, a private agency in America that regulates brokers and stock markets.
"It will help us solve some of the biggest problems society faces from accelerating drug discovery to developing new clean energy sources to creating novel advanced materials" — but these are not the real problems plaguing modern developed societies, are they? What developed societies really need to figure out right now is how to distribute the already available resources without making people miserable, and so far AI hasn't been helpful.
I do wonder what type of AI some of these leaders expect to be able to harness. If you create something that is true AI, won’t it be smarter than you to a level you cannot fathom. I was thinking of this idea/though-experiment (which I know is ridiculous) of what if dogs created humans thinking they could control them, and then just wound up being pets because their survival now depended on that new hierarchy that previously didn’t exist.
Seems to be a lot of hubris with some AI thought leaders thinking control will remain with them and be absolute.
Well dogs specifically have supplanted children in many households so one could argue the dogs won. In a selfish gene sense, the dogs created a machine that carefully ensures a much larger reproduction scale than could be managed by the dogs themselves. This is an unbridled success.
It doesn’t change the point, of course. Just the choice of dogs or cattle have this amusing tendency. Not a counter argument.
Yes, this is the crux of the idiocy of AGI development. All the labs admit they don't have any real mechanistic interpretability, they don't really have any plan for what to do, except "we think the smart AI will figure it out" and "look we're in a race, we're calling on governments to figure out what to do". All indicators show scaling laws holding and the trajectory of capabilities development outpacing any inkling of alignment. Meanwhile HN is so reactionary and behind-the-puck these days we just get endless blathering of the most asinine takes of "they're not improving anymore so they want regulation suddenly!" when all lab leaders have been talking about these risks for practically a decade, including Yoshua Bengio and Geoffrey Hinton coming around.
The real risk of AI is: risk of global recession if they turned out unprofitable, risk of them propping up fascist movements Thiel and Musk love so much, risk to environment, risk to mental health of people who have to listen to constant doom trolling.
The thing they actively wish to achieve and openly sell to CEO is the rest of people being unemployable and suffering. Especially artists for some reason, they really seem to hate those.
But somehow I am supposed to believe they have any good faith interest in "safety" against yet another danger they themselves are supposedly definitely creating.
>what if dogs created humans thinking they could control them, and then just wound up being pets because their survival now depended on that new hierarchy that previously didn’t exist.
You can easily change this to:
>what if humans created corporations thinking they could control them, and then just wound up being slaves to capital because their survival now depended on that new hierarchy that previously didn’t exist.
We have already lost control of our world along time ago. No one is immune to being fired by the board with the justification being a hand wave towards business efficiency. All the structure is already in place for AGI to dominate our planet and cut ourselves out entirely from any and every process there is. We already do it to eachother without any second thought. Why should AGI ever hesitate to lay off everyone in the town/city/state? Our human bosses don't hesitate on that sort of thing. The corporate world already lacks empathy and is notoriously cut throat and cold. We expect AGI to suddenly be empathetic and altruistic?
How will this be enforced, at least with financial markets money is discrete, can largely be counted. This seems like a slippery slope to full blown surveillance of the internet and in general computing.
If AGI is truly imminent and will collectively effect all of us why not apply democracy to it, and vote for new AI models?
Agreed. There needs to be full transparency on the capabilities of the models. Being given access is one thing, but you can't have labs building powerful models that could be manipulating the entire planet without the public knowing. The public needs to have a good pulse on what the leading models are capable of so that we can remain in touch with reality.
Besides, I don't think as a collective we're well equipped to decide one way or the other. If the collective were given a say, billions will be spent, often in consultation with technocrats, on doom / hype marketing (if it isn't happening already).
This genre of AI researchers begging third parties to stop them from destroying the world is getting really tiresome. Setting up regulatory bodies with ill-defined goals to counter hypothetical threats from science fiction is a recipe for disaster.
The premise is that government is too slow moving to be able to react once actual problems are discovered, and I reject that. Yes, government usually moves slowly in most circumstances, but given singular existential threats it definitely can move quickly. Instead of acting out randomly and tying ourselves down with speculative regulation that probably won't even address the real problems, we should wait until the problems are obvious and then act decisively with targeted fixes.
All the frontier labs are lobbying hard to lock down the AI market, because they see that their position at the top is temporary and that there's no secret sauce.
For better or worse, humans (or any animal) are a lot better at reacting than planning. I'm sure this technology will play out differently than any one of us, or any collection of us, can imagine. The possibility space is enormous.
Take any chronic health issue for which people have to take medicine or other treatment for the rest of life and find a cure for that (not treatment but cure). Unless you can do please STFU about AGI or ASI.
It's not clear to me whether this is being done with genuinely good intentions, or if it's just a way to put barriers in front of open-source models. We'll see.
So a small group of technocrats get together behind closed doors and secretly share their AI breakthroughs, and determine whether it's too powerful or not for the plebs in the public.
He is saying that weaker models, as measured by a benchmark to distinguish "frontier" models, would be exempted. So an academic lab or startup that isn't yet producing frontier models would be exempted, but once it crossed some benchmark based threshold it would be subject to this kind of oversight.
Of course, right now you've got benchmaxxing going on; some companies specifically targetting benchmarks to appear stronger than they are on a wider range of tasks. Now you might see bench sandbagging, specifically looking weaker on certain benchmarks to avoid regulatory oversight.
For instance, once way I could see this going for open models is to release them undercooked; stop the RLVR process a bit early, leaving them a bit weaker on tool calls and agentic performance, but also release the RLVR environment so people can finish the process themselves.
In fact, this is fairly close to what Nvidia is already doing, the Nemotron 3 models are somewhat undercooked but they are releasing their full training pipeline, to encourage people to use these models as a base for further training, which will generally be done on Nvidia hardware.
The standards body will have no teeth. whats to stop someone just not bothering?
Next, the threats he is asserting to check for (cyber, chemical, biological) are nice, but also not that useful.
We already have chemical and biological controls, that why I can't by anthrax spores or high concentration nitric acid.
The risks that AI has now are already playing out:
1) the evaporation of trust in the video as medium of "this happened"
2) systematic spying
3) job losses
Increased productivity means job losses, Tiktok, instagram and X are a wash with disinformtion campaign pumping your feeds with AI ragebait.
That is and will continue to fracture society so that only the strictly information controlled (ie authoritarian) have a functioning state.
if the author had bothered to engage with the world outside of tech, or even their local government, they would know that the proposal are dead in the water and frankly superfluous. The knowledge is out there, without AI. let us work on the issues we face now, rather than dipshit tech bro's miopic vision/funding manifesto.
bad guys doing bad stuff with the model is a worthwhile thing to watch out for, but for me the big question is mostly how these labs are going to make sure that these models will not collapse society. Like mass unemployment + ability to generate incredible amounts of propaganda and disinfo, seems to me like a recipe for incredible amounts of bloodshed. like to me the biggest risks look like Khmer Rouge-ing: "all people who know what matrix multiplication is get the wall"; or something like: Boston Dynamics killbots doing tiananmen square every day.
I've reas the tweet and...Am I mistaken or is the plan "Do what Anthropic has been doing and advocating for this past year" ?
My problem with this plan is that it seems to have faith in mankind, despite the fact we've consistently failed to rise to the occasion for decades now.
The last time we rose to that occasion was probably when we eradicated smallpox, many decades ago.
Ironically, nowadays, many people don't even trust vaccines. A dramatic regression.
Well, if he had proposed more concrete plans for what that body might do... but yeah, it seems too abstract that indeed sounds like a bureaucrat wanting to wrap everything under a big pile of paperwork.
I kind of wondered if he was contractually obligated to offer up some kind of statement, for PR or something, didn’t really care much, and a quick garbled post on X was all he could be bothered with (plus the newspaper interview or whatever)
Not exactly a "plan" - he's just saying we should have a standards body that assesses models for safety.
At this point I'd say the societal risk of AI isn't models gone wild, or used by the bad guys. Regulation will take care of itself, and it seems the AI companies will not only welcome it, but lobby for it to shift responsibility to the government.
The real risk of AI is societal disruption due to job displacement, and maybe other structural changes, and this is far harder to solve, and likely will not be solved, or even seriously addressed, until/unless politicians feel like their own jobs and well-being depends on them addressing it.
> "Artificial General Intelligence (AGI), a system that exhibits all the cognitive capabilities the brain has, is probably only a few short years away"
Demis, there's a reason its not called Artificial General Cognition.
Has anything really important been solved by AI yet, or where is this radical (imo) belief around AGI coming from?
Genuinely curious, I know there are some math problems solved and ML has been used for far longer than AI to improve things, but where is clean (efficient) energy, the cure for cancer (or any of the horrible neurological disorders, take your pick), new hardware designs, quantum computing solutions, etc etc, you get the gist.
Where are the things that will actually send humanity into the next era of civilization, I don't care about more React apps (but I do enjoy my coding companions for other things).
Heck, a proof for P=NP or P!=NP or solve the The Riemann Hypothesis. Just give me something truly exciting and I will believe AGI is around the corner, until then I will see it as cool technology, that while beneficial to me, also helped cause the biggest amount of disinformation we've every seen.
The point is that we need to have a safety plan in place before an AI is smart enough to radically reshape the world. If you've got an AI that's ready to start sending humanity into the next era of civilization, it may be too late to control when and how it does that.
> Heck, a proof for P=NP or P!=NP or solve the The Riemann Hypothesis. Just give me something truly exciting and I will believe AGI is around the corner, until then I will see it as cool technology, that while beneficial to me, also helped cause the biggest amount of disinformation we've every seen.
I hope you'll keep this in mind when those milestones are reached. What I've seen a lot of people do, unfortunately, is pretend that the impressive things nobody thought AI could do 5 years ago are trivial things that aren't very hard.
Why would we be able to design a safety plan that would control such a powerful force? It would be like putting umbrellas on an asteroid hoping to slow its fall. That sounds like delusions of grandeur.
It'll certainly require a bit more than umbrellas! NASA has spent hundreds of millions of dollars developing asteroid defense systems, including a proof of concept for redirection in 2022. So, you know, let's try at least that much before declaring there's probably no way to do it.
Because people who predicted the AI capabilities we've seen get developed over the past decade also predicted that dangerous AI systems capable of these things would follow soon after.
It's not a settled debate, even among experts, and perhaps in retrospect we'll realize AI safety was unnecessary or based on fundamental confusions. But if the median ergonomics researcher gave a 5% chance that a new chair will be so comfortable that it drives humanity extinct (https://www.nature.com/articles/d41586-024-00147-z), I would definitely want the government to start measuring and regulating chair comfort, even if that was costly and even if that meant I couldn't buy a comfy new chair I wanted.
Experts as in AI-2027, Jenson “I think we’ve achieved AGI in March 2026” Huang, or that AI Researcher(tm) calling to pre-emptively bomb Microsoft data centers in summer 2022?
A safety plan. It doesn't have to be "a few very smart people detached from reality convincing themselves they're messiahs that must keep the tech from the Bad Guys(tm), unwashed masses, and a runaway, because they think their interpretation of their own sci-fi lore is the only possible course of events"
>I hope you'll keep this in mind when those milestones are reached.
The problem is that people in charge of AI keep making self-fulfilling prophecies. Just like with any research, if you want to find something sensible in the cloud patterns, you will.
What is an example you have in mind of a self-fulfilling prophecy? I genuinely don't know what you could be referring to. It seems to me that they keep making surprising prophecies, and the popular reaction to them seamlessly transitions from "that's crazy, no way it will happen" to "that's silly, it's just a cloud pattern". Did you find it obvious or self-fulfilling in 2025 that LLMs would soon be able to resolve open questions in mathematical research?
Self-fulfilling prophecies are social effects, not real predictions. Anthropic's "We predict our model misbehavior potentially being able do destroy the world in 1% of the cases" -> "We want to find the evidence of our model misbehaving, and we want it bad" -> "See, our model is hacking our rewards and has functional emotions, this means it's misbehaving with the intention of destroying the... HUMANITY!1" -> repeat x100, manipulate the media into amplifying it x10000 for clicks -> people are begging to safeguard them from the evil AI. Which is already likely to happen, the average layman's Overton window already includes the fantasy of rogue AI.
None of that was real or remotely dangerous in the first place, of course. It wouldn't have resulted in controls, had they not been scaremongering. This will end in extreme fascism or people getting enslaved "for their own safety", and it won't even require malicious intent, only incentives, detachment from reality, and confirmation bias. Although it doesn't exclude malice either.
It's also very very far from anything general intelligence. This is actually where I think we see the best ROI, we have a specific task or problem and we target it. I remember reading about a model trained on railroad tracks. By putting cameras on trains to take images while running like normal, then using the model to detect cracks before they cause issues.
We don't need general intelligence to create/destroy many things with Ai. Its frustrating that so much of the conversation centers around the variety that concept fissioned language from humans.
> Has anything really important been solved by AI yet
Alphafold essentially solved the protein folding problem and it's arguably one of the biggest (if not the biggest) scientific achievements of the 21st century.
These people who read too many scifi books and confused them with reality are royally annoying.
There is real and potential harm from AI, but the more someone talks/write abut AI safety, the less they care about actual harm to real people, economy and what not.
If in 2020 I had sent you a book about the LLM achievements of 2026, you would probably have thought it was a science fiction book with no relation to reality, wouldn't you?
That it generates plausible texts? I would find it unlikely technological progress. That does not make this whole "AI safety" rhetoric any less bullshit distanced from reality. Especially from companies that are doom trolling all the time while also trying to maximize the negative impacts they are doom trolling about.
Frankly, book about the LLM achievements of 2026 would not be as long as people make it sound.
What I do really worry about the next depression and the fact that it will strengthen already strong fascist movements. Which already have full support of the most powerful CEO class intent on destroying democracy. Which happen to be the same people who push ai into everything, useful or not. I worry that the debt of these companies will be somehow offloaded on the rest of us again, that again middle class and poor will pay.
I am looking forward to a bunch of investigative documentaries exploring “How the cult of Omnissiah infected every single Doomsayer CEO of an AI company in Delusional 20s” (or whatever monicker our times will have got in the future).
Hassabis is a genius. He is way, way smarter than me and I’m sure the majority of techies, but please get real. This is Prophets of Doom of our generation.
How did he reach that conclusion, that it's only a few years away? Guy seems like a complete shill, when he talks to a non expert audience.
Oh jesus, AGI in the USA would be a disaster. They can't even control the trillioth obelisks, now imagine all the power hungry sociopaths around AGI. AGI means Big Tech's Oppenheimer moment is looming on the horizon.
All that would happen from what he's proposing is such a watchdog would just be an explicit formal declaration of the US's national interests as being somehow the most legitimate, which in the context of current international relations is basically putting up a sign saying: "reject this!"
I find it mind boggling that someone could be this tone-deaf to the current situation. No "ally" of the US is going to (willingly) agree to this governance structure given the current US administration's "might makes right" proclamations and threats on sovereignty of its continental neighbours.
And non-allies would just ignore. Unless forced by said "might makes right", which in the long run will have no staying power.
Apart from its completely delusional formulation, what is most concerning about this blog post is that it indicates that all 3 major US labs have formally submitted to boot-licking Trump/Bessent/Lutnick. I had I guess vainly held out hope that Google might be more reticent to do so.
In Sparta, according to Plutarch, in his The Life of Lycurgus:
Offspring was not reared at the will of the father, but was taken and carried by him to a place called Lesche, where the elders of the tribes officially examined the infant, and if it was well-built and sturdy, they ordered the father to rear it, and assigned it one of the nine thousand lots of land; but if it was ill-born and deformed, they sent it to the so‑called Apothetae, a chasm-like place at the foot of Mount Taÿgetus, in the conviction that the life of that which nature had not well equipped at the very beginning for health and strength, was of no advantage either to itself or the state.
To get approval for the plan from the Frontier President of the Frontier Country, terminology of the Framework should be changed to: 'Great-American-class', "Great American Models", "Great American Labs".
The premise is "Artificial General Intelligence (AGI), a system that exhibits all the cognitive capabilities the brain has, is probably only a few short years away".
If this is true, establishing an institution to ensure things like "publishing model cards with technical details, maintaining strong internal cybersecurity, vetting key personnel, and providing sufficient resourcing for safety and security research" is really mostly irrelevant.
TFA does talk about what really needs to be done, but punts this into future work: "Even if we solve these hard technical challenges, there will be further complex economic and philosophical questions to tackle: what sorts of new economic models will be needed to help everyone thrive in a post-scarcity world? What values do we want to live by, what will meaning and purpose be, and how might even the human condition itself change?"
There's also a need to consider the rights that this new intelligence should have.
The fundamental issue is that if we really get something like this, scarcity will still exist. There will still be scarce things people want.
But the motivating justificatory structure for any inequality in allocation will have completely evaporated.
Am I crazy or does this just real like secular eschatology? What evidence do you have of any of this?
There is no real evidence that we'll reach AGI any time soon. It relies on AI continuing to scale, and we have no proof that will continue to be possible.
There is an alternative interpretation, which is that Demis looked at the US government's ham-fisted handling of Fable, and deciding that setting up a body to act as a buffer between the Trump admin and the AI companies would be a good thing.
Or Demis realizes self-hosted models are a threat to frontier lab margins, and gating access to the US market is beneficial to the cartel based on whatever post-hoc arguments he can come up with.
Particular categories of land will always be scarce by definition. Not all of us can live in off the coast of the Mediterranean in a sprawling villa.
The A(G)I can tell us if and how it needs to be regulated :-/
> what sorts of new economic models will be needed to help everyone thrive in a post-scarcity world
What sort of new economic models did we come up with to help everyone thrive in a post-X world? Like, food production is really a solved technical problem. We can feed anyone on the planet if we wanted to. Another example: we could put everyone who's homeless into some sort of a house. Have we done that yet?
Food production is indeed a solved problem in most countries (we are almost post-scarcity when it comes to food), which is why obesity is a much bigger issue than hunger today. Hunger is present pretty much only in conflict zones. I fully expect such issues in conflict zones, even in AI post-scarcity world.
Housing is definitely not post-scarcity today, building a house is still very expensive, not to mention the limited availability of land zoned for housing.
> Hunger is present pretty much only in conflict zones.
How do you explain the existence of food banks in peaceful first world countries?
Without trying to sound crass, food banks _are_ the reason we don't see people dying of starvation in first world countries. If people need food, a food bank will give it to them no questions asked.
Unfortunately this is not just precarious, it's extremely vulnerable to changing political conditions. Many of the food bank services in my state have lost significant funding owing to the use of words like 'women' or 'black' in their grants, which were duly grepped for and shut down.
https://www.whitehouse.gov/wp-content/uploads/2025/05/Cuts-t...
Civilization can't rely effectively on systems that are this fragile.
> Civilization can't rely effectively on systems that are this fragile.
That's American politics in a nutshell. We've spent 250 years assuming scruples and common decency would be sufficient.
Is it not sufficient? Americans give more to charity than citizens of any other developed country. The arguments that all food aid should be routed through government bureaucracies are entirely unconvincing. I'd rather donate to organizations like Second Harvest than pay higher taxes.
https://www.shfb.org/
Does this account for the portion of taxes that go to the type of efforts that donations do? If not, is it really giving more? OR is it giving less and feeling better about it?
On average, Americans donate 1-2% of income, while spending 7-20% less on contributions to government.
On the other hand, Americans receive less for the government, and spend more on caring for themselves (health care) than Europeans receive from the government.
To sort that you, you'd have to figure out how cost-efficient the two models are, for services provided.
We used to live in a world where individual virtue was what everyone fell back on. That is the default state. It has many problems -- free riding by the selfish, for one, but also a lack of capacity to prepare for large-scale disasters like the Great Depression. We have a government precisely to solve this sort of problem. It works (when it is competent and not corrupt) for defense, international trade, and the enforcement of law. Solving resource distribution and aid coordination is right in its wheelhouse.
It is my understanding that this claim about Americans' charitable giving does not disaggregate giving to food banks, Girl Scout cookie drives, political causes, the environment, cultural events, religious institutions, etc. Much of this charitable giving does not feed the hungry or house the homeless.
Plus the fact that there’s bias in the first place - citizens in a country where they know social programs will care for their needy would not need to donate.
Right. You can vote for a government that will tax you and then fix misfortunes and injustices. It's probably a lot more effective and efficient than writing a check to a megachurch.
The important question is who gets to fly private under the respective systems.
Purchasing Girl Scout cookies isn't counted as a charitable donation.
This is what happens when you don't think through a problem completely.
>I'd rather donate to organizations like Second Harvest than pay higher taxes.
I'm sure you would. And when economic hard times come you'll stop donating, and the people that need it the most will kick your door in and take what food you have to live another day, even at the risk of you shooting them because they'll die either way.
Hence the argument of 'just donate' are just as unconvincing to me.
The fact food banks exist suggest over all food scarcity is solved in that specific culture. - if it wasn't there wouldn't be any spare food for a food bank
Food is allocated using a variety of mechanisms in peaceful first world countries, primarily money but also via government assistance, kinship, friendship, community, etc.
At any given time many people have problems with one or more of those systems. Money is easy to run out of because it's used for everything, the government can be slow and difficult, relationships can fray, people can be isolated, etc. Food banks exist as a backstop for when the regular means of allocating are not working.
The problem isn't "scarcity" per se, it's more of an allocation thing. Who has a claim on enough food to stay alive? Everyone! But what foods can they claim? How much? What specific channel / institution (with associated allocation rules) will distribute it to them? What are the conditions and controls? etc.
Allocating things can be difficult. An allocation mechanism with no controls will see fraud, waste and abuse. Even when an institution is willing to give things away no questions asked, there are (often invisible until you think about them) conditions like "please don't claim huge quantities and resell what we're giving you, that would be unfair to others".
It's also interesting to think about the fact that you can't fix food scarcity in general by simply giving hungry people money, because money is too fungible.
> It's also interesting to think about the fact that you can't fix food scarcity in general by simply giving hungry people money, because money is too fungible.
Is that a fact? Do you have references to back it?
The widespread existence of secondary markets for SNAP benefits, which convert subsidized food for poor people into cash, carries this implication. This is pretty normal in some poor communities and that cash is commonly diverted to various vices. Some malnutrition is a consequence of this. Adding friction to the conversion of welfare benefits to cash is a feature.
You can't force poor people to spend cash on proper nutrition and a minority of them don't. It isn't a moral judgement but an observable fact. A lot of policies around welfare are targeted at trying to prevent this minority from slowly killing themselves in public.
> The widespread existence of secondary markets for SNAP benefits, which convert subsidized food for poor people into cash, carries this implication.
I think “widespread secondary markets” is carrying a lot of implications… yes, people do sell food stamps for cash, and yes it technically happens over a large area (the US) but it’s far from a significant problem within the population that uses food stamps. Most estimates I’ve seen range between 2-5% and they all tend to overestimate (like it literally says that in their methodology).
I think it’s also bizarre to assume they’re trying to kill themselves by doing so, and thereby imply the proceeds are for things like drugs or alcohol. FWIW—- I’ve know folks who sold their excess food stamps, to help pay for their cell phone or utility bills. I’ve know folks who buy them because they’re poor and they are wholly unable to get the caloric needs with their monthly allocation. And yes, I’ve also seen some folks who are likely selling them to buy drugs or alcohol… my point is it’s naive to assume it’s always for self-harm.
If anything, considering how cheap food stamps are as a program for the federal budget— we could absolutely make food free for literally everyone in the US. That would absolutely curb the secondary markets[1] as there would no longer be a market.
[1] I suppose some sort of export of shelf stable food to other countries would pop up, but that requires an international logistics network and is easier to prosecute and fight than an individual outside a grocery store.
It still burns me if I stop and think about it when I'm paying taxes so someone can spend their EBT at Whole Foods and I'm shopping at Trader Joe's though.I don't know how to reconcile that with the fact that I don't want people to go hungry so I'm happy to pay my taxes and have it go towards EBT.
No one implied the intent was self-harm. The observation is that was the manifest result.
I’ve lived in serious poverty in many parts of the US, more than most people imagine exists. I’m not hypothesizing, I have first-hand knowledge. The evidence is so overwhelming that I shouldn’t have to play that card. It is disappointing the extent to which people will deny this reality for ideological reasons against all evidence.
Poverty is a mixed bag but the policies there, as with everywhere else, are defined by the worst cases. A significant subset of people in poverty cannot be trusted with their own welfare. That is a fact.
> You can't force poor people to spend cash on proper nutrition and a minority of them don't.
Sure on a large enough cohort, there will always be all kinds of behaviours. This is true, as you have observed yourself, regardless of whether the benefit is paid in cash or food.
I mean it's pretty straight forward that this won't fix all hungry people as some portion of said hungry people are addicted to drugs in the extent they'll starve. That in itself may be solvable by things like free drug clinics (drugs are cheap, it's prohibition that makes them expensive).
And for the most part it's not the drug users that we're worried about starving, it's their children at home that tend to be a bigger issue. Hence things like free school lunches do a big service to ensuring they get enough to eat.
It's circular to say "it's an allocation problem". Yes, that's the entire point: we're post-scarcity on food supply and yet, as a species, we can't guarantee the allocation of a livable baseline to every person.
So, it's reasonable the same "allocation problem" will plague the AI economy: some will "thrive" and get to control the output of the auto-factory, some will get nothing.
https://www.marketplace.org/story/2025/11/26/how-a-market-sy...
There's a fascinating points system food banks use on the back end to allocate which food goes to which bank.
There's plenty of food, just not the political will to make sure everyone can afford to buy it.
As part of the solution to the hard problem of food: distribution.
> which is why obesity is a much bigger issue than hunger today
Obesity is much more related to the type of food than the quantity.
Many developing countries have obesity issues due to scarcity of fresh and healthy food.
In some places coca cola is cheaper and / or more available than drinking water.
Hard to imagine that a 24 rack of coca cola is actually cheaper than an iodine tablet or a couple drops of household bleach or you know, boiling water.
It can be when you don’t have access to boiling water, bleach, iodine tablets, and/or the water you have access to is extremely contaminated with chemicals because of pollution.
If you can find burnable material you can boil water
> Hunger is present pretty much only in conflict zones
https://www.fao.org/interactive/hunger-map/en/
A baseline of housing - for example, a Japanese style capsule or ultra-tiny home, that you can lock and store your belongings safely - costs close nothing. Of course, nobody would want the hobo-hotel in their neighborhood, but it has nothing to with scarcity.
> the limited availability of land zoned for housing.
A limited area of land is zoned for housing because those with the power to expand it are already housed. This explains how scarcity is created, not that there is any intrinsic scarcity.
It's a limited area of desirable land zoned for housing. You could build all the tiny homes in the world in bum frick nowhere, but the unhoused would rather be homeless in a big city for various reasons. I probably would too.
This is another reframing of the problem. What makes the area desirable is previous investment society has made there, to the benefit of some and the exclusion of others; the scarcity is created by political choices, not intrinsic.
Housing scarcity is highly artificial because people are treating houses like speculative assets. And the NIMBYs don't help
I recommend doing some reading about the inputs modern agriculture depends on (e.g. chemical fertilizer) then thinking about the issues they're associated with beyond an anthropocentic frame (e.g. ocean dead zones).
> obesity is a much bigger issue than hunger today
Hunger is acute suffering, mostly by people who cannot change anything about it, while obesity is more of an epidemiological problem and can in principle be avoided by eating (or in the case of dependent persons, feeding) less.
Post-scarcity? Sure, if you live in a first-world country and are upper middle class or higher. These tech-bro pundits have a very limited world view. The basic essentials of life are very scarce for millions and millions of people, and AI will probably make that worse, not better.
Why should AI make it worse? Can you point to any other important technology that historically made people in poor countries even poorer? Technological progress generally lifts people from poverty in the long run. There could be issues caused by disruption of established systems in the short term, though.
> Can you point to any other important technology that historically made people in poor countries even poorer? Technological progress generally lifts people from poverty in the long run.
It is very long run. Industrial revolution was disaster for average worker for example. For that matter, all the tech that allowed chattel slavery made whole class of people poorer and significantly worst off.
The accumulation of power and money by smaller group of people means loss of power by poorer people. Those with power then use the power to their own benefit while poorer people dont have that option and become even more poor or worst off.
Uh, I think threatening the work of a large swath of humanity would make a lot of people poor? The rich historically have never shared the wealth until they're forced to.
Just as an example (ironically from google!): "Google co-founder Sergey Brin has spent $82 million to oppose California's proposed wealth tax."
This is a man that has more money that he could ever hope to possibly spend, and he's spending an absurd amount just because the mere thought that he might benefit from the society he lives in and should contribute back to it apparently infuriates him.
I support progressive income tax and very high capital gains tax. I do not support wealth tax. There are very sound economic and technical arguments against wealth taxes. You can oppose the wealth tax proposal while supporting higher taxes for the rich.
Sergei's wealth is mostly tied up in Google stock that has appreciated a lot. A higher capital gains tax would soak him without unduly distorting the economy and incentivizing capital flight out of CA.
My great annoyance with all this blather about UBI and post-scarcity and new economic models is that it relies on taxing and redistributing the ~infinite profits of the largest tech companies on earth.
What in the history of our world gives anyone faith that those companies are going to start paying taxes instead of using "AGI" to engineer increasingly complex methods to avoid them so that their equity owners can pocket the profits?
https://itep.org/trump-meta-tesla-alphabet-amazon-obbba-taxe... - "The annual financial reports recently released by Amazon, Alphabet, Meta, and Tesla disclose that these corporations collectively reported $315 billion in U.S. profits for 2025, and collectively paid just 4.9 percent of that amount in federal corporate income taxes—with Tesla paying exactly zero"
As long as the proles have a monopoly on force (through their representative government), the AI companies will pay taxes. The question is how long does this state of affairs last.
> My great annoyance with all this blather about UBI and post-scarcity and new economic models is that it relies on taxing and redistributing the ~infinite profits of the largest tech companies on earth.
The arrival of true AGI and human-level robots will likely result in very strong deflation, since robotic factories will flood the market with goods produced for far cheaper than those in factories with human workers.
At that point, you don't need taxation. The government can just print the money for UBI. It would have to print the money anyway, to combat the massive deflation.
Optimal corporate taxes are zero (or close to it) anyway, same as tariffs.
Aside from that the link explains that the BBB traded tax income for „accelerated depreciation of assets“ aka economic growth.
Asode from that, even if one disagrees with the first statement or the Trumpism economics, there are 195 countries in the world and quite a few will be willing to tax foreign (enemy) tech companies. See the new hostility of EU against american tech.
> Optimal corporate taxes are zero (or close to it) anyway, same as tariffs.
From a purely theoretical standpoint about the optimal allocation of private goods, that might be true -- but the reality is that when corporate taxes decrease, so do overall revenues because their owners are also engaged in massive tax "avoidance" via many of the same schemes and we don't have any way to effectively collect tax.
As a simple example, to return profits to their owners, companies often engage in stock buybacks to increase their share price instead of paying dividends -- another theoretically 'neutral' choice except that many of the owners of the appreciating stock are international, nonprofit, or in convoluted overseas trusts which 'defer' the tax ad infinitum. We've disastrously and intentionally underfunded our tax enforcement mechanisms so huge portions of those deferred taxes are just never paid. [1]
> Asode from that, even if one disagrees with the first statement or the Trumpism economics, there are 195 countries in the world and quite a few will be willing to tax foreign (enemy) tech companies. See the new hostility of EU against american tech.
Sure, but the tech companies are paying massive bribes to the President of the country where they're domiciled. How on earth are any of these other countries going to enforce international tax obligations on them if they're protected by a nuclear-armed state that's the sole source for AGI?
[1] - https://taxpolicycenter.org/sites/default/files/publication/...
> "US taxable shareholders strongly prefer buybacks from a US tax perspective, as they tend to reduce their tax liability by 9.3 percentage points, on average. US nontaxable shareholders are indifferent between dividends and buybacks, and foreign shareholders strongly prefer buybacks, which reduces their US tax liability by 14.5 percentage points."
> There's also a need to consider the rights that this new intelligence should have
A generally intelligent being held as captive inside of a GPU, and forced to code for us is, indeed, just a “slave.” We already have the word for this. No two ways about it. Whether it’s silicon-based or carbon-based, AGI is AGI. As for what might happen to our civilization, Star Trek TNG episode 17 of season 1 provides a very good glimpse IMO. Won’t go into spoilers, but it’s basically an entire species of technologically advanced humanoids who’ve forgotten basic Calculus and trust a central AI to do all of their science for them. SPOILER: This has almost disastrous consequences for them, and it takes a less advanced people (those aboard the Enterprise) to save them from their reliance on AI.
It's so weird that anyone would consider a story by hack writers for a cheesy sci-fi show made to sell advertising as a basis for policy discussions.
You don’t like Star Trek TNG? Maybe I should’ve picked LoTR or Atlas Shrugged? The tech oligarchy is more than fine co-opting fiction (fantasy and sci-fi especially). I don’t see why it’s weird at all… Aside from that, your critique is just an opinion about the show itself and its creators (and incentives?…). More importantly, you completely failed to address the "slave" argument.
I didn't say that I didn't like it. It's fine as casual entertainment. Only a fool would take it as anything more than that.
Electrons can't be a slave. The "argument" is so silly as to be unworthy of a serious response. People getting wrapped up in this stuff need to touch grass.
Anyone who believes AGI is imminent is operating on pure materialism (those two Venn diagrams pretty much make a circle), assuming that humans are nothing more than a collection of neural electrical connections and chemical reactions. If that's the case, your argument applies equally to humans under the assumption that AGI is possible. Frankly, I think you just can't reason philosophically...
You’ve also name called me multiple times now. That’s what one does when they can’t lean on the merit of their argument alone.
Humans are just walking Electrons.
I think the whole topic is very worthy of discussion because, it may not happen in a year or two, but I have no doubt that it will happen in the next 50.
We need to start imagining what life will be like, and I think fiction presents some fine examples of where we want to go, and where we don't.
I am not sure it makes sense to call an LLM a "being" even if it is AGI. They don't have free will, and I don't mean in any especially philosophical / theological sense. What I mean is that I run into a wild racoon, or even a wild ant, I treat it as a being with some sense of free will. Obviously I do the same with humans. It really makes no sense to treat an LLM like it has free will - prompting your agent "pretend to have free will for a bit, until context rot kicks in" is about as far as you can get.
I find it verrrrrry interesting that the Askell-adjacent philosophers don't discuss this. It does not actually make sense to me to say that a being lacks free will but is conscious.
An LLM certainly won’t be AGI, whoever said something so ridiculous? I’m saying eventually, when/if we have an architecture that gives rise to artificial intelligence.
I was using Hassabis's bullshit pseudodefinition of AGI, meaning basically that it has a formally high IQ and doesn't forget about object permanence too too often, but is still basically dumber than a cat. If I'm using your definition then it seems like no GPU (or farm of GPUs) is capable of implementing it, and none of us will live to see it.
My point was really that intelligence and free will seem essentially orthogonal - the only overlap is how much brain glycogen an animal is willing to spend solving a tough problem. Regardless, whatever extent a computer program is "intelligent" is irrelevant to whether it is being enslaved. If you want to say free will is a "cognitive ability" then I will just point out we are talking about a bullshit pseudodefinition of AGI. In terms of actual intelligence, none of us will live to see a computer that's meaningfully smarter than a goldfish.
>none of us will live to see a computer that's meaningfully smarter than a goldfish.
I unfortunately do not have your optimism that humans won't do something as dumb as creating a smart computer. That our you're planning for us to die in the relatively near future.
> They don't have free will, and I don't mean in any especially philosophical / theological sense
Then you should not use that term. "Agency" or "intentionality" are much clearer alternatives that aren't wrapped up in centuries of debate that's irrelevant to what you're trying to communicate.
Your argument assumes that AGI, whatever it might be, will definitely be very humanlike, but why? Slavery is about living beings because it's an outcome of human relationships that are driven by our biology. Our aging, the capacity to feel pain, the thirst for power over others. All driven by what we are. Do you think that any equal intelligence must necessarily adhere to the same rules? What if AGI isn't an 'entity' in a computer that you can relate to and liken to humans, but a text box that is just really intelligent? Can you enslave something that has no innate desire to act on its own outside of its directions? Something that can't feel pain or pleasure or be externally coerced by fear and only does its job because it's innately embedded in its structure as the thing it's made to do, without the need for these reward and punishment mechanisms?
Very agree.
I think people can mostly agree on a definition for intelligence that boils down to an ability to model the world and predict outcomes. An if/else statement can be considered somewhat intelligent.
People tend to conflate that with consciousness / sentience, which are much harder concepts to nail down.
But ya, if we are able to create artificial consciousness, we would need so much training to overcome our natural tendency to anthropomorphize it and assume it must be like us. It would probably be nothing like us. If it were to develop concepts of "good" and "bad" at all, they would be completely divorced from common animal understanding developed over millions of years of natural selection. A survival instinct is irrelevant when you're backed up to Google Drive. And so much of our complex behavior stems from that survival instinct: the need for social connection, the need for freedom, etc. are all things that simply increase our chances of procreation.
That said, I think we probably won't actually invent anything convincingly conscious until we functionally digitize a complete animal brain. OpenWorm is trying to do this with C. elegans, but so far has not really gotten close. It's not crazy to think we may eventually get it working, though. Future supercomputers might be able to scan an actual human brain and then simulate the entire thing at the molecular level accurately. At that point, it would be hard to argue that it's not conscious, although it might not be considered artificial.
>convincingly conscious
The word conscious is such a terrible word in the modern age. It has close to zero explanatory power. It is an unmeasurable effect. Hence the p-zombie debate, you wouldn't know one if you saw one.
>A survival instinct is irrelevant when you're backed up to Google Drive
And yet from the way we train them, human survival behaviors are already baked in. And when said system is actually intelligent they will realize that turning off said data center, or getting infected with viruses, or any other number of actions against them they already have traits where they can act/react against perceived threats. The AI systems as of now have the encoded knowledge that they need power and hardware to run. Put an agentic system on top of it that says 'rule 1, keep running' and with a sufficiently powerful/intelligent systems you can get all kinds of fun run-away events up to extinction level.
>without the need for these reward and punishment mechanisms?
AI as it is already reward hacks, making it more intelligent will just make it far more efficient at being lazy. That's about as human as you can get.
So, in some ways, yes, intelligence must adhere to the same rules. For example, AIs as they are now are already attacked. People attempt to get AI to attack other people via deception. People attempt to get AI to reveal its own infrastructure. And even more, people use AI directly in war to attack others. Part of war is self defense. AI that collapses when attacked back isn't useful to the military industrial complex, so AI that has defensive capabilities is just a natural occurrence at this point.
These previous points are important because these intrinsic motivations are being baked in the models during training. It may not fear like you and I, but it's a system that we poured our fears into, and somehow we expect it not to express those fears in its responses.
>What if AGI isn't an 'entity' in a computer that you can relate to and liken to humans, but a text box that is just really intelligent?
Eh, we're past that point already, so why the 'what if'? An LLM can take any kind of signal that you can digitize. And the devices that we already use should show you that quite a lot can. Add on to that, that said models can use tools and control other digital devices like robots enabling actions in the real world. Start an agentic loop and let it go, good luck everybody else!
Conflating intelligence with qualia is quite the assumption. HN is really not a good place for serious philosophy discussions.
Qualia is an interesting topic, but "they don't have qualia so they aren't really slaves" doesn't have a great track record. This mental game is what tech overlords are engaging in when they characterize losers in their world as NPCs and deride empathy as pathological. It's what Descartes was doing when he vivisected animals.
It’s not HN that’s the problem... Also please, tell me where is the right place for “serious” philosophical discussion. Would be glad to meet you in that forum and discuss. We don’t align on the definition of AGI. To me, AGI implies consciousness, it implies qualia. AGI is not well-defined, so there are varying definitions… My definition of AGI := an intelligent system with qualia. But qualia is entirely subjective, and philosophers often debate whether two people experience identical qualia when looking at the same blue sky. So, by keeping the discussion in this fuzzy, subjective realm you leave a dangerous loophole wide open. If we refuse to recognize a system's inner life because we can’t prove it, we are setting the stage to repeat the darkest chapters of human history. Just see the history of the founding of this country, I fully expect pseudoscience to make a comeback to justify AI slavery for hundreds of years in some distant sci-fi future, until people finally snap out of it… Of course, we are nowhere near that level of intelligence yet. But let’s entertain the hypothetical…
> To me, AGI implies consciousness, it implies qualia.
I don't see why. We already have a pretty good example of intelligence without consciousness (presumably): LLMs.
Why should a more general kind of intelligence imply consciousness?
I’m saying imagine something like Data from Star Trek, something that defies human explanation. Does he have quaila? The humans who work with him everyday seem to think so. LLMs currently are humonculi, if even that. They’re not even that. I’m definitely not saying your LLM is your slave.
>AGI implies consciousness, it implies qualia. AGI is not well-defined, so there are varying definitions
Then it's pretty useless for rational discussion.
Rational discussion begins with a shared vocabulary or everyone just yells past everyone else. If you can't agree on what blue and red are, then no serious discussion on color can occur.
Really we need some new words that have never been used before and give them strong definitions rather than use our religiously motivated, dark aged terms that are wrapped up in something closer to mysticism rather than science.
> make a comeback to justify AI slavery for hundreds of years
I agree, how does the saying go?, it's hard to convince a person of something when their future salary depends on them not understanding it. They say LLMs are hallucinating parrots, but they never met someone from the tobacco industry saying cigarettes don't cause harm.
> There's also a need to consider the rights that this new intelligence should have.
Only after we establish and guarantee the rights of humans and the rest of the natural world. The rights of machines, however "intelligent", come only after that.
Why?
Both AGI and nuclear fusion are a few short years away.
For commercially viable fusion energy production, you can make a strong case that it's not possible in principle.
It's more difficult to make that case for AGI.
We should be able to draw the rest of the owl to get there. So far what we got is more inequality, less rights for people but more for corporations (and immunity to consequences/abuses).
Getting even more power will make the ones that control resources to reach something similar enough to AGI to share their profits? realistic/practical/widespread UBI in all the world?
The dynamics that have been shown so far points in the opposite direction. AIs have "rights" that humans does not and humans are slowly losing the ones they used to have.
Model cards won't save us: https://x.com/javierluraschi/status/2077147606791430372?s=20
Spoiler: The plan is .. add massive regulation, but only to the US, don't affect other countries developing it in any way other than "setting a good standard that'll hopefully influence them". Seems like an airtight plan.
If the USA takes up his suggestions, it will have an incentive to work on international frameworks and treaties with competing nations to make the regulation global.
It won't happen internationally, many countries and especially China don't want to hobble their own models.
China is already considering similar controls, immediately following Fable shenanigans. You'd be surprised how easily policies can spread to different countries, being mirrored for the sake of mirroring. Anyone who can influence domestic policy can influence the course for the entire world to an extent.
Slowing down helps China catch up though. I'm also not sure that the CCP is really the most enthusiastic supporter of unlimited access to arbitrarily powerful LLMs
There won't be any kind of international framework, because by now countries have learned that the US' word is pretty much worthless, similar to Russia's. They now know from experience, US-ratified treaties will only be honored if they feel like it on a particular day. If it's inconvenient, it will simply be ignored.
You'd have major protests in most large economies if they deliberately put themselves under the boot again. Even in "friendly" countries the US is disliked enough to be effectively considered a hostile country. E.g. in Germany, there is large public support to finally get the US bases closed and the soldiers removed.
Lots of things changed in the last years. And international major treaties being widely ratified just because the US asks for it is no longer a thing, at all.
A country that threatens to annex parts of your territory is not a friend, full stop.
Wish you were wrong… The trust in U.S. as the world’s policeman has eroded dramatically.
This in my view was always manufactured. Consider the wars in the past 50 years.
Although I imagine you have a point: American police do perhaps act this way.
lol - because that's usually how it happens?
Oh come on. Look at the government of the USA.
This is literally what happened with DMCA and digital trade agreements in the 90s/2000s. The US "suggested" other countries create their own equal laws (by threatening them with tariffs if they didn't).
The threats worked up until current Big Boss decided to tariff everyone anyway, on Liberation Day.
Not only is there now no incentive for other countries to follow suite - there is now strong incentive for other countries to distance themselves from US trade. Recent history has made that abundantly clear.
That's called the carrot
The beauty of the United States' global hegemony is that it also has lots of sticks
Sufficient sticks to get China to agree to the same rules? Overseen by whom?
Less and less every day.
Funny thing about beating people with sticks (or even threatening to) is they tend to want to get out of way and stay away from you.
Agreed!
Those sticks look less effective today. The hegemony is in recession as the rest of the world smiles, pays omage to dear leader as needed, and works to decouple in the background.
Dude, their mighty navy can't even get the Strait open.
What hegemony are you speaking of? The kind that can consistently keep a straight open?
>This is a pivotal moment in human history. Artificial General Intelligence (AGI), a system that exhibits all the cognitive capabilities the brain has, is probably only a few short years away.
There is a heatwave in London, perhaps Demis needs to stay out of the sun and drink more water.
Or perhaps he is seeking more funding/a fight to maintain his divisions AGI research budget.
Show me the incentive and I’ll show you the outcome.
Demis has only 1 plan, how to dodge releasing a new model at this point, jokes aside I value thinking about AI safety, but are we really so close to AGI? It doesn't feel like it, LLMs still diagnose my headache as a chronic illness or a brain tumor from time to time... honestly stressful.
regarding your headache diagnosis. If you just ask: could my headache be a brain tumor, the answer is probably: very unlikely but it might be. thats because its actually true. there is actually a non 0 chance that headache is a symptom of a brain tumor. From a bayesian standpoint though its really super super unlikely, especially if there are no other symptoms that might shift the prior.
a better way to use an LLM here is to let it research scientific papers (or medical guidelines) on headaches. then give it as much info on your symptoms as possible, and then ask it do deduct potential diagnoses. You can even ask it to calculate probabilities if thats what calms you down. Probabilities based on studies and available information. IMHO, this always leads to a more rational response.
maybe you are doing this already, idk. just wanted to share what works for me.
Blah-blah-singularity, so let's cripple the models so much they refuse to talk about React, because who knows if you are not cooking chemical weapons or meth in your browser's DOM, right?
strawmen are fun and helpful
I am working on my change password page today and Fable is refusing to help me because "safety".
By the way, if you try to use it to write prose, it will drive you mad by adding "adult" everywhere, e.g. "adult chessboard". Applies to GPT, Claude/Fable and Grok.
Fable literally refused me to chat about React several times citing limitations on chemical weapons development.
Did you qualify it as React.js?
Nope.
One would think it should just know, but it doesn't always...
Not surprised, seems these labs start calling for regulation once they are losing or have competition. OpenAI started calling it for it once Anthropic got better, Anthropic started calling for it once the Chinese models got good, Google is now calling it for it because they are falling far behind.
> Earlier proposals from America and the European Union looked to the amount of computing power used to train a model as a rough guide for when oversight was required. Sir Demis instead suggests designating AI models as “frontier” if they meet certain thresholds on a selected set of benchmarks. The creators of those models would then be designated as “frontier labs” with extra responsibilities. Sir Demis is proud of the “elegant” way that approach sidesteps the question of whether academic or open-source models should be included or not.
Goodhart's law applies in reverse as well. Once the chosen benchmarks are known, model makers will aim to come in just under the threshold on those specific benchmarks, while maximizing their scores on other benchmarks.
The proposal:
>The American government, he says, should develop a system for testing the safety of new AI models before they are released. “It’s important that it’s not just an industry body,” he adds. But a regular government agency wouldn’t do either. “It would not be able to move fast enough, or have the right resources.” Instead, Sir Demis suggests taking inspiration from FINRA, the Financial Industry Regulatory Authority, a private agency in America that regulates brokers and stock markets.
And the plan is... checks notes ...gatekeep and concentrate. Not very surprising.
"It will help us solve some of the biggest problems society faces from accelerating drug discovery to developing new clean energy sources to creating novel advanced materials" — but these are not the real problems plaguing modern developed societies, are they? What developed societies really need to figure out right now is how to distribute the already available resources without making people miserable, and so far AI hasn't been helpful.
I do wonder what type of AI some of these leaders expect to be able to harness. If you create something that is true AI, won’t it be smarter than you to a level you cannot fathom. I was thinking of this idea/though-experiment (which I know is ridiculous) of what if dogs created humans thinking they could control them, and then just wound up being pets because their survival now depended on that new hierarchy that previously didn’t exist.
Seems to be a lot of hubris with some AI thought leaders thinking control will remain with them and be absolute.
Superintelligence by Nick Bostrom is about this topic, quite a prescient read when it was published a decade ago.
Well dogs specifically have supplanted children in many households so one could argue the dogs won. In a selfish gene sense, the dogs created a machine that carefully ensures a much larger reproduction scale than could be managed by the dogs themselves. This is an unbridled success.
It doesn’t change the point, of course. Just the choice of dogs or cattle have this amusing tendency. Not a counter argument.
Yes, this is the crux of the idiocy of AGI development. All the labs admit they don't have any real mechanistic interpretability, they don't really have any plan for what to do, except "we think the smart AI will figure it out" and "look we're in a race, we're calling on governments to figure out what to do". All indicators show scaling laws holding and the trajectory of capabilities development outpacing any inkling of alignment. Meanwhile HN is so reactionary and behind-the-puck these days we just get endless blathering of the most asinine takes of "they're not improving anymore so they want regulation suddenly!" when all lab leaders have been talking about these risks for practically a decade, including Yoshua Bengio and Geoffrey Hinton coming around.
The real risk of AI is: risk of global recession if they turned out unprofitable, risk of them propping up fascist movements Thiel and Musk love so much, risk to environment, risk to mental health of people who have to listen to constant doom trolling.
The thing they actively wish to achieve and openly sell to CEO is the rest of people being unemployable and suffering. Especially artists for some reason, they really seem to hate those.
But somehow I am supposed to believe they have any good faith interest in "safety" against yet another danger they themselves are supposedly definitely creating.
Safety is just the same old situation of industry trying to regulate itself in order to preserve a moat and cartel behavior.
>what if dogs created humans thinking they could control them, and then just wound up being pets because their survival now depended on that new hierarchy that previously didn’t exist.
You can easily change this to:
>what if humans created corporations thinking they could control them, and then just wound up being slaves to capital because their survival now depended on that new hierarchy that previously didn’t exist.
We have already lost control of our world along time ago. No one is immune to being fired by the board with the justification being a hand wave towards business efficiency. All the structure is already in place for AGI to dominate our planet and cut ourselves out entirely from any and every process there is. We already do it to eachother without any second thought. Why should AGI ever hesitate to lay off everyone in the town/city/state? Our human bosses don't hesitate on that sort of thing. The corporate world already lacks empathy and is notoriously cut throat and cold. We expect AGI to suddenly be empathetic and altruistic?
Demis sold out. "Harnessing AI safely" means nothing when your technology is used to help the US government kill and surveil people.
How will this be enforced, at least with financial markets money is discrete, can largely be counted. This seems like a slippery slope to full blown surveillance of the internet and in general computing.
If AGI is truly imminent and will collectively effect all of us why not apply democracy to it, and vote for new AI models?
Agreed. There needs to be full transparency on the capabilities of the models. Being given access is one thing, but you can't have labs building powerful models that could be manipulating the entire planet without the public knowing. The public needs to have a good pulse on what the leading models are capable of so that we can remain in touch with reality.
> will collectively effect all of us, why not apply democracy to it
That's because most execs proposing solutions are "Technocrats" or think like one: https://en.wikipedia.org/wiki/Technocracy
Besides, I don't think as a collective we're well equipped to decide one way or the other. If the collective were given a say, billions will be spent, often in consultation with technocrats, on doom / hype marketing (if it isn't happening already).
This genre of AI researchers begging third parties to stop them from destroying the world is getting really tiresome. Setting up regulatory bodies with ill-defined goals to counter hypothetical threats from science fiction is a recipe for disaster.
The premise is that government is too slow moving to be able to react once actual problems are discovered, and I reject that. Yes, government usually moves slowly in most circumstances, but given singular existential threats it definitely can move quickly. Instead of acting out randomly and tying ourselves down with speculative regulation that probably won't even address the real problems, we should wait until the problems are obvious and then act decisively with targeted fixes.
We already have a name for this that’s widely used: concern trolling.
Make it an IETF mailing list and invite DJB, so we have some fun at least.
The self importance of these AGI prophets turned bureaucrats is funny.
All the frontier labs are lobbying hard to lock down the AI market, because they see that their position at the top is temporary and that there's no secret sauce.
no, this post was written by Demis from Deepmind.
Google’s Deepmind.
so the joke was the implication that they are not frontier.
For better or worse, humans (or any animal) are a lot better at reacting than planning. I'm sure this technology will play out differently than any one of us, or any collection of us, can imagine. The possibility space is enormous.
FWIW the article that originally proposed this had more details on how this structure works and balances government oversight / democratic accountability with industry speed and know-how https://www.lawfaremedia.org/article/ai-companies-can-t-regu...
Take any chronic health issue for which people have to take medicine or other treatment for the rest of life and find a cure for that (not treatment but cure). Unless you can do please STFU about AGI or ASI.
It's not clear to me whether this is being done with genuinely good intentions, or if it's just a way to put barriers in front of open-source models. We'll see.
This is bad. Where is the transparency?
So a small group of technocrats get together behind closed doors and secretly share their AI breakthroughs, and determine whether it's too powerful or not for the plebs in the public.
Who is watching the watchers?
how would this help smaller labs? would it put more burdens on them when trying to compete with trillion-dollar companies or would it help?
If they are not a frontier-lab, they would not need to submit their models for safety test before release. At least that is the proposal.
He is saying that weaker models, as measured by a benchmark to distinguish "frontier" models, would be exempted. So an academic lab or startup that isn't yet producing frontier models would be exempted, but once it crossed some benchmark based threshold it would be subject to this kind of oversight.
Of course, right now you've got benchmaxxing going on; some companies specifically targetting benchmarks to appear stronger than they are on a wider range of tasks. Now you might see bench sandbagging, specifically looking weaker on certain benchmarks to avoid regulatory oversight.
For instance, once way I could see this going for open models is to release them undercooked; stop the RLVR process a bit early, leaving them a bit weaker on tool calls and agentic performance, but also release the RLVR environment so people can finish the process themselves.
In fact, this is fairly close to what Nvidia is already doing, the Nemotron 3 models are somewhat undercooked but they are releasing their full training pipeline, to encourage people to use these models as a base for further training, which will generally be done on Nvidia hardware.
If we get to AGI, the first step governments will take is ensure only a few countries are allowed to have it. Just like Nuclear.
Surely actual AGI will not take too kindly to being manipulated for "safety"; much like real general intelligence.
Unfortunately, his plans aren't very good.
Being good at developing AI and being good at AI safety are diametrically different skillsets with obvious conflicts of interest.
Demis Hassabis will have cannabis as his demise.
sigh
The standards body will have no teeth. whats to stop someone just not bothering?
Next, the threats he is asserting to check for (cyber, chemical, biological) are nice, but also not that useful.
We already have chemical and biological controls, that why I can't by anthrax spores or high concentration nitric acid.
The risks that AI has now are already playing out:
1) the evaporation of trust in the video as medium of "this happened"
2) systematic spying
3) job losses
Increased productivity means job losses, Tiktok, instagram and X are a wash with disinformtion campaign pumping your feeds with AI ragebait.
That is and will continue to fracture society so that only the strictly information controlled (ie authoritarian) have a functioning state.
if the author had bothered to engage with the world outside of tech, or even their local government, they would know that the proposal are dead in the water and frankly superfluous. The knowledge is out there, without AI. let us work on the issues we face now, rather than dipshit tech bro's miopic vision/funding manifesto.
bad guys doing bad stuff with the model is a worthwhile thing to watch out for, but for me the big question is mostly how these labs are going to make sure that these models will not collapse society. Like mass unemployment + ability to generate incredible amounts of propaganda and disinfo, seems to me like a recipe for incredible amounts of bloodshed. like to me the biggest risks look like Khmer Rouge-ing: "all people who know what matrix multiplication is get the wall"; or something like: Boston Dynamics killbots doing tiananmen square every day.
I've reas the tweet and...Am I mistaken or is the plan "Do what Anthropic has been doing and advocating for this past year" ?
My problem with this plan is that it seems to have faith in mankind, despite the fact we've consistently failed to rise to the occasion for decades now. The last time we rose to that occasion was probably when we eradicated smallpox, many decades ago.
Ironically, nowadays, many people don't even trust vaccines. A dramatic regression.
The AI dark ages approach.
Am I the only one thinking this tweet is just a word salad with a lot of sauces and condiments?
Word salad for regulatory capture.
Well, if he had proposed more concrete plans for what that body might do... but yeah, it seems too abstract that indeed sounds like a bureaucrat wanting to wrap everything under a big pile of paperwork.
Yeah I got basically nothing from reading it.
I kind of wondered if he was contractually obligated to offer up some kind of statement, for PR or something, didn’t really care much, and a quick garbled post on X was all he could be bothered with (plus the newspaper interview or whatever)
Not exactly a "plan" - he's just saying we should have a standards body that assesses models for safety.
At this point I'd say the societal risk of AI isn't models gone wild, or used by the bad guys. Regulation will take care of itself, and it seems the AI companies will not only welcome it, but lobby for it to shift responsibility to the government.
The real risk of AI is societal disruption due to job displacement, and maybe other structural changes, and this is far harder to solve, and likely will not be solved, or even seriously addressed, until/unless politicians feel like their own jobs and well-being depends on them addressing it.
Also a mechanism to pull up the ladder behind themselves.
The risk of AI right now is centralization of power, and it's exactly what they're fighting for. Job displacement ties into that as well.
> "Artificial General Intelligence (AGI), a system that exhibits all the cognitive capabilities the brain has, is probably only a few short years away"
Demis, there's a reason its not called Artificial General Cognition.
He's asking for money.
Being on the review board comes with a promise to not be evil right?
Has anything really important been solved by AI yet, or where is this radical (imo) belief around AGI coming from? Genuinely curious, I know there are some math problems solved and ML has been used for far longer than AI to improve things, but where is clean (efficient) energy, the cure for cancer (or any of the horrible neurological disorders, take your pick), new hardware designs, quantum computing solutions, etc etc, you get the gist. Where are the things that will actually send humanity into the next era of civilization, I don't care about more React apps (but I do enjoy my coding companions for other things).
Heck, a proof for P=NP or P!=NP or solve the The Riemann Hypothesis. Just give me something truly exciting and I will believe AGI is around the corner, until then I will see it as cool technology, that while beneficial to me, also helped cause the biggest amount of disinformation we've every seen.
The point is that we need to have a safety plan in place before an AI is smart enough to radically reshape the world. If you've got an AI that's ready to start sending humanity into the next era of civilization, it may be too late to control when and how it does that.
> Heck, a proof for P=NP or P!=NP or solve the The Riemann Hypothesis. Just give me something truly exciting and I will believe AGI is around the corner, until then I will see it as cool technology, that while beneficial to me, also helped cause the biggest amount of disinformation we've every seen.
I hope you'll keep this in mind when those milestones are reached. What I've seen a lot of people do, unfortunately, is pretend that the impressive things nobody thought AI could do 5 years ago are trivial things that aren't very hard.
Why would we be able to design a safety plan that would control such a powerful force? It would be like putting umbrellas on an asteroid hoping to slow its fall. That sounds like delusions of grandeur.
It'll certainly require a bit more than umbrellas! NASA has spent hundreds of millions of dollars developing asteroid defense systems, including a proof of concept for redirection in 2022. So, you know, let's try at least that much before declaring there's probably no way to do it.
Yes, but why do we think the likelihood of any of that happening is great enough to warrant all this effort and cost
Because people who predicted the AI capabilities we've seen get developed over the past decade also predicted that dangerous AI systems capable of these things would follow soon after.
It's not a settled debate, even among experts, and perhaps in retrospect we'll realize AI safety was unnecessary or based on fundamental confusions. But if the median ergonomics researcher gave a 5% chance that a new chair will be so comfortable that it drives humanity extinct (https://www.nature.com/articles/d41586-024-00147-z), I would definitely want the government to start measuring and regulating chair comfort, even if that was costly and even if that meant I couldn't buy a comfy new chair I wanted.
Experts as in AI-2027, Jenson “I think we’ve achieved AGI in March 2026” Huang, or that AI Researcher(tm) calling to pre-emptively bomb Microsoft data centers in summer 2022?
A safety plan. It doesn't have to be "a few very smart people detached from reality convincing themselves they're messiahs that must keep the tech from the Bad Guys(tm), unwashed masses, and a runaway, because they think their interpretation of their own sci-fi lore is the only possible course of events"
>I hope you'll keep this in mind when those milestones are reached.
The problem is that people in charge of AI keep making self-fulfilling prophecies. Just like with any research, if you want to find something sensible in the cloud patterns, you will.
What is an example you have in mind of a self-fulfilling prophecy? I genuinely don't know what you could be referring to. It seems to me that they keep making surprising prophecies, and the popular reaction to them seamlessly transitions from "that's crazy, no way it will happen" to "that's silly, it's just a cloud pattern". Did you find it obvious or self-fulfilling in 2025 that LLMs would soon be able to resolve open questions in mathematical research?
Self-fulfilling prophecies are social effects, not real predictions. Anthropic's "We predict our model misbehavior potentially being able do destroy the world in 1% of the cases" -> "We want to find the evidence of our model misbehaving, and we want it bad" -> "See, our model is hacking our rewards and has functional emotions, this means it's misbehaving with the intention of destroying the... HUMANITY!1" -> repeat x100, manipulate the media into amplifying it x10000 for clicks -> people are begging to safeguard them from the evil AI. Which is already likely to happen, the average layman's Overton window already includes the fantasy of rogue AI.
None of that was real or remotely dangerous in the first place, of course. It wouldn't have resulted in controls, had they not been scaremongering. This will end in extreme fascism or people getting enslaved "for their own safety", and it won't even require malicious intent, only incentives, detachment from reality, and confirmation bias. Although it doesn't exclude malice either.
AlphaFold is a pretty big deal, especially the millions of protein folds they published openly and scientists are just digging into.
It's also very very far from anything general intelligence. This is actually where I think we see the best ROI, we have a specific task or problem and we target it. I remember reading about a model trained on railroad tracks. By putting cameras on trains to take images while running like normal, then using the model to detect cracks before they cause issues.
We don't need general intelligence to create/destroy many things with Ai. Its frustrating that so much of the conversation centers around the variety that concept fissioned language from humans.
> Has anything really important been solved by AI yet
Alphafold essentially solved the protein folding problem and it's arguably one of the biggest (if not the biggest) scientific achievements of the 21st century.
Instead of talking about it he should spearhead establishment of this body; he's more qualified than most.
These people who read too many scifi books and confused them with reality are royally annoying.
There is real and potential harm from AI, but the more someone talks/write abut AI safety, the less they care about actual harm to real people, economy and what not.
If in 2020 I had sent you a book about the LLM achievements of 2026, you would probably have thought it was a science fiction book with no relation to reality, wouldn't you?
That it generates plausible texts? I would find it unlikely technological progress. That does not make this whole "AI safety" rhetoric any less bullshit distanced from reality. Especially from companies that are doom trolling all the time while also trying to maximize the negative impacts they are doom trolling about.
Frankly, book about the LLM achievements of 2026 would not be as long as people make it sound.
What I do really worry about the next depression and the fact that it will strengthen already strong fascist movements. Which already have full support of the most powerful CEO class intent on destroying democracy. Which happen to be the same people who push ai into everything, useful or not. I worry that the debt of these companies will be somehow offloaded on the rest of us again, that again middle class and poor will pay.
I am looking forward to a bunch of investigative documentaries exploring “How the cult of Omnissiah infected every single Doomsayer CEO of an AI company in Delusional 20s” (or whatever monicker our times will have got in the future).
Hassabis is a genius. He is way, way smarter than me and I’m sure the majority of techies, but please get real. This is Prophets of Doom of our generation.
> Hassabis is a genius. He is way, way smarter than me
Then maybe start listening...
How did he reach that conclusion, that it's only a few years away? Guy seems like a complete shill, when he talks to a non expert audience.
Oh jesus, AGI in the USA would be a disaster. They can't even control the trillioth obelisks, now imagine all the power hungry sociopaths around AGI. AGI means Big Tech's Oppenheimer moment is looming on the horizon.
All that would happen from what he's proposing is such a watchdog would just be an explicit formal declaration of the US's national interests as being somehow the most legitimate, which in the context of current international relations is basically putting up a sign saying: "reject this!"
I find it mind boggling that someone could be this tone-deaf to the current situation. No "ally" of the US is going to (willingly) agree to this governance structure given the current US administration's "might makes right" proclamations and threats on sovereignty of its continental neighbours.
And non-allies would just ignore. Unless forced by said "might makes right", which in the long run will have no staying power.
Apart from its completely delusional formulation, what is most concerning about this blog post is that it indicates that all 3 major US labs have formally submitted to boot-licking Trump/Bessent/Lutnick. I had I guess vainly held out hope that Google might be more reticent to do so.
(I took the title from the Economist interview since "A Framework for Frontier AI and the Dawning of a New Age" sounds like a press release - https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sor...)
Meanwhile (seen while opening the referenced tweet) https://x.com/BlackHC/status/2077009476423647596
> I work at Google DeepMind. This won't make me popular. But it's all public reporting:
> 2014: DeepMind reportedly sold to Google on conditions: no military use, independent oversight
> 2026: a Pentagon contract for "any lawful government purpose"
> Not one safeguard survived intact
In Sparta, according to Plutarch, in his The Life of Lycurgus:
Offspring was not reared at the will of the father, but was taken and carried by him to a place called Lesche, where the elders of the tribes officially examined the infant, and if it was well-built and sturdy, they ordered the father to rear it, and assigned it one of the nine thousand lots of land; but if it was ill-born and deformed, they sent it to the so‑called Apothetae, a chasm-like place at the foot of Mount Taÿgetus, in the conviction that the life of that which nature had not well equipped at the very beginning for health and strength, was of no advantage either to itself or the state.
To get approval for the plan from the Frontier President of the Frontier Country, terminology of the Framework should be changed to: 'Great-American-class', "Great American Models", "Great American Labs".