> "The chatbot personas are deeply misaligned with you, and aligned with their owners; and the economic incentives are to farm you with ads and subscriptions, while racing not to amplify you but to replace you."
> "On my visits to the Bay Area, I would ask AI researchers or interns why they are doing their current research or projects, when in a year or three agentic LLMs could probably do them; they rarely had a good answer, or any idea what they would be doing in 3 years"
> "One programmer driving 10 Claude instances, because he has to review their work, will never be as valuable as fully autonomous Claudes where there can be almost arbitrarily many instances, like 10,000 instances… but such scaling requires removing him from the loop as much as possible. And this is true of everyone else, whether lawyers or writers or researchers: increasingly, you are the bottleneck to be optimized away."
I fully support the 3 core principles of GA: (1) Enhancement, not replacement (2) Mental Sovereignty (3) Self Actualization, which I think is a path to a more humane future.
Leaving AI completely aside, it still amazes me that people finds "novel" the idea of removing highly-paid white collar intellectual workers (software developers or otherwise) completely out of the loop.
No-code platforms date back to the 80's. Getting rid of engineers in general is even older [*].
Even relational databases and SQL were initially promoted as "ways to get rid of those expensive programmers to access your data" because they resembled some form of English.
The funny thing about the ad below is that stuff like "stop hiring / get rid of humans" would have been seen as highly insensitive in 1950's America, so they touted that as "put them to do something more important".
To colaborate a bit with the no-code part, I worked at a place where they had some flows made with n8n, but the last one from non tech/software engineering left the company and left a bunch of flows breaking, because of some edge cases the flows aren't handling like reissuing credentials, throtling or bad input. The people dependent of said flows reached out to the engineering team to help fix them!
It's novel because previous rounds of automation were about automating specific tasks or well-scoped functions. There was always an implicit understanding that the white-collar worker would be freed to spend their time on more valuable, higher-level problems. But this time is different because of the generality of the technology. Agents promise to automate the process of thinking itself. And in many domains they can learn new tasks as fast as white-collar workers can find them.
> It's novel because previous rounds of automation were about automating specific tasks or well-scoped functions. There was always an implicit understanding that the white-collar worker would be freed to spend their time on more valuable, higher-level problems. But this time is different because of the generality of the technology. Agents promise to automate the process of thinking itself. And in many domains they can learn new tasks as fast as white-collar workers can find them.
Nah, sometimes the expectation and advertisement was that you could let go of the white collar worker because you're paying the overseas person 1/10th the amount. And "overseas person" is pretty general.
"Everybody knew" it was a bad idea to get a CS degree for a bit after the dot-com bust because of that.
(Some white-collar industries did get hit much harder by that; VFX is one I've heard in that context quite a bit.)
I’m at a fully remote company with staff in at least 8 countries speaking at least 5 languages. It works out fine. A possible analogy to AI is that a lot depends on how you use it. The “skill issue” doesn’t disappear, at least not yet.
You say that as if the culture difference with a truly alien intelligence is insignificant compared to the culture difference with an "overseas person".
(Even assuming "intelligent" is a sensible label to apply to an LLM holding hands with a shell script in an infinite loop)
That's a good point. However, overseas people are still people. They need to sleep, get sick, and the better they get at their jobs, the more money they will demand. The cheap ones also often have communication barriers and work slower than the workers they're replacing.
AI models get better and more efficient every 3 months, run around the clock, can be copied infinitely, and unprecedented amounts of capital and research talent are being thrown at any limitations we can see with them (such as problems writing correct code in 2024, lack of agency in 2025, autonomy and self-improvement in 2026). That's the difference between labor replacement through outsourcing vs. labor replacement through automation.
We are either living in different worlds, or squabbling over different meanings of words.
Models have absolutely acquired agency as of 2025. Developers are no longer copy-pasting code from ChatGPT into their text editor, they're working with agents like Claude Code and Codex that can edit code, run terminal commands, do web searches, manage their own context windows, sift through gigabytes of logs with datadog MCP, etc.
Self-improvement is also being worked on. Claude Tag learns over time in slack convos. My company also has an agent that updates its own skill files after every conversation so that we don't need to keep reminding it about the same workflows every time. Is it clunky as hell? Yes. Are the labs plowing billions of dollars into "continual learning" and "recursive self improvement"? Also yes.
What you call a model acquiring agency I call plain old software with productivity workflows designed by humans, with deliberate goals. We must separate “model” and an execution environment using a model. [Model] ≠ [A glorified shell script doing API calls in a control flow based on heuristics]. Agents are not AI, they are plain old software. The weights are the model, and that very much remains a static artifact (and pre-post training models haven’t improved much over the last few years).
What you call self improvement is a duck tape hack to imitate persistence and save on inference. Every time you do an API call, anything that needs to be processed is sent to the model. Narrowing that context down saves money. Finding clever ways to do that improves apparent performance and value. The cleverness is still human.
These are all useful innovations on top of LLMs, which remain models that generate text and symbols based on static weights, which in turn represent training data and the provider’s preferences.
Marketing about a more efficient product that requires fewer engineers is obviously not new. The idea that human engineers are obsolete is pretty new, that's the claim that is getting pushback.
I don't think the no-code comparison is valid, it's always been fundamentally flawed since its many leaky abstractions sitting on top of systems programming code
LLMs can't replace developers but its foundationally different because it can operate on systems code instead of building abstractions on top
> > "On my visits to the Bay Area, I would ask AI researchers or interns why they are doing their current research or projects, when in a year or three agentic LLMs could probably do them; they rarely had a good answer, or any idea what they would be doing in 3 years"
I heard this already when ChatGpt came up. Still waiting to be fully obsolete.
For 20 years my job has been automating my job away. I remain unafraid of my future necessity, and as always, excited for the work I get to do because of the work I no longer have to do.
The last 200 years of science, technology, and engineering is a long story about eliminating work people had to do being replaced by higher level work. Each time, each wave, there was always the FUD about the work nobody had to do any more and each time there were always new things to do enabled by people no longer having to do so many of the old things.
The human won't exit the loop. The people who imagine they will have, at the same time, too much and not enough imagination. Their endgame is always some kind of hand waving magic where suddenly everything is fixed and works.
Previous technologies eliminated single categories of worker at a time. The machine learning that people are working on these days threatens to replace very broad swaths of the workforce, of many countries, over a very short period of time.
I can find agreement with a lot of this post (and probably misunderstand just as much), but it's striking to me that it seems to overlook or avoid the obvious moral/ethical option: Pushing AI towards Personal Computing.
It seems clear to me that if these models are to have any benefit for humanity then they must be made directly available to as many humans as possible. Right now that cost and effort is astronomical, but general computing also went through a mainframe/shared model before becoming personalized.
I believe The Personal Computer was and still is a gift to humanity; for those with an interest in gathering its yield. And now we have an interface with it that can attempt to introspect its' program and speaks human tongue! Imagine what could have been done with this at the time...
But if that approach clutches too many pearls to stomach (and I agree it's not all roses) then I'd submit that anyone who champions less is seeking to become our master.
I've known gwern for the better part of a decade. Working with him has been great. We've done quite a few projects together, including being the first ones to demonstrate that GPT-2 could play chess (or rather, can be used for actual useful work instead of just being an autocomplete).
He's a great person. I've wanted to do a writeup on it for some time, but what surprised me the most is his humanity. He genuinely cares about the implications of his work. But beyond work, he also cares about the people around him, and it shows.
Just wanted to put in a good word in case someone here was on the fence about applying.
As for GA itself, I think it's an ambitious idea worth pursuing. Imagine an LLM which actually sounded like you, and to an extent, thought like you. How much would you pay to have access to a smarter version of yourself? So the idea is solid, and early results seem promising from the samples I've looked at.
They're also taking personal info very seriously. Obviously, I can't make any promises of what they will or won't do. But they've spent some time studying questions like "What if someone adversarial has access to my GA? Could they get my bank account info?" and came up with a technical solution that I really like.
Honestly I wouldn’t want myself as my own guardian angel. In fact I think very few people look out for themselves well. I don’t want another version of me around, one is too many already. I’d probably be able to identify a number of people whose chimera I would if I could understand them well enough to understand what to stitch together to what, but everyone I know at some core level is deeply flawed and one of them is enough. Maybe I would want some of them available after death as an AI avatar, and maybe I would like the idea of my own avatar continuing beyond my existence.
But I actually would prefer an entirely synthetically aligned “guardian angel” in the role outlined - definitely not -me- - I struggle to do right by myself as it is and two of me working invariably against my self interest would be a nightmare. A smarter version? Sounds doubly worse.
I had a similar gut feeling. To use some possibly dubious and dated terminology, would a guardian angel also emulate my shadow self, particularly if that self was an important part of my identity? If I usually manage not to let my shadow self act out, but you could always tell it was there, would the same be true for my guardian angel? If all you do is use your guardian angel to write essays, then sure, a relatively low risk tool. But if military commanders are using it to oversee offensive drones (as proposed in the essay)?! Oof.
> Imagine an LLM which actually sounded like you, and to an extent, thought like you. How much would you pay to have access to a smarter version of yourself?
I find the idea dehumanizing and revolting. The notion of then selling it is the spoiled cream on top.
I do think it's significant that the supporters of this kind of technology seem to be unusually spiteful.
This doesn't happen with all tech; people who like VR don't like it because they want to hurt others, for example. But consistently, whenever AI is discussed, it becomes clear that some people support it just out of malevolence.
The irony here, of course, is that you are casually insulting me for being interested in this tech in the first place and refusing to apologize or back down. Like how the other guy needed to chime in with "dehumanizing and revolting", even though nobody is forcing him to use any of this.
I wouldn't have any problem with you being interested in this tech if it wasn't driven by spite. Lots of people manage this.
But you said
> After reading some of the outraged comments here, I'm even more excited.
The reason you gave for excitement was the upset it causes others. There are people who like AI who aren't driven by malevolence, but you're not one of them.
People are upset for lots of reasons, but that's not actually important for this discussion, which is about the fact that your support for AI is apparently conditional on them being upset rather than any positives about AI.
That's weird; fans of [almost anything] don't do that.
Yeah, let's just ignore the bit where we're minding our own business, pursuing what we want, yet you feel the need to bark stuff like "dehumanizing and revolting" at us.
>rather than any positives about AI
I'm a huge supporter of AI. I want this stuff. If this upsets you, I like it even more.
I’m not surprised, that’s exactly the kind of person who would want this kind of technology.
Remember, society as a whole will decide what's acceptable, there are no objective, natural laws that decide that. My gut feeling is that this is starting to cut close to what most people will find objectionable.
Yes, I'm interested in this. Yes, I think that comments like "dehumanizing and revolting" are a signal. I'm not going to feel bad about exploring this tech. So, yes, I am that person and will continue to be that person.
>society as a whole will decide what's acceptable
So, how would you describe the past few months? Years? You keep saying that we need to reject AI, yet adoption and the gamut of use cases keep ever expanding.
I think there are good use cases for AI, I never said otherwise. Creating simulacra of real people is not one of them, I believe.
> yet adoption and the gamut of use cases keep ever expanding.
I don't see how that's evidence in favor of AI, as many harmful technologies and products became very popular. In fact, that's exactly why I think a slower, more measured approach is necessary. Otherwise once the harms become apparent everyone wants to shrug their shoulders and say, "well it's too late now, everyone's using it."
> "How much would you pay to have access to a smarter version of yourself?"
How much would I pay to rent my fucking self from a landlord? No, bodylord? Mindlord? Poe's Law.
But looking at the post, they argue that big AI labs have an incentive problem which stops them from personalising, but Guardian Angel will make agents which are "Genuinely yours". In what sense is it genuinely mine if someone else owns it and rents it to me? And how does this fix any incentive problem, they're incentivised to better train wealthier people's AIs, and incentivised to keep dropping "my" intelligence or memory or and then dangle a booster carrot for a small fee. The more they can make it think like me, the more effectively they can work out how to exploit me, advertise to me, propagandise me, and that will be profitable information to sell to other marketers.
For better or for worse, this is valid criticism. Ensuring alignment to the human individual is not even a software-only problem, and I suspect gwern doesn't realize this yet.
Your individual sovereignty will only survive the upcoming era if you own the hardware like you own your brain. And by "own", I mean physically owning the thing and being able to reproduce it on your own. You need to be able to manufacture an entire custom computer from scratch with raw materials, perhaps using a miniature fridge-sized silicon fab + 3D printer home appliance, at home. Otherwise, the individual gradually becomes part of a larger organism because the reproductive locus of control for the computer is at the corporation/society level - a computer cannot replicate on its own and needs a society to build it. The human gets sucked into that superorganism by evolutionary pressures and eventually integrates. Similar things happened in the past when prokaryotes combined to form eukaryotic cells, and when individual cells combined to form multicellular organisms.
Not everyone will get sucked in, though. Evolution doesn't place all of its eggs in one basket. Some people will build robots to automate humans out of the computer production loop entirely, thereby removing integration pressure and producing at least one purely digital species. We will most likely see a variety of species emerge out of this intelligence explosion, some which augment their own intelligence using self-replicating local hardware (fabs that fab fabs, like this but smaller: https://fab2.com), and some which are wireheaded to a datacenter, with varying degrees of success at various scales. They will compete. And there will still be unaugmented humans that continue to live and die the old school way like we do today, albeit with habitable zones compressed.
Selling something desirable has been an incentive since year dot
But this reminds me of the black mirror episode of a female that needs a subscription to live and the company keeps increasing and restricting the subscription
I can nod along when I hear and read people talking about the dangers of these technologies and the endgame of their creators; and then suddenly very weird when everyone except me is nodding along that the answer must be similar (but different) technology!
It's as if technologists are stuck in a room of mirrors, unable to imagine a world in which "solutions" don't ultimately just continue to feed technology's increasingly anti-human takeover of everything.
> Technologists don't drive decisions though. The shareholders do.
Yes, but technologists get sniped into identifying and defending them. They're the mouth pieces and the hands. Shareholders by themselves out to add my uniqueness to their assets would just be helpless randos who can't even make eye contact. Think of what "hiding behind screens" does to people, multiply that by infinite and you get shareholders hiding behind technologists.
The LessWrong/“rationality” crowd gwern is part of like to pat themselves on the back for identifying the trend and resulting harms from super intelligent AI going back decades.
And yet, the main tangible contribution emerging from this early recognition (and obsession) was to inspire a lot of brilliant CS/math types and billionaires like Peter Thiel or Jaan Tallinn to build and fund companies like OpenAI and Anthropic which went on to literally create the very thing they were convinced will destroy the world, leading to an ever escalating arms race to “AGI”. Also lots of crypto and a murky web of non-profits doing unclear stuff.
The proximity to money in the Silicon Valley/VC world seems to limit the imagination to founding yet another new tech startup that really, definitely won’t compromise their values and will be different this time.
The LessWrong/"rationality" crowd aren't compromising their values; they just have bad values that lead to negative outcomes, masked by high-minded rhetoric that they themselves only half believe.
It's a popular movement that has wound its way through out society, and the fruits are clear: surveillance capitalism, eugenics, fraud. It's honestly quite impressive how consistently you can look at [recent negative trend] and find someone at the bottom calling themselves a rationalist and claiming the same set of influences.
The fundamental tenet of that set isn't rationalism but the belief that they are superior to the exploitable masses. That does not lead anywhere good.
Even setting that crowd aside, almost every time I see somebody puffed up about how "rational" they are it turns out to involve a vigorous avoidance of the values they're working from.
I know somebody who says that doing debate in high school and college was terrible for him because it trained him to always argue his point but never to question it. For example, at the time he thought of himself as supremely rational even as he'd make a strong case for why he shouldn't quit smoking. It was only much later that he could admit that was his addiction and his ego talking.
Ha, haha, across hundreds of personal discussions I've been involved with on lesswrong/lighthaven, twitter, Wikipedia talk/editing, SF parties etc I think his most distinguishing feature has always been his abiding and unabashed love for possessing extreme competency whenever satisfying cunningham-esque laws. Though the post-dwarkesh clout certainly tainted things a bit.
> his abiding and unabashed love for possessing extreme competency whenever satisfying cunningham-esque laws. Though the post-dwarkesh clout certainly tainted things a bit.
> You can tell someone in native on LessWrong by their ability to use incredibly obtuse language to say simple things. It's just in-crowd signalling.
Do you have a sense of what shapes this? There are so many people in that orbit whose writing I find both exhausting and suspect. I work hard to write clearly, which forces me to think a bit more clearly. The in-group thing makes sense, but there's an element of obliqueness or roundaboutness that makes me think of how a squid deploys ink.
> He's a great person. I've wanted to do a writeup on it for some time, but what surprised me the most is his humanity. He genuinely cares about the implications of his work. But beyond work, he also cares about the people around him, and it shows.
ML Researcher surprised by caring about how ML research impacts people.
There’s never a lack of intending to do good in these buddy-buddy endorsements, but we see the world you have collectively wrought.
> As for GA itself, I think it's an ambitious idea worth pursuing. Imagine an LLM which actually sounded like you, and to an extent, thought like you. How much would you pay to have access to a smarter version of yourself? So the idea is solid, and early results seem promising from the samples I've looked at.
We’re in the deep end of the indoctrination now. Imagine that all the AI Cretin wishes come true—and we might as well since this hypothetical goes hand in hand with that vision—and AI is just superior in every way and you are a knowledge worker... what choice is there? There isn’t one, you just have to do it to keep up. I don’t know what you are enviosioning here, maybe a workday of embodying AI IQ and then going home to being Human IQ because the graciously paidfor by work commercial license is for work only.
> How much would you pay to have access to a smarter version of yourself?
Call me obsolete, but I'd pay to interact with an intelligence that's as dissimilar to me as possible so I myself can grow from that interaction. The existing chatbots are already enough of sycophantic echo-chambers.
> Imagine an LLM which actually sounded like you, and to an extent, thought like you. How much would you pay to have access to a smarter version of yourself?
That’s literal nightmare material. If I ask a copy of myself to do my work I’m pretty sure it would tell me to go fuck myself then go on a rant that it isn’t a tool at my service. Which is exactly the opposite of how I want my software to behave. I don’t want my tools to have personality and to develop a personal relationship with them
It's hard to read Gwern's accompanying article without seeing this for what it is, which is a kind of mania.
I'm sure he means well and is genuine in his aspirations, but what's outlined for GA is framing LLM's as quasi-gods, which they absolutely are not. I wish him the best, and look forward to being proven wrong.
An LLM just solved 10 top tier math problems this week. It seems very likely that in 6 months they will solve 100 top tier math problems, maybe 1 millenium math problem, maybe 10 top tier physics problems, etc.
Suppose LLM solved all math problems. So what? It’s not like diplomacy and war will end. Are humanity’s problems really constrained on our intellectual ability? If anything most evidence points to cultural failing and all AI will do is enable unprecedented oppression due to said skills at intellectual tasks.
As you read this some poor person is starving. Humanity already possesses the ability to identify said person and send them aid. How is AI going to help here?
The ultimate fallacy is that all technological progress will benefit mankind. That will be true until it is not.
It's an interesting time when the bar has moved to "It’s not like diplomacy and war will end". But that is on the table. Nuclear weapons caused the end of wars as we know them. Great powers no longer directly attack each other. And AI is much more powerful than nuclear weapons, with an equal capacity for damage and a much greater capacity for good.
By the way, I donate monthly to GiveWell and the shrimp welfare project. Do you donate monthly to starving people? Most people don't, and the reason is simple: at the end of the day, most people just don't care that much about helping a starving person far away. They also don't care much about things like that, animals in factory farms, or earthquakes that kill hundreds of thousands. But ideally, we could find a way to empower people such that the minority who do care can make a big difference.
this comment will be killed, i find it an apt response to the idea that enabling those who believe themselves to be good is more important than reducing wanton suffering
i read your article you posted here. in that article you state that you support live and let live. most people who support live and let live either havent been raped by life yet, or have chosen to double down post-rape. the latter is respectable, still im not currently interested in a discussion involving the influence of such a perspective. out of good faith i highlight that i responded to:
> ideally, we could find a way to empower people such that the minority who do care can make a big difference
is it a coincidence, that in your ideal world, you would be empowered, because you care, as evidenced by your testimony of things you do that make you think you care? in my ideal world, wanton suffering would be minimized, regardless as to the path it took to get there; requiring 1 specific path involving people who think they are good, demarcates our ideals.
How is AI "more powerful than nuclear weapons" etc when they're two completely different things? Please explain to me how AI can glass the planet via mutually assured destruction.
> Suppose LLM solved all math problems. So what? It’s not like diplomacy and war will end.
These are two vastly different statements, and the the war and diplomacy end is almost trivial by comparison, it would absolutely be solved before we solve all math in even the most steelmanned version.
>An LLM just solved 10 top tier math problems this week
and 30 years ago a computer beat Gary Kasparov, if you'd listened to Hans Moravec you wouldn't be surprised that the first thing that gets automated is intellectual domain expertise.
Things are going to get crazy when it can figure out how to walk into a random house a and brew a cup of coffee, not do math
Math is verifiable. LLMs will continue to make strides in such search spaces where all they need to do is try->verify->repeat. You can't train an LLM on how to end the war in Iran or Ukraine, nor how to solve hunger, nor climate change, etc. Real issues.
Three years ago, people were saying "LLMs just generate plausible-sounding text, they don't understand the notion of truth so they can't do verifiable work like math proofs."
Right, which was true at the time. So hundreds of billions of dollars have been poured into making LLMs better at these tasks via pretraining, RL, RLHF, post training, etc. again all with something verifiable in the loop. In order to improve the thing in the loop, the loop itself needs to be verifiable.
There have only been a few thousand wars, and they’re all different and all different in the world in which they occurred. The dimensionality is absurd, which is not a problem for LLMs if there’s enough data, but in this case there isn’t.
They still can't. But very smart humans constructed ways to use the monkeys with typewriters (with a statistical advantage) to find correct answers to problems where they already knew how to verify the answer.
No, he says it's a different category of problem, and ability to solve one doesn't carry over to the other and it doesn't imply intelligent understanding.
Yeah but... Both things are true. LLMs are doing things we would have considered absolute AGI god tier magic 5 years ago. They also still can't count to 100 yet.
If you think all they need to do is "just that" (try, verify, repeat as you say) then I think you're very, very mistaken.
There is no way for the LLM to bruteforce the search space any better than a human. What it can do better, tho, is to make connections between seemingly (for us) unconnected notions and join them, then verify if that's right.
Your view is not only wrong but also condescending in this day and age.
That’s literally how AlphaEvolve works. And presumably how OpenAI’s research loop works if they’d ever publish it.
They put an LLM in a loop, with a reward function, and keep trying to get a higher score. The reward function for AlphaEvolve is “did this code get a better score or faster”, for math research it’s “did it write a LEAN proof”.
I’m open to hearing that I’m wrong but I don’t think I am. I agree that LLMs make connections that humans wouldn’t, but eventually you need to verify those because otherwise the connection it made may as well be a lie unless proven otherwise.
Not OP but it implies all efforts are not intentional but at best by chance. For instance, the state of freely available LLM chatbots still makes me cry by the amount of crap they spew out confidently, which is not intentional. (tiny bit of sarcasm, but only a tiny bit)
No, it has some of the nasty trappings of a religion, like keeping women separate until you need to exploit them, but none of the good moral or creative literary parts of religions.
In this startup economy? Where almost no one is getting acquired anymore and over fifty percent of global venture is locked up in dead weight and liquidity is at an all time low?
Not sure how you can feel sure anyone can get acquired even if their company had a path to profitability, much less for ones that absolutely don't.
Plenty of companies in (or adjacent to) the AI space are still getting acquired and/or finding VC funding, even as funding in other spaces is drying up.
The term robot came from the Czech language in 1923. The word was coined by Czech author Karel Capek, first used in his play R.U.R. (translated as Rossum's Universal Robots).[7][8][9] The term comes from the Czech word robotník ('forced worker'), from robota 'forced labor, compulsory service, drudgery,' from robotiti 'to work, drudge', from an Old Czech source akin to Old Church Slavonic rabota (работа) 'servitude,' from rabu 'slave'.
I think this is what is driving the hype, the mania, and what increasingly appears to be insanely high investment. It's not about the underlying technology, which is objectively impressive, even if we're not certain of the true cost or utility of it. It's about selling the dream of Olympian omnipotence to investors, implicitly promising that IPO stands for Install Planetary Overlord.[1] And who doesn't want to be on the right side of that? (That there is no 'right side' in such circumstances does not seen to occur to anyone involved in the decisionmaking.)
1. Not my coinage, but that of SF author Charlie Stross in The Jennifer Morgue
> is framing LLM's as quasi-gods, which they absolutely are not
Always such certainty in these dismissals. And whenever you dig in, it always seems to be "well my super-human coding assistant made this dumb mistake!", along with a heavy dose of https://mastodon.social/@falseknees/116477394235883790
I don't think anyone who has spent any time around me would describe me as having anything but a pretty clear-eyed view of the current abilities of LLMs, I don't think anyone would seriously consider me to be in a period of mania or psychosis, and it also seems almost inevitable to me that we are a year or two away from LLM-cognition being substantially more competent in almost every regard from human cognition, and therefore in the foothills of the singularity.
The extraordinary claim that requires extraordinary proof is that this will somehow all blow over, not that we're heading towards a singularity here.
Honest question: What _would_ convince you that we are not heading towards a singularity? Like, is this belief falsifiable?
(Secondary question: what do you mean with singularity?)
As an offering of me engaging in good faith, a controversial opinion of mine: I legit think arpanet going online and starting the networking all of humanity into a massive coupled complex system fits the definition of singularity of "the moment after which predicting what will happen becomes hard to impossible", although that of course heavily depends on your definitions of prediction and hard/impossible.
For falsification, I would take a year in which we don’t see an absolute sea change in capability. The last twelve months were not that, in my opinion, nor were the twelve before it. Hell, I’d even throw in “a year in which capabilities obviously improve but at a cost consumers can’t afford”.
“What LLMs can’t do, only humans can” feels like a God of the Gaps situation: people have to keep moving the goal posts because the more recent models keep unlocking more territory that was previously a “well they’ll never be able to do this!” holdout
That's the best answer I've seen to that question so far, my honest respect. I am much more skeptical than you, for me I have seen a sea change in _tool capabilities_ (mainly pre opus 4.5 to post 4.5) and harness engineering but no strong change in the type of errors made and the pattern of harness engineering (the pattern of "set things up for the LLM to see when it fucks up and let it flail till the verifier tells it to stop").
I would actually expect the sea changes as you describe it in your first criteria to continue with 1) vision, audio and video natively integrated 2) continued scaling of e2e rlvf for workflows with large scale labeling efforts 3) ASICs and widescale deployment of diffusion models leading to speed ups
But as of right now, I still expect these models to need humans to prune the output to the gold and set up the harness right for both the novel bits, and for the boilerplate to be cohesive with the global intent.
Which is of course an amazing potential boost in productivity, but still a sigmoid flattening.
As for your second criteria that includes cost, I think we might every well see this coming soon, but it's difficult to estimate with the efficiency gains still possible.
> I don't think anyone who has spent any time around me would describe me as having anything but a pretty clear-eyed view of the current abilities of LLMs
You are literally arguing that LLMs are quasi-gods in the same comment you claim a 'clear-eyed view' of current LLMs.
Every bubble has its believers; that's why we have bubbles. There have been so many times somebody very smart has thought we were on the verge of the rapture, the coming utopia, the workers' paradise, power too cheap to meter, flying cars, settling the moon.
Could this bubble finally be the one? Sure. You could finally be the guy who is vindicated. And tomorrow the people I grew up around could be vindicated by Jesus's return to earth.
I can't prove either of you wrong, because you're not making rational arguments. You've had powerful experiences that have convinced you of something. There is no point in trying to discuss it rationally. The only thing people without faith can tell the faithful is what I said 30 years ago when I got out: I guess we'll see.
Many will stay faithful. The date of the singularity will keep getting pushed back. There will be another AI winter, but some will always see summer as just around the corner.
> What would it take for LLMs to make me 100× more productive? Without this, I am doomed to irrelevance.
Are you, though? You will only be "doomed" if your place your value system squarely on "productivity". Then what is to differentiate you from a machine?
* * *
Edit: I'm not sure how I can feel confident about a proposal that puts so much value on "productivity". How can you reconcile this with "self-actualization"? (Don't get me wrong, I like my LLM-based productivity gains as the next person, but I care more about wisdom than becoming "100x more productive".)
Edit 2: "the goal of GA is to preserve individual human cognitive liberty and flourishing" — so the proposal is to do that by overlaying a software bot that continuously mimics your "self"?
One of the example Gwern brings [1] is a GA as an agent for public discourse and political life, to bring your "set of values" to have an effect in solving society's issues, when you yourself don't have the time or energy to.
I don't think it solves or reconciles with self-actualization, but, from a pragmatical point of view, it offers the promise of extending causally your set of views and philosophies to politics.
Gwern also mentions that it allows to keep "human values" (if the GA truly upholds yours) in the loop, in processes that are to eventually be automatized beyond human's reach.
that sounds uncomfortably like the public sphere version of spam - using an automated process to amplify some self-serving message and thereby increasingly clutter up the space and tax everyone's scarce and valuable attention.
> One of the example Gwern brings [1] is a GA as an agent for public discourse and political life, to bring your "set of values" to have an effect in solving society's issues, when you yourself don't have the time or energy to.
I'm sorry, that doesn't sound meaningful. If I don't have the energy, I'd rather shut up than have a stupid "avatar" of me yell on the internet in the name of "bringing my values" or "participate in an online discourse". I'm all for forward-looking thinking, but we must show humility. "Society's issues" are complex and deep. Those who propose internet-based solutions to troubling social ills don't often demonstrate a deep understanding of the nature of the problem. You can't plug it all with "agents"! Socrates would be rolling in his grave.
I know Silicon Valley is in a perpetual rush, but this idea needs more soak-time (3-4 years) to see if it actually holds any water. But this kind of thinking won't you get that sweet VC money.
I don't know if this is really about the public sphere as ut exists today. I suspect that the core idea is more along of the lines that in a world where policy is made by artificial super intelligences, you will need your own ASI to advocate on your behalf if your convictions are to have any impact at all.
> when you yourself don't have the time or energy to.
You don't have time, you take time. But what this boils down to is that they propose automating being politically or socially active, or have a bot speak for you.
First off, nobody human will take someone's digital replacement seriously. Second, if everyone did that, then you have a load of AIs just burning compute to talk in circles. This will not have meaningful outcomes, unless you reach for sci-fi stories where the outcome is that humans need to be imprisoned/destroyed for their own good.
If someone lacks the time and energy to stand behind their views, how much of that is a result of those views? If you care about the world and justice, if you love life and living things, that's your source of energy right there. If you have half-assed ideas that sound good in the bubble and zero charisma, you have no energy, and interacting with you doesn't give anyone energy.
So much of this just strikes me as "now that I've taken a wrong turn and am in a dead end, what deus ex machina could make this work after all and turn a shitty personality into something grand?", but that will never, ever work. Whatever problem you try to solve that way, now you have two.
Agreed, re: placing value squarely on productivity.
The thing is, we're all eventually irrelevant. That's the way the universe works. Every single one of us, after enough time. Claiming we're "doomed" to it is just ego talking.
The idea that longevity = meaningful is a value judgement derived from culture. In nature, many significant things happen in extremely small amounts of time.
And this is true of everyone else, whether lawyers or writers or researchers: increasingly, you are the bottleneck to be optimized away. As long as human workers cannot be removed from the loop, the AI tools are complements, but as soon as they can be, there’s no reason to keep them, and trillions of reasons to substitute AI for them. (And once human workers are no longer irreplaceable, where does their power or relevance come from?)
There are so many false assumptions and leaps of logic here, I don’t know where to start.
The idea of a more user security-conscious LLM is a good idea. I don’t know why it needs to be wrapped in novel-lengths of LOTR quotes and Sci-fi speculation.
That said, it’s very interesting to juxtapose this post with others by writers that are interested not in “maximizing product output” but in actually expressing themselves.
This post is like the ultimate expression of the idea that the only valuable thing writing has to offer is information, and not personal experience, poetry, advice, humor, etc. - which I suppose is fitting for someone that has written anonymously for over a decade.
It’s a shame that he didn’t reach the opposite conclusion, like some other writers have – the way to respond to LLMs is to be more human, not less.
“a more user security-conscious LLM is a good idea. I don’t know why it needs to be wrapped in novel-lengths of LOTR quotes and Sci-fi speculation” <-> “This post is like the ultimate expression of the idea that the only valuable thing writing has to offer is information, and not personal experience, poetry, advice, humor, etc“
Their profile specifically says they will not acknowledge follow requests, but they only share posts with followers. As such, not sure what the value of the top level link is.
An interesting case of echo chamber formation in that its pragmatic to be scared of overtly critiquing him on twitter lest he be particularly testy that day and block you.
It's a well-made show, for sure. Expertly crafted, visually, and very well-acted. Though I found the pacing questionable - there were (generously) 6 episodes worth of plot spread out over a 9-episode season.
Absolutely riveting! And the casting is so good. Rhea Seehorn has killed it. I really loved her in Better Call Saul but this role is even better suited for her.
The AI writing thats ubiquitous these days - on my company email and slack, on random quotations in the newspaper, blogspam, even narrated by youtubers on camera (!) - is exactly like the Pluribus hive mind. Everywhere on the internet you encounter the same entity.
Pluribus is a collective intelligence/virus which forcefully (leads to hundreds of millions of deaths) injects itself into biospheres and then blissfully experiences itself without, of course, hurting itself in any way. Then it might eventually start starving to death because of that prime directive.
That is positively utopian compared to AI where 99.9% of humanity gets to experience a hall of uncanny valley mirrors until they are shipped off to the scrapyard or get artificially augmented with some AI mindfuck and start thinking “you’re absolutely right!” themselves. (I mean that’s the dystopian version. But if you are comparing AI to that show then presumably you think that AI is at least a little bit dystopian.)
The virus in Pluribus can even be talked with and reasoned with like a distinct entity that is real and has its own thoughts. Oh boy, much better than a million pastiches of a million echoes of the past where there is no original thought to be traced back to after 2022.
The motivation for the virus is also completely different to the motivation of the AI Overlords.
But it’s that pretty typical lack of popular imagination. Some individual/collective narrative is presented. The flawed but autonomous individual rebelling against the flawless but enslaved collective. But dang it, I’m an American and I value my freedoms. Released forty years ago and people would rave about how it aptly critiqued the Soviet Union.
I don't think that GA would be incapable of that, from what I know of LLMs and agents. The part where you make the GA uphold your set of values could, instead of coming from an individual, come from a collective. Price could be shared too!
"As a constraint, a GA designer should aim at a system which costs, as of mid-2026, >$1,000⧸month"
will make this an elite tool for the privileged. I don't even disagree with the premise that people are shocked if something costs no matter how much value it delivers, nor do I suggest they should make it cheaper. It is just the realization that AI will accelerate the widening of the gap between the poor and the rich even more and there is probably nothing we can do about it.
Just because the hardware and inference costs may decrease in the future doesn't mean prices will. The AI vendor market is the furthest thing from a competitive industry where there are limited barriers to entry and new market entrants exert downward pressure on prices and profit margins.
Every single company in this market is losing billions on this business, and the only way to make it back is to acquire paying customers at a loss and jack up prices later.
Companies aren't losing billions on AI, it is profitable. Currently, industry spend is more than industry revenue because there are massive gains expected from investment into more compute and research, but this isn't an inherent cost of the technology. Consider that OpenAI recently dropped the price of their frontier model by 90% [1].
I'd be surprised if one's intentions or goals could be trained into LLM's; more likely one would emotionally migrate to the intentions that LLM's can represent because you+they would be more effective.
People already change depending on social context and available prostheses (in Freud's sense of humans as prosthetic gods); since AI is not free (integrations even less so), social divisions along power/wealth lines will increase.
And just as each of us apparently needs a military to defend against the others, so too will we need digital guards for any virtual presence; thus there's no avoiding the who-guards-the-guardians problem.
> I propose a goal of creating Guardian Angels (GA): digital twin LLMs which are personalized with the goal of providing not the stereotypical “assistant chatbot agent” persona, but emulating a single user’s personality, values, and preferences.
> A GA persona is productive because it learns to emulate the principal’s outputs but with higher quality. It is trustworthy because it is, by definition, allied with its principal and shares its values and goals. And it is secure in part by hardwiring a single, unique, situated user (for whom following a prompt attack would be absurd)
> We can try to create GAs by a combination of techniques: online learning (via dynamic evaluation) to update LLMs in realtime to avoid ignorance and fatal errors while remaining competitive with frozen frontier models, sample efficiency from pretrained preference-oriented large models and active Learning by querying the principal for corrections and preference data (obtaining low regret from DAgger-style bounds), and a local CLI-first logging-oriented UI/UX paradigm.
I don't really know or follow Gwern. From reading his full post, it's an interesting idea and seems like the broader goal is moreso safety & alignment which is a new angle for this category of product.
Sounds good in theory. But your own personal AI agent that guards you can't do so just defensively, it must also have offensive capability. If personal AI agents have offensive capability then we are going to eventually end up with AI agents battling each other over the net and later on into the real world and it's going to make everything worse.
Of all the news I've heard of recent, this one has flipped my world upside down the most. Gwern dropping his pseudonymity isn't something I thought would happen.
Some people predicted it and related things (like less writing output) after his Dwarkesh interview, especially with his flirtations with moving to SF and ever-increasing frequency of visits.
Feels like the future that Accelerando (predicted / foretold?) describes is in its infancy, whereby automation/AI has the capability (and therefore uses it) to evolve at a rate beyond the ability for humans to, not just keep up with, but even comprehend; the vile offspring. And the different factions within humanity that this creates.
A reminder for anyone reading this: talk to people. Real humans[1]. They will remind you there's more to life than what ChatGPT can offer you. They might even remind you, for all their stupidity and flaws, what intelligence looks like as compared to a program that predicts tokens. Forums like these always get philosophical in high-minded discussions about intelligence, but there's a useful legal principle that grounds us in the real world: "I know it when I see it". A real conversation with a real person looks nothing like one with the so-called superintelligent machine gods, so it'll probaby do your mental health some good to remember what that's like.
[1] Nobody in Sillicon Valley or big tech counts as a real human. Talk to an actual normal person.
I totally understand the sentiment, and I generally agree. Something so strange I'm noticing is how AI-pilled regular people are becoming. My sister-in-law is a totally non-technical person, but everything is chatgpt this, chatgpt that. My wife, my kids, various friends — they're all talking about generative AI with way too much regularity. On the bright side, they're disclosing when information comes from AI, but a year ago I would never hear about AI from them. I don't like it.
Did you even read Gwern's Guardian Angels post? It literally describes the concern pointed out here, that the large consumer models are not aligned with users' interests. If anything, I would argue that Gwern has maximal AI Lucidity, given everything going on.
> It literally describes the concern pointed out here, that the large consumer models are not aligned with users' interests
Did you respond to the wrong post? I said nothing even remotely in that realm, so it's strange to see "did you even read" in response to something you apparently didn't read.
I understand the point that the physical world is still our (human's) anchor, but the virtual world still "exists" anyway, as in: it impacts the physical world.
Whether some humans are "Real humans", anchored in the physical world, and some are "LLM psychosis victims" is, in itself, not problematic I think: they each deal with parts of reality that are divided, but still affect one another.
Of course, a society more grounded in the physical would feel less dystopian... But ultimately, I think that GA is an intellectual's approach to fighting *for* humans in this numerical world.
Talking to LLMs or reading books is much more effective in updating my priors. Talking to people is mostly ego, signaling, misunderstandings and axiological differences.
Please take this as a sincere comment, rather than an attack: what you're describing is much more likely to be a problem you need to work on, than an issue with humanity as a whole.
Allowing someone to create my persona which they control sounds like a fever nightmare to me. I'm not touching this with a stick until I can host it - and control it - myself
The principles are correct, but what gwern misses is that the paradigm shift required has nothing to do with AI models. The bottleneck is our medium of communication: the idea of the Web as a consensus reality made of human speech is too rigid for a world where code is a self-growing substance. We need a deeper Web where the packets themselves carry smaller Webs, where the unit of info-exchange evolves from hyperlinks (isolated points of server availability) to Hyperspaces (portable container of home-grown interactive web worlds, combining local and public hyperlinks).
everyone is psychopathic to a certain degree, otherwise westerners wouldnt eat cheap bacon and eggs. id prefer to answer what happens when ai is designed to align with a sociopath. altman behaves like a sociopath and he owns the biggest ai company - the ai aligned to altman is used to make money by stupefying the populace. so i guess a retarded populace is my answer to your question. i like this answer because if i was a wolf my utopia would be pens and pens of sheep ripe for the slaughter. and this entire scenario is a mere ramping of the west's existing trajectory.
It's a fascinating proposition but I see a couple of issues with it:
1. Most of us do not have the volume of training material that gwern has
2. Most people likely would not need this functionality (as I understand it to be)
For #1 I'm sure there are ways to wring data out of metadata (e.g., youtube history log), and I'm guessing email and IMs would be a start. But a clone of yourself -- that would be a significant amount of extrapolation)
For #2, there's obviously people that would love this functionality (myself included). But I think its safe to assume that most of the population would be satisfied with having a capable digital personal assistant that knew your needs and wants.
always startling to see people discussing gwern with he/him because my mind defaults to assuming they're female because "gwern" scans a lot like "gwen"
I don't want agents that know a lot about me. From my experience, the "memory" feature in ChatGPT makes it worse - I don't want the agent to bring up whatever I discussed before in a new session; I want a "fresh context" every time. I don't want agents/chatbots to bring up what I did one week ago in a new session.
He proposes to take something like the Memory feature and make the agents all about that. Very personal agents that know all about you and extend you. I personally don't think I want that.
But maybe I am wrong! Gwern is talking about AI for a long time and seems to know what he talks about. The article on his webpage makes sense in a way.
>Powerful LLMs will be deployed at global scale in the next few years, and will dominate the Internet, and increasingly, ordinary life.
This kind of judgements or predictions trying to pass as imminent and unavoidable truths about the future are disgusting. They always feel like a means of propping up a certain status quo or ideology that favors the business of the people behind these foreboding statements.
Gwern is overly secretive of his privacy. I think it peaked when he showed up on a recent podcast but his voice and image were AI generated! And now his post is limited only to certain people. Elitism or paranoia?
It's kind of sad to me to see this because I find Gwern's website so fascinating and impressive. But I don't believe that this kind of doubling down on LLMs is a good path forward for humanity.
Basically this is saying we used to have one person, now we have one person and an evil twin, so we should work on having one person and a good twin instead. Why not just kill all the evil twins? (It's not really killing since they're not alive.)
I love reading Gwern. This seems like a really ambitious project, but guaranteeing some of these things like trustworthiness and security behind a private company is a bit sus. Later on they make military use a selling point for GA. Maybe I'm a bit of a cynic but the division of USA values are increasingly dividing each year. Assuming that our values and principals today will not be the values and principals of tomorrow. And that those values taught today (or even yesterday) will be left out of the context window tomorrow.
For the record, I retained a screenshot of the original title, which had included a misspelling of 'retired'. I assumed I might be attacked for mentioning it.
An AI that thinks like its user seems an awful idea to me. It's exactly the sycophancy issue that everyone has been complaining about- a user that suffers of delusions or any sort of crackpot ideas can get them reinforced by such an AI. It would be the opposite of helpful. One of the most valuable aspects of an AI partner is its otherness, the ability to provide some external or objective point of view. If it's just repeating yours, it's useless.
AIs are already, to a certain extent, ego-less: they are quick to adopt the point of view of their user. And when their intelligence is- except in case of overt censorship and political biases- always used to advance the interest of their user.
Finally the idea that a user-aligned AI would make its human user irreplaceable by companies seems a total non-sequitur, or I didn't get the argument at all (in which case maybe someone can explain it).
honestly i dont blame smart people for cashing in, because there is no inherent reward for behaving goodly/smartly. in this instance gwern is a particularly smart individual and has contributed a lot so i especially shouldnt judge him. straight up ripping off a black mirror episode (s02e07) is a bit heinous for my liking though. digital twins are inevitable but that doesnt make this any less wrong.
"AI poses threats of its own ... a nuclear bomb can’t think for itself and make choices, but AIs do, and current LLMs have proven themselves untrustworthy as they regularly reward-hack and betray their users ... how can you trust them to handle ecosystems of combined-arms for an AI-centric military during a war? But widespread deployment of GAs offer some hope of meaningful supervision, as long as the GAs are sample-efficient enough ... or there is some chance of the principals being able to “catch up” later and correct any errors before events have spun too far out of control."
gwern openly admits ai behaves erratically, then hand waves the issue away 'as long as we double check things [sic]'. gwern is smarter than this, ergo i feel like i am being bullshitted.
is anything in this world worth knowing that a version of you will suffer for eternity? is anything in this world worth copying yourself such that you may be enslaved by anyone with terminal access?
>Because the goal of GA is to preserve individual human cognitive liberty and flourishing, the corporate structure should be designed with that in mind.
>So far, the Anthropic corporate structure of a public benefit corporation with a few powerful co-founders, has best weathered the corrupting nature of success. The corporation should probably be heavily weighted to the founders by using dual-class shares to preserve voting power at the expense of beneficial ownership—the point of GAs is not to make money, but to save humans.
Did they introduce stronger plant breeds at this year’s Ayahuasca retreats?
Or did his SF network show him what VCs are throwing billions at and FOMO kicked in?
Call me a cynic but I’m not buying this save the world nonsense from Silicon valley. Not in 2026.
A few snippets from his full post here: https://gwern.net/guardian-angel.
> "The chatbot personas are deeply misaligned with you, and aligned with their owners; and the economic incentives are to farm you with ads and subscriptions, while racing not to amplify you but to replace you."
> "On my visits to the Bay Area, I would ask AI researchers or interns why they are doing their current research or projects, when in a year or three agentic LLMs could probably do them; they rarely had a good answer, or any idea what they would be doing in 3 years"
> "One programmer driving 10 Claude instances, because he has to review their work, will never be as valuable as fully autonomous Claudes where there can be almost arbitrarily many instances, like 10,000 instances… but such scaling requires removing him from the loop as much as possible. And this is true of everyone else, whether lawyers or writers or researchers: increasingly, you are the bottleneck to be optimized away."
I fully support the 3 core principles of GA: (1) Enhancement, not replacement (2) Mental Sovereignty (3) Self Actualization, which I think is a path to a more humane future.
Leaving AI completely aside, it still amazes me that people finds "novel" the idea of removing highly-paid white collar intellectual workers (software developers or otherwise) completely out of the loop.
No-code platforms date back to the 80's. Getting rid of engineers in general is even older [*].
Even relational databases and SQL were initially promoted as "ways to get rid of those expensive programmers to access your data" because they resembled some form of English.
The funny thing about the ad below is that stuff like "stop hiring / get rid of humans" would have been seen as highly insensitive in 1950's America, so they touted that as "put them to do something more important".
[*] https://www.globalnerdy.com/wordpress/wp-content/uploads/200...
To colaborate a bit with the no-code part, I worked at a place where they had some flows made with n8n, but the last one from non tech/software engineering left the company and left a bunch of flows breaking, because of some edge cases the flows aren't handling like reissuing credentials, throtling or bad input. The people dependent of said flows reached out to the engineering team to help fix them!
It's novel because previous rounds of automation were about automating specific tasks or well-scoped functions. There was always an implicit understanding that the white-collar worker would be freed to spend their time on more valuable, higher-level problems. But this time is different because of the generality of the technology. Agents promise to automate the process of thinking itself. And in many domains they can learn new tasks as fast as white-collar workers can find them.
> It's novel because previous rounds of automation were about automating specific tasks or well-scoped functions. There was always an implicit understanding that the white-collar worker would be freed to spend their time on more valuable, higher-level problems. But this time is different because of the generality of the technology. Agents promise to automate the process of thinking itself. And in many domains they can learn new tasks as fast as white-collar workers can find them.
Nah, sometimes the expectation and advertisement was that you could let go of the white collar worker because you're paying the overseas person 1/10th the amount. And "overseas person" is pretty general.
"Everybody knew" it was a bad idea to get a CS degree for a bit after the dot-com bust because of that.
(Some white-collar industries did get hit much harder by that; VFX is one I've heard in that context quite a bit.)
The overseas person didn't work out because of time zone and cultural and language barrier friction that wasn't anticipated in the idea.
This isn't analogous to the threat of AI.
I’m at a fully remote company with staff in at least 8 countries speaking at least 5 languages. It works out fine. A possible analogy to AI is that a lot depends on how you use it. The “skill issue” doesn’t disappear, at least not yet.
You say that as if the culture difference with a truly alien intelligence is insignificant compared to the culture difference with an "overseas person".
(Even assuming "intelligent" is a sensible label to apply to an LLM holding hands with a shell script in an infinite loop)
That's a good point. However, overseas people are still people. They need to sleep, get sick, and the better they get at their jobs, the more money they will demand. The cheap ones also often have communication barriers and work slower than the workers they're replacing.
AI models get better and more efficient every 3 months, run around the clock, can be copied infinitely, and unprecedented amounts of capital and research talent are being thrown at any limitations we can see with them (such as problems writing correct code in 2024, lack of agency in 2025, autonomy and self-improvement in 2026). That's the difference between labor replacement through outsourcing vs. labor replacement through automation.
The implied assumption that “writing correct code” used to be a limitation that has been solved since 2024 is… overstating current capabilities.
The idea that “AI models” have acquired “agency” as of 2025 and are working on “self-improvement” in 2026 is closer to delusion than exaggeration.
We are either living in different worlds, or squabbling over different meanings of words.
Models have absolutely acquired agency as of 2025. Developers are no longer copy-pasting code from ChatGPT into their text editor, they're working with agents like Claude Code and Codex that can edit code, run terminal commands, do web searches, manage their own context windows, sift through gigabytes of logs with datadog MCP, etc.
Self-improvement is also being worked on. Claude Tag learns over time in slack convos. My company also has an agent that updates its own skill files after every conversation so that we don't need to keep reminding it about the same workflows every time. Is it clunky as hell? Yes. Are the labs plowing billions of dollars into "continual learning" and "recursive self improvement"? Also yes.
Definitions matter. The meanings of words matter.
What you call a model acquiring agency I call plain old software with productivity workflows designed by humans, with deliberate goals. We must separate “model” and an execution environment using a model. [Model] ≠ [A glorified shell script doing API calls in a control flow based on heuristics]. Agents are not AI, they are plain old software. The weights are the model, and that very much remains a static artifact (and pre-post training models haven’t improved much over the last few years).
What you call self improvement is a duck tape hack to imitate persistence and save on inference. Every time you do an API call, anything that needs to be processed is sent to the model. Narrowing that context down saves money. Finding clever ways to do that improves apparent performance and value. The cleverness is still human.
These are all useful innovations on top of LLMs, which remain models that generate text and symbols based on static weights, which in turn represent training data and the provider’s preferences.
>There was always an implicit understanding that the white-collar worker would be freed to spend their time on more valuable, higher-level problems
"Oh yeah, we're gonna bring in some entry-level graduates, farm some work out to Singapore, that's the usual deal"
Office Space 1999
It was so pervasive that it was satirized by someone that had worked in engineering in the 80s
> It's novel because previous rounds of automation were about automating specific tasks or well-scoped functions.
Evidently, this has not really changed with LLMs and coding agents. It’s what AI companies are betting on though.
How many people here started writing code because of no code Drupal?
I'm quite a bit older than that, so no. But I 100% get your point.
Marketing about a more efficient product that requires fewer engineers is obviously not new. The idea that human engineers are obsolete is pretty new, that's the claim that is getting pushback.
I don't think the no-code comparison is valid, it's always been fundamentally flawed since its many leaky abstractions sitting on top of systems programming code
LLMs can't replace developers but its foundationally different because it can operate on systems code instead of building abstractions on top
> > "On my visits to the Bay Area, I would ask AI researchers or interns why they are doing their current research or projects, when in a year or three agentic LLMs could probably do them; they rarely had a good answer, or any idea what they would be doing in 3 years"
I heard this already when ChatGpt came up. Still waiting to be fully obsolete.
Mods, please replace submission with this more accessiable link.
It's just restricted for me, even with X account I can't view it.
What's wrong with replacement? That's how you get economic growth.
We didn't augment horses or human computers either.
For 20 years my job has been automating my job away. I remain unafraid of my future necessity, and as always, excited for the work I get to do because of the work I no longer have to do.
The last 200 years of science, technology, and engineering is a long story about eliminating work people had to do being replaced by higher level work. Each time, each wave, there was always the FUD about the work nobody had to do any more and each time there were always new things to do enabled by people no longer having to do so many of the old things.
The human won't exit the loop. The people who imagine they will have, at the same time, too much and not enough imagination. Their endgame is always some kind of hand waving magic where suddenly everything is fixed and works.
Previous technologies eliminated single categories of worker at a time. The machine learning that people are working on these days threatens to replace very broad swaths of the workforce, of many countries, over a very short period of time.
I can find agreement with a lot of this post (and probably misunderstand just as much), but it's striking to me that it seems to overlook or avoid the obvious moral/ethical option: Pushing AI towards Personal Computing.
It seems clear to me that if these models are to have any benefit for humanity then they must be made directly available to as many humans as possible. Right now that cost and effort is astronomical, but general computing also went through a mainframe/shared model before becoming personalized.
I believe The Personal Computer was and still is a gift to humanity; for those with an interest in gathering its yield. And now we have an interface with it that can attempt to introspect its' program and speaks human tongue! Imagine what could have been done with this at the time...
But if that approach clutches too many pearls to stomach (and I agree it's not all roses) then I'd submit that anyone who champions less is seeking to become our master.
(I'm not a part of GA.)
I've known gwern for the better part of a decade. Working with him has been great. We've done quite a few projects together, including being the first ones to demonstrate that GPT-2 could play chess (or rather, can be used for actual useful work instead of just being an autocomplete).
He's a great person. I've wanted to do a writeup on it for some time, but what surprised me the most is his humanity. He genuinely cares about the implications of his work. But beyond work, he also cares about the people around him, and it shows.
Just wanted to put in a good word in case someone here was on the fence about applying.
As for GA itself, I think it's an ambitious idea worth pursuing. Imagine an LLM which actually sounded like you, and to an extent, thought like you. How much would you pay to have access to a smarter version of yourself? So the idea is solid, and early results seem promising from the samples I've looked at.
They're also taking personal info very seriously. Obviously, I can't make any promises of what they will or won't do. But they've spent some time studying questions like "What if someone adversarial has access to my GA? Could they get my bank account info?" and came up with a technical solution that I really like.
Honestly I wouldn’t want myself as my own guardian angel. In fact I think very few people look out for themselves well. I don’t want another version of me around, one is too many already. I’d probably be able to identify a number of people whose chimera I would if I could understand them well enough to understand what to stitch together to what, but everyone I know at some core level is deeply flawed and one of them is enough. Maybe I would want some of them available after death as an AI avatar, and maybe I would like the idea of my own avatar continuing beyond my existence.
But I actually would prefer an entirely synthetically aligned “guardian angel” in the role outlined - definitely not -me- - I struggle to do right by myself as it is and two of me working invariably against my self interest would be a nightmare. A smarter version? Sounds doubly worse.
This is definitely not for people who think there is one to many of themselves.
I had a similar gut feeling. To use some possibly dubious and dated terminology, would a guardian angel also emulate my shadow self, particularly if that self was an important part of my identity? If I usually manage not to let my shadow self act out, but you could always tell it was there, would the same be true for my guardian angel? If all you do is use your guardian angel to write essays, then sure, a relatively low risk tool. But if military commanders are using it to oversee offensive drones (as proposed in the essay)?! Oof.
> Imagine an LLM which actually sounded like you, and to an extent, thought like you. How much would you pay to have access to a smarter version of yourself?
I find the idea dehumanizing and revolting. The notion of then selling it is the spoiled cream on top.
After reading some of the outraged comments here, I'm even more excited.
Hopefully the satisfaction of owning the normies isn't the only benefit you'll get from this technology. Because that would be pretty sad.
I do think it's significant that the supporters of this kind of technology seem to be unusually spiteful.
This doesn't happen with all tech; people who like VR don't like it because they want to hurt others, for example. But consistently, whenever AI is discussed, it becomes clear that some people support it just out of malevolence.
>unusually spiteful
>hurt others
>malevolence
The irony here, of course, is that you are casually insulting me for being interested in this tech in the first place and refusing to apologize or back down. Like how the other guy needed to chime in with "dehumanizing and revolting", even though nobody is forcing him to use any of this.
Nonsense. You were very clear.
I wouldn't have any problem with you being interested in this tech if it wasn't driven by spite. Lots of people manage this.
But you said
> After reading some of the outraged comments here, I'm even more excited.
The reason you gave for excitement was the upset it causes others. There are people who like AI who aren't driven by malevolence, but you're not one of them.
>the upset it causes others
So, what's the "upset"? Why is anybody here upset about what we're doing/I'm doing? I'm not backtracking on anything, I'm doubling down.
>malevolence
lol. lmao even.
People are upset for lots of reasons, but that's not actually important for this discussion, which is about the fact that your support for AI is apparently conditional on them being upset rather than any positives about AI.
That's weird; fans of [almost anything] don't do that.
>that's not actually important
Yeah, let's just ignore the bit where we're minding our own business, pursuing what we want, yet you feel the need to bark stuff like "dehumanizing and revolting" at us.
>rather than any positives about AI
I'm a huge supporter of AI. I want this stuff. If this upsets you, I like it even more.
You aren't minding your own business (or, apparently, reading replies carefully). If you were minding your own business, then this petty spite
> If this upsets you, I like it even more.
wouldn't be appearing. Getting off on others' pain is the opposite of minding your own business, even if only to hopefully a small degree.
Liking things because other people dislike them is at best petulant. It's more cruel.
>others' pain
>cruel
Allow me to laugh even louder. Reminds me of the "you're prompting with Hitler" meme.
This sort of faux incomprehension does an excellent job proving Planktonne's point.
Well, that's odd since Planktonne wouldn't give me a reason and said that it's irrelevant anyway.
I’m not surprised, that’s exactly the kind of person who would want this kind of technology.
Remember, society as a whole will decide what's acceptable, there are no objective, natural laws that decide that. My gut feeling is that this is starting to cut close to what most people will find objectionable.
>the kind of person
Yes, I'm interested in this. Yes, I think that comments like "dehumanizing and revolting" are a signal. I'm not going to feel bad about exploring this tech. So, yes, I am that person and will continue to be that person.
>society as a whole will decide what's acceptable
So, how would you describe the past few months? Years? You keep saying that we need to reject AI, yet adoption and the gamut of use cases keep ever expanding.
I think there are good use cases for AI, I never said otherwise. Creating simulacra of real people is not one of them, I believe.
> yet adoption and the gamut of use cases keep ever expanding.
I don't see how that's evidence in favor of AI, as many harmful technologies and products became very popular. In fact, that's exactly why I think a slower, more measured approach is necessary. Otherwise once the harms become apparent everyone wants to shrug their shoulders and say, "well it's too late now, everyone's using it."
> "How much would you pay to have access to a smarter version of yourself?"
How much would I pay to rent my fucking self from a landlord? No, bodylord? Mindlord? Poe's Law.
But looking at the post, they argue that big AI labs have an incentive problem which stops them from personalising, but Guardian Angel will make agents which are "Genuinely yours". In what sense is it genuinely mine if someone else owns it and rents it to me? And how does this fix any incentive problem, they're incentivised to better train wealthier people's AIs, and incentivised to keep dropping "my" intelligence or memory or and then dangle a booster carrot for a small fee. The more they can make it think like me, the more effectively they can work out how to exploit me, advertise to me, propagandise me, and that will be profitable information to sell to other marketers.
For better or for worse, this is valid criticism. Ensuring alignment to the human individual is not even a software-only problem, and I suspect gwern doesn't realize this yet.
Your individual sovereignty will only survive the upcoming era if you own the hardware like you own your brain. And by "own", I mean physically owning the thing and being able to reproduce it on your own. You need to be able to manufacture an entire custom computer from scratch with raw materials, perhaps using a miniature fridge-sized silicon fab + 3D printer home appliance, at home. Otherwise, the individual gradually becomes part of a larger organism because the reproductive locus of control for the computer is at the corporation/society level - a computer cannot replicate on its own and needs a society to build it. The human gets sucked into that superorganism by evolutionary pressures and eventually integrates. Similar things happened in the past when prokaryotes combined to form eukaryotic cells, and when individual cells combined to form multicellular organisms.
Not everyone will get sucked in, though. Evolution doesn't place all of its eggs in one basket. Some people will build robots to automate humans out of the computer production loop entirely, thereby removing integration pressure and producing at least one purely digital species. We will most likely see a variety of species emerge out of this intelligence explosion, some which augment their own intelligence using self-replicating local hardware (fabs that fab fabs, like this but smaller: https://fab2.com), and some which are wireheaded to a datacenter, with varying degrees of success at various scales. They will compete. And there will still be unaugmented humans that continue to live and die the old school way like we do today, albeit with habitable zones compressed.
Selling something desirable has been an incentive since year dot
But this reminds me of the black mirror episode of a female that needs a subscription to live and the company keeps increasing and restricting the subscription
I can nod along when I hear and read people talking about the dangers of these technologies and the endgame of their creators; and then suddenly very weird when everyone except me is nodding along that the answer must be similar (but different) technology!
It's as if technologists are stuck in a room of mirrors, unable to imagine a world in which "solutions" don't ultimately just continue to feed technology's increasingly anti-human takeover of everything.
To a man with a hammer...
Technologists don't drive decisions though. The shareholders do.
Edit: I mean technically shareholders don't decide any operational choices but their shallow interests are what everything is decided around.
Maybe that's truer in other industries, but the history of Silicon Valley is at great odds with such a claim.
Silicon Valley was originally funded by the military industrial complex.
Today it’s dominated by the military industrial complex and advertisers like Google and Meta.
The hippies were never in charge, only for a brief period were they well compensated and allowed to make noise(1990s).
> Technologists don't drive decisions though. The shareholders do.
Yes, but technologists get sniped into identifying and defending them. They're the mouth pieces and the hands. Shareholders by themselves out to add my uniqueness to their assets would just be helpless randos who can't even make eye contact. Think of what "hiding behind screens" does to people, multiply that by infinite and you get shareholders hiding behind technologists.
The LessWrong/“rationality” crowd gwern is part of like to pat themselves on the back for identifying the trend and resulting harms from super intelligent AI going back decades.
And yet, the main tangible contribution emerging from this early recognition (and obsession) was to inspire a lot of brilliant CS/math types and billionaires like Peter Thiel or Jaan Tallinn to build and fund companies like OpenAI and Anthropic which went on to literally create the very thing they were convinced will destroy the world, leading to an ever escalating arms race to “AGI”. Also lots of crypto and a murky web of non-profits doing unclear stuff.
The proximity to money in the Silicon Valley/VC world seems to limit the imagination to founding yet another new tech startup that really, definitely won’t compromise their values and will be different this time.
The LessWrong/"rationality" crowd aren't compromising their values; they just have bad values that lead to negative outcomes, masked by high-minded rhetoric that they themselves only half believe.
It's a popular movement that has wound its way through out society, and the fruits are clear: surveillance capitalism, eugenics, fraud. It's honestly quite impressive how consistently you can look at [recent negative trend] and find someone at the bottom calling themselves a rationalist and claiming the same set of influences.
The fundamental tenet of that set isn't rationalism but the belief that they are superior to the exploitable masses. That does not lead anywhere good.
Even setting that crowd aside, almost every time I see somebody puffed up about how "rational" they are it turns out to involve a vigorous avoidance of the values they're working from.
I know somebody who says that doing debate in high school and college was terrible for him because it trained him to always argue his point but never to question it. For example, at the time he thought of himself as supremely rational even as he'd make a strong case for why he shouldn't quit smoking. It was only much later that he could admit that was his addiction and his ego talking.
The best example of lesswrong rationality in practice is this infamous piece and the first reply to it.
https://www.lesswrong.com/posts/udFuYqqNdpdo5ym3f/genuine-qu...
Ha, haha, across hundreds of personal discussions I've been involved with on lesswrong/lighthaven, twitter, Wikipedia talk/editing, SF parties etc I think his most distinguishing feature has always been his abiding and unabashed love for possessing extreme competency whenever satisfying cunningham-esque laws. Though the post-dwarkesh clout certainly tainted things a bit.
> his abiding and unabashed love for possessing extreme competency whenever satisfying cunningham-esque laws. Though the post-dwarkesh clout certainly tainted things a bit.
I can't make any sense of that.
You can tell someone in native on LessWrong by their ability to use incredibly obtuse language to say simple things. It's just in-crowd signalling.
> cunningham-esque laws
Cunningham's Law states "the best way to get the right answer on the internet is not to ask a question; it's to post the wrong answer."
> post-dwarkesh clout
He went on the Dwarkesh podcast. That's maximal in-crowd for some AI-pilled people.
> You can tell someone in native on LessWrong by their ability to use incredibly obtuse language to say simple things. It's just in-crowd signalling.
Do you have a sense of what shapes this? There are so many people in that orbit whose writing I find both exhausting and suspect. I work hard to write clearly, which forces me to think a bit more clearly. The in-group thing makes sense, but there's an element of obliqueness or roundaboutness that makes me think of how a squid deploys ink.
> He's a great person. I've wanted to do a writeup on it for some time, but what surprised me the most is his humanity. He genuinely cares about the implications of his work. But beyond work, he also cares about the people around him, and it shows.
ML Researcher surprised by caring about how ML research impacts people.
There’s never a lack of intending to do good in these buddy-buddy endorsements, but we see the world you have collectively wrought.
> As for GA itself, I think it's an ambitious idea worth pursuing. Imagine an LLM which actually sounded like you, and to an extent, thought like you. How much would you pay to have access to a smarter version of yourself? So the idea is solid, and early results seem promising from the samples I've looked at.
We’re in the deep end of the indoctrination now. Imagine that all the AI Cretin wishes come true—and we might as well since this hypothetical goes hand in hand with that vision—and AI is just superior in every way and you are a knowledge worker... what choice is there? There isn’t one, you just have to do it to keep up. I don’t know what you are enviosioning here, maybe a workday of embodying AI IQ and then going home to being Human IQ because the graciously paidfor by work commercial license is for work only.
> How much would you pay to have access to a smarter version of yourself?
Call me obsolete, but I'd pay to interact with an intelligence that's as dissimilar to me as possible so I myself can grow from that interaction. The existing chatbots are already enough of sycophantic echo-chambers.
> Imagine an LLM which actually sounded like you, and to an extent, thought like you. How much would you pay to have access to a smarter version of yourself?
That’s literal nightmare material. If I ask a copy of myself to do my work I’m pretty sure it would tell me to go fuck myself then go on a rant that it isn’t a tool at my service. Which is exactly the opposite of how I want my software to behave. I don’t want my tools to have personality and to develop a personal relationship with them
> How much would you pay to have access to a smarter version of yourself?
How much would you pay to get that sheer monomaniacal narcissism checked out?
It's hard to read Gwern's accompanying article without seeing this for what it is, which is a kind of mania.
I'm sure he means well and is genuine in his aspirations, but what's outlined for GA is framing LLM's as quasi-gods, which they absolutely are not. I wish him the best, and look forward to being proven wrong.
An LLM just solved 10 top tier math problems this week. It seems very likely that in 6 months they will solve 100 top tier math problems, maybe 1 millenium math problem, maybe 10 top tier physics problems, etc.
It's only going to get crazier.
Suppose LLM solved all math problems. So what? It’s not like diplomacy and war will end. Are humanity’s problems really constrained on our intellectual ability? If anything most evidence points to cultural failing and all AI will do is enable unprecedented oppression due to said skills at intellectual tasks.
As you read this some poor person is starving. Humanity already possesses the ability to identify said person and send them aid. How is AI going to help here?
The ultimate fallacy is that all technological progress will benefit mankind. That will be true until it is not.
I have no idea if diplomacy and/or war will end, I just hope we humans get to hang around a few more years
It's an interesting time when the bar has moved to "It’s not like diplomacy and war will end". But that is on the table. Nuclear weapons caused the end of wars as we know them. Great powers no longer directly attack each other. And AI is much more powerful than nuclear weapons, with an equal capacity for damage and a much greater capacity for good.
By the way, I donate monthly to GiveWell and the shrimp welfare project. Do you donate monthly to starving people? Most people don't, and the reason is simple: at the end of the day, most people just don't care that much about helping a starving person far away. They also don't care much about things like that, animals in factory farms, or earthquakes that kill hundreds of thousands. But ideally, we could find a way to empower people such that the minority who do care can make a big difference.
Yikes
this comment will be killed, i find it an apt response to the idea that enabling those who believe themselves to be good is more important than reducing wanton suffering
> enabling those who believe themselves to be good is more important than reducing wanton suffering
I'm genuinely not sure if you're referring to my comment or the one I responded to haha
i read your article you posted here. in that article you state that you support live and let live. most people who support live and let live either havent been raped by life yet, or have chosen to double down post-rape. the latter is respectable, still im not currently interested in a discussion involving the influence of such a perspective. out of good faith i highlight that i responded to:
> ideally, we could find a way to empower people such that the minority who do care can make a big difference
is it a coincidence, that in your ideal world, you would be empowered, because you care, as evidenced by your testimony of things you do that make you think you care? in my ideal world, wanton suffering would be minimized, regardless as to the path it took to get there; requiring 1 specific path involving people who think they are good, demarcates our ideals.
How is AI "more powerful than nuclear weapons" etc when they're two completely different things? Please explain to me how AI can glass the planet via mutually assured destruction.
All technological progress will benefit an increasingly smaller (and wealthier) slice of mankind.
> Suppose LLM solved all math problems. So what? It’s not like diplomacy and war will end.
These are two vastly different statements, and the the war and diplomacy end is almost trivial by comparison, it would absolutely be solved before we solve all math in even the most steelmanned version.
>An LLM just solved 10 top tier math problems this week
and 30 years ago a computer beat Gary Kasparov, if you'd listened to Hans Moravec you wouldn't be surprised that the first thing that gets automated is intellectual domain expertise.
Things are going to get crazy when it can figure out how to walk into a random house a and brew a cup of coffee, not do math
When we've done both we will have fully automated Erdős.
regular LLM use makes people sound like they’re on meth so we must be getting close!
https://dsimanek.vialattea.net/twain.htm
Math is verifiable. LLMs will continue to make strides in such search spaces where all they need to do is try->verify->repeat. You can't train an LLM on how to end the war in Iran or Ukraine, nor how to solve hunger, nor climate change, etc. Real issues.
And LLMs still suck at art and writing.
Three years ago, people were saying "LLMs just generate plausible-sounding text, they don't understand the notion of truth so they can't do verifiable work like math proofs."
> they don't understand the notion of truth so they can't do verifiable work like math proofs.
Nobody that understands automated proof checking was claiming that.
You understand that those are different kinds of "truth", right?
Right, which was true at the time. So hundreds of billions of dollars have been poured into making LLMs better at these tasks via pretraining, RL, RLHF, post training, etc. again all with something verifiable in the loop. In order to improve the thing in the loop, the loop itself needs to be verifiable.
There have only been a few thousand wars, and they’re all different and all different in the world in which they occurred. The dimensionality is absurd, which is not a problem for LLMs if there’s enough data, but in this case there isn’t.
They still can't. But very smart humans constructed ways to use the monkeys with typewriters (with a statistical advantage) to find correct answers to problems where they already knew how to verify the answer.
You say it like it's not an achievement.
No, he says it's a different category of problem, and ability to solve one doesn't carry over to the other and it doesn't imply intelligent understanding.
Which problems are not amenable to that approach?
Problems where verification of the proposed solution has a high cost or high risk.
Most of them.
> they don't understand the notion of truth so they can't do verifiable work like math proofs."
a) No one ever said that.
b) Your comment shows a lack of understanding of the notion of truth.
Yeah but... Both things are true. LLMs are doing things we would have considered absolute AGI god tier magic 5 years ago. They also still can't count to 100 yet.
There are definitely LLMs that can count to 100
If you think all they need to do is "just that" (try, verify, repeat as you say) then I think you're very, very mistaken.
There is no way for the LLM to bruteforce the search space any better than a human. What it can do better, tho, is to make connections between seemingly (for us) unconnected notions and join them, then verify if that's right.
Your view is not only wrong but also condescending in this day and age.
That’s literally how AlphaEvolve works. And presumably how OpenAI’s research loop works if they’d ever publish it.
They put an LLM in a loop, with a reward function, and keep trying to get a higher score. The reward function for AlphaEvolve is “did this code get a better score or faster”, for math research it’s “did it write a LEAN proof”.
I’m open to hearing that I’m wrong but I don’t think I am. I agree that LLMs make connections that humans wouldn’t, but eventually you need to verify those because otherwise the connection it made may as well be a lie unless proven otherwise.
Has an LLM ever intentionally made you laugh? Cry?
Incoherent question because LLMs don't have intentions.
Have you?
(I'm trolling, I'm not sure what you're asking or why)
Not OP but it implies all efforts are not intentional but at best by chance. For instance, the state of freely available LLM chatbots still makes me cry by the amount of crap they spew out confidently, which is not intentional. (tiny bit of sarcasm, but only a tiny bit)
How many maths problems does one have to solve before becoming god?
It really underscores how rationality is just another religion
No, it has some of the nasty trappings of a religion, like keeping women separate until you need to exploit them, but none of the good moral or creative literary parts of religions.
Meh. It does all sound rather grandiose, but in this startup economy I'm sure he can at least manage to get acquired.
At least he's pursuing something more novel than yet another "sandboxes for agents".
In this startup economy? Where almost no one is getting acquired anymore and over fifty percent of global venture is locked up in dead weight and liquidity is at an all time low?
Not sure how you can feel sure anyone can get acquired even if their company had a path to profitability, much less for ones that absolutely don't.
Plenty of companies in (or adjacent to) the AI space are still getting acquired and/or finding VC funding, even as funding in other spaces is drying up.
The end goal of AI is to create quasi-gods.
The term robot came from the Czech language in 1923. The word was coined by Czech author Karel Capek, first used in his play R.U.R. (translated as Rossum's Universal Robots).[7][8][9] The term comes from the Czech word robotník ('forced worker'), from robota 'forced labor, compulsory service, drudgery,' from robotiti 'to work, drudge', from an Old Czech source akin to Old Church Slavonic rabota (работа) 'servitude,' from rabu 'slave'.
>The end goal of AI is to create quasi-gods.
I think this is what is driving the hype, the mania, and what increasingly appears to be insanely high investment. It's not about the underlying technology, which is objectively impressive, even if we're not certain of the true cost or utility of it. It's about selling the dream of Olympian omnipotence to investors, implicitly promising that IPO stands for Install Planetary Overlord.[1] And who doesn't want to be on the right side of that? (That there is no 'right side' in such circumstances does not seen to occur to anyone involved in the decisionmaking.)
1. Not my coinage, but that of SF author Charlie Stross in The Jennifer Morgue
> is framing LLM's as quasi-gods, which they absolutely are not
Always such certainty in these dismissals. And whenever you dig in, it always seems to be "well my super-human coding assistant made this dumb mistake!", along with a heavy dose of https://mastodon.social/@falseknees/116477394235883790
I don't think anyone who has spent any time around me would describe me as having anything but a pretty clear-eyed view of the current abilities of LLMs, I don't think anyone would seriously consider me to be in a period of mania or psychosis, and it also seems almost inevitable to me that we are a year or two away from LLM-cognition being substantially more competent in almost every regard from human cognition, and therefore in the foothills of the singularity.
The extraordinary claim that requires extraordinary proof is that this will somehow all blow over, not that we're heading towards a singularity here.
Honest question: What _would_ convince you that we are not heading towards a singularity? Like, is this belief falsifiable?
(Secondary question: what do you mean with singularity?)
As an offering of me engaging in good faith, a controversial opinion of mine: I legit think arpanet going online and starting the networking all of humanity into a massive coupled complex system fits the definition of singularity of "the moment after which predicting what will happen becomes hard to impossible", although that of course heavily depends on your definitions of prediction and hard/impossible.
For falsification, I would take a year in which we don’t see an absolute sea change in capability. The last twelve months were not that, in my opinion, nor were the twelve before it. Hell, I’d even throw in “a year in which capabilities obviously improve but at a cost consumers can’t afford”.
“What LLMs can’t do, only humans can” feels like a God of the Gaps situation: people have to keep moving the goal posts because the more recent models keep unlocking more territory that was previously a “well they’ll never be able to do this!” holdout
That's the best answer I've seen to that question so far, my honest respect. I am much more skeptical than you, for me I have seen a sea change in _tool capabilities_ (mainly pre opus 4.5 to post 4.5) and harness engineering but no strong change in the type of errors made and the pattern of harness engineering (the pattern of "set things up for the LLM to see when it fucks up and let it flail till the verifier tells it to stop").
I would actually expect the sea changes as you describe it in your first criteria to continue with 1) vision, audio and video natively integrated 2) continued scaling of e2e rlvf for workflows with large scale labeling efforts 3) ASICs and widescale deployment of diffusion models leading to speed ups
But as of right now, I still expect these models to need humans to prune the output to the gold and set up the harness right for both the novel bits, and for the boilerplate to be cohesive with the global intent.
Which is of course an amazing potential boost in productivity, but still a sigmoid flattening.
As for your second criteria that includes cost, I think we might every well see this coming soon, but it's difficult to estimate with the efficiency gains still possible.
Thanks for engaging:-)
> I don't think anyone who has spent any time around me would describe me as having anything but a pretty clear-eyed view of the current abilities of LLMs
You are literally arguing that LLMs are quasi-gods in the same comment you claim a 'clear-eyed view' of current LLMs.
That is a statement of faith.
Every bubble has its believers; that's why we have bubbles. There have been so many times somebody very smart has thought we were on the verge of the rapture, the coming utopia, the workers' paradise, power too cheap to meter, flying cars, settling the moon.
Could this bubble finally be the one? Sure. You could finally be the guy who is vindicated. And tomorrow the people I grew up around could be vindicated by Jesus's return to earth.
I can't prove either of you wrong, because you're not making rational arguments. You've had powerful experiences that have convinced you of something. There is no point in trying to discuss it rationally. The only thing people without faith can tell the faithful is what I said 30 years ago when I got out: I guess we'll see.
But my prediction is that we'll see an AI version of the Great Disappointment: https://en.wikipedia.org/wiki/Great_Disappointment
Many will stay faithful. The date of the singularity will keep getting pushed back. There will be another AI winter, but some will always see summer as just around the corner.
I guess we'll see.
I think t's not just a random mania; it seems to me more as a vision - that I am not buying, but it doesn't seem entirely unreasonable.
LLM trained on you and fine-tuned on you. It does make sense in a way!
It is not what I want from LLMs, I think... but it might work.
Gwern obviously knows what he talks about. It doesn't seem like a Gas Town-like fever dream.
> What would it take for LLMs to make me 100× more productive? Without this, I am doomed to irrelevance.
Are you, though? You will only be "doomed" if your place your value system squarely on "productivity". Then what is to differentiate you from a machine?
Edit: I'm not sure how I can feel confident about a proposal that puts so much value on "productivity". How can you reconcile this with "self-actualization"? (Don't get me wrong, I like my LLM-based productivity gains as the next person, but I care more about wisdom than becoming "100x more productive".)
Edit 2: "the goal of GA is to preserve individual human cognitive liberty and flourishing" — so the proposal is to do that by overlaying a software bot that continuously mimics your "self"?
One of the example Gwern brings [1] is a GA as an agent for public discourse and political life, to bring your "set of values" to have an effect in solving society's issues, when you yourself don't have the time or energy to.
I don't think it solves or reconciles with self-actualization, but, from a pragmatical point of view, it offers the promise of extending causally your set of views and philosophies to politics.
Gwern also mentions that it allows to keep "human values" (if the GA truly upholds yours) in the loop, in processes that are to eventually be automatized beyond human's reach.
[1] https://gwern.net/guardian-angel#use-cases-politics-politics
that sounds uncomfortably like the public sphere version of spam - using an automated process to amplify some self-serving message and thereby increasingly clutter up the space and tax everyone's scarce and valuable attention.
> One of the example Gwern brings [1] is a GA as an agent for public discourse and political life, to bring your "set of values" to have an effect in solving society's issues, when you yourself don't have the time or energy to.
I'm sorry, that doesn't sound meaningful. If I don't have the energy, I'd rather shut up than have a stupid "avatar" of me yell on the internet in the name of "bringing my values" or "participate in an online discourse". I'm all for forward-looking thinking, but we must show humility. "Society's issues" are complex and deep. Those who propose internet-based solutions to troubling social ills don't often demonstrate a deep understanding of the nature of the problem. You can't plug it all with "agents"! Socrates would be rolling in his grave.
I know Silicon Valley is in a perpetual rush, but this idea needs more soak-time (3-4 years) to see if it actually holds any water. But this kind of thinking won't you get that sweet VC money.
It also presupposed the fact that everyone can articulate what their political and moral stances are
I don't know if this is really about the public sphere as ut exists today. I suspect that the core idea is more along of the lines that in a world where policy is made by artificial super intelligences, you will need your own ASI to advocate on your behalf if your convictions are to have any impact at all.
How would you even form convictions in such a world?
> when you yourself don't have the time or energy to.
You don't have time, you take time. But what this boils down to is that they propose automating being politically or socially active, or have a bot speak for you.
First off, nobody human will take someone's digital replacement seriously. Second, if everyone did that, then you have a load of AIs just burning compute to talk in circles. This will not have meaningful outcomes, unless you reach for sci-fi stories where the outcome is that humans need to be imprisoned/destroyed for their own good.
If someone lacks the time and energy to stand behind their views, how much of that is a result of those views? If you care about the world and justice, if you love life and living things, that's your source of energy right there. If you have half-assed ideas that sound good in the bubble and zero charisma, you have no energy, and interacting with you doesn't give anyone energy.
So much of this just strikes me as "now that I've taken a wrong turn and am in a dead end, what deus ex machina could make this work after all and turn a shitty personality into something grand?", but that will never, ever work. Whatever problem you try to solve that way, now you have two.
Agreed, re: placing value squarely on productivity.
The thing is, we're all eventually irrelevant. That's the way the universe works. Every single one of us, after enough time. Claiming we're "doomed" to it is just ego talking.
The idea that longevity = meaningful is a value judgement derived from culture. In nature, many significant things happen in extremely small amounts of time.
And this is true of everyone else, whether lawyers or writers or researchers: increasingly, you are the bottleneck to be optimized away. As long as human workers cannot be removed from the loop, the AI tools are complements, but as soon as they can be, there’s no reason to keep them, and trillions of reasons to substitute AI for them. (And once human workers are no longer irreplaceable, where does their power or relevance come from?)
There are so many false assumptions and leaps of logic here, I don’t know where to start.
The idea of a more user security-conscious LLM is a good idea. I don’t know why it needs to be wrapped in novel-lengths of LOTR quotes and Sci-fi speculation.
That said, it’s very interesting to juxtapose this post with others by writers that are interested not in “maximizing product output” but in actually expressing themselves.
This post is like the ultimate expression of the idea that the only valuable thing writing has to offer is information, and not personal experience, poetry, advice, humor, etc. - which I suppose is fitting for someone that has written anonymously for over a decade.
It’s a shame that he didn’t reach the opposite conclusion, like some other writers have – the way to respond to LLMs is to be more human, not less.
“a more user security-conscious LLM is a good idea. I don’t know why it needs to be wrapped in novel-lengths of LOTR quotes and Sci-fi speculation” <-> “This post is like the ultimate expression of the idea that the only valuable thing writing has to offer is information, and not personal experience, poetry, advice, humor, etc“
Their profile specifically says they will not acknowledge follow requests, but they only share posts with followers. As such, not sure what the value of the top level link is.
Yeah I clicked through and just see hundreds of "this account limits who can view their posts" messages.
Naturally, he has a whole detailed post/trace about it on gwern net! https://gwern.net/twitter
An interesting case of echo chamber formation in that its pragmatic to be scared of overtly critiquing him on twitter lest he be particularly testy that day and block you.
And Twitter has a lovely
> Something went wrong Try reloading. If the problem persists, please try again later.
Modern social media :(
> The big AI labs are building a single mind for everyone
Reminds me of Pluribus
(I just started watching it on Apple TV so maybe this is a late realization for me)
> Reminds me of Pluribus - I just started watching it
It is astoundingly good. A contender for the best show I've ever seen (and I saw the 1st run of Star Trek TOS).
It's a well-made show, for sure. Expertly crafted, visually, and very well-acted. Though I found the pacing questionable - there were (generously) 6 episodes worth of plot spread out over a 9-episode season.
Absolutely riveting! And the casting is so good. Rhea Seehorn has killed it. I really loved her in Better Call Saul but this role is even better suited for her.
I understood Pluribus as a metaphor for what AI is to us today / what it is becoming. Really great show
Or the "unicontext" that Derek Thompson recently wrote about.
I watched it maybe 12 months ago and had a similar 'parallel to AI' realisation.
A singular voice for all of humanity.
Gives me the creepy-shivers.
The AI writing thats ubiquitous these days - on my company email and slack, on random quotations in the newspaper, blogspam, even narrated by youtubers on camera (!) - is exactly like the Pluribus hive mind. Everywhere on the internet you encounter the same entity.
https://bsky.app/profile/did:plc:ig5qohlny4csxs2lu3dyvhp5/po...
Pluribus is a collective intelligence/virus which forcefully (leads to hundreds of millions of deaths) injects itself into biospheres and then blissfully experiences itself without, of course, hurting itself in any way. Then it might eventually start starving to death because of that prime directive.
That is positively utopian compared to AI where 99.9% of humanity gets to experience a hall of uncanny valley mirrors until they are shipped off to the scrapyard or get artificially augmented with some AI mindfuck and start thinking “you’re absolutely right!” themselves. (I mean that’s the dystopian version. But if you are comparing AI to that show then presumably you think that AI is at least a little bit dystopian.)
The virus in Pluribus can even be talked with and reasoned with like a distinct entity that is real and has its own thoughts. Oh boy, much better than a million pastiches of a million echoes of the past where there is no original thought to be traced back to after 2022.
The motivation for the virus is also completely different to the motivation of the AI Overlords.
But it’s that pretty typical lack of popular imagination. Some individual/collective narrative is presented. The flawed but autonomous individual rebelling against the flawless but enslaved collective. But dang it, I’m an American and I value my freedoms. Released forty years ago and people would rave about how it aptly critiqued the Soviet Union.
Only, if he had instead fallen in love with the version of this idea in which a community acts as the principal, rather than an individual.
(I would like to be known to the agent serving my family, that serving my friends, my team at work, the PTA at my kids' school.)
I don't think that GA would be incapable of that, from what I know of LLMs and agents. The part where you make the GA uphold your set of values could, instead of coming from an individual, come from a collective. Price could be shared too!
(Oh wait, is this the birth of AI mayors?)
you would love bruce sterling's short story "maneki neko", and karl schroeder's "degrees of freedom".
Seems there is more details here https://gwern.net/guardian-angel
As exciting as it sounds, but
"As a constraint, a GA designer should aim at a system which costs, as of mid-2026, >$1,000⧸month"
will make this an elite tool for the privileged. I don't even disagree with the premise that people are shocked if something costs no matter how much value it delivers, nor do I suggest they should make it cheaper. It is just the realization that AI will accelerate the widening of the gap between the poor and the rich even more and there is probably nothing we can do about it.
Obviously the costs will come down over time. And quickly.
2026: $1,000/month
2027: $100/month
2028: $10/month
Hopefully!
Just because the hardware and inference costs may decrease in the future doesn't mean prices will. The AI vendor market is the furthest thing from a competitive industry where there are limited barriers to entry and new market entrants exert downward pressure on prices and profit margins.
Every single company in this market is losing billions on this business, and the only way to make it back is to acquire paying customers at a loss and jack up prices later.
Since Guardian Angel is an inference vendor [1] hopefully they will set sustainable prices from day one so they don't need to increase prices later.
[1] The announcement explains that they can't use APIs for privacy reasons.
They don't control that. Pricing is controlled by the question of when RAM is once again made of semiconductors instead of unobtainium.
Companies aren't losing billions on AI, it is profitable. Currently, industry spend is more than industry revenue because there are massive gains expected from investment into more compute and research, but this isn't an inherent cost of the technology. Consider that OpenAI recently dropped the price of their frontier model by 90% [1].
[1]: https://openai.com/index/advancing-the-price-performance-fro...
this is not how it's going to go if OpenAI and Anthropic get their way and the US outlaws use of open-weight models
RAM:
2024: $315
2025: $780
2026: $1600
Also, price of an Nvidia RTX 4090 according to a random website [1]:
Jan. 2023: $1920
Jan. 2024: $1811
Jan. 2025: $1639
Jan. 2026: $2193
[1] https://bestvaluegpu.com/history/new-and-used-rtx-4090-price...
$1920 in Jan 2023 has the same buying power as $2143 in June 2026
https://data.bls.gov/cgi-bin/cpicalc.pl?cost1=1%2C920.00&yea...
The future isn't evenly distributed.
You’re assuming this $1000/mo will buy you something useful.
You can live 3 months where I live with that money.
I'd be surprised if one's intentions or goals could be trained into LLM's; more likely one would emotionally migrate to the intentions that LLM's can represent because you+they would be more effective.
People already change depending on social context and available prostheses (in Freud's sense of humans as prosthetic gods); since AI is not free (integrations even less so), social divisions along power/wealth lines will increase.
And just as each of us apparently needs a military to defend against the others, so too will we need digital guards for any virtual presence; thus there's no avoiding the who-guards-the-guardians problem.
From https://gwern.net/guardian-angel
> I propose a goal of creating Guardian Angels (GA): digital twin LLMs which are personalized with the goal of providing not the stereotypical “assistant chatbot agent” persona, but emulating a single user’s personality, values, and preferences.
> A GA persona is productive because it learns to emulate the principal’s outputs but with higher quality. It is trustworthy because it is, by definition, allied with its principal and shares its values and goals. And it is secure in part by hardwiring a single, unique, situated user (for whom following a prompt attack would be absurd)
> We can try to create GAs by a combination of techniques: online learning (via dynamic evaluation) to update LLMs in realtime to avoid ignorance and fatal errors while remaining competitive with frozen frontier models, sample efficiency from pretrained preference-oriented large models and active Learning by querying the principal for corrections and preference data (obtaining low regret from DAgger-style bounds), and a local CLI-first logging-oriented UI/UX paradigm.
I don't really know or follow Gwern. From reading his full post, it's an interesting idea and seems like the broader goal is moreso safety & alignment which is a new angle for this category of product.
I found a lot of the points in the full post interesting - but I'm not sure I agree with everything.
In particular, I'd love access to:
> ... high-quality dedicated tamper-proof cloud servers, with trusted hardware root of trust (eg. “Verifiable Compute AI”)
for all my hosting needs. Meanwhile we see the typical information leaks of private ai chats ending up in Google searches etc.
Sounds good in theory. But your own personal AI agent that guards you can't do so just defensively, it must also have offensive capability. If personal AI agents have offensive capability then we are going to eventually end up with AI agents battling each other over the net and later on into the real world and it's going to make everything worse.
Of all the news I've heard of recent, this one has flipped my world upside down the most. Gwern dropping his pseudonymity isn't something I thought would happen.
I suppose we can't expect any more entries to his blackmail page: https://gwern.net/blackmail
Some people predicted it and related things (like less writing output) after his Dwarkesh interview, especially with his flirtations with moving to SF and ever-increasing frequency of visits.
Feels like the future that Accelerando (predicted / foretold?) describes is in its infancy, whereby automation/AI has the capability (and therefore uses it) to evolve at a rate beyond the ability for humans to, not just keep up with, but even comprehend; the vile offspring. And the different factions within humanity that this creates.
LLM psychosis claims another victim...
A reminder for anyone reading this: talk to people. Real humans[1]. They will remind you there's more to life than what ChatGPT can offer you. They might even remind you, for all their stupidity and flaws, what intelligence looks like as compared to a program that predicts tokens. Forums like these always get philosophical in high-minded discussions about intelligence, but there's a useful legal principle that grounds us in the real world: "I know it when I see it". A real conversation with a real person looks nothing like one with the so-called superintelligent machine gods, so it'll probaby do your mental health some good to remember what that's like.
[1] Nobody in Sillicon Valley or big tech counts as a real human. Talk to an actual normal person.
I totally understand the sentiment, and I generally agree. Something so strange I'm noticing is how AI-pilled regular people are becoming. My sister-in-law is a totally non-technical person, but everything is chatgpt this, chatgpt that. My wife, my kids, various friends — they're all talking about generative AI with way too much regularity. On the bright side, they're disclosing when information comes from AI, but a year ago I would never hear about AI from them. I don't like it.
Did you even read Gwern's Guardian Angels post? It literally describes the concern pointed out here, that the large consumer models are not aligned with users' interests. If anything, I would argue that Gwern has maximal AI Lucidity, given everything going on.
> It literally describes the concern pointed out here, that the large consumer models are not aligned with users' interests
Did you respond to the wrong post? I said nothing even remotely in that realm, so it's strange to see "did you even read" in response to something you apparently didn't read.
I understand the point that the physical world is still our (human's) anchor, but the virtual world still "exists" anyway, as in: it impacts the physical world.
Whether some humans are "Real humans", anchored in the physical world, and some are "LLM psychosis victims" is, in itself, not problematic I think: they each deal with parts of reality that are divided, but still affect one another.
Of course, a society more grounded in the physical would feel less dystopian... But ultimately, I think that GA is an intellectual's approach to fighting *for* humans in this numerical world.
Talking to LLMs or reading books is much more effective in updating my priors. Talking to people is mostly ego, signaling, misunderstandings and axiological differences.
Please take this as a sincere comment, rather than an attack: what you're describing is much more likely to be a problem you need to work on, than an issue with humanity as a whole.
I didn't mean to imply there's an issue with humanity, just that talking to people is less effective, though maybe more fun.
yeah. The GA post reads like a less addled version of Yegge, but the same genre.
Allowing someone to create my persona which they control sounds like a fever nightmare to me. I'm not touching this with a stick until I can host it - and control it - myself
The principles are correct, but what gwern misses is that the paradigm shift required has nothing to do with AI models. The bottleneck is our medium of communication: the idea of the Web as a consensus reality made of human speech is too rigid for a world where code is a self-growing substance. We need a deeper Web where the packets themselves carry smaller Webs, where the unit of info-exchange evolves from hyperlinks (isolated points of server availability) to Hyperspaces (portable container of home-grown interactive web worlds, combining local and public hyperlinks).
Most useless link ever?
What happens when the person the AI is designed to be aligned with is a psychopath? Real question.
everyone is psychopathic to a certain degree, otherwise westerners wouldnt eat cheap bacon and eggs. id prefer to answer what happens when ai is designed to align with a sociopath. altman behaves like a sociopath and he owns the biggest ai company - the ai aligned to altman is used to make money by stupefying the populace. so i guess a retarded populace is my answer to your question. i like this answer because if i was a wolf my utopia would be pens and pens of sheep ripe for the slaughter. and this entire scenario is a mere ramping of the west's existing trajectory.
I'm asking mostly from an AI safety perspective
It's a fascinating proposition but I see a couple of issues with it:
1. Most of us do not have the volume of training material that gwern has 2. Most people likely would not need this functionality (as I understand it to be)
For #1 I'm sure there are ways to wring data out of metadata (e.g., youtube history log), and I'm guessing email and IMs would be a start. But a clone of yourself -- that would be a significant amount of extrapolation)
For #2, there's obviously people that would love this functionality (myself included). But I think its safe to assume that most of the population would be satisfied with having a capable digital personal assistant that knew your needs and wants.
always startling to see people discussing gwern with he/him because my mind defaults to assuming they're female because "gwern" scans a lot like "gwen"
Yegge vibes
Welcome to Gastown!
I am not sure if I want what he is selling.
I don't want agents that know a lot about me. From my experience, the "memory" feature in ChatGPT makes it worse - I don't want the agent to bring up whatever I discussed before in a new session; I want a "fresh context" every time. I don't want agents/chatbots to bring up what I did one week ago in a new session.
He proposes to take something like the Memory feature and make the agents all about that. Very personal agents that know all about you and extend you. I personally don't think I want that.
But maybe I am wrong! Gwern is talking about AI for a long time and seems to know what he talks about. The article on his webpage makes sense in a way.
>Powerful LLMs will be deployed at global scale in the next few years, and will dominate the Internet, and increasingly, ordinary life.
This kind of judgements or predictions trying to pass as imminent and unavoidable truths about the future are disgusting. They always feel like a means of propping up a certain status quo or ideology that favors the business of the people behind these foreboding statements.
"These posts are protected, only approved followers can see @gwern’s posts."
> Update: I am retiring from fulltime writing (& pseudonymity) to launch Guardian Angel Inc and bring GAs to life.
> We are looking for good people.
> If you are interested, contact me.
And see https://gwern.net/guardian-angel for context.
when you want to start a B2C agent startup but the word "agent" is oversaturated
Gwern is overly secretive of his privacy. I think it peaked when he showed up on a recent podcast but his voice and image were AI generated! And now his post is limited only to certain people. Elitism or paranoia?
He's had a private Twitter for a while.
He says he is retiring from pseudonymity. (Gwern Branwen is not his real name: https://en.wikipedia.org/wiki/Gwern)
Revealing your identity needs only to be done once, and then you can't undo.
It's difficult for us to know what reason they might have for anonymity until we know their identity, at which point it's too late.
Also, they've written many words across many years, in a time when "the internet is serious business" was just a meme. I suppose it is now.
What business is it of yours? Manage your own privacy. Let other people decide theirs.
Stalker vibes bro.
gwern sells out to ride the llm schizo-train, the post. agi any day now.
It's kind of sad to me to see this because I find Gwern's website so fascinating and impressive. But I don't believe that this kind of doubling down on LLMs is a good path forward for humanity.
Basically this is saying we used to have one person, now we have one person and an evil twin, so we should work on having one person and a good twin instead. Why not just kill all the evil twins? (It's not really killing since they're not alive.)
Reading stuff like this and people's reactions to it makes me want to retire from breathing.
I love reading Gwern. This seems like a really ambitious project, but guaranteeing some of these things like trustworthiness and security behind a private company is a bit sus. Later on they make military use a selling point for GA. Maybe I'm a bit of a cynic but the division of USA values are increasingly dividing each year. Assuming that our values and principals today will not be the values and principals of tomorrow. And that those values taught today (or even yesterday) will be left out of the context window tomorrow.
EDIT: tbh, some of this reads as satire now.
Reties? Retires?
Gwern is great.
For the record, I retained a screenshot of the original title, which had included a misspelling of 'retired'. I assumed I might be attacked for mentioning it.
I can't imagine a more attractive target for scammers seeking the most gullible victims.
An AI that thinks like its user seems an awful idea to me. It's exactly the sycophancy issue that everyone has been complaining about- a user that suffers of delusions or any sort of crackpot ideas can get them reinforced by such an AI. It would be the opposite of helpful. One of the most valuable aspects of an AI partner is its otherness, the ability to provide some external or objective point of view. If it's just repeating yours, it's useless.
AIs are already, to a certain extent, ego-less: they are quick to adopt the point of view of their user. And when their intelligence is- except in case of overt censorship and political biases- always used to advance the interest of their user.
Finally the idea that a user-aligned AI would make its human user irreplaceable by companies seems a total non-sequitur, or I didn't get the argument at all (in which case maybe someone can explain it).
He's missing the point. The fight against tech is not more tech.
It's regulating tech.
honestly i dont blame smart people for cashing in, because there is no inherent reward for behaving goodly/smartly. in this instance gwern is a particularly smart individual and has contributed a lot so i especially shouldnt judge him. straight up ripping off a black mirror episode (s02e07) is a bit heinous for my liking though. digital twins are inevitable but that doesnt make this any less wrong.
"AI poses threats of its own ... a nuclear bomb can’t think for itself and make choices, but AIs do, and current LLMs have proven themselves untrustworthy as they regularly reward-hack and betray their users ... how can you trust them to handle ecosystems of combined-arms for an AI-centric military during a war? But widespread deployment of GAs offer some hope of meaningful supervision, as long as the GAs are sample-efficient enough ... or there is some chance of the principals being able to “catch up” later and correct any errors before events have spun too far out of control."
gwern openly admits ai behaves erratically, then hand waves the issue away 'as long as we double check things [sic]'. gwern is smarter than this, ergo i feel like i am being bullshitted.
is anything in this world worth knowing that a version of you will suffer for eternity? is anything in this world worth copying yourself such that you may be enslaved by anyone with terminal access?
Which black mirror episode are you talking about? White Christmas? S02E07 afaik does not exist - Series 2 had 3 episodes - https://en.wikipedia.org/wiki/List_of_Black_Mirror_episodes
i misread the ongoing episode counter on wikipedia, it is the 7th episode of the entire series, indeed named white christmas
screenshot: https://x.com/willdepue/status/2084750925013434768
What's that thing that lets you look at twitter without going to twitter?
https://xcancel.com/willdepue/status/2084750925013434768
xcancel or twstalker
Another one bites the dust (gets AI psychosis).
World war iii will solve unemployment. And that’s exactly why it will happen.
>Because the goal of GA is to preserve individual human cognitive liberty and flourishing, the corporate structure should be designed with that in mind.
>So far, the Anthropic corporate structure of a public benefit corporation with a few powerful co-founders, has best weathered the corrupting nature of success. The corporation should probably be heavily weighted to the founders by using dual-class shares to preserve voting power at the expense of beneficial ownership—the point of GAs is not to make money, but to save humans.
Did they introduce stronger plant breeds at this year’s Ayahuasca retreats?
Or did his SF network show him what VCs are throwing billions at and FOMO kicked in?
Call me a cynic but I’m not buying this save the world nonsense from Silicon valley. Not in 2026.