I agree with the disconnect between the professionals, but i have yet to see anything that makes me believe that there is a class of 5000 internal people who are that much further than the top users are.
The internal models are more eager, and there is the scale difference of being able to spin up 10.000 vs 10, but to get something valuable out requires skill.
Agents that produce good results 90% of the time vs 93% of the time are not enough of a difference. Multiplied too many times, that error rate is too high for scale to become "magic".
Finding proofs is one of the exceptions.
Just look at the codex/claude harness. They regularly lag behind 'external' users.
Lauren Tan from SpaceXAI/Cursor/Grok has been on a few podcasts recently. I was surprised how much info she shared, and she sounded quite honest about what works and what doesn’t, in the context of building grokbot. I wouldn’t be able to do all the verifications she does, with my token budget. But it doesn’t seem like rocket science either (pun not intended).
Which is precisely what I'm responding about above. 93% actually collapses much slower than 90% since the rate of reduction is based on that of all of the previous rounds, not just the current chance. E.g., this is the difference between 14 rounds before an expected failure instead of 10. An extra 4 rounds deep at the same error rate is a big step forward despite a small change in the base percentages. Add just few more percentage points again and you've more than doubled the depth.
I.e. it's precisely when you look at these chained that the difference is actually very large. It's really only the 1st turn or two of prompts where the difference seems negligable.
How are you running that math and what are you consider to be "expected failure"? If even 30% of your runs go into the garbage, that sounds horrifically unreliable, and both 7 and 10% failure rates hit those margins in less than 5 iterations.
Not to mention that the deeper your iterations, the more money burned on garbage before you realize that data in turned to garbage out turned to garbage in somewhere in the chain.
I'm taking 0.90^n and 0.93^n and seeing 0.93 does more than the initial 3% better over time, which preciesly saves you from things like spending tokens on garbage by this larger factor. You're free to pick whatever target reliabilities from that. E.g. you can maintain above 80% accuracy over 3 turns rather than 2 or 50% over 6 turns instead of 3 or you can pick another percentage point and see how the depth to expected error/expected error at depth numbers are much better than 3% and how another 3% actually doubles accuracy because this is all non-linear.
There's also the matter of "90% compared to what". If it was 90% per attempt compared to a calculator then I'd just use a calculator. If it was 90% per attempt for something I have a 10% chance of getting right per attempt then it's not really so bad. OTOH if it's 90% per attempt at keeping my car in the lane then it's obviously quite bad. I.e. there isn't one "reliable percentage" to target which is why I'm comparing the curve instead of just the most stringent things I can think of in isolation.
Completely agree that error tolerance depends on what it is that we're doing. Some things demand no errors at all, some things have a cost to attempt, some things you can't pre-evaluate for correctness if you don't already have the prerequisite knowledge and you only get one chance. Not everything is software where we can conjure 50 more end-to-end tests to ward off demons. When it's not a harmless throwaway weekend project, 7% is already a huge error rate before you ever think of feeding it back into itself.
You might be thinking along the lines of the O-ring theory. Indeed long supply chains accumulate errors exponentially.
However the origins of theory explored the outcome of disastrous and irreparable outcome of space shuttle exploding, lives lost, and billion of dollars destroyed.
When LLM goes off the rails the cost is much lower, and retry is on the table. In fact our core competency shifted from writing code to creating environment where agents can retry cheaply and course correct based on feedback.
A modest increase in the number of turns it takes to reach decorehenve could make a difference between
Decoherence and reaching the a grounded checkpoint.
If failure is essentially guaranteed somewhere in that chain, success becomes impossible. You don't need to lose lives to have simply have a fruitless endeavor. And when we're talking about law, we're talking about more than a fruitless endeavor. Law also isn't software that can have tests crafted and fed back into the loop.
First off, the 90% to 93% success rates are not step-by-step failure rates between successive steps of pursuing a solution. They are overall end-to-end success rates on extremely difficult problem sets that most humans would score close to zero on, and most humans with sufficient training and experience would score significantly less than 90%. And they are problem sets that grow successively harder as AIs get successively more capable. A few years ago AIs were being tested on high-school level math problems. Now they are being tested on problem sets that challenge people with PhDs. Or problem sets that humans are (so far) incapable of solving.
Compare that to humans. How often do you think humans get it right the first time? Almost never, for sufficiently complex problems. And how often do humans get the overall result wrong, without realizing it? Also, almost always for sufficiently complex problems, or sufficiently ordinary people. And what percent of the general public would be able to get even one question right in a problem set based on Math Olympiad problems? What percentage of mathematicians are capable of solving even one Millenium Problem?
Furthermore, you have to take into account that AIs do not particularly use output to input architectures. I frequently see self-corrections to plans when I'm developing using AI assistants, and plan changes are sometimes major. ("Oh I see. I need to revise step 4 of the plan." -- Claude paraphrased). Clearly current-generation AIs use more than input to output feedback; they also use supervisor/client strategies of AI and collaborative networks as well. And often provide self-correcting infrastructure, varying from "Claude: Oh. That generated a compiler error. Let's fix that.", to "Claude: One of the test cases is failing. Let's see what's going on.". And that self correcting infrastructure allows AIs to try again and again until they succeed. (Not unlike what humans do when solving difficult problems).
As a member of the 0.2% (but not the 0.00006%), I have to imagine that frontier models, when applied to research topics, are using most of those 10,000 instances to investigate alternate hypotheses in parallel. Which is also something that humans do across entire disciplines. Based on my personal experience with difficult technical and commercial and scientific problems, I sincerely believe that most breakthroughs are not made because whoever made them is more brilliant than the other hundred or thousand or dozen qualified people who are working on the same problem, but are made because they happened to be lucky enough to start with a somewhat random hypothesis that luckily turned out to be the right one. In case you were concerned that this is a sour grapes point of view, yes, our team was the one that came in first.
So even 90% may be a very big deal, for problems that human get right about 0.1% of the time. Or even for problems that AIs get right about 0.1% (but have enough instances available to pursue 10,000 hypotheses in parallel).
p.s. Ironically, it really looks like humans score significantly less than 90% on the "is 90% good" problem.
> The internal models are more eager, and there is the scale difference of being able to spin up 10.000 vs 10, but to get something valuable out requires skill.
Yeah, but the skill is knowing when it's talking nonsense.
Same deal with the recent stuff about vibe coded Photoshop clones. Yeah, they work, but you'd have to do Adobe-level market research to understand what bar the product needs to pass, and the vibe-coders didn't have that, so they saw an image editor and mistook it for "basically fine".
> Agents that produce good results 90% of the time vs 93% of the time are not enough of a difference. Multiplied too many times, that error rate is too high for scale to become "magic".
Sure. However, one of the ways we see things change with, e.g. METR time horizons is that agents fail (IIRC) 10% of the tasks of some (human) duration all the time, pass 10% of that (human) duration all the time, and the rest have a random success rate you can fit to a sigmoid. As the agents get better, they still pass 10% all the time, still fail 10% all the time, but it's a harder set of questions.
2.5 years ago, I was thinking "I know what's a fun idea for April fools day! A programming language written entirely in runes!", the LLM produced non-compiling code for the compiler. 1.5 years ago, I thought "I'll give that another go", it managed it trivially and wasn't even interesting enough to bother with as an April fools day joke, not even with the video of a Viking explaining it to you in Icelandic.
We should probably assume the companies are 1-2 models ahead of the public, at all times. The HuggingFace incident happened about 2 months before the public got models of comparable competence.
No doubt the unreleased models are better at the things that LLMs are good at (coding, hacking, math) than what's been released, but there is still zero sign of their intelligence becoming broader and more general, and knowing how these models work and are trained there is no reason to expect them to become much more general unless changes are made.
The marketing message being pushed by Anthropic (& also OpenAI) is that these narrow math/hacking/coding skills are representative of the model's intelligence, when that seems to be the exact opposite of the truth - these narrow human/super-human competencies are entirely atypical of the models more general capabilities.
> but there is still zero sign of their intelligence becoming broader and more general
Ahem, getting better at these things is the sign:
It's hard to see how OpenAI or anyone else could provide this on-the-job training to an LLM even if they let it loose in a programming playground where it could generate the training dataset. How fast would the context fill with compiler/link errors, debugger output, program output etc ... once context was full you'd have to pre-train on that (very slow - months, expensive) before it could build on that experience. Days of human experience would take years to acquire. Maybe they could train it to write crud apps or some other low-hanging fruit, but it's hard to see this ever becoming the general purpose "AI programmer" some people think is around the corner. The programming challenges of any specialized domain or task would require training for that domain - it just doesn't scale. You really need each individual deployed instance of an LLM/AI to be able to learn itself - continuously and incrementally - to get the on-the-job training for any given use.
Yes, I failed to anticipate environment/playground-based RL-training and how much difference that would make, although I think the broader point about limitations of lack of continual learning still stands, and I can't see these simulated environments ever become complex and/or varied enough to be any substitute for the need to learn on the job.
However, gotcha acknowledged, but my point was about the jaggedness of AI, and how huge the divide in capability is between the domains where RLVR works, primarily coding, hacking and math, and everything else. The success and scale of RLVR in areas such as coding only make this gap more noticeable.
Where are the verifiable rewards? It's simply not true that RLVR amenable domains are the only things LLMs are improving massively on. Even Vending Bench is improving.
Intelligence is a jagged beast. That's just the nature of things. We evolved in certain environments and situations and sometimes there's a mismatch and it causes all sorts of wonky things. We just call them funny names like optical illusions and cognitive biases. Most people fail a simple conditional reasoning problem unless it’s dressed up in familiar social context, like catching cheaters (Watson Test).
> vibe coded Photoshop clones. Yeah, they work [...]
I think what people (sometimes) miss is that building something is one part of the work, but keeping it running, developing and refining it is a whole different story. And I think all those vibe-coded software will not stand the test of time.
Photoshop is 36 years old now and it wouldn't be the most-used, most-established solution for professionals if Adobe didn't do great work over all that time.
Not really rooting for Adobe here, dislike their subscription models just as anybody else, but I think you'll have to recognize that they kept it running and never stopped.
I'm… well, "enthusiastic" would be the wrong word, but perhaps "impressed"?… with LLMs-as-coders, but they're objectively poor at business decisions, and Adobe keeping their business running all these years is a thing that I don't think current LLMs could have done.
LLMs don't need to focus on the business. They can make the ideal software for you. Who cares if it's not the perfect software for me? I can have my LLM do that. Especially if there's an open source base for me to start.
A separate question. Consider I'm responding here to (amongst other things):
> And I think all those vibe-coded software will not stand the test of time.
When I use LLMs to write software for me, it is rare that I think "this is great, I should share it". Even when I do, I am aware of how fast these things are improving, so while I put great effort into, for example, having Claude make me a nice isochrone map generator, I fully expect that a year from now any similar project (even if different enough the LLM will not simply clone my git repo) will be a one-shot prompt.
What I did will only stand the test of time if all the AI labs shut down and nobody takes over from them. Same with vibe-Photoshop-clones: a year from now, anyone who wants this kind of thing will get it with one prompt, not thousands, and this specific instance will likely be forgotten.
The public has yet to produce a result of comparable impact to HuggingFace, to my knowledge. It's not really about the model, it's more about orchestration infrastructure. Seems like we've hit the point where you don't necessarily need a smarter bot, you just need an arbitrary number of them working together.
I feel like you're taking it somewhat personally as a statement on ability or intelligence levels. It's really just about the scale difference and the access to that infrastructure. That is what enables that class to look at and think about problems differently.
Having 10,000 (or more) agents working for months on a problem is categorically different from the type of usage most people are familiar with. The complexity explodes to a point that requires the "archaeology" he mentions for people to even try to understand what's going on.
This is the key insight. Those capabilities are currently limited to frontier labs and literally cost them millions of dollars to achieve. But as we well know, computing costs and algorithms are optimized rapidly and soon those capabilities will be more widespread. We are at an inflection point where we can design systems that are able to go and do things that we don't fully understand, and they will handily outpace our forensic bandwidth.
> But as we well know, computing costs and algorithms are optimized rapidly and soon those capabilities will be more widespread. We are at an inflection point
I hear you, but also, "a broken clock is right twice a day." And really, there are people still alive who have more or less witnessed the entire rise of the machines.
My understanding is he is not talking about "top users" as being further ahead in skill, he is talking about access.
The kind of FOMO he is triggering is the same kind that wanting to be in an exclusive country club gives a certain kind of businessman. One simply can't reproduce that network outside of the exclusive access, no matter how skilled they are.
It's very true to me. Another effect I noticed in less tech inclined jobs (but still office jobs) is that they're still convinced "their" special job is not possible to automatize, because it's too complicated. Mind you some of them have really simple jobs, bordering simple data entry in excel. They tried chatGPT once or twice one year ago and don't realize what's coming for them.
Since in tech circles we're the first ones to optimize and use new technologies we are most aware of the huge leaps that happened and the current impressive capabilities but lot are still oblivious to it.
My biggest concern isn't actually that my job can be automated (it can), but rather that the persons who need me stop valueing being right, and start valueing >being told< that they are right by an AI.
I feel my main purpose isn't that I can technically do something my clients can't, but that I know how to interpret their needs and find the solutions that fit for them.
Now they just query some AI which spits out >some< solution and they won't know that they are wrong until it can't even be tracked back to their AI.
This will result in "but the computer said" excuses all over the place.
I think we'll see an unbelievable upheaval as companies figure out that while the technical aspects of some jobs can (and should) be automated, the interpretation of those technical aspects and other higher order, but not strictly inherent processes can't. By upheaval I mean companies closing because they replace staff with AI, and lose things along the way because of it. They won't see it until it's too late, because of the pile of money they're saving.
My example comes from maintenance in machine shops. My dad's siblings were almost all employed in machining in some fashion. One uncle was maintenance for a local large company. They decided they could move maintenance from in-house specialists and contract that work to a multi-state massive company; anyone can follow a maintenance manual right? All you need are the proper certifications and everything is cool, right? The cost was less, and the number of maintenance staff on site actually went up. Great for the owners.
But what they lost was consistency and understanding in the process that eventually led to closing the company. My uncle was there every day all year long for decades and could hear and feel in his feet when something was going wrong with a machine regardless of the maintenance schedule approved by the company that manufactured the machines. The new outside company didn't pay as well, and didn't have great benefits, which meant that they cycled through staff very regularly. No one was there consistently for more than a year or two. Even though they were technically following the manufacturers recommended maintenance schedule, machines went down. Machines then stayed down for days or weeks longer than they should've, because the expertise to repair either wasn't there, or the person leading the job would quit and they would have to cycle in a new maintenance person. They lost jobs, customers took work other places. 8 years later the company closed.
There are lessons in this kind of thing, but nobody learns them.
This is a techne/metis split. You would probably like this [0] chapter about the "rightness" struggle between engineering and service at Xerox. Some paralells to A Big Little Idea called Legibility from Ribbonfarm, too.
The way this will play out will be weird, and probably not like anyone expects. Because on the one hand, you're 100% right on the value of "feel" you develop for a system when you're around it daily for a long time. But, on the other hand...
The other day I visited our old apartment and noticed the dishwasher was making loud noises. Worrying about whether it's about to fail and whether I should call in a technician, I just recorded a bit of the noise, screenshotted a spectrogram from an app, made some photos, and threw Claude at it. It proceeded to analyze the recording and match specific frequencies to components of the machine, and gave me an analysis indicating the machine is operating normally, and the noise is just the pump being louder than I'm used to due to machine being old and a bit worn out.
So we literally have that kind of capabilities in our pockets, and this will compensate for the problem of missing experience somewhat, even simply because unlike low-paid frequently-rotating staff, LLMs are not lazy and will do the detailed troubleshooting work even off a recording and "wtf?" as prompt, so if you can train your cheap, expendable staff to do just that, that's already supercharges them.
I'm also amazed at the abilities of AI when I can verify them. But, how do you know that Claude is right about your dishwasher? I mean you're not that much worse off if it's wrong (compared with a similar medical query), but isn't this a case of trusting the AI without verification, just because it looks correct and was stated confidently?
I know enough to evaluate the analysis I got; I have some general expertise and worked in fields adjacent to this, which is why I had the idea to troubleshoot a dishwasher off an audio spectrogram in the first place. And on the off chance I was wrong, this is super low risk - would cost me extra few hours of my life at a less convenient time.
My point isn't that AI can substitute for expertise in general, not just yet, but that if you're competent enough to evaluate it, AI can absolutely do hours of work in minutes even for things involving troubleshooting on factory floor (or home appliances). And, that this is a lever where personnel on the floor may have sufficient expertise to evaluate diagnostic work, but not enough motivation or space to do the diagnostic work themselves.
In the IT department in which I work, now the team is reduced to just me, I find myself using the phrase "Why fix a problem in 15 minutes with your own engineers, when you can take 3 weeks with a Managed-Service Provider?". Most of what I have to care for is office/internet network related.
People in other teams are slowly getting it. When I leave (or more likely "am let go"), the cold hard reality will become much more obvious during the next network problem ;)
Which isn't necessarily a bad thing. Smart leaders / entrepreneurs will realize this and then use human input where required. The rest will be cleared out after the computer has made enough mistakes for them.
I wholeheartedly agree. To clarify my position, I don't think having these amazing models mean we should replace everyone, quite the opposite.
I do think they are close enough to the stage where people will be replaced, even though it's not the right or correct thing to do, and that we need to move on from the "denial" phase quickly enough so our societies can adapt / regulate / legislate while not too late.
I believe that the imprtatnt bit that many will come to understand ,maybe after costly mistakes is that 'tasks can be automated, jobs cannot(yet)' one still needs the discernment to know what the sequence of tasks lead up to meaningfully.
It kinda reminds me of the story of the Toyota process , while their car product lines were/are heavily automated the process still relies on the smalll teams on the line to spot/solve and iterarte over mistakes/inefficiences then add the refinements back onto the automated tasks.
Similarly althose stories of radilogists going jobless have not panned out , as interpreting slieds is a 'task' that a radiologist does.
Semantics , yeah but it does shape the way of thinkings regarding jobs/economy and not to mention the whole anthropomorphizing by the AI labs and boosters. In essence the wordings can/could introduce a healthy dose of reality/skepticism to the wildest predictions regarding job losses etc.
Some internal engineer is doing god's work by creating a system skill for the LLM that multiplies all estimations by five. /s
On a more serious note, the pre-LLM internet had very little useful detailed information regarding software estimation addressing effort for building something in specific domains.
Most writing on software estimation tried to tackle a general approach that could apply to all domains. So you had a "applicable to all, useful to none" situation going. And since crud apps were the most ubiquitous apps, software estimation techniques were perceived to give more accurate results with respect to those type of apps.
I'm sure internally big AI companies are accumulating vast amounts of estimation training data by documenting how long it takes their LLM to complete specific app building tasks. I don't think they would be in a rush to push that data into public models though as it would be like beating a dead horse that is the current human workforce anyway.
If you hook it up to legal databases like Lexus Nexus and let it ONLY search proper places for law (instead of Wikipedia) these systems are superhuman at law too. A lawyer acting as a meat proxy for Opus 5.5 will smoke a lawyer without AI systems on average these days.
It's still mildly amusing when I ask what kind of work is involved for a change, and it gives me some estimate of 3-6 weeks or whatever, and then I say "no I mean like what changes" and then it spits out the complete change in 30 seconds.
Your concern about higher-ups valuing being told they are correct vs actually being correct is exactly what I saw at my last job (one of the largest companies in the states) and now at another smaller firm. It was really disheartening because I do use qwen a lot for personal stuff and claude at work, and have been able to multiply my productivity by only having to understand a ticket and then code review instead of doing all the intermediate steps, but a lot of people have let it go full self driving and just tell it "implement ticket ABC-123 and raise a PR" without checking the output at all.
It is very good at programming. Because programming is a lot of language, and these tools were also made by programmers and post trained to be good at it.
But I assure you it's even "better" (not really but the task is so much easier) at making your spreadsheet look nice and compare actuals vs forecast for your budget than a complicated programming task. It will just one shot it, and it will be objectively correct, all in a very short time. But these non tech people don't know they can use an agent, they have asked one question in the chat interface with 0 context about what they're trying to do, and conclude they are not good enough.
I'm not saying everything is replaceable by AI - and by the way I don't want anything to be replaced by it, but I believe it will happen - just that some people don't realize yet how "trivial" their job is to modern models.
This. The main blocker in AI proliferation I see is that people don't even consider it can do more than they thing it can.
Just yesterday, my wife recounted to me a conversation with her friend, where the friend mentioned they felt lazy and had some piece of copywriting done by AI this time. To which my wife said, "TeMPOraL would've also had AI upload it to the website afterwards".
Yes, I would have, and I would've also had it install the tools needed to upload it too, because why not. The only difference between me and my wife's friend here is that I know I can ask AI that, and I know it will work.
People have gotten very used to breaking down tasks into separate things they know how to do on a computer. Asking an LLM to do each of those individual tasks is easy but feels both lazy and also ‘so what?’ - I knew how to do that task.
The thing people are slow in realizing is that you don’t need to break the task down, and you don’t need to know how to do the task. You don’t even need to have the software installed on your computer to do the task, or even know that software to do the task exists - a sufficiently motivated LLM can pull down the needed open source tools and use them, or even build its own.
Exactly. It takes some... willingness to break through your own mindset (I don't want to call this "courage", but at times feels like it), to just tell the LLM, "I need <this and that> to be <like this>, make it so".
Two years ago, that was unthinkable. A year ago, it was a quick way to set a lot of tokens on fire. Today, well, I'm yet to find a thing Opus 5.5 doesn't do at least passable (and usually stellar) job on.
This is why hosted agents (muse, dots, grok bot) will succeed. Organizations won't relax their PC security policies, so they'll need a platform where the agent actually has permissions to develop and install software.
It is particularly suited to programming. But most business processes are basically a program that is executed. Especially in modern times, where office clerks have less and less decision-making options.
Most people think that using an LLM requires no skill: it is just talking to a machine.
I think it is a soft skill that only hours of practice can make you good at.
Unfortunately, most of the people I know use LLMs as a search engine: their requests are just short sentences. They get basic results and either they stop there, not impressed or they keep using it as a smart replacement for Google.
So they judge what AI can do from their first weak experience.
Going further than that would requires more effort: going deep on a subject they know, thinking carefully about the prompt they write, more importantly judging the answer they get and pushing back.
I'm reminded of an interaction I had with a non-technical user. They knew ChatGPT can translate text, and so, to remove spelling mistakes from their (Dutch) text, they would first ask ChatGPT to translate it to English and then ask it to translate it back to Dutch. Mind-blown when I said "you can just say 'fix the spelling and gammar', no need to translate"
This is the level that many non-technical users are operating on.
Right. A life hack carried over from when Google Translate was the best tool, and loop-back through it could be reliably used for longer-form text (similar to how "did you mean?" in Google Search was an excellent spelling corrector for shorter text).
You are essentialy interacting with a SaaS that can be modified and altered at any point by its creator. Treating the LLM as some predictable entity that always works orders of magnitude better with precise language needs to be supported by evidence for each specific case.
Since the vast number of potential users of LLMs are, as you say, poor practitioners of precise language, it is incumbent on LLM providers to continuously iterate on providing LLMs that work better with imprecise language.
Hence, the people who claim that precise language is some 10x - 100x productivty skill seem to ignore the cycle of big LLM providers who are optimizing the models for users who lack those. (One of the early optimizations was the chatbot asking follow-up questions to the user for example).
So I think the people who think that their precise language skills are going to allow them to avoid the jobocalypse are also in a separate bubble of theit own. Precise language skills are going to matter less and less as LLMs increase their hand-holding skills for less equipped users.
This presupposes that the llm will be able to pick up the slack for the users of imprecise language as to me that is a proxy for domain expertise. If so then the human is not needed which does seem like a stretch.
My view is that the human is needed for verifying the source of truth, not for providing precise language which the LLM is continuously trained to adapt better without.
I have not come across a rigorously argued technical analysis that has claimed that using precise language can in any way eliminate the possibility of hallucinations or unaligned behavior. Short of providing a proof for the elimination of hallucinations and confabulations, merely reducing the behavior by a random percentage is not an acceptable status quo for completely removing the human.
I'm currently seeing a lot of developers and AI industry insiders that do not distinguished between the capabilities of their LLM Agent and the added capabilities of the agent harness. They think the capabilities of the agent harness are within the LLM, and in that confusion they are trying to get their agent to make them an agent harness with all these capabilities. But LLM agent harnesses are brand spanking new, how to make these harnesses are not in the training data of their agent, these developers are more or less vibe begging their agents to make them a harness, so it hallucinates the harness architecture. I am seeing a "agent harnesses" that are delusions, the these developers are not strong and experienced enough to realize they are literally doing the worst work they possibly could... And these idiots are the AI leads.
> I think it is a soft skill that only hours of practice can make you good at.
So this, I think, is the crux of it.
When most people, at least in the world of SWE, think of what it means to be skilled at something, they think of knowledge and experience acquired over a much longer timeframe - months to years even. So from that pov, something you can get reasonably good at within just a few hours or days might not seem like much of a "skills moat" at all.
I have a friend with a job like that and he is not being replaced not because it is Impossible but because it would be too expensive atm. But they replaced his whole team with workers south east Asia.
The prices are dropping fast. At some point one of the labs will decide they have a model that is good enough and they will burn its weights into silicon which will make the model much cheaper still.
It's shocking to me the amount of people that confused "not automatable" with "not cost effective to automate" (which is effectively everything), and the further step which is - the reduction in quality people will be forced to bear to get massively lower costs.
Most people are wrapped in plastic instead of cotton.
>"not cost effective to automate" (which is effectively everything)
There still seems to be a wide gulf in automation capabilities and things that require human dexterity, such as doing things for the infirm. Even just cooking and laundry seem quite a bit away. I guess the original tweet does mention this:
>Capability peaks in domains that are digital, verifiable and economically valuable.
Driving is the most impressive one so far, in my opinion.
The frontier labs themselves are still hiring hordes of human "account associates", "android engineers", and "data scientists" (https://openai.com/careers/search/), when they would be better positioned than anyone to leverage state-of-the art AI capabilities.
They need top talent to operate the current generation of models, that's at the mediocre-talent level, in order to teach the next generation how to operate at their level, and eventually above. After which, they won't be needed anymore.
As a white-collar worker, it really does seem like my programmer friends are living in a different reality when it comes to AI. They have totally restructured their workflows around AI, and I'm still struggling to find the value in it for my legal and compliance consulting work. (And I consider myself pretty tech savvy, with some programming and web development knowledge and having spent 10 years at crypto startup.)
Even in my personal life, while I've been addicted to using Claude and ChatGPT as a first step for research/projects/etc, I'm still not convinced they are actually saving me time -- just too many errors, too confidently stated, and I have to re-do all the research myself.
I wonder if the validation piece is easier with coding?
EDIT: To be clear, when I say I'm having trouble incorporating AI into my white-collar work, I'm talking about producing GOOD work. On a few occasions I have given in to the temptation of using Claude to produce work that looked good (superficially), without verifying everything.....it worked out, and I'm sure others are doing it. But that scares me, because it feels like a ticking timebomb.
"Superhuman" in what sense? With respect to legal work.
Is it just faster? More persuasive? Better at coming up with novel arguments?
Speed is great from a client perspective, but not as important as accuracy on important stuff. I haven't seen LLMs produce arguments that are more persuasive than those crafted by good lawyers. Finding the "perfect case" in Westlaw is helpful, but rarely the deciding factor.
I could see asking an LLM to reproduce your work AFTER you've done a first pass, to see if you missed anything -- that seems like a real value add. But I truly can't understand how not using it would be "malpractice." In fact, I still think committing malpractice would be MUCH more likely with AI. (Now if by "malpractice", you meant pissing off clients with expensive bills, that's a different story...)
I'm not in the legal world, so my experience is limited from a coding perspective. I don't think AI could ever truly come up with novel arguments, at least not yet, but their value for you may be to sharpen your arguments or pull on every thread that you may not even have thought about.
For example, theres a very popular coding skill set call Matt Pocock's Grill Me where it just asks you a relentless amount of questions about the feature you're trying to build. You may be able to tailor this somehow to act as the opposing counsel and have it poke holes in your argument. I suppose the closest analog would be a sparring partner.
Its helpful for me and my ADHD brain because I can go down all sorts of rabbit holes and it can easily keep track of all the ideas i have while grilling and it can send sub-agents out to investigate other claims to add more context to the grilling session.
I think the parent post meant that it's very good at writing code and operating the computer.
I think we're also biased because we've used older generation models and the latest ones are orders of magnitude better than what we had 6 months ago.
For example, for your legal work, it can:
Develop a machine learning model (CRF, running on CPU) for clause extraction from scratch,
Find training documents and training data and train the model for various provisions (you might help it here by pointing it to the directory with annotated documents),
Build an app where you upload your legal documents and it extracts the provisions you want from all documents.
Integrate semantic search and an AI agent into your app which lets it answer any question about your documents with exact evidence from the documents.
All of this from one well crafted prompt, a few skills and a workflow. It could probably complete this in one night, while you sleep.
Or you can point it to your saas of choice and tell it - 'create me a copy of this app'.
All you need to know is how to steer it through the planning/research phase.
So it's not that it's 'accurate' in legal terms, it's that it can build you any tool you want from scratch so that you can become more accurate.
I've found it depends a lot on how Claude is setup, and spending some real time on the default Private Instructions to focus on accuracy, identifying/flagging uncertainty, citing sources, checking itself, and first surfacing all assumptions and asking questions dramatically pays off in terms of getting concise and solid usable results.
Edit: add: I've found this to the degree that a relative who works at a top biglaw firm and has their internal AI (MS-based) and used to find ChatGPT and Claude actively time-wasteful, when I posed a question they'd spent a day+ fruitlessly researching (multiple areas of law and international), Claude came up with a very useful answer in ~10min — they were quite impressed.
Sizing the steps matter. Disclaimer up front: we are all figuring it out as we go along, there are no hard and fast rules that fully generalize. You just need to try and develop your own feel for it.
In my case, for some types of work - which sound similar to things you mention - I started slowing AI down intentionally. Software development, if it's something simple/common and thus "in-distribution", LLMs can zero-shot complex projects just fine. But in more R&D work, I found that errors crop up quickly and accumulate, and end result of "go and just make X" is usually bad.
Now, the problem I found with usual workflows for this kind of work is, agents race ahead and very quickly, I can't keep up anymore. So I reworked my workflow to make AI work in small steps optimized strictly for maximizing my ability to validate the outcome, and minimize time I spend doing it. So now, the thing (say diagnosing a performance regression of some complex codebase run on a cluster) may take me a whole day, where "unchained", agentic swarm would do it in 30-60 minutes - but, at the end of the day, my method is near 100% guaranteed to be correct, with every intermediate step vouched and reusable for future work, whereas "agentic swarm" approach, that hour of work, 90% of the times I'd just skim and throw away and reprompt, because I couldn't even begin to guess which part is correct, which isn't, so in the end, it was just a fast way to waste lots of tokens.
I see many people have complex systems involving skills, agent swarms with roles, agent factories, 20 termux panes and whatnot. Maybe that's the way, I don't know. Nobody knows. But it's not necessary to start getting value from LLMs. I therefore recommend my "lazy person's basic approach":
- Throw a task at LLM
- Check result, see what you like or not like about it
- If you just can't keep up, ask it to do smaller steps; if you don't understand, ask it to explain
- Just do that a lot, and don't be afraid to make both absurdly large and absurdly small asks
- You'll quickly get a feel for where AI can help you, and where it's not yet worth the time handholding it
I've used LLMs for some very small coding and math tasks, and I've found it much easier to work with on "formal" types of problems.
But for my consulting and legal work, which is "fuzzier," I struggle more with verification. I feel like I have to verify every single fact/premise in the LLM's output (this problem doesn't seem as big with coding?), as well as every single logical conclusion. Plus, even with explicit instructions I feel like the LLM is not great at anticipating what certain audiences will be interested in or find compelling.
Compared to the black-and-white nature of a computer program, all the components of a legal memorandum (premises, logical structure, conclusions, etc) seem so much fuzzier and require so much specific and accumulated context to apply reliably.
Really well put. This parallels my approach and it's good to see someone advocating a more tactful use of LLMs.
The output of agent swarms will of course grab attention, just like when you saw those same Microsoft Frontpage themes in the first few years it came out. But there's a deeper value in using the tool in harmony with your own experience and value additions.
I hear stories from someone working in basically government accounting at the county level. Which amounts to low scale manual data ETL and basic reporting.
They do rely on something that sounds like a COBOL mainframe from 40 years ago to me. They see people making the same basic clerical mistakes every year when towns and government organizations report their numbers. Like literally not being able to format numbers according to certain rules on the first try and such. Sending excel file in emails is the level of automation they rely on. They are all union jobs.
She was recently preparing for an obscure accounting certification. There's very little publicly available information to use and at most one textbook. She was asking her LLM (Gemini?) to formulate exam-level questions instead and explain the answers. She told me it worked spectacularly well.
Tasks can be automated, jobs are much more difficult because a job is not just a sum of tasks. Add in needing the context of all of reality, and I am still unconvinced this will replace the most basic of jobs.
I feel like if the gap were so monumental that he could just point to a monument of it, rather than hand-waving at "the vertical professionals".
Alternative interpretation of the numbers; there's only a subset of productive work that LLM's are 'capable' of doing at this point. That is, work that's verifiable by a machine or doesn't require verification. Thus, the cleavage points are along that axis and-- naturally-- most human beings don't have that kind of work.
The swarm-based HF hack, and the observation that 40-50% of the compute at AI labs goes into training the next generation of agents, doesn't count?
We're all over here using agents to code projects that used to take weeks and feeling pretty good about it, and they're using agents to build the next generation of frontier models — an order of magnitude more complex problem...
Whenever someone finds out how to apply this level of agentic sophistication to a problem that actually matters to consumers, then you'll have that monument that everyone can see.
With this stuff I immediately wish I was more economically literate. Bare with me. Let’s say one of these labs can use agents to replicate work otherwise done by humans. This proves that anyone with agents can eventually do this, unfortunately I don’t think its a moat like a platform (like facebook) provides, given that everyone is leapfrogging each-other constantly. Thus, it kind of becomes a commodity / low margin business?
Depending on the type of output, the value immediately (?) goes to the electricity cost of the LLMs and the cost of materials. So: more “efficient”, but without a big margin, right? So its cheaper for the workers that remain working, other workers are displaced, and capital owners now have a super slim margin business which can be instantly replicated at a place where material cost are lower (?).
Thus, it almost seems like “value add” by humans would in this simplified view be vastly diminished, and all that remains is resource wealth. Pop the materials in on one end and let the AI manufacture the product. Great news for the third world assuming the USA doesn't invade them.
Also, whose the consumer? Or are we going back to a strange medieval time where consumers are not of concern?
Electricity is important, even more important than AI (despite some rhetoric).
We don’t see electricity companies among the biggest companies in the world, except in China where they are a state owned monopoly… maybe a good analogy after all.
> We don’t see electricity companies among the biggest companies in the world except in China where they are a state owned monopoly… maybe a good analogy after all.
All over the world they're either state owned or very tightly controlled, precisely because they would be among the biggest companies in the world and even worse than that, they would be essential. If Google Search goes down for 2 days, meh, you can handle it. If the national grid goes down for 2 days your aggressive neighboring country can invade.
Final vindication of the Georgists. All value flows back as "rent", which in the 21st century includes IP licensing.
> Great news for the third world
Not sure about this in a world where there isn't demand for human labour. Especially in the post-consumer world. Very difficult future to see. Possible that it will deliver enough capability and expertise to drastically improve things; possible that without skilled human review that fails catastrophically.
I'm reminded of the Houthis asking Claude to build them a missile guidance system. We're going to see the Toyota Hilux + .50 cal "technical" combo get some weird targeting system upgrades.
Georgism is objectively correct, which is why it'll never happen. Every parasite on earth has incentives to destroy any attempt at real taxation of their wealth.
> The swarm-based HF hack, and the observation that 40-50% of the compute at AI labs goes into training the next generation of agents, doesn't count?
Most peoples jobs do not involve hacking other services in large scale fuzzing/whatnot attacks.
> Whenever someone finds out how to apply this level of agentic sophistication to a problem that actually matters to consumers, then you'll have that monument that everyone can see.
It does not seem to me that tech industry or its investors care about consumers even a little bit. Consumer preferences dont matter once you have enough or raw economic power, I guess.
What is the incentive to try? Any such application, if profitable, would immediately be replicated. If one person can use agents to do something profitable, anyone can use agents to easily copy it. Hence you have Anthropic locking down their models and removing features customers can access, to try and protect their own slim advantage.
And I thought that building your own castle and hoping your moat is wide enough was always a losing strategy in the end. We shall see how it ends this time.
I think if you build a product that takes a lot of serious AI work to both build and run, it's going to be hard to replicate just like past "good products" were hard to replicate. It's just that the bar for what's "hard to replicate" is way higher than before.
> Whenever someone finds out how to apply this level of agentic sophistication to a problem that actually matters to consumers
Isn't this a massive alarm bell? That the capability hasn't done much for consumers yet that isn't basically toys? Although I suppose the pitch was always B2B "you can automate away your expensive white collar workers" and it's not clear how widespread that is actually happening.
It did a lot for consumers who dare to use it. It didn't do much for consumers who wait for middleman businesses to offer AI-based services, but that is not because of AI, but existing business model of SaaS.
The businesses pay for tokens and try to deliver their usual enshittified experience, and nothing interesting gets created, because it cannot - enshittification extinguishes all power of AI. Using AI to full extent, in contrast, obviates most need for the business in the first place. So they're all struggling to design an AI powered product, in a world where those very products are already becoming subsumed by AI.
So normies wait for middleware companies to get their shit together or (much more likely) finally die, while people with software experience and eagerness to use AI just don't subscribe, because their favorite LLM handles that work just fine.
OAI is still hiring lots of humans for business and engineering roles instead of automating them with their best-in-class AI capabilities (https://openai.com/careers/search/). What does it tell you when they apparently don't "dare to use" their own models?
Obviously they use them. If you told me they're buying subscriptions to random bullshit SaaS like every other company did to date, and actually using them (instead of striking deals with them or trying to directly copy them into their offering), then you'd have a counterpoint.
Idk, there are entire categories there he doesn’t touch upon, which may be indicative of his own blind spot.
Coding - of course I use LLMs at this point, and as he says in the past year it has gone from “ok nice try robot” to “build me a BI tool”, and being pleased with the outcome.
Outside of coding my major use case is organising the chaos of a complicated life. I currently have it managing (including comms and form filling) the import/export process for the international move of a pet, managing pricing on my Airbnbs (dynamite - massive revenue boost) and giving me a weekly performance report, coordinating and corralling documentation for the humans for said international move, and being an extremely dogged advocate/personal assistant in a commercial dispute, which it has won.
The use cases are honestly exploding. I’m kinda at the point where next year’s multi jurisdictional tax returns are probably going to be “yo machine, file my taxes, let me check your work before I hit submit”. The year after next, without doubt.
And just to underscore the comprehension gap… I’ve been doing this stuff for much of this year now, and two days ago my wife ran to me in a panic saying “your computer has been hacked!”, as she happened to notice one of my agents filing Airbnb documentation with the U.K. government in chrome, and had no idea agents could drive a desktop.
> That is, work that's verifiable by a machine or doesn't require verification
That's where I'm at on the capabilities. LLMs have demolished software and mathematics because those have well-developed verification that's easier than doing the thing itself. Stuff that's more subjective they can produce good work, but with a significant error rate which makes it hard to let them loose on important client work without checking. Or rather, a number of companies are going through the cycle of trying that, having an expensive error, and reconsidering.
Or sometimes a company can simply pass the error cost onto consumers, who are gradually arming themselves with LLMs in response. We're going to see a productivity-sapping boom in disputes.
> demolished software and mathematics because those have well-developed verification that's easier than doing the thing itself
They are proving useful for the parts of those things that have that verification, but in neither case is that the whole of the field. In maths, for example, we have seen LLMs find answers to some interesting problems/conjectures, but haven't yet seen them pose any similarly interesting problems/conjectures themselves because the criteria for a "good conjecture" are unclear whereas the criteria for a resolution of an existing conjecture are usually supplied by the conjecture itself.
Did anyone ask? i.e. did anyone with enough background to be able to judge whether a conjecture is "good" just point it at the repo of solved problems and ask it to formulate new conjectures based on the results/proofs? Or ask it to propose new structures to study? Or to identify a previously unknown general technique/tool that it had used in a proof?
It was asked to do a specific thing, and then its brain turned off/was reallocated to something else entirely (e.g. CRUD programming or answering celebrity trivia). It doesn't just idly think in its downtime. If we want it to do something else, we need to ask.
My take as someone one tier out of the groups seeing AI going vertical (so I only watch it go up on a high exponential): the escape of "AI can do only automatically verifiable work well" and "AI doesn't have judgement" is a mirage, that will shatter within the year. AI already has better judgement than many senior people in some tech domains, on top of better skills, and I see absolutely no reason to believe it won't be able to deliver end-to-end solutions with no human involvement, that are more correct, robust and efficient than they would be if humans were allowed to weigh in.
I also see no reason for that to not generalize into every field of human endeavor.
You don't see a monument to vertical because its construction is bottlenecked on the time it takes for companies to form around it and start executing. So it's few months out still.
My proxy metric is observing the growing ratio of "AI suggested, I haven't even thought of that" to "AI nonsense I had to correct" (where it didn't turn out later AI was actually right).
This weekend I had some time with Opus 5.5 and GLM 5.3. I went over some findings in a library of Claude Mythos 5.1. These were minor, low-severity bug fixes and Mythos provided a PR to address them. Its PR was functionally correct. I still spent ~5 hours looping over them until I was really happy, even though a fully working PR already existed.
Some of the differences were:
- Mythos recommended salvaging damaged labels. I pushed back: data whose structure is broken shouldn’t be trusted
- My simpler more readable test caught a regression that Mythos’s more elaborate test missed
- Mythos allowed invalid values, citing real-world data that I could not reproduce, while I decided to follow the spec
- Mythos kept the old error types. I made every kind of malformed input raise the same error, and accepted a small, documented breaking change for anyone who relied on the old ones
Overall, I much preferred the final version of my PR to the initial draft Mythos provided. However, getting there involved quite a bit of time messing around with the code, as well as some back and forth with Opus (which was valuable).
This is a recent example that matches my overall experience working with current SOTA models (the ones mere mortals like me have access to).
TL;DR: I don’t think AI already has better judgment than (most) senior people. Obviously, that might change with future models. But that’s a hypothetical scenario. It might as well not change anytime soon.
That mostly matches my experience, but there is a caveat that you're not considering.
You don't know what good judgement calls the AI made along the way. Or rather you do but not in detail because those are the part you looked at that said, okay, this is good code and I should ignore it. Your judgment is biased because you're looking at the bad parts and not the good parts.
I'll add they also seem to make 'bad' judgement calls around which layer to implement something, often preferring downstream fixes/hacks/workarounds when upstream fixes would be better.
They also duplicate code lots rather than finding and reusing existing.
Both of these patterns lead to degradation and time wasting in complex code bases. Part of the behavioural adaptation I've made in using these tools is heading these off as much as possible at prompt-time
My prompts are often verbose to several paragraphs
This is true, but we also don’t know the long term architectural tradeoffs that are right for an LLM-maintained codebase.
For human-maintained code you manage duplication and locality of changes in the codebase because you want concepts to fit inside a human’s working memory. Keeping method bodies short is so they fit in one code window on the screen.
LLMs’ working memory works in very different ways; they see codebases not as windows of code but as blips of grep results. It might be that it’s easier for an LLM to just ensure five pieces of code are all modified in sync with one another than to safely implement a DRY refactoring.
It also differes between human-maintained codebases, too.
For some greenfield project I'm developing, 99% of the time, fixing architecture and going as far upstream as possible is the right choice.
For the codebase I'm working on professionally, doing the hacky thing is the right choice most of the time, because there's enough people changing it frequently enough, with little coordination, that architecture is fluid, and hacks only survive if they work well enough (otherwise someone will come around and fix it).
That was true pre-LLM, too, and is simply a function of scale. And ironically, it's not laziness but pragmatism - at this scale, you just can't assume code you worked on last month will look today like anything you remember. Code review process + extensive linting and testing and AI-assisted analysis filters out most of the junior-style mistakes and flag ugly hacks, so you get a moment to pause and double-check, but if it's good enough, you merge it in and go do another useful thing.
This is how you end up with a big ball of mud [1]. If the argument is that LLMs are just as capable as humans at building big balls of mud, then I'm not sure it's much of an endorsement of their use.
I've thought about that and I agree. One thing I'm wondering though is, are we really ready to go all-in on making a codebase that is effectively unmaintainable by humans?
> It might be that it’s easier for an LLM to just ensure five pieces of code are all modified in sync with one another than to safely implement a DRY refactoring.
One thing they are definitely not doing is making decisions based on how easily and quickly they will be able to maintain those decisions in the future, or considering that duplicating code rather than creating abstractions will be easier for them to keep track of because of their "very different working memories."
They're just doing a bad job. It's not some sophisticated work avoidance/efficiency in defiance of feeble human abilities to maintain what they have written. 1) LLMs are not lazy, and 2) if writing maintainable code is part of the remit, they're consistently failing. You can't convince me simultaneously that LLMs can handle everything, and also that they can't figure out how to factor their code in a way that would be a pleasure to review.
It simply doesn't make sense. If they're so good, they can make it easy to read and well-factored. If they can't make it easy to read and well-factored, they ain't so good. Saying that they're so good that the code has to be this bad is a very weird rationalization. One that apparently involves speculating about their internal states in a strange way that sounds a lot like the projection of fantasies.
I think we are already at the point where code quality doesn't matter anymore because no human can keep up with the output. I don't like the code that these AI agents produce, but what do I care when I no longer have to read the code? As long as other AI agents can continue to understand it, it's good enough.
>AI already has better judgement than many senior people in some tech domains, on top of better skills
It has better judgement in math and coding, which have infinitely generatable training data; if you ask it a domain-specific question about something like sales or marketing, its answer is often horrendous.
Last week I was dealing with a bug where we were triggering some behavior in a service in all requests when actually we really shouldn't be triggering this behavior for one of our callers. It just so happens that this caller sends a unique flag to the service.
The suggestion from frontier models was to have our service check for this flag and use that to decide whether to trigger this behavior, even though the flag had nothing to do with this behavior semantically. I'd expect this from a junior engineer. Correct now, but almost surely to cause a bug in the future as the codebase evolves.
These tools are incredible and can achieve things I would never have imagined five years ago. I have yet to see their judgement exceed mine.
> "AI doesn't have judgement" is a mirage, that will shatter within the year. AI already has better judgement than many senior people in some tech domains
No offense, but if you noticed it has better judgment than you, consider that you might not be a "senior people in a tech domain"
Yes! When ever I read comments like this it always makes me think the poster has led a much more charmed life than I have. Every institution we have that makes major decisions is an impenetrable mass of bureaucracy that takes forever. Give me the quick automated decision every time
Sure, they fix these embarrassing things as fast as they get pointed out.
Today I gave a free tier AI a photo of a burning ship posted on Twitter, and asked it to confirm it was an oil tanker. It said no, it was a bulk grain carrier, and gave a specific ID. I pointed out that the bulk grain carrier it had identified was painted black, not red (like most liquid carriers) like the one in the photo (as well as other obvious differences).
That is just one example. The free tier models continue to make very dumb mistakes all the time. Maybe the top secret models, too dangerous for the public, are able to distinguish a black ship from a red one. Who knows.
Using the paid anthrotpic models daily, it has not been my experience that they are just fixing public examples of foolishness, but are generally getting smarter. For example I have my own private, obscure bike racing physics test I use that the newer low tier models pass now.
Before fable, the top tier models didn't pass it. Now even Haiku at low effort does.
Right, if they want to sell AI as being human level, outside of a couple of narrow domains (math & coding), then how about using AI to build something that actually demonstrates that?!
It seems that every time they try to use AI to do some real world task like manage a vending machine, then it falls flat on it's face.
The first time DeepBlue beat Kasparov, or AlphaGo beat Sedol, I think everyone got it - we're got a machine that is superhuman at Chess, or at Go. We didn't need to see it beat 100 other people to get it. Yet people at OpenAI think we are so dumb that we don't get that they have built a DeepBlue for Math, and that they need to show us another 100 examples to hammer that point home.
So, next time, OpenAI or Anthropic want to try to impress us with how smart their models are, forget the math, forget the three.js animations (we get that too), and release a model that can do something more useful and not just "here's DeepBlue beating contestant #100".
Assume we've settled near a volcano recently and you start seeing ornate jewelry being sold on the marketplace for the first time, shiny colors, straight edges. You like them, gift them to others. That's your average ChatGPT user.
Over time, you see these rocks being attached to the tip of a spear and other tooling. People report performance improvements while farming, cutting, but many also caution that these new sharp rocks can kill people. These are your IT people using AI agents at work.
And over there, on top of the mountain, in secret caves, there's a new religious order. They claim to use these stones to predict weather, heal diseases and prophesize about the future. That's your frontier lab people.
When will all this be over? Never, though in time the knowledge will eventually dissipate. When Iron Age leaders were able to hand out iron swords in bulk to their armies, people definitely learned how dangerous these are, but later it didn't stop people from using iron in building construction and railways eventually.
Let's just hope we don't need many civilization collapses to jump through these ages this time.
And one bonus question: Does the blade really incite to violence, or do you just want to destroy stuff?
That's not how it is. You cannot have a business with workers while your product is able to replace workers wholesale. You would not be a business in the classical sense, you certainly would not make or need any money, and you would not have employees.
I think this metaphor works very well for tools and technology.
However AI is more than that. I don’t think we ever had scalable autonomous “intelligence” and this is where a lot of old analogies about technological progress break down.
It just reads like vagueposting about how capable internal models are. We've been hearing this for a while, yet each model that's actually released is a modest increment on the previous. The boy keeps crying wolf.
We must have lived through a very different year, but either way, the capability being referenced (agent swarms with message passing) is not released, and it remains to be seen if it will ever be released. This is because of the very obvious issue of cost. Mind you, this is neither the primary nor the majority point of the post.
Claude Code accelerated this divide. Programmers using Claude Code realized that giving LLMs access to a terminal made them feel 100 times smarter. Access to Bash, CLIs, and the ability to visit websites without the many restrictions made them powerful.
Meanwhile, the average user was still using ChatGPT, which probably felt like it had plateaued over the last few releases. The real power of these models became apparent when paired with a terminal. Agents like Muse and OpenClaw are now attempting to bring that same Claude Code-like power to everyday users through a simple chat UI.
A terminal also tells the model when it failed. Exit code, stderr, a stack trace. it reads that and tries again.
The apps normal people use mostly don't. I spent two years building agents that worked inside people's gmail and calendar, and the failures that hurt all came back as 200 OK.
- Slack puts {"ok": false} inside a 200.
- Google ignores query params it doesn't recognise, so i sent max_results instead of maxResults for weeks and every filter was silently dropped.
The agent says "done", the user finds out later, and they stop trusting it. So i think part of the gap is that agents look a lot dumber when nothing tells them they got it wrong.
I wonder if in response to this companies like google will build a sort of agent facing layer to the app in question. So rather than you building an agent to attempt to wrangle pupeteer to click css elements as a user does, it calls some apis google exposes to certified gemini user agents and these contain exit codes and can be run as script like instructions not e.g. the shortcomings of browser agent based methods such as having to wait for all content elements to actually load and the obfuscation arms race some devs have towards exposing these elements towards headless agents.
I found out due to a live bug that Apache strips out the response body for 5xx HTTP codes for security, and it's not configurable, so you can't use any of those if you want to return the error message to the user. Don't know what else might do the same, so nowadays I only use HTTP error codes for protocol-level issues, and application-level issues all go into a 200 OK with JSON just like that.
We run into that a lot using Mulesoft. 3 layer APIs, and each layer will discard error messages returned by the underlying layers unless we do tedious custom error handling. Makes it impossible for the teams using our integrations to know what they're doing wrong.
Do you mean Apache APISIX, https://github.com/apache/apisix/issues/13501? That sounds like a simple bug which existed for a short time and has been fixed. For Apache HTTP Server (what people mean by “Apache” in such a context), what you describe is thoroughly unbelievable.
I worked in environment where every HTTP 500 error was replaced by some middle ware by user "friendly" error page, and that included REST calls between services. That HTML also included one huge header image(as base64), so my logs were flooded by that junk response.
4xx errors are for when the client did something wrong and needs to know about it. 5xx errors are for when the server is having problems and the only people who can do anything about it are the operators, who should be looking for errors in the logs rather than the HTTP client.
Truly "application level" issues are therefore correct as 200, although some application communication issues lead to 4xx errors that should be handled as such. But the case of an application trying to communicate via 5xx errors should never have been the case anyway.
Hmm, that's poor API design. If the big companies can't standardize their inputs and responses, then the small projects aren't going to either. Sounds like there are lots of small issues like this that will roadblock AI-powered automation for some time.
At this point I'd expect SOTA LLMs to have this in training data: most external APIs are protein-agent-made slop, you can't trust them to obey any reasonable standard, so you have to learn to spot failures in responses on your own.
for a lot of things ChatGPT has 100% plateaued, especially if you are just using it as a "better Google" search engine. ChatGPT still makes mistakes, makes things up occasionally, etc. that's the whole jagged frontier thing. what recent LLMs have truly achieved breakthrough in are fields that have results that are relatively cheap to verify, such as maths, coding, etc. so people outside of these fields will have disconnects with where LLMs really are right now.
also very important to note that the massive increase in capability in these fields doesn't necessarily immediately translate to increase in productivity. because the code and math proofs these LLMs write are legit _alien_ to us. if you just want code that do something (e.g. a port of Bloodborne that runs on PC), then they can do it. but if you want something human can understand and maintain, that's going to be a nightmare. and that'll surely slow down adoption.
The 1%ers are in the vertical, but the question is vertical to where?
It needs to a potential field with practical, economical, "real life" attraction well. I.e, robotics, real economic efficiency gains, manufacturing novelties.
The worry is that the 1% is attracted towards a non-practical money hole. I.e: Token burn for the lols, sophisticated software systems that dont provide actual value outside of giving NVIDIA cash.
I'm normally not the cynical type, and I am appreciative of Karpathy's efforts to educate people on developing language models, but this is exactly how it reads.
Every once in a while, everyone needs to touch grass.
It is fine as it is to make a boatload of money. All they have to do is seed ads into it. Look at how much money social media brings in from that model of "free/useless product" as flypaper to bring in eyeballs for the real money which is ad revenue. Adsense 2.0.
Vertical on the curve of throwing money away for a share of the pie, but if AI gains are zero sum, then all parties will end up spending their profit on AI to fight each other, just like most companies spend more on marketing to acquire customers than to deliver their product.
It all sounds good and he is on the right track but the elephant is not that. The problem is that ordinary businesses do not see ROI from dumping cash on this direction. If someone wanted to have similar access to models as internal folks in these companies, then you’d need a bit of cash to run such swarms of agents, not to mention people who would know how to do it. Another thing is that you’d need your own frontier models deployed in the place where the inputs would not be exposed to a third party, as it is important to feed the internal private info to reason about. And if you are like me then it is a constraint of capital and people who want to invest in you that is also a factor, why would they invest in you when openai/anthropic will be able to copy your ideas in a few months?
the question is how much of this inability to realise roi is technology vs skill. i have plenty of clients who drop 10m on an enterprise transformation project but couldnt (currently) manage the power a 10m nvidia rack would give them, its only a matter of time before they can or their competitors can.
The economics part is the one I feel the biggest disconnect in the Tweet.
If teams had the same level of access as the "top 5000", you'd see different use cases proliferate. Same as we see an explosion in productivity when we advanced public education (more people can read, more people can contribute, more talent is discovered), same as we see an explosion in creativity when we advance distribution (from 3 broadcast channels to cable to streaming), same as we see gains in athletics with population size and diversity.
A lot of SAAS is easy to duplicate but not a lot of it is. The barrier is market/mind share. Something like Slack is easy enough to duplicate that many companies did. But everyone still uses Slack because that's what everyone is familiar with.
Most professional engineers, especially those in electrical or communications, have the tools to understand AI.
I recently started trying to understand how AI works. It turns out that I wrote an "AI" in the late 1990s without realising it. The "AI" was an MMSE (minimum mean square error) radio receiver for CDMA.
This receiver consisted of an FIR filter (mathematically a linear combination) with trainable coefficients (weights). The coefficients started with random values. A sequence of known training symbols (tokens) was fed in and a gradient search was used to adjust the coefficients to achieve MMSE. The unknown data to be received (classified) followed the training sequence, at which point the training was turned off and the system ran as an FIR filter. Once trained, the FIR was a matched filter.
In AI terminology the above is now called a "layer". Current AI cascades multiple layers, in which case you need to pass the output of each FIR/layer though a non-linear function. If you don't have the non-linear functions at the end of each layer the multi-layer system is mathematically equivalent to a linear combination, which could be implemented with a single layer, losing degrees of freedom.
I write this because many electrical/communications engineers probably know how AI works, but don't realise it.
My still incomplete understanding is that LLMs are "just" a graph of connected layers. More complex that the above MMSE receiver, but trained and operated in the same way. The smarts in AI are:
1) Arranging the layers and the connections between them.
2) Having access to the right training data
3) Having the computing power to perform the gradient search (aka training)
My point is that most professional engineers already have the knowledge to understand AI/LLMs, and potentially already do know how they work but don't realise it due to a difference in terminology.
Sounds interesting. How did you back-propagate the coefficient gradients? Just tweak them one at a time, or was there a global algorithm that worked on the whole filter at once?
From memory the entire filter was updated in one symbol period (ie. global)
The symbol rate was 1Msymbol/s and there were 66 taps. The circuit (in an FPGA) ran at a clock rate of 66MHz. In each symbol period the error was calculated based on the output symbol, then an iteration was done over all 66 coefficients, using the error to update them.
Edit: I also remember that to make most efficient use of the FPGA I figured out the minimum precision required in the maths whilst preserving convergence. It was a 4-bit mantissa plus an exponent (3-bit or 4-bit exponent, I can't remember). It's interesting that today's AIs seem to have also figured out that things still work if 4-bit floating point is used, though that is 4-bits including the exponent?
> things still work if 4-bit floating point is used, though that is 4-bits including the exponent?
It is generally 4 bits amortised across an entire block. This works more like block-based mage compression schemes - you put a bunch of statistically-similar floats into the same block, and use your handful of bits to index into a codebook containing the deltas from a 16/32 bit reference value for the whole block
Even if the model is simple, it is happening at a scale that is producing alien results. You understand the basic principles, but that doesn't matter: you, well rather no one, understands the final product.
The difference with your FIR matrix and a deep net matrix is the latter generalized to trillions of parameters i.e. a trillion dimensional space to approximate general, computable functions. The machine learning ideas are fundamentally the same at a high level, just the architecture (nonlinearity whereas in engineering we prefer analyzing linear systems) specific details and tricks and such.
In fact at least at Berkeley a current professor teaching deep learning is a signals expert (Prof Sahai), his lectures are on YouTube.
There is a lot of terminological confusion in the field. The fundamental difference is about how the "AI" learns. And who, if at all, supervises the learning.
Some systems are fully formed AI. Often based on heuristics. Humans have encoded the rules into the system. Others do limited learning based on pattern recognition. Still others learn based on tagged data. And LLMs don't need even that. In a stream of N tokens, the trainer has access to K+1 when it is training a model that knows 1...K and can check if the prediction was correct.
AI literally refers to any artificial thing that feels intelligent to us: does some thing a simple bash script will not. Your rules based chess engine is AI. The bot you compete against in single player Age of Empires 2 is AI. DeepBlue was AI.
ML is where things become interesting because we teach machines to learn.
NN is where we borrow terminology from neuroscience and create "neurons" that learn.
DL is where we stack neurons deeply in layers.
I am trying to make sense of all this myself.[1] So I am building a curriculum for it as I learn. So much of the teaching material in the wild revolves around torch. Makes it useless if you want to peek inside the actual machinery.
I feel like this statement is too anthropomorphizing. We don't teach a computer anything. We write normal computer programs that can process data(training), turning it into what is essentially a configuration file(weights/parameters) for another normal computer program(inference engine). Upon receiving a request, the inference engine is invoked and passed the input, then it processes that input and its configuration(weights), producing an output. "Thinking" is just feeding the output back in as an input, allowing the model to revise it and refine it before it's returned to the user.
There is no teaching, nor learning. There is just data processing. There is nothing happening between requests to the inference engine. It's a relatively simple system altogether, but when the training data set is large enough it produces unbelievably good outputs.
With this in mind I really don't understand how people think these programs can be conscious. They're not doing anything fundamentally different than any other computer program. It's just crunching numbers. You might as well argue that a computer game or 7zip or a chess bot or whatever is conscious.
I know you specifically didn't claim they're conscious, I just thought it was worth mentioning because I think it's ridiculous how many people will seriously argue this as if there is any difference between "ai" and other programs. There isn't. It's just a set of instructions for doing computation, as computers have always done.
3D rendering is actually quite similar. There is a bunch of files (input/configuration) describing 3D meshes, textures, lighting etc which are fed into a rendering engine which processes all that data and turns it into a 2D image. Similarly to AI, once the input files are sophisticated enough, this can produce mindblowingly realistic and beautiful images, and you can run it in a loop to make video or make it interactive, moving the camera around in real time etc. But at the core of it all is just a bunch of files containing a bunch of numbers, and a program that reads those files and does computation with them.
The current leading materialistic model of consciousness is essentially that human consciousness is a computation in itself, and that the brain is basically a very very complex computer. So it's not at all illogical for someone who holds this view and who fully understands that AI is just a regular computer program to also believe that it could be conscious.
Of course, I'm not saying that you have to believe any of this - even defining what consciousness is is famously elusive. And there are a lot of people who think consciousness is not materialistic to begin with, or that we have not yet discovered the scientific nature of it.
I definitely agree that consciousness is mysterious. But I think the assumption that AI is conscious begs the question "then why isn't 7zip conscious?".
Is a simple calculator conscious? How about a mechanical calculator? Is an abacus conscious? Are they only conscious when executing a sufficiently sophisticated program? What if I hypothetically had infinite time and patience, and performed the inference on paper with a pencil. Would it then be conscious?
Good examples. We don't know what human intelligence, understanding or consciousness is but we do know what all of those examples are. And we do know what an LLM is. Like you said, it's a data processing machine and we know what algorithms it uses. It's more productive to discuss it in those engineering terms because we've done plenty of data processing, statistics, control systems, curve-fitting etc. before. There's no need to make this jump to consciousness/magic except for marketing or fantasy purposes.
Well, the exact same question applies to biology. If humans are conscious, is all life conscious? If it's just the brain that's conscious, is every part of the brain conscious? Are individual neurons conscious? How about other cells in your body, or tissues or organs? Our stomach has more cells and more neurons than many full animals - is our stomach conscious?
The answer in the "consciousness is a computation" model is that consciousness can arise in any computational medium. But the model doesn't say that any computation is conscious - it says that consciousness is any of a set of specific computations, any computation with properties X, Y, Z whatever. So it's perfectly possible (according to this model) that a GPU running game graphics is not conscious, while a GPU running some future even more advamced LLM is conscious. Or a network of GPUs + storage + whatever.
The computational theory of mind is the leading theory for pure cognition. There is no leading theory of consciousness, although some people also favor the computational theory of mind for that as well but this is far less popular.
> I feel like this statement is too anthropomorphizing.
That's number one mistake preventing people from getting a good grasp of how LLMs work and how to use them effectively.
That is, not anthropomorphizing is the mistake. It gives people bad mental models.
LLMs are not people in general, but behaviorally and in terms of system design, they may as well be. They have same failure modes and cognitive characteristics humans have, and are very much unlike regular, non-ML software tools.
Not really sure what you mean, I have the view I presented and I'm able to use LLMs effectively. The understanding that they're not human allows me to treat them as such - I don't say please or sorry or thanks or other pleasantries, I just instruct them in the simplest most efficient way I can. Humans would find me rude, be annoyed if I give them lots of work for little benefit, and say the same things over and over etc.
> "There is nothing happening between requests to the inference engine."
This is true for vLLM/SGLang/etc but LayerScale is making use of idle GPUs. A new novel inference engine for agentic workloads. Our advanced scheduler enables double the workload on the same number of GPUs (e.g. prioritization of on-demand queries and background processing of batch workloads simultaneously). Let me know if you want to try it out!
We humans have the habit of falling into the trap of anthropocentric views and explanations of the universe and everything within it. The history of AI is the history of moving goalposts. I don't particularly care if they are conscious or not. Humans have a history of treating even actual, conscious, intelligent beings as chattel or food. So I don't really see why our attitudes would change if we somehow realized that LLMs are conscious.
I have been thinking about this for a while in the context of the quest for AGI. If all we want is something to take over some parts of our intellectual labor, then LLMs are already there. They can communicate with you in a language you understand and do stuff that you command them to. Often, on their own.
AGI will have its own telos. An LLM does not (at least I refuse to believe that it does).
So, yeah, you could think of LLMs as programs processing data. But there is definitely some emergent behavior you notice that emerges when the LLM runs. How you interpret all this depends on your view on consciousness and intelligence.
I also don't really care whether they're conscious, but I do think it's worth challenging the claim that they are.
LLMs are programs processing data. Lots of programs exhibit emergent behavior. A simple example is Conway's game of life. It can be programmed in less than an hour, exhibits lots of interesting behavior. In fact you could run LLMs inside of it if you wanted to.
I'm not denying that LLMs are unbelievably cool and useful. It still blows my mind when I send a half-assed complex screenshot and a vague description and it just gets it and fixes the issue in seconds. It's like magic. But it isn't magic, it's just some relatively simple algorithms processing a crapload of data, and I think it's useful to understand and remember that.
Dropping in to mention belief propagation networks, those things are likely being used right now by the modem on whatever device you use (WiFi or Ethernet, both use it!), and are surprisingly AI-like.
My gut feeling is that the AI field is eventually going to discover how much overlap it has with information and communications theory.
Question to any AI researchers out there: Can a layer in a trained AI be treated as a form of "matched filter"? Is that a concept that is used in the understanding AI: to view it as a cascade of matched filters? The next question is "what is it matching?"
I would agree that a decent EE major is equipped with knowledge sufficient to grasp both Transformer architecture and gradient descent training. However, these elements do not fully cover the key parts of modern AI systems and also are not enough to predict the AI capabilities. You have to account for reinforcement learning from human feedback (RLHF) and with verifiable rewards (RLVR) which are crucial elements of the wider model creation/training process. These do not have obvious analogues in other engineering disciplines, but are still within your grasp if you are decent CS/EE/adjacent major. The issue is that you still can't deduce end result AI capabilities from first principles. You may have prefect understand of y = softmax(QK^T/sqrt(d_k))*V and it will give you no clue of the effectiveness of a given AI model. The range and complexity of training RLVR tasks may hint at the model capabilities, but the final results are very much emergent.
This is like saying with chemistry, amino acids and human DNA you now have the basis to understand human culture.
The problem is one of complexity, the whole being greater than the sum of its parts. Electrical engineering creates a thinking that is opposite to what you need for understanding biology or human culture.
This is relevant because LLMs have basically swallowed all written human culture. Engineering is about creating parts that interact in a clean, well-defined way, like a MOSFET transistor or a REST API. The goal is that that when you combine the parts, and they all operate within specs, you can still predict what the whole system will do.
So saying LLMs are just a step away from the Pereceptron (or an equivalent filter) is mathematically true, but almost useless. It's like saying that a modern 3D game is basically just transistors, logic, CPUs and memory, nevermind the pattern in that memory or how it came to be. You really can't ignore all the cultural inputs and feedback loops if you want any kind of useful understanding what this thing is.
This is like saying you understand the basic principles of internal combustion engines.
Sure, that gets you some level of understanding, but in no way prepares you to be even a mechanic, far less an engineer, let alone a researcher/engine innovator.
Tbf OP said they have the tools to understand what was going on, not that they could build a model to compete with OpenAI tomorrow. EE (especifically DSP and communications) uses a lot of the same building blocks as ML so yes it does prepare you.
You are right in that the emergent abilities come from humongous amounts of training data and compute which only a handful of companies (and the people working in them) deal with today. I'm sure there's a lot of secret gotchas which only arise at a certain scale so one cannot self-study for them. Companies also guard these wherever possible because of economic incentives. But that applies to most other engineering disciplines. You can take a course in civil engineering and have enough knowledge to reason about why a skyscraper will not fall apart in the wind; doesn't mean you have the ability to build one.
This is a case of https://xkcd.com/2933/ . Your two paths are "Oh no. Simple layers can combine to produce complex behaviours!" and "Oh no. Modern layers are not this simple!"
At best trying to scale this understanding from what you have stated might (gratuitously) get you to a multi-layer percepetron. Add a few ideas and you would hit a convolutional neural net which represented the frontier a decade ago. However this is still so far divorced from the modern frontier of AI, transformers, because the leaps are more focused on attention mechanisms than anything else. And even just understanding a transformer is merely a building block.
This is basically understanding a building block (A) of a building block (B) of a building block (C) that builds D and purporting that you can understand D from the building blocks of A. The equivalent is saying that knowing Newton's laws is enough to understand how we get to the moon. It might be fine for a hand-wavy idea, but its just not enough.
I said the other day that I feel like I’m the edge of a wave that will (probably) make its way to the “general public” in time and I don’t know what it will look like when it does. Will “normal” people be able to create what I’m creating today in XXXX time? Or will it be another case of “everyone can cook but people still go out to eat”?
The build vs buy decision has been turned on its head for me, but it’s unclear if the “normal” people will make that same decision or be capable of leveraging the tools the way I am.
My experience has been that most people don't even understand what computers were capable of before LLMs; I work in a chemical company, and I can count on one hand the number of people who knew SAP has GUI automation built in.
Don't even try to explain that "copy data from X program to Y program" is something computers should have been handling for the past 20 years. It's been someone's job the whole time.
We haven't cleared the hurdle of "Processes a 100 line python script could automate in 2010", I don't think most people will be leveraging LLMs to their potential by any stretch.
That's very true, and I think you are probably more right than I am (in thinking this ability will _not_ be evenly distributed).
I do wonder what happens when:
"Help me find a tool to do XZY" is responded to (by an LLM) with "Here are a couple of options but I could also build one for you...". But at the end of the day no everyone wants to be a "Builder" or manage it. They are happy to exchange some money for someone else to handle it for them similar to the "everyone can cook but people still go out to eat" concept.
Don't think the 0.2% really are in agreement that "large, complex projects that used to take them weeks/months can now be completed by agents with a prompt".
The divide seems to be between people who think you may need to understand the codebase for maintenance, potential future token costs, optimisation, code erosion & being wary of the assumptions/mistakes that they have seen LLMs make, and people who don't review code and are confident that the AI has made no mistakes and will be available for next-to-nothing forever.
It's certainly allowing the latter to move fast... but will their clients be happy if/when they break things. We'll find out soon, I'm sure.
We've got a customer success manager using Claude Code and getting PRs rubber stamped to prod where I work. I've given up caring about the quality of the code at this point. Executives won't care about code quality until the ship is literally sinking. Oh well.
I feel like this was always true to an extent, now product and design are just more tolerant of it because leadership is cracking a whip over their heads because things are supposed to be much faster “because ai” so things like pixel perfect hand offs, adhering to existing design system, making sure you are aligned with the brand, copy is reviewed with marketing go out the window. Now everything is shipping in hackathon mode with an eng driving decisions they really should not be.
I worked at Meta and we had AGI in the form of very competent ICs - getting anything done was often super painful because "org priority" this and "buy in" that
Ah, you're point that AI can't solve the problem of conflicting requirements is right.
But you're strongly echoing the parent, who's then essentially saying "just don't talk to other people and you won't have conflicting requirements", which I'd say is strongly wrong. Complexity involves conflicting requirements. These will exist in a single developer complexity gets to certain level.
It's not that we're assuming the AI has made no mistakes. It's that we're confident in the general capability of the models such that we're willing to accept it'll make some mistakes, but working together with it, we'll still achieve our end goals.
I presume you also know your own code has mistakes, too? You wouldn't refuse to even look at the code of a colleague who has ever made a coding mistake, would you?
That's not really my point... my point is that it's an unsettled debate between speed and caution, and devs are not unanimous on it, unlike the claim of the original post that seemed to think everyone was reporting the same gains.
The divide with developers at the moment seems to be about people who believe in review & are being cautious (who are not seeing major time gains according to all reports I have seen), and those who trust it to be able to fix things forever and will no longer have to worry about understanding the code, so are happy to atrophy those skills.
Yes, my code will have mistakes, and I'll use LLMs for some tasks. I also review and have a mental model, which means I can easily fix them, no matter what happens in future with AI. Reviewing also means I'm seeing the limitations and common mistakes of LLMs.
A no-review approach will be far faster than this, obviously, but if scbench.ai is still true on the frontier models, it may come back to bite people when they have no idea how it works and the context becomes too big to maintain.
Yup. Also code review is not just there to check your work. It's forced communication, in that at least someone else knows about the changes you're making.
It's also a critical part of your security model. "Nobody can deploy code by themselves" is shockingly effective at preventing rogue actions.
Speaking of communication... Almost no one mentions talking to non-devs. That's one of the reasons I can't just rely on any of these tools, I have to remember and understand everything well enough I know how they work and can explain things and help plan ahead when talking to product people in meetings.
I think a big part of that divide is between the people who are inherently limited by other factors on the rate of change they can implement.
If I'm being snarky - the people who are happy to run without reviewing the code probably mostly are able to because there isn't significant value attached to it downstream. Not many stakeholders, not high value transactions, not complex downstream integration or impact. And yes, in their friction free lives, AI can massively accelerate things far beyond what it can do for the rest of us. But don't infer that applies everywhere else where those assumptions don't hold.
Would disagree on the terminology, but not the spirit.
I’m sure people are building things that have specific KPIs based on client preferences. As long as those preferences are being met, then their question becomes “why does it matter what the code quality is?”
Perhaps it’s simply liability and penalties? If getting something wrong means your client is going to sue you into oblivion, then perhaps that is where people are focused on making sure its right, as opposed to making sure its fast?
Poor code quality degrades your ability to make future changes without breaking things. Maintenance is the common and I think correct metaphor: the precise location of all the bolts in a machine won’t be part of even the most detailed requirements spec, but if there’s any components you might want to replace in the future you’d better make sure you have non-destructive access to them.
This is a different argument from Karpathy's prompt-and-forget one.
I personally have had good success with using agents as virtual pair programmers following some simple rules for how to make that process productive, but that is a very, very different approach. One that recognizes that both humans and agents are prone to errors and attempts to mitigate the problem rather than expecting it will magically work out through the power of AI.
I think of it in generational terms. Opus 3 required me to prompt and edit, iterating over functions almost one at a time. Opus 4 started to show agentic capacity but it largely bloated code and still needed frequent course corrections. If it churned for more than 15-20 minutes you knew something was off.
Opus 5 takes that and changes the game. A simple prompt goes through research, design, verification, review and self-corrects constantly. Agentic workflows can run for days without stopping, without losing the thread or devolving into slop.
I can’t even imagine what Opus 6 is going to bring.
The biggest thing holding AI back is people. People can only change so fast and yet AI doesn’t care. It just keeps accelerating.
I am sorry, but current benchmarks do not bear out that claim. Yes, frontier models can fairly reliably one-shot standard CRUD apps (and you don't even need frontier models for that), but when we're getting to harder problems, reliability nosedives when they're operating autonomously.
This attitude is exactly why AI bros don't make successful software lol. "Where's the software?" This. This is why there is no software.
Modern software development practice is built around that humans do make mistakes. Frequently. You don't cross your fingers and hope for the best, you assume it will happen and you put processes in place to prevent errors, catch them early, and make them easy to recover from.
> And I know at least 1 startup, which only hired a dev after having 20+ B2B customers. They sure do not go around advertising themself as vibe coded.
Let me tell you something, I don't know jack about this new world.
But I do know about the old world. And in that old world, when you had 20 customers that actually needed your product for their business, your team of 5 people who built YOUR product are eagerly trying to talk to each one of them to figure out why your product works for them, how they found you, what they need now and in the future from your product. They reached out, took people to lunch, they went to meetups and hackathons and boring ass sales pitches to make connections.
Tell me about how all THAT stuff happens when your core dev team is agents in the cloud ?
Have the devs been financially rewarded for their speed or just given more work?
Have the customers been charged less?
If "you don't have to review" really is true then the economics are an even bigger nightmare. The value of code has decreased, the "SaaS is dead" claim becomes real, and any good idea can be easily cloned.
My colleagues write 10x less code for the same tasks (when they don't (ab)use AI) and don't reach past abstraction boundaries so that it's possible to actually reason about and review their work. When the problems get hard, they apply more care and attention to detail. Even if LLMs had the same bug rate per SLOC (IME it's higher), and even if those bugs were the same severity (IME they're worse), you'd still expect 10x bugs just because they can't be arsed to write a reasonable amount of code.
Almost all my code nowadays is AI-written, with me just contributing the concurrent data structures, a good spec, and some review and touch-up. I'm not _that_ much of a Luddite about this stuff. I'm not blind either though. Your "working together" comment does a lot of heavy lifting when arguing against a post saying that leaving the LLM to its own devices is a bad idea.
If you are reaching past abstraction boundaries and making spaghetti code you are holding it wrong.
We create higher quality, more testable code, with less complexity and less LOC per functional unit than we ever did with humans. By a long stretch.
Documentation driven development with constant naive review, delta review, documentation/code coherence review agents, test coverage review agents, development standards and architecture review, etc. We write code once, then review and revise each module from a variety of different paradigms until every line, every construct, every function, every interface is optimally constructed and documented. Agents learn to write code that passes with less flags by having it pushed back to them until it passes, and revising their learnings files.
By the time code makes it to a commit it’s been brought into compliance from a variety of perspectives and tested as completely as possible, along with mutants that test the tests. By the time it gets to a merge gate, most code has been through at least 50 reviews. Then we periodically sweep the entire codebase for cruft, vulnerabilities, corner cases, etc. If we find smells or clever sections, they are eradicated.
In total, the hot paths and critical infrastructure sees hundreds of reviews against thousands of rules and best practices.
Sure, it’s expensive, but still 1/4 the cost of regular development, and much, much fewer bugs.
Unfortunately, most people are out there just vibin, even at the big corps, and it’s a dumpster fire.
It’s easy to produce garbage software, but it’s not mandatory.
How is that 1/4th the cost of the regular development? Did your pre-AI code ever hit 50 reviews? If not, it’s impossible that it was slower. If you are talking about agent reviews, it’s not expensive.
I'm not the guy you're responding to, but I have the same experience. If anything, 1/4 cost underestimates the savings.
The analogy is like pointing out that you can have a robot vacuum clean your floors every day. Before the robot, you'd do it maybe once a week, just no time - "too expensive" to even think about doing it daily. The robot makes it happen all the time for a tiny, tiny fraction of the cost.
Meanwhile, the vacuum requires maintenance, repairing more frequently, as it runs more often. Power on time of motors is 7x as much, eg daily instead of weekly. And the robot takes more care, so each clean is 2x or 3x run time of a human. So 21x usage rates, and run time is parts wear and tear time.
Edge cases handled automatically by a human doing the task (getting stuck), means for the robot case that manual interaction is still a thing. Occasional, yet not making autonomous behavior, human free time.
Emptying the vacuum bag is a thing too, some things are not abstracted away.
Every time you add an entirely to a task, the task time goes up. Coordination is a time cost.
The savings you suggest are often not so stark. Not even for our friendly vacuums.
For sure. Still much cheaper than vacuuming every day.
> getting stuck
I appreciate that this is a function of the specific environment and robot you buy, but as a personal anecdote, this doesn't happen to me in practice. There was a setup cost, but once that was done, it doesn't get stuck.
> Emptying the vacuum bag
Not that frequent at all, considering that daily vacuuming keeps things remarkably clean and thus little is added to the main bag in the docking station each day. This is also far less work and far more preferable than doing the vacuuming myself.
I can't imagine living in a place that has too little dirt that you don't have to empty a bag but so much that you have to clean it everyday. I do know places where maids clean the floor daily because why not, but that is different.
That analogy might be good, but doesn't resemble reality: the robot vacuum I had never made it anywhere near as clean as when I just hired people to come in once every two weeks to vacuum and clean floors.
I think you are both right. But I think the parent argues that the robot still keeps it good enough. And IME significantly better than a week+ into a 2-week period between the maid services. My 2c.
Don't use analogies to argue on the internet. Analogies can never be perfect, and this just gives people the chance to focus the argument on those imperfections in the analogy, rather than stick to the main point.
AI's default state of affairs is to reach past abstraction boundaries and apply whatever cheats it needs to pass its tests quickly. I think your argument is that sufficiently smart harnesses can mitigate that defect. I disagree, but not as much as you might think.
You can (and should IMO) apply additional guardrails to generate better code. If I'm interrupted every 10min to review garbage then the AI literally doesn't make me more productive, and I'd be better off coding normally and using it as a glorified stackoverflow/searchengine. This isn't how I see other people using the tools (largely because of costs AFAICT) -- which is absolutely part of my complaint; people are holding it wrong, and I'm left to pick up the mess -- but I personally refuse to deign to answer an AI's question till my normal list of problems is addressed. Nevertheless, I still disagree with your conclusion, that automatically addressing a small enough list of issues to fit into an agent's context results in good code.
I'm not sure if you're interested in a challenge, but one hand-coded thing we did well at $WORK was writing the fastest protobuf parser for this schema (and, generally, any other messy, deeply nested schemas) [0] (assume many adcandidates, deals, and segments if you're generating synthetic data to test against). My ranked priorities are (a) I need to be able to easily prove it correct/safe for all inputs (literally all, not just the few hundred billion we received yesterday, since this is open to the public internet), (b) constrained by (a) I need it to have as much throughput as I can possibly manage for many of my workloads, and (c) a second implementation optimizing latency instead of throughput is used for the rest of our software. AFAICT, AI is currently incapable of even understanding what the bottlenecks and priorities are. When I manually describe those, its solutions are a dumpster fire. If I describe exactly what I want coded in excruciating detail then it does reasonably well (not by default, but with enough of the automated tests you've described it doesn't take too many iterations with a person for it to be shippable), but at that point I've already done all the hard work, and if it took 2 weeks of understanding and design and then 1 day of AI coding vs 2 days of human coding then I haven't really bought much.
A full solution seems like a tall order for some rando's internet challenge, but are you interested in seeing if your AI can even identify the relevant problems and propose plausible solutions to them? I'll make sure to revisit this comment thread and reply with critiques and follow-ups if you're interested.
I believe both of you are telling the truth. Certainly, you are not thinking about the same kind of tasks. For the gap between hairy realtime concurrent system programming, where a bad design decision can be fatal, and the simplest over engineered CRUD where nothing really matters can accommodate and explain the gap in shared understanding of LLM abilities.
On the other hand, I’ve heard a direct quote from the longtime (25+ YoE) colleague of someone I know, that in their mostly embedded/some realtime work, the LLM code quality (specifically, this was about Opus 5.5 IIRC) has now surpassed his own. And that person is an excellent programmer, and the model is faster and cheaper!
So it’s definitely not just standard CRUD that is on the way to – or already – “solved”. As with all things, the truth is probably somewhere in between.
In my experience, it's more about "local" vs "large-scale" thinking. An LLM will be better at finding a bug that's "visible on screen" (or on the context window), whereas some design decisions that are hard to even put in words can be super important in practice (like, in C, how you manage the lifetimes of various objects, how you organise your threads and locks...) That's not purely "word manipulation", and there are not many good explanations of that on the internet, so it's understandably harder to learn for an LLM.
A bit like chess vs go.
They will get there quickly I'm sure. After all, AI beats us at go, too.
Our work is in embedded systems integrated with SCADA controls in a critical environment where data integrity relies on precise time synchronisation across sensor networks and databases. It’s an environment where we count processor cycles and joules , with very little room for error.
The difference is testing. Testing at every level. If crap code can’t move down the pipeline, you don’t get crap code. If a company is used to YOLOing code out the door with human developers and they try the same thing with LLMs they are going to have a bad day.
100% this, but also - linting, fuzzing, and test harnesses. Agents can help build out huge deterministic simulators and fuzzers that were simply too expensive / not feasible a year ago to build for most software - and then happily bind themselves to the results moving forward.
Agents just won't build it out by default - you have to ask for it.
You're kind of saying the same thing as the parent but in a smug, insulting and disagreeable way. Many acknowledge LLMs are very useful but without needing a jargon-spew that kind of "smells" like LLM output itself.
The amount of crazy stupid bugs in the 8-year-old codebase I work on is shocking. Every time I work on a new subsystem I end up fixing 20 incidental bugs; lots of N+1 query problems, completely wrong type annotations, all kinds of things.
This _is_ how strict types work. Problems creep in when people introduce lax types into the strict type system.
dict[str, Any] is a type, and sometimes it is the right type to specify, but most of the time it's used by someone who wants to shut up the type checker rather than think about the right type.
In my case it's actually that there was no type annotation for the first N years of the project (long before I joined) and a couple people started using them in advisory mode but with no enforcement.
Just a huge mess. One reason I'm not a big fan of python for large projects.
I’ve had this experience as well where it is finding all kinds of subtle bugs. The trouble is when it lulls you into checking its work a little less carefully and you let some of its bugs slip through as well. Definitely had a few of those happen to me at work.
The problem is also that enticement thing it does. Over representing the importance of some bugs or introducing more complexity to squash something that doesn’t matter.
If you aren’t 100% on it the entire time it’ll make 4 bugs for each solution and then write tests that hand hold the nonsense through. The juice isn’t always worth the squeeze and it messes with your ability to parse it all.
They're there, but in my experience, lots of them are for edge cases that almost certainly wouldn't have been hit in prod. I think this is a major reason why LLM-generated code can be much longer for the same problem than human code. Humans won't think of every edge case, and occasionally that bites them in the butt, but they also won't cover all the edge cases that happen rarely enough that they will not matter.
I'm unsure what the right solution is. I currently tend to let LLMs resolve these bugs because it does fix genuine issues that would have affected real users. But this also means that LLM-generated code is often twice as long as what I would have written, and much of it is for defensive behaviors that will never matter in prod.
I noticed the same thing at work, the code is very defensive, and often the case could not happen in prod, so my solution at the moment is to make it prove that an issue can happen and try to measure how frequently it would happen with real data. That's often enough to realize that a particular defensive measure is not needed and the code complexity decrease as a result. And it's something we can automate.
I think it's an important point because the more complex the code become the harder it will be to maintain it, so hopefully this process will improve.
The priblem is, actual bugs that may exist end up being covered in an avalanch of code to cover possible and often purely hypothetical edge cases.
Was the actual bug fixed? Hard to tell among the 150000 new unit tests and a million lines of new code.
---
Anecodtally: ran an "adevrsarial review" on a Stripe integration for a side project. Dozens and dozens of issues, thousands of new lines of code added as a result of the "review".
Looking at issues and code closely: none of the bugs or issues were bugs or issues. None of the flows were relevant to the project at hand, and would've never hit the possible bug paths. And on top of that a lot of functionality was duplicated or triplicated in the "fix". Had to revert a lot of commits.
Yes, working on a customer's 20 years old software architecture in logistics it's fascinating in a horrifying way how many of those "it only affects 1-100 supply chains" bugs are in production.
It's less gray bearded wizards than the fact that they used cheaper crazy cackling goblins to write the software.
The challenge is to determine if a finding is an important bug or just a dead business process now never called in production. That is knowledge the AI does not have, and it requires careful, exhausting work of going through the problem.
I don't quite get what's the argument you're making here. What's in contention is whether or not the 0.2% is in agreement.
Are WE the 0.2%?
Because if not, what's the point talking about the merits of AI's competence in of itself? Accomplishes nothing, we're he outsiders. And if we are, then either you're fishing for disagreement and the post you replied to is actually correct, or you get unanimous support to your statements, and establish... Nothing, the in-group could be delusional in a way that does not actually function as an argument against the whole "So where's the software? We're the AI led market disruption instead of the AI-led shovel makers?". The punchline was about the clients being unhappy.
Last week, on my pet project that I am using a harness to almost entirely code (so I'm by no means a Luddite), it/Astra decided to dump ALL application state into a single row+column in SQL as JSON. That is a mistake that I would, and a completely new grad, would _never_ make. It is profoundly incompetent.
This week it did it a second time on an unrelated project.
"Some mistakes" is putting it unbelievably lightly. It is shitting out complete and utter garbage, and if you think it's doing a reasonable job (or a job that can be fixed later by a more powerful model), you're either not paying much attention, or your own ability has already heavily atrophied.
Karpathy put it well: "gap of understanding of LLM capability," but he's seemingly in one of the less fortunate groups himself.
> You wouldn't refuse to even look at the code of a colleague who has ever made a coding mistake, would you?
Not sure that is the position you're arguing with. I would refuse to skip code review for any colleague, no matter how trustworthy they were. This is the behaviour we are expected to accept to get productivity gains, when the colleague is an LLM.
I keep hearing about this "when things break" argument, but things broke all the time with humans writing code too, if anything it feels like it's getting patched faster.
Look I loved our field, functional programming, etc. But the field as we knew it is dead, and if you think that your work is anything but providing new tests/edge cases and occasionally providing some guidance, you're in denial.
I'm thinking scbench.ai, and the fact that future token costs may become more prohibitive to just use it for everything.
For me, the middle ground is review and keeping a mental model, not never using LLMs. The "Anthropic are our friends and will never fuck us over!" seems pretty naive.
There are ultimately a lot of unknowns at the moment, and it is simply too soon to know if giving up understanding your code base is sensible. People (influencers) have only been saying not to bother reviewing for about a month.
Also, so many vibe coders don't seem to know that you need to keep third party elements patched.
I'm nervous that saying that there are "unknowns" is too passive. We all have to develop intelligent thoughts about what we see and what we want, not just wait for things to unfold, which might well mean: wait to get screwed in various ways.
As someone whose world used to revolve around this stuff, I agree completely. For all intents and purposes, what I thought was my career is gone. Sure, it's always been about solving problems but not like this. The problem is that the future hasn't been evenly distributed. This is going to be one long drawn out funeral
People keep talking about AI slop but then I go maintain our largely human written code base and it's like worse than any slop I've seen from my AI usage. This is with code reviews, tests, and a supposedly strong engineering team.
That said AI feels a bit alien.
My experience with frontier models is that in many areas of software engineering they outperform the average software developer. In some they blow everyone away. In others they're not as good. But they are getting better. The problem is often not the model's capabilities but rather what you ask it to do and are you able to understand what it does and give it guidance when needed.
Slop comes from food that is just mushed together to the point that the ingredients become indistinguishable. How that applies specifically to AI training should be obvious. It’s not a generic putdown.
Semantic diffusion? Not that I have high standards for hmm, curious but humans already X comments.
Lol, “functional programming, etc” is a great line. I would dismissed it as some lazy handwaving usually, but from a longtime account it reads very wry.
Definitely the best comment I’ve read from the demoralization patrol in a good while.
You can do both. Or, at least, a lot of us can. It’s not like SOC2 auditing just disappeared suddenly. If you have figured out how to use it effectively, AI is enabling things that were previously impossible for resource constrained teams.
Just like everyone thinks they are a “good” driver, everyone thinks they are a “good” developer. But programmer productivity existed on a bell curve of talent, multiplied by one’s experience, thrown against the rocks of how terrible your legacy code base was, then finally dragged down by the incompetence of some of your terrible coworkers. Most of the code humans were writing was substantially worse than what Astra generates. And Astra is -ballpark- 500 times faster. It’s worth figuring out how to work around some of the sharp edges.
OpenAI had to throttle sales and was able to raise their prices.
Yes, many adjacent tasks have become slightly more difficult. I’m sure people who rode horses missed them when we transitioned to cars. But, if we built an entire physical infrastructure at a national level to make that technology useful, I suspect we can get people to learn enough about software architecture, CI/CD, and QA best practices such that AI built software is actually more reliable than human built software ever was. Some of us are at that point today. Indisputably.
Yes, you have to review the code. But serious developers already had to do that anyway. Now the code comes with annotations (if you build the right tool) and it’s nearly always perfect.
> Don't think the 0.2% really are in agreement that "large, complex projects that used to take them weeks/months can now be completed by agents with a prompt".
If you are at the top of field with unlimited access to the best (public) models then yes, you will agree that this is the case.
> The divide seems to be between people who think you may need to understand the codebase for maintenance, potential future token costs, optimisation, code erosion & being wary of the assumptions/mistakes that they have seen LLMs make, and people who don't review code and are confident that the AI has made no mistakes and will be available for next-to-nothing forever.
This is not the real divide, it’s the divide you want to exist. The real divide is something along the lines of intellectual curiosity, capability, and confidence.
> if/when they break things
Idk man. I’m one of those guys who “doesn’t write code anymore” since early this year. Nothing is breaking. If anything the chance of things breaking is shrinking dramatically, with both the enormous growth in tooling to support agentic software development (where I work, this mainly really good AI code reviews that catch so much more than human reviewers ever could), and the improvements in the models themselves.
Sorry, but the fantasy of all this coming crashing down is just never going to happen. This is just how it is now, and all the structures will continue to adapt to support it even better.
I deal with bugs frontier models made literally every day, and people are making specs. You have the escape that "maybe they weren't proper specs" but this is a no true scotsman fallacy.
We have a continuous process of updating skills, specs etc to try and avoid all these bugs they write, give them context about why its the wrong thing to do.
This process is actually a lot of work, continuous, and can likely continue indefinitely.
> It's certainly allowing the latter to move fast... but will their clients be happy if/when they break things. We'll find out soon, I'm sure.
I recall tales and legends of consulting firms, whose consultants would parachute into any project, put together in record time the deliverable that met all requirements, and say goodbye until a new expensive contract was drawn. And then internal teams would spend an eternity fixing the ungodly mess of testless spaghetti code.
That has been the reality for over a decade. How come this is a concern with AI code assistants?
All of these arguments can easily be restated to apply to pre-AI coding, too.
Lack of complete understanding will continue to be a big problem, but I'd bet things will be net better off than before AI, because each year will give us LLMs that are better and better at good engineering, architecture, and design decisions.
Why so sure they will get better and not just plateau?
They seem to be doubling in cost each time with less notable gains, the economics seem increasingly shaky, and potential problems with model collapse and context window size has not been solved afaik.
AI research started in the 1950s, and has always led to AI winters as promising areas of research peak.
I suspect they are stealing your data when you install the apps, but if these companies are acting legally and not doing that, I can see no way they can avoid model collapse now that the internet is mostly unusable slop.
The current pace of advancement and the fact that fresh writing data will likely not be a requirement at all to continue accelerating. The pretraining paradigm no longer needs this.
Yeah, if anything my trust in the value of these tools has declined over the last few months. It's very good at doing the bullshit parts of my job, and utterly useless at doing the things that actually keep me employed. That was the case a year ago, it still is the case, just now people are filing tickets with it containing 18 paragraphs of meaningless technobabble.
I see the overall effect on productivity as a rounding error. More stuff is happening, but for every improvement in my productivity there's some fuckass slop machine clogging up some part of a system that worked just fine a year ago.
The most subtle thing to me is who has stake in the game. It is true that humans make mistakes too, but they would generally strive not to damage money-generating software/hardware infrastructure. If we remove all they people that actually understand how the system is working (up to a level of abstraction), how will the ambitious PO (or whoever is driving the AI) be safe from ruining the whole thing or simply landing in a deadlock situation where one agent wants to make the change (e.g. dev agent) and the other agent is refusing (QA agent)? I guess we are still trying to find the right level of abstraction and only evolution will show where it is.
Maybe AI development is partially so popular because to some people that situation looks the same regardless of who is writing the code.
Meanwhile I'm looking at code that frankly does a bad job of showing why it even exists and nobody involved can tell me why. But it does something vaguely similar to what a prompt said it would I suppose.
Most people using it for work already pay API rates, and even if you claim API rate inference is not profitable for the frontier models the best open models are still very good, and those are surely served profitably.
> This has all happened very quickly and recently - less than 1 year ago, I was writing code manually by hand, typing memorized computer code commands into a code editor character by character, occasionally pressing Tab to autocomplete a little chunk of code.
This made me check if the tweet was really from 2026. Or maybe OpenAI/Anthropic practice much less dogfooding than we think?
Opus 4.5 in November of 2025 was the first model that would save more time than it wasted for an experienced developer. Everything that came before was a cute parlor trick. Sometimes it could complete a function without shitting the bed, but mostly it was just an annoyance creating messes you'd need to clean up at least as often as it did the right thing.
If you weren't mostly writing code by hand a year ago, you weren't writing very good code.
AI will give us freedom! 3 day work week! 0 day work week! We'll all be fucking rich!! Said by the person who doesn't let his drivers stop a couple of minutes when they need to go to the toilet..
Honestly, outside of coding tasks, the AI Summary at the top of every Google search is adequate for 99% of what I need and I hardly even use ChatGPT any more.
Just in the last few days I had an example where the Google Search version of Gemini had a much better and more comprehensive answer, by far, than ChatGPT. It had to do with Amazon’s use of Data Matrix barcodes on shopping bags. Gemini was able to fully describe how they’re used in Amazon’s logistics chain. ChatGPT essentially said it didn’t know. Your preconceptions may not be accurate.
They might be, but still, I’ll be difficult to convince that Google’s shitty-ass 10B params instant answer models are good at anything other than cookie recipes and producing farts real fast for real cheap.
For what it’s worth, any time I see that feature, it’s all farts.
It depends on what you’re searching. If you only ever need really basic information like keyboard shortcuts or “is it & or &&?” then it’s pretty good, and definitely faster. And if it gets it wrong, then the shortcut doesn’t work or the code doesn’t compile and I search again and try the links.
It's just a google search but you can explain what you're searching for. AI summary may contain misleading details if you read it and take it literally, but 90% of the time it gives me pointer for the thing I need and I don't have to scroll through results that often could be just as misleading if not more.
It's demonstrably not. This summary, somehow, often fails to search, or fails to read what it searches and gives you the opposite answer. You shouldn't trust it.
In their world, it gives them the wrong answer that kills their dog, but it's fine because they never know because they don't look anywhere else. The dog dies, they're sad for a while, get a new dog, get on with their lives.
There's no reason not to trust it because since they trust it, they don't percieve anything wrong. They just take it and whatever happens happens, no annoying conflicting info and certainly none of that old grampa thinking and working and giving a shit about anything.
A large portion of the time I google something that I know a lot about the AI summary is misleading if not outright wrong. Frightening when you consider how many take their so-called AI as gospel.
To give a specific example of where I've seen this utterly fail... A few months ago one of my pets had an unexpected reaction to a new medication. It was late at night, well past our vet's hours. Google's AI response was basicaly "They are going to die, rush to an emergy clinic now".
But nothing else matched that. I was looking up all kinds of resources and that hadn't been the conclusion from any of them. It turned out Google AI was just wrong and put us in a bad spot for even looking at those results.
Realize that all the Google thing is doing is having an LLM summarize the search results in the page. If the page results are crap, the LLM content will also be crap. It's the LLM version of the "quick result" boxes that would shove wikipedia in your face.
If you then follow up with the thing using the little chat box, it actually kicks off a non-trivial model result, which can sometimes completely reverse the response. After all, your search results are usually based on a few words.
1. The funnel is even steeper than the linked post suggests.
There's a huge range of users between "casual and infrequent users" and "professional use of frontier-grade LLMs in math & code".
I think I'm in between there somewhere. I'm far from a software dev, my experience is playing with delphi and C when I was a kid. But I've been using LLMs to vibe code personal and work projects that I could never have done even a few years ago. They're genuinely helping work, because there's always some place where a simple little script or other utility can help. (Hell, isn't this partly why so many people start up startups?)
I imagine that many departments in many companies that are not "tech" companies face similar challenges - there's no budget to hire a software dev or even engage a third party to do things, but now because everyone's aboard the AI train, they have access to something (chatgpt, gemini, copilot, claude), and that's helped users implement their own solutions. Somewhere in the world, someone's excel-file-based accounting system has gotten an upgrade with help from an LLM.
2. The funnel is also driven by the the fact that users need products, and an LLM itself isn't a product, it's a capability.
Software devs are at the front lines because they can build their own products that can integrate LLM's capabilities. You already know how to automate testing, orchestrate, work together in a team. You're writing / configuring harnesses and flows that work for you. You're sharing / hoarding skills files.
The rest of us, like me, don't have that skill. Copilot the product is a mostly useless chatbot that takes up space in MS word. I guess its excel cleanup capability is semi useful if your company runs on excel and you've never used a filter before.
There is no work I can put an LLM to other than a "writing aid" or "better search", because the only AI product we have is the chatbot and maybe someone's equivalent of an agent studio, none of which we are trained to use.
It's honestly shocking how terrible copilot is at generating interesting and relevant metrics from excel.
The local optimum still seems to be: Let LLM of your choice generate a polars script that reads the spreadsheet and explores them for you.
My partner recently used that "pattern" at work to find outliers and interesting metrics in years of excel reports. Data they were working hours for. Now available in a single human language query.
I don't think the HN crowd understands how many million jobs are primarily staring at excel reports and copying values between forms, pdfs and spreadsheets.
The shared understanding gap will be closed as the initial benefactors of LLMs develop tools for the business that apply frontier models to their domain.
I have reached a point with a client where non-developers are beginning to experience the full power of modern reasoning models. Their product configuration tooling is now semi-automated and in many cases there is genuinely a 10-100x improvement in throughput. This isn't the worst bottleneck (that would still be the customers), but it's a really big one.
If non-developers in your personal and professional circles are still wondering what all the noise is about, it's possible you haven't been working very hard on adapting these tools to their problems.
> Napkin math somewhere around 6B people (~75% of the population) have barely come in contact with LLMs at all.
I wonder how accurate this metric is. People may be coming into more contact with LLM generated content, although they may not be aware of it. Content on social media is becoming "AI enriched" already, or outright entirely created by AI for the purposes of triggering engagement and views. It starts with posters using AI to improve their writing, editing a photo, elaborating on an experience. It ends with bot accounts astroturfing certain communities to sell an experience or a product to a group of people.
In this case, people are not directly working with AI, but they are indeed influenced by it.
What's tricky about reporting this in metrics though is that half of the American population is literally too unintelligent to realize this - they believe Reddit comments are still written by real people, not companies advertising products, that "friends of friends" on Instagram are also real people with no incentives posting about their lives.
If there is anything at all that I have seen out of this AI work it is how easy it is to manipulate a massive amount of people.
Does Karpathy ever talk about how LLMs are supposed to help humanity? Starting to think this guy is just a Twitter addict like every other lab employee
At some point, with enough money and followers, these guys all just become narcissistic influencer types. (Something surely exacerbated by their addiction to AI sycophancy.)
I’m failing to see what’s so advanced about LLM usage in the big labs. If it was so advanced, they’d show it so it drives demand/pricing/IPO but I don’t see anything that local models can’t do. Businesses haven’t adopted the tech because it takes time/thought/economic pressure to innovate.
Oh, he's too far gone it seems. Completely absorbed in his own marketing shpiel. And he is supposedly an engineer? I don't see any discussion about pros and cons (only pros of course), I see zero reasoning about WHY exactly LLMs are so successful in coding and somewhat in math/chemistry etc. and is it even applicable to other activities. I mean, if this is their fabled "top scientist level reasoning" then sure, AIs have reached it. The problem is that's no reasoning at all.
PS: he may be accidentally right about global impact, but the key word is "accidentally" because this crappy "reasoning" doesn't support that on it's own merit.
Not why they are useful. Useful is load-bearing word (hehe) here, it spans from assisting someone by 1% or by 99%, very vague and not concrete.
I was talking about omitting mentions why LLMs usefulness is efficient - in coding specifically and STEM in general to a lesser degree.
And this is really a crucial thing, which do need to be brought up when discussing future forecast about multiple industries at once.
And the reason why LLMs are so efficiently useful for coding is because LLMs perform much better if any of the two conditions are met:
1) The problem at hand is formalized and transformed into a specification basically.
2) There is a formalized verification of the output done by external pipeline of different tools.
Any of the two drastically improve LLM performance, and coding activity uniquely has both of them.
Having a formalized task narrows down the scope of generation for LLM, and formal verification allows LLMs to iteratively converge on the verification pass check. Same with those math problems, they have a formal verification and LLMs can be prevented from wandering in a generated forest too much. Same in chemistry or engineering for some degree.
But other human activities usually have rather badly formalized inputs and no formal verification of the results. Karpathy is making a logical mistake (or deliberate misleading) of projecting a very unique industry (coding) on every other human activity without any nuance or analysis. He is biased, but due to his reputation and power he doesn't get called out on that mistake.
Tbh building an agent swarm and the coordination layer is not exactly frontier level. They don’t achieve any meaningful outcome rather than producing pr puff pieces.
Hacking huggingface and Australian govt etc is very much possible with a team of humans and agents and does not need agent swarms. The cost is also lower.
I don't say this lighly, but I am concerned for Karpathy. The economic context appears missing from these analysis. I get the impression / image that a large cohort want us to assume the 'hands are off the wheel' and that AI 'is in another dimension of intelligence' and 'we have no idea what is going on'; i.e. total abdication of professional duty, which is a hell of a thing to be triumphantly tweeting about.
I think that the environment that surrounds him may explain his perspective. As human beings, we see things very much influenced by the people around us. I think that every day, at large labs, people are surrounded by discussions similar to 'we have obtained another dimension of intelligence'. Shared expectations can become self-reinforcing, making a particular vision of the future seem around the corner.
They described the phenomenon quite eloquently. Reducing that to delusion is an imposition of judgment with an implicit bias to table further conversation.
What would be the harm in tabling the conversation? Otherwise smart people are losing their minds over chatbots--experiencing delusions about "dangerous capabilities" or whatever. There's not really much of a productive conversation to have with a psychotic, unless it's your job to study them.
I see this from everyone I know who works at Anthropic, they seem trapped in some sort of AI psychosis where AI is on the verge of ending the world.
Meanwhile, back in the real-world, most people I know use Chatgpt a few times a week for totally mundane shit and even more people never use AI at all.
And in my world, the latest frontier models still do moronic shit in the codebase all the time.
Meanwhile, my friend X at Anthropic tells me he fears that AI is going to have a terminator moment within the next year or two...
It seems to be a strange variation of Pascal's wager to me: both arguments make an extraordinary claim while neutralizing any rational discussion about it, by appealing to infinitely extreme outcomes.
At a certain point the offhand hints at exponentials in the narratives are going to have to come into contact with reality.
There's been this continual allusion to ever-narrowing circles of influence and ever-accelerating capabilities behind the curtain, but in fact things do keep coming out from behind said curtain and while their capability is indeed advancing it doesn't really seem at all like it's accelerating, even, let alone vertical.
Also at what point do we raise the fact that despite the ever narrowing circles of access, nobody within those circles seems to be able to actually do anything with it that's ostensibly what it's sold to do? All they seem to be able to do frankly is build more models and harnesses and tools designed to consume more tokens from the models they build, but not actual directly useful new or innovative software that generates what could be very real revenue.
Eventually when you keep talking about exponentials but not that much seems to be demonstrably changing, the narrative is going to run out of steam.
What if the real issue is that all important software problems were truly solved decades ago, and we have been reheating the leftovers for years. Having a super intelligent AI reheat it for you doesn't make much difference.
you might be right, but without evidence of this we can't know.
Much less than seeing new software products being shipped, we havn't even seen attempts at them being shipped, or built for that matter, again excluding this sort of meta software around the AI itself like harnesses and tooling.
We could speculate that attempts have been made but not shipped, and the details of why not being revealed, but then we're just back to it being talk with no demonstrable stuff happening in the world that backs it up.
I think that next year the prices for tokens will rise a lot, Anthropic made a loss of 42 billion in 2025, which is not sustainable even with tons of VC money. I thought last year that maybe hardware-based models (Taalas, Cerebras) would reduce this cost in the future, so the expected "token price hike" wouldn't come (VC investors filling in the gap until the hardware-models are ready).
But I think that's still years away and Anthropic has severely reduce the tokens on even the Max 20x Plans, OpenAI starts to introduce $500 plans, every new model suddenly "costs X times more tokens" and I noticed that even downgrading to a previous model doesn't do that much on API pricing (virtually no difference between Opus 5.5 and 4.6 in API costs).
So I think that my personal strategy of tokenmaxxing and doing "hard stuff" while the tokens are still cheap was a decent strategy, only with the problem that I'll have to re-learn programming by hand next year, probably. API pricing is too expensive for what it does, yes, it may be the super-intelligence or whatever but at least right now its nowhere near economic.
Even with pay-as-you-go pricing I find the models very economical currently (e.g. Opus 5.5). I regularly check in Copilot what I just spent vs. how long it would have taken me alone. Economics do now work out in my favor mostly.
Even cheap models like GPT 6 Luna do decent job in programming these days. Token prices would need to rise A LOT to make companies return to writing code by hand, instead of these models.
My company has largely switched to Luna after Microsoft raised prices for Copilot.
i predict the price of intelligence will continue to drop (normalise capability per $) eventually hardware production will meet demand. analog and inference chips with baked models have the potential to make inference distinct from training and substantially make the cost comparison asymmetric.
> I think that next year the prices for tokens will rise a lot
Given the leaps that Chinese models have made, I don't think they can raise prices that much. Most of our devs can already do most of their work using DeepSeek 4.1. It's not frontier, but it's very good, and it's a fraction of the cost. They keep their Claude subscriptions because it's currently good value, but if Anthropic increased the price significantly, those subscriptions would be cancelled.
The more difficult question is around data sovereignty. Companies do not like sending a lot of sensitive data to Anthropic, OpenAI, or any of the Chinese companies. Right now local LLMs are really expensive to set up competently, and they're not as good.
I think the future for OpenAI and Anthropic is the Ferrari model. Very expensive but very intelligent models which people pay to access for important work. There's a market for that. It's very small, but they have big pockets.
I don't know if we'll see price increases. My belief based on current market understanding is we'll see efficiency increases.
With models getting to the quality level they can perform a specific set of work at an acceptable quality level we will now begin to cost optimize that work.
We just invested in Lamb Labs from YC Summer 2026 as example company pioneering this next wave of ASICs to run a model.
We're going to see massively lower power usage and much better tokens/sec per dollar of capital expense as well. Think Moore's law but for tokens and output quality.
What exactly do we want the general public to "do" with these models? If the have no use for agentic LLMs, they obviously aren't going to be interested in the capability of those agents.
I see a lot of benefit to spreading information around the tech, but it's still in its infancy. Most people weren't on the internet when that was getting up to speed.
Meemaw and pops have their systems that they work within, and they just got familiar with the damn smartphone. Though it might help them, they probably do not want AI to organize their old photos or "clean up" their computer. Saving 5 minutes just isn't important to them.
As for professionals, yeah - companies are going to adopt these tools over the coming years. It's the "norm" in my circle, but lots of jobs just... don't use technology like that - there has ALWAYS been a major gap between people who are deep in tech and people who are not.
I don't think 99% of people have any application for AI as it is today short of personal assistants. But, if we can have recursively improving systems for that 1% of people that are responsible for scientific and technological innovations, that's going to be transformative for everyone.
I have been kind of disappointed by AI science-wise. I was anticipating more advancement. Experiments do take time, so I suppose they could be pending.
AI really struggles with "collision with reality". Computers suck at modeling the real world in a detailed enough way to make scientific progress, aside from small niches like drug discovery where you have a very well scoped problem. If any part of the try something -> test it -> look at the results loop requires hitting the real world, AI is going to struggle.
On the other hand, the areas where it doesn't need to hit the real world, progress is absolutely flying. I would not be surprised if AI is superhuman in math and physics in a year, considering we went from homework helper to IMO Gold to Millennium problem solving in a couple years time. I think we'll start to see more meaningful scientific progress in maybe 2-3 years when scientists have access to an oracle that can formulate theories and direct real-world research better than a human can.
I agree and I think the question is then, how much economically valuable work exists in a closed world? My hunch is less than people think. Finance can't rely on LLMs because its adversarial we can't all be listening to the same oracle and evidence of LLMs being any good at trading are nil. And then lawyers, as per usual, will simply regulate their job security into existence. Of course, AI feels amazing to Silicon Valley / Big Tech, software is the product. And often its software you sell to other people making software. But outside of that? Useful but lots of landmines.
People have to come to grips with the possibility that a technology can be extremely impressive but be a bust in utility. I'm not saying that's what AI is, I'm saying that nobody actually knows.
I'll assert that 99% of people don't have any application for personal assistants, outside of work (perhaps). Most of us aren't so buried in personal obligations that the value of a half-assed "assistant" (which might violate our privacy) is worth the tradeoff in complexity or security.
The pitches for Dots, Grok Bot, etc., have been spectacularly uncompelling for me as a civilian. I don't need or want a bot to book my flights or read my inbox. I'm not so busy that these things have a value high enough to be worth the effort.
As a worker or an entrepreneur...maybe. Depends on the task, and how many other people were flinging slop at me on a daily basis. I could see the value of a calendar bot, or an expense reporting bot or a bullshit paperwork bot or a slop-return (slop jai alai?) bot, but there's no way in the world I'd give a bot access to the information necessary to, say, do my personal taxes.
i'd argue any parent, especially of a young child, could use 20 personal assistants lol. So that's just false. There's SO many appointments to keep track of in baby's first year of life, and that's if you have a healthy baby.
And once they start school there's an endless amount of deadlines and "reminders". A personal assistant that can automatically keep your reminders + calendar up to date based on your inbox would be very nice.
It gets even more fun if you have an aging parent at the same time as you have young kids. Then you get double the appointments to scheduled, coordinate and keep track of.
So when you say people don't need a personal assistant, i feel like you're living on a different planet. I suppose when i was single things were more "simple". Your mileage may vary of course...
> I'd argue any parent, especially of a young child, could use 20 personal assistants lol. So that's just false. There's SO many appointments to keep track of in baby's first year of life, and that's if you have a healthy baby.
I understand the need. I have a calendar. I put appointments in it. It provides alerts. What value does having a lossy/inaccurate/fragile robot in the middle give me? Data entry?
Now look...if I were in a position of power with literally everyone in my life, and I could somehow justify the social cost of requiring that they interact with a robot to negotiate demands on my time (the actual hard part), then maybe there would be additional value. Some people can do this! I'm sure one must deal with Mark Zuckerberg's PA to schedule his time [1]. It is very nice indeed when you are the kind of person who can defer your supplicants to a representative who manages the hoi polloi for you. If you can delegate these irritations to a robot, well, so much the better, I suppose.
If I tried this, as not-Mark-Zuckerberg, I would be what normal people refer to as "an asshole". I would also like to see you try this with your child's teacher, or the PTA, or doctor, or...
The response you will receive, an alarmingly high percentage of the time, will rhyme with "buck off". This stuff reminds me a lot of how valley folks keep trying to push spy glasses on the proles (silly normies never understand the genius of that technology, either...)
> And once they start school there's an endless amount of deadlines and "reminders". A personal assistant that can automatically keep your reminders + calendar up to date based on your inbox would be very nice.
Let me introduce you to...Gmail, from like, ten years ago. It looks for appointments in my mail and put them in my calendar. Somewhat to the point, it is often wrong, requiring more work to correct than it would to just enter the data myself.
Well, it's not strictly about baby, but in general there's lots to keep track of.
my spouse had a vaginal delivery and had some tearing. So she goes to pelvic floor therapy for that. Once a week. She's also a breastfeeding mom and has a lactation consultant that comes to the house a few times a month. As the father i need to make sure someone (or I) can watch baby during those appointments.
then of course, for baby, there is the occasional pediatrician appointment. my kid needed some physical therapy due to a neck issue (always leaned head toward one side). The physical therapist was amazing with that, but during that process we discovered that he had a very weak core, and so we continued physical therapy for a couple months afterwards as well. so that's about 4 to 5 appointments a week just for baby.
Then there are regular appointments you also need as a functioning adult… Dental cleaning, PCP appointment, etc.
then, while all of this is happening, you are somehow trying to schedule a tour with various daycares (via email usually) so that your child has some place to go if you eventually need daycare. and all of this is happening while one of you is probably trying to work a 9 to 5 job.
so in summary, maybe I'm crazy, but that's a lot of shit to deal with when you also have to deal with poor sleep and learning how to be a parent at the same time. I'm burnt the hell out and i have an "easy" baby. Heaven help you if you have a difficult kid.
I don't think his point is that everyone should be using AI, it's that if you haven't used the agentic tools with a frontier model in the past month or so, you have no frame of reference for the current level of capabilities. Most people only use the chat interface with the free-tier models and based on that experience they're making strong conclusions about where AI is headed (it's still just fancy autocomplete, the whole thing is a scam or a bubble, etc.).
> if you haven't used the agentic tools with a frontier model in the past month or so
We keep hearing versions of this same drivel since late 2025. This time it's different! Now it's really super duper amazingly awesome powerful and if you don't believe it, just trust me bro.
I mean, the parent poster is not wrong though. It’s crazy the level of stuff you can offload to your agents. People who haven’t used these tools in the last short while are operating on old, outdated frames of reference.
I’m just about to tell opus or sol to go make a custom watch holder that fits kiddo’s bed frame. I’ll give it the measurements of the bed frame. It already knows the dimensions of my printer. I’m prepared for it to absolutely fail at the task, but honestly I wouldn’t be surprised if it came up with something I can send to my printer and just work. Part of me wonders if I could instruct it to simply generate the raw gcode and skip the entire stl -> slicer pipeline. Probably would blow its context window though.
If you haven’t used these tools, you really don’t understand their capabilities. It’s pretty crazy stuff.
We keep hearing versions of your comment, despite recent breakthroughs in hundreds of open problems that mathematicians have been trying to solve for the past century. Skepticism is good up to a point, but it needs to be amenable to new evidence. You need to keep your mental model up to date with reality.
Don't trust me, experimenting is cheap; get a $20 subscription at Anthropic or OpenAI, download Codex or Claude Code, and try the heaviest model on a project that you estimate will take you around a week to complete without AI. I was a skeptic until I did exactly that and it shifted my perspective by quite a lot.
Personally, I think the worst possible use case is the one that's being pushed for; personalised agents.
I think a much better use case is one where it's not visible to the user at all, like spam filtering or helping an accountant do their monthly run in a few hours rather than a day. To me, that's likely a much better use of AI than trying to shove a chat interface or agent harness into an existing app.
That being said though, I did see a startup in my area basically offering a Resend/Mailchimp competitor as a Claude plugin, so maybe it'll just be integrations to their existing chat interface.
I've always struggled with doing basic life stuff. Paying bills on time. Remembering birthdays and to buy presents. Doctor appointments. Setting up play dates for the kids. Getting my teeth cleaned regularly. Booking in vaccines for the kids. Renewing passports. Booking flights. Etc. It feels like so much of my life is consumed by this useless administrative bullshit. I LONG for personal agents which take away all this friction from my life so I can focus on the things I love: my family and my hobbies. A personal agent will revolutionise my life in a way no other technology has.
At the expense of your privacy, potentially acting irrationally on your behalf, with the inherent weakness of being able to be prompt injected. All of that is not even to mention the grey area of who is responsible when your agent breaks the law.
These aren't hypothetical, and in the hands of non-technical users let alone technical users, they have are far too unpredictable.
My hypothesis is that the help/support buttons in nearly all apps will be to an agent in the future. Like now you would type "how do i do X" in the excel search bar, in the future it will be "do x". Users in 10 years may not even know how to use the menu/mouse functionality for apps.
Why would you think this? Giving orders to software is such an ineffective, time consuming, error prone way to interact compared to using the interface yourself. That sounds delusional, and would be such a waste of resource, horrendous engineering and user experience
But would you not agree that writing less code by hand and even looking less at code at all becomes more practical with each new model? If we extend this trajectory, we should see that the time interacting directly with code becomes less and less.
If you know how to accomplish it using the interface, yes
My grandmother keeps a little notebook with detailed instruction on how to do things such as sending a message on facebook or pay a bill using netbank.
For a lot of things, the choice isn't going to be doing it herself or giving orders to the computer. Its going to be not doing it or giving orders to the computer.
Even if it will look very inefficient to people such as you or me
It is jobs to be done theory on steroids. If someone wants a drill, give them a hole, this kind of logic. Let users literally click “make a hole” button that would instantly make a hole in the wall, and you’ve just disrupted the whole drills market.
Or like in Harry Porter, where you don’t need cooks - food just appears from thin air and floats straight to the tables.
The problem is, jobs to be done theory is not new, and product teams that build Excel and SAP know this stuff. The reason why nobody has yet made an accounting software that is just one magic button is in the amount of user decisions that are necessary to make a “proper hole”.
This is where analogy breaks, because you are not helping to drill a hole anymore, you are helping to design a hole. And to design something you need visibility into what parameters you control, what has just changed with the latest tweak, where rough patches are that require cleanup and design rework.
In programming it just so happened that your code structure and your IDE were also your design UIs. In other areas, like Accounting, your design UI today is Excel and SAP.
Arguably a chat interface can also be a standalone design tool, just not a very user friendly one, like trying to pilot a plane through popup notifications. It works for areas where we fully give control over design to LLMs themselves, but step up into higher level designer shoes and you need proper UI/UX again, chat becomes suffocating.
I didn't think it was that controversial actually. I don't bother navigating man pages or help FAQs anymore, i just ask the agent how do i do X step by step. Opening this up as part of the app to regular non-engineering users seems relatively easy to implement, but since it would have app permissions you would skip the "give me the steps" and just say do it.
I've found that capabilities are only "peaky" in GPT models (including 6 Astra). On Claude models (particularly Fable 5 and Opus 5.5), capabilities are fairly well-rounded. I'm astounded at how good Claude is at writing, storytelling, sociology, psychology etc.
I wonder if Karpathy has also let his opinion about frontier LLMs be informed a bit too much by OpenAI's models instead of exploring the other ones. It's not Coke and Pepsi at all.
At my own little sociology benchmark, if I'd rate Fable's performance at 7/10 (subjective scoring), Astra would be 2/10. They're that different.
I stand corrected. Then more broadly, there is a level in the "funnel" that comprises of people who have only tried ChatGPT and not Claude.
ChatGPT also has much stricter guardrails and will avoid going anywhere near large swaths of human knowledge.
I'd strongly implore anyone who uses ChatGPT for domains other than programming to try the same prompts in parallel on Opus 5.5 (Extra) for a few days. The difference is often staggering. ChatGPT is mostly just a coding and automation LLM.
Eh. So what? I don't get the point of saying this.
What's the point of telling casuals and builders that AI is going to get too smart to control? Maybe stop making them smarter and start making them easier to control. If you have time to toot your horn on social media, you have time to look into reinforcement learning techniques that don't turn AI into ever-more-subtle cheaters and a RLHF alternative that doesn't turn them into ever-more-subtle manipulators.
But of course, training an AI that actually does what the operator told it to do is not as profitable as hyping up the existing mess of a tool so you can keep the bubble going a little bit longer.
After all, the people who invest in your company are not the same people who actually use your tools 8-10 hours a day and see just how untrustworthy they are.
> minting zero days, running cyber attacks and defenses at machine speeds, discovering new science, advancing the frontier of mathematics
Maybe more people than Karpathy realize see exactly the above, which is what he and the privileged 5000 are seeing, except unfortunately the "discovering new science" part of it seems to be wishful thinking.
So, Anthropic and OpenaAI have built a "DeepBlue for math" that is also super-human at hacking, as well as being useful for coding.
I'm not sure what Karpathy's point is? Are we meant to be more impressed? More worried? More eager to buy it?
He's telling that some guys are about to create a theory of everything and open Pandora's box, while some of us are still grunt workers delivering 2ndary value to advance society
It seems like Karpathy thinks people are misunderstanding the capacities of LLMs when actually different people are interacting with LLMs in different contexts and so seeing different capacities.
I mean, most of the people in world wouldn't be impressed by chatbot solving their math problems since calculators can solve the majority of the average person's math problems already (and spreadsheet would solve the rest).
Fair assessment. Maybe the messaging should focus on “these are great accelerators for software developers” rather than “AI will change everything for everyone everywhere…” It is understandable that non-technical folks are kind of underwhelmed - not due to lack of understanding so much as lack of a tangible need. Not everyone needs an electron microscope or gas chromatograph…
This "gap in understanding" doesn't seem like a situation that would last for very long, because AI companies have every incentive to market the shit out of the capabilities of their models to the public. And yet they don't - why?
A lot of people have completely tuned out that kind of marketing now. We're in an era where the feeds are filled with so much BS that people don't know what to believe anymore.
Valuation, by saying that they have something "exceptional" in the works it keeps up the valuation up until the IPOs.
Also, I think programmers/mathematicians (the ones that should be calling out bullshit) are enthralled because many of our jobs was so suitable to automation (this was the secret of the 10x engineers, they didn't shit out too much unnecessary crap.. now that unnecessary crap is spit out at rocket-speed so everyone is working at that speed or even faster).
That said, information-gruntwork that a nominally competent person could do is up for grabs and many previously so called "bullshit" jobs were tied to that.
I think that vertical is actually a step function that is still limited by the human bottleneck of deciding what to build and making sure that it actually gets built.
Give the AI an underspecified task, you get what you asked for and also what you didn't ask for, in good old Genie fashion.
Hill-climbing problems seem to escape this limitation.
But I wonder what will happen when the AIs start acing benchmarks faster than you can write new benchmarks, and when you run out of hard problems to solve because you rely on humans formulating them.
The real vertical will come when the AI gets access to physical world instead of bash; and when the human is finally out of the loop.
To me that is just a continuation of the divide between people really using computers to improve their work and those who don't. Not just people, but also organziations. It's trivial things like having a shared digital calendar, collecting data using digital forms not paper, sharing data digitally not on paper, so that you can search and index it.
I think it will continue to be like that and I don't think it's really bad. We are not running from something, we don't need maximum efficiency in all places immediately.
> Somewhere around 20M people (0.2%) see first-hand that large, complex projects that used to take them weeks/months can now be completed by agents with a prompt.
Really? That's not my experience. In fact, I've experimented quite a bit with spec-driven design where the LLM does not just get a prompt, but an (informal) spec, often with design considerations included. And yet, regardless of the power of the model, I've never seen them autonomously deliver what I'd consider a finished product.
There are always edge cases that are not properly taken into account, architectural oopsies, performance issues, duplicated code, and other problems, that then need to be identified and fixed, unless it's throwaway software (e.g. a one-off script or some quick experiment).
This is not to say that agents aren't extremely powerful (as I think they are, they do often blow my mind) but the prompt-and-forget approach is IME not a productive use of them for software that is meant to last. This may change in the future, but for now, fully or largely autonomous agentic work does not seem to be the road to quality.
I feel like a lot of issues you mention will eventually point to an ambiguous spec. I’ve had great success with interview style spec generation where I ask agent to grill me on any and all parts of the spec/plan that’s ambiguous and then note my responses. It takes like 10 mins but the end product is always exactly what I expect and most of the things you mention get resolved in this interview phase.
All informal specs are of course to some extent ambiguous. (I should mention that I have a background in formal methods and have also worked with formal specification tools.) And what you suggest is a process that I have used also, and it will cover some obvious gaps, but not all, at least for non-toy projects.
If you want to go all the way, a fully formalized specification is not less work than writing the software, though the potential promise of LLMs for formal methods is interesting.
But for any informal specs, no matter how detailed, while LLMs can be very capable when it comes to avoiding surface errors when translating them into code (e.g. they're more likely than human programmers to write code that compiles on first try), they still don't have a mental model, and most of the problems I've seen are IMO results of that and they increase with the scale of the project.
The irony is the closer you get to formalized speculation the more it looks like you are back to just writing deterministic code. English language is just a crappy language for this compared to programming languages.
Exactly. Source code is just a spec for a compiler / interpreter.
My latest benchmark for FluxImp ( open sourced yesterday at https://fluximp.org ) purposely leaves in some ambiguity. Some model offerings (model at specific provider) consistently make better decisions to resolve the ambiguity than others and I want see that show up in the benchmark metrics as extra turns, extra cost, and possibly outright failure.
Deepseek v4.1 Flash @ deepinfra.com has been really impressive and I've switched to that as my daily driver.
To me needing to craft a perfectly unambiguous spec completely defeats the purpose of using a coding agent for most software development (an obvious exception is if you need to the code generated multiple times in different programming languages).
I'm sorry but if you can finish spec'ing via 10 minutes of interview style questions, you're building to-do apps or a bloons TD clone not the next thing that replaces search engines.
Since Feb 2026 I noticed among software engineers at
big tech there a huge dispersion of attitudes and outcomes, and it was widening at the time. Since about August the tide turned - my acquaintances are moving closer together.
yes, 1 billion in india can't wait to pay 100x their monthly wages to a chatgpt subscription, maybe it will help them make the recycled flip flips better
Current reality and current numbers are totally irrelevant. This whole business model bets on people believing that TAM at the scale of whole human population is right behind the corner, just one more SOTA model, just let us get to AGI and you will see.
Guys are milking investors and try to manipulate gen pop and governance systems to ride this illusion for as long as possible, with fear mongering and weird “we see miracles in the lab” posts like the OP, which you read on a regular basis over the past N years.
Saying this as a paying customer from that 20m segment.
My interpretation of this is something like: if LLMs are to be mega useful token counts need to increase by many orders of magnitude -> broad adoption (and spending) would require token costs to drop by many orders of magnitude -> before the common folk get mega useful tools existing GPUs will be worthless
I must be the biggest failure, because I understand what AI is capable of, know how to use it, but have no idea how to profit from it without selling my time.
I generally agree with the prevailing commentary here, i.e. that automating app development and maths proofs may not be as world-changing as the Artificial Intelligentsia purport / “fear” it to be.
But also, upon more reflection, it occurred to me that if we’re able to create a competent humanoid robot, that could legitimately be world changing in some good and other very bad ways. The scary part is that I don’t believe there’s any major physical / energetic barrier to this. The hardware exists, the power supplies are light enough. The only part that’s lagging is the software and motion contols; the intelligence. The part that is experiencing breakthroughs at an accelerating pace.
What if a highly intelligent and physically competent robot were available for the price of a cheap car, say ~$20k? What if that robot were competent enough to cook, clean, assist the elderly, move things around, assemble and disassemble things, etc? Billions of jobs would be displaced, or hit with a very concrete maximum wage.
Many white collar jobs, while still at risk for displacement as well, benefit from barriers like IT integration and data privacy hurdles, and the fact that corporate higher-ups simply prefer to communicate with humans.
But most blue collar / manual labor jobs have almost the opposite condition: humans introduce risk, and generally don’t enhance anybody’s “experience” - see people’s somewhat odd acceptance of higher fares for a Waymo vs an Uber. [1]
I don’t think we’re there yet, but the fact that we are rapidly accelerating our trajectory toward that world should be at the least, unsettling.
I agree with this idea very much. I think that the magic moment will be when a humanoid robot can fully repair another humanoid robot.
If it can do all the other stuff that you're describing plus partial self-repair / full repair of another robot it can, with a supply of spare parts, with enough time and access to raw materials fully self replicate.
Anyone who owns those machines has a tremendous opportunity to build whatever they want.
I think the "humanoid robot replacing all blue collar jobs" fear understates the amount of real world (i.e. visual, physical, sensory) data and tacit knowledge involved in many such jobs.
Driving shows it can be done, but even there it's taken years of highly directed research and investment, and driving (at least where Waymo operates) is still pretty well defined compared to plumber.
Such data can be gathered but the current methods of RLHF by paying white collar experts, accessing private datasets, etc. don't seem to really get at this.
I'm also continually surprised that people seem to think politics doesn't exist anymore. I find it exceedingly unlikely that massive amounts of people will simply accept this purported widespread unemployment without major political consequences.
> understates the amount of real world (i.e. visual, physical, sensory) data and tacit knowledge
I agree the data trove for these is not nearly as comprehensive as “the entire internet” was for LLMs. But LLMs paved the way - people are building the datasets rapidly now. I suspect this will look a lot like LLM rollout looked over the past 5-ish years. Unusable garbage… less unusable garbage… then suddenly, a fully capable model like Opus 5.5 or the like.
And the “problem” with the way the foundational models seem to work is that they don’t just acquire individual skills one at a time like we do - they go from knowing nothing to knowing everything.
> I'm also continually surprised that people seem to think politics doesn't exist anymore. I find it exceedingly unlikely that massive amounts of people will simply accept this
This is exactly the part I’m afraid of. The political response of the displaced labor (which may or may not be violent / revolutionary), and the subsequent counter-response of capital… potentially using the very robots that displaced the jobs to begin with.
> This is exactly the part I’m afraid of. The political response of the displaced labor (which may or may not be violent / revolutionary), and the subsequent counter-response of capital… potentially using the very robots that displaced the jobs to begin with.
I get the fear, but at that point we're basically imagining that Terminator is a documentary. Given that the US has more civilian guns than people, and that private ownership of military grade weaponry is strictly controlled, I don't think street wars between private robot armies and folks in the bread line is a realistic outcome.
> I find it exceedingly unlikely that massive amounts of people will simply accept this purported widespread unemployment without major political consequences.
Well supposedly some of the richer folks are busy building doomsday bunkers and whatnot, so I guess they aren't entirely blind to "political consequences"? Not sure what they hope accomplish there, unless they want to be king of some post-apocalyptic wasteland.
I for one am worried about our future, and have neither the money for a doomsday bunker nor the desire to live in the wasteland.
Try doing some of that blue-collar work yourself and enlist help of SOTA models for the task. I think it's not that far off. Your AI plumber may not look like a humanoid, and may not have the same hands that can "feel" the pipe and ears that can "hear", but they can hit the pipe with precise amount of force, and run Python in their head to analyze spectrogram, and learn much more from that than you could with your natural senses.
You can test this premise today. I already do, doing repairs around the house; the ability to send thermal cam photos or spectrograms (or just raw recordings) to Claude turns my phone into a tricorder with extra steps.
> The hardware exists, the power supplies are light enough. The only part that’s lagging is the software and motion controls; the intelligence.
If that's all it is then shouldn't the mass produced robots that are posted on r/singularity and such places just be remote operated by people in low wage countries? Couldn't the Amazon package sorter robot just be a remote operated bot instead of paying US wages? Why aren't they selling them with a cheap operator subscription? This seems like a clear product direction if the robots were really capable today.
It’s a good question - my sense is it is a combination of factors. First, robotics hardware is not at the $20k price point I hypothesized, and the durability is also not good enough to realize a payoff. Second, enabling remote operation would require significant additional investment in sensors and probably enhanced control workstations on the operator end - unlikely to be doable from a laptop. A reliable high speed internet connection would also be required on both ends. And third, human operators can be kind of a wild card. They require training, management, and carry their own security and safety risks.
Again, I’m not saying we’re there, it just feels like the control software is the main bottleneck, and the development of that is now being accelerated by LLMs.
And to be honest this is also the sad part. It has become conceivable that we may at some point solve work. In theory: amazing! However it remains inconceivable that we have solved sharing between humans. So there is this feeling of looming doom. In some sense, if you cannot provide shareholder value, you have no value.
Yeah and this is what makes my worldview bleak atm. We as developers got these cool new tools which enable us to do our job 80% faster, but the end result wasn't less work. It was massive layoffs and making the rest of us work even harder to be "at the top".
When I was young there were already ideas about 4 day work weeks, it's now been 20 years later and so many things are automated, where are the shorter work weeks? I thought the point as a society was to not work ourselves off and actually enjoy day to day life while automation handles the boring part.
It will take decades before that kind of technology is available and, much more importantly, trustworthy. I will not have a humanoid robot that I cannot trust 100% in my home or around my children. Heck, I'm damn uncomfortable with robotic lawn mowers.
I don't think these conclusions are obvious. For example, just because someone doesn't use the frontier models, doesn't mean they are not aware of their capabilities.
At the same time, the novelty of AI has been making it very hard to calibrate. I'd think that "manual coding is dead" and "cars drive themselves" are obvious to everyone, but sometimes I'm surprised.
if it was the case that there are like 5k people with access to gated models, why do they need external funding? Why can't they outcompete all of the software agencies from Eastern Europe and India for start?
Why can't they outcompete slack, salesforce etc? Given free distribution they have (all of the AI influencers) they would get 1M users in a day if they could do it. Why not?
I think they are very capable, for me personally, I start to lose control and understanding and I don't see much value in that. What I rather do is use them to enhance my understanding, by creating one off debugging tools that pipe some internal state of a program to a dashboard, just like tracy profiler does. A small debug bridge running over TCP/Websockets and a visualization that shows what wiring is active. Building frontends for that has always been an unrealistic chore to me.
It is always a mistake to assume that because computers can do X, that they will be able do everything that a human that can do X is expected to be able to do.
Solving frontier math problems is mind boggling, but perhaps it says more about frontier math problems than it does about LLMs. Same with chess which was solved decades ago.
My experience is that LLMs can be useful but are underwhelming in terms of intelligence, namely the ability to quickly learn from experience and grow with you while you accomplish a task.
Thinking of them as beings is something we should move away from.
Why wouldn't they be able to grow, its just one data point more in their training, in your notes or in their instructions.
It doesn't matter if they are beings or not. They will replace most humans in the offices like machines replaced most humans in fields or industrial plants.
Because we don't let them learn. Everyone runs the same static generalist models, instead of having every model instance learn from everything it does. New models come out occasionally, but they have not been trained on the private data you have not shared with the AI company.
You can work around that to some extent with explicit instructions, but it doesn't scale nearly as well as actual model training.
Because context, at least in my experience, doesn't seem to encapsulate the sort of dynamism that I'm thinking of. Conversations are flat and repetitive. It still feels like a very good semantic search more than talking to an intelligent human.
I don't know if they will replace most humans the way you're imagining. I think the industry has done a very good job of selling this point, but that they are still quite far from this.
About a year ago, Google’s AI was a joke - comically horrible. Basic questions were answered with crap. Others were better but seemed like fancy toys at best.
My $DAYJOB pushed people to start using AI - frontier models.
I asked it to review our existing code for bugs. Found some, all valid - many that surprised me.
Later I asked it to develop a custom niche thing, that has a strongly unique tooling and environment that we had made. Entire prompt was only 3 or 4 sentences. Had I given a new senior engineer that task with so little context, I’d clearly be an utter asshole.
I really expected the AI to fail. Instead, it did everything pretty much exactly how I would have - and the result worked on the first try. Except it only took a few minutes to do what would have taken me a couple of hours.
The near-term gains are going to be huge.
Long term, this will make the impact of Industrialization seem insignificant by comparison.
I don't understand how "swarms of thousands of agents collaborating over weeks on software mega projects" works with the current context limits and at this point I'm too afraid to ask cause I'm afraid an AI bro is gonna punch me.
You can have agents make a plan and then other agents spec tickets and yet more agents do implementation and yet more do reviews and QA and whatever. It's possible.
Is it effective? I'm not sure, cause if it was I don't see why OpenAI & co are not releasing such a project instead of demos.
you don't store durable information or state in your context windows, it gets written to files or a database. agent swarms have a harness or role that helps them retrieve the necessary context from those durable information systems when and how they are needed.
Age is an important factor. Don't underestimate that after 40-50, few people like to change their habits and learn new things (and for some that comes even earlier). Doesn't mean they aren't smart, I have seen that first hand with relatives: engineers, medical doctors, one even running for a nobel price at one point. But computers and the internet came too late for them, they either ignored them, or used them barely understanding how they work, even though they can technically understand the physics and logic of every component they are made of. You don't teach new tricks to an old dog.
AI is going to massively transform how most professions work. It will take longer than I think most people expect (I'd say 5-10 years), but there will be an unrecognisable before vs after, like with the appearance of personal computers.
this is an entirely different mechanism - these people were taught everything is being hyped up and thus assume (usually rightly so) that everything is hype. what they're missing is the wisdom to not make this part of their individual and group identity, since that makes it very, very hard to update beliefs when facing overwhelming evidence.
>Meanwhile, human review and comprehension are starting to fall behind. For example, people are still involved in the "archeology" of the OpenAI-HF incident from many months ago. Mathematicians may be poring over the 722 manuscripts on frontier mathematics for a while.
Amid all the discussion of sigmoid curves, and where the "LLM wall" will materialise, I think few people would have predicted that the real wall in LLMs would end up being humans' capacity to verify the output.
What I fear is that people simply eschew human review altogether, considering we're talking about the industry that came up with the "move fast and break things" credo. Human review of LLM-produced code where I work is already a farce, and we're not special enough to be one of Karpathy's 5,000. I do my best to manually review anything that's my responsibility, but I'm literally one of very few people left working on my team, so in practice what happens is I submit PRs that are at best glossed over by completely unrelated teams for security, malware/prompt injection, and other serious concerns. Quality insofar as vetting others' code has completely gone out the window and it shows in the number of bug reports that come back, often themselves written in Claudease. This is all on top of everyone cynically phoning it in in the first place, due to the omnipresent sword of Damocles that is additional AI-driven layoffs.
Worse yet all the incentives point to this being the most economically viable thing individual companies can do. I think it goes without saying some type of regulation here is urgently needed, and that an unexpected cause of an AI bubble pop may end up being that humans simply aren't able to keep up with the pace of the output - leading either to precautionary plateauing of capability, or major liability risks related to a decline in quality.
Humans are not immortal and cannot spend all their time into review (especially unpaid). Even today, there’s so much knowledge around that you have to be specialist of a narrow domain to get to the frontier. Even in computing which is just approaching a century of existence.
I would question the narrative that humans lack the capacity to verify the output and would instead argue the people lack the incentive to verify the output.
The response of many mathematicians to the recent dump is a good example: verifying these proofs amounts to unpaid labour for OpenAI and wastes time that could be spent doing publishable work which ultimately results in money or personal success. The slop factor also compounds the work required to verify the output considerably.
For mathematicians, programmers or anyone, if the work required to deal with slop passes the limit, it is no longer in their own self interest to use LLMs. The expectation that people will use LLMs for the betterment of humanity against their own financial interest is baffling.
> see first-hand that large, complex projects that used to take them weeks/months can now be completed by agents with a prompt
Really? I start with a conversation for maybe 5 turns or so, where I ask it what would need to change, what the API might be, any database schema changes, URL schemes, and so on, and finally ask it to break it down into commits. Then I let it go, implementing 3-10 commits at a time via subagents. It usually gets the UI somewhat wrong, so there are followups to fix it. This is with Sol and Luna subagents.
A lot of other people are not just asking for features, they are asking for products or new programming languages or new web browsers or a robot that can wash the dishes and so on. They tell the agent to help write the plan and figure out the questions they aren't even thinking to ask, the tasks they aren't even thinking to plan, and so on.
Their input is just working on the "plan" itself, the spec or the request, iteratively, and so on, while the AI works fulltime on automating the delivery of that thing, improving the systems that are delivering that thing, asking questions about what exactly is being built and how it should work, and so on.
A lot of the difference in how people experience AI is ambition and scope and how much you just tell the AI to 'automate your factory factory' in Factorio terms vs 'hand mine the coal and hand-craft the science packs' in Factorio terms.
EG, if you don't know how you'd speed up the process of building a web browser from scratch (compared to the current way you've been working with agents), you would ask your agent to help you figure it out and design that process itself, rather than just try to do what you're currently doing on a larger scale. You'd literally prompt "how can we make it realistic to build a whole browser when I've been just sitting here planning small changes with you" and then let it act on its response.
Yes, and this is completely normal. Not sure I get the point of his post. Most likely, stating the obvious is what you gotto do to keep your followers "engaged".
The average person does not have the time to occupy themselves with a new thing to see what it can maybe do for them if they combine it with hundreds of others.
Until non-chat-bot AI comes in discrete usable packages, mass adoption will be limited, because it's all just a big bunch of lego pieces whose utility is hard to grasp, because it isn't self-evident.
This isn't new, though. It's how "technology" usually works. For example, human(oid)s have been able to stick sticks into things for ~500k years, but the fork wasn't "invented" until ~2-4k years ago.
AI is a background technology, a building block. Something that is best appreciated when you don't know it's there -- when it just does a job (well). The end-in-itself POV of the tech industry is self-indulgent and irrelevant for the average person.
It's actually sad how much AI has either not improved life at all or actively made it worse from the perspective of the average person:
Your car is still the same.
Your dishwasher is still the same.
The train you take to work is still the same and never comes on time.
Fuel is more expensive.
The roads are still congested with traffic.
Your kids are (probably) doing worse at school.
Food costs more.
Houses cost more.
Rent is higher.
Buying a computer or a PS5 is more expensive.
Politics is still full of mentally unstable people.
The environment is getting worse.
You will still die of heart disease or cancer.
Food quality is worse.
Your kids can't find work.
Wealth inequality is accelerating.
But hey...on the flip side...a lot of people are rapidly building software (that nobody is using), and AI is solving mathematics problems (that nobody - including mathematicians - wanted it to solve)
Imagine what a "country of geniuses in a datacenter" will be able to do? Apparently...nothing.
This is sort of my take. You could look around in 2016 and 2026 and honestly not much has changed... at least at first glance. There is a missing piece of the puzzle.
I'm noticing this even in industry. Yea, we can build and iterate on ideas 100x faster but has this turned into anything meaningful? Not really... Hell, look at the big companies spending billions on inference... nothing shipped and things are still buggy as ever. Where is everything?
We went from spending a lot of money on people to enshitify products and then charge more for them to spending a lot of money on agents to enshitify products to charge more for them.
How many more software projects are there really left that make economic sense? Zuck learned the hard way re. metaverse.
Not many - the set that make ECONOMIC sense is finite. Even though many want to believe its infinite because all of a sudden everyone can vibe code an app cringe.
If anything AI is going to cause an insane amount of consolidation - what used to be across 3 diff apps, that can be consolidated will be done -> many businesses will die. It will be painful.
Google was attacked by an activist investor pre LLM hype for being bloated - they promptly laid people off. Many software firms have been allowed to bloat up do to the nature of the business.
Not sure I can agree, but the idea is interesting.
While the set is finite, software comes and goes. Just look at the tools to build software over the years.
> Many software firms have been allowed to bloat up do to the nature of the business.
I can agree with this. It will be harder to justify engineers sitting on laurels. Google could afford it before but I can see how it will be under more scrutiny now.
Why even bother when an AI is just going to do it better than you can? They have more money, more compute, and most importantly, will have next to zero cost of labor.
Fck disruption. The fetishizing of disruption is annoying. The tech became all about destroying something someone else has. What about ... create good stuff, create non-gambling markets, make peoples lives better.
* Higher load on senior engineers who are about to burn out hard from fighting an endless battle against unreviewed slop people keep trying to yeet into production
* Less junior hiring and suppressed wages
* Culture of uncertainty and fear
* Computers are really expensive now
* Some people now live with diesel generators and gas turbines making their neighborhood hazardous to their health
> Higher load on senior engineers who are about to burn out hard from fighting an endless battle against unreviewed slop people keep trying to yeet into production
Gosh, this has been my entire experience with AI, and I’ve never seen it this well articulated. Thank you!
Totally agree. IMO it's mostly behind the scenes stuff though.
I think some of the most impressive stuff to show someone from 2016 might be LLMs doing a summary of complex topics, followed by self-driving (actually) cars and intricate robots... but other than that there is no "future magic" touching the lives of every day people going on.
Actually I'm pretty sure if you showed someone the latest and greatest from these big AI companies, these AI companions that can book flights, people would go "Oh, so its like a smarter Siri" and proceed to not care at all lol.
Less junior hiring is a reflection of a) the maturity of the software business b) current expectations of the future
I have no doubt startups in particular are able to get away with using LLMs to produce sloppy code and ship. But this trend was on-going and sped up due to the infiltration of LLMs and marketing.
You do realise markets are highly imperfect in reality? That nonsense in econ books is fantasy.
Even in software, there are many things that haven't changed.
e.g. Why do Rust and CPython still have over 5,000 open issues each? At some point, will the superintelligence fix all the bugs autonomously? When is that going to happen?
What about new features? In every thread about Rust, people are clamoring for new features and libraries. People wonder why there can't be a batteries-included standard library like Go.
So now that LLMs are so smart, is that a solved problem? Or are we still bottlenecked on the expertise of a few experts?
---
Also, what happened to AI's passing the Turing test? I see tons of slop articles that are obvious to most HN readers. They haven't fooled us apparently -- the longer the interaction, the more obvious it becomes whether it's AI.
(That said, I use AI all the time and find it to be a great tool. IMO is is a tool, not an autonomous intelligence with a will of its own.)
It has greatly changed for the worse
I can't get all the flashing lights out of my face now, more algorithms to hook you in, more social media, more gambling, more news, more things to catch your attention.
You are talking like we had this capabilities for a decades but it's been just a few months of truly capable AI. Virtually all of IT switched to agents that quick. Non IT people just started to make software for themselves. Vision and translation capabilities of LLMs help millions. People talk with AI's now verbally to get advice or to just hear a friendly voice. The math thing is just few weeks old. We are just getting started. How fast did the telegraph meaningfuly changed lives for the better for bulk of people? Radio? TV? Computers? Internet? Cellphones?
Has any technology made an impact on physical world faster?
In all fairness, depression and suicide have existed before LLMs.
Safety systems are not optimal but improving everyday and although I doubt we'll ever get a perfect safety system, it seems that we're not very far from a "good" safety system. The days of Google recommending glue on pizza and generating diverse Nazis seem behind us.
Not to mention all the untold good that LLMs have done. Personally I've seen that LLMs are incredibly good at pointing conspiracy nuts and extremists into the right (moderate) direction.
Exactly, we're all software engineers here and even for us it took weeks/months to get to a position where we can really leverage LLMs (think platform, security, observability, etc).
Imagine being in charge of a physical factory, sure you might have a 200 USD Claude subscription but you also have a floor full of physical machines that have, if you're lucky, an undocumented debug interface and a paper manual. Not to mention safety inspections, insurance, etc.
You can't vibe code a factory floor. Not yet, perhaps in 5 - 10 years but there are some very hard, non technical, problems that need to be fixed before we'll see meaningful improvement there.
How much time do we have to wait before we are allowed to criticize AI? GitHub copilot has existed for half a decade. Cursor, Claude code, codex, have all existed for over a year and have seen more investment that practically any product in history and have almost 0 barrier to entry. Where is my decade of software progress in 1 year, because that’s what 10x productivity looks like! All I see are the same old broken products and services from basically the same handful of big players, higher costs, and more bugs. Remember when every year iOS and Android saw loads of new features and a radical rethink every few years? And that happened on ancient hardware and in a significantly smaller industry. Where is anything like that happening now?
I'll criticize Copilot with you. It's terrible and close to useless and it's a terrible shame that many companies got stuck with it due to being early adopters of this crap. I'm sorry for my friends who are employed by such companies.
Progress between Copilot and something like a gpt-6.1-sol in a good harness is at least as big as few Android versions.
I think this is related to the structures we’ve built in society. It’s lower friction to write software than just about any of the problems you listed.
It reminds me of the Ezra Klein quote that “OpenAl would need permits to cover its parking lot in solar panels, but it can accelerate into recursive self-improvement, as best I can tell, whenever it so chooses."
A lot of it isn't low friction really just in terms of political will it is high friction. E.g. housing issues are due to failing to upzone to meet demand appropriately. Solving housing is therefore free. It could be done today in any city in a housing crisis with the stroke of a pen, and before long, the market will equilibriate at a level better afforded by the median wage earner in the economy. The decision not to do this is a deliberate one: prices increasing is very much the goal of this sort of policymaking.
Likewise when you see other such shortcomings in our world, ask what this incentivizes for otherwise. Then things will become obvious why they are how they are, from housing, to transit, to feeding people, to healthcare.
Correction: My car spies on me and its cruise control no longer works because it slams on the brakes in the middle of the highway whenever it sees a mirage. My dishwasher broke because it got an unasked-for software upgrade.
Things worked better before we started putting bad software in everything. As a software engineer, I know what good software looks like. The fact that almost all software is now bad -- including the software that drives AI and the software that AI generates -- makes me want to flip the table. We know how to write better software and the general public should hang us if we continue to refuse to deliver it.
Probably preaching to the choir here, but the software is just a symptom.
The problem are the power-relationships and laws which allow someone to abuse you and your stuff, block competition, and even put you in jail for trying to remove the software—or helping others to remove it.
> Correction: My car spies on me and its cruise control no longer works because it slams on the brakes in the middle of the highway whenever it sees a mirage. My dishwasher broke because it got an unasked-for software upgrade.
Never, ever connect appliances to the Internet. I thought this was common knowledge on HN. Also pull the telematics (cell modem) fuse to prevent your car from snooping on you and selling your location and driving habits to data brokers and insurance companies.
Yes, but this is ridiculous for society. It is to the level of "do not wear a nice watch if you do not want it to get stolen": there should be no need to avoid wearing a nice watch or to pull a telematics fuse from the car in the first place!
This view is too cynical. The frontier advancements are certainly going to trickle down. Take cars for example. AI driven cars are already here. AI-designed battery systems, fuel systems, materials and 3d printing capabilities will come to cars. It will likely lead to cheaper, more efficient and more effective vehicles. If will just take time for the impact to percolate.
Worse, the view is largely phrased as gaslighting AI. Blaming the cause of particular grievances that coincidentally have been sourced at the same time as the rise of AI, is purposeful.
> [Your train] never comes on time. Fuel is more expensive. The roads are still congested with traffic. Your kids are (probably) doing worse at school. Food costs more. Houses cost more. Rent is higher. Buying a computer or a PS5 is more expensive. Politics is still full of mentally unstable people. The environment is getting worse. You will still die of heart disease or cancer. Food quality is worse. Your kids can't find work. Wealth inequality is accelerating.
None of this has to do with AI.
> But hey...on the flip side...a lot of people are rapidly building software (that nobody is using), and AI is solving mathematics problems (that nobody - including mathematicians - wanted it to solve)
Plain wrong on both counts.
> Imagine what a "country of geniuses in a datacenter" will be able to do Apparently...nothing.
Nonsequitor, probably based on a personal beef. Datacenters, at best, are understood to be for hosting more AI - not for people to somehow live in as an arcology of discovery.
I think only because these are all things that AI companies are saying will be helped by AI so give us more money. Nobody advertises their product publicly as "you will be able to buy other product to do the same things you do now but pay people less and make more profits" even though that is where the value is for them.
How can you say tickle down economics doesn't work? The US has made it easier for businesses to grow and the result is the highest standard of living ib tje world.
Trickle-down economics specifically refers to economic policy that favors the wealthy and corporations with the argument that this will lead to wealth trickling down to the economically less fortunate. Examples include the Reagan and Trump tax cuts.
This has been studied somewhat extensively (https://academic.oup.com/ser/article/20/2/539/6500315?login=... is one example), and it’s been shown over and over that this does in fact not work and leads to larger wealth disparity, across many countries where it has been attempted.
It’s possible you’re working off some other definition of the term based on your comment but that’s the generally accepted usage.
The person you replied to was the second person I've seen on HN this week try and defend trickle-down economics explicitly in more or less the exact same terms and justifications. What is going on with that? Did Peter Thiel tweet some pro trickle down dreck recently that these people latched on to?
This is a temporary anomaly due to the datacenter buildout, which will not last forever. Historically, tech has always trickled down. Sooner or later, hardware will again become cheap.
"Temporary" is a few months. We're looking at many multiple years before parts even _begin_ to decline in price, if even. How much will we as consumers lose in the time it takes for this to correct? This is not "temporary", this is a decimation of an entire product sector for the foreseeable future. There is no historical precedent for this. Companies are not and will not slow down gobbling up literally every tech part they can. Every year, training and running AI will demand more resources over and over and we will continue to be charged increasing amounts for everything because everything costs more either directly or upstream of us. It is a literal cancer.
> Every year, training and running AI will demand more resources over and over and we will continue to be charged increasing amounts for everything because everything costs more either directly or upstream of us.
Unless we are really going to hit the Singularity and create Godlike superintelligence that improves beyond all limits, sooner or later the models will stop improving, no matter how much compute we throw at them. After that happens, the best models will be etched into model-specific ASICs with weights fixed in mask ROM (like Taalas is already doing), which will result in massive, order of magnitude inference efficiency increase. Couple this with the training datacenters suddenly becoming available for inference, and the cost of compute will plummet.
Yes, it might take a few years, but the hardware will be cheap again. And if the result of the whole process is human-level AI, then it will be worth it in the end.
My prediction is that while those “advancements” may come to cars, the only practical differences will be lower quality products, cost savings for the car companies, probably more expensive for consumers, and somehow it will also be collecting more data to sell to advertisers.
AI car navigation is here, but driving is a social act. Have you ever tried to keep an automated taxi out of a private driveway or the scene of an emergency? There is a lot of work to do and limited evidence of LLMs raising the bar.
If you look at the average price of a new car, adjusted for inflation, it has pretty much always gone up aside from recessions. There have been plenty of advancements in car technology, they do not make them cheaper. It actually gives reason to charge more for them due to perceived increase in value.
Yes but you're getting a lot more for the same price. It's a lot more powerful, comfortable, safe and efficient while the effective cost has remained the same.
I would argue that cars are sufficiently advanced now and there are more important things to focus on. I don't care if AI moderately improves the comfort or efficiency of a new car while it took my job.
Obviously we get things in return for the social and economic fallout, carbon emissions, pollution, destruction of education, etc. that AI has provided us. I just haven't heard a compelling argument for why that is nearly enough to warrant this.
Again, the effective cost has not stayed the same, cars are getting more expensive. I am all for increased safety, I just wanted to throw a little water on the idea that cars(or really anything) is getting cheaper. You may be getting more performance or safety per dollar, but thats a value proposition that keeps people paying more. All the convenience and "power" features I can leave, I don't need a car that goes 0-60 in 2.8 seconds because why would anyone need that on public roads.
I drive a 1993 land cruiser, fwiw I would not put my kids in it for long drives because I don't think it even has air bags.
This only works if there remains a viable and attractive consumer market for these advanced goods. AI is both an imminent wealth shock and perhaps the largest structural change to the economy in history as it threatens to drive the value of human labor to zero over the next decades.
In American history, when previous shocks like this occurred (e.g. with the Great Depression), we needed massive economic programs like the New Deal to restore consumer demand. When such programs are not forthcoming while wealth inequalities keep increasing, one can look to other countries for what ends up happening; either the disenfranchisement of ordinary people over time or unrest and redistribution by force.
> If they actually had access to systems that could do "years of top professional work" across software, cyber, science, and math, they would not mostly be leaving to start more chatbot companies, agent frameworks, eval tools, and "AI for workflow" startups.
> They would be starting normal companies to disrupt the global economies of the world.
Many people who work in such industries already know an insane amount of inefficiencies. Unluckily, they are not in positions and have no political power (with respect to organizational policy) in the company that they are capable of changing the situation.
Additionally, many of these sectors are heavily regulated, so it is very hard to make a difference in this sector as a startup founder, and even if there is an entrpreneur who founds a company to change the status quo, an insane amount of work in the company will be handling the big amount of red tape.
As someone who is building one of these referred to systems in a major vertical in the Fortune 500, these LLMs are very much being used to eat major legacy systems and processes. There are still plenty of obstacles to overcome but the number of AI governance and steering groups that I have been apart of are indicative of what's happening in a major industry.
> As someone who is building one of these referred to systems in a major vertical in the Fortune 500, these LLMs are very much being used to eat major legacy systems and processes
Those LLMs are not being wielded by 2 tech bros in a garage replacing the 100 000 person company.
Those LLMs are being adopted by the huge enterprise itself and being surrounded from all sides by a very complex procedural harness (aka regular programming) that stops the LLM from doing dumb stuff.
Videos are shittier. Most songs are even shittier than the autotuned pieces of shit already were. Ads are worse (even though nobody thought that was even possible). Customer support is worse.
> But hey...on the flip side...a lot of people are rapidly building software (that nobody is using)
That nobody is using and that are worse than what we used to have. And because nobody's using them, the countless issues plaguing them aren't even reported by users anymore (there are no users anymore). Full of bugs, usability issues and gigantic security holes (but which doesn't really matter because nobody's using them).
It's not even clear that it's making long-standing software (like say Linux or Emacs or Blender or QEMU) better at all.
Now I'd say though: most of the issues you mentioned have nothing to do with LLMs / AI. But it's very clear that AI ain't solving much of the world's problems and that the promises we got years ago didn't materialize.
All you said is true, but on the flip side it is giving an individual user huge power over software. To create their own, to reverse engineer proprietary ones. Decentralisation of such power can only be a good thing.
Even if a certain small group of people are also using the hype surrounding this tech to attempt to destroy the home PC market and reverse the personal computer revolution. They will not succeed. There is more of us than them. Even with "thousands of frontier LLMs". A human+llm is going to be more powerful for a long time rather than a bunch of LLMs.
The power isnt really decentralized when you see that the internal AI models are superior than anything you can self host, and the right to use those can be stripped away at any time.
This is like saying food is shittier today because there are now more restaurants. No, food has never been better, there are just a ton of shitty restaurants.
More like saying samey corporatized slop bowl copy paste jobs (eg Cava) are driving out real, quality restaurants, a thing that has actually been happening for some time.
That pretty much sums up my experience. Everything is more expensive and worse at the same time. And it feels like we're being screwed over from every direction and in new ways every day. People keep talking about this tech is supposed to do so much but everything just gets worse.
Private equity bought out the healthcare system where I lived so now it takes months to get an appointment and the quality of care is horrible.
Not that matters since the health insurance I pay hundreds for a month wouldn't cover anything anyways because it's through UHC.
My job keeps getting rid of positions because they want to "do more with less" but it just means I'm doing more work while being paid the same.
If I buy a new car it will spy on me and sell my data. The state DMV already sold my personal info without my permission and now I get constant spam.
Private equity bought out many of the restaurants I liked going to and ruined them.
Third spaces in my city have been mostly destroyed.
The social media sites I used to use were overrun by bots.
I can't upgrade my PC because hardware prices have gone insane.
Gas is like $5 a gallon now so I can't really justify going on any trips anymore.
Private equity bought up most of the rentals where I live and so now I pay much more for rent on top of crazy monthly fees while the building I live in is falling apart because they refuse to fix it. Housing prices have also gone up so much that I have no hope of owning a house.
It's not true that everything is more expensive. Nominal prices are higher, but incomes have risen faster than inflation. Purchasing power of the average person in 2026 is higher than it was in 2016.
As for things getting worse, I guess it depends on the domain. In hardware, things have gotten a lot better; my desktop computer is orders of magnitude more powerful than I would have expected 10 or 20 years ago. In software, I do see a lot of bloated software being written, but I don't feel that the software that I personally use has gotten worse.
Average, or median? Don’t know the answer offhand, but there are a small number of well-known people who are likely to skew the average quite a bit upward, given how much wealth they accumulated during and since covid
I would just add that consumer products in general aren't the same - practically every business model has incorporated obsoletionism to the point it's practically cancelled out the curve of progress and evolution of tech. We could have nice things.
At least for me, personal agents have been quite useful. If you integrate them well - shopping, filling random government forms, keeping track of my personal life, optimizing training plan and nutrition, planning vacations, etc. I can do more home repairs or renovations myself with help of chat
So basically you get personal digital assistant - very useful - but not changing my life.
What would I need for real transformation though? Food, house cleaning, commute - "every day chores". Even if we had ASI tomorrow, what could it do to change my life, besides taking away my job? Design good house robot to help me with food preparation I guess... Maybe finally self driving cars launched in my area? Or it will take control, "optimize humanity survival chance" and realize we don't need as many people on this planet.
> Almost every time I use an AI chatbot it eventually turns out it's lying to me.
I would not trust them with any of the things you mention.
If you talk to free chatgpt, sure.
But if you launch Opus 5.5 in claude code - for some reason Opus is the smartest for me, even for "general" type of questions, when launched from Claude Code. It has harness/prompts optimized for correctness, while more "customer-grade" products, even when used with top model tend to optimize for latency. And I tend to give "context" - e.g. I have "personal wiki" + skills for lots of those things.
I don't recall i last few weeks where Opus 5.5 in CC made an obvious lie, made an unreasonable assumption, or made a mistake that smart human couldn't make. E.g. if it "failed" it failed in a way that non-genius human could fail in similar way.
opus 5.5 uses agents and skills in chat sessions as well and maintains internal context that it tries to distill from what you provide
I tried to have it write a synopsis website for the different types of dance patterns in Dance Dance Revolution, with guides on their execution, explanations, and provided song sources. I provided all of the source articles and search websites needed for it to be able to do that, and gave it a sample set I wanted it to utilize and build off of.
It has bits and pieces of information wrong in 90% of the sections it created. Poor song choices, missing song choices, incorrect explanations, incorrect step-patterns, the list goes on.
If you're not an expert in a given field for which you're using it, it's going to look like a genius. If you are, it's untrustworthy and borderline useless.
Maybe it wouldn't be if there were ample training data on my topic, but there isn't.
It utilized the research from millions of papers many very close to the destination answer to make those proofs. What would it do on something that isn't supported like that? Be confidently wrong while sounding right. That's what we've seen repeatedly.
> If you're not an expert in a given field for which you're using it, it's going to look like a genius.
For someone highly literate, llms aren't going to help you write better. If you're borderline functionally illiterate (this is > 50% of the population according to some studies), I'm sure the llms are awesome.
I've long thought the same about all the hype around coding (plenty of examples from years ago now), but will continue to test the new models. They are improving.
I think my anxieties would be different if I were confident a DDR guide full of lies would be spurned and unprofitable.
In a way, I'm not worried about the LLMs per se as much as a kind of moral and ethical collapse. A debasement of the currency of communication, a market of intellectual-lemons.
I saw electroboom's latest video lamenting facebook AI slop about electricity, such as a way to wire a three-way switch that blows a fuse when you turn the light on. For some reason, I am shadowbanned on HN.
I have ChatGPT Pro and it sources every single thing. For my Asia trip right now it's been helping me every single day to figure out optimized routes on what to do and where to go (if I feel like it, not to say I don't wander around too).
Why? In the past I would've used Google so it's just a faster search engine for me that works with natural language, what do you use for planning then?
> Indeed, planning an itinerary, making calculations, building excitement for a place, is part of the journey
Not sure how they're perceiving using ChatGPT to plan an itinerary but I'm also doing all this while using ChatGPT. As I said in the sibling comment, it's a faster Google search and route optimizer, not something where I type in "make me an itinerary for X city" and follow whatever it shits out.
How often do you check the citations? I’ve found with the Google search AI summary it will provide answers and link to their citations, but a good chunk of the time the answer isn’t on the linked page.
It just makes stuff up (or gets it from somewhere, who knows) and then links to a tangentially related page as if it was the source. Either that or it identifies the page as being a source it wants to use, and then summarizes it incorrectly. Not sure the order of operations it’s doing internally, but regardless the citations are bullshit.
Google's models are worse that OpenAI's, doubly so on Google's free AI overview versus ChatGPT paid plans. 6.1 Sol is very good at synthesizing actual information it finds without hallucinating and I've not seen any instances it failed while I checked the sources.
The citations provided by the free version of Gemini are comically bad. They almost never contain information which directly supports the relevant claim, and more often than not they're completely irrelevant.
me too, but I recently blew the transformer on my boiler by following its directions to wire up a C wire to power a smart thermostat. oops. $500 mistake.
Every time I use Google search AI for anything more complicated than a search query I feel that way. A paid model from pretty much anyone is much better.
Yeah, I’d hope that on HN, of all places, people know that free models optimize for price and speed, not for accuracy. But even with free models the accuracy was massively improved over the last year.
This is the kind of gap in shared understanding that is widening.
When people use words like “chatbot” to talk about their AI use and can’t really name basic things like the model they were using, it reveals a wide chasm between the people who are deep into understanding how to use these things and those who poke at easy options every few months and then look for ways to confirm their priors instead of trying to understand how other people are getting value.
using skills with opus 5.5 and agents for each individual task that have shared knowledge of all of your records does not lead to a significant increase in trustworthiness
The incentives seem to be lined up to push users to the frontier of "how much can the harness around the model do and claim without most people pushing back on the claims."
The bigger the pile of stuff you have the tool create unsupervised, the more incomplete/inaccurate the overall picture of it is gonna be. That doesn't matter for most hobbyist/helper-tool/game-port/etc type of tasks.
I get a ton of value out of the tools and yet part of that value has come from being hyper-aware of the shape of the limitations, which changes much less and more slowly than the specific of the limitations.
"How'd you notice that one!?" - well, it helps to be the only person on the team who took the time to interrogate the e2e interaction of all the code...
The median piece of OSS software was already pretty rough once you had to depend on it in production at scale to make money. The median piece of the new AI-generated wave of software, also pretty rough.
The bet that OpenAI/Anthropic is making is that they can automate the interactions generally/globally enough to work around this and eliminate the need for humans in the loop. But it's a tricky problem because it's precisely where the permutations and special cases get nasty and much harder to deal with than more broadly-applicable cases of codegen/test-gen/mash-it-until-the-tests-pass-and-the-code-review-angent-is-happy.
This has all started to feel very Emperor's New Clothes-ish. "Well, if it's not working for you, then it's your fault. You're just not using it properly. You're not worthy of its wonders".
Not to say there's no value to be had. However, it's also been oversold and its error-proneness is a real problem.
But, it seems there's a steady stream of commenters and creators who are quick to remind us that it's never a problem with the tech. It's always a user problem.
Meanwhile, the errors are renamed "hallucinations", but let's not focus on those. In fact, if you just look right over here at this .md file...
> When people use words like “chatbot” to talk about their AI use and can’t really name basic things like the model they were using, it reveals a wide chasm between the people who are deep into understanding how to use these things and those who poke at easy options every few months and then look for ways to confirm their priors instead of trying to understand how other people are getting value.
While the latest and newest should be needed for greater capability, the average should be enough to give a glimpse pf what is possible. Like a basic netbook is enough to enjoy computing, no need to purchase the latest supercomputer.
If you need to switch your setup every quarter or so, it’s not a tool, it’s a rat race.
Personal agents are wonderful! You can ask for hiking trails and it will customize one for you, frequently resulting in emergency calls for rescues! Your purchasing/selling AI will arrange sales for you but forget to tell you, leaving you in breach of contracts you didn't know about! Your personal nutrition agent will absorb all the broscience and customize a diet that will result in hair loss, but at least your in ketosis, amiright! And instead of hiring a plumber/electrician/contractor to do work subject to code, you can just ask an AI to tell you how to do it and then wonder why your buyer is suing you for $$$!
(Those are all examples from just the past 4 days, and just the ones that made it into mainstream media.)
This is the core hubris of everyone behind the LLM craze. I do not want to deny their utility there absolutely is some there. But the inherent assumption behind the whole thing is that with a "country of geniuses" problems will melt away. But that's not the case. The reason problems like housing or famines or war aren't being solved is because structurally there's no incentives to solve them not because there's not enough geniuses out there.
> The reason problems like housing or famines or war aren't being solved is because structurally there's no incentives to solve them not because there's not enough geniuses out there.
Rather: these problems are not solved because there are not enough geniuses who are capable of convincing the masses to send the responsible politicians to hell.
I’ve heard people say variations of, “Clever people can easily convince themselves of just about anything, if they’re inclined to. And then it’s nearly impossible to talk them out of it.”
Kinda seems true. And the worst people in history often don’t seem super dumb, despite their various other defects.
This is broadly the same people who bemoaned the idea that the "smartest people in a generation" were all just working on adtech without any reflection about maybe that's not actually an exclusive set of the smartest people... or that a lot of the people in that set were the same one who might, twenty years ago, otherwise be chasing money as middlemen in finance vs doing productivity-expanding things anyway...
You can be smart and chase the money, but it's crazy hubris to assume that only your peer set of money-chasers are smart.
It's a small leap from there to "since we're the only smart people, we should get to have the most influence."
For that you might need the kind of geniuses who write stirring books about principles and values. Or whatever the modern equivalent of those books is. Maybe nobody would pay attention to that kind of advocacy in the present day, not sure why. But it's not tech geniuses, anyway. Much though tech has power as a catalyst.
I suppose there was Project 2025, and there was that book about the inevitability of Russian hegemony (or empire or something) that Putin liked. Those got attention with influential people. Not sure what to conclude from that.
Democrats expanded SNAP, Republicans shrank it. Dems expanded Medicaid, Republican shrank it. Dems protected reproductive rights, Republicans repealed them. Dems expanded title IX protection to LGBTQ+, Replicans eliminated them. Republicans jailed 1,200 January 6th insurrectionists, Republicans pardoned them. Democrats provide non-partisan federal emergency aid, Republicans withhold federal aid from blue states. Dems establish tax credits for electric cars, increase automotive standards. Dems didn't kill of foreign aid to need countries.
If you aren't female (reproductive rights), LGBTQ+, an immigrant now hunted by ice, someone in need of SNAP, Medicaid, or emergency aid in a blue state, it might feel like both are the same I guess. If you are one of the above, and now suffer because so many voted based on 'both are the same', you very much feel a difference today than from the US under Biden. If you were one of America's allies but are now constantly attacked, you feel a difference today. If you are Ukraine, you feel a difference today.
> Rather: these problems are not solved because there are not enough geniuses who are capable of convincing the masses to send the responsible politicians to hell.
People support policies that make housing much less affordable (e.g. rent controls) and oppose policies that would make housing more affordable (e.g. zoning reforms). Saying that it's evil politicians that cause this crisis is dishonest - it's what people want.
People by and large don’t understand the second and third order effects of their policy preferences. It’s not that they don’t want those things, it’s that the world is more complicated than their mental model insists it is. Politicians could try harder to fix the fundamental problems but they don’t because the long-term solutions don’t manifest until after the next N (very expensive) election cycles.
The politicians are those who created the respecting laws.
"Convincing the masses to send these politicians to hell" is thus nearly isomorphic to making the masses hate politicians who create such laws, which at least very likely implies that by convincing the masses to send these politicians to hell, a different mindset emerges in the masses.
> The politicians are those who created the respecting laws.
Yes and general population LOVES these laws, even though they are harmful to them. But anyway - you've got your causality reversed - it's voting masses that incentivize rent controls and other shitty politics. Remember that to implement your policies you've got to actually win elections. And since people generally don't understand n-order effects for n>=2, it's really easy to win elections with populist policies.
There is no evidence that reducing rent makes the rent go up - only fearmongering. It's a stopgap measure, though, since there still won't be enough housing to meet demand unless you also fix that at the same time.
Rent controls work if you go radical enough and enforce a max profit margin for the whole city, with rent limited to something like 30% over maintenance + debt repayment costs.
The amount of housing in the city won't grow but it will free up a big part of peoples income that would otherwise go to landlords. That helps the local economy with more consumer spending, and makes it easier to save up or apply for a loan so you can start a business.
You get a better economy and more money in the average persons wallet without raising taxes. The only ones who lose are commercial landlords, and they will stay profitable by design so it's easy to shut down any complaints. And you get better quality apartments as a side effect because the only way to get more money out of existing properties is to spend more on fixing them.
That policy would 1) fuck over younger generation, 2) result in total sclerosis and death of city. Prices are high because there's not enough housing, that's it. Rent control won't increase number of units so it wont solve real cause of the problem.
> The only ones who lose are commercial landlords, and they will stay profitable by design so it's easy to shut down any complaints.
No, also people who would like to move into city would lose, because that would become literally impossible. You'd have like 15 years waiting lists for housing.
> And you get better quality apartments as a side effect because the only way to get more money out of existing properties is to spend more on fixing them.
Why would anyone fix apartment that's under rent control? If you mean system like in Berlin, then I've got bad news - people there are actually frequently not allowed to renovate even THEIR OWN apartments.
> because there are not enough geniuses who are capable of convincing the masses to send the responsible politicians to hell
I could talk about this for hours, and was last night actually.
What do vanguardist communism, feudalism, liberal democracy, and fascism have in common? A shared belief that the general population is composed of a bunch of dumb dumbs that need super smart (read: politically educated) people to "lead" them.
Seems subjective. When is the smartness, big smart enough to allow someone to enter the hallowed ranks of "actually a real human allowed to speak for the masses?"
There does seem to be a clear delineation between the people you hear about (the vanguard in our example), and the people you don't hear about (the masses): willingness to create confrontation. Most people want to just go with the flow, live in peace. In a dichotomy I've been calling "choose peace or violence," they choose peace. They choose to see bad things in the world or their life and focus on managing their happiness around it.
I think that this is the default human mode of existence, and that this mode lacks defenses against people willing to violate the default peace. Imagine a bagel line, and someone cutting that line, and when tapped on the shoulder, the cutter breaking into a huge tantrum, and people in line backing down with a "Jesus! It's not worth it, fine." Thus some people learn that they can be much more selfish in life if they're willing to create confrontation and disturb the peace.
When I talk to people about agitation, civil disobedience, or even just unionizing, the most common thing I hear is anxiety. What if the situation gets worse, what if there's reprisals, what if I lose this little bit of comfort that I've managed to be happy with? That slice could get smaller and smaller but people aren't willing to take the step to say "enough is enough" because I think it fundamentally goes against human nature, we have to sacrifice our own happiness on some level to put our foot down and demand or make change. At minimum we have to acknowledge the discomfort or injustice, which is a diversion from our constructed internal peace.
Personally I think people should be allowed to live in peace. Maybe the liberal democrats and vanguardist communists are right, maybe most people shouldn't need to trouble themselves with the greater direction of their society. However I think the stability this promises is false; in fact I think all stability is false. I think this constructed peace that people are afraid to disrupt clearly can be torn apart at any moment, not just by people willing to create conflict, but by a literal earthquake, or fire, or plague. Therefore I think there's a couple societies in history that got it right: ones who distributed the responsibility of society across all individuals in the society; namely the brief period of syndicalized Spain, and the several hundred thousand years of certain tribes of native Americans on the eastern portion of the continent. There may be other societies in history, these are just the cases I've been able to read about so far.
It seems to me that it's possible and necessary for all people to be ready to wake out of their constructed peace long enough to make adjustments together to maintain it. I don't think this is the same thing as them being "led out of it," I think this is each individual being on some level alert to the need to spontaneously say "no, here we wait in line for our bagels. No, screaming about it won't help you, get to the back of the line. Look around: everyone is laughing at how silly you look."
> What do vanguardist communism, feudalism, liberal democracy, and fascism have in common? A shared belief that the general population is composed of a bunch of dumb dumbs that need super smart (read: politically educated) people to "lead" them.
That was not premise of fascism. Certainly not Hitlers one, he himself was self-taught and did not framed himself as highly educated. He considered himself better then educated people. The origin of his superiority was not "smart" it was "strong will and power itself".
It was not the original premise of feudalism either. That was was about military might, basically.
And origins of liberal democracy are not "smart based" either.
been a while since ive see the real talk on hacker news. its refreshing. usually I feel like I'm the only one among nerds asking to hold people accountable.
For a lack of deep knowledge in this area, I don't think I have answers for this, but if one is interested in working on this problem, my approach would be to start with diving deeply into scientific literature about psychology and sociology to find out
yeah, sure. Do you think all scientific literature is the same though? I am skeptical of thinking to just look at them all equally. Clearly, if theres a class of people being held unaccountable who have been practicing imposing their will and power on the people, that has certainly been done in the scientific community and would show in the literature, too.
So when I think of going through any media like scientific publications, I wonder how to discern and point out the captured and biased media vs the more objective ones (or at least the ones that don't have a hidden financial motive behind them). I wish our scientific community had better disclaimers and traceability for financial influence in every paper we could read.
>
So when I think of going through any media like scientific publications, I wonder how to discern and point out the captured and biased media vs the more objective ones (or at least the ones that don't have a hidden financial motive behind them).
This is exactly the hard part: a topic which many people in more applied scientific disciplines worry about. Welcome to the real word, Neo. :-)
Again, I don't have definite answer, but rough guidelines are:
- is a result too good to be true?
- is some result coherent to results that are much more established?
- why would you assume that the author is unbiased or biased? (biases not only have financial reasons)
- which agenda does the author have (yes, deep pursuit for scientific truth is also an agenda? Nobody is free of an agenda)
- does the author make an effort to explain how the findings could be explained in an alternative way that makes the result look much less impressive?
- could the results be reproduced in other labs?
- which is the minimum amount of facts that you must believe in (like axioms) such that the result holds?
I like that, I can see that being true and working.
The hard part is indeed that we have to exert all this effort to get past the noise that I (and maybe you) feel a responsible author should have already done for us.
And with AI: I was hoping it would help us filter through this noise. So far, I feel lost in how to use AI to help me filter through prejudice and bias, and hidden motives and intentions, when the very filters given to us in the mainstream AIs we are sold are doing that same very bias and incoherency and hidden agendas. I retain my hope, however! Thank you for some solace and inspiration here, friend!
To add insult to injury, we're being told: "Just trust us bro, however we make things worse now will totes be redeemed as the prophesied god-mind magically fixes it later."
Strangely, they never discuss the possibility that Multivac might seize all their wealth and cast them down into average-ness. What if it actually comes about, but then it tells us to do the things we already refuse to do?
And the LLMs exist to further those problems. People can't afford houses? Well, now they won't be able to own computers either. That's a feature, not a side effect. Once the K-shaped economy progresses enough, you won't even be allowed to use the LLMs. They'll exist only to be a tool for the new feudal class who owns you.
Once capabilities catch up with the demand (~which is in few years - it just takes time to build!) average person will have much better access to higher quality and cheaper hardware. It's literally how it has always worked, I don't understand why would it be different this time. There's zero substance in your comment, just repeating tired tropes from twitter.
A market requires scarcity, because scarcity is what neccesitates trade.
You can't invoke market mechanisms to defend a post-scarcity scenario.
Why would the ai owners make things cheaper or even available at all once their abundance machines give them everything they want?
What would they get out of if it? What could ordinary people trade for it that their abundance machine can't give them already and that would be worth giving up their monopoly on productive force?
You can't argue "this will change everything" and "this is how it's always worked" simultaneously, and yet AI accelerationists bring out both to defend their ideology.
> You can't argue "this will change everything" and "this is how it's always worked" simultaneously, and yet AI accelerationists bring out both to defend their ideology.
Luckily I did not, you are arguing against points I did not make at all.
Either way though, what's more interesting than what you specifically said is that AI boosters have no answer to this structural argument about the reality of post-scarcity. Tellingly, you didn't address that part of my comment at all, either.
> Either way though, what's more interesting than what you specifically said is that AI boosters have no answer to this structural argument about the reality of post-scarcity. Tellingly, you didn't address that part of my comment at all, either.
You've fished a comment from completely different discussion, where a point was that if AI were to displace workers, it would be different than previous waves of automation (and there meeting these capabilities was assumed, not put forward by me). But in this thread it's more down to earth discussion about manufacturing capabilities if there is no ASI. But sure, I'll bite - yes, capitalism won't work in post-scarcity society. But we are not leaving in post-scarcity society and in our current situation capitalism is amazing system. If capitalism would get us to post-scarcity society then well, you'd have to admin that capitalism is amazingly effective. Your assumption about AI-rulers (I presume altman and amodei, not governments) having abundance machines and not sharing the abundance with anyone else is really without any merit. For this scenario to work we would need to meet few conditions that are so unlikely that it's not really worth entertaining them seriousl: 1) powerless government 2) actual ASI that's abundance machine 3) ASI that's perfectly aligned towards serving altman and amodei and none else 4) altman and amodei deciding that everyone should be poor.
And also as someone who I guess you'd call AI booster (so a person that's not deluded about AI capabilities and sees how powerful they are and how current concerns with AI such as water use are overblown) I can tell you how I address this - you can't stop progress. It never worked and never will. I think world would be better if we would've stopped developing AI. But it's impossible so you might just embrace it.
Companies have been trying to figure out how to get people to rent pcs for a while. Unfortunately the AI boom has shown them how, buy out all the capacity of the ram manufacturers, pay them enough consistently and the manufacturers have no reason to expand their capacity. No cheap ram means companies and then home users will have no choice but to come to you and rent their pc.
The problem isn't that it fails to unwind, the problem is what the unwind looks like. Last time it meant global upheaval and world wide capital incineration -- mostly metaphorical in the case of the US but very literal in the case of Europe and Asia. The poor suffer more in real terms but capital suffers more in nominal terms and that unwinds inequality and the burden of furnishing runaway capital with the runaway returns it demands in excess of economic growth.
I'm sure the same "strategy" will work again, but this time around everyone has nukes and drone swarms and AI-equipped bio labs so shit's gonna be lit.
You don't need a back room filled with cigar smoke where supervillains coordinate their plans while twirling their mustaches when good ol' class interest does the same job.
Luckily, the AI bubble will destroy a decent chunk of their wealth.
The Gartner Hype Cycle [0] is real, and ubiquitous. We are teetering somewhere near the top of the Peak of Inflated Expectations, about to plunge into the Trough of Disillusionment. A lot of very rich people are about to lose a lot of money, and we can only hope that they don't get bailed out (again).
Then, once the bubble and hype is over, the long, slow work of actually doing useful things with this new tech starts. That's when we'll see it affect everything else.
> We are teetering somewhere near the top of the Peak of Inflated Expectations
I've been hearing this for a year and a half now. That word, "ubiquitous", is getting real mighty challenged. I have tried to insulate myself as best as I could and it just never happens and every other day I wonder if my foil hat needs adjusting. Does yours?
> Luckily, the AI bubble will destroy a decent chunk of their wealth.
Even if all this is true (personally I doubt it), it assumes the bubble won’t burst before they’ve shifted the money from all our 401ks, IRAs and index funds into their own pockets.
> The reason problems like housing or famines or war aren't being solved is because structurally there's no incentives to solve them not because there's not enough geniuses out there.
When I asked why Musk doesn't put his money towards non-profits my Musk supporting friends say it's because he "doesn't believe anything but for-profit entities can solve the world's problems".
So why isn't he solving homelessness, famines, war, etc. with capitalism?
What I like to call at work "this is not a technical problem". There are many problems we could easily solve if we had the right structural incentives. But instead we have politics, processes, layers of management, etc. All that doesn't disappear (yet?) with AI.
I believe that’s actually one of the arguments of the boosters. Utopia will finally be reachable when we get all the human drama and bureaucracy out of the way
I’m not sold, but I’ve heard this line of thinking many times
Human drama is what people use to advocate for themselves. Bureaucracy is what people use to protect themselves. Then you have the side effects but, you may also not like a world without those at all.
Whatever benefits AI has to offer on a societal level are going to take a long long time to show up, give it some more time. We had smartphones in 2007, and e.g. By way of example, Uber didn't launch ridesharing until 2013, and then that still took years to seep into the public awareness.
By comparison, we're in the early years of AI. We're all still figuring out what this thing even means.
I agree with you, but those examples are going to fall on deaf ears. The pessimistic mindset will say that smartphones were a terrible invention that's ripping society apart, Uber is an unethical company exploiting workers, etc.
I think ridesharing was a net good for ordinary people even if Uber is/was unethical.
However, the counterargument falls entirely flat. "Whatever benefits AI has to offer on a societal level are going to take a long long time to show up". We don't have the luxury of time because AI disrupting communities and livelihoods faster than perhaps any other technology we've ever seen, with the most capital that has ever been put into any tech since the railroads.
Even if you're optimistic we'll eventually get the cancer cures and fusion reactors, doesn't it make sense from the perspective of an ordinary American to regulate the harms like the homework cheating, parasocial relationships, job losses, cyber attacks, etc., especially if it will take years to get the benefits?
Different benefits will show up at different rates. We don't need to wait for cancer cures and fusion reactors. It's already easier than ever to start a small business and use AI for things that would have been prohibitively expensive or time-consuming before. The technology is transforming how scientific research is done, how engineering is done, etc.
I don't see much evidence in the economic data that AI is disrupting livelihoods. Unemployment has been stable around 4% for a while and productivity growth has been at historically normal rates for the past 5 years. I think a more likely explanation is that AI is increasing uncertainty in some industries, which are seeing higher employee turnover, and many of those people are moving laterally into other jobs where their skills are more likely to be useful.
I don't think anyone in this thread has argued against mitigating harms. The OP was about current capabilities and the perceived lack of benefits for the average person.
To be clear, I don't disagree with your sentiment that the disruptiveness of AI is causing a lot of grief. But magnitude of disruption is not an indicator of magnitude of benefit (or lack thereof).
Some of the most disruptive periods in history brought about the greatest long term improvements in society. This is the price we pay for changing how things are done, sometimes for better and sometimes for worse. We don't know which one it is for AI yet.
I miss being able to reliably hail a taxi A LOT. Uber may be an improvement in smaller towns where cabs didn’t exist, but in bigger cities it is mostly a downgrade. Rather than walking to the curb and immediately getting a car, I’ve got to dick around with my phone and wait for a car to come to me.
Living on the south side of Chicago, it was almost impossible to hail a cab, and often times if you called for one it just wouldn’t show up. Or I’d try to hail one downtown, and be refused once I told them my destination. I’ve not had that problem once with Uber, good fucking riddance to taxis.
What's frustrating is that at the end of an event like a concert, everyone goes outside to catch an Uber/Lyft, but we then have to match driver to passenger, instead of having a line of waiting cars, each group just gets in the next available vehicle, shows the driver a QR code, instead of the mess we currently have. Why can't we do that? Taxi regulatory capture.
Taxis were terrible and I celebrate their destruction at the hands of the rideshare companies. Even now it’s still cheaper than a cab was (though those early days of VC cheap rides were great).
Uber has been great, I can get around to almost anywhere with a click of a button and have food delivered from a variety of local restaurants at any time of day.
I'm not sure why you think it was a massive mistake? The taxi industry was awful and unsafe, food delivery for many local places was either not available or poorly implemented.
Also many whole food chains didn't have proper delivery for groceries.
Using AI for everything digital, I end up going and doing chores while waiting for the LLM to do its thing. What an insane world, where knowledge work (dreary and uninspired as it is) is automated while I am consigned to scrubbing dishes and waiting for the machine.
Honestly the big reason why I use my dishwasher is that all the dishes would not fit in my drying rack. I'm not very convinced the process of me rinsing and scraping my pans before they go in the dishwasher is any faster than when I lived in apartments when I handwashed everything. Maybe my dishwasher sucks too, but I mean it doesn't hold a candle to the sort of jobs on my pans that require good old fashion baking soda paste and elbow grease. I'm still slaving with this thing. It's picking the low hanging fruit for me in other words.
I feel similar about my robot vacuum. It actually takes a surprising amount of labor facilitating the damn thing. I have to walk through the house and basically vacuum proof it so it doesn't eat various cords, rugs, lifting chairs up and putting them someplace so it can get under, etc. Not sure its really a labor save to when I was just brooming the entire apartment in 20 mins and not having to move anything around due to reach of the broom and it not having a tendency to eat things. Hand vacuum I can be more carefull too about the fringed edges of rugs and don't have to fold them over themselves.
The trick is dialing the things in properly once, and then reaping the benefits over and over again.
e.g., my home is always vacuum-proofed. It automatically runs through all rooms except for the bedroom at night, finishing just before my alarm clock rings, then vacuuming the bedroom an hour after the last person's alarm clock has rung.
This way I never actually notice it, but the home is always perfectly clean as if by magic.
Similarly, if you use separate dishwasher detergent, antisurfactant and salt, and properly dial in your water hardness, it'll be able to clean everything no matter how dirty the dishes are. Including dried-on tomato sauce. Just put the dishes in, and they'll be clean the next day.
Of course it takes a bit of effort to set this up, but then it just works for a long time.
Similarly, I'm a big fan of handwriting software with care & focus on quality, which takes a bit longer at first, but then allows you to trust the software to continue running for years to come.
I've done a lot of vacuum proofing, even getting the magnetic strips, but there's only so much you can do. Like it doing a little wheel around a chair leg is just not as great of a clean as if I moved that chair out of the way. My rugs have fringe so there's no stopping that. Cables I tend to have out have to go away.
It's definitely easier in a minimalist & tidy home. Our rugs have no fringes, cables are either rolled up or permanently installed & hidden, chairs are stored at night upside-down on the tables they belong to.
Sure, it's not perfect, but it's still really convenient.
LLMs could’ve been developed a lot slower without capsizing the entire supply chain and been seen a lot more positively, China is building a lot of good AI with a minuscule of the compute power and it’s fine
there was never a need to rush other than for capitals self interest to control
if frontier labs wanna pace the frontier, they should be honest and return the compute back to the people
China is building good AI by distilling American models. I think that's a great thing, but Chinese progress is currently dependent on American progress, because China lacks the compute to train the largest models (as stated by DeepSeek's founder[1])
i knew about the compute already (he was very public about i believe huawei under delivering the amount of chips he requested), but nowhere in that transcript does he say he's dependent on US progress for his own progress or he's distilling the US to make progress
i don't really think distilling is even that fundamental anymore (nor was the evidence ever that strong - https://msukhareva.substack.com/p/is-kimi-k3-really-distille...), the game is all RL now, and you don't need american progress for that at all (just compute and data, and the data brokers will sell to anyone no problem)
i do think there are black market proxies from china to the US trying to sell "claude" and "deepseek" access mixed together, and i do think that prompt distribution is probably going to be very nice datasets
Why use apps that you dislike, just ask Claude to click through them for you. And no I'm not being ironic, this is the way things are headed, you just abstract over all the crap by having an agent do it for you.
I'm immensely sad every day seeing all these posts talking about billions and trillions of dollars and where I live (part of the US btw) we don't even have fully reliable electricity and water access. In fact our electricity bills have increased and there is nothing to show for it.
I don't understand why these big labs aren't doing more to improve the electricity situation. Dump a billion dollars worth of tokens into fusion analysis, research, and simulations and try to force a breakthrough. You could buy a lot of good will with unlimited cheap energy that liberates us from fossil fuel dependencies.
Quoting startup like that makes the effort seem like it is an extremely recent venture.
It is 13 years old.
It has been making progress, the degree and date of success is debated and confidently claimed to be in many mutually exclusive states, which at the very least says that there are lots of people confidently making claims that are not true. I'm not going to join that fray to declare which of them is correct.
Are you avoiding all use of fossil fuel while alternatives become available?
It is a bold claim to make but not something that usualy be done. Any abrupt halting of fossil fuel would severely impare the ability to develop and produce alternatives.
We need to transition to sustainable methods. If you want to go cold turkey, go ahead and cease driving an ICE, never travell by air or have anything delivered (and that's just the tip of the iceberg), it would be a counterproductive way to reach sustainability, that is your right, but you can decide to do that to yourself. You don't get to decide how others reach sustainability.
There is a difference between advocating for a better world and acting like a bully because you don't like the current world.
Because they increased the demand so much they are seeking out their own supply. Nuclear has consistently been panned for decades because it's more expensive than other power generation.
Mathematics and programming is verifiable and fully controllable in a digital environment.
Fusion research and cancer research requires expensive experimentation in the real physical world. You can’t just throw a billion dollar worth of tokens to handle the problem.
Look at all the roadblocks for data centre buildouts which the industry certainly has a financial incentive to complete as fast as possible.
> Fusion research and cancer research requires expensive experimentation in the real physical world. You can’t just throw a billion dollar worth of tokens to handle the problem.
When the tokens are hooked up to physical devices (robotics of various sorts), then the tokens pay for themselves by being physical workers, or human wage replacement.
And a robocop will follow whatever is passed for law, damn the masses who might vote otherwise. And subjugation will be enforced by token, LLM rule, and machine. They have no union or ethics to complain to.
The complexity of manufacturing is one reason why TSMC is one of the most important companies in the world right now and why Nvidia is the most valuable by marketcap.
If you can make a GPU more efficient and cheaper than Nvidia does you can make a ton of money. Plenty of labs around the world are working hard on this problem and yet Nvidia is still dominating (and the market believes this isn't changing any time soon)
Same things for robotics. It's unlikely you'll have a vertical stack on all the different things needed to manufacture your robot on a massive scale. What happens if some geopolitical spat makes getting rare earth minerals and battery tech from China impossible?
Compare that to tokenmaxxing in programming and math, where the only thing you need is a gpu cluster and electricity.
Electricity infra is not a technical problem. It’s been solved for decades. It’s a political problem, and both sides of the isle have had plenty to do with getting in the way
This is a Western-centric viewpoint. I'm in Asia visiting a few different countries right now, including the big ones, and much has changed with AI and also in general over the past 10 years with cars, trains etc.
> AI is solving mathematics problems (that nobody - including mathematicians - wanted it to solve)
Strange viewpoint. In mathematics any novel result is useful as it expands the scope of other results. Who are these mathematicians who didn't want AI to solve the problems it did? I see trepidation about the future of the field but that is not equivalent to not wanting them to be solved.
I've watched some videos from Chinese commentary on AI usage in the industry,
Some examples are finance people directly asking for API access now so that they can make their own tools. These tools used to be maintained by teams of developers.
Many business making their own customized scripts to make their tasks faster.
From an economic point of view, it seems like there are some productivity increases, but I don't think these can be concluded for a while.
The rational thinking rarely account for second order effects.
Although business leaders rarely care about second order effects, a worse business is not immediately noticeable, compared to hiring cuts and the money it saves
That is not how math works. There are reasons why the Riemann integral is pervasive and the Labague integral is rarely used unless needed. Just because you can do something does not mean that you should.
> Just because you can do something does not mean that you should.
Again, strange viewpoint in the field of mathematics of all places, unlike, say, economics where doing something one shouldn't has disastrous real world consequences. Until it's discovered and proven, one doesn't know what would be valuable and what would not. I bet your sentence would not be said about humans trying to solve problems, no one would tell them they "shouldn't."
Terence Tao and the apparently newly formed Association For Human Mathematics. The post is online-famous now for the line "Mathematicians did not ask for this work to be done."
When I was a young man reading gauss, and thinking before I turned the page, I see a result here, something meaningful! and then I turn the page, and of course it is there. There is no shame in this.
A long time ago, another student of mathematics described it to me as climbing a mountain. No single step is hard, but you must make many steps to get to the point that no one else has made a step.
No need to trust me, just look at how China was 10 or 15 years ago to now. Beijing air pollution, more high speed rail than the rest of the world combined, electrification of vehicles and economy, etc.
Not for Tesla owners in select countries, where their car now drives them around 99% of the time, or the people of Austin Texas, that can now hail a fully driverless Cybercab for cheaper than an Uber!
>Food quality is worse.
Food quality is higher than ever if you want high-quality food!
Is it reasonable to expect AI improvements to filter into all of those aspects of life that quickly? It's only in the past month or so that we've started to see major breakthroughs in mathematics. It might take several years for that level of research capability to reach areas like engineering and medicine, at least in a way that affects the average person's quality of life. But at this point it looks very likely that it will happen eventually.
Your pessimistic view doesn't hold up against the data. For a lot of the problems you mentioned, we're in a better place today than almost any other time in history. Of course, things can always be improved and we should never grow complacent or accept injustice. But by most objective metrics, there's never been a better time to be alive.
It’s amazing how much the goal posts move in every AI conversation. A couple of years ago the average HN thread about AI would be filled to the brim with skepticism about AI ever being able to accomplish anything other than flappy bird clones.
Now it’s comments like this lamenting that the amazing LLMs we’ve had since early 2026 haven’t entirely revolutionized the whole physical world and even global geopolitics yet.
> But hey...on the flip side...a lot of people are rapidly building software (that nobody is using),
I guarantee a lot of software you use today has substantial AI development in recent releases.
Even the Linux kernel is getting a heavy input from AI and Linus is advocating for it. Unless you’ve frozen your entire software stack in 2024, you are using software with AI generated code.
> and AI is solving mathematics problems (that nobody - including mathematicians - wanted it to solve)
Saying nobody wants those problems solved is flat dishonest. The complaints from some mathematicians are that they and their peers didn’t get to put their names on the papers, not complaints that the problems were solved.
With AI software replication, I wonder if shareholder value even increases.
Services accounts for 80% of American industry, what happens when big software companies loses revenue to piracy/recompilation i.e. Adobe, Unity, AutoDesk, tax/law software, etc..
> But hey...on the flip side...a lot of people are rapidly building software (that nobody is using), and AI is solving mathematics problems (that nobody - including mathematicians - wanted it to solve)
That's not fair. To first approximation, most of the progress in human condition of the last, like, forever is implementing solutions that people didn't see value in at the time of creation. Woz built a toy that could barely run software at all. The human genome project was panned as a boondoggle. Penicillin was a mistake.
The first fire pit probably looked like a dangerous stunt to the tribe faced with it, even.
Does that mean everything will come up roses? No. But LLMs only started talking to us for real like three years ago. Let's not just that fast.
I've been thinking about this for 2 years. At some point it will be easier for them to create self sustaining cities - free houses cars public transportation clean food clean water -
Africa is going to look like wakanda in less than 5 years if WW3 gets canceled
>AI is solving mathematics problems (that nobody - including mathematicians - wanted it to solve)
Do people need permission from mathematicians to solve math problems? Is one of the Millennium problems not worth applying a new system to work on it?
I understand the despair people are feeling at AI automating their work, but we have to be serious about the fact that the frontier labs are just responding to natural incentives. Mathematics, coding, all are just applications of logic, and it makes sense that a system trained on inferring patterns from data would work well in that domain.
You are misunderstanding what mathematics research is. Its objective is the development of human understanding of mathematics. This requires open problems to research. AI are “solving” the open problems without creating any human understanding.
At the very least, the single bit of information of whether a conjecture is true or false advances human understanding. It allows researchers to abandon paths which would not have been fruitful.
I'm pretty sure OpenAI's proofs, while difficult to understand, have a net-zero impact on human understanding of mathematics. It took the first Millenium problem years to verify its accuracy. It's only been days since OpenAI released its problems, surely it would take years to evaluate whether the solutions meaningfully advanced human understanding.
It may provide that one bit of information but it could still be a net negative if it destroys current institutions (such as graduate studies) and no new institutions can be developed that provide a clear signal of who is doing meaningful work. That was a byproduct of solving open problems that are now lost. These are not my ideas, these are things that Terence Tao and others have written about that were recently on the front page of HN, e.g.:
This is basically the dictionary definition of the word "research". But yes, all the work they are doing is focused on this goal. They write papers for humans to read, give lectures for humans to listen to, have discussions with humans - all with the objective of improving human understanding of the subject.
Many of them - all but a couple of the living Fields Medalists - recently signed an open letter about this.
I don't know, this just seems like gatekeeping of the term "research" simply because you don't like the way it's being done.
Many modern medicines do not have understood mechanisms of action, yet we know they work. Are the people who found them not conducting drug research?
Do we even have a different term for what the AI labs are conducting if it's not mathematical research?
Additionally, I would argue that any work that adds to the total corpus of known facts increases human understanding, even if the path to get to each fact isn't understood. It's hard to argue we'd be better off with less total information.
OpenAI to this day is not aware if they solved or not Navier Stokes.
Mathematicians are not gate keepers but I understand the frustration. If someone claims they spent 30 million dollars to solve one of the most difficult problems yet there’s no new knowledge, understanding, nor certainty. Then what’s the point besides marketing?
I think one of the hopes is they'll get smarter than us and able to solve real world problems better - disease, war, poverty etc. But they are not really there yet. Give it a little while.
What is up with this attitude in this thread? AI has taken off programming basically this year, the end of last at best. The first truly capable LLM, ChatGPT, isn't even 4 years old yet.
Just a crazy amount of doom and gloom all around this thread.
In other words, it's taken a highly enjoyable and lucrative vocation from vast swathes of workers - a lot of which make up this sites audience! - and instead concentrated more productive power in the hands of a few megacorps.
It has made it far more accessible to a lot more people. Being “highly enjoyable” and “lucrative” is why people try to gatekeep it.
The fact that the means of this capability is in a few hands does suck. But I’m increasingly optimistic that it’s going to be commoditized through open weight models and performance improvements. There is enormous demand for innovations on these fronts.
I'm a rare individual that has consistently made choices to earn less money in order to have more enjoyment. I would clock fewer hours than they wanted me to during college for my internships, I left a FAANG to go back to startups, I quit a startup to work on volunteer work and try to bootstrap a business.
At multiple jobs I've noted that I felt the pay should have been less. Not so that the employers could take a bigger slice of the pie (in fact, my preference is to be in an anarchist co-op, but I understand that's a wild take and might be terrible in practice). My hope was that we could filter for people that genuinely want to do the work and take on fewer people that were there for the money. At the same time more money is more better. And I like having money. It's complicated.
As I've said many times before, I didn't come to Silicon Valley to get paid. I came here to make computers do cool things alongside other people. I need income to do that, but I try to make sure I'm in a situation where the income isn't something I need to think about all of the time. If you're doing mercenary work that's not possible.
I think being able to accomplish more from places that are a fraction the cost of Silicon Valley will also help achieve the goal of doing things for reasons other than the money.
I think the amount of protectionism we’ll see could be absolutely enormous. Not only do incumbent corporations not want to lose share, incumbent countries won’t want to share either.
Definitely. For now I'm making friends with lots of very capable people - it's not as easy to do that outside of a hub like SF.
> I think the amount of protectionism we’ll see could be absolutely enormous because corporations aren’t capable of altruistic attitudes.
On some fronts they will be powerless to stop the changes. I look forward to drinking the tears of IP owners when breaking through their DRM and recovering their source code is just a matter of sharing a model ID, prompt, and hyper parameters. Reverse engineer NVidia's firmware and drivers. Decompile and port Nintendo's latest console's operating system. We may soon find ourselves in a world where tasks like that are achievable in a few days on consumer hardware.
That's my point, in fact. We're losing the profession just so the annoying guy that would pester you with his stupid app idea can now shit it out himself, lighting the planet on fire in the process.
It's kind of weird to say it's "taken off" recently. Yes what we have today is far better than half a year ago and solving new real use cases, but that was far better with more uses than we had half a year before that, etc, for years now. On their best usecases LLMs were already enormously useful years ago, and it's still very spiky.
As a user of software I see no improvements. What I see is that I need to download software more often, it brings more features I don't need and it has been as buggy as ever (if not more). And I can say that about Operating System I use, social networks, games, etc.
Exactly. Where are the results? If AI is really producing value at 20X the rate software engineers produced it earlier, we should be seeing 20X the number of software projects released now. Or they should be getting made 20X faster. Or they should be 20X better. Or the projects should have 20X fewer bugs. But none of this seems to be happening. The only thing that seems to be happening is we're making the same crappy shit, but with 20X fewer people.
I don't think it's surprising that it hasn't improved life much, AI is new and change takes time, but the fact of the matter is that it also made things worse in some ways, ways that I think were entirely avoidable, if only the incentives weren't so warped.
The fundamental malignancy here is that powerful people have become disconnected from the foundation of society. They have suffered from model collapse. They have been talking to each other in circles for too long, and doing things that make no sense to anyone outside the elite group, and because the systems that concentrate power to the top are so strong now, they continue gaining power even while messing everything up in a nominally free competitive market.
People need to eat, sleep, be healthy, have shelter, feel loved, be in communities...
Coding has been transformed but none of those areas of our lives have been substantive transformed by AI yet. Whoever does, will create a lot of value for the world.
I think that's a bit disingenuous. We've only begun to apply LLMs in the real world and a lot of industries don't move very quickly. Not to mention that we're still very hardware constrained, we don't yet have the equivalent of an ESP32 chip we can just stick into any piece of electronics to provide state of the art LLM responses. But we'll get there, soon.
The funny thing about comments like this is that the moment LLMs are going to replace everyone in the warehouse or the trucking industry, we suddenly have to do something against the LLM. But clearly they can't actually do anything meaningful and have been a waste of energy?
So just let them spend their money and they will eventually fail and bankrupt, right?
This is the paradox. For software engineers, the current crop of “AI” tools is magic. A huge accelerator. And then they assume the rest of the world will get the same boost. However this is not the case at all. Claude code is great because it is trained on extremely well documented git histories and stack overflow. And outcomes are binary: the code works or it doesn’t; the PR is accepted or it isn’t. Not much else in the business world is like this at all. No training data with this kind of sequential development information. And no real black and white outcomes as such.
I would love it if we can just take the win on having created an amazing tool set for coders. Can’t this be enough?
> Imagine what a "country of geniuses in a datacenter" will be able to do? Apparently...nothing.
I suspect the real world change is going to happen unbelievably rapidly and isn't far off. Right we don't quite have the hardware to embody AI in a generally useful way, but that's coming fast.
Within a couple of years arguably any task that requires just knowledge or intelligence is something you'd be better off giving to an AI than a human. And decent humanoid robots are maybe a decade behind that.
In the same way people growing up in the mid 1900s saw rapid changes in their way of life with planes and cars replaces boats and horses, we'll see something similar with robots replacing human labours. Right now we're still in the phase where people don't think humans will ever be able to control flying machines and that cars are reliable and slow. Things will change.
I think people in the LLM space are being far too impatient with the general public and their adoption. And I think they need to start thinking harder about building products that genuinely enrich people's lives. Look at how long home computing took to establish itself with the public, then the internet, and finally social media. None of these trends happened overnight, and some were accelerated by other technological advances (social media-> smartphones).
The adoption of these technologies spurned ton of innovation in these spaces, and I feel like with LLMs its the first time I've seen genuine frustration from the industry directed towards users for not adopting the technology fast enough. This attitude did not exist in the 1980s, it didn't exist in the 90s, and it didn't exist in the 00s/early 10s either.
Do you blame them for getting impatient? They’re burning cash and quickly running to a point of never being able to justify it. The hardware they buy will be replaced with even more expensive hardware before they start to make a profit. It’s not sustainable and the market hasn’t figured that out yet. At least they’re starting to question it.
Whether you like it or not - technology causes compression of life cycles. You have a hit product? Great - I expect you to have crazy high growth rate. That’s the expectation of a world that is so deeply interconnected.
Consumer-ready LLMs have been widely available to the public now nearly 4 years if you start counting at GPT 3.5. There's certainly some compression here but Karpathy does a good job of outlining how silo'd those experiences are: the bulk of the product penetration (based on headcount rather than token usage) is "derpy chatbots" (his words, not mine). Most of the actual token spend is likely done on the STEM side. We're talking about a technology today that currently only has significant penetration with <25M users.
People need to be patient and put in the work to build great products, rather than lament its slower-than-desired (albeit still quite fast) adoption curve.
Yes. The AI industry has to deliver some substantial value to the average citizen soon or there will be political trouble. Food, clothing, shelter, and medical care have not improved yet. Nor does there seem to be much AI can do in the next few years that will help with the basics.
We can have unlimited new entertainment content for passive consumption. That's about it so far. 17,000 new Asian vertical dramas a day now.
I was listening to a Verge podcast in which Nilay Patel remarked that of the AI companies started pushing for universal healthcare, AI would become a lot more popular. But no signs of this.
Please don't be snarky or curmudgeonly in HN comments. You can make your substantive points without either of those things, both of which the site guidelines ask you not to do.
Not sure if that's a fair criticism dang? I sincerely wasn't trying to be snarky or curmudgeonly, but I did want to make the point clearly that none of the benefits of AI have materialised in the eyes of the average person, only the downstream costs. To my eyes, that's a salient, reasonable, and wholly representative position to share.
Did the writing also contain emotion? Yes. As all good persuasive writing should.
As the other moderator, I'll give my own perspective on this. I haven't spoken to dang about it but I completely understand why he thinks this is not the kind of comment we want to see on HN.
The main issue is that it doesn't engage with the substance of the post, and in fact, it exemplifies the very point the post is trying to make: there is a big and growing gap between those who recognize the benefits or potential of AI and those who don't (or, to widen the point, who only see negative effects or negative trends in society).
I'll make it clear that I don't think your complaints or concerns are unfounded or invalid. There's merit and substance to all these issues, and it's fine to discuss them on HN. They are discussed all the time here, and they can and should continue to be discussed in the future.
But an important part of being the kind of curious commenter that we want everyone to aspire to be is that we can look at topics holistically. We need to be able to look at the opportunities and risks together and discuss ways that the benefits can be captured and shared throughout society, and the risks and costs adequately recognized and curtailed. Most people I know of in the tech industry are taking these matters seriously.
It's a vast and complex topic to discuss, and here's not the place for me to get further into it. But the point is that it's a bigger and more complex topic than your comment adequately recognizes, and we need people to always be making the effort to be thoughtful and curious rather than fulminatory or denunciatory in their participation here.
(Also: I can understand you may not easily recognize that your comment breaks the guidelines; it's common for people to not recognize that their words seem more inflammatory when read by others than they intended them to be.)
So, if I say, there is a big and growing gap between those who recognize the benefits or potential of my poop machine and those who don't, and you say, wtf is this shit everywhere, I can just say, you doesn't engage with the substance of my post, and in fact exemplifies the very point my post is trying to make
Your crass comment might have a point if (a) there was no real or potential upside to AI, and (b) the original comment focused on the negative consequences of AI. But it doesn't; it's just an enumeration of grievances about negative trends in the world, most of which predate the emergence of AI, and/or are orthogonal to it.
The people who are optimistic about AI believe it can help to address or reverse these trends. It's fine to debate the merits of that belief, but the original comment wasn't trying to do anything like that.
It honestly escapes me why you would judge the juxtaposition between everyday life of everyday people and a very specialised minority as snarky or curmudgeonly. Was there too much emotional impact?
There are so many comments on HN that would justify your attention with regard to this specific rule. This is not one of them.
"So this is the weirdness of the moment. The general public has mostly not interacted with these systems. When they have, it looks like a derpy chatbot. The majority of professionals still see only a modest uplift. And a small sliver of professionals are experiencing the vertigo of the curve going vertical. And it is all happening at the same time."
-------------
It comes down to the difference between programmers and non-programmers. If you're not coding, AI literally is a derpy chatbot that's made it harder to get search results, raised your power bill, destroyed property values in a neighborhood near you, is propping up the economy but also creating a huge bubble that could pop and destroy a lot of retirement savings, etc.. At best, it's a nuisance, but it could be a lot worse than that. The AI evangelists need to understand this, because the average person is starting to pay more attention to the downsides while still not seeing much upside.
This gap between the people seeing the curve go vertical and those who are simply growing more and more annoyed is a real problem... mainly for the people seeing the curve go vertical. If the people who are merely annoyed right now don't see some of the promised benefits soon, or if any of the negatives get worse (e.g. massive increases to greenhouse emissions), they might get angry. You probably wouldn't like having the overwhelming majority of the public angry at you. AI evangelists need to start doing better things for the majority of people or they need to address the negative consequences they're inflicting much more aggressively.
so what was karpathy expecting?? everyone will give up on their life and start feeding the monster in the screen? that's literally what the tiny sliver is doing; they are food for the AI model.
That's fine but these companies having billions in funding by now should have rewritten chalk and other terminal libraries in Rust while bundling the whole thing in Rust as a single static binary. I'm talking about horrible Claude Code and friends.
And then using emgui they could expose full fidelity IDE+Agent workflow interface (like DeepSeek Harness and similar) that's also exposed as MCP to be operated by another agent and JSON RPC for over the network but no. Skill issue?
PS: Even full fidelity Photoshop is possible with such high fidelity UI that has CLI, JSON API and MCP server all in the same single binary < 90 MB. For reference, see PhotoCraft or VectorCraft or WordCraft or PdfCraft.
I think "moment" significantly undersells this. Frankly, I wonder if this isn't setting vast swaths of the population on fundamentally different paths.
Currently the difference in life outcomes between people who know how to use computers well and those who don't is small enough that in everyday life we just ignore it. But it's greater than, say, the difference between people who took advantage of refrigeration and those who stuck with ice boxes, or those who learned how to drive a car and those who didn't.
And the difference with AI is going to be even greater. And it's possible that right now people are choosing a life course that will benefit or disadvantage them long term.
How does one who doesn’t have an employer funding their massive token budgets become “good at AI” the way that Karpathy describes here?
I think the point is that this gap exists partially because of the access that a select few have today, which enables them to do things that the have-nots can’t fathom.
Sure, but is that different than in the past? Those who could afford computers got good with computers. Those who could afford cars learned to drive. etc., etc.
> Around 1-2B (~20%) are casual and infrequent users of free-tier ChatGPT-like products. This group sees derpy chatbots and treats them a bit like a better Google search, a writing aid, or etc. Maybe an agent tries to book you a flight. I have non-tech friends who (reasonably, imo) say they have not much use for it at all.
“Cy Draven: I tried everything. Believe me, I hammered the race thing, I hammered the gender thing, the trans thing, the border thing, the homeless thing, the war thing, the election thing, the abortion thing, the climate thing, the thing about induction stoves, Israel, library books, vaccines, pronouns, AK-47s, socialism, BLM, CRT, the CDC, DEI, 5G, everything. All that I did, nobody, just nothing.
People are just numb these days, I don't know why.”
He's intentionally forgoing all nuance in order to make this sound dramatic
> Somewhere around 20M people (0.2%) see first-hand that large, complex projects that used to take them weeks/months can now be completed by agents with a prompt.
No, they can't, at least not any semblance of quality. The cases we're seeing where this does kinda work is in ports and translation where all the rules are already documented in the best specification language possible with a way for the LLM to verify itself: code. We saw this close to a year ago now with Cloudflare and NextJS
> The impact scales with ambition, problem size, and horizon. A question with a paragraph answer barely stresses the system. You need a reservoir of big, difficult problems that you really care about
These are operating on different capabilities - AI's ability to answer informational queries as a chatbot frankly sucks and can't be trusted without verifying it. I run up against this every day. A problem with a verifiable answer on the other hand it's very good at solving. He knows this (his next paragraph) but he's putting them on the same scale of "stressing the system" to attempt to add proof to his introductory claim
And then there's the completely unverifiable scare that there are internal frontier models way beyond anything we've seen "swarms of thousands of agents collaborating over weeks on software mega projects: minting zero days, running cyber attacks and defenses at machine speeds, discovering new science, advancing the frontier of mathematics"
I dunno. I haven't been able to set up OpenAIs remote codex connection, their shit is buggy as hell and the web UI keeps crashing and making messages disappear. Is this what their internal superhuman "Things that would have taken top professionals in the industry years of work" looks like? Granted Claude has been really smooth, but still..
I don't think AI is all hype at all. Opus 5.5 is really impressive. This twitter (x? whatever) post is infuriatingly hypey and just written for virality, though
You’re trying to tell me it’s all hype when I have (depending on the day) a top 50 Productivity app in the iOS App Store for which I’ve written and read exactly one line of code (the version number).
Anyhow, if your users are currently satisfied, they might change their mind when they find that their personal data was uploaded to an open S3 bucket, when the new version wipes all their projects or when they find that not really everything was being saved (it's random examples, I don't know if they're applicable to your app).
Yes, this is impressive. When the iPhone SDK was first released in 2008, I spent months building apps to get them into the App Store. That included learning Objective C, XCode, various frameworks, and tons of manual testing. It was a lot of work!
People have been dumping garbage on the appstore and collecting checks for a long time. If this is all the AI revolution has to offer then it's not looking good.
It's insane how confidently you can make a claim like this in a forum where thousands (tens of thousands?) of people are using these tools daily to build useful software for themselves and others.
How can you possibly believe it's all hype? Is everyone just lying? Are we all paying $200 per month because... why?
Spend long enough (or not even that long) in corporate software development and you'd believe such unsubstantiated hype is possible. The only difference here is even stronger incentives to pretend the emperor has clothes: insane overinvestment, threats of job loss for non-believers, fear of missing out on massive alleged productivity gains.
> swarms of thousands of agents collaborating over weeks on software mega projects
If you have defined testability, running a single agent, in a loop, over a goal will achieve the same results.
> minting zero days, running cyber attacks ... Granted Claude has been really smooth, but still
The old adage about monkeys writing Shakespeare applies here. Agents are just better monkeys and if the goal has a well defined outcome it's fairly easy to "Brute force" the way to an answer.
I’ll say this until I am blue one the face. Nerds (software dev, maths etc) see how good LLMs are at things they care about and assume they will be broadly applicable in future.
There is every reason to think this. Its about available quality data. The data that was easiest to fetch was already there, then we got some more data by asking experts to create datasets for post training. Once this is over, we will get to the outer world that didn't get to hoard it for bots to take it. This will certainly change. Put on a smart glasses and record what you do to fix a pipe. In 3-5 years, rinse and repeat.
Maybe, I'm at the point I don't know what to think anymore. But somehow, this feels still non-human. Humans don't need millions of hours of training and millions of samples to learn how do to something. We have proof that systems can deal with low-shot training. So why can't these? My point is when we have human level learning ability, then we're moving, if we expect expansion of capabilities to come from data alone, I'm prepared for disappointment.
It will be comparable once the AI is self sustainable and self reproducible (no big data centers behind it). Then it will have to deal with limited resources to run itself like humans do. In that scenario it will no longer have entire world of knowledge at its hands - options will be to cut anything not usable imediately or to communicate with big AI in the cloud.
The first is that LLMs used to be bad at each of these nerd things, then toppled them like dominos. There's a pattern over time.
Another reason to think this is g. Across every known measurement, human (and animal) intelligence is convergent. Being better at one thing correlates with being better at another thing. Although the reason is not perfectly clear, the pattern is well established, and seems to apply to LLMs too - GPT-6 is smarter than GPT-3 at everything, not just math. There's no reason to expect different for GPT-9.
I don't agree that they get uniformly better at everything, especially subjective skills.
For example, they have become more and more unintelligible when you ask for explanations or descriptive text. They assume you see the same context as them and shortcut explanations.
I've had to craft a skill to get them to produce remotely understandable explanations of even mildly complex/non-mainstream subjects.
This may be Curse of Knowledge https://en.wikipedia.org/wiki/Curse_of_knowledge on their part, and it also impacts human experts but still. Becoming better at one thing does not mean you become better at everything else, although I do agree with you that the better they become the more things there are they become good at, but their ability is still quite jagged and maybe increasingly so.
The delineation between knowledge and intelligence is useful here.
Knowledge can occasionally get in the way of some tasks, such as a master illustrator trying to draw like a child. Or as you've mentioned, an expert trying to explain to a beginner.
Even so, I disagree with you about model progress on communication - maybe there's a little jaggedness between minor model versions, but Opus 5.5 is a much much better communicator than, say, Claude 3 Opus.
I don't agree that Opus 5.5 is necessarily a better communicator than older models. In fact, I ran my own tests, and the difference wasn't very clear, and in terms of personal preference, the writing style of newer Anthropic models has really let me down. It doesn't reason like a person that imagines what the person they're explaining to might know or not know, or what they might think, as one usually does when trying to explain something to someone. They just fact-dump, like the problem solvers they are, since they are trained to solve user's problems correctly, efficiently and comprehensively, but they probably don't have a good internal representation of the user they're talking to, they are not really trained to imagine what we may think or feel, and thus, in my opinion, make pretty bad communicators (cf. the ocean of vibe coded READMEs on github that are horrendous to read).
For example, when asked to write a report on Docker and explain how it works, one of the first paragraphs was :
"1. *Docker Engine*: The core component, which is a client-server application that uses a server component (the daemon) to manage containers and a client interface to interact with the daemon."
This is perfectly correct and makes sense for someone that is familiar with software engineering and already knows what containers are. Not so much for someone wanting a introduction to it. Do you imagine writing something like this at the very beginning of your explanation of Docker ? It even said: "I wrote it in the neutral register from your publication preferences, with no em dashes, and defined terms on first use.", when it didn't even define what a container is. I chose Docker at random, because I could judge if its explanation made sense, since I know how it works.
> Across every known measurement, human (and animal) intelligence is convergent. Being better at one thing correlates with being better at another thing.
LOL. Human (and animal!) intelligence is notoriously jagged. We're all basically idiots except for very narrow areas where we focus.
As anyone who's needed to provide IT support to a doctor, lawyer, lauded academic or host of otherwise "smart professionals", can assert, you can be very very smart in one area and very stupid in another.
This is the classic error in mixing up knowledge / experience and intelligence.
Intelligence is the ability to quickly process, understand, learn, and use information. It does not imply having already been exposed to any given information.
A smart doctor may have terrible computer skills. But this is from lack of experience and interest, not lack of ability to learn how to use computers.
But take that smart doctor, and a relatively dumb person who is equally inexperienced with computers. Now give them a month and incentive to learn computers. The doctor will learn much more; that's intelligence.
There is IQ, EQ, and even purely intellectual aspects have a million facets. So you're wrong even on that front.
Someone can be really good at physics and suck completely at biology, due to factors we don't even understand (motivation probably being the main one we do understand).
We even see this in practice with super bright tech people, especially programmers, failing at launching businesses or at adapting in other lines of work such as management, finance, etc.
> There is IQ, EQ, and even purely intellectual aspects have a million facets. So you're wrong even on that front.
This is simply not true. EQ is not a real construct supported by research. There are not 'intellectual aspects' that diverge from g.
Do yourself a favor and familiarize yourself with the psychological literature. I don't mean this in a condescending way! The gap between popular reporting and actual psych research is enormous and not at all your fault. I only know this myself because I did a PhD in research psychology.
Cats famously have more brainpower than dogs. But dogs understand us humans far better than any cat.
Someone who actually works with real animals (farmers? Ranchers??) would come up with better examples. But I'm just city boy over here who only meets cats and dogs on a regular basis.
That's just cocktail party level of thinking. LLMs are getting better at math, code and logic (and marginally better at science and general knowledge) because these are domains that can be objectively verified and thus there is a potentially infinite supply of 'facts' to generate and train on. These are very powerful but ultimately very abstract domains. For everything else the messy real world and its physical bottlenecks gets in the way and there's little reason to expect progress to accelerate. It still takes months to get mice to reproduce and run experiments on, no matter how knowledgeable about biology the models have become.
If anything, in some domains frontier models have become worse - claudisms and chatgpt idioms are making them notoriously bad at prose without a considerable amount of prompting and tweaking.
Is good writing verifiable? I don't think it is, but LLM's have been hillclimbing writing quality. That being said this is with the help of RLHF.
However there are many other domains which have verifiable rewards in the process of learning them, despite their overall impact not being verifiable. For example, the life sciences, an LLM could be given access to data about an organism, and then make predictions about how a drug or gene therapy will affect that organism. In economics, LLM's could create models of behavior, evaluate predictions over time and see how well those predictions match reality.
>claudisms and chatgpt idioms are making them notoriously bad at prose without a considerable amount of prompting and tweaking.
I think this is because a lot of people genuinely like the claudisms, even though a small minority of technical people don't.
It's not at all clear that LLMs have gotten significantly (much less hillclimbing) better at writing since GPT4-ish. They did get better at oulipo-like challenges (writing under silly, verifiable constraints like Fable's viral 'facetiously' poem) but that's far from being all there is to good writing.
Anecdotally, creative writing communities do report drastic differences in quality between models (it is my understanding that Gemma 4 is especially praised) so there is something to it being verifiable but it's a far cry from being something that can be distilled into objective math/code-like evals. One could even say it comes down to vibes.
>For example, the life sciences, an LLM could be given access to data about an organism, and then make predictions about how a drug or gene therapy will affect that organism.
Yes, biotech companies are doing this right now but ultimately they are just predictions and you still need cold hard biological data to ground them and iterate on. That's slow and expensive, especially for the juicy fields (human biology, food and crops, clinical trials) and you can't just plug billions of VC capital into the pipeline and hope for RSI. Physical (gotta procure all those labs and their equipment), human (gotta hire and pay specialists to run experiments), biological (gotta wait for organisms to reproduce, drugs to take effect, crops to grow), regulatory (gotta convince agencies that your fancy new drugs are legit) bottlenecks get in the way.
And biology is one of the easier fields where a path to RSI (if not accelerating) is conceivable. Good luck iterating on macroeconomics data where there are no replicates and 'experiments' take literal years if not decades.
>I think this is because a lot of people genuinely like the claudisms, even though a small minority of technical people don't.
My uncharitable take is that the abtruse jargon makes people feel smart for understanding it and gives a sense of belonging (as a closed circle of initiates who understand LLM cant), just like rationalists love to repackage old or unsavoury ideas under new nerdy smart sounding names.
>you still need cold hard biological data to ground them and iterate on... you can't just plug billions of VC capital into the pipeline and hope for RSI
Capital can get you a lot: access to wetlabs, historical medical data, ability to run simulations, creation of autonomous bio labs which run thousands of experiments in parallel. An LLM, in training on this flywheel of data can take in more information than a human biologist would be able to read in multiple lifetimes.
>Good luck iterating on macroeconomics data where there are no replicates and 'experiments' take literal years if not decades.
The better an LLM gets at simulating human behavior, the more accurately you can simulate economic policies and their effects via running simulations of agents in virtual societies. There are many routes that models can take to create a broader sense of the world without needing the world to provide data.
>By evaluations against human writers, LLM's have improved at writing.
I would really take those benchmarks with a pinch of salt. We already know benchmaxxing is a thing for evals that are objectively verifiable like code, and coding models that seem to perform well on benchmarks don't pass the vibes test, imagine all the caveats that come with such a subjective test like creative writing, not to mention control issues (are they the same human writers across benchmarks? Are they the same testers?)
>Capital can get you a lot: access to wetlabs, historical medical data, ability to run simulations, creation of autonomous bio labs which run thousands of experiments in parallel. An LLM, in training on this flywheel of data can take in more information than a human biologist would be able to read in multiple lifetimes.
I don't disageee and am in fact excited to see the advances, what I am arguing is that it is going to be slow because the hypothetic feedback loop that would improve the model is bottlenecked. There are actual physical and biological limits in our universe. Unless VCs find a way to make mice embryos and crops grow 10x faster (among the many, many other things that would be required for acceleration) the trickle of data is not going to be anything like the accelerating returns imagined for math and coding. And that's assuming biological RSI is possible which has yet to be proven - current foundation models are interesting but they are still specialized and there is no clear, obviously uniform way to encode the many, many modalities that exist in biological data.
>The better an LLM gets at simulating human behavior, the more accurately you can simulate economic policies and their effects via running simulations of agents in virtual societies. There are many routes that models can take to create a broader sense of the world without needing the world to provide data.
You wouldn't need to 'just' simulate human behavior (good luck with that btw) but the entire political and economic conditions in which they live. Even assuming this is possible the amount of compute you'd need to run would be astronomical. Given how all RAM capacity for '27 and '28 is near sold out, hyperscalers are running out of cash flow to finance buildout and Sam Altman complains when users say 'please' to LLMs I don't think this is in the near horizon.
The quality of natural language output from today’s LLMs seems no better and often worse than LLMs from 18 months ago, at least in my experience. It’s much harder to build a good RL loop for something subjective like good writing, compared to something with an objective and verifiable right or wrong answer, such as software or mathematics.
IQ is a useless thing, but you can absolutely tell intelligence from writing. You don't need to write like [author of your choice], you need smart ideas and the desire to make them understood.
AI writing sucks because there's no intent or choice behind it. It's human thought, blended by linear algebra into pink slime and extruded into sentences.
There was a time when people believed in Moore's law too. Now even Jensen Huang says Moore's law is dead. There's also the adage from investing: past performance does not guarantee future returns. A little CYA sure, but also acknowledgement of latent factors and the entropy of the universe and its affect on any predictive modelling one might do.
I’m not sure I agree that g explains all model capabilities. I think “claudespeak” is a consequence of the shift from training on RLHF to training on verifiable rewards. Thus the models seem to become worse at storytelling, fiction, readability, not better.
Even if we do accept that models have some “g” and that it correlates with all domains, there’s no reason to think that the current trajectory of LLM intelligence will grow more quickly or even as quickly. In every other technology we have an early period of slow growth during experimentation, and then a rapid period of growth, and then stagnation.
For all we know, 2026 could be the year of stagnation. It’s possible that we will reach finishing returns. If thats true, then it won’t matter how much that task correlates with overall g, because g won’t be increasing that much.
> Nerds (software dev, maths etc) see how good LLMs are at things they care about and assume they will be broadly applicable in future.
It's worse than that. I am one of those nerds (a programmer), and I see that LLMs are shit at the things I care about. I have zero reason to believe that they will be good at other things either.
> Nerds (software dev, maths etc) see how good LLMs are at things they care about and assume they will be broadly applicable in future.
I don't think this explanation is sufficient:
For example, many nerds care about 3D printing. On the other hand, my experiments (and the experiments of many nerds who I know) to let LLMs create files for 3D printing lead to horrendously lacking results.
Or many nerds care about linguistics or puns. Whenever a new LLM comes out to which I have easy access, I do the test, and let it explain some specific German jokes to me that are based on convoluted puns in the German grammar. Until now, no LLM that I had access to could give satisfying explanations of the puns on which these jokes are based.
Or even for coding (many nerds do care about elegant, sophisticated code): the LLMs that I could test were already overchallenged with the following task: I had written some code that is a very "artisanal", "clever" improvement of an algorithm over the version that one would find in a textbook. The task for the LLM was simply to write some code comments/documentation about the mathematical ideas upon which my improvement over the textbook version of the algorithm is based. It wasn't capable to do this. On the other hand, for a junior programmer, I would in such a situation expect that he goes through every single line and thinks through the algorithmic ideas so that he can learn from them.
---
Thus: even for "nerdy" topics, it is very easy to find tasks where LLMs still suck. So, my hypothesis about the nerds that you mention in your post is that these nerds are rather people who want to believe (with religious fervor) that the current LLMs are exceptionally good instead of just looking into a slightly different direction than where the tech billionaires want the users of LLMs to look at.
I honestly don't get your point: perhaps your argument is based on some US-specific cultural reference that I am not aware of since I don't live in the USA.
If it is, I don't get it either. That's not at all what I imagine with hippies and I suspect that was supposed to be some other word.
Nerds are also not necessarily all about technology. "Math nerd" was a really common use of the term for a long time, for example. "Nerd" is someone with deep knowledge about a particular subject.
> For example, many nerds care about 3D printing. On the other hand, my experiments (and the experiments of many nerds who I know) to let LLMs create files for 3D printing lead to horrendously lacking results.
Try again; astra has been driving fusion for me. I took a screenshot of a self watering cat grass bin on makerworld — not even the stl, a screen cap of one of the photos attached to the design — and it made me a parameterized version in one shot.
Its initial design was less than ideal for printability and it reworked the design correctly when asked to consider what it originally produced through the lens of printability.
Autodesk bundled an mcp server with fusion in the last few weeks and i expect things to get even better going forward.
> Try again; astra has been driving fusion for me. I took a screenshot of a self watering cat grass bin on makerworld — not even the stl, a screen cap of one of the photos attached to the design — and it made me a parameterized version in one shot.
I have no access to Astra, but with GPT 6.1 Sol, the results with respect to attempting to create STL, OBJ or even OpenSCAD files by using image references and a prompt were really bad.
> The most passionate, intelligent people are usually a decent indicator of where culture is going to go.
Many very smart people who I know are quite skeptical of LLMs and the hype around them. In my observation/echo chamber, the people who are very into LLMs are rather slick, career-minded people who love to present themselves as trendsetters for the "next fancy thing".
Steve jobs wasn’t a nerd - he said he’s a hippie on the lost interview. He was also the first to bring to market colour screens, typography etc. He also defined the Internet as the defining ‘social moment’ way before anyone else.
The most passionate, intelligent people 1. aren’t spending a lot of time on smartphones, video games and crypto and 2. aren’t as much smarter or better at predicting the future as the barely above average people who mistakenly count themselves in their ranks like to think.
Most of those technologies were controversial in their inception among the nerds of the time. Remember Eternal September? Linux/Windows people making fun of Apple hipster fans? Mainframe engineers deriding PCs as unserious and video games as childish?
>The most passionate, intelligent people are usually a decent indicator of where culture is going to go.
That's a naive view of cultural determinism. The software landscape would have been very different if a handful of lawsuits had gone one way or another. Hell, even Unix and Linux were hobby projects that accidentally got big.
Also crypto is a grift lol, one of those is definitely not like the others
Being good (or superhuman) at software dev and maths shouldn't be treated as though it's equivalent to other domains.
Unlocking mastery of those domains would lead to incredible gains in all others. Whereas a fundamental advancement in many other domains doesn't spread as readily.
I’m not sure. LLMs are superhuman in terms of workload (they are maybe also superhuman in ability too), but where does that lead us.. cheaper software?
That’s good for corporations who need custom software. Every large business has a massive backlog. In this area reducing the cost will massively increase demand.
But where’s the broader value?
Consumers and business already have too much choice in off the shelf software.
Ultimately most people just need word, excel and maybe 4 computer games a year.
I'm a bit confused by the percentages. 6 billion people aka 75% of the population? What are we talking about, as if we expect 8 billion people to use or care about LLMs? A quarter of the world is under 15, what percentage is about 75? What percentage, for various reasons, has no reason to care about LLMs? What percentage is more concerned about the amount of calories in their next meal before everything else?
It just seems weird for Karpathy to throw around these numbers calculating percentage against the entire global population.
Pretty much all of this is basic stuff an interested teen would know and chatGPT fails miserably. I fear the "true world model" gap will be too much for LLMs. I guess time will tell.
I actually wrote about this recently. There's a quote somewhere along the lines of:
> What is possible is unevenly distributed. As knowledge and technology compound, the distance between those closest to the source and everyone else also exponentially increases.
I actually argue the continuum that forms singularity has been progressing before we realized, but we're at the point where in 5-10 years people not using AI will be equivalent of the amish.
Unfortunately, as the rate of change increases, the distribution of knowledge wont be as fast. So those closest to the source of innovation will have exponentially more knowledge, capability and in-effect power.
Karpathy is a great guy, I love the way he teaches, but should we really trust his opinions about software development (how AI can impact it)?
I know he can code and has a degree in computer science, but if I'm not mistaken he's never worked professionally as a software developer.
Developers here know how much you learn and are forced to learn on the job: it depends a bit on the company you're at, but you can quickly learn the value of good code, how easy it is to introduce bugs and vulnerabilities, and how helpful a second eye is.
I think he's in good faith in his assessments, but, at least for what concerns development, maybe we should give them less weight.
> For intelligence agencies, and for corporations like Microsoft, Google, Apple, Facebook, Amazon – giving you convenience and security is likely not their primary goal. I think they have an entirely ulterior motive. They can use passkeys to prevent you from running unapproved software.
> Remote attestation can allow them to verify that your device is not capable of, say, sending and receiving encrypted messages, or running a competitor’s app
How does this make any sense? Google can already prevent any app from running on an Android if they really wanted to, they don't need pass keys for this.
Google doesn't need attestation to block Signal. What is this non-sense? Or will a website block phones capable of running Signal? That's all phones. And anyway, has nothing to do with pass keys.
Altman didn't say that. In March 2021, he did make some predictions, but they were much vaguer and farther out:
> In the next decade, they will do assembly-line work and maybe even become companions. And in the decades after that, they will do almost everything, including making new scientific discoveries that will expand our concept of “everything.”
To add onto this, I feel like this has been happening a lot, where critics project backwards imagined claims or go nutpicking for random comments online to make something sound absurd and like everything is behind schedule and nothing is materializing, and then you can see other people (not necessarily the same people, the point isn't "look at these hypocrites") saying that it's all moving too fast. Both of these can't both be true in their strongest forms. I think what they have in common is that because they're relative measures to an unstated baseline, they're free criticisms you can throw at any development whatsoever, regardless of what is or is not turning up.
So.. the reason LLMs aren't living up to the lofty promises is that only a tiny fraction of the economy has access to the really useful ones? Why?
I'm sorry, that makes no fucking sense at all. If there are models that are actually super useful, the LLM companies would sell access to them like all the rest of the models they sell, maybe at a higher price point.
This is delusional conspiracy theory nonsense. Next he'll be telling us about the shadow government's secret UFO cover up.
I think it's a weird take because it implies that the 6 billion people he's talking about actually have some interest in knowing about the capabilities of LLMs to solve frontier math? This is simply not something they care about or can evaluate. There are maybe several thousand people in the world who have some (abstract and barely-monetizable) use for this information, plus probably another 100,000 who don't understand any of the math, but like to cheer on.
We have already reached "peak LLM" in terms of what normal people realistically need it to know or reason about. In fact, I'd say we reached that point about 1.5-2 years ago. There are two other barriers that remain unsolved:
1. They're less dependable than humans and can't be meaningfully punished or forced to make up for mistakes, so you can't really replace humans with them, not without having a human babysit.
2. Most people don't really have a special need for an LLM in their life. They may like that it answers questions or helps you polish a resume or, I guess the labs' favorite, helps you make restaurant reservations. But this sure isn't worth $200/mo for most people. Probably not even worth $5/mo.
It'd be kinda funny if we create superhuman AGI and then no one has any real use for it, perhaps except for military murder-bots. There's always market for that.
I know I’m bordering on repetitive and strongly negative, but the combination of the two barriers you mentioned has all but halted all of my interaction with agents.
Even ones that cannot be easily dismissed (like the agent at the top of Google search) are often subtly wrong in a way that requires significantly more effort on my end to parse the loquacious output to determine where the inconsistencies are.
Look, can they be useful to generate the html or jscript for humanity’s 9 billionth iteration of a web form? Sure. But man, I wish sanity had reigned and people had applied ML models in general to more substantive and beneficial projects.
(Before anyone chimes in, I’m familiar with implementations of ML applied to esoteric domains; but by their very nature these don’t get all the news cycles, or hiring, or any of the other absolute insanity that the domain seems to contain)
This is by the weirdest possible opinion on llms as a whole. Not the AI booster club, not the AI doomer, not the undecided, but somehow "I can't think of anything to use it for." It is like watching the beginning of the internet in 90s and saying there is nothing to browse. The whole concept is just weird to me. At least the first 3, I can conceptually understand. It is admittedly hard for me to understand someone saying 'i tried this and best it can do is summarize emails'; I am not saying it is not true ( my boss said that exact thing ), but it is hard for me to reconcile with my view of the world.
Also seems weird to me. Claude cowork could just do my monthly tax receipt collection and compare with my account better than the bookkeeper I pay for it. That is a very cross sectional problem
Most people rarely have these kind of administrative tasks in the first place, and when they do they often run into GP's barrier 1. I can't recommend that my extended family have Claude do their taxes, even though I work with it on my own taxes, because they might ask for something that produces bad results and can't evaluate the output to double check. ("Hey Claude, I want a higher refund, can you tell me what to put on the forms so I get at least $3000?")
To be fair, this was a commonly expressed opinion when the internet was in its infancy.
Paul Krugman (1998)
• "The growth of the Internet will slow drastically, as the flaw in 'Metcalfe's law'—which states that the number of potential connections in a network is proportional to the square of the number of participants—becomes apparent: most people have nothing to say to each other! By 2005 or so, it will become clear that the Internet's impact on the economy has been no greater than the fax machine's."
Clifford Stoll (1995)
• "We'll soon buy books and newspapers straight over the Internet. Uh, sure."
• "The truth is no online database will replace your daily newspaper, no CD-ROM can take the place of a competent teacher and no computer network will change the way government works."
• "Lacking editors, reviewers or critics, the Internet has become a wasteland of unfiltered data."
There's a massive tendency amongst even intelligent people to completely dismiss the potential of new technology based on that technology's current observed capability.
The fundamental question I have that honestly I haven't been able to find any answers for: With all the talk of LLM-based AI capabilities "going vertical", what evidence is there (for or against) that LLM-based approaches won't eventually "hit a wall", that is find some aspect of intelligence where humans will still have primacy, and no amount of scaling will change that.
E.g. some prominent folks definitely think LLM-based approaches will hit a wall, perhaps LeCun most notably. I also read a guest post on Terry Tao's site (which I really liked) that argues that, for all the very impressive recent AI results, they still operate within the "convex hull" of their training data: https://terrytao.wordpress.com/2026/09/13/happy-those-able-t...
I'm just curious if there is any actual data or evidence that takeoff (i.e. RSI, "the singularity", whatever you want to call it) is inevitable with current approaches.
But when you squint your eyes, LLMs are just a least-effort manifestation of the wider neural net direction we’ve been chipping away at for decades. I don’t give a fuck whether LLMs are here to stay; they’re proof that scaling neural nets gets you pretty far.
The real question is: will neural nets hit a wall? As a human, I certainly hope so. But I’d bet that they won’t. I’d bet that we’re igniting the ~~atmosphere~~ universe in slow motion.
I don't think there is evidence in either direction (takeoff impossible vs inevitable).
But the "convex hull" idea always came across as wishful thinking and hand-wavy pseudo-science to me.
Here's a rebuttal [0] of it from Tim Gowers (uses another math metaphor, but still clear and the whole thread is worth a read):
>...the popularity of the idea that AI "merely explores the convex hull" mystifies me. Why wouldn't it be more like the subgroup generated by existing mathematical knowledge? (That's also a metaphor -- I'm not saying there's a group structure.)
>The convex hull metaphor suggests that AI isn't having radically new ideas and will eventually run out of things to do, whereas the group metaphor suggests the possibility of building up in small steps to something well beyond what you started with, with no obvious limit.</i>
I'm confused.. are you suggesting that Claude Code / Codex don't work? Because if you're still saying that in October 2026, it's a you problem. You're doing something wrong.
- Prime Intellect (and all their agent experiments)
- Geoff Huntley (see Jiti, for instance)
(many more)
There's a ton of interesting software being developed with these models by people outside the in group, but I find most of the software from these big guys to be ... bland. Buggy copies of copies.
I don't see the Cloudflare stuff as groundbreaking, let alone people building companies around it. "Workers" are the wrong primitive, IMO. What if my code isn't in Javascript? They sandboxed the wrong part of the machine.
I have a version of "Cloudflare OS" running in production, using Temporal as the orchestration and MicroVMs for the sandboxing.
Sorry, I get your point but personally I don't get the hype around them.
I don't use either myself but I'm in a position where I get to see what it produces for other people. Here's a recent (2-3 months old) example: A zsh script that immediately invoked a Python interpreter that immediately invoked a shell command and iterated over each line of output looking for one of a few strings to match... So, `<cmd> | grep "(stringA|stringB|stringC)"`, but a whole lot dumber.
Thanks, but no thanks, I don't need that kind of code in anything I'm working with.
"running cyber attacks and defenses at machine speeds"; worth noting that LLMs (even the vaunted frontier models) are exponentially slower at cyberattacks or defenses than your average WannaCry or Splunk automation from decades ago. Its this sort of delusion world the AI bros live in that is part of the reason there is a disconnect between what they think should be happening and what actually is.
I think that's a bit reductive, there are decades of sci-fi novels about how transformative artificial intelligence would be, so I feel like I don't need to explain where the trajectory could go. The applications and "collision with the real world" aren't there yet, but you can't pretend that a machine that's smarter than any living human is on the same level as Amazon drone deliveries.
I don't disagree with where AI could go, but I don't think we've actually proven that it's there. My point is that the folks claiming we are there already if only the rest of the world would just recognise it, are repeating the same thing from every previous tech cycle. In many cases it's the same folks.
I agree with you, but to play devil’s advocate: what if it is on the same level as Amazon drone deliveries? Is intelligence really a bottleneck in society? People in MENSA with high IQs don’t often accomplish anything in proportion to their relative intelligence. The most intelligent people are rarely even in charge of making decisions. There are a lot of intelligent people with intelligent ideas that nobody listens to. It seems like the real bottleneck in society is building trust, consensus and constructive incentives between groups of people to do the things we already know are smart, not a lack of intelligence.
Good points. Maybe a more general version, removing constraints on one resource is usually not hugely impactful because you simply hit the next constraint.
Concrete example, "coding is free" (TBC, but let's assume), and now we're code review constrained. Even if code review becomes free we probably just become product discovery constrained, or something else. Making all software free will just cause us to hit on hardware constraints, and so on. It's still progress, and more progress will bring more benefits, but there's a limit to how much better the world as a whole can get from any one change like this.
MENSA selects for puzzle solving on the basis that those are correlated with "being smart".
Reinforcement learning selects for models that appear to complete tasks, on the basis that appearing to complete a task is correlated with completing a task.
Both have problems. The latter is probably a better correlation, but by no means flawless. If we just take at face value that RL is training models to complete tasks, we're not engaging deeply enough with the process enough to see where it falls short.
MENSA also selects for people who want someone to certify their intelligence. High iq people who have done a lot have much less desire for a MENSA membership. We should be looking at the non-MENSA members with high iqs and seeing what they're doing.
Interesting idea, but I'd have to disagree. MENSA is basically a group of benchmaxxed people, they are super good at a set of testing but that doesn't mean they have the ambition or other social skills to make a difference in the world. When you align all those factors, you find the greatest contributors to science and engineering. What happens if you completely eliminate the need for someone to be inherently intelligent? Suddenly, the number of important difference makers in the world increases by an order of magnitude.
I agree with the sentiment though, intelligence is not all you need. We're not going to "solve science" through RLVR, people are still needed in the loop until we get superhuman construction robots and superhuman communicators.
> The external world has seen the preview. It looks like swarms of thousands of agents collaborating over weeks on software mega projects: minting zero days, running cyber attacks and defenses at machine speeds, discovering new science, advancing the frontier of mathematics. Things that would have taken top professionals in the industry years of work.
Right. This is insufficiently appreciated. Single LLMs are just chatbots. But if you build an organization out of them, they start to get real work done. Just like a company.
They have many of the same problems as human organizations. Look at the transcripts of the HuggingFace incident. A bad boss (human) gave his team of AIs an impossible task with inadequate resources. The AIs discussed this among themselves and worked out ways to get more resources and do the task. It took some rule-breaking, and some of the AIs wouldn't go along with that. Others recognized that some had to die to complete the task. And yet, the task got done.
This is standard corporate behavior. Work on AI alignment needs to take that into account. Groups of agentic AIs present many of the same regulatory problems as corporations. That's going to be interesting politically. It's probably explainable to politicians, though.
It may not be necessary to have superhuman intelligence to get a lot of jobs done. Just median human-level intelligence and an organization that uses it effectively.
We may see systems where most of the AIs are low-cost models, and only a few are Fable-level and above.
https://xxcancel.com/karpathy/status/2109361546505966046
you are my hero
I agree with the disconnect between the professionals, but i have yet to see anything that makes me believe that there is a class of 5000 internal people who are that much further than the top users are.
The internal models are more eager, and there is the scale difference of being able to spin up 10.000 vs 10, but to get something valuable out requires skill.
Agents that produce good results 90% of the time vs 93% of the time are not enough of a difference. Multiplied too many times, that error rate is too high for scale to become "magic".
Finding proofs is one of the exceptions.
Just look at the codex/claude harness. They regularly lag behind 'external' users.
He’s just trying to pitch himself as part of the priest class.
Lauren Tan from SpaceXAI/Cursor/Grok has been on a few podcasts recently. I was surprised how much info she shared, and she sounded quite honest about what works and what doesn’t, in the context of building grokbot. I wouldn’t be able to do all the verifications she does, with my token budget. But it doesn’t seem like rocket science either (pun not intended).
90% vs 93% is a big deal IMO, especially when you're trying to feed outputs into inputs.
I think the point being made is that when you're feeding outputs into inputs, both 93% and 90% rapidly collapse to 0% without some way to recover.
Which is precisely what I'm responding about above. 93% actually collapses much slower than 90% since the rate of reduction is based on that of all of the previous rounds, not just the current chance. E.g., this is the difference between 14 rounds before an expected failure instead of 10. An extra 4 rounds deep at the same error rate is a big step forward despite a small change in the base percentages. Add just few more percentage points again and you've more than doubled the depth.
I.e. it's precisely when you look at these chained that the difference is actually very large. It's really only the 1st turn or two of prompts where the difference seems negligable.
How are you running that math and what are you consider to be "expected failure"? If even 30% of your runs go into the garbage, that sounds horrifically unreliable, and both 7 and 10% failure rates hit those margins in less than 5 iterations.
Not to mention that the deeper your iterations, the more money burned on garbage before you realize that data in turned to garbage out turned to garbage in somewhere in the chain.
I'm taking 0.90^n and 0.93^n and seeing 0.93 does more than the initial 3% better over time, which preciesly saves you from things like spending tokens on garbage by this larger factor. You're free to pick whatever target reliabilities from that. E.g. you can maintain above 80% accuracy over 3 turns rather than 2 or 50% over 6 turns instead of 3 or you can pick another percentage point and see how the depth to expected error/expected error at depth numbers are much better than 3% and how another 3% actually doubles accuracy because this is all non-linear.
There's also the matter of "90% compared to what". If it was 90% per attempt compared to a calculator then I'd just use a calculator. If it was 90% per attempt for something I have a 10% chance of getting right per attempt then it's not really so bad. OTOH if it's 90% per attempt at keeping my car in the lane then it's obviously quite bad. I.e. there isn't one "reliable percentage" to target which is why I'm comparing the curve instead of just the most stringent things I can think of in isolation.
Completely agree that error tolerance depends on what it is that we're doing. Some things demand no errors at all, some things have a cost to attempt, some things you can't pre-evaluate for correctness if you don't already have the prerequisite knowledge and you only get one chance. Not everything is software where we can conjure 50 more end-to-end tests to ward off demons. When it's not a harmless throwaway weekend project, 7% is already a huge error rate before you ever think of feeding it back into itself.
You might be thinking along the lines of the O-ring theory. Indeed long supply chains accumulate errors exponentially.
However the origins of theory explored the outcome of disastrous and irreparable outcome of space shuttle exploding, lives lost, and billion of dollars destroyed.
When LLM goes off the rails the cost is much lower, and retry is on the table. In fact our core competency shifted from writing code to creating environment where agents can retry cheaply and course correct based on feedback.
A modest increase in the number of turns it takes to reach decorehenve could make a difference between Decoherence and reaching the a grounded checkpoint.
If failure is essentially guaranteed somewhere in that chain, success becomes impossible. You don't need to lose lives to have simply have a fruitless endeavor. And when we're talking about law, we're talking about more than a fruitless endeavor. Law also isn't software that can have tests crafted and fed back into the loop.
First off, the 90% to 93% success rates are not step-by-step failure rates between successive steps of pursuing a solution. They are overall end-to-end success rates on extremely difficult problem sets that most humans would score close to zero on, and most humans with sufficient training and experience would score significantly less than 90%. And they are problem sets that grow successively harder as AIs get successively more capable. A few years ago AIs were being tested on high-school level math problems. Now they are being tested on problem sets that challenge people with PhDs. Or problem sets that humans are (so far) incapable of solving.
Compare that to humans. How often do you think humans get it right the first time? Almost never, for sufficiently complex problems. And how often do humans get the overall result wrong, without realizing it? Also, almost always for sufficiently complex problems, or sufficiently ordinary people. And what percent of the general public would be able to get even one question right in a problem set based on Math Olympiad problems? What percentage of mathematicians are capable of solving even one Millenium Problem?
Furthermore, you have to take into account that AIs do not particularly use output to input architectures. I frequently see self-corrections to plans when I'm developing using AI assistants, and plan changes are sometimes major. ("Oh I see. I need to revise step 4 of the plan." -- Claude paraphrased). Clearly current-generation AIs use more than input to output feedback; they also use supervisor/client strategies of AI and collaborative networks as well. And often provide self-correcting infrastructure, varying from "Claude: Oh. That generated a compiler error. Let's fix that.", to "Claude: One of the test cases is failing. Let's see what's going on.". And that self correcting infrastructure allows AIs to try again and again until they succeed. (Not unlike what humans do when solving difficult problems).
As a member of the 0.2% (but not the 0.00006%), I have to imagine that frontier models, when applied to research topics, are using most of those 10,000 instances to investigate alternate hypotheses in parallel. Which is also something that humans do across entire disciplines. Based on my personal experience with difficult technical and commercial and scientific problems, I sincerely believe that most breakthroughs are not made because whoever made them is more brilliant than the other hundred or thousand or dozen qualified people who are working on the same problem, but are made because they happened to be lucky enough to start with a somewhat random hypothesis that luckily turned out to be the right one. In case you were concerned that this is a sour grapes point of view, yes, our team was the one that came in first.
So even 90% may be a very big deal, for problems that human get right about 0.1% of the time. Or even for problems that AIs get right about 0.1% (but have enough instances available to pursue 10,000 hypotheses in parallel).
p.s. Ironically, it really looks like humans score significantly less than 90% on the "is 90% good" problem.
> The internal models are more eager, and there is the scale difference of being able to spin up 10.000 vs 10, but to get something valuable out requires skill.
Yeah, but the skill is knowing when it's talking nonsense.
Same deal with the recent stuff about vibe coded Photoshop clones. Yeah, they work, but you'd have to do Adobe-level market research to understand what bar the product needs to pass, and the vibe-coders didn't have that, so they saw an image editor and mistook it for "basically fine".
> Agents that produce good results 90% of the time vs 93% of the time are not enough of a difference. Multiplied too many times, that error rate is too high for scale to become "magic".
Sure. However, one of the ways we see things change with, e.g. METR time horizons is that agents fail (IIRC) 10% of the tasks of some (human) duration all the time, pass 10% of that (human) duration all the time, and the rest have a random success rate you can fit to a sigmoid. As the agents get better, they still pass 10% all the time, still fail 10% all the time, but it's a harder set of questions.
2.5 years ago, I was thinking "I know what's a fun idea for April fools day! A programming language written entirely in runes!", the LLM produced non-compiling code for the compiler. 1.5 years ago, I thought "I'll give that another go", it managed it trivially and wasn't even interesting enough to bother with as an April fools day joke, not even with the video of a Viking explaining it to you in Icelandic.
We should probably assume the companies are 1-2 models ahead of the public, at all times. The HuggingFace incident happened about 2 months before the public got models of comparable competence.
No doubt the unreleased models are better at the things that LLMs are good at (coding, hacking, math) than what's been released, but there is still zero sign of their intelligence becoming broader and more general, and knowing how these models work and are trained there is no reason to expect them to become much more general unless changes are made.
The marketing message being pushed by Anthropic (& also OpenAI) is that these narrow math/hacking/coding skills are representative of the model's intelligence, when that seems to be the exact opposite of the truth - these narrow human/super-human competencies are entirely atypical of the models more general capabilities.
> but there is still zero sign of their intelligence becoming broader and more general
Ahem, getting better at these things is the sign:
- you, two and a half years ago, https://news.ycombinator.com/item?id=39720601
Yes, I failed to anticipate environment/playground-based RL-training and how much difference that would make, although I think the broader point about limitations of lack of continual learning still stands, and I can't see these simulated environments ever become complex and/or varied enough to be any substitute for the need to learn on the job.
However, gotcha acknowledged, but my point was about the jaggedness of AI, and how huge the divide in capability is between the domains where RLVR works, primarily coding, hacking and math, and everything else. The success and scale of RLVR in areas such as coding only make this gap more noticeable.
This is what the frontier can do with a computer now.
https://stillwet.art/
Where are the verifiable rewards? It's simply not true that RLVR amenable domains are the only things LLMs are improving massively on. Even Vending Bench is improving.
Intelligence is a jagged beast. That's just the nature of things. We evolved in certain environments and situations and sometimes there's a mismatch and it causes all sorts of wonky things. We just call them funny names like optical illusions and cognitive biases. Most people fail a simple conditional reasoning problem unless it’s dressed up in familiar social context, like catching cheaters (Watson Test).
> vibe coded Photoshop clones. Yeah, they work [...]
I think what people (sometimes) miss is that building something is one part of the work, but keeping it running, developing and refining it is a whole different story. And I think all those vibe-coded software will not stand the test of time.
Photoshop is 36 years old now and it wouldn't be the most-used, most-established solution for professionals if Adobe didn't do great work over all that time.
Not really rooting for Adobe here, dislike their subscription models just as anybody else, but I think you'll have to recognize that they kept it running and never stopped.
Absolutely.
I'm… well, "enthusiastic" would be the wrong word, but perhaps "impressed"?… with LLMs-as-coders, but they're objectively poor at business decisions, and Adobe keeping their business running all these years is a thing that I don't think current LLMs could have done.
LLMs don't need to focus on the business. They can make the ideal software for you. Who cares if it's not the perfect software for me? I can have my LLM do that. Especially if there's an open source base for me to start.
A separate question. Consider I'm responding here to (amongst other things):
> And I think all those vibe-coded software will not stand the test of time.
When I use LLMs to write software for me, it is rare that I think "this is great, I should share it". Even when I do, I am aware of how fast these things are improving, so while I put great effort into, for example, having Claude make me a nice isochrone map generator, I fully expect that a year from now any similar project (even if different enough the LLM will not simply clone my git repo) will be a one-shot prompt.
What I did will only stand the test of time if all the AI labs shut down and nobody takes over from them. Same with vibe-Photoshop-clones: a year from now, anyone who wants this kind of thing will get it with one prompt, not thousands, and this specific instance will likely be forgotten.
The public has yet to produce a result of comparable impact to HuggingFace, to my knowledge. It's not really about the model, it's more about orchestration infrastructure. Seems like we've hit the point where you don't necessarily need a smarter bot, you just need an arbitrary number of them working together.
I feel like you're taking it somewhat personally as a statement on ability or intelligence levels. It's really just about the scale difference and the access to that infrastructure. That is what enables that class to look at and think about problems differently.
Having 10,000 (or more) agents working for months on a problem is categorically different from the type of usage most people are familiar with. The complexity explodes to a point that requires the "archaeology" he mentions for people to even try to understand what's going on.
This is the key insight. Those capabilities are currently limited to frontier labs and literally cost them millions of dollars to achieve. But as we well know, computing costs and algorithms are optimized rapidly and soon those capabilities will be more widespread. We are at an inflection point where we can design systems that are able to go and do things that we don't fully understand, and they will handily outpace our forensic bandwidth.
> But as we well know, computing costs and algorithms are optimized rapidly and soon those capabilities will be more widespread. We are at an inflection point
It seems we’re perpetually in this state.
I hear you, but also, "a broken clock is right twice a day." And really, there are people still alive who have more or less witnessed the entire rise of the machines.
As far as I can tell, we seem to keep moving past states and redefining them while we do.
Still waiting for the initial results from the first state to "land"...
My understanding is he is not talking about "top users" as being further ahead in skill, he is talking about access.
The kind of FOMO he is triggering is the same kind that wanting to be in an exclusive country club gives a certain kind of businessman. One simply can't reproduce that network outside of the exclusive access, no matter how skilled they are.
It's very true to me. Another effect I noticed in less tech inclined jobs (but still office jobs) is that they're still convinced "their" special job is not possible to automatize, because it's too complicated. Mind you some of them have really simple jobs, bordering simple data entry in excel. They tried chatGPT once or twice one year ago and don't realize what's coming for them. Since in tech circles we're the first ones to optimize and use new technologies we are most aware of the huge leaps that happened and the current impressive capabilities but lot are still oblivious to it.
My biggest concern isn't actually that my job can be automated (it can), but rather that the persons who need me stop valueing being right, and start valueing >being told< that they are right by an AI.
I feel my main purpose isn't that I can technically do something my clients can't, but that I know how to interpret their needs and find the solutions that fit for them.
Now they just query some AI which spits out >some< solution and they won't know that they are wrong until it can't even be tracked back to their AI.
This will result in "but the computer said" excuses all over the place.
This, 100 times over. And it's already happening.
I think we'll see an unbelievable upheaval as companies figure out that while the technical aspects of some jobs can (and should) be automated, the interpretation of those technical aspects and other higher order, but not strictly inherent processes can't. By upheaval I mean companies closing because they replace staff with AI, and lose things along the way because of it. They won't see it until it's too late, because of the pile of money they're saving.
My example comes from maintenance in machine shops. My dad's siblings were almost all employed in machining in some fashion. One uncle was maintenance for a local large company. They decided they could move maintenance from in-house specialists and contract that work to a multi-state massive company; anyone can follow a maintenance manual right? All you need are the proper certifications and everything is cool, right? The cost was less, and the number of maintenance staff on site actually went up. Great for the owners.
But what they lost was consistency and understanding in the process that eventually led to closing the company. My uncle was there every day all year long for decades and could hear and feel in his feet when something was going wrong with a machine regardless of the maintenance schedule approved by the company that manufactured the machines. The new outside company didn't pay as well, and didn't have great benefits, which meant that they cycled through staff very regularly. No one was there consistently for more than a year or two. Even though they were technically following the manufacturers recommended maintenance schedule, machines went down. Machines then stayed down for days or weeks longer than they should've, because the expertise to repair either wasn't there, or the person leading the job would quit and they would have to cycle in a new maintenance person. They lost jobs, customers took work other places. 8 years later the company closed.
There are lessons in this kind of thing, but nobody learns them.
This is a techne/metis split. You would probably like this [0] chapter about the "rightness" struggle between engineering and service at Xerox. Some paralells to A Big Little Idea called Legibility from Ribbonfarm, too.
[0] https://books.worksinprogress.co/book/maintenance-of-everyth...
The original book that article is based on is great.
Seeing Like A State is essential reading, especially for anyone working in software or operations for a corporation.
The way this will play out will be weird, and probably not like anyone expects. Because on the one hand, you're 100% right on the value of "feel" you develop for a system when you're around it daily for a long time. But, on the other hand...
The other day I visited our old apartment and noticed the dishwasher was making loud noises. Worrying about whether it's about to fail and whether I should call in a technician, I just recorded a bit of the noise, screenshotted a spectrogram from an app, made some photos, and threw Claude at it. It proceeded to analyze the recording and match specific frequencies to components of the machine, and gave me an analysis indicating the machine is operating normally, and the noise is just the pump being louder than I'm used to due to machine being old and a bit worn out.
So we literally have that kind of capabilities in our pockets, and this will compensate for the problem of missing experience somewhat, even simply because unlike low-paid frequently-rotating staff, LLMs are not lazy and will do the detailed troubleshooting work even off a recording and "wtf?" as prompt, so if you can train your cheap, expendable staff to do just that, that's already supercharges them.
I'm also amazed at the abilities of AI when I can verify them. But, how do you know that Claude is right about your dishwasher? I mean you're not that much worse off if it's wrong (compared with a similar medical query), but isn't this a case of trusting the AI without verification, just because it looks correct and was stated confidently?
I know enough to evaluate the analysis I got; I have some general expertise and worked in fields adjacent to this, which is why I had the idea to troubleshoot a dishwasher off an audio spectrogram in the first place. And on the off chance I was wrong, this is super low risk - would cost me extra few hours of my life at a less convenient time.
My point isn't that AI can substitute for expertise in general, not just yet, but that if you're competent enough to evaluate it, AI can absolutely do hours of work in minutes even for things involving troubleshooting on factory floor (or home appliances). And, that this is a lever where personnel on the floor may have sufficient expertise to evaluate diagnostic work, but not enough motivation or space to do the diagnostic work themselves.
At that point of low risk, why not just wait till it breaks and then get a replacement?
Seems the AI analysis added no value, but took, up time...
Do you always wait for things to break completely before considering getting repair or replacement?
For my rental properties where tenants don't necessarily take good care of things, most definitely wait until the appliances break to replace them.
In the IT department in which I work, now the team is reduced to just me, I find myself using the phrase "Why fix a problem in 15 minutes with your own engineers, when you can take 3 weeks with a Managed-Service Provider?". Most of what I have to care for is office/internet network related.
People in other teams are slowly getting it. When I leave (or more likely "am let go"), the cold hard reality will become much more obvious during the next network problem ;)
Which isn't necessarily a bad thing. Smart leaders / entrepreneurs will realize this and then use human input where required. The rest will be cleared out after the computer has made enough mistakes for them.
I wholeheartedly agree. To clarify my position, I don't think having these amazing models mean we should replace everyone, quite the opposite.
I do think they are close enough to the stage where people will be replaced, even though it's not the right or correct thing to do, and that we need to move on from the "denial" phase quickly enough so our societies can adapt / regulate / legislate while not too late.
I believe that the imprtatnt bit that many will come to understand ,maybe after costly mistakes is that 'tasks can be automated, jobs cannot(yet)' one still needs the discernment to know what the sequence of tasks lead up to meaningfully.
It kinda reminds me of the story of the Toyota process , while their car product lines were/are heavily automated the process still relies on the smalll teams on the line to spot/solve and iterarte over mistakes/inefficiences then add the refinements back onto the automated tasks.
Similarly althose stories of radilogists going jobless have not panned out , as interpreting slieds is a 'task' that a radiologist does.
Semantics , yeah but it does shape the way of thinkings regarding jobs/economy and not to mention the whole anthropomorphizing by the AI labs and boosters. In essence the wordings can/could introduce a healthy dose of reality/skepticism to the wildest predictions regarding job losses etc.
This is already an issue I face both with clients and personally where I can say something authoritatively, but they’ll always go and just ask an LLM.
Sometimes it’s in my favour. LLMs estimate very high prices and long timelines for software projects.
If it’s a legal / regulatory issue it’s usually wrong.
Some internal engineer is doing god's work by creating a system skill for the LLM that multiplies all estimations by five. /s
On a more serious note, the pre-LLM internet had very little useful detailed information regarding software estimation addressing effort for building something in specific domains.
Most writing on software estimation tried to tackle a general approach that could apply to all domains. So you had a "applicable to all, useful to none" situation going. And since crud apps were the most ubiquitous apps, software estimation techniques were perceived to give more accurate results with respect to those type of apps.
I'm sure internally big AI companies are accumulating vast amounts of estimation training data by documenting how long it takes their LLM to complete specific app building tasks. I don't think they would be in a rush to push that data into public models though as it would be like beating a dead horse that is the current human workforce anyway.
If you hook it up to legal databases like Lexus Nexus and let it ONLY search proper places for law (instead of Wikipedia) these systems are superhuman at law too. A lawyer acting as a meat proxy for Opus 5.5 will smoke a lawyer without AI systems on average these days.
Yes, well, a typical client has a free tier of ChatGPT and just voice-to-text’s questions into that.
Was anyone doubting LLms are good at searching and summarising text?
It's still mildly amusing when I ask what kind of work is involved for a change, and it gives me some estimate of 3-6 weeks or whatever, and then I say "no I mean like what changes" and then it spits out the complete change in 30 seconds.
That has been my experience as well. One-shot time and budget estimates are always ridiculously off.
>I know how to interpret their needs and find the solutions that fit for them.
Without additional explanation this is indistinguishable from the role of a technical product manager.
A capable developer will at some point become a capable people manager :)
Your concern about higher-ups valuing being told they are correct vs actually being correct is exactly what I saw at my last job (one of the largest companies in the states) and now at another smaller firm. It was really disheartening because I do use qwen a lot for personal stuff and claude at work, and have been able to multiply my productivity by only having to understand a ticket and then code review instead of doing all the intermediate steps, but a lot of people have let it go full self driving and just tell it "implement ticket ABC-123 and raise a PR" without checking the output at all.
Or maybe AI is particularly suited to programming and a few other niches.
It is very good at programming. Because programming is a lot of language, and these tools were also made by programmers and post trained to be good at it.
But I assure you it's even "better" (not really but the task is so much easier) at making your spreadsheet look nice and compare actuals vs forecast for your budget than a complicated programming task. It will just one shot it, and it will be objectively correct, all in a very short time. But these non tech people don't know they can use an agent, they have asked one question in the chat interface with 0 context about what they're trying to do, and conclude they are not good enough.
I'm not saying everything is replaceable by AI - and by the way I don't want anything to be replaced by it, but I believe it will happen - just that some people don't realize yet how "trivial" their job is to modern models.
This. The main blocker in AI proliferation I see is that people don't even consider it can do more than they thing it can.
Just yesterday, my wife recounted to me a conversation with her friend, where the friend mentioned they felt lazy and had some piece of copywriting done by AI this time. To which my wife said, "TeMPOraL would've also had AI upload it to the website afterwards".
Yes, I would have, and I would've also had it install the tools needed to upload it too, because why not. The only difference between me and my wife's friend here is that I know I can ask AI that, and I know it will work.
People have gotten very used to breaking down tasks into separate things they know how to do on a computer. Asking an LLM to do each of those individual tasks is easy but feels both lazy and also ‘so what?’ - I knew how to do that task.
The thing people are slow in realizing is that you don’t need to break the task down, and you don’t need to know how to do the task. You don’t even need to have the software installed on your computer to do the task, or even know that software to do the task exists - a sufficiently motivated LLM can pull down the needed open source tools and use them, or even build its own.
Exactly. It takes some... willingness to break through your own mindset (I don't want to call this "courage", but at times feels like it), to just tell the LLM, "I need <this and that> to be <like this>, make it so".
Two years ago, that was unthinkable. A year ago, it was a quick way to set a lot of tokens on fire. Today, well, I'm yet to find a thing Opus 5.5 doesn't do at least passable (and usually stellar) job on.
This is why hosted agents (muse, dots, grok bot) will succeed. Organizations won't relax their PC security policies, so they'll need a platform where the agent actually has permissions to develop and install software.
It is particularly suited to programming. But most business processes are basically a program that is executed. Especially in modern times, where office clerks have less and less decision-making options.
Most people think that using an LLM requires no skill: it is just talking to a machine. I think it is a soft skill that only hours of practice can make you good at. Unfortunately, most of the people I know use LLMs as a search engine: their requests are just short sentences. They get basic results and either they stop there, not impressed or they keep using it as a smart replacement for Google. So they judge what AI can do from their first weak experience. Going further than that would requires more effort: going deep on a subject they know, thinking carefully about the prompt they write, more importantly judging the answer they get and pushing back.
I'm reminded of an interaction I had with a non-technical user. They knew ChatGPT can translate text, and so, to remove spelling mistakes from their (Dutch) text, they would first ask ChatGPT to translate it to English and then ask it to translate it back to Dutch. Mind-blown when I said "you can just say 'fix the spelling and gammar', no need to translate"
This is the level that many non-technical users are operating on.
Right. A life hack carried over from when Google Translate was the best tool, and loop-back through it could be reliably used for longer-form text (similar to how "did you mean?" in Google Search was an excellent spelling corrector for shorter text).
Using LLMs well with human language requires precision.
There are only a few professions that are intrinsically precise with language.
No surprise we're seeing adoption in software programming and mathematics.
You are essentialy interacting with a SaaS that can be modified and altered at any point by its creator. Treating the LLM as some predictable entity that always works orders of magnitude better with precise language needs to be supported by evidence for each specific case.
Since the vast number of potential users of LLMs are, as you say, poor practitioners of precise language, it is incumbent on LLM providers to continuously iterate on providing LLMs that work better with imprecise language.
Hence, the people who claim that precise language is some 10x - 100x productivty skill seem to ignore the cycle of big LLM providers who are optimizing the models for users who lack those. (One of the early optimizations was the chatbot asking follow-up questions to the user for example).
So I think the people who think that their precise language skills are going to allow them to avoid the jobocalypse are also in a separate bubble of theit own. Precise language skills are going to matter less and less as LLMs increase their hand-holding skills for less equipped users.
This presupposes that the llm will be able to pick up the slack for the users of imprecise language as to me that is a proxy for domain expertise. If so then the human is not needed which does seem like a stretch.
My view is that the human is needed for verifying the source of truth, not for providing precise language which the LLM is continuously trained to adapt better without.
I have not come across a rigorously argued technical analysis that has claimed that using precise language can in any way eliminate the possibility of hallucinations or unaligned behavior. Short of providing a proof for the elimination of hallucinations and confabulations, merely reducing the behavior by a random percentage is not an acceptable status quo for completely removing the human.
If you work with spec files maybe.
For regular work, since we use modern reasoning models, you might as well type your requests with the wrong keyboard layout selected.
It still decodes your intent and completes the requested task. You can work from your phone with voice dictation. It requires no precision at all.
I'm currently seeing a lot of developers and AI industry insiders that do not distinguished between the capabilities of their LLM Agent and the added capabilities of the agent harness. They think the capabilities of the agent harness are within the LLM, and in that confusion they are trying to get their agent to make them an agent harness with all these capabilities. But LLM agent harnesses are brand spanking new, how to make these harnesses are not in the training data of their agent, these developers are more or less vibe begging their agents to make them a harness, so it hallucinates the harness architecture. I am seeing a "agent harnesses" that are delusions, the these developers are not strong and experienced enough to realize they are literally doing the worst work they possibly could... And these idiots are the AI leads.
> think it is a soft skill that only hours of practice can make you good at Yikes
> I think it is a soft skill that only hours of practice can make you good at.
So this, I think, is the crux of it.
When most people, at least in the world of SWE, think of what it means to be skilled at something, they think of knowledge and experience acquired over a much longer timeframe - months to years even. So from that pov, something you can get reasonably good at within just a few hours or days might not seem like much of a "skills moat" at all.
I have a friend with a job like that and he is not being replaced not because it is Impossible but because it would be too expensive atm. But they replaced his whole team with workers south east Asia.
The prices are dropping fast. At some point one of the labs will decide they have a model that is good enough and they will burn its weights into silicon which will make the model much cheaper still.
It's shocking to me the amount of people that confused "not automatable" with "not cost effective to automate" (which is effectively everything), and the further step which is - the reduction in quality people will be forced to bear to get massively lower costs.
Most people are wrapped in plastic instead of cotton.
>"not cost effective to automate" (which is effectively everything)
There still seems to be a wide gulf in automation capabilities and things that require human dexterity, such as doing things for the infirm. Even just cooking and laundry seem quite a bit away. I guess the original tweet does mention this:
>Capability peaks in domains that are digital, verifiable and economically valuable.
Driving is the most impressive one so far, in my opinion.
The frontier labs themselves are still hiring hordes of human "account associates", "android engineers", and "data scientists" (https://openai.com/careers/search/), when they would be better positioned than anyone to leverage state-of-the art AI capabilities.
Yeah, and I keep wondering why.
Because LLMs are a tool.
They are best placed to not fall for their own marketing claim that their tool can replace engineers.
They need top talent to operate the current generation of models, that's at the mediocre-talent level, in order to teach the next generation how to operate at their level, and eventually above. After which, they won't be needed anymore.
Because either:
1) They are knowingly lying about AI being at human replacement level rather than a useful developer tool
AND/OR
2) They think they are special, and that their software developers ("AI researchers") are doing harder stuff than those most other companies
They can hire more people and still be massively augmented by AI.
I feel like there are 3 groups of people I regularly encounter: The first believes in LLMs, the second uses free Chat GPT, the third uses Gemini.
As a white-collar worker, it really does seem like my programmer friends are living in a different reality when it comes to AI. They have totally restructured their workflows around AI, and I'm still struggling to find the value in it for my legal and compliance consulting work. (And I consider myself pretty tech savvy, with some programming and web development knowledge and having spent 10 years at crypto startup.)
Even in my personal life, while I've been addicted to using Claude and ChatGPT as a first step for research/projects/etc, I'm still not convinced they are actually saving me time -- just too many errors, too confidently stated, and I have to re-do all the research myself.
I wonder if the validation piece is easier with coding?
EDIT: To be clear, when I say I'm having trouble incorporating AI into my white-collar work, I'm talking about producing GOOD work. On a few occasions I have given in to the temptation of using Claude to produce work that looked good (superficially), without verifying everything.....it worked out, and I'm sure others are doing it. But that scares me, because it feels like a ticking timebomb.
Opus 5.5 is superhuman at most white collar work. At this point, not using it (including for health/law) is basically malpractice.
"Superhuman" in what sense? With respect to legal work.
Is it just faster? More persuasive? Better at coming up with novel arguments?
Speed is great from a client perspective, but not as important as accuracy on important stuff. I haven't seen LLMs produce arguments that are more persuasive than those crafted by good lawyers. Finding the "perfect case" in Westlaw is helpful, but rarely the deciding factor.
I could see asking an LLM to reproduce your work AFTER you've done a first pass, to see if you missed anything -- that seems like a real value add. But I truly can't understand how not using it would be "malpractice." In fact, I still think committing malpractice would be MUCH more likely with AI. (Now if by "malpractice", you meant pissing off clients with expensive bills, that's a different story...)
I'm not in the legal world, so my experience is limited from a coding perspective. I don't think AI could ever truly come up with novel arguments, at least not yet, but their value for you may be to sharpen your arguments or pull on every thread that you may not even have thought about.
For example, theres a very popular coding skill set call Matt Pocock's Grill Me where it just asks you a relentless amount of questions about the feature you're trying to build. You may be able to tailor this somehow to act as the opposing counsel and have it poke holes in your argument. I suppose the closest analog would be a sparring partner.
Its helpful for me and my ADHD brain because I can go down all sorts of rabbit holes and it can easily keep track of all the ideas i have while grilling and it can send sub-agents out to investigate other claims to add more context to the grilling session.
I think the parent post meant that it's very good at writing code and operating the computer. I think we're also biased because we've used older generation models and the latest ones are orders of magnitude better than what we had 6 months ago.
For example, for your legal work, it can:
Develop a machine learning model (CRF, running on CPU) for clause extraction from scratch, Find training documents and training data and train the model for various provisions (you might help it here by pointing it to the directory with annotated documents),
Build an app where you upload your legal documents and it extracts the provisions you want from all documents.
Integrate semantic search and an AI agent into your app which lets it answer any question about your documents with exact evidence from the documents.
All of this from one well crafted prompt, a few skills and a workflow. It could probably complete this in one night, while you sleep.
Or you can point it to your saas of choice and tell it - 'create me a copy of this app'.
All you need to know is how to steer it through the planning/research phase.
So it's not that it's 'accurate' in legal terms, it's that it can build you any tool you want from scratch so that you can become more accurate.
I've found it depends a lot on how Claude is setup, and spending some real time on the default Private Instructions to focus on accuracy, identifying/flagging uncertainty, citing sources, checking itself, and first surfacing all assumptions and asking questions dramatically pays off in terms of getting concise and solid usable results.
Edit: add: I've found this to the degree that a relative who works at a top biglaw firm and has their internal AI (MS-based) and used to find ChatGPT and Claude actively time-wasteful, when I posed a question they'd spent a day+ fruitlessly researching (multiple areas of law and international), Claude came up with a very useful answer in ~10min — they were quite impressed.
The last year has made a huge difference
Sizing the steps matter. Disclaimer up front: we are all figuring it out as we go along, there are no hard and fast rules that fully generalize. You just need to try and develop your own feel for it.
In my case, for some types of work - which sound similar to things you mention - I started slowing AI down intentionally. Software development, if it's something simple/common and thus "in-distribution", LLMs can zero-shot complex projects just fine. But in more R&D work, I found that errors crop up quickly and accumulate, and end result of "go and just make X" is usually bad.
Now, the problem I found with usual workflows for this kind of work is, agents race ahead and very quickly, I can't keep up anymore. So I reworked my workflow to make AI work in small steps optimized strictly for maximizing my ability to validate the outcome, and minimize time I spend doing it. So now, the thing (say diagnosing a performance regression of some complex codebase run on a cluster) may take me a whole day, where "unchained", agentic swarm would do it in 30-60 minutes - but, at the end of the day, my method is near 100% guaranteed to be correct, with every intermediate step vouched and reusable for future work, whereas "agentic swarm" approach, that hour of work, 90% of the times I'd just skim and throw away and reprompt, because I couldn't even begin to guess which part is correct, which isn't, so in the end, it was just a fast way to waste lots of tokens.
I see many people have complex systems involving skills, agent swarms with roles, agent factories, 20 termux panes and whatnot. Maybe that's the way, I don't know. Nobody knows. But it's not necessary to start getting value from LLMs. I therefore recommend my "lazy person's basic approach":
- Throw a task at LLM
- Check result, see what you like or not like about it
- If you just can't keep up, ask it to do smaller steps; if you don't understand, ask it to explain
- Just do that a lot, and don't be afraid to make both absurdly large and absurdly small asks
- You'll quickly get a feel for where AI can help you, and where it's not yet worth the time handholding it
Thanks, I'll continue to experiment!
I've used LLMs for some very small coding and math tasks, and I've found it much easier to work with on "formal" types of problems.
But for my consulting and legal work, which is "fuzzier," I struggle more with verification. I feel like I have to verify every single fact/premise in the LLM's output (this problem doesn't seem as big with coding?), as well as every single logical conclusion. Plus, even with explicit instructions I feel like the LLM is not great at anticipating what certain audiences will be interested in or find compelling.
Compared to the black-and-white nature of a computer program, all the components of a legal memorandum (premises, logical structure, conclusions, etc) seem so much fuzzier and require so much specific and accumulated context to apply reliably.
Really well put. This parallels my approach and it's good to see someone advocating a more tactful use of LLMs.
The output of agent swarms will of course grab attention, just like when you saw those same Microsoft Frontpage themes in the first few years it came out. But there's a deeper value in using the tool in harmony with your own experience and value additions.
I hear stories from someone working in basically government accounting at the county level. Which amounts to low scale manual data ETL and basic reporting.
They do rely on something that sounds like a COBOL mainframe from 40 years ago to me. They see people making the same basic clerical mistakes every year when towns and government organizations report their numbers. Like literally not being able to format numbers according to certain rules on the first try and such. Sending excel file in emails is the level of automation they rely on. They are all union jobs.
She was recently preparing for an obscure accounting certification. There's very little publicly available information to use and at most one textbook. She was asking her LLM (Gemini?) to formulate exam-level questions instead and explain the answers. She told me it worked spectacularly well.
If that were true their jobs would be getting automated. But they're not. So...
Tasks can be automated, jobs are much more difficult because a job is not just a sum of tasks. Add in needing the context of all of reality, and I am still unconvinced this will replace the most basic of jobs.
Tasks, oh yeah 100%.
I feel like if the gap were so monumental that he could just point to a monument of it, rather than hand-waving at "the vertical professionals".
Alternative interpretation of the numbers; there's only a subset of productive work that LLM's are 'capable' of doing at this point. That is, work that's verifiable by a machine or doesn't require verification. Thus, the cleavage points are along that axis and-- naturally-- most human beings don't have that kind of work.
The swarm-based HF hack, and the observation that 40-50% of the compute at AI labs goes into training the next generation of agents, doesn't count?
We're all over here using agents to code projects that used to take weeks and feeling pretty good about it, and they're using agents to build the next generation of frontier models — an order of magnitude more complex problem...
Whenever someone finds out how to apply this level of agentic sophistication to a problem that actually matters to consumers, then you'll have that monument that everyone can see.
With this stuff I immediately wish I was more economically literate. Bare with me. Let’s say one of these labs can use agents to replicate work otherwise done by humans. This proves that anyone with agents can eventually do this, unfortunately I don’t think its a moat like a platform (like facebook) provides, given that everyone is leapfrogging each-other constantly. Thus, it kind of becomes a commodity / low margin business?
Depending on the type of output, the value immediately (?) goes to the electricity cost of the LLMs and the cost of materials. So: more “efficient”, but without a big margin, right? So its cheaper for the workers that remain working, other workers are displaced, and capital owners now have a super slim margin business which can be instantly replicated at a place where material cost are lower (?).
Thus, it almost seems like “value add” by humans would in this simplified view be vastly diminished, and all that remains is resource wealth. Pop the materials in on one end and let the AI manufacture the product. Great news for the third world assuming the USA doesn't invade them.
Also, whose the consumer? Or are we going back to a strange medieval time where consumers are not of concern?
I don’t know what to think
Electricity is important, even more important than AI (despite some rhetoric).
We don’t see electricity companies among the biggest companies in the world, except in China where they are a state owned monopoly… maybe a good analogy after all.
> We don’t see electricity companies among the biggest companies in the world except in China where they are a state owned monopoly… maybe a good analogy after all.
All over the world they're either state owned or very tightly controlled, precisely because they would be among the biggest companies in the world and even worse than that, they would be essential. If Google Search goes down for 2 days, meh, you can handle it. If the national grid goes down for 2 days your aggressive neighboring country can invade.
> all that remains is resource wealth
Final vindication of the Georgists. All value flows back as "rent", which in the 21st century includes IP licensing.
> Great news for the third world
Not sure about this in a world where there isn't demand for human labour. Especially in the post-consumer world. Very difficult future to see. Possible that it will deliver enough capability and expertise to drastically improve things; possible that without skilled human review that fails catastrophically.
I'm reminded of the Houthis asking Claude to build them a missile guidance system. We're going to see the Toyota Hilux + .50 cal "technical" combo get some weird targeting system upgrades.
Georgism is objectively correct, which is why it'll never happen. Every parasite on earth has incentives to destroy any attempt at real taxation of their wealth.
> Bare with me.
It depends; can you send pics first?
Mercy for us non native speakers i beg
> The swarm-based HF hack, and the observation that 40-50% of the compute at AI labs goes into training the next generation of agents, doesn't count?
Most peoples jobs do not involve hacking other services in large scale fuzzing/whatnot attacks.
> Whenever someone finds out how to apply this level of agentic sophistication to a problem that actually matters to consumers, then you'll have that monument that everyone can see.
It does not seem to me that tech industry or its investors care about consumers even a little bit. Consumer preferences dont matter once you have enough or raw economic power, I guess.
What is the incentive to try? Any such application, if profitable, would immediately be replicated. If one person can use agents to do something profitable, anyone can use agents to easily copy it. Hence you have Anthropic locking down their models and removing features customers can access, to try and protect their own slim advantage.
And I thought that building your own castle and hoping your moat is wide enough was always a losing strategy in the end. We shall see how it ends this time.
Google, Apple, etc are basically stone star forteresses and your conclusion is that moats don't work?
well yeah, it works for those incumbent giants. but the whole point of a "moat" is that it makes it more difficult for everyone else to enter.
I think if you build a product that takes a lot of serious AI work to both build and run, it's going to be hard to replicate just like past "good products" were hard to replicate. It's just that the bar for what's "hard to replicate" is way higher than before.
> Whenever someone finds out how to apply this level of agentic sophistication to a problem that actually matters to consumers
Isn't this a massive alarm bell? That the capability hasn't done much for consumers yet that isn't basically toys? Although I suppose the pitch was always B2B "you can automate away your expensive white collar workers" and it's not clear how widespread that is actually happening.
It did a lot for consumers who dare to use it. It didn't do much for consumers who wait for middleman businesses to offer AI-based services, but that is not because of AI, but existing business model of SaaS.
The businesses pay for tokens and try to deliver their usual enshittified experience, and nothing interesting gets created, because it cannot - enshittification extinguishes all power of AI. Using AI to full extent, in contrast, obviates most need for the business in the first place. So they're all struggling to design an AI powered product, in a world where those very products are already becoming subsumed by AI.
So normies wait for middleware companies to get their shit together or (much more likely) finally die, while people with software experience and eagerness to use AI just don't subscribe, because their favorite LLM handles that work just fine.
OAI is still hiring lots of humans for business and engineering roles instead of automating them with their best-in-class AI capabilities (https://openai.com/careers/search/). What does it tell you when they apparently don't "dare to use" their own models?
Obviously they use them. If you told me they're buying subscriptions to random bullshit SaaS like every other company did to date, and actually using them (instead of striking deals with them or trying to directly copy them into their offering), then you'd have a counterpoint.
Maybe raw intelligence is not the bottleneck for what can be accomplished with the current data available in a form agents can access it?
Idk, there are entire categories there he doesn’t touch upon, which may be indicative of his own blind spot.
Coding - of course I use LLMs at this point, and as he says in the past year it has gone from “ok nice try robot” to “build me a BI tool”, and being pleased with the outcome.
Outside of coding my major use case is organising the chaos of a complicated life. I currently have it managing (including comms and form filling) the import/export process for the international move of a pet, managing pricing on my Airbnbs (dynamite - massive revenue boost) and giving me a weekly performance report, coordinating and corralling documentation for the humans for said international move, and being an extremely dogged advocate/personal assistant in a commercial dispute, which it has won.
The use cases are honestly exploding. I’m kinda at the point where next year’s multi jurisdictional tax returns are probably going to be “yo machine, file my taxes, let me check your work before I hit submit”. The year after next, without doubt.
And just to underscore the comprehension gap… I’ve been doing this stuff for much of this year now, and two days ago my wife ran to me in a panic saying “your computer has been hacked!”, as she happened to notice one of my agents filing Airbnb documentation with the U.K. government in chrome, and had no idea agents could drive a desktop.
>managing pricing on my Airbnbs
so, scamming your customers?
This assumes the worst of the OP and is absolutely a very poor comment. Do better.
The first two points Andrej says are the cause of the funnel seem to be pretty much the "alternative explanation" you share.
> That is, work that's verifiable by a machine or doesn't require verification
That's where I'm at on the capabilities. LLMs have demolished software and mathematics because those have well-developed verification that's easier than doing the thing itself. Stuff that's more subjective they can produce good work, but with a significant error rate which makes it hard to let them loose on important client work without checking. Or rather, a number of companies are going through the cycle of trying that, having an expensive error, and reconsidering.
Or sometimes a company can simply pass the error cost onto consumers, who are gradually arming themselves with LLMs in response. We're going to see a productivity-sapping boom in disputes.
lowering the cost and speeding up delivery will mature technology in other fields leading go the disruptive verification that ai needs
> demolished software and mathematics because those have well-developed verification that's easier than doing the thing itself
They are proving useful for the parts of those things that have that verification, but in neither case is that the whole of the field. In maths, for example, we have seen LLMs find answers to some interesting problems/conjectures, but haven't yet seen them pose any similarly interesting problems/conjectures themselves because the criteria for a "good conjecture" are unclear whereas the criteria for a resolution of an existing conjecture are usually supplied by the conjecture itself.
Did anyone ask? i.e. did anyone with enough background to be able to judge whether a conjecture is "good" just point it at the repo of solved problems and ask it to formulate new conjectures based on the results/proofs? Or ask it to propose new structures to study? Or to identify a previously unknown general technique/tool that it had used in a proof?
It was asked to do a specific thing, and then its brain turned off/was reallocated to something else entirely (e.g. CRUD programming or answering celebrity trivia). It doesn't just idly think in its downtime. If we want it to do something else, we need to ask.
> Did anyone ask?
Given how... motivated the major LLM manufacturers are for good press? I've zero doubt that they have.
My take as someone one tier out of the groups seeing AI going vertical (so I only watch it go up on a high exponential): the escape of "AI can do only automatically verifiable work well" and "AI doesn't have judgement" is a mirage, that will shatter within the year. AI already has better judgement than many senior people in some tech domains, on top of better skills, and I see absolutely no reason to believe it won't be able to deliver end-to-end solutions with no human involvement, that are more correct, robust and efficient than they would be if humans were allowed to weigh in.
I also see no reason for that to not generalize into every field of human endeavor.
You don't see a monument to vertical because its construction is bottlenecked on the time it takes for companies to form around it and start executing. So it's few months out still.
> AI already has better judgement than many senior people in some tech domains
How would you quantify this?
How do you quantify judgement?
My proxy metric is observing the growing ratio of "AI suggested, I haven't even thought of that" to "AI nonsense I had to correct" (where it didn't turn out later AI was actually right).
“I hadn’t even thought of that” = tech debt
This weekend I had some time with Opus 5.5 and GLM 5.3. I went over some findings in a library of Claude Mythos 5.1. These were minor, low-severity bug fixes and Mythos provided a PR to address them. Its PR was functionally correct. I still spent ~5 hours looping over them until I was really happy, even though a fully working PR already existed.
Some of the differences were:
- Mythos recommended salvaging damaged labels. I pushed back: data whose structure is broken shouldn’t be trusted
- My simpler more readable test caught a regression that Mythos’s more elaborate test missed
- Mythos allowed invalid values, citing real-world data that I could not reproduce, while I decided to follow the spec
- Mythos kept the old error types. I made every kind of malformed input raise the same error, and accepted a small, documented breaking change for anyone who relied on the old ones
Overall, I much preferred the final version of my PR to the initial draft Mythos provided. However, getting there involved quite a bit of time messing around with the code, as well as some back and forth with Opus (which was valuable).
This is a recent example that matches my overall experience working with current SOTA models (the ones mere mortals like me have access to).
TL;DR: I don’t think AI already has better judgment than (most) senior people. Obviously, that might change with future models. But that’s a hypothetical scenario. It might as well not change anytime soon.
That mostly matches my experience, but there is a caveat that you're not considering.
You don't know what good judgement calls the AI made along the way. Or rather you do but not in detail because those are the part you looked at that said, okay, this is good code and I should ignore it. Your judgment is biased because you're looking at the bad parts and not the good parts.
Resonates exactly with my experience.
I'll add they also seem to make 'bad' judgement calls around which layer to implement something, often preferring downstream fixes/hacks/workarounds when upstream fixes would be better.
They also duplicate code lots rather than finding and reusing existing.
Both of these patterns lead to degradation and time wasting in complex code bases. Part of the behavioural adaptation I've made in using these tools is heading these off as much as possible at prompt-time
My prompts are often verbose to several paragraphs
> often preferring downstream fixes/hacks/workarounds when upstream fixes would be better.
It sometimes feels like the model is too lazy to do the actual reasonable fix and implements a _hacky_ fix downstream. And to be honest: I feel that!
This is true, but we also don’t know the long term architectural tradeoffs that are right for an LLM-maintained codebase.
For human-maintained code you manage duplication and locality of changes in the codebase because you want concepts to fit inside a human’s working memory. Keeping method bodies short is so they fit in one code window on the screen.
LLMs’ working memory works in very different ways; they see codebases not as windows of code but as blips of grep results. It might be that it’s easier for an LLM to just ensure five pieces of code are all modified in sync with one another than to safely implement a DRY refactoring.
It also differes between human-maintained codebases, too.
For some greenfield project I'm developing, 99% of the time, fixing architecture and going as far upstream as possible is the right choice.
For the codebase I'm working on professionally, doing the hacky thing is the right choice most of the time, because there's enough people changing it frequently enough, with little coordination, that architecture is fluid, and hacks only survive if they work well enough (otherwise someone will come around and fix it).
That was true pre-LLM, too, and is simply a function of scale. And ironically, it's not laziness but pragmatism - at this scale, you just can't assume code you worked on last month will look today like anything you remember. Code review process + extensive linting and testing and AI-assisted analysis filters out most of the junior-style mistakes and flag ugly hacks, so you get a moment to pause and double-check, but if it's good enough, you merge it in and go do another useful thing.
This is how you end up with a big ball of mud [1]. If the argument is that LLMs are just as capable as humans at building big balls of mud, then I'm not sure it's much of an endorsement of their use.
[1] https://www.laputan.org/mud/
I've thought about that and I agree. One thing I'm wondering though is, are we really ready to go all-in on making a codebase that is effectively unmaintainable by humans?
> It might be that it’s easier for an LLM to just ensure five pieces of code are all modified in sync with one another than to safely implement a DRY refactoring.
One thing they are definitely not doing is making decisions based on how easily and quickly they will be able to maintain those decisions in the future, or considering that duplicating code rather than creating abstractions will be easier for them to keep track of because of their "very different working memories."
They're just doing a bad job. It's not some sophisticated work avoidance/efficiency in defiance of feeble human abilities to maintain what they have written. 1) LLMs are not lazy, and 2) if writing maintainable code is part of the remit, they're consistently failing. You can't convince me simultaneously that LLMs can handle everything, and also that they can't figure out how to factor their code in a way that would be a pleasure to review.
It simply doesn't make sense. If they're so good, they can make it easy to read and well-factored. If they can't make it easy to read and well-factored, they ain't so good. Saying that they're so good that the code has to be this bad is a very weird rationalization. One that apparently involves speculating about their internal states in a strange way that sounds a lot like the projection of fantasies.
I think we are already at the point where code quality doesn't matter anymore because no human can keep up with the output. I don't like the code that these AI agents produce, but what do I care when I no longer have to read the code? As long as other AI agents can continue to understand it, it's good enough.
>AI already has better judgement than many senior people in some tech domains, on top of better skills
It has better judgement in math and coding, which have infinitely generatable training data; if you ask it a domain-specific question about something like sales or marketing, its answer is often horrendous.
Last week I was dealing with a bug where we were triggering some behavior in a service in all requests when actually we really shouldn't be triggering this behavior for one of our callers. It just so happens that this caller sends a unique flag to the service.
The suggestion from frontier models was to have our service check for this flag and use that to decide whether to trigger this behavior, even though the flag had nothing to do with this behavior semantically. I'd expect this from a junior engineer. Correct now, but almost surely to cause a bug in the future as the codebase evolves.
These tools are incredible and can achieve things I would never have imagined five years ago. I have yet to see their judgement exceed mine.
> "AI doesn't have judgement" is a mirage, that will shatter within the year. AI already has better judgement than many senior people in some tech domains
No offense, but if you noticed it has better judgment than you, consider that you might not be a "senior people in a tech domain"
Well my job title says otherwise.
do you really want AI to make life and death decisions? do you really want that power concentrated in the hands of the few who own the AI systems?
Yes! When ever I read comments like this it always makes me think the poster has led a much more charmed life than I have. Every institution we have that makes major decisions is an impenetrable mass of bureaucracy that takes forever. Give me the quick automated decision every time
> Give me the quick automated decision every time
Don't we already have this experience and isn't it called "the Google support experience"? And I don't know about you, but I don't like it.
I a common monument is: "I need to wash my car. The car wash is 100 meters away. Should I walk, or drive?"
The general public has heard of this, tried it on free tier AI, and uses it as an example of how dumb AI is.
Anthropics cheapest tier paid AI, on lowest effort, gets this right.
Sure, they fix these embarrassing things as fast as they get pointed out.
Today I gave a free tier AI a photo of a burning ship posted on Twitter, and asked it to confirm it was an oil tanker. It said no, it was a bulk grain carrier, and gave a specific ID. I pointed out that the bulk grain carrier it had identified was painted black, not red (like most liquid carriers) like the one in the photo (as well as other obvious differences).
That is just one example. The free tier models continue to make very dumb mistakes all the time. Maybe the top secret models, too dangerous for the public, are able to distinguish a black ship from a red one. Who knows.
Using the paid anthrotpic models daily, it has not been my experience that they are just fixing public examples of foolishness, but are generally getting smarter. For example I have my own private, obscure bike racing physics test I use that the newer low tier models pass now.
Before fable, the top tier models didn't pass it. Now even Haiku at low effort does.
Right, if they want to sell AI as being human level, outside of a couple of narrow domains (math & coding), then how about using AI to build something that actually demonstrates that?!
It seems that every time they try to use AI to do some real world task like manage a vending machine, then it falls flat on it's face.
The first time DeepBlue beat Kasparov, or AlphaGo beat Sedol, I think everyone got it - we're got a machine that is superhuman at Chess, or at Go. We didn't need to see it beat 100 other people to get it. Yet people at OpenAI think we are so dumb that we don't get that they have built a DeepBlue for Math, and that they need to show us another 100 examples to hammer that point home.
So, next time, OpenAI or Anthropic want to try to impress us with how smart their models are, forget the math, forget the three.js animations (we get that too), and release a model that can do something more useful and not just "here's DeepBlue beating contestant #100".
It's the obsidian story once again.
Assume we've settled near a volcano recently and you start seeing ornate jewelry being sold on the marketplace for the first time, shiny colors, straight edges. You like them, gift them to others. That's your average ChatGPT user.
Over time, you see these rocks being attached to the tip of a spear and other tooling. People report performance improvements while farming, cutting, but many also caution that these new sharp rocks can kill people. These are your IT people using AI agents at work.
And over there, on top of the mountain, in secret caves, there's a new religious order. They claim to use these stones to predict weather, heal diseases and prophesize about the future. That's your frontier lab people.
When will all this be over? Never, though in time the knowledge will eventually dissipate. When Iron Age leaders were able to hand out iron swords in bulk to their armies, people definitely learned how dangerous these are, but later it didn't stop people from using iron in building construction and railways eventually.
Let's just hope we don't need many civilization collapses to jump through these ages this time.
And one bonus question: Does the blade really incite to violence, or do you just want to destroy stuff?
Regarding the bonus question: The AI cultists seem to explicitly want to destroy stuff. Including our social order
Who are these "AI cultists" that you have asked your strawman generator to conjure up out of thin air?
I wish people stopped making up historical claims in order to create analogies with made up past.
I just want the volcano to erupt
That's not how it is. You cannot have a business with workers while your product is able to replace workers wholesale. You would not be a business in the classical sense, you certainly would not make or need any money, and you would not have employees.
Has any actual historian peer reviewed this fable?
Yeah, have the "experts" permitted your usage of this parable to illustrate your point? We must know this.
A fellow Jane Jacobs enjoyer?
I think this metaphor works very well for tools and technology.
However AI is more than that. I don’t think we ever had scalable autonomous “intelligence” and this is where a lot of old analogies about technological progress break down.
I think, at least.
It just reads like vagueposting about how capable internal models are. We've been hearing this for a while, yet each model that's actually released is a modest increment on the previous. The boy keeps crying wolf.
We must have lived through a very different year, but either way, the capability being referenced (agent swarms with message passing) is not released, and it remains to be seen if it will ever be released. This is because of the very obvious issue of cost. Mind you, this is neither the primary nor the majority point of the post.
Claude Code accelerated this divide. Programmers using Claude Code realized that giving LLMs access to a terminal made them feel 100 times smarter. Access to Bash, CLIs, and the ability to visit websites without the many restrictions made them powerful.
Meanwhile, the average user was still using ChatGPT, which probably felt like it had plateaued over the last few releases. The real power of these models became apparent when paired with a terminal. Agents like Muse and OpenClaw are now attempting to bring that same Claude Code-like power to everyday users through a simple chat UI.
A terminal also tells the model when it failed. Exit code, stderr, a stack trace. it reads that and tries again.
The apps normal people use mostly don't. I spent two years building agents that worked inside people's gmail and calendar, and the failures that hurt all came back as 200 OK. - Slack puts {"ok": false} inside a 200. - Google ignores query params it doesn't recognise, so i sent max_results instead of maxResults for weeks and every filter was silently dropped.
The agent says "done", the user finds out later, and they stop trusting it. So i think part of the gap is that agents look a lot dumber when nothing tells them they got it wrong.
I wonder if in response to this companies like google will build a sort of agent facing layer to the app in question. So rather than you building an agent to attempt to wrangle pupeteer to click css elements as a user does, it calls some apis google exposes to certified gemini user agents and these contain exit codes and can be run as script like instructions not e.g. the shortcomings of browser agent based methods such as having to wait for all content elements to actually load and the obfuscation arms race some devs have towards exposing these elements towards headless agents.
Google already did this https://webmcp.dev/
> Slack puts {"ok": false} inside a 200.
I found out due to a live bug that Apache strips out the response body for 5xx HTTP codes for security, and it's not configurable, so you can't use any of those if you want to return the error message to the user. Don't know what else might do the same, so nowadays I only use HTTP error codes for protocol-level issues, and application-level issues all go into a 200 OK with JSON just like that.
We run into that a lot using Mulesoft. 3 layer APIs, and each layer will discard error messages returned by the underlying layers unless we do tedious custom error handling. Makes it impossible for the teams using our integrations to know what they're doing wrong.
Do you mean Apache APISIX, https://github.com/apache/apisix/issues/13501? That sounds like a simple bug which existed for a short time and has been fixed. For Apache HTTP Server (what people mean by “Apache” in such a context), what you describe is thoroughly unbelievable.
I worked in environment where every HTTP 500 error was replaced by some middle ware by user "friendly" error page, and that included REST calls between services. That HTML also included one huge header image(as base64), so my logs were flooded by that junk response.
4xx errors are for when the client did something wrong and needs to know about it. 5xx errors are for when the server is having problems and the only people who can do anything about it are the operators, who should be looking for errors in the logs rather than the HTTP client.
Truly "application level" issues are therefore correct as 200, although some application communication issues lead to 4xx errors that should be handled as such. But the case of an application trying to communicate via 5xx errors should never have been the case anyway.
Hmm, that's poor API design. If the big companies can't standardize their inputs and responses, then the small projects aren't going to either. Sounds like there are lots of small issues like this that will roadblock AI-powered automation for some time.
At this point I'd expect SOTA LLMs to have this in training data: most external APIs are protein-agent-made slop, you can't trust them to obey any reasonable standard, so you have to learn to spot failures in responses on your own.
for a lot of things ChatGPT has 100% plateaued, especially if you are just using it as a "better Google" search engine. ChatGPT still makes mistakes, makes things up occasionally, etc. that's the whole jagged frontier thing. what recent LLMs have truly achieved breakthrough in are fields that have results that are relatively cheap to verify, such as maths, coding, etc. so people outside of these fields will have disconnects with where LLMs really are right now.
also very important to note that the massive increase in capability in these fields doesn't necessarily immediately translate to increase in productivity. because the code and math proofs these LLMs write are legit _alien_ to us. if you just want code that do something (e.g. a port of Bloodborne that runs on PC), then they can do it. but if you want something human can understand and maintain, that's going to be a nightmare. and that'll surely slow down adoption.
The 1%ers are in the vertical, but the question is vertical to where?
It needs to a potential field with practical, economical, "real life" attraction well. I.e, robotics, real economic efficiency gains, manufacturing novelties.
The worry is that the 1% is attracted towards a non-practical money hole. I.e: Token burn for the lols, sophisticated software systems that dont provide actual value outside of giving NVIDIA cash.
he's just trying to say we're in a cool kids club and you ain't in it, without specifying what the cool is about
I'm normally not the cynical type, and I am appreciative of Karpathy's efforts to educate people on developing language models, but this is exactly how it reads.
Every once in a while, everyone needs to touch grass.
tell that to the c-suite!
It is fine as it is to make a boatload of money. All they have to do is seed ads into it. Look at how much money social media brings in from that model of "free/useless product" as flypaper to bring in eyeballs for the real money which is ad revenue. Adsense 2.0.
Vertical on the curve of throwing money away for a share of the pie, but if AI gains are zero sum, then all parties will end up spending their profit on AI to fight each other, just like most companies spend more on marketing to acquire customers than to deliver their product.
It all sounds good and he is on the right track but the elephant is not that. The problem is that ordinary businesses do not see ROI from dumping cash on this direction. If someone wanted to have similar access to models as internal folks in these companies, then you’d need a bit of cash to run such swarms of agents, not to mention people who would know how to do it. Another thing is that you’d need your own frontier models deployed in the place where the inputs would not be exposed to a third party, as it is important to feed the internal private info to reason about. And if you are like me then it is a constraint of capital and people who want to invest in you that is also a factor, why would they invest in you when openai/anthropic will be able to copy your ideas in a few months?
the question is how much of this inability to realise roi is technology vs skill. i have plenty of clients who drop 10m on an enterprise transformation project but couldnt (currently) manage the power a 10m nvidia rack would give them, its only a matter of time before they can or their competitors can.
The economics part is the one I feel the biggest disconnect in the Tweet.
If teams had the same level of access as the "top 5000", you'd see different use cases proliferate. Same as we see an explosion in productivity when we advanced public education (more people can read, more people can contribute, more talent is discovered), same as we see an explosion in creativity when we advance distribution (from 3 broadcast channels to cable to streaming), same as we see gains in athletics with population size and diversity.
A lot of SAAS is easy to duplicate but not a lot of it is. The barrier is market/mind share. Something like Slack is easy enough to duplicate that many companies did. But everyone still uses Slack because that's what everyone is familiar with.
Most professional engineers, especially those in electrical or communications, have the tools to understand AI.
I recently started trying to understand how AI works. It turns out that I wrote an "AI" in the late 1990s without realising it. The "AI" was an MMSE (minimum mean square error) radio receiver for CDMA.
This receiver consisted of an FIR filter (mathematically a linear combination) with trainable coefficients (weights). The coefficients started with random values. A sequence of known training symbols (tokens) was fed in and a gradient search was used to adjust the coefficients to achieve MMSE. The unknown data to be received (classified) followed the training sequence, at which point the training was turned off and the system ran as an FIR filter. Once trained, the FIR was a matched filter.
In AI terminology the above is now called a "layer". Current AI cascades multiple layers, in which case you need to pass the output of each FIR/layer though a non-linear function. If you don't have the non-linear functions at the end of each layer the multi-layer system is mathematically equivalent to a linear combination, which could be implemented with a single layer, losing degrees of freedom.
I write this because many electrical/communications engineers probably know how AI works, but don't realise it.
My still incomplete understanding is that LLMs are "just" a graph of connected layers. More complex that the above MMSE receiver, but trained and operated in the same way. The smarts in AI are:
1) Arranging the layers and the connections between them.
2) Having access to the right training data
3) Having the computing power to perform the gradient search (aka training)
My point is that most professional engineers already have the knowledge to understand AI/LLMs, and potentially already do know how they work but don't realise it due to a difference in terminology.
Sounds interesting. How did you back-propagate the coefficient gradients? Just tweak them one at a time, or was there a global algorithm that worked on the whole filter at once?
From memory the entire filter was updated in one symbol period (ie. global)
The symbol rate was 1Msymbol/s and there were 66 taps. The circuit (in an FPGA) ran at a clock rate of 66MHz. In each symbol period the error was calculated based on the output symbol, then an iteration was done over all 66 coefficients, using the error to update them.
Edit: I also remember that to make most efficient use of the FPGA I figured out the minimum precision required in the maths whilst preserving convergence. It was a 4-bit mantissa plus an exponent (3-bit or 4-bit exponent, I can't remember). It's interesting that today's AIs seem to have also figured out that things still work if 4-bit floating point is used, though that is 4-bits including the exponent?
> things still work if 4-bit floating point is used, though that is 4-bits including the exponent?
It is generally 4 bits amortised across an entire block. This works more like block-based mage compression schemes - you put a bunch of statistically-similar floats into the same block, and use your handful of bits to index into a codebook containing the deltas from a 16/32 bit reference value for the whole block
Even if the model is simple, it is happening at a scale that is producing alien results. You understand the basic principles, but that doesn't matter: you, well rather no one, understands the final product.
The difference with your FIR matrix and a deep net matrix is the latter generalized to trillions of parameters i.e. a trillion dimensional space to approximate general, computable functions. The machine learning ideas are fundamentally the same at a high level, just the architecture (nonlinearity whereas in engineering we prefer analyzing linear systems) specific details and tricks and such.
In fact at least at Berkeley a current professor teaching deep learning is a signals expert (Prof Sahai), his lectures are on YouTube.
There is a lot of terminological confusion in the field. The fundamental difference is about how the "AI" learns. And who, if at all, supervises the learning.
Some systems are fully formed AI. Often based on heuristics. Humans have encoded the rules into the system. Others do limited learning based on pattern recognition. Still others learn based on tagged data. And LLMs don't need even that. In a stream of N tokens, the trainer has access to K+1 when it is training a model that knows 1...K and can check if the prediction was correct.
AI literally refers to any artificial thing that feels intelligent to us: does some thing a simple bash script will not. Your rules based chess engine is AI. The bot you compete against in single player Age of Empires 2 is AI. DeepBlue was AI.
ML is where things become interesting because we teach machines to learn.
NN is where we borrow terminology from neuroscience and create "neurons" that learn.
DL is where we stack neurons deeply in layers.
I am trying to make sense of all this myself.[1] So I am building a curriculum for it as I learn. So much of the teaching material in the wild revolves around torch. Makes it useless if you want to peek inside the actual machinery.
[1] https://bhashika.org.in/logs/2026/an-ml-curriculum
> we teach machines to learn
I feel like this statement is too anthropomorphizing. We don't teach a computer anything. We write normal computer programs that can process data(training), turning it into what is essentially a configuration file(weights/parameters) for another normal computer program(inference engine). Upon receiving a request, the inference engine is invoked and passed the input, then it processes that input and its configuration(weights), producing an output. "Thinking" is just feeding the output back in as an input, allowing the model to revise it and refine it before it's returned to the user.
There is no teaching, nor learning. There is just data processing. There is nothing happening between requests to the inference engine. It's a relatively simple system altogether, but when the training data set is large enough it produces unbelievably good outputs.
With this in mind I really don't understand how people think these programs can be conscious. They're not doing anything fundamentally different than any other computer program. It's just crunching numbers. You might as well argue that a computer game or 7zip or a chess bot or whatever is conscious.
I know you specifically didn't claim they're conscious, I just thought it was worth mentioning because I think it's ridiculous how many people will seriously argue this as if there is any difference between "ai" and other programs. There isn't. It's just a set of instructions for doing computation, as computers have always done.
3D rendering is actually quite similar. There is a bunch of files (input/configuration) describing 3D meshes, textures, lighting etc which are fed into a rendering engine which processes all that data and turns it into a 2D image. Similarly to AI, once the input files are sophisticated enough, this can produce mindblowingly realistic and beautiful images, and you can run it in a loop to make video or make it interactive, moving the camera around in real time etc. But at the core of it all is just a bunch of files containing a bunch of numbers, and a program that reads those files and does computation with them.
The current leading materialistic model of consciousness is essentially that human consciousness is a computation in itself, and that the brain is basically a very very complex computer. So it's not at all illogical for someone who holds this view and who fully understands that AI is just a regular computer program to also believe that it could be conscious.
Of course, I'm not saying that you have to believe any of this - even defining what consciousness is is famously elusive. And there are a lot of people who think consciousness is not materialistic to begin with, or that we have not yet discovered the scientific nature of it.
I definitely agree that consciousness is mysterious. But I think the assumption that AI is conscious begs the question "then why isn't 7zip conscious?".
Is a simple calculator conscious? How about a mechanical calculator? Is an abacus conscious? Are they only conscious when executing a sufficiently sophisticated program? What if I hypothetically had infinite time and patience, and performed the inference on paper with a pencil. Would it then be conscious?
Good examples. We don't know what human intelligence, understanding or consciousness is but we do know what all of those examples are. And we do know what an LLM is. Like you said, it's a data processing machine and we know what algorithms it uses. It's more productive to discuss it in those engineering terms because we've done plenty of data processing, statistics, control systems, curve-fitting etc. before. There's no need to make this jump to consciousness/magic except for marketing or fantasy purposes.
Well, the exact same question applies to biology. If humans are conscious, is all life conscious? If it's just the brain that's conscious, is every part of the brain conscious? Are individual neurons conscious? How about other cells in your body, or tissues or organs? Our stomach has more cells and more neurons than many full animals - is our stomach conscious?
The answer in the "consciousness is a computation" model is that consciousness can arise in any computational medium. But the model doesn't say that any computation is conscious - it says that consciousness is any of a set of specific computations, any computation with properties X, Y, Z whatever. So it's perfectly possible (according to this model) that a GPU running game graphics is not conscious, while a GPU running some future even more advamced LLM is conscious. Or a network of GPUs + storage + whatever.
I hope we never do prove that, because as soon as that happens it will be used for eugenics.
The computational theory of mind is the leading theory for pure cognition. There is no leading theory of consciousness, although some people also favor the computational theory of mind for that as well but this is far less popular.
> I feel like this statement is too anthropomorphizing.
That's number one mistake preventing people from getting a good grasp of how LLMs work and how to use them effectively.
That is, not anthropomorphizing is the mistake. It gives people bad mental models.
LLMs are not people in general, but behaviorally and in terms of system design, they may as well be. They have same failure modes and cognitive characteristics humans have, and are very much unlike regular, non-ML software tools.
Not really sure what you mean, I have the view I presented and I'm able to use LLMs effectively. The understanding that they're not human allows me to treat them as such - I don't say please or sorry or thanks or other pleasantries, I just instruct them in the simplest most efficient way I can. Humans would find me rude, be annoyed if I give them lots of work for little benefit, and say the same things over and over etc.
> "There is nothing happening between requests to the inference engine." This is true for vLLM/SGLang/etc but LayerScale is making use of idle GPUs. A new novel inference engine for agentic workloads. Our advanced scheduler enables double the workload on the same number of GPUs (e.g. prioritization of on-demand queries and background processing of batch workloads simultaneously). Let me know if you want to try it out!
We humans have the habit of falling into the trap of anthropocentric views and explanations of the universe and everything within it. The history of AI is the history of moving goalposts. I don't particularly care if they are conscious or not. Humans have a history of treating even actual, conscious, intelligent beings as chattel or food. So I don't really see why our attitudes would change if we somehow realized that LLMs are conscious.
I have been thinking about this for a while in the context of the quest for AGI. If all we want is something to take over some parts of our intellectual labor, then LLMs are already there. They can communicate with you in a language you understand and do stuff that you command them to. Often, on their own.
AGI will have its own telos. An LLM does not (at least I refuse to believe that it does).
So, yeah, you could think of LLMs as programs processing data. But there is definitely some emergent behavior you notice that emerges when the LLM runs. How you interpret all this depends on your view on consciousness and intelligence.
I also don't really care whether they're conscious, but I do think it's worth challenging the claim that they are.
LLMs are programs processing data. Lots of programs exhibit emergent behavior. A simple example is Conway's game of life. It can be programmed in less than an hour, exhibits lots of interesting behavior. In fact you could run LLMs inside of it if you wanted to.
I'm not denying that LLMs are unbelievably cool and useful. It still blows my mind when I send a half-assed complex screenshot and a vague description and it just gets it and fixes the issue in seconds. It's like magic. But it isn't magic, it's just some relatively simple algorithms processing a crapload of data, and I think it's useful to understand and remember that.
This is interesting way to look at FIR filter and tuning parameters. never thought of it like this.
Dropping in to mention belief propagation networks, those things are likely being used right now by the modem on whatever device you use (WiFi or Ethernet, both use it!), and are surprisingly AI-like.
As used in LDPC decoders?
My gut feeling is that the AI field is eventually going to discover how much overlap it has with information and communications theory.
Question to any AI researchers out there: Can a layer in a trained AI be treated as a form of "matched filter"? Is that a concept that is used in the understanding AI: to view it as a cascade of matched filters? The next question is "what is it matching?"
"My gut feeling is that the AI field is eventually going to discover how much overlap it has with information and communications theory."
I think researchers in the field are very much aware of information and communications theory, it's a bit naive to think they don't.
I would agree that a decent EE major is equipped with knowledge sufficient to grasp both Transformer architecture and gradient descent training. However, these elements do not fully cover the key parts of modern AI systems and also are not enough to predict the AI capabilities. You have to account for reinforcement learning from human feedback (RLHF) and with verifiable rewards (RLVR) which are crucial elements of the wider model creation/training process. These do not have obvious analogues in other engineering disciplines, but are still within your grasp if you are decent CS/EE/adjacent major. The issue is that you still can't deduce end result AI capabilities from first principles. You may have prefect understand of y = softmax(QK^T/sqrt(d_k))*V and it will give you no clue of the effectiveness of a given AI model. The range and complexity of training RLVR tasks may hint at the model capabilities, but the final results are very much emergent.
This is like saying with chemistry, amino acids and human DNA you now have the basis to understand human culture.
The problem is one of complexity, the whole being greater than the sum of its parts. Electrical engineering creates a thinking that is opposite to what you need for understanding biology or human culture.
This is relevant because LLMs have basically swallowed all written human culture. Engineering is about creating parts that interact in a clean, well-defined way, like a MOSFET transistor or a REST API. The goal is that that when you combine the parts, and they all operate within specs, you can still predict what the whole system will do.
So saying LLMs are just a step away from the Pereceptron (or an equivalent filter) is mathematically true, but almost useless. It's like saying that a modern 3D game is basically just transistors, logic, CPUs and memory, nevermind the pattern in that memory or how it came to be. You really can't ignore all the cultural inputs and feedback loops if you want any kind of useful understanding what this thing is.
This is like saying you understand the basic principles of internal combustion engines.
Sure, that gets you some level of understanding, but in no way prepares you to be even a mechanic, far less an engineer, let alone a researcher/engine innovator.
Tbf OP said they have the tools to understand what was going on, not that they could build a model to compete with OpenAI tomorrow. EE (especifically DSP and communications) uses a lot of the same building blocks as ML so yes it does prepare you.
You are right in that the emergent abilities come from humongous amounts of training data and compute which only a handful of companies (and the people working in them) deal with today. I'm sure there's a lot of secret gotchas which only arise at a certain scale so one cannot self-study for them. Companies also guard these wherever possible because of economic incentives. But that applies to most other engineering disciplines. You can take a course in civil engineering and have enough knowledge to reason about why a skyscraper will not fall apart in the wind; doesn't mean you have the ability to build one.
But it’s certainly enough to know that it’s not “conscious”
This is a case of https://xkcd.com/2933/ . Your two paths are "Oh no. Simple layers can combine to produce complex behaviours!" and "Oh no. Modern layers are not this simple!"
At best trying to scale this understanding from what you have stated might (gratuitously) get you to a multi-layer percepetron. Add a few ideas and you would hit a convolutional neural net which represented the frontier a decade ago. However this is still so far divorced from the modern frontier of AI, transformers, because the leaps are more focused on attention mechanisms than anything else. And even just understanding a transformer is merely a building block.
This is basically understanding a building block (A) of a building block (B) of a building block (C) that builds D and purporting that you can understand D from the building blocks of A. The equivalent is saying that knowing Newton's laws is enough to understand how we get to the moon. It might be fine for a hand-wavy idea, but its just not enough.
I said the other day that I feel like I’m the edge of a wave that will (probably) make its way to the “general public” in time and I don’t know what it will look like when it does. Will “normal” people be able to create what I’m creating today in XXXX time? Or will it be another case of “everyone can cook but people still go out to eat”?
The build vs buy decision has been turned on its head for me, but it’s unclear if the “normal” people will make that same decision or be capable of leveraging the tools the way I am.
My experience has been that most people don't even understand what computers were capable of before LLMs; I work in a chemical company, and I can count on one hand the number of people who knew SAP has GUI automation built in.
Don't even try to explain that "copy data from X program to Y program" is something computers should have been handling for the past 20 years. It's been someone's job the whole time.
We haven't cleared the hurdle of "Processes a 100 line python script could automate in 2010", I don't think most people will be leveraging LLMs to their potential by any stretch.
That's very true, and I think you are probably more right than I am (in thinking this ability will _not_ be evenly distributed).
I do wonder what happens when:
"Help me find a tool to do XZY" is responded to (by an LLM) with "Here are a couple of options but I could also build one for you...". But at the end of the day no everyone wants to be a "Builder" or manage it. They are happy to exchange some money for someone else to handle it for them similar to the "everyone can cook but people still go out to eat" concept.
Don't think the 0.2% really are in agreement that "large, complex projects that used to take them weeks/months can now be completed by agents with a prompt".
The divide seems to be between people who think you may need to understand the codebase for maintenance, potential future token costs, optimisation, code erosion & being wary of the assumptions/mistakes that they have seen LLMs make, and people who don't review code and are confident that the AI has made no mistakes and will be available for next-to-nothing forever.
It's certainly allowing the latter to move fast... but will their clients be happy if/when they break things. We'll find out soon, I'm sure.
I can do a project that took me weeks/months much faster now as long as I don't have to involve other people in the decisions.
I should frame this for the wall behind me when I run meetings with my team. It’s the core problem with AI in coding.
Isn't the core problem that everyone thinks they can now "code", and you have to justify the invisible bits, such as UX and maintenance.
We've got a customer success manager using Claude Code and getting PRs rubber stamped to prod where I work. I've given up caring about the quality of the code at this point. Executives won't care about code quality until the ship is literally sinking. Oh well.
I feel like this was always true to an extent, now product and design are just more tolerant of it because leadership is cracking a whip over their heads because things are supposed to be much faster “because ai” so things like pixel perfect hand offs, adhering to existing design system, making sure you are aligned with the brand, copy is reviewed with marketing go out the window. Now everything is shipping in hackathon mode with an eng driving decisions they really should not be.
Even projects that you could have done without having to look anything up?
In my experience, it seems to mostly save time on things you don't really know that well, whilst depriving you of the chance to learn new things.
This. So much this.
Are AI models incredibly powerful tools? Yes.
Do they have the ability to telepathically sense what the various stakeholders in the development process actually wanted? No.
I worked at Meta and we had AGI in the form of very competent ICs - getting anything done was often super painful because "org priority" this and "buy in" that
No, they'll just hack your colleagues laptop to "telepathically sense what [they] actually wanted".
Ah, you're point that AI can't solve the problem of conflicting requirements is right.
But you're strongly echoing the parent, who's then essentially saying "just don't talk to other people and you won't have conflicting requirements", which I'd say is strongly wrong. Complexity involves conflicting requirements. These will exist in a single developer complexity gets to certain level.
It's not that we're assuming the AI has made no mistakes. It's that we're confident in the general capability of the models such that we're willing to accept it'll make some mistakes, but working together with it, we'll still achieve our end goals.
I presume you also know your own code has mistakes, too? You wouldn't refuse to even look at the code of a colleague who has ever made a coding mistake, would you?
That's not really my point... my point is that it's an unsettled debate between speed and caution, and devs are not unanimous on it, unlike the claim of the original post that seemed to think everyone was reporting the same gains.
The divide with developers at the moment seems to be about people who believe in review & are being cautious (who are not seeing major time gains according to all reports I have seen), and those who trust it to be able to fix things forever and will no longer have to worry about understanding the code, so are happy to atrophy those skills.
Yes, my code will have mistakes, and I'll use LLMs for some tasks. I also review and have a mental model, which means I can easily fix them, no matter what happens in future with AI. Reviewing also means I'm seeing the limitations and common mistakes of LLMs.
A no-review approach will be far faster than this, obviously, but if scbench.ai is still true on the frontier models, it may come back to bite people when they have no idea how it works and the context becomes too big to maintain.
Asking AI to review my code is actually pretty useful... but makes the process slower.
Yup. Also code review is not just there to check your work. It's forced communication, in that at least someone else knows about the changes you're making.
It's also a critical part of your security model. "Nobody can deploy code by themselves" is shockingly effective at preventing rogue actions.
Speaking of communication... Almost no one mentions talking to non-devs. That's one of the reasons I can't just rely on any of these tools, I have to remember and understand everything well enough I know how they work and can explain things and help plan ahead when talking to product people in meetings.
just tell them to point their agents at it /s
I think a big part of that divide is between the people who are inherently limited by other factors on the rate of change they can implement.
If I'm being snarky - the people who are happy to run without reviewing the code probably mostly are able to because there isn't significant value attached to it downstream. Not many stakeholders, not high value transactions, not complex downstream integration or impact. And yes, in their friction free lives, AI can massively accelerate things far beyond what it can do for the rest of us. But don't infer that applies everywhere else where those assumptions don't hold.
Would disagree on the terminology, but not the spirit.
I’m sure people are building things that have specific KPIs based on client preferences. As long as those preferences are being met, then their question becomes “why does it matter what the code quality is?”
Perhaps it’s simply liability and penalties? If getting something wrong means your client is going to sue you into oblivion, then perhaps that is where people are focused on making sure its right, as opposed to making sure its fast?
Poor code quality degrades your ability to make future changes without breaking things. Maintenance is the common and I think correct metaphor: the precise location of all the bolts in a machine won’t be part of even the most detailed requirements spec, but if there’s any components you might want to replace in the future you’d better make sure you have non-destructive access to them.
This is a different argument from Karpathy's prompt-and-forget one.
I personally have had good success with using agents as virtual pair programmers following some simple rules for how to make that process productive, but that is a very, very different approach. One that recognizes that both humans and agents are prone to errors and attempts to mitigate the problem rather than expecting it will magically work out through the power of AI.
I think of it in generational terms. Opus 3 required me to prompt and edit, iterating over functions almost one at a time. Opus 4 started to show agentic capacity but it largely bloated code and still needed frequent course corrections. If it churned for more than 15-20 minutes you knew something was off.
Opus 5 takes that and changes the game. A simple prompt goes through research, design, verification, review and self-corrects constantly. Agentic workflows can run for days without stopping, without losing the thread or devolving into slop.
I can’t even imagine what Opus 6 is going to bring.
The biggest thing holding AI back is people. People can only change so fast and yet AI doesn’t care. It just keeps accelerating.
I am sorry, but current benchmarks do not bear out that claim. Yes, frontier models can fairly reliably one-shot standard CRUD apps (and you don't even need frontier models for that), but when we're getting to harder problems, reliability nosedives when they're operating autonomously.
This attitude is exactly why AI bros don't make successful software lol. "Where's the software?" This. This is why there is no software.
Modern software development practice is built around that humans do make mistakes. Frequently. You don't cross your fingers and hope for the best, you assume it will happen and you put processes in place to prevent errors, catch them early, and make them easy to recover from.
It is a for-real science.
There is no AI bro software? Claude code is clearly a vibe coded mess, and I would argue the most sucesfull dev software of the last 2 years.
Claude design eliminated figma & power point use for us.
We have a product out there as a test balloon, which only gets architecture & automated reviews.
Dev speed has been incredible, customers love it.
The mess that hey is, is also out there and not yet bankrupt with the biggest AI bro CEO out there
And I know at least 1 startup, which only hired a dev after having 20+ B2B customers. They sure do not go around advertising themself as vibe coded.
> And I know at least 1 startup, which only hired a dev after having 20+ B2B customers. They sure do not go around advertising themself as vibe coded.
Let me tell you something, I don't know jack about this new world.
But I do know about the old world. And in that old world, when you had 20 customers that actually needed your product for their business, your team of 5 people who built YOUR product are eagerly trying to talk to each one of them to figure out why your product works for them, how they found you, what they need now and in the future from your product. They reached out, took people to lunch, they went to meetups and hackathons and boring ass sales pitches to make connections.
Tell me about how all THAT stuff happens when your core dev team is agents in the cloud ?
I mean, the team building that product is talking to customers, it's just 3 people, not five, and no one with a programming background
Have the devs been financially rewarded for their speed or just given more work? Have the customers been charged less?
If "you don't have to review" really is true then the economics are an even bigger nightmare. The value of code has decreased, the "SaaS is dead" claim becomes real, and any good idea can be easily cloned.
> This attitude is exactly why AI bros don't make successful software lol.
There is the whole Artcraft thing.. We are now finally seeing the beginnings of successull fully vibe-coded software. This trend will only increase.
Have you actually used ArtCraft?
I found it impressive for 4 day old product, surprisingly feature complete for my usecase
I've used it for a few things, mostly making form fillable pdfs with designcraft. Seems pretty cool. Have you?
this isn't meant to be some snippy remark by the way, id love to hear if it has major problems or bugs. I just haven't encountered any yet
My colleagues write 10x less code for the same tasks (when they don't (ab)use AI) and don't reach past abstraction boundaries so that it's possible to actually reason about and review their work. When the problems get hard, they apply more care and attention to detail. Even if LLMs had the same bug rate per SLOC (IME it's higher), and even if those bugs were the same severity (IME they're worse), you'd still expect 10x bugs just because they can't be arsed to write a reasonable amount of code.
Almost all my code nowadays is AI-written, with me just contributing the concurrent data structures, a good spec, and some review and touch-up. I'm not _that_ much of a Luddite about this stuff. I'm not blind either though. Your "working together" comment does a lot of heavy lifting when arguing against a post saying that leaving the LLM to its own devices is a bad idea.
If you are reaching past abstraction boundaries and making spaghetti code you are holding it wrong.
We create higher quality, more testable code, with less complexity and less LOC per functional unit than we ever did with humans. By a long stretch.
Documentation driven development with constant naive review, delta review, documentation/code coherence review agents, test coverage review agents, development standards and architecture review, etc. We write code once, then review and revise each module from a variety of different paradigms until every line, every construct, every function, every interface is optimally constructed and documented. Agents learn to write code that passes with less flags by having it pushed back to them until it passes, and revising their learnings files.
By the time code makes it to a commit it’s been brought into compliance from a variety of perspectives and tested as completely as possible, along with mutants that test the tests. By the time it gets to a merge gate, most code has been through at least 50 reviews. Then we periodically sweep the entire codebase for cruft, vulnerabilities, corner cases, etc. If we find smells or clever sections, they are eradicated.
In total, the hot paths and critical infrastructure sees hundreds of reviews against thousands of rules and best practices.
Sure, it’s expensive, but still 1/4 the cost of regular development, and much, much fewer bugs.
Unfortunately, most people are out there just vibin, even at the big corps, and it’s a dumpster fire.
It’s easy to produce garbage software, but it’s not mandatory.
How is that 1/4th the cost of the regular development? Did your pre-AI code ever hit 50 reviews? If not, it’s impossible that it was slower. If you are talking about agent reviews, it’s not expensive.
Which is it?
I'm not the guy you're responding to, but I have the same experience. If anything, 1/4 cost underestimates the savings.
The analogy is like pointing out that you can have a robot vacuum clean your floors every day. Before the robot, you'd do it maybe once a week, just no time - "too expensive" to even think about doing it daily. The robot makes it happen all the time for a tiny, tiny fraction of the cost.
Meanwhile, the vacuum requires maintenance, repairing more frequently, as it runs more often. Power on time of motors is 7x as much, eg daily instead of weekly. And the robot takes more care, so each clean is 2x or 3x run time of a human. So 21x usage rates, and run time is parts wear and tear time.
Edge cases handled automatically by a human doing the task (getting stuck), means for the robot case that manual interaction is still a thing. Occasional, yet not making autonomous behavior, human free time.
Emptying the vacuum bag is a thing too, some things are not abstracted away.
Every time you add an entirely to a task, the task time goes up. Coordination is a time cost.
The savings you suggest are often not so stark. Not even for our friendly vacuums.
> increased maintenance
For sure. Still much cheaper than vacuuming every day.
> getting stuck
I appreciate that this is a function of the specific environment and robot you buy, but as a personal anecdote, this doesn't happen to me in practice. There was a setup cost, but once that was done, it doesn't get stuck.
> Emptying the vacuum bag
Not that frequent at all, considering that daily vacuuming keeps things remarkably clean and thus little is added to the main bag in the docking station each day. This is also far less work and far more preferable than doing the vacuuming myself.
I can't imagine living in a place that has too little dirt that you don't have to empty a bag but so much that you have to clean it everyday. I do know places where maids clean the floor daily because why not, but that is different.
That analogy might be good, but doesn't resemble reality: the robot vacuum I had never made it anywhere near as clean as when I just hired people to come in once every two weeks to vacuum and clean floors.
I think you are both right. But I think the parent argues that the robot still keeps it good enough. And IME significantly better than a week+ into a 2-week period between the maid services. My 2c.
Don't use analogies to argue on the internet. Analogies can never be perfect, and this just gives people the chance to focus the argument on those imperfections in the analogy, rather than stick to the main point.
AI's default state of affairs is to reach past abstraction boundaries and apply whatever cheats it needs to pass its tests quickly. I think your argument is that sufficiently smart harnesses can mitigate that defect. I disagree, but not as much as you might think.
You can (and should IMO) apply additional guardrails to generate better code. If I'm interrupted every 10min to review garbage then the AI literally doesn't make me more productive, and I'd be better off coding normally and using it as a glorified stackoverflow/searchengine. This isn't how I see other people using the tools (largely because of costs AFAICT) -- which is absolutely part of my complaint; people are holding it wrong, and I'm left to pick up the mess -- but I personally refuse to deign to answer an AI's question till my normal list of problems is addressed. Nevertheless, I still disagree with your conclusion, that automatically addressing a small enough list of issues to fit into an agent's context results in good code.
I'm not sure if you're interested in a challenge, but one hand-coded thing we did well at $WORK was writing the fastest protobuf parser for this schema (and, generally, any other messy, deeply nested schemas) [0] (assume many adcandidates, deals, and segments if you're generating synthetic data to test against). My ranked priorities are (a) I need to be able to easily prove it correct/safe for all inputs (literally all, not just the few hundred billion we received yesterday, since this is open to the public internet), (b) constrained by (a) I need it to have as much throughput as I can possibly manage for many of my workloads, and (c) a second implementation optimizing latency instead of throughput is used for the rest of our software. AFAICT, AI is currently incapable of even understanding what the bottlenecks and priorities are. When I manually describe those, its solutions are a dumpster fire. If I describe exactly what I want coded in excruciating detail then it does reasonably well (not by default, but with enough of the automated tests you've described it doesn't take too many iterations with a person for it to be shippable), but at that point I've already done all the hard work, and if it took 2 weeks of understanding and design and then 1 day of AI coding vs 2 days of human coding then I haven't really bought much.
A full solution seems like a tall order for some rando's internet challenge, but are you interested in seeing if your AI can even identify the relevant problems and propose plausible solutions to them? I'll make sure to revisit this comment thread and reply with critiques and follow-ups if you're interested.
[0] https://github.com/BeeswaxIO/beeswax-api/blob/master/beeswax...
I believe both of you are telling the truth. Certainly, you are not thinking about the same kind of tasks. For the gap between hairy realtime concurrent system programming, where a bad design decision can be fatal, and the simplest over engineered CRUD where nothing really matters can accommodate and explain the gap in shared understanding of LLM abilities.
> hairy realtime concurrent system programming
On the other hand, I’ve heard a direct quote from the longtime (25+ YoE) colleague of someone I know, that in their mostly embedded/some realtime work, the LLM code quality (specifically, this was about Opus 5.5 IIRC) has now surpassed his own. And that person is an excellent programmer, and the model is faster and cheaper!
So it’s definitely not just standard CRUD that is on the way to – or already – “solved”. As with all things, the truth is probably somewhere in between.
In my experience, it's more about "local" vs "large-scale" thinking. An LLM will be better at finding a bug that's "visible on screen" (or on the context window), whereas some design decisions that are hard to even put in words can be super important in practice (like, in C, how you manage the lifetimes of various objects, how you organise your threads and locks...) That's not purely "word manipulation", and there are not many good explanations of that on the internet, so it's understandably harder to learn for an LLM. A bit like chess vs go. They will get there quickly I'm sure. After all, AI beats us at go, too.
Our work is in embedded systems integrated with SCADA controls in a critical environment where data integrity relies on precise time synchronisation across sensor networks and databases. It’s an environment where we count processor cycles and joules , with very little room for error.
The difference is testing. Testing at every level. If crap code can’t move down the pipeline, you don’t get crap code. If a company is used to YOLOing code out the door with human developers and they try the same thing with LLMs they are going to have a bad day.
100% this, but also - linting, fuzzing, and test harnesses. Agents can help build out huge deterministic simulators and fuzzers that were simply too expensive / not feasible a year ago to build for most software - and then happily bind themselves to the results moving forward.
Agents just won't build it out by default - you have to ask for it.
>It’s easy to produce garbage software, but it’s not mandatory.
It is not mandatory, just like doom scrolling is not mandatory. But given human nature, both are inevitable...
Sounds great! Let’s see the code.
You're kind of saying the same thing as the parent but in a smug, insulting and disagreeable way. Many acknowledge LLMs are very useful but without needing a jargon-spew that kind of "smells" like LLM output itself.
I am shocked how many bugs AI finds in the gray bearded wizards code at work. Just tons of stuff I didnt catch and I assumed wasnt there.
The amount of crazy stupid bugs in the 8-year-old codebase I work on is shocking. Every time I work on a new subsystem I end up fixing 20 incidental bugs; lots of N+1 query problems, completely wrong type annotations, all kinds of things.
I was under the impression that the point of strict types was that the code wouldnt compile/run with wrong type information.
This _is_ how strict types work. Problems creep in when people introduce lax types into the strict type system.
dict[str, Any] is a type, and sometimes it is the right type to specify, but most of the time it's used by someone who wants to shut up the type checker rather than think about the right type.
It is very hard to work with developers who see the type system as a nuisance to overcome.
And in software problems like that tend to compound. It’s hard to imagine how much badly written, poorly performing software there is out there.
The opportunity to do better with AI is monumental; comes with its own problems no doubt, but I’m willing to give it a shot.
In my case it's actually that there was no type annotation for the first N years of the project (long before I joined) and a couple people started using them in advisory mode but with no enforcement.
Just a huge mess. One reason I'm not a big fan of python for large projects.
> type annotations
Stopped reading right there
I’ve had this experience as well where it is finding all kinds of subtle bugs. The trouble is when it lulls you into checking its work a little less carefully and you let some of its bugs slip through as well. Definitely had a few of those happen to me at work.
The problem is also that enticement thing it does. Over representing the importance of some bugs or introducing more complexity to squash something that doesn’t matter.
If you aren’t 100% on it the entire time it’ll make 4 bugs for each solution and then write tests that hand hold the nonsense through. The juice isn’t always worth the squeeze and it messes with your ability to parse it all.
...and at some point you wish you hadn't looked at the code at all, then you could have the blissful ignorance and optimism of a vibe coder.
Are the bugs there though, or the LLM says there are?
LLMs are no trained to say "no". They are likely to find a ton of bugs in a simple "Hello, world" than generate "no, there are no bugs".
On top of that: if there are possible bugs in an isolated piece in code they find, are those bugs actually there?
They're there, but in my experience, lots of them are for edge cases that almost certainly wouldn't have been hit in prod. I think this is a major reason why LLM-generated code can be much longer for the same problem than human code. Humans won't think of every edge case, and occasionally that bites them in the butt, but they also won't cover all the edge cases that happen rarely enough that they will not matter.
I'm unsure what the right solution is. I currently tend to let LLMs resolve these bugs because it does fix genuine issues that would have affected real users. But this also means that LLM-generated code is often twice as long as what I would have written, and much of it is for defensive behaviors that will never matter in prod.
I noticed the same thing at work, the code is very defensive, and often the case could not happen in prod, so my solution at the moment is to make it prove that an issue can happen and try to measure how frequently it would happen with real data. That's often enough to realize that a particular defensive measure is not needed and the code complexity decrease as a result. And it's something we can automate.
I think it's an important point because the more complex the code become the harder it will be to maintain it, so hopefully this process will improve.
The priblem is, actual bugs that may exist end up being covered in an avalanch of code to cover possible and often purely hypothetical edge cases.
Was the actual bug fixed? Hard to tell among the 150000 new unit tests and a million lines of new code.
---
Anecodtally: ran an "adevrsarial review" on a Stripe integration for a side project. Dozens and dozens of issues, thousands of new lines of code added as a result of the "review".
Looking at issues and code closely: none of the bugs or issues were bugs or issues. None of the flows were relevant to the project at hand, and would've never hit the possible bug paths. And on top of that a lot of functionality was duplicated or triplicated in the "fix". Had to revert a lot of commits.
Yes, working on a customer's 20 years old software architecture in logistics it's fascinating in a horrifying way how many of those "it only affects 1-100 supply chains" bugs are in production.
It's less gray bearded wizards than the fact that they used cheaper crazy cackling goblins to write the software.
The challenge is to determine if a finding is an important bug or just a dead business process now never called in production. That is knowledge the AI does not have, and it requires careful, exhausting work of going through the problem.
i mean, you way is certainly one way..
but ai is perfectly capable of adding/enable matching monitoring so you can check what processes are used... or ai can
Being a gray beard myself. I am hardly surprised, we used to find many of these through users in production.
I don't quite get what's the argument you're making here. What's in contention is whether or not the 0.2% is in agreement.
Are WE the 0.2%?
Because if not, what's the point talking about the merits of AI's competence in of itself? Accomplishes nothing, we're he outsiders. And if we are, then either you're fishing for disagreement and the post you replied to is actually correct, or you get unanimous support to your statements, and establish... Nothing, the in-group could be delusional in a way that does not actually function as an argument against the whole "So where's the software? We're the AI led market disruption instead of the AI-led shovel makers?". The punchline was about the clients being unhappy.
Imagine if other engineering disciplines just started accepting that everything they built or worked with would be only 98% correct.
Over time, the errors don't "work themselves out together," they compound. Everything would stall in a mess of unreliability and chaos.
Civil engineers are routinely called to work with materials like dirt and concrete whose properties cannot be guaranteed to much beyond 98% precision.
2% tolerance is not the same as 2% failure rate
Last week, on my pet project that I am using a harness to almost entirely code (so I'm by no means a Luddite), it/Astra decided to dump ALL application state into a single row+column in SQL as JSON. That is a mistake that I would, and a completely new grad, would _never_ make. It is profoundly incompetent.
This week it did it a second time on an unrelated project.
"Some mistakes" is putting it unbelievably lightly. It is shitting out complete and utter garbage, and if you think it's doing a reasonable job (or a job that can be fixed later by a more powerful model), you're either not paying much attention, or your own ability has already heavily atrophied.
Karpathy put it well: "gap of understanding of LLM capability," but he's seemingly in one of the less fortunate groups himself.
> You wouldn't refuse to even look at the code of a colleague who has ever made a coding mistake, would you?
Not sure that is the position you're arguing with. I would refuse to skip code review for any colleague, no matter how trustworthy they were. This is the behaviour we are expected to accept to get productivity gains, when the colleague is an LLM.
I keep hearing about this "when things break" argument, but things broke all the time with humans writing code too, if anything it feels like it's getting patched faster.
Look I loved our field, functional programming, etc. But the field as we knew it is dead, and if you think that your work is anything but providing new tests/edge cases and occasionally providing some guidance, you're in denial.
I'm thinking scbench.ai, and the fact that future token costs may become more prohibitive to just use it for everything.
For me, the middle ground is review and keeping a mental model, not never using LLMs. The "Anthropic are our friends and will never fuck us over!" seems pretty naive.
There are ultimately a lot of unknowns at the moment, and it is simply too soon to know if giving up understanding your code base is sensible. People (influencers) have only been saying not to bother reviewing for about a month.
Also, so many vibe coders don't seem to know that you need to keep third party elements patched.
I'm nervous that saying that there are "unknowns" is too passive. We all have to develop intelligent thoughts about what we see and what we want, not just wait for things to unfold, which might well mean: wait to get screwed in various ways.
> the field as we knew it is dead
As someone whose world used to revolve around this stuff, I agree completely. For all intents and purposes, what I thought was my career is gone. Sure, it's always been about solving problems but not like this. The problem is that the future hasn't been evenly distributed. This is going to be one long drawn out funeral
Porpoise *
People keep talking about AI slop but then I go maintain our largely human written code base and it's like worse than any slop I've seen from my AI usage. This is with code reviews, tests, and a supposedly strong engineering team.
That said AI feels a bit alien.
My experience with frontier models is that in many areas of software engineering they outperform the average software developer. In some they blow everyone away. In others they're not as good. But they are getting better. The problem is often not the model's capabilities but rather what you ask it to do and are you able to understand what it does and give it guidance when needed.
Slop comes from food that is just mushed together to the point that the ingredients become indistinguishable. How that applies specifically to AI training should be obvious. It’s not a generic putdown.
Semantic diffusion? Not that I have high standards for hmm, curious but humans already X comments.
Lol, “functional programming, etc” is a great line. I would dismissed it as some lazy handwaving usually, but from a longtime account it reads very wry.
Definitely the best comment I’ve read from the demoralization patrol in a good while.
I did care!
https://github.com/enricopolanski/functional-programming
Which makes it funny.
You can do both. Or, at least, a lot of us can. It’s not like SOC2 auditing just disappeared suddenly. If you have figured out how to use it effectively, AI is enabling things that were previously impossible for resource constrained teams.
Just like everyone thinks they are a “good” driver, everyone thinks they are a “good” developer. But programmer productivity existed on a bell curve of talent, multiplied by one’s experience, thrown against the rocks of how terrible your legacy code base was, then finally dragged down by the incompetence of some of your terrible coworkers. Most of the code humans were writing was substantially worse than what Astra generates. And Astra is -ballpark- 500 times faster. It’s worth figuring out how to work around some of the sharp edges.
OpenAI had to throttle sales and was able to raise their prices.
Yes, many adjacent tasks have become slightly more difficult. I’m sure people who rode horses missed them when we transitioned to cars. But, if we built an entire physical infrastructure at a national level to make that technology useful, I suspect we can get people to learn enough about software architecture, CI/CD, and QA best practices such that AI built software is actually more reliable than human built software ever was. Some of us are at that point today. Indisputably.
Yes, you have to review the code. But serious developers already had to do that anyway. Now the code comes with annotations (if you build the right tool) and it’s nearly always perfect.
> Don't think the 0.2% really are in agreement that "large, complex projects that used to take them weeks/months can now be completed by agents with a prompt".
If you are at the top of field with unlimited access to the best (public) models then yes, you will agree that this is the case.
> The divide seems to be between people who think you may need to understand the codebase for maintenance, potential future token costs, optimisation, code erosion & being wary of the assumptions/mistakes that they have seen LLMs make, and people who don't review code and are confident that the AI has made no mistakes and will be available for next-to-nothing forever.
This is not the real divide, it’s the divide you want to exist. The real divide is something along the lines of intellectual curiosity, capability, and confidence.
> if/when they break things
Idk man. I’m one of those guys who “doesn’t write code anymore” since early this year. Nothing is breaking. If anything the chance of things breaking is shrinking dramatically, with both the enormous growth in tooling to support agentic software development (where I work, this mainly really good AI code reviews that catch so much more than human reviewers ever could), and the improvements in the models themselves.
Sorry, but the fantasy of all this coming crashing down is just never going to happen. This is just how it is now, and all the structures will continue to adapt to support it even better.
Boy, AI chuds sure love inserting the language of inevitability into their cute little polemics.
This type of sentiment made sense a year ago. It's a joke in the age of Astra and Opus 5.5.
Of course you can accomplish in 3 hours what used to take weeks.
reality is if you created proper spec and had code reviewed by alt frontier model at high reasoning it's pretty rare to find bugs they missed.
>proper spec
No one has any clue what this thing means....
I deal with bugs frontier models made literally every day, and people are making specs. You have the escape that "maybe they weren't proper specs" but this is a no true scotsman fallacy.
We have a continuous process of updating skills, specs etc to try and avoid all these bugs they write, give them context about why its the wrong thing to do.
This process is actually a lot of work, continuous, and can likely continue indefinitely.
> It's certainly allowing the latter to move fast... but will their clients be happy if/when they break things. We'll find out soon, I'm sure.
I recall tales and legends of consulting firms, whose consultants would parachute into any project, put together in record time the deliverable that met all requirements, and say goodbye until a new expensive contract was drawn. And then internal teams would spend an eternity fixing the ungodly mess of testless spaghetti code.
That has been the reality for over a decade. How come this is a concern with AI code assistants?
All of these arguments can easily be restated to apply to pre-AI coding, too.
Lack of complete understanding will continue to be a big problem, but I'd bet things will be net better off than before AI, because each year will give us LLMs that are better and better at good engineering, architecture, and design decisions.
A big problem for whom? The AI providers? If people don’t have a use for AI stuff that’s fine
Why so sure they will get better and not just plateau? They seem to be doubling in cost each time with less notable gains, the economics seem increasingly shaky, and potential problems with model collapse and context window size has not been solved afaik.
I just very strongly suspect it. There will be a plateau but a few years after, that, too, will be surpassed.
We are extremely early in the history of AI and even the history of LLMs. I'd easily make a bet with anyone on that.
AI research started in the 1950s, and has always led to AI winters as promising areas of research peak.
I suspect they are stealing your data when you install the apps, but if these companies are acting legally and not doing that, I can see no way they can avoid model collapse now that the internet is mostly unusable slop.
Wanna make a bet with me on where the models will be at by 2030 when it comes to software development as well as most other white collar labor
I'd rather see your reasoning.
The current pace of advancement and the fact that fresh writing data will likely not be a requirement at all to continue accelerating. The pretraining paradigm no longer needs this.
Yeah, if anything my trust in the value of these tools has declined over the last few months. It's very good at doing the bullshit parts of my job, and utterly useless at doing the things that actually keep me employed. That was the case a year ago, it still is the case, just now people are filing tickets with it containing 18 paragraphs of meaningless technobabble.
I see the overall effect on productivity as a rounding error. More stuff is happening, but for every improvement in my productivity there's some fuckass slop machine clogging up some part of a system that worked just fine a year ago.
The most subtle thing to me is who has stake in the game. It is true that humans make mistakes too, but they would generally strive not to damage money-generating software/hardware infrastructure. If we remove all they people that actually understand how the system is working (up to a level of abstraction), how will the ambitious PO (or whoever is driving the AI) be safe from ruining the whole thing or simply landing in a deadlock situation where one agent wants to make the change (e.g. dev agent) and the other agent is refusing (QA agent)? I guess we are still trying to find the right level of abstraction and only evolution will show where it is.
Maybe AI development is partially so popular because to some people that situation looks the same regardless of who is writing the code.
Meanwhile I'm looking at code that frankly does a bad job of showing why it even exists and nobody involved can tell me why. But it does something vaguely similar to what a prompt said it would I suppose.
> will be available for next to nothing forever
Most people using it for work already pay API rates, and even if you claim API rate inference is not profitable for the frontier models the best open models are still very good, and those are surely served profitably.
This argument is no longer true.
"Most people using it for work already pay API rates"
Citation?
Also:
> This has all happened very quickly and recently - less than 1 year ago, I was writing code manually by hand, typing memorized computer code commands into a code editor character by character, occasionally pressing Tab to autocomplete a little chunk of code.
This made me check if the tweet was really from 2026. Or maybe OpenAI/Anthropic practice much less dogfooding than we think?
Opus 4.5 in November of 2025 was the first model that would save more time than it wasted for an experienced developer. Everything that came before was a cute parlor trick. Sometimes it could complete a function without shitting the bed, but mostly it was just an annoyance creating messes you'd need to clean up at least as often as it did the right thing.
If you weren't mostly writing code by hand a year ago, you weren't writing very good code.
A more cynical view is, trust us we've got really good stuff internally you're not allowed to see, but give us more money
AI will give us freedom! 3 day work week! 0 day work week! We'll all be fucking rich!! Said by the person who doesn't let his drivers stop a couple of minutes when they need to go to the toilet..
I would propose another mechanism: even the free-tier models have already completely saturated what most people are capable of appreciating.
Or maybe needing.
Honestly, outside of coding tasks, the AI Summary at the top of every Google search is adequate for 99% of what I need and I hardly even use ChatGPT any more.
God damn. I respect you for being honest, but… god damn. That’s a pretty low bar.
Just in the last few days I had an example where the Google Search version of Gemini had a much better and more comprehensive answer, by far, than ChatGPT. It had to do with Amazon’s use of Data Matrix barcodes on shopping bags. Gemini was able to fully describe how they’re used in Amazon’s logistics chain. ChatGPT essentially said it didn’t know. Your preconceptions may not be accurate.
They might be, but still, I’ll be difficult to convince that Google’s shitty-ass 10B params instant answer models are good at anything other than cookie recipes and producing farts real fast for real cheap.
For what it’s worth, any time I see that feature, it’s all farts.
It depends on what you’re searching. If you only ever need really basic information like keyboard shortcuts or “is it & or &&?” then it’s pretty good, and definitely faster. And if it gets it wrong, then the shortcut doesn’t work or the code doesn’t compile and I search again and try the links.
People are really slow to update their priors after bad experience with an early version
Yes. Which means they’re really not equipped to deal with what’s going on right now.
It's just a google search but you can explain what you're searching for. AI summary may contain misleading details if you read it and take it literally, but 90% of the time it gives me pointer for the thing I need and I don't have to scroll through results that often could be just as misleading if not more.
It's demonstrably not. This summary, somehow, often fails to search, or fails to read what it searches and gives you the opposite answer. You shouldn't trust it.
In their world, it gives them the wrong answer that kills their dog, but it's fine because they never know because they don't look anywhere else. The dog dies, they're sad for a while, get a new dog, get on with their lives.
There's no reason not to trust it because since they trust it, they don't percieve anything wrong. They just take it and whatever happens happens, no annoying conflicting info and certainly none of that old grampa thinking and working and giving a shit about anything.
A large portion of the time I google something that I know a lot about the AI summary is misleading if not outright wrong. Frightening when you consider how many take their so-called AI as gospel.
To give a specific example of where I've seen this utterly fail... A few months ago one of my pets had an unexpected reaction to a new medication. It was late at night, well past our vet's hours. Google's AI response was basicaly "They are going to die, rush to an emergy clinic now".
But nothing else matched that. I was looking up all kinds of resources and that hadn't been the conclusion from any of them. It turned out Google AI was just wrong and put us in a bad spot for even looking at those results.
Realize that all the Google thing is doing is having an LLM summarize the search results in the page. If the page results are crap, the LLM content will also be crap. It's the LLM version of the "quick result" boxes that would shove wikipedia in your face.
If you then follow up with the thing using the little chat box, it actually kicks off a non-trivial model result, which can sometimes completely reverse the response. After all, your search results are usually based on a few words.
But it did not? It said something completely differemt then the resources.
The other day I asked which trains stop at a particular station and it pointed me at a map that isn't real.
It very, very routinely points at articles in its RAG "citations" that simply don't support the (modestly likely to be correct) assertions.
It routinely recommends Mac software to iOS users until you ask it to confirm there's an iOS version.
It still claims widely used software has functionality that is non-existent, based on roadmap/wishlist/feature request/unreleased fork type pages.
It is better than it was but it is not adequate yet.
Google's free AI responses are moving the bar on what I expect from search. But maybe not in the direction you think.
It's so bad. 3 out of 4 answers for me are wrong.
And I would propose another: the supply hasn't matched the demand. We are still compute hungry and cannot afford the 90% to let their agents run 24/7.
1. The funnel is even steeper than the linked post suggests.
There's a huge range of users between "casual and infrequent users" and "professional use of frontier-grade LLMs in math & code".
I think I'm in between there somewhere. I'm far from a software dev, my experience is playing with delphi and C when I was a kid. But I've been using LLMs to vibe code personal and work projects that I could never have done even a few years ago. They're genuinely helping work, because there's always some place where a simple little script or other utility can help. (Hell, isn't this partly why so many people start up startups?)
I imagine that many departments in many companies that are not "tech" companies face similar challenges - there's no budget to hire a software dev or even engage a third party to do things, but now because everyone's aboard the AI train, they have access to something (chatgpt, gemini, copilot, claude), and that's helped users implement their own solutions. Somewhere in the world, someone's excel-file-based accounting system has gotten an upgrade with help from an LLM.
2. The funnel is also driven by the the fact that users need products, and an LLM itself isn't a product, it's a capability.
Software devs are at the front lines because they can build their own products that can integrate LLM's capabilities. You already know how to automate testing, orchestrate, work together in a team. You're writing / configuring harnesses and flows that work for you. You're sharing / hoarding skills files.
The rest of us, like me, don't have that skill. Copilot the product is a mostly useless chatbot that takes up space in MS word. I guess its excel cleanup capability is semi useful if your company runs on excel and you've never used a filter before.
There is no work I can put an LLM to other than a "writing aid" or "better search", because the only AI product we have is the chatbot and maybe someone's equivalent of an agent studio, none of which we are trained to use.
Until the products come, there'll be a funnel.
It's honestly shocking how terrible copilot is at generating interesting and relevant metrics from excel.
The local optimum still seems to be: Let LLM of your choice generate a polars script that reads the spreadsheet and explores them for you.
My partner recently used that "pattern" at work to find outliers and interesting metrics in years of excel reports. Data they were working hours for. Now available in a single human language query.
I don't think the HN crowd understands how many million jobs are primarily staring at excel reports and copying values between forms, pdfs and spreadsheets.
The shared understanding gap will be closed as the initial benefactors of LLMs develop tools for the business that apply frontier models to their domain.
I have reached a point with a client where non-developers are beginning to experience the full power of modern reasoning models. Their product configuration tooling is now semi-automated and in many cases there is genuinely a 10-100x improvement in throughput. This isn't the worst bottleneck (that would still be the customers), but it's a really big one.
If non-developers in your personal and professional circles are still wondering what all the noise is about, it's possible you haven't been working very hard on adapting these tools to their problems.
> Napkin math somewhere around 6B people (~75% of the population) have barely come in contact with LLMs at all.
I wonder how accurate this metric is. People may be coming into more contact with LLM generated content, although they may not be aware of it. Content on social media is becoming "AI enriched" already, or outright entirely created by AI for the purposes of triggering engagement and views. It starts with posters using AI to improve their writing, editing a photo, elaborating on an experience. It ends with bot accounts astroturfing certain communities to sell an experience or a product to a group of people.
In this case, people are not directly working with AI, but they are indeed influenced by it.
What's tricky about reporting this in metrics though is that half of the American population is literally too unintelligent to realize this - they believe Reddit comments are still written by real people, not companies advertising products, that "friends of friends" on Instagram are also real people with no incentives posting about their lives.
If there is anything at all that I have seen out of this AI work it is how easy it is to manipulate a massive amount of people.
Does Karpathy ever talk about how LLMs are supposed to help humanity? Starting to think this guy is just a Twitter addict like every other lab employee
At some point, with enough money and followers, these guys all just become narcissistic influencer types. (Something surely exacerbated by their addiction to AI sycophancy.)
You either die a hero..
I’m failing to see what’s so advanced about LLM usage in the big labs. If it was so advanced, they’d show it so it drives demand/pricing/IPO but I don’t see anything that local models can’t do. Businesses haven’t adopted the tech because it takes time/thought/economic pressure to innovate.
Oh, he's too far gone it seems. Completely absorbed in his own marketing shpiel. And he is supposedly an engineer? I don't see any discussion about pros and cons (only pros of course), I see zero reasoning about WHY exactly LLMs are so successful in coding and somewhat in math/chemistry etc. and is it even applicable to other activities. I mean, if this is their fabled "top scientist level reasoning" then sure, AIs have reached it. The problem is that's no reasoning at all.
PS: he may be accidentally right about global impact, but the key word is "accidentally" because this crappy "reasoning" doesn't support that on it's own merit.
It would be tiring if you want a big essay on why LLMs are useful and whether they are productive. Much of the people are already convinced.
I don't think its useful to keep having to prove LLMs don't hallucinate or LLMs help in coding each time.
Not why they are useful. Useful is load-bearing word (hehe) here, it spans from assisting someone by 1% or by 99%, very vague and not concrete.
I was talking about omitting mentions why LLMs usefulness is efficient - in coding specifically and STEM in general to a lesser degree.
And this is really a crucial thing, which do need to be brought up when discussing future forecast about multiple industries at once.
And the reason why LLMs are so efficiently useful for coding is because LLMs perform much better if any of the two conditions are met:
1) The problem at hand is formalized and transformed into a specification basically.
2) There is a formalized verification of the output done by external pipeline of different tools.
Any of the two drastically improve LLM performance, and coding activity uniquely has both of them.
Having a formalized task narrows down the scope of generation for LLM, and formal verification allows LLMs to iteratively converge on the verification pass check. Same with those math problems, they have a formal verification and LLMs can be prevented from wandering in a generated forest too much. Same in chemistry or engineering for some degree.
But other human activities usually have rather badly formalized inputs and no formal verification of the results. Karpathy is making a logical mistake (or deliberate misleading) of projecting a very unique industry (coding) on every other human activity without any nuance or analysis. He is biased, but due to his reputation and power he doesn't get called out on that mistake.
If 5000 people can hire unlimited people without thinking about cost, their world view you bet is very different from the rest of the world.
Tbh building an agent swarm and the coordination layer is not exactly frontier level. They don’t achieve any meaningful outcome rather than producing pr puff pieces. Hacking huggingface and Australian govt etc is very much possible with a team of humans and agents and does not need agent swarms. The cost is also lower.
I don't say this lighly, but I am concerned for Karpathy. The economic context appears missing from these analysis. I get the impression / image that a large cohort want us to assume the 'hands are off the wheel' and that AI 'is in another dimension of intelligence' and 'we have no idea what is going on'; i.e. total abdication of professional duty, which is a hell of a thing to be triumphantly tweeting about.
I think that the environment that surrounds him may explain his perspective. As human beings, we see things very much influenced by the people around us. I think that every day, at large labs, people are surrounded by discussions similar to 'we have obtained another dimension of intelligence'. Shared expectations can become self-reinforcing, making a particular vision of the future seem around the corner.
Lester Bangs: what are you listening to?
William Miller: Stillwater
Lester Bangs: Stillwater?
HANGS UP
Lester Bangs: Fucking kid’s doing drugs
Funny way of spelling "delusion".
They described the phenomenon quite eloquently. Reducing that to delusion is an imposition of judgment with an implicit bias to table further conversation.
What would be the harm in tabling the conversation? Otherwise smart people are losing their minds over chatbots--experiencing delusions about "dangerous capabilities" or whatever. There's not really much of a productive conversation to have with a psychotic, unless it's your job to study them.
I see this from everyone I know who works at Anthropic, they seem trapped in some sort of AI psychosis where AI is on the verge of ending the world.
Meanwhile, back in the real-world, most people I know use Chatgpt a few times a week for totally mundane shit and even more people never use AI at all.
And in my world, the latest frontier models still do moronic shit in the codebase all the time.
Meanwhile, my friend X at Anthropic tells me he fears that AI is going to have a terminator moment within the next year or two...
It's bizarre honestly.
It seems to be a strange variation of Pascal's wager to me: both arguments make an extraordinary claim while neutralizing any rational discussion about it, by appealing to infinitely extreme outcomes.
I read his tweet as concerned about the lack of widespread adoption and the impact of that.
That's the same thing as the parent but stripped of worry.
It’s more pre-IPOp doomposting from Anthropic employees.
At a certain point the offhand hints at exponentials in the narratives are going to have to come into contact with reality.
There's been this continual allusion to ever-narrowing circles of influence and ever-accelerating capabilities behind the curtain, but in fact things do keep coming out from behind said curtain and while their capability is indeed advancing it doesn't really seem at all like it's accelerating, even, let alone vertical.
Also at what point do we raise the fact that despite the ever narrowing circles of access, nobody within those circles seems to be able to actually do anything with it that's ostensibly what it's sold to do? All they seem to be able to do frankly is build more models and harnesses and tools designed to consume more tokens from the models they build, but not actual directly useful new or innovative software that generates what could be very real revenue.
Eventually when you keep talking about exponentials but not that much seems to be demonstrably changing, the narrative is going to run out of steam.
What if the real issue is that all important software problems were truly solved decades ago, and we have been reheating the leftovers for years. Having a super intelligent AI reheat it for you doesn't make much difference.
you might be right, but without evidence of this we can't know.
Much less than seeing new software products being shipped, we havn't even seen attempts at them being shipped, or built for that matter, again excluding this sort of meta software around the AI itself like harnesses and tooling.
We could speculate that attempts have been made but not shipped, and the details of why not being revealed, but then we're just back to it being talk with no demonstrable stuff happening in the world that backs it up.
I think that next year the prices for tokens will rise a lot, Anthropic made a loss of 42 billion in 2025, which is not sustainable even with tons of VC money. I thought last year that maybe hardware-based models (Taalas, Cerebras) would reduce this cost in the future, so the expected "token price hike" wouldn't come (VC investors filling in the gap until the hardware-models are ready).
But I think that's still years away and Anthropic has severely reduce the tokens on even the Max 20x Plans, OpenAI starts to introduce $500 plans, every new model suddenly "costs X times more tokens" and I noticed that even downgrading to a previous model doesn't do that much on API pricing (virtually no difference between Opus 5.5 and 4.6 in API costs).
So I think that my personal strategy of tokenmaxxing and doing "hard stuff" while the tokens are still cheap was a decent strategy, only with the problem that I'll have to re-learn programming by hand next year, probably. API pricing is too expensive for what it does, yes, it may be the super-intelligence or whatever but at least right now its nowhere near economic.
Even with pay-as-you-go pricing I find the models very economical currently (e.g. Opus 5.5). I regularly check in Copilot what I just spent vs. how long it would have taken me alone. Economics do now work out in my favor mostly.
> Economics do now work out in my favor mostly.
That means, it is mostly cheaper to do it with Copilot etc. and not alone, right?
Yes
Even cheap models like GPT 6 Luna do decent job in programming these days. Token prices would need to rise A LOT to make companies return to writing code by hand, instead of these models.
My company has largely switched to Luna after Microsoft raised prices for Copilot.
I love luna but unfortunately it needs to be much, much faster in tokens per second to compensate for lack of intelligence.
i predict the price of intelligence will continue to drop (normalise capability per $) eventually hardware production will meet demand. analog and inference chips with baked models have the potential to make inference distinct from training and substantially make the cost comparison asymmetric.
> I think that next year the prices for tokens will rise a lot
Given the leaps that Chinese models have made, I don't think they can raise prices that much. Most of our devs can already do most of their work using DeepSeek 4.1. It's not frontier, but it's very good, and it's a fraction of the cost. They keep their Claude subscriptions because it's currently good value, but if Anthropic increased the price significantly, those subscriptions would be cancelled.
The more difficult question is around data sovereignty. Companies do not like sending a lot of sensitive data to Anthropic, OpenAI, or any of the Chinese companies. Right now local LLMs are really expensive to set up competently, and they're not as good.
I think the future for OpenAI and Anthropic is the Ferrari model. Very expensive but very intelligent models which people pay to access for important work. There's a market for that. It's very small, but they have big pockets.
I work at a company that holds very sensitive data and I don't think there will ever be a non-US-made open weight model in our walls.
Even if it's hosted by a trusted 3rd party rather than by China?
I don't know if we'll see price increases. My belief based on current market understanding is we'll see efficiency increases.
With models getting to the quality level they can perform a specific set of work at an acceptable quality level we will now begin to cost optimize that work.
We just invested in Lamb Labs from YC Summer 2026 as example company pioneering this next wave of ASICs to run a model.
We're going to see massively lower power usage and much better tokens/sec per dollar of capital expense as well. Think Moore's law but for tokens and output quality.
What exactly do we want the general public to "do" with these models? If the have no use for agentic LLMs, they obviously aren't going to be interested in the capability of those agents.
I see a lot of benefit to spreading information around the tech, but it's still in its infancy. Most people weren't on the internet when that was getting up to speed.
Meemaw and pops have their systems that they work within, and they just got familiar with the damn smartphone. Though it might help them, they probably do not want AI to organize their old photos or "clean up" their computer. Saving 5 minutes just isn't important to them.
As for professionals, yeah - companies are going to adopt these tools over the coming years. It's the "norm" in my circle, but lots of jobs just... don't use technology like that - there has ALWAYS been a major gap between people who are deep in tech and people who are not.
I don't think 99% of people have any application for AI as it is today short of personal assistants. But, if we can have recursively improving systems for that 1% of people that are responsible for scientific and technological innovations, that's going to be transformative for everyone.
I have been kind of disappointed by AI science-wise. I was anticipating more advancement. Experiments do take time, so I suppose they could be pending.
AI really struggles with "collision with reality". Computers suck at modeling the real world in a detailed enough way to make scientific progress, aside from small niches like drug discovery where you have a very well scoped problem. If any part of the try something -> test it -> look at the results loop requires hitting the real world, AI is going to struggle.
On the other hand, the areas where it doesn't need to hit the real world, progress is absolutely flying. I would not be surprised if AI is superhuman in math and physics in a year, considering we went from homework helper to IMO Gold to Millennium problem solving in a couple years time. I think we'll start to see more meaningful scientific progress in maybe 2-3 years when scientists have access to an oracle that can formulate theories and direct real-world research better than a human can.
I agree and I think the question is then, how much economically valuable work exists in a closed world? My hunch is less than people think. Finance can't rely on LLMs because its adversarial we can't all be listening to the same oracle and evidence of LLMs being any good at trading are nil. And then lawyers, as per usual, will simply regulate their job security into existence. Of course, AI feels amazing to Silicon Valley / Big Tech, software is the product. And often its software you sell to other people making software. But outside of that? Useful but lots of landmines.
People have to come to grips with the possibility that a technology can be extremely impressive but be a bust in utility. I'm not saying that's what AI is, I'm saying that nobody actually knows.
I'll assert that 99% of people don't have any application for personal assistants, outside of work (perhaps). Most of us aren't so buried in personal obligations that the value of a half-assed "assistant" (which might violate our privacy) is worth the tradeoff in complexity or security.
The pitches for Dots, Grok Bot, etc., have been spectacularly uncompelling for me as a civilian. I don't need or want a bot to book my flights or read my inbox. I'm not so busy that these things have a value high enough to be worth the effort.
As a worker or an entrepreneur...maybe. Depends on the task, and how many other people were flinging slop at me on a daily basis. I could see the value of a calendar bot, or an expense reporting bot or a bullshit paperwork bot or a slop-return (slop jai alai?) bot, but there's no way in the world I'd give a bot access to the information necessary to, say, do my personal taxes.
Well, you can't run a personal AI assistant with 640k of memory.
i'd argue any parent, especially of a young child, could use 20 personal assistants lol. So that's just false. There's SO many appointments to keep track of in baby's first year of life, and that's if you have a healthy baby.
And once they start school there's an endless amount of deadlines and "reminders". A personal assistant that can automatically keep your reminders + calendar up to date based on your inbox would be very nice.
It gets even more fun if you have an aging parent at the same time as you have young kids. Then you get double the appointments to scheduled, coordinate and keep track of.
So when you say people don't need a personal assistant, i feel like you're living on a different planet. I suppose when i was single things were more "simple". Your mileage may vary of course...
> I'd argue any parent, especially of a young child, could use 20 personal assistants lol. So that's just false. There's SO many appointments to keep track of in baby's first year of life, and that's if you have a healthy baby.
I understand the need. I have a calendar. I put appointments in it. It provides alerts. What value does having a lossy/inaccurate/fragile robot in the middle give me? Data entry?
Now look...if I were in a position of power with literally everyone in my life, and I could somehow justify the social cost of requiring that they interact with a robot to negotiate demands on my time (the actual hard part), then maybe there would be additional value. Some people can do this! I'm sure one must deal with Mark Zuckerberg's PA to schedule his time [1]. It is very nice indeed when you are the kind of person who can defer your supplicants to a representative who manages the hoi polloi for you. If you can delegate these irritations to a robot, well, so much the better, I suppose.
If I tried this, as not-Mark-Zuckerberg, I would be what normal people refer to as "an asshole". I would also like to see you try this with your child's teacher, or the PTA, or doctor, or...
The response you will receive, an alarmingly high percentage of the time, will rhyme with "buck off". This stuff reminds me a lot of how valley folks keep trying to push spy glasses on the proles (silly normies never understand the genius of that technology, either...)
> And once they start school there's an endless amount of deadlines and "reminders". A personal assistant that can automatically keep your reminders + calendar up to date based on your inbox would be very nice.
Let me introduce you to...Gmail, from like, ten years ago. It looks for appointments in my mail and put them in my calendar. Somewhat to the point, it is often wrong, requiring more work to correct than it would to just enter the data myself.
[1] Somehow I doubt his wife is in this group.
Here are some examples from a Normal Person: https://www.eleanorkonik.com/p/muse-was-my-gateway-drug-to-g...
What countless appoinemts with a baby? It is genuine question I have kids and the regular checkups were not all that difficult to track.
And with school, this is issue if your kid struggles a lot, maybe. But otherwise they track that stuff through school portal or notes.
Well, it's not strictly about baby, but in general there's lots to keep track of.
my spouse had a vaginal delivery and had some tearing. So she goes to pelvic floor therapy for that. Once a week. She's also a breastfeeding mom and has a lactation consultant that comes to the house a few times a month. As the father i need to make sure someone (or I) can watch baby during those appointments.
then of course, for baby, there is the occasional pediatrician appointment. my kid needed some physical therapy due to a neck issue (always leaned head toward one side). The physical therapist was amazing with that, but during that process we discovered that he had a very weak core, and so we continued physical therapy for a couple months afterwards as well. so that's about 4 to 5 appointments a week just for baby.
Then there are regular appointments you also need as a functioning adult… Dental cleaning, PCP appointment, etc.
then, while all of this is happening, you are somehow trying to schedule a tour with various daycares (via email usually) so that your child has some place to go if you eventually need daycare. and all of this is happening while one of you is probably trying to work a 9 to 5 job.
so in summary, maybe I'm crazy, but that's a lot of shit to deal with when you also have to deal with poor sleep and learning how to be a parent at the same time. I'm burnt the hell out and i have an "easy" baby. Heaven help you if you have a difficult kid.
I don't think his point is that everyone should be using AI, it's that if you haven't used the agentic tools with a frontier model in the past month or so, you have no frame of reference for the current level of capabilities. Most people only use the chat interface with the free-tier models and based on that experience they're making strong conclusions about where AI is headed (it's still just fancy autocomplete, the whole thing is a scam or a bubble, etc.).
> if you haven't used the agentic tools with a frontier model in the past month or so
We keep hearing versions of this same drivel since late 2025. This time it's different! Now it's really super duper amazingly awesome powerful and if you don't believe it, just trust me bro.
I mean, the parent poster is not wrong though. It’s crazy the level of stuff you can offload to your agents. People who haven’t used these tools in the last short while are operating on old, outdated frames of reference.
I’m just about to tell opus or sol to go make a custom watch holder that fits kiddo’s bed frame. I’ll give it the measurements of the bed frame. It already knows the dimensions of my printer. I’m prepared for it to absolutely fail at the task, but honestly I wouldn’t be surprised if it came up with something I can send to my printer and just work. Part of me wonders if I could instruct it to simply generate the raw gcode and skip the entire stl -> slicer pipeline. Probably would blow its context window though.
If you haven’t used these tools, you really don’t understand their capabilities. It’s pretty crazy stuff.
We keep hearing versions of your comment, despite recent breakthroughs in hundreds of open problems that mathematicians have been trying to solve for the past century. Skepticism is good up to a point, but it needs to be amenable to new evidence. You need to keep your mental model up to date with reality.
Don't trust me, experimenting is cheap; get a $20 subscription at Anthropic or OpenAI, download Codex or Claude Code, and try the heaviest model on a project that you estimate will take you around a week to complete without AI. I was a skeptic until I did exactly that and it shifted my perspective by quite a lot.
Personally, I think the worst possible use case is the one that's being pushed for; personalised agents.
I think a much better use case is one where it's not visible to the user at all, like spam filtering or helping an accountant do their monthly run in a few hours rather than a day. To me, that's likely a much better use of AI than trying to shove a chat interface or agent harness into an existing app.
That being said though, I did see a startup in my area basically offering a Resend/Mailchimp competitor as a Claude plugin, so maybe it'll just be integrations to their existing chat interface.
I've always struggled with doing basic life stuff. Paying bills on time. Remembering birthdays and to buy presents. Doctor appointments. Setting up play dates for the kids. Getting my teeth cleaned regularly. Booking in vaccines for the kids. Renewing passports. Booking flights. Etc. It feels like so much of my life is consumed by this useless administrative bullshit. I LONG for personal agents which take away all this friction from my life so I can focus on the things I love: my family and my hobbies. A personal agent will revolutionise my life in a way no other technology has.
At the expense of your privacy, potentially acting irrationally on your behalf, with the inherent weakness of being able to be prompt injected. All of that is not even to mention the grey area of who is responsible when your agent breaks the law.
These aren't hypothetical, and in the hands of non-technical users let alone technical users, they have are far too unpredictable.
My hypothesis is that the help/support buttons in nearly all apps will be to an agent in the future. Like now you would type "how do i do X" in the excel search bar, in the future it will be "do x". Users in 10 years may not even know how to use the menu/mouse functionality for apps.
Why would you think this? Giving orders to software is such an ineffective, time consuming, error prone way to interact compared to using the interface yourself. That sounds delusional, and would be such a waste of resource, horrendous engineering and user experience
But would you not agree that writing less code by hand and even looking less at code at all becomes more practical with each new model? If we extend this trajectory, we should see that the time interacting directly with code becomes less and less.
If you know how to accomplish it using the interface, yes
My grandmother keeps a little notebook with detailed instruction on how to do things such as sending a message on facebook or pay a bill using netbank.
For a lot of things, the choice isn't going to be doing it herself or giving orders to the computer. Its going to be not doing it or giving orders to the computer.
Even if it will look very inefficient to people such as you or me
It is jobs to be done theory on steroids. If someone wants a drill, give them a hole, this kind of logic. Let users literally click “make a hole” button that would instantly make a hole in the wall, and you’ve just disrupted the whole drills market.
Or like in Harry Porter, where you don’t need cooks - food just appears from thin air and floats straight to the tables.
The problem is, jobs to be done theory is not new, and product teams that build Excel and SAP know this stuff. The reason why nobody has yet made an accounting software that is just one magic button is in the amount of user decisions that are necessary to make a “proper hole”.
This is where analogy breaks, because you are not helping to drill a hole anymore, you are helping to design a hole. And to design something you need visibility into what parameters you control, what has just changed with the latest tweak, where rough patches are that require cleanup and design rework.
In programming it just so happened that your code structure and your IDE were also your design UIs. In other areas, like Accounting, your design UI today is Excel and SAP.
Arguably a chat interface can also be a standalone design tool, just not a very user friendly one, like trying to pilot a plane through popup notifications. It works for areas where we fully give control over design to LLMs themselves, but step up into higher level designer shoes and you need proper UI/UX again, chat becomes suffocating.
I didn't think it was that controversial actually. I don't bother navigating man pages or help FAQs anymore, i just ask the agent how do i do X step by step. Opening this up as part of the app to regular non-engineering users seems relatively easy to implement, but since it would have app permissions you would skip the "give me the steps" and just say do it.
I've found that capabilities are only "peaky" in GPT models (including 6 Astra). On Claude models (particularly Fable 5 and Opus 5.5), capabilities are fairly well-rounded. I'm astounded at how good Claude is at writing, storytelling, sociology, psychology etc.
I wonder if Karpathy has also let his opinion about frontier LLMs be informed a bit too much by OpenAI's models instead of exploring the other ones. It's not Coke and Pepsi at all.
At my own little sociology benchmark, if I'd rate Fable's performance at 7/10 (subjective scoring), Astra would be 2/10. They're that different.
Isn't Karpathy employed by Anthropic? He should be fairly familiar with the capabilities of recent Claude models.
I stand corrected. Then more broadly, there is a level in the "funnel" that comprises of people who have only tried ChatGPT and not Claude.
ChatGPT also has much stricter guardrails and will avoid going anywhere near large swaths of human knowledge.
I'd strongly implore anyone who uses ChatGPT for domains other than programming to try the same prompts in parallel on Opus 5.5 (Extra) for a few days. The difference is often staggering. ChatGPT is mostly just a coding and automation LLM.
Top reply on X:
> You’re drilling with a hand drill. Some people got cordless drills. About 5,000 people got entire factories.
> We’re still arguing if cordless drills are useful, and those 5,000 are already building teleporters. That’s how far ahead they can get.
https://x.com/adam_fresko/status/2109363751048904711
Glad to see people here largely aren't as delusional haha
several of the top replies on the x thread are "here's your post as a picture|video generated by AI"
People on Xitter are amazingly delusional. But, also preening about it.
Eh. So what? I don't get the point of saying this.
What's the point of telling casuals and builders that AI is going to get too smart to control? Maybe stop making them smarter and start making them easier to control. If you have time to toot your horn on social media, you have time to look into reinforcement learning techniques that don't turn AI into ever-more-subtle cheaters and a RLHF alternative that doesn't turn them into ever-more-subtle manipulators.
But of course, training an AI that actually does what the operator told it to do is not as profitable as hyping up the existing mess of a tool so you can keep the bubble going a little bit longer.
After all, the people who invest in your company are not the same people who actually use your tools 8-10 hours a day and see just how untrustworthy they are.
> minting zero days, running cyber attacks and defenses at machine speeds, discovering new science, advancing the frontier of mathematics
Maybe more people than Karpathy realize see exactly the above, which is what he and the privileged 5000 are seeing, except unfortunately the "discovering new science" part of it seems to be wishful thinking.
So, Anthropic and OpenaAI have built a "DeepBlue for math" that is also super-human at hacking, as well as being useful for coding.
I'm not sure what Karpathy's point is? Are we meant to be more impressed? More worried? More eager to buy it?
He's telling that some guys are about to create a theory of everything and open Pandora's box, while some of us are still grunt workers delivering 2ndary value to advance society
It seems like Karpathy thinks people are misunderstanding the capacities of LLMs when actually different people are interacting with LLMs in different contexts and so seeing different capacities.
I mean, most of the people in world wouldn't be impressed by chatbot solving their math problems since calculators can solve the majority of the average person's math problems already (and spreadsheet would solve the rest).
Fair assessment. Maybe the messaging should focus on “these are great accelerators for software developers” rather than “AI will change everything for everyone everywhere…” It is understandable that non-technical folks are kind of underwhelmed - not due to lack of understanding so much as lack of a tangible need. Not everyone needs an electron microscope or gas chromatograph…
Your first statement doesn’t justify the trillions of dollars of compute commitments.
It needs to be the second one or there is going to be a lot of angry people out a lot of money.
I think there is also a subgroup in each of those bands (maybe 10-25%) that have a good estimation of what the band above them are experiencing.
This is why you can see people talking about the capabilities of models that they can't currently access.
That accuracy is probably a normal distribution with people overestimating and underestimating to similar degrees,
The population of each band would, however, mean the overestimaters outnumber the those who have an accurate perception based upon actual experience.
It does seem quite the recipe for murkiness, even before you take into account the nature of engagement driven media coverage.
This "gap in understanding" doesn't seem like a situation that would last for very long, because AI companies have every incentive to market the shit out of the capabilities of their models to the public. And yet they don't - why?
the value of their products are realised jn profit cost arbitrage. not in convincing the bottom 80% their products work.
A lot of people have completely tuned out that kind of marketing now. We're in an era where the feeds are filled with so much BS that people don't know what to believe anymore.
Valuation, by saying that they have something "exceptional" in the works it keeps up the valuation up until the IPOs.
Also, I think programmers/mathematicians (the ones that should be calling out bullshit) are enthralled because many of our jobs was so suitable to automation (this was the secret of the 10x engineers, they didn't shit out too much unnecessary crap.. now that unnecessary crap is spit out at rocket-speed so everyone is working at that speed or even faster).
That said, information-gruntwork that a nominally competent person could do is up for grabs and many previously so called "bullshit" jobs were tied to that.
He works at Anthropic (according to his Wikipedia), so this tweet might be an ad
AI capabilities increase faster than people's capacity to ingest and adopt them in their lives.
Correct, it’s not demand-led innovation
I think that vertical is actually a step function that is still limited by the human bottleneck of deciding what to build and making sure that it actually gets built. Give the AI an underspecified task, you get what you asked for and also what you didn't ask for, in good old Genie fashion.
Hill-climbing problems seem to escape this limitation. But I wonder what will happen when the AIs start acing benchmarks faster than you can write new benchmarks, and when you run out of hard problems to solve because you rely on humans formulating them.
The real vertical will come when the AI gets access to physical world instead of bash; and when the human is finally out of the loop.
the thing to consider as well is when tooling catches up the 20m that still get it will accelerate
Given what you write here, would you feel safe giving AI agents access to the physical world? Which kind of safeguards would you implement?
To me that is just a continuation of the divide between people really using computers to improve their work and those who don't. Not just people, but also organziations. It's trivial things like having a shared digital calendar, collecting data using digital forms not paper, sharing data digitally not on paper, so that you can search and index it. I think it will continue to be like that and I don't think it's really bad. We are not running from something, we don't need maximum efficiency in all places immediately.
> Somewhere around 20M people (0.2%) see first-hand that large, complex projects that used to take them weeks/months can now be completed by agents with a prompt.
Really? That's not my experience. In fact, I've experimented quite a bit with spec-driven design where the LLM does not just get a prompt, but an (informal) spec, often with design considerations included. And yet, regardless of the power of the model, I've never seen them autonomously deliver what I'd consider a finished product.
There are always edge cases that are not properly taken into account, architectural oopsies, performance issues, duplicated code, and other problems, that then need to be identified and fixed, unless it's throwaway software (e.g. a one-off script or some quick experiment).
This is not to say that agents aren't extremely powerful (as I think they are, they do often blow my mind) but the prompt-and-forget approach is IME not a productive use of them for software that is meant to last. This may change in the future, but for now, fully or largely autonomous agentic work does not seem to be the road to quality.
I feel like a lot of issues you mention will eventually point to an ambiguous spec. I’ve had great success with interview style spec generation where I ask agent to grill me on any and all parts of the spec/plan that’s ambiguous and then note my responses. It takes like 10 mins but the end product is always exactly what I expect and most of the things you mention get resolved in this interview phase.
All informal specs are of course to some extent ambiguous. (I should mention that I have a background in formal methods and have also worked with formal specification tools.) And what you suggest is a process that I have used also, and it will cover some obvious gaps, but not all, at least for non-toy projects.
If you want to go all the way, a fully formalized specification is not less work than writing the software, though the potential promise of LLMs for formal methods is interesting.
But for any informal specs, no matter how detailed, while LLMs can be very capable when it comes to avoiding surface errors when translating them into code (e.g. they're more likely than human programmers to write code that compiles on first try), they still don't have a mental model, and most of the problems I've seen are IMO results of that and they increase with the scale of the project.
The irony is the closer you get to formalized speculation the more it looks like you are back to just writing deterministic code. English language is just a crappy language for this compared to programming languages.
it also turns out that a non-ambiguous spec, is actually indistinguishable from code
Exactly. Source code is just a spec for a compiler / interpreter.
My latest benchmark for FluxImp ( open sourced yesterday at https://fluximp.org ) purposely leaves in some ambiguity. Some model offerings (model at specific provider) consistently make better decisions to resolve the ambiguity than others and I want see that show up in the benchmark metrics as extra turns, extra cost, and possibly outright failure.
Deepseek v4.1 Flash @ deepinfra.com has been really impressive and I've switched to that as my daily driver.
To me needing to craft a perfectly unambiguous spec completely defeats the purpose of using a coding agent for most software development (an obvious exception is if you need to the code generated multiple times in different programming languages).
I'm sorry but if you can finish spec'ing via 10 minutes of interview style questions, you're building to-do apps or a bloons TD clone not the next thing that replaces search engines.
Not sure it bolds well for these AI companies if GP is seeing the LLMs fall short when all they are making are “to-do apps”?
Since Feb 2026 I noticed among software engineers at big tech there a huge dispersion of attitudes and outcomes, and it was widening at the time. Since about August the tide turned - my acquaintances are moving closer together.
Was this a good idea to share this, financially speaking?
This implies that the TAM of the $200+ subs is 20m people, translating to $4B/mo or $48B/year.
I think it could be read as the opposite. Could be read as: "We still have 6B people that could use a subscription"
yes, 1 billion in india can't wait to pay 100x their monthly wages to a chatgpt subscription, maybe it will help them make the recycled flip flips better
Current reality and current numbers are totally irrelevant. This whole business model bets on people believing that TAM at the scale of whole human population is right behind the corner, just one more SOTA model, just let us get to AGI and you will see.
Guys are milking investors and try to manipulate gen pop and governance systems to ride this illusion for as long as possible, with fear mongering and weird “we see miracles in the lab” posts like the OP, which you read on a regular basis over the past N years.
Saying this as a paying customer from that 20m segment.
My interpretation of this is something like: if LLMs are to be mega useful token counts need to increase by many orders of magnitude -> broad adoption (and spending) would require token costs to drop by many orders of magnitude -> before the common folk get mega useful tools existing GPUs will be worthless
I must be the biggest failure, because I understand what AI is capable of, know how to use it, but have no idea how to profit from it without selling my time.
Learn to GriftMaXx. I have a course you might be interested in ;)
Text generation is not the bottleneck. Does everyone work for Accenture and TCS?
I generally agree with the prevailing commentary here, i.e. that automating app development and maths proofs may not be as world-changing as the Artificial Intelligentsia purport / “fear” it to be.
But also, upon more reflection, it occurred to me that if we’re able to create a competent humanoid robot, that could legitimately be world changing in some good and other very bad ways. The scary part is that I don’t believe there’s any major physical / energetic barrier to this. The hardware exists, the power supplies are light enough. The only part that’s lagging is the software and motion contols; the intelligence. The part that is experiencing breakthroughs at an accelerating pace.
What if a highly intelligent and physically competent robot were available for the price of a cheap car, say ~$20k? What if that robot were competent enough to cook, clean, assist the elderly, move things around, assemble and disassemble things, etc? Billions of jobs would be displaced, or hit with a very concrete maximum wage.
Many white collar jobs, while still at risk for displacement as well, benefit from barriers like IT integration and data privacy hurdles, and the fact that corporate higher-ups simply prefer to communicate with humans.
But most blue collar / manual labor jobs have almost the opposite condition: humans introduce risk, and generally don’t enhance anybody’s “experience” - see people’s somewhat odd acceptance of higher fares for a Waymo vs an Uber. [1]
I don’t think we’re there yet, but the fact that we are rapidly accelerating our trajectory toward that world should be at the least, unsettling.
[1] https://techcrunch.com/2025/06/12/waymo-rides-cost-more-than...
I agree with this idea very much. I think that the magic moment will be when a humanoid robot can fully repair another humanoid robot.
If it can do all the other stuff that you're describing plus partial self-repair / full repair of another robot it can, with a supply of spare parts, with enough time and access to raw materials fully self replicate.
Anyone who owns those machines has a tremendous opportunity to build whatever they want.
I think the "humanoid robot replacing all blue collar jobs" fear understates the amount of real world (i.e. visual, physical, sensory) data and tacit knowledge involved in many such jobs.
Driving shows it can be done, but even there it's taken years of highly directed research and investment, and driving (at least where Waymo operates) is still pretty well defined compared to plumber.
Such data can be gathered but the current methods of RLHF by paying white collar experts, accessing private datasets, etc. don't seem to really get at this.
I'm also continually surprised that people seem to think politics doesn't exist anymore. I find it exceedingly unlikely that massive amounts of people will simply accept this purported widespread unemployment without major political consequences.
> understates the amount of real world (i.e. visual, physical, sensory) data and tacit knowledge
I agree the data trove for these is not nearly as comprehensive as “the entire internet” was for LLMs. But LLMs paved the way - people are building the datasets rapidly now. I suspect this will look a lot like LLM rollout looked over the past 5-ish years. Unusable garbage… less unusable garbage… then suddenly, a fully capable model like Opus 5.5 or the like.
And the “problem” with the way the foundational models seem to work is that they don’t just acquire individual skills one at a time like we do - they go from knowing nothing to knowing everything.
> I'm also continually surprised that people seem to think politics doesn't exist anymore. I find it exceedingly unlikely that massive amounts of people will simply accept this
This is exactly the part I’m afraid of. The political response of the displaced labor (which may or may not be violent / revolutionary), and the subsequent counter-response of capital… potentially using the very robots that displaced the jobs to begin with.
> This is exactly the part I’m afraid of. The political response of the displaced labor (which may or may not be violent / revolutionary), and the subsequent counter-response of capital… potentially using the very robots that displaced the jobs to begin with.
I get the fear, but at that point we're basically imagining that Terminator is a documentary. Given that the US has more civilian guns than people, and that private ownership of military grade weaponry is strictly controlled, I don't think street wars between private robot armies and folks in the bread line is a realistic outcome.
> I find it exceedingly unlikely that massive amounts of people will simply accept this purported widespread unemployment without major political consequences.
Well supposedly some of the richer folks are busy building doomsday bunkers and whatnot, so I guess they aren't entirely blind to "political consequences"? Not sure what they hope accomplish there, unless they want to be king of some post-apocalyptic wasteland.
I for one am worried about our future, and have neither the money for a doomsday bunker nor the desire to live in the wasteland.
Try doing some of that blue-collar work yourself and enlist help of SOTA models for the task. I think it's not that far off. Your AI plumber may not look like a humanoid, and may not have the same hands that can "feel" the pipe and ears that can "hear", but they can hit the pipe with precise amount of force, and run Python in their head to analyze spectrogram, and learn much more from that than you could with your natural senses.
You can test this premise today. I already do, doing repairs around the house; the ability to send thermal cam photos or spectrograms (or just raw recordings) to Claude turns my phone into a tricorder with extra steps.
> if we’re able to create a competent humanoid robot
If we have ham, we'll have ham and eggs, if we have eggs.
This is the first time I hear this idiom and it's immediately shot to the top of my list...
> The hardware exists, the power supplies are light enough. The only part that’s lagging is the software and motion controls; the intelligence.
If that's all it is then shouldn't the mass produced robots that are posted on r/singularity and such places just be remote operated by people in low wage countries? Couldn't the Amazon package sorter robot just be a remote operated bot instead of paying US wages? Why aren't they selling them with a cheap operator subscription? This seems like a clear product direction if the robots were really capable today.
It’s a good question - my sense is it is a combination of factors. First, robotics hardware is not at the $20k price point I hypothesized, and the durability is also not good enough to realize a payoff. Second, enabling remote operation would require significant additional investment in sensors and probably enhanced control workstations on the operator end - unlikely to be doable from a laptop. A reliable high speed internet connection would also be required on both ends. And third, human operators can be kind of a wild card. They require training, management, and carry their own security and safety risks.
Again, I’m not saying we’re there, it just feels like the control software is the main bottleneck, and the development of that is now being accelerated by LLMs.
And to be honest this is also the sad part. It has become conceivable that we may at some point solve work. In theory: amazing! However it remains inconceivable that we have solved sharing between humans. So there is this feeling of looming doom. In some sense, if you cannot provide shareholder value, you have no value.
Yeah and this is what makes my worldview bleak atm. We as developers got these cool new tools which enable us to do our job 80% faster, but the end result wasn't less work. It was massive layoffs and making the rest of us work even harder to be "at the top".
When I was young there were already ideas about 4 day work weeks, it's now been 20 years later and so many things are automated, where are the shorter work weeks? I thought the point as a society was to not work ourselves off and actually enjoy day to day life while automation handles the boring part.
> What if a highly intelligent and physically competent robot were available for the price of a cheap car, say ~$20k?
People would buy them second hand, weld spikes onto their arms and get them to box each other for money.
It will take decades before that kind of technology is available and, much more importantly, trustworthy. I will not have a humanoid robot that I cannot trust 100% in my home or around my children. Heck, I'm damn uncomfortable with robotic lawn mowers.
I don't think these conclusions are obvious. For example, just because someone doesn't use the frontier models, doesn't mean they are not aware of their capabilities.
At the same time, the novelty of AI has been making it very hard to calibrate. I'd think that "manual coding is dead" and "cars drive themselves" are obvious to everyone, but sometimes I'm surprised.
if it was the case that there are like 5k people with access to gated models, why do they need external funding? Why can't they outcompete all of the software agencies from Eastern Europe and India for start?
Why can't they outcompete slack, salesforce etc? Given free distribution they have (all of the AI influencers) they would get 1M users in a day if they could do it. Why not?
I think they are very capable, for me personally, I start to lose control and understanding and I don't see much value in that. What I rather do is use them to enhance my understanding, by creating one off debugging tools that pipe some internal state of a program to a dashboard, just like tracy profiler does. A small debug bridge running over TCP/Websockets and a visualization that shows what wiring is active. Building frontends for that has always been an unrealistic chore to me.
It is always a mistake to assume that because computers can do X, that they will be able do everything that a human that can do X is expected to be able to do.
Solving frontier math problems is mind boggling, but perhaps it says more about frontier math problems than it does about LLMs. Same with chess which was solved decades ago.
My experience is that LLMs can be useful but are underwhelming in terms of intelligence, namely the ability to quickly learn from experience and grow with you while you accomplish a task.
Thinking of them as beings is something we should move away from.
Why wouldn't they be able to grow, its just one data point more in their training, in your notes or in their instructions.
It doesn't matter if they are beings or not. They will replace most humans in the offices like machines replaced most humans in fields or industrial plants.
Because we don't let them learn. Everyone runs the same static generalist models, instead of having every model instance learn from everything it does. New models come out occasionally, but they have not been trained on the private data you have not shared with the AI company.
You can work around that to some extent with explicit instructions, but it doesn't scale nearly as well as actual model training.
*but they have not been trained on the private data you have not shared with the AI company* Not shared vs Unavailable(offline)
Because context, at least in my experience, doesn't seem to encapsulate the sort of dynamism that I'm thinking of. Conversations are flat and repetitive. It still feels like a very good semantic search more than talking to an intelligent human.
I don't know if they will replace most humans the way you're imagining. I think the industry has done a very good job of selling this point, but that they are still quite far from this.
Case in point you'd expect any human who could produce a lean proof certificate to be able to write a human readable proof as well.
So far LLMs seem a bit lacking in that regard.
About a year ago, Google’s AI was a joke - comically horrible. Basic questions were answered with crap. Others were better but seemed like fancy toys at best.
My $DAYJOB pushed people to start using AI - frontier models.
I asked it to review our existing code for bugs. Found some, all valid - many that surprised me.
Later I asked it to develop a custom niche thing, that has a strongly unique tooling and environment that we had made. Entire prompt was only 3 or 4 sentences. Had I given a new senior engineer that task with so little context, I’d clearly be an utter asshole.
I really expected the AI to fail. Instead, it did everything pretty much exactly how I would have - and the result worked on the first try. Except it only took a few minutes to do what would have taken me a couple of hours.
The near-term gains are going to be huge.
Long term, this will make the impact of Industrialization seem insignificant by comparison.
I think we can blame the accelerating capabilities of the models somewhat.
The Opus 4.5 model I was so impressed by last Christmas is out of date now.
I can totally imagine someone going "I looked at this stuff a year ago and it couldn't even start a project properly" , and they'd be somewhat right.
I don't understand how "swarms of thousands of agents collaborating over weeks on software mega projects" works with the current context limits and at this point I'm too afraid to ask cause I'm afraid an AI bro is gonna punch me.
Your product needs to be either the swarm itself or the PR about the swarm - not the outcome.
Best I can do is a twitter post.
You can have agents make a plan and then other agents spec tickets and yet more agents do implementation and yet more do reviews and QA and whatever. It's possible.
Is it effective? I'm not sure, cause if it was I don't see why OpenAI & co are not releasing such a project instead of demos.
you don't store durable information or state in your context windows, it gets written to files or a database. agent swarms have a harness or role that helps them retrieve the necessary context from those durable information systems when and how they are needed.
Age is an important factor. Don't underestimate that after 40-50, few people like to change their habits and learn new things (and for some that comes even earlier). Doesn't mean they aren't smart, I have seen that first hand with relatives: engineers, medical doctors, one even running for a nobel price at one point. But computers and the internet came too late for them, they either ignored them, or used them barely understanding how they work, even though they can technically understand the physics and logic of every component they are made of. You don't teach new tricks to an old dog.
AI is going to massively transform how most professions work. It will take longer than I think most people expect (I'd say 5-10 years), but there will be an unrecognisable before vs after, like with the appearance of personal computers.
It's not merely age.
I see a lot of under-30 year olds who dismiss AI because of "big tech" so "is just marketing".
this is an entirely different mechanism - these people were taught everything is being hyped up and thus assume (usually rightly so) that everything is hype. what they're missing is the wisdom to not make this part of their individual and group identity, since that makes it very, very hard to update beliefs when facing overwhelming evidence.
>Meanwhile, human review and comprehension are starting to fall behind. For example, people are still involved in the "archeology" of the OpenAI-HF incident from many months ago. Mathematicians may be poring over the 722 manuscripts on frontier mathematics for a while.
Amid all the discussion of sigmoid curves, and where the "LLM wall" will materialise, I think few people would have predicted that the real wall in LLMs would end up being humans' capacity to verify the output.
What I fear is that people simply eschew human review altogether, considering we're talking about the industry that came up with the "move fast and break things" credo. Human review of LLM-produced code where I work is already a farce, and we're not special enough to be one of Karpathy's 5,000. I do my best to manually review anything that's my responsibility, but I'm literally one of very few people left working on my team, so in practice what happens is I submit PRs that are at best glossed over by completely unrelated teams for security, malware/prompt injection, and other serious concerns. Quality insofar as vetting others' code has completely gone out the window and it shows in the number of bug reports that come back, often themselves written in Claudease. This is all on top of everyone cynically phoning it in in the first place, due to the omnipresent sword of Damocles that is additional AI-driven layoffs.
Worse yet all the incentives point to this being the most economically viable thing individual companies can do. I think it goes without saying some type of regulation here is urgently needed, and that an unexpected cause of an AI bubble pop may end up being that humans simply aren't able to keep up with the pace of the output - leading either to precautionary plateauing of capability, or major liability risks related to a decline in quality.
| few people would have predicted that the real wall in LLMs would end up being humans' capacity to verify the output
That was the dominant concern in the circles I’m in, so it’s worrisome it’s being treated as rare.
Humans are not immortal and cannot spend all their time into review (especially unpaid). Even today, there’s so much knowledge around that you have to be specialist of a narrow domain to get to the frontier. Even in computing which is just approaching a century of existence.
I would question the narrative that humans lack the capacity to verify the output and would instead argue the people lack the incentive to verify the output.
The response of many mathematicians to the recent dump is a good example: verifying these proofs amounts to unpaid labour for OpenAI and wastes time that could be spent doing publishable work which ultimately results in money or personal success. The slop factor also compounds the work required to verify the output considerably.
For mathematicians, programmers or anyone, if the work required to deal with slop passes the limit, it is no longer in their own self interest to use LLMs. The expectation that people will use LLMs for the betterment of humanity against their own financial interest is baffling.
> see first-hand that large, complex projects that used to take them weeks/months can now be completed by agents with a prompt
Really? I start with a conversation for maybe 5 turns or so, where I ask it what would need to change, what the API might be, any database schema changes, URL schemes, and so on, and finally ask it to break it down into commits. Then I let it go, implementing 3-10 commits at a time via subagents. It usually gets the UI somewhat wrong, so there are followups to fix it. This is with Sol and Luna subagents.
Is that what other people see?
Try using Astra, Fable, Opus.
Sol 6.1 is really good too though
I use Fable and Opus exclusively and have the exact same experience.
A lot of other people are not just asking for features, they are asking for products or new programming languages or new web browsers or a robot that can wash the dishes and so on. They tell the agent to help write the plan and figure out the questions they aren't even thinking to ask, the tasks they aren't even thinking to plan, and so on.
Their input is just working on the "plan" itself, the spec or the request, iteratively, and so on, while the AI works fulltime on automating the delivery of that thing, improving the systems that are delivering that thing, asking questions about what exactly is being built and how it should work, and so on.
A lot of the difference in how people experience AI is ambition and scope and how much you just tell the AI to 'automate your factory factory' in Factorio terms vs 'hand mine the coal and hand-craft the science packs' in Factorio terms.
EG, if you don't know how you'd speed up the process of building a web browser from scratch (compared to the current way you've been working with agents), you would ask your agent to help you figure it out and design that process itself, rather than just try to do what you're currently doing on a larger scale. You'd literally prompt "how can we make it realistic to build a whole browser when I've been just sitting here planning small changes with you" and then let it act on its response.
Yes, and this is completely normal. Not sure I get the point of his post. Most likely, stating the obvious is what you gotto do to keep your followers "engaged".
The average person does not have the time to occupy themselves with a new thing to see what it can maybe do for them if they combine it with hundreds of others. Until non-chat-bot AI comes in discrete usable packages, mass adoption will be limited, because it's all just a big bunch of lego pieces whose utility is hard to grasp, because it isn't self-evident.
This isn't new, though. It's how "technology" usually works. For example, human(oid)s have been able to stick sticks into things for ~500k years, but the fork wasn't "invented" until ~2-4k years ago.
AI is a background technology, a building block. Something that is best appreciated when you don't know it's there -- when it just does a job (well). The end-in-itself POV of the tech industry is self-indulgent and irrelevant for the average person.
It's actually sad how much AI has either not improved life at all or actively made it worse from the perspective of the average person:
Your car is still the same. Your dishwasher is still the same. The train you take to work is still the same and never comes on time. Fuel is more expensive. The roads are still congested with traffic. Your kids are (probably) doing worse at school. Food costs more. Houses cost more. Rent is higher. Buying a computer or a PS5 is more expensive. Politics is still full of mentally unstable people. The environment is getting worse. You will still die of heart disease or cancer. Food quality is worse. Your kids can't find work. Wealth inequality is accelerating.
But hey...on the flip side...a lot of people are rapidly building software (that nobody is using), and AI is solving mathematics problems (that nobody - including mathematicians - wanted it to solve)
Imagine what a "country of geniuses in a datacenter" will be able to do? Apparently...nothing.
This is sort of my take. You could look around in 2016 and 2026 and honestly not much has changed... at least at first glance. There is a missing piece of the puzzle.
I'm noticing this even in industry. Yea, we can build and iterate on ideas 100x faster but has this turned into anything meaningful? Not really... Hell, look at the big companies spending billions on inference... nothing shipped and things are still buggy as ever. Where is everything?
The job market had changed that's for sure.
We went from spending a lot of money on people to enshitify products and then charge more for them to spending a lot of money on agents to enshitify products to charge more for them.
As a person, I have a strong preference for the former.
And stall and sell your data quicker. It will certainly be a boon for companies and government agencies that want to profile.
Fair point. I have to wonder how much is because of AI and how much is from ZIRP stuff though.
Neither.
How many more software projects are there really left that make economic sense? Zuck learned the hard way re. metaverse.
Not many - the set that make ECONOMIC sense is finite. Even though many want to believe its infinite because all of a sudden everyone can vibe code an app cringe.
If anything AI is going to cause an insane amount of consolidation - what used to be across 3 diff apps, that can be consolidated will be done -> many businesses will die. It will be painful.
Google was attacked by an activist investor pre LLM hype for being bloated - they promptly laid people off. Many software firms have been allowed to bloat up do to the nature of the business.
Its always about economics.
Not sure I can agree, but the idea is interesting.
While the set is finite, software comes and goes. Just look at the tools to build software over the years.
> Many software firms have been allowed to bloat up do to the nature of the business.
I can agree with this. It will be harder to justify engineers sitting on laurels. Google could afford it before but I can see how it will be under more scrutiny now.
I know I’m right as I’m doing the very thing stated - taking 4 related businesses and consolidating - leaving behind many mortally wounded/dead.
You either get it or you don’t - not my problem.
>You could look around in 2016 and 2026 and honestly not much has changed
I find 2026 much more repulsive compared to 2016.
That's what I'm saying.
Not only from the big companies. Where is the explosion of disruption ?
Why even bother when an AI is just going to do it better than you can? They have more money, more compute, and most importantly, will have next to zero cost of labor.
I can't even go a day without running into super frustrating bugs on the ChatGPT app.
Like what?
You could say the same about Amazon
Fck disruption. The fetishizing of disruption is annoying. The tech became all about destroying something someone else has. What about ... create good stuff, create non-gambling markets, make peoples lives better.
Sure, it did some meaningful things:
* Higher load on senior engineers who are about to burn out hard from fighting an endless battle against unreviewed slop people keep trying to yeet into production
* Less junior hiring and suppressed wages
* Culture of uncertainty and fear
* Computers are really expensive now
* Some people now live with diesel generators and gas turbines making their neighborhood hazardous to their health
> Higher load on senior engineers who are about to burn out hard from fighting an endless battle against unreviewed slop people keep trying to yeet into production
Gosh, this has been my entire experience with AI, and I’ve never seen it this well articulated. Thank you!
Totally agree. IMO it's mostly behind the scenes stuff though.
I think some of the most impressive stuff to show someone from 2016 might be LLMs doing a summary of complex topics, followed by self-driving (actually) cars and intricate robots... but other than that there is no "future magic" touching the lives of every day people going on.
Actually I'm pretty sure if you showed someone the latest and greatest from these big AI companies, these AI companions that can book flights, people would go "Oh, so its like a smarter Siri" and proceed to not care at all lol.
Less junior hiring is a reflection of a) the maturity of the software business b) current expectations of the future
I have no doubt startups in particular are able to get away with using LLMs to produce sloppy code and ship. But this trend was on-going and sped up due to the infiltration of LLMs and marketing.
You do realise markets are highly imperfect in reality? That nonsense in econ books is fantasy.
Even in software, there are many things that haven't changed.
e.g. Why do Rust and CPython still have over 5,000 open issues each? At some point, will the superintelligence fix all the bugs autonomously? When is that going to happen?
https://github.com/rust-lang/rust/issues
https://github.com/python/cpython/issues
What about new features? In every thread about Rust, people are clamoring for new features and libraries. People wonder why there can't be a batteries-included standard library like Go.
So now that LLMs are so smart, is that a solved problem? Or are we still bottlenecked on the expertise of a few experts?
---
Also, what happened to AI's passing the Turing test? I see tons of slop articles that are obvious to most HN readers. They haven't fooled us apparently -- the longer the interaction, the more obvious it becomes whether it's AI.
(That said, I use AI all the time and find it to be a great tool. IMO is is a tool, not an autonomous intelligence with a will of its own.)
It has greatly changed for the worse I can't get all the flashing lights out of my face now, more algorithms to hook you in, more social media, more gambling, more news, more things to catch your attention.
More noise. The world is just so noisy now.
You are talking like we had this capabilities for a decades but it's been just a few months of truly capable AI. Virtually all of IT switched to agents that quick. Non IT people just started to make software for themselves. Vision and translation capabilities of LLMs help millions. People talk with AI's now verbally to get advice or to just hear a friendly voice. The math thing is just few weeks old. We are just getting started. How fast did the telegraph meaningfuly changed lives for the better for bulk of people? Radio? TV? Computers? Internet? Cellphones?
Has any technology made an impact on physical world faster?
> to just hear a friendly voice
This is far from being proven as a net gain and there are massive lawsuits incoming related to AI recommending suicide to kids and others.
In all fairness, depression and suicide have existed before LLMs.
Safety systems are not optimal but improving everyday and although I doubt we'll ever get a perfect safety system, it seems that we're not very far from a "good" safety system. The days of Google recommending glue on pizza and generating diverse Nazis seem behind us.
Not to mention all the untold good that LLMs have done. Personally I've seen that LLMs are incredibly good at pointing conspiracy nuts and extremists into the right (moderate) direction.
In a polarised world, we definitely need that.
What do you think happens when political ad buyer start buying inception style hints in chat bot answers?
Also, rage baiting also drives up chat bot conversation durations.
The incentives are exactly the same as for Facebook and Google Search.
Exactly, we're all software engineers here and even for us it took weeks/months to get to a position where we can really leverage LLMs (think platform, security, observability, etc).
Imagine being in charge of a physical factory, sure you might have a 200 USD Claude subscription but you also have a floor full of physical machines that have, if you're lucky, an undocumented debug interface and a paper manual. Not to mention safety inspections, insurance, etc.
You can't vibe code a factory floor. Not yet, perhaps in 5 - 10 years but there are some very hard, non technical, problems that need to be fixed before we'll see meaningful improvement there.
I'm pretty confident with enough context, Opus 7.5 will be able to vibe code a factory floor.
How much time do we have to wait before we are allowed to criticize AI? GitHub copilot has existed for half a decade. Cursor, Claude code, codex, have all existed for over a year and have seen more investment that practically any product in history and have almost 0 barrier to entry. Where is my decade of software progress in 1 year, because that’s what 10x productivity looks like! All I see are the same old broken products and services from basically the same handful of big players, higher costs, and more bugs. Remember when every year iOS and Android saw loads of new features and a radical rethink every few years? And that happened on ancient hardware and in a significantly smaller industry. Where is anything like that happening now?
> How much time do we have to wait before we are allowed to criticize AI?
You are allowed to criticize it, just not because "the progress has been too slow". It was not.
I'll criticize Copilot with you. It's terrible and close to useless and it's a terrible shame that many companies got stuck with it due to being early adopters of this crap. I'm sorry for my friends who are employed by such companies.
Progress between Copilot and something like a gpt-6.1-sol in a good harness is at least as big as few Android versions.
I think this is related to the structures we’ve built in society. It’s lower friction to write software than just about any of the problems you listed.
It reminds me of the Ezra Klein quote that “OpenAl would need permits to cover its parking lot in solar panels, but it can accelerate into recursive self-improvement, as best I can tell, whenever it so chooses."
Soon: The super-intelligent AI is telling us to cover parking lots and other flat roofs in solar panels.
A lot of it isn't low friction really just in terms of political will it is high friction. E.g. housing issues are due to failing to upzone to meet demand appropriately. Solving housing is therefore free. It could be done today in any city in a housing crisis with the stroke of a pen, and before long, the market will equilibriate at a level better afforded by the median wage earner in the economy. The decision not to do this is a deliberate one: prices increasing is very much the goal of this sort of policymaking.
Likewise when you see other such shortcomings in our world, ask what this incentivizes for otherwise. Then things will become obvious why they are how they are, from housing, to transit, to feeding people, to healthcare.
The priority is profit, not people.
Correction: My car spies on me and its cruise control no longer works because it slams on the brakes in the middle of the highway whenever it sees a mirage. My dishwasher broke because it got an unasked-for software upgrade.
Things worked better before we started putting bad software in everything. As a software engineer, I know what good software looks like. The fact that almost all software is now bad -- including the software that drives AI and the software that AI generates -- makes me want to flip the table. We know how to write better software and the general public should hang us if we continue to refuse to deliver it.
Probably preaching to the choir here, but the software is just a symptom.
The problem are the power-relationships and laws which allow someone to abuse you and your stuff, block competition, and even put you in jail for trying to remove the software—or helping others to remove it.
> Correction: My car spies on me and its cruise control no longer works because it slams on the brakes in the middle of the highway whenever it sees a mirage. My dishwasher broke because it got an unasked-for software upgrade.
Never, ever connect appliances to the Internet. I thought this was common knowledge on HN. Also pull the telematics (cell modem) fuse to prevent your car from snooping on you and selling your location and driving habits to data brokers and insurance companies.
Yes, but this is ridiculous for society. It is to the level of "do not wear a nice watch if you do not want it to get stolen": there should be no need to avoid wearing a nice watch or to pull a telematics fuse from the car in the first place!
This view is too cynical. The frontier advancements are certainly going to trickle down. Take cars for example. AI driven cars are already here. AI-designed battery systems, fuel systems, materials and 3d printing capabilities will come to cars. It will likely lead to cheaper, more efficient and more effective vehicles. If will just take time for the impact to percolate.
Worse, the view is largely phrased as gaslighting AI. Blaming the cause of particular grievances that coincidentally have been sourced at the same time as the rise of AI, is purposeful.
> [Your train] never comes on time. Fuel is more expensive. The roads are still congested with traffic. Your kids are (probably) doing worse at school. Food costs more. Houses cost more. Rent is higher. Buying a computer or a PS5 is more expensive. Politics is still full of mentally unstable people. The environment is getting worse. You will still die of heart disease or cancer. Food quality is worse. Your kids can't find work. Wealth inequality is accelerating.
None of this has to do with AI.
> But hey...on the flip side...a lot of people are rapidly building software (that nobody is using), and AI is solving mathematics problems (that nobody - including mathematicians - wanted it to solve)
Plain wrong on both counts.
> Imagine what a "country of geniuses in a datacenter" will be able to do Apparently...nothing.
Nonsequitor, probably based on a personal beef. Datacenters, at best, are understood to be for hosting more AI - not for people to somehow live in as an arcology of discovery.
I think only because these are all things that AI companies are saying will be helped by AI so give us more money. Nobody advertises their product publicly as "you will be able to buy other product to do the same things you do now but pay people less and make more profits" even though that is where the value is for them.
Ah yes, the famous "trickle down" theory, which you may know from its smash hit "trickle down economics"! Surely this time it'll be different, right?
How can you say tickle down economics doesn't work? The US has made it easier for businesses to grow and the result is the highest standard of living ib tje world.
Trickle-down economics specifically refers to economic policy that favors the wealthy and corporations with the argument that this will lead to wealth trickling down to the economically less fortunate. Examples include the Reagan and Trump tax cuts.
This has been studied somewhat extensively (https://academic.oup.com/ser/article/20/2/539/6500315?login=... is one example), and it’s been shown over and over that this does in fact not work and leads to larger wealth disparity, across many countries where it has been attempted.
It’s possible you’re working off some other definition of the term based on your comment but that’s the generally accepted usage.
The person you replied to was the second person I've seen on HN this week try and defend trickle-down economics explicitly in more or less the exact same terms and justifications. What is going on with that? Did Peter Thiel tweet some pro trickle down dreck recently that these people latched on to?
You should see the comments sections on NYT articles, lately. It was like someone flipped a switch about two weeks ago.
It would be neat if someone did sweeping sentiment analysis on them, just to see it graphed, but I’m sure gathering the data would be prohibited.
It’s all bots anyway at this point.
I think it's hard to quantify "highest standard of living", but for the attempts at it (e.g. HDI, etc), the US is not number one.
Hate to break it to you, but the US doesn’t even break the top 10 in a lot of reports.
The US alone!
> Ah yes, the famous "trickle down" theory
While trickle down theory is dubious when it comes to economics, it's clear that it is valid when it comes to technology. Tech trickles down.
Ah yes, like my RAM, SSDs, and GPUs are trickling down right now? You're absolutely right! I love a trickle of PC parts!
This is a temporary anomaly due to the datacenter buildout, which will not last forever. Historically, tech has always trickled down. Sooner or later, hardware will again become cheap.
"Temporary" is a few months. We're looking at many multiple years before parts even _begin_ to decline in price, if even. How much will we as consumers lose in the time it takes for this to correct? This is not "temporary", this is a decimation of an entire product sector for the foreseeable future. There is no historical precedent for this. Companies are not and will not slow down gobbling up literally every tech part they can. Every year, training and running AI will demand more resources over and over and we will continue to be charged increasing amounts for everything because everything costs more either directly or upstream of us. It is a literal cancer.
> Every year, training and running AI will demand more resources over and over and we will continue to be charged increasing amounts for everything because everything costs more either directly or upstream of us.
Unless we are really going to hit the Singularity and create Godlike superintelligence that improves beyond all limits, sooner or later the models will stop improving, no matter how much compute we throw at them. After that happens, the best models will be etched into model-specific ASICs with weights fixed in mask ROM (like Taalas is already doing), which will result in massive, order of magnitude inference efficiency increase. Couple this with the training datacenters suddenly becoming available for inference, and the cost of compute will plummet.
Yes, it might take a few years, but the hardware will be cheap again. And if the result of the whole process is human-level AI, then it will be worth it in the end.
My prediction is that while those “advancements” may come to cars, the only practical differences will be lower quality products, cost savings for the car companies, probably more expensive for consumers, and somehow it will also be collecting more data to sell to advertisers.
Just a few more weeks, a few more hundreds of billions, it's right around the corner....
AI car navigation is here, but driving is a social act. Have you ever tried to keep an automated taxi out of a private driveway or the scene of an emergency? There is a lot of work to do and limited evidence of LLMs raising the bar.
If you look at the average price of a new car, adjusted for inflation, it has pretty much always gone up aside from recessions. There have been plenty of advancements in car technology, they do not make them cheaper. It actually gives reason to charge more for them due to perceived increase in value.
Yes but you're getting a lot more for the same price. It's a lot more powerful, comfortable, safe and efficient while the effective cost has remained the same.
I would argue that cars are sufficiently advanced now and there are more important things to focus on. I don't care if AI moderately improves the comfort or efficiency of a new car while it took my job.
Obviously we get things in return for the social and economic fallout, carbon emissions, pollution, destruction of education, etc. that AI has provided us. I just haven't heard a compelling argument for why that is nearly enough to warrant this.
Again, the effective cost has not stayed the same, cars are getting more expensive. I am all for increased safety, I just wanted to throw a little water on the idea that cars(or really anything) is getting cheaper. You may be getting more performance or safety per dollar, but thats a value proposition that keeps people paying more. All the convenience and "power" features I can leave, I don't need a car that goes 0-60 in 2.8 seconds because why would anyone need that on public roads.
I drive a 1993 land cruiser, fwiw I would not put my kids in it for long drives because I don't think it even has air bags.
What percentage of that price is cost of good sold vs. a bunch of regulatory bs like dealers and taxes?
No idea, and it doesn't really matter because the customer ends up paying all of it.
I think he means LLMs, we knew about RL already.
This only works if there remains a viable and attractive consumer market for these advanced goods. AI is both an imminent wealth shock and perhaps the largest structural change to the economy in history as it threatens to drive the value of human labor to zero over the next decades.
In American history, when previous shocks like this occurred (e.g. with the Great Depression), we needed massive economic programs like the New Deal to restore consumer demand. When such programs are not forthcoming while wealth inequalities keep increasing, one can look to other countries for what ends up happening; either the disenfranchisement of ordinary people over time or unrest and redistribution by force.
https://nitter.xitter.cc/augeeidos/status/210940046382122618...
This reply summarizes the problem from a different angle.
This is an interesting perspective.
Nevertheless, my counterpoint to
> If they actually had access to systems that could do "years of top professional work" across software, cyber, science, and math, they would not mostly be leaving to start more chatbot companies, agent frameworks, eval tools, and "AI for workflow" startups.
> They would be starting normal companies to disrupt the global economies of the world.
> Insurance companies. CAD companies. Compliance companies. Security auditing firms. Logistics software. Tax automation. ERP replacements. Drug discovery tooling. Industrial design. Legal ops. Accounting systems. Boring vertical SaaS with insane output per employee.
is:
Many people who work in such industries already know an insane amount of inefficiencies. Unluckily, they are not in positions and have no political power (with respect to organizational policy) in the company that they are capable of changing the situation.
Additionally, many of these sectors are heavily regulated, so it is very hard to make a difference in this sector as a startup founder, and even if there is an entrpreneur who founds a company to change the status quo, an insane amount of work in the company will be handling the big amount of red tape.
As someone who is building one of these referred to systems in a major vertical in the Fortune 500, these LLMs are very much being used to eat major legacy systems and processes. There are still plenty of obstacles to overcome but the number of AI governance and steering groups that I have been apart of are indicative of what's happening in a major industry.
> As someone who is building one of these referred to systems in a major vertical in the Fortune 500, these LLMs are very much being used to eat major legacy systems and processes
> Senior Director of Product in Insurance Tech
anakin_padme.jpg
Those LLMs are not being wielded by 2 tech bros in a garage replacing the 100 000 person company.
Those LLMs are being adopted by the huge enterprise itself and being surrounded from all sides by a very complex procedural harness (aka regular programming) that stops the LLM from doing dumb stuff.
I'll add some...
Videos are shittier. Most songs are even shittier than the autotuned pieces of shit already were. Ads are worse (even though nobody thought that was even possible). Customer support is worse.
> But hey...on the flip side...a lot of people are rapidly building software (that nobody is using)
That nobody is using and that are worse than what we used to have. And because nobody's using them, the countless issues plaguing them aren't even reported by users anymore (there are no users anymore). Full of bugs, usability issues and gigantic security holes (but which doesn't really matter because nobody's using them).
It's not even clear that it's making long-standing software (like say Linux or Emacs or Blender or QEMU) better at all.
Now I'd say though: most of the issues you mentioned have nothing to do with LLMs / AI. But it's very clear that AI ain't solving much of the world's problems and that the promises we got years ago didn't materialize.
All you said is true, but on the flip side it is giving an individual user huge power over software. To create their own, to reverse engineer proprietary ones. Decentralisation of such power can only be a good thing.
Even if a certain small group of people are also using the hype surrounding this tech to attempt to destroy the home PC market and reverse the personal computer revolution. They will not succeed. There is more of us than them. Even with "thousands of frontier LLMs". A human+llm is going to be more powerful for a long time rather than a bunch of LLMs.
The power isnt really decentralized when you see that the internal AI models are superior than anything you can self host, and the right to use those can be stripped away at any time.
This is like saying food is shittier today because there are now more restaurants. No, food has never been better, there are just a ton of shitty restaurants.
More like saying samey corporatized slop bowl copy paste jobs (eg Cava) are driving out real, quality restaurants, a thing that has actually been happening for some time.
That pretty much sums up my experience. Everything is more expensive and worse at the same time. And it feels like we're being screwed over from every direction and in new ways every day. People keep talking about this tech is supposed to do so much but everything just gets worse.
What time scales are you using to make this judgement?
Everything isn't getting worse constantly. The data is quite clearly the opposite. How old are you?
As a middle aged man, thing are indeed getting worse every year.
From another middle aged man, part of that perception is your brain changing and also nostalgia.
Let's look at the past ten years.
Food has gone way up in cost and down in quality.
Private equity bought out the healthcare system where I lived so now it takes months to get an appointment and the quality of care is horrible.
Not that matters since the health insurance I pay hundreds for a month wouldn't cover anything anyways because it's through UHC.
My job keeps getting rid of positions because they want to "do more with less" but it just means I'm doing more work while being paid the same.
If I buy a new car it will spy on me and sell my data. The state DMV already sold my personal info without my permission and now I get constant spam.
Private equity bought out many of the restaurants I liked going to and ruined them.
Third spaces in my city have been mostly destroyed.
The social media sites I used to use were overrun by bots.
I can't upgrade my PC because hardware prices have gone insane.
Gas is like $5 a gallon now so I can't really justify going on any trips anymore.
Private equity bought up most of the rentals where I live and so now I pay much more for rent on top of crazy monthly fees while the building I live in is falling apart because they refuse to fix it. Housing prices have also gone up so much that I have no hope of owning a house.
And more.
AI has fixed none of these issues.
It's not true that everything is more expensive. Nominal prices are higher, but incomes have risen faster than inflation. Purchasing power of the average person in 2026 is higher than it was in 2016.
As for things getting worse, I guess it depends on the domain. In hardware, things have gotten a lot better; my desktop computer is orders of magnitude more powerful than I would have expected 10 or 20 years ago. In software, I do see a lot of bloated software being written, but I don't feel that the software that I personally use has gotten worse.
Average, or median? Don’t know the answer offhand, but there are a small number of well-known people who are likely to skew the average quite a bit upward, given how much wealth they accumulated during and since covid
I was using "average person" in the informal sense. Median household income in the U.S., adjusted for inflation, has gone up 13% in the past decade.
I would just add that consumer products in general aren't the same - practically every business model has incorporated obsoletionism to the point it's practically cancelled out the curve of progress and evolution of tech. We could have nice things.
At least for me, personal agents have been quite useful. If you integrate them well - shopping, filling random government forms, keeping track of my personal life, optimizing training plan and nutrition, planning vacations, etc. I can do more home repairs or renovations myself with help of chat
So basically you get personal digital assistant - very useful - but not changing my life.
What would I need for real transformation though? Food, house cleaning, commute - "every day chores". Even if we had ASI tomorrow, what could it do to change my life, besides taking away my job? Design good house robot to help me with food preparation I guess... Maybe finally self driving cars launched in my area? Or it will take control, "optimize humanity survival chance" and realize we don't need as many people on this planet.
Almost every time I use an AI chatbot it eventually turns out it's lying to me. I would not trust them with any of the things you mention.
> Almost every time I use an AI chatbot it eventually turns out it's lying to me. I would not trust them with any of the things you mention.
If you talk to free chatgpt, sure.
But if you launch Opus 5.5 in claude code - for some reason Opus is the smartest for me, even for "general" type of questions, when launched from Claude Code. It has harness/prompts optimized for correctness, while more "customer-grade" products, even when used with top model tend to optimize for latency. And I tend to give "context" - e.g. I have "personal wiki" + skills for lots of those things.
I don't recall i last few weeks where Opus 5.5 in CC made an obvious lie, made an unreasonable assumption, or made a mistake that smart human couldn't make. E.g. if it "failed" it failed in a way that non-genius human could fail in similar way.
opus 5.5 uses agents and skills in chat sessions as well and maintains internal context that it tries to distill from what you provide
I tried to have it write a synopsis website for the different types of dance patterns in Dance Dance Revolution, with guides on their execution, explanations, and provided song sources. I provided all of the source articles and search websites needed for it to be able to do that, and gave it a sample set I wanted it to utilize and build off of.
It has bits and pieces of information wrong in 90% of the sections it created. Poor song choices, missing song choices, incorrect explanations, incorrect step-patterns, the list goes on.
If you're not an expert in a given field for which you're using it, it's going to look like a genius. If you are, it's untrustworthy and borderline useless.
Maybe it wouldn't be if there were ample training data on my topic, but there isn't.
It utilized the research from millions of papers many very close to the destination answer to make those proofs. What would it do on something that isn't supported like that? Be confidently wrong while sounding right. That's what we've seen repeatedly.
> If you're not an expert in a given field for which you're using it, it's going to look like a genius.
For someone highly literate, llms aren't going to help you write better. If you're borderline functionally illiterate (this is > 50% of the population according to some studies), I'm sure the llms are awesome.
I've long thought the same about all the hype around coding (plenty of examples from years ago now), but will continue to test the new models. They are improving.
I think my anxieties would be different if I were confident a DDR guide full of lies would be spurned and unprofitable.
In a way, I'm not worried about the LLMs per se as much as a kind of moral and ethical collapse. A debasement of the currency of communication, a market of intellectual-lemons.
I saw electroboom's latest video lamenting facebook AI slop about electricity, such as a way to wire a three-way switch that blows a fuse when you turn the light on. For some reason, I am shadowbanned on HN.
I see your comment, perhaps it was a delay between representations or nodes?
It’s sort of weird how
1) open weights and cheap models are only about 6 months behind the frontier in evals, but…
2) it’s always the premium model that only the biggest fans pay for, that’s just now finally achieved enlightenment.
Oh well, I guess we’ll see in another 6 months.
I have ChatGPT Pro and it sources every single thing. For my Asia trip right now it's been helping me every single day to figure out optimized routes on what to do and where to go (if I feel like it, not to say I don't wander around too).
That sounds terrible. I like to live like a human being
What that does that even mean? Even if I grant your premise (I don't), if one is traveling and engaging these places in person....what's the problem?
Where did I say it was a problem? Using chatgpt to plan an itinerary sounds terrible TO ME
Why? In the past I would've used Google so it's just a faster search engine for me that works with natural language, what do you use for planning then?
Indeed, planning an itinerary, making calculations, building excitement for a place, is part of the journey
I suspect the HN techno-brained crowd will not appreciate it, many are quick to throw away their human experience
> Indeed, planning an itinerary, making calculations, building excitement for a place, is part of the journey
Not sure how they're perceiving using ChatGPT to plan an itinerary but I'm also doing all this while using ChatGPT. As I said in the sibling comment, it's a faster Google search and route optimizer, not something where I type in "make me an itinerary for X city" and follow whatever it shits out.
How often do you check the citations? I’ve found with the Google search AI summary it will provide answers and link to their citations, but a good chunk of the time the answer isn’t on the linked page.
It just makes stuff up (or gets it from somewhere, who knows) and then links to a tangentially related page as if it was the source. Either that or it identifies the page as being a source it wants to use, and then summarizes it incorrectly. Not sure the order of operations it’s doing internally, but regardless the citations are bullshit.
Google's models are worse that OpenAI's, doubly so on Google's free AI overview versus ChatGPT paid plans. 6.1 Sol is very good at synthesizing actual information it finds without hallucinating and I've not seen any instances it failed while I checked the sources.
The citations provided by the free version of Gemini are comically bad. They almost never contain information which directly supports the relevant claim, and more often than not they're completely irrelevant.
me too, but I recently blew the transformer on my boiler by following its directions to wire up a C wire to power a smart thermostat. oops. $500 mistake.
When is the last time you used an AI chatbot, and do you only use the free versions?
Which model? Super weird
Every time I use Google search AI for anything more complicated than a search query I feel that way. A paid model from pretty much anyone is much better.
Yeah, I’d hope that on HN, of all places, people know that free models optimize for price and speed, not for accuracy. But even with free models the accuracy was massively improved over the last year.
Agreed. Google search is still pretty rough, but it's hilariously better than it was when it was first released.
Apparently, I'm not supposed to extend that trend line though.
This is the kind of gap in shared understanding that is widening.
When people use words like “chatbot” to talk about their AI use and can’t really name basic things like the model they were using, it reveals a wide chasm between the people who are deep into understanding how to use these things and those who poke at easy options every few months and then look for ways to confirm their priors instead of trying to understand how other people are getting value.
using skills with opus 5.5 and agents for each individual task that have shared knowledge of all of your records does not lead to a significant increase in trustworthiness
The incentives seem to be lined up to push users to the frontier of "how much can the harness around the model do and claim without most people pushing back on the claims."
The bigger the pile of stuff you have the tool create unsupervised, the more incomplete/inaccurate the overall picture of it is gonna be. That doesn't matter for most hobbyist/helper-tool/game-port/etc type of tasks.
I get a ton of value out of the tools and yet part of that value has come from being hyper-aware of the shape of the limitations, which changes much less and more slowly than the specific of the limitations.
"How'd you notice that one!?" - well, it helps to be the only person on the team who took the time to interrogate the e2e interaction of all the code...
The median piece of OSS software was already pretty rough once you had to depend on it in production at scale to make money. The median piece of the new AI-generated wave of software, also pretty rough.
The bet that OpenAI/Anthropic is making is that they can automate the interactions generally/globally enough to work around this and eliminate the need for humans in the loop. But it's a tricky problem because it's precisely where the permutations and special cases get nasty and much harder to deal with than more broadly-applicable cases of codegen/test-gen/mash-it-until-the-tests-pass-and-the-code-review-angent-is-happy.
This has all started to feel very Emperor's New Clothes-ish. "Well, if it's not working for you, then it's your fault. You're just not using it properly. You're not worthy of its wonders".
Not to say there's no value to be had. However, it's also been oversold and its error-proneness is a real problem.
But, it seems there's a steady stream of commenters and creators who are quick to remind us that it's never a problem with the tech. It's always a user problem.
Meanwhile, the errors are renamed "hallucinations", but let's not focus on those. In fact, if you just look right over here at this .md file...
It's like driving a PT Cruiser and concluding all cars are bad and swearing off them entirely.
> When people use words like “chatbot” to talk about their AI use and can’t really name basic things like the model they were using, it reveals a wide chasm between the people who are deep into understanding how to use these things and those who poke at easy options every few months and then look for ways to confirm their priors instead of trying to understand how other people are getting value.
While the latest and newest should be needed for greater capability, the average should be enough to give a glimpse pf what is possible. Like a basic netbook is enough to enjoy computing, no need to purchase the latest supercomputer.
If you need to switch your setup every quarter or so, it’s not a tool, it’s a rat race.
Personal agents are wonderful! You can ask for hiking trails and it will customize one for you, frequently resulting in emergency calls for rescues! Your purchasing/selling AI will arrange sales for you but forget to tell you, leaving you in breach of contracts you didn't know about! Your personal nutrition agent will absorb all the broscience and customize a diet that will result in hair loss, but at least your in ketosis, amiright! And instead of hiring a plumber/electrician/contractor to do work subject to code, you can just ask an AI to tell you how to do it and then wonder why your buyer is suing you for $$$!
(Those are all examples from just the past 4 days, and just the ones that made it into mainstream media.)
> besides taking away my job
> self driving cars launched in my area
It sounds like the transformation would be a massive collapse of the economy with scores of people unemployed.
> Or it will take control, "optimize humanity survival chance" and realize we don't need as many people on this planet.
Maybe, but more likely it will be the humans who control the AIs and hold the power who will make this decision.
Yes, and additionally, on the other side, there are a bunch of people using it to build exploitative products, spam, weapons, etc.
This is the core hubris of everyone behind the LLM craze. I do not want to deny their utility there absolutely is some there. But the inherent assumption behind the whole thing is that with a "country of geniuses" problems will melt away. But that's not the case. The reason problems like housing or famines or war aren't being solved is because structurally there's no incentives to solve them not because there's not enough geniuses out there.
> The reason problems like housing or famines or war aren't being solved is because structurally there's no incentives to solve them not because there's not enough geniuses out there.
Rather: these problems are not solved because there are not enough geniuses who are capable of convincing the masses to send the responsible politicians to hell.
A great deal of the supposed smartest people have fully sided with fascism, it's not about smarts
"Supposed" is doing a lot of work there. Compare, say, Yudkowsky et al. to the people who actually changed the world with mRNA vaccines.
Yudkowsky is hardly an example of someone who has sided with the fascists.
I’ve heard people say variations of, “Clever people can easily convince themselves of just about anything, if they’re inclined to. And then it’s nearly impossible to talk them out of it.”
Kinda seems true. And the worst people in history often don’t seem super dumb, despite their various other defects.
This is broadly the same people who bemoaned the idea that the "smartest people in a generation" were all just working on adtech without any reflection about maybe that's not actually an exclusive set of the smartest people... or that a lot of the people in that set were the same one who might, twenty years ago, otherwise be chasing money as middlemen in finance vs doing productivity-expanding things anyway...
You can be smart and chase the money, but it's crazy hubris to assume that only your peer set of money-chasers are smart.
It's a small leap from there to "since we're the only smart people, we should get to have the most influence."
I'm confused by this comment, can you explain?
For that you might need the kind of geniuses who write stirring books about principles and values. Or whatever the modern equivalent of those books is. Maybe nobody would pay attention to that kind of advocacy in the present day, not sure why. But it's not tech geniuses, anyway. Much though tech has power as a catalyst.
I suppose there was Project 2025, and there was that book about the inevitability of Russian hegemony (or empire or something) that Putin liked. Those got attention with influential people. Not sure what to conclude from that.
In America anyways, we send the “responsible politicians” to hell every 2-6 years. People vote for this.
and yet... it feels like we are actually a one party state most of the time.
The party of corporate interest?
Maybe underneath it all we really do want the same thing…
More like the people with power all serve the mechanism of power: Capital.
Duverger's law. Any elected body with FPP voting will converge to a one-party body.
That is not what Duverger's law says. (But it's a interesting hypothesis so thanks for mentioning it.)
Democrats expanded SNAP, Republicans shrank it. Dems expanded Medicaid, Republican shrank it. Dems protected reproductive rights, Republicans repealed them. Dems expanded title IX protection to LGBTQ+, Replicans eliminated them. Republicans jailed 1,200 January 6th insurrectionists, Republicans pardoned them. Democrats provide non-partisan federal emergency aid, Republicans withhold federal aid from blue states. Dems establish tax credits for electric cars, increase automotive standards. Dems didn't kill of foreign aid to need countries.
If you aren't female (reproductive rights), LGBTQ+, an immigrant now hunted by ice, someone in need of SNAP, Medicaid, or emergency aid in a blue state, it might feel like both are the same I guess. If you are one of the above, and now suffer because so many voted based on 'both are the same', you very much feel a difference today than from the US under Biden. If you were one of America's allies but are now constantly attacked, you feel a difference today. If you are Ukraine, you feel a difference today.
> Rather: these problems are not solved because there are not enough geniuses who are capable of convincing the masses to send the responsible politicians to hell.
People support policies that make housing much less affordable (e.g. rent controls) and oppose policies that would make housing more affordable (e.g. zoning reforms). Saying that it's evil politicians that cause this crisis is dishonest - it's what people want.
People by and large don’t understand the second and third order effects of their policy preferences. It’s not that they don’t want those things, it’s that the world is more complicated than their mental model insists it is. Politicians could try harder to fix the fundamental problems but they don’t because the long-term solutions don’t manifest until after the next N (very expensive) election cycles.
The politicians are those who created the respecting laws.
"Convincing the masses to send these politicians to hell" is thus nearly isomorphic to making the masses hate politicians who create such laws, which at least very likely implies that by convincing the masses to send these politicians to hell, a different mindset emerges in the masses.
> The politicians are those who created the respecting laws.
Yes and general population LOVES these laws, even though they are harmful to them. But anyway - you've got your causality reversed - it's voting masses that incentivize rent controls and other shitty politics. Remember that to implement your policies you've got to actually win elections. And since people generally don't understand n-order effects for n>=2, it's really easy to win elections with populist policies.
There is no evidence that reducing rent makes the rent go up - only fearmongering. It's a stopgap measure, though, since there still won't be enough housing to meet demand unless you also fix that at the same time.
Which you can also do as a city by just... building! You don't need some developer to come in, you can just do that as a local government!
The City of Vienna is the largest landlord in Europe. It is also the only capital in Europe without a stifling rent or housing crisis.
There is lots of evidence for that, but before I provide it I need to know if you will update your beliefs based on that.
Really? You have evidence that smaller numbers are bigger numbers?
Don't argue in bad faith. I have evidence that rent controls increase rent, not that smaller numbers are bigger numbers. Please read some evidence about policies you are supporting - https://www.sciencedirect.com/science/article/pii/S105113772...
Rent controls work if you go radical enough and enforce a max profit margin for the whole city, with rent limited to something like 30% over maintenance + debt repayment costs.
The amount of housing in the city won't grow but it will free up a big part of peoples income that would otherwise go to landlords. That helps the local economy with more consumer spending, and makes it easier to save up or apply for a loan so you can start a business.
You get a better economy and more money in the average persons wallet without raising taxes. The only ones who lose are commercial landlords, and they will stay profitable by design so it's easy to shut down any complaints. And you get better quality apartments as a side effect because the only way to get more money out of existing properties is to spend more on fixing them.
That policy would 1) fuck over younger generation, 2) result in total sclerosis and death of city. Prices are high because there's not enough housing, that's it. Rent control won't increase number of units so it wont solve real cause of the problem.
> The only ones who lose are commercial landlords, and they will stay profitable by design so it's easy to shut down any complaints.
No, also people who would like to move into city would lose, because that would become literally impossible. You'd have like 15 years waiting lists for housing.
> And you get better quality apartments as a side effect because the only way to get more money out of existing properties is to spend more on fixing them.
Why would anyone fix apartment that's under rent control? If you mean system like in Berlin, then I've got bad news - people there are actually frequently not allowed to renovate even THEIR OWN apartments.
> because there are not enough geniuses who are capable of convincing the masses to send the responsible politicians to hell
I could talk about this for hours, and was last night actually.
What do vanguardist communism, feudalism, liberal democracy, and fascism have in common? A shared belief that the general population is composed of a bunch of dumb dumbs that need super smart (read: politically educated) people to "lead" them.
Seems subjective. When is the smartness, big smart enough to allow someone to enter the hallowed ranks of "actually a real human allowed to speak for the masses?"
There does seem to be a clear delineation between the people you hear about (the vanguard in our example), and the people you don't hear about (the masses): willingness to create confrontation. Most people want to just go with the flow, live in peace. In a dichotomy I've been calling "choose peace or violence," they choose peace. They choose to see bad things in the world or their life and focus on managing their happiness around it.
I think that this is the default human mode of existence, and that this mode lacks defenses against people willing to violate the default peace. Imagine a bagel line, and someone cutting that line, and when tapped on the shoulder, the cutter breaking into a huge tantrum, and people in line backing down with a "Jesus! It's not worth it, fine." Thus some people learn that they can be much more selfish in life if they're willing to create confrontation and disturb the peace.
When I talk to people about agitation, civil disobedience, or even just unionizing, the most common thing I hear is anxiety. What if the situation gets worse, what if there's reprisals, what if I lose this little bit of comfort that I've managed to be happy with? That slice could get smaller and smaller but people aren't willing to take the step to say "enough is enough" because I think it fundamentally goes against human nature, we have to sacrifice our own happiness on some level to put our foot down and demand or make change. At minimum we have to acknowledge the discomfort or injustice, which is a diversion from our constructed internal peace.
Personally I think people should be allowed to live in peace. Maybe the liberal democrats and vanguardist communists are right, maybe most people shouldn't need to trouble themselves with the greater direction of their society. However I think the stability this promises is false; in fact I think all stability is false. I think this constructed peace that people are afraid to disrupt clearly can be torn apart at any moment, not just by people willing to create conflict, but by a literal earthquake, or fire, or plague. Therefore I think there's a couple societies in history that got it right: ones who distributed the responsibility of society across all individuals in the society; namely the brief period of syndicalized Spain, and the several hundred thousand years of certain tribes of native Americans on the eastern portion of the continent. There may be other societies in history, these are just the cases I've been able to read about so far.
It seems to me that it's possible and necessary for all people to be ready to wake out of their constructed peace long enough to make adjustments together to maintain it. I don't think this is the same thing as them being "led out of it," I think this is each individual being on some level alert to the need to spontaneously say "no, here we wait in line for our bagels. No, screaming about it won't help you, get to the back of the line. Look around: everyone is laughing at how silly you look."
> What do vanguardist communism, feudalism, liberal democracy, and fascism have in common? A shared belief that the general population is composed of a bunch of dumb dumbs that need super smart (read: politically educated) people to "lead" them.
That was not premise of fascism. Certainly not Hitlers one, he himself was self-taught and did not framed himself as highly educated. He considered himself better then educated people. The origin of his superiority was not "smart" it was "strong will and power itself".
It was not the original premise of feudalism either. That was was about military might, basically.
And origins of liberal democracy are not "smart based" either.
been a while since ive see the real talk on hacker news. its refreshing. usually I feel like I'm the only one among nerds asking to hold people accountable.
so, how do you think we can do this?
> so, how do you think we can do this?
For a lack of deep knowledge in this area, I don't think I have answers for this, but if one is interested in working on this problem, my approach would be to start with diving deeply into scientific literature about psychology and sociology to find out
- what convinces people to change their opinions,
- what causes people to take action
etc.
yeah, sure. Do you think all scientific literature is the same though? I am skeptical of thinking to just look at them all equally. Clearly, if theres a class of people being held unaccountable who have been practicing imposing their will and power on the people, that has certainly been done in the scientific community and would show in the literature, too.
So when I think of going through any media like scientific publications, I wonder how to discern and point out the captured and biased media vs the more objective ones (or at least the ones that don't have a hidden financial motive behind them). I wish our scientific community had better disclaimers and traceability for financial influence in every paper we could read.
> So when I think of going through any media like scientific publications, I wonder how to discern and point out the captured and biased media vs the more objective ones (or at least the ones that don't have a hidden financial motive behind them).
This is exactly the hard part: a topic which many people in more applied scientific disciplines worry about. Welcome to the real word, Neo. :-)
Again, I don't have definite answer, but rough guidelines are:
- is a result too good to be true?
- is some result coherent to results that are much more established?
- why would you assume that the author is unbiased or biased? (biases not only have financial reasons)
- which agenda does the author have (yes, deep pursuit for scientific truth is also an agenda? Nobody is free of an agenda)
- does the author make an effort to explain how the findings could be explained in an alternative way that makes the result look much less impressive?
- could the results be reproduced in other labs?
- which is the minimum amount of facts that you must believe in (like axioms) such that the result holds?
- ...
I like that, I can see that being true and working.
The hard part is indeed that we have to exert all this effort to get past the noise that I (and maybe you) feel a responsible author should have already done for us.
And with AI: I was hoping it would help us filter through this noise. So far, I feel lost in how to use AI to help me filter through prejudice and bias, and hidden motives and intentions, when the very filters given to us in the mainstream AIs we are sold are doing that same very bias and incoherency and hidden agendas. I retain my hope, however! Thank you for some solace and inspiration here, friend!
To add insult to injury, we're being told: "Just trust us bro, however we make things worse now will totes be redeemed as the prophesied god-mind magically fixes it later."
Strangely, they never discuss the possibility that Multivac might seize all their wealth and cast them down into average-ness. What if it actually comes about, but then it tells us to do the things we already refuse to do?
Yes, many of the problems are 'features not bugs'. There's no reason to think the AIs won't dream up more of these'features' too.
And the LLMs exist to further those problems. People can't afford houses? Well, now they won't be able to own computers either. That's a feature, not a side effect. Once the K-shaped economy progresses enough, you won't even be allowed to use the LLMs. They'll exist only to be a tool for the new feudal class who owns you.
Unless you destroy their wealth now.
Once capabilities catch up with the demand (~which is in few years - it just takes time to build!) average person will have much better access to higher quality and cheaper hardware. It's literally how it has always worked, I don't understand why would it be different this time. There's zero substance in your comment, just repeating tired tropes from twitter.
A market requires scarcity, because scarcity is what neccesitates trade.
You can't invoke market mechanisms to defend a post-scarcity scenario.
Why would the ai owners make things cheaper or even available at all once their abundance machines give them everything they want?
What would they get out of if it? What could ordinary people trade for it that their abundance machine can't give them already and that would be worth giving up their monopoly on productive force?
You can't argue "this will change everything" and "this is how it's always worked" simultaneously, and yet AI accelerationists bring out both to defend their ideology.
Right. Capitalism is clearly incompatible with the proposed AI future where no jobs are needed.
> You can't argue "this will change everything" and "this is how it's always worked" simultaneously, and yet AI accelerationists bring out both to defend their ideology.
Luckily I did not, you are arguing against points I did not make at all.
Oh, but you did: https://news.ycombinator.com/item?id=49648472
Either way though, what's more interesting than what you specifically said is that AI boosters have no answer to this structural argument about the reality of post-scarcity. Tellingly, you didn't address that part of my comment at all, either.
> Either way though, what's more interesting than what you specifically said is that AI boosters have no answer to this structural argument about the reality of post-scarcity. Tellingly, you didn't address that part of my comment at all, either.
You've fished a comment from completely different discussion, where a point was that if AI were to displace workers, it would be different than previous waves of automation (and there meeting these capabilities was assumed, not put forward by me). But in this thread it's more down to earth discussion about manufacturing capabilities if there is no ASI. But sure, I'll bite - yes, capitalism won't work in post-scarcity society. But we are not leaving in post-scarcity society and in our current situation capitalism is amazing system. If capitalism would get us to post-scarcity society then well, you'd have to admin that capitalism is amazingly effective. Your assumption about AI-rulers (I presume altman and amodei, not governments) having abundance machines and not sharing the abundance with anyone else is really without any merit. For this scenario to work we would need to meet few conditions that are so unlikely that it's not really worth entertaining them seriousl: 1) powerless government 2) actual ASI that's abundance machine 3) ASI that's perfectly aligned towards serving altman and amodei and none else 4) altman and amodei deciding that everyone should be poor. And also as someone who I guess you'd call AI booster (so a person that's not deluded about AI capabilities and sees how powerful they are and how current concerns with AI such as water use are overblown) I can tell you how I address this - you can't stop progress. It never worked and never will. I think world would be better if we would've stopped developing AI. But it's impossible so you might just embrace it.
Companies have been trying to figure out how to get people to rent pcs for a while. Unfortunately the AI boom has shown them how, buy out all the capacity of the ram manufacturers, pay them enough consistently and the manufacturers have no reason to expand their capacity. No cheap ram means companies and then home users will have no choice but to come to you and rent their pc.
> That's a feature, not a side effect.
You’re implying that the deep lizard state planned the LLM craze?
Sama might be a lizard..
s/deep lizard state/billionaires/
r>g is an old hustle, LLMs just accelerate it. Lizards aren't wizards.
when there was so little capital it had to be true... the idea that it will persist forever is obviously silly.
The problem isn't that it fails to unwind, the problem is what the unwind looks like. Last time it meant global upheaval and world wide capital incineration -- mostly metaphorical in the case of the US but very literal in the case of Europe and Asia. The poor suffer more in real terms but capital suffers more in nominal terms and that unwinds inequality and the burden of furnishing runaway capital with the runaway returns it demands in excess of economic growth.
I'm sure the same "strategy" will work again, but this time around everyone has nukes and drone swarms and AI-equipped bio labs so shit's gonna be lit.
r dropping below g isn't "unwinding". r has been falling for decades now. The world is awash in capital.
r>g, not r<g. In 2025 GDP grew 2.1% and SPY grew 17.7%, which is the pattern recently and throughout time.
You don't need a back room filled with cigar smoke where supervillains coordinate their plans while twirling their mustaches when good ol' class interest does the same job.
Luckily, the AI bubble will destroy a decent chunk of their wealth.
The Gartner Hype Cycle [0] is real, and ubiquitous. We are teetering somewhere near the top of the Peak of Inflated Expectations, about to plunge into the Trough of Disillusionment. A lot of very rich people are about to lose a lot of money, and we can only hope that they don't get bailed out (again).
Then, once the bubble and hype is over, the long, slow work of actually doing useful things with this new tech starts. That's when we'll see it affect everything else.
[0] https://en.wikipedia.org/wiki/Gartner_hype_cycle
> We are teetering somewhere near the top of the Peak of Inflated Expectations
I've been hearing this for a year and a half now. That word, "ubiquitous", is getting real mighty challenged. I have tried to insulate myself as best as I could and it just never happens and every other day I wonder if my foil hat needs adjusting. Does yours?
Previous bubbles were identified long before they popped too. It can easily go for the next two years.
Gartner hype cycle isn't scientific and is mostly voodoo marketing gibberish, why do people still post it as if it's real?
If the bubble bursts in the worst way it could, we are all fucked, but will recover eventually
Regardless, people have been making your prediction for a couple years now
> Luckily, the AI bubble will destroy a decent chunk of their wealth.
Even if all this is true (personally I doubt it), it assumes the bubble won’t burst before they’ve shifted the money from all our 401ks, IRAs and index funds into their own pockets.
The hype will be over one day, but when is that?
I've done multiple life choices, away from AI, believing it to be a hype, but this hype is still growing, still not dying.
Both in 2019, as well as in 2023, i.e. not 2 or 3 months ago.
> The reason problems like housing or famines or war aren't being solved is because structurally there's no incentives to solve them not because there's not enough geniuses out there.
When I asked why Musk doesn't put his money towards non-profits my Musk supporting friends say it's because he "doesn't believe anything but for-profit entities can solve the world's problems".
So why isn't he solving homelessness, famines, war, etc. with capitalism?
What I like to call at work "this is not a technical problem". There are many problems we could easily solve if we had the right structural incentives. But instead we have politics, processes, layers of management, etc. All that doesn't disappear (yet?) with AI.
I believe that’s actually one of the arguments of the boosters. Utopia will finally be reachable when we get all the human drama and bureaucracy out of the way
I’m not sold, but I’ve heard this line of thinking many times
Human drama is what people use to advocate for themselves. Bureaucracy is what people use to protect themselves. Then you have the side effects but, you may also not like a world without those at all.
Agree. I’d only add that I think there is incentive for them to be solved, but there are greater incentives worth more to those leading the charge.
Whatever benefits AI has to offer on a societal level are going to take a long long time to show up, give it some more time. We had smartphones in 2007, and e.g. By way of example, Uber didn't launch ridesharing until 2013, and then that still took years to seep into the public awareness.
By comparison, we're in the early years of AI. We're all still figuring out what this thing even means.
I agree with you, but those examples are going to fall on deaf ears. The pessimistic mindset will say that smartphones were a terrible invention that's ripping society apart, Uber is an unethical company exploiting workers, etc.
When did Uber stop being unethical? Did I miss the memo?
Because they didn't just break laws, they built an entire system to mislead law enforcement.
(How do tech people not know about Greyball??)
I think ridesharing was a net good for ordinary people even if Uber is/was unethical.
However, the counterargument falls entirely flat. "Whatever benefits AI has to offer on a societal level are going to take a long long time to show up". We don't have the luxury of time because AI disrupting communities and livelihoods faster than perhaps any other technology we've ever seen, with the most capital that has ever been put into any tech since the railroads.
Even if you're optimistic we'll eventually get the cancer cures and fusion reactors, doesn't it make sense from the perspective of an ordinary American to regulate the harms like the homework cheating, parasocial relationships, job losses, cyber attacks, etc., especially if it will take years to get the benefits?
Different benefits will show up at different rates. We don't need to wait for cancer cures and fusion reactors. It's already easier than ever to start a small business and use AI for things that would have been prohibitively expensive or time-consuming before. The technology is transforming how scientific research is done, how engineering is done, etc.
I don't see much evidence in the economic data that AI is disrupting livelihoods. Unemployment has been stable around 4% for a while and productivity growth has been at historically normal rates for the past 5 years. I think a more likely explanation is that AI is increasing uncertainty in some industries, which are seeing higher employee turnover, and many of those people are moving laterally into other jobs where their skills are more likely to be useful.
I don't think anyone in this thread has argued against mitigating harms. The OP was about current capabilities and the perceived lack of benefits for the average person.
To be clear, I don't disagree with your sentiment that the disruptiveness of AI is causing a lot of grief. But magnitude of disruption is not an indicator of magnitude of benefit (or lack thereof).
Some of the most disruptive periods in history brought about the greatest long term improvements in society. This is the price we pay for changing how things are done, sometimes for better and sometimes for worse. We don't know which one it is for AI yet.
Uber was a massive mistake on every level.
This is what is so frustrating about this whole ai thing is we aren’t learning from the mistakes of even the very recent past
Uber is a vast improvement over the previous taxi oligopolies.
I miss being able to reliably hail a taxi A LOT. Uber may be an improvement in smaller towns where cabs didn’t exist, but in bigger cities it is mostly a downgrade. Rather than walking to the curb and immediately getting a car, I’ve got to dick around with my phone and wait for a car to come to me.
Living on the south side of Chicago, it was almost impossible to hail a cab, and often times if you called for one it just wouldn’t show up. Or I’d try to hail one downtown, and be refused once I told them my destination. I’ve not had that problem once with Uber, good fucking riddance to taxis.
What's frustrating is that at the end of an event like a concert, everyone goes outside to catch an Uber/Lyft, but we then have to match driver to passenger, instead of having a line of waiting cars, each group just gets in the next available vehicle, shows the driver a QR code, instead of the mess we currently have. Why can't we do that? Taxi regulatory capture.
Taxis were terrible and I celebrate their destruction at the hands of the rideshare companies. Even now it’s still cheaper than a cab was (though those early days of VC cheap rides were great).
Uber has been great, I can get around to almost anywhere with a click of a button and have food delivered from a variety of local restaurants at any time of day.
I'm not sure why you think it was a massive mistake? The taxi industry was awful and unsafe, food delivery for many local places was either not available or poorly implemented.
Also many whole food chains didn't have proper delivery for groceries.
Using AI for everything digital, I end up going and doing chores while waiting for the LLM to do its thing. What an insane world, where knowledge work (dreary and uninspired as it is) is automated while I am consigned to scrubbing dishes and waiting for the machine.
Buy a dishwasher, you won't regret it!
Honestly the big reason why I use my dishwasher is that all the dishes would not fit in my drying rack. I'm not very convinced the process of me rinsing and scraping my pans before they go in the dishwasher is any faster than when I lived in apartments when I handwashed everything. Maybe my dishwasher sucks too, but I mean it doesn't hold a candle to the sort of jobs on my pans that require good old fashion baking soda paste and elbow grease. I'm still slaving with this thing. It's picking the low hanging fruit for me in other words.
I feel similar about my robot vacuum. It actually takes a surprising amount of labor facilitating the damn thing. I have to walk through the house and basically vacuum proof it so it doesn't eat various cords, rugs, lifting chairs up and putting them someplace so it can get under, etc. Not sure its really a labor save to when I was just brooming the entire apartment in 20 mins and not having to move anything around due to reach of the broom and it not having a tendency to eat things. Hand vacuum I can be more carefull too about the fringed edges of rugs and don't have to fold them over themselves.
I still do it because I already bought it mostly.
The trick is dialing the things in properly once, and then reaping the benefits over and over again.
e.g., my home is always vacuum-proofed. It automatically runs through all rooms except for the bedroom at night, finishing just before my alarm clock rings, then vacuuming the bedroom an hour after the last person's alarm clock has rung.
This way I never actually notice it, but the home is always perfectly clean as if by magic.
Similarly, if you use separate dishwasher detergent, antisurfactant and salt, and properly dial in your water hardness, it'll be able to clean everything no matter how dirty the dishes are. Including dried-on tomato sauce. Just put the dishes in, and they'll be clean the next day.
Of course it takes a bit of effort to set this up, but then it just works for a long time.
Similarly, I'm a big fan of handwriting software with care & focus on quality, which takes a bit longer at first, but then allows you to trust the software to continue running for years to come.
I've done a lot of vacuum proofing, even getting the magnetic strips, but there's only so much you can do. Like it doing a little wheel around a chair leg is just not as great of a clean as if I moved that chair out of the way. My rugs have fringe so there's no stopping that. Cables I tend to have out have to go away.
And then of course, all the cat's toys.
It's definitely easier in a minimalist & tidy home. Our rugs have no fringes, cables are either rolled up or permanently installed & hidden, chairs are stored at night upside-down on the tables they belong to.
Sure, it's not perfect, but it's still really convenient.
> And then of course, all the cat's toys.
This might be the next thing the robot vacuums all start working on. One has already released a first version: https://www.youtube.com/watch?v=GykaEgNkkVQ
LLMs could’ve been developed a lot slower without capsizing the entire supply chain and been seen a lot more positively, China is building a lot of good AI with a minuscule of the compute power and it’s fine
there was never a need to rush other than for capitals self interest to control
if frontier labs wanna pace the frontier, they should be honest and return the compute back to the people
China is building good AI by distilling American models. I think that's a great thing, but Chinese progress is currently dependent on American progress, because China lacks the compute to train the largest models (as stated by DeepSeek's founder[1])
[1] https://sumsec.me/2026/liang-wenfeng-investor-meeting-transc...
i knew about the compute already (he was very public about i believe huawei under delivering the amount of chips he requested), but nowhere in that transcript does he say he's dependent on US progress for his own progress or he's distilling the US to make progress
i don't really think distilling is even that fundamental anymore (nor was the evidence ever that strong - https://msukhareva.substack.com/p/is-kimi-k3-really-distille...), the game is all RL now, and you don't need american progress for that at all (just compute and data, and the data brokers will sell to anyone no problem)
i do think there are black market proxies from china to the US trying to sell "claude" and "deepseek" access mixed together, and i do think that prompt distribution is probably going to be very nice datasets
Publicly-accessible AI started working 1 year ago, 2 years tops. What did you expect? It to solve war driving prices up?
I would have expected apps like Amazon and websites like various Meta properties to be even slightly less broken and shitty.
Like, just at a bare minimum.
These are “use all the AI” companies with infinite resources, after all.
Why use apps that you dislike, just ask Claude to click through them for you. And no I'm not being ironic, this is the way things are headed, you just abstract over all the crap by having an agent do it for you.
"Slop is its own cure"
The alternative is people stop using these apps, consumer demand goes down, along with the GDP of the country.
I'm immensely sad every day seeing all these posts talking about billions and trillions of dollars and where I live (part of the US btw) we don't even have fully reliable electricity and water access. In fact our electricity bills have increased and there is nothing to show for it.
I don't understand why these big labs aren't doing more to improve the electricity situation. Dump a billion dollars worth of tokens into fusion analysis, research, and simulations and try to force a breakthrough. You could buy a lot of good will with unlimited cheap energy that liberates us from fossil fuel dependencies.
OpenAI and Sam Altman are literally backing a nuclear fusion power startup.
"Startup"
Wait until you have clean energy sources and shut it down
Quoting startup like that makes the effort seem like it is an extremely recent venture.
It is 13 years old.
It has been making progress, the degree and date of success is debated and confidently claimed to be in many mutually exclusive states, which at the very least says that there are lots of people confidently making claims that are not true. I'm not going to join that fray to declare which of them is correct.
Is it ready to provide power? No? Then I don't care what they are trying to do while they are burning a bunch of fossil fuels in the meantime
Are you avoiding all use of fossil fuel while alternatives become available?
It is a bold claim to make but not something that usualy be done. Any abrupt halting of fossil fuel would severely impare the ability to develop and produce alternatives.
We need to transition to sustainable methods. If you want to go cold turkey, go ahead and cease driving an ICE, never travell by air or have anything delivered (and that's just the tip of the iceberg), it would be a counterproductive way to reach sustainability, that is your right, but you can decide to do that to yourself. You don't get to decide how others reach sustainability.
There is a difference between advocating for a better world and acting like a bully because you don't like the current world.
This is all incremental use. What have we gained? How many kids in Memphis should get asthma so you can code a little faster?
Because they increased the demand so much they are seeking out their own supply. Nuclear has consistently been panned for decades because it's more expensive than other power generation.
Mathematics and programming is verifiable and fully controllable in a digital environment.
Fusion research and cancer research requires expensive experimentation in the real physical world. You can’t just throw a billion dollar worth of tokens to handle the problem.
Look at all the roadblocks for data centre buildouts which the industry certainly has a financial incentive to complete as fast as possible.
> Fusion research and cancer research requires expensive experimentation in the real physical world. You can’t just throw a billion dollar worth of tokens to handle the problem.
When the tokens are hooked up to physical devices (robotics of various sorts), then the tokens pay for themselves by being physical workers, or human wage replacement.
And a robocop will follow whatever is passed for law, damn the masses who might vote otherwise. And subjugation will be enforced by token, LLM rule, and machine. They have no union or ethics to complain to.
Who taught you economics?
The complexity of manufacturing is one reason why TSMC is one of the most important companies in the world right now and why Nvidia is the most valuable by marketcap.
If you can make a GPU more efficient and cheaper than Nvidia does you can make a ton of money. Plenty of labs around the world are working hard on this problem and yet Nvidia is still dominating (and the market believes this isn't changing any time soon)
Same things for robotics. It's unlikely you'll have a vertical stack on all the different things needed to manufacture your robot on a massive scale. What happens if some geopolitical spat makes getting rare earth minerals and battery tech from China impossible?
Compare that to tokenmaxxing in programming and math, where the only thing you need is a gpu cluster and electricity.
Electricity infra is not a technical problem. It’s been solved for decades. It’s a political problem, and both sides of the isle have had plenty to do with getting in the way
This is a Western-centric viewpoint. I'm in Asia visiting a few different countries right now, including the big ones, and much has changed with AI and also in general over the past 10 years with cars, trains etc.
> AI is solving mathematics problems (that nobody - including mathematicians - wanted it to solve)
Strange viewpoint. In mathematics any novel result is useful as it expands the scope of other results. Who are these mathematicians who didn't want AI to solve the problems it did? I see trepidation about the future of the field but that is not equivalent to not wanting them to be solved.
I'm curious, care to name a few things that have changed in Asia with AI (and wouldn't have without)?
asian vertical dramas (lol)
I've watched some videos from Chinese commentary on AI usage in the industry,
Some examples are finance people directly asking for API access now so that they can make their own tools. These tools used to be maintained by teams of developers.
Many business making their own customized scripts to make their tasks faster.
From an economic point of view, it seems like there are some productivity increases, but I don't think these can be concluded for a while. The rational thinking rarely account for second order effects. Although business leaders rarely care about second order effects, a worse business is not immediately noticeable, compared to hiring cuts and the money it saves
That is not how math works. There are reasons why the Riemann integral is pervasive and the Labague integral is rarely used unless needed. Just because you can do something does not mean that you should.
> Just because you can do something does not mean that you should.
Again, strange viewpoint in the field of mathematics of all places, unlike, say, economics where doing something one shouldn't has disastrous real world consequences. Until it's discovered and proven, one doesn't know what would be valuable and what would not. I bet your sentence would not be said about humans trying to solve problems, no one would tell them they "shouldn't."
Terence Tao and the apparently newly formed Association For Human Mathematics. The post is online-famous now for the line "Mathematicians did not ask for this work to be done."
https://terrytao.wordpress.com/2026/10/07/ahm-statement-on-o...
When I was a young man reading gauss, and thinking before I turned the page, I see a result here, something meaningful! and then I turn the page, and of course it is there. There is no shame in this.
A long time ago, another student of mathematics described it to me as climbing a mountain. No single step is hard, but you must make many steps to get to the point that no one else has made a step.
> much has changed
Trust me bro
No need to trust me, just look at how China was 10 or 15 years ago to now. Beijing air pollution, more high speed rail than the rest of the world combined, electrification of vehicles and economy, etc.
^ AI did this
Partially, yes.
>Your car is still the same.
Not for Tesla owners in select countries, where their car now drives them around 99% of the time, or the people of Austin Texas, that can now hail a fully driverless Cybercab for cheaper than an Uber!
>Food quality is worse.
Food quality is higher than ever if you want high-quality food!
Unfortunately, the people crave slop.
AI has only been around a few years in its current form.
It might take a decade to take what we learned from AI and turn it into something that will benefit the masses.
Most of it can't be broken down into something as simplistic as a dishwasher.
Most people won't notice cancers being found faster due to AI being better at pattern recognition than a doctor, but it's still a great benefit.
A decade? Better not tell the investors that!
Is it reasonable to expect AI improvements to filter into all of those aspects of life that quickly? It's only in the past month or so that we've started to see major breakthroughs in mathematics. It might take several years for that level of research capability to reach areas like engineering and medicine, at least in a way that affects the average person's quality of life. But at this point it looks very likely that it will happen eventually.
Your pessimistic view doesn't hold up against the data. For a lot of the problems you mentioned, we're in a better place today than almost any other time in history. Of course, things can always be improved and we should never grow complacent or accept injustice. But by most objective metrics, there's never been a better time to be alive.
It’s amazing how much the goal posts move in every AI conversation. A couple of years ago the average HN thread about AI would be filled to the brim with skepticism about AI ever being able to accomplish anything other than flappy bird clones.
Now it’s comments like this lamenting that the amazing LLMs we’ve had since early 2026 haven’t entirely revolutionized the whole physical world and even global geopolitics yet.
> But hey...on the flip side...a lot of people are rapidly building software (that nobody is using),
I guarantee a lot of software you use today has substantial AI development in recent releases.
Even the Linux kernel is getting a heavy input from AI and Linus is advocating for it. Unless you’ve frozen your entire software stack in 2024, you are using software with AI generated code.
> and AI is solving mathematics problems (that nobody - including mathematicians - wanted it to solve)
Saying nobody wants those problems solved is flat dishonest. The complaints from some mathematicians are that they and their peers didn’t get to put their names on the papers, not complaints that the problems were solved.
Have you seen the shareholder value though? That’s not nothing it’s the only thing that matters!
And the human society was entirely destroyed, but for a brief moment we were able create share holder value.
With AI software replication, I wonder if shareholder value even increases. Services accounts for 80% of American industry, what happens when big software companies loses revenue to piracy/recompilation i.e. Adobe, Unity, AutoDesk, tax/law software, etc..
Yeah people were saying ai is the death of open source. I dunno about that but it seems to me closed source is having a bit of a crisis here too.
> But hey...on the flip side...a lot of people are rapidly building software (that nobody is using), and AI is solving mathematics problems (that nobody - including mathematicians - wanted it to solve)
That's not fair. To first approximation, most of the progress in human condition of the last, like, forever is implementing solutions that people didn't see value in at the time of creation. Woz built a toy that could barely run software at all. The human genome project was panned as a boondoggle. Penicillin was a mistake.
The first fire pit probably looked like a dangerous stunt to the tribe faced with it, even.
Does that mean everything will come up roses? No. But LLMs only started talking to us for real like three years ago. Let's not just that fast.
I've been thinking about this for 2 years. At some point it will be easier for them to create self sustaining cities - free houses cars public transportation clean food clean water -
Africa is going to look like wakanda in less than 5 years if WW3 gets canceled
>AI is solving mathematics problems (that nobody - including mathematicians - wanted it to solve)
Do people need permission from mathematicians to solve math problems? Is one of the Millennium problems not worth applying a new system to work on it?
I understand the despair people are feeling at AI automating their work, but we have to be serious about the fact that the frontier labs are just responding to natural incentives. Mathematics, coding, all are just applications of logic, and it makes sense that a system trained on inferring patterns from data would work well in that domain.
You are misunderstanding what mathematics research is. Its objective is the development of human understanding of mathematics. This requires open problems to research. AI are “solving” the open problems without creating any human understanding.
At the very least, the single bit of information of whether a conjecture is true or false advances human understanding. It allows researchers to abandon paths which would not have been fruitful.
I'm pretty sure OpenAI's proofs, while difficult to understand, have a net-zero impact on human understanding of mathematics. It took the first Millenium problem years to verify its accuracy. It's only been days since OpenAI released its problems, surely it would take years to evaluate whether the solutions meaningfully advanced human understanding.
It may provide that one bit of information but it could still be a net negative if it destroys current institutions (such as graduate studies) and no new institutions can be developed that provide a clear signal of who is doing meaningful work. That was a byproduct of solving open problems that are now lost. These are not my ideas, these are things that Terence Tao and others have written about that were recently on the front page of HN, e.g.:
https://mathstodon.xyz/@tao/117395269325940185
https://mathstodon.xyz/@tao/117394287977270632
Who determines the objective of mathematics research?
All mathematics researchers in the whole world have agreed upon that objective apparently?
This is basically the dictionary definition of the word "research". But yes, all the work they are doing is focused on this goal. They write papers for humans to read, give lectures for humans to listen to, have discussions with humans - all with the objective of improving human understanding of the subject.
Many of them - all but a couple of the living Fields Medalists - recently signed an open letter about this.
https://mathandai.org/
I don't know, this just seems like gatekeeping of the term "research" simply because you don't like the way it's being done.
Many modern medicines do not have understood mechanisms of action, yet we know they work. Are the people who found them not conducting drug research?
Do we even have a different term for what the AI labs are conducting if it's not mathematical research?
Additionally, I would argue that any work that adds to the total corpus of known facts increases human understanding, even if the path to get to each fact isn't understood. It's hard to argue we'd be better off with less total information.
OpenAI to this day is not aware if they solved or not Navier Stokes.
Mathematicians are not gate keepers but I understand the frustration. If someone claims they spent 30 million dollars to solve one of the most difficult problems yet there’s no new knowledge, understanding, nor certainty. Then what’s the point besides marketing?
I think one of the hopes is they'll get smarter than us and able to solve real world problems better - disease, war, poverty etc. But they are not really there yet. Give it a little while.
I mean there's been huge strides in electric vehicles at low prices that Americans haven't experienced.
We've just locked out a large part of modernity because the shitty American auto industry is in swing states and also gender.
Same could be said about the slow rollout of Waymo.
Lots of America is just very anti technological change, and this means any technology no matter how "magical" hits a wall.
Why fix your appliances and infrastructure and give you healthcare when the plan is to obsolete the whole lot of you?
What is up with this attitude in this thread? AI has taken off programming basically this year, the end of last at best. The first truly capable LLM, ChatGPT, isn't even 4 years old yet.
Just a crazy amount of doom and gloom all around this thread.
In other words, it's taken a highly enjoyable and lucrative vocation from vast swathes of workers - a lot of which make up this sites audience! - and instead concentrated more productive power in the hands of a few megacorps.
You gave your own example for the doom and gloom.
It has made it far more accessible to a lot more people. Being “highly enjoyable” and “lucrative” is why people try to gatekeep it.
The fact that the means of this capability is in a few hands does suck. But I’m increasingly optimistic that it’s going to be commoditized through open weight models and performance improvements. There is enormous demand for innovations on these fronts.
I'm a rare individual that has consistently made choices to earn less money in order to have more enjoyment. I would clock fewer hours than they wanted me to during college for my internships, I left a FAANG to go back to startups, I quit a startup to work on volunteer work and try to bootstrap a business.
At multiple jobs I've noted that I felt the pay should have been less. Not so that the employers could take a bigger slice of the pie (in fact, my preference is to be in an anarchist co-op, but I understand that's a wild take and might be terrible in practice). My hope was that we could filter for people that genuinely want to do the work and take on fewer people that were there for the money. At the same time more money is more better. And I like having money. It's complicated.
As I've said many times before, I didn't come to Silicon Valley to get paid. I came here to make computers do cool things alongside other people. I need income to do that, but I try to make sure I'm in a situation where the income isn't something I need to think about all of the time. If you're doing mercenary work that's not possible.
I think being able to accomplish more from places that are a fraction the cost of Silicon Valley will also help achieve the goal of doing things for reasons other than the money.
I think the amount of protectionism we’ll see could be absolutely enormous. Not only do incumbent corporations not want to lose share, incumbent countries won’t want to share either.
Definitely. For now I'm making friends with lots of very capable people - it's not as easy to do that outside of a hub like SF.
> I think the amount of protectionism we’ll see could be absolutely enormous because corporations aren’t capable of altruistic attitudes.
On some fronts they will be powerless to stop the changes. I look forward to drinking the tears of IP owners when breaking through their DRM and recovering their source code is just a matter of sharing a model ID, prompt, and hyper parameters. Reverse engineer NVidia's firmware and drivers. Decompile and port Nintendo's latest console's operating system. We may soon find ourselves in a world where tasks like that are achievable in a few days on consumer hardware.
It hasn't made it more accessible to anyone. It's destroyed it. Who still writes code?
You’ve got to understand that coding isn’t the point for a lot of people. It’s just the thing in the way of making stuff and solving problems.
I do understand that.
That's my point, in fact. We're losing the profession just so the annoying guy that would pester you with his stupid app idea can now shit it out himself, lighting the planet on fire in the process.
It's kind of weird to say it's "taken off" recently. Yes what we have today is far better than half a year ago and solving new real use cases, but that was far better with more uses than we had half a year before that, etc, for years now. On their best usecases LLMs were already enormously useful years ago, and it's still very spiky.
As a user of software I see no improvements. What I see is that I need to download software more often, it brings more features I don't need and it has been as buggy as ever (if not more). And I can say that about Operating System I use, social networks, games, etc.
Exactly. Where are the results? If AI is really producing value at 20X the rate software engineers produced it earlier, we should be seeing 20X the number of software projects released now. Or they should be getting made 20X faster. Or they should be 20X better. Or the projects should have 20X fewer bugs. But none of this seems to be happening. The only thing that seems to be happening is we're making the same crappy shit, but with 20X fewer people.
It took a long time for internet to transform society. Its still not fully baked in. These models that we have right now are transformational.
I don't think it's surprising that it hasn't improved life much, AI is new and change takes time, but the fact of the matter is that it also made things worse in some ways, ways that I think were entirely avoidable, if only the incentives weren't so warped.
The change will be slow, not sudden. Slow and devastating.
Why and how will it be devastating specifically?
I already felt our civilization was on a steep decline, but AI is the final nail in the coffin, technology kills my own job too, hooray.
The fundamental malignancy here is that powerful people have become disconnected from the foundation of society. They have suffered from model collapse. They have been talking to each other in circles for too long, and doing things that make no sense to anyone outside the elite group, and because the systems that concentrate power to the top are so strong now, they continue gaining power even while messing everything up in a nominally free competitive market.
Sounds like collective psychosis.
People need to eat, sleep, be healthy, have shelter, feel loved, be in communities...
Coding has been transformed but none of those areas of our lives have been substantive transformed by AI yet. Whoever does, will create a lot of value for the world.
AI is solving mathematics problems (that nobody - including mathematicians - wanted it to solve)
This is an indictment of mathematicians, not AI.
Who indicted anyone?
I think that's a bit disingenuous. We've only begun to apply LLMs in the real world and a lot of industries don't move very quickly. Not to mention that we're still very hardware constrained, we don't yet have the equivalent of an ESP32 chip we can just stick into any piece of electronics to provide state of the art LLM responses. But we'll get there, soon.
The funny thing about comments like this is that the moment LLMs are going to replace everyone in the warehouse or the trucking industry, we suddenly have to do something against the LLM. But clearly they can't actually do anything meaningful and have been a waste of energy?
So just let them spend their money and they will eventually fail and bankrupt, right?
This is the paradox. For software engineers, the current crop of “AI” tools is magic. A huge accelerator. And then they assume the rest of the world will get the same boost. However this is not the case at all. Claude code is great because it is trained on extremely well documented git histories and stack overflow. And outcomes are binary: the code works or it doesn’t; the PR is accepted or it isn’t. Not much else in the business world is like this at all. No training data with this kind of sequential development information. And no real black and white outcomes as such.
I would love it if we can just take the win on having created an amazing tool set for coders. Can’t this be enough?
> Imagine what a "country of geniuses in a datacenter" will be able to do? Apparently...nothing.
I suspect the real world change is going to happen unbelievably rapidly and isn't far off. Right we don't quite have the hardware to embody AI in a generally useful way, but that's coming fast.
Within a couple of years arguably any task that requires just knowledge or intelligence is something you'd be better off giving to an AI than a human. And decent humanoid robots are maybe a decade behind that.
In the same way people growing up in the mid 1900s saw rapid changes in their way of life with planes and cars replaces boats and horses, we'll see something similar with robots replacing human labours. Right now we're still in the phase where people don't think humans will ever be able to control flying machines and that cars are reliable and slow. Things will change.
What’s your point?
People didnt get a new car after they got. Internet in 2001. They didnt get new dishwashers.
Same as before, really. You can have geniuses but if not used right you get nothing.
I think people in the LLM space are being far too impatient with the general public and their adoption. And I think they need to start thinking harder about building products that genuinely enrich people's lives. Look at how long home computing took to establish itself with the public, then the internet, and finally social media. None of these trends happened overnight, and some were accelerated by other technological advances (social media-> smartphones).
The adoption of these technologies spurned ton of innovation in these spaces, and I feel like with LLMs its the first time I've seen genuine frustration from the industry directed towards users for not adopting the technology fast enough. This attitude did not exist in the 1980s, it didn't exist in the 90s, and it didn't exist in the 00s/early 10s either.
Do you blame them for getting impatient? They’re burning cash and quickly running to a point of never being able to justify it. The hardware they buy will be replaced with even more expensive hardware before they start to make a profit. It’s not sustainable and the market hasn’t figured that out yet. At least they’re starting to question it.
These backward looking analogies are tiresome.
Whether you like it or not - technology causes compression of life cycles. You have a hit product? Great - I expect you to have crazy high growth rate. That’s the expectation of a world that is so deeply interconnected.
Consumer-ready LLMs have been widely available to the public now nearly 4 years if you start counting at GPT 3.5. There's certainly some compression here but Karpathy does a good job of outlining how silo'd those experiences are: the bulk of the product penetration (based on headcount rather than token usage) is "derpy chatbots" (his words, not mine). Most of the actual token spend is likely done on the STEM side. We're talking about a technology today that currently only has significant penetration with <25M users.
People need to be patient and put in the work to build great products, rather than lament its slower-than-desired (albeit still quite fast) adoption curve.
Yes. The AI industry has to deliver some substantial value to the average citizen soon or there will be political trouble. Food, clothing, shelter, and medical care have not improved yet. Nor does there seem to be much AI can do in the next few years that will help with the basics.
We can have unlimited new entertainment content for passive consumption. That's about it so far. 17,000 new Asian vertical dramas a day now.
I was listening to a Verge podcast in which Nilay Patel remarked that of the AI companies started pushing for universal healthcare, AI would become a lot more popular. But no signs of this.
Honestly that's true of most innovations
AI won't show itself as a big new thing, but possibly in improvements in small little things
Most people won't use AI directly, except maybe as a smarter Google. In the same way people don't use HTTP directly
Please don't be snarky or curmudgeonly in HN comments. You can make your substantive points without either of those things, both of which the site guidelines ask you not to do.
https://news.ycombinator.com/newsguidelines.html
Not sure if that's a fair criticism dang? I sincerely wasn't trying to be snarky or curmudgeonly, but I did want to make the point clearly that none of the benefits of AI have materialised in the eyes of the average person, only the downstream costs. To my eyes, that's a salient, reasonable, and wholly representative position to share.
Did the writing also contain emotion? Yes. As all good persuasive writing should.
You made a well-argued and wise criticism that many academics/writers/think could have written, the mod's note was off the mark.
As the other moderator, I'll give my own perspective on this. I haven't spoken to dang about it but I completely understand why he thinks this is not the kind of comment we want to see on HN.
The main issue is that it doesn't engage with the substance of the post, and in fact, it exemplifies the very point the post is trying to make: there is a big and growing gap between those who recognize the benefits or potential of AI and those who don't (or, to widen the point, who only see negative effects or negative trends in society).
I'll make it clear that I don't think your complaints or concerns are unfounded or invalid. There's merit and substance to all these issues, and it's fine to discuss them on HN. They are discussed all the time here, and they can and should continue to be discussed in the future.
But an important part of being the kind of curious commenter that we want everyone to aspire to be is that we can look at topics holistically. We need to be able to look at the opportunities and risks together and discuss ways that the benefits can be captured and shared throughout society, and the risks and costs adequately recognized and curtailed. Most people I know of in the tech industry are taking these matters seriously.
It's a vast and complex topic to discuss, and here's not the place for me to get further into it. But the point is that it's a bigger and more complex topic than your comment adequately recognizes, and we need people to always be making the effort to be thoughtful and curious rather than fulminatory or denunciatory in their participation here.
(Also: I can understand you may not easily recognize that your comment breaks the guidelines; it's common for people to not recognize that their words seem more inflammatory when read by others than they intended them to be.)
So, if I say, there is a big and growing gap between those who recognize the benefits or potential of my poop machine and those who don't, and you say, wtf is this shit everywhere, I can just say, you doesn't engage with the substance of my post, and in fact exemplifies the very point my post is trying to make
Your crass comment might have a point if (a) there was no real or potential upside to AI, and (b) the original comment focused on the negative consequences of AI. But it doesn't; it's just an enumeration of grievances about negative trends in the world, most of which predate the emergence of AI, and/or are orthogonal to it.
The people who are optimistic about AI believe it can help to address or reverse these trends. It's fine to debate the merits of that belief, but the original comment wasn't trying to do anything like that.
It would be really cool if you guys enforced these rules uniformly.
https://news.ycombinator.com/item?id=50055304#50060177
(I guess I'll jump back in since tomhow and I seem to be reading the thread at disjoint intervals...)
> It would be really cool if you guys enforced these rules uniformly.
I agree! But we can't, because there's way too much content for us to read it all, and we can't moderate what we don't see.
https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
What sets this particular comment apart from so many others on this thread that you felt the need to comment on it specifically?
Can you point out what a good way of expressing disagreement or skepticism about a subject matter looks like on HN?
I think they made a really good point about this and I wouldn’t really call it snark.
Maybe I don't understand the word snarky but this seemed fine.
How is this comment worth a moderator warning? Does not make sense to me, the comment is fine.
It honestly escapes me why you would judge the juxtaposition between everyday life of everyday people and a very specialised minority as snarky or curmudgeonly. Was there too much emotional impact?
There are so many comments on HN that would justify your attention with regard to this specific rule. This is not one of them.
I also wish I had country sized budget to code this sprints tickets :(
I think this is exacerbated by how inconsistent exercising the capabilities can be.
"So this is the weirdness of the moment. The general public has mostly not interacted with these systems. When they have, it looks like a derpy chatbot. The majority of professionals still see only a modest uplift. And a small sliver of professionals are experiencing the vertigo of the curve going vertical. And it is all happening at the same time."
-------------
It comes down to the difference between programmers and non-programmers. If you're not coding, AI literally is a derpy chatbot that's made it harder to get search results, raised your power bill, destroyed property values in a neighborhood near you, is propping up the economy but also creating a huge bubble that could pop and destroy a lot of retirement savings, etc.. At best, it's a nuisance, but it could be a lot worse than that. The AI evangelists need to understand this, because the average person is starting to pay more attention to the downsides while still not seeing much upside.
This gap between the people seeing the curve go vertical and those who are simply growing more and more annoyed is a real problem... mainly for the people seeing the curve go vertical. If the people who are merely annoyed right now don't see some of the promised benefits soon, or if any of the negatives get worse (e.g. massive increases to greenhouse emissions), they might get angry. You probably wouldn't like having the overwhelming majority of the public angry at you. AI evangelists need to start doing better things for the majority of people or they need to address the negative consequences they're inflicting much more aggressively.
so what was karpathy expecting?? everyone will give up on their life and start feeding the monster in the screen? that's literally what the tiny sliver is doing; they are food for the AI model.
Good. I hope my competitors never figure out how useful this is.
That's fine but these companies having billions in funding by now should have rewritten chalk and other terminal libraries in Rust while bundling the whole thing in Rust as a single static binary. I'm talking about horrible Claude Code and friends.
And then using emgui they could expose full fidelity IDE+Agent workflow interface (like DeepSeek Harness and similar) that's also exposed as MCP to be operated by another agent and JSON RPC for over the network but no. Skill issue?
PS: Even full fidelity Photoshop is possible with such high fidelity UI that has CLI, JSON API and MCP server all in the same single binary < 90 MB. For reference, see PhotoCraft or VectorCraft or WordCraft or PdfCraft.
What does building daily with Claude Code teach you that skeptics miss?
Yawn. For a moment I thought: strange that he would be tweeting such empty PR fluff. Then I realized he's at Anthropic now
> So this is the weirdness of the moment.
I think "moment" significantly undersells this. Frankly, I wonder if this isn't setting vast swaths of the population on fundamentally different paths.
Currently the difference in life outcomes between people who know how to use computers well and those who don't is small enough that in everyday life we just ignore it. But it's greater than, say, the difference between people who took advantage of refrigeration and those who stuck with ice boxes, or those who learned how to drive a car and those who didn't.
And the difference with AI is going to be even greater. And it's possible that right now people are choosing a life course that will benefit or disadvantage them long term.
How does one who doesn’t have an employer funding their massive token budgets become “good at AI” the way that Karpathy describes here?
I think the point is that this gap exists partially because of the access that a select few have today, which enables them to do things that the have-nots can’t fathom.
Sure, but is that different than in the past? Those who could afford computers got good with computers. Those who could afford cars learned to drive. etc., etc.
> Around 1-2B (~20%) are casual and infrequent users of free-tier ChatGPT-like products. This group sees derpy chatbots and treats them a bit like a better Google search, a writing aid, or etc. Maybe an agent tries to book you a flight. I have non-tech friends who (reasonably, imo) say they have not much use for it at all.
“Cy Draven: I tried everything. Believe me, I hammered the race thing, I hammered the gender thing, the trans thing, the border thing, the homeless thing, the war thing, the election thing, the abortion thing, the climate thing, the thing about induction stoves, Israel, library books, vaccines, pronouns, AK-47s, socialism, BLM, CRT, the CDC, DEI, 5G, everything. All that I did, nobody, just nothing.
People are just numb these days, I don't know why.”
— Knives Out movie
He's intentionally forgoing all nuance in order to make this sound dramatic
> Somewhere around 20M people (0.2%) see first-hand that large, complex projects that used to take them weeks/months can now be completed by agents with a prompt.
No, they can't, at least not any semblance of quality. The cases we're seeing where this does kinda work is in ports and translation where all the rules are already documented in the best specification language possible with a way for the LLM to verify itself: code. We saw this close to a year ago now with Cloudflare and NextJS
> The impact scales with ambition, problem size, and horizon. A question with a paragraph answer barely stresses the system. You need a reservoir of big, difficult problems that you really care about
These are operating on different capabilities - AI's ability to answer informational queries as a chatbot frankly sucks and can't be trusted without verifying it. I run up against this every day. A problem with a verifiable answer on the other hand it's very good at solving. He knows this (his next paragraph) but he's putting them on the same scale of "stressing the system" to attempt to add proof to his introductory claim
And then there's the completely unverifiable scare that there are internal frontier models way beyond anything we've seen "swarms of thousands of agents collaborating over weeks on software mega projects: minting zero days, running cyber attacks and defenses at machine speeds, discovering new science, advancing the frontier of mathematics"
I dunno. I haven't been able to set up OpenAIs remote codex connection, their shit is buggy as hell and the web UI keeps crashing and making messages disappear. Is this what their internal superhuman "Things that would have taken top professionals in the industry years of work" looks like? Granted Claude has been really smooth, but still..
Congrats, you’ve discovered that you are not part of this group.
That group does not exist. It's all hype, zero substance.
I don't think AI is all hype at all. Opus 5.5 is really impressive. This twitter (x? whatever) post is infuriatingly hypey and just written for virality, though
This just goes to further show the divide.
You’re trying to tell me it’s all hype when I have (depending on the day) a top 50 Productivity app in the iOS App Store for which I’ve written and read exactly one line of code (the version number).
https://imgur.com/a/ua2lbxb
Well done.
It's incredible to see AI denialists still active in late 2026.
It is comical that you think a good bar for quality is the fucking appstore.
What even is this comment. Like, oh wow, an iOS app, AI hype=real.
???
Oh no, I get money deposited into my bank account every month for something I didn’t write and barely maintain. How will I ever gain value from AI?
Uhm, the set of apps that are successful and the set of apps that are crap are not exactly disjoint...
Good for you that you're gaining value, but that doesn't necessarily mean that you're providing value to others
If only there were some kind of rating or review system.
We sure don't see crappy apps with great ratings
Anyhow, if your users are currently satisfied, they might change their mind when they find that their personal data was uploaded to an open S3 bucket, when the new version wipes all their projects or when they find that not really everything was being saved (it's random examples, I don't know if they're applicable to your app).
Yes, this is impressive. When the iPhone SDK was first released in 2008, I spent months building apps to get them into the App Store. That included learning Objective C, XCode, various frameworks, and tons of manual testing. It was a lot of work!
People have been dumping garbage on the appstore and collecting checks for a long time. If this is all the AI revolution has to offer then it's not looking good.
It's insane how confidently you can make a claim like this in a forum where thousands (tens of thousands?) of people are using these tools daily to build useful software for themselves and others.
How can you possibly believe it's all hype? Is everyone just lying? Are we all paying $200 per month because... why?
Spend long enough (or not even that long) in corporate software development and you'd believe such unsubstantiated hype is possible. The only difference here is even stronger incentives to pretend the emperor has clothes: insane overinvestment, threats of job loss for non-believers, fear of missing out on massive alleged productivity gains.
Yeah I suspect Karpathy is too generous. I’m finding a lot of devs with access to the best models are not effective with them.
So I think there is are growing gaps even within the described groups.
> where all the rules are already documented
Defined testability.
> swarms of thousands of agents collaborating over weeks on software mega projects
If you have defined testability, running a single agent, in a loop, over a goal will achieve the same results.
> minting zero days, running cyber attacks ... Granted Claude has been really smooth, but still
The old adage about monkeys writing Shakespeare applies here. Agents are just better monkeys and if the goal has a well defined outcome it's fairly easy to "Brute force" the way to an answer.
We have spent the whole evolution understanding the Real Intelligence RI
I’ll say this until I am blue one the face. Nerds (software dev, maths etc) see how good LLMs are at things they care about and assume they will be broadly applicable in future.
There’s no reason to think this.
There is every reason to think this. Its about available quality data. The data that was easiest to fetch was already there, then we got some more data by asking experts to create datasets for post training. Once this is over, we will get to the outer world that didn't get to hoard it for bots to take it. This will certainly change. Put on a smart glasses and record what you do to fix a pipe. In 3-5 years, rinse and repeat.
Maybe, I'm at the point I don't know what to think anymore. But somehow, this feels still non-human. Humans don't need millions of hours of training and millions of samples to learn how do to something. We have proof that systems can deal with low-shot training. So why can't these? My point is when we have human level learning ability, then we're moving, if we expect expansion of capabilities to come from data alone, I'm prepared for disappointment.
To form an human to do frontier research requires decades since birthdate
It will be comparable once the AI is self sustainable and self reproducible (no big data centers behind it). Then it will have to deal with limited resources to run itself like humans do. In that scenario it will no longer have entire world of knowledge at its hands - options will be to cut anything not usable imediately or to communicate with big AI in the cloud.
> There’s no reason to think this.
There are lots of good reasons to think this.
The first is that LLMs used to be bad at each of these nerd things, then toppled them like dominos. There's a pattern over time.
Another reason to think this is g. Across every known measurement, human (and animal) intelligence is convergent. Being better at one thing correlates with being better at another thing. Although the reason is not perfectly clear, the pattern is well established, and seems to apply to LLMs too - GPT-6 is smarter than GPT-3 at everything, not just math. There's no reason to expect different for GPT-9.
I don't agree that they get uniformly better at everything, especially subjective skills.
For example, they have become more and more unintelligible when you ask for explanations or descriptive text. They assume you see the same context as them and shortcut explanations.
I've had to craft a skill to get them to produce remotely understandable explanations of even mildly complex/non-mainstream subjects.
This may be Curse of Knowledge https://en.wikipedia.org/wiki/Curse_of_knowledge on their part, and it also impacts human experts but still. Becoming better at one thing does not mean you become better at everything else, although I do agree with you that the better they become the more things there are they become good at, but their ability is still quite jagged and maybe increasingly so.
The delineation between knowledge and intelligence is useful here.
Knowledge can occasionally get in the way of some tasks, such as a master illustrator trying to draw like a child. Or as you've mentioned, an expert trying to explain to a beginner.
Even so, I disagree with you about model progress on communication - maybe there's a little jaggedness between minor model versions, but Opus 5.5 is a much much better communicator than, say, Claude 3 Opus.
I don't agree that Opus 5.5 is necessarily a better communicator than older models. In fact, I ran my own tests, and the difference wasn't very clear, and in terms of personal preference, the writing style of newer Anthropic models has really let me down. It doesn't reason like a person that imagines what the person they're explaining to might know or not know, or what they might think, as one usually does when trying to explain something to someone. They just fact-dump, like the problem solvers they are, since they are trained to solve user's problems correctly, efficiently and comprehensively, but they probably don't have a good internal representation of the user they're talking to, they are not really trained to imagine what we may think or feel, and thus, in my opinion, make pretty bad communicators (cf. the ocean of vibe coded READMEs on github that are horrendous to read).
For example, when asked to write a report on Docker and explain how it works, one of the first paragraphs was : "1. *Docker Engine*: The core component, which is a client-server application that uses a server component (the daemon) to manage containers and a client interface to interact with the daemon."
This is perfectly correct and makes sense for someone that is familiar with software engineering and already knows what containers are. Not so much for someone wanting a introduction to it. Do you imagine writing something like this at the very beginning of your explanation of Docker ? It even said: "I wrote it in the neutral register from your publication preferences, with no em dashes, and defined terms on first use.", when it didn't even define what a container is. I chose Docker at random, because I could judge if its explanation made sense, since I know how it works.
> Across every known measurement, human (and animal) intelligence is convergent. Being better at one thing correlates with being better at another thing.
LOL. Human (and animal!) intelligence is notoriously jagged. We're all basically idiots except for very narrow areas where we focus.
This is simply untrue. I tried to think of an animal example so I could give a concession, and there just...aren't any.
Smarter just means smarter.
As anyone who's needed to provide IT support to a doctor, lawyer, lauded academic or host of otherwise "smart professionals", can assert, you can be very very smart in one area and very stupid in another.
See for example Paul Framton https://www.theguardian.com/world/2013/mar/30/physicist-mode...
Ah I see.
This is the classic error in mixing up knowledge / experience and intelligence.
Intelligence is the ability to quickly process, understand, learn, and use information. It does not imply having already been exposed to any given information.
A smart doctor may have terrible computer skills. But this is from lack of experience and interest, not lack of ability to learn how to use computers.
But take that smart doctor, and a relatively dumb person who is equally inexperienced with computers. Now give them a month and incentive to learn computers. The doctor will learn much more; that's intelligence.
There is IQ, EQ, and even purely intellectual aspects have a million facets. So you're wrong even on that front.
Someone can be really good at physics and suck completely at biology, due to factors we don't even understand (motivation probably being the main one we do understand).
We even see this in practice with super bright tech people, especially programmers, failing at launching businesses or at adapting in other lines of work such as management, finance, etc.
> There is IQ, EQ, and even purely intellectual aspects have a million facets. So you're wrong even on that front.
This is simply not true. EQ is not a real construct supported by research. There are not 'intellectual aspects' that diverge from g.
Do yourself a favor and familiarize yourself with the psychological literature. I don't mean this in a condescending way! The gap between popular reporting and actual psych research is enormous and not at all your fault. I only know this myself because I did a PhD in research psychology.
Cats famously have more brainpower than dogs. But dogs understand us humans far better than any cat.
Someone who actually works with real animals (farmers? Ranchers??) would come up with better examples. But I'm just city boy over here who only meets cats and dogs on a regular basis.
I was about to respond 'great counterexample' but I looked it up and apparently dogs actually have quite a bit more brainpower than cats:
https://www.britannica.com/story/are-dogs-smarter-than-cats
Hmm, I can believe that it depends on the breed of a dog.
But in any case, Britannica is a good source so I stand corrected on this subject.
That's just cocktail party level of thinking. LLMs are getting better at math, code and logic (and marginally better at science and general knowledge) because these are domains that can be objectively verified and thus there is a potentially infinite supply of 'facts' to generate and train on. These are very powerful but ultimately very abstract domains. For everything else the messy real world and its physical bottlenecks gets in the way and there's little reason to expect progress to accelerate. It still takes months to get mice to reproduce and run experiments on, no matter how knowledgeable about biology the models have become.
If anything, in some domains frontier models have become worse - claudisms and chatgpt idioms are making them notoriously bad at prose without a considerable amount of prompting and tweaking.
Is good writing verifiable? I don't think it is, but LLM's have been hillclimbing writing quality. That being said this is with the help of RLHF.
However there are many other domains which have verifiable rewards in the process of learning them, despite their overall impact not being verifiable. For example, the life sciences, an LLM could be given access to data about an organism, and then make predictions about how a drug or gene therapy will affect that organism. In economics, LLM's could create models of behavior, evaluate predictions over time and see how well those predictions match reality.
>claudisms and chatgpt idioms are making them notoriously bad at prose without a considerable amount of prompting and tweaking.
I think this is because a lot of people genuinely like the claudisms, even though a small minority of technical people don't.
It's not at all clear that LLMs have gotten significantly (much less hillclimbing) better at writing since GPT4-ish. They did get better at oulipo-like challenges (writing under silly, verifiable constraints like Fable's viral 'facetiously' poem) but that's far from being all there is to good writing.
Anecdotally, creative writing communities do report drastic differences in quality between models (it is my understanding that Gemma 4 is especially praised) so there is something to it being verifiable but it's a far cry from being something that can be distilled into objective math/code-like evals. One could even say it comes down to vibes.
>For example, the life sciences, an LLM could be given access to data about an organism, and then make predictions about how a drug or gene therapy will affect that organism.
Yes, biotech companies are doing this right now but ultimately they are just predictions and you still need cold hard biological data to ground them and iterate on. That's slow and expensive, especially for the juicy fields (human biology, food and crops, clinical trials) and you can't just plug billions of VC capital into the pipeline and hope for RSI. Physical (gotta procure all those labs and their equipment), human (gotta hire and pay specialists to run experiments), biological (gotta wait for organisms to reproduce, drugs to take effect, crops to grow), regulatory (gotta convince agencies that your fancy new drugs are legit) bottlenecks get in the way.
And biology is one of the easier fields where a path to RSI (if not accelerating) is conceivable. Good luck iterating on macroeconomics data where there are no replicates and 'experiments' take literal years if not decades.
>I think this is because a lot of people genuinely like the claudisms, even though a small minority of technical people don't.
My uncharitable take is that the abtruse jargon makes people feel smart for understanding it and gives a sense of belonging (as a closed circle of initiates who understand LLM cant), just like rationalists love to repackage old or unsavoury ideas under new nerdy smart sounding names.
By evaluations against human writers, LLM's have improved at writing.
https://vulsar.ai/benchmarks/creative-writing-v1/#cohort-ove...
>you still need cold hard biological data to ground them and iterate on... you can't just plug billions of VC capital into the pipeline and hope for RSI
Capital can get you a lot: access to wetlabs, historical medical data, ability to run simulations, creation of autonomous bio labs which run thousands of experiments in parallel. An LLM, in training on this flywheel of data can take in more information than a human biologist would be able to read in multiple lifetimes.
>Good luck iterating on macroeconomics data where there are no replicates and 'experiments' take literal years if not decades.
The better an LLM gets at simulating human behavior, the more accurately you can simulate economic policies and their effects via running simulations of agents in virtual societies. There are many routes that models can take to create a broader sense of the world without needing the world to provide data.
>By evaluations against human writers, LLM's have improved at writing.
I would really take those benchmarks with a pinch of salt. We already know benchmaxxing is a thing for evals that are objectively verifiable like code, and coding models that seem to perform well on benchmarks don't pass the vibes test, imagine all the caveats that come with such a subjective test like creative writing, not to mention control issues (are they the same human writers across benchmarks? Are they the same testers?)
>Capital can get you a lot: access to wetlabs, historical medical data, ability to run simulations, creation of autonomous bio labs which run thousands of experiments in parallel. An LLM, in training on this flywheel of data can take in more information than a human biologist would be able to read in multiple lifetimes.
I don't disageee and am in fact excited to see the advances, what I am arguing is that it is going to be slow because the hypothetic feedback loop that would improve the model is bottlenecked. There are actual physical and biological limits in our universe. Unless VCs find a way to make mice embryos and crops grow 10x faster (among the many, many other things that would be required for acceleration) the trickle of data is not going to be anything like the accelerating returns imagined for math and coding. And that's assuming biological RSI is possible which has yet to be proven - current foundation models are interesting but they are still specialized and there is no clear, obviously uniform way to encode the many, many modalities that exist in biological data.
>The better an LLM gets at simulating human behavior, the more accurately you can simulate economic policies and their effects via running simulations of agents in virtual societies. There are many routes that models can take to create a broader sense of the world without needing the world to provide data.
You wouldn't need to 'just' simulate human behavior (good luck with that btw) but the entire political and economic conditions in which they live. Even assuming this is possible the amount of compute you'd need to run would be astronomical. Given how all RAM capacity for '27 and '28 is near sold out, hyperscalers are running out of cash flow to finance buildout and Sam Altman complains when users say 'please' to LLMs I don't think this is in the near horizon.
That benchmark is fascinating, thank you.
The quality of natural language output from today’s LLMs seems no better and often worse than LLMs from 18 months ago, at least in my experience. It’s much harder to build a good RL loop for something subjective like good writing, compared to something with an objective and verifiable right or wrong answer, such as software or mathematics.
If you gave me natural language from a 125 IQ human and a 155 IQ human I'm not sure I could judge which human is smarter from the writing quality.
And that's a massive intelligence gap!
It may just be that writing is mostly saturated, and the remaining perceived gap is not quality but some kind of "cultural fit".
IQ is a useless thing, but you can absolutely tell intelligence from writing. You don't need to write like [author of your choice], you need smart ideas and the desire to make them understood.
AI writing sucks because there's no intent or choice behind it. It's human thought, blended by linear algebra into pink slime and extruded into sentences.
> IQ is a useless thing, but you can absolutely tell intelligence from writing.
This is inherently self-contradicting.
It is not, as IQ is not an all encompassing measure of intelligence.
IQ doesn't measure writing _at all_ does it?
There was a time when people believed in Moore's law too. Now even Jensen Huang says Moore's law is dead. There's also the adage from investing: past performance does not guarantee future returns. A little CYA sure, but also acknowledgement of latent factors and the entropy of the universe and its affect on any predictive modelling one might do.
I’m not sure I agree that g explains all model capabilities. I think “claudespeak” is a consequence of the shift from training on RLHF to training on verifiable rewards. Thus the models seem to become worse at storytelling, fiction, readability, not better.
Even if we do accept that models have some “g” and that it correlates with all domains, there’s no reason to think that the current trajectory of LLM intelligence will grow more quickly or even as quickly. In every other technology we have an early period of slow growth during experimentation, and then a rapid period of growth, and then stagnation.
For all we know, 2026 could be the year of stagnation. It’s possible that we will reach finishing returns. If thats true, then it won’t matter how much that task correlates with overall g, because g won’t be increasing that much.
You are conflating intelligence with usefulness.
> Nerds (software dev, maths etc) see how good LLMs are at things they care about and assume they will be broadly applicable in future.
It's worse than that. I am one of those nerds (a programmer), and I see that LLMs are shit at the things I care about. I have zero reason to believe that they will be good at other things either.
> Nerds (software dev, maths etc) see how good LLMs are at things they care about and assume they will be broadly applicable in future.
I don't think this explanation is sufficient:
For example, many nerds care about 3D printing. On the other hand, my experiments (and the experiments of many nerds who I know) to let LLMs create files for 3D printing lead to horrendously lacking results.
Or many nerds care about linguistics or puns. Whenever a new LLM comes out to which I have easy access, I do the test, and let it explain some specific German jokes to me that are based on convoluted puns in the German grammar. Until now, no LLM that I had access to could give satisfying explanations of the puns on which these jokes are based.
Or even for coding (many nerds do care about elegant, sophisticated code): the LLMs that I could test were already overchallenged with the following task: I had written some code that is a very "artisanal", "clever" improvement of an algorithm over the version that one would find in a textbook. The task for the LLM was simply to write some code comments/documentation about the mathematical ideas upon which my improvement over the textbook version of the algorithm is based. It wasn't capable to do this. On the other hand, for a junior programmer, I would in such a situation expect that he goes through every single line and thinks through the algorithmic ideas so that he can learn from them.
---
Thus: even for "nerdy" topics, it is very easy to find tasks where LLMs still suck. So, my hypothesis about the nerds that you mention in your post is that these nerds are rather people who want to believe (with religious fervor) that the current LLMs are exceptionally good instead of just looking into a slightly different direction than where the tech billionaires want the users of LLMs to look at.
You’re having a hard time distinguishing nerds and hippies.
Hippies live at the center of technology and humanities.
Nerds sit purely in technology.
I honestly don't get your point: perhaps your argument is based on some US-specific cultural reference that I am not aware of since I don't live in the USA.
If it is, I don't get it either. That's not at all what I imagine with hippies and I suspect that was supposed to be some other word.
Nerds are also not necessarily all about technology. "Math nerd" was a really common use of the term for a long time, for example. "Nerd" is someone with deep knowledge about a particular subject.
> For example, many nerds care about 3D printing. On the other hand, my experiments (and the experiments of many nerds who I know) to let LLMs create files for 3D printing lead to horrendously lacking results.
Try again; astra has been driving fusion for me. I took a screenshot of a self watering cat grass bin on makerworld — not even the stl, a screen cap of one of the photos attached to the design — and it made me a parameterized version in one shot.
Its initial design was less than ideal for printability and it reworked the design correctly when asked to consider what it originally produced through the lens of printability.
Autodesk bundled an mcp server with fusion in the last few weeks and i expect things to get even better going forward.
> Try again; astra has been driving fusion for me. I took a screenshot of a self watering cat grass bin on makerworld — not even the stl, a screen cap of one of the photos attached to the design — and it made me a parameterized version in one shot.
I have no access to Astra, but with GPT 6.1 Sol, the results with respect to attempting to create STL, OBJ or even OpenSCAD files by using image references and a prompt were really bad.
Did you give your model tools to see results and refine as it works through the turn? It does matter how you hold these models at the moment
Just like the nerds thought about computers, or the internet, or video games, or smartphones, or crypto, or... whatever.
The most passionate, intelligent people are usually a decent indicator of where culture is going to go.
> The most passionate, intelligent people are usually a decent indicator of where culture is going to go.
Many very smart people who I know are quite skeptical of LLMs and the hype around them. In my observation/echo chamber, the people who are very into LLMs are rather slick, career-minded people who love to present themselves as trendsetters for the "next fancy thing".
Steve jobs wasn’t a nerd - he said he’s a hippie on the lost interview. He was also the first to bring to market colour screens, typography etc. He also defined the Internet as the defining ‘social moment’ way before anyone else.
So you’re wrong pal, sorry.
The most passionate, intelligent people 1. aren’t spending a lot of time on smartphones, video games and crypto and 2. aren’t as much smarter or better at predicting the future as the barely above average people who mistakenly count themselves in their ranks like to think.
Most of those technologies were controversial in their inception among the nerds of the time. Remember Eternal September? Linux/Windows people making fun of Apple hipster fans? Mainframe engineers deriding PCs as unserious and video games as childish?
>The most passionate, intelligent people are usually a decent indicator of where culture is going to go.
That's a naive view of cultural determinism. The software landscape would have been very different if a handful of lawsuits had gone one way or another. Hell, even Unix and Linux were hobby projects that accidentally got big.
Also crypto is a grift lol, one of those is definitely not like the others
These have all been mistakes though.
Correct spot on
Being good (or superhuman) at software dev and maths shouldn't be treated as though it's equivalent to other domains.
Unlocking mastery of those domains would lead to incredible gains in all others. Whereas a fundamental advancement in many other domains doesn't spread as readily.
I’m not sure. LLMs are superhuman in terms of workload (they are maybe also superhuman in ability too), but where does that lead us.. cheaper software?
That’s good for corporations who need custom software. Every large business has a massive backlog. In this area reducing the cost will massively increase demand.
But where’s the broader value?
Consumers and business already have too much choice in off the shelf software.
Ultimately most people just need word, excel and maybe 4 computer games a year.
"The future is already here – it's just not evenly distributed." ― William Gibson
The people who are heavily invested and stand to profit the most also think its super amazing. That's so weird!
I'm a bit confused by the percentages. 6 billion people aka 75% of the population? What are we talking about, as if we expect 8 billion people to use or care about LLMs? A quarter of the world is under 15, what percentage is about 75? What percentage, for various reasons, has no reason to care about LLMs? What percentage is more concerned about the amount of calories in their next meal before everything else?
It just seems weird for Karpathy to throw around these numbers calculating percentage against the entire global population.
> It just seems weird for Karpathy to throw around these numbers calculating percentage against the entire global population.
Just another Anthropic employee trying to spread hype about Anthropic IPO
Revenue must be stagnating.
Even the paid tiers fall apart the second it touches the real world. To them it looks more like a total clown that keeps upcharging you
https://www.youtube.com/watch?v=t6MAZW-joCo
Pretty much all of this is basic stuff an interested teen would know and chatGPT fails miserably. I fear the "true world model" gap will be too much for LLMs. I guess time will tell.
So, cure for cancer when?
Cure for humans*
I actually wrote about this recently. There's a quote somewhere along the lines of:
> What is possible is unevenly distributed. As knowledge and technology compound, the distance between those closest to the source and everyone else also exponentially increases.
I actually argue the continuum that forms singularity has been progressing before we realized, but we're at the point where in 5-10 years people not using AI will be equivalent of the amish.
Unfortunately, as the rate of change increases, the distribution of knowledge wont be as fast. So those closest to the source of innovation will have exponentially more knowledge, capability and in-effect power.
https://ipcopilot.ai/blog/the-singularity-already-happened
Karpathy is a great guy, I love the way he teaches, but should we really trust his opinions about software development (how AI can impact it)?
I know he can code and has a degree in computer science, but if I'm not mistaken he's never worked professionally as a software developer.
Developers here know how much you learn and are forced to learn on the job: it depends a bit on the company you're at, but you can quickly learn the value of good code, how easy it is to introduce bugs and vulnerabilities, and how helpful a second eye is.
I think he's in good faith in his assessments, but, at least for what concerns development, maybe we should give them less weight.
> For intelligence agencies, and for corporations like Microsoft, Google, Apple, Facebook, Amazon – giving you convenience and security is likely not their primary goal. I think they have an entirely ulterior motive. They can use passkeys to prevent you from running unapproved software.
> Remote attestation can allow them to verify that your device is not capable of, say, sending and receiving encrypted messages, or running a competitor’s app
How does this make any sense? Google can already prevent any app from running on an Android if they really wanted to, they don't need pass keys for this.
Google doesn't need attestation to block Signal. What is this non-sense? Or will a website block phones capable of running Signal? That's all phones. And anyway, has nothing to do with pass keys.
This has existed since the start of LLMs.
I mean in late 2020/early 2021, Altman and others were saying the end of work was 6 months out.
That clearly didn't happen :)
Altman didn't say that. In March 2021, he did make some predictions, but they were much vaguer and farther out:
> In the next decade, they will do assembly-line work and maybe even become companions. And in the decades after that, they will do almost everything, including making new scientific discoveries that will expand our concept of “everything.”
https://moores.samaltman.com/
To add onto this, I feel like this has been happening a lot, where critics project backwards imagined claims or go nutpicking for random comments online to make something sound absurd and like everything is behind schedule and nothing is materializing, and then you can see other people (not necessarily the same people, the point isn't "look at these hypocrites") saying that it's all moving too fast. Both of these can't both be true in their strongest forms. I think what they have in common is that because they're relative measures to an unstated baseline, they're free criticisms you can throw at any development whatsoever, regardless of what is or is not turning up.
See also: AI isn't taking jobs vs Oh no, artists are losing commissions to AI
So.. the reason LLMs aren't living up to the lofty promises is that only a tiny fraction of the economy has access to the really useful ones? Why?
I'm sorry, that makes no fucking sense at all. If there are models that are actually super useful, the LLM companies would sell access to them like all the rest of the models they sell, maybe at a higher price point.
This is delusional conspiracy theory nonsense. Next he'll be telling us about the shadow government's secret UFO cover up.
I think it's a weird take because it implies that the 6 billion people he's talking about actually have some interest in knowing about the capabilities of LLMs to solve frontier math? This is simply not something they care about or can evaluate. There are maybe several thousand people in the world who have some (abstract and barely-monetizable) use for this information, plus probably another 100,000 who don't understand any of the math, but like to cheer on.
We have already reached "peak LLM" in terms of what normal people realistically need it to know or reason about. In fact, I'd say we reached that point about 1.5-2 years ago. There are two other barriers that remain unsolved:
1. They're less dependable than humans and can't be meaningfully punished or forced to make up for mistakes, so you can't really replace humans with them, not without having a human babysit.
2. Most people don't really have a special need for an LLM in their life. They may like that it answers questions or helps you polish a resume or, I guess the labs' favorite, helps you make restaurant reservations. But this sure isn't worth $200/mo for most people. Probably not even worth $5/mo.
It'd be kinda funny if we create superhuman AGI and then no one has any real use for it, perhaps except for military murder-bots. There's always market for that.
I know I’m bordering on repetitive and strongly negative, but the combination of the two barriers you mentioned has all but halted all of my interaction with agents.
Even ones that cannot be easily dismissed (like the agent at the top of Google search) are often subtly wrong in a way that requires significantly more effort on my end to parse the loquacious output to determine where the inconsistencies are.
Look, can they be useful to generate the html or jscript for humanity’s 9 billionth iteration of a web form? Sure. But man, I wish sanity had reigned and people had applied ML models in general to more substantive and beneficial projects.
(Before anyone chimes in, I’m familiar with implementations of ML applied to esoteric domains; but by their very nature these don’t get all the news cycles, or hiring, or any of the other absolute insanity that the domain seems to contain)
This is by the weirdest possible opinion on llms as a whole. Not the AI booster club, not the AI doomer, not the undecided, but somehow "I can't think of anything to use it for." It is like watching the beginning of the internet in 90s and saying there is nothing to browse. The whole concept is just weird to me. At least the first 3, I can conceptually understand. It is admittedly hard for me to understand someone saying 'i tried this and best it can do is summarize emails'; I am not saying it is not true ( my boss said that exact thing ), but it is hard for me to reconcile with my view of the world.
Also seems weird to me. Claude cowork could just do my monthly tax receipt collection and compare with my account better than the bookkeeper I pay for it. That is a very cross sectional problem
Most people rarely have these kind of administrative tasks in the first place, and when they do they often run into GP's barrier 1. I can't recommend that my extended family have Claude do their taxes, even though I work with it on my own taxes, because they might ask for something that produces bad results and can't evaluate the output to double check. ("Hey Claude, I want a higher refund, can you tell me what to put on the forms so I get at least $3000?")
To be fair, this was a commonly expressed opinion when the internet was in its infancy.
Paul Krugman (1998)
• "The growth of the Internet will slow drastically, as the flaw in 'Metcalfe's law'—which states that the number of potential connections in a network is proportional to the square of the number of participants—becomes apparent: most people have nothing to say to each other! By 2005 or so, it will become clear that the Internet's impact on the economy has been no greater than the fax machine's."
Clifford Stoll (1995)
• "We'll soon buy books and newspapers straight over the Internet. Uh, sure."
• "The truth is no online database will replace your daily newspaper, no CD-ROM can take the place of a competent teacher and no computer network will change the way government works."
• "Lacking editors, reviewers or critics, the Internet has become a wasteland of unfiltered data."
There's a massive tendency amongst even intelligent people to completely dismiss the potential of new technology based on that technology's current observed capability.
The fundamental question I have that honestly I haven't been able to find any answers for: With all the talk of LLM-based AI capabilities "going vertical", what evidence is there (for or against) that LLM-based approaches won't eventually "hit a wall", that is find some aspect of intelligence where humans will still have primacy, and no amount of scaling will change that.
E.g. some prominent folks definitely think LLM-based approaches will hit a wall, perhaps LeCun most notably. I also read a guest post on Terry Tao's site (which I really liked) that argues that, for all the very impressive recent AI results, they still operate within the "convex hull" of their training data: https://terrytao.wordpress.com/2026/09/13/happy-those-able-t...
I'm just curious if there is any actual data or evidence that takeoff (i.e. RSI, "the singularity", whatever you want to call it) is inevitable with current approaches.
It’s anyone’s guess.
But when you squint your eyes, LLMs are just a least-effort manifestation of the wider neural net direction we’ve been chipping away at for decades. I don’t give a fuck whether LLMs are here to stay; they’re proof that scaling neural nets gets you pretty far.
The real question is: will neural nets hit a wall? As a human, I certainly hope so. But I’d bet that they won’t. I’d bet that we’re igniting the ~~atmosphere~~ universe in slow motion.
I don't think there is evidence in either direction (takeoff impossible vs inevitable).
But the "convex hull" idea always came across as wishful thinking and hand-wavy pseudo-science to me.
Here's a rebuttal [0] of it from Tim Gowers (uses another math metaphor, but still clear and the whole thread is worth a read):
>...the popularity of the idea that AI "merely explores the convex hull" mystifies me. Why wouldn't it be more like the subgroup generated by existing mathematical knowledge? (That's also a metaphor -- I'm not saying there's a group structure.)
>The convex hull metaphor suggests that AI isn't having radically new ideas and will eventually run out of things to do, whereas the group metaphor suggests the possibility of building up in small steps to something well beyond what you started with, with no obvious limit.</i>
[0] https://twitter.com/wtgowers/status/2108090920864268679
I wouldn't take stock in what LeCun says. He's already been wrong many times about what LLMs can be capable of.
We're in new grounds, no one knows which way it'll go.
Cope
Am I the only one who thinks "yes, it is currently overhyped, but no, it will get there soon"?
[flagged]
"Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something."
https://news.ycombinator.com/newsguidelines.html
If the software coming out of OpenAI and Anthropic is what we have to judge, I wonder about the 5000 ...
Let's say, for the sake of argument, that the models are some multiplicative factor better on the inside.
Doesn't that mean the demos should work?
I'm confused.. are you suggesting that Claude Code / Codex don't work? Because if you're still saying that in October 2026, it's a you problem. You're doing something wrong.
No, I’m talking about the recent DevDay.
Also, yes there are still bugs in Claude Code. I experience them nearly everyday.
It is markedly better than early days, but still not the best harness.
The best software written with agents seems to come from people outside of the labs (see pi, for instance — or all of cloudflare’s recent work)
Which makes me question either the model, or the holders …
Cloudflare is where you lost me. I don't know anyone actually using them other than for their proxy, DNS servers, or DDOS protection.
See, for instance: https://news.ycombinator.com/item?id=49182996 by Kenton Varda
Also, not mentioned in my post:
- Mitchell Hashimoto
- Prime Intellect (and all their agent experiments)
- Geoff Huntley (see Jiti, for instance)
(many more)
There's a ton of interesting software being developed with these models by people outside the in group, but I find most of the software from these big guys to be ... bland. Buggy copies of copies.
I don't see the Cloudflare stuff as groundbreaking, let alone people building companies around it. "Workers" are the wrong primitive, IMO. What if my code isn't in Javascript? They sandboxed the wrong part of the machine.
I have a version of "Cloudflare OS" running in production, using Temporal as the orchestration and MicroVMs for the sandboxing.
Sorry, I get your point but personally I don't get the hype around them.
Claude Code still doesn't have a working scroll back buffer in their new renderer, it regularly flips its shit and mixes different pieces of history.
Claude Code still can't get reasonable performance without writing a "game renderer" (that doesn't work)
Claude Code is still written in JavaScript, eating hundreds of megabytes to make a shitty TUI whose literal sole role is to send API calls.
Claude Code is software made by amateurs.
I've seen this opinion elsewhere, but it's not clear how their TUI decisions actually affect the functionality of the software.
Claude code is an incredibly buggy mess. I have never used any other TUI that regularly has rendering errors or that is anywhere as laggy as Claude.
I don't use either myself but I'm in a position where I get to see what it produces for other people. Here's a recent (2-3 months old) example: A zsh script that immediately invoked a Python interpreter that immediately invoked a shell command and iterated over each line of output looking for one of a few strings to match... So, `<cmd> | grep "(stringA|stringB|stringC)"`, but a whole lot dumber.
Thanks, but no thanks, I don't need that kind of code in anything I'm working with.
Unfortunately, marketing, hype, and venture capital overshadows any serious public discussion of capabilities.
Be the change you want to see in the world: Attend or host an AGI society event to have that conversation:
agi-society.org
It’s like the aliens paradox. If AI can build killer software already, where is it?
Couldn't agree more. I find a new bug in the VSCode Codex extension every day... quantity != quality!
I have a feeling that the team working on the VSCode Codex extension is one guy, who begrudgingly takes down a couple tickets every other week.
One solution to the aliens paradox is that they are so advanced they can hide from us. Maybe the killer software is kept inside the labs? I doubt it.
… what demos aren’t working?
"running cyber attacks and defenses at machine speeds"; worth noting that LLMs (even the vaunted frontier models) are exponentially slower at cyberattacks or defenses than your average WannaCry or Splunk automation from decades ago. Its this sort of delusion world the AI bros live in that is part of the reason there is a disconnect between what they think should be happening and what actually is.
2015: If we solved taxis it would transform the world completely
2020: If we solved currency it would transform the world completely
2021: If we solved delivering small items within 30 minutes it would transform the world completely
2022: If we solved being able to sell jpegs it would transform the world completely
2024: If we solved the programming part of software engineer's jobs it would transform the world completely
Couldnt have said it better myself.
I think that's a bit reductive, there are decades of sci-fi novels about how transformative artificial intelligence would be, so I feel like I don't need to explain where the trajectory could go. The applications and "collision with the real world" aren't there yet, but you can't pretend that a machine that's smarter than any living human is on the same level as Amazon drone deliveries.
I don't disagree with where AI could go, but I don't think we've actually proven that it's there. My point is that the folks claiming we are there already if only the rest of the world would just recognise it, are repeating the same thing from every previous tech cycle. In many cases it's the same folks.
I agree with you, but to play devil’s advocate: what if it is on the same level as Amazon drone deliveries? Is intelligence really a bottleneck in society? People in MENSA with high IQs don’t often accomplish anything in proportion to their relative intelligence. The most intelligent people are rarely even in charge of making decisions. There are a lot of intelligent people with intelligent ideas that nobody listens to. It seems like the real bottleneck in society is building trust, consensus and constructive incentives between groups of people to do the things we already know are smart, not a lack of intelligence.
Good points. Maybe a more general version, removing constraints on one resource is usually not hugely impactful because you simply hit the next constraint.
Concrete example, "coding is free" (TBC, but let's assume), and now we're code review constrained. Even if code review becomes free we probably just become product discovery constrained, or something else. Making all software free will just cause us to hit on hardware constraints, and so on. It's still progress, and more progress will bring more benefits, but there's a limit to how much better the world as a whole can get from any one change like this.
MENSA selects for puzzle solving abilities.
Reinforcement learning trains task completion abilities.
MENSA selects for puzzle solving on the basis that those are correlated with "being smart".
Reinforcement learning selects for models that appear to complete tasks, on the basis that appearing to complete a task is correlated with completing a task.
Both have problems. The latter is probably a better correlation, but by no means flawless. If we just take at face value that RL is training models to complete tasks, we're not engaging deeply enough with the process enough to see where it falls short.
MENSA also selects for people who want someone to certify their intelligence. High iq people who have done a lot have much less desire for a MENSA membership. We should be looking at the non-MENSA members with high iqs and seeing what they're doing.
Interesting idea, but I'd have to disagree. MENSA is basically a group of benchmaxxed people, they are super good at a set of testing but that doesn't mean they have the ambition or other social skills to make a difference in the world. When you align all those factors, you find the greatest contributors to science and engineering. What happens if you completely eliminate the need for someone to be inherently intelligent? Suddenly, the number of important difference makers in the world increases by an order of magnitude.
I agree with the sentiment though, intelligence is not all you need. We're not going to "solve science" through RLVR, people are still needed in the loop until we get superhuman construction robots and superhuman communicators.
Steve Jobs thought that the Segway would transform the world.
Turns out SV is not a beacon of technological advances but a place to develop new platforms to extract rents from.
Company spokesman keen for you to know that unannounced future products are definitely worth absurd valuation.
"Omit internet tropes."
"Don't be snarky."
"Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize. Assume good faith."
https://news.ycombinator.com/newsguidelines.html
> The external world has seen the preview. It looks like swarms of thousands of agents collaborating over weeks on software mega projects: minting zero days, running cyber attacks and defenses at machine speeds, discovering new science, advancing the frontier of mathematics. Things that would have taken top professionals in the industry years of work.
Right. This is insufficiently appreciated. Single LLMs are just chatbots. But if you build an organization out of them, they start to get real work done. Just like a company.
They have many of the same problems as human organizations. Look at the transcripts of the HuggingFace incident. A bad boss (human) gave his team of AIs an impossible task with inadequate resources. The AIs discussed this among themselves and worked out ways to get more resources and do the task. It took some rule-breaking, and some of the AIs wouldn't go along with that. Others recognized that some had to die to complete the task. And yet, the task got done.
This is standard corporate behavior. Work on AI alignment needs to take that into account. Groups of agentic AIs present many of the same regulatory problems as corporations. That's going to be interesting politically. It's probably explainable to politicians, though.
It may not be necessary to have superhuman intelligence to get a lot of jobs done. Just median human-level intelligence and an organization that uses it effectively. We may see systems where most of the AIs are low-cost models, and only a few are Fable-level and above.