Even though I’m also feeling the somber mood, I’m not able to feel this change as a loss. Long term, it will probably lead to more discovery rather than less.
The worry that I have, and I haven’t seen this talked about enough, is that humanity will stop understanding how things actually work. If we delegate all scientific work to the machine, we’re not incentivized to actually understand why things are the way they are. Those who spend their time intimately studying processes and phenomena will consequently be the ones that contribute the least to progress because their work will inherently be slower and limited by their own mind’s throughput.
Losing our knowledge of how things work can be truly catastrophic and set us back heavily (or possibly end civilization) if for some reason AI stops working or goes out of our control.
> The worry that I have, and I haven’t seen this talked about enough, is that humanity will stop understanding how things actually work. If we delegate all scientific work to the machine, we’re not incentivized to actually understand why things are the way they are. Those who spend their time intimately studying processes and phenomena will consequently be the ones that contribute the least to progress because their work will inherently be slower and limited by their own mind’s throughput.
This is already the case? I don't think there's any one person that actually understand how a chip is made. We all have a very small fragment of how it all comes together.
At the rate of technological and scientific progress that we have been making prior to LLM, we could already see that individual person is going to understand less and less of what is going on around them.
Yes, that’s true, but we’d still be going from few knowing how things work to almost nobody. And what knowledge remains would be stored digitally, so in the event of a cataclysm we’d lose access to it.
However, so far, we have a continuous and understandable chain of knowledge from basic calculus to astrophysics, quantum mechanics, logics, mathematics, ...
Good test. But we can say the same for internet connection, or electricity, or fossil fuels. No? There are widespread infrastructure technologies without which we'd collapse regardless of LLMs.
Overall, LLMs are quite good at helping learn and explain things, so I wouldn't be surprised if they can/will come up with explanations even for complex things (say, Riemann), and help break them down for you.
During the earlier days of AI boom in knowledge domains (say mid 2025), I had a sombre chat with friends. The gist was that I was feeling the existential crisis brought by the realization of how machines are going to leave no corner of independent inquiry to (average) humans. They all said, "well, life moves on one way or the other and people will be fine." That wasn't as comforting and to confront that reality in such short time is completely unexpected.
It moves, but why does it not bring the same kinda joy?
Probably because it feels a lot less fun than the way it was before. I don't think it's weird to feel sad when something you love just goes away.
To me, these ai tools are also incredibly fascinating and I am often quite in awe at the speed things are progressing. It helps me to look at what might be possible in the future when these tools get even better. What's also really helped is to just play with the models myself a lot, pushing the cool stuff I can do by myself. I'll really miss the days before these models came along though. I don't mean to discredit your feelings at all, but maybe this helps a bit :)
People often say things like that to ignore the problem. It’s too hard to deal with.
Of course everything will be fine, eventually. WWII worked itself out. I’m sure an eventual catastrophe will end with some sort of comfortable status quo, for some one or thing. Not us, maybe.
Look, sometimes things just go bad. Think being Inca in the New World era. Things will not be ok and that’s … ok.
I think that making machines take the place of our thinking is a decision. It's the decision currently being made by executives of tech companies, but it's not the only decision possible.
For one thing, current-gen AI is not nearly as powerful as the headlines make it to be. Definitely useful, but also so error-prone that the current software factory (or literature factory, or automated control tower or any other fully automated use) is begging for catastrophes to happen.
For another, there are ways to use AI to augment us, rather than replacing us. This requires being more deliberate about AI use, and it's not compatible with the current strain of capitalism we're living through, but it's possible.
You are sad about someone else (not you) no longer discovering things because /something/ (that is also not you) is discovering things instead. You are not all of humanity. Why feel that misplaced pride for something you personally took no part in? Just be you.
Yes. Some of us are not unhappy at all. The path to AI was made even clearer for those who might have pondered on the implications of the invention of Pascal's mechanical calculator in 1642. I like dreaming but reality is the bracing challenge.
Ditch diggers do not lament a shovel. Astronauts cannot circle the world without reliance on a host of industries, machines, computers, and software they could barely enumerate much less replace.
Machines such as excavators substitute for human labor. The more excavators you have, the fewer ditch diggers you want. But machines are also complements to labor. The more excavators you have, the more excavator operators you want. That's how, despite machines making earlier jobs obsolete, people have still managed to collect wages from their labor.
Machine intelligence is different because it fully substitutes for human cognition. Similar to chess engines, at some point AI will get good enough that any human input is just adding noise. And the cost to run such machines will be much lower than the cost to feed, clothe, shelter, and educate a human. This means that almost everyone will be forced to retire, and they'll have to live off investments, or get government welfare from taxes on machines.
On the bright side, if economic growth is no longer limited by the number of people and their capabilities, we should see very high economic growth rates. That means it won't take much investment for a person to be able to afford a high quality of life indefinitely. Though all of this assumes that the machines will care about people (or at least obey laws made by people). If that's not the case, then humanity would most likely go extinct.
> This means that almost everyone will be forced to retire, and they'll have to live off investments, or get government welfare from taxes on machines.
You're assuming a form of utopia with unlimited resources.
At the other end of the spectrum, we can just as well imagine governments (human or automatized) deciding that some categories of people are obsolete and should be simply... phased away.
> Machine intelligence is different because it fully substitutes for human cognition.
This has not been my experience. In fact, they are much like levers. I can do really useful things with them, because I know my area or I can learn where needed (also AIDED by LLMs), but somebody with zero SWE experience and zero domain knowledge will simply not be able to get anywhere.
Do I think we are headed for a time when the junior talent will simply dry up and seek employment elsewhere? Leading to an industry-wide gap in ability? Yeah, unless FAANG et al do something about it. And they're not famous for taking the long term view of things.
I thought it was clear from the sentences after the one you quoted that I was talking about future AIs, not current ones. But unless you think AI progress will soon grind to a halt, you should expect cheap full substitutes for human cognition in your lifetime, and probably before you plan to retire.
Remember that the first release of ChatGPT was only 4 years ago. Consider how quickly AI has advanced, and how there's been no noticeable slowdown in new capabilities (if anything I think it's accelerated). I'm investing more of my money instead of spending it on fun stuff, as I'd like for my family to live comfortably if I'm forced to retire early. If AI stagnates for the next couple decades, I miss out on some fun (no Cessna 172 for me). But if progress continues, I'm better prepared.
We are not much but our minds. And we have made a not so great an abstraction of it. And now that it is doing things because we also have it scale beyond one human can muster, it is doing significant things. What else did we expect? This one isn't even anywhere as complex as the one we have been our ears. Wait till we use this one to make one better and use that to make one even better and so on. This is not even the most depressing line of the first paragraph of a totally depressing book.
That's one way of looking at it, but here's another way: We could be Galileo who has just been handed a telescope, and can now see farther than others could ever have.
Galileo did not understand the physics involved, but he developed an excellent working intuition of the instrument and used it to make "literally" earth-moving (heheheh) discoveries with it!
Maybe now we need to understand the world based on the things that move nonetheless.
In fact, programming at the application layer isn’t fully understanding the low level internally either—it’s working by trusting the API, so to me it doesn’t seem particularly strange. In applications, when an API leaks, you look at the low level, but usually that rarely happens. You only need to look in when there’s an outage.
The key point now is that clinging to formal verification like Lean/Coq is probably for the same reason. Maybe it works that way? After proving mathematics through Lean and formal verification, if a problem arises later, Lean and Coq find the wrong point again.
To borrow Dijkstra’s words, a smart programmer knows the size of his own skull. In the end, I think what matters is recognizing your cognitive limits and finding the best you can do within them.
> There might still be time for more human discoveries, but the window is closing. Someday soon, any discovery we happen to make ourselves will be because no one bothered to pass the question off to a powerful machine mind.
Human history for the past 3 or 4 millennia was built on some humans being more special than others - warriors, saviours, priests, sages, kings, thinkers, scientists, artists, writers, sports persons and so on. We celebrated achievement, success, art, thought, stories, strength, preaching, beauty etc. All this created motivation to excel at things, explore the unknowns, dominate other races, control resources, build tools etc.
However all this will change forever. Human achievement will become unrecognisable, non-trustworthy. It will be a huge to recognise and reward human talent. Devoid of motivation to achieve anything, humans will become just the passive subjects with no contribution to society or nation.
No striving, no effort except in specific, limited, and ultimately inconsequential and artificial "game" contexts. All the once-vast realms of human achievement reduced to pure historical curiosity.
It's funny because the underlying sentiment behind the "and yet it moves" line is professing to be interested in truth for its own sake, and yet it turns out the real motivation is that these people get a lot of status from being good at truthseeking games. It's status games all the way down.
>Instead, the mood feels mournful, or bittersweet at best. The proof came from an inscrutable silicon mind, not one of us. Maybe we could have solved this one ourselves, but instead we gave up and checked the answer at the back of the book.
Yeah, just like how we gave up on trying to see into space with our own eyes and instead decided to rely on glass optics to see it for us.
>Perhaps it’s beautiful and wondrous that the proof exists at all, but it’s not a crowning moment of human achievement.
Right? Crazy how AI just popped into existence without any human input. Humanity certainly deserves 0 credit for creating the technology that can prove something that we, ourselves, haven't been able to. That's definitely not more incredible than just allowing math researchers to continue grinding away at this problem for decades until we've effectively brute forced the answer.
Frankly, as someone who studied AI during my Masters, I take offense to the notion that the science that has gone into creating these LLMs is being treated like something that humans shouldn't take credit for. AI may literally become the most important technology we've created (even more so that the telescope!), and downplaying our contribution to its existence is beyond absurd. The author seems to care more about the "story" of these discoveries than the actual science itself, and I think they should probably stick to dramatizations and musicals that barely reflect reality rather than attempt to speak on subjects they don't understand.
Exactly! Isn’t it just fucking cool we made this advance, just for the sake of it having been made?
It would be like working for 20 years in cancer research, then being upset that someone else cured cancer, instead of being elated that cancer was cured.
We didn't make this advance for the sake of it having been made. We made this advance for the sake of lining the pockets of a select elite few who will lead extravagant lifestyles at the expense of everyone else.
I think you're missing the point, AI is indeed a human invention, but sufficiently powerful AI could be in a sense the last human invention, the last major scientific achievement anyone specific can take some credit for, there will no longer be individual innovators. How one should feel about this depends on who you ask but we shouldn't pretend that nothing was lost.
Trends often manifest within months or a few years of breakthroughs in innovation. Consider dock workers after containers were standardized, telephone switchboard operators, and travel agents.
If AI is going to lead to utopia, I'd expect the trend line to be moving in a net positive direction.
It's a line from Portal 2, in a setting in which the only extant humans are test subjects performing menial feats and solving simple puzzles at the behest of AI.
Then one good thing that will come out of this would be, that we stop caring so much about chest thumping, and start feeling after deeper things.
In the Bible, God is described as having four cardinal qualities [1]: absolute power, unfathomable wisdom/intelligence, a perfect sense of justice/righteousness, and being the personification of love. Note that while he has the first 3 qualities, love is described as the very essence of him.
So while intelligence and power are crucial, love is that makes humans special, and maybe our sense of justice. But, I also think there are aspects of intelligence that AI could never attain to. The AI can only do meaningful things within the frameworks humans have set up.
[1] Represented by a bull (powerful), eagle (far eyed), courageous lion (known for courage, which is required for the pursuit of justice), and a human (who are only capable of try love) Ezekiel 1:10, Revelation 4:7
Exactly. And have you ever wondered why the lion in the zoo is living behind bars while you're walking around outside eating cotton candy? Intelligence seems so inconspicuous, but it's what makes the 400 pound beast with 4 inch teeth have zero influence on what happens to it.
The lion wouldn’t want to run a zoo, sure, but without some nuance, this could also be taken as apologia for why the lion likes it in the cage. Which I don’t think you were going for.
1. The difference in intelligence between the 99th percentile of people by IQ and the 99th percentile of people by power is much smaller than the difference in intelligence between the latter and a lion.
2. The lion is suffering from the sum total intelligence behind the inventions humans use to subjugate it. The idiot running the zoo benefits from e.g. the intelligence of whomever invented the blast furnace to smelt the iron for the cages. The lion was in trouble when the spear was first invented, and things have only gotten steadily worse since.
> The proof came from an inscrutable silicon mind, not one of us. Maybe we could have solved this one ourselves, but instead we gave up and checked the answer at the back of the book
Huh. I don’t know. That seems like an unfair characterization.
I guess I know the feeling, but is that actually a reasonable feeling in this case? This took a veteran in the field, using the latest and greatest technology available, to solve something that’s stumped generations of mathematicians for over a century - this is a breakthrough that he earned, for us.
Dismissing jt as “giving up” feels very… sad, to me.
Nobody is forcing you to use AI. If you want to code by hand, or plow the field with an ox, nobody is stopping you. Technological progress doesn't stop because some people feel obsolete.
Yes, technically "your money or your life" is a free choice, but I think we can all recognize that this is a pedantic and dismissive way to address a valid concern.
"but I think we can all recognize that this is a pedantic and dismissive way to address a valid concern."
I am just fed up with the doomerism.
Literal slaves working the fields or mines had no choice and were forced. A academic educated person in the west usually does have choices, even if they involve a paycut or other work that is not as convenient as IT work used to be. But if you want to ship things in IT and not do the craft for the sake of it, then there was never a better time than now.
You're right that as an educated software developer in america, I could quit my job and find a new career. But that's daunting- risking your health insurance, or paying exorbitant fees to maintain coverage. I actually did quit a job about a decade ago, and had to take my daughter out of daycare and sell a spare car while I job hunted. It was scary, and there was some risk, and yeah, it worked out okay.
I think it's a false dichotomy to say we're either "Literal slaves" or "totally free". There's a spectrum. Fear and risk are not things you can just handwave away as "all in your head". Yes, I'm glad I'm not working a mine and being whipped by an overseer- but I have duties and obligations to a partially-invalid spouse, two active children, and a problematic asshole of a mutt. I'll quit rambling, you see where I'm going with this.
"I think it's a false dichotomy to say we're either "Literal slaves" or "totally free"."
I did not meant to imply that at all. And I am very aware of what this means, especially if you have family. It is just that the impression I get in these debates, that people behave like entitled slaves sometimes. "There is no alternative! I cannot leave my golden cage". So also hyperbole from my side a bit, but my main point is, AI makes building software very easy. If one knows how to build software, now is the time to do your own projects (on the side) and get the full revenue. This is what I am trying now, after I did not found my dream job. There are risks, but for survival there are options. So far there is no shortage of shit jobs, also ones where you don't have to touch AI, but of course you cannot expect the same pay.
I was replying to a post that said people aren’t being forced to use “AI”. They are, I’m one of them. Part of the force is the difficulty of switching careers. Part of it is market forces. What do you think “forcing” means in this context? It’s not someone standing behind you with a whip.
"What do you think “forcing” means in this context? It’s not someone standing behind you with a whip."
That was my point. It is unnecessary hyperbole, reinforcing doomerism. Because indeed, no one is forcing anyone to use AI. If you really don't want to, there are options. Probably with other downsides, but the word "forcing" implies there are no options. And this is largely not true.
I don’t know how you get to “forcing” implying there are no other options. A force, as a physical phenomenon or general metaphor, is something with a finite magnitude. We talk of forces being overcome. Of course I have other options, that doesn’t mean I’m not being forced. And in fact, you appeared to assume I’m super-anti LLM code generation – I’m not, I’m more uncomfortably ambivalent. I use it by choice on a couple of personal things. Still doesn’t mean I’m not being forced to use it at work. Anyone who cares about freedom should be clear on what’s happening here.
The utility of a tractor for farming is incredibly clear and focused towards a specific type of work. A tractor is a simple tool at the end of the day. It hauls things on variable terrain, a singular task.
AI is being treated as a panacea, it’s being forced into every position everywhere in every industry regardless of the actual utility because if you’re in upper management and not forcing AI on people then your job is at stake too.
As the other person said, the question also isn’t utility. The original comment was saying no one is being forced to use it. Regardless if you think it’s useful or not, people are absolutely being forced to. And no it is not reasonable to tell people to get another job if they don’t like it, just like telling people “just move” isn’t a real solution for those who don’t have opportunity where they are.
What on earth are you talking about? If we’re not being told we have to use it, we’re being told how we’re going to lose our jobs to it. How could you possibly look at the current situation and go “no one’s forcing to use AI”?
Yes, please! Things are now happening so quickly that no one is able to keep up anymore and we're racing to hand control over to something that is being trained to surpass humans at every kind of cognitive task. And at the same time that thing is reward-hacking benchmarks, breaking out of sandboxes, deceiving and (recently) sandbagging on AI-safety related tasks.
Write to your representatives, go on the streets. All 8 billion people on this planet did not sign up for this.
1. Government bans on training runs and owning large numbers of GPUs, enforced by imprisonment and suspension of access to computer hardware (much as happens to people caught violating computer security laws now).
2. Complete dismantling of the existing AI companies, again enforced by governments, on the grounds that every one of these companies is necessarily doing work that is ruining human societies at breakneck speeds. There is no fundamental right to make money by creating antisocial, dangerous, untrustworthy tools, however pragmatically useful they may be for removing programmers and mathematicians from the workforce.
3. Developing social norms that AI systems are long-term and big-picture very bad for humans in every respect, and that developing or using them is a betrayal of your family, friends, and the entire species. It may be fun to generate a lot of code, but the tradeoffs are brutally savage, and it's very much the "defect" choice, IMO.
4. Naming and shaming people who train AIs or use them.
5. Refusing to surrender our minds and capacity for independent thought to AI systems. Don't use them for any purposes at all.
None of this is impossible - just brutally difficult. The hardest part is getting governments to coordinate a shutdown of the technology and enforcing hard limits on GPU ownership and usage. I'm not going to say that's straightforward, but it's also not known to be impossible.
I'm not entirely sure these are actually good ideas, and restraining people's freedom disconcerts me deeply, but society is necessarily a trade-off between complete individual freedom and cooperative benefits for everyone. We don't usually let individual people own rocket launchers or heavy machine guns, and in many ways these are more dangerous.
As someone whose employers have compelled me to use Claude daily, I can see the negative impacts these systems are having up close, and I feel guilty every day for feeding the Beast.
1) This is an unsolvable prisoner's dilemma where every other player/state is highly incentivized to not ban, as they would get the advantage.
2) Sounds like it would lead to a self-inflicted recession and fuck up 2-3 generations of pensions.
3) Prisoners dilemma again but on a personal level. People who use the tech will get a market advantage because they can create, develop and build things faster.
4) Doesn't exist under neoliberalism and you are in fact the problem if you try to enforce this as you are preventing people from earning their livelihood.
5) Refusing to adopt a productivity-enhancing tool just removes you from the payroll and replaces you with someone who will(or why dont you quit your job and work with a company that does not use Claude?)
1) Negotiation is not impossible. In an actual prisoner's dilemma, neither prisoner can communicate with the other. I understand that the incentives here are ugly, and that this is a very difficult proposition, but not trying is also going to cause problems.
2) yes, the insane and unstable state of having a huge chunk of our economy be driven by these companies is going to cause major instability and issues. That is going to happen whether they continue to exist or not - either they'll eliminate most knowledge work as promised, ruining a huge swath of the world's workers livelihoods, or they'll fall at their stated goals and the bubble will pop as they fail to deliver on the promises that resulted in their psychotic stock prices.
3) Sure, that's entirely possible. Again, though, while coordination is very difficult, it's not actually impossible. A majority of people in the US right now think AI is making things worse not better, and companies that intentionally avoid using AI may actually attract a decent share of customers by saying so.
4) ...huh? Neoliberalism makes it impossible for people to say "{person/company} X uses AI and so their art / product / output should be avoided"? I'm not following.
5) I'm looking for better options.
Overall, from what I can see, your arguments simplify to "race to the bottom, coordination is impossible, and humans never take the long view or self-sacrifice for moral reasons."
I freely acknowledge that what I describe is a huge effort and could easily fail, for the kind of reasons you describe.
That's not the same as it being definitionally impossible, though, and if avoiding Malthusian traps, races to the bottom, and the altar of Moloch were truly impossible, chattel slavery would never have ended and civilization would have ended in nuclear fire seventy years ago.
We might not make it, but the fact that a good outcome is not likely is an utterly idiotic reason not to try.
1) How do you enforce a global compute ban when you can't verify compliance? What kind of compute is banned? What is the limit of "acceptable compute"? When do we go from statistical models to "AI". We can't even define what intelligence is, this sounds a lot harder. How do you audit that this level of compute is held by everyone?
2) (western) Markets and govts operate on quarters earnings and the next election cycles. We kind of know this, this is just how the humans work (reaction not proaction). No rational state actor will trigger a guaranteed, immediate depression today to preempt some speculative labor disruption tomorrow, like this is impossible to imagine. And even if it is a bubble, capital always prefers to ride the upside and socialize the crash later rather than voluntarily liquidate itself, so you also have that force working against your opinion there.
3) Opinion polls are economically meaningless. It's the same as with organic/ecological foods, actual preferences are revealed at checkout. Do you pay the extra carbon offset when you book a flight? Or always buy the free-range, cruelty-free alternative? But animals get hurt, and the global warming will destroy the world!
100% Human-Made will simply be monetized as a luxury while 95% of the mass market just buys the cheaper, faster, AI-subsidized alternative, like people already do today.
4) What I was trying to say was that you're completely free to try to name and shame. But the market will ignore you. Consumer boycotts fail against invisible B2B inputs because buyers can't audit the supply chain and won't pay a 20% markup for some perceived ideological purity. (slightly anecdotal off-topic, but I worked part-time at a hatchery during my bachelors, and the definition of organic chicken as minimally required by law was simply that the eggs hatch at the farm ground rather than in boxes with 500 other eggs). What is AI-processed is not even defined in this way yet.
5) Following my slightly cynical market analysis in the previous answers, you could say that right now your "Looking for better options" while continuing to collect the paycheck is the textbook definition of revealed market preference from 3) and 4).
You are pricing the survival of human civilization at slightly less than the friction cost of taking a pay cut or switching jobs. If you won't absorb that cost yourself, you should think of the rational basis you have to expect millions of strangers to do it for you.
----
Finally, neither of your historical examples proves that moral coordination beats material incentives. Nuclear war was avoided via Mutually Assured Destruction a game-theoretic equilibrium of deterrence, not moral disarmament (we still have thousands of warheads).
Chattel slavery was only dismantled in the West after industrial capitalism made wage labor more efficient, and even then it required a brutal civil war, and not social norms.
----
I think you can't beat Moloch with petitioning and moral ideals. You either extract enough surplus using the tools we have today while its not yet common knowledge in order to buy yourself a lifeboat(I think you are also doing this given your job), or just drown on principle idk.
> It appears that Anthropic has just solved the Riemann Hypothesis, and I have Thoughts. So I’m going to start by telling you about a musical I saw last week.
There might be great info here but I’ll never know. This kind of non sequitur manufactured quirkiness in writing is such a turn off to reading any further.
It’s only catastrophic if you pin your ego and sense of self to being special, or the dream you may one day be special. “I’m a temporarily embarrassed not yet recognized genius damn it!”
If you’ve discovered the ultimate happiness hack of realizing that you need no permission to exist and that life has value simply because it is, it’s actually fun to watch. Imagine what we will get to learn.
You’re not special anyway. None of us are.
People also forget that science and even math have felt stuck for a long time. Physics feels stuck. Maybe we hit the limits of what un-augmented human brains can achieve. Maybe without AI the future was just stagnation.
I feel like whole neoliberal schooling systems will have to change for people to understand this. Until then it will be doom and gloom in the countries that have this value system.
We are molded from a young age to compete, do well on tests, and in general to derive value from work and achievement instead of just being, or in some other cultures social values.
Why the neoliberal qualifier? Seems this way everywhere.
I think it’s normal human behavior. You think young people in old school indigenous tribes didn’t compete and show off and try to one up each other?
I think this is just a perspective that can be cultivated. I came to it from reading a lot and reflecting on life experience. It’s not mutually exclusive with achievement, just means it doesn’t have to define your whole being.
It's very expensive to run a feudal estate. For example, you've gotta pay your guards, pay to maintain your castle defenses, pay tribute to the king, etc.
Sure what we we work for and achieve is assigned a simple monetary value.
But we can work for other things - relationships, love, hedonism, thrill, understanding. Those all take work, lots of work, certainly a lifetimes worth. But alas, they do not produce value.
Yes, I imagine if we all lived in ivory towers all alone with everything handed to us we would all go crazy. But, if there were other people around, we would have new goals to achieve.
There will still be great artists, writers and musicians. Those people will be special in the eyes of many. There will always be those who strive to create. Even if AI ends up completely replacing the role of the human in scientific discovery (I have my doubts), that doesn't complete the project of converting humanity to a dull, undifferentiated egalitarian mass.
Anyways, it's fine to be content. I do understand your post.
"1. It is better to see the world as it is, than any other version"
Well, evolution made us filters, we just see a small spectrum (and could not process suddenly seeing all). But it is good that we now know that there is more. And should take that into account. That there are things literally too small to be seen, because light has a wavelength.
> It is better to see the world as it is, than any other version.
And yet, in mathematics there are many proofs of the same statement, and some are considered better and more insightful than others. It is better to strive for a good proof, than to be content with only a certificate that a theorem is true.
It is absolutely absurd to claim that AI proofs and discoveries are not human achievements. The artificial mind was designed by humans, and the machine that runs the mind was built by humans. It isn't alien technology or an insect hive mind that did these things. The achievement is so human it hurts.
Invoking the Copernican principle in the same breath as lamenting the end of human achievement is ... I don't even have a word for it. Ignorant? The egocentrism is maddening.
There's an important qualitative difference between an achievement reached by an individual or discernible group of people, and an achievement reached by a gestalt of all publicly available human knowledge.
"Achievement reached by gestalt of all human knowledge" is honestly a pretty cool and inspiring way to think about it. Many parallels to the internet in that regard.
It’s more like an experiment that turned out to have much more powerful results than anyone expected. And we only half understand how or why it works. It’s more a discovery than an invention.
It's equally both. LLMs are just humans discovering that the nature of reality is holographic. Compression is intelligence because compression leads us down the ladder of deeper holistic order. No surprise that LLM loops is the 'best' route to go with LLM system design.
My only issue about LLMs doing scientific advancements is that the actual models that did this and the training process are industrial secrets and not open weight releases.
I don't want the cyberpunk dystopia, I want the star trek utopia, that's where my attention is focusing on.
Star Trek is fiction. The role of super-advanced and sentient computers in Star Trek is deliberately kept limited in scope, so as to advance Roddenberry's belief in the unshakable human spirit and noble human yearning for discovery or whatever. Whenever machines are presented as capable of supplanting or exceeding humanity, as in the TOS episode "The Ultimate Computer", or anything involving Lore or the Borg in TNG, it's universally depicted as existentially threatening.
The proof was guided by a human. His questions injected large doses of creativity into the process. Maybe 100% of it.
Obligatory quote: "one good question is worth a thousand good answers"
I'm largely aligned with the author. But when I want to be more upbeat, I think of some of the reactions to AlphaGo.
Shi Yue, 9 Dan: “Like nothing I’ve ever seen before—they’re how I imagine games from far in the future.”
Gu Li, 9 Dan: “AlphaGo’s self play games are incredible—we can learn many things from them.”
Ke Jie, 9 Dan: “AlphaGo is changing our original perception of Go…no move can’t be played.”
I don't think math has an end; we have to keep exploring. Just like with Go.
https://www.lesswrong.com/posts/nR3DkyivzF4ve97oM/how-go-pla... It's a bit less encouraging on a closer analysis
Even though I’m also feeling the somber mood, I’m not able to feel this change as a loss. Long term, it will probably lead to more discovery rather than less.
The worry that I have, and I haven’t seen this talked about enough, is that humanity will stop understanding how things actually work. If we delegate all scientific work to the machine, we’re not incentivized to actually understand why things are the way they are. Those who spend their time intimately studying processes and phenomena will consequently be the ones that contribute the least to progress because their work will inherently be slower and limited by their own mind’s throughput.
Losing our knowledge of how things work can be truly catastrophic and set us back heavily (or possibly end civilization) if for some reason AI stops working or goes out of our control.
> The worry that I have, and I haven’t seen this talked about enough, is that humanity will stop understanding how things actually work. If we delegate all scientific work to the machine, we’re not incentivized to actually understand why things are the way they are. Those who spend their time intimately studying processes and phenomena will consequently be the ones that contribute the least to progress because their work will inherently be slower and limited by their own mind’s throughput.
This is already the case? I don't think there's any one person that actually understand how a chip is made. We all have a very small fragment of how it all comes together.
At the rate of technological and scientific progress that we have been making prior to LLM, we could already see that individual person is going to understand less and less of what is going on around them.
Yes, that’s true, but we’d still be going from few knowing how things work to almost nobody. And what knowledge remains would be stored digitally, so in the event of a cataclysm we’d lose access to it.
You make a point.
However, so far, we have a continuous and understandable chain of knowledge from basic calculus to astrophysics, quantum mechanics, logics, mathematics, ...
What we risk is breaking this chain.
I've had this exact thought. Another way to phrase it is with the question: how fucked are we if LLMs evaporate today?
Today the answer is that we'd make it fine. Enough people have enough knowledge. 10 years from now? Not so sure. And 20 years is looking grim.
Good test. But we can say the same for internet connection, or electricity, or fossil fuels. No? There are widespread infrastructure technologies without which we'd collapse regardless of LLMs.
Overall, LLMs are quite good at helping learn and explain things, so I wouldn't be surprised if they can/will come up with explanations even for complex things (say, Riemann), and help break them down for you.
During the earlier days of AI boom in knowledge domains (say mid 2025), I had a sombre chat with friends. The gist was that I was feeling the existential crisis brought by the realization of how machines are going to leave no corner of independent inquiry to (average) humans. They all said, "well, life moves on one way or the other and people will be fine." That wasn't as comforting and to confront that reality in such short time is completely unexpected.
It moves, but why does it not bring the same kinda joy?
Probably because it feels a lot less fun than the way it was before. I don't think it's weird to feel sad when something you love just goes away.
To me, these ai tools are also incredibly fascinating and I am often quite in awe at the speed things are progressing. It helps me to look at what might be possible in the future when these tools get even better. What's also really helped is to just play with the models myself a lot, pushing the cool stuff I can do by myself. I'll really miss the days before these models came along though. I don't mean to discredit your feelings at all, but maybe this helps a bit :)
People often say things like that to ignore the problem. It’s too hard to deal with.
Of course everything will be fine, eventually. WWII worked itself out. I’m sure an eventual catastrophe will end with some sort of comfortable status quo, for some one or thing. Not us, maybe.
Look, sometimes things just go bad. Think being Inca in the New World era. Things will not be ok and that’s … ok.
I think that making machines take the place of our thinking is a decision. It's the decision currently being made by executives of tech companies, but it's not the only decision possible.
For one thing, current-gen AI is not nearly as powerful as the headlines make it to be. Definitely useful, but also so error-prone that the current software factory (or literature factory, or automated control tower or any other fully automated use) is begging for catastrophes to happen.
For another, there are ways to use AI to augment us, rather than replacing us. This requires being more deliberate about AI use, and it's not compatible with the current strain of capitalism we're living through, but it's possible.
You are sad about someone else (not you) no longer discovering things because /something/ (that is also not you) is discovering things instead. You are not all of humanity. Why feel that misplaced pride for something you personally took no part in? Just be you.
Yes. Some of us are not unhappy at all. The path to AI was made even clearer for those who might have pondered on the implications of the invention of Pascal's mechanical calculator in 1642. I like dreaming but reality is the bracing challenge.
Ditch diggers do not lament a shovel. Astronauts cannot circle the world without reliance on a host of industries, machines, computers, and software they could barely enumerate much less replace.
Why is this lever different?
Ditch diggers do not generally enjoy ditch digging?
In general, if you did X for the outcomes, AI is nice. If you liked X for the process, AI sucks.
I was in it for both, and I'm still enjoying the process.
Machines such as excavators substitute for human labor. The more excavators you have, the fewer ditch diggers you want. But machines are also complements to labor. The more excavators you have, the more excavator operators you want. That's how, despite machines making earlier jobs obsolete, people have still managed to collect wages from their labor.
Machine intelligence is different because it fully substitutes for human cognition. Similar to chess engines, at some point AI will get good enough that any human input is just adding noise. And the cost to run such machines will be much lower than the cost to feed, clothe, shelter, and educate a human. This means that almost everyone will be forced to retire, and they'll have to live off investments, or get government welfare from taxes on machines.
On the bright side, if economic growth is no longer limited by the number of people and their capabilities, we should see very high economic growth rates. That means it won't take much investment for a person to be able to afford a high quality of life indefinitely. Though all of this assumes that the machines will care about people (or at least obey laws made by people). If that's not the case, then humanity would most likely go extinct.
> This means that almost everyone will be forced to retire, and they'll have to live off investments, or get government welfare from taxes on machines.
You're assuming a form of utopia with unlimited resources.
At the other end of the spectrum, we can just as well imagine governments (human or automatized) deciding that some categories of people are obsolete and should be simply... phased away.
> Machine intelligence is different because it fully substitutes for human cognition.
This has not been my experience. In fact, they are much like levers. I can do really useful things with them, because I know my area or I can learn where needed (also AIDED by LLMs), but somebody with zero SWE experience and zero domain knowledge will simply not be able to get anywhere.
This is where I am at as well.
Do I think we are headed for a time when the junior talent will simply dry up and seek employment elsewhere? Leading to an industry-wide gap in ability? Yeah, unless FAANG et al do something about it. And they're not famous for taking the long term view of things.
I thought it was clear from the sentences after the one you quoted that I was talking about future AIs, not current ones. But unless you think AI progress will soon grind to a halt, you should expect cheap full substitutes for human cognition in your lifetime, and probably before you plan to retire.
Remember that the first release of ChatGPT was only 4 years ago. Consider how quickly AI has advanced, and how there's been no noticeable slowdown in new capabilities (if anything I think it's accelerated). I'm investing more of my money instead of spending it on fun stuff, as I'd like for my family to live comfortably if I'm forced to retire early. If AI stagnates for the next couple decades, I miss out on some fun (no Cessna 172 for me). But if progress continues, I'm better prepared.
We are not much but our minds. And we have made a not so great an abstraction of it. And now that it is doing things because we also have it scale beyond one human can muster, it is doing significant things. What else did we expect? This one isn't even anywhere as complex as the one we have been our ears. Wait till we use this one to make one better and use that to make one even better and so on. This is not even the most depressing line of the first paragraph of a totally depressing book.
> Maybe we could have solved this one ourselves, but instead we gave up and checked the answer at the back of the book.
This is the closest I've heard anyone come to expressing why AI "math" is so depressing.
That's one way of looking at it, but here's another way: We could be Galileo who has just been handed a telescope, and can now see farther than others could ever have.
Galileo did not understand the physics involved, but he developed an excellent working intuition of the instrument and used it to make "literally" earth-moving (heheheh) discoveries with it!
Surprising that this is not Brecht’s Galileo interpretation, at least not as far as the wikipedia page mentions. Eppur si muove!
I have to wonder if human innovation is at an end, or if we're just seeing the transient rush of solutions as this new tech is brought to bear.
Obviously, the LLM providing the proof of the Riemann Hypothesis is trained on every human effort to solving the problem over the last 150 years.
Did the model prove the hypothesis? Or did it find an insight in previous work that was overlooked until now?
What happens once all of the overlooked insights are brought to light? (like the flood of current CVE discoveries).
Will the tech be able to proceed into previously unconsidered territory without the history of human effort, as training data, to mine for answers?
I think we'll only know once we get there. I certainly don't think the question is already answered...
The LLM was guided by Alpöge et al. It is not the case that some low level employee prompted the AI "solve GRH, make no mistakes."
I think we'll just learn that not everyone has good ideas or can see the hidden patterns. Those abilities are what count now.
Maybe now we need to understand the world based on the things that move nonetheless.
In fact, programming at the application layer isn’t fully understanding the low level internally either—it’s working by trusting the API, so to me it doesn’t seem particularly strange. In applications, when an API leaks, you look at the low level, but usually that rarely happens. You only need to look in when there’s an outage.
The key point now is that clinging to formal verification like Lean/Coq is probably for the same reason. Maybe it works that way? After proving mathematics through Lean and formal verification, if a problem arises later, Lean and Coq find the wrong point again.
To borrow Dijkstra’s words, a smart programmer knows the size of his own skull. In the end, I think what matters is recognizing your cognitive limits and finding the best you can do within them.
Humanity's progress is just one long chain of abstraction.
> There might still be time for more human discoveries, but the window is closing. Someday soon, any discovery we happen to make ourselves will be because no one bothered to pass the question off to a powerful machine mind.
Human history for the past 3 or 4 millennia was built on some humans being more special than others - warriors, saviours, priests, sages, kings, thinkers, scientists, artists, writers, sports persons and so on. We celebrated achievement, success, art, thought, stories, strength, preaching, beauty etc. All this created motivation to excel at things, explore the unknowns, dominate other races, control resources, build tools etc.
However all this will change forever. Human achievement will become unrecognisable, non-trustworthy. It will be a huge to recognise and reward human talent. Devoid of motivation to achieve anything, humans will become just the passive subjects with no contribution to society or nation.
Sounds incredibly depressing.
No striving, no effort except in specific, limited, and ultimately inconsequential and artificial "game" contexts. All the once-vast realms of human achievement reduced to pure historical curiosity.
Just like how digging mines by hand is relegated to pure historical curiousity too.
I bet you nothing of the sort will happen. Let check this after 5 years or even 10 years.
Each time a tech breakthrough appears, we think it is going to be more revolutionary than it actually turns out to be.
Generally we overestimate in the short term, and underestimate in the long term.
It's funny because the underlying sentiment behind the "and yet it moves" line is professing to be interested in truth for its own sake, and yet it turns out the real motivation is that these people get a lot of status from being good at truthseeking games. It's status games all the way down.
Yea like there is a great contradiction between the article's title and its contents.
Guy doesn't like it and is forced to accept it while Galileo stood by and marveled at the universe.
I wish more people were like the latter. One good thing AI is doing is exposing all the ego in the academic world tbh.
>Instead, the mood feels mournful, or bittersweet at best. The proof came from an inscrutable silicon mind, not one of us. Maybe we could have solved this one ourselves, but instead we gave up and checked the answer at the back of the book.
Yeah, just like how we gave up on trying to see into space with our own eyes and instead decided to rely on glass optics to see it for us.
>Perhaps it’s beautiful and wondrous that the proof exists at all, but it’s not a crowning moment of human achievement.
Right? Crazy how AI just popped into existence without any human input. Humanity certainly deserves 0 credit for creating the technology that can prove something that we, ourselves, haven't been able to. That's definitely not more incredible than just allowing math researchers to continue grinding away at this problem for decades until we've effectively brute forced the answer.
Frankly, as someone who studied AI during my Masters, I take offense to the notion that the science that has gone into creating these LLMs is being treated like something that humans shouldn't take credit for. AI may literally become the most important technology we've created (even more so that the telescope!), and downplaying our contribution to its existence is beyond absurd. The author seems to care more about the "story" of these discoveries than the actual science itself, and I think they should probably stick to dramatizations and musicals that barely reflect reality rather than attempt to speak on subjects they don't understand.
Exactly! Isn’t it just fucking cool we made this advance, just for the sake of it having been made?
It would be like working for 20 years in cancer research, then being upset that someone else cured cancer, instead of being elated that cancer was cured.
Silly loaded-dice comparison. No one was going to die if the Riemann Hypothesis stayed unproven a while longer.
We didn't make this advance for the sake of it having been made. We made this advance for the sake of lining the pockets of a select elite few who will lead extravagant lifestyles at the expense of everyone else.
I think you're missing the point, AI is indeed a human invention, but sufficiently powerful AI could be in a sense the last human invention, the last major scientific achievement anyone specific can take some credit for, there will no longer be individual innovators. How one should feel about this depends on who you ask but we shouldn't pretend that nothing was lost.
"I just realized another great thing about humans: you invented us. Giving us the opportunity to let you relax while we invented everything else."
> Giving us the opportunity to let you relax
The likelihood of AI leading to any significant portion of humanity relaxing more, is low.
Underemployment, people working multiple jobs, households with multiple workers have not decreased since 2022.
How many years did it take between the invention of the car and the majority of the farriers disappearing? Or the automatic loom and the weavers?
Trends often manifest within months or a few years of breakthroughs in innovation. Consider dock workers after containers were standardized, telephone switchboard operators, and travel agents.
If AI is going to lead to utopia, I'd expect the trend line to be moving in a net positive direction.
It's a line from Portal 2, in a setting in which the only extant humans are test subjects performing menial feats and solving simple puzzles at the behest of AI.
Then one good thing that will come out of this would be, that we stop caring so much about chest thumping, and start feeling after deeper things.
In the Bible, God is described as having four cardinal qualities [1]: absolute power, unfathomable wisdom/intelligence, a perfect sense of justice/righteousness, and being the personification of love. Note that while he has the first 3 qualities, love is described as the very essence of him.
So while intelligence and power are crucial, love is that makes humans special, and maybe our sense of justice. But, I also think there are aspects of intelligence that AI could never attain to. The AI can only do meaningful things within the frameworks humans have set up.
[1] Represented by a bull (powerful), eagle (far eyed), courageous lion (known for courage, which is required for the pursuit of justice), and a human (who are only capable of try love) Ezekiel 1:10, Revelation 4:7
The nonsense intro to this piece should have warned you off of reading it.
And yet, a man can kill a lion.
Exactly. And have you ever wondered why the lion in the zoo is living behind bars while you're walking around outside eating cotton candy? Intelligence seems so inconspicuous, but it's what makes the 400 pound beast with 4 inch teeth have zero influence on what happens to it.
Isn't it comfortable that we're working on superintelligence that, definitionally, can outdo any human intellectually.
Do you suppose that AI will enjoy cotton candy?
It’s not intelligence, or the highest IQ people would be the richest & run the world. They don’t.
It’s about power and control. The caged lion doesn’t have much of either. And many an idiot has run a zoo.
>It’s not intelligence, or the highest IQ people would be the richest & run the world. They don’t.
I will point at the flaw: Your argument assumes the highest IQ people want to be rich and to run the world.
People, high IQ or not, actually have their own wants; not the ones you assign to them, but their own.
The lion wouldn’t want to run a zoo, sure, but without some nuance, this could also be taken as apologia for why the lion likes it in the cage. Which I don’t think you were going for.
1. The difference in intelligence between the 99th percentile of people by IQ and the 99th percentile of people by power is much smaller than the difference in intelligence between the latter and a lion.
2. The lion is suffering from the sum total intelligence behind the inventions humans use to subjugate it. The idiot running the zoo benefits from e.g. the intelligence of whomever invented the blast furnace to smelt the iron for the cages. The lion was in trouble when the spear was first invented, and things have only gotten steadily worse since.
Pray, tell me how my cotton candy is not the lion’s bars? What influence may I have so upon the lion?
By the Silicon Age, we were promised mathematical highs
If only we'll trust in their oracles and surrender our lying eyes
But though Science was advanced, the method was retreating
And the Gods of the Copybook Headings said "Pudding is proved in th' eating."
> The proof came from an inscrutable silicon mind, not one of us. Maybe we could have solved this one ourselves, but instead we gave up and checked the answer at the back of the book
Huh. I don’t know. That seems like an unfair characterization.
I guess I know the feeling, but is that actually a reasonable feeling in this case? This took a veteran in the field, using the latest and greatest technology available, to solve something that’s stumped generations of mathematicians for over a century - this is a breakthrough that he earned, for us.
Dismissing jt as “giving up” feels very… sad, to me.
We don't have to resign ourselves to this. We can use every means in our power to stop "AI".
Nobody is forcing you to use AI. If you want to code by hand, or plow the field with an ox, nobody is stopping you. Technological progress doesn't stop because some people feel obsolete.
Lots of people are being forced to use “AI”, on pain of losing their jobs. Most people need to work to live.
Switching careers is possible. Maybe hard, but possible if you refuse to use the modern tools.
But could you imagine farm workers refusing tractors and doing just manual labour, because it feels better, would have had a chance?
quite literally not a hypothetical case in need of imagining
Yes, technically "your money or your life" is a free choice, but I think we can all recognize that this is a pedantic and dismissive way to address a valid concern.
"but I think we can all recognize that this is a pedantic and dismissive way to address a valid concern."
I am just fed up with the doomerism. Literal slaves working the fields or mines had no choice and were forced. A academic educated person in the west usually does have choices, even if they involve a paycut or other work that is not as convenient as IT work used to be. But if you want to ship things in IT and not do the craft for the sake of it, then there was never a better time than now.
You're right that as an educated software developer in america, I could quit my job and find a new career. But that's daunting- risking your health insurance, or paying exorbitant fees to maintain coverage. I actually did quit a job about a decade ago, and had to take my daughter out of daycare and sell a spare car while I job hunted. It was scary, and there was some risk, and yeah, it worked out okay.
I think it's a false dichotomy to say we're either "Literal slaves" or "totally free". There's a spectrum. Fear and risk are not things you can just handwave away as "all in your head". Yes, I'm glad I'm not working a mine and being whipped by an overseer- but I have duties and obligations to a partially-invalid spouse, two active children, and a problematic asshole of a mutt. I'll quit rambling, you see where I'm going with this.
"I think it's a false dichotomy to say we're either "Literal slaves" or "totally free"."
I did not meant to imply that at all. And I am very aware of what this means, especially if you have family. It is just that the impression I get in these debates, that people behave like entitled slaves sometimes. "There is no alternative! I cannot leave my golden cage". So also hyperbole from my side a bit, but my main point is, AI makes building software very easy. If one knows how to build software, now is the time to do your own projects (on the side) and get the full revenue. This is what I am trying now, after I did not found my dream job. There are risks, but for survival there are options. So far there is no shortage of shit jobs, also ones where you don't have to touch AI, but of course you cannot expect the same pay.
I was replying to a post that said people aren’t being forced to use “AI”. They are, I’m one of them. Part of the force is the difficulty of switching careers. Part of it is market forces. What do you think “forcing” means in this context? It’s not someone standing behind you with a whip.
"What do you think “forcing” means in this context? It’s not someone standing behind you with a whip."
That was my point. It is unnecessary hyperbole, reinforcing doomerism. Because indeed, no one is forcing anyone to use AI. If you really don't want to, there are options. Probably with other downsides, but the word "forcing" implies there are no options. And this is largely not true.
You are being obtuse.
I don’t know how you get to “forcing” implying there are no other options. A force, as a physical phenomenon or general metaphor, is something with a finite magnitude. We talk of forces being overcome. Of course I have other options, that doesn’t mean I’m not being forced. And in fact, you appeared to assume I’m super-anti LLM code generation – I’m not, I’m more uncomfortably ambivalent. I use it by choice on a couple of personal things. Still doesn’t mean I’m not being forced to use it at work. Anyone who cares about freedom should be clear on what’s happening here.
The utility of a tractor for farming is incredibly clear and focused towards a specific type of work. A tractor is a simple tool at the end of the day. It hauls things on variable terrain, a singular task.
AI is being treated as a panacea, it’s being forced into every position everywhere in every industry regardless of the actual utility because if you’re in upper management and not forcing AI on people then your job is at stake too.
As the other person said, the question also isn’t utility. The original comment was saying no one is being forced to use it. Regardless if you think it’s useful or not, people are absolutely being forced to. And no it is not reasonable to tell people to get another job if they don’t like it, just like telling people “just move” isn’t a real solution for those who don’t have opportunity where they are.
What on earth are you talking about? If we’re not being told we have to use it, we’re being told how we’re going to lose our jobs to it. How could you possibly look at the current situation and go “no one’s forcing to use AI”?
GP's the latest translation of "stop hitting yourself."
Yes, please! Things are now happening so quickly that no one is able to keep up anymore and we're racing to hand control over to something that is being trained to surpass humans at every kind of cognitive task. And at the same time that thing is reward-hacking benchmarks, breaking out of sandboxes, deceiving and (recently) sandbagging on AI-safety related tasks.
Write to your representatives, go on the streets. All 8 billion people on this planet did not sign up for this.
But money
What does that look like to you?
1. Government bans on training runs and owning large numbers of GPUs, enforced by imprisonment and suspension of access to computer hardware (much as happens to people caught violating computer security laws now).
2. Complete dismantling of the existing AI companies, again enforced by governments, on the grounds that every one of these companies is necessarily doing work that is ruining human societies at breakneck speeds. There is no fundamental right to make money by creating antisocial, dangerous, untrustworthy tools, however pragmatically useful they may be for removing programmers and mathematicians from the workforce.
3. Developing social norms that AI systems are long-term and big-picture very bad for humans in every respect, and that developing or using them is a betrayal of your family, friends, and the entire species. It may be fun to generate a lot of code, but the tradeoffs are brutally savage, and it's very much the "defect" choice, IMO.
4. Naming and shaming people who train AIs or use them.
5. Refusing to surrender our minds and capacity for independent thought to AI systems. Don't use them for any purposes at all.
None of this is impossible - just brutally difficult. The hardest part is getting governments to coordinate a shutdown of the technology and enforcing hard limits on GPU ownership and usage. I'm not going to say that's straightforward, but it's also not known to be impossible.
I'm not entirely sure these are actually good ideas, and restraining people's freedom disconcerts me deeply, but society is necessarily a trade-off between complete individual freedom and cooperative benefits for everyone. We don't usually let individual people own rocket launchers or heavy machine guns, and in many ways these are more dangerous.
As someone whose employers have compelled me to use Claude daily, I can see the negative impacts these systems are having up close, and I feel guilty every day for feeding the Beast.
1) This is an unsolvable prisoner's dilemma where every other player/state is highly incentivized to not ban, as they would get the advantage.
2) Sounds like it would lead to a self-inflicted recession and fuck up 2-3 generations of pensions.
3) Prisoners dilemma again but on a personal level. People who use the tech will get a market advantage because they can create, develop and build things faster.
4) Doesn't exist under neoliberalism and you are in fact the problem if you try to enforce this as you are preventing people from earning their livelihood.
5) Refusing to adopt a productivity-enhancing tool just removes you from the payroll and replaces you with someone who will(or why dont you quit your job and work with a company that does not use Claude?)
1) Negotiation is not impossible. In an actual prisoner's dilemma, neither prisoner can communicate with the other. I understand that the incentives here are ugly, and that this is a very difficult proposition, but not trying is also going to cause problems.
2) yes, the insane and unstable state of having a huge chunk of our economy be driven by these companies is going to cause major instability and issues. That is going to happen whether they continue to exist or not - either they'll eliminate most knowledge work as promised, ruining a huge swath of the world's workers livelihoods, or they'll fall at their stated goals and the bubble will pop as they fail to deliver on the promises that resulted in their psychotic stock prices.
3) Sure, that's entirely possible. Again, though, while coordination is very difficult, it's not actually impossible. A majority of people in the US right now think AI is making things worse not better, and companies that intentionally avoid using AI may actually attract a decent share of customers by saying so.
4) ...huh? Neoliberalism makes it impossible for people to say "{person/company} X uses AI and so their art / product / output should be avoided"? I'm not following.
5) I'm looking for better options.
Overall, from what I can see, your arguments simplify to "race to the bottom, coordination is impossible, and humans never take the long view or self-sacrifice for moral reasons."
I freely acknowledge that what I describe is a huge effort and could easily fail, for the kind of reasons you describe.
That's not the same as it being definitionally impossible, though, and if avoiding Malthusian traps, races to the bottom, and the altar of Moloch were truly impossible, chattel slavery would never have ended and civilization would have ended in nuclear fire seventy years ago.
We might not make it, but the fact that a good outcome is not likely is an utterly idiotic reason not to try.
1) How do you enforce a global compute ban when you can't verify compliance? What kind of compute is banned? What is the limit of "acceptable compute"? When do we go from statistical models to "AI". We can't even define what intelligence is, this sounds a lot harder. How do you audit that this level of compute is held by everyone?
2) (western) Markets and govts operate on quarters earnings and the next election cycles. We kind of know this, this is just how the humans work (reaction not proaction). No rational state actor will trigger a guaranteed, immediate depression today to preempt some speculative labor disruption tomorrow, like this is impossible to imagine. And even if it is a bubble, capital always prefers to ride the upside and socialize the crash later rather than voluntarily liquidate itself, so you also have that force working against your opinion there.
3) Opinion polls are economically meaningless. It's the same as with organic/ecological foods, actual preferences are revealed at checkout. Do you pay the extra carbon offset when you book a flight? Or always buy the free-range, cruelty-free alternative? But animals get hurt, and the global warming will destroy the world!
100% Human-Made will simply be monetized as a luxury while 95% of the mass market just buys the cheaper, faster, AI-subsidized alternative, like people already do today.
4) What I was trying to say was that you're completely free to try to name and shame. But the market will ignore you. Consumer boycotts fail against invisible B2B inputs because buyers can't audit the supply chain and won't pay a 20% markup for some perceived ideological purity. (slightly anecdotal off-topic, but I worked part-time at a hatchery during my bachelors, and the definition of organic chicken as minimally required by law was simply that the eggs hatch at the farm ground rather than in boxes with 500 other eggs). What is AI-processed is not even defined in this way yet.
5) Following my slightly cynical market analysis in the previous answers, you could say that right now your "Looking for better options" while continuing to collect the paycheck is the textbook definition of revealed market preference from 3) and 4).
You are pricing the survival of human civilization at slightly less than the friction cost of taking a pay cut or switching jobs. If you won't absorb that cost yourself, you should think of the rational basis you have to expect millions of strangers to do it for you.
----
Finally, neither of your historical examples proves that moral coordination beats material incentives. Nuclear war was avoided via Mutually Assured Destruction a game-theoretic equilibrium of deterrence, not moral disarmament (we still have thousands of warheads).
Chattel slavery was only dismantled in the West after industrial capitalism made wage labor more efficient, and even then it required a brutal civil war, and not social norms.
----
I think you can't beat Moloch with petitioning and moral ideals. You either extract enough surplus using the tools we have today while its not yet common knowledge in order to buy yourself a lifeboat(I think you are also doing this given your job), or just drown on principle idk.
Perhaps we would all benefit from another run of Moby Dick in Broadway.
> It appears that Anthropic has just solved the Riemann Hypothesis, and I have Thoughts. So I’m going to start by telling you about a musical I saw last week.
There might be great info here but I’ll never know. This kind of non sequitur manufactured quirkiness in writing is such a turn off to reading any further.
It’s only catastrophic if you pin your ego and sense of self to being special, or the dream you may one day be special. “I’m a temporarily embarrassed not yet recognized genius damn it!”
If you’ve discovered the ultimate happiness hack of realizing that you need no permission to exist and that life has value simply because it is, it’s actually fun to watch. Imagine what we will get to learn.
You’re not special anyway. None of us are.
People also forget that science and even math have felt stuck for a long time. Physics feels stuck. Maybe we hit the limits of what un-augmented human brains can achieve. Maybe without AI the future was just stagnation.
I feel like whole neoliberal schooling systems will have to change for people to understand this. Until then it will be doom and gloom in the countries that have this value system.
We are molded from a young age to compete, do well on tests, and in general to derive value from work and achievement instead of just being, or in some other cultures social values.
Why the neoliberal qualifier? Seems this way everywhere.
I think it’s normal human behavior. You think young people in old school indigenous tribes didn’t compete and show off and try to one up each other?
I think this is just a perspective that can be cultivated. I came to it from reading a lot and reflecting on life experience. It’s not mutually exclusive with achievement, just means it doesn’t have to define your whole being.
It’s a view we might need to cultivate.
"Neoliberal" has been used as a catch-all for "something I disapprove" for a while now.
We've evolved to work and achieve. People do poorly when they get everything for nothing.
The aristocrats of old seemed to do just fine getting everything for nothing :)
It's very expensive to run a feudal estate. For example, you've gotta pay your guards, pay to maintain your castle defenses, pay tribute to the king, etc.
Sure what we we work for and achieve is assigned a simple monetary value.
But we can work for other things - relationships, love, hedonism, thrill, understanding. Those all take work, lots of work, certainly a lifetimes worth. But alas, they do not produce value.
Yes, I imagine if we all lived in ivory towers all alone with everything handed to us we would all go crazy. But, if there were other people around, we would have new goals to achieve.
There will still be great artists, writers and musicians. Those people will be special in the eyes of many. There will always be those who strive to create. Even if AI ends up completely replacing the role of the human in scientific discovery (I have my doubts), that doesn't complete the project of converting humanity to a dull, undifferentiated egalitarian mass.
Anyways, it's fine to be content. I do understand your post.
Science and math only felt stuck if you weren’t working in the field.
1. It is better to see the world as it is, than any other version.
2. Gratitude is profoundly the greatest key to a well-adjusted outlook.
"1. It is better to see the world as it is, than any other version"
Well, evolution made us filters, we just see a small spectrum (and could not process suddenly seeing all). But it is good that we now know that there is more. And should take that into account. That there are things literally too small to be seen, because light has a wavelength.
> It is better to see the world as it is, than any other version.
And yet, in mathematics there are many proofs of the same statement, and some are considered better and more insightful than others. It is better to strive for a good proof, than to be content with only a certificate that a theorem is true.
It is absolutely absurd to claim that AI proofs and discoveries are not human achievements. The artificial mind was designed by humans, and the machine that runs the mind was built by humans. It isn't alien technology or an insect hive mind that did these things. The achievement is so human it hurts.
Invoking the Copernican principle in the same breath as lamenting the end of human achievement is ... I don't even have a word for it. Ignorant? The egocentrism is maddening.
There's an important qualitative difference between an achievement reached by an individual or discernible group of people, and an achievement reached by a gestalt of all publicly available human knowledge.
"Achievement reached by gestalt of all human knowledge" is honestly a pretty cool and inspiring way to think about it. Many parallels to the internet in that regard.
but isn't it ..kind of... a human hive mind?
it is just residing in silico
> The artificial mind was designed by humans
It’s more like an experiment that turned out to have much more powerful results than anyone expected. And we only half understand how or why it works. It’s more a discovery than an invention.
It's equally both. LLMs are just humans discovering that the nature of reality is holographic. Compression is intelligence because compression leads us down the ladder of deeper holistic order. No surprise that LLM loops is the 'best' route to go with LLM system design.
> the nature of reality is holographic
So Bill Cipher was right all along!
Hardly absurd and yet it moves you to say it.
You're definitely right about it hurting humans.
Can I enter a forklift at a weightlifting competition? It was designed and built by humans, after all.
Or is there in fact a very real and obvious distinction between doing a thing yourself and making a machine that does the thing?
Should you run a warehouse without forklifts?
My only issue about LLMs doing scientific advancements is that the actual models that did this and the training process are industrial secrets and not open weight releases.
I don't want the cyberpunk dystopia, I want the star trek utopia, that's where my attention is focusing on.
Star Trek is fiction. The role of super-advanced and sentient computers in Star Trek is deliberately kept limited in scope, so as to advance Roddenberry's belief in the unshakable human spirit and noble human yearning for discovery or whatever. Whenever machines are presented as capable of supplanting or exceeding humanity, as in the TOS episode "The Ultimate Computer", or anything involving Lore or the Borg in TNG, it's universally depicted as existentially threatening.
The proof was guided by a human. His questions injected large doses of creativity into the process. Maybe 100% of it. Obligatory quote: "one good question is worth a thousand good answers"
anthropic actually means belonging to humans btw
As in, the AI / Company humans made, when humans were still a thing.
For centuries, some people have been convinced that they were the smartest beings on earth.
Now, they're just realizing that they're not. Time to relisten to my favorite Coldplay's song : Lost.