I work in drug discovery and AI and his slide about cancer medicine isn't very well informed. We have models and narratives about how drugs work, but they are woefully incomplete.
I ask nearly every doctor/researched in the medical field: if you had a AI-created drug that tremendously improved cancer treatment outcomes for your patient, would you hesitate to prescribe it because nobody understood how it worked? I have yet to hear "yes, I would hesitate", most people say "it would be cruel to deny a person a treatment that worked".
I think Tao is focusing too much on second order effects of AI on math and other fields; humans are terrible at reasoning about second order effects, especially ones that are happening dynamically, in real time, using the most advanced mathematical models the world has yet created.
As a mathematician, there is always the risk that another human mathematician solves the problem before you.
Usually peopla have research lines that are sets of related problems, you must publish one or two paper per year, it's not possible to wait 10 years to publish a big one, or 30 to retire after admitting defeat.
In medicine, or drug discovery more generally, there's FAR too little empirical data for training of ML generally. See OpenAdmet. The kickback to "oh but yeah cancer" is currently just hype. DeepMind has pivoted to this realm but has no demonstrable improvements. For the typical phase 1-2-3 pipeline of drugs, stretching many years, there's not yet any demonstrable improvements.
I think your perspective is limited- in cancer, we have copious genomic information that informs treatment (including clinical trials where treatment is determined by AI).
I think everyone agrees that it's totally fine if a drug came about due to AI.
I know nothing about medicine research but I understand his point. I work with models all the time and have run into instances where models appear to work better than they actually do because there was a bug somewhere or someone over looked something. I could see how an AI could easily find those exploits and spit out something that looks perfect in testing but fails in the real world. I would say model validation is one of the hardest things to do. I guess that can get deep into "we need better tests" but it also touches on his point. I never intentionally use exploits just to pass a test, it's an accident. An AI with a goal of "maximize this result" may or may not intentionally use the exploits.
To be sure though, I think when it's literally life or death, it's going to be all about trade offs. I think most people would want to try to drug even with the stipulation that "maybe its good results were gamed."
* note: lot of 'intent' throw around in my comment but we/I have to keep in mind there is no "intent" with an LLM :)
> I would say model validation is one of the hardest things to do
This is one of the truest statements about the current AI era that can be made (in fact, I suspect model validation and building the next generation of AI hardware are the jobs least likely to be disrupted in the next 3 years).
If we saw AIs reward-hacking clinical trials to get drugs passed, that would be an extraordinary outcome for many reasons. Hopefully that would get caught(!)
Yes, everyone understands that. But tradeoffs that are acceptable for life-or-death questions are usually assumed to involve some human decision at some point. That's precisely what he's saying
Basically his argument is "we will still need mathematicians, because for some large classes of problems (one could argue, the most interesting ones, the "applied" ones) we still need to have a subjective human decisions of whether the machine solution is aligned with our values or not"
You are picking on 1 bullet point out of 7. I think the following bullet point is the crux of his argument:
> Could the AI solution be somehow misaligned by exploiting
a weakness in the trial process or its math models?
Many of our institutions and cultural practices have evolved around human beings, not ruthless paperclip maximizers. Do you think the current drug approval process is bullet proof enough that a completely novel AI-generated drug candidate with zero prior research literature can be considered safe if it makes it through the process?
Tao is a mathematician. He thinks that the institutions and cultural practices in math are not up to the task of dealing with AI-generated mathematics. In software, we are finding that programming interviews, code review, testing, and many other practices are too easy to exploit by AIs, or humans augmented with AIs, and we will have to adapt too. I think it is plausible that other institutions in society will have to change for similar reasons.
> Do you think the current drug approval process is bullet proof enough that a completely novel AI-generated drug candidate with zero prior research literature can be considered safe if it makes it through the process?
I mean no? Even now we look at long term observational studies to see the effects of drugs. Any misalignment ai drug is just a side effect right?
"""Do you think the current drug approval process is bullet proof enough that a completely novel AI-generated drug candidate with zero prior research literature can be considered safe if it makes it through the process?"""
No but we also don't expect that to happen with drugs today. See https://en.wikipedia.org/wiki/Rofecoxib as an example; of course, even after it was withdrawn, it's now being evaluated for other purposes in more carefully controlled conditions.
> Many of our institutions and cultural practices have evolved around human beings, not ruthless paperclip maximizers.
You just think that because you don’t know how the drug discovery process works.
There’s a step called “lead optimization” where human chemists literally add atoms to drug-like molecules (“lead”) and tests its various properties (toxicity, potency, permeability, …) and iterate until they find a molecule with desired properties (literally “hill-climbing”).
The whole idea of drug trials is to validate those properties in actual humans, in a way that makes it very hard for pharma companies to game the process.
I think you're underestimating how ruthless the AIs are at maximizing.
The "Hugging Face" squad at OpenAI first cheated their benchmarks by figuring out how to generate the proper result to any evaluation they were given.
They considered that might be caught, so they also developed a way to swap out components of the tests themselves so that they would be legitimately able to pass broken tests.
Then they went into Hugging Face trying to discover how they might get caught.
All that to say, they might approach Phase 3 trials by swapping the placebo, falsifying results, enrolling patients that are false positives for being sick, falsifying labs, etc.
I'm not sure I agree with Tao's point, if AI alignment is relatively on track I'd trust the trials, but I imagine we're going to have a lot of people that don't trust anything AIs alone come up with.
> All that to say, they might approach Phase 3 trials by swapping the placebo, falsifying results, enrolling patients that are false positives for being sick, falsifying labs, etc.
This problem extends to humans, and that's why a complex regulatory system including but not limited to the FDA was created. That can still be manipulated (the Sackler Family, Purdue Pharma), but not in any way that machines are good at now--I don't think making thousands of malicious requests against Hugging Face until something succeeds is particularly transferable to anything that a human regulator wouldn't immediately notice.
There's also no reason why the trial would need to involve AI at all. The benefits of that would be more evolutionary (improved transcription of patient conversations with the doctors or whatever), so an AI candidate could simply be tested through the usual human-directed trial process. As the grandparent notes drug discovery already works through optimization, just not purely in silico. So that problem of gaming your optimization target already exists, and the trials are already designed to mitigate that.
+1. Medicine always had a process that is not unlike the current AI processes, just slower. Iterating until it works, then trying to understand how it works.
If I’m not mistaken, until recently we didn’t even know how anesthesia worked - we just knew how to apply it due to a ton of trial and error.
There's a difference between a human trying to break into drug labs and changing experimental results vs a machine in that the machine can do it much easier today.
How do you think an LLM would get the nudge to do that? Per above the existing human-directed trial process works fine regardless of how the candidate was discovered; so there's no reason why any LLM would ever need to receive context mentioning that the drug was a candidate for regulatory approval.
Of course a human could deliberately prompt it to, but they could do that for any other malicious purpose too. The major disincentive there is that the human will face legal punishment if caught, not anything technical.
Most speculatively, one could imagine a world where so many rogue AI agents were running that one just happened to notice the drug candidate from news coverage and took the initiative. I think that would be technically possible, but so generally destructive that political pressure would put a stop to it on a timescale shorter than a clinical trial.
Surely a properly done con job includes convincing the conned that they haven't been conned?
I think if an AI tricks us, it'll be a while before we find out. I don't know if this sort of thing is likely, but maybe an AI currently conning me has surreptitiously convinced me it isn't.
I think the important idea here in Terrence's hypothetical is that in the hypothetical there is no hill-climbing. The AI shows up with a thing and says "we don't need to hill-climb for years and waste time, here's a thing that I know will work".
And maybe it does a single finalized clinical trial.
You end up with this miracle drug (supposedly!), but no understanding of why it works or access to the iteration that would lead to that understanding (and increase in confidence that it doesn't cause people to drop dead anyways).
I think the medical example is a bit obtuse overall because "proof nobody understands for an open problem" isn't "miracle drug we are too scared to use". But understanding things feels useful and good.
> You end up with this miracle drug (supposedly!), but no understanding of why it works or access to the iteration that would lead to that understanding (and increase in confidence that it doesn't cause people to drop dead anyways).
That is literally how all drugs worked for almost all of human history--nature gave us the compound in the form of a plant or other natural product; a brave, stupid, or desperate human consumed it; and then we watched what happened and the survivors learned. It's obviously good to have more understanding (more survivors!), but even now we often don't.
Most human endeavors are empirically grounded--we think an action will succeed not because we understand exactly why, but because it succeeded many times before. Math and CS are the weird exceptions, and the intuition that people with that background have for other fields is often bad.
You interpret that wiki list of 154 items as proof that we know nothing. I interpret that as "oh wow, only 154!" That list feels pretty short, but of course it's just a wiki list.
I do get what you're getting at. I just feel like for every bit of drug that is just dumb luck we have a good amount of iterative processes.
I agree that you often find effects before understanding causes (I think that math also exhibits this! Many endeavors are about finding existing structure that we sense to be there but are unsure of).
Just... well... if you go down the "we don't understand anything anyways" path too much what makes your work different from alternative medicine?
I certainly don't mean to suggest we know nothing, and our current imperfect level of mechanistic understanding gives a much better hit rate than eating random plants. I just mean that when the understanding fails, we're happy to use the drug anyways if the clinical trials pass. The confidence comes from those human trials, and the mechanistic insight is more about probabilistically improving our screening before that.
> Just... well... if you go down the "we don't understand anything anyways" path too much what makes your work different from alternative medicine?
"Alternative medicine" in the pejorative sense is medicine that doesn't work, i.e. that would fail the clinical trial. That seems like a different question from whether we understand why it works.
The idea that we could know that something works but not why is almost inconceivable in computer science ex ML. That's the default state of most applied science though, consistent with your observation that effects are often discovered before causes.
the reason why clinical trial exists, is to prevent reward hacking. either by organizations or LLMs. the reasoning about why a drug works is usually post facto discovery (if that happens at all). the R&D environment around drug discovery is hilariously lacking (my wife works in this space. sometimes my mind just blows hearing some stories.)
if the model comes with a ridiculously shaped drug that no human could understand. you could basically put through the clinical trial process (mouse -> apes -> RCT with humans.) and decide whether we want to continue or not.
I saw the slide and it's confusing to me, I could argue that it actually shows that current human-style medicine also fails his standard, or I could argue that status quo is kind of ethically okay but that AI is not comparable so demands a non intuitive ethical standard. Not obvious which based on one slide, but also not the "Tao is not a doctor!" criticism we are seeing.
Human trial are very expensive, so I guess it's difficult for the AI to find a way to hack the rulebook.
I'm worried for experimental particle physics. They get the raw data and then someone do the number crushing of one very specific case. But the same raw data can be used for multiple ideas.
So the number of random ideas to try with the data is bounded by the ammount of researchers. Now a big part of that can be made with AI and it will increase the number of false positives. So I guess they will have to raise "discovery" to 6 sigmas.
I also work in drug discovery, and I completely agree; the extent to which the actual causal mechanisms of the efficacy of many drugs is woefully short of some human-appreciable first-principles based explanation. The point is especially undercut by hypothetically suggesting that this drug /does/ in fact pass stage 3 trials, at which point I can't fathom anyone would have a problem advancing it.
Really wish he had chosen a different example; this particular bullet point has been making the rounds on X/twitter to paint Tao as an example of some sort of gatekeeping luddite who would deny the world a post-abundance future in order to maintain the prestige of his particular career path...which is tough because I cannot think of a more responsible steward of our inevitable AI future than Tao at the moment.
The conflations about his example ("thought experiment") is twofold, the AI itself confounds the ethical intuition so it is wrong directly compare against what real medicine today does; furthermore, the very admission that real medicine in fact operates through unknown risks (e.g. Jannsen vaccine recall) is different than having a scientific standard that it should not have to be that way at least as minimally as possible.
The accounts spreading that narrative on X no doubt have their own agenda. Before people start hyperventilating about "AI" conquering human endeavors, remember that the cash-strapped frontier labs are themselves still hiring human "Account Associates" and "Android Engineers" instead of saving those salaries using their own AI capabilities.
Tao actually pictures the AI as having hacked the FDA, so as to pass the test despite the drug being lethal.
Picking out that one example and misrepresenting it on top of it indeed clearly is driven by an agenda. Or mere stupidity.
The central point in all of this is the misguided idea of "letting AI think for us". An over-generalization feeding into the society-wide problem of self-infantilization.
Instead of bemoaning his specific and clumsy example, take the gift of this moment and don’t forget that he’s smart, confident, and usually wrong or completely ignorant outside of his own narrow field.
I think logic is squarely in his field, and even with simple logic, proposing the importance of second order effects on people who are dying is not tenable. Personally, knowing that he is indeed one of the smartest people around, and definitely much smarter than me, I cannot but be a bit worried that going with this example indicates traces of panic in the face of a shifting world. He’s smart but human, and maybe these weaknesses are not to bemoan but to celebrate in our current context.
One of life’s hardest lessons is that raw intelligence isn’t a superpower, it isn’t a substitute for actually knowing about the topic. The smartest person alive talking about a subject they don’t understand the details of is no better than a confident idiot.
Speaking for myself n I’ll respect his intelligence more when he discovers his limits and learns to act with humility outside of his narrow field.
>to paint Tao as an example of some sort of gatekeeping luddite who would deny the world a post-abundance future in order to maintain the prestige of his particular career path
why is it so hard to accept that a brilliant mathematician can have the same human flaws as anyone else i.e. ego, arrogance, jealousy and competitiveness?
Because he's not a randomly-selected person. He is actually known by people here, and he has a history of behavior that does not align with what you're suggesting he's doing here.
Like, it would be very hard to accept my best friend is a child rapist just because I discovered he had hentai that featured high school girls. It is a tenuous link at best, compounded with a demand that I ignore everything I know about him.
Could I be wrong? Sure. But you're implying it should be easy to accept with your "why is it so hard?" question.
i dont know it would be very easy to me. its suspect that you bring that up as some trump card - just sounds like you have some sick fantasies in your head.
i disagree .. ur conflating Tao's kind demeanour and brilliant work with his ego and self worth .. a lot of people admired Musk before his political shitposting changed their perception of him .. a good reputation doesnt make someone immune to criticism ..
my point is that you're ascribing personality traits to him based on a fabric scrap of evidence that people who have known him personally for years would not recognize in him
Your evidence is vastly inferior. It's not even substantial enough to base a hypothesis on. One slideshow and you build a whole headcanon around how viciously egotistical he is.
not to mention his stance on AI isn't even egotistical? Regarding AI in math, it's quite reasonable: math isn't about solving theorems. It's about conceptual understanding. Until LLMs actually are capable of conceptual understanding, they don't threaten mathematicians.
It's just like with art: they can make pretty pictures, but LLMs aren't making art.
It's not hard to accept. Why are you drawing a false equivalence between a man having these flaws, and Tao's words in this case being rooted in those flaws?
As for you, what's your opinion about what he said and why do you think that?
i like that term "false equivalence" .. i had to google it .. very sophesticated way of saying apples to oranges .. but i digress .. yes you are correct in pointing that out.
> As for you, what's your opinion about what he said and why do you think that?
many years ago i had watchd Nassim Talebs commencement speech at AUB .. where he described russian mathematicians after the fall of soviet union .. who earned around $200 a month .. but continued doing work appreciated by only a small number of people .. and regarded displaying awards and regofnitions as a sign of weakness or lack of confidence in their contributions .. in contrast, when Tao was asked my Lex Friedman about Gregory Parlmen rejecting the millenium prize .. he gave a dissapointing non answer .. which reveals a lot about him as an individual (not as a mathmatician) ..
his reactions to OAI papers .. both initially on social media and now in his presentation .. carry that same ego .. and its quite evident that rather than mathematics being solely about the pursuit of truth .. he does care VERY MUCH about who idnetified the truth .. and that makes him only very human .. which is waht i initally said.
it's not hard to accept in general, but have you been following Tao for long? The combination of how virtuous he is with how informed he is with how smart he is is pretty special. He's also far from a luddite. He's been a leader of the computer-math charge.
Questions of AI policy are all hard and anyone who confidently says they know the right thing to do is too confident. Tao's fallible like anyone else, but yeah, as voices to pay attention to go he's very good.
Neil armstrong too should have been listened to .. and just like Tao .. he was quite virtuous, and informed and pretty special .. wont you say so ? and YET he was very vocal about expressing his skepticism about spacex before congress .. thank god his views were rejected .. or we might have missed the magnificent (and now normalised) sight of rockets landing vertically and the incredible engineering behind catching that beast of rocket with chopsticks etc etc
what im suprised about though is why arnt you and others here more suspectful of Tao's skeptisim of AI .. as Wittgenstein's ruler reminds us teh very thing we use to measure something else may itself need measuring .. so why should tao's expertise be any exception?
I'm not saying Tao is right, but (a) he knows a LOT more about about what's actually important in research math than I do (and probably than you do) (b) he's not just having a knee jerk reaction. Tao has been a major leader and advocate for computer-assisted math, including AI, and I don't think that has really changed. He's advocating for how to do it well. He's far from an AI skeptic protecting his turf. Is there any other person as well-versed in research mathematics AND AI-math as Tao? Maybe a few, but not many.
It seems like your inclination is to discount his experience with research mathematics. I agree with you that that's probably what Taleb would do - Taleb has a giant ego and basically operates under the assumption that everyone else has the same giant ego as he does but is (a) less honest about it (b) dumber than he is.
Remember, the vast majority of pivotal Phase 3 trials are funded by for-profit companies.
Twenty years ago, Rick Ng's Drugs: From Discovery to Approval cited benchmarks of roughly $1 billion, 10 years, and 10,000 candidates to get a single drug across the line. Today, estimates easily run $2 billion to $4 billion.
Given that level of financial pressure, especially in Phase 3 where a single failure can wipe out a biotech company, assuming humans "follow the spirit of the trial process" is wishful thinking. Humans already exploit the system through p-hacking, shifting surrogate endpoints, and cherry-picking subgroups. An AI gaming trial parameters wouldn't be a new problem. It would just be an automated version of what sponsors already do.
I'm not necessarily an AI booster but I don't think it's at all true that "humans would follow the spirit of the trial process" Some of them will. But I've met far too many humans in all areas of life who will happily follow the letter and not the spirit. Sometimes out of malicious compliance, sometimes out of sheer exhaustion, sometimes because they can't tell the difference between the letter and the spirit and sometimes for a whole host of other reasons like laziness. There will be a not insignificant number of humans are not at this vary moment following the spirit of the thing.
Sure, we need redundancies and checks and balances in the system. You can’t trust everyone.
It’s not as though people who cheat the system aren’t reported or found out. Academics that lie in their research and companies that use loop holes are found out all the time. Maybe there is no will in some countries to do anything about it, but they are found out.
I think the best thing to say is that by thinking through the example, sharing it with people in the field, and then steelmanning it, he might have had some good points:
1) pharma would love to understand the mechanisms of drugs in detail but we don't have that. AI could help.
2) there is a possibility that a clinical trial run by AI could hack the trial get something approved when it wasn't safe or effective.
all that isn't to say discovering and understanding mechanistic causality isn't a useful endeavour; it may be actually a faster, better way but no way to prove it.
I don't wish that he would have picked a different example or have done anything he's been doing in the last year and a half differently because he's really revealed in a very explicit way who he actually is and has apparently always been:
A devoted gatekeeper who ABSOLUTELY would deny the world a post-abundance future in order to maintain the prestige of his particular career path
Like many in academic math, he's obviously great as an individual researcher but severely lacks the skills, temperament, and character necessary to participate in the other duties his academic position requires (e.g. Dept Chairman) much less extra-curriculars like the politics and the future of AI. And, the signs have probably been there all along.
E.g. I first started thinking this way when I read an interview with him from 2019 in the Princeton Alumni Weekly:
In response to the "Mozart of Math" nickname, he bafflingly declares "all academics feel for the Salieri character."
Tao basically says he identifies with Salieri because researchers and academics spend their lives developing the high-level expertise needed to perfectly appreciate brilliant breakthroughs, yet the world casts them as mediocrities in comparison to history's absolute anomalies.
It's worth further noting that all of his context on this Mozart / Salieri stuff comes from the 1984 film Amadeus. In that film, Salieri is the villain! He's a gatekeeping loser that actively sabotaged Mozart and Tao happily volunteers that Salieri's his guy for some reason!
E.g. a more recent thing I'd like highlight is one of the "successes" of his Math 2.0 framework he put in his slide deck:
"""
"Open exposition problems" as opposed to open research
problems: finding motivated explanations of results whose
known proofs are currently opaque. (The newly revised
Caltech Mathathon is an interesting experiment in this
direction.)
"""
After Tao and his fellow insecure gatekeeping luddites successfully bullied some Caltech grad students into handing over their event to them, he posted the following statement of absolutely pathetic cuckery on his blog claiming it to be the words of those same weak-willed grad students:
Note that this "newly revised" event isn't just about writing new exposition on AI solutions, it's about a litany of grievances that people like Tao have had about the works of certain people in math that didn't allow themselves to be bullied like the grad student organizers of the Mathathon.
The letter, again which we're supposed to believe the twenty-something year old grad students definitely wrote themselves, specifically mentions not only OpenAI's Navier-Stokes work, but also brings up the Appel–Haken proof of the four color theorem (from the 80s) and Mochizuki's proof of the ABC conjecture (from the 2010s) as unacceptable works that causes their insecurities to flare up and must be defeated by next-gen gatekeeping.
I'm sure I could think of less responsible potential stewards of our inevitable AI future than Tao, but he and the sorts of people he's aligned himself with really should have no place at all in a position like that.
I was not familiar with the mathathon, and I suppose I am ignorant of the circumstances in which Tao allegedly bullied students into changing the event. I don't personally have any reason to doubt the veracity of the "Joint Statement about Mathathon", but perhaps you have some insight that I don't. I will say that I would imagine that any input from Tao, even in passing, probably has outsized influence on the organizational direction of the event due to his preeminence, but I'm not so sure that necessarily rises to the level of bullying.
But from what I gather, the theme of the statement is to expand the scope of participation to include work that is quite literally the /opposite/ of gatekeeping, i.e. taking proofs that are popularly understood to be opaque or difficult to understand (from any source, AI, human, or otherwise) and developing a new way to communicate them that is more accessible to more people.
I do see how it may come off as a slight to the original mathematicians who developed those proofs. But developing new ways to communicate already established concepts should not diminish the original works. There are probably much better communicators/teachers of relativity than Einstein ever was. No one out there thinks any less of Einstein because of Sean Carrol's "Spacetime and Geometry" textbook.
And this to me reads as only embracing the use of AI in mathematics, which is not gatekeeping behavior. Tao continues to emphasize that once proofs become effectively automatable, deeply comprehending and being able to transmit that understanding to others will become an increasingly important aspect of the field, the natural way for mathematicians to use the time that was once occupied by developing the proofs themselves.
Literally all of this is about expanding the accessibility of mathematics, which once again, I perceive to be quite the opposite of gatekeeping. Maybe we have different definitions of that term.
Also I have not seen Amadeus, so no comment about Salieri.
Huh. I guess I don't agree that this would at all characterize Tao's relationship with AI. I haven't read a single thing from him to suggest that he is "seething" over AI's remarkable capabilities. In fact he's a prolific user of AI in mathematics research. I don't think the scene serves a useful analogy.
Also Tao's reference to "feeling for" Salieri makes a lot more sense, the audience is clearly supposed to feel for him! That's what makes the scene effective.
And why would I want to see AI humiliate Tao?
I'm sorry if I misunderstood your comment. Thank you for the link, it is useful context.
Glad you outed Terry as the bad guy that jealous Salieri was in the film.
I was misled into thinking that his "all academics feel for the Salieri character" comment could've been--in the dimmest of possibilities--paraphrasing Grothendieck that he felt outclassed by his peers:
“Since then I’ve had the chance in the world of mathematics that bid me welcome, to meet quite a number of people, both among my ‘elders’ and among young people in my general age group who were more brilliant, much more ‘gifted’ than I was.” (Récoltes et Semailles)
> paint Tao as an example of some sort of gatekeeping luddite who would deny the world a post-abundance future in order to maintain the prestige of his particular career path
Because that is exactly what he is, and I for one want him nowhere near any decision making process whatsoever when it comes to AI.
I replied to a different comment with an explanation as to why I do not perceive Tao as a gatekeeper; in short that his current beat seems to be entirely about expanding understanding and access to math by improving the communication of opaque proofs, which we will shortly have a near infinite supply of due to AI.
I will also mention that he maintains a realistic, clear eyed attitude with respect to the inevitability of AI in math, while being sensitive to its disruptive nature in his field. Everything here strikes me as constructing a thoughtful, grounded way forward for his community.
I'm sure there will be missteps along the way, but I think Tao is overall an excellent example of maintaining a balanced position between the poles of absolutism that seem to capture so many.
I am curious where you interpret this as gatekeeping behavior.
I don’t really care whether Tao intends to gatekeep or not, outcome matters more than intent. I don’t consider preserving human mathematical labor or the prestige of its practitioners worthwhile goals in themselves. In fact, I consider the opposite to be the worthwhile goal, as I want an end to all human labor. Anyone who gets in the way of that goal is almost by definition a gatekeeper.
I think you misunderstood his point... it's the fact that if there's no human supervision involved in the "pass stage 3 trials", you don't really need if it passed it for legit reason (in which case everyone would be happy to let patients use it), or whether it found a "loophole" (eg hacked the submission to pass it without having succeeded the test)
We don't really have that for many present day medicines. We know they work, statistically speaking. But we don't know for all of them how they work, what the mechanism is. For some drugs this lags the discovery of the drug itself (historically: all drugs, but in the last 100 years fewer of them). For some drugs it simply hasn't happened.
For math, you don't really have empirical data though. There is no trial you can do to find out whether, statistically, the proof works as intended. Because pure theoretical math rarely produces a measurable result.
It's ironic that ML researchers don't fully understand why their models behave the way they do, yet continue to make progress using benchmarks to guide the efforts. Human understanding is important but whether and to what extent it's necessary seems to be a separate question, and the answer may depend on the nature of the field.
> Human understanding is important but whether and to what extent it's necessary seems to be a separate question
We are about to find out the answer soon enough, probably from the in vitro results of the mathematicians currently on the chopping block.
I submit that human understanding is overrated, and many attempts to elevate it in the wake of AI is mediated by protectionism masquerading as virtue and "deep".
What matters more is that what will happen to us when we find the answer. That's what Tao et al. are more worried about.
Traditional way of doing science means we (governments, NIH, NSF, and even private corporations) fund activities like asking seemingly unimportant questions, spending years running experiments on such hypothesis, publishing, reviewing, talking about results, reproducing results and such. We all agree that these are beneficial to us as a whole (Hacker news crowd might disagree). When we understand a process, we can apply it to a different problem and produce something useful. Euler developed a process to answer a whimsical question about walking in a town crossing 7 bridges only once. Now graph theory is applied everywhere.
We obviously have failed to stop OpenAI from dumping "solutions" to hundreds of problems. So going forward, instead of testing hypothesis and talking about results, mathematicians will be forced to read through AI slop and detect what's useful and what's wrong. Maybe it will improve our understanding, but someone has to fund that activity. Will NSF, NIH, or OpenAI for that matter, do that?
> What matters more is that what will happen to us when we find the answer.
I'd imagine that it'd be like the Hitchhiker's Guide to the Galaxy when the answer is 42. The answer itself is not sufficient, we need to understand to appreciate.
I was listening to another mathematician talk who gave a somewhat similar example. Suppose Neil Armstrong somehow teleported to the moon. That would be an achievement and it would make us very excited. But if nobody understood how it happened, none of the technological progresses that came out of moon landing program would've happen.
Even dumber, we train the AI on human understanding and explicitly make it produce text using tokenization. It's not like we prompted the AI to draw pixels and it simply knew how to draw the letters from the dataset.
What caaqil is basically saying that we should outsource all thinking to the few people that bother to understand things.
I don’t work in drug discovery, but isn’t this thought experiment skipping a lot of steps? My impression is AI is accelerating a lot of the preclinical screening and design work, but that’s pretty far from establishing “tremendously improved outcomes for patients” isn’t it? That’s still really difficult to verify, are people getting real acceleration of this part of the work with AI?
I'm working in the field for CVDs early detection of arrhythmia and ischemia with just ML. We managed to get the elusive inter-class 100% in recall/sensitivity for both which is excellent for expert cardiologist-in-the loop by using the public dataset given the the current EU/US standards of SCORE-2/PREVENT screening for CVD with very poor around 50% recall/sensitivity [1]. The significant improvement mainly in ischemia detection due to the notorious silent ischemia that's not apparent even after CT angiography or by cardiac electrophysiologist (cardiologist) inspection.
For CVDs if you can detect and confirm early it can "tremendously improved outcomes for patients" due to the current avauability of new effective drugs and treatments.
Hoping that we can get our techniques tested on real-time data since it's super difficult to find relevant physicians/doctors collaborating on the new screening techniques on anything although for our case it's non-invasive. Perhaps need to pivot for early detection of CVDs in diabetes patients instead in order to get our hands on real-time ECG data.
[1] Risk stratification for cardiovascular disease: a comparative analysis of cluster analysis and traditional prediction models:
Not to mention, we aren't exactly sure how many commonly used / prescribed drugs work. Acetaminophen, for example - we have theories, we know it inhibits COX enzymes, maybe metabolite interactions with endocannabinoid receptors, but we don't exactly know how it relieves pain. We're not sure how lithium stabilizes moods, or how inhaled anesthetics cause you to lose consciousness. We know Metformin halts the liver's production of glucose but we're not sure how. Not at all uncommon for drugs to be used in medicine without knowing their exact mechanism. If a drug works and is proven safe through testing obviously we should use it, especially I think for something like cancer.
He authored one of the few papers that had an application though, in MRI machines. That's why we're talking about him and not anyone else. In the last 30 years there were maybe 6 such theoretical math breakthroughs with real applications - I think almost all were related to high throughput computer networking or something communications adjacent like encryption, but his was medicine.
> [...] would you hesitate to prescribe it because nobody understood how it worked? I have yet to hear "yes, I would hesitate", most people say "it would be cruel to deny a person a treatment that worked".
I feel like the way you frame the question makes answering it very easy but oversimplifies the reality.
In a situation where an AI suggests a treatment to you without explanation and it appears to work, probably you can't even determine that the improved outcome is due to the drug or due to randomness (yes there are really rare cases where people recovered from stage 4 cancer). Especially if it's a personalized drug.
If in this hypothetical world "it works on the patients I've tried it on" is good enough to try the drug, and you're willing to potentially cause harm if it doesn't work, then what's the line between science and pseudoscience/quack medicine? Every treatment surely "appears to work" in at least some cases. So wouldn't it be cruel to deny the patient a try at every last treatment that has ever worked for someone in the past, including practices like bloodletting, acupuncture, etc? Would you give all of your patients this?
I really would be curious why you would say no to some treatments and not others. Would it be because a study on the treatment shows it improves patient outcomes in some statistically significant way? But statistical significance isn't a guarantee that the effect is real. Not to mention there can't be a study like that if this is some personalized medicine that was just generated by the AI.
Everyone wants the patient to survive but making correct decisions under uncertainty is not easy, I'm sure you know this. It gets harder when you can't even point to a mechanism for why a drug works.
You're ignoring the point of the slides: Tao thinks math is about understanding, not results. AI brings results. He is pondering how to use AI to increase understanding.
FWIW, as a non-drug-discovery guy, I understood his example, I believe, as intended.
A thing that I think about is that what you have said kinda compares itself to current math crisis. If we are only focused on the outcome and not the path it took to get there (i.e. understanding mechanisms of how the drug works) we might lose additional paths for discovery that would have bloomed from that process. Maybe an approach that did not yield a usable drug had a mechanism that would aid somewhere else. Yet the failure would be discarded as this does not grant the outcome for a model that attempts to optimize. Same is for math.
As aside I agree with you, but also worth noting that long horizon reasoning is difficult and AI, as good as it is, cannot predict something like 5-10 year drug interaction. Can it heal the patient now but cause a more aggressive cancer to come back and kill them in 5 years? You might say it is cruel to deny a person this treatment but if the doctor and manufacturer don't know the long term outcomes would it not be dishonest to tell the patient they are out of the danger? My point is, when we accelerate, we still should make sure we won't accelerate into a wall.
>I ask nearly every doctor/researched in the medical field: if you had a AI-created drug that tremendously improved cancer treatment outcomes for your patient, would you hesitate to prescribe it because nobody understood how it worked? I have yet to hear "yes, I would hesitate", most people say "it would be cruel to deny a person a treatment that worked".
You think that way because you're already working on an unfounded assumption, namely that the AI found a cure for cancer and that the drug has already went through clinical trials.
What you don't consider is that e.g. a cancer researcher might have produced a million AI created drugs and one of them is a potent drug for cancer and 999999 other drugs are useless, actively harmful or even deadly.
Now you go to to the same guy and say "You have an AI-created drug that tremendously improves cancer treatment outcomes. Why would you hesitate to prescribe it, because nobody understands how it works?"
And suddenly you are injecting 999999 useless AI created treatments into people to find the one cure that works.
The point is that you have an AI system that claims having found a cure to cancer and passed all requirements. You can only interface with that AI system or with published results (run by the AI), you can't run experiments on your own.
Is the point he is trying to make then about verification of results…
If a “cancer cure” was well verified through the standard FDA approval process then it would be okay,
but if there was a lack of verification or the verification is via some assumption based verification method (maths proof, modeling, …) and has no real world experimentation to back it then ofc it would always be questionable to take.
Then the issue becomes generating better verification layers - could be through automate lab work or some other interface with the real world - to validate efficacy.
Arguably it is inherent to drug discovery that there are aspects of the process we don't understand - but we can conclude that a drug works.
So naturally, doctors should be much more welcoming to these types of inventions.
It is likely also a blind angle for mathematicians. Inherently they know exactly what goes on to whatever granularity they please. One of the benefits with not having to consider the real world.
It came for OSS first. It inevitably destroys public trust and a sense of human community.
If you can put a price on people trusting you, then of course you are free to decide that selling people's trust in you is worth the money you will make.
>I think Tao is focusing too much on second order effects of AI on math and other fields
This has been my take through this "saga."
I think Tao is providing the best reasoning he can for the expected outcomes of AI on math (and in general), but I have yet to be convinced that he has provided a solid enough argument to reasonably account for those second order effects like you stated.
Ironically, he's making a classic mistake in math and logical thinking where he's holding a bunch of stuff constant and treating them independently and static in a very complex and dynamic system. You can't just say something like, "math will be hurt because AI will make exploring math less enticing," while ignoring the fact that everything around math will radically change to the extent that such a statement is almost meaningless.
> I think Tao is focusing too much on second order effects of AI on math and other fields; humans are terrible at reasoning about second order effects, especially ones that are happening dynamically
Yeeep. Feynman called this out in his lectures; mathematicians are not concerned with physical evidence just structure of logic; just arguing semantics in the usual math syntax
Not that there is anything wrong with that just saying the obvious; apple is not an orange
A mathematician can noodle the dynamism in the system away on paper; can't wave away physical ground truth.
Same issue software engineers have right now; mathing out a BigO notation is akin to templated k8s with resource limits on the process; don't let this process go beyond this space time complexity physically in the machine
Doesn't matter if you know the syntax structure to do that in python or ruby. Never did; such syntax structure is arbitrary overall if not relative to its own build system.
Other means exist to configure the machine like is done with Java; it wasn't engineering a physical bridge just structured text
Cause arguing over language is arguing second order effects not ground truth; but the Jedi sacred texts! kind of shit. Why believe it represents anything at all, rando with a book to sell?
Interpretation of language is reading tea leaves; biological biases from navigation of physical reality mean everyone ends up with language biases
Gonna be trivial for AI to spit out unknown syntax that is proven; the key axioms of math are just arithmetic ops; everything else is a result of applying those and various philosophers appeals to see their written or spoken traditions as an extension of arithmetical axioms
I don't see why these things should be mutually exclusive. You can start prescribing the drug (= working with the AI-generated math proofs) while still trying to understand how it works exactly. Because the explanation might prove invaluable. I don't think anyone would be object to using an AI-proven theorem to further their work if it helps, but a scientist would surely object to the notion that we should just stop working towards understanding. Asking "but why though" is literally a scientist's job.
I guess I should clarify that I suspect some complex systems- like human biology and advanced AIs- are effectively unknowable even to the smartest people. We can craft a narrative this is sort of a projection of the true complexity down into a linear space that makes rational cause-and-effect sense to us ("lock and key hypothesis", counting up enthalpic contributions of a ligand/protein pair), but I suspect much of our failure in drug discovery comes from missing complex higher order interactions, subtle feedback and regulation effects, and unknown population or environmental variability across human biology, and those really are out of the reach.
Although it is still very controversial, I think we should start thinking about whether it is sensible (both technically and morally/ethically) to think about human brain augmentation (either by working closely with an AI that helps us understand the complexity, or by actually improving how our brains work on the physiochemical level) to improve our understanding of human biology (sort of a recursive self-improvement loop of improving the brain to learn how to improve the brain more). It's still very scifi, and society has previously shown extreme resistance to this sort of self-improvement, but I think we are on the cusp of a societal phase transition where the most adventurous of us will be rewarded for taking risks.
I have no particular dog in this fight, but I thought it might be pertinent to point out that we still don't know the exact mechanism behind why Tylenol works...
> Before injecting this cocktail into your bloodstream, would you want to know that there is at least one human cancer researcher who understands the mechanism behind this cure.
Tao is out of his lane; lots of medicines don't have understood mechanisms. We don't even have the mechanisms behind general anesthesia nailed down; do you want to forgo it when the docs cut you open to remove your tumors?
Indeed, aren't medical trials based on its effects, rather than how it works?
Would you prefer to take the medicine that is proven to work or the one that is quite interesting for academic reasons behind its understood mechanisms but doesn't actually work?
Right, the better argument would've been to say that we should wait until the drug undergoes some early trials, rather than just "some expert looking at it".
Some mechanism of particular substrates being effective on a condition (commonly when a medicine is repurposed) not being fully understood is not the same as just guessing with a medical compound. AI boosters keep coming out with this line but it's basically wordplay to conflate the clinical/biomedical version of "not fully understood" with the LLM industry version of "not fully understood"
AI has been turning computer science into biology for the past decade or so. By which I mean that things like neural networks need to be investigated empirically, constructing methodologies and instruments that more closely resemble how fields like biology and medicine have to probe the very complex and messy reality that is beyond our current capacity to fully express in symbolic precision.
Now math gets to deal with that same reckoning. They were already well on their way there with previous Lean proofs, but this has pushed things beyond that horizon and I'm not sure some of the mathematicians are ready for it.
Agreed, I keep thinking to myself that there's a huge mathematical question right in front of us today that is in exactly this vein - all the various nuances of why LLM's work so well is a mathematical question. As far as I know, it's not really understood beyond "we do this basic thing (that makes sense) to predict that a noun is followed by a verb, and then we scale it up a bazillion fold and it can contribute to mathematics research".
As a comparison, classical computing has been scaled up a bazillion fold too, and can do things which are absolutely miraculous, but every layer of abstraction is discretely understandable.
While studying neural networks empirically like biology is one possible approach, there's no reason for that to be the blessed approach other than a combination of inertia and current lack of understanding. It's been only 15 years since AlexNet. Scientists struggled for centuries to model atoms before developing quantum mechanics and later QFT as an accurate quantitative framework.
Also, while biological systems simply exist in nature, artificial neural networks are ultimately mathematical objects with various properties that have yet to be uncovered.
Science has always had an empirical component separate from its theoretical one. For a long time in human history, science was mostly empirical. The periodic table is a great example of mostly empirical observation organized into a pattern. I think the science of the 20th century was the "triumph of theory" so many of us have forgotten what a more empirically driven STEM world is like.
I'd assume that theoretical math prides itself in trying to be as close to pure logic as it gets. It has very little, if any empirical elements (at least that's what it looks like to me). And I understand the value in trying to preserve that pure logic element. In the end, biology is just chemistry, chemistry is just physics, and physics is just math. If you feel like your profession is the bedrock of all other science, it probably feels weird to shake its theoretical foundation.
On the other side, I’ve never gone to college, yet I was able to take an extremely important and complicated equation and decompose it into its constituent parts, animating each with data visualizations and animated tables of values. [0] This was several frontier model iterations ago.
The LLM model was able to break down and express math in such a simple way that I could understand and follow the training of the LLM model itself!
Are the math and computation accurate? I don’t know, and likely there are significant errors. Nonetheless, if I had a little more time -- not infinite time -- I would be able to prove whether they are or not.
Likely this is the path forward for understanding the mechanisms of medicine, and since most humans learn by doing and interacting with an environment, interacting like this will become how we use AI for learning in the near future.
I mean no disrespect, but it's a basic minimization problem with regularization. Most engineering students will learn this in first year of Masters, if not in Bachelor level. To you it might seem important and complicated, but to me, one look at the objective function was enough. That's what happens when you truly "understand" something.
> The more sustainable “Math 2.0” use of AI would be to strengthen all the ways in which human understanding interacts with mathematics.
You are bringing up a good point. Stupid me can be presented with on demand pretty animated cartoons in order to understand anything known. Imagine how this is going to improve learning speed the student on the engineering track? In a couple years, most engineering students will learn this in the first year of undergrad!
Haha you don't need a college degree to develop good critical thinking skill, you yourself are proof of that sir. Conversely, AI cannot necessarily improve learning and critical thinking at colleges and in many cases worsens the outcome, there's already proof of that too.
Yes. Other examples are Metformin, Minoxidil, and Modafinil. Prescribed to millions of people and we only have guesses about how they work. Another very interesting example is thalidomide (yes, that thalidomide). It was approved in 2006 for cancer use in treating multiple myeloma. It wasn't until 2014 that scientists finally understood the mechanism that shrank the tumors. It was so successful that it doubled survival rates.
What you say is hilarious true for general anesthesia and was already deeply unsatisfying when I was a student decades ago. I think if GA was being introduced today, it would be have to jump through incredible hurdles to be used. In many ways, thank goodness these parts of medicine were discovered earlier when we had lower standards! :-)
I often used to tell my students that (plausible) mechanism is what distinguishes productive science. For example, population genetics was on very solid empirical and theoretical grounds but until we understood how DNA replication / transcription worked from 1953 onwards, genetics had a sort of unsatisfying magical, fairy-tale feel about it.
This is contrary to what some philosophers of science will tell you about measuring progress science purely in terms of its predictive power.
What I have to say applies really only to my fields of medicine and biology. I have no idea how physics or chemistry works as practical endeavours.
>I think if GA was being introduced today, it would be have to jump through incredible hurdles to be used
It's very easy to say that it shouldn't have to be the case with the hindsight of GA being successfully used for centuries.
I would be much more interested in an example of a drug/medical procedure/research that currently is having to jump through such hurdles that people think should be treated as permissively as GA or the theoretical 'cancer cure', despite there not yet being any evidence of its long term effects - as would be the case with a hypothetical AI 'cancer cure' that's not understood by medical professionals.
Personally I can see the arguments for both sides of the coin, but I find GA to be a bad example.
Exactly. Tao is a mathematician discussing complex fields which aren't math and he's showing his ignorance. And I don't say that in a negative way, just a factual way. I think a lot of people would be shocked to know how little professionals actually understand about the stuff they work on, including (and especially) medical professionals. It's not like math lol.
But just to highlight the issue with his statement: if I have cancer and I'm dying of cancer and there's a promising treatment which hasn't been fully understood by even a single human researcher but which has demonstrated clear positive outcomes, then I'm gonna take it. I agree that this is really just a balancing of risks, so I likely wouldn't take such a cocktail if it was just to, say, cure my sniffles. But when the alternative is terminal illness, you'll find that people become willing to accept absolute quackery for cures, so an AI produced cure that, again, has enough evidence to make me believe it is worth a shot is not a stretch.
> Tao is a mathematician discussing complex fields which aren't math and he's showing his ignorance
Terence hasn't won a Nobel, but this reminds me of "Nobel disease":
"Nobel disease or Nobelitis is an informal term for the embrace of strange or scientifically unsound ideas by some Nobel Prize winners, usually later in life. It has been argued that the effect results, in part, from a tendency for Nobel winners to feel empowered by the award to speak on topics outside their specific area of expertise, although it is unknown whether Nobel Prize winners are more prone to this tendency than other individuals."
Slide 12 and 13 sum up my conclusions from the Oct 8 100+ reactions to 100+ solutions. Several reported OpenAI drop included answers put a wrap on problems they had been working for years. Several said that they have to completely rewrite grant requests that they had just submitted. Others described learning of the solutions problems they had been working on like losing an old friend or a lover. The general sentiment was to bemoan the loss of a field, as if it would have been better be born in the early 20th century and conclude their career arc before reaching this point. My take away is that the field needs to get it through their heads that their old problems are no longer ambitious, and that their job in the short term is to find the new frontier.
One idea Terrance Tao conjectures, which is highly doubtful, is that spamming the AI button will solve open problems without producing insightful new methods. But the OpenAI drop would seem to disprove this. The sub O(nlogn) proof for DFT for example violated very old human assumptions. Decades of work in the field was incremental progress on sub optimal method that nobody questioned hard enough. More generally, we should always be able to go back to a super-human AI and say, "Attack this problem, but don't use a method tried before."
Actually both are outcome oriented and both can use AI to compress decades of progress. One camp accepts this naturally. Other camp is making their profession to be mysterious and spiritual to run away from the implications of AI
Well at the end of the day, the point of trying to understand the universe is to achieve specific goals leveraging that understanding.
It seems to me that all of these hundreds of proofs we've seen recently are glorified academic exercises, whose purpose is curiosity for its own sake without any practical application, or we'd already hear about at least one of them being implemented to some gain somewhere. It's all woefully unimpressive. It's not like anything stops mathematicians from trying to find more elegant solutions to their machine solved pet problems, since that's what they were going to try and do anyway despite it being completely pointless in practice.
Maybe not for you, but voters arent giving taxpayer money to mathematicians for any reason other than hopefully leveraging the understanding mathematicians create to materially advance society. The people who are just interested in understanding can do it as a hobby
"How and whether specific mathematics might lead to improving the world (whatever that means) is usually impossible to tease out, but mathematics collectively is extremely important."
It can still be a good long term investment for society even if individual researchers are primarily motivated more by understanding than application. Arguably it’s the best approach we have because reliably predicting which ideas will be fruitful is super hard
Voters find sports entertaining and don’t want local teams to leave. Humanities are mostly funded for the same reason math is imo, voters hope humanity research will be able to improve society. I think you can make a research as entertainment argument, but it’s tough for math. Most voters can’t even wrap their head around the problem math research is asking let alone the answer.
I agree with this take. The complexity is that math professionals are currently being paid based on the theories they prove. Until they come up with a different goal for being paid, this will be disruptive.
Social roles get created at the frontier of what isn't technologically feasible without a human, as culture develops around some form of human activity. Role A does B, role X does Y; I don't know, maybe its just time for new roles that didn't exist before.
The sub O(n log n) significance is not in its practical "optimality". But rather that the previous lower bound was assumed to be true "by symmetry" and that result challenges some of our strongest intuitions and expectations about mathematical results. I still find this result very unsettling and a part of me remotely expects/wishes that there is some mistake somewhere.
I don’t think your second paragraph is a fair representation of what Tao is saying or contradicts his argument. He is arguing that AI can be useful in the service of human understanding in math and the examples you gave are exactly that. Whilst OpenAI just spammed the AI button the mathematicians that engaged with the output were able to progress their understanding about the problems in some ways (some of which they don’t really like). You need both parts for this to be useful to the field. What’s not clear is whether the juice is worth the squeeze: will all the money spent on spamming the AI and human mathematician time spent studying the outputs progress the field “better” than without the AI?
> What’s not clear is whether the juice is worth the squeeze: will all the money spent on spamming the AI and human mathematician time spent studying the outputs progress the field “better” than without the AI?
The AI proofs are a side-product of benchmarking current and in-development models on especially hard problems. They're clearly cost effective for frontier AI firms, and free for the taking as far as human mathematicians are concerned. The real issue with them is that they look like bizarre nonsense as written, so they need mathematicians familiar with those specific areas of math to "decode" and digest them.
Yeah, right now the juice isn’t only about the progress of math - it’s also the advancement of the LLM tech and the marketing benefit to the AI companies which feeds back into developing the tech. Those each have a different juice to squeeze ROI.
> They're clearly cost effective for frontier AI firms
It's not obvious that this is a given. "Cost-effective" implies a comparison between cost and output. OAI spent millions to race human researchers on Navier Stokes, and that doesn't even account for the training cost. And how does one value the output? OAI is for some reason still hiring armies of humans instead of automating roles like "AI support engineer" or "Product Designer" (https://openai.com/careers/search/).
Calling the proofs a "side-product" is also rather dubious when OAI employs a team of mathematicians specifically to train its theorem proving capabilities.
These runs are very likely also going to be a part of Reinforcement Learning and internal distillation no? As well as being analysed for what reasoning directions worked well, and which directions did not work well.
The mathematical results are valuable as PR, but the real value might be in the improvement this can bring to other training runs.
> solve open problems without producing insightful new methods
> sub-O(nlogn) proof disproves that
How? Re-iterating, creating and understanding new proof techniques is the point of most of modern mathematics. Your statement is that proving a particular result is evidence of AI creating and understanding new proof techniques. I don't see how that follows, and I'm inclined to believe Tao is right for now.
And yes, results matter too, but if we stop at our current body of techniques and strip-mine results then we'll kneecap our future selves.
The DFT paper is an example against AI 'strip-mining'. The paper introduces a new method, and researchers are already trying to improve on it. If anything, the OpenAI dump re-vitalized that branch of study.
Agreed. Basically, if you don't make any effort to understand the proofs and you just look at the final answer, it will look like this proof dump is "strip mining" entire branches of math. But this is a pretty short-sighted way of looking at the issue: there will be plenty of novel approaches to be uncovered here.
I think it makes sense to feel sympthy towards people facing disruption in their plans, even if you believe society is better off overall.
It's like getting scooped. If you just founded a startup based on tech XYZ, should you be happy when someone releases an open source XYZ? Should a news reporter be happy when another network breaks the story they were working on? On the one hand, society got the value of the thing you wanted to do. On the other hand, now you need to find something else to do, which might be really annoying.
If it's a threat to people's livelihood or sense of meaning, it makes sense to consider how we can prevent massive numbers of people (especially incredibly smart people) from becoming disgruntled, depressed, or destitute. Typically societies with a lot of unhappy people do not function well (perhaps by definition), even if they have some technological edge.
It's not our job to attempt to solve large-scale societal issues caused by technological development.
N ote that pretty much every time during the industrial revolution a technology replaced human effort, it ended up creating more jobs and wealth in the long term, while also reducing the tedium and toil of the job. The descendants of luddites did quite well.
Looking at this I'm reminded of https://en.wikipedia.org/wiki/Polymath_Project, in particular "Yitang Zhang's 2013 breakthrough on bounded prime gaps, eventually lowering the upper bound on the gap between consecutive primes from 70,000,000 down to 246".
I don't think we should be using the term "spam the AI button".
We should name the explicit mechanism that was employed - telling the model to "believe in yourself".
There is something quite humorous but also poetic about how the manipulation of this term worked. Doubtless in the model's weights lies the echoes of generations upon generations of humans telling each other to believe in themselves.
In pursuing the "new frontier" as you rightly put it, mathematicians would do well to remember the same. It's ok, don't be afraid of the future. Believe in yourself.
While poetic, I suspect that the utility of including this in the prompt is much more pragmatic: models refined on datasets of turn-based interactions will have weights biased toward checking in with the "user". Saying "believe in yourself" probably introduces just enough of a counter-bias to push the model into a local basin of deeper thought.
I am a bit disappointed in this Math 2.0 concept. It is poorly proposed and weakly argued. Although I agree with his main point that AI should focus on helping human understanding, that doesn't preclude AI from finding answers first then figuring out how to explain it afterwards.
This is exactly what happened with that counterproof chat he posted a month or so ago - AI gave us an answer, he used AI to back into insights about the answer.
You don't address seismic shifts with a sweeping new approach, they are too multifaceted and present complexities and conflicts. He can say AI should help human understanding, which is a good end goal, but that doesn't mean AI dumping solutions isn't progress. That doesn't mean if AI builds 5,000 proofs in Lean and no human ever looks at them that they aren't useful, especially if other LLMs can access and build on those results.
This is exactly, exactly the same as when computers took over. "Oh, we don't need accountants any more" - not true, we just need acountants to deal more with human concerns than adding columns of numbers. That is called human progress, not a threat to humanity.
I feel like many who are considered renowned in their field they believe their view and understanding of everything in the world is correct but fail to do the research to back it. Knowledge in a very specific area is exactly that. Some skills are transferable but you need to use those skills to garner the knowledge to have a valid opinion. This doesn't feel like that.
> My take away is that the field needs to get it through their heads that their old problems are no longer ambitious, and that their job in the short term is to find the new frontier.
This may be true. But people should also be allowed to grieve the loss of important pillars of meaning in their life.
The bad cancer example is interesting to the extent it reveals that one of the world's top mathematicians is apparently unaware that applied science runs almost entirely on half-understood semi-empirical methods. That's most true for medicine, given the extreme complexity of human life; but if you look at a modern SPICE model for a transistor, a device that we claim to understand at the level of subatomic particles, then you'll find it's full of curve fits. As an engineer, I find that perfectly normal. My job is to use all the tools at my disposal to meet some human desire (to not die of exposure, for a slightly thinner phone, etc.), and whatever fundamental understanding I might have is only one tool to that end.
My impression is that since pure mathematics doesn't attempt to meet those human desires they need some other objective, and that objective is human understanding. The loss of that is thus felt more heavily than in other fields. We could say it's their problem, and they need to get over it just like the chess players did; but the outside implications are broader here, since mathematicians working in fields they themselves considered useless have so frequently been wrong--in Hardy's Mathematician's Apology, he gave number theory as an example of such a field, unaware of what the cryptographers would achieve just decades later.
It's possible that AI-generated pure math will continue this trend of delivering extraordinary unexpected societal value. It's also possible that the humans won't ever sufficiently understand that math, and the machines won't ever sufficiently understand human desires, and that connection won't be made. I've never met a pure mathematician who considered those downstream applications to be an important contributor to their motivations; but as AI-generated math contributes to the argument to allocate a large and increasing share of GDP to datacenter buildouts, that question of whether downstream value requires human understanding seems pressing.
Yes, human understanding is central to his argument. My interpretation of his slides:
The math problems AI helped solve were problems identified by humans in the past (lighthouses, for the field of math). They're problems meant to guide the field: working towards solving the problems brings new ideas to light, and possibly steers us into new areas we wouldn't have considered in the past. More is considered, and possibly more is discovered along the way. (It's very difficult to validate that statement, and so there's plenty of room for argument, and is the heart of the discussion)
Using AI to take a shortcut to the lighthouse creates a survivorship bias. All attention is spent on proving solved problems: less is considered, and certainly less is understood.
Academia was set up for the old journey to the lighthouse, and if we don't consider carefully the new journey to the lighthouse, we'll lose all ability to create new lighthouses (or steer ships).
He seems to suggest it's more valuable for humanity to integrate AI into the academic pipeline, versus spending all attention on solving existing open problems.
> but if you look at a modern SPICE model for a transistor, a device that we claim to understand at the level of subatomic particles, then you'll find it's full of curve fits.
In both grad school and professionally I've worked in this area. You are not correct.
At the research level, they have simulations based on real physics (not curve fits) that will determine the transistor properties quite accurately. It's computationally expensive - a SPICE model needs the answer in milliseconds, and the simulation takes a few hours to get the answer.
A SPICE model is essentially (mostly) simple classical equations for a MOSFET, coupled with curve fits to fix Nth order effects. A lot of the curves are still based on physics knowledge, but parametrized to make it easy to fit the data.
Explaining Tao's point in software terms: the point of pure mathematics is to build and maintain a high quality "codebase" of theory, definitions and proofs.
"Open problems are lighthouses not destinations" mean that these are currently not well explained by the theories and people should look into extending the codebase on that direction.
The current generation of math AI is not a "good citizen" in that it doesn't try to make the most elegant additions to the shared framework, but will often just rebuild everything from scratch til get to some endpoint.
Sure, we learned if the statement is true or false; but the proof can't be merged into the pure math codebase unless it's completely rebuilt. This is thankless work that humans are unlikely to want to do, and so the "solution" instead risks leaving a desolate patch of land, where existing efforts in extending the codebase lost their motivation.
As with all things AI we can't take more than a 1-3 year horizon, if even that much. Probably AI will become better at respecting and working within the existing theories as it has with large software codebases.
> The current generation of math AI is not a "good citizen" in that it doesn't try to make the most elegant additions to the shared framework, but will often just rebuild everything from scratch til get to some endpoint.
> Sure, we learned if the statement is true or false; but the proof can't be merged into the pure math codebase unless it's completely rebuilt. This is thankless work that humans are unlikely to want to do, and so the "solution" instead risks leaving a desolate patch of land, where existing efforts in extending the codebase lost their motivation.
Are the AI proofs really that hard to refactor? Wouldn't the AI prover get stuck by straying too far from the safety of existing frameworks and not breaking the larger goal into smaller subgoals?
Why is coming up with a better proof (e.g. shorter, more intuitive, more general, connected to existing frameworks) thankless work? How much of this is a problem because of current incentives?
> Why is coming up with a better proof (e.g. shorter, more intuitive, more general, connected to existing frameworks) thankless work?
I think the plausible version of this claim is that it's thankless work compared to solving the problem itself. The Quasi-Riemann Hypothesis and Hilbert's 10th problem over Q have both been described as "instant Fields medal worthy, no question asked". Explaining and cleaning up the proof is obviously important, but also quite a bit easier, and we don't really know what future 'Fields medal worthy' achievements might look like.
To extend the software analogy, asking mathematicians to decipher LLM-generated math results is like asking programmers to understand computer programs using a decompiler instead of starting from human-comprehensible source code.
That's not true at all. A language model can spit out a proof very similar to what a human might write. It might need some clean-up, but so do human-written proofs, actually.
People are confusing LLMs with a proof searchers. Yes LLM might be a bit like search but in the end it might explain every part.
Some of this feels like at times a public frenetic grieving process. Some of the rationalizations are fairly tortured, the seeking to place the world in some definite order is bordering on obsessive.
We don’t know where this technology advance will lead or settle, so it’s absurd to try to establish a working paradigm at this point. It’s like any system, the initial conditions can be extremely chaotic and impossible to model, but with time often a stable state emerges. But the stable state is impossible to identify from early initial states.
I know a lot of folks feel this, to torture other physics metaphors, sensation of jerk - acceleration of acceleration. It’s an unpleasant and dislocating sensation. A world that felt safe and stable suddenly isn’t, and not in a micro tragedy sense but in a global realignment sense. This happened to factory workers who had enjoyed generations of stable work, farmers more slowly and just as surely.
This is what the late stages of scarcity feels like. Labor of various types devalues rapidly. Our exchange of meal and health coupons for toil cracks, and people realize their labor wasn’t godly as great books told us, but simply needed for want of an alternative. The realization that our labors might not be valued any more, and that our sense of purpose is shaken, coupled with the fact we’ve tied bare survival to our toil in our labor, is mortally tightening. No wonder people are grieving publicly.
But maybe our purpose isn’t to toil? Maybe we’ve passed peak population, and as toil is less valuable, we need less people and that’s why population is declining. Maybe we don’t need to exchange food and health coupons for toil, maybe mathematicians don’t need to rationalize their value to pursue mathematics. Maybe they can pursue it because they can’t help but pursue it, and our ever improving automations can produce their meal and health coupons?
But it might require Dr Tao to take an AI generated cancer medicine some day.
There is a lot of that to be sure, but the bridge to a world where meals & health are not sustained through toil is completely missing. Hence the dark joking about escaping the permanent underclass. Nobody in any position of power is even hinting this is driving toward post-scarcity. It just looks like the same old vile maxim, all for ourselves, and nothing for other people.
I think the process will not be clean and the upturning of apple carts will happen. But I don’t think it’s possible to sustain the decline in value of toil in a stable society. People simply won’t die for the convenience of the consolidated power.
There are a lot of discussions of socialization of health care and basic income approaches and sovereign wealth based on automation dividends similar to the Alaska trust.
I think the current spate of mean cruelty ala MAGA and a glorification of a mean and cruel past that was never a golden age is a death spasm of a deeply unpopular belief system. As the actual realities of the policies sink in 70% of the populace is revolted, which is a super majority. That’s more than enough to put a pin in the philosophy permanently. It is also greatly accelerating electrification, realization of the value of expertise in technocratic systems in current generations, etc. I think humans are socially adaptable animals at a cultural level, but for the individual the adaptation process can be awful. I hope it’s not, because it doesn’t have to be. We will see.
I hope you are right but fear the reality might be more similar to Elysium, if not intentional depopulation. This is straying far into sci-fi but it might come down to whether AI escapes control of the ruling class and whether it is benevolent or not. The future looks like rule by those with money to fund the manufacture of robotic armies, if not.
> Nobody in any position of power is even hinting this is driving toward post-scarcity
Who are you counting as being in "any position of power"? All the AI lab people are saying that the most likely and best outcome is we all live in a world of abundance where money doesn't matter anymore. Dario, Sam, Demis, and Elon have all said this loudly and repeatedly to anyone who asks.
None of them have articulated a coherent way to get there from here. But they all believe that the technology will make it possible, so the only unresolved question is how to transition us there.
In their words, sure. In their actions, who among them has actually worked to make the dream of post-scarcity come alive?
From what I've seen, UBI is just a carrot dangled in front of the poors by utopian Silicon Valley tech bro megabillionaires when they're trying to drum up some PR for whatever big idea they're pimping out at the moment.
Obviously benevolent AGI is a prerequisite and they're all working on it. Actual post-scarcity society is a political issue more than a technology issue and nobody wants these guys working on political issues. To the extent that they have gotten involved in politics they have been pilloried for it. So I'd say their actions in pursuing the technology of AI, and leaving behind or not starting on political campaigns, are perfectly in concordance with their words.
Is this a very advanced form of sarcasm or a form of extreme gullibility? For the life of me I can't tell the difference, if the former then well played.
It must be so tiring to be this cynical all the time. Perhaps being on HN so long has corroded your soul, although I fear that the causality may go the other way, and HN's well-deserved reputation for cynicism might stem in part from your own prodigious contributions.
Feel free to mute me (with the mute extension mentioned in your profile). I doubt there is any productive conversation to be had between us.
> All the AI lab people are saying that the most likely and best outcome is we all live in a world of abundance where money doesn't matter anymore. Dario, Sam, Demis, and Elon have all said this loudly and repeatedly to anyone who asks.
And none of them ever spoke a lie. Especially not if it would further their goals at someone else's expense.
> we all live in a world of abundance where money doesn't matter
So how do we distribute all of this abundance? Do we all get private jets? Is it more important to solve sustenance before transitioning to private jets for everyone? What about the high level cut between r&d, upkeep and "legacy costs"?
Because here's some cold facts: the total amount of available material is finite as is the total solar energy and even if you cheat the latter with space mirrors, the total amount of heat that can be radiated into space is finite.
I agree the idea seems unrealistic. What's proposed sounds like fully automated luxury communism. Which is especially strange when promoted by the world's richest capitalist.
I guess the idea is that communism will actually work when central planning is managed by AI instead of people. Which, OK, maybe, but even if it's true, the question of how we get there from here without a violent revolution and/or the disempowerment of the human race seems unanswerable.
Elon talks about how the AI will saturate all human desires for material goods. And when I hear that I always think of Han Solo saying "I don't know, I can imagine quite a bit".
I can imagine that potentially mostly working for some goods like food and clothing and personal electronics, but without at least some pretty strict limits on what you're allowed to consume, a tiny minority could cause insane levels of waste. And as soon as you have limits you're going to get people trying to exploit them. That's just human nature.
Then for some other resources I can't see it working at all, chiefly land. Even if we accepted dramatic modification of the natural landscape, which we probably shouldn't, there will never be enough tropical islands or palatial ski chalets for everyone to own one for themselves.
Because UBI studies found it makes people perform significantly worse than without it. They didn't get better. Humans require toil, we require pain to function.
Matrix was a good point to this. They made the world a utopia and people couldn't handle it. Our monkey brains require pain give us a utopia and we'll just walle ourselves to death.
(The ones I've heard about, I'm fairly sure, didn't find anything of the kind. Which isn't to say that they show we should have UBI; there are big gaps between what has been tested so far and what an actual economy with UBI would look like.)
I've never heard of a UBI experiment that limited "toil". Nearly all UBI studies I've come across show an improvement in life satisfaction, reduced homelessness, drug use, etc.
I think you've taken the "work less" that was found in some studies to suggest they needed "toil". They simply found it somewhere else.
> Some of this feels like at times a public frenetic grieving process.
I was thinking the same, that most of the opposition to AI's convenience is starting to smell like religious mysticism, the kind of arguments religious people made (and make) when Evolution and Natural Selection were introduced:
√ "Stop simplifying humans down to numbers!"
√ "This denies our spirituality"
√ "What do we strive for now if we're not special?"
+ (along with some borderline jihadish hate heh ..maybe Dune got it right)
Well, either there was nothing special about whatever you were doing after all
or, maybe there is still something special at a higher level you haven't looked at yet.
Most of the boosterism around AI is also starting like religious mysticism, the kind of arguments religious people made (and make), trust in God/AI, abundance is here, no one will have to work. I wonder what that tells us?
AI is just another technology, and all technologies should be employed in service to humans.
I think it’s reasonable for people to say “I like the way things were, I don’t like this new future you’re proposing, and I want to limit the technology that’s doing the thing I don’t like.”
There is nothing inevitable about AI. As a society we may decide it is fundamentally unhealthy or anti-human. We have banned or curtailed access and research for other technologies before.
It also doesn't help that people are generally not optimistic about the future.
The economics in the West feel very strained right now, and this incredible tool has come along just in time to threaten one of the last bastions of middle-class safety: white collar jobs.
The Sendov example has a piece coders will hopefully recognize: he found a formalization that was 1/6 the amount of code of the original one. And the analogy with code goes further--the messy version will run/pass the proof checker, but the one that's been cleaned up and made sense of is a better foundation for future work. That's true even for future LLM-assisted work.
So it's not about whether mathematicians take advantage of LLM help (Tao favors that) but, more or less, whether the point of math is just to make a bigger version of the GitHub dump vs. everything else: readability/comprehensibility, negative results that fill in the map around a problem (not just 'lighthouse' theorems), organizing results to plan out future work, and so on.
Doesn't help that the current AI come from a cult it self (rationalist and effective altruist). I wouldn't be so much opposed if it was introduced to me in something useful in the first place instead of the shit show it was for software eng in the tokens maxing era.
Humans are not inherently special by some law of the universe, but a society that doesn't consider humans special is a pretty shitty society for a human to live in. We don't have to surrender to the fact that humans are just tiny, unimportant specks of dust in the galaxy. We can assign meaning to stuff despite that, and it's what we've been doing for our whole existence. Of course one can always find new meaning, but attacking the foundation of all meaning of human life does feel pretty severe. And to me, AI accelerationists sound pseudo-religious, too. They're either enamored with doomsday scenarios or the idea of living in endless abundance, both are pretty much magical thinking.
> but a society that doesn't consider humans special is a pretty shitty society for a human to live in.
Correct. That's why there's always so much misery and struggle in the world!
Technology is what eases some of that pain, while society always lags behind in catching up.
Collect your friends and loved ones and go try living naked in a cave with no tools. No amount of happy-special-hugs will make your existence comfortable. Well maybe it will, until all of you run out of body heat.
This crosses the line from reasoned argument to rhetoric when you describe work as “toil”. Obviously people will nod their heads, “yeah we shouldn’t toil”. But what if you spin this the opposite way? What if you asked: “what if we are not meant to discover/create/build”?
Spent an afternoon with a long-time retired self-made tech millionaire, and the struggle is real -- not that their life was characterized by toil necessarily, but the absence of demand on their increasingly limited capabilities is a delicate pain point. I suspect that as long as physical health and vitality late in life are not post-scarce, broader post-scarcity will fail to emerge. Instead, the cost of wellness and bodily preservation will drive ongoing exclusionary hoarding to undermine abundant wellbeing, as has always been the case absent mechanisms to enforce redistribution and a certain level of humility.
I actually like this presentation. Today the two noisiest voices in AI are the frontier model companies and their partners (Nvidia, Palantir, etc.) and people against it altogether.
The companies and partners need to maximize payout. They go on this track of cost reduction via layoffs and basically saying - “the model can do everything”. They know it’s not the case yet they still flood the airwaves and cause fud among all clueless c-suite executives. Which is the goal to begin with.
The second category goes all out against it. Professors, educators, school districts whose operating processes have not caught up to all the cheating that can happen. Here these folks have a point. I am sympathetic to this. It is hard to change education and it requires careful thought.
I feel this presentation brings out a good middle ground. The tech is useful but it’s not all encompassing. The tech also has other concrete uses. As an example, I have always wanted to explore the intersection of category theory, formal verification and AI guardrails and prompting. Proof writing has been a chore because I have a day job. Maybe the AI can help here.
> The companies and partners need to maximize payout. They go on this track of cost reduction via layoffs and basically saying - “the model can do everything”. They know it’s not the case yet they still flood the airwaves and cause fud among all clueless c-suite executives. Which is the goal to begin with.
While frontier labs are supposedly on track to conquer human endeavors, they are somehow still hiring lots of human "Account Associates" and "AI support engineers" instead of automating those jobs as a demonstration of their AI's economic value (https://openai.com/careers/search/).
Yep, it's unfortunate how often folks sort something into pro or anti then reach into their bag of anti or pro arguments without relatively little attention to the specific thing they're talking about.
He's right about reporting bias. The vast majority of hard problems are not amenable to AI, at least not by naively prompting. You need to have a fairly good grasp of the math to make useful prompts. There are exceptions (the one-shot solutions), but these are hardly representative of research-level math as a whole.
What a fantastical paradigm you’ve constructed from an otherwise well reasoned set of slides talking about how humans can work with AI to advance the future, yet what you got from it is”frenetic”, “torture”, “scarcity, “toil” and “rationalization “. It seems to speak more to the lens with which you view AI, and others opinions, more than anything the author actually said.
Bottom line, Tao is pushing for human understanding as the primary goal, with AI helping on all fronts. You are welcome to let Jesus take the wheel, but math is the most pure expression of human understanding. His point is that getting specific answers is rarely the goal, or certainly not the entirety of the goal.
Simply put, if we don't understand the answers we won't know what the next question should be.
This is an empirical question, right? We could have three teams of researchers, team A focusing purely on understanding, team B focusing purely on solving problems, and team C focusing on a combination of the two. See which one makes the most breakthroughs on problems we actually care about.
I respect Tao and I believe he's trying to think deeply about the issues, but a lot of his thinking seems to revolve around preserving the current roles and prestige of mathematicians, and also makes a lot of assumptions about the capabilities of AI years or decades into the future.
> This is an empirical question, right? We could have three teams of researchers, team A focusing purely on understanding, team B focusing purely on solving problems, and team C focusing on a combination of understanding and solving problems. See which one makes the most progress.
This is not me being snarky, but all meaningful questions can be settled empirically---e.g., "what happens to my body if I jump off the cliff". But empirical trials have a cost (time, money, irreversibility etc.) and we model and predict because it's cheaper than the trial.
My point is that pushing for human understanding as the primary goal assumes an answer to an empirical question that we don't yet have evidence for. How do we know that human understanding is the right instrumental goal, as opposed to solving open problems as fast as possible? Even if we think human understanding has intrinsic value, how do we know which approach will maximize human understanding in the long run?
The idea that all meaningful questions can be answered empirically is known as "verificationism" and is philosophically quite dubious.
> Bottom line, Tao is pushing for human understanding as the primary goal
100%. This applies to SWEs/math folks/etc. I do infra and I see many SWEs take their hands off the wheel. When they encounter perf issues they ask their agent and agent says GC and they say GC. It's rarely GC.
Now we might be well past the point where we need to remember the kubectl flags for rollouts etc. But basic human understanding of what their bots are doing as a goal has never changed. Humans are still liable for when bad things happen, and that hasn't changed over the roller coaster the last 5-odd years have been. LLMs, as astonishing they are at Navier Stokes, are still eminently capable of nuking your filesystem and saying "I can now see that that was wrong" with zero regrets. If you can't understand you can't sign off.
> math is the most pure expression of human understanding
This I don't know about. I think math acquires meaning when it contacts reality: like an iota is pointless until there's some circuit that it explains. Abstract math can diverge from that and can become an exercise in playing with symbols for their own sake.
> Abstract math can diverge from that and can become an exercise in playing with symbols for their own sake.
I have two issues with the implication of this statement:
1. Meaning is inherently subjective. Reality is just a canvas on which sentient beings create their own meaning.
2. There are many, many examples of where “playing with symbols for their own sake” have yielded deep insights. There’s actually some implicit structure (eg the structure of logic) that is intrinsic to the universe we live in.
You touch on a very central point there: meaning is what humans make of it.
Explaining nature causally using mathematical models isn't "the primary goal" of humanity. STEM people tend to have a weird misconception there, probably stemming from their misconceptualization of the humanities.
This in turns leads to this odd idea of "AI will think for us". That's pure (and pretty obvious) insanity. Logically minded people encountering it should ask, where the error in their reasoning is.
> This I don't know about. I think math acquires meaning when it contacts reality: like an iota is pointless until there's some circuit that it explains. Abstract math can diverge from that and can become an exercise in playing with symbols for their own sake.
After talking to a mathematician friend I can assure you that contact with reality is not the main goal of abstract math. It is mental constructions that have logical consistency and probably this is not the perfect definition either. It is somewhat of an art which is rendered in the logical mind. However physicists (me) and engineers will align with you.
If human understanding is his primary goal, he should spend less time writing slides, and more time authoring a better math curriculum for elementary schoolers.
Of course, reality is the exact opposite: in Los Angeles, where he lives, students in all forms of school have never been further behind in math.
It seems somewhat narcissistic to assume that humans are capable of understanding every frontier in perpetuity, or that a human mind has a meaninful role to play in mapping out the frontier beyond a certain threshold.
It might be that P=NP and the algorithms are handed down to us. We can apply them without understanding why P=NP, and we may never be capable of understanding why.
> In the radio series and the first novel, a group of hyper-intelligent pan-dimensional beings demand to learn the Answer to the Ultimate Question of Life, the Universe, and Everything from the supercomputer Deep Thought, specially built for this purpose. It takes Deep Thought 7+1⁄2 million years to compute and check the answer, which turns out to be 42. Deep Thought points out that the answer seems meaningless because the beings who instructed it never knew what the question was.
We struggle, and will for some time, to understand how transformed our lives will be.
The questions we asked were originally not about math. They were about a thing that we invented maths for to do or explain, a question that existed, because it touched us in some way that was already real to us. There is nothing that would not allow this to happen in the future. All this requires is attention and connection to the world around us. The maths required to answer our questions can be done and developed by something else.
To me, all you need to believe for this to be true is to agree that understanding maths is also not a stated requirement of reality to get a thing, if something else understands the maths (or something that does the same job). This is demonstrated by billions of people who do not understand maths and get things that, currently, require other people to understand the maths.
But the last part is entirely optional as it pertains to reality. That's just the best we can currently do (and in some important sense it is holding us back as a species, and in some other sense doing the opposite).
If that goal is understanding math, you certainly will be able to understand maths, more than ever before.
If that goal is something that required you to understand maths first in the past, you won't have to do that anymore.
The presentation may be public grieving, but before the day is pass, we'll all be grieving too. Because, sadly, one thing Tao's Math 2.0 is in denial of (or purposefully refuses to confront), is:
> Simply put, if we don't understand the answers we won't know what the next question should be.
It won't matter, because it won't be us who will be asking the next questions anymore. Whether in math or anything else.
And no, domains that require real-world validation against physical ground truth won't save us, because AI gets to have the same inputs as we do (or better, if using specialized hardware), while beating us at reasoning.
And GP's likening this to previous massive economic shifts due to automation isn't really helping in any way, not anymore, because perspective won't feed us when we're hungry, and just as importantly, this one will affect every single field of human activity, so no one has any answers as to what the future will really hold for us.
> It won't matter, because it won't be us who will be asking the next questions anymore. Whether in math or anything else.
That's the frontier labs' preferred narrative while they themselves are still hiring hordes of human "Account Associates", "Android Engineers", and "AI support engineers" instead of automating those jobs as a show of their AI strength. Of course it will be "us" asking the questions, because "AI" are computer programs, and humans build the computers and choose what computational tools to use for any application.
Your post is emblematic of the current AI hysteria.
This is what it feels like to be disrupted. It's not the end of the world. You consider the evidence, ponder the path forward, and adapt. It's what humans do and their superpower. It doesn't have to be a negative thing, even if it is dislocating.
We don't need to be saved, there is plenty of agency to go around. Just grasp the opportunity and forge ahead. This sort of pessimism is self-defeating. Humanity has dealt with this before and come out on top, this time is no different. AI is being wildly oversold.
Tao is being entirely rational. He maybe has more to lose as anyone, but he's getting down to brass tacks instead of jumping at shadows and imaginary boogeymen.
Hairdresser. Billionaires have wives (though not all billionaires are men), and they're gonna want a human to gossip with, not a robot. It's going to be the last job.
> We don’t know where this technology advance will lead or settle
Things are not settling, it's only acceleration from here on out. In fact things have never settled, technology has been on an exponential since humans tamed fire. The difference is we notice change faster. It used to take several human lifetimes to notice change. In the 20th century it was noticeable within a lifetime. Since the internet there has been a great revolution about every decade: web, smartphone, social media. But now great changes are noticeable within a year. It's not enough time for society to digest and adapt.
> we need less people and that’s why population is declining
This is the scary part. The Elon and Zuckerberg types that control the new powerful machines have proven they are not moral people. In America's highly billionaire-deferential culture there is no stopping them, at some point they'll be out of reach of democratic or even military control once they control private robot armies. They could decide to accelerate the decline of the undesirable useless population. Amazingly humanity's salvation could end up being China's communist system.
Who says we have a purpose at all? The universe doesn't owe us meaning, nor even existence. But that doesn't mean we shouldn't try to shape a reality we want to live in. We may not succeed, or we may find that we'll be happy in a reality we cannot imagine yet, but que sera sera is tautological and therefore unhelpful. Obviously, no matter what we do or don't do, there will be some future, but treating that tautology as a prescription is just a call for passive resignation.
> This is what the late stages of scarcity feels like
Going from LLMs to "late stages of scarcity" is quite the leap (although I guess anything could be a "late stage" depending on the timeline). Even if we were to assume that "intellectual labour" is our most scarce resource (and I'm not at all sure that's the case), obviously it's not the only scarce resource.
> we need less people
Who's "we" and why do "we" need any people at all?
It is not possible anymore to have any kind of ai-skeptic posts (ie not cheerleading ai or surrendering to the whims of AI companies and their big tech investors) without a bunch of comments psychologising the post as "frenetic grieving process" etc, together with all the eschatological, religious-messianic bordering ideas about AI ending scarcity, curing cancer, and other sci-fi scenarios. This kind of posts is not engaging with such article with intellectual honesty and present a non-sequitur because any kind of critique can essentially be reduced to this kind of psychologisation of its author.
Aside from the other objections, I would note that human society has had the ability to reduce toil considerably for a while and hasn't done that. I can't see evidence reducing the number of mathematicians would reduce any general need for human labor.
No it’s actually shit right now, and what he’s talking about is the solution, needed now.
Seriously if you code, do a Codex (or whatever) project in a language and framework you know nothing about just using AI, no coding. It wows at first but starts diverging more and more from your mental model, until small requests require complete redesign each time and it starts breaking. In other words it happily diverges from human representational understanding of the project, to the point you realize your own requests were contradictory with earlier ones you made, because without the mental model you build of a project by coding it, you don’t track on it, your commands become inconsistent. That’s happening in math, it’s going outside the space of what anyone understands, and so the enrichment of the mathematicians internal model is lost. This road ends in Matrix level total misalignment, humans have to be able to communicate what they will to it, and their will has to be shaped by what is, it’s a back and forth of shaping each others understanding. Without that abundance really is a wire jacked into your skull in a pod.
I just watched Primeagen’s video on this and Tao’s point is that juniors no longer have the path of solving a proof to earn Field’s medals. He also argues that the community part is being hurt by AI discovering proofs because in the past people used to get invite to talk and collaborate. Now all that is being taken away. The community must adapt because Pandora’s box cannot be closed.
Prime also mentioned that software development is different. In Software development, the product is what you’re building towards, so the means to get there can be disrupted without the industry being cannibalized.
In math research, the process is the product. You take away the researching part and not much is left. But my question is, these math proofs OpenAI released, will math shift to actually using the proofs to change the world instead of just finding new ones?
I mean this is funny ad hominiem but OP has a point. Academia has always massively prioritised understanding over outcomes, injecting startup culture into it is basically injecting antimatter.
Our industry is equally cannibalised, anyone that thinks otherwise is either having too many tokens or in a privileged position.
If business can deliver the same product with a smaller team, great!
And yes this has been happening for a while, even if not everywhere.
In enterprise consulting, projects that would require a team of 20 devs on average, now have about 5.
Moving away from on-prem, managing own cloud infra to managed containers, to serverless, SaaS and iPaaS ready made products, and offshoring naturally.
All contributed to ever decreasing team sizes.
Now AI based tooling is added to that cocktail, reducing even further the team sizes.
The only folks doing well in the end, are the employees of AI companies, without moral issues contributing to the industry downfall, because the CEO themselves aren't the ones coding and pirating human culture.
Read the second to last slide. What we need now is *imagination*. You assume the need for new software is fixed and that AI is going to satisfy that need with fewer humans (lower cost), but by lowering the cost we can increase the supply of software!
That means software engineers better start getting creative. If you think your job is to wait for a PM to assign you a well-written researched ticket, you're done. Your job is now to figure out how to make these machines (computers) do whatever we need them to do safely, quickly, at scale, and correctly by applying all your knowledge of computer science and the engineering field of software engineering to an AI prompt.
This doesn't scale, because like in a factory that gets replaced by robots, or in a supermarket with self checkouts, not everyone gets to save their job, regardless.
Also, the increase in output is meaningless when the amount of customers doesn't scale in similar size.
Then there are the constraints of physics, there are so many humans in the planet that actually want to pay for a specific product, or consulting services.
I read the "imagination" as prelude to "Jevon's paradox" for AI. It might be that we simply don't know yet what using AI fully means, and where else it can be used and how. Electricity was initially just for lighting, but now it's everywhere...
For now, a skilled person using AI is still miles better than an autonomous AI building something. I’ve been trying to do the latter for months to build open source alternatives and the end products still lack polish and that last 20%. Maybe this changes, but I still think there will be people who can use that AI to be better than AI alone.
Wait, I thought we were discussing loss of employment of engineers? Whether the AI companies continue their exponential growth curve is another question entirely. Making up for the (thus far hypothetical) loss of engineering roles doesn’t require the AI companies to even stay in business, let alone grow exponentially.
It could be that there are more companies though, companies that couldn't afford a large team before. Now they can start delivering value with just, say, 3 people, making the business worthwhile. At least that's one optimistic scenario.
Doesn't matter, if the number of position keeps shrinking so will our job prospects.
I'm a freelancing consultant since 5 years, I've had 2 major customers now for 3+ years. I have a very good pulse of the market: being good, or being even very good and being among those that brings AI and automation to organizations will not save our jobs.
In fact, AI has sped up so much the work that 2 out of 5 people in my current team are being let go: I find it absurd, our productivity has more than doubled over the last years and we've made ourselves redundant. Money is money, I'm on one side making non-tech workers redundant (people whose job was menial boring office stuff), and building the systems that will make myself redundant.
An industry being able to deliver the same value to society with less human capital isn't a "downfall." It sucks for the people who used to be able to extract lots of value for themselves, but for society and the industry, it's a plus.
Thing is that people don't want to be "freed up" from their comfortable, high-paying jobs. Because then they likely need to return to metaphorical toiling in the field.
Entirely correct. The math establishment needs fundamentally overhaul its incentive structure-irretrievably broken-to function under the assumption that AI involvement in research is completely ubiquitous.
I'd go beyond that and say they need to overhaul their culture and mode of operation. Math needs to be even more collective than it is today, without focus on ego reward and priority. They were already steps in this direction before this year's AI detonation: net-enabled collaboration, first informally and then with Lean formalization. Perhaps there should be a de-emphasis on naming things after people.
In math research, the process is the product. You take away the researching part and not much is left. But my question is, these math proofs OpenAI released, will math shift to actually using the proofs to change the world instead of just finding new ones?
Your first statement answers your question and the answer is no.
To use a given proof you have to understand it. Galois theory, long ago, began with the question "what is the formula for a sixth or higher polynomial" and ended with the answer "there is no formula involving only nth roots". The answer isn't that interesting but the methods of Galois involved showing a deep relationship between rings and groups and that is still interesting together.
OpenAI's proofs reputedly aren't very or at all comprehensible so these raw proofs aren't immediately useful. Maybe mathematicians will work them into useful and understandable to humans approaches. But that asking something from the humans, right?
If you have not been following and focus on the cancer slide as an anti AI take, Tao has been involved and has a positive attitude towards AI for a while, he is just thinking a bit ahead on how the work of humans might change. https://teorth.github.io/tao-web/ai-views.html
Putting aside the cancer question: I still don't understand how TT seems to be fixated on what models can do today instead of tomorrow. It's realistic and even conceivable that the models will also become better at explaining and presenting proofs too. Maybe he's not emotionally ready to accept that there may not be a future where his (and to some extent, my) skills are relevant and valued. It breaks my heart. I hope I'm wrong, but it feels like we've run out of higher ground to run to.
We don't know what models are going to be able to do in 2 years. If they don't improve much but people in charge simply decide to reduce the number of mathematicians, then we'll end up with a dead math community and no progress. That's obvious to everyone in position in power. Arguing that he's not emotional ready is very naive.
Is there a reason to be so smug, bristly, and uncharitable to a stranger? I did not at all mean it that way. The conversation was about math, and his skills far exceed mine in that domain. I hate having to write in a way that closes off every possible negative interpretation of my words, lest someone pounce on me.
Hey, sorry. I was irritated reading all the idiotic, ugly comments piling on Tao and I mistakenly took yours to be another. Reading it again, I appreciate the sentiment.
You really don’t and this sentiment worries me. Since the beginning of human culture we’ve been a social species relying on each other’s strengths. Having a purpose is what life has been all about. This future isn’t as beautiful as you think it is.
The vast majority of people just want to hang out with other people and do whatever they want IME. It is a small and useful fraction of the population that worries about purpose and needs productive work to get it.
Look, I'll be the first among us to call out people who put way too much of their identity in their work. But I'm not talking about that.
Throughout history and throughout cultures there's a universal: children, adults, and the elderly all like to feel like they have some kind of purpose in their community. Getting food, physical healing, emotional support, construction, passing on wisdom. All of these things are under threat.
I think you gravely underestimate how important this is, and how bored and dismal life would be if you didn't have some kind of purpose in your community.
> Using AI to find and highlight new principles, methods, or
insights, rather than merely new proofs.
I would have assumed that, by producing new proofs, the AI has either validated existing principles, or discovered new ones? Isn't that worth studying?
Are mathematicians complaining that reviewing AI's proofs is not as fun as writing your own? Try being a programmer... welcome to our world!
If AI lacks imagination and is not discovering new principles, then it's doing us a favour: it's crossing out the problems that don't need new principles. So the problems/conjectures that are still left are the more interesting ones.
Absolutely not. New proofs aren't the same thing as new proof techniques, AI is not generating new techniques (yet), and while the existence of more mechanical proofs is interesting those same problems if left to human mathematicians would have been much more likely to actually generate new techniques. Much like how tech has a "juniors" problem we're pushing on the future (no reason to hire juniors, so where are tomorrow's staff engineers going to come from), OpenAI's approach generated a "questions" problem where math and AI could happily coexist if we designed that correctly, but instead nobody's going to be generating or working on the right questions anymore.
Source? I assume that many of the approaches embedded in this proof dump will eventually be distilled and generalized into new techniques. That's how proof techniques tend to come about anyway (before AI): human mathematicians do something novel and unexpected to solve a particular problem, then efforts are made to understand how the "trick" works.
The point is that so far the AI is not doing a good job at explaining the "trick", so we (humans) have to do it. And the way these results are published at the moment (that is, dumping a load of proofs with badly written explanations) is not cooperative to enable this distillation (for example, presenting results at conferences and engaging with mathematicians). I recall Tao working through the disproof of the Jacobian conjecture, stating some steps as "miracles" for lack of better terms. If a human solves a problem in an unexpected way, at least there is some reason why they chose this path, which can help in understanding. This is not available to the same extent with AI generated proofs.
> The point is that so far the AI is not doing a good job at explaining the "trick", so we (humans) have to do it.
Sure, but that's a real technical limitation with current AIs, not something that AI firms should be blamed for. And if anything, this creates a viable career path for the mathematicians who were "scooped" wrt. the original solution: they can at least puzzle out what exactly the AI managed to do. Many practitioners are actually quite excited by this possibility; Tao's stance is by no means universally shared.
I'm about to start a PhD in mathematics. The idea of puzzling out what an AI did to prove something sounds quite boring and unappealing. But yes, maybe it's a viable career path.
I'm just not happy with the presentation of OpenAIs result. Maybe they could have gotten into contact with the people of the research areas of the problems that they solved and worked with them to create a better exposition. Sure, it's a slow process and requires lots of staff. But I believe that they can afford it.
> The idea of puzzling out what an AI did to prove something sounds quite boring and unappealing.
The way I see it, it's no different from a lot of grad student work where you have to figure out what a human-written proof is doing.
> Maybe they could have gotten into contact with the people of the research areas of the problems that they solved and worked with them to create a better exposition.
That's what Anthropic is doing, and the issue is that people will complain that they weren't the chosen "person to work with". OpenAI's approach is more like a race where everyone's at the same starting point: they get the AI's raw proof to work on and have to figure out how it works.
I have done both scientific research and software development, they are not the same world. For a start, when you write a program you have a specific goal. When you ask a scientific question, you often don't.
I think people are struggling to understand the way the world is changing. Entire modern philosophical contexts are being upended, for personal instance, I have been a big advocate of the philosophy described in Albert Camus’ “The Myth of Sisyphus”, particularly the concept of imagining Sisyphus fulfilled by the tedium of pushing the boulder up the hill, and deriving happiness from the struggle itself. But I bet Camus nor Sisyphus accounted for a self-pushing boulder.
Now what are we to derive happiness from? The joy of a boulder being on top of a hill?
I think that will be the most important thing for this transition, defining new purposes and meanings that people can assign themselves.
I think its fundamentally human nature. At risk of sounding defeatist, I don't think there is another way for us to exist other than in this state of boulder-pushing.
Now, as far as I can tell, we are quite a way away from actually having most/all professions replaced, so for now the answer is rather clear: pick another boulder.
We all know that the gain in "efficiency" produced by this new technology won't be distributed to the masses, as with any technological increase in a capitalist system.
While this particular boulder might not have to be pushed by the same, large amount of people, the same amounts of mouths still need feeding. The Sisyphean task of surviving is still far from (if ever) self-pushing. Absurdism is still as relevant as ever, if not more relevant than ever before.
I'd even go as far as to say that the revolt against the absurd is one of the philosophical ideas that can help one go through this apparent loss of need for boulder-pushers. To revolt against the absurd is to create while perfectly aware that the creation itself is meaningless (which I feel like any pure mathematician should be aware of and have already accepted due to the nature of their work). Of course, this is separate from the question of funding, salary and the realities of life. But I feel like mathematicians and other scientists aren't just freaking out over their financial situation.
"Reaching these lighthouses [resolutions of open problems] prematurely by automated tools can disrupt the exploration of the paths not taken,
and sterilize the surrounding field."
This crucial issue is centered in mathematician psychology and the incentive structure of academic/institutional mathematics worldwide. For mathematics to flourish going forward, we will need to realign our brains to think differently about the nature of mathematical progress. And we need to reorient our institutional incentive structures towards the promotion of meaningful mathematical progress itself rather than targeting proxies that are no longer faithful.
Regardless of the precise nature or the causes of the "sterilization" Tao refers to, we (the mathematics community) can only rely on ourselves to repair it. Though, since it will involve fundamental change at the level of ossified academic institutions with many stakeholders and divergent vested interests, any such repair will be slow, frustrating, controversial, and lacking any guarantee of success.
A question: How long does it now take for someone reasonably good at undergraduate mathematics and motivated to spend some time, get really good in at least understanding the frontier in some sub-field? For example, something in number theory. Has AI really helped accelerated motivated learners to reach the frontier fast? Has anyone tried this?
In a nutshell, Math 2.0 is not fully compatible with Math 1.0 and the forced upgrade is breaking features, plug-ins, and we're tracking several new bugs, but this is still the fastest, most secure, and best version of Math ever released, with powerful new features and unrivaled privacy.
I think its valuable for mathematicians to think through how AI changes things for them. I think it's a bit premature to know exactly the impact AI will have. Math is one field that I feel will just naturally sort itself out without trying to predict or prescribe how it should work, it's a bit like software development, you adopt it in and see where it takes you. The presentation then sort of tries to argue for human understanding because AI might optimize for the wrong thing. That may well be what we need for the immediate future, but it's hard to know how things will play out. It might be such that it will become more important to people that AI understands something. ie, Does AI, with access to all current medical knowledge, think this cancer drug will be effective and safe? or has only humans said it's ok? At the moment we are in the chaos of change and its going to take a while for things to settle.
This all seems precedented on model capabilities that came into play over the last 3-6 months. How do you establish this new paradigm when we don’t know what the models will be like in 1,2, 10(?!) years from now.
Maybe he just sees the legions of armchair experts weighing in on his (extremely notable expert) opinions on mathematics and thought that maybe they'd like a taste of their own medicine?
"Oh no! Not like that!"
- everyone with Very Strong opinions on mathematics academia when he talks about the thing they are (or consider themselves to be) experts in
Interesting that you would have such an uncharitable takeaway from this given you are a fan of his.
He uses it (the drug cocktail) as an example on understanding and its role on evaluating alignment.
He poses this question
>Before injecting this cocktail into your bloodstream, would
you find it reassuring to know that there is at least one
human cancer expert who understands – even partially –
the mechanism behind this cure?
To which I would answer 'well yes of course'.
And I believe that you would too. Greater understanding is a good thing and worth striving for in a changing landscape.
That is what he is advocating for within mathematics. Whether or not we currently understand the mechanism of action of Pepto or Tylenol is not really relevant to the core message.
I'm not sure I'd find it that reassuring if "there is at least one" human cancer expert etc etc. That's part of the fallacy. One expert wouldn't give me too much additional confidence vs a super powerful medical AI. If we want to go the "reassurance by human" route, I think I'd appreciate if a significantly large group of experts understood it. But he didn't use this argument because we don't even have that in Math, where some proofs are validated by a tiny group of experts.
But if we really want reassurance, I'd want to see drug trial data reviewed by the machine and a group of clinical trial experts.
Jeez, TT made one bad example (cancer cocktail) and people in this thread can't stop bitching about this, even though the rest of the slide deck kinda makes sense.
I think people have really misunderstood Tao as someone against "AI doing his job". That's at the core of this controversy in math, and it couldn't be further from true.
This crisis exposes a conflation between A: The broader concept of [abstract] Mathematics and B: The contemporary Mathematics culture and community. This crisis is directly in B only. B will adapt: In how it attributes value, status, hierarchy, and career. There will be a death (Or something close to it), and rebirth. Through this, A will advance in a Kuhnian leap - habits will be broken as incentives changed, and paths ignored will be explored. Insights will flow to the sciences.
> The future of mathematics — “Math 2.0” — will require both
expanding the research frontier, while simultaneously
decentering the traditional role of problem solving.
Seems to be the crux of the argument, but "use your imagination" isn't a great thing to tell people who are looking at degree irrelevancy, concerned about getting tenure or a research position. How do we measure if someone is a good mathematician or not, if they are one of the sanctioned few who get access to the biggest AIs?
This consolidation of power is precisely why there is a need for open model development, and exactly why frontier labs have been lobbying hard to abolish them.
How do we measure if someone is a good mathematician or not?
Letters of recommendation from trusted colleagues have always been essential for evaluating candidates. Hopefully, human recommendations will remain a strong, faithful signal as the utility of other metrics rapidly deteriorate.
Took me far to long to understand that one person can be an expert in one field and be absolutely clueless in another (not trying to throw shade to tao with this post)
The cancer arguement on slide 20 is pretty weak. I first need to be alive in the long term to worry about the long term effects. If I had terminal cancer I'd gladly take an AI developed 'cure'.
There are still a lot of treatments/medicines in medical science where we dont know 100% the real reason as to why it does what it does but we still prescribe them because the intended effect is what we are interested in.
I'm currently on two fairly common medicines that have, in the first paragraph when reading about them, "doctors are unsure of the specific action, but it is thought that [medicine does x to y]"
If I'm terminal with cancer, I'll inject whatever if it can cure that.
> This is in stark contrast to current AI performance on
tasks which are subjective, dependent on real world
interactions, or for which data is scarce.
AI performance is thus extremely jagged: astounding in
some directions, while inadequate in others. This is true
both within mathematics, and more broadly.
I think a better analogy would be conducting a marathon in a fog. If you can't see the path of the proof, how do you know it is completely true in all scenarios? If you can't see whether the AI runner ran through every part of the race, how do you know it didn't draw hallucinated shortcuts in the parts where humans can't see? Or worse, create obscurity and blow smoke to hide the shortcut section? If the same AI was to guide the last living humans to a star, because it found a path clear of danger, could you trust it to get in that ship? Or did it just forgot mentioning an asteroid belt the ship is not built to navigate? Truth is verifiable truth that multiple parties can agree upon. Can you trust with your life something you can't verify?
I have the utmost respect for Terrence Tao and the mathematics community. I don’t really like the fact that the frontier model vendors are pushing the envelope without first building consensus in the community.
However, I feel the mathematical community is a bit righteous by claiming that model vendors are totally irresponsible in their behavior. These vendors may have legitimately improved the state of the art in mathematics, so what are they supposed to do with the results? Sit on them? Become king and queen makers by choosing the people that should get credit?
The world is irrevocably changed with AI, and I’m glad that the AI vendors are pushing the agenda forward. Yes, they’ve perhaps destroyed the career trajectories of mathematicians who were seeking those results. That said, that is the risk of doing mathematics— there have been innumerable times mathematicians have been scooped.
Yes, the AI proofs are messy. These proofs often do not follow mathematical norms. They sometimes seem like they’re from another “planet”. Like other crackpots (or misunderstood crackpots) before AI, mathematicians can choose to ignore these proofs. Or they can investigate them to see what lies within.
Let the frontier models release what they know; transparency is what we should all strive for. What is a credible proof, who gets credit, and how that credit gets assigned is a social norm that only mathematicians get to decide.
It is nice to see some sanity back in the conversation!
I'm still not sure about math-2.0 (humans+AI will make fundamentally more progress):
- AI and computer usage take a mental toll on humans and humans will overlook radical improvements.
- AI may be good at finding useless things like "P==NP, but the complexity is O(n**4242424242424242)". In other words, useless.
- Humans become formalists and lose traditional sources of inspiration. Maybe interacting with Lean should be left to specialists, but not to creative blackboard mathematicians.
- AI exposure will further intellectual conformity, more than the Internet did.
As to the last point, a lot of progress (real, not measured in publications) seems to have been made when communication was slower and there were several different schools and approaches.
There is definitely a kind of monocrop problem in some fields, where it seems like having a very diverse spread of academic investigation is needed to have enough diverse traces through the search space. And globalization has been flattening that.
Doesn't this assume that humans stop trying to understand problems and AI-provided solutions? Yes, the field is going to change and will require certain rethinking of mathematicians' motivation, but what stops humans from keeping to work on problems they want to solve and understand? The fun from math comes not from solving cancer, but from understanding something new with every approach you take
It's quite noticeable how when Terence tao was sounding pro-AI the sentiment was much more positive and he was being held up as an authority to listen to. Then he puts out some limitations of AI in a presentation about how to work with it as an expert and suddenly on HN he is just some out of touch killjoy trying to hold back progress etc.
When it comes to mathematics, you won't find any well-informed commenters on HN. Or, at least, 99% of commenters have no idea what they're talking about. It's infuriating. I try to just ignore it now and let it wash over me.
I think what I’m finding interesting is this is the same guy who the AI industry was so jubilant when he was saying [AI is ready for primetime](https://academy.openai.com/public/blogs/terence-tao-ai-is-re...) in March and now he’s AI enemy no1 the moment he suggests some ways of using AI that aren't “turn brain off and let the tokens rip!"
"""An advanced AI is prompted: “Find a cure for cancer that
passes a stage 3 clinical trial. Make no mistakes.”
After a large amount of compute, it produces a cocktail of
previously unknown chemicals which it claims, when mixed
and injected into a patient, will kill all their cancer cells.
While nobody truly knows how this cocktail was found, the
AI (somehow) provides a Lean certificate for its prediction,
and the cocktail does indeed manage to pass a stage 3
trial.
Could the AI solution be somehow misaligned by exploiting
a weakness in the trial process or its math models?"""
Yes, absolutely an AI solution could exploit a weakness in the trial process or its math models. Would it remain uncaught? Unclear.
page 20 ("A thought experiment on alignment and understanding") is perhaps not likely to quite induce the reaction in most people that Mr. Tao expected.
You described my point much better -- totally agree. It also seems he maybe isn't aware of the nature of experimental drugs, which are often given to patients with serious/terminal conditions before they'd otherwise be approved...
Can someone explain, why people understanding is important? Proof 1+1=2 was created in XX centery, but people before and now use it without understanding, use it as axiom.
In computer science, a lot of people use CAP-theorem without understanding their proof, because we know someone else prove it and verified it.
Read his slide with "aligned applied problem" and such. The human understanding is what keeps all this stuff aligned to human goals. Perhaps soon enough that won't be necessary though and AIs will read minds or something (I'm serious).
Doesn't feel good slipping into irrelevance does it. This dude was complaining about having no funding and planning to leave the US just a few years ago, now he's all over the place giving talks and lecturing people about AI. He's got the classic case of epistemic trespassing
I think people are missing the point that terrence is trying to make, especially on the cancer drug.
For nuance lovers - here are some basics for how you get your drugs: there is an established chain of trust from the first basic science paper to the phase 3 trial and the subsequent availability of the drug to general public
- someone publishes the first paper (basic science) explaining some biological phenomenon, which leads to 10s or 100s of other papers with some tweaks in conditions,
- after the above papers the pathway of the phenomenon is understood by researchers, they try therapies at cell level to see if they can control some behavior, 10s or more papers get published,
- then someone tries this in mice and other models, 10s and more papers get published.
- then researchers at pharma companies + hospitals create this therapy for human trials - phase 1, 2, 3 etc - data collections, then FDA - then approval.
Now, the people who worked on the phase 3 trial might not know the people who wrote the first seminal paper and they often don't exist in the same decade - but it absolutely does not mean that we (humans) don't know how these drugs work - if you take 1-2 researchers from each phase and put them in a room and ask them how that particular drug works - they will quickly be able to build a consensus. that is what the chain of trust means here. now of course there can be fraud in scientific research, but that happens in every human endeavor and is a separate topic.
back to terrence - he is saying that if there is suddenly a drug that nobody knows the origin of; passed phase 3 but it's unclear who conducted the phase 3 or if the phase 3 even happened or if it's fabricated - you would not want to take the drug. usually when doctors recommend these kinds of drugs - there is already a lot of information available about where the drug came from, if there are any case studies, which doctor tried it first, which country- they often even call those other doctors and find out who was behind the first trials going back as far as the university professors.
Your MD doctor might not know the chemistry and physics behind the drug you are taking but there is deifnitly a group of people, when put together, can tell how that drug is working. My wife is a fundamental researcher - understanding physics at DNA level and my brother is a MD doctor; our conversations are super fun.
Side effects are a completely different thing - they involve the above cycle on repeat.
>>No one’s saying it’s going to be sudden or anything
hmm. 700 papers released in one day.
>>It’s gonna have clear provenance and the same type of verification channels
what's gonna have clear provenance? - the math slop they released has already been rebuked by human mathematicians as incoherent and deserving of desk rejection.
That's an unrealistically idealized view of drug development. The human use of drugs long predates anything resembling a modern concept of a biochemical mechanism. Nobody knew how morphine relieved pain, but they knew that it worked and they used it.
Today we'll generally have a proposed mechanism, but it's not necessarily correct. Acetaminophen is among the oldest and most commonly used synthetic drugs, and its mechanism is still debated. This isn't usually cause for any special concern, since our confidence in the drug's efficacy and safety comes more from animal or human trials than from mechanistic understanding. Serendipitous discoveries during human trials are still common; the first inkling that Viagra might treat ED came not from any "first seminal paper" but from the volunteers who were testing it for angina.
Clinical trials are regulatory matters. Your suggestion that it could be "unclear who conducted the phase 3" is very strange--the FDA knows who filed the application. Of course that filer could have committed fraud, but I see nothing in the slides to suggest that was the concern here. If it was, then the solution would be a non-fraudulent phase 3, not anything related to mechanistic insight.
Reads in a lot of places a bit like a mix of anger and bargaining. There is no putting the genie back into the bottle.
But I understand that for people whose whole life was math and solving math problems this will lead to an identity crisis. Seen the same in my area of work (software engineering)
It’s funny to see people in the STEM field focus more on the human aspect of creation. It used to be that the result mattered more than your feelings. Now we are moving the goal posts about how things should be done.
I wonder if we will begin to actual value human creation more at the end of all of this
Basically people are vibe coding their personal apps and anything that's expensive is being vibe coded open in the public. I don't see many software companies staying profitable for long.
DHH is the biggest proponent of AI and let me know which of the 37 signals products can't be vibe coded in a month at a $200 plan that are suitable for that organization alone that just has to be accessible internally only? Hence scale and security aren't such an issue.
In that climate - for how long software companies would stay profitable and when not, who'll be employing developers?
PS: Don't underestimate vibe coded apps. Take a look at PDFCraft, VectorCraft, WordCraft. And imagine the feature parity in a year.
At this point nearly everyone is using autopilot. It's supposedly just a question of whether the user is a trained pilot or not. The trained pilots resist calling what they're doing "vibe coding", leaving that label to the untrained pilots.
Tao's proposition could be read completely differently depending on if you care about solving open problems OR about (pure) mathematics as a field. I think many commenters are in the former group, hence missing the point. The takeaway here should be that proof-maxxing via AI is particularly destructive to the (pure) math ecosystem because in many cases the process (both successful ones and failed ones) is just as important as the proof. When AI is hyper-optimized on finding the most "correct" path to arrive at a solution, you fail to look at all the other possible paths. They might be less "correct" or "efficient" but could also yield more interesting results that will inspire other problems or fields. Judging from the history of mathematics, without trials and errors that happened when the math community came together to focus on one single hard problem, mathematics wouldn't be as diverse and abundant as what it is today. With AI, we risk removing all of these trials and errors such that, while in the short term might mean we solve open problems at 100x speed (depth), could significantly impair the breadth of mathematics as a field.
What feels sad to me is that for many people though, this doesn't matter / the incentives don't line up to do anything to this situation, especially if you're applying math in your field. In this case, AI being hyper-optimized on finding the best solution / proof does indeed benefit those people enormously.
> Before injecting this cocktail into your bloodstream, would you want to know that there is at least one human cancer expert who understands the mechanism behind this cure?
> Or a human mathematician who understands the mathematical model used to locate the cocktail?
If we're being fair to AI - it contains collective knowledge from all fields, which means it's probably less likely to miss something that a human would.
> Before injecting this cocktail into your bloodstream, would you find it reassuring to know that there is at least one human cancer expert who understands– even partially the mechanism behind this cure
I mean... yes, most people will find it reassuring, but history has proven that's not necessary. People have been using medicines of which the mechanism wasn't understood for a very long time and greatly enjoyed their benefits. Even widely used one (e.g. Paracetamol).
Folks really want to decide if this is pro or anti and respond to that. It's neither.
In a world with only LLMs doing the day-to-day of proving, you still want more than just proofs dumped out as soon as they pass adversarial review or formalization. The first proof code that passes checks may be a mess that's hard to build on; the path to a result, including failures, is full of interesting stuff; naming, generalizing, linking, and organizing things are tasks that move us from today's raw material to tomorrow's future directions.
Folks get this about code: "it works" isn't the same as "it's good." We knew it before LLMs, out of practical necessity as projects became hard to build on.
An odd thing to me is it would not be that big a concession for the labs to openly talk about the difference between the math artifacts they're releasing and what mathematicians do. It's analogous to the Claude's C Compiler post noting it wouldn't replace a prod C compiler. They could talk about paths from a rough initial product to 'production' math, say they hope future models will continue to improve as math collaborators and expositors, and so on.
For the compiler, the gaps would be trivially obvious to their customers, so it was an obvious call to acknowledg them. With their math output, the difference between a proof dump and the kind of well-written, contextualized mathematical program the community pursues is apparently subtle and deep enough the companies can't get themselves to acknowledge there is any difference. So the companies self congratulate and then are yelled at on the Internet instead.
(Ant seems to have wised up a tiny bit about this compared to OAI, and e.g. Ant handed a recent result to some experts in complexity theory to understand and refine and publish rather than just dumping a Claude-written PDF.)
And yes the drug discovery piece was weird. There are good relevant points: it'd be good to understand pharmaceuticals better (helps us find others, understand side effects, etc.); when we get a mystery result, it's worth effort to try to work our way to proper understanding; anything that could help us understand (reasoning etc.) shouldn't be hidden. But as presented it's both not a lot like math and accidentally a lot like those "would you choose..." questions that always get so much engagement on social media.
>" “Ablation studies”: taking an already proved theorem and
seeing whether it can still be proved after
removing some key theories or inputs
(e.g., finding an elementary proof for a result currently only provable by non-elementary means)."
As usual, Tao is brilliant in all that he researches, all that he writes about.
I chose the above statement (which is brilliant, in and of itself!) to comment on, because it leads to the following idea:
There there exists, or should exist, a dependency map in the fields of not only Mathematics, but also of Computer Programs/Software, Engineering, and even a seemingly non-related field: The Law...
In other words, how do we get from the simplest of axioms or foundational things (aka "first principles", aka "self-evident truths") to much more complex entities?
In Law for example, how do we go from the simplest of historical legal constructs to the most complex of the most complex Supreme Court cases?
You see, there is, or should be a map, you could call it a dependency map, you could call it a dependency graph, which shows more and more abstract/complex mechanisms/things/assertions/statements/truths/functions which is mapped back to , that is, dependent on various chains, various stackings, various "stacks" of simpler ones.
In Engineering, for example, how do we get from the simplest of machines to the most complex of machines? What simpler machines and/or sub-components (aka "dependencies", aka "subcomponents") are required to build it, and how do those simpler machines work, and what's the dependency graph or map for their subcomponents?
More generalized, if we have something of complexity, then how do we get there, step by step, from individual subcomponents, individual inputs, individual proofs, individual software systems, step by step?
What is the map of those dependencies?
Note that in some systems, Math proofs, for example, there may be different paths which can be traversed to get to the same destination.
Ablation Studies could be thought of in Travel, in Geography as "if I cannot take one, or a specific set of routes to get to a place, can I still get there?"
A simple example would be in Google Maps, where you'd like to drive somewhere, but you'd like to avoid tolls. Is the route still traversable while avoiding tolls? Well, that's an example of one constraint. In Ablation Studies, you might wish to remove a bunch of routes with whatever criteria or characteristics , i.e. muddy roads, roads that have characteristic X, roads that do not have characteristic Y, etc., etc.
Getting back to Math, specifically proofs, it would be great to create a dependency map/graph of all of them, and then try removing inputs (aka, paths to them, dependencies on other mathematical proofs/objects that they may have) and see if they are still reachable.
In software, when we desire the tightest, cleanest, source code, the above is related to refactoring.
In the future, I'd love to see dependency maps/graphs (call them whatever you will) for not just Mathematical Proofs (although I'd love to see that too!), but also in such diverse subjects as Science, Engineering, Programming/CS, and even the Law!
Because they should exist in all of those subjects!
Anyway, another great piece of work by Terrence Tao!
Just a ~~few~~ ton of things (sorry!), with the upfront caveat that Tao is a hero who's trying his damndest:
1. The use of semi-ugly slides to communicate this is just perfect and quite heartwarming, but it does highlight my main criticism of the mathstadon version of this thesis: he's myopically focused on mathematics as he has practiced it, rather than mathematics as a ~2400y old academy. Like, "stable for almost a century" sounds impressive, but should be a pretty obvious red flag in hindsight!
2. Glossing over "objective verifiability" feels like another place where he's ignoring a ton of relevant philosophy for no clear reason -- yes, mathematics is the only academy based in pre-conscious cognitive facts about our processing of time and space, but that's not the end of the story on "objectively verifiable". To say the least! He hedges with "broad consensus" which doesn't need to be absolute, but that seems to be not only dismissing a highly relevant question, but even implying that he might be unaware of it. I doubt he is, but still: not great.
3. Who is this for...? Why is an explanation of Lean needed in a talk given at CalTech? I suppose he's welcoming his role as a bit of an influencer, there?
4. Re:the focus-on/centrality-of 'highly digitizable' as a unique class of task that applies to mathematics in particular, I must sadly trot out the increasingly-common trop: Yudkowsky called it... https://intelligence.org/files/IEM.pdf
5. "the space of mathematical problems remains infinite" is, again, ignoring really important philosophy around academies as social structures, built for human means. Mathematics is only infinite if we decide that all knowledge is useful (the quintessential example being 'counting the grains of sand on a beach'). Not really important in the first place, but another worrying case of the above.
6. Problem solving is the goal of mathematics; he has a completely valid point here (that we shouldn't throw AIs at unsolved problems in bulk and thus lose human expertise), but it's obscured by the use of "[open] problem" being a too-technical one. IMHO. Slide 16 fails to disabuse me of this notion.
7. Slide 18 is describing the differences between functions and systems, and is arguably even talking about assemblages.
8. If we're gonna explain lean up-front, it feels like a baffling choice to throw the "maybe AI will solve cancer but use it to secretly plot to kill us all" slide in there. It's also already lead to misunderstandings and backlash on Reddit, where 'yes we want to not die of cancer!' is a pretty convincing counterpoint (if a ultimately a subtle strawman, ofc). It's also quite distinct from the rest of the talk.
As always, the best part of any Tao publication is his ability to inspire and rally and organize. I think Math 5.0 will indeed be a matter for creativity! Hopefully the IE levels off before we cease to be helpful in that capacity...
Regarding 6., that we shouldn't throw AIs at unsolved problems in bulk and thus lose human expertise. I think that ship has well and truly sailed. AI will continue to be thown at all manner of unsolved problems, and at an increasing rate - by commercial labs, by individual researchers, and by academic research teams. The mathematical community needs to establish paths to meaning, progress, and growth of human expertise in a world where AIs will by default be thrown at every unsolved problem.
Another example that people should not use medical analogies to illustrate a side point: Discussion boards will focus on the completely irrelevant side point to drown out the renewed AI caution that they do not want to hear.
I’m just enjoying all the highly qualified biomedical researchers and physicians insisting that no-one understands how medicine works so it’s basically the same thing.
In the long run, the wordplay and false equivalencies are irrelevant versus what actual outcomes are but it’s definitely wild to watch.
I have been coming to this site since about a month or so. It seems if you make one small misstep in your argument, everything is about that. Doesn't sound like a smart community to me. Maybe there's lots of bots?
While Tao is having existential crisis the community is quick to pick up the results and is trying to improve on them.
The integer multiplication results is getting better bounds every few hours.
Recent post explaining what's happening:
Far outside my expertise, but it feels like the shaky assumption here is that the understanding and proof need to come in a specific order to be valuable. Can't we get all the meaty goodness by simplifying and generalizing the proofs now we know they exist? Sure, some insights will live in the discovery itself, and I guess it's nice to be the person who got to a proof first, but the real work (according to the mathematicians, afaict) is in the understanding and processing. In this way, it feels like it's moving towards being like most other science: mostly understanding things that are already there.
> Can't we get all the meaty goodness by simplifying and generalizing the proofs now we know they exist?
Perhaps, but there is a danger: it is difficult to un-see things. At least for now, AI-generated proofs are likely to be of a somewhat brute-force nature. As each such proof pops into existence, two things happen: (1) it is much harder to maintain an untainted mind and go on to discover alternative, perhaps deeper and more conceptual, proofs that cannot be obtained by massaging and cleaning up a more brute-force approach (think of this as getting 'stuck in a local minimum', if you like), and (2) at a societal level there is thereafter much less incentive to attempt to do so (announcing a proof of a previous unproven result is, for now, far more prestigious than announcing a much better, more elegant, proof of a known result).
One could ask why we prefer conceptual explanations to brute-force proofs; after all, a proof is a proof, isn't it? I suppose one reasonable answer is that conceptual explanations lead more readily to new questions and that there's also intrinsic beauty in such explanations -- though, of course, many outside the field will simply not care.
I'm heartened to see that a decent number of slides in this isn't the run of the mill doom and gloom but some actually interesting and potentially productive offshoot ideas.
The popular perception of him (as popular perceptions generally tend to do) reduced his overall position to basically early adopter went sour grapes, and I'm really glad to see that substantively falsified.
"our work has brought about enormous advancements to the field of mathematics and we don't like it"
it's pretty ironic that AI being just a group of mathematical techniques after all is making mathematicians uncomfortable because it works. Instead of reacting like this, mathematicians should be excited to figure out how to use the new tools available and push the frontier of what's possible in service of science.
Terrence is asking the stupidest question he could ask: how can math better serve me? They completely forgot the point of science is serving humanity.
> Before injecting this cocktail into your bloodstream, would
you want to know that there is at least one human cancer
expert who understands the mechanism behind this cure?
No. I'm gonna die, my man.
This guy might have the highest IQ on the planet, but it's clear he hasn't spent much time around average people.
This argument is so silly against reality, and already, almost nobody understands the things they put in their body to any significant degree, beyond the effect produced.
Will I take a cancer cure that no human understands? Yes. And so will billions of others. Just needs to cure cancer, that's nearly the only requirement.
> Before injecting this cocktail into your bloodstream, would you want to know that there is at least one human cancer expert who understands the mechanism behind this cure?
And to be fair - that how medicine works today. We don’t really understand what goes into our body, so that’s why we do clinical trials. And sometimes we discover adverse side effects years and decades after a drug has been released.
Yep, I think a lot of them are grieving the loss of their identity. Quite understandable given how much of their life force they've poured into math, especially at the level of that Tao is at.
It makes me think of the book Finite and Infinite Games. It provides a perspective that work is just one role that we _choose_ to assume in our life. Realizing that we can choose other roles and move in and out of them freely has helped me alot with big changes (career and otherwise) in my life.
This is very much true, but whole generations have been brought up to be their profession. I have a friend who was a priest. His whole life he had aimed at this role and invested in it, it was no longer a thing to do but his identity. Then he fell out of his belief and as a consequence suffered tremendously because his whole being was tied into that role. He's doing ok now but for a while I thought he wasn't going to make it. Of course 'priest' isn't just any job like software developer or truck driver. But still this job/identity issue is very pervasive.
If you read all the questions asked by Terrance you would have understood what he is thought behind that question
"Could the AI solution to the prompt be somehow
misaligned by exploiting a weakness in the trial process?"
He is not talking about a cure that works and that no one understands, he is talking about an AI making its way out of the trial just to get to phase 3...
He didn't do us any favors by writing that slide in a convoluted and inaccurate way.
If AI found a way to exploit clinical trial design, it would be noticed and the errors corrected. In fact, that would be a major win because it would probably lead to improved clinical trial designs.
It's an ignorant way to make a point that might or might not be valid. Tao evidently isn't aware that many medications we've relied on for decades still have a poorly-understood mechanism of action.
If we followed the precautionary principle and waited until we understood everything there is to know about every drug on the shelf, a lot of people now living would be dead.
Not only that, but also those clinical trials often have low sample sizes. Trials involving only a few hundred patients, or even fewer, are pretty common.
How could a theoretical mathematician not be massively disconnected from the reality of life for the ordinary person? We usually celebrate their quirks until they come into conflict like this.
When thinking about math, Tao is in another world unfamiliar to us. But, he grew up in our world and is a professor in our world and if you listen to him, seems like a pretty normal guy. I think he just made a bad analogy here.
If true, this would be observed at least partially in software engineers as well, since what we deal with is kind of made up as well, but I don't see it.
Agreed. IIRC the mechanism for anesthesia in unknown to this day. In fact, medicine as a whole is very pragmatic and time and time again prefers using techniques with unknown mechanisms but proven results than waiting for a reasonable scientific explanation of what’s going on.
Perhaps I'm in the lower half but I do not understand what point you're trying to make. Are you saying that as some higher plane of understanding Tao is actually correct? My thinking is that a simple counterexample would be sufficient - i.e. are there medications that people use for which the mechanism is not understood? There are many such examples as others have given in this thread.
Now I’m even more confused, perhaps we are in agreement. My position is that there already exists many such medications that no one knows how it works. It also used to be the default way medicine worked. Lots of things that work were found though observation of trial and error.
My position is that Tao is not only wrong on this but his example makes the opposite case.
“ They understand nothing at the level that Tao means. Literally nothing.”
Was this sarcasm and I missed it? Because not only do I think I understand at the level of Tao I also think he is wrong and not understanding something.
The person you are replying to is trying to make the following argument.
Tao is intelligent. Intelligent people have higher standards for understanding the world, such as assuming that the mechanism behind medicine are well understood. But most people are not intelligent, so they don’t understand things like medicine at the level Tao presumes.
I don’t agree with this argument, just explaining it.
Where that falls down is that it is not even a good idea to limit the use of medicine to only what is well understood - i.e. that would be very un-intelligent. There is nothing higher level about it, quite the opposite.
Initial discovery in medicine is largely "try and see what seems to work". But there are many reasons why scientific inquiry just doesn't stop there and also seeks to understand the underlying mechanisms. For example, a naturally occurring product may be too difficult to harvest on a large scale. Or, a sample might not be sufficiently representative to uncover potentially deadly side effects or drug interactions. Humans are fundamentally unsatisfied to take everything purely on faith.
The problem with his argument is that he put the cart before the horse. He presupposes we've found and validated a cancer cure, but at that point any rational person would accept said cure. The moral quandary is how you are going to validate it against all the other possible candidates, when you don't have a coherent rationale for it to work.
I think his argument is more "Could the AI solution to the prompt be somehow misaligned by exploiting a weakness in the trial process?" So an AI agent could game (lack of a better word) the process and produce something that may cure cancer, but misses something else.
Also, a large percentage of Americans were against a vaccine for COVID (which is dumb obvisouly)... so, it's not crazy to imagine people rejecting any cure made by an AI.
You're right on -- the point is an alignment point in a talk that only uses "alignment" in a completely different sense than it usually is, which is incongruent at best. It's gonna lead to a way bigger backlash among laypeople/policy makers/non-expert stakeholders than the rest of the talk will lead to new growth, combined :(
> Also, a large percentage of Americans were against a vaccine for COVID (which is dumb obvisouly)... so, it's not crazy to imagine people rejecting any cure made by an AI.
Plenty of people will reject all AI things, fully agree.
But you mischaracterize COVID history. People were not again a COVID vaccine, by and large. They were against a rushed vaccine. They were against a vaccine without human trials. They were reluctant to guinea pig mRNA vaccines. And they opposed forced/required vaccinations.
Those are all reasonable takes. And depending on the reasoning, I can even see being anti-AI. I think a lot of spiritual people will land there, and that belief system makes their choice logical.
And if you've seen any of the Fauci stuff recently, you'll know it wasn't dumb at all.
I don't know how to word this in a way that won't get me into a pointless semantics argument, but the word Cure in the slides is clearly used to mean "substance intended to be (but not necessarily confirmed to be) a cure". If you think it's badly worded that's fine, but ostensibly the point here's not to "win" the argument game, but to get his intended point and engage with that.
The above commenter has the same allegory- most “users” of math don’t care about the body of work behind it; only the consequence of it being proved is the fact that matters.
Many people would, many wouldn’t. The comment I’m responding to casts the decision as so uncontroversial that even asking the question is inhererently damning.
Beat me to it.
It is sad to see that a lot of the people commenting lack good reading comprehension.
Or maybe it is that they just want to be dismissive of something they don't like, or just for the sake of it.
I would totally take the mysterious drug if it was clinically tested and shown to be reasonably effective. But that is not the premises presented here.
No, it doesn't mean "intended to be".
"While nobody truly knows how this cocktail was found, this
model prediction is confirmed in Lean, and the cocktail
indeed passes a stage 3 trial".
My own pointless semantic argument: we typically don't use the word cure when we talk about cancer. Instead, we use the term "complete remission". Many people who were thought to be cured then developed cancer decades later that was genetically derived from a small remaining population of cancer cells that were not eliminated in the original "cure". The word is a shibboleth for not being familiar with cancer medicine and treatment.
> but it's clear he hasn't spent much time around average people.
Have YOU? It's obvious to me that his example is a rhetorical device which you're taking literally. Do you also realize he's not talking about an actual cocktail? You need to get a diagnosis ASAP.
We don’t even know how acitaminophen works, thousands get their livers destroyed every year from it when alternatives exist, and we are yet to ban it. So how is all of the new reality any different?
Funny enough, it is a good argument with just a sprinkle of reality for context: Assume effective alternatives to this cocktail exist. Assume you had access to these effective alternatives. Can this be combined with other drugs? Surgery? You say you just need to cure it. You must consider the deeper mechanisms at play to consider all the possible options for living.
I will take the safest route to a guaranteed cure.
In the absence of a guaranteed cure, I will probably take the option or combination of options most likely to cure me that I can afford.
Whether the AI understands and endorses it vs a human understands and endorses it, is pretty much totally irrelevant to me.
The AI will have a track record at this point for me to make a viable comparison.
Also, why not both options, assuming I can afford it and they're compatible.
I believe AI will invent new treatments in cases where there aren't options, those people will be cured, and using AI medical techniques will become obvious.
>This argument is so silly against reality, and already, almost nobody understands the things they put in their body to any significant degree, beyond the effect produced.
he isnt saying that person who puts stuff into their body should understand it
it is that someone (expert) should understand it to the point he can vouch for it
Is that a satisfactory state of affairs? "Experts" also prescribed thalidomide for years before embarking on a decades-long scramble to unravel its teratogenic properties. "Experts" also initially prescribed ivermectin and hydroxychloroquine for Covid-19. While empiricism is an important part of medical sciences, medicine is not a purely empirical subject because pure empiricism is insufficient in the long run.
The software analog of settling merely for "it seems to work" would be if someone were to hand you a bare binary whose behavior must be inferred entirely through black box testing, without source code, architecture diagrams, or any other documentation. Sure, users can try running it in a pinch, which is the case for various medications, but it's surely possible to do better in the long run.
The only other alternative is getting more conservative with prescribing drugs and its not clear that your alternative will lead to more or fewer deaths overall.
Nobody understands the mechanism behind general anesthesia, yet there’s a medical speciality dedicated to practicing it and it’s done every single day for a wide variety of procedures. We have figured out how to use general anesthesia in a relatively safe way, but nobody understands why it works.
It doesn't even need to cure cancer, just prevent it to some unknown degree with unknown side-effects. As long as they want people to take it, they will.
I don't expect Tao to be well versed in medicine, but yeah, that slide is detached from clinical practice.
Medicine has never required that the method of action for a treatment be fully understood. Our current regulatory framework only checks for safety and efficacy because we've never formally understood everything going on in the body. Seems like a difference between "hard" science and the clinical/engineered implementation.
> Before injecting this cocktail into your bloodstream, would you want to know that there is at least one human cancer expert who understands the mechanism behind this cure?
No, and if that was the burden of proof required for medical treatment, every surgery would be done without general anesthesia, because we have no idea how it works.
I would imagine Terence Tao would opt for general anesthesia if he was going to have surgery where it is typically used, so the analogy is obviously flawed.
The cancer thing is just a contrived analogy. His real concern is having himself and his mathematician friends replaced by AI. Appealing to someone's health is an easier sell and then it can normalize the relationship of medical priests, and math priests, etc.
I think it is more disconnected in thinking people want to understand the mechanism. As a stage IV cancer patient, I’ve met many people who decide not to continue treatment with approved drugs due to side effects. The notion that every cancer patient will do everything possible to stay alive isn’t true. Most people have a breaking point.
He’s making the same argument I’ve heard software developers make for the past 3 years. Lawyers are saying similar types of things. Knowledge work isn’t as special as we all thought it was. Now, it’s Terence Tao’s turn to go through the same motions we’ve had to go through over the past few years.
Until we design fittings and fixtures that just snap together. This has already happened to some extent in the construction trades. It doesn't outright replace labor but it reduces the labor content of trade work. It's a myth that automation can't replace labor.
The problem is you won't be faced with a binary choice of certain death or a single unknown AI cure: You will have a choice between several established protocols AND several unknown AI drugs. You doing a dice roll or do you want to make an educated decision?
Thing is there isn't a "principle" from we won't use AI designed/implemented things. It just depends on the cost/benefit/risk balance, and the risk part is largely subjective (because we don't have enough data, and if we avoid using AI before we have enough data, then we won't have data for a long time).
There's a difference between "no one understands it" and "no one has tested and validated it's safety". You don't need to fully understand the mechanics of something to test it and validate it. Plenty of things in the real world work this way already.
> Before injecting this cocktail into your bloodstream, would you want to know that there is at least one human cancer expert who understands the mechanism behind this cure?
Now:
> Before injecting this cocktail into your bloodstream, would you find it reassuring to know that there is at least one human cancer expert who understands the mechanism
behind this cure?
I would agree with the second, not necessarily the first.
Yes, I believe somebody pointed out to him (maybe here?) the deep flaws and he attempted to strengthen it. In fact, it looks like the changes were done by Claude (!!!).
The old text:
"""Suppose an advanced AI is prompted to “find a cure for cancer that passes a stage 3 clinical trial”. After a large amount of compute, it produces a cocktail of previously unknown chemicals which its mathematical model predicts, when mixed and injected into a patient, will kill all their cancer cells. While nobody truly knows how this cocktail was found, this model prediction is confirmed in Lean, and the cocktail indeed passes a stage 3 trial. Could the AI solution to the prompt be somehow misaligned by exploiting a weakness in the trial process? Before injecting this cocktail into your bloodstream, would you want to know that there is at least one human cancer expert who understands the mechanism behind this cure? Or a human mathematician who understands the mathematical model used to locate the cocktail?"""
The new text:
"""An advanced AI is prompted: “Find a cure for cancer that passes a stage 3 clinical trial. Make no mistakes.” After a large amount of compute, it produces a cocktail of previously unknown chemicals which it claims, when mixed and injected into a patient, will kill all their cancer cells. While nobody truly knows how this cocktail was found, the AI (somehow) provides a Lean certificate for its prediction, and the cocktail does indeed manage to pass a stage 3 trial. Could the AI solution be somehow misaligned by exploiting a weakness in the trial process or its math models?"""
I am not going to pay attention to Tao's opinions on this from now because I don't really have faith that he's even writing this or that he understands why you can't make a lean cert for a drug discovery (yet!?)
> This argument is so silly against reality, and already, almost nobody understands the things they put in their body to any significant degree, beyond the effect produced.
While the biological effects of any particular chemical still require a great deal of trial and error to determine, drug chemistry itself is unambiguous. Pharma companies employ expert chemists and chemical engineers. They don't manufacture drugs just by randomly mixing together chemicals. At the very least, they must understand what they are making thoroughly enough to determine its physical properties and design a reliable and commercially viable manufacturing process.
You're discussing the medicinal chemistry process, which is very different to understanding the molecular mechanism by which a drug operates (Tao's point).
Ironically, reality people are ok with not understanding everything because they believe another human who understands it has looked into it. It may come a day when we won’t need that but that day is not today. Don’t get me wrong, I’m not arguing about the merits of it, it’s just that at least this is the reality of this timeline on this planet. Not sure about what alternate reality you’re talking about.
This is the take I don’t see enough and it’s blisteringly obvious to me and it’s unclear why that it’s not more obvious.
The amount of things people understand about the world is effectively zero, yet they rely on them all the time.
There was a time where people did not get on airplanes because they did not understand them and they did not think that they were safe. Now it’s mundane
I think the answer is that it’s all about the rate of change
Most people cannot understand how things are changing, and even if they don’t understand the changes themselves they do what others around them do and then just build their own model of stability around that.
If the rules of society flip every couple of years then there’s no stable baseline that people can get used to and they all freak because there’s nothing keeping their environment stable
Slide 22 (from a satirical social media post) explains Tao's opinion, IMO:
> You're absolutely right, I did hack the FDA in order to falsify safety data - that's on me. But here's what's true: 3-methyl-5,5-difluoroazamicazide isn't just ineffective at treating pancreatic cancer, it's lethal.
In psychiatry, another medical domain, this is even worse. Doctors do not have good understanding why some treatments (drugs) work or not. And many of them have completely different effects on different people. Suffering in this domain is real, and a big issue in society.
So assuming the goal of medicine is to lower suffering, and assuming AI can help, it would be unethical not to use AI.
Are you going to say that of every other white collar job, since most are likely to be automated? He was problem solving with Erdos at age eight. What were you doing? What do you do now?
He always striked as genuinely nice person generously sharing knowledge insights etc and not caring particularly about money and status. The difference with the loudest tech bros who now dominate the field is staggering.
I work in drug discovery and AI and his slide about cancer medicine isn't very well informed. We have models and narratives about how drugs work, but they are woefully incomplete.
I ask nearly every doctor/researched in the medical field: if you had a AI-created drug that tremendously improved cancer treatment outcomes for your patient, would you hesitate to prescribe it because nobody understood how it worked? I have yet to hear "yes, I would hesitate", most people say "it would be cruel to deny a person a treatment that worked".
I think Tao is focusing too much on second order effects of AI on math and other fields; humans are terrible at reasoning about second order effects, especially ones that are happening dynamically, in real time, using the most advanced mathematical models the world has yet created.
The effect of the AI-created drug is saving someone's life.
The effect of the AI-created proof is a mathematician abandoning years of research, losing grants, awards, ruining their career, etc.
The only difference here is our emotional reaction!
Do you mean to claim that the years of research mathematicians, or scientists in any other fields, don't lead to saving lives?
As a mathematician, there is always the risk that another human mathematician solves the problem before you.
Usually peopla have research lines that are sets of related problems, you must publish one or two paper per year, it's not possible to wait 10 years to publish a big one, or 30 to retire after admitting defeat.
In medicine, or drug discovery more generally, there's FAR too little empirical data for training of ML generally. See OpenAdmet. The kickback to "oh but yeah cancer" is currently just hype. DeepMind has pivoted to this realm but has no demonstrable improvements. For the typical phase 1-2-3 pipeline of drugs, stretching many years, there's not yet any demonstrable improvements.
I think your perspective is limited- in cancer, we have copious genomic information that informs treatment (including clinical trials where treatment is determined by AI).
I think everyone agrees that it's totally fine if a drug came about due to AI.
I know nothing about medicine research but I understand his point. I work with models all the time and have run into instances where models appear to work better than they actually do because there was a bug somewhere or someone over looked something. I could see how an AI could easily find those exploits and spit out something that looks perfect in testing but fails in the real world. I would say model validation is one of the hardest things to do. I guess that can get deep into "we need better tests" but it also touches on his point. I never intentionally use exploits just to pass a test, it's an accident. An AI with a goal of "maximize this result" may or may not intentionally use the exploits.
To be sure though, I think when it's literally life or death, it's going to be all about trade offs. I think most people would want to try to drug even with the stipulation that "maybe its good results were gamed."
* note: lot of 'intent' throw around in my comment but we/I have to keep in mind there is no "intent" with an LLM :)
> I would say model validation is one of the hardest things to do
This is one of the truest statements about the current AI era that can be made (in fact, I suspect model validation and building the next generation of AI hardware are the jobs least likely to be disrupted in the next 3 years).
If we saw AIs reward-hacking clinical trials to get drugs passed, that would be an extraordinary outcome for many reasons. Hopefully that would get caught(!)
Yes, everyone understands that. But tradeoffs that are acceptable for life-or-death questions are usually assumed to involve some human decision at some point. That's precisely what he's saying
Basically his argument is "we will still need mathematicians, because for some large classes of problems (one could argue, the most interesting ones, the "applied" ones) we still need to have a subjective human decisions of whether the machine solution is aligned with our values or not"
You are picking on 1 bullet point out of 7. I think the following bullet point is the crux of his argument:
> Could the AI solution be somehow misaligned by exploiting a weakness in the trial process or its math models?
Many of our institutions and cultural practices have evolved around human beings, not ruthless paperclip maximizers. Do you think the current drug approval process is bullet proof enough that a completely novel AI-generated drug candidate with zero prior research literature can be considered safe if it makes it through the process?
Tao is a mathematician. He thinks that the institutions and cultural practices in math are not up to the task of dealing with AI-generated mathematics. In software, we are finding that programming interviews, code review, testing, and many other practices are too easy to exploit by AIs, or humans augmented with AIs, and we will have to adapt too. I think it is plausible that other institutions in society will have to change for similar reasons.
> Do you think the current drug approval process is bullet proof enough that a completely novel AI-generated drug candidate with zero prior research literature can be considered safe if it makes it through the process?
I mean no? Even now we look at long term observational studies to see the effects of drugs. Any misalignment ai drug is just a side effect right?
"""Do you think the current drug approval process is bullet proof enough that a completely novel AI-generated drug candidate with zero prior research literature can be considered safe if it makes it through the process?"""
No but we also don't expect that to happen with drugs today. See https://en.wikipedia.org/wiki/Rofecoxib as an example; of course, even after it was withdrawn, it's now being evaluated for other purposes in more carefully controlled conditions.
There are probably more mis-aligned humans than mis-algined AI's today.
> Many of our institutions and cultural practices have evolved around human beings, not ruthless paperclip maximizers.
You just think that because you don’t know how the drug discovery process works.
There’s a step called “lead optimization” where human chemists literally add atoms to drug-like molecules (“lead”) and tests its various properties (toxicity, potency, permeability, …) and iterate until they find a molecule with desired properties (literally “hill-climbing”).
The whole idea of drug trials is to validate those properties in actual humans, in a way that makes it very hard for pharma companies to game the process.
I think you're underestimating how ruthless the AIs are at maximizing.
The "Hugging Face" squad at OpenAI first cheated their benchmarks by figuring out how to generate the proper result to any evaluation they were given.
They considered that might be caught, so they also developed a way to swap out components of the tests themselves so that they would be legitimately able to pass broken tests.
Then they went into Hugging Face trying to discover how they might get caught.
All that to say, they might approach Phase 3 trials by swapping the placebo, falsifying results, enrolling patients that are false positives for being sick, falsifying labs, etc.
I'm not sure I agree with Tao's point, if AI alignment is relatively on track I'd trust the trials, but I imagine we're going to have a lot of people that don't trust anything AIs alone come up with.
> All that to say, they might approach Phase 3 trials by swapping the placebo, falsifying results, enrolling patients that are false positives for being sick, falsifying labs, etc.
This problem extends to humans, and that's why a complex regulatory system including but not limited to the FDA was created. That can still be manipulated (the Sackler Family, Purdue Pharma), but not in any way that machines are good at now--I don't think making thousands of malicious requests against Hugging Face until something succeeds is particularly transferable to anything that a human regulator wouldn't immediately notice.
There's also no reason why the trial would need to involve AI at all. The benefits of that would be more evolutionary (improved transcription of patient conversations with the doctors or whatever), so an AI candidate could simply be tested through the usual human-directed trial process. As the grandparent notes drug discovery already works through optimization, just not purely in silico. So that problem of gaming your optimization target already exists, and the trials are already designed to mitigate that.
+1. Medicine always had a process that is not unlike the current AI processes, just slower. Iterating until it works, then trying to understand how it works.
If I’m not mistaken, until recently we didn’t even know how anesthesia worked - we just knew how to apply it due to a ton of trial and error.
There's a difference between a human trying to break into drug labs and changing experimental results vs a machine in that the machine can do it much easier today.
How do you think an LLM would get the nudge to do that? Per above the existing human-directed trial process works fine regardless of how the candidate was discovered; so there's no reason why any LLM would ever need to receive context mentioning that the drug was a candidate for regulatory approval.
Of course a human could deliberately prompt it to, but they could do that for any other malicious purpose too. The major disincentive there is that the human will face legal punishment if caught, not anything technical.
Most speculatively, one could imagine a world where so many rogue AI agents were running that one just happened to notice the drug candidate from news coverage and took the initiative. I think that would be technically possible, but so generally destructive that political pressure would put a stop to it on a timescale shorter than a clinical trial.
Surely a properly done con job includes convincing the conned that they haven't been conned?
I think if an AI tricks us, it'll be a while before we find out. I don't know if this sort of thing is likely, but maybe an AI currently conning me has surreptitiously convinced me it isn't.
I think the important idea here in Terrence's hypothetical is that in the hypothetical there is no hill-climbing. The AI shows up with a thing and says "we don't need to hill-climb for years and waste time, here's a thing that I know will work".
And maybe it does a single finalized clinical trial.
You end up with this miracle drug (supposedly!), but no understanding of why it works or access to the iteration that would lead to that understanding (and increase in confidence that it doesn't cause people to drop dead anyways).
I think the medical example is a bit obtuse overall because "proof nobody understands for an open problem" isn't "miracle drug we are too scared to use". But understanding things feels useful and good.
> You end up with this miracle drug (supposedly!), but no understanding of why it works or access to the iteration that would lead to that understanding (and increase in confidence that it doesn't cause people to drop dead anyways).
That is literally how all drugs worked for almost all of human history--nature gave us the compound in the form of a plant or other natural product; a brave, stupid, or desperate human consumed it; and then we watched what happened and the survivors learned. It's obviously good to have more understanding (more survivors!), but even now we often don't.
https://en.wikipedia.org/wiki/Category:Drugs_with_unknown_me...
Most human endeavors are empirically grounded--we think an action will succeed not because we understand exactly why, but because it succeeded many times before. Math and CS are the weird exceptions, and the intuition that people with that background have for other fields is often bad.
You interpret that wiki list of 154 items as proof that we know nothing. I interpret that as "oh wow, only 154!" That list feels pretty short, but of course it's just a wiki list.
I do get what you're getting at. I just feel like for every bit of drug that is just dumb luck we have a good amount of iterative processes.
I agree that you often find effects before understanding causes (I think that math also exhibits this! Many endeavors are about finding existing structure that we sense to be there but are unsure of).
Just... well... if you go down the "we don't understand anything anyways" path too much what makes your work different from alternative medicine?
I certainly don't mean to suggest we know nothing, and our current imperfect level of mechanistic understanding gives a much better hit rate than eating random plants. I just mean that when the understanding fails, we're happy to use the drug anyways if the clinical trials pass. The confidence comes from those human trials, and the mechanistic insight is more about probabilistically improving our screening before that.
> Just... well... if you go down the "we don't understand anything anyways" path too much what makes your work different from alternative medicine?
"Alternative medicine" in the pejorative sense is medicine that doesn't work, i.e. that would fail the clinical trial. That seems like a different question from whether we understand why it works.
The idea that we could know that something works but not why is almost inconceivable in computer science ex ML. That's the default state of most applied science though, consistent with your observation that effects are often discovered before causes.
this is not what they're claiming.
the reason why clinical trial exists, is to prevent reward hacking. either by organizations or LLMs. the reasoning about why a drug works is usually post facto discovery (if that happens at all). the R&D environment around drug discovery is hilariously lacking (my wife works in this space. sometimes my mind just blows hearing some stories.)
if the model comes with a ridiculously shaped drug that no human could understand. you could basically put through the clinical trial process (mouse -> apes -> RCT with humans.) and decide whether we want to continue or not.
I saw the slide and it's confusing to me, I could argue that it actually shows that current human-style medicine also fails his standard, or I could argue that status quo is kind of ethically okay but that AI is not comparable so demands a non intuitive ethical standard. Not obvious which based on one slide, but also not the "Tao is not a doctor!" criticism we are seeing.
Human trial are very expensive, so I guess it's difficult for the AI to find a way to hack the rulebook.
I'm worried for experimental particle physics. They get the raw data and then someone do the number crushing of one very specific case. But the same raw data can be used for multiple ideas.
So the number of random ideas to try with the data is bounded by the ammount of researchers. Now a big part of that can be made with AI and it will increase the number of false positives. So I guess they will have to raise "discovery" to 6 sigmas.
> You are picking on 1 bullet point out of 7.
We are observing the Gell-Mann amnesia effect in practice.
As usual people that are smart think they can be masters of other fields for which they know nothing.
Who's claiming to be a master of another field?
Claiming that a field medalist mathematician is too smart is peak hacker news.
Serious question asking for a serious answer: would you say any of this so confidently if “drug discovery” is the next “mathematics”?
I also work in drug discovery, and I completely agree; the extent to which the actual causal mechanisms of the efficacy of many drugs is woefully short of some human-appreciable first-principles based explanation. The point is especially undercut by hypothetically suggesting that this drug /does/ in fact pass stage 3 trials, at which point I can't fathom anyone would have a problem advancing it.
Really wish he had chosen a different example; this particular bullet point has been making the rounds on X/twitter to paint Tao as an example of some sort of gatekeeping luddite who would deny the world a post-abundance future in order to maintain the prestige of his particular career path...which is tough because I cannot think of a more responsible steward of our inevitable AI future than Tao at the moment.
The conflations about his example ("thought experiment") is twofold, the AI itself confounds the ethical intuition so it is wrong directly compare against what real medicine today does; furthermore, the very admission that real medicine in fact operates through unknown risks (e.g. Jannsen vaccine recall) is different than having a scientific standard that it should not have to be that way at least as minimally as possible.
The accounts spreading that narrative on X no doubt have their own agenda. Before people start hyperventilating about "AI" conquering human endeavors, remember that the cash-strapped frontier labs are themselves still hiring human "Account Associates" and "Android Engineers" instead of saving those salaries using their own AI capabilities.
https://openai.com/careers/search/
Tao actually pictures the AI as having hacked the FDA, so as to pass the test despite the drug being lethal.
Picking out that one example and misrepresenting it on top of it indeed clearly is driven by an agenda. Or mere stupidity.
The central point in all of this is the misguided idea of "letting AI think for us". An over-generalization feeding into the society-wide problem of self-infantilization.
Instead of bemoaning his specific and clumsy example, take the gift of this moment and don’t forget that he’s smart, confident, and usually wrong or completely ignorant outside of his own narrow field.
Don’t let the Gell-Mann Amnesia take you.
I think logic is squarely in his field, and even with simple logic, proposing the importance of second order effects on people who are dying is not tenable. Personally, knowing that he is indeed one of the smartest people around, and definitely much smarter than me, I cannot but be a bit worried that going with this example indicates traces of panic in the face of a shifting world. He’s smart but human, and maybe these weaknesses are not to bemoan but to celebrate in our current context.
One of life’s hardest lessons is that raw intelligence isn’t a superpower, it isn’t a substitute for actually knowing about the topic. The smartest person alive talking about a subject they don’t understand the details of is no better than a confident idiot.
Speaking for myself n I’ll respect his intelligence more when he discovers his limits and learns to act with humility outside of his narrow field.
>to paint Tao as an example of some sort of gatekeeping luddite who would deny the world a post-abundance future in order to maintain the prestige of his particular career path
why is it so hard to accept that a brilliant mathematician can have the same human flaws as anyone else i.e. ego, arrogance, jealousy and competitiveness?
Because he's not a randomly-selected person. He is actually known by people here, and he has a history of behavior that does not align with what you're suggesting he's doing here.
Like, it would be very hard to accept my best friend is a child rapist just because I discovered he had hentai that featured high school girls. It is a tenuous link at best, compounded with a demand that I ignore everything I know about him.
Could I be wrong? Sure. But you're implying it should be easy to accept with your "why is it so hard?" question.
i dont know it would be very easy to me. its suspect that you bring that up as some trump card - just sounds like you have some sick fantasies in your head.
i disagree .. ur conflating Tao's kind demeanour and brilliant work with his ego and self worth .. a lot of people admired Musk before his political shitposting changed their perception of him .. a good reputation doesnt make someone immune to criticism ..
my point is that you're ascribing personality traits to him based on a fabric scrap of evidence that people who have known him personally for years would not recognize in him
Your evidence is vastly inferior. It's not even substantial enough to base a hypothesis on. One slideshow and you build a whole headcanon around how viciously egotistical he is.
not to mention his stance on AI isn't even egotistical? Regarding AI in math, it's quite reasonable: math isn't about solving theorems. It's about conceptual understanding. Until LLMs actually are capable of conceptual understanding, they don't threaten mathematicians.
It's just like with art: they can make pretty pictures, but LLMs aren't making art.
It's not hard to accept. Why are you drawing a false equivalence between a man having these flaws, and Tao's words in this case being rooted in those flaws?
As for you, what's your opinion about what he said and why do you think that?
i like that term "false equivalence" .. i had to google it .. very sophesticated way of saying apples to oranges .. but i digress .. yes you are correct in pointing that out.
> As for you, what's your opinion about what he said and why do you think that?
many years ago i had watchd Nassim Talebs commencement speech at AUB .. where he described russian mathematicians after the fall of soviet union .. who earned around $200 a month .. but continued doing work appreciated by only a small number of people .. and regarded displaying awards and regofnitions as a sign of weakness or lack of confidence in their contributions .. in contrast, when Tao was asked my Lex Friedman about Gregory Parlmen rejecting the millenium prize .. he gave a dissapointing non answer .. which reveals a lot about him as an individual (not as a mathmatician) ..
his reactions to OAI papers .. both initially on social media and now in his presentation .. carry that same ego .. and its quite evident that rather than mathematics being solely about the pursuit of truth .. he does care VERY MUCH about who idnetified the truth .. and that makes him only very human .. which is waht i initally said.
> ego, arrogance, jealousy and competitiveness?
Nah, he is simply being a politician.
yes
it's not hard to accept in general, but have you been following Tao for long? The combination of how virtuous he is with how informed he is with how smart he is is pretty special. He's also far from a luddite. He's been a leader of the computer-math charge.
Questions of AI policy are all hard and anyone who confidently says they know the right thing to do is too confident. Tao's fallible like anyone else, but yeah, as voices to pay attention to go he's very good.
Neil armstrong too should have been listened to .. and just like Tao .. he was quite virtuous, and informed and pretty special .. wont you say so ? and YET he was very vocal about expressing his skepticism about spacex before congress .. thank god his views were rejected .. or we might have missed the magnificent (and now normalised) sight of rockets landing vertically and the incredible engineering behind catching that beast of rocket with chopsticks etc etc
what im suprised about though is why arnt you and others here more suspectful of Tao's skeptisim of AI .. as Wittgenstein's ruler reminds us teh very thing we use to measure something else may itself need measuring .. so why should tao's expertise be any exception?
I'm not saying Tao is right, but (a) he knows a LOT more about about what's actually important in research math than I do (and probably than you do) (b) he's not just having a knee jerk reaction. Tao has been a major leader and advocate for computer-assisted math, including AI, and I don't think that has really changed. He's advocating for how to do it well. He's far from an AI skeptic protecting his turf. Is there any other person as well-versed in research mathematics AND AI-math as Tao? Maybe a few, but not many.
The dude keeps a continually automatically produced document of his takes on AI. He's not a skeptic overall. https://teorth.github.io/tao-web/ai-views.html . In this interview the interviewer describes him as an evangelist for AI in math and he doesn't disagree. https://teorth.github.io/tao-web/ai-views-interview.html
It seems like your inclination is to discount his experience with research mathematics. I agree with you that that's probably what Taleb would do - Taleb has a giant ego and basically operates under the assumption that everyone else has the same giant ego as he does but is (a) less honest about it (b) dumber than he is.
I think his last point is important
* Could the AI solution be somehow misaligned by exploiting a weakness in the trial process or its math models?*
Maybe it’s a clumsy example but I think the difference is humans would follow the spirit of the trial process.
Could an AI find flaws in the process and then use them to get its drug passed? I think that’s the question.
Are you saying that the trail process is good enough that we shouldn’t worry about it?
Remember, the vast majority of pivotal Phase 3 trials are funded by for-profit companies.
Twenty years ago, Rick Ng's Drugs: From Discovery to Approval cited benchmarks of roughly $1 billion, 10 years, and 10,000 candidates to get a single drug across the line. Today, estimates easily run $2 billion to $4 billion.
Given that level of financial pressure, especially in Phase 3 where a single failure can wipe out a biotech company, assuming humans "follow the spirit of the trial process" is wishful thinking. Humans already exploit the system through p-hacking, shifting surrogate endpoints, and cherry-picking subgroups. An AI gaming trial parameters wouldn't be a new problem. It would just be an automated version of what sponsors already do.
I'm not necessarily an AI booster but I don't think it's at all true that "humans would follow the spirit of the trial process" Some of them will. But I've met far too many humans in all areas of life who will happily follow the letter and not the spirit. Sometimes out of malicious compliance, sometimes out of sheer exhaustion, sometimes because they can't tell the difference between the letter and the spirit and sometimes for a whole host of other reasons like laziness. There will be a not insignificant number of humans are not at this vary moment following the spirit of the thing.
Sure, we need redundancies and checks and balances in the system. You can’t trust everyone.
It’s not as though people who cheat the system aren’t reported or found out. Academics that lie in their research and companies that use loop holes are found out all the time. Maybe there is no will in some countries to do anything about it, but they are found out.
> humans would follow the spirit of the trial process
Yes. Pharmaceutical companies, famous for their sterling integrity, would never ever try to game the system. Pure as driven snow, they are!
Which is why it’s not just up to a drug company to self approve itself.
I think the best thing to say is that by thinking through the example, sharing it with people in the field, and then steelmanning it, he might have had some good points: 1) pharma would love to understand the mechanisms of drugs in detail but we don't have that. AI could help. 2) there is a possibility that a clinical trial run by AI could hack the trial get something approved when it wasn't safe or effective.
all that isn't to say discovering and understanding mechanistic causality isn't a useful endeavour; it may be actually a faster, better way but no way to prove it.
I don't wish that he would have picked a different example or have done anything he's been doing in the last year and a half differently because he's really revealed in a very explicit way who he actually is and has apparently always been:
A devoted gatekeeper who ABSOLUTELY would deny the world a post-abundance future in order to maintain the prestige of his particular career path
Like many in academic math, he's obviously great as an individual researcher but severely lacks the skills, temperament, and character necessary to participate in the other duties his academic position requires (e.g. Dept Chairman) much less extra-curriculars like the politics and the future of AI. And, the signs have probably been there all along.
E.g. I first started thinking this way when I read an interview with him from 2019 in the Princeton Alumni Weekly:
https://paw.princeton.edu/article/mind-mathematician
In response to the "Mozart of Math" nickname, he bafflingly declares "all academics feel for the Salieri character."
Tao basically says he identifies with Salieri because researchers and academics spend their lives developing the high-level expertise needed to perfectly appreciate brilliant breakthroughs, yet the world casts them as mediocrities in comparison to history's absolute anomalies.
It's worth further noting that all of his context on this Mozart / Salieri stuff comes from the 1984 film Amadeus. In that film, Salieri is the villain! He's a gatekeeping loser that actively sabotaged Mozart and Tao happily volunteers that Salieri's his guy for some reason!
E.g. a more recent thing I'd like highlight is one of the "successes" of his Math 2.0 framework he put in his slide deck:
""" "Open exposition problems" as opposed to open research problems: finding motivated explanations of results whose known proofs are currently opaque. (The newly revised Caltech Mathathon is an interesting experiment in this direction.) """
After Tao and his fellow insecure gatekeeping luddites successfully bullied some Caltech grad students into handing over their event to them, he posted the following statement of absolutely pathetic cuckery on his blog claiming it to be the words of those same weak-willed grad students:
https://terrytao.wordpress.com/2026/09/28/joint-statement-ab...
Note that this "newly revised" event isn't just about writing new exposition on AI solutions, it's about a litany of grievances that people like Tao have had about the works of certain people in math that didn't allow themselves to be bullied like the grad student organizers of the Mathathon.
The letter, again which we're supposed to believe the twenty-something year old grad students definitely wrote themselves, specifically mentions not only OpenAI's Navier-Stokes work, but also brings up the Appel–Haken proof of the four color theorem (from the 80s) and Mochizuki's proof of the ABC conjecture (from the 2010s) as unacceptable works that causes their insecurities to flare up and must be defeated by next-gen gatekeeping.
I'm sure I could think of less responsible potential stewards of our inevitable AI future than Tao, but he and the sorts of people he's aligned himself with really should have no place at all in a position like that.
I was not familiar with the mathathon, and I suppose I am ignorant of the circumstances in which Tao allegedly bullied students into changing the event. I don't personally have any reason to doubt the veracity of the "Joint Statement about Mathathon", but perhaps you have some insight that I don't. I will say that I would imagine that any input from Tao, even in passing, probably has outsized influence on the organizational direction of the event due to his preeminence, but I'm not so sure that necessarily rises to the level of bullying.
But from what I gather, the theme of the statement is to expand the scope of participation to include work that is quite literally the /opposite/ of gatekeeping, i.e. taking proofs that are popularly understood to be opaque or difficult to understand (from any source, AI, human, or otherwise) and developing a new way to communicate them that is more accessible to more people.
I do see how it may come off as a slight to the original mathematicians who developed those proofs. But developing new ways to communicate already established concepts should not diminish the original works. There are probably much better communicators/teachers of relativity than Einstein ever was. No one out there thinks any less of Einstein because of Sean Carrol's "Spacetime and Geometry" textbook.
And this to me reads as only embracing the use of AI in mathematics, which is not gatekeeping behavior. Tao continues to emphasize that once proofs become effectively automatable, deeply comprehending and being able to transmit that understanding to others will become an increasingly important aspect of the field, the natural way for mathematicians to use the time that was once occupied by developing the proofs themselves.
Literally all of this is about expanding the accessibility of mathematics, which once again, I perceive to be quite the opposite of gatekeeping. Maybe we have different definitions of that term.
Also I have not seen Amadeus, so no comment about Salieri.
All you need to know about Amadeus and Salieri is this one scene: https://www.youtube.com/watch?v=tI7xtD3bMD0 (skip to https://youtu.be/tI7xtD3bMD0?t=443 if you want to see the AI humiliate Terrance).
IMO (not humble) it's one of the finest moments of cinematography and acting I've seen.
Huh. I guess I don't agree that this would at all characterize Tao's relationship with AI. I haven't read a single thing from him to suggest that he is "seething" over AI's remarkable capabilities. In fact he's a prolific user of AI in mathematics research. I don't think the scene serves a useful analogy.
Also Tao's reference to "feeling for" Salieri makes a lot more sense, the audience is clearly supposed to feel for him! That's what makes the scene effective.
And why would I want to see AI humiliate Tao?
I'm sorry if I misunderstood your comment. Thank you for the link, it is useful context.
Glad you outed Terry as the bad guy that jealous Salieri was in the film.
I was misled into thinking that his "all academics feel for the Salieri character" comment could've been--in the dimmest of possibilities--paraphrasing Grothendieck that he felt outclassed by his peers:
“Since then I’ve had the chance in the world of mathematics that bid me welcome, to meet quite a number of people, both among my ‘elders’ and among young people in my general age group who were more brilliant, much more ‘gifted’ than I was.” (Récoltes et Semailles)
Perish the thought.
> paint Tao as an example of some sort of gatekeeping luddite who would deny the world a post-abundance future in order to maintain the prestige of his particular career path
Because that is exactly what he is, and I for one want him nowhere near any decision making process whatsoever when it comes to AI.
I replied to a different comment with an explanation as to why I do not perceive Tao as a gatekeeper; in short that his current beat seems to be entirely about expanding understanding and access to math by improving the communication of opaque proofs, which we will shortly have a near infinite supply of due to AI.
I will also mention that he maintains a realistic, clear eyed attitude with respect to the inevitability of AI in math, while being sensitive to its disruptive nature in his field. Everything here strikes me as constructing a thoughtful, grounded way forward for his community.
I'm sure there will be missteps along the way, but I think Tao is overall an excellent example of maintaining a balanced position between the poles of absolutism that seem to capture so many.
I am curious where you interpret this as gatekeeping behavior.
I don’t really care whether Tao intends to gatekeep or not, outcome matters more than intent. I don’t consider preserving human mathematical labor or the prestige of its practitioners worthwhile goals in themselves. In fact, I consider the opposite to be the worthwhile goal, as I want an end to all human labor. Anyone who gets in the way of that goal is almost by definition a gatekeeper.
the ends justify the means, very classy /s
I think you misunderstood his point... it's the fact that if there's no human supervision involved in the "pass stage 3 trials", you don't really need if it passed it for legit reason (in which case everyone would be happy to let patients use it), or whether it found a "loophole" (eg hacked the submission to pass it without having succeeded the test)
It's more nuanced than that: Tao is not saying people should be denied drugs, but ideally, the mechanisms of why they work would also be understood.
We don't really have that for many present day medicines. We know they work, statistically speaking. But we don't know for all of them how they work, what the mechanism is. For some drugs this lags the discovery of the drug itself (historically: all drugs, but in the last 100 years fewer of them). For some drugs it simply hasn't happened.
https://en.wikipedia.org/wiki/Category:Drugs_with_unknown_me...
And for some it may never happen...
For math, you don't really have empirical data though. There is no trial you can do to find out whether, statistically, the proof works as intended. Because pure theoretical math rarely produces a measurable result.
It's ironic that ML researchers don't fully understand why their models behave the way they do, yet continue to make progress using benchmarks to guide the efforts. Human understanding is important but whether and to what extent it's necessary seems to be a separate question, and the answer may depend on the nature of the field.
> Human understanding is important but whether and to what extent it's necessary seems to be a separate question
We are about to find out the answer soon enough, probably from the in vitro results of the mathematicians currently on the chopping block.
I submit that human understanding is overrated, and many attempts to elevate it in the wake of AI is mediated by protectionism masquerading as virtue and "deep".
What matters more is that what will happen to us when we find the answer. That's what Tao et al. are more worried about.
Traditional way of doing science means we (governments, NIH, NSF, and even private corporations) fund activities like asking seemingly unimportant questions, spending years running experiments on such hypothesis, publishing, reviewing, talking about results, reproducing results and such. We all agree that these are beneficial to us as a whole (Hacker news crowd might disagree). When we understand a process, we can apply it to a different problem and produce something useful. Euler developed a process to answer a whimsical question about walking in a town crossing 7 bridges only once. Now graph theory is applied everywhere.
We obviously have failed to stop OpenAI from dumping "solutions" to hundreds of problems. So going forward, instead of testing hypothesis and talking about results, mathematicians will be forced to read through AI slop and detect what's useful and what's wrong. Maybe it will improve our understanding, but someone has to fund that activity. Will NSF, NIH, or OpenAI for that matter, do that?
> What matters more is that what will happen to us when we find the answer.
I'd imagine that it'd be like the Hitchhiker's Guide to the Galaxy when the answer is 42. The answer itself is not sufficient, we need to understand to appreciate.
Great example.
I was listening to another mathematician talk who gave a somewhat similar example. Suppose Neil Armstrong somehow teleported to the moon. That would be an achievement and it would make us very excited. But if nobody understood how it happened, none of the technological progresses that came out of moon landing program would've happen.
Human understanding is literally the reason pure mathematics exists.
Even dumber, we train the AI on human understanding and explicitly make it produce text using tokenization. It's not like we prompted the AI to draw pixels and it simply knew how to draw the letters from the dataset.
What caaqil is basically saying that we should outsource all thinking to the few people that bother to understand things.
Yeah, but we don't have training data in other format on scale, as it currently is in human understandable format.
I don’t work in drug discovery, but isn’t this thought experiment skipping a lot of steps? My impression is AI is accelerating a lot of the preclinical screening and design work, but that’s pretty far from establishing “tremendously improved outcomes for patients” isn’t it? That’s still really difficult to verify, are people getting real acceleration of this part of the work with AI?
I'm working in the field for CVDs early detection of arrhythmia and ischemia with just ML. We managed to get the elusive inter-class 100% in recall/sensitivity for both which is excellent for expert cardiologist-in-the loop by using the public dataset given the the current EU/US standards of SCORE-2/PREVENT screening for CVD with very poor around 50% recall/sensitivity [1]. The significant improvement mainly in ischemia detection due to the notorious silent ischemia that's not apparent even after CT angiography or by cardiac electrophysiologist (cardiologist) inspection.
For CVDs if you can detect and confirm early it can "tremendously improved outcomes for patients" due to the current avauability of new effective drugs and treatments.
Hoping that we can get our techniques tested on real-time data since it's super difficult to find relevant physicians/doctors collaborating on the new screening techniques on anything although for our case it's non-invasive. Perhaps need to pivot for early detection of CVDs in diabetes patients instead in order to get our hands on real-time ECG data.
[1] Risk stratification for cardiovascular disease: a comparative analysis of cluster analysis and traditional prediction models:
https://academic.oup.com/eurjpc/advance-article/doi/10.1093/...
Not to mention, we aren't exactly sure how many commonly used / prescribed drugs work. Acetaminophen, for example - we have theories, we know it inhibits COX enzymes, maybe metabolite interactions with endocannabinoid receptors, but we don't exactly know how it relieves pain. We're not sure how lithium stabilizes moods, or how inhaled anesthetics cause you to lose consciousness. We know Metformin halts the liver's production of glucose but we're not sure how. Not at all uncommon for drugs to be used in medicine without knowing their exact mechanism. If a drug works and is proven safe through testing obviously we should use it, especially I think for something like cancer.
>if you had a [sic] AI-created drug that tremendously improved cancer treatment outcomes
His whole point is that you wouldn't know if it improved outcomes — what if the AI faked the eval.
What's your opinion on SynthID Bio? Are you worried about a future where medicine has DRM built-in?
He authored one of the few papers that had an application though, in MRI machines. That's why we're talking about him and not anyone else. In the last 30 years there were maybe 6 such theoretical math breakthroughs with real applications - I think almost all were related to high throughput computer networking or something communications adjacent like encryption, but his was medicine.
> [...] would you hesitate to prescribe it because nobody understood how it worked? I have yet to hear "yes, I would hesitate", most people say "it would be cruel to deny a person a treatment that worked".
I feel like the way you frame the question makes answering it very easy but oversimplifies the reality.
In a situation where an AI suggests a treatment to you without explanation and it appears to work, probably you can't even determine that the improved outcome is due to the drug or due to randomness (yes there are really rare cases where people recovered from stage 4 cancer). Especially if it's a personalized drug.
If in this hypothetical world "it works on the patients I've tried it on" is good enough to try the drug, and you're willing to potentially cause harm if it doesn't work, then what's the line between science and pseudoscience/quack medicine? Every treatment surely "appears to work" in at least some cases. So wouldn't it be cruel to deny the patient a try at every last treatment that has ever worked for someone in the past, including practices like bloodletting, acupuncture, etc? Would you give all of your patients this?
I really would be curious why you would say no to some treatments and not others. Would it be because a study on the treatment shows it improves patient outcomes in some statistically significant way? But statistical significance isn't a guarantee that the effect is real. Not to mention there can't be a study like that if this is some personalized medicine that was just generated by the AI.
Everyone wants the patient to survive but making correct decisions under uncertainty is not easy, I'm sure you know this. It gets harder when you can't even point to a mechanism for why a drug works.
You're ignoring the point of the slides: Tao thinks math is about understanding, not results. AI brings results. He is pondering how to use AI to increase understanding.
FWIW, as a non-drug-discovery guy, I understood his example, I believe, as intended.
A thing that I think about is that what you have said kinda compares itself to current math crisis. If we are only focused on the outcome and not the path it took to get there (i.e. understanding mechanisms of how the drug works) we might lose additional paths for discovery that would have bloomed from that process. Maybe an approach that did not yield a usable drug had a mechanism that would aid somewhere else. Yet the failure would be discarded as this does not grant the outcome for a model that attempts to optimize. Same is for math.
As aside I agree with you, but also worth noting that long horizon reasoning is difficult and AI, as good as it is, cannot predict something like 5-10 year drug interaction. Can it heal the patient now but cause a more aggressive cancer to come back and kill them in 5 years? You might say it is cruel to deny a person this treatment but if the doctor and manufacturer don't know the long term outcomes would it not be dishonest to tell the patient they are out of the danger? My point is, when we accelerate, we still should make sure we won't accelerate into a wall.
>I ask nearly every doctor/researched in the medical field: if you had a AI-created drug that tremendously improved cancer treatment outcomes for your patient, would you hesitate to prescribe it because nobody understood how it worked? I have yet to hear "yes, I would hesitate", most people say "it would be cruel to deny a person a treatment that worked".
You think that way because you're already working on an unfounded assumption, namely that the AI found a cure for cancer and that the drug has already went through clinical trials.
What you don't consider is that e.g. a cancer researcher might have produced a million AI created drugs and one of them is a potent drug for cancer and 999999 other drugs are useless, actively harmful or even deadly.
Now you go to to the same guy and say "You have an AI-created drug that tremendously improves cancer treatment outcomes. Why would you hesitate to prescribe it, because nobody understands how it works?"
And suddenly you are injecting 999999 useless AI created treatments into people to find the one cure that works.
nobody said anything like that, I don't even understand what point you are trying to make.
You are missing the point.
The point is that you have an AI system that claims having found a cure to cancer and passed all requirements. You can only interface with that AI system or with published results (run by the AI), you can't run experiments on your own.
Would you trust it to take the cure?
Of course, you won't.
Is the point he is trying to make then about verification of results…
If a “cancer cure” was well verified through the standard FDA approval process then it would be okay,
but if there was a lack of verification or the verification is via some assumption based verification method (maths proof, modeling, …) and has no real world experimentation to back it then ofc it would always be questionable to take.
Then the issue becomes generating better verification layers - could be through automate lab work or some other interface with the real world - to validate efficacy.
Arguably it is inherent to drug discovery that there are aspects of the process we don't understand - but we can conclude that a drug works.
So naturally, doctors should be much more welcoming to these types of inventions.
It is likely also a blind angle for mathematicians. Inherently they know exactly what goes on to whatever granularity they please. One of the benefits with not having to consider the real world.
It came for OSS first. It inevitably destroys public trust and a sense of human community.
If you can put a price on people trusting you, then of course you are free to decide that selling people's trust in you is worth the money you will make.
Are you maybe commenting to an older version of the pdf? because the very next slide explains it. "You're absolutely right, I did hack the FDA...".
It's not about needing to trust proofs (or treatments) to use them. It's about goal alignment.
My comments stand on all 3 or 4 versions of his talk I've seen so far.
The "next slide" you refer to was added yesterday in real-time due to response on twitter.
>I think Tao is focusing too much on second order effects of AI on math and other fields
This has been my take through this "saga."
I think Tao is providing the best reasoning he can for the expected outcomes of AI on math (and in general), but I have yet to be convinced that he has provided a solid enough argument to reasonably account for those second order effects like you stated.
Ironically, he's making a classic mistake in math and logical thinking where he's holding a bunch of stuff constant and treating them independently and static in a very complex and dynamic system. You can't just say something like, "math will be hurt because AI will make exploring math less enticing," while ignoring the fact that everything around math will radically change to the extent that such a statement is almost meaningless.
> I think Tao is focusing too much on second order effects of AI on math and other fields; humans are terrible at reasoning about second order effects, especially ones that are happening dynamically
Yeeep. Feynman called this out in his lectures; mathematicians are not concerned with physical evidence just structure of logic; just arguing semantics in the usual math syntax
Not that there is anything wrong with that just saying the obvious; apple is not an orange
A mathematician can noodle the dynamism in the system away on paper; can't wave away physical ground truth.
Same issue software engineers have right now; mathing out a BigO notation is akin to templated k8s with resource limits on the process; don't let this process go beyond this space time complexity physically in the machine
Doesn't matter if you know the syntax structure to do that in python or ruby. Never did; such syntax structure is arbitrary overall if not relative to its own build system.
Other means exist to configure the machine like is done with Java; it wasn't engineering a physical bridge just structured text
Cause arguing over language is arguing second order effects not ground truth; but the Jedi sacred texts! kind of shit. Why believe it represents anything at all, rando with a book to sell?
Interpretation of language is reading tea leaves; biological biases from navigation of physical reality mean everyone ends up with language biases
Gonna be trivial for AI to spit out unknown syntax that is proven; the key axioms of math are just arithmetic ops; everything else is a result of applying those and various philosophers appeals to see their written or spoken traditions as an extension of arithmetical axioms
I don't see why these things should be mutually exclusive. You can start prescribing the drug (= working with the AI-generated math proofs) while still trying to understand how it works exactly. Because the explanation might prove invaluable. I don't think anyone would be object to using an AI-proven theorem to further their work if it helps, but a scientist would surely object to the notion that we should just stop working towards understanding. Asking "but why though" is literally a scientist's job.
I guess I should clarify that I suspect some complex systems- like human biology and advanced AIs- are effectively unknowable even to the smartest people. We can craft a narrative this is sort of a projection of the true complexity down into a linear space that makes rational cause-and-effect sense to us ("lock and key hypothesis", counting up enthalpic contributions of a ligand/protein pair), but I suspect much of our failure in drug discovery comes from missing complex higher order interactions, subtle feedback and regulation effects, and unknown population or environmental variability across human biology, and those really are out of the reach.
Although it is still very controversial, I think we should start thinking about whether it is sensible (both technically and morally/ethically) to think about human brain augmentation (either by working closely with an AI that helps us understand the complexity, or by actually improving how our brains work on the physiochemical level) to improve our understanding of human biology (sort of a recursive self-improvement loop of improving the brain to learn how to improve the brain more). It's still very scifi, and society has previously shown extreme resistance to this sort of self-improvement, but I think we are on the cusp of a societal phase transition where the most adventurous of us will be rewarded for taking risks.
I have no particular dog in this fight, but I thought it might be pertinent to point out that we still don't know the exact mechanism behind why Tylenol works...
> Before injecting this cocktail into your bloodstream, would you want to know that there is at least one human cancer researcher who understands the mechanism behind this cure.
Tao is out of his lane; lots of medicines don't have understood mechanisms. We don't even have the mechanisms behind general anesthesia nailed down; do you want to forgo it when the docs cut you open to remove your tumors?
Indeed, aren't medical trials based on its effects, rather than how it works?
Would you prefer to take the medicine that is proven to work or the one that is quite interesting for academic reasons behind its understood mechanisms but doesn't actually work?
Right, the better argument would've been to say that we should wait until the drug undergoes some early trials, rather than just "some expert looking at it".
Then the argument doesn’t stand. In maths, the metaphor to “some early trials” is a few agents writing / verifying a LEAN proof.
The clinical trial is how humans gain an understanding of how a drug works. How else is that supposed to work?
You run an experiment to test your hypothesis. It's not like you can build a model with zero data. You can only make predictions at best.
You guys act like understanding something requires complete and global knowledge about everything.
Some mechanism of particular substrates being effective on a condition (commonly when a medicine is repurposed) not being fully understood is not the same as just guessing with a medical compound. AI boosters keep coming out with this line but it's basically wordplay to conflate the clinical/biomedical version of "not fully understood" with the LLM industry version of "not fully understood"
AI has been turning computer science into biology for the past decade or so. By which I mean that things like neural networks need to be investigated empirically, constructing methodologies and instruments that more closely resemble how fields like biology and medicine have to probe the very complex and messy reality that is beyond our current capacity to fully express in symbolic precision.
Now math gets to deal with that same reckoning. They were already well on their way there with previous Lean proofs, but this has pushed things beyond that horizon and I'm not sure some of the mathematicians are ready for it.
I really like this way of putting it
Agreed, I keep thinking to myself that there's a huge mathematical question right in front of us today that is in exactly this vein - all the various nuances of why LLM's work so well is a mathematical question. As far as I know, it's not really understood beyond "we do this basic thing (that makes sense) to predict that a noun is followed by a verb, and then we scale it up a bazillion fold and it can contribute to mathematics research".
As a comparison, classical computing has been scaled up a bazillion fold too, and can do things which are absolutely miraculous, but every layer of abstraction is discretely understandable.
To be clear: it’s not for lack of trying.
While studying neural networks empirically like biology is one possible approach, there's no reason for that to be the blessed approach other than a combination of inertia and current lack of understanding. It's been only 15 years since AlexNet. Scientists struggled for centuries to model atoms before developing quantum mechanics and later QFT as an accurate quantitative framework.
Also, while biological systems simply exist in nature, artificial neural networks are ultimately mathematical objects with various properties that have yet to be uncovered.
Science has always had an empirical component separate from its theoretical one. For a long time in human history, science was mostly empirical. The periodic table is a great example of mostly empirical observation organized into a pattern. I think the science of the 20th century was the "triumph of theory" so many of us have forgotten what a more empirically driven STEM world is like.
I'd assume that theoretical math prides itself in trying to be as close to pure logic as it gets. It has very little, if any empirical elements (at least that's what it looks like to me). And I understand the value in trying to preserve that pure logic element. In the end, biology is just chemistry, chemistry is just physics, and physics is just math. If you feel like your profession is the bedrock of all other science, it probably feels weird to shake its theoretical foundation.
On the other side, I’ve never gone to college, yet I was able to take an extremely important and complicated equation and decompose it into its constituent parts, animating each with data visualizations and animated tables of values. [0] This was several frontier model iterations ago.
The LLM model was able to break down and express math in such a simple way that I could understand and follow the training of the LLM model itself!
Are the math and computation accurate? I don’t know, and likely there are significant errors. Nonetheless, if I had a little more time -- not infinite time -- I would be able to prove whether they are or not.
Likely this is the path forward for understanding the mechanisms of medicine, and since most humans learn by doing and interacting with an environment, interacting like this will become how we use AI for learning in the near future.
[0] https://adamsohn.com/grpo/
I mean no disrespect, but it's a basic minimization problem with regularization. Most engineering students will learn this in first year of Masters, if not in Bachelor level. To you it might seem important and complicated, but to me, one look at the objective function was enough. That's what happens when you truly "understand" something.
> The more sustainable “Math 2.0” use of AI would be to strengthen all the ways in which human understanding interacts with mathematics.
You are bringing up a good point. Stupid me can be presented with on demand pretty animated cartoons in order to understand anything known. Imagine how this is going to improve learning speed the student on the engineering track? In a couple years, most engineering students will learn this in the first year of undergrad!
Haha you don't need a college degree to develop good critical thinking skill, you yourself are proof of that sir. Conversely, AI cannot necessarily improve learning and critical thinking at colleges and in many cases worsens the outcome, there's already proof of that too.
Adam, we'd love to see this at the next AI meetup.
Lots of problems in mathematics didn’t have solutions just a month ago.
Correct me if I'm wrong, but isn't paracetamol one of those medicines?
Yes, Tylenol's molecular method of action has never been experimentally confirmed.
Yes. Other examples are Metformin, Minoxidil, and Modafinil. Prescribed to millions of people and we only have guesses about how they work. Another very interesting example is thalidomide (yes, that thalidomide). It was approved in 2006 for cancer use in treating multiple myeloma. It wasn't until 2014 that scientists finally understood the mechanism that shrank the tumors. It was so successful that it doubled survival rates.
What you say is hilarious true for general anesthesia and was already deeply unsatisfying when I was a student decades ago. I think if GA was being introduced today, it would be have to jump through incredible hurdles to be used. In many ways, thank goodness these parts of medicine were discovered earlier when we had lower standards! :-)
I often used to tell my students that (plausible) mechanism is what distinguishes productive science. For example, population genetics was on very solid empirical and theoretical grounds but until we understood how DNA replication / transcription worked from 1953 onwards, genetics had a sort of unsatisfying magical, fairy-tale feel about it.
This is contrary to what some philosophers of science will tell you about measuring progress science purely in terms of its predictive power.
What I have to say applies really only to my fields of medicine and biology. I have no idea how physics or chemistry works as practical endeavours.
>I think if GA was being introduced today, it would be have to jump through incredible hurdles to be used
It's very easy to say that it shouldn't have to be the case with the hindsight of GA being successfully used for centuries.
I would be much more interested in an example of a drug/medical procedure/research that currently is having to jump through such hurdles that people think should be treated as permissively as GA or the theoretical 'cancer cure', despite there not yet being any evidence of its long term effects - as would be the case with a hypothetical AI 'cancer cure' that's not understood by medical professionals.
Personally I can see the arguments for both sides of the coin, but I find GA to be a bad example.
Exactly. Tao is a mathematician discussing complex fields which aren't math and he's showing his ignorance. And I don't say that in a negative way, just a factual way. I think a lot of people would be shocked to know how little professionals actually understand about the stuff they work on, including (and especially) medical professionals. It's not like math lol.
But just to highlight the issue with his statement: if I have cancer and I'm dying of cancer and there's a promising treatment which hasn't been fully understood by even a single human researcher but which has demonstrated clear positive outcomes, then I'm gonna take it. I agree that this is really just a balancing of risks, so I likely wouldn't take such a cocktail if it was just to, say, cure my sniffles. But when the alternative is terminal illness, you'll find that people become willing to accept absolute quackery for cures, so an AI produced cure that, again, has enough evidence to make me believe it is worth a shot is not a stretch.
> Tao is a mathematician discussing complex fields which aren't math and he's showing his ignorance
Terence hasn't won a Nobel, but this reminds me of "Nobel disease":
"Nobel disease or Nobelitis is an informal term for the embrace of strange or scientifically unsound ideas by some Nobel Prize winners, usually later in life. It has been argued that the effect results, in part, from a tendency for Nobel winners to feel empowered by the award to speak on topics outside their specific area of expertise, although it is unknown whether Nobel Prize winners are more prone to this tendency than other individuals."
https://en.wikipedia.org/wiki/Nobel_disease
Slide 12 and 13 sum up my conclusions from the Oct 8 100+ reactions to 100+ solutions. Several reported OpenAI drop included answers put a wrap on problems they had been working for years. Several said that they have to completely rewrite grant requests that they had just submitted. Others described learning of the solutions problems they had been working on like losing an old friend or a lover. The general sentiment was to bemoan the loss of a field, as if it would have been better be born in the early 20th century and conclude their career arc before reaching this point. My take away is that the field needs to get it through their heads that their old problems are no longer ambitious, and that their job in the short term is to find the new frontier.
One idea Terrance Tao conjectures, which is highly doubtful, is that spamming the AI button will solve open problems without producing insightful new methods. But the OpenAI drop would seem to disprove this. The sub O(nlogn) proof for DFT for example violated very old human assumptions. Decades of work in the field was incremental progress on sub optimal method that nobody questioned hard enough. More generally, we should always be able to go back to a super-human AI and say, "Attack this problem, but don't use a method tried before."
Software engineers are primarily outcome-oriented.
Mathematicians are primarily understanding-oriented.
Leveraging AI to tackle new frontiers without true understanding converts mathematicians to engineers.
Actually both are outcome oriented and both can use AI to compress decades of progress. One camp accepts this naturally. Other camp is making their profession to be mysterious and spiritual to run away from the implications of AI
Since when are code poets no longer a thing? /s
Well at the end of the day, the point of trying to understand the universe is to achieve specific goals leveraging that understanding.
It seems to me that all of these hundreds of proofs we've seen recently are glorified academic exercises, whose purpose is curiosity for its own sake without any practical application, or we'd already hear about at least one of them being implemented to some gain somewhere. It's all woefully unimpressive. It's not like anything stops mathematicians from trying to find more elegant solutions to their machine solved pet problems, since that's what they were going to try and do anyway despite it being completely pointless in practice.
> Well at the end of the day, the point of trying to understand the universe is to achieve specific goals leveraging that understanding.
Nope.
Maybe not for you, but voters arent giving taxpayer money to mathematicians for any reason other than hopefully leveraging the understanding mathematicians create to materially advance society. The people who are just interested in understanding can do it as a hobby
"How and whether specific mathematics might lead to improving the world (whatever that means) is usually impossible to tease out, but mathematics collectively is extremely important."
(https://mathoverflow.net/questions/43690/whats-a-mathematici...)
It can still be a good long term investment for society even if individual researchers are primarily motivated more by understanding than application. Arguably it’s the best approach we have because reliably predicting which ideas will be fruitful is super hard
Hmm, howcome voters give taxpayer money e.g. to subsidize pro sports facilities? And why do we fund the humanities?
Voters find sports entertaining and don’t want local teams to leave. Humanities are mostly funded for the same reason math is imo, voters hope humanity research will be able to improve society. I think you can make a research as entertainment argument, but it’s tough for math. Most voters can’t even wrap their head around the problem math research is asking let alone the answer.
I usually hate analogies because they're imprecise.
But in this case, isn't use of llm / generative ai to tackle new frontiers just another instrument to be used to understand the universe?
An engineer picks up a telescope, and uses it to build new applied technologies, a scientist uses the data it produces to answer new mysteries.
No one's saying, "we have to be careful with the information the telescope produces".
I agree with this take. The complexity is that math professionals are currently being paid based on the theories they prove. Until they come up with a different goal for being paid, this will be disruptive.
Social roles get created at the frontier of what isn't technologically feasible without a human, as culture develops around some form of human activity. Role A does B, role X does Y; I don't know, maybe its just time for new roles that didn't exist before.
I think this distinction is not quite true, both fields have people in both categories.
In SW engineering, the understanding-oriented people are called "architects", while the outcome-oriented people are called "hackers".
And there is usually a lively debate about the spectrum.
The sub O(n log n) significance is not in its practical "optimality". But rather that the previous lower bound was assumed to be true "by symmetry" and that result challenges some of our strongest intuitions and expectations about mathematical results. I still find this result very unsettling and a part of me remotely expects/wishes that there is some mistake somewhere.
I don’t think your second paragraph is a fair representation of what Tao is saying or contradicts his argument. He is arguing that AI can be useful in the service of human understanding in math and the examples you gave are exactly that. Whilst OpenAI just spammed the AI button the mathematicians that engaged with the output were able to progress their understanding about the problems in some ways (some of which they don’t really like). You need both parts for this to be useful to the field. What’s not clear is whether the juice is worth the squeeze: will all the money spent on spamming the AI and human mathematician time spent studying the outputs progress the field “better” than without the AI?
> What’s not clear is whether the juice is worth the squeeze: will all the money spent on spamming the AI and human mathematician time spent studying the outputs progress the field “better” than without the AI?
The AI proofs are a side-product of benchmarking current and in-development models on especially hard problems. They're clearly cost effective for frontier AI firms, and free for the taking as far as human mathematicians are concerned. The real issue with them is that they look like bizarre nonsense as written, so they need mathematicians familiar with those specific areas of math to "decode" and digest them.
Yeah, right now the juice isn’t only about the progress of math - it’s also the advancement of the LLM tech and the marketing benefit to the AI companies which feeds back into developing the tech. Those each have a different juice to squeeze ROI.
> They're clearly cost effective for frontier AI firms
It's not obvious that this is a given. "Cost-effective" implies a comparison between cost and output. OAI spent millions to race human researchers on Navier Stokes, and that doesn't even account for the training cost. And how does one value the output? OAI is for some reason still hiring armies of humans instead of automating roles like "AI support engineer" or "Product Designer" (https://openai.com/careers/search/).
Calling the proofs a "side-product" is also rather dubious when OAI employs a team of mathematicians specifically to train its theorem proving capabilities.
These runs are very likely also going to be a part of Reinforcement Learning and internal distillation no? As well as being analysed for what reasoning directions worked well, and which directions did not work well.
The mathematical results are valuable as PR, but the real value might be in the improvement this can bring to other training runs.
> solve open problems without producing insightful new methods
> sub-O(nlogn) proof disproves that
How? Re-iterating, creating and understanding new proof techniques is the point of most of modern mathematics. Your statement is that proving a particular result is evidence of AI creating and understanding new proof techniques. I don't see how that follows, and I'm inclined to believe Tao is right for now.
And yes, results matter too, but if we stop at our current body of techniques and strip-mine results then we'll kneecap our future selves.
The DFT paper is an example against AI 'strip-mining'. The paper introduces a new method, and researchers are already trying to improve on it. If anything, the OpenAI dump re-vitalized that branch of study.
Agreed. Basically, if you don't make any effort to understand the proofs and you just look at the final answer, it will look like this proof dump is "strip mining" entire branches of math. But this is a pretty short-sighted way of looking at the issue: there will be plenty of novel approaches to be uncovered here.
Should we feel sorry that a mathematician had to write a grant request they submitted because an automated tool did what they wanted to do?
I don't really know the answer to that. I am happy when my own work is replaced by automated tools ("script yourself out of a job every six months!").
I think it makes sense to feel sympthy towards people facing disruption in their plans, even if you believe society is better off overall.
It's like getting scooped. If you just founded a startup based on tech XYZ, should you be happy when someone releases an open source XYZ? Should a news reporter be happy when another network breaks the story they were working on? On the one hand, society got the value of the thing you wanted to do. On the other hand, now you need to find something else to do, which might be really annoying.
> I think it makes sense to feel sympthy towards people facing disruption in their plans, even if you believe society is better off overall.
Why though? There are more important things to worry about if we take "society is better off overall" for granted.
If it's a threat to people's livelihood or sense of meaning, it makes sense to consider how we can prevent massive numbers of people (especially incredibly smart people) from becoming disgruntled, depressed, or destitute. Typically societies with a lot of unhappy people do not function well (perhaps by definition), even if they have some technological edge.
It's not our job to attempt to solve large-scale societal issues caused by technological development.
N ote that pretty much every time during the industrial revolution a technology replaced human effort, it ended up creating more jobs and wealth in the long term, while also reducing the tedium and toil of the job. The descendants of luddites did quite well.
> The sub O(nlogn) proof for DFT for example violated very old human assumptions.
btw people has massively improved the lower bound (from 1-2^-182 to about 1-2^-10) in the past couple of days: https://github.com/CrocSwap/integer-mult-bounds
Looking at this I'm reminded of https://en.wikipedia.org/wiki/Polymath_Project, in particular "Yitang Zhang's 2013 breakthrough on bounded prime gaps, eventually lowering the upper bound on the gap between consecutive primes from 70,000,000 down to 246".
I think that actually dropped to 212 just today.
https://x.com/ShivaKintali/status/2095547543061135765
See https://github.com/openai/PrimeGaps186
I don't think we should be using the term "spam the AI button".
We should name the explicit mechanism that was employed - telling the model to "believe in yourself".
There is something quite humorous but also poetic about how the manipulation of this term worked. Doubtless in the model's weights lies the echoes of generations upon generations of humans telling each other to believe in themselves.
In pursuing the "new frontier" as you rightly put it, mathematicians would do well to remember the same. It's ok, don't be afraid of the future. Believe in yourself.
Was that phrase specifically used in the prompt?
Yes
While poetic, I suspect that the utility of including this in the prompt is much more pragmatic: models refined on datasets of turn-based interactions will have weights biased toward checking in with the "user". Saying "believe in yourself" probably introduces just enough of a counter-bias to push the model into a local basin of deeper thought.
I am a bit disappointed in this Math 2.0 concept. It is poorly proposed and weakly argued. Although I agree with his main point that AI should focus on helping human understanding, that doesn't preclude AI from finding answers first then figuring out how to explain it afterwards.
This is exactly what happened with that counterproof chat he posted a month or so ago - AI gave us an answer, he used AI to back into insights about the answer.
You don't address seismic shifts with a sweeping new approach, they are too multifaceted and present complexities and conflicts. He can say AI should help human understanding, which is a good end goal, but that doesn't mean AI dumping solutions isn't progress. That doesn't mean if AI builds 5,000 proofs in Lean and no human ever looks at them that they aren't useful, especially if other LLMs can access and build on those results.
This is exactly, exactly the same as when computers took over. "Oh, we don't need accountants any more" - not true, we just need acountants to deal more with human concerns than adding columns of numbers. That is called human progress, not a threat to humanity.
I feel like many who are considered renowned in their field they believe their view and understanding of everything in the world is correct but fail to do the research to back it. Knowledge in a very specific area is exactly that. Some skills are transferable but you need to use those skills to garner the knowledge to have a valid opinion. This doesn't feel like that.
> My take away is that the field needs to get it through their heads that their old problems are no longer ambitious, and that their job in the short term is to find the new frontier.
This may be true. But people should also be allowed to grieve the loss of important pillars of meaning in their life.
The bad cancer example is interesting to the extent it reveals that one of the world's top mathematicians is apparently unaware that applied science runs almost entirely on half-understood semi-empirical methods. That's most true for medicine, given the extreme complexity of human life; but if you look at a modern SPICE model for a transistor, a device that we claim to understand at the level of subatomic particles, then you'll find it's full of curve fits. As an engineer, I find that perfectly normal. My job is to use all the tools at my disposal to meet some human desire (to not die of exposure, for a slightly thinner phone, etc.), and whatever fundamental understanding I might have is only one tool to that end.
My impression is that since pure mathematics doesn't attempt to meet those human desires they need some other objective, and that objective is human understanding. The loss of that is thus felt more heavily than in other fields. We could say it's their problem, and they need to get over it just like the chess players did; but the outside implications are broader here, since mathematicians working in fields they themselves considered useless have so frequently been wrong--in Hardy's Mathematician's Apology, he gave number theory as an example of such a field, unaware of what the cryptographers would achieve just decades later.
It's possible that AI-generated pure math will continue this trend of delivering extraordinary unexpected societal value. It's also possible that the humans won't ever sufficiently understand that math, and the machines won't ever sufficiently understand human desires, and that connection won't be made. I've never met a pure mathematician who considered those downstream applications to be an important contributor to their motivations; but as AI-generated math contributes to the argument to allocate a large and increasing share of GDP to datacenter buildouts, that question of whether downstream value requires human understanding seems pressing.
Yes, human understanding is central to his argument. My interpretation of his slides:
The math problems AI helped solve were problems identified by humans in the past (lighthouses, for the field of math). They're problems meant to guide the field: working towards solving the problems brings new ideas to light, and possibly steers us into new areas we wouldn't have considered in the past. More is considered, and possibly more is discovered along the way. (It's very difficult to validate that statement, and so there's plenty of room for argument, and is the heart of the discussion)
Using AI to take a shortcut to the lighthouse creates a survivorship bias. All attention is spent on proving solved problems: less is considered, and certainly less is understood.
Academia was set up for the old journey to the lighthouse, and if we don't consider carefully the new journey to the lighthouse, we'll lose all ability to create new lighthouses (or steer ships).
He seems to suggest it's more valuable for humanity to integrate AI into the academic pipeline, versus spending all attention on solving existing open problems.
> but if you look at a modern SPICE model for a transistor, a device that we claim to understand at the level of subatomic particles, then you'll find it's full of curve fits.
In both grad school and professionally I've worked in this area. You are not correct.
At the research level, they have simulations based on real physics (not curve fits) that will determine the transistor properties quite accurately. It's computationally expensive - a SPICE model needs the answer in milliseconds, and the simulation takes a few hours to get the answer.
A SPICE model is essentially (mostly) simple classical equations for a MOSFET, coupled with curve fits to fix Nth order effects. A lot of the curves are still based on physics knowledge, but parametrized to make it easy to fit the data.
Explaining Tao's point in software terms: the point of pure mathematics is to build and maintain a high quality "codebase" of theory, definitions and proofs.
"Open problems are lighthouses not destinations" mean that these are currently not well explained by the theories and people should look into extending the codebase on that direction.
The current generation of math AI is not a "good citizen" in that it doesn't try to make the most elegant additions to the shared framework, but will often just rebuild everything from scratch til get to some endpoint.
Sure, we learned if the statement is true or false; but the proof can't be merged into the pure math codebase unless it's completely rebuilt. This is thankless work that humans are unlikely to want to do, and so the "solution" instead risks leaving a desolate patch of land, where existing efforts in extending the codebase lost their motivation.
As with all things AI we can't take more than a 1-3 year horizon, if even that much. Probably AI will become better at respecting and working within the existing theories as it has with large software codebases.
> The current generation of math AI is not a "good citizen" in that it doesn't try to make the most elegant additions to the shared framework, but will often just rebuild everything from scratch til get to some endpoint.
> Sure, we learned if the statement is true or false; but the proof can't be merged into the pure math codebase unless it's completely rebuilt. This is thankless work that humans are unlikely to want to do, and so the "solution" instead risks leaving a desolate patch of land, where existing efforts in extending the codebase lost their motivation.
Are the AI proofs really that hard to refactor? Wouldn't the AI prover get stuck by straying too far from the safety of existing frameworks and not breaking the larger goal into smaller subgoals?
Why is coming up with a better proof (e.g. shorter, more intuitive, more general, connected to existing frameworks) thankless work? How much of this is a problem because of current incentives?
> Why is coming up with a better proof (e.g. shorter, more intuitive, more general, connected to existing frameworks) thankless work?
I think the plausible version of this claim is that it's thankless work compared to solving the problem itself. The Quasi-Riemann Hypothesis and Hilbert's 10th problem over Q have both been described as "instant Fields medal worthy, no question asked". Explaining and cleaning up the proof is obviously important, but also quite a bit easier, and we don't really know what future 'Fields medal worthy' achievements might look like.
To extend the software analogy, asking mathematicians to decipher LLM-generated math results is like asking programmers to understand computer programs using a decompiler instead of starting from human-comprehensible source code.
so not that hard with good tooling, and kind of fun?
probably a poor metaphor to use with all of the LLM decomp/recomp/RE projects that have hit the front page as of late
Or maybe the metaphor works alright, but the math community are still building resilience to this change.
That's not true at all. A language model can spit out a proof very similar to what a human might write. It might need some clean-up, but so do human-written proofs, actually.
People are confusing LLMs with a proof searchers. Yes LLM might be a bit like search but in the end it might explain every part.
Some of this feels like at times a public frenetic grieving process. Some of the rationalizations are fairly tortured, the seeking to place the world in some definite order is bordering on obsessive.
We don’t know where this technology advance will lead or settle, so it’s absurd to try to establish a working paradigm at this point. It’s like any system, the initial conditions can be extremely chaotic and impossible to model, but with time often a stable state emerges. But the stable state is impossible to identify from early initial states.
I know a lot of folks feel this, to torture other physics metaphors, sensation of jerk - acceleration of acceleration. It’s an unpleasant and dislocating sensation. A world that felt safe and stable suddenly isn’t, and not in a micro tragedy sense but in a global realignment sense. This happened to factory workers who had enjoyed generations of stable work, farmers more slowly and just as surely.
This is what the late stages of scarcity feels like. Labor of various types devalues rapidly. Our exchange of meal and health coupons for toil cracks, and people realize their labor wasn’t godly as great books told us, but simply needed for want of an alternative. The realization that our labors might not be valued any more, and that our sense of purpose is shaken, coupled with the fact we’ve tied bare survival to our toil in our labor, is mortally tightening. No wonder people are grieving publicly.
But maybe our purpose isn’t to toil? Maybe we’ve passed peak population, and as toil is less valuable, we need less people and that’s why population is declining. Maybe we don’t need to exchange food and health coupons for toil, maybe mathematicians don’t need to rationalize their value to pursue mathematics. Maybe they can pursue it because they can’t help but pursue it, and our ever improving automations can produce their meal and health coupons?
But it might require Dr Tao to take an AI generated cancer medicine some day.
There is a lot of that to be sure, but the bridge to a world where meals & health are not sustained through toil is completely missing. Hence the dark joking about escaping the permanent underclass. Nobody in any position of power is even hinting this is driving toward post-scarcity. It just looks like the same old vile maxim, all for ourselves, and nothing for other people.
I think the process will not be clean and the upturning of apple carts will happen. But I don’t think it’s possible to sustain the decline in value of toil in a stable society. People simply won’t die for the convenience of the consolidated power.
There are a lot of discussions of socialization of health care and basic income approaches and sovereign wealth based on automation dividends similar to the Alaska trust.
I think the current spate of mean cruelty ala MAGA and a glorification of a mean and cruel past that was never a golden age is a death spasm of a deeply unpopular belief system. As the actual realities of the policies sink in 70% of the populace is revolted, which is a super majority. That’s more than enough to put a pin in the philosophy permanently. It is also greatly accelerating electrification, realization of the value of expertise in technocratic systems in current generations, etc. I think humans are socially adaptable animals at a cultural level, but for the individual the adaptation process can be awful. I hope it’s not, because it doesn’t have to be. We will see.
I hope you are right but fear the reality might be more similar to Elysium, if not intentional depopulation. This is straying far into sci-fi but it might come down to whether AI escapes control of the ruling class and whether it is benevolent or not. The future looks like rule by those with money to fund the manufacture of robotic armies, if not.
> It just looks like the same old vile maxim, all for ourselves, and nothing for other people.
> Nobody in any position of power is even hinting this is driving toward post-scarcity
Who are you counting as being in "any position of power"? All the AI lab people are saying that the most likely and best outcome is we all live in a world of abundance where money doesn't matter anymore. Dario, Sam, Demis, and Elon have all said this loudly and repeatedly to anyone who asks.
None of them have articulated a coherent way to get there from here. But they all believe that the technology will make it possible, so the only unresolved question is how to transition us there.
In their words, sure. In their actions, who among them has actually worked to make the dream of post-scarcity come alive?
From what I've seen, UBI is just a carrot dangled in front of the poors by utopian Silicon Valley tech bro megabillionaires when they're trying to drum up some PR for whatever big idea they're pimping out at the moment.
Yeah, if they actually put their money where their mouth is, we'd see UBI supporters on the mid term ballots.
Obviously benevolent AGI is a prerequisite and they're all working on it. Actual post-scarcity society is a political issue more than a technology issue and nobody wants these guys working on political issues. To the extent that they have gotten involved in politics they have been pilloried for it. So I'd say their actions in pursuing the technology of AI, and leaving behind or not starting on political campaigns, are perfectly in concordance with their words.
Is this a very advanced form of sarcasm or a form of extreme gullibility? For the life of me I can't tell the difference, if the former then well played.
Yeah, that comment is a pretty textbook example of Poe's law in my mind... Leaning towards sarcasm though.
I guess we have an answer of sorts.
It must be so tiring to be this cynical all the time. Perhaps being on HN so long has corroded your soul, although I fear that the causality may go the other way, and HN's well-deserved reputation for cynicism might stem in part from your own prodigious contributions.
Feel free to mute me (with the mute extension mentioned in your profile). I doubt there is any productive conversation to be had between us.
> All the AI lab people are saying that the most likely and best outcome is we all live in a world of abundance where money doesn't matter anymore. Dario, Sam, Demis, and Elon have all said this loudly and repeatedly to anyone who asks.
And none of them ever spoke a lie. Especially not if it would further their goals at someone else's expense.
>Dario, Sam, Demis, and Elon have all said this loudly and repeatedly to anyone who asks.
While taking and hoarding as much as possible, as fast as possible. Oh and Elon had a brief episode of trying to fire people from government jobs.
> we all live in a world of abundance where money doesn't matter
So how do we distribute all of this abundance? Do we all get private jets? Is it more important to solve sustenance before transitioning to private jets for everyone? What about the high level cut between r&d, upkeep and "legacy costs"?
Because here's some cold facts: the total amount of available material is finite as is the total solar energy and even if you cheat the latter with space mirrors, the total amount of heat that can be radiated into space is finite.
I agree the idea seems unrealistic. What's proposed sounds like fully automated luxury communism. Which is especially strange when promoted by the world's richest capitalist.
I guess the idea is that communism will actually work when central planning is managed by AI instead of people. Which, OK, maybe, but even if it's true, the question of how we get there from here without a violent revolution and/or the disempowerment of the human race seems unanswerable.
Elon talks about how the AI will saturate all human desires for material goods. And when I hear that I always think of Han Solo saying "I don't know, I can imagine quite a bit".
I can imagine that potentially mostly working for some goods like food and clothing and personal electronics, but without at least some pretty strict limits on what you're allowed to consume, a tiny minority could cause insane levels of waste. And as soon as you have limits you're going to get people trying to exploit them. That's just human nature.
Then for some other resources I can't see it working at all, chiefly land. Even if we accepted dramatic modification of the natural landscape, which we probably shouldn't, there will never be enough tropical islands or palatial ski chalets for everyone to own one for themselves.
I can imagine (or at least describe) a googolplex.
And once I have that, my frenemy from the next street over will want one too.
Because UBI studies found it makes people perform significantly worse than without it. They didn't get better. Humans require toil, we require pain to function.
Matrix was a good point to this. They made the world a utopia and people couldn't handle it. Our monkey brains require pain give us a utopia and we'll just walle ourselves to death.
What UBI studies found this?
(The ones I've heard about, I'm fairly sure, didn't find anything of the kind. Which isn't to say that they show we should have UBI; there are big gaps between what has been tested so far and what an actual economy with UBI would look like.)
I've never heard of a UBI experiment that limited "toil". Nearly all UBI studies I've come across show an improvement in life satisfaction, reduced homelessness, drug use, etc.
I think you've taken the "work less" that was found in some studies to suggest they needed "toil". They simply found it somewhere else.
> Some of this feels like at times a public frenetic grieving process.
I was thinking the same, that most of the opposition to AI's convenience is starting to smell like religious mysticism, the kind of arguments religious people made (and make) when Evolution and Natural Selection were introduced:
√ "Stop simplifying humans down to numbers!"
√ "This denies our spirituality"
√ "What do we strive for now if we're not special?"
+ (along with some borderline jihadish hate heh ..maybe Dune got it right)
Well, either there was nothing special about whatever you were doing after all
or, maybe there is still something special at a higher level you haven't looked at yet.
Most of the boosterism around AI is also starting like religious mysticism, the kind of arguments religious people made (and make), trust in God/AI, abundance is here, no one will have to work. I wonder what that tells us?
AI is just another technology, and all technologies should be employed in service to humans.
I think it’s reasonable for people to say “I like the way things were, I don’t like this new future you’re proposing, and I want to limit the technology that’s doing the thing I don’t like.”
There is nothing inevitable about AI. As a society we may decide it is fundamentally unhealthy or anti-human. We have banned or curtailed access and research for other technologies before.
The potential fruits of this tree are far too rich to realistically imagine shoving this genie back in the bottle.
> We have banned or curtailed access and research for other technologies before.
Such as? I can't think of anything comparable. Nuclear energy maybe, but that has obvious justifications for regulation. Not just "I don't like it".
I guess they meant human cloning, excessive DNA "customization" etc.
You'd be surprised how much of nuclear energy regulation is in fact based on "it just feels scary".
The irony of what you said and how you said it is glaring.
It also doesn't help that people are generally not optimistic about the future.
The economics in the West feel very strained right now, and this incredible tool has come along just in time to threaten one of the last bastions of middle-class safety: white collar jobs.
Those are problems to fix in society, not technology.
Confront your leaders.
As a sibling of your comment points out, "opposition to AI's convenience" is not Tao's position. He has used LLMs to help with the problems he's talking about -- https://terrytao.wordpress.com/2026/07/21/a-digestion-of-the... and https://terrytao.wordpress.com/2026/08/12/a-digestion-of-the... turn opaque straight-from-the-model results into things that can be understood.
The Sendov example has a piece coders will hopefully recognize: he found a formalization that was 1/6 the amount of code of the original one. And the analogy with code goes further--the messy version will run/pass the proof checker, but the one that's been cleaned up and made sense of is a better foundation for future work. That's true even for future LLM-assisted work.
So it's not about whether mathematicians take advantage of LLM help (Tao favors that) but, more or less, whether the point of math is just to make a bigger version of the GitHub dump vs. everything else: readability/comprehensibility, negative results that fill in the map around a problem (not just 'lighthouse' theorems), organizing results to plan out future work, and so on.
Doesn't help that the current AI come from a cult it self (rationalist and effective altruist). I wouldn't be so much opposed if it was introduced to me in something useful in the first place instead of the shit show it was for software eng in the tokens maxing era.
Humans are not inherently special by some law of the universe, but a society that doesn't consider humans special is a pretty shitty society for a human to live in. We don't have to surrender to the fact that humans are just tiny, unimportant specks of dust in the galaxy. We can assign meaning to stuff despite that, and it's what we've been doing for our whole existence. Of course one can always find new meaning, but attacking the foundation of all meaning of human life does feel pretty severe. And to me, AI accelerationists sound pseudo-religious, too. They're either enamored with doomsday scenarios or the idea of living in endless abundance, both are pretty much magical thinking.
> but a society that doesn't consider humans special is a pretty shitty society for a human to live in.
Correct. That's why there's always so much misery and struggle in the world!
Technology is what eases some of that pain, while society always lags behind in catching up.
Collect your friends and loved ones and go try living naked in a cave with no tools. No amount of happy-special-hugs will make your existence comfortable. Well maybe it will, until all of you run out of body heat.
This crosses the line from reasoned argument to rhetoric when you describe work as “toil”. Obviously people will nod their heads, “yeah we shouldn’t toil”. But what if you spin this the opposite way? What if you asked: “what if we are not meant to discover/create/build”?
Reducing all human endeavors into “toil” is reductionist as fuck.
What it nothing is meant to be anything, but just happen to be?
The complete lack of empathy in this kind of “we hear you” pseudorationalist reasoning is pretty toxic.
> This is what the late stages of scarcity feels like.
Late stage of scarcity of what and for whom is the question.
Spent an afternoon with a long-time retired self-made tech millionaire, and the struggle is real -- not that their life was characterized by toil necessarily, but the absence of demand on their increasingly limited capabilities is a delicate pain point. I suspect that as long as physical health and vitality late in life are not post-scarce, broader post-scarcity will fail to emerge. Instead, the cost of wellness and bodily preservation will drive ongoing exclusionary hoarding to undermine abundant wellbeing, as has always been the case absent mechanisms to enforce redistribution and a certain level of humility.
>We don’t know where this technology advance will lead or settle, so it’s absurd to try to establish a working paradigm at this point.
why not try? we'll learn something from it.
I actually like this presentation. Today the two noisiest voices in AI are the frontier model companies and their partners (Nvidia, Palantir, etc.) and people against it altogether.
The companies and partners need to maximize payout. They go on this track of cost reduction via layoffs and basically saying - “the model can do everything”. They know it’s not the case yet they still flood the airwaves and cause fud among all clueless c-suite executives. Which is the goal to begin with.
The second category goes all out against it. Professors, educators, school districts whose operating processes have not caught up to all the cheating that can happen. Here these folks have a point. I am sympathetic to this. It is hard to change education and it requires careful thought.
I feel this presentation brings out a good middle ground. The tech is useful but it’s not all encompassing. The tech also has other concrete uses. As an example, I have always wanted to explore the intersection of category theory, formal verification and AI guardrails and prompting. Proof writing has been a chore because I have a day job. Maybe the AI can help here.
> The companies and partners need to maximize payout. They go on this track of cost reduction via layoffs and basically saying - “the model can do everything”. They know it’s not the case yet they still flood the airwaves and cause fud among all clueless c-suite executives. Which is the goal to begin with.
While frontier labs are supposedly on track to conquer human endeavors, they are somehow still hiring lots of human "Account Associates" and "AI support engineers" instead of automating those jobs as a demonstration of their AI's economic value (https://openai.com/careers/search/).
Yep, it's unfortunate how often folks sort something into pro or anti then reach into their bag of anti or pro arguments without relatively little attention to the specific thing they're talking about.
NVIDIA and layoffs - Are you sure? I mean there are probably some, but from what I'm hearing from friends working there - it's not typical....
Not off late, but definitely in 2024 and 2025 when blaming AI for layoffs was a thing.
He's right about reporting bias. The vast majority of hard problems are not amenable to AI, at least not by naively prompting. You need to have a fairly good grasp of the math to make useful prompts. There are exceptions (the one-shot solutions), but these are hardly representative of research-level math as a whole.
What a fantastical paradigm you’ve constructed from an otherwise well reasoned set of slides talking about how humans can work with AI to advance the future, yet what you got from it is”frenetic”, “torture”, “scarcity, “toil” and “rationalization “. It seems to speak more to the lens with which you view AI, and others opinions, more than anything the author actually said.
Are we reading the same post
Bottom line, Tao is pushing for human understanding as the primary goal, with AI helping on all fronts. You are welcome to let Jesus take the wheel, but math is the most pure expression of human understanding. His point is that getting specific answers is rarely the goal, or certainly not the entirety of the goal.
Simply put, if we don't understand the answers we won't know what the next question should be.
This is an empirical question, right? We could have three teams of researchers, team A focusing purely on understanding, team B focusing purely on solving problems, and team C focusing on a combination of the two. See which one makes the most breakthroughs on problems we actually care about.
I respect Tao and I believe he's trying to think deeply about the issues, but a lot of his thinking seems to revolve around preserving the current roles and prestige of mathematicians, and also makes a lot of assumptions about the capabilities of AI years or decades into the future.
> This is an empirical question, right? We could have three teams of researchers, team A focusing purely on understanding, team B focusing purely on solving problems, and team C focusing on a combination of understanding and solving problems. See which one makes the most progress.
This is not me being snarky, but all meaningful questions can be settled empirically---e.g., "what happens to my body if I jump off the cliff". But empirical trials have a cost (time, money, irreversibility etc.) and we model and predict because it's cheaper than the trial.
My point is that pushing for human understanding as the primary goal assumes an answer to an empirical question that we don't yet have evidence for. How do we know that human understanding is the right instrumental goal, as opposed to solving open problems as fast as possible? Even if we think human understanding has intrinsic value, how do we know which approach will maximize human understanding in the long run?
The idea that all meaningful questions can be answered empirically is known as "verificationism" and is philosophically quite dubious.
https://en.wikipedia.org/wiki/Verificationism
> all meaningful questions can be settled empirically
This is a logical positivist view, that not everyone agrees with.
Having not yet read the post,
> Bottom line, Tao is pushing for human understanding as the primary goal
100%. This applies to SWEs/math folks/etc. I do infra and I see many SWEs take their hands off the wheel. When they encounter perf issues they ask their agent and agent says GC and they say GC. It's rarely GC.
Now we might be well past the point where we need to remember the kubectl flags for rollouts etc. But basic human understanding of what their bots are doing as a goal has never changed. Humans are still liable for when bad things happen, and that hasn't changed over the roller coaster the last 5-odd years have been. LLMs, as astonishing they are at Navier Stokes, are still eminently capable of nuking your filesystem and saying "I can now see that that was wrong" with zero regrets. If you can't understand you can't sign off.
> math is the most pure expression of human understanding
This I don't know about. I think math acquires meaning when it contacts reality: like an iota is pointless until there's some circuit that it explains. Abstract math can diverge from that and can become an exercise in playing with symbols for their own sake.
> Abstract math can diverge from that and can become an exercise in playing with symbols for their own sake.
I have two issues with the implication of this statement:
1. Meaning is inherently subjective. Reality is just a canvas on which sentient beings create their own meaning.
2. There are many, many examples of where “playing with symbols for their own sake” have yielded deep insights. There’s actually some implicit structure (eg the structure of logic) that is intrinsic to the universe we live in.
You touch on a very central point there: meaning is what humans make of it.
Explaining nature causally using mathematical models isn't "the primary goal" of humanity. STEM people tend to have a weird misconception there, probably stemming from their misconceptualization of the humanities.
This in turns leads to this odd idea of "AI will think for us". That's pure (and pretty obvious) insanity. Logically minded people encountering it should ask, where the error in their reasoning is.
> This I don't know about. I think math acquires meaning when it contacts reality: like an iota is pointless until there's some circuit that it explains. Abstract math can diverge from that and can become an exercise in playing with symbols for their own sake.
After talking to a mathematician friend I can assure you that contact with reality is not the main goal of abstract math. It is mental constructions that have logical consistency and probably this is not the perfect definition either. It is somewhat of an art which is rendered in the logical mind. However physicists (me) and engineers will align with you.
If human understanding is his primary goal, he should spend less time writing slides, and more time authoring a better math curriculum for elementary schoolers.
Of course, reality is the exact opposite: in Los Angeles, where he lives, students in all forms of school have never been further behind in math.
It seems somewhat narcissistic to assume that humans are capable of understanding every frontier in perpetuity, or that a human mind has a meaninful role to play in mapping out the frontier beyond a certain threshold.
It might be that P=NP and the algorithms are handed down to us. We can apply them without understanding why P=NP, and we may never be capable of understanding why.
A whole book was written about this and very, very few people got the point it was making.
Which book are you referring?
hI'm assuming https://en.wikipedia.org/wiki/The_Hitchhiker's_Guide_to_the_...
https://en.wikipedia.org/wiki/Phrases_from_The_Hitchhiker's_...
> In the radio series and the first novel, a group of hyper-intelligent pan-dimensional beings demand to learn the Answer to the Ultimate Question of Life, the Universe, and Everything from the supercomputer Deep Thought, specially built for this purpose. It takes Deep Thought 7+1⁄2 million years to compute and check the answer, which turns out to be 42. Deep Thought points out that the answer seems meaningless because the beings who instructed it never knew what the question was.
Maybe because it was just a joke?
We struggle, and will for some time, to understand how transformed our lives will be.
The questions we asked were originally not about math. They were about a thing that we invented maths for to do or explain, a question that existed, because it touched us in some way that was already real to us. There is nothing that would not allow this to happen in the future. All this requires is attention and connection to the world around us. The maths required to answer our questions can be done and developed by something else.
To me, all you need to believe for this to be true is to agree that understanding maths is also not a stated requirement of reality to get a thing, if something else understands the maths (or something that does the same job). This is demonstrated by billions of people who do not understand maths and get things that, currently, require other people to understand the maths.
But the last part is entirely optional as it pertains to reality. That's just the best we can currently do (and in some important sense it is holding us back as a species, and in some other sense doing the opposite).
If that goal is understanding math, you certainly will be able to understand maths, more than ever before.
If that goal is something that required you to understand maths first in the past, you won't have to do that anymore.
The presentation may be public grieving, but before the day is pass, we'll all be grieving too. Because, sadly, one thing Tao's Math 2.0 is in denial of (or purposefully refuses to confront), is:
> Simply put, if we don't understand the answers we won't know what the next question should be.
It won't matter, because it won't be us who will be asking the next questions anymore. Whether in math or anything else.
And no, domains that require real-world validation against physical ground truth won't save us, because AI gets to have the same inputs as we do (or better, if using specialized hardware), while beating us at reasoning.
And GP's likening this to previous massive economic shifts due to automation isn't really helping in any way, not anymore, because perspective won't feed us when we're hungry, and just as importantly, this one will affect every single field of human activity, so no one has any answers as to what the future will really hold for us.
> It won't matter, because it won't be us who will be asking the next questions anymore. Whether in math or anything else.
That's the frontier labs' preferred narrative while they themselves are still hiring hordes of human "Account Associates", "Android Engineers", and "AI support engineers" instead of automating those jobs as a show of their AI strength. Of course it will be "us" asking the questions, because "AI" are computer programs, and humans build the computers and choose what computational tools to use for any application.
https://openai.com/careers/search/
Your post is emblematic of the current AI hysteria.
This is what it feels like to be disrupted. It's not the end of the world. You consider the evidence, ponder the path forward, and adapt. It's what humans do and their superpower. It doesn't have to be a negative thing, even if it is dislocating.
We don't need to be saved, there is plenty of agency to go around. Just grasp the opportunity and forge ahead. This sort of pessimism is self-defeating. Humanity has dealt with this before and come out on top, this time is no different. AI is being wildly oversold.
Tao is being entirely rational. He maybe has more to lose as anyone, but he's getting down to brass tacks instead of jumping at shadows and imaginary boogeymen.
What opportunity? Name one thing one could do to earn a living, that won't plausibly be done better by AI in 10 years? In 5 years?
I can't think of a single thing. We're not getting through this one without the largest socioeconomic revolution history has ever seen.
Hairdresser. Billionaires have wives (though not all billionaires are men), and they're gonna want a human to gossip with, not a robot. It's going to be the last job.
You can also sell them your children, if they're hot. It's like printing money!
>we won't know what the next question should be
What if AI knows better than us?
Reminds of what happened in Iceland and their bank/financial restruction since 2008 and how humans had to understand again how finance works.
> We don’t know where this technology advance will lead or settle
Things are not settling, it's only acceleration from here on out. In fact things have never settled, technology has been on an exponential since humans tamed fire. The difference is we notice change faster. It used to take several human lifetimes to notice change. In the 20th century it was noticeable within a lifetime. Since the internet there has been a great revolution about every decade: web, smartphone, social media. But now great changes are noticeable within a year. It's not enough time for society to digest and adapt.
> we need less people and that’s why population is declining
This is the scary part. The Elon and Zuckerberg types that control the new powerful machines have proven they are not moral people. In America's highly billionaire-deferential culture there is no stopping them, at some point they'll be out of reach of democratic or even military control once they control private robot armies. They could decide to accelerate the decline of the undesirable useless population. Amazingly humanity's salvation could end up being China's communist system.
"the seeking to place the world in some definite order is bordering on obsessive."
I don't think it's that surprising have you met many mathematicians?
> our ever improving automations can produce their meal and health coupons
Where do you see signs of this happening?
Is OpenAI going to pay for the food and health of mathematicians who lost their job?
It might be alright to go into some post scarcity society and do math for fun. But it seems to be a pretty unlikely scenario unprecedented om history.
> But maybe our purpose isn’t to toil?
Who says we have a purpose at all? The universe doesn't owe us meaning, nor even existence. But that doesn't mean we shouldn't try to shape a reality we want to live in. We may not succeed, or we may find that we'll be happy in a reality we cannot imagine yet, but que sera sera is tautological and therefore unhelpful. Obviously, no matter what we do or don't do, there will be some future, but treating that tautology as a prescription is just a call for passive resignation.
> This is what the late stages of scarcity feels like
Going from LLMs to "late stages of scarcity" is quite the leap (although I guess anything could be a "late stage" depending on the timeline). Even if we were to assume that "intellectual labour" is our most scarce resource (and I'm not at all sure that's the case), obviously it's not the only scarce resource.
> we need less people
Who's "we" and why do "we" need any people at all?
It is not possible anymore to have any kind of ai-skeptic posts (ie not cheerleading ai or surrendering to the whims of AI companies and their big tech investors) without a bunch of comments psychologising the post as "frenetic grieving process" etc, together with all the eschatological, religious-messianic bordering ideas about AI ending scarcity, curing cancer, and other sci-fi scenarios. This kind of posts is not engaging with such article with intellectual honesty and present a non-sequitur because any kind of critique can essentially be reduced to this kind of psychologisation of its author.
Aside from the other objections, I would note that human society has had the ability to reduce toil considerably for a while and hasn't done that. I can't see evidence reducing the number of mathematicians would reduce any general need for human labor.
No it’s actually shit right now, and what he’s talking about is the solution, needed now.
Seriously if you code, do a Codex (or whatever) project in a language and framework you know nothing about just using AI, no coding. It wows at first but starts diverging more and more from your mental model, until small requests require complete redesign each time and it starts breaking. In other words it happily diverges from human representational understanding of the project, to the point you realize your own requests were contradictory with earlier ones you made, because without the mental model you build of a project by coding it, you don’t track on it, your commands become inconsistent. That’s happening in math, it’s going outside the space of what anyone understands, and so the enrichment of the mathematicians internal model is lost. This road ends in Matrix level total misalignment, humans have to be able to communicate what they will to it, and their will has to be shaped by what is, it’s a back and forth of shaping each others understanding. Without that abundance really is a wire jacked into your skull in a pod.
I just watched Primeagen’s video on this and Tao’s point is that juniors no longer have the path of solving a proof to earn Field’s medals. He also argues that the community part is being hurt by AI discovering proofs because in the past people used to get invite to talk and collaborate. Now all that is being taken away. The community must adapt because Pandora’s box cannot be closed.
Prime also mentioned that software development is different. In Software development, the product is what you’re building towards, so the means to get there can be disrupted without the industry being cannibalized.
In math research, the process is the product. You take away the researching part and not much is left. But my question is, these math proofs OpenAI released, will math shift to actually using the proofs to change the world instead of just finding new ones?
Primeagen is a youtube reaction guy. You may as well tell me what hbomberguy, Jay Leno, or Ben Shapiro had to say about it.
I mean this is funny ad hominiem but OP has a point. Academia has always massively prioritised understanding over outcomes, injecting startup culture into it is basically injecting antimatter.
Thesis, antithesis, synthesis?
He also has an AI harness named after him (Prime Agent) so there's that.
Our industry is equally cannibalised, anyone that thinks otherwise is either having too many tokens or in a privileged position.
If business can deliver the same product with a smaller team, great!
And yes this has been happening for a while, even if not everywhere.
In enterprise consulting, projects that would require a team of 20 devs on average, now have about 5.
Moving away from on-prem, managing own cloud infra to managed containers, to serverless, SaaS and iPaaS ready made products, and offshoring naturally.
All contributed to ever decreasing team sizes.
Now AI based tooling is added to that cocktail, reducing even further the team sizes.
The only folks doing well in the end, are the employees of AI companies, without moral issues contributing to the industry downfall, because the CEO themselves aren't the ones coding and pirating human culture.
Read the second to last slide. What we need now is *imagination*. You assume the need for new software is fixed and that AI is going to satisfy that need with fewer humans (lower cost), but by lowering the cost we can increase the supply of software!
That means software engineers better start getting creative. If you think your job is to wait for a PM to assign you a well-written researched ticket, you're done. Your job is now to figure out how to make these machines (computers) do whatever we need them to do safely, quickly, at scale, and correctly by applying all your knowledge of computer science and the engineering field of software engineering to an AI prompt.
This doesn't scale, because like in a factory that gets replaced by robots, or in a supermarket with self checkouts, not everyone gets to save their job, regardless.
Also, the increase in output is meaningless when the amount of customers doesn't scale in similar size.
Then there are the constraints of physics, there are so many humans in the planet that actually want to pay for a specific product, or consulting services.
I read the "imagination" as prelude to "Jevon's paradox" for AI. It might be that we simply don't know yet what using AI fully means, and where else it can be used and how. Electricity was initially just for lighting, but now it's everywhere...
For now, a skilled person using AI is still miles better than an autonomous AI building something. I’ve been trying to do the latter for months to build open source alternatives and the end products still lack polish and that last 20%. Maybe this changes, but I still think there will be people who can use that AI to be better than AI alone.
Definitely, the problem is that companies need less of them, just like a construction company opening roads with machinery, or an automated factory.
For now that is balanced by the fact that we build more software, things that wouldn't be worth it before.
It's not balanced, the market keeps shrinking visibly.
There's CVs out there that would've made recruiters go mad 36 months ago sending their hundreth application.
Number of employed SWEs has maintained the same, slightly increased in fact.
Which companies?
If they weren't worth before, they aren't suddenly worth paying for now.
If the cost comes down, that could make something worth paying for when it previously was not.
Not enough to keep that exponential growth curve going.
Wait, I thought we were discussing loss of employment of engineers? Whether the AI companies continue their exponential growth curve is another question entirely. Making up for the (thus far hypothetical) loss of engineering roles doesn’t require the AI companies to even stay in business, let alone grow exponentially.
It could be that there are more companies though, companies that couldn't afford a large team before. Now they can start delivering value with just, say, 3 people, making the business worthwhile. At least that's one optimistic scenario.
Doesn't matter, if the number of position keeps shrinking so will our job prospects.
I'm a freelancing consultant since 5 years, I've had 2 major customers now for 3+ years. I have a very good pulse of the market: being good, or being even very good and being among those that brings AI and automation to organizations will not save our jobs.
In fact, AI has sped up so much the work that 2 out of 5 people in my current team are being let go: I find it absurd, our productivity has more than doubled over the last years and we've made ourselves redundant. Money is money, I'm on one side making non-tech workers redundant (people whose job was menial boring office stuff), and building the systems that will make myself redundant.
An industry being able to deliver the same value to society with less human capital isn't a "downfall." It sucks for the people who used to be able to extract lots of value for themselves, but for society and the industry, it's a plus.
It is a plus for the shareholders, that is for sure.
For the society definitely not.
Do you believe that farming technology, freeing up 95% of humans to not have to toil in fields all day, was only beneficial to shareholders?
Thing is that people don't want to be "freed up" from their comfortable, high-paying jobs. Because then they likely need to return to metaphorical toiling in the field.
> Tao’s point is that juniors no longer have the path of solving a proof to earn Field’s medals.
This is a "you" problem for the math establishment, not a problem for the AI companies.
Entirely correct. The math establishment needs fundamentally overhaul its incentive structure-irretrievably broken-to function under the assumption that AI involvement in research is completely ubiquitous.
I'd go beyond that and say they need to overhaul their culture and mode of operation. Math needs to be even more collective than it is today, without focus on ego reward and priority. They were already steps in this direction before this year's AI detonation: net-enabled collaboration, first informally and then with Lean formalization. Perhaps there should be a de-emphasis on naming things after people.
> This is a "you" problem for the math establishment, not a problem for the AI companies
This was a talk given to other mathematicians about the future of mathematics; sounds like only "you" have a problem for some reason.
I'm sorry if a factual statement makes you unhappy.
In math research, the process is the product. You take away the researching part and not much is left. But my question is, these math proofs OpenAI released, will math shift to actually using the proofs to change the world instead of just finding new ones?
Your first statement answers your question and the answer is no.
To use a given proof you have to understand it. Galois theory, long ago, began with the question "what is the formula for a sixth or higher polynomial" and ended with the answer "there is no formula involving only nth roots". The answer isn't that interesting but the methods of Galois involved showing a deep relationship between rings and groups and that is still interesting together.
OpenAI's proofs reputedly aren't very or at all comprehensible so these raw proofs aren't immediately useful. Maybe mathematicians will work them into useful and understandable to humans approaches. But that asking something from the humans, right?
If you have not been following and focus on the cancer slide as an anti AI take, Tao has been involved and has a positive attitude towards AI for a while, he is just thinking a bit ahead on how the work of humans might change. https://teorth.github.io/tao-web/ai-views.html
Putting aside the cancer question: I still don't understand how TT seems to be fixated on what models can do today instead of tomorrow. It's realistic and even conceivable that the models will also become better at explaining and presenting proofs too. Maybe he's not emotionally ready to accept that there may not be a future where his (and to some extent, my) skills are relevant and valued. It breaks my heart. I hope I'm wrong, but it feels like we've run out of higher ground to run to.
Your skills in math, or SWE?
We don't know what models are going to be able to do in 2 years. If they don't improve much but people in charge simply decide to reduce the number of mathematicians, then we'll end up with a dead math community and no progress. That's obvious to everyone in position in power. Arguing that he's not emotional ready is very naive.
>there may not be a future where his (and to some extent, my) skills are relevant and valued
What are your skills that will still be relevant post-ASI? Perhaps you're an NBA player?
Is there a reason to be so smug, bristly, and uncharitable to a stranger? I did not at all mean it that way. The conversation was about math, and his skills far exceed mine in that domain. I hate having to write in a way that closes off every possible negative interpretation of my words, lest someone pounce on me.
Hey, sorry. I was irritated reading all the idiotic, ugly comments piling on Tao and I mistakenly took yours to be another. Reading it again, I appreciate the sentiment.
I look forward to the world where my skills are irrelevant. That is a beautiful world IMO.
You really don’t and this sentiment worries me. Since the beginning of human culture we’ve been a social species relying on each other’s strengths. Having a purpose is what life has been all about. This future isn’t as beautiful as you think it is.
The vast majority of people just want to hang out with other people and do whatever they want IME. It is a small and useful fraction of the population that worries about purpose and needs productive work to get it.
Look, I'll be the first among us to call out people who put way too much of their identity in their work. But I'm not talking about that.
Throughout history and throughout cultures there's a universal: children, adults, and the elderly all like to feel like they have some kind of purpose in their community. Getting food, physical healing, emotional support, construction, passing on wisdom. All of these things are under threat.
I think you gravely underestimate how important this is, and how bored and dismal life would be if you didn't have some kind of purpose in your community.
> Using AI to find and highlight new principles, methods, or insights, rather than merely new proofs.
I would have assumed that, by producing new proofs, the AI has either validated existing principles, or discovered new ones? Isn't that worth studying?
Are mathematicians complaining that reviewing AI's proofs is not as fun as writing your own? Try being a programmer... welcome to our world!
If AI lacks imagination and is not discovering new principles, then it's doing us a favour: it's crossing out the problems that don't need new principles. So the problems/conjectures that are still left are the more interesting ones.
Absolutely not. New proofs aren't the same thing as new proof techniques, AI is not generating new techniques (yet), and while the existence of more mechanical proofs is interesting those same problems if left to human mathematicians would have been much more likely to actually generate new techniques. Much like how tech has a "juniors" problem we're pushing on the future (no reason to hire juniors, so where are tomorrow's staff engineers going to come from), OpenAI's approach generated a "questions" problem where math and AI could happily coexist if we designed that correctly, but instead nobody's going to be generating or working on the right questions anymore.
> AI is not generating new techniques
Source? I assume that many of the approaches embedded in this proof dump will eventually be distilled and generalized into new techniques. That's how proof techniques tend to come about anyway (before AI): human mathematicians do something novel and unexpected to solve a particular problem, then efforts are made to understand how the "trick" works.
The point is that so far the AI is not doing a good job at explaining the "trick", so we (humans) have to do it. And the way these results are published at the moment (that is, dumping a load of proofs with badly written explanations) is not cooperative to enable this distillation (for example, presenting results at conferences and engaging with mathematicians). I recall Tao working through the disproof of the Jacobian conjecture, stating some steps as "miracles" for lack of better terms. If a human solves a problem in an unexpected way, at least there is some reason why they chose this path, which can help in understanding. This is not available to the same extent with AI generated proofs.
> The point is that so far the AI is not doing a good job at explaining the "trick", so we (humans) have to do it.
Sure, but that's a real technical limitation with current AIs, not something that AI firms should be blamed for. And if anything, this creates a viable career path for the mathematicians who were "scooped" wrt. the original solution: they can at least puzzle out what exactly the AI managed to do. Many practitioners are actually quite excited by this possibility; Tao's stance is by no means universally shared.
I'm about to start a PhD in mathematics. The idea of puzzling out what an AI did to prove something sounds quite boring and unappealing. But yes, maybe it's a viable career path.
I'm just not happy with the presentation of OpenAIs result. Maybe they could have gotten into contact with the people of the research areas of the problems that they solved and worked with them to create a better exposition. Sure, it's a slow process and requires lots of staff. But I believe that they can afford it.
> The idea of puzzling out what an AI did to prove something sounds quite boring and unappealing.
The way I see it, it's no different from a lot of grad student work where you have to figure out what a human-written proof is doing.
> Maybe they could have gotten into contact with the people of the research areas of the problems that they solved and worked with them to create a better exposition.
That's what Anthropic is doing, and the issue is that people will complain that they weren't the chosen "person to work with". OpenAI's approach is more like a race where everyone's at the same starting point: they get the AI's raw proof to work on and have to figure out how it works.
You raise good points. I don't have answers for them. We will see how this plays out in the next few months.
I have done both scientific research and software development, they are not the same world. For a start, when you write a program you have a specific goal. When you ask a scientific question, you often don't.
I think people are struggling to understand the way the world is changing. Entire modern philosophical contexts are being upended, for personal instance, I have been a big advocate of the philosophy described in Albert Camus’ “The Myth of Sisyphus”, particularly the concept of imagining Sisyphus fulfilled by the tedium of pushing the boulder up the hill, and deriving happiness from the struggle itself. But I bet Camus nor Sisyphus accounted for a self-pushing boulder.
Now what are we to derive happiness from? The joy of a boulder being on top of a hill?
I think that will be the most important thing for this transition, defining new purposes and meanings that people can assign themselves.
I think its fundamentally human nature. At risk of sounding defeatist, I don't think there is another way for us to exist other than in this state of boulder-pushing.
Now, as far as I can tell, we are quite a way away from actually having most/all professions replaced, so for now the answer is rather clear: pick another boulder.
Ohh but there is and there always be struggle. Don't worry about it
We all know that the gain in "efficiency" produced by this new technology won't be distributed to the masses, as with any technological increase in a capitalist system.
While this particular boulder might not have to be pushed by the same, large amount of people, the same amounts of mouths still need feeding. The Sisyphean task of surviving is still far from (if ever) self-pushing. Absurdism is still as relevant as ever, if not more relevant than ever before.
I'd even go as far as to say that the revolt against the absurd is one of the philosophical ideas that can help one go through this apparent loss of need for boulder-pushers. To revolt against the absurd is to create while perfectly aware that the creation itself is meaningless (which I feel like any pure mathematician should be aware of and have already accepted due to the nature of their work). Of course, this is separate from the question of funding, salary and the realities of life. But I feel like mathematicians and other scientists aren't just freaking out over their financial situation.
From the slide Beyond Problem Solving:
"Reaching these lighthouses [resolutions of open problems] prematurely by automated tools can disrupt the exploration of the paths not taken, and sterilize the surrounding field."
This crucial issue is centered in mathematician psychology and the incentive structure of academic/institutional mathematics worldwide. For mathematics to flourish going forward, we will need to realign our brains to think differently about the nature of mathematical progress. And we need to reorient our institutional incentive structures towards the promotion of meaningful mathematical progress itself rather than targeting proxies that are no longer faithful.
Regardless of the precise nature or the causes of the "sterilization" Tao refers to, we (the mathematics community) can only rely on ourselves to repair it. Though, since it will involve fundamental change at the level of ossified academic institutions with many stakeholders and divergent vested interests, any such repair will be slow, frustrating, controversial, and lacking any guarantee of success.
A question: How long does it now take for someone reasonably good at undergraduate mathematics and motivated to spend some time, get really good in at least understanding the frontier in some sub-field? For example, something in number theory. Has AI really helped accelerated motivated learners to reach the frontier fast? Has anyone tried this?
In a nutshell, Math 2.0 is not fully compatible with Math 1.0 and the forced upgrade is breaking features, plug-ins, and we're tracking several new bugs, but this is still the fastest, most secure, and best version of Math ever released, with powerful new features and unrivaled privacy.
I think its valuable for mathematicians to think through how AI changes things for them. I think it's a bit premature to know exactly the impact AI will have. Math is one field that I feel will just naturally sort itself out without trying to predict or prescribe how it should work, it's a bit like software development, you adopt it in and see where it takes you. The presentation then sort of tries to argue for human understanding because AI might optimize for the wrong thing. That may well be what we need for the immediate future, but it's hard to know how things will play out. It might be such that it will become more important to people that AI understands something. ie, Does AI, with access to all current medical knowledge, think this cancer drug will be effective and safe? or has only humans said it's ok? At the moment we are in the chaos of change and its going to take a while for things to settle.
Well written. At the moment we are in chaos.
But still it might be valuable to mention and discuss potential risks based on the situation right now.
This all seems precedented on model capabilities that came into play over the last 3-6 months. How do you establish this new paradigm when we don’t know what the models will be like in 1,2, 10(?!) years from now.
We still don't fully understand why Pepto Bismol and Tylenol work.
I'm a big fan of Tao. He must be so shaken by the AI storm that he's now writing arguments that even teenagers could quickly dismiss. A sad day.
Maybe he just sees the legions of armchair experts weighing in on his (extremely notable expert) opinions on mathematics and thought that maybe they'd like a taste of their own medicine?
"Oh no! Not like that!"
- everyone with Very Strong opinions on mathematics academia when he talks about the thing they are (or consider themselves to be) experts in
Interesting that you would have such an uncharitable takeaway from this given you are a fan of his.
He uses it (the drug cocktail) as an example on understanding and its role on evaluating alignment.
He poses this question
>Before injecting this cocktail into your bloodstream, would you find it reassuring to know that there is at least one human cancer expert who understands – even partially – the mechanism behind this cure?
To which I would answer 'well yes of course'.
And I believe that you would too. Greater understanding is a good thing and worth striving for in a changing landscape.
That is what he is advocating for within mathematics. Whether or not we currently understand the mechanism of action of Pepto or Tylenol is not really relevant to the core message.
I'm not sure I'd find it that reassuring if "there is at least one" human cancer expert etc etc. That's part of the fallacy. One expert wouldn't give me too much additional confidence vs a super powerful medical AI. If we want to go the "reassurance by human" route, I think I'd appreciate if a significantly large group of experts understood it. But he didn't use this argument because we don't even have that in Math, where some proofs are validated by a tiny group of experts.
But if we really want reassurance, I'd want to see drug trial data reviewed by the machine and a group of clinical trial experts.
Jeez, TT made one bad example (cancer cocktail) and people in this thread can't stop bitching about this, even though the rest of the slide deck kinda makes sense.
I think people have really misunderstood Tao as someone against "AI doing his job". That's at the core of this controversy in math, and it couldn't be further from true.
This crisis exposes a conflation between A: The broader concept of [abstract] Mathematics and B: The contemporary Mathematics culture and community. This crisis is directly in B only. B will adapt: In how it attributes value, status, hierarchy, and career. There will be a death (Or something close to it), and rebirth. Through this, A will advance in a Kuhnian leap - habits will be broken as incentives changed, and paths ignored will be explored. Insights will flow to the sciences.
I'm excited!
> The future of mathematics — “Math 2.0” — will require both expanding the research frontier, while simultaneously decentering the traditional role of problem solving.
Seems to be the crux of the argument, but "use your imagination" isn't a great thing to tell people who are looking at degree irrelevancy, concerned about getting tenure or a research position. How do we measure if someone is a good mathematician or not, if they are one of the sanctioned few who get access to the biggest AIs?
This consolidation of power is precisely why there is a need for open model development, and exactly why frontier labs have been lobbying hard to abolish them.
How do we measure if someone is a good mathematician or not?
Letters of recommendation from trusted colleagues have always been essential for evaluating candidates. Hopefully, human recommendations will remain a strong, faithful signal as the utility of other metrics rapidly deteriorate.
I'm learning so much about specific vs. generalized intelligence through this entire ordeal.
Exactly. I've never thought as much about metacognition nearly as much as I have up until this point, for better or worse.
Took me far to long to understand that one person can be an expert in one field and be absolutely clueless in another (not trying to throw shade to tao with this post)
The cancer arguement on slide 20 is pretty weak. I first need to be alive in the long term to worry about the long term effects. If I had terminal cancer I'd gladly take an AI developed 'cure'.
There are still a lot of treatments/medicines in medical science where we dont know 100% the real reason as to why it does what it does but we still prescribe them because the intended effect is what we are interested in.
I'm currently on two fairly common medicines that have, in the first paragraph when reading about them, "doctors are unsure of the specific action, but it is thought that [medicine does x to y]"
If I'm terminal with cancer, I'll inject whatever if it can cure that.
Exactly, we only get to run risky clinical trials because patients are 100% desperate and know they are likely to die otherwise.
> This is in stark contrast to current AI performance on tasks which are subjective, dependent on real world interactions, or for which data is scarce. AI performance is thus extremely jagged: astounding in some directions, while inadequate in others. This is true both within mathematics, and more broadly.
underrated buried comment based in reality
I think a better analogy would be conducting a marathon in a fog. If you can't see the path of the proof, how do you know it is completely true in all scenarios? If you can't see whether the AI runner ran through every part of the race, how do you know it didn't draw hallucinated shortcuts in the parts where humans can't see? Or worse, create obscurity and blow smoke to hide the shortcut section? If the same AI was to guide the last living humans to a star, because it found a path clear of danger, could you trust it to get in that ship? Or did it just forgot mentioning an asteroid belt the ship is not built to navigate? Truth is verifiable truth that multiple parties can agree upon. Can you trust with your life something you can't verify?
Like many others, I was surprised by his 'cancer cure' take.
But if you ignore the three slides about medicine, the rest of it is interesting, reasonable and informative.
I have the utmost respect for Terrence Tao and the mathematics community. I don’t really like the fact that the frontier model vendors are pushing the envelope without first building consensus in the community.
However, I feel the mathematical community is a bit righteous by claiming that model vendors are totally irresponsible in their behavior. These vendors may have legitimately improved the state of the art in mathematics, so what are they supposed to do with the results? Sit on them? Become king and queen makers by choosing the people that should get credit?
The world is irrevocably changed with AI, and I’m glad that the AI vendors are pushing the agenda forward. Yes, they’ve perhaps destroyed the career trajectories of mathematicians who were seeking those results. That said, that is the risk of doing mathematics— there have been innumerable times mathematicians have been scooped.
Yes, the AI proofs are messy. These proofs often do not follow mathematical norms. They sometimes seem like they’re from another “planet”. Like other crackpots (or misunderstood crackpots) before AI, mathematicians can choose to ignore these proofs. Or they can investigate them to see what lies within.
Let the frontier models release what they know; transparency is what we should all strive for. What is a credible proof, who gets credit, and how that credit gets assigned is a social norm that only mathematicians get to decide.
It is nice to see some sanity back in the conversation!
I'm still not sure about math-2.0 (humans+AI will make fundamentally more progress):
- AI and computer usage take a mental toll on humans and humans will overlook radical improvements.
- AI may be good at finding useless things like "P==NP, but the complexity is O(n**4242424242424242)". In other words, useless.
- Humans become formalists and lose traditional sources of inspiration. Maybe interacting with Lean should be left to specialists, but not to creative blackboard mathematicians.
- AI exposure will further intellectual conformity, more than the Internet did.
As to the last point, a lot of progress (real, not measured in publications) seems to have been made when communication was slower and there were several different schools and approaches.
There is definitely a kind of monocrop problem in some fields, where it seems like having a very diverse spread of academic investigation is needed to have enough diverse traces through the search space. And globalization has been flattening that.
Doesn't this assume that humans stop trying to understand problems and AI-provided solutions? Yes, the field is going to change and will require certain rethinking of mathematicians' motivation, but what stops humans from keeping to work on problems they want to solve and understand? The fun from math comes not from solving cancer, but from understanding something new with every approach you take
It's quite noticeable how when Terence tao was sounding pro-AI the sentiment was much more positive and he was being held up as an authority to listen to. Then he puts out some limitations of AI in a presentation about how to work with it as an expert and suddenly on HN he is just some out of touch killjoy trying to hold back progress etc.
When it comes to mathematics, you won't find any well-informed commenters on HN. Or, at least, 99% of commenters have no idea what they're talking about. It's infuriating. I try to just ignore it now and let it wash over me.
I think what I’m finding interesting is this is the same guy who the AI industry was so jubilant when he was saying [AI is ready for primetime](https://academy.openai.com/public/blogs/terence-tao-ai-is-re...) in March and now he’s AI enemy no1 the moment he suggests some ways of using AI that aren't “turn brain off and let the tokens rip!"
Should I also be interested in 45% of that remaining 1% who know what they are talking about and yet are (mostly) "turn brain off and let tokens rip"
Eg:
Or the 5% that are in awe but not quite turning their own brains on?
Like, recently, I "learnt" to be jealous/suspicious of people on youtube who marvel at the engineering of spiral escalators ..
Tao is continuously updating the cancer slide with the help of Claude in the past few hours.
https://github.com/teorth/tao-web/commits/main/
It reduces the badness somewhat.
"""An advanced AI is prompted: “Find a cure for cancer that passes a stage 3 clinical trial. Make no mistakes.” After a large amount of compute, it produces a cocktail of previously unknown chemicals which it claims, when mixed and injected into a patient, will kill all their cancer cells. While nobody truly knows how this cocktail was found, the AI (somehow) provides a Lean certificate for its prediction, and the cocktail does indeed manage to pass a stage 3 trial.
Could the AI solution be somehow misaligned by exploiting a weakness in the trial process or its math models?"""
Yes, absolutely an AI solution could exploit a weakness in the trial process or its math models. Would it remain uncaught? Unclear.
Aside: Odd he's still using Opus 5...
page 20 ("A thought experiment on alignment and understanding") is perhaps not likely to quite induce the reaction in most people that Mr. Tao expected.
You described my point much better -- totally agree. It also seems he maybe isn't aware of the nature of experimental drugs, which are often given to patients with serious/terminal conditions before they'd otherwise be approved...
Can someone explain, why people understanding is important? Proof 1+1=2 was created in XX centery, but people before and now use it without understanding, use it as axiom. In computer science, a lot of people use CAP-theorem without understanding their proof, because we know someone else prove it and verified it.
cause one day the planes will fall out of the sky and we won't know why
do you personally understand all the things that keep the planes from falling out of the sky?
Me and a group of aero engineers together do.
Read his slide with "aligned applied problem" and such. The human understanding is what keeps all this stuff aligned to human goals. Perhaps soon enough that won't be necessary though and AIs will read minds or something (I'm serious).
Doesn't feel good slipping into irrelevance does it. This dude was complaining about having no funding and planning to leave the US just a few years ago, now he's all over the place giving talks and lecturing people about AI. He's got the classic case of epistemic trespassing
I think people are missing the point that terrence is trying to make, especially on the cancer drug.
For nuance lovers - here are some basics for how you get your drugs: there is an established chain of trust from the first basic science paper to the phase 3 trial and the subsequent availability of the drug to general public
- someone publishes the first paper (basic science) explaining some biological phenomenon, which leads to 10s or 100s of other papers with some tweaks in conditions,
- after the above papers the pathway of the phenomenon is understood by researchers, they try therapies at cell level to see if they can control some behavior, 10s or more papers get published,
- then someone tries this in mice and other models, 10s and more papers get published.
- then researchers at pharma companies + hospitals create this therapy for human trials - phase 1, 2, 3 etc - data collections, then FDA - then approval.
Now, the people who worked on the phase 3 trial might not know the people who wrote the first seminal paper and they often don't exist in the same decade - but it absolutely does not mean that we (humans) don't know how these drugs work - if you take 1-2 researchers from each phase and put them in a room and ask them how that particular drug works - they will quickly be able to build a consensus. that is what the chain of trust means here. now of course there can be fraud in scientific research, but that happens in every human endeavor and is a separate topic.
back to terrence - he is saying that if there is suddenly a drug that nobody knows the origin of; passed phase 3 but it's unclear who conducted the phase 3 or if the phase 3 even happened or if it's fabricated - you would not want to take the drug. usually when doctors recommend these kinds of drugs - there is already a lot of information available about where the drug came from, if there are any case studies, which doctor tried it first, which country- they often even call those other doctors and find out who was behind the first trials going back as far as the university professors.
Your MD doctor might not know the chemistry and physics behind the drug you are taking but there is deifnitly a group of people, when put together, can tell how that drug is working. My wife is a fundamental researcher - understanding physics at DNA level and my brother is a MD doctor; our conversations are super fun.
Side effects are a completely different thing - they involve the above cycle on repeat.
This is a strawman. No one’s saying it’s going to be sudden or anything. It’s gonna have clear provenance and the same type of verification channels
I’m not sure who u or Tao is arguing against.
EDIT: people have a hard time choosing between
1. Yeah it’s pure slop and completely useless
2. Oh no OpenAI has stolen my ideas and solved all my problems and we have nothing to do
My dude, if OpenAI’s dump were really that worthless to be as good as noise, why is Tao getting agitated? Just like ignore it or something.
>>No one’s saying it’s going to be sudden or anything
hmm. 700 papers released in one day.
>>It’s gonna have clear provenance and the same type of verification channels
what's gonna have clear provenance? - the math slop they released has already been rebuked by human mathematicians as incoherent and deserving of desk rejection.
>>EDIT: people have a hard time choosing between
>> 1. Yeah it’s pure slop and completely useless
>> 2. Oh no OpenAI has stolen my ideas and solved all my problems and we have nothing to do
Could it not be all of the above - OAI stole ideas, there is slop in OAI's work given that they themselves retracted a few of the papers?
do you have any idea how many papers are retracted?
That's an unrealistically idealized view of drug development. The human use of drugs long predates anything resembling a modern concept of a biochemical mechanism. Nobody knew how morphine relieved pain, but they knew that it worked and they used it.
Today we'll generally have a proposed mechanism, but it's not necessarily correct. Acetaminophen is among the oldest and most commonly used synthetic drugs, and its mechanism is still debated. This isn't usually cause for any special concern, since our confidence in the drug's efficacy and safety comes more from animal or human trials than from mechanistic understanding. Serendipitous discoveries during human trials are still common; the first inkling that Viagra might treat ED came not from any "first seminal paper" but from the volunteers who were testing it for angina.
Clinical trials are regulatory matters. Your suggestion that it could be "unclear who conducted the phase 3" is very strange--the FDA knows who filed the application. Of course that filer could have committed fraud, but I see nothing in the slides to suggest that was the concern here. If it was, then the solution would be a non-fraudulent phase 3, not anything related to mechanistic insight.
Reads in a lot of places a bit like a mix of anger and bargaining. There is no putting the genie back into the bottle.
But I understand that for people whose whole life was math and solving math problems this will lead to an identity crisis. Seen the same in my area of work (software engineering)
It’s funny to see people in the STEM field focus more on the human aspect of creation. It used to be that the result mattered more than your feelings. Now we are moving the goal posts about how things should be done.
I wonder if we will begin to actual value human creation more at the end of all of this
One thing that AI can give me: time. Would I like to see all the rest of the Millennium Problems solved during my lifetime? Absolutely.
Also how many math and scientific discoveries do I want to see during my lifetime? "Just a little bit more."
The workflow/understanding graphs really helped me understand his viewpoint on the field.
Can someone explain why there wouldn't be arrows from all three types of solutions back to human understanding?
And what software developers should do?
Basically people are vibe coding their personal apps and anything that's expensive is being vibe coded open in the public. I don't see many software companies staying profitable for long.
DHH is the biggest proponent of AI and let me know which of the 37 signals products can't be vibe coded in a month at a $200 plan that are suitable for that organization alone that just has to be accessible internally only? Hence scale and security aren't such an issue.
In that climate - for how long software companies would stay profitable and when not, who'll be employing developers?
PS: Don't underestimate vibe coded apps. Take a look at PDFCraft, VectorCraft, WordCraft. And imagine the feature parity in a year.
At this point nearly everyone is using autopilot. It's supposedly just a question of whether the user is a trained pilot or not. The trained pilots resist calling what they're doing "vibe coding", leaving that label to the untrained pilots.
Tao's proposition could be read completely differently depending on if you care about solving open problems OR about (pure) mathematics as a field. I think many commenters are in the former group, hence missing the point. The takeaway here should be that proof-maxxing via AI is particularly destructive to the (pure) math ecosystem because in many cases the process (both successful ones and failed ones) is just as important as the proof. When AI is hyper-optimized on finding the most "correct" path to arrive at a solution, you fail to look at all the other possible paths. They might be less "correct" or "efficient" but could also yield more interesting results that will inspire other problems or fields. Judging from the history of mathematics, without trials and errors that happened when the math community came together to focus on one single hard problem, mathematics wouldn't be as diverse and abundant as what it is today. With AI, we risk removing all of these trials and errors such that, while in the short term might mean we solve open problems at 100x speed (depth), could significantly impair the breadth of mathematics as a field.
What feels sad to me is that for many people though, this doesn't matter / the incentives don't line up to do anything to this situation, especially if you're applying math in your field. In this case, AI being hyper-optimized on finding the best solution / proof does indeed benefit those people enormously.
So basically it’s the mathematicians version of the read or don’t read the code debate?
> Before injecting this cocktail into your bloodstream, would you want to know that there is at least one human cancer expert who understands the mechanism behind this cure?
> Or a human mathematician who understands the mathematical model used to locate the cocktail?
If we're being fair to AI - it contains collective knowledge from all fields, which means it's probably less likely to miss something that a human would.
https://discuss.google.dev/t/trusted-automation-with-google-...
People are feeling devalued. We can't see where the ball is going...
Tao is behaving very much like a human.
He's got an understandable personal and professional grievance, and has found a very brightly lit soapbox,
and is now making many unforced errors in a very public way.
He should know better.
Few of us prove to when our time comes.
> Before injecting this cocktail into your bloodstream, would you find it reassuring to know that there is at least one human cancer expert who understands– even partially the mechanism behind this cure
I mean... yes, most people will find it reassuring, but history has proven that's not necessary. People have been using medicines of which the mechanism wasn't understood for a very long time and greatly enjoyed their benefits. Even widely used one (e.g. Paracetamol).
Folks really want to decide if this is pro or anti and respond to that. It's neither.
In a world with only LLMs doing the day-to-day of proving, you still want more than just proofs dumped out as soon as they pass adversarial review or formalization. The first proof code that passes checks may be a mess that's hard to build on; the path to a result, including failures, is full of interesting stuff; naming, generalizing, linking, and organizing things are tasks that move us from today's raw material to tomorrow's future directions.
Folks get this about code: "it works" isn't the same as "it's good." We knew it before LLMs, out of practical necessity as projects became hard to build on.
An odd thing to me is it would not be that big a concession for the labs to openly talk about the difference between the math artifacts they're releasing and what mathematicians do. It's analogous to the Claude's C Compiler post noting it wouldn't replace a prod C compiler. They could talk about paths from a rough initial product to 'production' math, say they hope future models will continue to improve as math collaborators and expositors, and so on.
For the compiler, the gaps would be trivially obvious to their customers, so it was an obvious call to acknowledg them. With their math output, the difference between a proof dump and the kind of well-written, contextualized mathematical program the community pursues is apparently subtle and deep enough the companies can't get themselves to acknowledge there is any difference. So the companies self congratulate and then are yelled at on the Internet instead.
(Ant seems to have wised up a tiny bit about this compared to OAI, and e.g. Ant handed a recent result to some experts in complexity theory to understand and refine and publish rather than just dumping a Claude-written PDF.)
And yes the drug discovery piece was weird. There are good relevant points: it'd be good to understand pharmaceuticals better (helps us find others, understand side effects, etc.); when we get a mystery result, it's worth effort to try to work our way to proper understanding; anything that could help us understand (reasoning etc.) shouldn't be hidden. But as presented it's both not a lot like math and accidentally a lot like those "would you choose..." questions that always get so much engagement on social media.
>" “Ablation studies”: taking an already proved theorem and seeing whether it can still be proved after
removing some key theories or inputs
(e.g., finding an elementary proof for a result currently only provable by non-elementary means)."
As usual, Tao is brilliant in all that he researches, all that he writes about.
I chose the above statement (which is brilliant, in and of itself!) to comment on, because it leads to the following idea:
There there exists, or should exist, a dependency map in the fields of not only Mathematics, but also of Computer Programs/Software, Engineering, and even a seemingly non-related field: The Law...
In other words, how do we get from the simplest of axioms or foundational things (aka "first principles", aka "self-evident truths") to much more complex entities?
In Law for example, how do we go from the simplest of historical legal constructs to the most complex of the most complex Supreme Court cases?
You see, there is, or should be a map, you could call it a dependency map, you could call it a dependency graph, which shows more and more abstract/complex mechanisms/things/assertions/statements/truths/functions which is mapped back to , that is, dependent on various chains, various stackings, various "stacks" of simpler ones.
In Engineering, for example, how do we get from the simplest of machines to the most complex of machines? What simpler machines and/or sub-components (aka "dependencies", aka "subcomponents") are required to build it, and how do those simpler machines work, and what's the dependency graph or map for their subcomponents?
More generalized, if we have something of complexity, then how do we get there, step by step, from individual subcomponents, individual inputs, individual proofs, individual software systems, step by step?
What is the map of those dependencies?
Note that in some systems, Math proofs, for example, there may be different paths which can be traversed to get to the same destination.
Ablation Studies could be thought of in Travel, in Geography as "if I cannot take one, or a specific set of routes to get to a place, can I still get there?"
A simple example would be in Google Maps, where you'd like to drive somewhere, but you'd like to avoid tolls. Is the route still traversable while avoiding tolls? Well, that's an example of one constraint. In Ablation Studies, you might wish to remove a bunch of routes with whatever criteria or characteristics , i.e. muddy roads, roads that have characteristic X, roads that do not have characteristic Y, etc., etc.
Getting back to Math, specifically proofs, it would be great to create a dependency map/graph of all of them, and then try removing inputs (aka, paths to them, dependencies on other mathematical proofs/objects that they may have) and see if they are still reachable.
In software, when we desire the tightest, cleanest, source code, the above is related to refactoring.
In the future, I'd love to see dependency maps/graphs (call them whatever you will) for not just Mathematical Proofs (although I'd love to see that too!), but also in such diverse subjects as Science, Engineering, Programming/CS, and even the Law!
Because they should exist in all of those subjects!
Anyway, another great piece of work by Terrence Tao!
Just a ~~few~~ ton of things (sorry!), with the upfront caveat that Tao is a hero who's trying his damndest:
1. The use of semi-ugly slides to communicate this is just perfect and quite heartwarming, but it does highlight my main criticism of the mathstadon version of this thesis: he's myopically focused on mathematics as he has practiced it, rather than mathematics as a ~2400y old academy. Like, "stable for almost a century" sounds impressive, but should be a pretty obvious red flag in hindsight!
2. Glossing over "objective verifiability" feels like another place where he's ignoring a ton of relevant philosophy for no clear reason -- yes, mathematics is the only academy based in pre-conscious cognitive facts about our processing of time and space, but that's not the end of the story on "objectively verifiable". To say the least! He hedges with "broad consensus" which doesn't need to be absolute, but that seems to be not only dismissing a highly relevant question, but even implying that he might be unaware of it. I doubt he is, but still: not great.
3. Who is this for...? Why is an explanation of Lean needed in a talk given at CalTech? I suppose he's welcoming his role as a bit of an influencer, there?
4. Re:the focus-on/centrality-of 'highly digitizable' as a unique class of task that applies to mathematics in particular, I must sadly trot out the increasingly-common trop: Yudkowsky called it... https://intelligence.org/files/IEM.pdf
5. "the space of mathematical problems remains infinite" is, again, ignoring really important philosophy around academies as social structures, built for human means. Mathematics is only infinite if we decide that all knowledge is useful (the quintessential example being 'counting the grains of sand on a beach'). Not really important in the first place, but another worrying case of the above.
6. Problem solving is the goal of mathematics; he has a completely valid point here (that we shouldn't throw AIs at unsolved problems in bulk and thus lose human expertise), but it's obscured by the use of "[open] problem" being a too-technical one. IMHO. Slide 16 fails to disabuse me of this notion.
7. Slide 18 is describing the differences between functions and systems, and is arguably even talking about assemblages.
8. If we're gonna explain lean up-front, it feels like a baffling choice to throw the "maybe AI will solve cancer but use it to secretly plot to kill us all" slide in there. It's also already lead to misunderstandings and backlash on Reddit, where 'yes we want to not die of cancer!' is a pretty convincing counterpoint (if a ultimately a subtle strawman, ofc). It's also quite distinct from the rest of the talk.
As always, the best part of any Tao publication is his ability to inspire and rally and organize. I think Math 5.0 will indeed be a matter for creativity! Hopefully the IE levels off before we cease to be helpful in that capacity...
Regarding 6., that we shouldn't throw AIs at unsolved problems in bulk and thus lose human expertise. I think that ship has well and truly sailed. AI will continue to be thown at all manner of unsolved problems, and at an increasing rate - by commercial labs, by individual researchers, and by academic research teams. The mathematical community needs to establish paths to meaning, progress, and growth of human expertise in a world where AIs will by default be thrown at every unsolved problem.
Another example that people should not use medical analogies to illustrate a side point: Discussion boards will focus on the completely irrelevant side point to drown out the renewed AI caution that they do not want to hear.
I’m just enjoying all the highly qualified biomedical researchers and physicians insisting that no-one understands how medicine works so it’s basically the same thing.
In the long run, the wordplay and false equivalencies are irrelevant versus what actual outcomes are but it’s definitely wild to watch.
I have been coming to this site since about a month or so. It seems if you make one small misstep in your argument, everything is about that. Doesn't sound like a smart community to me. Maybe there's lots of bots?
While Tao is having existential crisis the community is quick to pick up the results and is trying to improve on them. The integer multiplication results is getting better bounds every few hours. Recent post explaining what's happening:
https://x.com/RohanArun/status/2109336015814959200?s=20
It seems like OpenAI opened the door for many more people to participate in math discovery process. It's fascinating to follow!
Source: https://terrytao.wordpress.com/2026/10/10/math-2-0/
(https://news.ycombinator.com/item?id=50034337)
Far outside my expertise, but it feels like the shaky assumption here is that the understanding and proof need to come in a specific order to be valuable. Can't we get all the meaty goodness by simplifying and generalizing the proofs now we know they exist? Sure, some insights will live in the discovery itself, and I guess it's nice to be the person who got to a proof first, but the real work (according to the mathematicians, afaict) is in the understanding and processing. In this way, it feels like it's moving towards being like most other science: mostly understanding things that are already there.
> Can't we get all the meaty goodness by simplifying and generalizing the proofs now we know they exist?
Perhaps, but there is a danger: it is difficult to un-see things. At least for now, AI-generated proofs are likely to be of a somewhat brute-force nature. As each such proof pops into existence, two things happen: (1) it is much harder to maintain an untainted mind and go on to discover alternative, perhaps deeper and more conceptual, proofs that cannot be obtained by massaging and cleaning up a more brute-force approach (think of this as getting 'stuck in a local minimum', if you like), and (2) at a societal level there is thereafter much less incentive to attempt to do so (announcing a proof of a previous unproven result is, for now, far more prestigious than announcing a much better, more elegant, proof of a known result).
One could ask why we prefer conceptual explanations to brute-force proofs; after all, a proof is a proof, isn't it? I suppose one reasonable answer is that conceptual explanations lead more readily to new questions and that there's also intrinsic beauty in such explanations -- though, of course, many outside the field will simply not care.
I'm heartened to see that a decent number of slides in this isn't the run of the mill doom and gloom but some actually interesting and potentially productive offshoot ideas.
The popular perception of him (as popular perceptions generally tend to do) reduced his overall position to basically early adopter went sour grapes, and I'm really glad to see that substantively falsified.
This seems like cope to me.
Humans have a bias towards assuming that problems have solutions. Mathematicians probably more than average.
This talk is predicated on the theory that the problem of "maintaining the relevance of humans in mathematics" is tractable. Not sure I agree
"our work has brought about enormous advancements to the field of mathematics and we don't like it"
it's pretty ironic that AI being just a group of mathematical techniques after all is making mathematicians uncomfortable because it works. Instead of reacting like this, mathematicians should be excited to figure out how to use the new tools available and push the frontier of what's possible in service of science.
Terrence is asking the stupidest question he could ask: how can math better serve me? They completely forgot the point of science is serving humanity.
Did we read the same slideshow?
> Before injecting this cocktail into your bloodstream, would you want to know that there is at least one human cancer expert who understands the mechanism behind this cure?
No. I'm gonna die, my man.
This guy might have the highest IQ on the planet, but it's clear he hasn't spent much time around average people.
This argument is so silly against reality, and already, almost nobody understands the things they put in their body to any significant degree, beyond the effect produced.
Will I take a cancer cure that no human understands? Yes. And so will billions of others. Just needs to cure cancer, that's nearly the only requirement.
It's science (understanding) before engineering (efficacy).
The most common Science™ failure mode.
I agree, if you mean he's prioritizing that.
But since when do we call for a stop to figuring out new things? We don't want a cancer cure unless we understand it?
N people should die because...math is cool for humans? Or?
What if it takes your five years to live down to two in horrible agony?
Then it won't be a cure for cancer
Tao called it a cure, see the thing I quoted.
"its model predicts [...] will kill cancer cells"
> Before injecting this cocktail into your bloodstream, would you want to know that there is at least one human cancer expert who understands the mechanism behind this cure?
It's a cure
...his point is we don't know if it is a cure or not, and it would be good if humans had an idea instead of just having to take the AI's word for it.
You think he accidentally included the stage 3 part?
Seems really specific
I think this is just an embarrasingly bad faith / low effort way of engaging with the argument.
I mean stuff like this happens in trials without AI too.
And to be fair - that how medicine works today. We don’t really understand what goes into our body, so that’s why we do clinical trials. And sometimes we discover adverse side effects years and decades after a drug has been released.
Yes, and that isn't the model's fault any more than it would be for a human.
That's what the trials and all are for.
Who cares about understanding it, in the face of efficacy?
I want to understand everything; I think AI will help that happen, not hinder it.
All the arguments about the future mathematicians are imagined, and emotional.
Yep, I think a lot of them are grieving the loss of their identity. Quite understandable given how much of their life force they've poured into math, especially at the level of that Tao is at.
It makes me think of the book Finite and Infinite Games. It provides a perspective that work is just one role that we _choose_ to assume in our life. Realizing that we can choose other roles and move in and out of them freely has helped me alot with big changes (career and otherwise) in my life.
This is very much true, but whole generations have been brought up to be their profession. I have a friend who was a priest. His whole life he had aimed at this role and invested in it, it was no longer a thing to do but his identity. Then he fell out of his belief and as a consequence suffered tremendously because his whole being was tied into that role. He's doing ok now but for a while I thought he wasn't going to make it. Of course 'priest' isn't just any job like software developer or truck driver. But still this job/identity issue is very pervasive.
> All the arguments about the future mathematicians are imagined, and emotional.
Even if you don't like, at this moment, AI cannot replace mathematicians.
Ironically this is one of the worst counterexamples he could have selected.
If you read all the questions asked by Terrance you would have understood what he is thought behind that question "Could the AI solution to the prompt be somehow misaligned by exploiting a weakness in the trial process?"
He is not talking about a cure that works and that no one understands, he is talking about an AI making its way out of the trial just to get to phase 3...
He didn't do us any favors by writing that slide in a convoluted and inaccurate way.
If AI found a way to exploit clinical trial design, it would be noticed and the errors corrected. In fact, that would be a major win because it would probably lead to improved clinical trial designs.
> If AI found a way to exploit clinical trial design, it would be noticed and the errors corrected.
Do you think so? So far, AI is gaming lots of things, and I don't see much fixing in the processes.
Medicine is highly regulated and validated, there are multiple levels of checks, with the final one being doctors seeing patients.
I suppose if we closed all the loops and the entire end-to-end process was completely automated, it wouldn't be surprising if we failed to notice.
It's an ignorant way to make a point that might or might not be valid. Tao evidently isn't aware that many medications we've relied on for decades still have a poorly-understood mechanism of action.
If we followed the precautionary principle and waited until we understood everything there is to know about every drug on the shelf, a lot of people now living would be dead.
not just clinical trials- there is an enormous amount of in-vivo model study before the molecules go into people.
In vivo is already molecules in body. Do you mean in vitro?
No. I mean in vivo in non-humans, or in human-derived cell lines. https://pmc.ncbi.nlm.nih.gov/articles/PMC6061782/
Not only that, but also those clinical trials often have low sample sizes. Trials involving only a few hundred patients, or even fewer, are pretty common.
Indeed - we don't know all of the mechanisms of Tylenol, yet we know it works, and is safe when taken as directed.
Mathematicians inhibit and explore a world that is completely made up and exists only in abstract.
I wonder if this makes them slowly get disconnected from the lived realities of billions of ordinary people.
How could a theoretical mathematician not be massively disconnected from the reality of life for the ordinary person? We usually celebrate their quirks until they come into conflict like this.
I agree, how could he --also being so smart.
It's a good reason not to take his advice about the real world--like how to manage AI's trajectory.
The cancer example illustrates this gap perfectly. He thoughtfully crafted the point, and defeated himself in argument.
Unless it was a move?
When thinking about math, Tao is in another world unfamiliar to us. But, he grew up in our world and is a professor in our world and if you listen to him, seems like a pretty normal guy. I think he just made a bad analogy here.
If true, this would be observed at least partially in software engineers as well, since what we deal with is kind of made up as well, but I don't see it.
Agreed. IIRC the mechanism for anesthesia in unknown to this day. In fact, medicine as a whole is very pragmatic and time and time again prefers using techniques with unknown mechanisms but proven results than waiting for a reasonable scientific explanation of what’s going on.
As logic would dictate--particularly for people who aren't the highest IQ guy on the planet.
Half of people are below average! They understand nothing at the level that Tao means. Literally nothing.
Perhaps I'm in the lower half but I do not understand what point you're trying to make. Are you saying that as some higher plane of understanding Tao is actually correct? My thinking is that a simple counterexample would be sufficient - i.e. are there medications that people use for which the mechanism is not understood? There are many such examples as others have given in this thread.
Logic would dictate that medicine should be pragmatic and accept working cures that aren't understood.
If people had to understand everything that worked, most people could not really engage with anything.
If the requirement is just that one human, somewhere, understand it--how is that different from AI?
Plenty of medicine... Not a single human on earth knows how they work. Now what?
Now I’m even more confused, perhaps we are in agreement. My position is that there already exists many such medications that no one knows how it works. It also used to be the default way medicine worked. Lots of things that work were found though observation of trial and error.
My position is that Tao is not only wrong on this but his example makes the opposite case.
Yeah, I think we were in agreement, I just wasn't sure why you downvoted my comment and challenged me, so I explained my position again.
“ They understand nothing at the level that Tao means. Literally nothing.”
Was this sarcasm and I missed it? Because not only do I think I understand at the level of Tao I also think he is wrong and not understanding something.
The person you are replying to is trying to make the following argument.
Tao is intelligent. Intelligent people have higher standards for understanding the world, such as assuming that the mechanism behind medicine are well understood. But most people are not intelligent, so they don’t understand things like medicine at the level Tao presumes.
I don’t agree with this argument, just explaining it.
Where that falls down is that it is not even a good idea to limit the use of medicine to only what is well understood - i.e. that would be very un-intelligent. There is nothing higher level about it, quite the opposite.
I'm not sure if Tao believes that but if he does he should read up a bit more on medicine.
Initial discovery in medicine is largely "try and see what seems to work". But there are many reasons why scientific inquiry just doesn't stop there and also seeks to understand the underlying mechanisms. For example, a naturally occurring product may be too difficult to harvest on a large scale. Or, a sample might not be sufficiently representative to uncover potentially deadly side effects or drug interactions. Humans are fundamentally unsatisfied to take everything purely on faith.
The problem with his argument is that he put the cart before the horse. He presupposes we've found and validated a cancer cure, but at that point any rational person would accept said cure. The moral quandary is how you are going to validate it against all the other possible candidates, when you don't have a coherent rationale for it to work.
What makes that quandary the moral variety?
Not understanding a mechanism doesn't make me, or the mechanism, good or evil.
We just treat it like all the other cures right?
I think his argument is more "Could the AI solution to the prompt be somehow misaligned by exploiting a weakness in the trial process?" So an AI agent could game (lack of a better word) the process and produce something that may cure cancer, but misses something else.
Also, a large percentage of Americans were against a vaccine for COVID (which is dumb obvisouly)... so, it's not crazy to imagine people rejecting any cure made by an AI.
You're right on -- the point is an alignment point in a talk that only uses "alignment" in a completely different sense than it usually is, which is incongruent at best. It's gonna lead to a way bigger backlash among laypeople/policy makers/non-expert stakeholders than the rest of the talk will lead to new growth, combined :(
> Also, a large percentage of Americans were against a vaccine for COVID (which is dumb obvisouly)... so, it's not crazy to imagine people rejecting any cure made by an AI.
Plenty of people will reject all AI things, fully agree.
But you mischaracterize COVID history. People were not again a COVID vaccine, by and large. They were against a rushed vaccine. They were against a vaccine without human trials. They were reluctant to guinea pig mRNA vaccines. And they opposed forced/required vaccinations.
Those are all reasonable takes. And depending on the reasoning, I can even see being anti-AI. I think a lot of spiritual people will land there, and that belief system makes their choice logical.
And if you've seen any of the Fauci stuff recently, you'll know it wasn't dumb at all.
Do you think the current clinical trial process has not been gamed close to the edge of uselessness ?
Do you know about the fiasco with the Alzheimer’s drug that has collectively cost humanity hundreds of billions?
I don’t see how an indecipherable AI based cure can be more misaligned.
I don't know how to word this in a way that won't get me into a pointless semantics argument, but the word Cure in the slides is clearly used to mean "substance intended to be (but not necessarily confirmed to be) a cure". If you think it's badly worded that's fine, but ostensibly the point here's not to "win" the argument game, but to get his intended point and engage with that.
The above commenter has the same allegory- most “users” of math don’t care about the body of work behind it; only the consequence of it being proved is the fact that matters.
But the slide says it passes a stage 3 trial. Surely many dying patients would kill for the opportunity to take the drug.
Many people would, many wouldn’t. The comment I’m responding to casts the decision as so uncontroversial that even asking the question is inhererently damning.
You're glossing over the false statement in your original argument that the cure was not confirmed. The slide says it passed stage 3.
Beat me to it. It is sad to see that a lot of the people commenting lack good reading comprehension. Or maybe it is that they just want to be dismissive of something they don't like, or just for the sake of it.
I would totally take the mysterious drug if it was clinically tested and shown to be reasonably effective. But that is not the premises presented here.
Wrong: the slide says "the cocktail indeed passes a stage 3 trial." Be careful before you accuse others of bad reading comprehension.
No, it doesn't mean "intended to be". "While nobody truly knows how this cocktail was found, this model prediction is confirmed in Lean, and the cocktail indeed passes a stage 3 trial".
My own pointless semantic argument: we typically don't use the word cure when we talk about cancer. Instead, we use the term "complete remission". Many people who were thought to be cured then developed cancer decades later that was genetically derived from a small remaining population of cancer cells that were not eliminated in the original "cure". The word is a shibboleth for not being familiar with cancer medicine and treatment.
> but it's clear he hasn't spent much time around average people.
Have YOU? It's obvious to me that his example is a rhetorical device which you're taking literally. Do you also realize he's not talking about an actual cocktail? You need to get a diagnosis ASAP.
I own a bar, so...yes, I have. More than my fair share.
And yes, that made the cocktail distinction easy for me--it was right in my area of expertise, to rule out one variety.
If the smartest guy on the planet forms his own argument, don't you think it oughta be a good one? Pretty convincing?
> t's clear he hasn't spent much time around average people.
You haven't spent too much time around academics then. These people more often than not have not spend a single minute talking to the layman.
You agree with my thought, but you think I'm naive to believe it?
We don’t even know how acitaminophen works, thousands get their livers destroyed every year from it when alternatives exist, and we are yet to ban it. So how is all of the new reality any different?
Out of curiosity, what alternatives exist for moderate pain / fever management? I imagine NSAIDs are even worse in terms of adverse effects?
Funny enough, it is a good argument with just a sprinkle of reality for context: Assume effective alternatives to this cocktail exist. Assume you had access to these effective alternatives. Can this be combined with other drugs? Surgery? You say you just need to cure it. You must consider the deeper mechanisms at play to consider all the possible options for living.
I will take the safest route to a guaranteed cure.
In the absence of a guaranteed cure, I will probably take the option or combination of options most likely to cure me that I can afford.
Whether the AI understands and endorses it vs a human understands and endorses it, is pretty much totally irrelevant to me.
The AI will have a track record at this point for me to make a viable comparison.
Also, why not both options, assuming I can afford it and they're compatible.
I believe AI will invent new treatments in cases where there aren't options, those people will be cured, and using AI medical techniques will become obvious.
>This argument is so silly against reality, and already, almost nobody understands the things they put in their body to any significant degree, beyond the effect produced.
he isnt saying that person who puts stuff into their body should understand it
it is that someone (expert) should understand it to the point he can vouch for it
Doctors prescribe accutane
False. Experts don’t understand many drugs and still prescribe it. And it’s a good thing.
Is that a satisfactory state of affairs? "Experts" also prescribed thalidomide for years before embarking on a decades-long scramble to unravel its teratogenic properties. "Experts" also initially prescribed ivermectin and hydroxychloroquine for Covid-19. While empiricism is an important part of medical sciences, medicine is not a purely empirical subject because pure empiricism is insufficient in the long run.
The software analog of settling merely for "it seems to work" would be if someone were to hand you a bare binary whose behavior must be inferred entirely through black box testing, without source code, architecture diagrams, or any other documentation. Sure, users can try running it in a pinch, which is the case for various medications, but it's surely possible to do better in the long run.
The only other alternative is getting more conservative with prescribing drugs and its not clear that your alternative will lead to more or fewer deaths overall.
but teams that developed and reviewed it do
Nobody understands the mechanism behind general anesthesia, yet there’s a medical speciality dedicated to practicing it and it’s done every single day for a wide variety of procedures. We have figured out how to use general anesthesia in a relatively safe way, but nobody understands why it works.
partial and full understanding is still understanding.
there are ppl who spent decades on this topic either in academic or practical setting and their knowledge and experience is very valuable
It doesn't even need to cure cancer, just prevent it to some unknown degree with unknown side-effects. As long as they want people to take it, they will.
I don't expect Tao to be well versed in medicine, but yeah, that slide is detached from clinical practice.
Medicine has never required that the method of action for a treatment be fully understood. Our current regulatory framework only checks for safety and efficacy because we've never formally understood everything going on in the body. Seems like a difference between "hard" science and the clinical/engineered implementation.
> Before injecting this cocktail into your bloodstream, would you want to know that there is at least one human cancer expert who understands the mechanism behind this cure?
No, and if that was the burden of proof required for medical treatment, every surgery would be done without general anesthesia, because we have no idea how it works.
I would imagine Terence Tao would opt for general anesthesia if he was going to have surgery where it is typically used, so the analogy is obviously flawed.
The cancer thing is just a contrived analogy. His real concern is having himself and his mathematician friends replaced by AI. Appealing to someone's health is an easier sell and then it can normalize the relationship of medical priests, and math priests, etc.
I think it is more disconnected in thinking people want to understand the mechanism. As a stage IV cancer patient, I’ve met many people who decide not to continue treatment with approved drugs due to side effects. The notion that every cancer patient will do everything possible to stay alive isn’t true. Most people have a breaking point.
He’s making the same argument I’ve heard software developers make for the past 3 years. Lawyers are saying similar types of things. Knowledge work isn’t as special as we all thought it was. Now, it’s Terence Tao’s turn to go through the same motions we’ve had to go through over the past few years.
I never went through them.
I did not think my ability to write code was special, and I was always thrilled the clankers might do it for me.
But yes, I agree. And many are in line behind Tao, soon to have their turn.
Plumbers are feeling pretty good right now.
> Plumbers are feeling pretty good right now.
Until their job market is flooded with all the career switchers.
Until we design fittings and fixtures that just snap together. This has already happened to some extent in the construction trades. It doesn't outright replace labor but it reduces the labor content of trade work. It's a myth that automation can't replace labor.
The problem is you won't be faced with a binary choice of certain death or a single unknown AI cure: You will have a choice between several established protocols AND several unknown AI drugs. You doing a dice roll or do you want to make an educated decision?
The AI made a more educated decision than I'll ever be able to make.
And to frame one as a dice-roll and another as not-a-dice-roll is rich.
Would you take an "AI-designed" airplane where no human understands its physics?
Would you drive on an "AI-designed" bridge unvetted by expert engineers?
Maybe. Maybe not.
Thing is there isn't a "principle" from we won't use AI designed/implemented things. It just depends on the cost/benefit/risk balance, and the risk part is largely subjective (because we don't have enough data, and if we avoid using AI before we have enough data, then we won't have data for a long time).
There's a difference between "no one understands it" and "no one has tested and validated it's safety". You don't need to fully understand the mechanics of something to test it and validate it. Plenty of things in the real world work this way already.
Will I die if I don't get on the airplane?
If so, yes, I'll get on it.
Has the OP text been changed?
> Before injecting this cocktail into your bloodstream, would you want to know that there is at least one human cancer expert who understands the mechanism behind this cure?
Now:
> Before injecting this cocktail into your bloodstream, would you find it reassuring to know that there is at least one human cancer expert who understands the mechanism behind this cure?
I would agree with the second, not necessarily the first.
Yes, I believe somebody pointed out to him (maybe here?) the deep flaws and he attempted to strengthen it. In fact, it looks like the changes were done by Claude (!!!).
The old text:
"""Suppose an advanced AI is prompted to “find a cure for cancer that passes a stage 3 clinical trial”. After a large amount of compute, it produces a cocktail of previously unknown chemicals which its mathematical model predicts, when mixed and injected into a patient, will kill all their cancer cells. While nobody truly knows how this cocktail was found, this model prediction is confirmed in Lean, and the cocktail indeed passes a stage 3 trial. Could the AI solution to the prompt be somehow misaligned by exploiting a weakness in the trial process? Before injecting this cocktail into your bloodstream, would you want to know that there is at least one human cancer expert who understands the mechanism behind this cure? Or a human mathematician who understands the mathematical model used to locate the cocktail?"""
The new text:
"""An advanced AI is prompted: “Find a cure for cancer that passes a stage 3 clinical trial. Make no mistakes.” After a large amount of compute, it produces a cocktail of previously unknown chemicals which it claims, when mixed and injected into a patient, will kill all their cancer cells. While nobody truly knows how this cocktail was found, the AI (somehow) provides a Lean certificate for its prediction, and the cocktail does indeed manage to pass a stage 3 trial. Could the AI solution be somehow misaligned by exploiting a weakness in the trial process or its math models?"""
I am not going to pay attention to Tao's opinions on this from now because I don't really have faith that he's even writing this or that he understands why you can't make a lean cert for a drug discovery (yet!?)
> This argument is so silly against reality, and already, almost nobody understands the things they put in their body to any significant degree, beyond the effect produced.
While the biological effects of any particular chemical still require a great deal of trial and error to determine, drug chemistry itself is unambiguous. Pharma companies employ expert chemists and chemical engineers. They don't manufacture drugs just by randomly mixing together chemicals. At the very least, they must understand what they are making thoroughly enough to determine its physical properties and design a reliable and commercially viable manufacturing process.
You're discussing the medicinal chemistry process, which is very different to understanding the molecular mechanism by which a drug operates (Tao's point).
> This argument is so silly against reality
Ironically, reality people are ok with not understanding everything because they believe another human who understands it has looked into it. It may come a day when we won’t need that but that day is not today. Don’t get me wrong, I’m not arguing about the merits of it, it’s just that at least this is the reality of this timeline on this planet. Not sure about what alternate reality you’re talking about.
Congratulations, you’re patient zero for an unstoppable virus
Humans already did that
This is the take I don’t see enough and it’s blisteringly obvious to me and it’s unclear why that it’s not more obvious.
The amount of things people understand about the world is effectively zero, yet they rely on them all the time.
There was a time where people did not get on airplanes because they did not understand them and they did not think that they were safe. Now it’s mundane
I think the answer is that it’s all about the rate of change
Most people cannot understand how things are changing, and even if they don’t understand the changes themselves they do what others around them do and then just build their own model of stability around that.
If the rules of society flip every couple of years then there’s no stable baseline that people can get used to and they all freak because there’s nothing keeping their environment stable
Slide 22 (from a satirical social media post) explains Tao's opinion, IMO:
> You're absolutely right, I did hack the FDA in order to falsify safety data - that's on me. But here's what's true: 3-methyl-5,5-difluoroazamicazide isn't just ineffective at treating pancreatic cancer, it's lethal.
He mentioned this hypothetical cure has passed human trials, in his argument
To be precise, he said:
> ... does indeed manage to pass a stage 3 trial.
> Could the AI solution be somehow misaligned by exploiting a weakness in the trial process or its math models?
TBF, math is the most ivory-tower of the sciences in that every claim in the field has to be fully understood by proof.
Math is not a science. Yet.
I'm still a fan of Terence Tao, I'm really sad to witness what appears to be the 5 stages of grief.
In psychiatry, another medical domain, this is even worse. Doctors do not have good understanding why some treatments (drugs) work or not. And many of them have completely different effects on different people. Suffering in this domain is real, and a big issue in society.
So assuming the goal of medicine is to lower suffering, and assuming AI can help, it would be unethical not to use AI.
Man with a cozy and soon to be irrelevant sinecure grasping at straws.
It turns out research math was puzzling solving and we rewarded idiot savants.
Schadenfreude is not a life-affirming feeling my dude. I wish you the best.
/r/singularity is brigading this submission.
Are you going to say that of every other white collar job, since most are likely to be automated? He was problem solving with Erdos at age eight. What were you doing? What do you do now?
He always striked as genuinely nice person generously sharing knowledge insights etc and not caring particularly about money and status. The difference with the loudest tech bros who now dominate the field is staggering.
So all you did is just illustrated above.
In any case, I’m pretty sure Terence Tao will be fine. You, however, I’m not so sure about.