@AcerFur (former openai math employee on twitter) claims there are issues with this proof: https://x.com/AcerFur/status/2123853069599547786
Maybe repairable, but either way, alpoge/anthropic certainly has the claim to the first proof
@AcerFur (former openai math employee on twitter) claims there are issues with this proof: https://x.com/AcerFur/status/2123853069599547786
Maybe repairable, but either way, alpoge/anthropic certainly has the claim to the first proof
What value is there in just dumping something into an LLM and saying it produced errors?
edit: to all the people responding that LLMs are good at math, I'm going to paste a reply I wrote downthread...
I'm not arguing that LLMs aren't useful tools for mathematicians, I'm questioning the quality of discourse that amounts to "I pasted this into the LLM and it said there are errors"
Have you tried doing this recently? For my work on software at least, the latest LLMs have a very low false positive rate for bugs. They almost never hallucinate issues. My understanding is the same applies to math.
I'm not arguing that LLMs aren't useful tools for mathematicians, I'm questioning the quality of discourse that amounts to "I pasted this into the LLM and it said there are errors".
There's value in knowing whether a claim is true or false. Plugging things into an LLM doesn't necessarily give 100% certainty or tell us why something is true or false in a human-comprehensible way, but it can move our credences and suggest directions for investigation.
If you're not even gonna share what the supposed errors were, the value is pretty much infinitesimal.
Surely, GPT said something more elaborate than "nuh uh".
That has nothing to do with anything. This post is just a tweet from some guy who said ChatGPT found errors, with no mention of what those purported errors are. This is a complete crap post and should be flagged.
classic HN isolated demand for rigor. i found it interesting
Think of it as a (fuzzy) compiler.
Now think about complaining that someone "merely pasted my code into a compiler and said that there are errors in my code."
That's nuts, right? If a compiler -- a kind of proof engine -- finds errors, there are errors. Fix them! Don't complain about the usage of a compiler, the errors were real, they're there, they need to be fixed.
The fuzziness part is both useful and a liability, but the error rate is dropping.
Some numbers for you:
A year ago, I ran a huge legacy codebase through Gemini Pro 2.5 and it spat out several thousand bugs, but manual inspection showed that 80% were false positives and only 20% were worth fixing.
I did a similar exercise with GPT 5.6 more recently, and the hit rate was 80% valid and 20% false positive. The human developer got into an argument saying that they thought all 100% were false positives. He lost. The AI could easily produce "repro" snippets that triggered every issue (mostly security vulnerabilities, but some correctness ones too).
These days, if you're publishing anything with verifiable claims in it, and not running it through an AI first, it's the direct equivalent of uploading open source code to GitHub without bothering to check that it compiles first.
It's sloppy.
I write a program that compiles and appears to work and has been checked over by other programmers for months. Someone posts a tweet on X saying that ChatGPT found errors in the program, with nary a hint as to what those errors are.
That's comparable to what we have here. It's a completely crap post and should be flagged.
Pre-chatgpt 5.5 it hallucinated quiet a bit on hard math. Starting from 5.6 it is incredibly good.
I would guess that LLMs are much worse at math without having a theorem prover to validate approaches with. Since this draft is not formalized, an LLM is operating purely on the text which is more tenuous.
I’m doing it daily and it finds a bunch of issues which are not issues in practice.
I suspect LLMs are much better trained and more reliable at bug identification in react apps than draft-state proofs on the bleeding edge of math, on problems that are specifically highlighted as extremely difficult to solve without error in their training.
Cool story.
I have the opposite experience in my line of work.
What's your line of work?
> the latest LLMs have a very low false positive rate for bugs.
Man people live in different worlds, astra for me finds 25/6 bugs all the time.
That has nothing to do with anything. You posted a tweet from some guy who said ChatGPT found errors, with no mention of what those purported errors are. This is a complete crap post and should be flagged.
Also,
> alpoge/anthropic certainly has the claim to the first proof
is nonsense ... many "proofs" of RH have been published that turned out to be wrong. We don't yet know whether that is the case for Alpöge, and we won't know until/unless the Millenium Prize is handed out in two years.
FYI Alpoge's proof of RH (and GRH) has been verified now. The Lean was checked to be a valid proof and the statement is correct. There is no reasonable doubt it's wrong.
It's being hurriedly announced as fully human work. I wonder if the author had carefully avoided ever running the full argument through frontier LLMs (OpenAI/Anthropic) for fear of it being stolen or incorporated into AI training.
This may be the first time this proof is being proof-read by frontier models
(this is pure speculation, but provides an answer to your question)
I also suspect the author actively tried to avoid using AI at all.
Another possible reason: AI will correct your mistakes by using ideas of others. A proof will unlikely remain human-only, even if you just use AI to check for your errors.
Seriously. At least be constructive. Share what the errors were. Make an effort to understand them, anything. That tweet just amounts to low effort FUD.
Very high value. The LLMs are excellent at auditing math proofs.
Can confirm that current GPT (with no guidance) took 5 minutes to find what it claims are errors with counterexamples. (I have nowhere near the level of state-of-the-art mathematical knowledge and experience needed to validate them. I'm posting this solely as additional confirmation of "there may be issues with this proof that an actual mathematician together with LLM assistance would be able to properly evaluate".)
And that is the problem.
AI claims but too few people can check.
Won‘t get better from now on
I mean I guess there is incentive for some human curiosity and trying to learn more than ever, small bit but still to get a standard for verification.
I wouldn't be surprised if these models were aligned to extreme skepticism (to the point of hallucination) when e.g. a user proposes a solution to a Millennium Prize Problem.
1) There's a meaningful difference between "I'm not buying it based on my priors" and "I'm claiming to have a counterexample". I'm looking forward to seeing analysis by other mathematicians.
2) Any model with properly updated priors will have less skepticism of a Millennium Prize Problem being solved now.
Did you try the same with the other LLM assisted proofs? I would be curious what your GPT model says about them. I fed Alpöge and Anthropic statement and details to Gemini 4 hydrogen, and it said that there are "major problems and red flags in both the text and the current verification status", and that it is impossible to verify the 250k pages of slop, and there are there are "Extraordinary, unsubstantiated mathematical leaps in the sketch". So I would take these surface level reviews by LLMs with a grain of salt.
Anyhow, I hope that they are both right!
The LLM-authored proof is too large to fit in the context of any of the models available to consumers
You know this kind of off-hand remark with minimal effort outside of pasting into GPT or retweeting another’s 2 cents degrades what was always implicit in human effort, that it is of worth.
We are not machines, we _care_ about the man or woman behind the result, your chiming in, expending a few minutes of your time, is becoming of a dilettante. It depresses me to think that this is not only becoming more prevalent but common.
I’m embarrassed on your behalf. Here is to a future where the man in the arena is slung by formally verified arrows.
Guess you're not yet on board with the new future where we are all called to do nothing but CONSOOM and not bother at all about the source or process used to make what we're consooming.
Ok, would you rather nobody comment on the correctness of this proof until a human expert spends weeks finding the same issue?
https://XNotMe.com/AcerFur/status/2123853069599547786
This lacks much in the way of credentials.
> but either way, alpoge/anthropic certainly has the claim to the first proof
Is that a thing? Don't people care more about the proof's elegance and insight than its timing?
Yes. That has always been the beauty of it all: the human discovery. Perhaps mathematics awards should be abolished; instead, recognition should be given to the human being in their capacity as a "prompter" or an "AI assistant."
Darwin only published because Wallace came to him (wrote to him, to be precise - Wallace was living in Indonesia at the time) and said that he'd figured out how species are born. Darwin had been writing what was basically a giant proof of it but wasn't going to publish until his enormous book was done. When he realized he was potentially being scooped, he a) made sure Wallace got credit at the Royal Academy too by having them read both papers back to back and b) published the Origin instead of his giant tome.
Today everybody knows Darwin's name and only nerds know Wallace. Timing (and seniority) matters!
Yeah, it's a thing, for example the clay institute award goes to first proof, and if you look at any pre-AI news about theorems being proven it's always about who's first. Are you asking this question in good faith?
> Are you asking this question in good faith?
Yes, I'm just not into maths. I think Terry Tao highlighted the difference between a proof that teaches you "A is true" vs a proof that enlightens you and has broad implications.