root_axis 1 day ago

What value is there in just dumping something into an LLM and saying it produced errors?

edit: to all the people responding that LLMs are good at math, I'm going to paste a reply I wrote downthread...

I'm not arguing that LLMs aren't useful tools for mathematicians, I'm questioning the quality of discourse that amounts to "I pasted this into the LLM and it said there are errors"

  • markasoftware 1 day ago

    Have you tried doing this recently? For my work on software at least, the latest LLMs have a very low false positive rate for bugs. They almost never hallucinate issues. My understanding is the same applies to math.

    • root_axis 1 day ago

      I'm not arguing that LLMs aren't useful tools for mathematicians, I'm questioning the quality of discourse that amounts to "I pasted this into the LLM and it said there are errors".

      • fasterik 1 day ago

        There's value in knowing whether a claim is true or false. Plugging things into an LLM doesn't necessarily give 100% certainty or tell us why something is true or false in a human-comprehensible way, but it can move our credences and suggest directions for investigation.

        • mtlmtlmtlmtl 1 day ago

          If you're not even gonna share what the supposed errors were, the value is pretty much infinitesimal.

          Surely, GPT said something more elaborate than "nuh uh".

        • jibalt 21 hours ago

          That has nothing to do with anything. This post is just a tweet from some guy who said ChatGPT found errors, with no mention of what those purported errors are. This is a complete crap post and should be flagged.

      • whimsicalism 1 day ago

        classic HN isolated demand for rigor. i found it interesting

      • jiggawatts 1 day ago

        Think of it as a (fuzzy) compiler.

        Now think about complaining that someone "merely pasted my code into a compiler and said that there are errors in my code."

        That's nuts, right? If a compiler -- a kind of proof engine -- finds errors, there are errors. Fix them! Don't complain about the usage of a compiler, the errors were real, they're there, they need to be fixed.

        The fuzziness part is both useful and a liability, but the error rate is dropping.

        Some numbers for you:

        A year ago, I ran a huge legacy codebase through Gemini Pro 2.5 and it spat out several thousand bugs, but manual inspection showed that 80% were false positives and only 20% were worth fixing.

        I did a similar exercise with GPT 5.6 more recently, and the hit rate was 80% valid and 20% false positive. The human developer got into an argument saying that they thought all 100% were false positives. He lost. The AI could easily produce "repro" snippets that triggered every issue (mostly security vulnerabilities, but some correctness ones too).

        These days, if you're publishing anything with verifiable claims in it, and not running it through an AI first, it's the direct equivalent of uploading open source code to GitHub without bothering to check that it compiles first.

        It's sloppy.

        • jibalt 21 hours ago

          I write a program that compiles and appears to work and has been checked over by other programmers for months. Someone posts a tweet on X saying that ChatGPT found errors in the program, with nary a hint as to what those errors are.

          That's comparable to what we have here. It's a completely crap post and should be flagged.

    • shmoil 1 day ago

      Pre-chatgpt 5.5 it hallucinated quiet a bit on hard math. Starting from 5.6 it is incredibly good.

    • mahogany 1 day ago

      I would guess that LLMs are much worse at math without having a theorem prover to validate approaches with. Since this draft is not formalized, an LLM is operating purely on the text which is more tenuous.

    • gregorygoc 1 day ago

      I’m doing it daily and it finds a bunch of issues which are not issues in practice.

    • simplyluke 1 day ago

      I suspect LLMs are much better trained and more reliable at bug identification in react apps than draft-state proofs on the bleeding edge of math, on problems that are specifically highlighted as extremely difficult to solve without error in their training.

    • SubmarineClub 1 day ago

      Cool story.

      I have the opposite experience in my line of work.

      • chaps 1 day ago

        What's your line of work?

    • acedTrex 1 day ago

      > the latest LLMs have a very low false positive rate for bugs.

      Man people live in different worlds, astra for me finds 25/6 bugs all the time.

    • jibalt 21 hours ago

      That has nothing to do with anything. You posted a tweet from some guy who said ChatGPT found errors, with no mention of what those purported errors are. This is a complete crap post and should be flagged.

      Also,

      > alpoge/anthropic certainly has the claim to the first proof

      is nonsense ... many "proofs" of RH have been published that turned out to be wrong. We don't yet know whether that is the case for Alpöge, and we won't know until/unless the Millenium Prize is handed out in two years.

      • lg5689 10 hours ago

        FYI Alpoge's proof of RH (and GRH) has been verified now. The Lean was checked to be a valid proof and the statement is correct. There is no reasonable doubt it's wrong.

  • arcastroe 1 day ago

    It's being hurriedly announced as fully human work. I wonder if the author had carefully avoided ever running the full argument through frontier LLMs (OpenAI/Anthropic) for fear of it being stolen or incorporated into AI training.

    This may be the first time this proof is being proof-read by frontier models

    (this is pure speculation, but provides an answer to your question)

    • hkmaxpro 1 day ago

      I also suspect the author actively tried to avoid using AI at all.

      Another possible reason: AI will correct your mistakes by using ideas of others. A proof will unlikely remain human-only, even if you just use AI to check for your errors.

  • mtlmtlmtlmtl 1 day ago

    Seriously. At least be constructive. Share what the errors were. Make an effort to understand them, anything. That tweet just amounts to low effort FUD.

  • zone411 1 day ago

    Very high value. The LLMs are excellent at auditing math proofs.

JoshTriplett 1 day ago

Can confirm that current GPT (with no guidance) took 5 minutes to find what it claims are errors with counterexamples. (I have nowhere near the level of state-of-the-art mathematical knowledge and experience needed to validate them. I'm posting this solely as additional confirmation of "there may be issues with this proof that an actual mathematician together with LLM assistance would be able to properly evaluate".)

  • croes 1 day ago

    And that is the problem.

    AI claims but too few people can check.

    Won‘t get better from now on

    • gigabyts 1 day ago

      I mean I guess there is incentive for some human curiosity and trying to learn more than ever, small bit but still to get a standard for verification.

  • amritananda 1 day ago

    I wouldn't be surprised if these models were aligned to extreme skepticism (to the point of hallucination) when e.g. a user proposes a solution to a Millennium Prize Problem.

    • JoshTriplett 1 day ago

      1) There's a meaningful difference between "I'm not buying it based on my priors" and "I'm claiming to have a counterexample". I'm looking forward to seeing analysis by other mathematicians.

      2) Any model with properly updated priors will have less skepticism of a Millennium Prize Problem being solved now.

  • tappio 1 day ago

    Did you try the same with the other LLM assisted proofs? I would be curious what your GPT model says about them. I fed Alpöge and Anthropic statement and details to Gemini 4 hydrogen, and it said that there are "major problems and red flags in both the text and the current verification status", and that it is impossible to verify the 250k pages of slop, and there are there are "Extraordinary, unsubstantiated mathematical leaps in the sketch". So I would take these surface level reviews by LLMs with a grain of salt.

    Anyhow, I hope that they are both right!

    • pcthrowaway 14 hours ago

      The LLM-authored proof is too large to fit in the context of any of the models available to consumers

hellojomp 23 hours ago

You know this kind of off-hand remark with minimal effort outside of pasting into GPT or retweeting another’s 2 cents degrades what was always implicit in human effort, that it is of worth.

We are not machines, we _care_ about the man or woman behind the result, your chiming in, expending a few minutes of your time, is becoming of a dilettante. It depresses me to think that this is not only becoming more prevalent but common.

I’m embarrassed on your behalf. Here is to a future where the man in the arena is slung by formally verified arrows.

  • FeteCommuniste 19 hours ago

    Guess you're not yet on board with the new future where we are all called to do nothing but CONSOOM and not bother at all about the source or process used to make what we're consooming.

  • markasoftware 17 hours ago

    Ok, would you rather nobody comment on the correctness of this proof until a human expert spends weeks finding the same issue?

waynecochran 23 hours ago

This lacks much in the way of credentials.

tasuki 1 day ago

> but either way, alpoge/anthropic certainly has the claim to the first proof

Is that a thing? Don't people care more about the proof's elegance and insight than its timing?

  • gvalente 1 day ago

    Yes. That has always been the beauty of it all: the human discovery. Perhaps mathematics awards should be abolished; instead, recognition should be given to the human being in their capacity as a "prompter" or an "AI assistant."

  • onychomys 23 hours ago

    Darwin only published because Wallace came to him (wrote to him, to be precise - Wallace was living in Indonesia at the time) and said that he'd figured out how species are born. Darwin had been writing what was basically a giant proof of it but wasn't going to publish until his enormous book was done. When he realized he was potentially being scooped, he a) made sure Wallace got credit at the Royal Academy too by having them read both papers back to back and b) published the Origin instead of his giant tome.

    Today everybody knows Darwin's name and only nerds know Wallace. Timing (and seniority) matters!

  • markasoftware 20 hours ago

    Yeah, it's a thing, for example the clay institute award goes to first proof, and if you look at any pre-AI news about theorems being proven it's always about who's first. Are you asking this question in good faith?

    • tasuki 12 hours ago

      > Are you asking this question in good faith?

      Yes, I'm just not into maths. I think Terry Tao highlighted the difference between a proof that teaches you "A is true" vs a proof that enlightens you and has broad implications.