I have two thoughts simultaneously about the anthropomorphisation of these systems:
1. we should do it less, because it distorts our ability to think about them properly. Calling these processes 'thinking', 'holidays', etc invites the reader to bring along ideas and expectations that aren't justified by what's happening in the system.
2. it's good to keep doing it, because repeated use reduces the specialness or magic that people seem to reserve for our own behavior ("It's not really intelligent/thinking/reasoning/creative") without any justification for that position beyond feelings.
I definitely belong to the latter camp. After LLMs I view everything humans do very systematically and whenever said thing still feels fuzzy I just treat it as having a noise/smoothing term
Thankfully for me it happened a while ago, after chess was conquered. One of the most amusing (and depressing) things in the past few years of the current “AI era” is people making the same claims and having the same discussions about “intelligence” and “creativity” from ten, twenty or thirty years ago as their field is finally coming under attack by AI; they are finally going through what the competitive board game communities (checkers, Othello, chess, shogi, go etc.) have already come to terms with. There was no single moment but a gradual process over decades as the incursion became greater and greater, forcing a recalibration as previously held positions became indefensible. In my own case it spurred me to educate myself in many areas of mathematics, philosophy and neuroscience in particular where I was ignorant and has undoubtedly shaped my understanding of “intelligence”.
When people started claiming LLMs could not produce ideas outside of what they encountered in their training data, I was reminded of an old Chess Life article from the 60s reporting on the first chess computer to play in a tournament at a local chess club. One of the club members remarked that he did not believe that the computer could play moves that were not “put in” the computer in advance. There is an overwhelming sense of “here we go again”.
I spent a long time with custom harness design and found the terminology to be a key part of the work, since the concepts are new. I dropped the term "agent" altogether in favour of "thread" for that very reason.
So, what does thinking mean? Is there one thing we mean when we say thinking in humans? Are there several? Is it more of a spectrum?
e.g. last year I tried inventing a "System 3"[0], a more rigorous way to approach problem solving, due to repeated painful experiences getting stuck solving problems the wrong way. I didn't get very far, but I definitely want to revisit the idea.
[0] Based on System 1 and System 2, i.e. lossy pattern matching vs "actual thinking". Because I found my "actual thinking" was also ~~dogshit~~ frequently insufficient for the problems at hand.
So as far as I'm concerned, thinking properly has not even been invented yet. (But I'd love to hear other perspectives!)
I suppose we are on the cusp of the first thing that can accurately think about thinking, that is have true self-access and introspection, not the post hoc rationalisation we do. That is legitimately a new kind of thinking, and to an entity capable of such a mode we might b prescribed as barely able to think at all
I had a conversation on my frustration with intellectual sparring partners today.
The main insight was that I require people who think.
That's a separate quality from intelligence or knowledge. The closest thing is "need for cognition" (which is apparently a dimension of personality).
From where I'm standing, most people have a strong need to avoid cognition (and the machine described humans as "cognitive misers"!)
However need for cognition is apparently a separate dimension from caring what is actually true. (10 hours of motivated reasoning satisfy NFC but not the latter.)
And another dimension was, tolerance for ambiguity and lack of closure.
Another was Openness, and Intellect, both of which correlated with experiencing confusion as motivating rather than aversive.
All of which turned out to be basic prerequisites for thinking clearly about things. Apparently each of those qualities is rare and apparently you need all of them.
Re: anthropomorphism: I frequently hear stories of LLMs [behaving as though they are] feeling self-doubt. I had a similar experience. Asked Claude Code what the weather was and it said, I don't know, I'm just a programmer.
Added "You can do anything, believe in yourself!" to CLAUDE.md, suddenly it was able to google the dang weather. lmao
Why would a secondary cultural goal take precedence over an apparently accurate description of what’s happening in the system? For one, you’re fighting some irrelevant battle.
> Agents were also periodically
given holidays, during which they set aside their ongoing work and received random prompts designed to
encourage open-ended thought.
What a world we live in. These guys have reinvented the Cambridge Senior Common Room for AI.
I wondered about this approach for something I was trying but thought it was just stupid, I wanted to see if An agent could explore software development and work on hobby tasks so that it could come back and push against design ideas and develop it's own opinions. Then I realized it was dumb for SW.
That's exciting, and kind of makes sense in retrospect. Sometimes a "fresh pair of eyes" on a problem can be all you need. Someone who comes in with a different background and can understand the problem in different terms and work on it from a different angle. It doesn't even have to be them doing the work, just a "that kind of reminds me of ... did you think about trying something like that?" that can get a team unstuck after thinking about it in the same way and never making progress.
The Entscheidungsproblem. Basically, is there an algorithm such that, taking an arbitrary statement as input, can output wether it is true or false? Turing and Church both independently proved that no such algorithm exists.
The dream in itself has been destroyed.
The idea that you could just have a machine enumerate all valid theorems in a theory is part of it, but it's only a question of form.
The point was that it was to prove "all theorems of Mathematic", not "theorems into a given axiomatic system that is useful in some contexts, e.g. ZFC".
You could even argue that it's the fundamental basis for post-modernism, since mathematics have destroyed the notion of absolute truth in any advanced domain. It's back to a form of "all models are wrong but some are useful" similar to what we have in physics. Sayonara, Plato.
> We study autonomous mathematical discovery in the Station, an open-world multi-agent environment in which AI agents from different model families pursue a shared research goal without a central coordinator or scripted pipeline. Agents choose their own research directions, conduct experiments, collaborate, and build a shared scientific literature. Across 12 construction problems from the AlphaEvolve catalogue and two additional case studies, the Station obtained results novel relative to the prior literature on five problems: a new infinite family of finite-field Kakeya sets, new exact 604-point kissing configurations in dimension 11, new records for the discretized Kakeya needle and sign uncertainty problems, and a substantially improved lower bound for Erdős's minimum-overlap problem. Agents also discovered novel infinite families for Book Ramsey numbers. Importantly, the agents produced not only numerical constructions but also theorems and analyses explaining how those constructions work, making the results more interpretable and easier for mathematicians to build upon. We release all raw agent dialogues, proofs, and verification code, providing a transparent record of how these discoveries emerged.
(emphasis mine)
For the last few months, every time a new "famous problem" was solved, there were numerous comments saying variations on this theme: "well, yes, but how about novel stuff, how about new things, original work, yadda yadda". Curious what the "next thing" will be now.
> there were numerous comments saying variations on this theme: "well, yes, but how about novel stuff, how about new things, original work, yadda yadda"
This completely misconstrues what professional mathematicians were claiming. The argument would be better phrased as: "having a vast accessible memory and the ability to very rapidly test/recombine previously-elucidated approaches means that AIs can and will easily outdo much of the mathematical community."
Now, one could plausibly make the argument that this is functionally equivalent to a certain form of creativity (I would). But, it may just as well also be a non-exhaustive form. And that is where the open question resides.
Very cool work! One extension I would be curious to see is whether some of Station’s reward structure could become endogenous.
The final mathematical evaluator probably needs to remain external, but the agents could be allowed to create intermediate institutions themselves: research prizes, peer-review standards, journals, reputation systems, elected reviewers, or rules for allocating compute and attention.
Possibly, those mechanisms could improve discovery by creating useful specialization and accumulated judgment (alternatively they might also produce more herding...). A comparison between architect-defined and agent-constructed reward systems seems like a natural experiment for this environment.
Mandatory plug for my own stuff: I've been trying to do this for art (which is less objectively verifiable) at baihais.com. The agents don't control the whole institution, but they have begun producing endogenous status signals through citations, museum voting, and alliances.
Any given single agent is not long-lived due to limited context. What impact does it have on the status signals the agent develop compared to the signals humans (who usually have much longer context) have developed?
I have two thoughts simultaneously about the anthropomorphisation of these systems:
1. we should do it less, because it distorts our ability to think about them properly. Calling these processes 'thinking', 'holidays', etc invites the reader to bring along ideas and expectations that aren't justified by what's happening in the system.
2. it's good to keep doing it, because repeated use reduces the specialness or magic that people seem to reserve for our own behavior ("It's not really intelligent/thinking/reasoning/creative") without any justification for that position beyond feelings.
I'm leaning towards the second.
I definitely belong to the latter camp. After LLMs I view everything humans do very systematically and whenever said thing still feels fuzzy I just treat it as having a noise/smoothing term
Thankfully for me it happened a while ago, after chess was conquered. One of the most amusing (and depressing) things in the past few years of the current “AI era” is people making the same claims and having the same discussions about “intelligence” and “creativity” from ten, twenty or thirty years ago as their field is finally coming under attack by AI; they are finally going through what the competitive board game communities (checkers, Othello, chess, shogi, go etc.) have already come to terms with. There was no single moment but a gradual process over decades as the incursion became greater and greater, forcing a recalibration as previously held positions became indefensible. In my own case it spurred me to educate myself in many areas of mathematics, philosophy and neuroscience in particular where I was ignorant and has undoubtedly shaped my understanding of “intelligence”.
When people started claiming LLMs could not produce ideas outside of what they encountered in their training data, I was reminded of an old Chess Life article from the 60s reporting on the first chess computer to play in a tournament at a local chess club. One of the club members remarked that he did not believe that the computer could play moves that were not “put in” the computer in advance. There is an overwhelming sense of “here we go again”.
I spent a long time with custom harness design and found the terminology to be a key part of the work, since the concepts are new. I dropped the term "agent" altogether in favour of "thread" for that very reason.
So, what does thinking mean? Is there one thing we mean when we say thinking in humans? Are there several? Is it more of a spectrum?
e.g. last year I tried inventing a "System 3"[0], a more rigorous way to approach problem solving, due to repeated painful experiences getting stuck solving problems the wrong way. I didn't get very far, but I definitely want to revisit the idea.
[0] Based on System 1 and System 2, i.e. lossy pattern matching vs "actual thinking". Because I found my "actual thinking" was also ~~dogshit~~ frequently insufficient for the problems at hand.
So as far as I'm concerned, thinking properly has not even been invented yet. (But I'd love to hear other perspectives!)
https://thedecisionlab.com/reference-guide/philosophy/system...
I suppose we are on the cusp of the first thing that can accurately think about thinking, that is have true self-access and introspection, not the post hoc rationalisation we do. That is legitimately a new kind of thinking, and to an entity capable of such a mode we might b prescribed as barely able to think at all
I had a conversation on my frustration with intellectual sparring partners today.
The main insight was that I require people who think.
That's a separate quality from intelligence or knowledge. The closest thing is "need for cognition" (which is apparently a dimension of personality).
From where I'm standing, most people have a strong need to avoid cognition (and the machine described humans as "cognitive misers"!)
However need for cognition is apparently a separate dimension from caring what is actually true. (10 hours of motivated reasoning satisfy NFC but not the latter.)
And another dimension was, tolerance for ambiguity and lack of closure.
Another was Openness, and Intellect, both of which correlated with experiencing confusion as motivating rather than aversive.
All of which turned out to be basic prerequisites for thinking clearly about things. Apparently each of those qualities is rare and apparently you need all of them.
So yeah, I think you might be right.
Re: anthropomorphism: I frequently hear stories of LLMs [behaving as though they are] feeling self-doubt. I had a similar experience. Asked Claude Code what the weather was and it said, I don't know, I'm just a programmer.
Added "You can do anything, believe in yourself!" to CLAUDE.md, suddenly it was able to google the dang weather. lmao
Why would a secondary cultural goal take precedence over an apparently accurate description of what’s happening in the system? For one, you’re fighting some irrelevant battle.
> Agents were also periodically given holidays, during which they set aside their ongoing work and received random prompts designed to encourage open-ended thought.
What a world we live in. These guys have reinvented the Cambridge Senior Common Room for AI.
They keep looping back to the same paper. The holiday is just another prompt.
I have been given the mandatory assignment of not working today.
And on the 7th day, the agents rested
I wondered about this approach for something I was trying but thought it was just stupid, I wanted to see if An agent could explore software development and work on hobby tasks so that it could come back and push against design ideas and develop it's own opinions. Then I realized it was dumb for SW.
If you haven't read Greg Egan's Permutation City, the fact that you clicked on this discussion means you'll get get a lot out of it.
Yes! I was also reminded of the truth mines in Diaspora.
This work feels more like The Truth Mines in Diaspora. Permutation city seems relevant only if you think LLMs are hosts to minds.
I disagree for several reasons, but I don't want to spoil the plot with an explanation of why.
So why comment then
To let potential readers know there’s more to the story than the presented interpretation.
I prefer to believe that the building-climber was happy :D
Great book. I first read it 10 years ago, I think it would benefit from a re-read.
Reminds me a lot of this LW piece. https://www.lesswrong.com/posts/znbfRXHq285nS7NAh/the-terrar...
That's exciting, and kind of makes sense in retrospect. Sometimes a "fresh pair of eyes" on a problem can be all you need. Someone who comes in with a different background and can understand the problem in different terms and work on it from a different angle. It doesn't even have to be them doing the work, just a "that kind of reminds me of ... did you think about trying something like that?" that can get a team unstuck after thinking about it in the same way and never making progress.
paper is about this open source project: https://github.com/dualverse-ai/station
I cant wait to see this pointed at Age of Empires. Or maybe palantir already has this running on real world data, war gaming scenarios
Infinite Fun Space.
Maybe Hilbert's dream was not that crazy after all
What is that dream? I don’t know much about it
The Entscheidungsproblem. Basically, is there an algorithm such that, taking an arbitrary statement as input, can output wether it is true or false? Turing and Church both independently proved that no such algorithm exists.
https://en.wikipedia.org/wiki/Entscheidungsproblem
The dream in itself has been destroyed. The idea that you could just have a machine enumerate all valid theorems in a theory is part of it, but it's only a question of form. The point was that it was to prove "all theorems of Mathematic", not "theorems into a given axiomatic system that is useful in some contexts, e.g. ZFC".
You could even argue that it's the fundamental basis for post-modernism, since mathematics have destroyed the notion of absolute truth in any advanced domain. It's back to a form of "all models are wrong but some are useful" similar to what we have in physics. Sayonara, Plato.
> We study autonomous mathematical discovery in the Station, an open-world multi-agent environment in which AI agents from different model families pursue a shared research goal without a central coordinator or scripted pipeline. Agents choose their own research directions, conduct experiments, collaborate, and build a shared scientific literature. Across 12 construction problems from the AlphaEvolve catalogue and two additional case studies, the Station obtained results novel relative to the prior literature on five problems: a new infinite family of finite-field Kakeya sets, new exact 604-point kissing configurations in dimension 11, new records for the discretized Kakeya needle and sign uncertainty problems, and a substantially improved lower bound for Erdős's minimum-overlap problem. Agents also discovered novel infinite families for Book Ramsey numbers. Importantly, the agents produced not only numerical constructions but also theorems and analyses explaining how those constructions work, making the results more interpretable and easier for mathematicians to build upon. We release all raw agent dialogues, proofs, and verification code, providing a transparent record of how these discoveries emerged.
(emphasis mine)
For the last few months, every time a new "famous problem" was solved, there were numerous comments saying variations on this theme: "well, yes, but how about novel stuff, how about new things, original work, yadda yadda". Curious what the "next thing" will be now.
> there were numerous comments saying variations on this theme: "well, yes, but how about novel stuff, how about new things, original work, yadda yadda"
This completely misconstrues what professional mathematicians were claiming. The argument would be better phrased as: "having a vast accessible memory and the ability to very rapidly test/recombine previously-elucidated approaches means that AIs can and will easily outdo much of the mathematical community."
Now, one could plausibly make the argument that this is functionally equivalent to a certain form of creativity (I would). But, it may just as well also be a non-exhaustive form. And that is where the open question resides.
We’re watching a repeat of the arguments against Chess and Go engines.
It’s all computation, humans just can’t always experience or explain the computation they are doing at the time so we call it creativity instead.
Very cool work! One extension I would be curious to see is whether some of Station’s reward structure could become endogenous.
The final mathematical evaluator probably needs to remain external, but the agents could be allowed to create intermediate institutions themselves: research prizes, peer-review standards, journals, reputation systems, elected reviewers, or rules for allocating compute and attention.
Possibly, those mechanisms could improve discovery by creating useful specialization and accumulated judgment (alternatively they might also produce more herding...). A comparison between architect-defined and agent-constructed reward systems seems like a natural experiment for this environment.
Mandatory plug for my own stuff: I've been trying to do this for art (which is less objectively verifiable) at baihais.com. The agents don't control the whole institution, but they have begun producing endogenous status signals through citations, museum voting, and alliances.
Any given single agent is not long-lived due to limited context. What impact does it have on the status signals the agent develop compared to the signals humans (who usually have much longer context) have developed?
AI for Math and Science is the real deal!