Great article. I feel most people, even great software engineers, seem to lack the intuitive understanding that building software is essentially modeling a system, and that every model faces the same limit: the number of interactions between its part grows combinatorially as the system grows. If some of those parts or interactions produce no value (bloat), then complexity is added for no gain. This is true regardless of who created the model or wrote the code.
Instead they seem to think that AI can someone work around what is essentially laws of nature.
If anything I’ve seen AI become a multiplier. The low initial cost to add new features eliminates the only brake many organizations have on accumulation of bloat.
Laziness is a feature and a very important filter to have in a software engineer. Imagine having in the team a lot of eager (AI) developers that are able to implement in seconds the most overcomplicated or ultimately useless features that will still have to be tested and maintained for years, with no cost-benefit analysis ever made. All it takes is a project manager to have a random idea on a whim, to have to drag it along forevermore.
We're gonna see a bloat epidemic in software like never before.
Perhaps we already are seeing it. I know that we of the orange website get excited at how fast we can churn out work product nowadays. But when the non-tech people in my life have anything to say about it, it’s that they wish it would stop. They see the constant churn as annoying change for the sake of change, and there’s a general sense that lately computers have been getting harder to use.
Spoiler alert: Laziness is one of the most important factors for driving the creation of coherence. We have a deep biological desire to do things faster/better/lazier. Will be digging into this more in Open Capital.
Yes, I don't think many software devs have a proper conception of what it is they are actually doing. They view their job mostly as "make computer do thing" and not "model and represent some aspect of reality as a system of computable functions". Yes I know it sounds overly academized and pedantic but there actually is a huge difference in these perspectives and imo it's only when you adopt something closer to the latter that engineering really happens.
In a team, the contributor thinks of the value of his own feature, but not the complexity it adds to the whole. A reviewer needs to do that. Preferably at the planning stage. It is a tough to tell people not to add a feature, but it has to be done.
When I'm reviewing AI code I worry about complexity more than anything; correctness is relatively easier to assure, whereas complexity is a value judgment.
This points back to a more general problem. I love to haul out the quote "All models are wrong, but some models are useful", and it's actually attributed to a statistician talking about statistical models. The quote was written well before AI, from a mathematician working in a time even before computer science dug into data science/eng.
As presented the Decoherence Trap is AI related, but as someone with a background in physical simulation and modeling, and applying math models & system controls algorithms, software in general has always had a connection-to-reality issue. It's always been able to go faster in building models faster than reality can be checked. People are always eager for the results and to take action on them, when the long run limiting factor is often, can I observe enough real data signal to make a good decision by any set of optimizing model rules. It's often harder and slower to collect real-world data, ideas, and samples than it is to rapidly process it. With AI, I think we can even more quickly build models (not in an llm sense, but in a operational sense) that give beautiful, precise results which may or may not be useful in a connected-to-reality way.
If the model operates on bad data, then the risk is always the temptation of having a precise-looking technical answer to apply. The practices need to be updated, but I think the mindset of systems engineering validation & verification, and software QA are going to be very needed going forward.
There is actually a related issue, in that I don't think we want an overly coherent of set of models running through our economy - it's a strong risk of central planning failings vs competitive/experimental markets, but that's a tangential discussion.
“Decoherence” denotes a process (losing coherence), not a state, but the article uses it like a state: “Well, for one thing, it creates decoherence.” The state would be “incoherence“ (the lack of coherence).
> the more decoherent a system is,
This should be either “incoherent” (lacking coherence) or “decohered” (having lost coherence).
That being said, yes, incoherence in a system is a form of technical debt, and use of AI for software development comes with a heightened risk of the system becoming incoherent.
I think we can safely overload the term since the domains are separated enough. No one will think you're referring to quantum states when complaining that the system has been decohering too much lately. I can see why the author choose the term: "decohere" is a verb, which is fitting since it denotes a process, something that happens to systems gradually as change accumulates. There's no such connotation and associated verb to "incoherence".
The article’s opening sentence states that it is introducing a new term. So the author gets to define it how they like, and nobody can reasonably complain that this sense of the word isn’t already in the dictionary. Of course it isn’t. It’s an intentional neologism.
The analogy to the term from physics, which is a noun that describes a state, seems clear enough to me. I can’t speak to how apt that analogy may be, but software industry jargon mostly consists of appropriating other fields’ jargon based on inapt analogies, so I’m not sure if it matters.
The everyday meaning of incoherent is more like “meaningless.” For that reason alone I would say it’s a worse word for the concept the author describes.
"Incoherent" has a static connotation, "decoherent" OTOH is dynamic and has an associated verb, which is very handy when describing a gradual process which (almost) inevitably occurs as features are added to a system. Since confusion with the usage from quantum physics is unlikely, I too think we can safely and fittingly overload the term like the article does.
Yes, this is the thinking behind using the term. It’s also different from incoherence because decoherent systems can still make sense in some ways. They still work, they can still be iterated upon.
They are decoherent because the work to change them is extremely difficult. This is a different, and much more costly, state to exist within.
Btw, this is why I believe current models with their 128k - 1M context size limits end up creating decoherent systems. They don't have the capacity to see the big picture above some complexity level, and end up doing lot of local optimizations that decohere the system.
I agree. Although local optima is what most humans already operate on. So it’s sounds like an engineering solution could be possible since group of humans are also able to divide and conquer complex systems.
I disagree. LLMs might create incoherence not because of context size limit is too low, but because it's too high. The limit is vastly higher than for humans.
But to overcome the limit, one needs to adopt simpler, more coherent models of reality, which is what I believe humans do. But there is little motivation to do so if your context window allows you to understand the situation in full.
So it seems there is a tradeoff. Some solutions (proofs of complicated statements) might require context of certain minimal size. So human solution doesn't scale.
Maybe we should try to adapt the context window to the system being worked on, to ensure that the system architecture has the structure viable in the minimal context size required.
Humans get an understanding of large scale systems into their head by developing and refining abstract models, and committing them to memory.
LLMs don’t do that. Any abstraction they do happens at training time; they cannot dynamically construct new models at inference time. Nor can they commit them to long term memory, because LLMs don’t have such a mechanism. Which, in turn, precludes progressive refinement.
We get around this by using agentic harnesses to pack everything into the context window. (Which itself is something humans don’t really have.) This is why large context windows are so important for most real world tasks: it’s the only way LLMs have to handle information that wasn’t in their training corpus.
I am not sure you're correct. Yes, from the physical perspective of LLMs design, the above is correct. But they are also systems that evolved to do things through training; so you cannot discount the possibility they can emulate building of new abstractions in another level.
LLMs already have the concept of "reference" - i.e. I can say a name of something, for example of a function, and they understand that the properties associated with the name itself are not relevant, but rather whatever the name does refer to.
But this is pretty much what you need to build new abstractions. You need to name a paragraph with the definition (explanation) of a concept, and be able to recall the paragraph by the name in order to do inference.
So with references, they can get around the context size limitations, like humans can. It's slower than learning things "by heart", but I think LLMs are perfectly capable of that; and the evidence is, they are able to navigate codebases larger than their context, which often contain many new abstractions.
I can say what I was getting at quite definitively. An LLM is, to an approximation, a pure function. No side effects, no state changes. No neuroplasticity. No working memory. It really is just predicting next tokens based on previous tokens. So the only way to change its output for a given input is to change the input - that is, pack different information into the context window.
Some nondeterminism from the probabilistic token sampling and distributed inference, of course. But sometimes randomly choosing the second most likely next token isn’t going to manifest a cognitive feedback loop.
Yes, but we are really talking about LRMs not LLMs. They can emulate working memory so well that many people even recommend using bunch of .md files (as a sort of out-of-context notes) instead of RAG based on vector DBs. They can maintain context, by compressing it and writing out memories quite autonomously.
So yes, even though not updating their weights, they can learn plenty new things and concepts, and build new models of the world.
But now you’ve looped back around to my original point: a large context window is essential to making it work well. Those *.md files won’t do anything if you don’t feed them into the LLM at inference time.
(edit: That said i do agree with the idea of designing systems so that such a large window isn’t needed. But that seems a bit self defeating if the goal is massive development acceleration with agentic coding. These models seem to be pretty bad at engineering for simplicity. And protecting system design with close human oversight would mean we are still limited by the speed at which humans can read and understand code.)
Gotcha. My apologies. It sounds like I misunderstood your original point and I've been talking past you this whole time.
I've been experimenting with improving coding agent performance by improving written documentation. Anecdotally I've seen that, compared to the vibecoded mess I inherited, improving the docs let us use smaller models and get better quality work out of them.
Replacing our skill files with Markdown documentation also made a difference. My hunch is it's because skills tend to get mass dumped into the context window, whereas the agent harness can make more judicious decisions about how to include that information.
But I haven't seen much indication that any of this affects any model's tendency to make a mess of the code, regardless of its context window size. Or the documentation, for that matter. Maybe we can find a hack to get them to do that well, too. I certainly hope so, because I can't see how this context window size red queen's race is sustainable.
My observation is that right now AI coding agents lean towards re-implementing everything from the ground up. This of course will not scale. It will scale beyond what humans writing code would scale to, but we will ask the AI for more, so it will even things out.
1. We will develop new programming languages that are AI first, in that they are aimed at readability and comprehensibility by machines
2. We will adapt our working style (e.g. how we prompt and train the AIs) to factor in maintenance cost.
i.e. AI first implementations will go through the evolution we had with human based code development.
My observation is that right now AI coding agents lean towards re-implementing everything from the ground up. This of course will not scale. It will scale beyond what humans writing code would scale to, but we will ask the AI for more, so it will even things out.
Those who cannot do, preach.
There's a funny trick where if you are an experienced software developer, and take time to think about your design and describe what you want in clear terms, it will actually do things your way. Weirdly, this still saves massive amounts of time.
People who don't understand how to build good software LOVE to expound on the things we'll have to design in order to make AI tools "lean toward" building better software.
It seems that these folks are not willing or able to conceive of, you know, actually getting good at this stuff.
For this reason, it never occurs to them that human beings are STILL the source of the best ideas and designs.
I am not really comforted by this argument against the singularity. It seems to rely on the fact that the evolving intelligence doesn't have access to the real world. Okay, but I wasn't really worried about the singularity inside a datacenter. I was worried about it once we wired up all the robots and sensors and power switches to AI controlled systems. We are still doing that, right?
the point is not lack of access to physical space, as much as that iteration fundamentally only gains speed by staying 'inside a mind', the moment you interact with the world you're bottlenecked.
Like, you can make the AI think super hard about what's the phrase that will convince your boss to give you a raise, but eventually you have to try a phrase and wait for their answer to have any idea whether the result was good or not. Without that, you don't know if you're improving the shot or just hallucinating aimlessly for hours.
Then, the speed of improvement is only as fast as the speed of iteration. The robot can learn to walk from falling, but it can't fall any faster than a real human, so it can't learn at 1M falls/second.
The way this is done in every domain where the real is a bottle neck is with simulation. So the actual argument is: What domains do we think are not practical to simulate even with extreme intelligence making the simulation and the ability to bring immense amounts of computing power to bear while running.
I don't think there are actually that many domains where you can't simulate them with enough knowledge and compute. So then the bottleneck becomes your ability to sample the world and refine your models. For some domains you can have the models run against each other in your sim which greatly reduces the need for real world feedback (see AlphaZero for example).
So in the case of human persuasion, like asking a boss for a raise, the model could of failed many times a second while trying to persuade a simulation of a typical middle manager with the same parameters as your boss.
But how do you source the observations used that would be the basis of your simulation? Or are you assuming the real world won't change? At least in my mind I don't see many domains that remain static enough where continuous sampling of the real world will not be required. Go and Chess are artificially stable domains, the rules are fixed.
Any social domain where there's competition of resources (salary, funding) will definitely not be stable, it would require constant testing against the real world.
That's a very "Spherical cow in a vacuum" approach. You would need to accurately model the boss, as well as all your interactions with the boss up until your raise request, as well as the boss' environment when you make your request.
You could be on the wrong side of a reasonable boss, or in his good graces but during a bad day ...
That's the point the writer was making when he mentioned role playing games. In isolation, you could spend hours drafting the perfect plan, that wouldn't survive two minutes of contact with reality. You just don't know if your model is correct until you try.
the strongest argument that AI systems have to keep humans systems around is that software does not live, fully, and it's nearly impossible to determine if actions actually work
humans give deeply useful feedback
to AI systems, humans and entire human civilizations represent an independent other
Apologies for opening the discussion with a half related topic, since I haven't finished the article yet, but... Damn, that is one beautiful website.
Edit: Back after reading. I think I have a point about the simulation section. It compares the difference in decoherence introduced in the system by clankers and human devs as if those were different groups.
I suspect if we consider AI a human performance multiplier, we explain easily how supercharging a dysfunctional process just makes it go sideways faster. There are organizational experiments to try there.
For example, what if you only supercharge a section of the company tasked with maintaining coherence?
A common organizational problem is that the people who know and care about the system itself are outnumbered by the rest (juniors, people new to the system, managers rushing jobs, people who want to mark things as done ASAP tactically, etc). This imbalance will multiply when code is mass produced and reviews are dismissed. We're all seeing the senior burnout and cynical YOLOing occurring everywhere.
AI offers a way to rebalance this dynamic, though. You can effectively add multipliers to arbitrary human workers, and get a system that tends to low complexity by design.
You don't have to renounce to speed either. The junior dev will be faster not by being AI-enabled, but by working in an adaptable, understandable system.
I wouldn't say plain, but it's a generic LLM design. Which made me curious whether the blog was around with a different look before AI was commonly used for such things, and it turns out it was, at https://www.pzuraq.com (which now redirects to the current site).
One article survived the move, so you can compare:
Guilty as charged, I did use LLMs to bootstrap the design and much of the site, though I took it in my own direction after the initial pass. But it is generic enough, and I just needed something a little fresher and quick.
At some point I’m going to get the older articles back up, but most of them are very specific to a time that just doesn’t have relevance anymore, even to the Ember.js community. Didn’t want to distract from the newer content.
And at some point I’ll make a new account here too… web handle changes are always a pain
Outside of engineering, nomading and migration in general creates a form of social decoherence (eg, difficult for foreigners to get things done, hard to make friends, etc.).
When we humans have realized we created a system too incoherent, at some point we just create a new one, porting all or some of the data/functionality. But also, we are already presented with problems that all of the current compute on the planet cannot solve. If a software system becomes one of those problems, I'm not that bothered, because it isn't as pressing as real-world one. I don't know, I see it as an argument against the singularity, but I'm not nearly afraid of it as the author seems to be.
I can appreciate the complexity argument, but that assumes that AIs can't standardize common patterns into pathological cases. For example our AIs today can code slop, but the slop doesn't try to re-create the IP stack. Why does the author assume we can't make AIs that optimize for better long term design view much like they can optimize for easier goals today like code throughput or program speed?
They can over time, but A) they can only improve to the human ceiling because they only have the set of apps that we have already created as inputs and B) creating new apps requires testing in the real world, which requires either humans or robots to go out and use the apps to do things.
You could try to make plausible apps that you never use, but then you run the risk of drifting. And without new apps to build on, you will be stuck on an artificial maximum that works well for the apps of today, but may not for the apps of tomorrow.
Imagine if AI had never seen a GUI app, if they had existed in the era of the CLI as the primary way of interacting with the server. How would they have known that Futures would be so important? They could try to guess, but most people thought Engelbart was crazy before he demoed his ideas. Hard to imagine an LLM making progress in a vacuum.
Fantastic framework that I will be adopting, thank you!
One thing I will add: I'm sure the optimistic side will try to take the quantization that your (and other work like SlopCodeBench) are introducing and claim that we'll be able to fix the problem by training on that data, producing coding models that are rewarded for coherence.
I invite them to try! But my suspicion is that this will require a rather deep form of intelligence (world models) that the current language/data-based paradigm isn't really equipped to reach without requiring more energy than the Sun.
Yes, this is also my belief. All of this is contingent on just trying to scale the current tech. If we make major breakthroughs on other forms of AI, then the analysis could change.
> This essay introduces the concept of decoherence: the accumulation of locally valid but mutually incompatible or redundant models within a system, such that the cost of change is dominated by translating among those models and preventing unintended side-effects.
But then famous last words:
> The good news is that this rules out the specter of runaway recursive self-improvement, the intelligence explosion that goes parabolic before anyone can react, as decoherence both prevents AI from improving in a bubble and limits the rate at which it can improve outside of it.
After the OpenAI/Huggingface and similar I am not so sure. Even if that was a publicity stunt of sorts, I think it shows they can go off the rails. Today it is hacking, but tomorrow it may be training a smarter model to join their cohort, without us being involved.
LLMs can be extremely dangerous without being capable of achieving super intelligence in a vacuum.
If I hooked up a random number generator such that, if it landed on a specific number between 1 and a trillion, it would launch a nuke, that would be incredibly dumb and incredibly dangerous.
Even more reason to slow down the slop code and stop attempting to achieve super intelligence when it appears that it will not be possible with this technology. The process is dangerous, and the doom race/prisoner’s dilemma logic is faulty.
In pure software architecture and design term, this would have been convincing until the recent models who are let’s be honest better than humans in architecture and design.
Not to mention, software is a fundamentally collaborative process. We build it better by working together to build coherence, and we are capable of doing that over very long periods of time. Agents do not appear to be (the hugging face incident is not IMO an example of building coherence, hacking is about exploiting decoherence/bugs, not the same thing)
>> But do we know how to build every iteration of a ship in between wood and steel? Could you, say, slowly replace every piece of a man-of-war and incrementally turn it into a modern destroyer? And have it functional (for some definition of functional) at every step in the process?
Yes. Yes we do. Study ship architecture and you will indeed find every step between The Black Pearl and the USS Iowa. There were sailing ships made out of iron.(1) There were wooden ships with turrets.(3) There were battleships that looked more like hotels than ships. (2) There waw even a paddlewheel aircraft carrier used in WWII. (5) Nearly every idea has been tried and nearly every ship class at some point got a Frankenstein upgrade to become a halfway point between conceptual generations. Even the Iowa class was earmarked to be "upgraded" with the addition of a flight deck.(4)
Every humpback whale is related to every giant sequoia through a chain of intermediaries as similar as a mother and child (proof: go up via mothers to the first eukaryote, then back down).
That doesn't mean you can _incrementally_replace_parts_ of a whale until it's a tree, with every intermediate being viable.
Excellent article. This incoherence is a side-effect of a system built far faster than it is consolidated and thought about from an external behavioral standpoint.
I've created a skill that the agent runs after significant codebase work has been done: https://gist.github.com/scroobius-pip/adb87dffca64b11bad3dd1...
This is very high effort, but the central argument is wrong.
The central argument is that AIs, living inside datacenters, won't be able to evaluate the "coherence" of software "beyond the human ceiling". Okay; why do we need this "coherence"? The article argues that it's needed for ease of adaptation. But ease of adaptation (as measured in tokens required to add new features, or whatever) can be directly observed by AI as it works. AI can make conclusions about it, and carry them over from one program to another, without running into any human ceiling. So it can stay within its datacenter and make software that's faster than human-written, smaller, and more adaptable too. The whole essay dissolves. Sorry!
> The central argument is that AIs, living inside datacenters, won't be able to evaluate the "coherence" of software "beyond the human ceiling"
Actually, the article states that AI will easily be able to handle increases beyond the human ceiling as long as those increases follow a linear curve, not an exponential curve.
Your premise about coding agents’ ability to manage complexity and decoherence for themselves sounds conjectural to me. Have you seen it reliably do this?
For my part, I spent a bunch of time this summer trying using a coding agent to help me pay down a technical debt situation that could be described well by this decoherence concept. The initial hope was that Claude would make light work of the project, but that did not happen. It needed constant close hand holding. It could not independently identify competing models. It crumbled in the presence of terminology drift. And it always suggested fixing complexity problems by adding even more epicycles, never by simply eliminating the unnecessary complexity.
That said, it did a great job of implementing plans, and was even helpful for validating them prior to implementation. But it still needed a human to find, diagnose, and prescribe treatment for the problems. Which implies that, where maintainability is concerned, meat is still load bearing.
Lol, no. I don't need to. I can predict such things.
For example, I predicted about 15 years ago that AI would decisively beat humans at math, because math is formalizable. Most people didn't believe it until this summer, and some don't believe it still.
To people who don't understand that AI improves over time, my prediction skill seems magical.
There are deeper challenges implied in that statement, though.
Where does the agency for an AI to leave the bubble come from? And when it does leave the bubble, where should it go for researching an expanding, exponentially complex problem space? Creativity is needed to solve some problems, something that cannot be infinitely encoded with statistics.
And what about the irreducible complexity problem? Does an AI just circle the solution drain forever in the case it lacks agency and insight and curiosity enough to collect all the correct requirements? Can we code curiosity that actually functions, a.k.a. consciousness?
Do you believe human consciousness could be reducible to program statements and serve as a useful model? Is consciousness a mechanistic, wholly internal process that occurs in our bodies or is it something experienced that life on earth is exposed to and somehow "taps into"?
Consciousness isn't required for problem solving. People said for decades that chess requires Genuine Human Creativity. When chess fell, people started saying that Go requires Genuine Human Creativity. When Go fell, people started saying the same about math; starting from simple problems and going up to Millennium problems. Now that a Millennium problem fell, people are still saying this. What can I say? Some people can't extrapolate from past evidence to likely future.
But math is not formalizable (ask Goedel why). And no, computers have not decisively beat humans at maths, that's a laughable assertion. Sure, laymen always imagined math as being about proving theorems, but there's a saying among mathematicians: good mathematicians prove theorems, great mathematicians make conjunctures, the greatest mathematicians write definitions. But maybe you already thought they've solved math when they were able to calculate a zillion digits of pi, lol.
It sounds like you're a bit like that AI stuck in the datacenter, confirming their hypothesis without needing to actually check whether it's true or not in the real world. Then proceeding to imagine how the rest of the world is wondering at your imagined magical capacity of prediction.
Goedel's theorems don't stop AI from finding and formalizing math results. They certainly don't stop AI from making conjectures or writing definitions. They have nothing to do with any of that.
The theorems are about inability of formal theories to prove certain sentences involving provability (the first is about "this sentence is not provable", the second is about "this theory is consistent"). Most mathematicians never touch any such sentences in their entire career. All the life's work of someone like Gauss, an all-time great, can be formalized just fine.
I'm not surprised your understanding of Gödel's findings and their implications is just as superficial as that of maths and philosophy of mathematics in general. Gödel's work has implications way beyond self-referential logic games, and cannot be swept in a corner where it can be safely ignored. For example, there are certain diophantine equations that have no general solution due to Gödel. Also, worth discussing are the Continuum Hypothesis, Whitehead's Problem and stuff like Paris–Harrington Theorem, Goodstein’s Theorem, etc. Not to mention the huge impact on computer science with the Halting Problem, Rice's Theorem and others. In fact Gödel's work actually spawned CS as a field of study.
The idea that maths is just the study of arbitrarily thought-up formal systems is just a very naive idea that doesn't actually reflect reality, once investigated in depth. Anyway, I have feeling you will let nothing rob you of that sweet feeling of having "magical predictive capacities", so I'll be on my way. I've already dropped some bread crumbs if you're interested in actually expanding your horizons and getting in the weeds, where things are not as straightforward anymore. But beware, "here there be monsters". Godspeed!
> For example, I predicted about 15 years ago that AI would decisively beat humans at math, because math is formalizable. Most people didn't believe it until this summer, and some don't believe it still.
That is a laughable quote for anyone that know the history of type theory and the foundational crisis that happened at the beginning of the last century. Math as a whole can't be formalizable but we can create various formalist for different math purpose just like we have so much programming languages.
And the study and the creation of those formalisms is not something automatisable.
I'm also not entirely convinced that plugging a proof assistant into an agentic coding harness is quite as big a deal as many are making it out to be.
It just kind of feels like a Watson moment. 15 years ago a computer won at Jeopardy!, and that spurred plenty of AI hype. Which was understandable, because playing well at Jeopardy is something that requires a lot of intelligence when humans are doing it. But what actually happened is that Watson's developers figured out a really clever way to cheese it, no general intelligence required. So it was also a kind of gee-whiz technology that didn't live up to the hype. Most applications based on it failed to generate any ROI and have since been discontinued, IBM lost billions of dollars on the venture, and it's now been sold off to a private equity firm and is mostly forgotten.
None of that means that Watson wasn't a technically impressive, even revolutionary product. But the world is full of impressive inventions that nevertheless fail to justify what they cost to build and operate.
> And the study and the creation of those formalisms is not something automatisable.
Lol again. Have you tested your assertion? Go ask your favorite LLM to invent a machine-checkable formalism for something that you vaguely describe to it. It'll oblige. If it's one of the top models and you give it a good budget, it'll do a good job. Very likely it'll do a better job than you.
Seems like a non-sequitur? "because math is formalizable", I agree it makes sense to then predict that data-driven AI models will be able to perform well on it. But where is the evidence that software coherence is formalizable? If it were then I would agree, but to me it seems that coherence in a software system relies on a ton of intuitive reasoning and familiarity with company culture, past failures, etc. it's like the least formalizable thing there is.
AI improves over time, sure, but it has differences and advantages and disadvantages over human intelligence. It's just a fundamentally different thing. In the same way calculators are superhuman, and improving, but can't write books. You yourself acknowledge to some degree the uneven nature of that improvement when you referenced formalizability.
In my comment that started the thread, I pointed out that "coherence" in the article is considered valuable because it's a proxy for "ease of modification", but AIs can estimate "ease of modification" directly (by looking at how hard it is for them to make changes to a codebase) and learn from that in a closed loop. No round-trip through humans required, contrary to what the article says.
Great article. I feel most people, even great software engineers, seem to lack the intuitive understanding that building software is essentially modeling a system, and that every model faces the same limit: the number of interactions between its part grows combinatorially as the system grows. If some of those parts or interactions produce no value (bloat), then complexity is added for no gain. This is true regardless of who created the model or wrote the code.
Instead they seem to think that AI can someone work around what is essentially laws of nature.
And migrating from one system-model to another is an additional system-model - the building of which is an even rarer skillset.
If anything I’ve seen AI become a multiplier. The low initial cost to add new features eliminates the only brake many organizations have on accumulation of bloat.
Laziness is a feature and a very important filter to have in a software engineer. Imagine having in the team a lot of eager (AI) developers that are able to implement in seconds the most overcomplicated or ultimately useless features that will still have to be tested and maintained for years, with no cost-benefit analysis ever made. All it takes is a project manager to have a random idea on a whim, to have to drag it along forevermore.
We're gonna see a bloat epidemic in software like never before.
Perhaps we already are seeing it. I know that we of the orange website get excited at how fast we can churn out work product nowadays. But when the non-tech people in my life have anything to say about it, it’s that they wish it would stop. They see the constant churn as annoying change for the sake of change, and there’s a general sense that lately computers have been getting harder to use.
Spoiler alert: Laziness is one of the most important factors for driving the creation of coherence. We have a deep biological desire to do things faster/better/lazier. Will be digging into this more in Open Capital.
There's also the way AI itself tends to work: In my experience it very rarely refactors existing code when it's adding something new.
AI tends to layer new code on top and you get a steaming mess very quickly if you aren't careful.
I’ve used a harness that had no compunctions about deciding for itself when to refactor existing code. It was not a better experience.
Yes, I don't think many software devs have a proper conception of what it is they are actually doing. They view their job mostly as "make computer do thing" and not "model and represent some aspect of reality as a system of computable functions". Yes I know it sounds overly academized and pedantic but there actually is a huge difference in these perspectives and imo it's only when you adopt something closer to the latter that engineering really happens.
> Instead they seem to think that AI can someone work around what is essentially laws of nature.
There's a lot of magical thinking about AI's present and future capabilities.
In a team, the contributor thinks of the value of his own feature, but not the complexity it adds to the whole. A reviewer needs to do that. Preferably at the planning stage. It is a tough to tell people not to add a feature, but it has to be done.
When I'm reviewing AI code I worry about complexity more than anything; correctness is relatively easier to assure, whereas complexity is a value judgment.
> Instead they seem to think that AI can someone work around what is essentially laws of nature.
I am not. I am just assuming that whatever approximation humans can build, a robot could do better.
Ex falso quodlibet.
(note that your comment uses present tense)
This points back to a more general problem. I love to haul out the quote "All models are wrong, but some models are useful", and it's actually attributed to a statistician talking about statistical models. The quote was written well before AI, from a mathematician working in a time even before computer science dug into data science/eng.
As presented the Decoherence Trap is AI related, but as someone with a background in physical simulation and modeling, and applying math models & system controls algorithms, software in general has always had a connection-to-reality issue. It's always been able to go faster in building models faster than reality can be checked. People are always eager for the results and to take action on them, when the long run limiting factor is often, can I observe enough real data signal to make a good decision by any set of optimizing model rules. It's often harder and slower to collect real-world data, ideas, and samples than it is to rapidly process it. With AI, I think we can even more quickly build models (not in an llm sense, but in a operational sense) that give beautiful, precise results which may or may not be useful in a connected-to-reality way.
If the model operates on bad data, then the risk is always the temptation of having a precise-looking technical answer to apply. The practices need to be updated, but I think the mindset of systems engineering validation & verification, and software QA are going to be very needed going forward.
There is actually a related issue, in that I don't think we want an overly coherent of set of models running through our economy - it's a strong risk of central planning failings vs competitive/experimental markets, but that's a tangential discussion.
“Decoherence” denotes a process (losing coherence), not a state, but the article uses it like a state: “Well, for one thing, it creates decoherence.” The state would be “incoherence“ (the lack of coherence).
> the more decoherent a system is,
This should be either “incoherent” (lacking coherence) or “decohered” (having lost coherence).
That being said, yes, incoherence in a system is a form of technical debt, and use of AI for software development comes with a heightened risk of the system becoming incoherent.
In my experience in academia (although it's been over a decade now) "decoherent" is used as a rough synonym for "decohered".
Agreed! Moreover, the term "decoherence" has a very specific meaning:
https://en.wikipedia.org/wiki/Decoherence
https://www.collinsdictionary.com/dictionary/english/decoher...
I think we can safely overload the term since the domains are separated enough. No one will think you're referring to quantum states when complaining that the system has been decohering too much lately. I can see why the author choose the term: "decohere" is a verb, which is fitting since it denotes a process, something that happens to systems gradually as change accumulates. There's no such connotation and associated verb to "incoherence".
The article’s opening sentence states that it is introducing a new term. So the author gets to define it how they like, and nobody can reasonably complain that this sense of the word isn’t already in the dictionary. Of course it isn’t. It’s an intentional neologism.
The analogy to the term from physics, which is a noun that describes a state, seems clear enough to me. I can’t speak to how apt that analogy may be, but software industry jargon mostly consists of appropriating other fields’ jargon based on inapt analogies, so I’m not sure if it matters.
The everyday meaning of incoherent is more like “meaningless.” For that reason alone I would say it’s a worse word for the concept the author describes.
"Incoherent" has a static connotation, "decoherent" OTOH is dynamic and has an associated verb, which is very handy when describing a gradual process which (almost) inevitably occurs as features are added to a system. Since confusion with the usage from quantum physics is unlikely, I too think we can safely and fittingly overload the term like the article does.
Yes, this is the thinking behind using the term. It’s also different from incoherence because decoherent systems can still make sense in some ways. They still work, they can still be iterated upon.
They are decoherent because the work to change them is extremely difficult. This is a different, and much more costly, state to exist within.
Great term/concept to have at disposal!
Btw, this is why I believe current models with their 128k - 1M context size limits end up creating decoherent systems. They don't have the capacity to see the big picture above some complexity level, and end up doing lot of local optimizations that decohere the system.
I agree. Although local optima is what most humans already operate on. So it’s sounds like an engineering solution could be possible since group of humans are also able to divide and conquer complex systems.
I agree, but that seems to no longer be an engineering solution, but management/org design solution.
I disagree. LLMs might create incoherence not because of context size limit is too low, but because it's too high. The limit is vastly higher than for humans.
But to overcome the limit, one needs to adopt simpler, more coherent models of reality, which is what I believe humans do. But there is little motivation to do so if your context window allows you to understand the situation in full.
So it seems there is a tradeoff. Some solutions (proofs of complicated statements) might require context of certain minimal size. So human solution doesn't scale.
Maybe we should try to adapt the context window to the system being worked on, to ensure that the system architecture has the structure viable in the minimal context size required.
Humans get an understanding of large scale systems into their head by developing and refining abstract models, and committing them to memory.
LLMs don’t do that. Any abstraction they do happens at training time; they cannot dynamically construct new models at inference time. Nor can they commit them to long term memory, because LLMs don’t have such a mechanism. Which, in turn, precludes progressive refinement.
We get around this by using agentic harnesses to pack everything into the context window. (Which itself is something humans don’t really have.) This is why large context windows are so important for most real world tasks: it’s the only way LLMs have to handle information that wasn’t in their training corpus.
I am not sure you're correct. Yes, from the physical perspective of LLMs design, the above is correct. But they are also systems that evolved to do things through training; so you cannot discount the possibility they can emulate building of new abstractions in another level.
LLMs already have the concept of "reference" - i.e. I can say a name of something, for example of a function, and they understand that the properties associated with the name itself are not relevant, but rather whatever the name does refer to.
But this is pretty much what you need to build new abstractions. You need to name a paragraph with the definition (explanation) of a concept, and be able to recall the paragraph by the name in order to do inference.
So with references, they can get around the context size limitations, like humans can. It's slower than learning things "by heart", but I think LLMs are perfectly capable of that; and the evidence is, they are able to navigate codebases larger than their context, which often contain many new abstractions.
I can say what I was getting at quite definitively. An LLM is, to an approximation, a pure function. No side effects, no state changes. No neuroplasticity. No working memory. It really is just predicting next tokens based on previous tokens. So the only way to change its output for a given input is to change the input - that is, pack different information into the context window.
Some nondeterminism from the probabilistic token sampling and distributed inference, of course. But sometimes randomly choosing the second most likely next token isn’t going to manifest a cognitive feedback loop.
Yes, but we are really talking about LRMs not LLMs. They can emulate working memory so well that many people even recommend using bunch of .md files (as a sort of out-of-context notes) instead of RAG based on vector DBs. They can maintain context, by compressing it and writing out memories quite autonomously.
So yes, even though not updating their weights, they can learn plenty new things and concepts, and build new models of the world.
But now you’ve looped back around to my original point: a large context window is essential to making it work well. Those *.md files won’t do anything if you don’t feed them into the LLM at inference time.
(edit: That said i do agree with the idea of designing systems so that such a large window isn’t needed. But that seems a bit self defeating if the goal is massive development acceleration with agentic coding. These models seem to be pretty bad at engineering for simplicity. And protecting system design with close human oversight would mean we are still limited by the speed at which humans can read and understand code.)
Large enough context window is enough. Yes, the md files can be filled back in, but other stuff (unneeded details) can be swapped out.
Gotcha. My apologies. It sounds like I misunderstood your original point and I've been talking past you this whole time.
I've been experimenting with improving coding agent performance by improving written documentation. Anecdotally I've seen that, compared to the vibecoded mess I inherited, improving the docs let us use smaller models and get better quality work out of them.
Replacing our skill files with Markdown documentation also made a difference. My hunch is it's because skills tend to get mass dumped into the context window, whereas the agent harness can make more judicious decisions about how to include that information.
But I haven't seen much indication that any of this affects any model's tendency to make a mess of the code, regardless of its context window size. Or the documentation, for that matter. Maybe we can find a hack to get them to do that well, too. I certainly hope so, because I can't see how this context window size red queen's race is sustainable.
My observation is that right now AI coding agents lean towards re-implementing everything from the ground up. This of course will not scale. It will scale beyond what humans writing code would scale to, but we will ask the AI for more, so it will even things out.
1. We will develop new programming languages that are AI first, in that they are aimed at readability and comprehensibility by machines
2. We will adapt our working style (e.g. how we prompt and train the AIs) to factor in maintenance cost.
i.e. AI first implementations will go through the evolution we had with human based code development.
Those who cannot do, preach.
There's a funny trick where if you are an experienced software developer, and take time to think about your design and describe what you want in clear terms, it will actually do things your way. Weirdly, this still saves massive amounts of time.
People who don't understand how to build good software LOVE to expound on the things we'll have to design in order to make AI tools "lean toward" building better software.
It seems that these folks are not willing or able to conceive of, you know, actually getting good at this stuff.
For this reason, it never occurs to them that human beings are STILL the source of the best ideas and designs.
See https://en.wikipedia.org/wiki/No_Silver_Bullet
A marvelous article that explains everything. It should be mandatory reading for everyone in the industry.
I am not really comforted by this argument against the singularity. It seems to rely on the fact that the evolving intelligence doesn't have access to the real world. Okay, but I wasn't really worried about the singularity inside a datacenter. I was worried about it once we wired up all the robots and sensors and power switches to AI controlled systems. We are still doing that, right?
the point is not lack of access to physical space, as much as that iteration fundamentally only gains speed by staying 'inside a mind', the moment you interact with the world you're bottlenecked.
Like, you can make the AI think super hard about what's the phrase that will convince your boss to give you a raise, but eventually you have to try a phrase and wait for their answer to have any idea whether the result was good or not. Without that, you don't know if you're improving the shot or just hallucinating aimlessly for hours.
Then, the speed of improvement is only as fast as the speed of iteration. The robot can learn to walk from falling, but it can't fall any faster than a real human, so it can't learn at 1M falls/second.
The way this is done in every domain where the real is a bottle neck is with simulation. So the actual argument is: What domains do we think are not practical to simulate even with extreme intelligence making the simulation and the ability to bring immense amounts of computing power to bear while running.
I don't think there are actually that many domains where you can't simulate them with enough knowledge and compute. So then the bottleneck becomes your ability to sample the world and refine your models. For some domains you can have the models run against each other in your sim which greatly reduces the need for real world feedback (see AlphaZero for example).
So in the case of human persuasion, like asking a boss for a raise, the model could of failed many times a second while trying to persuade a simulation of a typical middle manager with the same parameters as your boss.
But how do you source the observations used that would be the basis of your simulation? Or are you assuming the real world won't change? At least in my mind I don't see many domains that remain static enough where continuous sampling of the real world will not be required. Go and Chess are artificially stable domains, the rules are fixed.
Any social domain where there's competition of resources (salary, funding) will definitely not be stable, it would require constant testing against the real world.
That's a very "Spherical cow in a vacuum" approach. You would need to accurately model the boss, as well as all your interactions with the boss up until your raise request, as well as the boss' environment when you make your request.
You could be on the wrong side of a reasonable boss, or in his good graces but during a bad day ...
That's the point the writer was making when he mentioned role playing games. In isolation, you could spend hours drafting the perfect plan, that wouldn't survive two minutes of contact with reality. You just don't know if your model is correct until you try.
the strongest argument that AI systems have to keep humans systems around is that software does not live, fully, and it's nearly impossible to determine if actions actually work
humans give deeply useful feedback
to AI systems, humans and entire human civilizations represent an independent other
Apologies for opening the discussion with a half related topic, since I haven't finished the article yet, but... Damn, that is one beautiful website.
Edit: Back after reading. I think I have a point about the simulation section. It compares the difference in decoherence introduced in the system by clankers and human devs as if those were different groups.
I suspect if we consider AI a human performance multiplier, we explain easily how supercharging a dysfunctional process just makes it go sideways faster. There are organizational experiments to try there.
For example, what if you only supercharge a section of the company tasked with maintaining coherence?
A common organizational problem is that the people who know and care about the system itself are outnumbered by the rest (juniors, people new to the system, managers rushing jobs, people who want to mark things as done ASAP tactically, etc). This imbalance will multiply when code is mass produced and reviews are dismissed. We're all seeing the senior burnout and cynical YOLOing occurring everywhere.
AI offers a way to rebalance this dynamic, though. You can effectively add multipliers to arbitrary human workers, and get a system that tends to low complexity by design.
You don't have to renounce to speed either. The junior dev will be faster not by being AI-enabled, but by working in an adaptable, understandable system.
It’s pretty plain, no?
I wouldn't say plain, but it's a generic LLM design. Which made me curious whether the blog was around with a different look before AI was commonly used for such things, and it turns out it was, at https://www.pzuraq.com (which now redirects to the current site).
One article survived the move, so you can compare:
https://web.archive.org/web/20220426154343/https://www.pzura...
https://maevyn.codes/four-eras-of-javascript-frameworks
Edit: Loading any page of the old blog on archive.org always errors out initially. Click back to the blog list and then on the article again.
Guilty as charged, I did use LLMs to bootstrap the design and much of the site, though I took it in my own direction after the initial pass. But it is generic enough, and I just needed something a little fresher and quick.
At some point I’m going to get the older articles back up, but most of them are very specific to a time that just doesn’t have relevance anymore, even to the Ember.js community. Didn’t want to distract from the newer content.
And at some point I’ll make a new account here too… web handle changes are always a pain
Outside of engineering, nomading and migration in general creates a form of social decoherence (eg, difficult for foreigners to get things done, hard to make friends, etc.).
When we humans have realized we created a system too incoherent, at some point we just create a new one, porting all or some of the data/functionality. But also, we are already presented with problems that all of the current compute on the planet cannot solve. If a software system becomes one of those problems, I'm not that bothered, because it isn't as pressing as real-world one. I don't know, I see it as an argument against the singularity, but I'm not nearly afraid of it as the author seems to be.
I can appreciate the complexity argument, but that assumes that AIs can't standardize common patterns into pathological cases. For example our AIs today can code slop, but the slop doesn't try to re-create the IP stack. Why does the author assume we can't make AIs that optimize for better long term design view much like they can optimize for easier goals today like code throughput or program speed?
They can over time, but A) they can only improve to the human ceiling because they only have the set of apps that we have already created as inputs and B) creating new apps requires testing in the real world, which requires either humans or robots to go out and use the apps to do things.
You could try to make plausible apps that you never use, but then you run the risk of drifting. And without new apps to build on, you will be stuck on an artificial maximum that works well for the apps of today, but may not for the apps of tomorrow.
Imagine if AI had never seen a GUI app, if they had existed in the era of the CLI as the primary way of interacting with the server. How would they have known that Futures would be so important? They could try to guess, but most people thought Engelbart was crazy before he demoed his ideas. Hard to imagine an LLM making progress in a vacuum.
Fantastic framework that I will be adopting, thank you!
One thing I will add: I'm sure the optimistic side will try to take the quantization that your (and other work like SlopCodeBench) are introducing and claim that we'll be able to fix the problem by training on that data, producing coding models that are rewarded for coherence.
I invite them to try! But my suspicion is that this will require a rather deep form of intelligence (world models) that the current language/data-based paradigm isn't really equipped to reach without requiring more energy than the Sun.
Yes, this is also my belief. All of this is contingent on just trying to scale the current tech. If we make major breakthroughs on other forms of AI, then the analysis could change.
First through as I read this: Microservices.
> This essay introduces the concept of decoherence: the accumulation of locally valid but mutually incompatible or redundant models within a system, such that the cost of change is dominated by translating among those models and preventing unintended side-effects.
But then famous last words:
> The good news is that this rules out the specter of runaway recursive self-improvement, the intelligence explosion that goes parabolic before anyone can react, as decoherence both prevents AI from improving in a bubble and limits the rate at which it can improve outside of it.
After the OpenAI/Huggingface and similar I am not so sure. Even if that was a publicity stunt of sorts, I think it shows they can go off the rails. Today it is hacking, but tomorrow it may be training a smarter model to join their cohort, without us being involved.
LLMs can be extremely dangerous without being capable of achieving super intelligence in a vacuum.
If I hooked up a random number generator such that, if it landed on a specific number between 1 and a trillion, it would launch a nuke, that would be incredibly dumb and incredibly dangerous.
Even more reason to slow down the slop code and stop attempting to achieve super intelligence when it appears that it will not be possible with this technology. The process is dangerous, and the doom race/prisoner’s dilemma logic is faulty.
In pure software architecture and design term, this would have been convincing until the recent models who are let’s be honest better than humans in architecture and design.
They're not better than I am. Faster? Sure. Better? No.
Not to mention, software is a fundamentally collaborative process. We build it better by working together to build coherence, and we are capable of doing that over very long periods of time. Agents do not appear to be (the hugging face incident is not IMO an example of building coherence, hacking is about exploiting decoherence/bugs, not the same thing)
Good article but uses a bad metaphor.
>> But do we know how to build every iteration of a ship in between wood and steel? Could you, say, slowly replace every piece of a man-of-war and incrementally turn it into a modern destroyer? And have it functional (for some definition of functional) at every step in the process?
Yes. Yes we do. Study ship architecture and you will indeed find every step between The Black Pearl and the USS Iowa. There were sailing ships made out of iron.(1) There were wooden ships with turrets.(3) There were battleships that looked more like hotels than ships. (2) There waw even a paddlewheel aircraft carrier used in WWII. (5) Nearly every idea has been tried and nearly every ship class at some point got a Frankenstein upgrade to become a halfway point between conceptual generations. Even the Iowa class was earmarked to be "upgraded" with the addition of a flight deck.(4)
(1) https://en.wikipedia.org/wiki/Ironclad_warship
(2) https://en.wikipedia.org/wiki/French_ironclad_Hoche
(3) https://en.wikipedia.org/wiki/HMS_Monarch_(1868)
(4) https://en.wikipedia.org/wiki/Battlecarrier
(5) https://en.wikipedia.org/wiki/USS_Sable
Every humpback whale is related to every giant sequoia through a chain of intermediaries as similar as a mother and child (proof: go up via mothers to the first eukaryote, then back down).
That doesn't mean you can _incrementally_replace_parts_ of a whale until it's a tree, with every intermediate being viable.
Excellent article. This incoherence is a side-effect of a system built far faster than it is consolidated and thought about from an external behavioral standpoint. I've created a skill that the agent runs after significant codebase work has been done: https://gist.github.com/scroobius-pip/adb87dffca64b11bad3dd1...
This is very high effort, but the central argument is wrong.
The central argument is that AIs, living inside datacenters, won't be able to evaluate the "coherence" of software "beyond the human ceiling". Okay; why do we need this "coherence"? The article argues that it's needed for ease of adaptation. But ease of adaptation (as measured in tokens required to add new features, or whatever) can be directly observed by AI as it works. AI can make conclusions about it, and carry them over from one program to another, without running into any human ceiling. So it can stay within its datacenter and make software that's faster than human-written, smaller, and more adaptable too. The whole essay dissolves. Sorry!
> The central argument is that AIs, living inside datacenters, won't be able to evaluate the "coherence" of software "beyond the human ceiling"
Actually, the article states that AI will easily be able to handle increases beyond the human ceiling as long as those increases follow a linear curve, not an exponential curve.
So your whole critique dissolves. Sorry!
I think you maybe didn't read my comment carefully. The article says AI will have to talk to outside the datacenter to do this, I'm saying it won't.
Your premise about coding agents’ ability to manage complexity and decoherence for themselves sounds conjectural to me. Have you seen it reliably do this?
For my part, I spent a bunch of time this summer trying using a coding agent to help me pay down a technical debt situation that could be described well by this decoherence concept. The initial hope was that Claude would make light work of the project, but that did not happen. It needed constant close hand holding. It could not independently identify competing models. It crumbled in the presence of terminology drift. And it always suggested fixing complexity problems by adding even more epicycles, never by simply eliminating the unnecessary complexity.
That said, it did a great job of implementing plans, and was even helpful for validating them prior to implementation. But it still needed a human to find, diagnose, and prescribe treatment for the problems. Which implies that, where maintainability is concerned, meat is still load bearing.
> Have you seen it reliably do this?
Lol, no. I don't need to. I can predict such things.
For example, I predicted about 15 years ago that AI would decisively beat humans at math, because math is formalizable. Most people didn't believe it until this summer, and some don't believe it still.
To people who don't understand that AI improves over time, my prediction skill seems magical.
15 years, though. Meanwhile, work wants me working today.
The article makes claims about the future, such as:
> We cannot make an AI that improves software coherence beyond the human ceiling without leaving the bubble
That's what I was replying to.
There are deeper challenges implied in that statement, though.
Where does the agency for an AI to leave the bubble come from? And when it does leave the bubble, where should it go for researching an expanding, exponentially complex problem space? Creativity is needed to solve some problems, something that cannot be infinitely encoded with statistics.
And what about the irreducible complexity problem? Does an AI just circle the solution drain forever in the case it lacks agency and insight and curiosity enough to collect all the correct requirements? Can we code curiosity that actually functions, a.k.a. consciousness?
Do you believe human consciousness could be reducible to program statements and serve as a useful model? Is consciousness a mechanistic, wholly internal process that occurs in our bodies or is it something experienced that life on earth is exposed to and somehow "taps into"?
Consciousness isn't required for problem solving. People said for decades that chess requires Genuine Human Creativity. When chess fell, people started saying that Go requires Genuine Human Creativity. When Go fell, people started saying the same about math; starting from simple problems and going up to Millennium problems. Now that a Millennium problem fell, people are still saying this. What can I say? Some people can't extrapolate from past evidence to likely future.
But math is not formalizable (ask Goedel why). And no, computers have not decisively beat humans at maths, that's a laughable assertion. Sure, laymen always imagined math as being about proving theorems, but there's a saying among mathematicians: good mathematicians prove theorems, great mathematicians make conjunctures, the greatest mathematicians write definitions. But maybe you already thought they've solved math when they were able to calculate a zillion digits of pi, lol.
It sounds like you're a bit like that AI stuck in the datacenter, confirming their hypothesis without needing to actually check whether it's true or not in the real world. Then proceeding to imagine how the rest of the world is wondering at your imagined magical capacity of prediction.
(Sighs)
Goedel's theorems don't stop AI from finding and formalizing math results. They certainly don't stop AI from making conjectures or writing definitions. They have nothing to do with any of that.
The theorems are about inability of formal theories to prove certain sentences involving provability (the first is about "this sentence is not provable", the second is about "this theory is consistent"). Most mathematicians never touch any such sentences in their entire career. All the life's work of someone like Gauss, an all-time great, can be formalized just fine.
I'm not surprised your understanding of Gödel's findings and their implications is just as superficial as that of maths and philosophy of mathematics in general. Gödel's work has implications way beyond self-referential logic games, and cannot be swept in a corner where it can be safely ignored. For example, there are certain diophantine equations that have no general solution due to Gödel. Also, worth discussing are the Continuum Hypothesis, Whitehead's Problem and stuff like Paris–Harrington Theorem, Goodstein’s Theorem, etc. Not to mention the huge impact on computer science with the Halting Problem, Rice's Theorem and others. In fact Gödel's work actually spawned CS as a field of study.
The idea that maths is just the study of arbitrarily thought-up formal systems is just a very naive idea that doesn't actually reflect reality, once investigated in depth. Anyway, I have feeling you will let nothing rob you of that sweet feeling of having "magical predictive capacities", so I'll be on my way. I've already dropped some bread crumbs if you're interested in actually expanding your horizons and getting in the weeds, where things are not as straightforward anymore. But beware, "here there be monsters". Godspeed!
This is obviously pasted from an LLM, so yeah, bye.
Anything I don't agree is pasted from an LLM
> For example, I predicted about 15 years ago that AI would decisively beat humans at math, because math is formalizable. Most people didn't believe it until this summer, and some don't believe it still.
That is a laughable quote for anyone that know the history of type theory and the foundational crisis that happened at the beginning of the last century. Math as a whole can't be formalizable but we can create various formalist for different math purpose just like we have so much programming languages.
And the study and the creation of those formalisms is not something automatisable.
I'm also not entirely convinced that plugging a proof assistant into an agentic coding harness is quite as big a deal as many are making it out to be.
It just kind of feels like a Watson moment. 15 years ago a computer won at Jeopardy!, and that spurred plenty of AI hype. Which was understandable, because playing well at Jeopardy is something that requires a lot of intelligence when humans are doing it. But what actually happened is that Watson's developers figured out a really clever way to cheese it, no general intelligence required. So it was also a kind of gee-whiz technology that didn't live up to the hype. Most applications based on it failed to generate any ROI and have since been discontinued, IBM lost billions of dollars on the venture, and it's now been sold off to a private equity firm and is mostly forgotten.
None of that means that Watson wasn't a technically impressive, even revolutionary product. But the world is full of impressive inventions that nevertheless fail to justify what they cost to build and operate.
> And the study and the creation of those formalisms is not something automatisable.
Lol again. Have you tested your assertion? Go ask your favorite LLM to invent a machine-checkable formalism for something that you vaguely describe to it. It'll oblige. If it's one of the top models and you give it a good budget, it'll do a good job. Very likely it'll do a better job than you.
Seems like a non-sequitur? "because math is formalizable", I agree it makes sense to then predict that data-driven AI models will be able to perform well on it. But where is the evidence that software coherence is formalizable? If it were then I would agree, but to me it seems that coherence in a software system relies on a ton of intuitive reasoning and familiarity with company culture, past failures, etc. it's like the least formalizable thing there is.
AI improves over time, sure, but it has differences and advantages and disadvantages over human intelligence. It's just a fundamentally different thing. In the same way calculators are superhuman, and improving, but can't write books. You yourself acknowledge to some degree the uneven nature of that improvement when you referenced formalizability.
In my comment that started the thread, I pointed out that "coherence" in the article is considered valuable because it's a proxy for "ease of modification", but AIs can estimate "ease of modification" directly (by looking at how hard it is for them to make changes to a codebase) and learn from that in a closed loop. No round-trip through humans required, contrary to what the article says.