No this isn't mindless reinventing --- this is finally having clear empiric evidence for something that we knew the entire time. This is a huge relief!
Next up is demonstrating the AI is more productive with better programming languages.
> This raises a question the programming-language community has not previously had reason to ask: should error-message detail be calibrated differently for AI agents than for humans?
> We investigate this question through a controlled experiment using Shplait, an ML-style statically typed language. We construct a suite of programs containing a single deliberate type error each, and measure how often an AI agent repairs them under ablation: a detailed error context using the unification stack; a proximate error location; a minimal type error; and a dynamic (test suite) error only. An automated oracle uses a test suite to classify each repair attempt as a type error, semantically incorrect, or semantically correct.
> We find concrete evidence that more detailed error messages generally improve an agent's ability to fix type errors. We also find that the presence of a type system appears to help more than only test suite failure reports.
As an example, 110v American outlet plugs are almost certainly not the most efficient way to power devices and appliances; but, because they are so standard and are good enough, using them massively eases the burden of making, buying and selling powered appliances in the US. Similar story with USB C.
Outlets benefit from standards, however. Every device (well kind of) uses the same few plug types, and a different plug would be a disadvantage.
On the other hand, having a bunch of different languages is an advantage. People can pick the best tool for the job, and experiment with new language features.
True, and to my point, "popular" isn't necessarily "better".
> because they are so standard and are good enough, using them massively eases the burden of making, buying and selling powered appliances
Yes, but LLMs can do the heavy lifting when analyzing the "better programming languages" for productivity. There will be a bias toward popularity in their training sources, though.
You might be surprised. I've had an AI start spitting out none functioning python code. I've rewritten it with Go. The requirements are that every change has to pass compilation, linting and tests. Same requirements as Python. Go keeps the AI on the rails better. The compilation failing provides quick and probably better feedback than a whole dictionary of tests.
> Next up is demonstrating the AI is more productive with better programming languages.
This is hard to do given the current reliance on model weights in its programming language knowledge. You’ll get your best results for python or some variation of python. But
I’m not really sure how to read those charts, but it’s completely possible that python did not work well for the approach they were trying.
I would be the first person to give up python for kotlin or some other more natively typed higher performing language. But the python bias is so far too big to ignore.
(It's an interesting question though - my prediction would be that AI is best in the languages it consumed the most examples of, and I reckon the size of the StackOverflow site for each language would be a rough proxy for that).
To be clear, the guy who wrote this article is Martin Fowler. Fowler was the author of a book literally titled "Refactoring", which he wrote over 20 years ago (it didn't invent the term "refactoring", but it's widely credited as popularizing it).
In other words, this is not some new AI-bandwagon-jumping blogger trying to rebrand old practices as something new ... this is the guy who coined the very term "refactoring"! He's not pretending what's old is new, he's arguing that best practices from 20+ years ago remain relevant (and showing receipts).
It’s an easy mistake to make, given the name of the website, but Fowler frequently hosts guest authors. This article was written by Giles Edwards-Alexander, ThoughtWorks CTO for EMEA.
It’s much easier to get your AI agents to do something consistently than it is to get your human colleagues to do it. Heck, personally we know these things are the right things to do, but we are just too busy and our minds wander to what we think our more productive uses of our time. That completely changes when it’s agents being instructed to do the work instead.
The crazy thing is that it s looking like AI might be able to write software better than humans eventually simply because they do not get bored of doing tedious tasks. Many things that we know work but don’t practice because humans aren’t very scriptable are now viable and can be easily applied with agents. For example, I’m finding a lot of success in using separate agents to write implementation and tests from a common specification and then using an auditor to run the tests so neither agent is contaminated by the other’s work. This is just part of the clean room engineering process that was developed for people by IBM in the 1980s, and it was shown useful then, but with AI it can be widely and consistently applied.
They'd be 10x better than us already if task tedium was the problem. It's design sense that they're missing.
I find that in the areas where people think that LLMs excel at coding and don't like doing manually it's usually because the human was inclined to slop out repetitive boilerplate and thought that was the only way. Tests are usually like this, sadly.
LLMs are definitely good at providing reams of duct tape (which is drudge work) to patch up those bits of the code base where the code sucks. The problem is that duct tape is not the most architecturally sound construction material.
> They'd be 10x better than us already if task tedium was the problem... the areas where people think that LLMs excel at coding and don't like doing manually it's usually because the human was inclined to slop out repetitive boilerplate and thought that was the only way.
There is so much work out there that is repetitive / boilerplate / tedium. If you get to personally work on interesting work more than 50% of the time (pre-LLM) I'd say your job is #blessed.
IME if a technical task is tedious and repetitive it is nearly always because the system was badly designed or because it wasnt automated properly.
My job is automation and system design, so if I can't fix or work around these things that reflects poorly upon my skills.
Some people treat writing tests as inherently boring because their test frameworks usually suck. If they don't suck and a test is a very close approximation of a spec, it's not boring at all, especially if you use it as a means of codifying a spec before implementation.
Don't disagree that that's the system not working well.
You're lucky to be in a system that works out for you. Not everyone is in such a lucky position. LLMs help automate the tedium out of their job, and hopefully, have a bit more extra energy to make a better system within their immediate locus of control.
The thing is, humans make do. Given lacking, outdated documentation somewhere on sharepoint, big picture revealed somewhere in passing at some meetings or water cooler talks, underprioritizing refactoring humans still deliver something even if at reduced quality and timelines.
AIs either execute extremely poorly or don't work at all without these. Decent engineering practices are suddenly a hard prerequisite with AIs, not just some longer term improvement.
On a related note, this is unironically good argument to integrate AIs into the workflow even if there's zero net benefit in your use case. It's a perfect excuse to bring the proper engineering practices in.
There is a big difference pre and post AI, namely that humans have quite good long term context management.
But now that everyone is coding with AI, all agents need to acquire context every session. Thus the value of doing best practices is much higher and benefits are there immediately.
I really enjoy refactoring. Like, doing it by hand (5 miles uphill in the snow both ways) rather than AI. I don't even understand why I enjoy it, because if done right, there's no visible change. When people ask me why I'm so pumped up after having worked on the codebase all day long, I can't give them any sort of answer that makes sense (coming from a small business with a smaller dev team). It's just... "I'm future-proofing our website and we won't see any direct results today but things will be so much easier going forward..."
There's something about the puzzle. Looking at my old, deranged coding workarounds that tried to solve problems that have already been solved a thousand times before with established paradigms; and then moving them toward said best practices; and doing it in a way that no NEW technical debt is created. It's just satisfying.
I think one of the best learning experiences for me has been the fact that I created a bunch of sloppy shit by hand, auth and all, which forced me to learn things the hard way. All along the way, people were shouting from the rooftops: "use established libraries, dummy!", which is the same advice I would give to somebody today. But by doing things the hard way, I learned so much more about the inner workings. And, I've also given myself a decade's worth of refactoring work, which I really enjoy!
I have often done huge refactors over the last 20 years on systems where there are none. It's very satisfying turning a 1,000 line class file into 100 lines.
But all my refactors have been in statically typed languages.
To speculate on the reason why, which the GP asked, I think perhaps it is two things. It's the same satisfaction from tidying a room or your workbench. But it's also slightly narcissistic unfortunately. I think you've shown you're better than the other programmers. Much, much, better when you end up with massive reductions in TLOC.
And with tests, the irony is that the resulting code is often so much easier to reason about, you start spotting really obvious bad assumptions in the original implementation.
It's one of the reasons I've always been fairly skeptical about the true value of unit tests (integration, I get). That and the fact that projects I've worked on that did have unit tests catch like 1 bug a year. Maybe it's just the size of systems I traditionally work on (smaller teams, or even 1 person teams, so man-years worth of effort rather than decades or centuries).
Me as well. One of my formative experiences was reading Refactoring (Fowler). At the time I felt skeptical that it could make a difference --- I had mostly written scientific/research code at this point. I decided as an experiment to apply it to a dissatisfaction I had with the structure of a codebase and it was a revelation. It really made it really concrete what people meant by "a codebase is a system", made me see code from a higher level, almost like a continuous tissue or web you could pull and prod.
It's quite sad to think this experience might be on its way out, and it really highlights that "junior problem" from AI. There is just no way around diving in to get an intuition for things. Naur warned us of this 40 years ago but the lesson never seems to get learned.
Ha, many years back I had to write a template processor for some system configuration handling. Manager told me last guy did most of the work so test and wrap it up. I instead had to throw it out as it was so crappy, wrote much smaller code, with no dependency and really put soul in logging where it showed cleanly how tool is navigating all directories, picking placeholders and replacing with values and so on.
No one really cared but it satisfied me and make me really happy.
This is such a nice piece, this is how people should write about AI. Specific, grounded to how the tools are actually being used, and quantitative. There is so much bad AI commentary that is incredibly vague, divorced from any actual use cases, or written by people who don't actually use the tools. It is good to see a critique that is "here is a thing AI is bad at and measurements to show it" rather than gesturing "here is why I think AI is problematic for society."
It is rather different but another piece of research I liked for the same reason was this report that interviewed Boko Haram members about how they used AI to assist terrorism. You get these interminable online debates that are so unproductive and reporting that is specific is such a breath of fresh air.
I think this is one of those things where a human in the loop is indispensable
An agentic refactoring pass does make sense cause one LLM reviewing work can spot things the ‘generator’ LLM missed while focused on the initial task output
But can the reviewer agent ever actually have in mind what this project actually is? And how the code all comes together to do the work involved? In other words what parts of the code are redundant or can be made more elegant
Asking coding agents to refactor your codebase is maybe like asking trauma surgeons to increase your exercise capacity. The agents are gonna need a really holistic POV to do this properly
I guess part of my point is that just splitting big files into multiple files is only refactoring in a superficial sense without having a theory of what code belongs together and what can be extracted into utility functions etc. Is splitting files actually like decomposing factors or is it like splitting a larger number into smaller numbers that still eventually get added together
A good example of what I mean is that agents often don’t ~actually~ understand the whole system anyway. They might implement a system to store and calculate something that is already being fetched via API. Humans often have a dual perspective — a holistic sense of the project and (when applying our mind to a task) a precise scalpel: ‘oh if we just look at this this JSON it has a key with this data already’
I was able to largely refactor something that was going to take months in a week or so by being the narrating human in the loop.
It was helped by having refactoring experience and approaches to codebases by others, and in my case, being the original architect and being able to speak to the original and current intents, where needed.
This was using a less common, but capable and easy language for the LLM without a ton of dependancy brittleness to manage.. once the effort to remove javascript/python bias was in place, it became so powerful that once the lightbulb went on, it really got cruising.
The project was playing in the world of JSR-223 languages, where you could script in many popular languages, but it all got to run in the JVM, which was an environmental requirement.
Your take is outdated. Agents are excellent at refactoring now. If you told me what you wrote during an interview, that would be grounds to not continue.
It's very important that you are using coding agents with the latest frontier models and know exactly what it can't and can't do if you want to be hired in this market.
"Refactor this code spin up subagents" try that in codex to start your journey of improving your agentic coding skills but if you are too lazy to do even then there's just no hope for you.
I am currently working 3 remote jobs thanks to agentic coding and one of the jobs require me to interview and hire other devs. I find it amusing how many are irrationaly resistant and unwilling to use agentic coding. They are also the loudest in the room with opinions that are outdated. On the flip side, it's just as funny how to see how complicated engineers try to make agentic coding. It doesn't have to be and that's not its design. Don't get in its way as much as possible. The frontier models are good enough that adding complexity only wastes more tokens. We've come a long way from the summer you were fiddling with Sonnet 4.5 in Cursor or windsurf.
The job market isn't suddenly going to swoop in and save you by banning AI agents. Keep up or get left behind. Don't let other people's opinions about AI agents hinder your own progress, chances are they have no skin in the game.
> "Refactor this code spin up subagents" try that in codex
The deferral of all agency as a product owner over to the stochastic-word-box-in-a-do-loop feels like the kind of thing that, if you mentioned you do that in an interview, would be grounds to not continue.
Re: your updated comment, I'm glad you've found success with this pattern of behavior, but I'd suggest that you might be extrapolating "I haven't been punished yet for this shortcut" to "This is the future of work and those who disagree are luddites".
lol - I guess, if I heard that in an interview that would be grounds not to continue.
"enhance"
fwiw codex is very good. arguably better than the sonnet equivalent (value-wise). I use both daily, professionally. And "refactor this code spin up subagents" is hardly an expert-level "I wouldn't hire you" flex.
EDIT: agree with your comment updates. I'd just say it more positively leaning especially to people who are stressed financially. SOTA has advanced a lot so counter-intuitively being behind isn't really behind, it can almost be a way to leap-frog, lean on the Agents. But definitely you have to commit to putting in the reps and learning and not just mindlessly instructing AI to "fix it". Think that's where the discussion splits.
> Asking coding agents to refactor your codebase is maybe like asking trauma surgeons to increase your exercise capacity. The agents are gonna need a really holistic POV to do this properly
If you ask it to implement a specific kind of refactoring over a specific section of code it seems that current LLMs can do just fine. Even things as complex as "use `functools.partial` to implement the Command pattern here, rather than dataclasses".
> Is splitting files actually like decomposing factors or is it like splitting a larger number into smaller numbers that still eventually get added together
The file boundaries represent logical subsystem boundaries in the code, making it easier to reason about. The training data abundantly represents the idea of treating the contents of another file as opaque by default while other functionality in the same file can be used freely. I think it's reasonable to assume that there's something objective about the benefits that humans get from this, and it isn't just a consequence of how human cognition works.
The benefits go beyond reducing token consumption. Compact contexts also foster better reasoning, enable intelligence across more layers if you can load them in a single context, and writing software to enable this, results in more correct software, software that _generalizes_, that has higher probability of being correct not just for tested cases, but for the interpolations and extrapolations of these cases.
Refactoring towards good abstractions is more powerful than people realize. There's information theoretic bayesian math to back this up.
It's a bit of a divine coincidence that software that is more economically and energy efficient to process and run tends to also be more correct.
Imagine what happens when human readability is discarded as a goal. LLMs are already very good at inferring meaning with little context. If the objective function is reduction of token consumption, it's hard to know where we will end up.
I've been budgeting myself explicit "slop removal" time. In fact I'm in it right now one window to my right here.
I still get a big win from AI on the net, but you do need to budget some time to clean up. I'm still on team "read every line".
In fact this is a case where I deliberately deferred some review because I was a blocker for another team. Now that I've got something to them I'm going back and I'm going to eat a bigger chunk of debt than I normally would, but it's worth it for unblocking the other team sooner. AI has made tech debt easier to take out, in all senses of that term.
It's totally worth doing. Especially since it only takes one or two manual improvements before you can just say "ok, replicate this new pattern throughout the codebase. Go."
Great piece, but it misses the elephant in the room: the lion's share of economic benefit from refactoring will come from the fact that it makes it easier for humans to understand. That means 3am pages get resolved faster, fewer bugs will end up in prod, and your team can ship faster than your competition - gaining a leg up in the market. Most importantly, folks will be more comfortable accepting responsibility and ownership of a system when they understand it... this means that if/when something goes wrong, people will more quickly jump in and fix it, and when things could be better, folks will jump in and improve it.
the punchline: "Refactoring reduces token consumption"
I appreciate the effort to quantify the benefit rather than pontificate. It's worth mentioning Martin Fowler wrote a whole book on refactoring [1], in which he states, "to refactor, the essential precondition is [...] solid tests", which I think is the real benefit here, AI or not. Good tests protect against regressions, whether human or robot. They also help encode the spec, which humans and robots can read.
I think the real punchline was that the value saved was on the order of cents!
> Every single change that touches the data access layer from this point forward now costs significantly less.
> How much of a saving? Assuming Sonnet 5 pricing at the time of writing of $3/MTok, 39.7 cents.
Now consider OpenAI's price drop, and open models, and consider that in the long run tokens will get cheaper. And think that the refactor needs to be human guided at a price of what for a senior developer - $100/hour?
Tangent, but I have a theory that refactoring is one of the best symptoms of a healthy dev team.
It's partly that refactors themselves have benefits, but I think more that the benefits to refactoring aren't visible to something like product-owners, feature tickets, etc.
If teams are refactoring to ensure the health of the overall software, it's a tell-tale sign that developers are happy making recommendations for good software, and that those recommendations are being taken seriously.
I think Martin Folwer might have actually coined the term "software rot" - either way, as an issue it happens most severely when a team either aren't motivated or empowered to build their vision of high quality software. When a team can follow their judgement of excellence, that's usually a great sign!
(and yes, obviously this can go to far, there are probably some teams who rewrote all their stuff in Ruby then Node then Rust and now something else to be "agent native", but in the coorporate world, I see a lot less of this than teams who just don't feel like they have permission to improve things)
Agent code can only be read and understood by agents. We’ve reached the point where people just can’t comprehend these giant code blobs.
Feature, bug or emergent property?
I don’t think the distinction matters as much as the reality.
We’re being locked into using the AI tooling bc the code was generated with AI tooling.
These giant files of doom were being generated by humans anyway and were very hard to work with. With LLMs it’s at last manageable or feasible to edit, refactor etc.
I honestly think LLMs are going to save us from ourselves as the codebases became too large and “messy” for humans to comprehend. (Mono repos of doom)
On a personal level these giant files are abhorrent but that’s just personal taste and I don’t think any of the Martin Fowler refactor/cleanup stuff is going to matter at all anymore. Kinda sad on some level.
There was always bad code. Extend that to the general challenge of a bad hire doing net-negative things for the company. AI is a new problem because it 1000x a bad hire's blast radius. Personally, I'd even state that AI tends to turn an average and sometimes-but-not-always-good hire into bad hires because finally they get to do the thing they've always wanted at lightning speed in a way that previously the company red-tape wouldn't allow.
Clearly not true. I got codebases that are 100% done by LLM. I have no problem reading the code base and find where things are. Not harder to find stuff than when I done it manually.
Interesting that the amount of code remained essentially unchanged. In my experience it is not unusual that refactoring messy code cuts the number of lines in half.
I don’t know. LOC to me is indispensable for certain refactoring goals in my opinion.
Taking a component and turning it from 7K lines to 3K lines and maintaining functionality obviously means there’s less complexity introduced, less to go wrong now, and less overhead to modify in the future.
Sure it can go the other way, the component needs to support something it might need, we need to adjust larger patterns, this function needs to be refactored into something more robust.
But lines of code is a pretty decent metric of success for “trimming down and cleaning up” style refactoring, to me at least. It’s not everything of course, but it’s definitely an indicator.
LOC can be one measure, but not always the sole or best ones.
Since LLMs are word generators, and have a propensity to generating words, they need to be shaped to understand simplest is best, more isn't more, and less isn't more always.
Trimming down and cleaning up could be formatting, standardization, commenting, or even some basic re-architecting that was overdue.
One of the biggest benefits of llms for refactoring I'm finding is reducing technical debt.
That's LOC in a single file, that's a bit different from total LOC as a metric.
Total LOC is a garbage metric. Things like reducing line count in specific files or components is a big benefit, but those lines are often moved, not dropped.
A couple of days ago I split a single class which did 5 barely-connected things, where each of the 5 entry points had another 5-10 private helper methods (none of which were shared between entry points), into 5 separate classes which each do one thing - total number of lines went up (because they all have the same boilerplate import statements at the top), but being able to focus on one entry point and it's helpers at a time has made things much easier to work with
I did not mean it as a metric but as an indicator what was wrong with the code.
- the code is essentially good but all is in one file, you split it up, lines of code stay the same
- the code is essentially good but lacks some structure, for a function that does five things directly, you extract the functionality into five functions and call them from the original function, lines of code goes very slightly up
But once the code is actually bad - code duplication, bad abstractions, inefficient language use, ... - I would generally expect the lines of code to significantly drop. What scenarios are there where the code is actually bad but refactoring does not reduce the lines of code? It is certainly possible but at moment I am having a hard time comming up with a good example.
> What scenarios are there where the code is actually bad but refactoring does not reduce the lines of code?
IMX: the scenarios where people hold higher standards than what you describe :) (i.e. such that "the function does five things" is deemed "actually bad". Of course, that does depend on the refactoring not causing an unacceptable performance hit, which can happen depending on the environment.)
Don't get me wrong, a function doing five things is bad. I consider code not good if you write if(condition) instead of if (condition) or a +b instead of a + b. Not that it is really bad on its own, but it indicates to me that the code was written with little care and I should probably expect bigger issues.
What I really want to get at is the distinction between leaving the actual code untouched and just moving it a bit around - to other functions, other classes, other files - and having to change the code - from deduplicating to completely rewritting it.
Yeah, I sorta expected the punchline to be "this 17,000 lines of code became 2,000" but it is still about the same quantity. Kinda interesting. Also interesting that the last change has by far the largest effect (4x reduction in tokens) and that also corresponded to the biggest reduction in single file LOC
Yep, this was a real surprise to me, too. That’s why I included that top line in the figure. It also doesn’t line up with the agent’s predictions when it laid out the refactoring before proceeding.
There’s further work to do to understand exactly what’s going on here.
After reading a fair amount of OSS code, it's rather glaring that the data structures and algorithms are more like atoms than molecules in software design. We do have some molecules in the Design patterns, but they are more suitable to the OOP universe, like the collection api (filter, map, take,...) when dealing with groups of objects or the reactive api for dealing with concurrent tasks.
Most of software development is looking at some process and then decomposing it recursively until you get to those molecules/atoms of the computing world. Coding them is trivial, and while you can gain a certain boost from the AI, after a while you no longer have to write that much code. It will turn into a balancing act where the introduction of a new concept has to be done carefully.
I have a blanket, unignorable 'files must be <= 1000 lines' lint for my Rust projects for exactly this reason. Exploring large files is _costly_ for agents, so it's much better to lean into the filesystem hierarchy to a greater extent than you would normally with a team of human developers.
This is interesting, but I think it would be even more interesting to see a comparison of the token cost of adding a new feature, in the original codebase vs the refactored one. You'd think/hope that the cost of adding the new feature would be lower when starting from the refactored codebase, indicating that refactoring makes economic sense in the long run by lowering the cost of adding new features.
That's what I've shown in this article. The token consumption shown is for adding a new feature (the same feature each time -- in a sub-agent) after every refactoring is applied. I didn't actually capture the token cost of the refactoring. That was a miss, and I'm making sure I do that in the future.
Kind of you to engage! Ahh, ok, I misunderstood. Very cool! I think maybe it would have helped me if you had given a one-or-two-sentence description of the "representative change" in the main text, to make it a bit more concrete up-front? But possibly just a 'me' issue...
It’s interesting that cyclomatic complexity or cognitive complexity scales with token usage. A codebase that enforces upper bounds on one of these two also (potentially) helps AI agents stay efficient.
Can't agree more. Feels like LLMs are not able to understand your requirements at a high level and always add complexity by default. Really need a handbook to guide them to do those cleanups. Obviously they know the knowledge in their weights but they just don't know how to apply it.
The author said that Claude was not good at refactoring. But the author was using Sonnet 5. I supect Opus 5 would have done a lot better at refactoring. It would have come up with a multstep approach, i suspect.
I used Opus 4.8 to create the refactoring plans. I converted token savings using Sonnet 5 pricing at the time of writing as I'd expect to use that for more 'straightforward' feature additions.
I can very much relate to the experiences described in the article. In my projects, I have multiple Claude skills and rules aiming at making my files (the code, but also the Claude files themselves) more token-effective. I have seen huge differences in token usage before and after invoking those skills, which I regularly do as part of having Claude audit my projects; although I haven't measured them. Those projects are (like the one in the article) fully developed by Claude Code, so I found such audits and mindful token usage very necessary.
Opus/Fable 5 (or really, any thinking model) do a better job of refactoring than Sonnet would. I think the takeaways are still valid, just not as valid when using a more expensive model at a higher effort level.
I think the name of the game is going to be "how do we maximize output while keeping token counts low". For example, finding ways to replace AI workflows, even if minor ones, with scripts/code.
Anthropic told me that I have till August 19th to be get my act together becuase they are going to reduce my token count by 50%. lol. I have been abusing my Pro Max allowance and need to start being less wasteful.
Articles like this, can help us come up with ideas on how to do it.
Great point. Using good software architecture and clean code will deliver direct and measurable economic value to projects and reduce AI costs right from the start, whether during refactoring or when planning a logical reorganization.
Being a bit cheeky here -- the amount of comments on this post is a telltale sign of how common / desirable refactoring is for the majority of developers (xD). It isn't our fault though, maybe if those pesky managers read this article then they would understand..
I've been experimenting with not only refactoring but rewriting history. I'm always moving around commits, inserting, splitting things up. The amount of times I've instructed to "tell the story right the first time"...
More quantitative and qualitative analysis to come
I have been enforcing this myself through AGENTS files and other explicit instructions. What’s crazy is that none of the existing coding harnesses include such instructions by default. Two lines in Claude Code and Codex and the entire world would be writing better, more efficient code. Makes me believe in the conspiracy that these companies want more verbose code and longer files because it directly results in higher token usage.
I find it funny how the best practices for programmers, ignored in most IT companies, get reinvented as the best practices for AIs.
Boring: The documentation should be in code, not in external Word documents uploaded to the company SharePoint server.
Exciting: The documentation for the AI should be in code, not in external Word documents uploaded to the company SharePoint server.
Boring: You should give your developers the big picture of the project, not just micromanage them using Jira tasks.
Exciting: You should give your AI the big picture of the project in CLAUDE.md, not just micromanage it using prompts.
Boring: Refactoring makes your developers more productive in long term.
Exciting: Refactoring makes your AI more productive in long term.
No this isn't mindless reinventing --- this is finally having clear empiric evidence for something that we knew the entire time. This is a huge relief!
Next up is demonstrating the AI is more productive with better programming languages.
I've already seen at least one promising experiment about how static checking helps LLMs: https://arxiv.org/abs/2606.01522
Key parts of the abstract:
> This raises a question the programming-language community has not previously had reason to ask: should error-message detail be calibrated differently for AI agents than for humans?
> We investigate this question through a controlled experiment using Shplait, an ML-style statically typed language. We construct a suite of programs containing a single deliberate type error each, and measure how often an AI agent repairs them under ablation: a detailed error context using the unification stack; a proximate error location; a minimal type error; and a dynamic (test suite) error only. An automated oracle uses a test suite to classify each repair attempt as a type error, semantically incorrect, or semantically correct.
> We find concrete evidence that more detailed error messages generally improve an agent's ability to fix type errors. We also find that the presence of a type system appears to help more than only test suite failure reports.
> Next up is demonstrating the AI is more productive with better programming languages.
That actually would be pretty exciting!
But, wouldn't AI be biased toward more popular languages, since those will by nature of their popularity provide more sources for training material?
Even still, the AI could demonstrate which algorithms or maybe patterns and techniques are more productive, in the context of the popular languages.
That’s not necessarily a bad thing.
As an example, 110v American outlet plugs are almost certainly not the most efficient way to power devices and appliances; but, because they are so standard and are good enough, using them massively eases the burden of making, buying and selling powered appliances in the US. Similar story with USB C.
Outlets benefit from standards, however. Every device (well kind of) uses the same few plug types, and a different plug would be a disadvantage.
On the other hand, having a bunch of different languages is an advantage. People can pick the best tool for the job, and experiment with new language features.
> That’s not necessarily a bad thing.
True, and to my point, "popular" isn't necessarily "better".
> because they are so standard and are good enough, using them massively eases the burden of making, buying and selling powered appliances
Yes, but LLMs can do the heavy lifting when analyzing the "better programming languages" for productivity. There will be a bias toward popularity in their training sources, though.
You might be surprised. I've had an AI start spitting out none functioning python code. I've rewritten it with Go. The requirements are that every change has to pass compilation, linting and tests. Same requirements as Python. Go keeps the AI on the rails better. The compilation failing provides quick and probably better feedback than a whole dictionary of tests.
For sure. Its code output is definitely not to be fully trusted. Oftentimes, you can point at where it made a mistake and it usually fixes it too.
i can't wait for that one haha
> Next up is demonstrating the AI is more productive with better programming languages.
This is hard to do given the current reliance on model weights in its programming language knowledge. You’ll get your best results for python or some variation of python. But
Not true. Python performance is actually pretty low, and Elixir, Kotlin and C# trounce it handily.
Do you have citations? Everything I’ve read have said they have an overwhelming bias to Python; eg https://arxiv.org/html/2503.17181v1 and https://arxiv.org/html/2505.13004v1
Agents love writing Python, but it turns out they're not amazing at it. https://autocodebench.github.io/
I’m not really sure how to read those charts, but it’s completely possible that python did not work well for the approach they were trying.
I would be the first person to give up python for kotlin or some other more natively typed higher performing language. But the python bias is so far too big to ignore.
Haha, queue a big argument over what “better” means for programming languages.
They're the ones AI is more productive in....
(It's an interesting question though - my prediction would be that AI is best in the languages it consumed the most examples of, and I reckon the size of the StackOverflow site for each language would be a rough proxy for that).
If it was as simple as that we would have far fewer programming languages even for humans today.
Exactly
To be clear, the guy who wrote this article is Martin Fowler. Fowler was the author of a book literally titled "Refactoring", which he wrote over 20 years ago (it didn't invent the term "refactoring", but it's widely credited as popularizing it).
In other words, this is not some new AI-bandwagon-jumping blogger trying to rebrand old practices as something new ... this is the guy who coined the very term "refactoring"! He's not pretending what's old is new, he's arguing that best practices from 20+ years ago remain relevant (and showing receipts).
The website is martinfowler.com, but the article seems to be written by Giles Edwards-Alexander.
Oh, I stand corrected (thanks for the catch)!
It’s an easy mistake to make, given the name of the website, but Fowler frequently hosts guest authors. This article was written by Giles Edwards-Alexander, ThoughtWorks CTO for EMEA.
My namesake, er, not quite ;).
This has been a soul crushing part of the AI craze - we can finally fund all the devx work we wanted to do, for all the wrong reasons.
(It is nice that I can make something try our CLI a hundred times in an hour to test that new flag ergonomics, though)
It’s much easier to get your AI agents to do something consistently than it is to get your human colleagues to do it. Heck, personally we know these things are the right things to do, but we are just too busy and our minds wander to what we think our more productive uses of our time. That completely changes when it’s agents being instructed to do the work instead.
The crazy thing is that it s looking like AI might be able to write software better than humans eventually simply because they do not get bored of doing tedious tasks. Many things that we know work but don’t practice because humans aren’t very scriptable are now viable and can be easily applied with agents. For example, I’m finding a lot of success in using separate agents to write implementation and tests from a common specification and then using an auditor to run the tests so neither agent is contaminated by the other’s work. This is just part of the clean room engineering process that was developed for people by IBM in the 1980s, and it was shown useful then, but with AI it can be widely and consistently applied.
They'd be 10x better than us already if task tedium was the problem. It's design sense that they're missing.
I find that in the areas where people think that LLMs excel at coding and don't like doing manually it's usually because the human was inclined to slop out repetitive boilerplate and thought that was the only way. Tests are usually like this, sadly.
LLMs are definitely good at providing reams of duct tape (which is drudge work) to patch up those bits of the code base where the code sucks. The problem is that duct tape is not the most architecturally sound construction material.
> They'd be 10x better than us already if task tedium was the problem... the areas where people think that LLMs excel at coding and don't like doing manually it's usually because the human was inclined to slop out repetitive boilerplate and thought that was the only way.
There is so much work out there that is repetitive / boilerplate / tedium. If you get to personally work on interesting work more than 50% of the time (pre-LLM) I'd say your job is #blessed.
IME if a technical task is tedious and repetitive it is nearly always because the system was badly designed or because it wasnt automated properly.
My job is automation and system design, so if I can't fix or work around these things that reflects poorly upon my skills.
Some people treat writing tests as inherently boring because their test frameworks usually suck. If they don't suck and a test is a very close approximation of a spec, it's not boring at all, especially if you use it as a means of codifying a spec before implementation.
Don't disagree that that's the system not working well.
You're lucky to be in a system that works out for you. Not everyone is in such a lucky position. LLMs help automate the tedium out of their job, and hopefully, have a bit more extra energy to make a better system within their immediate locus of control.
Hey look if putting a sticker of Elsa on it or saying it’s for the AI gets the kids excited to eat their vegetables just quietly take the win.
if it works, it works
The thing is, humans make do. Given lacking, outdated documentation somewhere on sharepoint, big picture revealed somewhere in passing at some meetings or water cooler talks, underprioritizing refactoring humans still deliver something even if at reduced quality and timelines.
AIs either execute extremely poorly or don't work at all without these. Decent engineering practices are suddenly a hard prerequisite with AIs, not just some longer term improvement.
On a related note, this is unironically good argument to integrate AIs into the workflow even if there's zero net benefit in your use case. It's a perfect excuse to bring the proper engineering practices in.
There is a big difference pre and post AI, namely that humans have quite good long term context management.
But now that everyone is coding with AI, all agents need to acquire context every session. Thus the value of doing best practices is much higher and benefits are there immediately.
and furthermore, codebases can reach a size where good patterns matter much much faster
(so through this mechanism, the pain of not refactoring becomes apparent more often)
I really enjoy refactoring. Like, doing it by hand (5 miles uphill in the snow both ways) rather than AI. I don't even understand why I enjoy it, because if done right, there's no visible change. When people ask me why I'm so pumped up after having worked on the codebase all day long, I can't give them any sort of answer that makes sense (coming from a small business with a smaller dev team). It's just... "I'm future-proofing our website and we won't see any direct results today but things will be so much easier going forward..."
There's something about the puzzle. Looking at my old, deranged coding workarounds that tried to solve problems that have already been solved a thousand times before with established paradigms; and then moving them toward said best practices; and doing it in a way that no NEW technical debt is created. It's just satisfying.
I think one of the best learning experiences for me has been the fact that I created a bunch of sloppy shit by hand, auth and all, which forced me to learn things the hard way. All along the way, people were shouting from the rooftops: "use established libraries, dummy!", which is the same advice I would give to somebody today. But by doing things the hard way, I learned so much more about the inner workings. And, I've also given myself a decade's worth of refactoring work, which I really enjoy!
Probably it's just the same dopamine hits that make things like watching the Windows 98 Defragger soothing.
I'm curious how heavily you have test suites built up as guardrails while doing this, to prevent regressions and such?
I have often done huge refactors over the last 20 years on systems where there are none. It's very satisfying turning a 1,000 line class file into 100 lines.
But all my refactors have been in statically typed languages.
To speculate on the reason why, which the GP asked, I think perhaps it is two things. It's the same satisfaction from tidying a room or your workbench. But it's also slightly narcissistic unfortunately. I think you've shown you're better than the other programmers. Much, much, better when you end up with massive reductions in TLOC.
And with tests, the irony is that the resulting code is often so much easier to reason about, you start spotting really obvious bad assumptions in the original implementation.
It's one of the reasons I've always been fairly skeptical about the true value of unit tests (integration, I get). That and the fact that projects I've worked on that did have unit tests catch like 1 bug a year. Maybe it's just the size of systems I traditionally work on (smaller teams, or even 1 person teams, so man-years worth of effort rather than decades or centuries).
Me as well. One of my formative experiences was reading Refactoring (Fowler). At the time I felt skeptical that it could make a difference --- I had mostly written scientific/research code at this point. I decided as an experiment to apply it to a dissatisfaction I had with the structure of a codebase and it was a revelation. It really made it really concrete what people meant by "a codebase is a system", made me see code from a higher level, almost like a continuous tissue or web you could pull and prod.
It's quite sad to think this experience might be on its way out, and it really highlights that "junior problem" from AI. There is just no way around diving in to get an intuition for things. Naur warned us of this 40 years ago but the lesson never seems to get learned.
I enjoy studying different refactoring patterns and real-world examples.
> I can't give them any sort of answer that makes sense ...
Pride of workmanship. For those who understand, no answer is needed. For those who don't, no answer will work.
Ha, many years back I had to write a template processor for some system configuration handling. Manager told me last guy did most of the work so test and wrap it up. I instead had to throw it out as it was so crappy, wrote much smaller code, with no dependency and really put soul in logging where it showed cleanly how tool is navigating all directories, picking placeholders and replacing with values and so on.
No one really cared but it satisfied me and make me really happy.
This is such a nice piece, this is how people should write about AI. Specific, grounded to how the tools are actually being used, and quantitative. There is so much bad AI commentary that is incredibly vague, divorced from any actual use cases, or written by people who don't actually use the tools. It is good to see a critique that is "here is a thing AI is bad at and measurements to show it" rather than gesturing "here is why I think AI is problematic for society."
It is rather different but another piece of research I liked for the same reason was this report that interviewed Boko Haram members about how they used AI to assist terrorism. You get these interminable online debates that are so unproductive and reporting that is specific is such a breath of fresh air.
https://casp.ac/reports/ai-enabled-terrorism
Yes, more like this please, universe!
I’ll try to keep it up.
I think this is one of those things where a human in the loop is indispensable
An agentic refactoring pass does make sense cause one LLM reviewing work can spot things the ‘generator’ LLM missed while focused on the initial task output
But can the reviewer agent ever actually have in mind what this project actually is? And how the code all comes together to do the work involved? In other words what parts of the code are redundant or can be made more elegant
Asking coding agents to refactor your codebase is maybe like asking trauma surgeons to increase your exercise capacity. The agents are gonna need a really holistic POV to do this properly
I guess part of my point is that just splitting big files into multiple files is only refactoring in a superficial sense without having a theory of what code belongs together and what can be extracted into utility functions etc. Is splitting files actually like decomposing factors or is it like splitting a larger number into smaller numbers that still eventually get added together
A good example of what I mean is that agents often don’t ~actually~ understand the whole system anyway. They might implement a system to store and calculate something that is already being fetched via API. Humans often have a dual perspective — a holistic sense of the project and (when applying our mind to a task) a precise scalpel: ‘oh if we just look at this this JSON it has a key with this data already’
I was able to largely refactor something that was going to take months in a week or so by being the narrating human in the loop.
It was helped by having refactoring experience and approaches to codebases by others, and in my case, being the original architect and being able to speak to the original and current intents, where needed.
This was using a less common, but capable and easy language for the LLM without a ton of dependancy brittleness to manage.. once the effort to remove javascript/python bias was in place, it became so powerful that once the lightbulb went on, it really got cruising.
The project was playing in the world of JSR-223 languages, where you could script in many popular languages, but it all got to run in the JVM, which was an environmental requirement.
https://en.wikipedia.org/wiki/Scripting_for_the_Java_Platfor...
Your take is outdated. Agents are excellent at refactoring now. If you told me what you wrote during an interview, that would be grounds to not continue.
It's very important that you are using coding agents with the latest frontier models and know exactly what it can't and can't do if you want to be hired in this market.
Interesting that you said the word ‘using’. Why is that? If an agent could refactor excellently without a human who is the user
"Refactor this code spin up subagents" try that in codex to start your journey of improving your agentic coding skills but if you are too lazy to do even then there's just no hope for you.
I am currently working 3 remote jobs thanks to agentic coding and one of the jobs require me to interview and hire other devs. I find it amusing how many are irrationaly resistant and unwilling to use agentic coding. They are also the loudest in the room with opinions that are outdated. On the flip side, it's just as funny how to see how complicated engineers try to make agentic coding. It doesn't have to be and that's not its design. Don't get in its way as much as possible. The frontier models are good enough that adding complexity only wastes more tokens. We've come a long way from the summer you were fiddling with Sonnet 4.5 in Cursor or windsurf.
The job market isn't suddenly going to swoop in and save you by banning AI agents. Keep up or get left behind. Don't let other people's opinions about AI agents hinder your own progress, chances are they have no skin in the game.
> "Refactor this code spin up subagents" try that in codex
The deferral of all agency as a product owner over to the stochastic-word-box-in-a-do-loop feels like the kind of thing that, if you mentioned you do that in an interview, would be grounds to not continue.
Re: your updated comment, I'm glad you've found success with this pattern of behavior, but I'd suggest that you might be extrapolating "I haven't been punished yet for this shortcut" to "This is the future of work and those who disagree are luddites".
lol - I guess, if I heard that in an interview that would be grounds not to continue.
fwiw codex is very good. arguably better than the sonnet equivalent (value-wise). I use both daily, professionally. And "refactor this code spin up subagents" is hardly an expert-level "I wouldn't hire you" flex.
EDIT: agree with your comment updates. I'd just say it more positively leaning especially to people who are stressed financially. SOTA has advanced a lot so counter-intuitively being behind isn't really behind, it can almost be a way to leap-frog, lean on the Agents. But definitely you have to commit to putting in the reps and learning and not just mindlessly instructing AI to "fix it". Think that's where the discussion splits.
> Asking coding agents to refactor your codebase is maybe like asking trauma surgeons to increase your exercise capacity. The agents are gonna need a really holistic POV to do this properly
If you ask it to implement a specific kind of refactoring over a specific section of code it seems that current LLMs can do just fine. Even things as complex as "use `functools.partial` to implement the Command pattern here, rather than dataclasses".
> Is splitting files actually like decomposing factors or is it like splitting a larger number into smaller numbers that still eventually get added together
The file boundaries represent logical subsystem boundaries in the code, making it easier to reason about. The training data abundantly represents the idea of treating the contents of another file as opaque by default while other functionality in the same file can be used freely. I think it's reasonable to assume that there's something objective about the benefits that humans get from this, and it isn't just a consequence of how human cognition works.
The benefits go beyond reducing token consumption. Compact contexts also foster better reasoning, enable intelligence across more layers if you can load them in a single context, and writing software to enable this, results in more correct software, software that _generalizes_, that has higher probability of being correct not just for tested cases, but for the interpolations and extrapolations of these cases.
Refactoring towards good abstractions is more powerful than people realize. There's information theoretic bayesian math to back this up.
It's a bit of a divine coincidence that software that is more economically and energy efficient to process and run tends to also be more correct.
It's all about reducing the entropy of your code. https://benoitessiambre.com/entropy.html
> It's all about reducing the entropy of your code
In all corners of our world and the universe at large, reducing entropy in anything can be thought of as “building.”
Imagine what happens when human readability is discarded as a goal. LLMs are already very good at inferring meaning with little context. If the objective function is reduction of token consumption, it's hard to know where we will end up.
It's very interesting to have some data on this.
It matches my experience which is that LLMs greatly benefit from well factored code, but are not particularly adept at creating such code.
Much like most human developers I suppose!
I've been budgeting myself explicit "slop removal" time. In fact I'm in it right now one window to my right here.
I still get a big win from AI on the net, but you do need to budget some time to clean up. I'm still on team "read every line".
In fact this is a case where I deliberately deferred some review because I was a blocker for another team. Now that I've got something to them I'm going back and I'm going to eat a bigger chunk of debt than I normally would, but it's worth it for unblocking the other team sooner. AI has made tech debt easier to take out, in all senses of that term.
It is also pretty decent, in my experience, at being guided into how to fix tech debt. Some other people's experience varies: https://news.ycombinator.com/item?id=49035455 YMMV.
It's totally worth doing. Especially since it only takes one or two manual improvements before you can just say "ok, replicate this new pattern throughout the codebase. Go."
Out of the box, this is what I've found too.
I've had some luck while refactoring by helping it shape how to refactor, which can improve how well factored code should look like.
Providing examples of well factored code can go a long way, even if it's an open source repo of what to do / not to do.
Great piece, but it misses the elephant in the room: the lion's share of economic benefit from refactoring will come from the fact that it makes it easier for humans to understand. That means 3am pages get resolved faster, fewer bugs will end up in prod, and your team can ship faster than your competition - gaining a leg up in the market. Most importantly, folks will be more comfortable accepting responsibility and ownership of a system when they understand it... this means that if/when something goes wrong, people will more quickly jump in and fix it, and when things could be better, folks will jump in and improve it.
the punchline: "Refactoring reduces token consumption"
I appreciate the effort to quantify the benefit rather than pontificate. It's worth mentioning Martin Fowler wrote a whole book on refactoring [1], in which he states, "to refactor, the essential precondition is [...] solid tests", which I think is the real benefit here, AI or not. Good tests protect against regressions, whether human or robot. They also help encode the spec, which humans and robots can read.
[1] https://www.oreilly.com/library/view/refactoring-improving-t...
(Just to note that although the article is on martinfowler.com Martin is not the author. It's attributed to Thoughtworks CTO Giles Edwards-Alexander.)
Thanks for catching that. Updated my post
Oh. That's kind of disappointing.
I think the real punchline was that the value saved was on the order of cents!
> Every single change that touches the data access layer from this point forward now costs significantly less.
> How much of a saving? Assuming Sonnet 5 pricing at the time of writing of $3/MTok, 39.7 cents.
Now consider OpenAI's price drop, and open models, and consider that in the long run tokens will get cheaper. And think that the refactor needs to be human guided at a price of what for a senior developer - $100/hour?
> I think the real punchline was that the value saved was on the order of cents!
If the agents can read less tokens in the future, and make changes more effectively then this would add up over time.
Tangent, but I have a theory that refactoring is one of the best symptoms of a healthy dev team.
It's partly that refactors themselves have benefits, but I think more that the benefits to refactoring aren't visible to something like product-owners, feature tickets, etc.
If teams are refactoring to ensure the health of the overall software, it's a tell-tale sign that developers are happy making recommendations for good software, and that those recommendations are being taken seriously.
I think Martin Folwer might have actually coined the term "software rot" - either way, as an issue it happens most severely when a team either aren't motivated or empowered to build their vision of high quality software. When a team can follow their judgement of excellence, that's usually a great sign!
(and yes, obviously this can go to far, there are probably some teams who rewrote all their stuff in Ruby then Node then Rust and now something else to be "agent native", but in the coorporate world, I see a lot less of this than teams who just don't feel like they have permission to improve things)
Agent code can only be read and understood by agents. We’ve reached the point where people just can’t comprehend these giant code blobs.
Feature, bug or emergent property?
I don’t think the distinction matters as much as the reality.
We’re being locked into using the AI tooling bc the code was generated with AI tooling.
These giant files of doom were being generated by humans anyway and were very hard to work with. With LLMs it’s at last manageable or feasible to edit, refactor etc.
I honestly think LLMs are going to save us from ourselves as the codebases became too large and “messy” for humans to comprehend. (Mono repos of doom)
On a personal level these giant files are abhorrent but that’s just personal taste and I don’t think any of the Martin Fowler refactor/cleanup stuff is going to matter at all anymore. Kinda sad on some level.
> Agent code can only be read and understood by agents.
This just fundamentally isn't true and if this is your perspective then you're using LLMs wrong.
Fundamentally, the code can be understood by humans, but the volume of the generated code outpaces human capacity to understand it.
There was always bad code. Extend that to the general challenge of a bad hire doing net-negative things for the company. AI is a new problem because it 1000x a bad hire's blast radius. Personally, I'd even state that AI tends to turn an average and sometimes-but-not-always-good hire into bad hires because finally they get to do the thing they've always wanted at lightning speed in a way that previously the company red-tape wouldn't allow.
> These giant files of doom were being generated by humans anyway
Right, I've never seen an agent produce code anywhere near as bad as some of the human-generated code that I've worked on.
And, if your agent is producing huge files or functions, you can just tell it not to and it'll comply.
Clearly not true. I got codebases that are 100% done by LLM. I have no problem reading the code base and find where things are. Not harder to find stuff than when I done it manually.
Interesting that the amount of code remained essentially unchanged. In my experience it is not unusual that refactoring messy code cuts the number of lines in half.
It can go both ways. Lines of code is a terrible metric for pretty much any goal that refactorings are meant to tackle.
I don’t know. LOC to me is indispensable for certain refactoring goals in my opinion.
Taking a component and turning it from 7K lines to 3K lines and maintaining functionality obviously means there’s less complexity introduced, less to go wrong now, and less overhead to modify in the future.
Sure it can go the other way, the component needs to support something it might need, we need to adjust larger patterns, this function needs to be refactored into something more robust.
But lines of code is a pretty decent metric of success for “trimming down and cleaning up” style refactoring, to me at least. It’s not everything of course, but it’s definitely an indicator.
LOC can be one measure, but not always the sole or best ones.
Since LLMs are word generators, and have a propensity to generating words, they need to be shaped to understand simplest is best, more isn't more, and less isn't more always.
Trimming down and cleaning up could be formatting, standardization, commenting, or even some basic re-architecting that was overdue.
One of the biggest benefits of llms for refactoring I'm finding is reducing technical debt.
That's LOC in a single file, that's a bit different from total LOC as a metric.
Total LOC is a garbage metric. Things like reducing line count in specific files or components is a big benefit, but those lines are often moved, not dropped.
A couple of days ago I split a single class which did 5 barely-connected things, where each of the 5 entry points had another 5-10 private helper methods (none of which were shared between entry points), into 5 separate classes which each do one thing - total number of lines went up (because they all have the same boilerplate import statements at the top), but being able to focus on one entry point and it's helpers at a time has made things much easier to work with
I did not mean it as a metric but as an indicator what was wrong with the code.
- the code is essentially good but all is in one file, you split it up, lines of code stay the same
- the code is essentially good but lacks some structure, for a function that does five things directly, you extract the functionality into five functions and call them from the original function, lines of code goes very slightly up
But once the code is actually bad - code duplication, bad abstractions, inefficient language use, ... - I would generally expect the lines of code to significantly drop. What scenarios are there where the code is actually bad but refactoring does not reduce the lines of code? It is certainly possible but at moment I am having a hard time comming up with a good example.
> What scenarios are there where the code is actually bad but refactoring does not reduce the lines of code?
IMX: the scenarios where people hold higher standards than what you describe :) (i.e. such that "the function does five things" is deemed "actually bad". Of course, that does depend on the refactoring not causing an unacceptable performance hit, which can happen depending on the environment.)
Don't get me wrong, a function doing five things is bad. I consider code not good if you write if(condition) instead of if (condition) or a +b instead of a + b. Not that it is really bad on its own, but it indicates to me that the code was written with little care and I should probably expect bigger issues.
What I really want to get at is the distinction between leaving the actual code untouched and just moving it a bit around - to other functions, other classes, other files - and having to change the code - from deduplicating to completely rewritting it.
Quality of lines > Quantity of lines.
In some projects, the code might not have much, or enough testing, or documentation/commenting.
Code is largely for others and the future if the creators of it want to move onto other projects.
Yeah, I sorta expected the punchline to be "this 17,000 lines of code became 2,000" but it is still about the same quantity. Kinda interesting. Also interesting that the last change has by far the largest effect (4x reduction in tokens) and that also corresponded to the biggest reduction in single file LOC
Yep, this was a real surprise to me, too. That’s why I included that top line in the figure. It also doesn’t line up with the agent’s predictions when it laid out the refactoring before proceeding.
There’s further work to do to understand exactly what’s going on here.
> Add a new ItemWatchStore public async trait to the Firestore layer, following existing patterns exactly. The trait must have three methods:
async fn watch_item(&self, item_id: &str, user_id: &str) -> Result<()> async fn unwatch_item(&self, item_id: &str, user_id: &str) -> Result<()> async fn watched_items_for_user(&self, user_id: &str) -> Result<Vec<String>>
This shows the limitations of vibe coding. It takes someone with a long history of software development to prompt for something like this.
Even though the model is writing 100% of the code, still needed someone with a lot of programming knowledge to write the prompt.
After reading a fair amount of OSS code, it's rather glaring that the data structures and algorithms are more like atoms than molecules in software design. We do have some molecules in the Design patterns, but they are more suitable to the OOP universe, like the collection api (filter, map, take,...) when dealing with groups of objects or the reactive api for dealing with concurrent tasks.
Most of software development is looking at some process and then decomposing it recursively until you get to those molecules/atoms of the computing world. Coding them is trivial, and while you can gain a certain boost from the AI, after a while you no longer have to write that much code. It will turn into a balancing act where the introduction of a new concept has to be done carefully.
I have a blanket, unignorable 'files must be <= 1000 lines' lint for my Rust projects for exactly this reason. Exploring large files is _costly_ for agents, so it's much better to lean into the filesystem hierarchy to a greater extent than you would normally with a team of human developers.
This is interesting, but I think it would be even more interesting to see a comparison of the token cost of adding a new feature, in the original codebase vs the refactored one. You'd think/hope that the cost of adding the new feature would be lower when starting from the refactored codebase, indicating that refactoring makes economic sense in the long run by lowering the cost of adding new features.
That's what I've shown in this article. The token consumption shown is for adding a new feature (the same feature each time -- in a sub-agent) after every refactoring is applied. I didn't actually capture the token cost of the refactoring. That was a miss, and I'm making sure I do that in the future.
Kind of you to engage! Ahh, ok, I misunderstood. Very cool! I think maybe it would have helped me if you had given a one-or-two-sentence description of the "representative change" in the main text, to make it a bit more concrete up-front? But possibly just a 'me' issue...
It’s interesting that cyclomatic complexity or cognitive complexity scales with token usage. A codebase that enforces upper bounds on one of these two also (potentially) helps AI agents stay efficient.
Can't agree more. Feels like LLMs are not able to understand your requirements at a high level and always add complexity by default. Really need a handbook to guide them to do those cleanups. Obviously they know the knowledge in their weights but they just don't know how to apply it.
The author said that Claude was not good at refactoring. But the author was using Sonnet 5. I supect Opus 5 would have done a lot better at refactoring. It would have come up with a multstep approach, i suspect.
I used Opus 4.8 to create the refactoring plans. I converted token savings using Sonnet 5 pricing at the time of writing as I'd expect to use that for more 'straightforward' feature additions.
Opus 5 is old news and has lots of problems. Minuet 6.9 is the one to use, and fixes all the problems people have with LLMs.
>The goal of refactoring an agentic code base is to spend tokens now in refactoring to make token consumption for future work lower.
Couldn't have said it better myself. Deciding to refactor should be like a discounted cash flow analysis but for tokens!
I can very much relate to the experiences described in the article. In my projects, I have multiple Claude skills and rules aiming at making my files (the code, but also the Claude files themselves) more token-effective. I have seen huge differences in token usage before and after invoking those skills, which I regularly do as part of having Claude audit my projects; although I haven't measured them. Those projects are (like the one in the article) fully developed by Claude Code, so I found such audits and mindful token usage very necessary.
Interesting take-away:
> Claude is unable to look at code, look at refactorings in general and work out which are suitable to apply: a human needs to actively guide it.
Claude is happy to produce a very large Rust file. But you need human guidance to make it smaller.
"I would have written a shorter letter but did not have the time"
Opus/Fable 5 (or really, any thinking model) do a better job of refactoring than Sonnet would. I think the takeaways are still valid, just not as valid when using a more expensive model at a higher effort level.
I think the name of the game is going to be "how do we maximize output while keeping token counts low". For example, finding ways to replace AI workflows, even if minor ones, with scripts/code.
Anthropic told me that I have till August 19th to be get my act together becuase they are going to reduce my token count by 50%. lol. I have been abusing my Pro Max allowance and need to start being less wasteful.
Articles like this, can help us come up with ideas on how to do it.
> and used tiktoken to approximate tokens, by dividing character count by four.
This is just the type of thing that stands out.
You used a tokenizer library to determine the number of tokens in your text by dividing its length by 4.
That makes no sense!
Great point. Using good software architecture and clean code will deliver direct and measurable economic value to projects and reduce AI costs right from the start, whether during refactoring or when planning a logical reorganization.
Was hoping for a process diagram on how refactoring removes bottlenecks during software developement and allows you to ship faster etc :)
Still a nice writeup and love how these meta analysises (presumeably) done via AI can now easily capture metrics that inform your workflow.
> Claude is unable to look at code, look at refactorings in general and work out which are suitable to apply
I am just sitting here waiting for Grady Booch to write "Architecture!"
Make it work, make it right, make it fast, in that order
Make it right is the refactoring
Putting this in agent.md didnt help?
"always give ultimate decent code in term of maintanibility, cleaness, and perf".
I wonder, how close can we get to a reasonable simulator for testing software engineering practices before inflicting them onto the real world?
Being a bit cheeky here -- the amount of comments on this post is a telltale sign of how common / desirable refactoring is for the majority of developers (xD). It isn't our fault though, maybe if those pesky managers read this article then they would understand..
It's been on HN for 45 minutes. It's before 9am on the west coast
You can tell this is BS because there's no way that app is actually functioning at all. Fully vibe-coded, all that? No, I don't believe it.
I've been experimenting with not only refactoring but rewriting history. I'm always moving around commits, inserting, splitting things up. The amount of times I've instructed to "tell the story right the first time"...
More quantitative and qualitative analysis to come
> Claude.ai was better than Claude Code
Why is this something users have to distinguish? Why can't these generative AIs choose good names for things?
I have been enforcing this myself through AGENTS files and other explicit instructions. What’s crazy is that none of the existing coding harnesses include such instructions by default. Two lines in Claude Code and Codex and the entire world would be writing better, more efficient code. Makes me believe in the conspiracy that these companies want more verbose code and longer files because it directly results in higher token usage.
Pray do share those two lines