AI coders think they’re 20% faster — but they’re actually 19% slower
Model Evaluation and Threat Research is an AI research charity that looks into the threat of AI agents! That sounds a bit AI doomsday cult, and they take funding from the AI doomsday cult organisat…
I feel this -- we had a junior dev on our project who started using AI for coding, without management approval BTW (it was a small company and we didn't yet have a policy specifically for it. Alas.)
I got the fun task, months later, of going through an entire component that I'm almost certain was 'vibe coded' -- it "worked" the first time the main APIs were called, but leaked and crashed on subsequent calls. It used double- and even triple-pointers to data structures, which the API vendor's documentation upon some casual reading indicated could all be declared statically and re-used (this was an embedded system); needless arguments; mallocs and frees everywhere for no good reason (again due to all of the un-needed dynamic storage involving said double/triple pointers to stuff). It was a horrible mess.
It should have never gotten through code review, but the senior devs were themselves overloaded with work (another, separate problem) ...
I took two days and cleaned it all up, much simpler, no mem leaks, and could actually be, you know, used more than once.
Fucking mess, and LLMs (don't call it "AI") just allow those who are lazy and/or inexperienced to skate through short-term tasks, leaving huge technical debt for those that have to clean up after.
If you're doing job interviews, ensure the interviewee is not connected to LLMs in any way and make them do the code themselves. No exceptions. Consider blocking LLMs from your corp network as well and ban locally-installed things like Ollama.
It should have never gotten through code review, but the senior devs were themselves overloaded with work
Ngl, as much as I dislike AI, I think this is really the bigger issue. Hiring a junior and then merging his contributions without code reviewing is a disaster waiting to happen with or without AI.
It used double- and even triple-pointers to data structures
(old song, to the tune of My Favourite Things)
🎶 "Pointers to pointers to pointers to strings,
this code does some rather unusual things...!" 🎶
I talked to Microsoft Copilot 3 times for work related reasons because I couldn't find something in documentation. I was lied to 3 times. It either made stuff up about how the thing I asked about works or even invented entirely new configuration settings
Claude AI does this ALL the time too. It NEEDS to give a solution, it rarely can say "I don't know" so it will just completely make up a solution that it thinks is right without actually checking to see the solution exists. It will make/dream up programs or libraries that don't and have never existed OR it will tell you something can do something when it has never been able to do that thing ever.
And that's just how all these LLMs have been built. they MUST provide a solution so they all lie. they've been programmed this way to ensure maximum profits. Github Copilot is a bit better because it's with me in my code so it's suggestions, most of the time, actually work because it can see the context and whats around it. Claude is absolute garbage, MS Copilot is about the same caliber if not worse than Claude, and Chatgpt is only good for content writing or bouncing ideas off of.
Are you using Claude web chat or Claude code? Because my experience with it is vastly different eve when using the same underlying model. Clause code isn't perfect and gets stuff wrong, but it can run the project check the output and realize it's mistake and fix it in many cases. It doesn't fix logic flaws, but it can fix hallucinations of library methods that don't exist.
In fairness the msdn documentation is prone to this also.
By "this" I mean having what looks like a comprehensive section about the thing you want but the actual information you need isn't there, but you need to read the whole thing to find out.
I'll quote myself from some time ago:
The entire article is based on the flawed premise, that "AI" would improve the performance of developers. From my daily observation the only people increasing their throughput with "AI" are inexperienced and/or bad developers. So, create terrible code faster with "AI". Suggestions by copilot are >95% garbage (even for trivial stuff) just slowing me down in writing proper code (obviously I disabled it precisely for that reason). And I spend more time on PRs to filter out the "AI" garbage inserted by juniors and idiots. "AI" is killing the productivity of the best developers even if they don't use it themselves, decreases code quality leading to more bugs (more time wasted) and reducing maintainability (more time wasted). At this point I assume ignorance and incompetence of everybody talking about benefits of "AI" for software development. Oh, you have 15 years of experience in the field and "AI" has improved your workflow? You sucked at what you've been doing for 15 years and "AI" increases the damage you are doing which later has to be fixed by people who are more competent.
from some time ago
It's a fair statement and personal experience, but a question is, does this change with tool changes and user experience? Which makes studies like OP important.
Your >95% garbage claim may very well be an isolated issue due to tech or lib or llm usage patters or whatnot. And it may change over time, with different models or tooling.
At this point I assume ignorance and incompetence of everybody talking about benefits of "AI" for software development.
The study was centered on bugfixing large established projects. This task is not really the one that AI helpers excel at.
Also small number of participants (16) , the participants were familiar with the code base and all tasks seems to be smaller in completion time can screw results.
Thus the divergence between studio results and many people personal experience that would experience increase of productivity because they are doing different tasks in a different scenario.
The study was centered on bugfixing large established projects. This task is not really the one that AI helpers excel at.
"AI is good for Hello World projects written in javascript."
Managers will still fire real engineers though.
I find it more useful doing large language transformations and delving into unknown patterns, languages or environments.
If I know a source head to toe, and I'm proficient with that environment, it's going to offer little help. Specially if it's a highly specialized problem.
Since SVB crash there have been firings left and right. I suspect AI is only an excuse for them.
familiar with the code base
Call me crazy but I think developers should understand what they're working on, and using LLM tools doesn't provide a shortcut there.
You have to get familiar with the codebase at some point. When you are unfamiliar, in my experience, LLMs can provide help understanding it. Copying large portions of code you don't really understand and asking for an analysis and explanation.
Not so far ago I used it on assembly code. It would have taken ages to decipher what it was doing by myself. The AI sped up the process.
But once you are very familiar with a established project you had work a lot with, I don't even bother asking LLMs anything, as in my experience, I come up with better answers quicker.
At the end of the day we must understand that a LLM is more or less an statistical autocomplete trained on a large dataset. If your solution is not on the dataset the thing is not going to really came up with a creative solution. And the thing is not going to run a debugger on your code either, afaik.
When I use it the question I ask myself the most before bothering is "is the solution likely to be on the training dataset?" or "is it a task that can be solved as a language problem?"
Reading the paper, AI did a lot better than I would expect. It showed experienced devs working on a familiar code base got 19% slower. It's telling that they thought they had been more productive, but the result was not that bad tbh.
I wish we had similar research for experienced devs on unfamiliar code bases, or for inexperienced devs, but those would probably be much harder to measure.
I don't understand your point. How is it good that the developers thought they were faster? Does that imply anything at all in LLMs' favour? IMO that makes the situation worse because we're not only fighting inefficiency, but delusion.
20% slower is substantial. Imagine the effect on the economy if 20% of all output was discarded (or more accurately, spent using electricity).
I'm not saying it's good, I'm saying I expected it to be even worse.
Does that imply anything at all in LLMs’ favour?
Yes it suggest lower cognitive load.
1% slowdown is pretty bad. You'd still do better just not using it. 19% is huge!
Don’t give yourselves to these unnatural men - machine men with machine minds and machine hearts! You are not machines! You are men!
You are not cattle!
You are men!
You have the love of humanity in your hearts!
You don’t hate!
Only the unloved hate - the unloved and the unnatural!
Soldiers!
Don’t fight for slavery! Fight for liberty!
In the 17th Chapter of St Luke it is written: “the Kingdom of God is within man” - not one man nor a group of men, but in all men!
In you!
I love the old Melody Sheep version.
Paolo Nutini's song 'Iron Sky' samples this same speech
https://en.wikipedia.org/wiki/Iron_Sky_%28song%29?wprov=sfla1
Would ai coders even get faster over time or just stay stagnant since they aren't learning anything about what they're doing
People are a bad judge of their own skill and overrely on tools and assistants when present. See also: car adas systems making drivers less skillful. More news at 11.
See also: car adas systems making drivers less skillful.
But also making traffic safer
Think we need to introduce a mandatory period where you need to drive an old car with no ABS when you've just gotten your license. I mean for me that was called being a broke-ass student, but nowadays cars with no ABS are starting to cost more than cars with ABS, traction control and even ESP, because the 80s and early 90s cars where these things were optional, are now classics, whereas you can get a BMW or Audi that was made this century for like 500-800 euros if you're brave or just want to move in to your garage full time.
Now the interesting question is what it really means when less experienced programmers think they are 100% faster.
Someone told me the best use of AI was writing unit tests and I died on the inside.
Why? That is a great use for AI. I'm guessing you are imagining that people are just blindly asking for unit tests and not even reading the results? Obviously don't do that.
Of course that’s what they’re doing. That’s the whole point. Generate a bunch of plausible-looking BS and move on.
Writing one UT (actually writing, not pressing tab) gives you ideas for other tests.
And unit tests are not some boring chore. When doing TDD, they help inform and guide the design. If the LLM is doing that thinking for you, too, you’re just flying blind. “Yeah, that looks about right.”
Can’t wait for this shit to show up in medical devices.
🎵Had a great day out,
Callin' my name like Ferris Bueller,
Time to wrap this up,
I'm getting 19℅ slower! 🎵
The only time it really helps me is when I'm following a pretty clear pattern and the auto-complete spares me from copy-pasting or just retyping the same thing over and over. Otherwise I'm double-checking everything it wrote, and I have to understand it to test it, and that probably takes most of my time. Furthermore, it usually doesn't take the entire codebase into account so it looks like it was written by someone who didn't know our team or company standards as well as our proprietary code.
If there's a clear pattern, regex is your friend. I use it for complex find-and-replace actions, or to generate code based on a template and a list of values (find a value, replace it with the template including the value). Full control over the output, more reliable even than manual copy-pasting.
It’s hard to even call them specialists, they are at the level of cashiers, for whom the computer does everything, and sometimes they do something at the level of communicating with clients and that’s all. I'm certainly not a professional, but I think the main message is clear.
It would be interesting to see another study focusing on cognitive load. Maybe the AI let's you offload some amount of thinking so you reserve that energy for things it's bad at. But I could see how that would potentially be a wash as you need to clearly specify your requirements in the prompt, which is a different cognitive load.
I seem to recall a separate study showing that it just encouraged users to think more lazily, not more critically.
My velocity has taken an unreasonable rocket trajectory. Deploying internal tooling, agent creation, automation. I have teams/swarms that tackle so many things, and do it well. I understand there are issues, but learning how to use the tools is critical to improving performance, blindly expecting the tools to be sci-fi super coders is unrealistic.
I wish AI was never invented, but surely this isn't ture.
I've been able to solve coding issues that usually took me hours in minutes.
Wish it wasn't so, but it's been my reality
The experienced developers in the study believed they were 20% faster. There's a chance you also measured your efficiency more subjectively than you think you did.
I suspect that unless you were considerably more rigorous in testing your efficiency than they were, you might just be in a time flies when you're having fun kind of situation.
Ai-only vibe coders. As a development manager I can tell you that AI-augmented actual developers who know how to write software and what good and bad code looks like are unquestionably faster. GitHub Copilot makes creating a suite of unit tests and documentation for a class take very little time.
Try reading the article.
The article is a blog post summarizing the actual research. The researchers' summary says:
We do not provide evidence that: AI systems do not currently speed up many or most software developers. We do not claim that our developers or repositories represent a majority or plurality of software development work.
The research shows that under their tested scenario and assumptions, devs were less productive.
The takeaway from this study is to measure and benchmark what's important to your team. However many development teams have been doing that, albeit not in a formal study format, and finding AI improves productivity. It is not (only) "vibe productivity".
And certainly I agree with the person you replied to: anecdotally, AI makes my devs more productive by cutting out the most grindy parts, like writing mocks for tests or getting that last missing coverage corner. So we have some measuring and validation to do.
I did, thank you. Terms therein like "they spend more time prompting the AI" genuinely do not apply to a code copilot, like the one provided by GitHub, because it infers its prompt based on what you're doing and the context of the file and application and creates an autocomplete based on its chat completion, which you can accept or ignore like any autocomplete.
You can start writing test templates and it will fill them out for you, and then write the next tests based on the inputs of your methods and the imports in the test class. You can write a whole class without any copilot usage and then start writing the xmldocs and it will autocomplete them for you based on work you already did. Try it for yourself if you haven't already, it's pretty useful.
I read the article (not the study only the abstract) and they were getting paid an hourly rate. It did not mention anything about whether or not they had expirence in using llms to code. I feel there is a sweet spot, has to do with context window size etc.
I was not consistently better a year and a half ago but now i know the limits caveats and methods.
I think this is a very difficult thing to quantify but haters gonna latch on to this, same as the study that said “ai makes you stupid” and “llms cant reason”… its a cool tool that has limits.
