AI isn’t ready to replace human coders for debugging, researchers say
Even when given access to tools, AI agents can’t reliably debug software.
Laughs in Cobol.
But the only way to learn debugging is to have experience coding. So if we let AI do the coding then all the entry level coding jobs go away and no one learns to debug.
This isn't just a code thing. This is all kinds of professions. AI will kill the entry level which will prevent new people from getting experience which will have downstream effects throughout entire industries.
It already started happening before LLM AI. Have you heard the joke that we were teaching our parents how to use printers and PCs with mouse and keyboard and now we have to do the same with our children? It’s really not a joke. We are the last generation that have seen it all evolving before our eyes, we know the fundamentals of each layer of abstraction the current technology is built upon. It was natural process for us to learn all of this and now suddenly we expect "fresh people" to grasp 50 years or so of progress in 5 or so years?
Interesting times ahead of us.
Good point
Have you used any AI for programming? There is 0 chance entry level jobs will be replaced. AI only works well if what it needs to do is well defined, as a dev that is almost never the case. Also companies understand that to create a senior dev they need a junior dev they can train. Also cooperations do not trust Google, openAI, meta, ect with their intellectual property. My company made it a firedable offense if they catch you uploading IP to an AI.
Also companies understand that to create a senior dev they need a junior dev they can train.
We live in a world where every company wants people that can hit the ground running, requires 5 years of experience for an entry level job on a language that's only been out for three years. On the job training died long ago.
The junior devs are my job are way better at debugging than AI, lol. Granted they are top talent hires because no one else can break in these days.
In my experience, LLMs are good for code snippets and input on best practices.
I use it as a tool to speed up my work, but I don't see it replacing even entry jobs any time soon.
So, AI gets to create problems, and actually capable people get to deal with the consequences. Yeah that sounds about right
And it'll be used to suppress wages, because "you're not making new stuff, just fixing some problems in existing code." That you have to rewrite most of it is conveniently not counted.
That's at least what was tried with movie writers.
I’m actually quite enjoying watching the LLM evangelists fall into the trough of despair after their initial inflated expectations of what they thought stochastic text generation would achieve for the business. After a while you get used to the waves of magic bullet solutions that promise to revolutionise the industry but introduce as many new problems as they solve.
So we “fixed” the easiest part of software development (writing code) and now humans have to clean up the AI slop.
I’ll bet this lovely new career field comes with a pay cut.
I would charge more. Fixing my own code is easier than fixing someone elses code.
I think I might go insane if that was my career.
But trust me Bro, AGI is around the corner. In the meantime have this new groundbreaking feature https://decrypt.co/314380/chatgpt-total-recall-openai-memory-upgrade /s
LLMs are so fundamentally different to AGI, it's a wonder people believe that balderdash
As a very experienced python developer, I have tried using chatgpt for debugging and vibe coding multiple times and you just end up going in circles and never get to a working solution. It ends up being a lot faster just to do it yourself
Thank you! So many morons saying you can just use Generative AI to build whatever you need. That’s a no..
Absolutely agree. I just use it for some simple stuff like "every nth row in a pandas dataframe slice a string from x to y if column z is True" or something like that. These logics take time to write, and GPT usually comes up with a right solution or one that doesn't need a lot of modification.
But debugging or analyzing an error? No thanks
I'm full luddite on this. And fuck all of us.
"Give me some good warning message css" was a pretty nice use case. It's a nice tool that's near the importance of Google search.
But you have to know when its answers are good and when they're useless or harmful. That requires a developer.
They really want to enforce that quote. The cleverly as possible part doesn’t really apply. Let the idiot write the code and the more experienced person debug it. I feel like we’ve seen this with airline pilots already. Huge shortage mainly caused by retirement and regulation changes making it harder to get into the field. I guess their hope is by the time that happens with programmers AI doesn’t suck.
At least this won’t be true anymore.
I occasionally check what various code generators will do if I don't immediately know the answer is almost always wrong, but recently it might have been correct, but surprisingly convoluted. It had it broken down into about 6 functions iterating through many steps to walk it through various intermediate forms. It seemed odd to me that such a likely operation was quite so involved, so I did a quick Internet search, ignored the AI generated result and saw the core language built-in designed to handle my use case directly. There was one detail that was not clear in the documentation, so I went back to the LLM to ask that question and it gave the exact wrong answer.
I am willing to buy that with IDE integration or can probably have much richer function completion for small easy stuff I know to do and save some time, but I just haven't gotten used to the idea of asking for help on things I already know how to do.
I've found the same thing.
Whenever I ask an LLM for a pointer, I end up spending just as long (if not longer) refining the question than just figuring it out myself it's doing a search on SO it in other online resources.
But even the IDE integration is getting annoying. I write a class with some functionality baked in, and the whole time it's promoting me with a shit load of irrelevant suggested code. I get the class done, then I go to spin up a unit test. It knows which class I'm trying to create a unit test for, which is cool. But then the suggested code is usually completely wrong or it's much more convoluted than it needs to be. In the latter case, the first several characters of the suggested code is good, but then there's several lines after it of shite. And hitting tab injects all of it in, which then requires me to delete it all. So almost every time I end up hitting escape anyway.
I've heard a few people rave about 'vibe coding' - usually people with no or little programming experience. I have to assume that generated code was either for very simple atomic actions and/or it's spaghettified, inefficient garbage.
"AI" is good for pattern matching, generating boiler plate / template code and text, and generating images. Maybe also translation. That's about it. And it's of course often flawed/inaccurate so it needs human oversight. Everything else is like a sales scam. A very profitable one.
Can AI fix itself so that it gets better at a task? I don't see how that could be possible, it would just fall into a feed back loop where it gets stranger and stranger.
Personally, I will always lie to AI when asked for feed back.
It is worse. People can't even fix AI so it gets better at a task.
That's been one of the things that has really stumped a team that wanted to go all in on some AI offering. They go to customer evaluations and really there's just nothing they can do about the problems reported. They can try to train and hope for the best, but that likely won't work and could also make other things worse.
Yeah, they're cooked.
It’s always the people that don’t have a clue.
It’s also always the people that think they’ll get some benefit out of AI taking over. When they’re absolutely part of the group that’ll be replaced by AI.
It's a cargo cult. They don't understand, but they like what it promises, so they blindly worship. Sceptics become unbelievers, visionaries become prophets and collateral damages become sacrifices.
They may use different terms, but if some job became obsolete, that's just the price of a better future to them. And when the day of Revelation comes, they'll surely be among the faithful delivered from the shackles of human labour to enjoy the paradise built on this technology. Any day now...
No shit.
Are those researchers human or is this just an Ai that’s too lazy to do the work?
Color me shocked.
the tool can't replace the person or whatever
Ars Technica would die of an aneurysm if it stopped posting about generative AI for even 30 seconds
as they're the authority on tech, and all they write about is shitty generative AI from 2017, that means shitty generative AI from 2017 is the only tech worth writing about
Well, now they're just subverting expectations left and right, aren't they!
