We Asked 100+ AI Models to Write Code. The Results: AI-generated Code That Works, But Isn’t Safe

I find that very difficult to believe. If for no other reason that there is an implementation in the wiki page for Levenshtein distance (and wiki is known to be very prominant in the training sets used for foundational models), and that trying it just now and it gave a perfectly functional implementation.

I'm primarily transfixed, not by the example in your comment, but that you don't voice the "th" in "with".

Depends on their definition of "working" .

I tried asking an AI to make a basic webrtc client to make audio calls - something that has hundreds of examples on the web about how to do it from the first line of code to the very last. It did generate a complete webrtc client for audio calls I could launch and see working, it just had a couple tiny bugs:

• you needed an user id to call someone and one was only generated when you call (effectively meaning you can only call people if they are calling someone)
• if you fixed the above and managed to make a call between two users, the audio was exchanged but never played.

Technically speaking, all of the small parts worked, they just didn't work together. I can totally see someone ignoring that fact and treating this as an example of "working code".

I was surprised by that sentence, too.

But I see from my AI-using coworkers that there are different values in use for "it works".

Yeah, I've found AI generated code to be hit or miss. It's been fine to good for boilerplate stuff that I'm too lazy to do myself, but is super easy CS 101 type stuff. Anything that's more specialized requires the LLM to be hand-held in the best case. More often than not, though, I just take the wheel and code the thing myself.

By the way, I think it's cool that you use Old English characters in your writing. In school I used to do the same in my notes to write faster and smaller.