ChatGPT gets code questions wrong 52% of the time
ChatGPT gets code questions wrong 52% of the time
But its suggestions are so annoyingly plausible
52% is an understatement
1One of the main reasons was how detailed ChatGPT’s answers are. In many cases, participants did not mind the length if they are getting useful information from lengthy and detailed answers. Also, positive sentiments and politeness of the answers were the other two reasons.
Man, this answer is long, detailed, polite... it's great!
Sure, but it's wrong. It's just complete bullshit.
Yeah, sure.... still...
1Title feels misleading, it gets stack overflow questions wrong 52% of the time
However it got 77% of easy Leetcode questions correct. Also I believe that's first try, which is not generally how chatgpt should be used.
Also also, you should probably be using a coding specific model if you want good coding results
0Every leetcode question has been answered a billion times and you train it on those billions of answers, it should get those right.
1