A study of GPT-3.5-based ChatGPT's ability to solve 728 coding problems: fairly good at solving problems that existed before 2021, but struggles with newer ones
The 728-problem evaluation adds a time dimension to concerns raised by a prior analysis of ChatGPT answers on Stack Overflow, which found frequent incorrect information and excessive verbosity. Here, the gap is not simply answer quality: performance differs between older and more recent programming problems.
It also qualifies broad model-to-model comparisons such as reported performance drift across ChatGPT versions. Coding capability should be assessed against the age and currency of the task set, not treated as a fixed attribute of a model.
First-order effects
Developers using GPT-3.5-based ChatGPT for recently introduced coding problems have a clearer reason to treat outputs as less dependable and require validation before use.
The study narrows the model's apparent strength to problems likely represented in its pre-2021 knowledge, rather than establishing uniform coding competence.
Second-order effects
Teams evaluating coding assistants will need benchmarks that distinguish older, well-represented tasks from current ones; aggregate scores can obscure this gap.
Tool vendors and buyers may place greater weight on review workflows and fresh-task evaluations when deciding where older models can be deployed.
Third-order effects
If temporal gaps persist across coding models, competitive advantage will shift toward systems that can reliably work with current software ecosystems, not merely reproduce solutions to established problems.
The result supports a more governed model for AI-assisted development: claims of coding capability will increasingly need task-specific, time-aware evaluation rather than broad demonstrations.
The trend: Coding copilots are moving from generalized benchmark claims toward evaluation based on freshness, reliability, and the controls surrounding production use.
How does an #AI code generator compare to a human programmer? — Testing GPT-3.5's ability to address 728 coding problems from the LeetCode testing platform in 5 programming languages — https://spectrum.ieee.org/... #GenAI via — @ieeespectrum
In other words, ChatGPT is largely useless for generating code for real and enduring economically interesting software-intensive systems. https://spectrum.ieee.org/...
Do LLMs understand programming problems? The data would suggest the answer is no. When presented with problems in the training data set, ChatGPT did okay'ish (though even its best results would get any programmer fired). But on newer problems, it choked. https://spectrum.ieee.org…
“As coding evolves, ChatGPT has not been exposed yet to new problems and solutions. It lacks the critical thinking skills of a human and can only address problems it has previously encountered.” AI: good for pattern matching, bad for critical thinking https://spectrum.ieee.org/..…
Exploring ChatGPT's role in coding: useful for generating code snippets, explaining code, and aiding in debugging. Always review its suggestions! #Coding #AI 🔗https://spectrum.ieee.org/ ...
ChatGPT's coding skills are impressive, but not perfect. A new study shows it excels at solving older coding problems, but struggles with newer ones, suggesting it lacks the critical thinking skills... #AI #MachineLearning #ArtificialIntelligence #NLP #LLM https://spectrum.ieee.o…
🤖 How good is ChatGPT at coding? A study in IEEE Transactions on Software Engineering shows its success rate ranges from 0.66% to 89%, depending on task difficulty and language. It excels at older problems but struggles with newer ones. #AI #Coding #Cha... https://spectrum.ieee.o…
Essentially, as coding evolves, ChatGPT has not been exposed yet to new problems and solutions. It lacks the critical thinking skills of a human and can only address problems it has previously encountered. https://spectrum.ieee.org/...