A study of GPT-3.5-based ChatGPT's ability to solve 728 coding problems: fairly good at solving problems that existed before 2021, but struggles with newer ones
Developers using GPT-3.5-based ChatGPT for coding get a clearer reason to scrutinize answers involving post-2021 problems, rather than applying the same confidence to older and newer tasks.
The result narrows what this particular system’s strong performance on established coding problems can demonstrate; it does not establish equivalent utility on newer material.
Second-order effects
Teams evaluating coding assistants will need test sets that reflect the age and stack of their own work, alongside checks for correctness, instead of relying on broad coding benchmarks.
Providers face added pressure to distinguish genuine reasoning and up-to-date problem solving from performance that may be stronger on material resembling older public examples.
Third-order effects
If temporal gaps recur across evaluations, coding-assistant procurement and deployment will shift toward continuous, version-specific evaluation rather than one-time headline benchmark scores.
The larger market may increasingly separate assistants that serve as drafting aids from those trusted for newer production work, with verification remaining central where error costs are high.
The trend: AI coding tools are moving from general demonstrations of competence toward evidence-based assessment of how reliably each model performs on current, real-world tasks.
Do LLMs understand programming problems? The data would suggest the answer is no. When presented with problems in the training data set, ChatGPT did okay'ish (though even its best results would get any programmer fired). But on newer problems, it choked. https://spectrum.ieee.org…
In other words, ChatGPT is largely useless for generating code for real and enduring economically interesting software-intensive systems. https://spectrum.ieee.org/...
“As coding evolves, ChatGPT has not been exposed yet to new problems and solutions. It lacks the critical thinking skills of a human and can only address problems it has previously encountered.” AI: good for pattern matching, bad for critical thinking https://spectrum.ieee.org/..…
Exploring ChatGPT's role in coding: useful for generating code snippets, explaining code, and aiding in debugging. Always review its suggestions! #Coding #AI 🔗https://spectrum.ieee.org/ ...
ChatGPT's coding skills are impressive, but not perfect. A new study shows it excels at solving older coding problems, but struggles with newer ones, suggesting it lacks the critical thinking skills... #AI #MachineLearning #ArtificialIntelligence #NLP #LLM https://spectrum.ieee.o…
🤖 How good is ChatGPT at coding? A study in IEEE Transactions on Software Engineering shows its success rate ranges from 0.66% to 89%, depending on task difficulty and language. It excels at older problems but struggles with newer ones. #AI #Coding #Cha... https://spectrum.ieee.o…
Essentially, as coding evolves, ChatGPT has not been exposed yet to new problems and solutions. It lacks the critical thinking skills of a human and can only address problems it has previously encountered. https://spectrum.ieee.org/...