How LLMs are taught to reason, the downsides of “chain of thought”, how models might solve problems without reasoning, and evaluating LLMs for general reasoning
What should we believe about the reasoning abilities of today's large language models? X: @mattbeane , @rtk254 , @rodneyabrooks , and @piesaac X: Matt Beane / @mattbeane : VERY well written summary of latest science on whether genAI can reason. Devious and rather decisive tests say, for now: pretty much no. Ronen Tamari / @rtk254 : @MelMitchell1 Interesting post. The Python - ThonPy test presented reminds me of @LakeBrenden 's challenges for RL in “Building machines that learn and think like people” (2017). Clearly we've made progress since then, but how far have we actually gotten on human-like model-building? [image] Rodney Brooks / @rodneyabrooks : As always Melanie Mitchell has is very insightful and precise, and decomposes difficult questions into pieces that can be analyzed. “Can Large Language Models Reason?”. @piesaac : The gist of this seems to say probably not, and that LLMs seem to mostly rely on memorization and pattern matching
Context & Ripple Effects
This is an early articulation of the dispute over whether fluent model outputs constitute general reasoning or sophisticated pattern matching. Later coverage sharpened the empirical side: Apple researchers reported no evidence of formal reasoning in language models, while [[a:875667|OpenAI's o1 showed materially stronger reasoning performance but retained spatial weaknesses]].
The story matters because it separates task success, visible chain-of-thought, and general reasoning—distinctions that remain central as reasoning becomes a product feature and as subsequent work tests large reasoning models on classic problems.
First-order effects
- It gives model evaluators a reason to test transferable problem-solving rather than infer general reasoning from fluent explanations or isolated benchmark wins.
- It puts chain-of-thought under scrutiny as both a training/evaluation tool and an imperfect indicator of how a model reached an answer.
Second-order effects
- Model developers face pressure to publish evaluations that distinguish memorized or pattern-based success from robust performance on altered problem formulations; later tests of leading models and reasoning models on classic tasks reinforce that demand.
- Teams deploying LLMs in high-consequence workflows are pushed toward task-specific validation and output checks rather than treating a narrated rationale as sufficient assurance.
Third-order effects
- If reasoning claims continue to outpace robust evidence, competition may shift from general-intelligence rhetoric toward demonstrable reliability on bounded tasks and independently reproducible evaluations.
- The field may increasingly treat explanation quality and reasoning capability as separate measurement problems, particularly where models can produce convincing but unfaithful rationales.
The trend: LLM development is moving from treating chain-of-thought and benchmark performance as proxies for reasoning toward more adversarial, task-specific tests of reliable problem-solving.