Three case studies using GPT-4o, OpenAI o1, and Claude 3.5 Sonnet for historical research show that the models are now good enough for PhD-level analysis
I had a bit of a glimpse this morning, when I used chatgpt to identify a manuscript that ended up in my backlog stack somehow. 👇 Dr. Erika Graham-Goering / @jeanneologist : Came across this and am genuinely alarmed at the idea that any of this is evidence for an LLM being a “very good historian” (a phrase which, incidentally, I previously heard used only by a pompous old bore trying to wow me with how many History Facts™ he knew while I was an underpaid museum guide). Thomas Ptacek / @sockpuppet.org : Nicholas Carlini wrote a super-popular post about working in software and security with LLMs ("Why I Use AI"); here's an almost equivalent post by a historian about using LLMs to do good history work: resobscura.substack.com/p/the- leadin... @davidcrespo : people need to frequently update their intuitions about things that are changing quickly — “Even a year ago, when I requested further reading on a topic, ChatGPT would either invent fake sources or recommend terrible ones (like a History Channel website). That is no longer the case.” X: Dave Winer / @davewiner : The leading AI models are now very good historians. https://resobscura.substack.com/ ...
Context & Ripple Effects
This debate extends the earlier view that generative systems marked a step change in AI use cases, while leaving the acceptable error rate unresolved for knowledge work. The case studies move that question into historical research rather than general-purpose prompting.
It also sharpens a split in the surrounding coverage: advocates argue models should be assessed by what they can do, as in the case for judging LLMs on demonstrated use cases, while critics challenge whether strong-looking outputs prove scholarly competence.
First-order effects
- GPT-4o, OpenAI o1, and Claude 3.5 Sonnet gain a concrete—though contested—set of historical-research examples that users can scrutinize beyond generic capability claims.
- The historian’s objection keeps the reported PhD-level standard from functioning as a settled endorsement; source quality, method, and error checking remain central to interpreting the cases.
Second-order effects
- Researchers and instructors evaluating LLM-assisted research will face pressure to separate useful discovery and reading-list support from reliable historical interpretation, echoing earlier classroom experiences with students fact-checking AI output.
- Model providers compete not only on broad benchmarks but on whether users can validate outputs in specialist workflows where persuasive mistakes carry high costs.
Third-order effects
- If domain-specific case studies become the common evidence base, AI evaluation may shift from claims of general intelligence toward repeatable, expert-reviewed workflow tests.
- The enduring divide will be less about whether LLMs can produce analysis and more about what human verification is required before that analysis can count as scholarly work.
The trend: LLM adoption is moving from broad demonstrations toward contested, domain-level assessments of where assisted work is useful and where expert oversight remains indispensable.