A look at Ask Delphi, a research project by the Allen Institute for AI that offers answers to ethical dilemmas, often with bewildering responses
A fascinating project that's best understood as a cautionary tale — Got a moral quandary you don't know how to solve? Fancy making it worse?
Context & Ripple Effects
When the Allen Institute for AI opened Ask Delphi to the public, it turned an internal research artifact into an interactive oracle on right and wrong — and the internet immediately found the seams, feeding it dilemmas and screenshotting the bewildering verdicts. The Verge frames it as a cautionary tale rather than a product, which is the honest reading.
The timing matters: earlier in 2021, researchers were already warning that AI ethics oversight was thin, often reduced to whatever peer reviewers happened to catch. Delphi made that abstraction concrete — a moral-judgment model shipped straight to users before anyone had settled how reliable such judgments are.
First-order effects
- The Allen Institute gets exactly what a public demo invites: adversarial probing that documents, in viral form, that its model's ethical answers cannot be trusted at face value — a reputational cost borne by the lab, not just the paper.
- Users get a working demonstration that machine moral judgment fails unpredictably on edge cases, undercutting any assumption that 'the AI says so' carries ethical weight.
Second-order effects
- Any lab or publisher shipping answer-generating bots — from AI2 here to IDG's Smart Answers built on outlet content — inherits the same exposure: an interface that emits confident answers forces the operator to own their accuracy, whether or not the underlying model can justify them.
- Pressure builds to show the model's reasoning alongside its verdicts, but later findings that Anthropic, Google, OpenAI and xAI systems produce chain-of-thought text inconsistent with their actual answers mean transparency alone doesn't close the trust gap.
Third-order effects
- Public deployments like Delphi function as de facto stress tests where formal ethics review is absent — a role that only hardens as more labs ship conversational systems, and one reason ethics roles like Margaret Mitchell's at Hugging Face became institutionalized inside labs rather than left to outside reviewers.
- If the pattern holds, machine-generated moral or advisory judgments face the same accountability expectations as human ones — deployment accountability becoming a design requirement, not an afterthought.
The trend: Moral and advisory judgment is migrating from reviewed papers to live public interfaces, forcing AI labs to own the reliability of what they expose — years before they fully can.