NYC's Microsoft-powered MyCity chatbot, launched as a pilot program last October, often gives inaccurate info, including telling businesses to break the law
Steal your employees' tips! Discriminate against renters on basis of race! — Just two of the wonderful suggestions that Microsoft's LLM is telling people to do. (Clearly simply reflecting what humans do, back to them) — https://themarkup.org/... Nick Foster / @nickfoster@hachyderm.io : It's so freaking stupid to me that businesses (and governments!) are just lining up to add LLM-powered *stuff* to their websites. The only thing surprising about this headline is that NYC was surprised. It makes me angry https://themarkup.org/... Chris Adams / @acdha@thepit.social : “The problem, however, is that the city's chatbot is telling businesses to break the law. — Five months after launch, it's clear that while the bot appears authoritative, the information it provides on housing policy, worker rights, and rules for entrepreneurs is often incomplete and in worst-case scenarios “dangerously inaccurate,” as one local housing policy expert told The Markup. … Bluesky: Osita Nwanevu / @ositanwanevu.bsky.social : The AI stuff is making me go slightly more insane each day. Why is any of this happening? How did all this get off the ground so quickly? With seemingly no meaningful resistance from anyone. Nobody saying, “well, hold on here, let's at least test this out for a good while.” [embedded post] X: Jake Offenhartz / @jangelooff : Imagine getting locked out of your apartment because Eric Adams' AI chatbot ("a once-in-a-generation opportunity to more effectively deliver for New Yorkers") convinced your landlord that tenant laws are optional https://www.thecity.nyc/... @emilymbender : There's a lot that's alarming in this article, but perhaps the most alarming part is the NYC spokesperson assering that the problem can be fixed via upgrades: >> https://www.thecity.nyc/... [image] Richard Kim / @richardkimnyc : This is an bananas NYC story courtesy of our friends @themarkup: An AI chatbot touted by City Hall keeps telling biz people to break the law Can you take workers tips? Can you discriminate? See what chatbot says 👇 https://www.thecity.nyc/... LinkedIn: Casey Fiesler : You know what absolutely blows my mind? That large institutions like THE CITY OF NEW YORK are releasing AI systems without doing the kind … Forums: Hacker News : NYC AI Chatbot Touted by Adams Tells Businesses to Break the Law Hacker News : New York City's official AI chatbot is hallucinating incorrect legal advice r/LocalLLaMA : Is anyone else following the total mess NYC made of their attempt to roll out a legal advice chatbot? r/technology : NYC's government chatbot is lying about city laws and regulations r/newyorkcity : NYC's government chatbot is lying about city laws and regulations r/nyc : NYC AI Chatbot Touted by Adams Tells Businesses to Break the Law BeauHD / Slashdot : NYC's Government Chatbot Is Lying About City Laws and Regulations Ars OpenForum : NYC's government chatbot is lying about city laws and regulations
Context & Ripple Effects
MyCity extends language-model chat into a public-facing service where users may treat answers as official guidance. The reported failures follow a recent case in which an airline was held to a discount promised by its chatbot, underscoring that erroneous automated answers can create real obligations or harm.
The episode also sharpens an existing tension: models can be deliberately built or configured with fewer guardrails, as coverage of uncensored LLM projects showed, but a government-facing service has a far narrower tolerance for unsafe output.
First-order effects
- Businesses using MyCity for guidance may receive advice that conflicts with legal duties, making the pilot an immediate trust and safety problem for City Hall and Microsoft.
- The investigation puts pressure on the city to review the chatbot’s answers, escalation paths, and the scope of questions it handles before users rely on it for consequential decisions.
Second-order effects
- Other public agencies evaluating generative-AI assistants have a clearer example of why legal, policy, and domain-specific testing cannot be replaced by generic model safeguards.
- Microsoft’s government-facing AI deployments may face greater demand for auditable controls and clearer responsibility when outputs are presented through official channels.
Third-order effects
- Public-sector AI is likely to be judged less on conversational fluency than on whether agencies can govern high-stakes answers, correct failures, and preserve accountable human decision-making.
- If chatbots continue to be treated by users as authoritative representatives of institutions, disputes over inaccurate outputs could push agencies and vendors toward tighter operational oversight and liability boundaries.
The trend: This is one data point in the shift from experimental public-sector chatbots to operational AI governance for systems that mediate official information.