Critics say California's AI safety bill, which requires “kill switches” for AI models, will force AI startups to leave, harm open-source models, and more
Financial Times
Context & Ripple Effects
This report captures the early fault line in California’s AI-policy debate: a safety requirement aimed at model control was immediately framed by critics as a constraint on startup formation and open development. The concern later became organized industry opposition, including a letter backed by Y Combinator and 140 AI startups.
The subsequent coverage suggests the objections had practical legislative relevance: lawmakers later weakened the proposal through amendments that reduced the attorney general’s power to sue. That makes this less a one-off dispute over safeguards than a test of how state rules can govern rapidly evolving AI development.
First-order effects
AI developers covered by the proposal would need to account for a shutdown capability, adding a compliance and design requirement to model development and deployment.
Startups and open-source developers face immediate uncertainty over whether California remains a workable base for building or releasing advanced models, according to the bill’s critics.
Industry groups and major AI companies gain an incentive to coordinate around a federal rather than state-by-state framework, a position later voiced in OpenAI’s call for federal-level regulation.
Third-order effects
If high-capability AI rules are set primarily at the state level, developers may increasingly design governance, release practices, and organizational structures around the most consequential jurisdiction rather than a uniform national standard.
The lasting policy challenge is likely to be separating obligations for the most capable models from rules that unintentionally raise barriers for smaller firms, researchers, and open-source releases.
The trend: AI governance is shifting from broad calls for safety toward contested implementation rules that determine who can develop, release, and operate advanced models.
The proposed rules include - annual tests that to certify the AI model is “safe” - required reporting of “safety incidents” to the government - a “kill switch” to turn off the AI system in case it goes rogue - Fines up to 30% of model development costs https://www.ft.com/...
In the least surprising turn of events, companies that were saying is a dangerous threat to humanity that needs to be regulated are now complaining about the proposed regulations. The rules do seem overly broad and in some cases based on based on science fiction concerns. [image]
Almost all these companies said they supported regulation for AI. Now that a modest amount of regulation in the US is possible, they all oppose it. Why should we trust anything these companies say?
Weird the way a bunch of people that are calling for Universal Basic Income freak out at the first sign that they might need to spend a few dollars on paperwork and internal testing.
Great point from @GaryMarcus . If we don't make the AI companies liable for large harms, the public (and taxpayers) are liable. https://garymarcus.substack.com/ ... [image]
Excuse me for thinking that a lot of people here talking about regulations and CA SB-1047 don't know the slightest thing about how the actual world works. Take for example explosives, which are very much a dual-use technology. We don't say, “go ahead, manufacture whatever the...
Interesting to see Anthropic joining TechNet, the trade group opposing SB 1047. That means OpenAI, Anthropic, Google, Meta, Amazon, Apple, IBM, and Andreessen Horowitz all now belong to orgs opposing the bill. Hardly looking like regulatory capture!
The effort to protect innovation and open source continues. I believe we're all better off if anyone can carry out basic AI research and share their innovations. Right now, I'm deeply concerned about California's proposed law SB-1047. It's a long, complex bill with many parts... …
If this CA AI bill passes, it will be a huge boon for VC investors in WA, TX, MA, NY, and FL. Easier for entrepreneurs to relocate than to be exposed to such a large unknown new risk.
SB 1047 bill, assigning liability for unintended use to model developers, is total idiocy. No thought given to how this will cripple open source and weaken CA's position as the global heart of innovation.
If you think that AI safety is a grift by AI companies to hype up their products, I think SB 1047 is a brilliant move. “Oh, you're telling me your products might be catastrophically dangerous? Thanks! We will establish that if they are, you are liable.”
“Imagine if we passed a law that says if anyone uses a motor in a harmful way, the *motor manufacturer* is liable.” This would be ridiculous. Well that is what California's SB-1047 is proposing for AI models. Regulate the application. Not the math.
Yup. Those behind SB 1047 have summarily ignored all substantive feedback and are trying to cram horrible legislation through with the aim of political points and sci fi X-risk protection. We all need to be more vocal. https://press.airstreet.com/ ...
I share @AndrewYNg's serious concerns with much of the open-source and broader AI community about California's SB-1047 proposal. Among many issues, the covered models definitions, shutdown capability, and enormous cost for compliance would be a huge blow to both CA and US...
new on @airstreetpress: sb-1047 - california's proposed ai law will be a disaster for start-ups and open source but its opponents are partly to blame for this we need to organize better, ditch loser arguments like regulatory capture, and begin learning from our opponents 🧵 [image…