Critics say California's AI safety bill, which requires “kill switches” for AI models, will force AI startups to leave, harm open-source models, and more
Tech companies launch fightback against proposed law to introduce ‘kill switch’ on powerful artificial intelligence models
Financial Times
Context & Ripple Effects
The proposal put California at the center of a widening dispute over whether frontier-model safeguards can be imposed through state law without changing where and how models are built. The initial industry resistance later hardened into a joint startup opposition letter, showing that the concern extended beyond a few large companies.
The argument is not simply over a technical control: critics frame the requirement as a potential constraint on startup formation and open-model distribution. That conflict subsequently made the bill’s enforcement design a target for legislative amendments that reduced the attorney general’s power.
First-order effects
AI companies and startups face an immediate lobbying and compliance fight over a proposed requirement to retain a shutdown capability for powerful models.
Open-source developers become a central constituency in the debate, as critics argue that obligations designed around controlled deployment may be difficult to reconcile with broadly distributed models.
Second-order effects
The opposition raises the political cost of California acting alone, encouraging lawmakers to narrow scope or enforcement rather than accept the proposal unchanged.
Large AI companies can use the dispute to press for a national framework; OpenAI later explicitly argued that AI safety rules should be set federally, rather than through California-specific requirements.
Third-order effects
If similar proposals persist, safety controls may become a differentiator between labs able to document and govern model access and smaller or open-development projects that cannot readily do so.
The episode points to a continuing jurisdictional contest: state-level AI rules can shape product and deployment choices even as industry advocates seek a single federal standard.
The trend: AI governance is moving from broad safety principles toward operational obligations that test the compatibility of frontier-model controls with startup and open-source development.
The proposed rules include - annual tests that to certify the AI model is “safe” - required reporting of “safety incidents” to the government - a “kill switch” to turn off the AI system in case it goes rogue - Fines up to 30% of model development costs https://www.ft.com/...
In the least surprising turn of events, companies that were saying is a dangerous threat to humanity that needs to be regulated are now complaining about the proposed regulations. The rules do seem overly broad and in some cases based on based on science fiction concerns. [image]
Almost all these companies said they supported regulation for AI. Now that a modest amount of regulation in the US is possible, they all oppose it. Why should we trust anything these companies say?
Weird the way a bunch of people that are calling for Universal Basic Income freak out at the first sign that they might need to spend a few dollars on paperwork and internal testing.
Great point from @GaryMarcus . If we don't make the AI companies liable for large harms, the public (and taxpayers) are liable. https://garymarcus.substack.com/ ... [image]
Excuse me for thinking that a lot of people here talking about regulations and CA SB-1047 don't know the slightest thing about how the actual world works. Take for example explosives, which are very much a dual-use technology. We don't say, “go ahead, manufacture whatever the...
Interesting to see Anthropic joining TechNet, the trade group opposing SB 1047. That means OpenAI, Anthropic, Google, Meta, Amazon, Apple, IBM, and Andreessen Horowitz all now belong to orgs opposing the bill. Hardly looking like regulatory capture!
The effort to protect innovation and open source continues. I believe we're all better off if anyone can carry out basic AI research and share their innovations. Right now, I'm deeply concerned about California's proposed law SB-1047. It's a long, complex bill with many parts... …
If this CA AI bill passes, it will be a huge boon for VC investors in WA, TX, MA, NY, and FL. Easier for entrepreneurs to relocate than to be exposed to such a large unknown new risk.
SB 1047 bill, assigning liability for unintended use to model developers, is total idiocy. No thought given to how this will cripple open source and weaken CA's position as the global heart of innovation.
If you think that AI safety is a grift by AI companies to hype up their products, I think SB 1047 is a brilliant move. “Oh, you're telling me your products might be catastrophically dangerous? Thanks! We will establish that if they are, you are liable.”
“Imagine if we passed a law that says if anyone uses a motor in a harmful way, the *motor manufacturer* is liable.” This would be ridiculous. Well that is what California's SB-1047 is proposing for AI models. Regulate the application. Not the math.
Yup. Those behind SB 1047 have summarily ignored all substantive feedback and are trying to cram horrible legislation through with the aim of political points and sci fi X-risk protection. We all need to be more vocal. https://press.airstreet.com/ ...
I share @AndrewYNg's serious concerns with much of the open-source and broader AI community about California's SB-1047 proposal. Among many issues, the covered models definitions, shutdown capability, and enormous cost for compliance would be a huge blow to both CA and US...
new on @airstreetpress: sb-1047 - california's proposed ai law will be a disaster for start-ups and open source but its opponents are partly to blame for this we need to organize better, ditch loser arguments like regulatory capture, and begin learning from our opponents 🧵 [image…