California pushes strict AI safety bill as Big Tech pushes back

While Republicans in Milwaukee are promising to tear up federal AI limits, California is trying to write the toughest playbook in the nation.

Allies of Donald Trump are already pitching Manhattan Projects for military AI and vowing to scrap federal rules. But California's Democratic-controlled Legislature is doing the exact opposite: pushing forward a sweeping bill to force the world's best-funded AI labs to run mandatory safety tests before shipping frontier models.

The bill, authored by San Francisco state Senator Scott Wiener, targets models trained with massive amounts of computing power. Before releasing them to the public, companies would have to certify that their software can't be used for catastrophic harms, such as whipping up chemical or biological weapons, hacking critical infrastructure, or taking down power grids.

Tech giants break cover

Silicon Valley usually loves to talk about responsible AI, but this bill crossed a clear line for them.

Instead of hiding behind trade groups, major tech companies are publicly putting their names on protest letters. Alice Friend, Google's head of AI and emerging tech policy, wrote directly to California lawmakers, arguing the provisions are technically infeasible and would punish developers even when they act responsibly.

Meta and Microsoft have also sent letters voicing serious concerns. Dylan Hoffman, executive director at TechNet, called the move to step out from behind industry associations an extraordinary step that shows just how high the stakes are.

Under the bill, companies would submit detailed safety reports to a newly minted Frontier Model Division, which would hold the authority to expand which models fall under regulation. To appease the startup community, the bill also proposes building a state-run cloud computing cluster so researchers and small founders aren't at the mercy of Big Tech's cloud bills.

Nicol Turner Lee from the Brookings Institution noted that while tech leaders previously made voluntary safety pledges in Washington, Wiener's bill turns those into legally binding mandates with real accountability, which has rattled the entire industry.

Why it matters

For builders and researchers, the promise of state-funded compute sounds great, but the liability question is a massive headache. If base-model providers fear state-level legal blowback for catastrophic misuse, open-source weights and raw developer access could be locked down significantly. Founders relying on open models to move fast might suddenly find fewer unrestricted tools on the shelf.

For the AI landscape, California is once again proving that when Washington stalls, Sacramento takes the wheel. Over 450 AI bills have surfaced across statehouses this year, but Wiener's proposal is the one setting the tone. If Trump wins in November and repeals Biden's federal AI executive order, California's framework will effectively become the de facto national standard, forcing global tech giants to comply regardless of what the federal government does.

We think targeting catastrophic bioweapon and infrastructure risks makes sense on paper, but empowering an administrative body to constantly move the goalposts on covered models introduces genuine uncertainty. Pushing for strict legal liability before the industry even agrees on how to evaluate catastrophic capability looks like a heavy-handed shortcut that could drive ambitious teams to incorporate elsewhere.