California takes another swing at regulating AI giants

Last September, California Governor Gavin Newsom vetoed Senate Bill 1047, which would have forced developers of $100M+ AI models to run mandatory safety tests. Silicon Valley celebrated, while safety advocates were crushed. Newsom called the bill too blunt and promised an alternative.

On Tuesday, we finally got it. A 52-page report from a powerhouse group of AI academics delivers a blunt reality check: tech giants are simply not capable of understanding their own models' dangers alone.

Ditching compute metrics for real scrutiny

Co-authored by Stanford HAI’s Fei-Fei Li, UC Berkeley’s Jennifer Tour Chayes, and Carnegie Endowment head Mariano-Florentino Cuéllar, the report outlines a new governance framework. The central premise is simple: self-regulation is failing.

The authors note that since Newsom’s veto, chain-of-thought reasoning capabilities have exploded. Evidence linking frontier models to chemical, biological, radiological, and nuclear (CBRN) weapon risks has also grown, with top labs even self-reporting alarming capability spikes.

Critically, the authors ditch the old approach of using compute thresholds as the primary regulatory trigger. Compute requirements change constantly, and raw training power ignores how tools behave in the wild. Instead, the framework focuses on initial risk evaluations and downstream impacts.

To fix the systemic opacity around safety testing, the report pushes for mandatory third-party evaluations, whistleblower protections, and public incident reporting. Independent researchers have unmatched diversity and scale compared to in-house teams, but they need protection. The report demands legal safe harbor so companies cannot use restrictive terms of service to sue or silence researchers who uncover flaws.

The black box problem

Independent evaluations sound great on paper, but they require deep access to models and internal data—something companies fight tooth and nail.

Even vetted partners are getting locked out. Safety evaluation firm Metr recently posted that OpenAI gave them significantly less time to test its o3 model than previous systems, along with restricted access to data and internal reasoning chains. Metr noted these constraints made robust capability assessments nearly impossible. OpenAI claimed it is exploring better data-sharing methods, but the episode shows just how tight corporate control remains.

Earlier this month, Anthropic CEO Dario Amodei called for federal rules requiring frontier labs to publish risk mitigation plans on their websites. But Tuesday’s report dismisses mere self-disclosure as inadequate. Developers alone cannot see their own blind spots.

Why it matters

California’s new blueprint is trying to transform AI safety from a corporate PR exercise into an enforceable, independent discipline.

For researchers and white-hat evaluators, this is a massive deal. A legal safe harbor would finally protect independent testers from copyright and terms-of-service lawsuits when they probe commercial models. For startups, moving away from rigid compute-based thresholds avoids penalizing efficient architectures simply because they cross an arbitrary training budget.

We think shifting away from raw compute metrics is a smart, modern pivot. Compute definitions age terribly fast as inference and post-training techniques evolve. Focus on real-world impact makes far more technical sense. However, the gaping flaw remains enforcement: forcing secretive labs to grant outsiders meaningful access inside proprietary reasoning systems runs straight into trade-secret defenses.

There is also a ticking political clock. A Republican-controlled Congress and allies like OpenAI are pushing for a 10-year federal moratorium on state AI laws. Lead author Scott Singer argues California can harmonize common-sense policies across states, but the state will have to fight Washington to make any of this stick.