OpenAI and Anthropic open models to US safety watchdogs

For the first time, the biggest AI labs are letting the feds kick the tires on their frontier models before the public ever sees them.

On Thursday, OpenAI and Anthropic signed formal agreements with the U.S. AI Safety Institute, which sits inside the Commerce Department's NIST. Under the memorandums of understanding, government researchers will get access to major new models both before and after their release to evaluate safety risks and collaborate on mitigations.

Government gets the keys before launch

The deal goes beyond a simple demo. The U.S. AI Safety Institute said it will provide concrete safety feedback to both companies and work closely with its counterpart agency in the UK.

OpenAI CEO Sam Altman posted on X that he was happy to reach the agreement for pre-release testing. Anthropic co-founder Jack Clark noted that tapping the government's expertise would strengthen their ability to identify risks before mass deployment. The institute's director, Elizabeth Kelly, called the pact an important milestone, though she added it is just the start.

The timing is wild for both companies. OpenAI is reportedly negotiating a fresh funding round at a valuation topping $100 billion led by Thrive Capital, while its weekly active users just doubled over the past year to 200 million. Anthropic, heavily backed by Amazon, was recently valued at $18.4 billion.

Running from California's hard law?

The federal handshake didn't happen in a vacuum. Just a day earlier, California lawmakers passed SB 1047, a controversial bill that would mandate strict safety protocols and kill switches for frontier models before training begins. Governor Gavin Newsom has until September 30 to sign or veto it.

Both OpenAI and Anthropic have pushed back against SB 1047, arguing that heavy-handed state rules would stifle innovation and hurt open-source builders. By voluntarily partnering with NIST under the Biden administration's October 2023 executive order, the frontier labs are setting up an alternative path: voluntary collaboration instead of statutory liability.

They are also trying to clean up an optics problem. In June, current and former OpenAI staffers published an open letter warning that financial incentives make internal corporate governance insufficient for AI safety. Around the same time, reports surfaced that the FTC and DOJ were preparing antitrust inquiries into OpenAI, Microsoft, and Nvidia.

Why it matters

This is a classic regulatory head-fake, but a very effective one.

By inviting the federal safety institute into their testing pipeline, OpenAI and Anthropic are trying to kill two birds with one stone. They show policymakers in Washington that industry leaders can be responsible partners, while handing Governor Newsom a massive talking point if he chooses to veto California's SB 1047.

For enterprise users, this pre-release vetting is a net positive. Knowing a foundation model has gone through testing by both U.S. and UK safety bodies removes a layer of legal and compliance anxiety before deploying these tools in production.

However, we think there are real downsides. These are non-binding agreements. If the institute flags a severe capability risk, there is no statutory mechanism here to force a company to halt a deployment. Furthermore, as safety evaluations become institutionalized between government labs and well-funded giants, the testing overhead will naturally favor deep-pocketed incumbents, leaving open-source creators and smaller startups out in the cold.