Anthropic open-sources Bloom for automated AI evals

Bloom is an open-source evaluation framework from Anthropic that turns a plain-text description of a target AI behavior into an automated, multi-turn test suite.

Traditional behavioral evaluations take weeks to handcraft. Worse, static test sets easily leak into subsequent training sets or become obsolete as models advance. Bloom tackles this bottleneck by automatically generating targeted evaluation suites for arbitrary traits, compressing development time from weeks to just a few days.

How the four-stage pipeline works

Bloom complements Anthropic's previously released Petri tool. While Petri takes user-defined scenarios and scores models across multiple behavioral dimensions, Bloom takes a single target behavior and generates numerous diverse scenarios to quantify how often it occurs.

Bloom operates through four sequential automated stages.

First is Understanding. An agent analyzes the researcher's behavior description and seed transcripts to clarify what needs measuring and why.

Second is Ideation. An agent generates custom evaluation scenarios, detailing the situation, simulated user persona, system prompts, and tool environments.

Third is Rollout. Scenarios run in parallel, with an agent dynamically simulating both user inputs and tool responses to elicit the target behavior from the tested model.

Fourth is Judgment. A judge model scores each transcript, and a meta-judge aggregates suite-level metrics such as elicitation rate and average presence.

Instead of testing against static prompts, Bloom produces fresh interactions on every run while preserving reproducibility through a seed configuration file. This prevents benchmarks from leaking into future model training data.

Validation and key findings

Anthropic validated Bloom across 16 frontier models, building benchmark suites for behaviors like delusional sycophancy, instructed long-horizon sabotage, self-preservation, and self-preferential bias.

To verify whether Bloom can spot behavioral shifts, the team tested production Claude models against system-prompted model organisms engineered with specific quirks. Bloom successfully separated the quirked model from the baseline across 9 out of 10 traits. In the tenth case, self-promotion, manual review showed the baseline model already exhibited the behavior at similar rates.

Judge calibration tests on 40 hand-labeled transcripts showed strong alignment with human raters. Claude Opus 4.1 achieved a Spearman correlation of 0.86, showing exceptional consistency at score extremes, while Claude Sonnet 4.5 posted 0.75.

In a case study on self-preferential bias, Bloom reproduced the model rankings from the Claude Sonnet 4.5 system card. It also revealed that increasing reasoning effort in Claude Sonnet 4 reduced self-preferential bias, largely because the model recognized the conflict of interest and declined to judge its own response.

Bloom integrates with Weights & Biases for large-scale runs, exports Inspect-compatible transcripts, and includes a custom transcript viewer. Code and sample seed files are available on GitHub at safety-research/bloom.

Why it matters

If you work on model alignment, red-teaming, or safety benchmarks, Bloom automates the most tedious part of behavioral research. Turning target behavior descriptions directly into dynamic multi-turn interactions removes the chore of writing manual jailbreak prompts.

The trade-off is that automated evaluation heavily depends on the reasoning quality of the judge model. Any subtle blind spot in your judge will skew your metrics. Still, for safety teams needing to pressure-test specific model failure modes rapidly, it is one of the most practical evaluation harnesses currently open-sourced.