Sakana AI adds evidence checks to its open-source AI scientist
Sakana AI just released The Agentic Scientist v3 under an Apache 2.0 license, turning markdown research briefs into automated experiments, figures, and draft paper sections. The biggest upgrade is built-in evidence checks—running parallel baselines and ablations to test whether findings hold up before humans review them.
The Agentic Scientist v3 is an open-source research platform from Sakana AI that turns a simple markdown research brief into runnable code experiments, figures, and draft paper sections.
The system runs coding agents in parallel to handle scientific discovery from end to end, licensed under Apache 2.0 with a companion paper on arXiv.
What it does
The big shift in this third-generation release is built-in verification.
Previous AI research agents often fell into a trap: optimizing benchmarks for higher numbers without actually investigating the underlying mechanics.
To fix that, the new system automatically builds comparison baselines and runs ablation studies alongside your main experiment.
It stress-tests whether an apparent improvement actually holds up, pulls the metrics into clean figures, and writes structured paper sections based on the evidence.
All the orchestration code is out in the open under Apache 2.0, meaning anyone can audit how the checks work or adapt the pipelines to their own lab setup.
Keep humans in the loop
Automating the loop does not mean scientists can walk away.
Sakana notes that researchers must still evaluate the outputs. In early evaluations, mis-scored candidate experiments highlighted why scientific judgment cannot simply be delegated to agents.
The system takes care of the parallel plumbing and tedious sanity checks, but the human remains the reviewer of record.
Why it matters
AI-generated science is facing mounting pushback over reproducibility, with too many tools churning out unvetted synthetic claims. Sakana's approach is smart because it bakes evidence-gathering, baseline generation, and ablation directly into the workflow instead of treating them as afterthoughts.
If your team spends half its research time wiring up standard baselines and sanity-checking experimental variations, this repo is well worth spinning up. Just don't treat the draft sections as final truth—it is a fast, tireless junior assistant, not the principal investigator.
Sources (1)
- runtimewire.com(opens in a new tab)Originalruntimewire.com