Google Labs launches Stax to take the guesswork out of LLM evals

Every AI developer knows the feeling. You tweak a prompt, run a couple of sample queries, and the output just *feels* a little crisper. But is it actually better? You have no real idea.

Google Labs just teamed up with Google DeepMind to launch Stax, an experimental tool designed to pull developers out of the endless cycle of "vibe testing" and bring real engineering rigor to LLM evals.

Firing the vibes, hiring an autorater

Traditional code is deterministic: if two plus two doesn't equal four, the test fails. LLMs are non-deterministic, meaning traditional unit testing breaks down completely.

Building an evaluation pipeline from scratch is usually a nightmare of juggling CSV files, wrangling API limits, parsing JSON, and writing glue code. So people just eyeball the results and ship with their fingers crossed.

Stax simplifies the process with autoraters—a scalable implementation of the LLM-as-a-judge technique. Powered by Gemini, these raters grade another model's outputs against specific instructions.

You can upload existing CSV datasets or construct prompts and ground-truth pairs right inside the web interface.

Out of the box, Stax includes pre-built autoraters for common evaluation criteria like coherence, factuality, and conciseness. You can see repeatable benchmark numbers in minutes.

Codifying your secret sauce

The real standout feature in Stax is custom autoraters. Public LLM leaderboards might tell you which model is smarter at high school math, but they won't tell you if an agent matches your brand voice or follows your compliance rules.

Stax lets you write your own grading rubrics in plain text. You define what a good answer looks like, and the autorater runs that rubric across your entire test suite.

The tool is currently live and free to experiment with at stax.withgoogle.com, with a dedicated Discord community for feedback.

Why it matters

If you are building AI agents or wrapping LLMs for production, vibe-testing is an active liability. Stax provides an accessible middle ground between expensive human labeling and complex custom test harnesses like LangSmith or Promptfoo. It is absolutely worth uploading a few dozen difficult edge cases right now to see where your prompts actually fail.

Strategically, Google is smart to lean into evaluation infrastructure. By weaponizing DeepMind's internal eval playbook and tying autoraters closely to Gemini, Google is positioning its ecosystem as the adult in the room for serious engineering.

That said, autoraters aren't magic. LLM-as-a-judge approaches carry well-documented risks like length bias and self-preference. Stax makes automated grading frictionless, but we believe teams that rely on it blindly without periodic human calibration run the risk of mistaking algorithmic agreement for real-world reliability.