Claude gets no-code evals for Agent Skills

Most people building custom skills for AI agents aren't engineers. They are lawyers, project managers, or domain experts who know their workflows inside and out, but have zero easy ways to tell if an update to the underlying model just quietly broke their setup.

Anthropic is tackling that reliability headache with a major upgrade to its skill-creator tool for Claude. The release brings software testing discipline like evals, automated benchmarks, and A/B testing directly into a no-code workflow.

The features are now live across Claude.ai, Cowork, and as a Claude Code plugin on GitHub.

Bringing software testing to non-coders

Anthropic divides agent skills into two camps: capability uplift skills (teaching Claude tricks the base model can't reliably do, like generating structured files) and encoded preference skills (locking in a specific company SOP, like vetting an NDA).

The new skill-creator allows authors to write evals by setting up test prompts, attaching sample files, and specifying what good output looks like. The system then runs tests automatically to spot regressions before they hit real work.

Anthropic noted that its own PDF skill previously struggled to place text onto non-fillable forms without coordinates. Evals isolated exactly where the model failed, allowing the team to ship a fix anchored to extracted text coordinates.

A new benchmark mode tracks pass rates, latency, and token consumption across updates. Results stay with the user and can live locally or plug straight into standard developer CI pipelines.

Blind testing and smarter triggers

To keep test runs quick and clean, skill-creator now spins up independent agents in parallel, preventing leftover context from bleeding across prompts.

It also adds comparator agents for blind A/B testing. Two versions of a skill—or a skill versus bare prompting—are evaluated without the judging agent knowing which is which, cutting out creator bias when tweaking instructions.

Trigger precision got an upgrade too. Because writing a prompt trigger that neither over-fires nor ignores relevant requests is notoriously tricky, a new description optimizer analyzes past prompts to suggest cleaner triggers. Anthropic tested it across six public document skills and improved trigger accuracy on five of them.

Why it matters

We think this is one of Anthropic's smartest product updates in months. In the agent race, everyone focuses on making bots easy to create, while ignoring how brittle they become when foundation models update underneath them.

For creators and workplace operators, this bridges a massive gap. You no longer need an engineering squad to verify whether your custom business logic still functions after a model refresh. If you run business-critical processes inside Claude, testing this now is an obvious win.

It also introduces a clever reality check: capability uplift skills eventually get made redundant by smarter base models. Running regular evals gives teams clear data showing when Claude has natively outgrown a custom skill, letting you retire technical debt.

The catch? Creating high-quality evaluation sets still requires rigorous thinking. The interface may be no-code, but curating edge cases remains hard work that busy teams might still be tempted to skip.