AI just beat law professors at their own game

For years, the standard defense against AI in higher education has been simple: sure, language models can ace multiple-choice bar exams, but they can't handle nuanced reasoning, subjective judgment, or real analytical ambiguity.

A new study led by Stanford Law School just put that argument to a brutal blind test.

Researchers led by Professor Julian Nyarko at Stanford's liftlab, alongside scholars from Yale, NYU, and the University of Chicago, published a study titled "Law Professors Prefer AI Over Peer Answers." The setup was straightforward: 16 law professors from 14 top schools went head-to-head against AI on real-world contract law questions.

The results were a landslide. Across 2,918 blind comparisons, the professors chose AI-generated responses over answers written by their own faculty peers 75% of the time.

2,918 blind matchups

The study deliberately avoided simple legal trivia. Instead, the researchers crafted 40 representative contract law questions typical of post-class office hours—the kind of scenarios that demand deep critical thinking, balancing competing doctrines, and explaining complex concepts to students.

Participating faculty wrote their own answers. Google's Gemini 2.5 Pro and NotebookLM generated alternative responses, which were calibrated in length and format to look just like human output.

The 16 professors then blindly graded nearly 3,000 randomized matchups without knowing whether an answer came from a colleague or a machine.

AI won three out of every four matchups. In fact, only a single elite human professor managed to tie with the models across the evaluation.

Even more striking was the pedagogical damage check. When asked whether an answer might confuse or mislead students, professors flagged peer-written answers as harmful 12% of the time. The AI responses? Flagged only 3.5% of the time.

Claude Opus takes the crown

As Yale Law School professor and co-author Sarath Sanga put it, in most fields AI tests have a single right answer, but law is about competing defensible positions. The experiment proved that AI can meet the latent professional standard lawyers use when judging arguments.

The team also ran tests with an AI stand-in judge across nine additional commercial models. Anthropic's Claude Opus 4.7 took the top spot, and every single model evaluated beat the professors' baseline.

Even when context constraints hampered the AI's full performance, professors still consistently preferred its explanations over human writing.

Why it matters

This study dismantles the widespread belief that LLMs are only useful for rote administrative tasks and factual recall. For students and independent learners, it shows that round-the-clock, elite-tier legal tutoring is technically viable right now.

For the legal and professional services industry, the implications are massive. If commercial models can out-reason law professors on their home turf, automated legal analysis and complex corporate contract review will advance much faster than skeptics predicted.

We think the study's design was brilliant because it relied entirely on blind peer review from domain experts rather than synthetic benchmarks. That said, as Nyarko emphasized, knowing the answer isn't the same as pedagogy, and nobody is calling for firing the faculty. But blanket skepticism against AI tutoring is no longer tenable; the conversation has officially shifted to how educators deploy these systems without eroding student critical thinking.