Law Professors Prefer AI Over Peer Answers - Journal Article - Stanford Law School

logo-footer logo-full logo-stanford-university logo menu-close menu<br>Close IconIcon with an X to denote closing.<br>Play IconPlay icon in a circular border.

Skip to main content

Search

Abstract

Large language models (LLMs) are increasingly promoted as educational tutors, yet most evaluations focus on domains with a single ground truth. Many disciplines, however, hinge on judgment: reasoning, weighing ambiguity, and reaching defensible conclusions. Law provides a sharp test. We conducted a blinded evaluation of short-answer tutoring in contracts courses with sixteen U.S. law professors. Participants created 40 representative questions, wrote answers, and judged 2,918 anonymized comparisons between human and LLM responses. Professors rated LLMs far higher than their peers (average win rate = 75.33%), with models performing similarly to the best instructor. LLM responses were also rarely flagged as harmful (3.53%, vs 12.06% for professors). Preferences for LLM answers were consistent across evaluators and reflected shared professional standards. Our evaluation can be reliably extended to additional models by employing a separate LLM as a judge, rendering expert agreements an effective, scalable method to evaluate AI tutors in judgment-rich domains.

Details

Author(s):

Julian Nyarko<br>Alejandro Salinas<br>Carly Frieders<br>Neel Guha<br>Sibo Ma<br>Ralph Anzivino<br>Ian Ayres<br>Oren Bar-Gill<br>Omri Ben-Shahar<br>Stephen Friedman<br>George Geis<br>Sue Guan<br>Christoph Henkel<br>Stephanie Hoffer<br>Gregory Klass<br>Larasz Moody-Villarose<br>Sarath Sanga<br>Keith Sharfman<br>Justin Simard<br>Rebecca Stone<br>David Wishnick

Publish Date:<br>May 27, 2026

Publication Title:

Social Science Research Network

Format:

Journal Article

Page(s) 61

Citation(s):

Julian Nyarko, Alejandro Salinas, Carly Frieders, Neel Guha, Sibo Ma, Ralph Anzivino, Ian Ayres, Oren Bar-Gill, Omri Ben-Shahar, Stephen Friedman, George Geis, Sue Guan, Christoph Henkel, Stephanie Hoffer, Gregory Klass, Larasz Moody-Villarose, Sarath Sanga, Keith Sharfman, Justin Simard, Rebecca Stone & David Wishnick, Law Professors Prefer AI Over Peer Answers, Social Science Research Network 61 (2026).

Related Organization(s):

Stanford Law AI Initiative

Attachment(s):

Download Now

Link(s):

Read More

Other Publications By

Julian Nyarko

Stanford Law AI Initiative

Back to the Top