method · first seen 4 Oct 2026
LLM-as-a-judge
Also: GRM, LaJ, LLMaJ, GEMBA, LLMaaJ, LLM judge, AI judges, LLM critic
Using language models to grade the outputs of other models, with judge reliability measured against each other.
LLM-as-a-judge means using a language model to score or compare the outputs of other models, in place of human raters or lexical metrics such as ROUGE. The sources treat it as standard infrastructure and focus on its reliability. One paper systematically compares judge prompts, rating scales and models. Others show that the choice of judge shifts LoCoMo memory scores, that an 8B critic can degrade AutoML search, and that unanimity-based model verifiers fail in multi-agent proof search. Applied work uses judges for clinical notes, hiring screens and coding benchmarks, and a small GitHub RAG repo audits its own judge.
The idea has been mainstream since roughly 2023. The live research question is the judge's calibration and failure modes, because conclusions in many papers depend on unreported judge design choices. Standardizing and auditing judges could make benchmark results comparable across studies.
Analyst note (established), 17 Nov 2026.
Mentions per week
PapersCodeDiscussion
Rank 29 on 14 Nov 2026 · 136 mentions in 28 days · 131 independent groups · papers → code → discussion · first seen 6 weeks ago · acceleration +0.02
Earliest sources
- Viva La Vida: Verification and Accumulation Failures in Multi-Agent Proof SearchPapers · 4 Oct 2026 · Benji Xu, Ken Zheng, Noah Han
- Priced Guidance: Can Language Models Generate Future Research Ideas?Papers · 4 Oct 2026 · Kaiyue Wen, Tengyu Ma, Percy Liang
- How Much Do LLM-as-a-Judge Design Choices Matter? A Systematic Comparison of Prompt Designs, Rating Scales, and ModelsPapers · 4 Oct 2026 · Laurène Vaugrante, Thilo Hagendorff
- MemStrata: 95% and 90.91% Source-Aware Accuracy on LongMemEval-500 and LoCoMo-1540 with a Local Qwen 3.8 27B Q4_K_M ReaderPapers · 4 Oct 2026 · Neeraj Yadav
- EnGRICH: Enhancing Generative Reward Modeling with Critiques from HumansPapers · 4 Oct 2026 · Xuancheng Li, Beining Wang, Haitao Li, Heng Wang, Yujia Zhou, Qingyi Pan, Blaze…
- VERA: Verdict-Conditioned Reliability for Adaptive LLM JudgesPapers · 4 Oct 2026 · Qiushui Xu, Syamil Mohd Razak, Tao Yuan, Piotr Habas
Latest sources
- Right Reason, Wrong Advice: Evaluating Recourse in LLM Explanations of Denied RequestsPapers · 16 Nov 2026 · Arunkumar V
- Validating LLM-Based Essay Scorers: A Comparative Analysis in Critical Online Reasoning in Medical EducationPapers · 16 Nov 2026 · Daniel Baumartz, Alexander Mehler, Mevlüt Bagci, Maxim Konca
- Jev-as-a-Reward: Validating Probabilistic Safety Judges for Reinforcement LearningPapers · 16 Nov 2026 · Eunkyeong Lee, Junhyung Moon, Seunggwan Hong, Dongsuk Oh
- Tight PAC-Bayes Generalisation Guarantees for Large Language Model JudgesPapers · 16 Nov 2026 · Tom A. Lamb, Philip H. S. Torr, Tim G. J. Rudner
- Same Papers, Different Verdicts: Tracing Version Drift in LLM Peer ReviewPapers · 16 Nov 2026 · Jizhou Guo
- Can AI Judge You? An Empirical Audit of Deployment Readiness in LLM-assisted Hiring SystemsPapers · 15 Nov 2026 · Spandan Roy, Subhabrata Majumdar, Soudeep Deb
- Grounding Language Models in Speech Does Not Fix "AI Flavor": Calibrated LLM Judges, Goodhart Traps, and a Preference-Tuning FixPapers · 15 Nov 2026 · Dongwu Qian
- Objectify, Don't Delete: Complaint-Derived Evaluation of Ambient Clinical NotesPapers · 14 Nov 2026 · Nikita Gupta, Narayanan Unny Edakunni, Vikram Okaly, Suranjana Datta, Navya Penu…
- What Moves a LoCoMo Score? Memory Context, Answer Prompts and JudgesPapers · 14 Nov 2026 · Nikhil Kumar Jha, Amitoj Cheema
- The Critic Can Hurt: On the Safety of Critics in LLM-Guided AutoML SearchPapers · 14 Nov 2026 · Gaoyuan Du, Xiaoyang Liu, Rex Zhou, Dengdu Jiang, Rong Fu, Meng Ding, Defu Cao
- Benchmark Factory: Agentic Benchmark Creation via Dataset Curation and Label-Free Quality AnchorsPapers · 13 Nov 2026 · Shervin Shahidi, James Poon, Hang Xie, Kenneth O. Xu, Robin Håkanson, Vikram Na…
- Do Verifier Judgments Survive Abstraction? Measuring Verification Transport for LLM Agent EvaluationPapers · 13 Nov 2026 · Disha Sheshanarayana, Sai Linisha Reddy Goli