Early-Signal Detector

method · first seen 4 Oct 2026

LLM-as-a-judge

Also: GRM, LaJ, LLMaJ, GEMBA, LLMaaJ, LLM judge, AI judges, LLM critic

Using language models to grade the outputs of other models, with judge reliability measured against each other.

LLM-as-a-judge means using a language model to score or compare the outputs of other models, in place of human raters or lexical metrics such as ROUGE. The sources treat it as standard infrastructure and focus on its reliability. One paper systematically compares judge prompts, rating scales and models. Others show that the choice of judge shifts LoCoMo memory scores, that an 8B critic can degrade AutoML search, and that unanimity-based model verifiers fail in multi-agent proof search. Applied work uses judges for clinical notes, hiring screens and coding benchmarks, and a small GitHub RAG repo audits its own judge.

The idea has been mainstream since roughly 2023. The live research question is the judge's calibration and failure modes, because conclusions in many papers depend on unreported judge design choices. Standardizing and auditing judges could make benchmark results comparable across studies.

Analyst note (established), 17 Nov 2026.

Mentions per week

04919 May11 Aug10 NovPapersCodeDiscussion

PapersCodeDiscussion

Rank 29 on 14 Nov 2026 · 136 mentions in 28 days · 131 independent groups · papers → code → discussion · first seen 6 weeks ago · acceleration +0.02

Earliest sources

Latest sources

Mentioned together with

retrieval-augmented generationmachine translation evaluationreward hackinglow-rank adaptationhallucination detectiondirect preference optimizationGUI agentgroup relative policy optimizationreward modelingreasoning models