method · first seen 7 Oct 2026
inter-rater reliability
Also: kappa, Cohen's kappa, Fleiss' kappa, inter-rater agreement
Statistical measures of agreement between multiple annotators or classifiers labeling the same items.
Mentions per week
PapersCodeDiscussion
Earliest sources
- Nobody Truly Agrees on Sentiment: Humans, Bespoke Tools, and LLMs Struggle with Social Media TextsPapers · 7 Oct 2026 · Himarsha R. Jayanetti, Sivakanesan Dhanushkanda, Shuai Hao, Michael L. Nelson, M…
- BenCoDiC: A Large-Scale Contextual Cyberbullying Corpus for Bengali Political News Discourse FULL VERSIONPapers · 9 Oct 2026 · Md. Nesarul Hoque, Rudra Pratap Deb Nath, Umme Salma, Shima Chakraborty, Md. Han…
- Can LLM Annotators Match Human Consistency? Benchmarking Human Agreement and Optimizing Multi-Agent with Contrastive KnowledgePapers · 23 Oct 2026 · Xiaopeng Ke, Zhixin Wang, Xuebo Liu, Xinyu Ma, Xing Zheng, Bowen Song, Weiqiang…
- Swapping the Judge Changes the Ranking: A Preregistered Study of Cross-Provider Replication in LLM-Judged System ComparisonsPapers · 3 Nov 2026 · Sangdo Han, Moonsoo Kim
- Beyond the Keyword: A Human-Annotated Benchmark for Arabic Anxiety-Related DiscoursePapers · 5 Nov 2026 · Shimaa Ibrahim, Mabrouka Bessghaier, Md. Rafiul Biswas, Insaf Elouard, Wajdi Zag…
- TripleGrader: A Web-Based Platform Supporting Collaborative Triple Annotation with Quality Grading and Inter-Rater Agreement AssessmentPapers · 6 Nov 2026 · Rajan Lamichhane, Songhui Yue
Latest sources
- TripleGrader: A Web-Based Platform Supporting Collaborative Triple Annotation with Quality Grading and Inter-Rater Agreement AssessmentPapers · 6 Nov 2026 · Rajan Lamichhane, Songhui Yue
- Beyond the Keyword: A Human-Annotated Benchmark for Arabic Anxiety-Related DiscoursePapers · 5 Nov 2026 · Shimaa Ibrahim, Mabrouka Bessghaier, Md. Rafiul Biswas, Insaf Elouard, Wajdi Zag…
- Swapping the Judge Changes the Ranking: A Preregistered Study of Cross-Provider Replication in LLM-Judged System ComparisonsPapers · 3 Nov 2026 · Sangdo Han, Moonsoo Kim
- Can LLM Annotators Match Human Consistency? Benchmarking Human Agreement and Optimizing Multi-Agent with Contrastive KnowledgePapers · 23 Oct 2026 · Xiaopeng Ke, Zhixin Wang, Xuebo Liu, Xinyu Ma, Xing Zheng, Bowen Song, Weiqiang…
- BenCoDiC: A Large-Scale Contextual Cyberbullying Corpus for Bengali Political News Discourse FULL VERSIONPapers · 9 Oct 2026 · Md. Nesarul Hoque, Rudra Pratap Deb Nath, Umme Salma, Shima Chakraborty, Md. Han…
- Nobody Truly Agrees on Sentiment: Humans, Bespoke Tools, and LLMs Struggle with Social Media TextsPapers · 7 Oct 2026 · Himarsha R. Jayanetti, Sivakanesan Dhanushkanda, Shuai Hao, Michael L. Nelson, M…