phenomenon · first seen 5 Oct 2026
benchmark contamination
Also: benchmark gaming, benchmark leakage, data contamination, evaluation leakage, benchmarking dilemma, benchmark overfitting, evaluation circularity, test-set leakage audit
Evaluation benchmark content leaking into model training or retrieval, inflating measured performance.
Benchmark contamination refers to evaluation data appearing in a model's training set, being retrievable at test time, or leaking through the evaluation pipeline, so that reported scores overstate real capability. The sources show it applied across many domains: an audit framework for agentic leaderboards that separates training-time exposure, evaluation-time retrieval and scaffold leakage; BenchChecker for bug-fixing benchmarks; studies on medical exam memorization, Gene Ontology circularity in gene embeddings, temporal leakage in intrusion detection, and deepfake fusion evaluation; plus Hacker News discussion of benchmark overfitting and leaderboard credibility. The newer angle is retrieval-time leakage by web-searching agents, but the core concern has been widely studied for years.
As agents gain web access and benchmarks stay public, contamination audits determine whether leaderboard gains reflect real capability, which affects model selection and research claims. The slight rise here mostly reflects ongoing attention to a long-standing evaluation problem rather than a new idea.
Analyst note (established), 17 Nov 2026.
Mentions per week
PapersCodeDiscussion
Rank 115 on 14 Nov 2026 · 14 mentions in 28 days · 14 independent groups · papers → code → discussion · first seen 6 weeks ago · acceleration -0.29
Earliest sources
- Measurement-First Auditing of Agentic Leaderboards: Contamination Susceptibility, Matched-Control Re-evaluation, and Scorer ValidationPapers · 5 Oct 2026 · Dishu Yang, Qi Su, Hongbo Qin, Hansong Zhang
- Why machines will still not rule the worldPapers · 8 Oct 2026 · Jobst Landgrebe, Barry Smith
- Glasseek: A Transparent Recipe for Training and Trustworthy Evaluation of DeepSearch AgentsPapers · 12 Oct 2026 · Cong Pang, Jiaqi Su, Chengrui Wu, Kuihao Yan, Tianrui Zhang, Zixuan Qiu, Chao Li…
- BenchChecker: Assessing the Credibility of Bug-Fixing Benchmarks for LLMsPapers · 14 Oct 2026 · Di Wu, Xu He, Shu Wang, Kun Sun
- What Does LALM Post-training Learn? Rethinking Audio Understanding and Reasoning CapabilitiesPapers · 14 Oct 2026 · Hua-Hsuan Chen, Xuanjun Chen, Wei-Chung Lu, Chih-Kai Yang, Yu-Xiang Lin, Sung-Fe…
- Beyond Static Leaderboards: Capability-Focused Evaluation with Structural Diversity for Time Series Foundation ModelsPapers · 19 Oct 2026 · Hongyin Zhang, Mingyi Xiang, Rongzhao Chen, Yankai Wang, Jialin Qiao, Xiangdong…
Latest sources
- Benchleaking Is Accelerating: Benchmark contamination has worsened sharply in the past month -- from post-training down to pre-training base modelsPapers · 16 Nov 2026 · Yiwei Qin, Muhang Xie, Xuefeng Li, Mohan Jiang, Dayuan Fu, Tiantian Mi, Qipeng G…
- ST-GAT: A Leakage-Free Spatio-Temporal Graph Attention Framework for Robust Network Intrusion DetectionPapers · 14 Nov 2026 · Nikhil Parkar, Sanskar Pandey, Siddhi Prajapati, Suganeshwari G, Syed Ibrahim
- Does a Self-Learning Migration Agent Learn? An Audited Evaluation of Evidence-Graded Experience Memory for Oracle-to-PostgreSQL MigrationPapers · 12 Nov 2026 · Oleg Grynets, Vasyl Lyashkevych, Valeriia Oliferuk, Vladyslav Babenko, Danylo Ko…
- Next-Level Benchmarking Overfitting – Thomas NeumannDiscussion · 11 Nov 2026 · artrax · 68 points
- Honest Fusion Evaluation for Cross-Domain Deepfake Detection: Leakage Audits and Tail RobustnessPapers · 11 Nov 2026 · Luke Collins, Chang-Tsun Li, Duc Thanh Nguyen
- When gene summaries reveal the answers: Gene Ontology circularity in gene embedding benchmarksPapers · 8 Nov 2026 · Jiawei Wu, Huishu Wang, Junlin Chen, Hongyi Wang
- MaxxBench – "Nice DeepSWE score. But is it true?"Discussion · 7 Nov 2026 · antiuniverse · 409 points
- FinForesight: A Diagnostic Backtesting Benchmark for Deep Research Systems in Financial ForecastingPapers · 1 Nov 2026 · Fengyuan Liu, Mukai Li, Jieshun You, Yilin Zheng, Chenjia Min, Yizhou Zhang, Har…
- From Small to Large: Assessing Medical Reasoning, Benchmark Contamination, and Model Size in Language ModelsPapers · 30 Oct 2026 · Navapat Nananukul, Mayank Kejriwal
- Artificial Analysis v5 and the benchmarker's dilemmaDiscussion · 28 Oct 2026 · 6thbit · 73 points
- The State and Fate of Multilingual, Contextual Evaluation in the NLP WorldPapers · 27 Oct 2026 · Manan Uppadhyay, Himanshu Beniwal, Prashant Kodali, Sunayana Sitaram
- ALIAS: Evaluating Performance and Memorization in LLM-Written TLA+ SpecificationsPapers · 21 Oct 2026 · Arslan Bisharat, Mujtaba Nazari, Khushboo Bhadauria, Brian Ortiz, Eric Spencer,…