Early-Signal Detector

phenomenon · first seen 5 Oct 2026

benchmark contamination

Also: benchmark gaming, benchmark leakage, data contamination, evaluation leakage, benchmarking dilemma, benchmark overfitting, evaluation circularity, test-set leakage audit

Evaluation benchmark content leaking into model training or retrieval, inflating measured performance.

Benchmark contamination refers to evaluation data appearing in a model's training set, being retrievable at test time, or leaking through the evaluation pipeline, so that reported scores overstate real capability. The sources show it applied across many domains: an audit framework for agentic leaderboards that separates training-time exposure, evaluation-time retrieval and scaffold leakage; BenchChecker for bug-fixing benchmarks; studies on medical exam memorization, Gene Ontology circularity in gene embeddings, temporal leakage in intrusion detection, and deepfake fusion evaluation; plus Hacker News discussion of benchmark overfitting and leaderboard credibility. The newer angle is retrieval-time leakage by web-searching agents, but the core concern has been widely studied for years.

As agents gain web access and benchmarks stay public, contamination audits determine whether leaderboard gains reflect real capability, which affects model selection and research claims. The slight rise here mostly reflects ongoing attention to a long-standing evaluation problem rather than a new idea.

Analyst note (established), 17 Nov 2026.

Mentions per week

0419 May11 Aug10 NovPapersCodeDiscussion

PapersCodeDiscussion

Rank 115 on 14 Nov 2026 · 14 mentions in 28 days · 14 independent groups · papers → code → discussion · first seen 6 weeks ago · acceleration -0.29

Earliest sources

Latest sources

Mentioned together with

SWE-benchartificial general intelligencedeep search agentsautomated program repairlarge audio-language modelstime series foundation modeltext-attributed graph node classificationCorrect and SmoothTLA+consistent identifier renaming