method · first seen 4 Oct 2026
automated benchmark generation
Also: automatic benchmark generation, automatic benchmark construction
Creating evaluation queries and relevance labels automatically from data structure instead of manual annotation.
Mentions per week
PapersCodeDiscussion
Earliest sources
- AutoSciBench: Autonomous Benchmark Generation for Evaluating Scientific AgentsPapers · 4 Oct 2026 · Dongki Kim, Namkyeong Lee, Surag Nair, Carl Edwards, Xiner Li, Edward De Brouwer…
- MS-Exam-Gen: An Automatic Benchmark Construction Toolkit for Evaluating LLMs on Multiple Sclerosis-MRI Imaging KnowledgePapers · 5 Oct 2026 · Abdul Basit, Muhammad Abdullah Hanif, Muhammad Shafique
- FASHION-RQ: Mining Hierarchical Semantic Structure for Automated Evaluation of Fashion RetrievalPapers · 13 Oct 2026 · Tripti Shukla, Shwetha Somasundaram, Akshat Gupta, Rohit Garg, Paul Duff, Ian An…
- BenchMeUp: Building Software Engineering Benchmarks from Natural-Language SpecificationsPapers · 27 Oct 2026 · Islem Bouzenia, Michael Pradel
Latest sources
- BenchMeUp: Building Software Engineering Benchmarks from Natural-Language SpecificationsPapers · 27 Oct 2026 · Islem Bouzenia, Michael Pradel
- FASHION-RQ: Mining Hierarchical Semantic Structure for Automated Evaluation of Fashion RetrievalPapers · 13 Oct 2026 · Tripti Shukla, Shwetha Somasundaram, Akshat Gupta, Rohit Garg, Paul Duff, Ian An…
- MS-Exam-Gen: An Automatic Benchmark Construction Toolkit for Evaluating LLMs on Multiple Sclerosis-MRI Imaging KnowledgePapers · 5 Oct 2026 · Abdul Basit, Muhammad Abdullah Hanif, Muhammad Shafique
- AutoSciBench: Autonomous Benchmark Generation for Evaluating Scientific AgentsPapers · 4 Oct 2026 · Dongki Kim, Namkyeong Lee, Surag Nair, Carl Edwards, Xiner Li, Edward De Brouwer…