benchmark · first seen 5 Oct 2026
SWE-bench
Also: SWE-bench Verified, software engineering benchmark
Benchmark of real GitHub issues used to evaluate whether models can resolve software engineering tasks.
SWE-bench is a benchmark built from real GitHub issues and their pull requests, where a model or agent must produce a patch that makes the repository's hidden tests pass. Variants such as SWE-bench Verified, Live and Pro are now the default yardstick for coding agents. In these sources it appears mostly as a reference point rather than a new idea. Papers on contamination auditing, termination lag, failure prediction and debugger interfaces use it as their testbed. Others, including a JetBrains study of IDE sessions, a repository-policy compliance benchmark and an expert-authored task set, argue that it misses what real contributions require. A forked mini-swe-agent repo cites its Verified score.
The benchmark itself is not new. The visible trend is a growing body of work that questions whether passing issue-derived unit tests reflects mergeable, maintainable code, and that work may reshape how coding agents are evaluated.
Analyst note (established), 17 Nov 2026.
Mentions per week
PapersCodeDiscussion
Rank 145 on 14 Nov 2026 · 29 mentions in 28 days · 29 independent groups · papers → discussion → code · first seen 6 weeks ago · acceleration -0.07
Earliest sources
- Measurement-First Auditing of Agentic Leaderboards: Contamination Susceptibility, Matched-Control Re-evaluation, and Scorer ValidationPapers · 5 Oct 2026 · Dishu Yang, Qi Su, Hongbo Qin, Hansong Zhang
- Correct Code, Broken Contributions? SWE-CC: Benchmarking Repository Policy Compliance for Coding AgentsPapers · 5 Oct 2026 · Hai Dang Truong, Rayner Goh, Thanh Le-Cong, Yintong Huo
- Code Understanding is a Bottleneck for Coding AgentsPapers · 7 Oct 2026 · Nishant Balepur, Kiran Tomlinson, Tobias Schnabel
- Coding-Agent Benchmarks Should Match Their Users' Task FlowsPapers · 7 Oct 2026 · Igor Slinko, Yaroslav Golubev, Sergey Titov
- TestGRAD: Evolving Test Suites via Failure Pattern Momentum for SWE-Agent EnsemblePapers · 7 Oct 2026 · Pengfei He, Jiayuan Zhou, Shaowei Wang, Ruiqi Pan
- SWE-Journey: Towards More Realistic Evaluation of Coding Assistants through Long-Horizon, Multi-Turn InteractionPapers · 8 Oct 2026 · Hexuan Deng, Yue Wang, Wenyu Jiang, Cheng Yang, Haolin Yang, Zhaohua Zhang, Chen…
Latest sources
- Beyond Leaderboard Ranks: Selecting Software Engineering Agents Under Budget ConstraintsPapers · 16 Nov 2026 · Srinath Srinivasan, Tim Menzies
- Eliciting and Mitigating Specification Satisficing in Coding AgentsPapers · 12 Nov 2026 · Ruixin Wang, Tianyi Zhang
- Termination Lag: Coding Agents Solve Early but Stop LatePapers · 11 Nov 2026 · Musa Cim, Burak Topcu, Mahmut Taylan Kandemir
- ExpertCrowd-SWE: Measuring Frontier Coding Agents on Expert-Authored Engineering Tasks in Unfamiliar CodebasesPapers · 9 Nov 2026 · Josip Vrdoljak, Akash Singh, Bas Steunebrink, Alex Yarotsky, John Lorenzo, Achil…
- Repository-Context-Guided Image Understanding for Agentic Software DevelopmentPapers · 9 Nov 2026 · Fumiki Yohena, Toshiaki Sota, Shota Suzuki, Tomoko Okamoto, Michiaki Tatsubori
- kIieret/SWE-agent-mini-swe-agentCode · 5 Nov 2026 · kIieret · 34 stars
- Pdb or Not to Be: Comparing Debugging Interfaces With Pass$_{X}$@$k$Papers · 5 Nov 2026 · Ryan Magnuson, Ellie Kitanidis, Ariel Lee
- Hard Issue, Weak Model, or Bad Run? Rethinking Failure Prediction for Coding AgentsPapers · 5 Nov 2026 · Xingyue Shi, Wesley K. G. Assunção
- From Watching to Steering: Enabling Human-Coding Agent Collaboration through Execution TracesPapers · 3 Nov 2026 · Xiaoke Yan, Jiaying Lei, Danqing Shi, Zichen Wang, Nan Cao
- Beyond Surface Metrics: Multi-Condition Requirements in Enterprise Maintenance and AI Coding BenchmarksPapers · 2 Nov 2026 · Heeyong Eun
- CR-Bench-v2: Evaluating the Real-World Utility of AI Code Review AgentsPapers · 2 Nov 2026 · Kristen Pereira, Neelabh Sinha, Rajat Ghosh, Debojyoti Dutta
- Delivered, Not Used: Evaluating Agent Memory at Its Point of UsePapers · 1 Nov 2026 · Changran Xu, Zeju Li, Qiang Xu