Early-Signal Detector

benchmark · first seen 5 Oct 2026

SWE-bench

Also: SWE-bench Verified, software engineering benchmark

Benchmark of real GitHub issues used to evaluate whether models can resolve software engineering tasks.

SWE-bench is a benchmark built from real GitHub issues and their pull requests, where a model or agent must produce a patch that makes the repository's hidden tests pass. Variants such as SWE-bench Verified, Live and Pro are now the default yardstick for coding agents. In these sources it appears mostly as a reference point rather than a new idea. Papers on contamination auditing, termination lag, failure prediction and debugger interfaces use it as their testbed. Others, including a JetBrains study of IDE sessions, a repository-policy compliance benchmark and an expert-authored task set, argue that it misses what real contributions require. A forked mini-swe-agent repo cites its Verified score.

The benchmark itself is not new. The visible trend is a growing body of work that questions whether passing issue-derived unit tests reflects mergeable, maintainable code, and that work may reshape how coding agents are evaluated.

Analyst note (established), 17 Nov 2026.

Mentions per week

01719 May11 Aug10 NovPapersCodeDiscussion

PapersCodeDiscussion

Rank 145 on 14 Nov 2026 · 29 mentions in 28 days · 29 independent groups · papers → discussion → code · first seen 6 weeks ago · acceleration -0.07

Earliest sources

Latest sources

Mentioned together with

coding agentsbenchmark contaminationuser simulationcontext compactionself-evolving agentsLLM-as-a-judgesuccessive halvingrepository-level code agent memoryruntime monitoring of coding agentsDarwin Gödel Machine