The headline numbers are disputed, self-reported, or under revision, so treat the ranking as a claim, not a settled fact.
SWE-bench Verified
SWE-bench Verified checks whether a model can fix a real bug that a developer actually filed on GitHub, by editing a real open-source codebase until the project's own tests pass.
It is a human-screened set of 500 issues from popular Python projects, built so a passing solution has to genuinely resolve the issue, not just look plausible.
The model is handed a real GitHub issue (for example, a reported crash in a popular Python library) and the full repository, and it must edit the code so the hidden fail-to-pass tests for that issue go green, the same bar a human maintainer's pull request has to clear.
This is the benchmark closest to the job people imagine AI doing: not finishing a line of code, but reading a messy real project and shipping a working fix. It matters because in early 2026 OpenAI publicly stopped reporting it, saying the test had become a memorization exam rather than a measure of real skill, a rare case of a lab walking away from a number it was winning.
Percent of the 500 issues resolved, meaning the project's fail-to-pass tests all pass after the model's edit. The benchmark is maintained by researchers at Princeton (the human-validated Verified subset was built in collaboration with OpenAI). The headline figures here come from a third-party board, vals.ai, since the official board lists self-submitted entries the operators do not independently re-run.
Early agents in 2023 resolved only a low single-digit percentage of issues. The prior high before the June 2026 frontier wave was Claude Mythos at about 93.9 percent (Anthropic, self-reported). Top systems now claim the low-to-mid 90s, but see the warning below about why those numbers are disputed.
SWE-bench Verified is the clearest case on this page of a contested number. In February 2026 OpenAI announced it would stop reporting the benchmark, saying that at least 59 percent of audited tasks had flawed tests that reject correct fixes, and that every major model had effectively seen the 500 tasks and their solutions during training, so high scores partly reflect memorization. The Princeton maintainers did not retire it; the board is still live, and OpenAI now points to the held-out SWE-bench Pro instead. The current number-one, Claude Fable 5 at 95 percent, is itself disputed: the model was suspended from public access in June 2026 under a US export-control directive, and a newer board ranks a different model first. Treat the top of this board as a moving, contested claim.
The benchmark creates the need to know. The catalog explains the ideas behind it: