What coding benchmarks actually say under the hood
Most people meet a coding benchmark as a single number. "This agent solves 45% of real GitHub issues." The number gets quoted, a leaderboard shifts, and everyone moves on.
But a whole family of benchmarks has grown up around the original, SWE-bench, and each new one changes just one thing and measures what happens. Read them side by side and they stop being a scoreboard. They start to read like a manual for getting better results at lower cost. The surprising part: changing the model is usually the last and smallest lever, not the first.
Here is the family, and what each one was really testing.
| Benchmark | What it changed | What it found |
|---|---|---|
| SWE-bench | Can an AI fix real GitHub issues, checked by running the real tests | Became the standard ruler everything else builds on |
| SWE-agent | Changed the tool interface, not the model | Better interface design alone tripled the fix rate |
| SWE-Bench Pro | Harder, longer, more realistic tasks | Even top models solve under 45%. The old "easy" set was flattering everyone |
| SWD-Bench | Gave the agent better auto-written docs | Good documentation raised the solve rate by about 20% |
| CodeScout | Improved the problem description before the agent starts | Steady gains of roughly 8 to 13%, biggest for weaker models |
| SWE-Bench 5G | Injected domain spec docs (telecom standards) | Big help on tasks that needed that knowledge, nothing on generic bug fixes |
| SWE-Skills-Bench | Injected 49 different "skill" documents | 80% did nothing. Only the narrowly-matched, actionable ones helped |
| SWE-ABS | Re-checked "solved" patches with tougher tests | About 1 in 5 previously "solved" patches were false alarms |
| SWE-ContextBench | Reused context from related past tasks | Good retrieval helps a lot and cuts cost. Noisy retrieval actively hurts |
| SWE-Edit | Split "read the code" and "edit the code" into separate helpers | Small accuracy gain and lower cost, just from cleaner division of labor |
| Claw-SWE-Bench | Isolated the harness design as its own variable | Harness quality moves the score as much as the model does |
The SWE-bench family, each entry changing one thing.
Pull the threads together and the cost-and-performance story lands on a handful of clear, repeating levers. Here they are, cheapest first.
1. Fix the plumbing before you touch the model
This is the biggest and cheapest win in the whole table. SWE-agent's interface redesign tripled the fix rate with the same model underneath. Claw-SWE-Bench watched one fixed model jump from 19% to 73% purely from better packaging, mostly by letting the agent edit files directly instead of hand-typing a patch as a block of text. SWE-Edit split "look at the code" and "change the code" into two helpers with their own clean workspace, and got both a small accuracy bump and a real cost cut at the same time. That last part is rare: almost everywhere else, accuracy and cost trade off against each other.
None of this needs a smarter model or more context. It is wiring. If your setup has any mechanical friction, like fragile patch formatting or one step's mess leaking into the next, that is the first thing to fix.
2. The right context beats more context, and the wrong context costs you
SWE-ContextBench found that well-matched context improves accuracy and cuts cost, especially on hard tasks. But badly-chosen context gave little benefit, and sometimes made things worse. SWE-Skills-Bench found that 80% of the knowledge documents it handed the agent did nothing at all.
3. Not every task deserves the expensive treatment
SWE-Bench 5G showed that heavy domain context helped a lot on tasks that genuinely needed it, and did essentially nothing on plain mechanical fixes. Put that next to lever #2 (the wrong context can hurt) and the move is obvious: add a cheap triage step. Quickly sort whether a task is mechanical or judgment-heavy, and only spend your expensive tools, deep retrieval, extra reasoning turns, on the ones that need it. Applying the full treatment to every task wastes money on the easy ones and buries your real gains on the hard ones.
4. Bigger models stop paying off fast
The cost curves make this concrete. In one head-to-head, the top model scored 78% for about $1,400 on a full run, while a cheap model scored 70% for about $8. That is roughly 170 times cheaper for an eight-point gap. If you care about cost per solved issue rather than the peak score, the honest choice is usually a cheaper base model with a good scaffold and good context around it, not the priciest model on the shelf. Worth checking before you assume you need a frontier model at all.
5. Put stable knowledge in the model, changing knowledge in retrieval
Every token you put in a prompt gets paid for on every single call, forever. Something you bake into the model itself gets paid for once, at training time. So the split writes itself. Things that rarely change, like known design patterns and architecture choices, are good candidates to train in. Things that change every week, like the current API surface or this sprint's file layout, belong in retrieval where they can update cheaply.
Get the split backwards in either direction and you lose. Either you pay a recurring token tax for something that never changes, or you bake in something that goes stale and quietly misleads the model. A related study on teaching a model to spot a pattern found that training it in beat retrieval alone, because recognizing a pattern in the wild is a different skill from being able to look up its definition.
6. A cheap, honest test loop is its own saving
A slimmed-down version of one benchmark reproduced the full results within half a percentage point at less than a quarter of the cost. That is not about your live system. It is about your own development budget. A fast, trustworthy test loop lets you run more experiments for the same spend, so you can actually answer questions like "is this extra layer earning its keep?" If every test cycle is expensive, you run them rarely, and you end up trusting numbers you did not really check.
7. Watch for wins that are not real
SWE-ABS found that about 1 in 5 "solved" patches on a top leaderboard were false alarms once the tests were tightened. Separately, one benchmark had a data leak that inflated every model's score until it was caught and patched. The cost angle: if your scorecard is gameable or leaky, you will spend real engineering effort chasing a number that does not reflect real ability. That is the most expensive kind of waste, because it stays invisible until someone bothers to check.
Takeaway
Read as a group, the SWE-bench family quietly agrees on an order of operations for "solve more, spend less":
- Fix the plumbing first. It is the cheapest lever and often the biggest.
- Feed the right small amount of context, and be willing to feed none.
- Triage tasks, so the expensive treatment only lands where it pays off.
- Split stable knowledge (into the model) from changing knowledge (into retrieval).
- Keep a cheap, honest test loop so you can tell what is actually working.
Notice what is missing from the top of that list: buy a bigger model. It is on the list, but it is near the bottom, and it is usually the smallest and most expensive win. That is the whole point of reading benchmarks under the hood instead of off the leaderboard. The headline number tells you who is ahead. The design underneath tells you how to get there for less.
It is also the thesis we build on. Prefex is a proxy that sits in front of your model and works these levers automatically: routing easy turns to cheaper models, keeping the prompt cache warm, and trimming context that is not pulling its weight. Better plumbing and better context, without changing the model you already trust.