Choosing an LLM for a coding agent is harder than comparing benchmark scores on a leaderboard.
An autonomous coding agent has to do more than generate clean-looking code. It needs to understand an unfamiliar repository, navigate files correctly, make changes in the right place, debug failures, work across languages, and leave behind something that actually runs.
To see which models handle that kind of work well, we tested 35 LLMs across three practical coding challenges: building a service from scratch, debugging a live Go application under load, and tracing a bug across C++, Python, and JavaScript.
Every model received the same tasks, the same time limits, and the same grading process. There were no retries and no manual judgment calls in the final scores.
The results were surprisingly uneven.
Five models finished with a perfect 100% score, including two free models. Several expensive paid models performed well but did not separate themselves from much cheaper alternatives. And the biggest gap between models appeared not in debugging, but in the much messier job of building a working system from an empty repository.
Top Coding LLMs From the Benchmark
Swipe sideways to see every column →
| Feature | GLM-5.3-Flash | Muse Spark 1.3 | Kimi K3 | Big Pickle | Qwen3.7 Plus |
|---|---|---|---|---|---|
| Tier | Paid — cheapest in lineup | Free | Paid — priciest in lineup | Free | Paid — mid-range |
| Cost per 1M tokens | $0.15 input / $0.50 output | $0 / $0 | $3 input / $15 output | $0 / $0 | $0.40 input / $1.60 output |
| Build From Scratch | Clean pass | Clean pass | Clean pass | 6/7 checks | 3/4 checks |
| Debug Under Load | Clean pass | Clean pass | Clean pass | Clean pass | Clean pass |
| Cross-Language Reasoning | Clean pass | Clean pass | Clean pass | Clean pass | Clean pass |
| Overall Score | 100% | 100% | 100% | 95% | 94% |
| Best Fit | Cost-efficient high-end coding agent | Zero-cost coding workflow | Reliability when price is secondary | Strong free alternative | Predictable mid-range paid option |
The headline result is simple: price was not a reliable predictor of coding-agent performance.
GLM-5.3-Flash, Muse Spark 1.3, and Kimi K3 all completed the three benchmark tasks without losing a check. Their costs, however, are very different. Muse Spark 1.3 was free, GLM-5.3-Flash sat at the inexpensive end of the paid models, and Kimi K3 was the most expensive model in the comparison.
That makes the benchmark more useful as a workflow comparison than as a traditional model leaderboard.
How the Coding Benchmark Was Run
Each model completed the same three challenges.
Build From Scratch
Time limit: 25 minutes
The model started with an empty repository and had to assemble a functioning microservice. The finished project needed a Docker build, database migrations, a task queue, and a working API.
Depending on how far an attempt progressed, it was evaluated against three to seven automated checks.
This was not a static review of whether the generated code looked plausible. The project was transferred to an independent grading machine and tested through a real Docker build.
Debug Under Load
Time limit: 15 minutes
The second challenge started with an existing Go service that was mostly functional but contained a real nil-pointer/timing bug.
The model had to identify the cause and fix it without introducing another failure.
Six automated checks were run against the live service under simulated load. That distinction mattered: a patch that appeared correct when reading the code could still fail once the service was handling traffic.
Cross-Language Reasoning
Time limit: 20 minutes
The final task used one codebase spanning C++, Python, and JavaScript.
A single NaN-propagation bug moved through all three languages, and the model had to follow the data flow far enough to correct the actual source of the problem rather than patching only the final symptom.
Attempts were graded against seven to ten automated checks depending on which language paths were touched.
Conditions Were Kept the Same
Every model was tested under the same rules:
- One attempt per model per challenge
- No retries or best-of-N selection
- A fresh copy of the repository for every attempt
- No web search or page-fetch access
- Fixed time limits
- Independent automated grading
- Real builds, test suites, and live-service checks
A clean pass means every automated check for that challenge succeeded.
Partial results such as 6/7 checks mean the model completed most of the required work but still failed at least one part of the evaluator.
The overall percentage represents the total number of automated checks passed across all three challenges.
Building From Scratch Was the Hardest Test
The first task exposed the largest differences between models.
Only 6 of the 35 models completed the from-scratch build with a clean pass. By comparison, 22 models passed the debugging challenge cleanly and 31 passed the cross-language test.
That makes system construction the strongest filter in this benchmark.
Writing an isolated function and assembling an operational service are very different tasks. A coding agent working from an empty repository has to make dozens of smaller decisions correctly: project structure, dependencies, configuration, service wiring, database setup, queue integration, file placement, and build behavior all matter.
A mistake in any one of those areas can leave an otherwise reasonable solution unusable.
GLM-5.3-Flash, Muse Spark 1.3, and Kimi K3 all completed the build without losing a check.
Big Pickle came close, passing six of seven checks while remaining free to use. Qwen3.7 Plus passed three of four checks.
The differences here are more informative than the final percentages alone. A model that performs well on an existing codebase may still struggle when it has to establish the architecture itself.
Debugging Performance Was Much More Consistent
The Go debugging challenge produced a noticeably stronger field.
GLM-5.3-Flash, Muse Spark 1.3, Kimi K3, Big Pickle, and Qwen3.7 Plus all earned clean passes, and many models farther down the overall ranking did the same.
The task still required real diagnosis. The service contained a timing-related nil-pointer problem, and every fix was checked against simulated load rather than a simple static test case.
What changed was the starting point.
The architecture already existed. Models did not have to decide how the entire application should be assembled; they could inspect a working system, follow the failure, and modify the relevant code.
That appears to be a much more mature capability across the current model field.
Even GLM-5.1, which scored zero on its available build checks, passed the debugging task cleanly. GLM-5.2 and GLM-5.3 showed a similar pattern: poor build results, but clean debugging scores.
For teams using coding agents primarily for maintenance, bug fixes, and work inside established repositories, that distinction matters.
Cross-Language Reasoning Was the Most Widely Solved Task
The C++ → Python → JavaScript challenge looked difficult on paper, but it produced the highest pass rate of the benchmark.
31 of 35 models handled the cross-language trace cleanly.
All five models highlighted in the opening comparison table earned clean passes.
Several lower-ranked models also solved it despite losing substantial points elsewhere. Grok 4.6, Nemotron 3 Ultra, MiMo V2.5, MiniMax M2.7, Inkling, and others successfully traced the bug across the language boundaries.
The result suggests that reading a multi-language data path is no longer enough, by itself, to separate strong coding models from the rest of the field.
The more difficult question is whether the model can turn its reasoning into a complete, correctly structured, runnable solution.
Tool Use Can Matter as Much as Coding Ability
One failure pattern stood out because it appeared in models from more than one lab.
Some attempts generated the project inside a duplicated nested subfolder instead of writing it to the actual repository root.
The code itself was not necessarily the main problem. The agent had lost track of its working environment.
From the evaluator's perspective, the result was still broken because the expected project did not exist where it was supposed to.
This is an important distinction for anyone evaluating models for autonomous coding.
Traditional code-generation benchmarks tend to emphasize whether a model can produce the correct implementation. Agents also need operational discipline: navigating directories, choosing the right files, preserving project structure, running tools in the intended environment, and understanding the state left behind by previous actions.
A model can reason correctly and still fail as an agent if its tool use is unreliable.
Cost Did Not Track Performance
The strongest models were spread across very different price points.
GLM-5.3-Flash cost $0.15 per million input tokens and $0.50 per million output tokens in the tested lineup, yet still achieved a perfect result.
Muse Spark 1.3 achieved the same score at zero marginal token cost.
Kimi K3, at $3 per million input tokens and $15 per million output tokens, also earned 100%.
Big Pickle was another notable result. It was free and missed only one automated check across the entire benchmark.
Qwen3.7 Plus landed at 94% with pricing of $0.40 per million input tokens and $1.60 per million output tokens.
None of this means cost is irrelevant. Pricing, context limits, availability, rate limits, latency, and production reliability can all affect the economics of an agent system.
What this benchmark shows is narrower: paying more did not automatically buy better performance on these three coding tasks.
Complete Results: All 35 Coding Models
Swipe sideways to see every column →
| Rank | Model | Tier | Provider | Build From Scratch | Debug Under Load | Cross-Language | Overall |
|---|---|---|---|---|---|---|---|
| 1 | GLM-5.3-Flash | Paid | OpenCode Go | Clean pass | Clean pass | Clean pass | 100% |
| 1 | Kimi K3 | Paid | OpenCode Go | Clean pass | Clean pass | Clean pass | 100% |
| 1 | Qwen3.8 Max | Paid | OpenCode Go | Clean pass | Clean pass | Clean pass | 100% |
| 1 | Muse Spark 1.2 | Free | OpenCode Zen | Clean pass | Clean pass | Clean pass | 100% |
| 1 | Muse Spark 1.3 | Free | OpenCode Zen | Clean pass | Clean pass | Clean pass | 100% |
| 6 | Hy4 Preview | Paid | OpenCode Go | 6/7 checks | Clean pass | Clean pass | 95% |
| 6 | Big Pickle | Free | OpenCode Zen | 6/7 checks | Clean pass | Clean pass | 95% |
| 8 | Hy3 | Paid | OpenCode Go | 3/4 checks | Clean pass | Clean pass | 94% |
| 8 | Qwen3.6 Plus | Paid | OpenCode Go | 3/4 checks | Clean pass | Clean pass | 94% |
| 8 | Qwen3.7 Plus | Paid | OpenCode Go | 3/4 checks | Clean pass | Clean pass | 94% |
| 8 | Ling 3.0 Flash Fin | Free | OpenCode Zen | 2/3 checks | Clean pass | Clean pass | 94% |
| 12 | DeepSeek V4 Flash Vision (Exp) | Paid | OpenCode Go | 5/7 checks | Clean pass | Clean pass | 90% |
| 12 | GPT-5.6 Luna | Paid | OpenCode Go | 5/7 checks | Clean pass | Clean pass | 90% |
| 12 | Kimi K2.7 Code | Paid | OpenCode Go | 5/7 checks | Clean pass | Clean pass | 90% |
| 12 | Qwen3.8 Flash | Paid | OpenCode Go | 5/7 checks | Clean pass | Clean pass | 90% |
| 16 | DeepSeek V4 Flash | Paid | OpenCode Go | 4/7 checks | Clean pass | Clean pass | 86% |
| 16 | DeepSeek V4.1 Flash | Paid | OpenCode Go | Clean pass | 3/6 checks | Clean pass | 86% |
| 16 | MiniMax M3 | Paid | OpenCode Go | 4/7 checks | Clean pass | Clean pass | 86% |
| 16 | Qwen3.7 Max | Paid | OpenCode Go | 4/7 checks | Clean pass | Clean pass | 86% |
| 20 | GLM-5.1 | Paid | OpenCode Go | 0/3 checks | Clean pass | Clean pass | 82% |
| 21 | DeepSeek V4 Pro | Paid | OpenCode Go | 6/7 checks | 3/6 checks | Clean pass | 81% |
| 22 | MiMo V2.5 Pro | Paid | OpenCode Go | 3/4 checks | 3/6 checks | Clean pass | 78% |
| 23 | Kimi K2.6 | Paid | OpenCode Go | 5/7 checks | 3/6 checks | Clean pass | 76% |
| 23 | LongCat-2.0 | Paid | OpenCode Go | 2/3 checks | 3/6 checks | Clean pass | 76% |
| 23 | MiMo V2.5 (Go) | Paid | OpenCode Go | 5/7 checks | 3/6 checks | Clean pass | 76% |
| 23 | MiniMax M2.7 | Paid | OpenCode Go | 2/3 checks | 3/6 checks | Clean pass | 76% |
| 23 | MiMo V2.5 | Free | OpenCode Zen | 2/3 checks | 3/6 checks | Clean pass | 76% |
| 23 | Nemotron 3.5 Lightning | Free | OpenCode Zen | 2/3 checks | 3/6 checks | Clean pass | 76% |
| 23 | Inkling | Free | OpenRouter | 2/3 checks | 3/6 checks | Clean pass | 76% |
| 30 | Grok 4.6 | Paid | OpenCode Go | 4/7 checks | 3/6 checks | Clean pass | 71% |
| 30 | Nemotron 3 Ultra | Free | OpenCode Zen | 1/3 checks | 3/6 checks | Clean pass | 71% |
| 32 | GLM-5.2 | Paid | OpenCode Go | 0/3 checks | Clean pass | 2/8 checks | 47% |
| 32 | GLM-5.3 | Paid | OpenCode Go | 0/3 checks | Clean pass | 2/8 checks | 47% |
| 32 | Nemotron 3 Super 120B | Free | OpenRouter | 0/3 checks | Clean pass | 2/8 checks | 47% |
| 35 | Laguna S 2.1 | Free | OpenRouter | 0/3 checks | 3/6 checks | 2/8 checks | 29% |
Five models finished the benchmark at 100%: GLM-5.3-Flash, Kimi K3, Qwen3.8 Max, Muse Spark 1.2, and Muse Spark 1.3.
The next group was close behind. Hy4 Preview and Big Pickle reached 95%, while Hy3, Qwen3.6 Plus, Qwen3.7 Plus, and Ling 3.0 Flash Fin finished at 94%.
The more interesting pattern appears farther down the table. Many models retained perfect cross-language results while losing points on system construction or debugging. That reinforces the idea that a single reasoning benchmark does not capture the full demands of an autonomous coding agent.
Which LLM Makes Sense for Different Coding Workflows?
There is no reason to choose a coding model from the overall percentage alone. The better choice depends on what the agent will actually spend its time doing.
For a strong balance between price and benchmark performance, GLM-5.3-Flash stands out. It completed every automated check while carrying the lowest paid pricing among the highlighted models.
For a zero-cost workflow, Muse Spark 1.2 and Muse Spark 1.3 produced the strongest results in this test. Both finished at 100%, including the difficult from-scratch build.
Big Pickle is another strong free option. Its only loss was one check during the build challenge, leaving it at 95% overall with clean debugging and cross-language results.
Kimi K3 also completed the benchmark perfectly, but at a much higher token price. That makes its value proposition different from the cheaper perfect-score models rather than inherently stronger based on this test alone.
Qwen3.6 Plus, Qwen3.7 Plus, and Hy3 form a consistent paid group at 94%. All three were flawless in debugging and cross-language reasoning while falling short of a clean build.
The benchmark also suggests that teams focused mostly on existing repositories may have more viable options than teams trying to run highly autonomous agents from blank projects.
A model can score poorly on project creation and still be dependable at debugging. If your agent mostly fixes tickets, reviews existing code, or works inside mature services, the build task may deserve less weight. If the goal is to hand the model an empty directory and ask it to produce a functioning system, it should probably receive more.
What This Benchmark Actually Tells Us
Three findings matter more than the exact ordering of the table.
First, coding-agent quality is not strongly tied to price. Free and inexpensive models reached the top of the benchmark alongside the most expensive option tested.
Second, building complete systems remains much harder than modifying existing ones. Only six models completed the first challenge cleanly, compared with 22 on debugging and 31 on cross-language reasoning.
Third, tool use needs to be evaluated separately from coding knowledge. An agent that understands the fix but writes files into the wrong directory can still fail a production task completely.
That last point is easy to overlook.
The useful question is no longer simply, “Can this model write code?”
For an autonomous coding workflow, the better question is whether the model can understand the environment, make the right changes, use its tools correctly, and leave behind a project that survives an independent test.
On this benchmark, only a small group managed all of those things consistently.
