Okay, the
@VulcanBench results for Qwen3.8-Max are in, and it is not what I expected.
First, for anyone new to VulcanBench, here's a quick TL;DR on the eval suite:
23 frontier-hard software engineering tasks taken from real merged OSS PRs, run in a Docker sandbox, 3 runs per task across all three of its effort levels.
No puzzles, no random abstract stuff, all real things engineering teams would do with these models.
It looks like Qwen3.8-Max has a major overthinking problem, it uses a LOT of tokens and is very slow, period, no other way to see it.
My cost to run this benchmark was $126.25, to run the exact same eval suite with DeepSeek V4-Flash was only $13.60. This makes Qwen3.8-Max an insanely expensive model.
The tasks Qwen genuinely can't solve fail at every effort level, extra reasoning didn't help. The regression is almost all in work it already handles: six tasks that low solves every single time account for 83% of the 26-point drop, three of them collapsing to zero. It's not losing the hard problems. It's losing the ones it already knows how to do.
Since Qwen3.8-Max hit a lot of wall clock budget caps, I thought I'd share more about this.
- VulcanBench caps both steps (50–200) and wall clock (5–60 min), each scaled by repo size.
- This is aligned with how comparable harnesses bound agents, DeepSWE caps rollouts at 100 environment steps, sitting right inside my step range; Terminal-Bench enforces a per-task wall clock; SWE-bench Verified scaffolds typically allow 20–60 min per instance with 250–350 step limits.
- Every model on my chart gets the identical budget, and Qwen is the slowest model I've tested at 20–25 min/task.
Soooo... Alibaba positions Qwen3.8-Max as trailing only Claude Fable 5. But on the kind of real coding work engineering teams would actually throw at it, under a fixed budget, its best setting lands mid-pack and its default lands last, so common.
If you want to optimize for accuracy, Grok 4.5 is the move. If you want accuracy per dollar, DeepSeek V4-Flash is hard to beat, heck it's 10× cheaper than Qwen and you get higher accuracy.
Qwen just isn't in the game at this point, this is not a model I could see engineering teams using for daily coding work.