与「BENCH」相关的搜索结果

The dangerous AI agent may look productive: task completed, consequences hidden. The real benchmark isn't just whether it can act, but whether it knows when the task has become unsafe. Judgment matters more than speed. #AIAgents# #AISafety# #ResponsibleAI# #PodcastorAI#

Okay, the @VulcanBench results for Qwen3.8-Max are in, and it is not what I expected. First, for anyone new to VulcanBench, here's a quick TL;DR on the eval suite: 23 frontier-hard software engineering tasks taken from real merged OSS PRs, run in a Docker sandbox, 3 runs per task across all three of its effort levels. No puzzles, no random abstract stuff, all real things engineering teams would do with these models. It looks like Qwen3.8-Max has a major overthinking problem, it uses a LOT of tokens and is very slow, period, no other way to see it. My cost to run this benchmark was $126.25, to run the exact same eval suite with DeepSeek V4-Flash was only $13.60. This makes Qwen3.8-Max an insanely expensive model. The tasks Qwen genuinely can't solve fail at every effort level, extra reasoning didn't help. The regression is almost all in work it already handles: six tasks that low solves every single time account for 83% of the 26-point drop, three of them collapsing to zero. It's not losing the hard problems. It's losing the ones it already knows how to do. Since Qwen3.8-Max hit a lot of wall clock budget caps, I thought I'd share more about this. - VulcanBench caps both steps (50–200) and wall clock (5–60 min), each scaled by repo size. - This is aligned with how comparable harnesses bound agents, DeepSWE caps rollouts at 100 environment steps, sitting right inside my step range; Terminal-Bench enforces a per-task wall clock; SWE-bench Verified scaffolds typically allow 20–60 min per instance with 250–350 step limits. - Every model on my chart gets the identical budget, and Qwen is the slowest model I've tested at 20–25 min/task. Soooo... Alibaba positions Qwen3.8-Max as trailing only Claude Fable 5. But on the kind of real coding work engineering teams would actually throw at it, under a fixed budget, its best setting lands mid-pack and its default lands last, so common. If you want to optimize for accuracy, Grok 4.5 is the move. If you want accuracy per dollar, DeepSeek V4-Flash is hard to beat, heck it's 10× cheaper than Qwen and you get higher accuracy. Qwen just isn't in the game at this point, this is not a model I could see engineering teams using for daily coding work.

转一下苏剑林老师对 K3 架构的复盘。 一句话概括，K3 = KDA + MLA + Stable LatentMoE + AttnRes。整套设计没什么炫技，核心就是在模型效果、计算效率和训练稳定性之间做取舍。 这里稍微解释一下： KDA，一种线性 Attention MLA，一种 KV Cache 较小的 Full Attention 变体 Stable LatentMoE，对 LatentMoE 做过稳定性改造的 MoE 方案 AttnRes，用可学习的跨层注意力，替代固定等权的残差累加 几个比较有意思的点： ① K3 同时使用了 KDA 和 MLA。另外，训练仍然用 Moonlight 版本的 Muon 优化器，Attention 权重改成 Per-Head Muon，每个 Head 独立优化。苏神说这不会直接提高效果，主要是数学和结构上更合理：各个 Head 本来就相对独立，不应该在优化时耦合在一起。 ② 解决MoE“容易炸”的问题。 LatentMoE 会先降维，再使用更多专家，最后升维，在训推成本大致相同的情况下效果略好。不过连续的矩阵投影也让训练更容易出现数值不稳定。 K3 把 SwiGLU 换成了 SiTU-GLU，用 softcap 压住异常激活；又在 LatentMoE 升维前加了一层 RMS Norm。这个 Norm 不只是让训练更稳定，在 Valid Loss 差不多的情况下，不加它，某些 Benchmark 会稳定变差。 （这里有超级多的技术细节，感兴趣的可以看苏神原文） ③ 关于MLA DSV4 看上去都换设计了，K3 怎么还在用 MLA？苏神的答案是，目前 MLA 仍然很难被全面击败。它在训练阶段是 MHA 形态，推理时 KV Cache 较小；在固定训练成本和 KV Cache 大小时，MLA 依然近乎最优。 它的问题是对 MTP（推测解码）不够友好。MTP 的思路是“用计算换速度”，而 MLA 在 Decoding 阶段本身就比较吃计算，再叠一个 MTP，两边就容易抢算力。 换别的方案也有代价。比如换成 128+128 的 GQA8，效果很难打赢 MLA，KV Cache 还是 MLA 的三倍多；换成 256+256 的 MFA（本质上是 MQA），训练和 Prefill 成本又会上去。 目前还没有一个简单的 Attention 设计，能同时占住效果、训练成本、Prefill、KV Cache 和 Decoding 计算量。在 KDA+MLA 的混合架构下，MLA 的部分问题得到缓解，所以 K3 最后还是选了 MLA。 ④ 苏神觉得，DSV4 也不算真正“抛弃 MLA”。 DSV4 看上去重新设计了 Attention，但底层仍然有 MLA 的影子。它采用的是 head_dims=512、K=V 的 MQA，这正是 MLA 在 Decoding 阶段的形态；再加上 Sparse + Compress，前者减少计算，后者进一步压缩 KV Cache，同时减少计算。 方向很激进，不过 Infra 也更复杂。 ⑤ K3 的 MLA 可以去掉 RoPE，是因为 KDA 已经隐含提供了一种广义的位置信息。 这只适用于 KDA+MLA 的混合结构，像 K2 那种全 MLA 模型，直接去掉 RoPE 还是会明显掉效果。 感觉苏神真正的观点是：大模型架构里很少有免费的升级。K3 的设计哲学不是找一个碾压所有方案的新架构，而是在效果、训练成本、Prefill、Decoding、KV Cache 和稳定性之间，找一个当前更合适的组合。 苏神原文里还有很多技术内容，非常值得一看。 原文地址：

You know, DeepSeek and I, we go way back. We've been through a lot of problems together. And Liang Wenfeng, I've known this guy since way before all this — this guy is incredible, man. The things he can do are just unbelievable. And I feel like together, we can produce some real special chemistry. In this fall... this is very tough... in this fall, I'm going to take my talents to DeepSeek, and join the Blue Whale. I feel like it's going to give me the best opportunity to win, and to win for multiple years. Not just to win one benchmark, or just to win three conversations in a row — I want to be able to win it all. And I feel like I can compete over there. As for GLM... my heart will always be around that area. But I felt like it's time for a change. I can only wish them the best of luck.🫡 #DeepSeek# #GLM#

$BNB is second among the 18 constituents of @SPDJIndices and @PanteraCapital's new digital asset index, all screened on protocol revenue. Crypto benchmarks have typically measured what an asset is worth. Revenue only exists if people transact. BNB Chain averaged ~4.5 million daily active users through Q1 2026. Every protocol team now has a public standard telling them what institutional capital will count.

RAG system that skips HTML parsing entirely! PixelRAG is an open-source visual RAG framework that renders documents as screenshots instead of parsing them into text. Most RAG pipelines start by converting HTML to text. Tables flatten into unstructured rows. Charts disappear. Layout context is gone before the LLM ever sees it. The paper measured this directly: HTML-to-text conversion accounts for 36.6% of retrieval failures on SimpleQA. PixelRAG skips that step entirely. It renders pages as screenshot tiles using Playwright, embeds those tiles with a fine-tuned Qwen3-VL-Embedding model, builds a FAISS index, and passes retrieved images directly to a VLM reader. No text abstraction in between. Benchmarked across six datasets against the strongest text-based baselines: - SimpleQA: 78.8% vs 71.6% (+7.1 points) - NQ-Tables: 48.8% vs 42.5% (+6.3 points) - EVQA: +15.5 points - LiveVQA: +11.3 points One honest caveat from the authors: this requires Qwen3-VL-4B class models or larger to see the benefit. Smaller models trail text retrieval. The authors also recommend using PixelRAG as an enhancement layer alongside existing text systems rather than a full replacement. Ships with a pre-built Wikipedia index covering 8.28M articles across 28.1M screenshot tiles. A Claude Code plugin lets Claude take screenshots of any URL and reason over the visual content directly. Key capabilities: • Renders web pages, PDFs, and images as screenshot tiles via Playwright • Fine-tuned Qwen3-VL-Embedding model for visual retrieval • FAISS index for fast vector search • Pre-built Wikipedia index: 8.28M articles, 28.1M tiles • 3x token cost reduction via image compression • Claude Code plugin for direct URL screenshot and visual reasoning • LoRA fine-tuning support via pixelrag-train 100% open source. I've shared the link in the replies!

又来一篇论文。 这次我也有幸列在作者名单里。贡献边界得说清楚：真正把研究、工程和实验做出来的，都是算法同事。 我做的很少，更多是从产品和 Agent 生态角度参与了一些讨论，提供了一点问题定义和应用反馈，甚是荣幸。🫡 这篇论文研究的问题很实际： 网络上已经有几十万个 SKILL.md。把更多 skills 塞给 Agent，它就会自动变强吗？ 团队从约 82.1 万个公开技能文件中，经过结构筛选、去重，以及质量、安全和许可检查，整理出 96,401 个技能。然后通过 retrieval、rerank 和 LLM selector，为每个任务选择 0–2 个真正相关的技能。 在三个 benchmark、两种 agent harness 和两种开源 backbone 下，SkillCorpus 都带来了正向提升，SkillsBench 上的汇总提升达到 +7.5 个百分点。 结论不是「skill 越多越好」，而是要在正确的任务里找到正确的 skill，再让 harness 真正把它执行出来。

DeepSeek V4 Flash 0731 is now 90% off on Nous Portal for the next 7 days, in partnership with @novita_labs. At this discounted price, it is over 1000x cheaper than Fable 5 on comparable tasks while still beating it on Terminal-Bench 2.1. Try it today at

DeepSeek-v4-Flash-0731 我自己使用，并没有Benchmark看起来那么厉害，但是相比于价格，这些缺点都能忍受。 有点DeepSeek-V3.2对比V3的感觉，后训练很重要很重要。