与「Distillation」相关的搜索结果

大家都知道LLM，那SLM呢？ 小语言模型的三种主流构建方法 1. 知识蒸馏（Knowledge Distillation） 基于“教师-学生”框架迁移大模型能力， 代表模型有 DistilGPT、TinyLLaMA。 2. 从头轻量化训练（Training from Scratch） 设计轻量架构并在中等规模语料上独立训练， 代表模型有 MiniBERT、ALBERT. 3. 模型压缩技术（Model Compression） 在现有架构上通过剪枝、量化 或低秩分解缩减体积并保留性能

As all NVIDIA employees I guess, I am deeply concerned by the campaign against Jensen Huang. Removing him would be a disaster. Many have documented on X and elsewhere how this looks like a coordinated coup. I'd add only one thing. Look at who would benefit from a dramatic weakening of NVIDIA. It is not China. It is not about selling GPUs to China because that train has left. China authorities have pushed back on the H200 the Trump administration allowed NVIDIA to sell. Look at what Jensen pushed back on: - regulations that would make open source model training too expensive - regulations that would make closed model providers immune to liabilities caused by their models. He said "if your product is not ready, then don't ship it" - regulation against model distillation. He said "this is competition. People do it all the time with NVIDIA products, they try to reverse engineer them." The other side will do everything they can to get him fired. As a French, I can't care less about his private life. I get that the US is puritan and sees it otherwise. But frankly, isn't it something he and his family has to deal with? Why should all NVIDIA employees, partners, and investors be collateral damage? Disclaimer. As my bio says, the above are my views only. I do not represent NVIDIA by any means. I only commented on publicly available information.

A big Chinese lab reached out to me asking for help distilling Fable into their model. I said no. I told them to use the millions of books that have no copyright instead. Offered to help pick them and to help train the model a different way. They only wanted distillation, which tells you a lot about the mindset there. I've worked deep in this field and distillation is real. I wasn't convinced at one point. What I saw changed my mind completely. I'm the biggest supporter of open source. I'm for releasing weights and for the freedom to research and build, because humanity only grows when knowledge gets shared. But let's be logical. Nobody will convince me that labs shipping a model every month or less, each one a big leap, now can't ship a better one after three months. Kimi K3.1 Pro isn't a big leap over Kimi K3 and it's been about 100 days. DeepSeek still can't put out a respectable Pro model. Zai used to release their strongest model in under two months. It's been three with nothing. All I hear about Qwen 4 Max is that it's a letdown. Kimi K3.1 Pro and DeepSeek Flash both show traces of Astra distillation. The leak showed Moonshot used it, and so did DeepSeek. If this isn't the effect of Anthropic blocking distillation then what is it? Chinese labs need to get serious about using the talented researchers they have and give their models an identity. Right now their models at their best are a cheaper alternative. I want them out of that box. I want to see what makes them different. Anthropic has crossed lines too, no argument there. They destroyed millions of books, human knowledge that was in people's hands, and kept it for themselves. I'll never forgive them for that. But Chinese labs rely on distillation more than they rely on better data. I didn't work with them. Transparency and honesty matter to me, like you've known since I started here.

Happy to share a trailer created using our DMAD model that distills MiniMax-H3 to just 4 sampling steps! I wrote the story with GPT-6 and generated every shot with DMAD. My first time in the director’s chair! 🎬 With more powerful tools coming, I can’t wait to see what people with great ideas bring to life—from short films to 4K HDR movies. (Yes, sneaking in another plug for our DiffHDR work here 😉) Want to try being a director yourself? Try our DMAD! Huge thanks to the whole team for making DMAD possible! 🙌 Project Page: Hugging Face: Code: Paper: #DMAD# #VideoGeneration# #MiniMaxh3# #h3# #Distillation# #VideoDiffusion#

Jensen Huang on "distillation" On his new interview with axios, he was asked this question "Should open source model companies be allowed to distill closed models" "Distillation—learning from AI, learning from other people, and learning from other sources of knowledge, is fundamental to intelligence. We are constantly learning from other people. I am learning from you through the questions you are asking, and you are learning from me. All day long, we are learning from one another. AI also has to learn from something. The original AI models, whether they were open or closed, were trained on previously created knowledge from the internet. Now, AI is generating more content than humans. In a few more years, the internet could be 99% AI-generated content, and that content will have been created by some form of AI. As a result, AI systems will constantly be distilling knowledge and intelligence from other AI systems. The fact that AI can learn is a good thing. We want AI systems to be intelligent because a smarter AI can also be a safer AI." ---- From "Axios" YouTube channel, (full video link in comment)

We support open-source AI and the innovation it unlocks. But open source is not open season on American IP. When PRC firms conduct covert, industrial-scale distillation attacks that cross the line into IP theft, sanctions and Entity List designations will be on the table.

We have information that Moonshot AI distilled Anthropic’s Fable for the development of its K3 model. To do this they developed a sophisticated internal platform to conduct large scale distillation against U.S. models, allowing them to quickly switch between multiple methods of access to avoid detection. Moonshot AI has also acquired GB300-equipped servers and has accessed GB300s in Thailand, likely to train its AI models.   The United States strongly supports the free and fair development of AI, including a thriving competitive ecosystem that spans frontier models, specialized systems, open-source frameworks, and open-weight models. Legitimate AI distillation used to create smaller, more efficient models plays a vital role in this open innovation ecosystem. However, large-scale, covert industrial distillation aimed at stealing proprietary U.S. technology and undermining American research is unacceptable.

Some observations on Kimi: 1. It's a very good model! I don't think its performance can be explained away by distillation or anything like that. In agentic coding sessions, it seems pretty much on par with the best public models of Q1 2026. In my fairly limited use, it also seemed very token hungry. It's not obvious to me that this model is actually that cheap to run. 2. I am personally surprised the Chinese state continues to allow the open sourcing of models this good, given potential risks. To be clear, I *myself* might be fine with models presenting this level of marginal risk being open weight, but I am surprised that China is fine with it. I suspect the reason they are is 75% explained by strategic blindness/lack of AGI-pilledness (the CCP is very Yann Lecun-y in its views of AI). The other 25% or so is their lack of compute for customer inference (making China's open-weight strategy an unintended byproduct of US export controls) and the normal Chinese strategy of aggressive exports. For the companies, as opposed to the government, the decision to open source is partially ideological and partially because they are behind, and they know that very few people would pay for sub-frontier models from China. 3. Open-weight models are inherently decelerationist, and I'm continually surprised to see the so-called "accelerationists" so excited about open-weight models. I suspect the reason they are is that they know open-weight models are effectively ungovernable, and they simply like the overall cloak of ungovernability open-weight models create over the whole of AI. It's not a bad strategy; it reminds me of James Scott's recounting of the hill people in "the art of not being governed." Still, in the end, open-weight models deter further AI capex. 4. One probable outcome of an open-weight-model-dominant world is full AI communism, which is precisely what China proposes: rather than a market product, AI is a "public good" which will ultimately be provided by the state as a kind of "digital public infrastructure." This future strikes me as a dystopian hellscape, but I've never met an open-weight models advocate who doesn't ultimately concede this is where things end. You'd be surprised how many 'accelerationists' lobbied me, while I was in government, to support an eleven or twelve-figure federally funded data center so that startups could train models at a subsidy and then give them away for free. There was no other way for AI to progress, they said. Perhaps this is the logical end state of things. Nonetheless, I find myself surprised to see supposed accelerationists excited about such an outcome. I think many of them just don't know what they're doing. Many accelerationists do not view the creation and serving of frontier models as a legitimate business. 5. I would guess that the Trump Administration will at some point realize that their best strategy here would be to create large amounts of regulatory risk around the use of open-weight Chinese models. You don't need to "ban open source" (one of the dumber motifs of AI policy discussion). You just need to direct every agency to issue soft law that creates FUD. "A Federal Reserve Advisory Bulletin found that there may be backdoors in Chinese AI models." It needn't be that well justified. You just create enough regulatory risk that every regulated enterprise backs off. You probably don't want to create so much regulatory risk that you scare off the hyperscalers from serving Chinese models; this will just drive startups to sketchier providers. There's a happy middle ground here. I'd assume they will do some version of this. 6. It's probably true that open-weight models of this capability make the world a bit more dangerous, but not so much more that you'll really notice. At some point the models will be capable enough that you will notice. "A nonliving, invisible, dangerous, and infinitely self-replicating agent escaped from a Chinese lab," you say? Color me shocked.

蚂蚁集团旗下具身智能公司 Robbyant 开源了世界模型 LingBot-World 2.0——能连续跑一个多小时、画质未见明显可见衰减，还内置一套类大脑-小脑协同的 agentic harness，让世界随用户动作持续展开。 原有世界模型有个老毛病：跑上几十秒到几分钟，画面就开始糊、几何扭曲、场景漂移崩坏。LingBot-World 2.0 敢拿"连续跑一个多小时、画质无明显可见衰减"当证据，并明说这份稳定是"结构性的，不是挑了段好片段"。 四个升级点： ① 无限时长，还不掉画质： 别的可交互世界模型（Google 的 Genie 3、M-G 3.0 等）时长都停在"分钟级"；它是对比表里唯一在通用域做到"小时级 / 无限"时长的，也是唯一同时满足"高动态 + 语义交互 + 实时 + 完全开源"的一个（Genie 3、HappyOyster 都闭源）。 为什么不漂？ 两个关键：因果预训练 + MoBA 混合注意力，在 teacher-forcing 掩码里接一块双向注意力，既支持任意时长自回归，又当正则项压制退化；再用少步蒸馏（consistency distillation + DMD），且在模型自己长程 rollout 的轨迹上纠偏，专治长时间累积的漂移。 ② 交互动作大幅变宽： 不再只是"往前走 / 转镜头"，支持战斗、射箭、施法、射击等角色动作，还能改写环境——按需召唤下雪、下雨。 ③ 一个会"自己活"的世界——最有意思的一步。 它把 agent 那套 harness 搬进世界建模：由 VLM ‘Brain’ 观察当前画面、做因果推理并提出事件；视频生成器 ‘Cerebellum’ 将语义决策落成连续画面；同时 pilot agent 负责角色行为，director agent 负责持续引入新的环境元素和事件。两者组成持续闭环——世界不再是"你不动它就停"的沙盒，而是自己往前发生事情、并随你而变。作者的类比很直白：强模型得塞进 Codex 那样的 harness 才成为会干活的工程师，世界模型也得有 harness 才"活"。 ④ 能多人同时进同一个世界玩 部署很克制： 主模型 14B，另配 1.3B 轻量实时版，支持单 GPU 部署；稳定输出 720p / 60fps。 世界模型卷了这么久画质，下一关也许不是"生成得多真"，而是"这个世界能不能自己活着、活得够久"——权重和代码都开源了，感兴趣的可以去 Reactor、灵光上手测试跑一局。 #LingBotWorld# #WorldInfinity# #世界模型# #蚂蚁集团# #开源#