alphaXiv (@askalphaxiv) “"Improved Large Language Diffusion Models" ByteDance just made bidirectional mas” — TopicDigg
alphaXiv
@askalphaxiv
High fidelity research
加入 November 2023
56 正在关注 48.4K 粉丝
"Improved Large Language Diffusion Models"
ByteDance just made bidirectional masked diffusion on-par with autoregessive LM!
This paper iLLaDA trains an 8B Transformer from scratch on 12T tokens, then keeps the same denoising objective for SFT on a 25B-token instruction corpus.
It improves LLaDA with GQA, tied embeddings, variable-length generation, confidence-based MCQ scoring, and packed-sequence diffusion SFT.
iLLaDA-Base raises the average score from 51.1 to 63.9 and slightly exceeds Qwen2.5 7B Base at 63.3, while iLLaDA-Instruct still trails Qwen2.5 Instruct without RL alignment.
显示更多
