This page organizes papers related to artificial intelligence published around the world. This page is summarized using Google Gemini and is operated on a non-profit basis. The copyright of the paper belongs to the author and the relevant institution. When sharing, simply cite the source.
This paper challenges the conventional wisdom that inference performance in large-scale language models (LLMs) requires a large dataset. By carefully selecting and resampling a high-quality dataset, the authors demonstrate that powerful inference performance can be achieved with just 2 trillion tokens. Using this data, they pre-trained the 4.2 trillion tokens obtained in this manner, and then post-trained the model to develop MobileLLM-R1. This model significantly outperformed models trained on existing open-source data, and MobileLLM-R1-950M achieved performance comparable to that of Qwen3-0.6B. The results, training recipe, data sources, data mixing ratios, and model checkpoints are made public to support future research.
Takeaways, Limitations
•
Takeaways:
◦
Demonstrates that the scale of data required to develop LLM inference capabilities can be significantly reduced.
◦
Emphasize the importance of data quality management and curation.
◦
We demonstrate that powerful inference capabilities can be achieved even with small models.
◦
Facilitating the reproducibility and extension of research by making training recipes, data, and models public.
•
Limitations:
◦
Additional analysis of the characteristics of the dataset used may be required.
◦
Performance comparisons across various inference benchmarks are still needed.
◦
Further validation of the model's generalization ability is needed.
