This paper explores how large-scale language models (LLMs) can acquire new language abilities and adapt to new domains through continuous pretraining (CPT). Specifically, we systematically analyze the impact of optimal selection of key hyperparameters, such as the mixing ratio of additional languages or domain corpora, on model performance. We perform CPT to improve Chinese proficiency using the Llama-3 8B and 70B models, and study the optimal correlation between the additional language mixing ratio (ALMR) and learning rate (LR) in the 8B model to derive optimal experimental settings. Through careful selection and fine-tuning of hyperparameters, we improve model performance not only on Chinese-related benchmarks but also in specific domains such as mathematics, coding, and emotional intelligence. We deploy the final 70B model in a real-world chat system, achieving satisfactory performance.
