In this paper, we present a quantitative measure of data quality, especially data diversity, in large-scale language model (LLM) pre-training. Previous LLM pre-training studies have mainly focused on model and dataset size expansion, but the importance of data quality has not been clearly defined. In response, we propose a metric called 'diversity coefficient' to measure the diversity of natural language data and the diversity of publicly available pre-training datasets. Through experiments on 44 models (in total) of various sizes (from 51M to 7B parameters) using GPT-2 and LLaMAv2, we show that the proposed diversity coefficient is correlated with the downstream model evaluation performance. In conclusion, the diversity coefficient is an important aspect of data quality and captures the causal relationship between data diversity and improved model performance.
