We conducted research on the phenomenon of memorization of training data in large-scale language models (LLMs). Specifically, we explored methods to characterize the difficulty of memorizing data, conducted experiments on the OLMo model, and proposed the entropy-memorization law. According to this law, there is a linear correlation between data entropy and memorization scores. Furthermore, through experiments on memorizing random strings (gibberish), we confirmed that random strings have lower entropy than training data. Based on these results, we developed a simple and effective dataset inference (DI) method that distinguishes between training and test data.
