In this paper, we propose a novel framework for evaluating the ability of large-scale language models (LLMs) to acquire new knowledge. The framework simulates a curious human being who encounters a sentence introducing scientific knowledge by inducing the LLM to generate questions for the first time. We evaluate the knowledge acquisition potential of the LLM by assessing the quality of the generated questions, and validate the validity of the scoring procedure through a controlled elimination study. We generate a synthetic dataset consisting of 1,101 sentences of varying difficulty in physics, chemistry, and mathematics, 300 general knowledge sentences, and 567 incorrect sentences, and validate the validity of the model evaluation through human evaluation (weighted Cohen's kappa approximately 0.7). We find that while large models such as GPT-4 and Mistral 8x7b are adept at generating consistent and relevant questions, the smaller Phi-2 model is equally or more effective. This suggests that model size is not the only factor determining knowledge acquisition potential. The proposed framework quantifies important model features that have been previously overlooked, and presents research opportunities for developing more knowledge-rich AI systems.
