This paper theoretically explores the benefits of tool-based language models (e.g., external retrieval, memory, and APIs). Specifically, we demonstrate the superiority of tool-based learning (external retrieval) over weighted learning (memorization) in terms of factual information representation. While the number of model parameters fundamentally limits the number of facts that can be memorized, we demonstrate that tool-based learning enables infinite fact representation through simple and efficient circuitry. Controlled experiments demonstrate that tool-based models outperform memorization models, demonstrating that teaching tool-based learning and general rules is more effective than memorizing facts in a pre-trained, large-scale language model. In conclusion, we demonstrate, through theoretical and experimental evidence, that tool-based workflows are not only practical but also theoretically superior in terms of scalability.
