This paper presents a skeptical perspective on claims that AI contributes to science, particularly the claim that AGI will cure all diseases or dramatically accelerate scientific discovery. It raises a key epistemological question: do large-scale language models (LLMs) generate new knowledge or simply reassemble fragments of memory? We propose "unlearning-as-ablation" as a testable method for answering this question. This involves removing a specific result and all supporting information (such as lemmas, alternative representations, and multi-step inferences) from the model, and then assessing whether the model can re-derive that result using only permitted axioms and tools. Success demonstrates generative capabilities beyond mere memorization, while failure demonstrates current limitations. This paper outlines a minimal pilot study demonstrating the feasibility of this method using mathematical and algorithmic examples, and discusses its potential extension to other fields such as physics and chemistry. This paper is an argumentative paper, focusing on conceptual and methodological contributions rather than empirical results. It aims to stimulate discussion on how principled ablation tests can help distinguish between AI reconstructing scientific knowledge and merely retrieving it, and how such tests can guide next-generation AI-for-Science benchmarks.
