This paper presents Scalable Logical Reasoning (SLR), an end-to-end framework for systematic evaluation and training of large-scale language models (LLMs). Based on a user's task specification, SLR automatically generates (i) instructional prompts for inductive reasoning tasks, (ii) executable verification programs (with verifiable rewards) for model output, and (iii) potential ground truth rules. This process is fully automated and scalable, requires no human annotation, and allows for precise control of task difficulty. Using SLR, we create SLR-Bench, a benchmark consisting of 19,000 prompts organized into 20 curriculum levels of increasing relational, arithmetic, and recursive complexity. Large-scale evaluations show that state-of-the-art LLMs readily generate syntactically valid rules but often fail to perform accurate logical reasoning. While recent inference LLMs have improved performance, they incur a very high test-time computational cost, exceeding $300 for 1,000 prompts. Finally, curriculum learning via SLR doubled the SLR-Bench accuracy of Llama-3-8B, reaching a level comparable to Gemini-Flash-Thinking at a significantly lower computational cost. Furthermore, this inference ability generalizes to various existing benchmarks, highlighting the effectiveness of SLR for downstream inference.
