In this paper, we investigate whether the improvement in mathematical reasoning performance is due to major changes in the transformer layer in large-scale language models that exhibit improved mathematical reasoning performance through post-training techniques (instruction tuning, reinforcement learning, and knowledge distillation) or to minor adjustments that do not significantly change the relative layer importance structure of the base model. We compare the base model, instruction-tuned models, knowledge-distilled models, and reinforcement learning models on mathematical reasoning benchmarks using layer-by-layer removal experiments. We show that mathematical reasoning generates a specific layer importance structure that persists across all post-training methods. Removing these important layers leads to an accuracy degradation of up to 80%, but the important layers do not appear in non-mathematical tasks such as factual representation. This suggests that mathematical reasoning requires special layers that are generated during pre-training, but other non-inference tasks do not. From an information-theoretic perspective, we find that these important layers are identical to the layers where the main representation transformation occurs.
