This paper addresses the role of exemplar learning or preference signal learning in the post-training phase, a key step for applying pre-trained large-scale language models (LLMs) to real-world tasks. We present a theoretical framework that unifies preference learning methods such as Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO), and show through rigorous mathematical derivations that both SFT and DPO operate in the same optimal policy-reward subspace, and that SFT is a special case of implicit reward learning. We point out that an important Limitations of conventional SFT is that the KL divergence term of distribution matching during optimization becomes constant for the policy, failing to constrain model updating. To address this, we propose a learning rate decay technique, which achieves performance enhancement (up to 25% relative improvement and 6% absolute win rate increase). In addition, we derive an alternative SFT objective function derived from various f-divergence functions that maintain the KL term during optimization, which further improves the model performance after DPO, and extend the theoretical relationship between the LLM logit and the Q-function in preference learning to the SFT context, providing mathematical derivation and experimental verification.
