This paper builds on the achievements of denoising-based generative models, particularly diffusion and flow-matching algorithms, to address the challenges of aligning the output distribution of generative models with complex sub-objectives such as human preference, compositional accuracy, and data compression ratio. To overcome the limitations of existing reinforcement learning (RL) fine-tuning methods, we reinterpret RL fine-tuning for diffusion models in terms of stochastic differential equations and implicit reward conditioning. We present Reinforcement Learning Guidance (RLG), an inference-time method that combines the outputs of a base model and an RL fine-tuned model via geometric means and applies classifier-free guidance (CFG). Theoretical analysis demonstrates that the guidance metric of RLG is mathematically equivalent to adjusting the KL-regularization coefficient in standard RL objectives, enabling dynamic control of alignment-quality trade-offs without additional training. Extensive experiments demonstrate that RLG consistently improves the performance of RL fine-tuned models across a variety of architectures, RL algorithms, and sub-tasks (including human preference, compositional control, compression ratio, and text rendering). Furthermore, RLG supports both interpolation and extrapolation, providing unprecedented flexibility in controlling generative alignment. In conclusion, this paper presents a practical and theoretically sound solution for improving and controlling diffusion model alignment during inference.
