This paper comprehensively reviews recent research trends on the alignment problem of large-scale language models (LLMs) from an inverse reinforcement learning (IRL) perspective. It highlights the differences between reinforcement learning techniques used in LLM alignment and those used in traditional reinforcement learning tasks, and in particular discusses the necessity of constructing neural network reward models from human data and the formal and practical implications of this paradigm shift. After introducing the basic concepts of reinforcement learning, we cover practical aspects of IRL for LLM alignment, including recent advances, key challenges and opportunities, datasets, benchmarks, evaluation metrics, infrastructures, and computationally efficient training and inference techniques. Based on the research results on sparse reward reinforcement learning, we suggest open challenges and future directions. By synthesizing various research results, we aim to provide a structured and critical overview of the field, highlight unresolved challenges, and suggest promising future directions for improving LLM alignment with RL and IRL techniques.
