In this paper, we propose a novel framework, Hard Preference Sampling (HPS), to align responses of large-scale language models (LLMs) to human preferences. Existing Plackett-Luce (PL) and Bradley-Terry (BT) model-based preference optimization methods have problems such as difficulty in handling harmful content, inefficient utilization of non-preferred responses, and high computational cost of PL. HPS solves these problems by introducing a training loss that prioritizes the most preferred responses and rejects all non-preferred and harmful responses. In particular, it enhances the rejection ability of the model by emphasizing “hard” non-preferred responses that are similar to preferred responses, and it maintains alignment quality while reducing computational overhead by utilizing a single-sample Monte Carlo sampling strategy. Theoretically, HPS improves the sample efficiency over existing PL methods and maximizes the compensation margin between preferred and non-preferred responses, ensuring a clearer distinction. Through experiments on the HH-RLHF and PKU-Safety datasets, we verify the effectiveness of HPS, achieving similar BLEU and reward scores while significantly improving the reward margin, thereby reducing the generation of harmful content.
