This paper presents Search-Based Preference Weighting (SPW), a novel method that integrates two types of human feedback—expert demonstrations and preferences—to address the challenges of reward function design in offline reinforcement learning. For each transition within a preference-labeled trajectory, SPW finds the most similar state-action pair from expert demonstrations and directly derives step-by-step importance weights based on their similarity scores. These weights guide standard preference learning, enabling accurate credit assignment, a challenge faced by existing methods. It demonstrates superior performance over existing methods on a robot manipulation task.
