This paper explores the use of Potential-Based Reward Shaping (PBRS) to address the sample inefficiency problem in Reinforcement Learning (RL). We highlight the difficulty of selecting an appropriate latent function and the bias inherent in using a finite horizon due to computational limitations. We then provide a theoretical rationale for why selecting an optimal value function as the latent function improves performance. We analyze the bias induced by a finite horizon in PBRS and, by leveraging abstraction to approximate the optimal value function, evaluate the sample efficiency and performance impact of PBRS in four environments, including a goal-directed navigation task and three Arcade Learning Environment (ALE) games. Experimental results demonstrate that a simple fully-connected network can achieve performance comparable to that of a CNN-based solution.
