This paper focuses on uncertainty quantification in reinforcement learning, particularly in Bayesian deep Q-learning. Unlike previous studies that primarily focused on improving the accuracy of posterior distribution approximations, this paper investigates the accuracy of the prior distribution and likelihood assumptions that constitute the posterior distribution. The paper demonstrates the "cold posterior effect" in Bayesian deep Q-learning, whereby lowering the temperature of the posterior distribution improves performance, contrary to theory. To elucidate the cause of this phenomenon, we verify assumptions regarding the likelihood and prior distributions commonly used in Bayesian model-free algorithms, and experimentally demonstrate that the Gaussian likelihood assumption is frequently violated. Consequently, developing more appropriate likelihood and prior distributions is crucial for future Bayesian reinforcement learning research, and we propose a method to improve the prior distribution in deep Q-learning for better performance.
