The continuous nature of the belief states of a POMDP poses significant computational challenges for optimal policy learning. In this paper, we consider an approach to solve the PORL problem by approximating the POMDP model as a finite-state MDP (Superstate MDP). We present theoretical guarantees linking the optimal value function of the Superstate MDP with the optimal value function of the original POMDP, achieving improved results compared to existing studies. Next, we propose a policy-based learning approach that utilizes linear function approximation to learn optimal policies for Superstate MDPs. The proposed approach treats the POMDP as an MDP and demonstrates that it can be approximately solved using TD learning and policy optimization. Here, the MDP states correspond to a finite history, and the approximation error decays exponentially with the length of this history. We also present a finite-time bound that explicitly quantifies the error incurred when applying standard TD learning in environments where the true dynamics are non-Markovian.
