This paper studies Markov decision processes (MDPs) influenced by external temporal processes to overcome the limitations of existing reinforcement learning algorithms, which primarily assume a static environment. We demonstrate that, when changes caused by external processes satisfy certain conditions, the problem can be solved by considering only a finite history of past events. To achieve this, we propose a policy iteration algorithm that considers both the current state of the environment and a finite history of past external process events, and conduct a theoretical analysis. While the algorithm does not guarantee convergence, it guarantees policy improvement in specific regions of the state space, depending on the errors caused by the approximate policy and value functions. Furthermore, we present the sample complexity of a least-squares policy evaluation and policy improvement algorithm that considers the approximation due to the integration of finite past temporal events. It is applicable to general discrete-time processes satisfying certain conditions, and we provide additional analysis of a discrete-time Hawkes process with Gaussian marks. We also present experimental results for policy evaluation and deployment in a traditional control environment.
