Determining the causes of success or failure of reinforcement learning (RL) policies is a challenging problem due to the complex and high-dimensional nature of agent-environment interactions. This study takes a causal perspective to explain the behavior of RL policies by considering states, actions, and rewards as variables in a low-level causal model. We introduce random perturbations to actions during policy execution and observe their effects on the cumulative rewards to learn a simplified high-level causal model that describes these relationships. To this end, we develop a nonlinear causal model reduction framework that guarantees approximate intervention consistency, meaning that the simplified high-level model responds to interventions in a similar way to the original complex system. We demonstrate that there exists a unique solution that achieves exact intervention consistency for a class of nonlinear causal models, ensuring that the learned explanations reflect meaningful causal patterns. Experiments on synthetic causal models and real-world RL tasks, including pendulum control and robotic table tennis, demonstrate that the proposed approach can uncover important behavioral patterns, biases, and failure modes in trained RL policies.
