This paper studies a novel problem called Future Off-Policy Evaluation (F-OPE) and Learning (F-OPL), which estimates and optimizes the future value of a policy in a nonstationary environment. For example, in e-commerce recommendation, the goal is to estimate and optimize the policy value in the next month using data collected with old policies in the previous month. A key challenge is that data relevant to the future environment are not observed in the past data. Existing methods either assume stationarity or rely on restrictive reward modeling assumptions, which introduce significant biases. To address these limitations, we propose a novel estimator, \textit{\textbf{O}ff-\textbf{P}olicy Estimator for the \textbf{F}uture \textbf{V}alue (\textbf{\textit{OPFV}})}, which is designed to accurately estimate the policy value at any future point in time. The key feature of OPFV is its ability to exploit useful structures in time series data. For example, it can exploit seasonal, weekly, or holiday effects that are consistent in both the past and future data, even though the future data may not be present in the past log. This estimator is the first to exploit this temporal structure with a novel type of importance weighting to enable effective F-OPE. Theoretical analysis reveals the conditions under which OPFV is low-biased. In addition, we extend this estimator by developing a novel policy-gradient method that learns future policies a priori using only past data. Experimental results show that the proposed method significantly outperforms existing methods in estimating and optimizing future policy values in various experimental settings under nonstationarity.
