This paper proposes Mean-Field Occupation-Measure Learning (MF-OML), an online mean-field reinforcement learning algorithm for computing approximate Nash equilibria of large-scale collective sequentially symmetric games. MF-OML is the first fully polynomial-time multi-agent reinforcement learning algorithm that provably solves Nash equilibria (with vanishing mean-field approximation errors as the number of players N tends to infinity) beyond zero-sum games and latent game variants. For games with strong Lasry-Lions monotonicity, it achieves a high-probability regret upper bound of $\tilde{O}(M^{3/4}+N^{-1/2}M)$, as measured by the cumulative deviation from the Nash equilibrium, and for games with only Lasry-Lions monotonicity, it achieves a regret upper bound of $\tilde{O}(M^{11/12}+N^{- 1/6}M)$, where M is the total number of episodes and N is the number of agents in the game. As a by-product, we obtain the first tractable globally convergent computational algorithm for computing approximate Nash equilibria of monotonic mean-field games.
