This paper proposes a conceptual revision of the three core principles of reinforcement learning (RL): the definition of the agent, the goal of learning, and the scope of the reward hypothesis. We reexamine these three principles from an evolutionary perspective, and first argue that evolutionary dynamics can operate in the brain throughout the life of an individual, in order to support the validity of RL as a model of biological learning. For the second principle (the goal of learning), we enrich the ‘adaptation rather than exploration’ perspective with evolutionary insights, and for the third principle (the scope of the reward hypothesis), we discuss scalar rewards and multi-objective problems through the analogy of evolutionary fitness. After discussing practical implications for exploration in RL, we address the absence of the first principle, the definition of the agent. We argue that the agent problem cannot be solved by the evolutionary paradigm alone, and suggest that the thermodynamics of nutrition and replication in the origin of life theory provide a promising foundation for understanding agents and resource-constrained reinforcement learning in biological systems.
