This paper presents a method for performing secure reinforcement learning under constraints expressed in natural language. Existing methods have the limitation of requiring manual design of cost functions for each constraint. In this paper, we propose the Trajectory-level Textual Constraints Translator (TTCT), which automatically generates cost functions using natural language constraints. TTCT learns by combining natural language constraints with trajectories, and experimental results demonstrate that it learns policies with lower violation rates than existing manually designed cost functions. Furthermore, we demonstrate zero-shot transfer capability, which can be applied to environments with changing constraints.