In this paper, we point out the Limitations (tradeoffs between conflicting objectives, low training efficiency, lack of scalability, and lack of explainability) of existing methods in solving multi-objective tasks in reinforcement learning (RL)-based fine-tuning of large-scale language models (LLMs), and propose a novel framework, EMORL (Ensemble Multi-Objective RL). EMORL fine-tunes multiple models with individual objectives, and aggregates the hidden states of these models after fine-tuning to improve efficiency and flexibility. In particular, we present the first hidden state aggregation method that integrates context information of multiple objectives, and a hierarchical grid search algorithm that finds the optimal weight combination. Through experiments on the counselor response generation task, we show that our proposed method significantly reduces training data consumption and time ($17,529\pm 1,650$ data points, $6,573\pm 147.43$ seconds, respectively) compared to existing methods, while maintaining similar performance on multi-objectives.
