To address the challenges of predicting conversion rates (CVR) for inactive users in large-scale e-commerce recommender systems, we propose the ChoirRec framework, which leverages the semantic capabilities of large-scale language models (LLMs) to organize user groups and improve CVR predictions for inactive users. ChoirRec consists of a semantic group generation module for filtering out noise signals, a group-aware hierarchical representation module for augmenting sparse user embeddings, and a group-aware multi-particle module that utilizes a dual-channel architecture and adaptive fusion mechanism for effective learning and utilization of group knowledge. In offline and online experiments on the Taobao platform, we demonstrated a 1.16% increase in offline GAUC and a 7.24% increase in order volume in online A/B tests, demonstrating its practical applicability.