The generation quality of large-scale language models (LLMs) is often improved using inference-time sequence-level scaling methods (e.g., Chain-of-Thought). In this paper, we introduce hyper-parallel scaling, a complementary framework that improves prediction quality at the token level. Hyper-parallel scaling computes and aggregates multiple output proposals for a single token in the model. In this study, we implement this approach in a Mixture-of-Experts (MoE) model, called Roster of Experts (RoE). RoE is a learning-free inference algorithm that transforms a single MoE into a dynamic MoE ensemble. RoE injects a controlled probabilistic element into the expert routing mechanism, allowing it to sample multiple different experts for each token and aggregate their outputs for a more accurate final prediction. To overcome computational costs, we introduce an efficient batching strategy and a specialized KV caching mechanism that minimizes compute and memory overhead. For example, using RoE, a 7B MoE model can match the performance of a 10.5B MoE model with 30% less computation during inference. These benefits can be achieved without fine-tuning model parameters.
