This paper reveals that the fundamental cause of mode collapse, a phenomenon in which diversity in LLM decreases during the post-training alignment process, lies not in algorithmic limitations, but in typicality bias within the preference data. The researchers theoretically established this finding and empirically verified it on a preference dataset. Based on this analysis, they propose Verbalized Sampling (VS), a simple, training-free prompting strategy that circumvents mode collapse. VS prompts the model to verbalize the probability distribution over a series of responses. Through various experiments, they demonstrate that VS significantly improves performance in various domains, including creative writing, conversation simulation, open-ended question-answering, and synthetic data generation, and particularly significantly increases diversity in the creative writing domain. Furthermore, they found that the effect of VS tends to be greater for better-performing models.
