Daily Arxiv

This is a page that curates AI-related papers published worldwide.
All content here is summarized using Google Gemini and operated on a non-profit basis.
Copyright for each paper belongs to the authors and their institutions; please make sure to credit the source when sharing.

Time Is a Feature: Exploiting Temporal Dynamics in Diffusion Language Models

Created by
  • Haebom

Author

Wen Wang, Bozhen Fang, Chenchen Jing, Yongliang Shen, Yangyi Shen, Qiuyu Wang, Hao Ouyang, Hao Chen, Chunhua Shen

Outline

Diffusion Large-Scale Language Models (dLLMs) generate text through iterative denoising, but current decoding strategies discard rich intermediate predictions for the final output. This study uncovers a temporal oscillation phenomenon where correct answers emerge during the intermediate stage and are subsequently overwritten during the denoising stage. To address this issue, we propose two complementary methods that leverage temporal consistency. First, Temporal Self-Consistency Voting (TSV), a training-free test-time decoding strategy, aggregates predictions from the denoising stage to select the most consistent output. Second, Temporal Consistency Reinforcement (TCR), a post-training method that encourages stable generation using Temporal Semantic Entropy (TSE), a measure of semantic stability in intermediate predictions, as a reward signal. Experimental results on several benchmarks demonstrate the effectiveness of the proposed method. Using only negative TSE compensation, we observe a remarkable average performance improvement of 24.7% over the existing dLLM on the Countdown dataset. Combined with accuracy compensation, we achieved absolute performance improvements of 2.0% on GSM8K, 4.3% on MATH500, 6.6% on SVAMP, and 25.3% on Countdown. These results highlight the untapped potential of dLLM's temporal dynamics and provide two simple yet effective tools for exploiting it.

Takeaways, Limitations

Takeaways:
We identify the temporal oscillation phenomenon occurring during the intermediate generation process of dLLM and propose two effective methods to improve it (temporal self-consistency voting and temporal consistency enhancement).
We experimentally demonstrate that leveraging temporal consistency can significantly improve the performance of dLLM (significant performance improvements on the GSM8K, MATH500, SVAMP, and Countdown datasets).
By providing a new understanding and utilization of the temporal dynamics of dLLM, it provides important Takeaways for future dLLM research and development.
Limitations:
The effectiveness of the proposed method may be limited to specific datasets and models. Additional experiments on diverse datasets and models are needed.
A detailed description of the definition and calculation of temporal semantic entropy (TSE) is lacking. Further analysis of the generalizability and limitations of TSE is needed.
Analysis of the computational complexity of temporal self-consistent voting and temporal consistency enhancement methods is lacking. Further consideration is needed for their efficiency in practical applications.
👍