Daily Arxiv

This page organizes papers related to artificial intelligence published around the world.
This page is summarized using Google Gemini and is operated on a non-profit basis.
The copyright of the paper belongs to the author and the relevant institution. When sharing, simply cite the source.

DrDiff: Dynamic Routing Diffusion with Hierarchical Attention for Breaking the Efficiency-Quality Trade-off

Created by
  • Haebom

Author

Jusheng Zhang, Yijia Fan, Kaitong Cai, Zimeng Huang, Xiaofei Sun, Jian Wang, Chengpei Tang, Keze Wang

Outline

This paper introduces DrDiff, a novel framework for long-text generation. DrDiff overcomes the trade-off between efficiency and quality through three key techniques. First, we design a dynamic expert scheduling mechanism that intelligently allocates computational resources during the diffusion process based on text complexity, enabling more efficient handling of text generation tasks with varying difficulty. Second, we introduce a hierarchical sparse attention (HSA) mechanism that adaptively adjusts attention patterns based on varying input lengths, reducing computational complexity from O($n^2$) to O($n$) while maintaining model performance. Finally, we propose a soft absorption guidance optimization strategy, combined with DPM-solver++, that significantly improves generation speed by reducing the diffusion step. Through comprehensive experiments on various long-text generation benchmarks, we demonstrate the superiority of DrDiff over existing state-of-the-art methods.

Takeaways, Limitations

Takeaways:
Efficient allocation of computational resources based on text complexity through a dynamic expert scheduling mechanism.
Maintain performance by reducing attention computation complexity with the hierarchical sparse attention (HSA) mechanism.
Reduced diffusion step and increased production rate through soft absorption guidance optimization strategy.
Demonstrated superior performance over existing SOTA methods in various long-form text generation benchmarks.
Limitations:
The specific Limitations is not specified in the paper (in the Abstract).
👍