Daily Arxiv

This is a page that curates AI-related papers published worldwide.
All content here is summarized using Google Gemini and operated on a non-profit basis.
Copyright for each paper belongs to the authors and their institutions; please make sure to credit the source when sharing.

Morse: Dual-Sampling for Lossless Acceleration of Diffusion Models

Created by
  • Haebom

Author

Chao Li, Jiawei Fan, Anbang Yao

Outline

In this paper, we present Morse, a simple dual-sampling framework that improves the speed of diffusion models without loss. Morse reconstructs the iterative generation process (from noise to data) by utilizing fast jump sampling and an adaptive residual feedback strategy. Two models, Dash and Dot, interact. The Dash model is an existing pre-trained diffusion model that operates in a jump-sampling manner to secure space for improved sampling efficiency. The Dot model is much faster than the Dash model and is trained to generate residual feedback conditioned on the observations at the current jump-sampling point in the trajectory of the Dash model, thereby improving the noise estimate to easily match the next-stage Dash model estimate without jump sampling. By temporally cross-linking the outputs of the Dash and Dot models, Morse flexibly achieves the desired image generation performance while improving the overall runtime efficiency. The weight sharing strategy between the Dash and Dot models makes it efficient in both training and inference. It shows lossless speedups of 1.78x to 3.31x over nine baseline diffusion models on six image generation tasks, and shows that it can also generalize to the Latent Consistency Model (LCM-SDXL) tuned for few-step text-to-image synthesis. The code and models are available at https://github.com/deep-optimization/Morse .

Takeaways, Limitations

Takeaways:
We present Morse, an effective double-sampling framework that improves the speedup of diffusion models from 1.78x to 3.31x without loss.
Significantly improve sampling efficiency through fast jump sampling and adaptive residual feedback strategies.
Improving training and inference efficiency through weight sharing strategy between Dash and Dot models.
It also shows applicability to other diffusion models such as LCM-SDXL.
Ensuring reproducibility and usability through open code and models.
Limitations:
The extent to which Morse's performance is improved may vary depending on the underlying diffusion model used and the task at hand.
It may only be applicable to certain types of diffusion models (needs to verify performance on models other than those covered in the paper).
Further analysis of the detailed effects and limitations of weight sharing strategies may be needed.
Performance evaluation is needed for more complex image generation tasks or high-resolution image generation.
👍