Daily Arxiv

This is a page that curates AI-related papers published worldwide.
All content here is summarized using Google Gemini and operated on a non-profit basis.
Copyright for each paper belongs to the authors and their institutions; please make sure to credit the source when sharing.

Efficiently Serving Large Multimodal Models Using EPD Disaggregation

Created by
  • Haebom

Author

Gursimran Singh, Xinglu Wang, Yifan Hu, Timothy Yu, Linzi Xing, Wei Jiang, Zhefeng Wang, Xiaolong Bai, Yi Li, Ying Xiong, Yong Zhang, Zhenan Fan

Outline

In this paper, we propose a distributed Encode-Prefill-Decode (EPD) framework to address the performance degradation issue of large-scale multi-modal models (LMMs). LMMs process diverse inputs such as images, audios, and videos, but their multi-modal encoding steps incur increased computation and memory overhead, which degrades key service-level objectives (SLOs) such as response time. The EPD distributed framework addresses these issues by separating the encoding, prefill, and decoding steps into dedicated resources. Through multimedia token caching, encoding load parallelization, an optimal resource allocation module, and a role switching mechanism, it significantly improves memory efficiency, batch size, number of images per request, and KV cache size, thereby improving the SLO achievement rate and response time.

Takeaways, Limitations

•
Takeaways:
◦
A novel architecture that effectively addresses the performance degradation problem of LMMs is presented.
◦
Performance improvements in memory efficiency, batch size, and request processing speed (up to 15x memory reduction, up to 22x batch size increase, up to 10x image throughput increase, 2.2x KV cache increase, up to 71% response time reduction, 90-100% SLO achievement rate improvement)
◦
Presentation and experimental verification of various optimization techniques (multimedia token caching, encoding load parallelization, optimal resource allocation, role switching)
◦
Improving accessibility through open source code disclosure
•
Limitations:
◦
Additional costs and complexities may arise in implementing and operating the EPD distributed framework.
◦
Only experimental results for specific LMMs are presented, so further research is needed on generalizability to other LMMs or diverse environments.
◦
Long-term performance and stability verification in an actual service environment is required.
👍
Made with Slashpage