Daily Arxiv

This is a page that curates AI-related papers published worldwide.
All content here is summarized using Google Gemini and operated on a non-profit basis.
Copyright for each paper belongs to the authors and their institutions; please make sure to credit the source when sharing.

KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Provider

Created by
  • Haebom

Author

Jiahao Wang, Jinbo Han, Xingda Wei, Sijie Shen, Dingyan Zhang, Chenguang Fang, Rong Chen, Wenyuan Yu, Haibo Chen

Outline

In this paper, we present the first systematic characterization of the KV$ workload patterns of real-world LLM service providers, given that intermediate result caching (KV$) plays a critical role in improving performance in large-scale language model (LLM) service provisioning, but system design decisions such as cache eviction policies are highly workload-dependent. We make several observations that have not been addressed in previous studies focusing on synthetic workloads: that KV$ reuse across requests is asymmetric, that reuse across single-turn requests is as important as reuse across multi-turn requests, that reuse times and probabilities vary across all requests, but patterns are predictable for specific request categories, and that the overall cache size is appropriate for an ideal cache hit ratio. Based on these characterizations, we propose a workload-aware cache eviction policy that improves service performance on real-world trace data, especially when cache capacity is limited.

Takeaways, Limitations

•
Takeaways:
◦
We systematically analyze the KV$ caching workload characteristics in a real LLM service environment for the first time, overcoming the limitations of existing synthetic data-based studies.
◦
We demonstrate the importance of KV$ reuse in both single-turn and multi-turn requests, and discover predictable reuse patterns across request categories.
◦
We propose a novel cache eviction policy that takes workload characteristics into account, suggesting the potential for performance improvement in real environments.
•
Limitations:
◦
Because the study results are based on data from a specific LLM service provider, generalizability to other service providers or LLM architectures may be limited.
◦
The performance of the proposed cache eviction policy needs to be further verified through extensive testing in real environments.
◦
More comprehensive studies on various LLM architectures and workloads are needed.
👍
Made with Slashpage