Daily Arxiv

This is a page that curates AI-related papers published worldwide.
All content here is summarized using Google Gemini and operated on a non-profit basis.
Copyright for each paper belongs to the authors and their institutions; please make sure to credit the source when sharing.

LagKV: Lag-Relative Information of the KV Cache Tells Which Tokens Are Important

Created by
  • Haebom

Author

Manlai Liang, JiaMing Zhang, Xiong Li, Jinlong Li

Outline

In this paper, we propose LagKV, a novel KV compression strategy that does not rely on attention weights, to address the issue of increasing key-value (KV) cache size in long-text inference for large-scale language models. While existing attention-weighted methods require major modifications of the inference infrastructure and significant computational overhead, LagKV achieves efficient compression without any attention computation by simply using comparisons between KVs. On the RULER benchmark results, LagKV outperforms SnapKV and StreamingLLM, and in particular, it outperforms the attention-weighted method $H_2O$ by more than 50% on the 64-character password retrieval task. The source code is available on GitHub.

Takeaways, Limitations

•
Takeaways:
◦
We propose a novel KV compression strategy, LagKV, that does not rely on attention weights, thereby solving the computational overhead and infrastructure modification problems of existing methods, which are Limitations.
◦
It demonstrates superior performance over other existing KV compression methods on the RULER benchmark.
◦
In particular, it showed more than 50% performance improvement compared to $H_2O$ in the 64-digit password search task.
◦
It provides a simple way to integrate easily into mainstream inference platforms.
•
Limitations:
◦
Performance in benchmark environments other than the RULER benchmark presented in this paper requires additional verification.
◦
Further analysis is needed to determine whether LagKV's performance gains may be biased towards specific tasks or datasets.
◦
Further research is needed on generalization performance across a variety of large-scale language models and hardware environments.
👍
Made with Slashpage