Daily Arxiv

This is a page that curates AI-related papers published worldwide.
All content here is summarized using Google Gemini and operated on a non-profit basis.
Copyright for each paper belongs to the authors and their institutions; please make sure to credit the source when sharing.

NoWag: A Unified Framework for Shape Preserving Compression of Large Language Models

Created by
  • Haebom

Author

Lawrence Liu, Inesh Chakrabarti, Yixiao Li, Mengdi Wang, Tuo Zhao, Lin F. Yang

Outline

In this paper, we propose Normalized Weight and Activation Guided Compression (NoWag), a unified framework for zero-shot shape-preserving compression algorithms to address the high computational and memory requirements that limit the deployment of large-scale language models (LLMs) in resource-constrained environments. NoWag compresses Llama-2 7B/13B/70B and Llama-3 8B/70B models using two shape-preserving compression methods: vector quantization (NoWag-VQ) and unstructured/semi-structured pruning (NoWag-P). Experimental results show that NoWag-VQ significantly outperforms state-of-the-art zero-shot vector quantization methods, and NoWag-P is competitive with state-of-the-art methods. This suggests commonalities between different compression paradigms and provides inspiration for future research. The source code is available on GitHub.

Takeaways, Limitations

•
Takeaways:
◦
NoWag provides a novel unified framework for efficient compression of LLM in zero-shot settings.
◦
NoWag-VQ outperforms existing state-of-the-art zero-shot vector quantization methods.
◦
NoWag-P performs competitively with existing state-of-the-art pruning methods.
◦
We reveal commonalities between different compression paradigms, namely vector quantization and pruning, and suggest future research directions.
•
Limitations:
◦
The experimental results presented in this paper are limited to specific LLM models (Llama-2, Llama-3) and further studies are needed to determine their generalizability.
◦
There is a lack of detailed analysis and interpretation of the performance improvements of NoWag.
◦
Lack of performance evaluation on various hardware platforms.
👍
Made with Slashpage