This paper proposes QSpec, a novel quantization paradigm that improves upon widely used quantization techniques for accelerating Large Language Model (LLM) inference and reducing memory usage. QSpec decouples efficiency and quality by combining low-precision joint quantization for fast drafting and high-precision weight-only quantization for accurate verification. QSpec minimizes transition costs by reusing weights and KV caches across stages without retraining or auxiliary models. It achieves up to 1.64x speedup compared to high-precision-based models and up to 1.55x performance improvement over conventional speculative decoding methods in batch environments. Furthermore, QSpec supports plug-and-play deployment and works well across a variety of model sizes, quantization methods, and workloads.
