QuantX is a collection of custom quantization recipes for LLM and VLM. It enables quantization up to 3-bit resolution with minimal performance degradation. QuantX's quantization strategy ensures a flexible trade-off between execution speed, memory requirements, and model accuracy, while considering hardware-specific constraints for efficient dequantization during inference. Experimental results show that QuantX achieves within 6% of the performance of the unquantized model for 3-bit quantized LlaVa-v1.6 on several end-user tasks, outperforming recently published state-of-the-art quantization techniques. We also integrate specific QuantX techniques into the popular Llama.cpp framework and demonstrate their feasibility in terms of execution time compared to mainstream quantization techniques in Llama.cpp. Finally, this paper provides insights into the LLM quantization process that led to the various recipes and options integrated into QuantX.
