This paper highlights that securely aligned large-scale language models (LLMs) are vulnerable to harmful fine-tuning attacks. A small amount of harmful data mixed into the fine-tuning dataset can break the secure alignment of the LLM. We show that existing defenses are ineffective under certain training hyperparameters (e.g., high learning rates or a large number of training epochs). Therefore, we propose Antidote, a post-fine-tuning solution that is independent of the training hyperparameters used during the fine-tuning phase. Antidote is based on the principle of removing harmful parameters to recover harmful models from harmful behavior. Experimentally, we demonstrate that Antidote reduces harmful scores while maintaining the accuracy of downstream tasks by introducing a one-time pruning step that removes harmful weights responsible for generating harmful content. The code is available on GitHub.
