This paper addresses privacy and ethical concerns raised by the rapid development of Zero-Shot Text-to-Speech (ZS-TTS) technology, particularly the potential for unwanted individual voice cloning. To address this, we propose a method for selectively removing speaker information from a ZS-TTS system. Specifically, we propose a novel machine learning unlearning framework, Teacher-Guided Unlearning (TGU), which trains a model to forget the voice of a specific speaker while retaining the ability to generate voices from other speakers. Furthermore, we introduce randomness to ensure that the forgotten speaker's voice cannot be traced, and we propose a new evaluation metric, Speaker-Zero Retrain Forgetting (spk-ZRF), to evaluate the model's ability to ignore prompts related to the forgotten speaker. Experimental results demonstrate that TGU prevents voice cloning for the forgotten speaker while maintaining the speech quality of other speakers.
