In this paper, we propose a low-rank and sparse model merging (LoRS-Merging) technique to address the linguistic diversity problem in multilingual speech-to-text (S2T) tasks. Existing multilingual multi-task learning approaches aim to jointly optimize multiple speech recognition and translation tasks across multiple languages, but suffer from high computational cost, language interference, suboptimal training configurations, and limited scalability. LoRS-Merging combines low-rank and sparse pruning to remove redundant parameters while preserving essential structures, thereby mitigating language interference and improving scalability. Experimental results on ten languages show that LoRS-Merging outperforms multilingual multi-task learning, sequential learning, and other merging methods by more than 20%. Therefore, LoRS-Merging suggests a scalable and effective complement to existing multilingual learning strategies for S2T applications.
