Although the Mamba State-Space Model (SSM) outperforms state-of-the-art (SOTA) Transformer Large-Scale Language Models (LLMs) in many tasks and is widely applied, a key challenge for the stable training of recurrent-based deep models (e.g., SSMs) is their sensitivity to recurrent dynamics. In this paper, we empirically investigate the sensitivity of Mamba to recurrent dynamics under common fine-tuning methods, such as mixed-precision fine-tuning (MPFT) and parameter-efficient fine-tuning (PEFT). We demonstrate that the Mamba LLM is highly robust to variations in the combination of MPFT and PEFT, while the Transformer LLM can deviate significantly from the full-precision model under different combinations of MPFT and PEFT. We attribute the Mamba LLM's robustness to recurrent dynamics, and we demonstrate that its stability is guaranteed using dynamical systems theory (specifically, Lyapunov stability). Finally, we complement recent work by exploring the in-context learning (ICL) capabilities of the Mamba LLM for natural language processing tasks using MPFT and PEFT.
