This paper proposes AutoSteer, an inference-time arbitration technique for improving the safety of multimodal large-scale language models (MLLMs). AutoSteer consists of three core components: a Safety Awareness Score (SAS), an adaptive safety explorer, and a lightweight rejection head, all without fine-tuning the underlying model. The SAS automatically identifies safety-related differences between layers within the model, the adaptive safety explorer estimates the likelihood of harmful outputs from intermediate representations, and the rejection head selectively adjusts output when safety risks are detected. Experimental results using LLaVA-OV and Chameleon demonstrate that AutoSteer significantly reduces the attack success rate (ASR) against textual, visual, and multimodal threats while maintaining general functionality. Therefore, AutoSteer can be established as a practical, interpretable, and effective framework for the secure deployment of multimodal AI systems.
