We present JALMBench, a comprehensive benchmark for evaluating the security of audio language models (ALMs). JALMBench comprises a dataset containing 11,316 text samples and 245,355 audio samples (over 1,000 hours), supporting 12 major ALMs, four text-based attack methods, four audio-based attack methods, and five defense methods. Using JALMBench, we provide in-depth analysis of attack effectiveness, topic sensitivity, voice diversity, and architecture, and explore attack mitigation strategies at the prompt and response levels.