This paper presents a study on jailbreak attacks that exploit vulnerabilities in large-scale language models (LLMs) to induce malicious content generation. In particular, we focus on a common attack method that confuses LLMs by using ambiguous prompts, and analyze the attention weight distribution to reveal the internal relationship between the input prompts and outputs of LLMs. Using statistical analysis methods, we define new metrics such as attention strength (Attn_SensWords), context-dependent score (Attn_DepScore), and attention distribution entropy (Attn_Entropy), and utilize them to propose an attention-based attack (ABA) strategy inspired by the "deception attack" strategy. ABA works by switching the attention distribution of LLMs using overlapping prompts to focus attention on benign parts. In addition, we present an attention-based defense (ABD) strategy based on ABA to improve the robustness of LLMs by adjusting the attention distribution. Through experimental results, we verify the effectiveness of ABA and ABD, and show that the attention weight distribution has a significant impact on the output of LLM.
