This paper presents a scalable framework, X-Teaming, to address the safety risks of language models (LMs) in multi-round interactions. X-Teaming systematically explores how seemingly innocuous interactions escalate to harmful outcomes and generates corresponding attack scenarios. Using collaborative agents for planning, attack optimization, and validation, it achieves state-of-the-art multi-round jailbreak effectiveness and diversity with a success rate of up to 98.1% on leading open- and closed-source models. Specifically, it achieves a 96.2% attack success rate against the state-of-the-art Claude 3.7 Sonnet model, which was previously considered nearly immune to single-round attacks. Furthermore, we introduce XGuard-Train, an open-source multi-round safety training dataset consisting of 30,000 interactive jailbreaks, which is 20 times larger than previous state-of-the-art resources.
