In this paper, we propose a novel framework, Hierarchical Adversarial Resilient Multi-Agent Reinforcement Learning (HAMARL), for enhancing cybersecurity of cyber-physical systems (CPS) that play a critical role in infrastructures in various fields such as manufacturing, energy distribution, and autonomous driving systems. Since conventional rule-based intrusion detection and single-agent reinforcement learning are ineffective against sophisticated cyber threats such as adaptive and zero-day attacks, HAMARL adopts a hierarchical structure consisting of local agents responsible for subsystem security and a global coordinator that supervises and optimizes the defense strategy of the entire system. In addition, it integrates an adversarial training loop designed to simulate and predict evolving cyber threats, enabling proactive defense adaptation. Extensive experimental evaluations conducted on a simulated industrial IoT testbed demonstrate that HAMARL significantly outperforms conventional multi-agent reinforcement learning approaches, significantly improving attack detection accuracy, reducing response time, and ensuring operational continuity. The results highlight the effectiveness of combining hierarchical multi-agent coordination and adversarial awareness training to enhance the resilience and security of next-generation CPSs.
