This paper presents a theoretical analysis and experimental verification of the interplay between the compressibility of neural networks (e.g., neuron-level sparsity, spectral compressibility, etc.) and their robustness against adversarial attacks. We show that compressed neural networks introduce a small number of high-sensitivity directions in the representation space, which can be exploited by adversarial attackers to generate effective interference. We provide simple yet informative bounds on the robustness of $L_\infty$ and $L_2$, and show that the vulnerabilities arise regardless of the compression method (e.g., regularization, structural bias, implicit learning dynamics, etc.). Experimental evaluations on synthetic and real tasks confirm the theoretical predictions, and show that these vulnerabilities persist under adversarial training and transfer learning, contributing to the emergence of universal adversarial interference. In conclusion, we reveal a fundamental tension between structural compressibility and robustness, and suggest new directions for the design of efficient and secure models.
