Tsingenopoulos, Ilias; Rimmer, Vera; Preuveneers, Davy; Pierazzi, Fabio; Cavallaro, Lorenzo; Joosen, Wouter; (2025) The Adaptive Arms Race: Redefining Robustness in AI Security. In: Proceedings of The 28th International Symposium on Research in Attacks, Intrusions and Defenses (RAID 2025). Research in Attacks, Intrusions and Defenses (RAID): Gold Coast, Australia. (In press).

Text raid25-amg.pdf - Accepted Version Access restricted to UCL open access staff until 6 April 2026. Download (1MB)

Abstract

Despite considerable efforts on making them robust, real-world AI-based systems remain vulnerable to decision based attacks, as definitive proofs of their operational robustness have so far proven intractable. Canonical robustness evaluation relies on adaptive attacks, which leverage complete knowledge of the defense and are tailored to bypass it. This work broadens the notion of adaptivity, which we employ to enhance both attacks and defenses, showing how they can benefit from mutual learning through interaction. We introduce a framework for adaptively optimizing black-box attacks and defenses under the competitive game they form. To assess robustness reliably, it is essential to evaluate against realistic and worst-case attacks. We thus enhance attacks and their evasive arsenal together using reinforcement learning (RL), apply the same principle to defenses, and evaluate them first independently and then jointly under a multi-agent perspective. We find that active defenses, those that dynamically control system responses, are an essential complement to model hardening against decision-based attacks; that these defenses can be circumvented by adaptive attacks, something that elicits defenses being adaptive too. Our findings, supported by an extensive theoretical and empirical investigation, confirm that adaptive adversaries pose a serious threat to black-box AI-based systems, rekindling the proverbial arms race. Notably, our approach outperforms the state-of-the-art black-box attacks and defenses, while bringing them together to render effective insights into the robustness of real-world deployed ML-based systems.

Download activity - last month

Export as JSONCSVXML

Download activity - last 12 months

Export as XMLCSVJSON

Downloads by country - last 12 months

Export as CSVXMLJSON

Archive Staff Only

View Item