UCL Discovery
UCL home » Library Services » Electronic resources » UCL Discovery

Who Watches the Watchers: A Multi-Task Benchmark for Anomaly Detection

Demetriou, K; Becker, I; Hailes, S; (2022) Who Watches the Watchers: A Multi-Task Benchmark for Anomaly Detection. In: Proceedings of the 8th International Conference on Information Systems Security and Privacy - ICISSP. (pp. pp. 579-586). International Conference on Information Systems Security and Privacy (ICISSP): Online conference. Green open access

[thumbnail of _PUBLIC_ACCESS___Who_Watches_the_Watchers__A_Multi_Task_Benchmark_for_Anomaly_Detection__ICISSP_2022_.pdf]
Preview
Text
_PUBLIC_ACCESS___Who_Watches_the_Watchers__A_Multi_Task_Benchmark_for_Anomaly_Detection__ICISSP_2022_.pdf - Published Version

Download (144kB) | Preview

Abstract

A driver in the rise of IoT systems has been the relative ease with which it is possible to create specialized-but- adaptable deployments from cost-effective components. Such components tend to be relatively unreliable and resource poor, but are increasingly widely connected. As a result, IoT systems are subject both to component failures and to the attacks that are an inevitable consequence of wide-area connectivity. Anomaly detection systems are therefore a cornerstone of effective operation; however, in the literature, there is no established common basis for the evaluation of anomaly detection systems for these environments. No common set of benchmarks or metrics exists and authors typically provide results for just one scenario. This is profoundly unhelpful to designers of IoT systems, who need to make a choice about anomaly detection that takes into account both ease of deployment and likely detection performance in their context. To address this problem, we introduce Aftershoc k, a multi-task benchmark. We adapt and standardize an array of datasets from the public literature into anomaly detection-specific benchmarks. We then proceed to apply a diverse set of existing anomaly detection algorithms to our datasets, producing a set of performance baselines for future comparisons. Results are reported via a dedicated online platform located at https://aftershock. dev, allowing system designers to evaluate the general applicability and practical utility of various anomaly detection models. This approach of public evaluation against common criteria is inspired by the immensely useful community resources found in areas such as natural language processing, recommender systems, and reinforcement learning. We collect, adapt, and make available 10 anomaly detection tasks which we use to evaluate 6 state-of-the-art solutions as well as common baselines. We offer researchers a submission system to evaluate future solutions in a transparent manner and we are actively engaging with academic and industry partners to expand the set of available tasks. Moreover, we are exploring options to add hardware-in-the-loop. As a community contribution, we invite researchers to train their own models (or those reported by others) on the public development datasets available on the online platform, submitting them for independent evaluation and reporting results against others.

Type: Proceedings paper
Title: Who Watches the Watchers: A Multi-Task Benchmark for Anomaly Detection
Event: 8th International Conference on Information Systems Security and Privacy (ICISSP)
ISBN-13: 978-989-758-553-1
Open access status: An open access version is available from UCL Discovery
DOI: 10.5220/0010915000003120
Publisher version: https://doi.org/10.5220/0010915000003120
Language: English
Additional information: This is an Open Access paper published under a Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0) Licence (https://creativecommons.org/licenses/by-nc-nd/4.0/).
Keywords: Anomaly Detection, Benchmark, Performance Metrics, Datasets, Reproducibility
UCL classification: UCL
UCL > Provost and Vice Provost Offices > UCL BEAMS
UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science
UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science
UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Security and Crime Science
URI: https://discovery.ucl.ac.uk/id/eprint/10140855
Downloads since deposit
34Downloads
Download activity - last month
Download activity - last 12 months
Downloads by country - last 12 months

Archive Staff Only

View Item View Item