Jimenez, M;
Rwemalika, R;
Papadakis, M;
Sarro, F;
Le Traon, Y;
Harman, M;
(2019)
The Importance of Accounting for Real-World Labelling When Predicting Software Vulnerabilities.
In: Dumas, M and Pfahl, D and Apel, S and Russo, A, (eds.)
Proceedings of the 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE 2019).
(pp. pp. 695-705).
ACM (Association for Computing Machinery): New York, NY, USA.
Preview |
Text
FSE19.pdf - Accepted Version Download (2MB) | Preview |
Abstract
Previous work on vulnerability prediction assume that predictive models are trained with respect to perfect labelling information (includes labels from future, as yet undiscovered vulnerabilities). In this paper we present results from a comprehensive empirical study of 1,898 real-world vulnerabilities reported in 74 releases of three security-critical open source systems (Linux Kernel, OpenSSL and Wiresark). Our study investigates the effectiveness of three previously proposed vulnerability prediction approaches, in two settings: with and without the unrealistic labelling assumption. The results reveal that the unrealistic labelling assumption can profoundly mis- lead the scientific conclusions drawn; suggesting highly effective and deployable prediction results vanish when we fully account for realistically available labelling in the experimental methodology. More precisely, MCC mean values of predictive effectiveness drop from 0.77, 0.65 and 0.43 to 0.08, 0.22, 0.10 for Linux Kernel, OpenSSL and Wiresark, respectively. Similar results are also obtained for precision, recall and other assessments of predictive efficacy. The community therefore needs to upgrade experimental and empirical methodology for vulnerability prediction evaluation and development to ensure robust and actionable scientific findings.
| Type: | Proceedings paper |
|---|---|
| Title: | The Importance of Accounting for Real-World Labelling When Predicting Software Vulnerabilities |
| Event: | 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering ( (ESEC/FSE 2019), 26-30 August 2019, Tallinn, Estonia |
| Location: | Tallinn, ESTONIA |
| Dates: | 26 August 2019 - 30 August 2019 |
| ISBN-13: | 978-1-4503-5572-8 |
| Open access status: | An open access version is available from UCL Discovery |
| DOI: | 10.1145/3338906.3338941 |
| Publisher version: | https://doi.org/10.1145/3338906.3338941 |
| Language: | English |
| Additional information: | This version is the author accepted manuscript. For information on re-use, please refer to the publisher’s terms and conditions. |
| Keywords: | Software Vulnerabilities, Machine Learning, Prediction Modelling |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > UCL BEAMS UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10083408 |
Archive Staff Only
 |
View Item |
