Morger, A;
Svensson, F;
Arvidsson McShane, S;
Gauraha, N;
Norinder, U;
Spjuth, O;
Volkamer, A;
(2021)
Assessing the calibration in toxicological in vitro models with conformal prediction.
Journal of Cheminformatics
, 13
(1)
10.1186/s13321-021-00511-5.
Preview |
Text
Svensson_s13321-021-00511-5.pdf - Published Version Download (2MB) | Preview |
Abstract
Machine learning methods are widely used in drug discovery and toxicity prediction. While showing overall good performance in cross-validation studies, their predictive power (often) drops in cases where the query samples have drifted from the training data’s descriptor space. Thus, the assumption for applying machine learning algorithms, that training and test data stem from the same distribution, might not always be fulfilled. In this work, conformal prediction is used to assess the calibration of the models. Deviations from the expected error may indicate that training and test data originate from different distributions. Exemplified on the Tox21 datasets, composed of chronologically released Tox21Train, Tox21Test and Tox21Score subsets, we observed that while internally valid models could be trained using cross-validation on Tox21Train, predictions on the external Tox21Score data resulted in higher error rates than expected. To improve the prediction on the external sets, a strategy exchanging the calibration set with more recent data, such as Tox21Test, has successfully been introduced. We conclude that conformal prediction can be used to diagnose data drifts and other issues related to model calibration. The proposed improvement strategy—exchanging the calibration data only—is convenient as it does not require retraining of the underlying model.
| Type: | Article |
|---|---|
| Title: | Assessing the calibration in toxicological in vitro models with conformal prediction |
| Open access status: | An open access version is available from UCL Discovery |
| DOI: | 10.1186/s13321-021-00511-5 |
| Publisher version: | https://doi.org/10.1186/s13321-021-00511-5 |
| Language: | English |
| Additional information: | This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data. |
| Keywords: | Toxicity prediction, Conformal prediction, Data drifts, Applicability domain, Calibration plots, Tox21 datasets |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Brain Sciences UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Brain Sciences > UCL Queen Square Institute of Neurology UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Brain Sciences > UCL Queen Square Institute of Neurology > Department of Neuromuscular Diseases |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10128642 |
Archive Staff Only
 |
View Item |
