Morger, A;
Garcia de Lomana, M;
Norinder, U;
Svensson, F;
Kirchmair, J;
Mathea, M;
Volkamer, A;
(2022)
Studying and mitigating the effects of data drifts on ML model performance at the example of chemical toxicity data.
Scientific Reports
, 12
, Article 7244. 10.1038/s41598-022-09309-3.
Preview |
Text
s41598-022-09309-3.pdf - Published Version Download (2MB) | Preview |
Abstract
Machine learning models are widely applied to predict molecular properties or the biological activity of small molecules on a specific protein. Models can be integrated in a conformal prediction (CP) framework which adds a calibration step to estimate the confidence of the predictions. CP models present the advantage of ensuring a predefined error rate under the assumption that test and calibration set are exchangeable. In cases where the test data have drifted away from the descriptor space of the training data, or where assay setups have changed, this assumption might not be fulfilled and the models are not guaranteed to be valid. In this study, the performance of internally valid CP models when applied to either newer time-split data or to external data was evaluated. In detail, temporal data drifts were analysed based on twelve datasets from the ChEMBL database. In addition, discrepancies between models trained on publicly-available data and applied to proprietary data for the liver toxicity and MNT in vivo endpoints were investigated. In most cases, a drastic decrease in the validity of the models was observed when applied to the time-split or external (holdout) test sets. To overcome the decrease in model validity, a strategy for updating the calibration set with data more similar to the holdout set was investigated. Updating the calibration set generally improved the validity, restoring it completely to its expected value in many cases. The restored validity is the first requisite for applying the CP models with confidence. However, the increased validity comes at the cost of a decrease in model efficiency, as more predictions are identified as inconclusive. This study presents a strategy to recalibrate CP models to mitigate the effects of data drifts. Updating the calibration sets without having to retrain the model has proven to be a useful approach to restore the validity of most models.
| Type: | Article |
|---|---|
| Title: | Studying and mitigating the effects of data drifts on ML model performance at the example of chemical toxicity data |
| Location: | England |
| Open access status: | An open access version is available from UCL Discovery |
| DOI: | 10.1038/s41598-022-09309-3 |
| Publisher version: | https://doi.org/10.1038/s41598-022-09309-3 |
| Language: | English |
| Additional information: | This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. |
| Keywords: | Computational biology and bioinformatics, Drug discovery |
| UCL classification: | UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Brain Sciences UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Brain Sciences > UCL Queen Square Institute of Neurology > Department of Neuromuscular Diseases UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences UCL UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Brain Sciences > UCL Queen Square Institute of Neurology |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10149273 |
Archive Staff Only
 |
View Item |
