UCL Discovery
UCL home » Library Services » Electronic resources » UCL Discovery

Why rankings of biomedical image analysis competitions should be interpreted with care

Maier-Hein, L; Eisenmann, M; Reinke, A; Onogur, S; Stankovic, M; Scholz, P; Arbel, T; ... Kopp-Schneider, A; + view all (2018) Why rankings of biomedical image analysis competitions should be interpreted with care. Nature Communications , 9 , Article 5217. 10.1038/s41467-018-07619-7. Green open access

[thumbnail of s41467-018-07619-7.pdf]
Preview
Text
s41467-018-07619-7.pdf - Published Version

Download (833kB) | Preview

Abstract

International challenges have become the standard for validation of biomedical image analysis methods. Given their scientific impact, it is surprising that a critical analysis of common practices related to the organization of challenges has not yet been performed. In this paper, we present a comprehensive analysis of biomedical image analysis challenges conducted up to now. We demonstrate the importance of challenges and show that the lack of quality control has critical consequences. First, reproducibility and interpretation of the results is often hampered as only a fraction of relevant information is typically provided. Second, the rank of an algorithm is generally not robust to a number of variables such as the test data used for validation, the ranking scheme applied and the observers that make the reference annotations. To overcome these problems, we recommend best practice guidelines and define open research questions to be addressed in the future.

Type: Article
Title: Why rankings of biomedical image analysis competitions should be interpreted with care
Open access status: An open access version is available from UCL Discovery
DOI: 10.1038/s41467-018-07619-7
Publisher version: http://doi.org/10.1038/s41467-018-07619-7
Language: English
Additional information: Copyright © The Author(s) 2018. Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/ licenses/by/4.0/.
UCL classification: UCL
UCL > Provost and Vice Provost Offices > UCL BEAMS
UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science
UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science
URI: https://discovery.ucl.ac.uk/id/eprint/10065068
Downloads since deposit
59Downloads
Download activity - last month
Download activity - last 12 months
Downloads by country - last 12 months

Archive Staff Only

View Item View Item