UCL Discovery
UCL home » Library Services » Electronic resources » UCL Discovery

Using automated text classification to explore uncertainty in NICE appraisals for drugs for rare diseases

Wiedmann, Lea; Blumenau, Jack; Carroll, Orlagh; Cairns, John; (2024) Using automated text classification to explore uncertainty in NICE appraisals for drugs for rare diseases. International Journal of Technology Assessment in Health Care 10.1017/s0266462323002805. (In press). Green open access

[thumbnail of Using automated text classification.pdf]
Preview
Text
Using automated text classification.pdf

Download (470kB) | Preview

Abstract

Objective: This study examined the application, feasibility, and validity of supervised learning models for text classification in appraisals for rare disease treatments (RDTs) in relation to uncertainty, and analyzed differences between appraisals based on the classification results. Methods: We analyzed appraisals for RDTs (n = 94) published by the National Institute for Health and Care Excellence (NICE) between January 2011 and May 2023. We used Naïve Bayes, Lasso, and Support Vector Machine models in a binary text classification task (classifying paragraphs as either referencing uncertainty in the evidence base or not). To illustrate the results, we tested hypotheses in relation to the appraisal guidance, advanced therapy medicinal product (ATMP) status, disease area, and age group. Results: The best performing (Lasso) model achieved 83.6 percent classification accuracy (sensitivity = 74.4 percent, specificity = 92.6 percent). Paragraphs classified as referencing uncertainty were significantly more likely to arise in highly specialized technology (HST) appraisals compared to appraisals from the technology appraisal (TA) guidance (adjusted odds ratio = 1.44, 95 percent CI 1.09, 1.90, p = 0.004). There was no significant association between paragraphs classified as referencing uncertainty and appraisals for ATMPs, non-oncology RDTs, and RDTs indicated for children only or adults and children. These results were robust to the threshold value used for classifying paragraphs but were sensitive to the choice of classification model. Conclusion: Using supervised learning models for text classification in NICE appraisals for RDTs is feasible, but the results of downstream analyses may be sensitive to the choice of classification model.

Type: Article
Title: Using automated text classification to explore uncertainty in NICE appraisals for drugs for rare diseases
Open access status: An open access version is available from UCL Discovery
DOI: 10.1017/s0266462323002805
Publisher version: http://dx.doi.org/10.1017/s0266462323002805
Language: English
Additional information: Copyright © The Author(s), 2024. This is an Open Access article, distributed under the terms of the Creative Commons Attribution licence (http://creativecommons.org/licenses/by/4.0), which permits unrestricted re-use, distribution and reproduction, provided the original article is properly cited.
Keywords: health technology assessment; supervised learning; uncertainty; rare diseases; National Institute for Health and Care Excellence
UCL classification: UCL
UCL > Provost and Vice Provost Offices > UCL SLASH
UCL > Provost and Vice Provost Offices > UCL SLASH > Faculty of S&HS
UCL > Provost and Vice Provost Offices > UCL SLASH > Faculty of S&HS > Dept of Political Science
URI: https://discovery.ucl.ac.uk/id/eprint/10185212
Downloads since deposit
3Downloads
Download activity - last month
Download activity - last 12 months
Downloads by country - last 12 months

Archive Staff Only

View Item View Item