Wiedmann, Lea;
Blumenau, Jack;
Carroll, Orlagh;
Cairns, John;
(2024)
Using automated text classification to explore uncertainty in NICE appraisals for drugs for rare diseases.
International Journal of Technology Assessment in Health Care
10.1017/s0266462323002805.
(In press).
Preview |
Text
Using automated text classification.pdf Download (470kB) | Preview |
Abstract
Objective: This study examined the application, feasibility, and validity of supervised learning models for text classification in appraisals for rare disease treatments (RDTs) in relation to uncertainty, and analyzed differences between appraisals based on the classification results. Methods: We analyzed appraisals for RDTs (n = 94) published by the National Institute for Health and Care Excellence (NICE) between January 2011 and May 2023. We used Naïve Bayes, Lasso, and Support Vector Machine models in a binary text classification task (classifying paragraphs as either referencing uncertainty in the evidence base or not). To illustrate the results, we tested hypotheses in relation to the appraisal guidance, advanced therapy medicinal product (ATMP) status, disease area, and age group. Results: The best performing (Lasso) model achieved 83.6 percent classification accuracy (sensitivity = 74.4 percent, specificity = 92.6 percent). Paragraphs classified as referencing uncertainty were significantly more likely to arise in highly specialized technology (HST) appraisals compared to appraisals from the technology appraisal (TA) guidance (adjusted odds ratio = 1.44, 95 percent CI 1.09, 1.90, p = 0.004). There was no significant association between paragraphs classified as referencing uncertainty and appraisals for ATMPs, non-oncology RDTs, and RDTs indicated for children only or adults and children. These results were robust to the threshold value used for classifying paragraphs but were sensitive to the choice of classification model. Conclusion: Using supervised learning models for text classification in NICE appraisals for RDTs is feasible, but the results of downstream analyses may be sensitive to the choice of classification model.
| Type: | Article |
|---|---|
| Title: | Using automated text classification to explore uncertainty in NICE appraisals for drugs for rare diseases |
| Open access status: | An open access version is available from UCL Discovery |
| DOI: | 10.1017/s0266462323002805 |
| Publisher version: | http://dx.doi.org/10.1017/s0266462323002805 |
| Language: | English |
| Additional information: | Copyright © The Author(s), 2024. This is an Open Access article, distributed under the terms of the Creative Commons Attribution licence (http://creativecommons.org/licenses/by/4.0), which permits unrestricted re-use, distribution and reproduction, provided the original article is properly cited. |
| Keywords: | health technology assessment; supervised learning; uncertainty; rare diseases; National Institute for Health and Care Excellence |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > UCL SLASH UCL > Provost and Vice Provost Offices > UCL SLASH > Faculty of S&HS UCL > Provost and Vice Provost Offices > UCL SLASH > Faculty of S&HS > Dept of Political Science |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10185212 |
Archive Staff Only
 |
View Item |
