eprintid: 10158158 rev_number: 7 eprint_status: archive userid: 699 dir: disk0/10/15/81/58 datestamp: 2022-10-28 13:38:39 lastmod: 2022-10-28 13:38:39 status_changed: 2022-10-28 13:38:39 type: article metadata_visibility: show sword_depositor: 699 creators_name: Kummervold, Per E creators_name: Martin, Sam creators_name: Dada, Sara creators_name: Kilich, Eliz creators_name: Denny, Chermain creators_name: Paterson, Pauline creators_name: Larson, Heidi J title: Categorizing Vaccine Confidence With a Transformer-Based Machine Learning Model: Analysis of Nuances of Vaccine Sentiment in Twitter Discourse ispublished: pub divisions: C10 divisions: G88 divisions: B02 divisions: UCL divisions: D16 keywords: computer science, information technology, public health, health humanities, vaccines, machine learning note: Copyright © Per E Kummervold, Sam Martin, Sara Dada, Eliz Kilich, Chermain Denny, Pauline Paterson, Heidi J Larson. Originally published in JMIR Medical Informatics (https://medinform.jmir.org), 08.10.2021. This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Medical Informatics, is properly cited. The complete bibliographic information, a link to the original publication on https://medinform.jmir.org/, as well as this copyright and license information must be included. abstract: Background: Social media has become an established platform for individuals to discuss and debate various subjects, including vaccination. With growing conversations on the web and less than desired maternal vaccination uptake rates, these conversations could provide useful insights to inform future interventions. However, owing to the volume of web-based posts, manual annotation and analysis are difficult and time consuming. Automated processes for this type of analysis, such as natural language processing, have faced challenges in extracting complex stances such as attitudes toward vaccination from large amounts of text. / Objective: The aim of this study is to build upon recent advances in transposer-based machine learning methods and test whether transformer-based machine learning could be used as a tool to assess the stance expressed in social media posts toward vaccination during pregnancy. / Methods: A total of 16,604 tweets posted between November 1, 2018, and April 30, 2019, were selected using keyword searches related to maternal vaccination. After excluding irrelevant tweets, the remaining tweets were coded by 3 individual researchers into the categories Promotional, Discouraging, Ambiguous, and Neutral or No Stance. After creating a final data set of 2722 unique tweets, multiple machine learning techniques were trained on a part of this data set and then tested and compared with the human annotators. / Results: We found the accuracy of the machine learning techniques to be 81.8% (F score=0.78) compared with the agreed score among the 3 annotators. For comparison, the accuracies of the individual annotators compared with the final score were 83.3%, 77.9%, and 77.5%. / Conclusions: This study demonstrates that we are able to achieve close to the same accuracy in categorizing tweets using our machine learning models as could be expected from a single human coder. The potential to use this automated process, which is reliable and accurate, could free valuable time and resources for conducting this analysis, in addition to informing potentially effective and necessary interventions. date: 2021-10 date_type: published publisher: JMIR PUBLICATIONS, INC official_url: https://doi.org/10.2196/29584 oa_status: green full_text_type: pub language: eng primo: open primo_central: open_green verified: verified_manual elements_id: 1984179 doi: 10.2196/29584 medium: Electronic pii: v9i10e29584 lyricists_name: Martin, Samantha lyricists_id: SMARB73 actors_name: Flynn, Bernadette actors_id: BFFLY94 actors_role: owner funding_acknowledgements: [European Commission]; 797876 [funding scheme MSCA-IF-EF-ST]; [GlaxoSmithKline]; [Cloud TPUs (Tensor Processing Units) from Google's TPU Research Cloud] full_text_status: public publication: JMIR Medical Informatics volume: 9 number: 10 article_number: e29584 pages: 10 event_location: Canada issn: 2291-9694 citation: Kummervold, Per E; Martin, Sam; Dada, Sara; Kilich, Eliz; Denny, Chermain; Paterson, Pauline; Larson, Heidi J; (2021) Categorizing Vaccine Confidence With a Transformer-Based Machine Learning Model: Analysis of Nuances of Vaccine Sentiment in Twitter Discourse. JMIR Medical Informatics , 9 (10) , Article e29584. 10.2196/29584 <https://doi.org/10.2196/29584>. Green open access document_url: https://discovery.ucl.ac.uk/id/eprint/10158158/1/PDF%20%282%29.pdf