Kusa, Wojciech;
Lipani, Aldo;
Knoth, Petr;
Hanbury, Allan;
(2023)
VoMBaT: A Tool for Visualising Evaluation Measure Behaviour in High-Recall Search Tasks.
In: Chen, Hsin-Hsi and Duh, Wei-Jou (Edward) and Huang, Hen-Hsen and Kato, Makoto P and Mothe, Josiane and Poblete, Barbara, (eds.)
SIGIR '23: Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval.
Association for Computing Machinery (ACM): New York, NY, USA.
Preview |
Text
3539618.3591802.pdf - Published Version Download (1MB) | Preview |
Abstract
The objective of High-Recall Information Retrieval (HRIR) is to retrieve as many relevant documents as possible for a given search topic. One approach to HRIR is Technology-Assisted Review (TAR), which uses information retrieval and machine learning techniques to aid the review of large document collections. TAR systems are commonly used in legal eDiscovery and systematic literature reviews. Successful TAR systems are able to find the majority of relevant documents using the least number of assessments. Commonly used retrospective evaluation assumes that the system achieves a specific, fixed recall level first, and then measures the precision or work saved (e.g., precision at r% recall). This approach can cause problems related to understanding the behaviour of evaluation measures in a fixed recall setting. It is also problematic when estimating time and money savings during technology-assisted reviews. This paper presents a new visual analytics tool to explore the dynamics of evaluation measures depending on recall level. We implemented 18 evaluation measures based on the confusion matrix terms, both from general IR tasks and specific to TAR. The tool allows for a comparison of the behaviour of these measures in a fixed recall evaluation setting. It can also simulate savings in time and money and a count of manual vs automatic assessments for different datasets depending on the model quality. The tool is open-source, and the demo is available under the following URL: https://vombat.streamlit.app.
| Type: | Proceedings paper |
|---|---|
| Title: | VoMBaT: A Tool for Visualising Evaluation Measure Behaviour in High-Recall Search Tasks |
| Event: | 46th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '23) |
| ISBN-13: | 9781450394086 |
| Open access status: | An open access version is available from UCL Discovery |
| DOI: | 10.1145/3539618.3591802 |
| Publisher version: | http://doi.org/10.1145/3539618.3591802 |
| Language: | English |
| Additional information: | This is an Open Access paper published under a Creative Commons Attribution 4.0 International (CC BY 4.0) Licence (https://creativecommons.org/licenses/by/4.0/). |
| Keywords: | visual analytics, technology-assisted reviews, systematic reviews, citation screening, evaluation, evaluation measures, TNR |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > UCL BEAMS UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Civil, Environ and Geomatic Eng |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10174082 |
Archive Staff Only
 |
View Item |
