Liu, Linqing;
(2024)
Towards Generalized Open Domain Question Answering Systems.
Doctoral thesis (Ph.D), UCL (University College London).
Preview |
Text
Liu_10192104_thesis_revised.pdf Download (4MB) | Preview |
Abstract
Generalization remains a paramount yet unresolved challenge for open-domain question answering (ODQA) systems, impeding their capacity to adeptly handle novel queries and responses beyond the confines of their training data. This thesis conducts a comprehensive exploration of ODQA generalization. We commence with a meticulous investigation into the underlying challenges. Drawing upon studies on systematic generalization, we introduce and annotate questions according to three categories that measure different levels and kinds of generalization: training set overlap, compositional generalization and novel-entity generalization. When evaluating six popular parametric and non-parametric models, we find non-parametric models demonstrate proficiency with novel entities but encounter difficulties with compositional generalization. Noteworthy correlations emerge, such as a positive association between question pattern frequency and test accuracy, juxtaposed with a strong negative correlation between entity frequency and test accuracy, attributable to closely related distractors. Factors influencing generalization include cascading errors originating from the retrieval component, question pattern frequency, and entity prevalence. Building on these insights, the focus pivots towards the enhancement of passage retrieval. We propose a novel contextual clue sampling strategy using language models to address the vocabulary mismatch challenge in lexical retrieval for ODQA. This two-step method, comprising filtering and fusion, generates a diverse set of query expansion terms, yielding retrieval accuracy similar to dense methods while notably reducing the index size. The subsequent phase concentrates on refining reader models in ODQA through flat minima optimization techniques, incorporating Stochastic Weight Averaging (SWA) and Sharpness Aware Minimization (SAM). Rigorous benchmarking under- scores the impact of dataset characteristics and model architecture on optimizer effectiveness, with SAM particularly excelling in Natural Language Processing tasks. The combination of SWA and SAM yields additional gains, underscoring the pivotal role of flatter minimizers in fostering enhanced generalization for reader models in ODQA.
| Type: | Thesis (Doctoral) |
|---|---|
| Qualification: | Ph.D |
| Title: | Towards Generalized Open Domain Question Answering Systems |
| Open access status: | An open access version is available from UCL Discovery |
| Language: | English |
| Additional information: | Copyright © The Author 2024. Original content in this thesis is licensed under the terms of the Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0) Licence (https://creativecommons.org/licenses/by-nc/4.0/). Any third-party copyright material present remains the property of its respective owner(s) and is licensed under its existing terms. Access may initially be restricted at the author’s request. |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > UCL BEAMS UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10192104 |
Archive Staff Only
 |
View Item |
