UCL Discovery
UCL home » Library Services » Electronic resources » UCL Discovery

The role of visual cues in speech comprehension: evidence from brain and behaviour

Krason, Anna Malgorzata; (2023) The role of visual cues in speech comprehension: evidence from brain and behaviour. Doctoral thesis (Ph.D), UCL (University College London). Green open access

[thumbnail of PhDThesis_Krason.pdf]
Preview
Text
PhDThesis_Krason.pdf - Other

Download (11MB) | Preview

Abstract

Face-to-face communication is multimodal. It comprises a plethora of linguistic and non-linguistic cues, such as gestures, face and body movements, eye gaze, and prosody, that modulate language processing. However, these cues have been primarily studied in isolation and it remains unclear whether they interact and, if so, how they affect speech comprehension. This thesis aims to assess the role and the relationship of two multimodal cues: co-speech gestures and mouth movements. The claims presented in this thesis are supported by findings from studies on neurotypical adults and people with post-stroke aphasia. First, I developed a novel quantification of how informative mouth and facial movements are and provided informativeness norms for more than 2,000 English words. Such quantification allows studying the effects of speech and gestures while controlling for the invariably present mouth movements. Second, I investigated the impact of iconic gestures (i.e., gestures referring to features and properties of objects or actions in an imagistic way) and mouth informativeness on word comprehension under clear and challenging listening conditions. I found that both cues contribute to speech comprehension and that listeners dynamically process visual cues depending on their informativeness and listening conditions. Next, I showed that gestures and mouth movements also benefit discourse comprehension, but they do so differently: Listeners always take gestures into account, whereas mouth movements are only beneficial in challenging listening conditions. Finally, I investigated the impact of mouth movements on speech comprehension in aphasia. I showed that people with aphasia benefit less from mouth movements than neurotypical adults and that this effect is contingent upon lesion locations. This thesis brings together research on gestures and mouth movements and demonstrates that visual cues benefit speech comprehension interactively and dynamically depending on cue informativeness, listening conditions, and neuroanatomical profiles of people with aphasia.

Type: Thesis (Doctoral)
Qualification: Ph.D
Title: The role of visual cues in speech comprehension: evidence from brain and behaviour
Open access status: An open access version is available from UCL Discovery
Language: English
Additional information: Copyright © The Author 2023. Original content in this thesis is licensed under the terms of the Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0) Licence (https://creativecommons.org/licenses/by-nc/4.0/). Any third-party copyright material present remains the property of its respective owner(s) and is licensed under its existing terms. Access may initially be restricted at the author’s request.
UCL classification: UCL
UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences
UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Brain Sciences
UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Brain Sciences > Div of Psychology and Lang Sciences
URI: https://discovery.ucl.ac.uk/id/eprint/10166528
Downloads since deposit
21Downloads
Download activity - last month
Download activity - last 12 months
Downloads by country - last 12 months

Archive Staff Only

View Item View Item