UCL Discovery
UCL home » Library Services » Electronic resources » UCL Discovery

More than words: word predictability, prosody, gesture and mouth movements in natural language comprehension

Zhang, Y; Frassinelli, D; Tuomainen, J; Skipper, JI; Vigliocco, G; (2021) More than words: word predictability, prosody, gesture and mouth movements in natural language comprehension. Proceedings of the Royal Society B: Biological Sciences , 288 (1955) , Article 20210500. 10.1098/rspb.2021.0500. Green open access

[thumbnail of rspb.2021.0500.pdf]
Preview
Text
rspb.2021.0500.pdf - Published version

Download (1MB) | Preview

Abstract

The ecology of human language is face-to-face interaction, comprising cues such as prosody, co-speech gestures and mouth movements. Yet, the multimodal context is usually stripped away in experiments as dominant paradigms focus on linguistic processing only. In two studies we presented video-clips of an actress producing naturalistic passages to participants while recording their electroencephalogram. We quantified multimodal cues (prosody, gestures, mouth movements) and measured their effect on a well-established electroencephalographic marker of processing load in comprehension (N400). We found that brain responses to words were affected by informativeness of co-occurring multimodal cues, indicating that comprehension relies on linguistic and non-linguistic cues. Moreover, they were affected by interactions between the multimodal cues, indicating that the impact of each cue dynamically changes based on the informativeness of other cues. Thus, results show that multimodal cues are integral to comprehension, hence, our theories must move beyond the limited focus on speech and linguistic processing.

Type: Article
Title: More than words: word predictability, prosody, gesture and mouth movements in natural language comprehension
Location: England
Open access status: An open access version is available from UCL Discovery
DOI: 10.1098/rspb.2021.0500
Publisher version: https://doi.org/10.1098/rspb.2021.0500
Language: English
Additional information: Copyright © 2021 The Authors. Published by the Royal Society under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, provided the original author and source are credited.
Keywords: N400, comprehensions, gesture, mouth, prosody, surprisal
UCL classification: UCL
UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences
UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Brain Sciences
UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Brain Sciences > Div of Psychology and Lang Sciences
UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Brain Sciences > Div of Psychology and Lang Sciences > Experimental Psychology
UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Brain Sciences > Div of Psychology and Lang Sciences > Speech, Hearing and Phonetic Sciences
URI: https://discovery.ucl.ac.uk/id/eprint/10131972
Downloads since deposit
17Downloads
Download activity - last month
Download activity - last 12 months
Downloads by country - last 12 months

Archive Staff Only

View Item View Item