Scholes, C;
Skipper, JI;
Johnston, A;
(2020)
The interrelationship between the face and vocal tract configuration during audiovisual speech.
Proceedings of the National Academy of Sciences of the United States of America
10.1073/pnas.2006192117.
(In press).
Preview |
Text
2006192117.full.pdf - Published Version Download (1MB) | Preview |
Abstract
It is well established that speech perception is improved when we are able to see the speaker talking along with hearing their voice, especially when the speech is noisy. While we have a good understanding of where speech integration occurs in the brain, it is unclear how visual and auditory cues are combined to improve speech perception. One suggestion is that integration can occur as both visual and auditory cues arise from a common generator: the vocal tract. Here, we investigate whether facial and vocal tract movements are linked during speech production by comparing videos of the face and fast magnetic resonance (MR) image sequences of the vocal tract. The joint variation in the face and vocal tract was extracted using an application of principal components analysis (PCA), and we demonstrate that MR image sequences can be reconstructed with high fidelity using only the facial video and PCA. Reconstruction fidelity was significantly higher when images from the two sequences corresponded in time, and including implicit temporal information by combining contiguous frames also led to a significant increase in fidelity. A "Bubbles" technique was used to identify which areas of the face were important for recovering information about the vocal tract, and vice versa, on a frame-by-frame basis. Our data reveal that there is sufficient information in the face to recover vocal tract shape during speech. In addition, the facial and vocal tract regions that are important for reconstruction are those that are used to generate the acoustic speech signal.
| Type: | Article |
|---|---|
| Title: | The interrelationship between the face and vocal tract configuration during audiovisual speech |
| Location: | United States |
| Open access status: | An open access version is available from UCL Discovery |
| DOI: | 10.1073/pnas.2006192117 |
| Publisher version: | https://doi.org/10.1073/pnas.2006192117 |
| Language: | English |
| Additional information: | © 2020 the Author(s). Published by PNAS. This open access article is distributed under Creative Commons Attribution License 4.0 (http://creativecommons.org/licenses/by/4.0/). |
| Keywords: | PCA, audiovisual, speech |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Brain Sciences UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Brain Sciences > Div of Psychology and Lang Sciences UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Brain Sciences > Div of Psychology and Lang Sciences > Experimental Psychology |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10117272 |
Archive Staff Only
 |
View Item |
