Ge, Yao;
Tang, Chong;
Li, Haobo;
Chen, Zikang;
Wang, Jingyan;
Li, Wenda;
Cooper, Jonathan;
... Abbasi, Qammer; + view all
(2023)
A comprehensive multimodal dataset for contactless lip reading and acoustic analysis.
Scientific Data
, 10
, Article 895. 10.1038/s41597-023-02793-w.
Preview |
Text
s41597-023-02793-w.pdf Download (3MB) | Preview |
Abstract
Small-scale motion detection using non-invasive remote sensing techniques has recently garnered significant interest in the field of speech recognition. Our dataset paper aims to facilitate the enhancement and restoration of speech information from diverse data sources for speakers. In this paper, we introduce a novel multimodal dataset based on Radio Frequency, visual, text, audio, laser and lip landmark information, also called RVTALL. Specifically, the dataset consists of 7.5 GHz Channel Impulse Response (CIR) data from ultra-wideband (UWB) radars, 77 GHz frequency modulated continuous wave (FMCW) data from millimeter wave (mmWave) radar, visual and audio information, lip landmarks and laser data, offering a unique multimodal approach to speech recognition research. Meanwhile, a depth camera is adopted to record the landmarks of the subject’s lip and voice. Approximately 400 minutes of annotated speech profiles are provided, which are collected from 20 participants speaking 5 vowels, 15 words, and 16 sentences. The dataset has been validated and has potential for the investigation of lip reading and multimodal speech recognition.
| Type: | Article |
|---|---|
| Title: | A comprehensive multimodal dataset for contactless lip reading and acoustic analysis |
| Open access status: | An open access version is available from UCL Discovery |
| DOI: | 10.1038/s41597-023-02793-w |
| Publisher version: | https://doi.org/10.1038/s41597-023-02793-w |
| Language: | English |
| Additional information: | Open Access: This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > UCL BEAMS UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Security and Crime Science |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10181045 |
Archive Staff Only
 |
View Item |
