Automated Lip Reading to Predict Visemes using Multimodal Convolutional Neural Network with Audio-Visual Features

Khalid Mahboob; Umm-e-Laila; Sana Alam; Muhammad Abbas; Muhammad Asghar Khan; Sidra Fatima

Author(s)

Khalid Mahboob College of Computer Science and Information Systems, Institute of Business Management, Karachi, Pakistan

Umm-e-Laila College of Computer Science and Information Systems, Institute of Business Management, Karachi, Pakistan

Sana Alam College of Computer Science and Information Systems, Institute of Business Management, Karachi, Pakistan

Muhammad Abbas College of Computer Science and Information Systems, Institute of Business Management, Karachi, Pakistan

Muhammad Asghar Khan College of Computer Science and Information Systems, Institute of Business Management, Karachi, Pakistan

Sidra Fatima Computer Engineering Department, Sir Syed University of Engineering & Technology, Karachi Country, Pakistan

Abstract

The process of interpreting sentences based on the movements of a speaker's lips is referred to as lip reading. Traditionally, this task has been approached in two stages using conventional methods: first, by generating or learning audio-visual features, and second, by making predictions. While contemporary deep lip reading techniques benefit from end-to-end trainable datasets, much of the existing research on these models tends to concentrate solely on word classification rather than predicting sequences at the sentence level. Long sentences may be lip-read by humans, as studies have shown. This study emphasizes the value of temporal considerations by highlighting the components that are important for capturing temporal context in instances when communication channels are unclear. In the paper, a lipreading system for viseme prediction is shown. The system uses a Convolutional Neural Network (CNN) with a recurrent network, spatiotemporal convolutions, and the connectionist temporal classification loss. A variablelength series of video frames is efficiently mapped to text using an end-to-end training procedure. Both visual and auditory qualities are evaluated using the CNN architecture. The CNN model outperforms trained human lip readers and achieves accuracies of 72.8% CER and 80.8% WER (unseen speakers with audio), whereas 46.2% CER and 56.6% WER (unseen speakers without audio), which are reasonable accuracies on the GRID corpus by splitting test at the level of the sentences.

Publication Details

Page(s) 38-50

DOI DOI not available

Published Journal: Journal of Information & Communication Technology (JICT), Volume: 17, Issue: 2, Year: 2023

Request the full text from PASTIC

Keywords

Accuracy, Convolutional Neural Network, Model, lip, reading, visemes

References

Li Z,Liu F,Yang W,Peng S,Zhou J. .2021 .A Survey of Convolutional Neural Networks: Analysis, Applications, and Prospects. IEEE Transactions on Neural Networks and LearningSystems, 3084827 : 1-21.

Pujari S,Sneha S,Vinusha R,Bhuvaneshwari P,Rashaswini C. .2021 .A Survey on Deep Learning based LipReading Techniques. Proceedings of the Third International Conference on Intelligent Communication Technologies and Virtual Mobile Networks (ICICV, : 1286-93.

Oghbaie M,Sabaghi A,Hashemifard K,Akbari M.,Advances M. .2021 .. Computer Vision, : 1-31.

.2020 .. , 9(2) : 85-112.

Fenghour S,Chen D,Guo K,Li B,Xiao P. Deep LearningBased Automated B .2021 .. IEEE Access, 9 : 121184-205.

Nandini M.S.,Bhajantri N.U. . .A comprehension of contemporary efort for tracking of lip.. , : .

.2020 .. International Journal of Bioinformatics Applications, 16(1) : 85.

McMurray B,Jongman A .2011 .What information is necessary for speech categorization? Harnessing variability in the speech signal by integrating cues computed relative to expectations. Psychol Rev, 118(2) : 219-46.

Fenghour S,Chen D,Guo K,Li B,Xiao P. .2021 .An efective conversion of visemes to words for high-performance automatic lipreading. Sensors, 21(23) : .

Lu Y,Li H Y .2019 .Automatic lip-reading system based on deep convolutional neural network and attention-based long short-term memory. Applied Sciences (Switzerland), 9(8) : .

.2021 .deep neural networks. -3027-905-916., 3027 : 905-16.

.2021 .Detection with Deep Learning Convolutional Neural Networks Using Video-Based Features. , 1489(4) : 4.

Miled M.,Messaoud M.A.B.,Bouzid A. .2022 .Lip reading of words with lip segmentation and deep learning. Multimedia Tools and Applications, : .

SMART .2019 .Reading Using Deep Learning. Proceedings of the 2019 8th International Conference on System Modelingand Advancement in Research Trends, 9117224 : 240-3.

Jeon S,Elsharkawy A,Kim MS .2022 .Lipreading architecture based on multiple convolutional neural networks for sentence-level visual speech recognition. Sensors, 22(1) : .

Ma P,Wang Y,Shen J,Petridis S,Pantic M,WACV M .2021 .Lip-reading with densely connected temporal convolutional networks. Proceedings - 2021 IEEE Winter Conference on Applications of Computer Vision, 00290 : 2856-65.

Nambeesan AS,Payyappilly C,Edwin JC,Alex P JJ,S. LIP P JJ .2021 .Reading Using Facial Feature Extraction and Deep Learning. International Journal of Innovative Science and Research Technology, 6(7) : 92-6.

Corrector Lip .1883 .1. Journal of Physics: Conference Series, : 1742-6596.

Cues Visual .2020 .. IEEE Access, 3040906 : 215516-30.

.2022 .provided and atomic layers provided. 2022 2nd International Conference on Consumer Electronics and Computer Engineering, 9712718 : 676-9.

Analysis Multispectral Image .2020 .. , 10 : .

ICSIMA .2019 .Coding Framework for Modeling of Pooling Operation of Convolution Neural Networks. 2019 IEEE 6th International Conference on Smart Instrumentation, Measurement and Application, 9057334 : 27-9.

Jameel SM,Hashmani MA,Alhussain H,Rehman M,Budiman A .2020 .An optimized deep convolutional neural network architecture for concept drifted image classification. Advances in Intelligent Systems and Computing, 932 : 70.

Zhu Q,Zu X .2022 .Fully Convolutional Neural Network Structure and Its Loss Function for Image Classification. IEEE Access, 3163849 : 35541-9.

Ting J,Song C,Huang H,Tian T. . .A Comprehensive Dataset for Machine-Learning-based. , : .

Lip-Reading Algorithm .2022 .. , 01 : 1444-9.

Huang H,Song C,Ting J,Tian T,Hong C,Di Z . .. A Novel Machine Lip Reading Model., : .

.2022 .. Procedia Computer Science, 01 : 1432-7.

Johnston B,Chazal P de. B .2018 .A review of image-based automatic facial landmark identification techniques. Eurasip Journal on Image and Video Processing, : .

Lalitha SD,Thyagharajan KK .2016 .A study on lip localization techniques used for lip reading from a video. International Journal of Applied Engineering Research, 11(1) : 611-5.

Sukritha SN,Mohan M. .2021 .Analysis on Lip Reading Techniques and Image Concatenation Concepts. Control and Information Sciences, Proceedings, 9484920 : .

Chen T,Rao RR . .Audio-Visual Integration in Multimodal Communication. , : .

Neeraja K,Srinivas Rao K,Praneeth G,ICCES G .2021 .Deep Learning based Lip Movement Technique for Mute. Proceedings of the 6th International Conference on Communication and Electronics Systems, 9489122 : 1446-50.

Jeon S,Kim MS . .End-to-End Lip-Reading Open CloudBased Speech Architecture. , : .

.2022 .. , 22(8) : .

Joshi VS .2021 .FYEO : A Character Level Model for Lip Reading. , : .

ICSCC .2021 .Applications for a Smart World. , 9528104 : 257-62.

Sarhan AM,Elshennawy NM,Ibrahim DM,HLR-Net DM .2021 .A hybrid lip-reading model based on deep convolutional neural networks. Computers, Materials and Continua, 016509(2) : 1531-49.