Audio-Visual Emotion Recognition Using Multilevel Fusion

Muhammad Shoaib; Sana Ul Haq; Muhammad Saeed Shah; Imtiaz Rasool; Mohammad Omer Farooq

Author(s)

Muhammad Shoaib Department of Electronics, University of Peshawar,Peshawar 25120,Pakistan

Sana Ul Haq Department of Electronics, University of Peshawar,Peshawar 25120,Pakistan

Muhammad Saeed Shah Department of Electronics, University of Peshawar,Peshawar 25120,Pakistan

Imtiaz Rasool Department of Electronics, University of Peshawar,Peshawar 25120,Pakistan

Mohammad Omer Farooq Department of Electronics, University of Peshawar,Peshawar 25120,Pakistan

Abstract

In the affective computing domain, many researchers have worked on automatic human emotion recognition in recent years. Unimodal techniques, i.e., audio, visual or physiological signals, have been used in most emotion recognition research. The research indicates that one modality can trump the other when it comes to classification accuracy. Some emotions may have better classification accuracy in one modality, while others may be easily separated in the other. In the proposed research, emotion recognition is performed using both unimodal and bimodal techniques. Experiments were performed using six emotions of an audio-visual interactive emotional dyadic motion capture (IEMOCAP) database. The classification was performed using three different feature selection methods and seven various classification techniques. The recognition accuracy of 64.54% was obtained for the audio modality, and 96.77% for the visual modality using rotation forest classifier. For the bimodal approach, the best accuracy of 96.04% was obtained for feature-level fusion using the rotation forest classifier. The decision-level fusion resulted in the best performance of 97.60% for the product rule, while obtained an accuracy of 97.51% for the sum rule. The bimodal approach provided better results in comparison to unimodal approach, and the decision-level fusion provided better results.

Publication Details

Page(s) 39-51

DOI DOI not available

Published Journal: The Sciencetech, Volume: 5, Issue: 1, Year: 2024

Request the full text from PASTIC

Keywords

Sum Rule, Decision-Level Fusion, Product Rule, Classification, Emotion Recognition

References

Alluhaidan A. S.,Saidani O.,Jahangir R.,Nauman M. A.,Neffati O. S. .2023 .Speech Emotion Recognition through Hybrid Features and Convolutional Neural Network. Applied Sciences, 13(8) : 4750.

Araño K. A.,Gloor P.,Orsenigo C.,Vercellis C. .2021 .When old meets new: emotion recognition from speech signals. Cognitive Computation, 13 : 771-783.

Avots E.,Sapiński T.,Bachmann M.,Kamińska D. .2019 .Audiovisual emotion recognition in wild. Machine Vision and Applications, 30(5) : 975-985.

Azhagusundari B.,Thanamani A. S. .2013 .Feature selection based on information gain. International Journal of Innovative Technology and Exploring Engineering, 2(2) : 18-21.

Bhanusree Y.,Kumar S. S.,Rao A. K. .2023 .Time-Distributed Attention-Layered Convolution Neural Network with Ensemble Learning using Random Forest Classifier for Speech Emotion Recognition. Journal of Information and Communication Technology, 22(1) : 49-76.

Bota P.,Wang C.,Fred A.,Silva H. .2020 .Emotion assessment using feature fusion and decision fusion classification based on physiological data: Are we there yet?. Sensors, 20(17) : 4723.

Breiman L. .2001 .Random forests. Machine learning, 45 : 5-32.

Büchlmann P.,Yu B. .2002 .. The Annals of Statistics, 30(4) : 927-961.

Busso C.,Bulut M.,Lee C.-C.,Kazemzadeh A.,Mower E.,Kim S. .2008 .IEMOCAP: Interactive emotional dyadic motion capture database. Language resources and evaluation, 42 : 335-359.

Chalapathi M. V.,Kumar M. R.,Sharma N.,Shitharth S. .2022 .Ensemble Learning by High-Dimensional Acoustic Features for Emotion Recognition from Speech Audio Signal. Security and Communication Networks, : 1-10.

Chang C.-C.,Lin C.-J. .2011 .LIBSVM: a library for support vector machine. ACM transactions on intelligent systems and technology (TIST), 2(3) : 1-27.

Chen M.,He X.,Yang J.,Zhang H. .2018 .3-D convolutional recurrent neural networks with attention model for speech emotion recognition. IEEE Signal Processing Letters, 25(10) : 1440-1444.

Ekman P. .1993 .Facial expression. American Psychologist, 48(4) : 384-392.

Ekman P.,Friesen W. V. .1978 .Facial action coding system: a technique for the measurement of facial movement. Consulting Psychologists., : .

Eyben F.,Wollmer M.,Schuller B. .2009 .OpenEAR-Introducing the Munich Open-Source Emotion and Affect Recognition Toolkit. International Conference on Affective Computing and Intelligent Interaction and Workshops, : 1-6.

Gama J. .2004 .Functional trees. Machine learning, 55 : 219-250.

Hajarolasvadi N.,Demirel H. .2020 .Deep emotion recognition based on audio-visual correlation. IET Computer Vision, 14(7) : 517-527.

Hall M. A. .1999 .Correlation-based feature selection for machine learning. , : .

Haq S. U.,Asif M.,Ali A.,Jan T.,Ahmad N.,Khan Y. .2015 .Audio-visual emotion classification using filter and wrapper feature selection approaches. , 47(1) : 67-72.

Haq S.,,Jackson P. J. .2011 .Multimodal Emotion Recognition. IGI Global., : 398-423.

Jain A.,Nandakumar K.,Ross A. .2005 .Score normalization in multimodal biometric systems. Pattern Recognition, 38(12) : 2270-2285.

Jain Y. K.,Bhandare S. K. .2011 .Min max normalization based data perturbation method for privacy protection. International Journal of Computer & Communication Technology, 2(8) : 45-50.

Khalil R. A.,Jones E.,Babar M. I.,Jan T.,Zafar M. H.,Alhussain T. .2019 .Speech emotion recognition using deep learning techniques: A review. IEEE Access, 7 : 117327-117345.

Khare S. K.,Blanes-Vidal V.,Nadimi E. S.,Acharya U. R. .2024 .Emotion recognition and artificial intelligence: A systematic review (2014-2023) and research recommendations. Information Fusion, 102 : .

Liu S.,McGree J.,Ge Z.,Xie Y. .2016 .Computational and Statistical Methods for Analysing Big Data with Applications. , : .

Ma Y.,Hao Y.,Chen M.,Chen J.,Lu P.,Košir A. .2019 .Audiovisual emotion fusion (AVEF): A deep efficient weighted approach. Information Fusion, 46 : 184-192.

Mannepalli K.,Sastry P. N.,Suman M. .2022 .Emotion recognition in speech signals using optimization based multi-SVNN classifier. Journal of King, 34(2) : 384-397.

Mohan M.,Dhanalakshmi P.,Kumar R. S. .2023 .Speech Emotion Classification Using Ensemble Models with MFCC. Procedia Computer Science, 218 : 1857-1868.

Nie W.,Ren M.,Nie J.,,Zhao S. .2020 .C-GCN: Correlation based graph convolutional network for audio-video emotion recognition. IEEE Transactions on Multimedia, 23 : 3793-3804.

Rodriguez J. J.,Kuncheva L. I.,Alonso C. J. .2006 .Rotation Forest: A New Classifier Ensemble Method. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(10) : 1619-1630.

Samadiani N.,Huang G.,Cai B.,Luo W.,Chi W.,Xiang Y. .1863 .A review on automatic facial expression recognition systems assisted by multimodal sensor data. Sensors, 19(8) : .

Tan L.,Yu K.,Lin X.,Srivastava G.,Lin J. C.-W. .2021 .Speech emotion recognition enhanced traffic efficiency solution for autonomous vehicles in a 5G-enabled space--air-- ground integrated intelligent transportation system. IEEE Transactions on Intelligent Transportation Systems, 23(3) : 2830-2842.

The MathWorks Inc. .2024 .The MathWorks Inc. , : .

Thiam P.,Kestler H. A.,Schwenker F. .2020 .Two-stream attention network for pain recognition from video sequences. Sensors, 20(3) : 839.

Tzirakis P.,Trigeorgis G.,Nicolaou M. A.,Schuller B. W.,Zafeiriou S. .2017 .End-to-end multimodal emotion recognition using deep neural networks. IEEE Journal of selected topics in signal processing, 11(8) : 1301-1309.

Wang Y.,Guan L.,Venetsanopoulos A. N. .2012 .Kernel crossmodal factor analysis for information fusionwith application to bimodal emotion recognition. IEEE Transactions on Multimedia, 14(3) : 597-607.

Wang Y.,Wu J.,Hoashi K. .2019 .Multi-attention fusion network for video-based emotion recognition. International Conference on Multimodal Interaction., : .

Witten I. H.,Frank E.,Hall M. A.,Pal C. J. .2016 .Data Mining: Practical Machine Learning Tools and Techniques. , : .

Yuvaraj R.,Murugappan M.,Ibrahim N. M.,Sundaraj K.,Omar M. I.,Mohamad K. .2014 .Detection of emotions in Parkinson's disease using higher order spectral features from brain's electrical activity. Biomedical Signal Processing and Control, 14 : 108-116.

Zeng Z.,Tu J.,Pianfetti B. M.,Huang T. S. .2008 .-visual affective expression recognition through multistream fused HMM. IEEE Transactions on multimedia, 10(4) : 570-577.

Zhang S.,Li L.,Zhao Z. .2012 .Audio-visual emotion recognition based on facial expression and affective speech. International Conference on Multimedia and Signal Processing, : 46-52.

Zhang S.,Huang S.,Gao T.,Tian W.,Q. W. .2017 .Learning affective features with a hybrid deep model for audio--visual emotion recognition. IEEE Transactions on Circuits and Systems for Video Technology, 28(10) : 3030-3043.

Zhao G.,Barnard M.,Pietikainen M. .2009 .Lipreading with local spatiotemporal descriptors. IEEE Transactions on Multimedia, 11(7) : 1254-1265.

Zhao J.,Mao X.,Chen L. .2019 .Speech emotion recognition using deep 1D & 2D CNN LSTM networks. Biomedical signal processing and control, 47 : 312-323.

Zhou H.,Du J.,Wang Y.,Liu Q.,Lee Q.-F.,C.-H. Q.-F. .2021 .Information fusion in attention networks using adaptive and multilevel factorized bilinear pooling for audio-visual emotion recognition. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29 : 2617-2629.