Coupled Multimodal Emotional Feature Analysis Based on Broad-Deep Fusion Networks in Human-Robot Interaction

Basit Öğe Kaydı
dc.authoridli, min/0000-0002-5773-8976
dc.authoridHirota, Kaoru/0000-0002-8118-9815
dc.authoridWu, Min/0000-0002-0668-8315
dc.contributor.authorChen, Luefeng
dc.contributor.authorLi, Min
dc.contributor.authorWu, Min
dc.contributor.authorPedrycz, Witold
dc.contributor.authorHirota, Kaoru
dc.date.accessioned2024-05-19T14:39:39Z
dc.date.available2024-05-19T14:39:39Z
dc.date.issued2023
dc.departmentİstinye Üniversitesien_US
dc.description.abstractA coupled multimodal emotional feature analysis (CMEFA) method based on broad-deep fusion networks, which divide multimodal emotion recognition into two layers, is proposed. First, facial emotional features and gesture emotional features are extracted using the broad and deep learning fusion network (BDFN). Considering that the bi-modal emotion is not completely independent of each other, canonical correlation analysis (CCA) is used to analyze and extract the correlation between the emotion features, and a coupling network is established for emotion recognition of the extracted bi-modal features. Both simulation and application experiments are completed. According to the simulation experiments completed on the bimodal face and body gesture database (FABO), the recognition rate of the proposed method has increased by 1.15% compared to that of the support vector machine recursive feature elimination (SVMRFE) (without considering the unbalanced contribution of features). Moreover, by using the proposed method, the multimodal recognition rate is 21.22%, 2.65%, 1.61%, 1.54%, and 0.20% higher than those of the fuzzy deep neural network with sparse autoencoder (FDNNSA), ResNet-101 + GFK, C3D + MCB + DBN, the hierarchical classification fusion strategy (HCFS), and cross-channel convolutional neural network (CCCNN), respectively. In addition, preliminary application experiments are carried out on our developed emotional social robot system, where emotional robot recognizes the emotions of eight volunteers based on their facial expressions and body gestures.en_US
dc.description.sponsorshipNational Natural Science Foundation of China [61973286, 61603356, 61773353]; 111 Project [B17040]; Fundamental Research Funds for the Central Universities, China University of Geosciences [2021063]en_US
dc.description.sponsorshipThis work was supported in part by the National Natural Science Foundation of China under Grant 61973286, Grant 61603356, and Grant 61773353; in part by the 111 Project under Grant B17040; and in part by the Fundamental Research Funds for the Central Universities, China University of Geosciences under Grant 2021063. (Corresponding author: Min Wu.) Luefengen_US
dc.identifier.doi10.1109/TNNLS.2023.3236320
dc.identifier.issn2162-237X
dc.identifier.issn2162-2388
dc.identifier.pmid37021991en_US
dc.identifier.urihttps://doi.org10.1109/TNNLS.2023.3236320
dc.identifier.urihttps://hdl.handle.net/20.500.12713/4824
dc.identifier.wosWOS:001128303600001en_US
dc.identifier.wosqualityN/Aen_US
dc.indekslendigikaynakWeb of Scienceen_US
dc.language.isoenen_US
dc.publisherIeee-Inst Electrical Electronics Engineers Incen_US
dc.relation.ispartofIeee Transactions on Neural Networks and Learning Systemsen_US
dc.relation.publicationcategoryMakale - Uluslararası Hakemli Dergi - Kurum Öğretim Elemanıen_US
dc.rightsinfo:eu-repo/semantics/closedAccessen_US
dc.snmz20240519_kaen_US
dc.subjectBroad Learningen_US
dc.subjectDeep Feature Fusionen_US
dc.subjectDeep Neural Networksen_US
dc.subjectHuman-Robot Interactionen_US
dc.subjectMultimodal Emotion Recognitionen_US
dc.titleCoupled Multimodal Emotional Feature Analysis Based on Broad-Deep Fusion Networks in Human-Robot Interactionen_US
dc.typeArticleen_US

Dosyalar

Koleksiyon

WoS İndeksli Yayınlar Koleksiyonu
PubMed İndeksli Yayınlar Koleksiyonu