Научная визуализация, 2026, том 18, номер 2, страницы 91 - 107, DOI: 10.26583/sv.18.2.08

MFASA: New Hierarchical Transformer Model with Dynamic Gating and Contrastive Alignment for Multimodal Emotion Recognition

Авторы: Li Bao^1,A, Huafeng Chen^2,B,C, Sergey Ablameyko^3,A,D

^A Belarusian State University, Minsk, 220030, Belarus

^B Zhejiang Shuren University, Hangzhou, 310015, China

^C International Science and Technology Cooperation Base of Zhejiang Province: Remote Sensing Image Processing and Application, Hangzhou 310000, China

^D United Institute for Informatics Problems, National Academy of Sciences of Belarus, Minsk, 220012, Belarus

¹ ORCID: 0009-0008-9010-8950, bil025204@gmail.com

² ORCID: 0000-0003-4229-4505, eric.hf.chen@hotmail.com

³ ORCID: 0000-0001-9404-1206, ablameyko@bsu.by

 

Аннотация

Multimodal emotion recognition is fundamentally constrained by static fusion weights, single-scale feature processing, and cross-modal semantic misalignment. We propose the Multimodal Fusion and Sentiment Analysis (MFASA) model, a hierarchical Transformer that extracts facial, video, and acoustic features via ResNet-18, 3D ResNet, and 1D CNNs, respectively. The proposed model introduces a six-stage progressive optimization process designed to achieve dynamic fusion of multimodal information within a unified Transformer backbone. Unlike traditional "concatenate-and-vote" pipelines, MFASA employs several innovative strategies: Dynamic Modality Gating (DMG) adaptively recalibrates modality contributions through continuous gating; Multi-Scale Residual Fusion (MSRF) preserves hierarchical features via parallel branches to retain micro-expressions; and Cross-Modal Contrastive Alignment (CMCA) enforces semantic consistency via principled contrastive constraints. Our work makes three major contributions: (1) adaptive neural modulation via DMG without information loss; (2) explicit hierarchical preservation through MSRF for fine-grained emotional cues; and (3) explicit cross-modal alignment via CMCA to enhance robustness under occlusion or noisy conditions. Our experiments on the EmoDB, RAVDESS, and CREMA-D datasets show that the MFASA model achieves an accuracy of 91.1%, surpassing the Transformer baseline by 6.8 percentage points, significantly improving emotion recognition performance.

 

Ключевые слова: Multimodal emotion recognition; Cross-modal Transformer; Dynamic modality gating; Contrastive alignment.