A Multimodal BiMamba Network with Test-Time Adaptation for Emotion Recognition Based on Physiological Signals

Ziyu Jia, Tingyu Du, Zhengyu Tian, Hongkai Li, Yong Zhang, Chenyu Liu

Chinese Academy of Sciences· Shanghai Key Laboratory of Data Science· Institute of Computing Technology, Chinese Academy of Sciences· Beijing Jiaotong University· Huzhou University· Nanyang Technological University

Emotion Recognition Mamba Network Test-Time Adaptation Multimodal Physiological Signals

⋅ NeurIPS ⋅ OpenReview

Abstract

Emotion recognition based on physiological signals plays a vital role in psychological health and human–computer interaction, particularly with the substantial advances in multimodal emotion recognition techniques.

However, two key challenges remain unresolved:

how to effectively model the intra-modal long-range dependencies and inter-modal correlations in multimodal physiological emotion signals, and
how to address the performance limitations resulting from missing multimodal data.

In this paper, we propose a multimodal bidirectional Mamba (BiMamba) network with test-time adaptation (TTA) for emotion recognition named BiM-TTA. Specifically, BiM-TTA consists of a multimodal BiMamba network and a multimodal TTA. The former includes intra-modal and inter-modal BiMamba modules, which model long-range dependencies along the time dimension and capture cross-modal correlations along the channel dimension, respectively. The latter (TTA) mitigates the amplified distribution shifts caused by missing multimodal data through two-level entropy-based sample filtering and mutual information sharing across modalities.

By addressing these challenges, BiM-TTA achieves state-of-the-art results on two multimodal emotion datasets.