Research on Multimodal Sentiment Analysis Method Based on Adversarial Optimization and Triplet Soft Contrastive Learning
-
摘要: 多模态情感分析是人机交互领域的关键环节，但在实际应用中往往忽视了各模态之间的时间依赖关系，导致时序信息未能得到充分利用。同时，现有工作多通过简单的特征拼接来融合多模态信息，跨模态交互有限，容易引入冗余特征。针对上述问题，提出一种对抗优化三重软对比学习的多模态情感分析方法。本方法构建了三重软对比学习框架，即引入软对比学习策略，为样本赋予连续权重，以刻画样本间的细微差异。并在模态内、模态间以及时间序列三个层面开展对比任务，全面捕捉多模态数据在空间与时间维度中的关联信息。同时结合生成对抗网络，提升样本质量，缓解特征冗余问题，从而优化多模态情感特征的学习过程与分类效果。实验结果表明，在CMU-MOSI数据集上准确率和F1 值分别提升了1.3%和1%；而在CMU-MOSEI数据集上准确率和F1 值相较于最新模型均提升了0.7%，验证了所提模型在多模态情感分析任务中的有效性。Abstract:
Objective Multimodal sentiment analysis has become an important research topic in the field of human–computer interaction, as it aims to understand human emotions by integrating information from multiple modalities such as text, audio, and video. However, existing methods often suffer from two key limitations. First, many approaches neglect the temporal dependency relationships among multimodal features, which leads to insufficient utilization of sequential information. Second, conventional multimodal fusion strategies usually rely on simple feature concatenation, which limits cross-modal interaction and may introduce redundant features. Moreover, due to the continuous and ambiguous nature of sentiment expressions, traditional hard contrastive learning strategies tend to treat sample relationships in a binary manner, which fails to capture the subtle semantic differences between samples. To address these challenges, paper proposes a multimodal sentiment analysis method based on adversarial optimization and triple soft contrastive learning. The proposed framework aims to enhance multimodal feature representation by modeling temporal dependencies and by introducing a soft contrastive learning mechanism that assigns continuous similarity weights between samples. Meanwhile, adversarial optimization is incorporated to improve the robustness and discriminative capability of the learned representations. Methods The proposed approach constructs a triple soft contrastive learning framework integrated with adversarial optimization. The overall architecture consists of three main components: multimodal feature extraction, temporal contrastive representation learning, and adversarial optimization. First, modality-specific encoders are used to extract representations from textual, acoustic, and visual inputs. These features are then aligned in a shared representation space to enable multimodal interaction. To capture temporal dependencies within each modality, a temporal contrastive learning strategy is introduced to model relationships between sequential features and encourage temporally consistent representations. Second, a soft contrastive learning mechanism is adopted to overcome the limitations of conventional hard contrastive learning. Instead of using binary labels to distinguish positive and negative pairs, the proposed method assigns continuous similarity weights according to the sentiment distance between samples. Design allows the model to better capture gradual emotional transitions and reduces the impact of ambiguous sentiment boundaries. Third, an adversarial optimization strategy is introduced to further enhance the robustness of the representation learning process. By constructing a dynamic adversarial training mechanism between representation learning and feature perturbation, the model is able to learn more discriminative and stable multimodal features. The overall training objective integrates the soft contrastive loss, temporal contrastive loss, and adversarial optimization process to achieve more effective multimodal representation learning. The overall framework of the proposed model is illustrated in 图 2. Results and Discussions To evaluate the effectiveness of the proposed method, extensive experiments are conducted on widely used multimodal sentiment analysis datasets. The experimental results demonstrate that the proposed adversarial optimization based triple soft contrastive learning framework significantly improves the performance of multimodal sentiment prediction. Compared with several representative baseline models, the proposed approach achieves competitive results in terms of classification accuracy, F1-score, and regression metrics. The experimental comparisons with existing multimodal methods are summarized in表 and表. The results show that incorporating soft contrastive learning enables the model to capture fine-grained emotional relationships between samples, while adversarial optimization enhances the robustness of feature representations against noisy inputs. Furthermore, ablation studies indicate that each component of the proposed framework contributes to performance improvement. In particular, the soft contrastive learning mechanism effectively alleviates the problem of hard boundary assumptions in traditional contrastive learning, while temporal contrastive modeling helps preserve sequential semantic information. The adversarial optimization module further strengthens the discriminative ability of the learned features. These findings confirm the effectiveness of integrating adversarial learning with contrastive representation learning for multimodal sentiment analysis. Conclusions TPaper proposes a multimodal sentiment analysis method based on adversarial optimization and triple soft contrastive learning. The proposed approach introduces a soft contrastive learning strategy to model continuous sentiment similarity between samples, while temporal contrastive learning captures sequential dependencies in multimodal data. In addition, adversarial optimization is incorporated to enhance representation robustness and improve generalization ability. Experimental results on benchmark datasets demonstrate that the proposed method achieves competitive performance and effectively improves multimodal sentiment representation learning. The proposed framework provides a promising direction for future research on robust and fine-grained multimodal sentiment analysis. -
表 2 参数配置对比实验
组别 ACC2(%) F1(%) MAE Corr 1 85.7 0.87 0.49 0.90 2 87.5 0.89 0.45 0.92 3 86.8 0.88 0.47 0.91 4 84.9 0.85 0.50 0.89 
下载: 导出CSV
表 3 数据集数据量
数据集 训练集(个) 验证集(个) 测试集(个) 总计(个) 语言 CMU-MOSI
CMU-MOSEI1284 16326 229
1871686 4659 2199 22856 英文
英文
下载: 导出CSV
表 4 损失函数配置对比实验
组别 λintra λinter λtem λgan ACC2(%) A 0.5 0.5 0.5 0.5 73.7 B 0.5 0.3 0.2 1.0 82.3 C 0.2 0.3 0.5 1.0 78.7 D 1.0 1.0 1.0 1.0 82.0 E 0.5 0.2 0.3 1.0 85.7 F 0.3 0.2 0.5 1.0 87.6
下载: 导出CSV
表 5 在CMU-MOSI数据集的实验结果
模型 MOSI Acc-2(%) F1(%) MAE Corr TFN[28] 80.8 74.5 0.901 0.700 LMF[29] 82.5 82.5 0.917 0.677 Self-MM[30] 85.9 85.9 0.713 0.798 CRIL[31] 86.9 86.8 0.695 0.812 MAG-BERT[32] 83.0 82.8 0.871 0.559 MISA[33] 83.5 83.5 0.790 0.593 HyCon[22] 86.4 86.4 0.644 0.832 ConFEDE[24] 86.5 86.5 0.708 0.796 MSA-HCL[34] 86.4 86.4 0.726 0.789 MLCL[35] 86.4 86.3 0.701 0.798 Ours 87.8 87.5 0.605 0.804
下载: 导出CSV
表 6 在CMU-MOSEI数据集的实验结果
模型 MOSEI Acc-2(%) F1(%) MAE Corr TFN[28] 82.6 82.1 0.593 0.700 LMF[29] 82.0 82.5 0.623 0.677 Self-MM[30] 85.3 85.1 0.530 0.765 CRIL[31] 86.2 86.1 0.529 0.767 MAG-BERT[32] 85.6 85.0 0.602 0.778 MISA[33] 85.5 85.3 0.555 0.756 HyCon[22] 86.5 86.4 0.590 0.792 ConFEDE[24] 86.8 86.9 0.522 0.780 MSA-HCL[34] 86.4 85.9 0.530 0.771 MLCL[35] 86.3 86.2 0.551 0.756 Ours 87.5 87.6 0.506 0.740
下载: 导出CSV
表 7 MOSI数据集上模块消融实验结果
模型 MOSI Acc-2(%) F1(%) MAE Corr w/o TemCL 74.4 75.9 0.642 0.801 w/o InterCL 79.9 78.6 0.625 0.802 w/o IntraCL 60.7 81.3 0.602 0.548 w/o SoftCL 86.2 87.4 0.548 0.742 w/o GAN 80.8 81.6 0.590 0.746 Ours 87.8 87.5 0.605 0.804
下载: 导出CSV
表 8 MOSEI数据集上模块消融实验结果
模型 MOSEI Acc-2(%) F1(%) MAE Corr w/o TemCL 76.0 79.7 0.597 0.845 w/o InterCL 81.3 83.4 0.587 0.860 w/o IntraCL 62.0 85.1 0.563 0.893 w/o SoftCL 85.5 83.9 0.490 0.881 w/o GAN 81.7 86.8 0.537 0.881 Ours 87.5 87.6 0.450 0.920
下载: 导出CSV
表 11 MOSI数据集和MOSEI数据集模态缺失鲁棒性实验
模型 MOSI MOSEI Acc-2(%) F1(%) MAE Corr Acc-2(%) F1(%) MAE Corr 全模态 87.8 87.5 0.605 0.804 87.5 87.6 0.506 0.740 缺失文本模态 82.3 82.1 0.720 0.690 81.5 81.3 0.642 0.635 缺失语音模态 86.9 86.8 0.625 0.785 86.2 86.3 0.535 0.720 缺失视觉模态 85.7 85.6 0.651 0.760 84.8 84.9 0.566 0.695
下载: 导出CSV
-
[1] 黄辰, 刘会杰, 张龑, 等. 带全局噪声增强的多模态超图学习引导用于模态信息缺失情感分析[J]. 电子与信息学报, 2025, 47(12): 5192–5202. doi: 10.11999/JEIT250649.HUANG Chen, LIU Huijie, ZHANG Yan, et al. Multimodal hypergraph learning guidance with global noise enhancement for sentiment analysis under missing modality information[J]. Journal of Electronics & Information Technology, 2025, 47(12): 5192–5202. doi: 10.11999/JEIT250649. [2] 张乐, 陈岩松, 张雷瀚. 大模型特征增强与多层次交叉融合的多模态情感分析方法[J]. 数据分析与知识发现, 2025, 9(8): 47–58. doi: 10.11925/infotech.2096-3467.2024.0625.ZHANG Le, CHEN Yansong, and ZHANG Leihan. A multimodal sentiment analysis method based on LLM feature enhancement and multi-level cross-fusion[J]. Data Analysis and Knowledge Discovery, 2025, 9(8): 47–58. doi: 10.11925/infotech.2096-3467.2024.0625. [3] 陈杰, 马静, 李晓峰, 等. 基于DR-Transformer模型的多模态情感识别研究[J]. 情报科学, 2022, 40(3): 117–125. doi: 10.13833/j.issn.1007-7634.2022.03.015.CHEN Jie, MA Jing, LI Xiaofeng, et al. Multi-modal emotion recognition based on DR-Transformer model[J]. Information Science, 2022, 40(3): 117–125. doi: 10.13833/j.issn.1007-7634.2022.03.015. [4] 林宜山, 左景, 卢树华. 基于音视频特征优化与跨模态Transformer的多模态情感分析[J]. 北京航空航天大学学报, 2026, 52(6): 2219–2228. doi: 10.13700/j.bh.1001-5965.2024.0247.LIN Yishan, ZUO Jing, and LU Shuhua. A multimodal sentiment analysis based on audio and video features optimization and cross-modal Transformer[J]. Journal of Beijing University of Aeronautics and Astronautics, 2026, 52(6): 2219–2228. doi: 10.13700/j.bh.1001-5965.2024.0247. [5] WU Yujin, DAOUDI M, and AMAD A. Transformer-based self-supervised multimodal representation learning for wearable emotion recognition[J]. IEEE Transactions on Affective Computing, 2024, 15(1): 157–172. doi: 10.1109/TAFFC.2023.3263907. [6] CAI Yujian, LI Xingguang, ZHANG Yingyu, et al. Multimodal sentiment analysis based on multi-layer feature fusion and multi-task learning[J]. Scientific Reports, 2025, 15(1): 2126. doi: 10.1038/s41598-025-85859-6. [7] 冯广, 周垣桦, 钟婷, 等. 结合自适应特征加权与权值优化策略的多模态情感分析[J]. 计算机工程与应用, 2026, 62(6): 194–204. doi: 10.3778/j.issn.1002-8331.2501-0164.FENG Guang, ZHOU Yuanhua, ZHONG Ting, et al. Multimodal sentiment analysis combining adaptive feature weighting and weight optimization strategy[J]. Computer Engineering and Applications, 2026, 62(6): 194–204. doi: 10.3778/j.issn.1002-8331.2501-0164. [8] 刘佳, 宋泓, 陈大鹏, 等. 非语言信息增强和对比学习的多模态情感分析模型[J]. 电子与信息学报, 2024, 46(8): 3372–3381. doi: 10.11999/JEIT231274.LIU Jia, SONG Hong, CHEN Dapeng, et al. A multimodal sentiment analysis model enhanced with non-verbal information and contrastive learning[J]. Journal of Electronics & Information Technology, 2024, 46(8): 3372–3381. doi: 10.11999/JEIT231274. [9] SCHULLER B, RIGOLL G, and LANG M. Speech emotion recognition: Features and classification[J]. Speech Communication, 2009, 51(10): 975–982. doi: 10.1016/j.dsp.2012.05.007. (查阅网上资料,未找到本条文献信息,请确认). [10] YE Jiaxin, WEN Xincheng, WEI Yujie, et al. Temporal modeling matters: A novel temporal emotional modeling approach for speech emotion recognition[C]. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, Rhodes Island, Greece, 2023: 1–5. doi: 10.1109/ICASSP49357.2023.10096370. [11] MIKOLOV T, CHEN Kai, CORRADO G, et al. Efficient estimation of word representations in vector space[C]. International Conference on Learning Representations, Scottsdale, USA, 2013: 1301–3781. (查阅网上资料, 未找到本条文页码, 请确认). [12] DEVLIN J, CHANG Mingwei, LEE K, et al. BERT: Pre-training of deep bidirectional Transformers for language understanding[C]. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis, Minnesota, 2019: 4171–4186. doi: 10.18653/v1/N19-1423. [13] WANG Yabing, HUANG Guimin, LI Maolin, et al. Automatically constructing a fine-grained sentiment lexicon for sentiment analysis[J]. Cognitive Computation, 2023, 15(1): 254–271. doi: 10.1007/s12559-022-10043-1. [14] JASSIM M A, ABD D H, and OMRI M N. A survey of sentiment analysis from film critics based on machine learning, lexicon and hybridization[J]. Neural Computing and Applications, 2023, 35(13): 9437–9461. doi: 10.1007/s00521-023-08359-6. [15] 曹银妮, 韩虎, 黄明伟, 等. 基于多视角融合表示的多模态方面级情感分析模型[J]. 数据分析与知识发现, 2025, 9(10): 54–67. doi: 10.11925/infotech.2096-3467.2024.1114.CAO Yinni, HAN Hu, HUANG Mingwei, et al. Multi-modal aspect-level sentiment analysis model with multi-view fusion representation[J]. Data Analysis and Knowledge Discovery, 2025, 9(10): 54–67. doi: 10.11925/infotech.2096-3467.2024.1114. [16] 赵川斌, 许伟华, 林博, 等. 融合视觉的多模态通信感知一体化关键技术及原型验证[J]. 电子与信息学报, 2026, 48(2): 487–498. doi: 10.11999/JEIT250685.ZHAO Chuanbin, XU Weihua, LIN Bo, et al. Vision enabled multimodal integrated sensing and communications: Key technologies and prototype validation[J]. Journal of Electronics & Information Technology, 2026, 48(2): 487–498. doi: 10.11999/JEIT250685. [17] YOU Quanzeng, JIN Hailin, and LUO Jiebo. Visual sentiment analysis by attending on local image regions[C]. Proceedings of the 31st AAAI Conference on Artificial Intelligence, San Francisco, USA, 2017: 231–237. [18] LIU Yunze, FAN Qingnan, ZHANG Shanghang, et al. Contrastive multimodal fusion with TupleInfoNCE[C]. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, Canada, 2021: 734–743. doi: 10.1109/ICCV48922.2021.00079. [19] YANG Liu, WU Zhenjie, HONG Junkun, et al. MCL: A contrastive learning method for multimodal data fusion in violence detection[J]. IEEE Signal Processing Letters, 2023, 30: 408–412. doi: 10.1109/LSP.2022.3227818. [20] GRILL J B, STRUB F, ALTCHÉ F, et al. Bootstrap your own latent a new approach to self-supervised learning[C]. Proceedings of the 34th International Conference on Neural Information Processing Systems, Vancouver, Canada, 2020: 1786. [21] WANG Huiru, LI Xiuhong, REN Zenyu, et al. Multimodal sentiment analysis representations learning via contrastive learning with condense attention fusion[J]. Sensors, 2023, 23(5): 2679. doi: 10.3390/s23052679. [22] MAI Sijie, ZENG Ying, ZHENG Shuangjia, et al. Hybrid contrastive learning of tri-modal representation for multimodal sentiment analysis[J]. IEEE Transactions on Affective Computing, 2023, 14(3): 2276–2289. doi: 10.1109/TAFFC.2022.3172360. [23] QUAN Zhibang, SUN Tao, SU Mengli, et al. Multimodal sentiment analysis based on nonverbal representation optimization network and contrastive interaction learning[C]. Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics, Prague, Czech Republic, 2022: 3086–3091. doi: 10.1109/SMC53654.2022.9945514. [24] YANG Jiuding, YU Yakun, NIU Di, et al. ConFEDE: Contrastive feature decomposition for multimodal sentiment analysis[C]. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, Toronto, Canada, 2023: 7617–7630. doi: 10.18653/v1/2023.acl-long.421. [25] WANG Senzhang, YAN Hao, DU Jinlong, et al. Adversarial hard negative generation for complementary graph contrastive learning[C]. SIAM International Conference on Data Mining, Austin, USA, 2023: 163–171. doi: 10.1137/1.9781611977653.ch19. (查阅网上资料,未找到本条文献出版地,请确认). [26] ZADEH A, ZELLERS R, PINCUS E, et al. MOSI: Multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos[EB/OL]. https://arxiv.org/abs/1606.06259, 2016. [27] ZADEH A, LIANG P P, PORIA S, et al. Multimodal language analysis in the wild: CMU-MOSEI dataset and interpretable dynamic fusion graph[C]. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, Melbourne, Australia, 2018: 2236–2246. doi: 10.18653/v1/P18-1208. [28] ZADEH A, CHEN Minghai, PORIA S, et al. Tensor fusion network for multimodal sentiment analysis[C]. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Copenhagen, Denmark, 2017: 1103–1114. doi: 10.18653/v1/D17-1115. [29] LIU Zhun, SHEN Ying, LAKSHMINARASIMHAN V B, et al. Efficient low-rank multimodal fusion with modality-specific factors[C]. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, Melbourne, Australia, 2018: 2247–2256. doi: 10.18653/v1/P18-1209. [30] YU Wenmeng, XU Hua, YUAN Ziqi, et al. Learning modality-specific representations with self-supervised multi-task learning for multimodal sentiment analysis[C]. Proceedings of the 35th AAAI Conference on Artificial Intelligence, 2021: 10790–10797. doi: 10.1609/aaai.v35i12.17289. (查阅网上资料,未找到本条文献出版地,请确认). [31] HUANG Jian, JI Yanli, YANG Yang, et al. Cross-modality representation interactive learning for multimodal sentiment analysis[C]. Proceedings of the 31st ACM International Conference on Multimedia, Ottawa, Canada, 2023: 426–434. doi: 10.1145/3581783.3612295. [32] RAHMAN W, HASAN K, LEE S, et al. Integrating multimodal information in large pretrained transformers[C]. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020: 2359–2369. doi: 10.18653/v1/2020.acl-main.214. (查阅网上资料,未找到本条文献出版地,请确认). [33] HAZARIKA D, ZIMMERMANN R, and PORIA S. MISA: Modality-invariant and -specific representations for multimodal sentiment analysis[C]. Proceedings of the 28th ACM International Conference on Multimedia, Seattle, USA, 2020: 1122–1131. [34] ZHAO Wang, ZHANG Yong, HUA Qiang, et al. MSA-HCL: Multimodal sentiment analysis model with hybrid contrastive learning[J]. Mathematical Foundations of Computing, 2025, 8(3): 433–447. doi: 10.3934/mfc.2024017. [35] ZHUANG Yan, BAI Wei, ZHANG Yanru, et al. Multi-level contrastive learning for multimodal sentiment analysis[J]. IEEE Transactions on Multimedia, 2025, 27: 9044–9058. doi: 10.1109/TMM.2025.3613116. -
下载:
图(6) / 表(11)
计量
- 文章访问数: 12
- HTML全文浏览量: 1
- PDF下载量: 0
- 被引次数: 0
