Research on Multimodal Sentiment Analysis Method Based on Adversarial Optimization and Triplet Soft Contrastive Learning

KANG Shouqiang; ZHANG Bohao; XIE Jinbao

doi:10.11999/JEIT260255

Article Contents

Article Navigation > Journal of Electronics & Information Technology > 2026 >

KANG Shouqiang, ZHANG Bohao, XIE Jinbao. Research on Multimodal Sentiment Analysis Method Based on Adversarial Optimization and Triplet Soft Contrastive Learning[J]. Journal of Electronics & Information Technology. doi: 10.11999/JEIT260255

Citation:

KANG Shouqiang, ZHANG Bohao, XIE Jinbao. Research on Multimodal Sentiment Analysis Method Based on Adversarial Optimization and Triplet Soft Contrastive Learning[J]. Journal of Electronics & Information Technology. doi: 10.11999/JEIT260255

Citation:

PDF( 2776 KB)

Research on Multimodal Sentiment Analysis Method Based on Adversarial Optimization and Triplet Soft Contrastive Learning

doi: 10.11999/JEIT260255 cstr: 32379.14.JEIT260255

1.
Department of Harbin University of Science and Technology Heilongjiang Provincial Key Laboratory of Pattern Recognition and Information Perception, Heilongjiang 150080, China
2.
College of Physics and Electronic Engineering, Hainan Normal University, Hainan 571158, China

Received Date: 2026-03-06
Accepted Date: 2026-09-13
Rev Recd Date: 2026-09-13

Available Online: 2026-09-18

Abstract

Abstract

Objective Multimodal sentiment analysis has become an important research topic in the field of human–computer interaction, as it aims to understand human emotions by integrating information from multiple modalities such as text, audio, and video. However, existing methods often suffer from two key limitations. First, many approaches neglect the temporal dependency relationships among multimodal features, which leads to insufficient utilization of sequential information. Second, conventional multimodal fusion strategies usually rely on simple feature concatenation, which limits cross-modal interaction and may introduce redundant features. Moreover, due to the continuous and ambiguous nature of sentiment expressions, traditional hard contrastive learning strategies tend to treat sample relationships in a binary manner, which fails to capture the subtle semantic differences between samples. To address these challenges, paper proposes a multimodal sentiment analysis method based on adversarial optimization and triple soft contrastive learning. The proposed framework aims to enhance multimodal feature representation by modeling temporal dependencies and by introducing a soft contrastive learning mechanism that assigns continuous similarity weights between samples. Meanwhile, adversarial optimization is incorporated to improve the robustness and discriminative capability of the learned representations. Methods The proposed approach constructs a triple soft contrastive learning framework integrated with adversarial optimization. The overall architecture consists of three main components: multimodal feature extraction, temporal contrastive representation learning, and adversarial optimization. First, modality-specific encoders are used to extract representations from textual, acoustic, and visual inputs. These features are then aligned in a shared representation space to enable multimodal interaction. To capture temporal dependencies within each modality, a temporal contrastive learning strategy is introduced to model relationships between sequential features and encourage temporally consistent representations. Second, a soft contrastive learning mechanism is adopted to overcome the limitations of conventional hard contrastive learning. Instead of using binary labels to distinguish positive and negative pairs, the proposed method assigns continuous similarity weights according to the sentiment distance between samples. Design allows the model to better capture gradual emotional transitions and reduces the impact of ambiguous sentiment boundaries. Third, an adversarial optimization strategy is introduced to further enhance the robustness of the representation learning process. By constructing a dynamic adversarial training mechanism between representation learning and feature perturbation, the model is able to learn more discriminative and stable multimodal features. The overall training objective integrates the soft contrastive loss, temporal contrastive loss, and adversarial optimization process to achieve more effective multimodal representation learning. The overall framework of the proposed model is illustrated in 图 2. Results and Discussions To evaluate the effectiveness of the proposed method, extensive experiments are conducted on widely used multimodal sentiment analysis datasets. The experimental results demonstrate that the proposed adversarial optimization based triple soft contrastive learning framework significantly improves the performance of multimodal sentiment prediction. Compared with several representative baseline models, the proposed approach achieves competitive results in terms of classification accuracy, F1-score, and regression metrics. The experimental comparisons with existing multimodal methods are summarized in表 and表. The results show that incorporating soft contrastive learning enables the model to capture fine-grained emotional relationships between samples, while adversarial optimization enhances the robustness of feature representations against noisy inputs. Furthermore, ablation studies indicate that each component of the proposed framework contributes to performance improvement. In particular, the soft contrastive learning mechanism effectively alleviates the problem of hard boundary assumptions in traditional contrastive learning, while temporal contrastive modeling helps preserve sequential semantic information. The adversarial optimization module further strengthens the discriminative ability of the learned features. These findings confirm the effectiveness of integrating adversarial learning with contrastive representation learning for multimodal sentiment analysis. Conclusions TPaper proposes a multimodal sentiment analysis method based on adversarial optimization and triple soft contrastive learning. The proposed approach introduces a soft contrastive learning strategy to model continuous sentiment similarity between samples, while temporal contrastive learning captures sequential dependencies in multimodal data. In addition, adversarial optimization is incorporated to enhance representation robustness and improve generalization ability. Experimental results on benchmark datasets demonstrate that the proposed method achieves competitive performance and effectively improves multimodal sentiment representation learning. The proposed framework provides a promising direction for future research on robust and fine-grained multimodal sentiment analysis.
- Multimodal Sentiment Analysis (MSA),
- Soft contrastive learning,
- Generative Adversarial Network (GAN),
- Multimodal learning,
- Feature fusion

FullText(HTML)

References(35)

References

[1]	黄辰, 刘会杰, 张龑, 等. 带全局噪声增强的多模态超图学习引导用于模态信息缺失情感分析[J]. 电子与信息学报, 2025, 47(12): 5192–5202. doi: 10.11999/JEIT250649. HUANG Chen, LIU Huijie, ZHANG Yan, et al. Multimodal hypergraph learning guidance with global noise enhancement for sentiment analysis under missing modality information[J]. Journal of Electronics & Information Technology, 2025, 47(12): 5192–5202. doi: 10.11999/JEIT250649.
[2]	张乐, 陈岩松, 张雷瀚. 大模型特征增强与多层次交叉融合的多模态情感分析方法[J]. 数据分析与知识发现, 2025, 9(8): 47–58. doi: 10.11925/infotech.2096-3467.2024.0625. ZHANG Le, CHEN Yansong, and ZHANG Leihan. A multimodal sentiment analysis method based on LLM feature enhancement and multi-level cross-fusion[J]. Data Analysis and Knowledge Discovery, 2025, 9(8): 47–58. doi: 10.11925/infotech.2096-3467.2024.0625.
[3]	陈杰, 马静, 李晓峰, 等. 基于DR-Transformer模型的多模态情感识别研究[J]. 情报科学, 2022, 40(3): 117–125. doi: 10.13833/j.issn.1007-7634.2022.03.015. CHEN Jie, MA Jing, LI Xiaofeng, et al. Multi-modal emotion recognition based on DR-Transformer model[J]. Information Science, 2022, 40(3): 117–125. doi: 10.13833/j.issn.1007-7634.2022.03.015.
[4]	林宜山, 左景, 卢树华. 基于音视频特征优化与跨模态Transformer的多模态情感分析[J]. 北京航空航天大学学报, 2026, 52(6): 2219–2228. doi: 10.13700/j.bh.1001-5965.2024.0247. LIN Yishan, ZUO Jing, and LU Shuhua. A multimodal sentiment analysis based on audio and video features optimization and cross-modal Transformer[J]. Journal of Beijing University of Aeronautics and Astronautics, 2026, 52(6): 2219–2228. doi: 10.13700/j.bh.1001-5965.2024.0247.
[5]	WU Yujin, DAOUDI M, and AMAD A. Transformer-based self-supervised multimodal representation learning for wearable emotion recognition[J]. IEEE Transactions on Affective Computing, 2024, 15(1): 157–172. doi: 10.1109/TAFFC.2023.3263907.
[6]	CAI Yujian, LI Xingguang, ZHANG Yingyu, et al. Multimodal sentiment analysis based on multi-layer feature fusion and multi-task learning[J]. Scientific Reports, 2025, 15(1): 2126. doi: 10.1038/s41598-025-85859-6.
[7]	冯广, 周垣桦, 钟婷, 等. 结合自适应特征加权与权值优化策略的多模态情感分析[J]. 计算机工程与应用, 2026, 62(6): 194–204. doi: 10.3778/j.issn.1002-8331.2501-0164. FENG Guang, ZHOU Yuanhua, ZHONG Ting, et al. Multimodal sentiment analysis combining adaptive feature weighting and weight optimization strategy[J]. Computer Engineering and Applications, 2026, 62(6): 194–204. doi: 10.3778/j.issn.1002-8331.2501-0164.
[8]	刘佳, 宋泓, 陈大鹏, 等. 非语言信息增强和对比学习的多模态情感分析模型[J]. 电子与信息学报, 2024, 46(8): 3372–3381. doi: 10.11999/JEIT231274. LIU Jia, SONG Hong, CHEN Dapeng, et al. A multimodal sentiment analysis model enhanced with non-verbal information and contrastive learning[J]. Journal of Electronics & Information Technology, 2024, 46(8): 3372–3381. doi: 10.11999/JEIT231274.
[9]	SCHULLER B, RIGOLL G, and LANG M. Speech emotion recognition: Features and classification[J]. Speech Communication, 2009, 51(10): 975–982. doi: 10.1016/j.dsp.2012.05.007. (查阅网上资料,未找到本条文献信息,请确认).
[10]	YE Jiaxin, WEN Xincheng, WEI Yujie, et al. Temporal modeling matters: A novel temporal emotional modeling approach for speech emotion recognition[C]. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, Rhodes Island, Greece, 2023: 1–5. doi: 10.1109/ICASSP49357.2023.10096370.
[11]	MIKOLOV T, CHEN Kai, CORRADO G, et al. Efficient estimation of word representations in vector space[C]. International Conference on Learning Representations, Scottsdale, USA, 2013: 1301–3781. (查阅网上资料, 未找到本条文页码, 请确认).
[12]	DEVLIN J, CHANG Mingwei, LEE K, et al. BERT: Pre-training of deep bidirectional Transformers for language understanding[C]. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis, Minnesota, 2019: 4171–4186. doi: 10.18653/v1/N19-1423.
[13]	WANG Yabing, HUANG Guimin, LI Maolin, et al. Automatically constructing a fine-grained sentiment lexicon for sentiment analysis[J]. Cognitive Computation, 2023, 15(1): 254–271. doi: 10.1007/s12559-022-10043-1.
[14]	JASSIM M A, ABD D H, and OMRI M N. A survey of sentiment analysis from film critics based on machine learning, lexicon and hybridization[J]. Neural Computing and Applications, 2023, 35(13): 9437–9461. doi: 10.1007/s00521-023-08359-6.
[15]	曹银妮, 韩虎, 黄明伟, 等. 基于多视角融合表示的多模态方面级情感分析模型[J]. 数据分析与知识发现, 2025, 9(10): 54–67. doi: 10.11925/infotech.2096-3467.2024.1114. CAO Yinni, HAN Hu, HUANG Mingwei, et al. Multi-modal aspect-level sentiment analysis model with multi-view fusion representation[J]. Data Analysis and Knowledge Discovery, 2025, 9(10): 54–67. doi: 10.11925/infotech.2096-3467.2024.1114.
[16]	赵川斌, 许伟华, 林博, 等. 融合视觉的多模态通信感知一体化关键技术及原型验证[J]. 电子与信息学报, 2026, 48(2): 487–498. doi: 10.11999/JEIT250685. ZHAO Chuanbin, XU Weihua, LIN Bo, et al. Vision enabled multimodal integrated sensing and communications: Key technologies and prototype validation[J]. Journal of Electronics & Information Technology, 2026, 48(2): 487–498. doi: 10.11999/JEIT250685.
[17]	YOU Quanzeng, JIN Hailin, and LUO Jiebo. Visual sentiment analysis by attending on local image regions[C]. Proceedings of the 31st AAAI Conference on Artificial Intelligence, San Francisco, USA, 2017: 231–237.
[18]	LIU Yunze, FAN Qingnan, ZHANG Shanghang, et al. Contrastive multimodal fusion with TupleInfoNCE[C]. Proceedings of the IEEE/CVF International Conference on Computer Vision, Montreal, Canada, 2021: 734–743. doi: 10.1109/ICCV48922.2021.00079.
[19]	YANG Liu, WU Zhenjie, HONG Junkun, et al. MCL: A contrastive learning method for multimodal data fusion in violence detection[J]. IEEE Signal Processing Letters, 2023, 30: 408–412. doi: 10.1109/LSP.2022.3227818.
[20]	GRILL J B, STRUB F, ALTCHÉ F, et al. Bootstrap your own latent a new approach to self-supervised learning[C]. Proceedings of the 34th International Conference on Neural Information Processing Systems, Vancouver, Canada, 2020: 1786.
[21]	WANG Huiru, LI Xiuhong, REN Zenyu, et al. Multimodal sentiment analysis representations learning via contrastive learning with condense attention fusion[J]. Sensors, 2023, 23(5): 2679. doi: 10.3390/s23052679.
[22]	MAI Sijie, ZENG Ying, ZHENG Shuangjia, et al. Hybrid contrastive learning of tri-modal representation for multimodal sentiment analysis[J]. IEEE Transactions on Affective Computing, 2023, 14(3): 2276–2289. doi: 10.1109/TAFFC.2022.3172360.
[23]	QUAN Zhibang, SUN Tao, SU Mengli, et al. Multimodal sentiment analysis based on nonverbal representation optimization network and contrastive interaction learning[C]. Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics, Prague, Czech Republic, 2022: 3086–3091. doi: 10.1109/SMC53654.2022.9945514.
[24]	YANG Jiuding, YU Yakun, NIU Di, et al. ConFEDE: Contrastive feature decomposition for multimodal sentiment analysis[C]. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, Toronto, Canada, 2023: 7617–7630. doi: 10.18653/v1/2023.acl-long.421.
[25]	WANG Senzhang, YAN Hao, DU Jinlong, et al. Adversarial hard negative generation for complementary graph contrastive learning[C]. SIAM International Conference on Data Mining, Austin, USA, 2023: 163–171. doi: 10.1137/1.9781611977653.ch19. (查阅网上资料,未找到本条文献出版地,请确认).
[26]	ZADEH A, ZELLERS R, PINCUS E, et al. MOSI: Multimodal corpus of sentiment intensity and subjectivity analysis in online opinion videos[EB/OL]. https://arxiv.org/abs/1606.06259, 2016.
[27]	ZADEH A, LIANG P P, PORIA S, et al. Multimodal language analysis in the wild: CMU-MOSEI dataset and interpretable dynamic fusion graph[C]. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, Melbourne, Australia, 2018: 2236–2246. doi: 10.18653/v1/P18-1208.
[28]	ZADEH A, CHEN Minghai, PORIA S, et al. Tensor fusion network for multimodal sentiment analysis[C]. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Copenhagen, Denmark, 2017: 1103–1114. doi: 10.18653/v1/D17-1115.
[29]	LIU Zhun, SHEN Ying, LAKSHMINARASIMHAN V B, et al. Efficient low-rank multimodal fusion with modality-specific factors[C]. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, Melbourne, Australia, 2018: 2247–2256. doi: 10.18653/v1/P18-1209.
[30]	YU Wenmeng, XU Hua, YUAN Ziqi, et al. Learning modality-specific representations with self-supervised multi-task learning for multimodal sentiment analysis[C]. Proceedings of the 35th AAAI Conference on Artificial Intelligence, 2021: 10790–10797. doi: 10.1609/aaai.v35i12.17289. (查阅网上资料,未找到本条文献出版地,请确认).
[31]	HUANG Jian, JI Yanli, YANG Yang, et al. Cross-modality representation interactive learning for multimodal sentiment analysis[C]. Proceedings of the 31st ACM International Conference on Multimedia, Ottawa, Canada, 2023: 426–434. doi: 10.1145/3581783.3612295.
[32]	RAHMAN W, HASAN K, LEE S, et al. Integrating multimodal information in large pretrained transformers[C]. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020: 2359–2369. doi: 10.18653/v1/2020.acl-main.214. (查阅网上资料,未找到本条文献出版地,请确认).
[33]	HAZARIKA D, ZIMMERMANN R, and PORIA S. MISA: Modality-invariant and -specific representations for multimodal sentiment analysis[C]. Proceedings of the 28th ACM International Conference on Multimedia, Seattle, USA, 2020: 1122–1131.
[34]	ZHAO Wang, ZHANG Yong, HUA Qiang, et al. MSA-HCL: Multimodal sentiment analysis model with hybrid contrastive learning[J]. Mathematical Foundations of Computing, 2025, 8(3): 433–447. doi: 10.3934/mfc.2024017.
[35]	ZHUANG Yan, BAI Wei, ZHANG Yanru, et al. Multi-level contrastive learning for multimodal sentiment analysis[J]. IEEE Transactions on Multimedia, 2025, 27: 9044–9058. doi: 10.1109/TMM.2025.3613116.