高级搜索

留言板

尊敬的读者、作者、审稿人, 关于本刊的投稿、审稿、编辑和出版的任何问题, 您可以本页添加留言。我们将尽快给您答复。谢谢您的支持!

姓名
邮箱
手机号码
标题
留言内容
验证码

视觉-语言预训练增强的音视频多模态表征与融合框架

孙超,  杜博

孙超, 杜博. 视觉-语言预训练增强的音视频多模态表征与融合框架[J]. 电子与信息学报. doi: 10.11999/JEIT260793
引用本文: 孙超, 杜博. 视觉-语言预训练增强的音视频多模态表征与融合框架[J]. 电子与信息学报. doi: 10.11999/JEIT260793
SUN Chao, DU Bo. Vision-Language Pretraining Enhanced Audio-Visual Multimodal Representation and Fusion Framework[J]. Journal of Electronics & Information Technology. doi: 10.11999/JEIT260793
Citation: SUN Chao, DU Bo. Vision-Language Pretraining Enhanced Audio-Visual Multimodal Representation and Fusion Framework[J]. Journal of Electronics & Information Technology. doi: 10.11999/JEIT260793

视觉-语言预训练增强的音视频多模态表征与融合框架

doi: 10.11999/JEIT260793 cstr: 32379.14.JEIT260793
基金项目: 中国博士后科学基金会与中国煤炭科工集团有限公司联合资助(特别资助)项目(2026T001ZGMK),国家自然科学基金青年学生基础研究项目(本科生)(626B1007),国家重点研发计划(2023YFC2705700)
详细信息
    作者简介:

    孙超:(chaosun@whu.edu.cn):男,博士后,研究方向为多模态学习、视频理解及医学人工智能

    杜博:(dubo@whu.edu.cn):男,教授,博士生导师,研究方向为计算机视觉、大模型及医学人工智能

    通讯作者:

    杜博 dubo@whu.edu.cn

  • 中图分类号: TN911.73; TP391.41; TN912.34

Vision-Language Pretraining Enhanced Audio-Visual Multimodal Representation and Fusion Framework

Funds: China Postdoctoral Science Foundation - CCTEG Joint Support Program (2026T001ZGMK), NSFC Young Students Basic Research Program (Undergraduate) (626B1007), National Key Research and Development Program of China (2023YFC2705700)
  • 摘要: 音视频事件定位旨在识别视频中音频与视觉语义同步出现的时间片段,并判断其事件类别,是多模态视频理解领域的重要研究任务。现有方法通常采用音频分支与视觉分支独立编码后进行特征融合的范式,虽然能够利用跨模态互补信息,但在真实开放场景中,音频环境噪声、视觉背景干扰以及音视频语义不一致等问题会在融合过程中持续累积,进而削弱模型的事件定位能力。针对上述问题,本文提出一种基于视觉-语言预训练微调与多模态互学习融合的双模态联觉框架(DMSF)。该框架首先利用视觉-语言预训练模型的先验知识构建音频与视觉编码器,并设计轻量级Adapter层对编码器进行任务适配,以增强音视频语义表征能力;其次,提出单模态噪声抑制模块(SNS),通过状态空间建模和门控残差自注意力增强模态内时序建模能力;再次,提出跨模态信息协同模块(CIC),采用多头跨模态注意力实现音频与视觉特征的细粒度语义对齐;最后,引入视觉辅助分支,并通过多模态互学习机制抑制融合噪声。本文在AVE数据集上的实验结果表明,所提方法在全监督与弱监督设置下均取得了优良性能,验证了其在音视频事件定位任务中的有效性和鲁棒性。
  • 图  1  DMSF模型整体架构图

    图  2  Adapter层结构图

    图  3  SNS和CIC模块结构图

    图  4  互学习损失的不同拓扑结构图

    图  5  超参数敏感性实验折线图

    图  6  SNS模块前后特征的t-SNE可视化图

    图  7  AVE数据集上模型可视化对比图

    表  1  DMSF与主流方法在AVE数据集上的对比实验结果

    模型 编码器 预训练数据集 参数量(M)↓ Acc.(%)↑ FLOPs(G)↓
    视觉 音频 视觉 音频 可学习 总参数量 全监督 弱监督 全监督
    AGVA[11] ResNet-152 VGGish ImageNet AudioSet 3.7 136.0 68.6 66.7 123.7
    AVSDN[13] ResNet-152 VGGish ImageNet AudioSet 8.0 140.3 72.6 67.3 125.2
    AV-Trans[14] VGG-19 VGGish ImageNet AudioSet 15.8 231.5 76.8 70.2 207.9
    CMRAN[12] ResNet-152 VGGish ImageNet AudioSet 15.9 148.2 77.4 72.9 127.0
    MM-Pyramid[20] ResNet-152 VGGish ImageNet AudioSet 44.0 176.3 77.8 73.5 130.5
    PSP[15] VGG-19 VGGish ImageNet AudioSet 1.7 217.4 77.8 73.5 205.4
    CMBS[16] ResNet-152 VGGish ImageNet AudioSet 14.4 216.7 79.3 74.2 126.8
    CACE-Net*[21] VGG-19 VGGish ImageNet AudioSet 50.1 50.1 80.8 − 206.2
    AVMoE*[25] ResNet-152 VGGish ImageNet AudioSet 383.1 384.2 14.9 − 132.6
    LAVISH[23] ViT-B-16 共享 ImageNet × 4.7 107.2 75.3 − 362.1
    ViT-L-14 共享 ImageNet × 14.5 340.1 78.1 − 1648.9
    EPMI[26] ViT-B-16 共享 ImageNet × 0.6 87.0 76.4 − 355.4
    ViT-L-14 共享 ImageNet × 0.8 305.0 76.9 − 1662.0
    DMSF(本文方法) ViT-B-16 ResNeXt CLIP AudioSet 18.8 101.7 78.7 74.3 223.6
    ViT-L-14 ResNeXt CLIP AudioSet 78.3 290.6 81.7 76.8 821.3
    下载: 导出CSV

    表  2  基于AVE数据集的模块消融实验结果

    模型Acc.(%)↑
    全监督弱监督
    w/o SNS & CIC72.2154.97
    w/o SNS79.7474.82
    w/o CIC81.0675.33
    DMSF(本文方法)81.7176.83
    下载: 导出CSV

    表  3  基于AVE数据集的互学习分支模式消融实验结果

    分支模式Acc.(%)↑
    AVAV全监督弱监督
    √××78.0173.67
    √×√81.7176.83
    √√×78.7071.86
    √√√80.7774.87
    下载: 导出CSV

    表  4  基于AVE数据集的SNS模块消融实验结果

    模型Acc.(%)↑
    全监督弱监督
    Transformer Encoder80.5275.66
    SSM81.0776.15
    SSM+Attention(本文方法)81.7176.83
    下载: 导出CSV

    表  5  基于AVE数据集的损失函数消融实验结果

    损失函数Acc.(%)↑
    全监督弱监督
    $ {\mathcal{L}}_{ce} $81.6375.09
    $ {\mathcal{L}}_{\text{mul}} $65.4655.70
    $ {\mathcal{L}}_{\text{ce}}+\alpha {\mathcal{L}}_{\text{mul}} $81.7176.83
    下载: 导出CSV
  • [1] 李玉鑑. 认知相对论—通向强人工智能之路[J]. 电子与信息学报, 2024, 46(2): 408–427. doi: 10.11999/JEIT230749.

    LI Yujian. Theory of cognitive relativity — the road to strong artificial intelligence[J]. Journal of Electronics & Information Technology, 2024, 46(2): 408–427. doi: 10.11999/JEIT230749.
    [2] 李哲, 王可, 王彪, 等. 人机融合智能决策: 概念、框架与应用[J]. 电子与信息学报, 2025, 47(10): 3439–3464. doi: 10.11999/JEIT250260.

    LI Zhe, WANG Ke, WANG Biao, et al. Human-machine fusion intelligent decision-making: Concepts, framework, and applications[J]. Journal of Electronics & Information Technology, 2025, 47(10): 3439–3464. doi: 10.11999/JEIT250260.
    [3] 李学龙. 多模态认知计算[J]. 中国科学: 信息科学, 2023, 53(1): 1–32. doi: 10.1360/SSI-2022-0226.

    LI Xuelong. Multi-modal cognitive computing[J]. Scientia Sinica (Informationis), 2023, 53(1): 1–32. doi: 10.1360/SSI-2022-0226.
    [4] 孙强, 陈远. 多层次时空特征自适应集成与特有-共享特征融合的双模态情感识别[J]. 电子与信息学报, 2024, 46(2): 574–587. doi: 10.11999/JEIT231110.

    SUN Qiang and CHEN Yuan. Bimodal emotion recognition with adaptive integration of multi-level spatial-temporal features and specific-shared feature fusion[J]. Journal of Electronics & Information Technology, 2024, 46(2): 574–587. doi: 10.11999/JEIT231110.
    [5] 孙立峰, 宋新航, 蒋树强, 等. 多模态协同感知与融合技术专题前言[J]. 软件学报, 2024, 35(5): 2099–2100. doi: 10.13328/j.cnki.jos.007030.

    SUN Lifeng, SONG Xinhang, JIANG Shuqiang, et al. Special topic on multimodal collaborative perception and fusion preface[J]. Journal of Software, 2024, 35(5): 2099–2100. doi: 10.13328/j.cnki.jos.007030.
    [6] 张北辰, 李亮, 查正军, 等. 基于跨模态对比学习的视觉问答主动学习方法[J]. 计算机学报, 2022, 45(8): 1730–1745. doi: 10.11897/SP.J.1016.2022.0173.

    ZHANG Beichen, LI Liang, ZHA Zhengjun, et al. Contrastive cross-modal representation learning based active learning for visual question answer[J]. Chinese Journal of Computers, 2022, 45(8): 1730–1745. doi: 10.11897/SP.J.1016.2022.0173.
    [7] 赵朝阳, 朱贵波, 王金桥. ChatGPT给语言大模型带来的启示和多模态大模型新的发展思路[J]. 数据分析与知识发现, 2023, 7(3): 26–35. doi: 10.11925/infotech.2096-3467.2023.0216.

    ZHAO Chaoyang, ZHU Guibo, and WANG Jinqiao. The inspiration brought by ChatGPT to LLM and the new development ideas of multi-modal large model[J]. Data Analysis and Knowledge Discovery, 2023, 7(3): 26–35. doi: 10.11925/infotech.2096-3467.2023.0216.
    [8] 彭敏轩, 梁艳. 基于双流门控视听融合的多模态语音增强[J]. 计算机系统应用, 2025, 34(11): 127–138. doi: 10.15888/j.cnki.csa.009996.

    PENG Minxuan and LIANG Yan. Multi-modal speech enhancement based on dual-stream gated audio-visual fusion[J]. Computer Systems and Applications, 2025, 34(11): 127–138. doi: 10.15888/j.cnki.csa.009996.
    [9] 章佳妍, 续欣莹, 谢珺, 等. 融合关键区域增强与双向语义引导的视听定位[J]. 计算机工程与应用, 2026, 62(3): 207–217. doi: 10.3778/j.issn.1002–8331.2411-0435.

    ZHANG Jiayan, XU Xinying, XIE Jun, et al. Audio-visual localization with key region enhancement and bidirectional semantic guidance[J]. Computer Engineering and Applications, 2026, 62(3): 207–217. doi: 10.3778/j.issn.1002-8331.2411-0435. doi: 10.3778/j.issn.1002–8331.2411-0435.
    [10] LIU Chen, YANG Liying, LI Peike, et al. Dynamic derivation and elimination: Audio visual segmentation with enhanced audio semantics[C]. 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, USA, 2025: 3131–3141. doi: 10.1109/CVPR52734.2025.00298.
    [11] VILAÇA L, YU Yi, and VIANA P. A survey of recent advances and challenges in deep audio-visual correlation learning[J]. ACM Computing Surveys, 2025, 57(12): 299. doi: 10.1145/3696445.
    [12] YE Qilang, YU Zitong, SHAO Rui, et al. CAT+: Investigating and enhancing audio-visual understanding in large language models[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025, 47(10): 8674–8690. doi: 10.1109/TPAMI.2025.3582389.
    [13] BANDARA T R, GOSTAR A K, ABEYSINGHE P, et al. Occlusion handling for intelligent vehicles using audio-visual multi-object tracking[J]. IEEE Transactions on Intelligent Transportation Systems, 2026, 27(9): 11578–11593. doi: 10.1109/TITS.2026.3697405.
    [14] TIAN Yapeng, SHI Jing, LI Bochen, et al. Audio-visual event localization in unconstrained videos[C]. 2018 15th European Conference on Computer Vision (ECCV), Munich, Germany, 2018: 252–268. doi: 10.1007/978-3-030-01216-8_16.
    [15] XU Haoming, ZENG Runhao, WU Qingyao, et al. Cross-modal relation-aware networks for audio-visual event localization[C]. 2020 28th ACM International Conference on Multimedia (ACM MM), Seattle, USA, 2020: 3893–3901. doi: 10.1145/3394171.3413581.
    [16] LIN Yanbo, LI Y J, and WANG Y C F. Dual-modality seq2seq network for audio-visual event localization[C]. 2019 44th IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Brighton, UK, 2019: 2002–2006. doi: 10.1109/ICASSP.2019.8683226.
    [17] LIN Yanbo and WANG Y C F. Audiovisual transformer with instance attention for audio-visual event localization[C]. 2020 15th Asian Conference on Computer Vision (ACCV), Kyoto, Japan, 2021: 274–290. doi: 10.1007/978-3-030-69544-6_17.
    [18] ZHOU Jinxing, ZHENG Liang, ZHONG Yiran, et al. Positive sample propagation along the audio-visual event line[C]. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, USA, 2021: 8432–8440. doi: 10.1109/CVPR46437.2021.00833.
    [19] XIA Yan and ZHAO Zhou. Cross-modal background suppression for audio-visual event localization[C]. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, USA, 2022: 19957–19966. doi: 10.1109/CVPR52688.2022.01936.
    [20] XUAN Hanyu, ZHANG Zhenyu, CHEN Shuo, et al. Cross-modal attention network for temporal inconsistent audio-visual event localization[C]. 2020 34th AAAI Conference on Artificial Intelligence (AAAI-20), New York, USA, 2020: 279–286. doi: 10.1609/aaai.v34i01.5361.
    [21] SUN Chao, CHEN Min, CHENG Jialiang, et al. SCLAV: Supervised cross-modal contrastive learning for audio-visual coding[C]. 2023 31st ACM International Conference on Multimedia (ACM MM), Ottawa, Canada, 2023: 261–270. doi: 10.1145/3581783.3613805.
    [22] SUN Chao, CHEN Min, ZHU Chuanbo, et al. Listen with seeing: Cross-modal contrastive learning for audio-visual event localization[J]. IEEE Transactions on Multimedia, 2025, 27: 2650–2665. doi: 10.1109/TMM.2025.3535359.
    [23] YU Jiashuo, CHENG Ying, ZHAO Ruiwei, et al. MM-pyramid: Multimodal pyramid attentional network for audio-visual event localization and video parsing[C]. 2022 30th ACM International Conference on Multimedia (ACM MM), Lisbon, Portugal, 2022: 6241–6249. doi: 10.1145/3503161.3547869.
    [24] HE Xiang, LIU Xiangxi, LI Yang, et al. CACE-Net: Co-guidance attention and contrastive enhancement for effective audio-visual event localization[C]. 2024 32nd ACM International Conference on Multimedia (ACM MM), Melbourne, Australia, 2024: 985–993. doi: 10.1145/3664647.3681503.
    [25] GUZHOV A, RAUE F, HEES J, et al. Audioclip: Extending clip to image, text and audio[C]. 2022 47th IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Singapore, Singapore, 2022: 976–980. doi: 10.1109/ICASSP43922.2022.9747631.
    [26] LIN Yanbo, SUNG Y L, LEI Jie, et al. Vision transformers are parameter-efficient audio-visual learners[C]. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, Canada, 2023: 2299–2309. doi: 10.1109/CVPR52729.2023.00228.
    [27] WANG Kai, TIAN Yapeng, and DIMITRIOS H. Towards efficient audio-visual learners via empowering pre-trained vision transformers with cross-modal adaptation[C]. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Seattle, USA, 2024: 1837–1846. doi: 10.1109/CVPRW63382.2024.00190.
    [28] CHENG Ying, LI Yang, HE Junjie, et al. Mixture of experts for audio-visual learning[C]. Proceedings of the 38th International Conference on Neural Information Processing Systems, Vancouver, Canada, 2024: 7. doi: 10.52202/079017-0007.
    [29] HUANG Longzhuo, LI Liang, FU Xueyang, et al. Efficient prompt-based multimodal interaction for audio-visual event localization[C]. Proceedings of the 2025 International Conference on Multimedia Retrieval, Chicago, USA, 2025: 506–515. doi: 10.1145/3731715.3733320.
  • 加载中
图(7) / 表(5)
计量
  • 文章访问数:  18
  • HTML全文浏览量:  2
  • PDF下载量:  0
  • 被引次数: 0
出版历程
  • 修回日期:  2026-11-05
  • 录用日期:  2026-11-05
  • 网络出版日期:  2026-11-20

目录

    /

    返回文章
    返回