A Lightweight Multi-Modal Semantic Segmentation Method Based on Dynamic Prompt and Knowledge Distillation

SHEN Bo; ZHANG Zhixiang; SHANG Zhengying; ZHU Zhenyu; YANG Gang

doi:10.11999/JEIT260729

Article Contents

Article Navigation > Journal of Electronics & Information Technology > 2025 >

SHEN Bo, ZHANG Zhixiang, SHANG Zhengying, ZHU Zhenyu, YANG Gang. A Lightweight Multi-Modal Semantic Segmentation Method Based on Dynamic Prompt and Knowledge Distillation[J]. Journal of Electronics & Information Technology. doi: 10.11999/JEIT260729

Citation:

SHEN Bo, ZHANG Zhixiang, SHANG Zhengying, ZHU Zhenyu, YANG Gang. A Lightweight Multi-Modal Semantic Segmentation Method Based on Dynamic Prompt and Knowledge Distillation[J]. Journal of Electronics & Information Technology. doi: 10.11999/JEIT260729

Citation:

SHEN Bo, ZHANG Zhixiang, SHANG Zhengying, ZHU Zhenyu, YANG Gang. A Lightweight Multi-Modal Semantic Segmentation Method Based on Dynamic Prompt and Knowledge Distillation[J]. Journal of Electronics & Information Technology. doi: 10.11999/JEIT260729

PDF( 2308 KB)

A Lightweight Multi-Modal Semantic Segmentation Method Based on Dynamic Prompt and Knowledge Distillation

doi: 10.11999/JEIT260729 cstr: 32379.14.JEIT260729

School of Computer Science, Northwestern Polytechnical University, Xi’an 710129, China

Received Date: 2026-06-02
Accepted Date: 2026-10-16
Rev Recd Date: 2026-10-11

Available Online: 2026-10-25

Abstract

Abstract

Objective Semantic segmentation assigns a semantic category to each image pixel and supports scene understanding in autonomous driving and robot navigation. Methods based on visible-light images can lose accuracy under low illumination, strong glare, and occlusion. Thermal infrared and depth images provide complementary information, but extracting features with two full backbone networks increases model size and computational cost, limiting deployment on embedded devices with restricted computing resources. Parameter-efficient adaptation reduces the number of weights updated during training, but a large pretrained backbone can still impose substantial inference overhead. Model compression and runtime optimization are therefore also needed. This study develops a multi-modal segmentation framework that combines dynamic visual prompts, structured knowledge distillation, and deployment optimization. The framework aims to improve the use of auxiliary-modality information with a limited number of trainable parameters, transfer the resulting representations to a smaller student network, and increase inference throughput across heterogeneous embedded platforms. Methods A Multi-Modal Feature Fusion Prompt Network (MFFPNet) is constructed using an ImageNet-pretrained Mix Transformer B5 (MiT-B5) backbone and a SegFormer decoder. Most pretrained backbone parameters remain frozen, while the prompt, fusion, normalization, and decoding components are adapted to the segmentation task. The encoder retains four hierarchical stages. A lightweight Feature Fusion Prompt (FFP) module encodes thermal infrared or depth information into dynamic visual prompts, which are added to visible-light features. The prompt generated at each stage is passed to the next stage after spatial and channel adjustment, allowing auxiliary information to contribute at multiple feature scales. Global learnable prompts provide an additional mechanism for adapting pretrained representations to the task (Fig. 1). Within the FFP module, spatial and channel attention assign different weights to the two modalities. Complementary feature reweighting is followed by cross-attention to capture cross-modal relationships and generate the visual prompt. The decoder aggregates outputs from all four stages to produce pixelwise predictions. Training combines label-smoothed cross-entropy loss and Dice loss to account for pixel-level classification and regional overlap. The backbone is initialized with pretrained weights, and training uses stochastic gradient descent with momentum and polynomial learning-rate decay. Data augmentation includes multi-scale resizing, random cropping, horizontal flipping, and photometric perturbations. For model compression, the trained MFFPNet serves as a frozen teacher in an offline distillation framework. MFFPNet-S uses MiT-B1 as its main student backbone, and MiT-B2 is also evaluated on NYUDv2 (Fig. 4). A feature adapter applies convolution, batch normalization, and a rectified linear unit to reduce differences between student and teacher features (Fig. 5). Distillation combines feature content loss, channel relation loss, spatial structure loss, and logit supervision. A confidence mask emphasizes teacher predictions in high-confidence regions, while a global supervision term and ground-truth labels preserve training information outside the selected regions. Combining local and global supervision preserves structural information while reducing the influence of low-confidence local predictions (Fig. 6). For deployment, Open Neural Network Exchange (ONNX) provides an intermediate model representation for platform-specific compilation. TensorRT uses half-precision inference on NVIDIA Jetson Orin NX, the Compute Architecture for Neural Networks (CANN) software stack supports Huawei Atlas 200I DK A2, and the Rockchip Neural Network (RKNN) toolchain supports RK3588. Mixed-precision inference on RK3588 retains higher precision for sensitive attention and output layers. Containerized environments and a common inference interface support deployment, while a three-stage asynchronous pipeline overlaps data loading, inference, and network communication. Results and Discussions The evaluation uses the MFNet dataset, which pairs Red-Green-Blue (RGB) images with thermal images, and the NYUDv2 dataset, which pairs RGB images with depth images. On MFNet, MFFPNet achieves a mean Intersection over Union (mIoU) of 63.3% with 9.4 million trainable parameters, approximately 10.4% of its 90.4 million total parameters. Its mIoU exceeds the reported values for CRM-B, DPLNet, and MixPrompt by 1.9, 5.0, and 3.2 percentage points, respectively. MixPrompt uses fewer trainable parameters, indicating a trade-off between segmentation accuracy and adaptation size (Table 2). Visual comparisons show improved pedestrian contours under challenging nighttime illumination (Fig. 2). On NYUDv2, MFFPNet achieves an mIoU of 60.6% under single-scale testing, compared with 58.3% for DPLNet under the same testing protocol. The reported multi-scale results for DPLNet and MixPrompt use a different evaluation protocol and therefore cannot be directly compared with the single-scale results (Table 3). Indoor examples show finer object boundaries and better separation of visually similar regions (Fig. 3). Ablation results show that the global prompt and full fusion module improve segmentation performance (Table 4). Removing cross-attention reduces the MFNet mIoU from 63.3% to 62.1% (Table 5). Applying the fusion module at all four stages yields higher mIoU than applying it only at the first or last stage, supporting the use of features at multiple resolutions (Table 6). On MFNet, the MiT-B1 student achieves an mIoU of 56.2% with approximately 18.05 million parameters and 19.6 Giga Floating-Point Operations (GFLOPs), compared with 63.3% mIoU, 90.4 million parameters, and 122.8 GFLOPs for the teacher. The student therefore retains approximately 88.8% of the teacher’s mIoU while using 20.0% of its parameter count and reducing computation by approximately 84%. Its measured inference speed is 23.9 frames per second (fps), compared with 10.0 fps for the teacher (Table 8). On NYUDv2, distillation increases the MiT-B1 student’s mIoU from 38.6% to 42.9%, a gain of 4.3 percentage points. The MiT-B2 student reaches 48.5%, providing higher accuracy with a larger model (Table 9). The MFNet distillation ablation also shows successive performance improvements as additional supervision components are introduced (Table 10). In synchronous deployment tests, the student achieves mIoUs of 55.7%, 51.5%, and 49.4% at 48.6, 31.5, and 28.6 fps on Jetson Orin NX, Atlas 200I DK A2, and RK3588, respectively; the RK3588 result is obtained with mixed-precision inference. The 8-bit Integer (INT8) configuration on RK3588 reaches 31.2 fps but reduces mIoU to 41.1%, illustrating the accuracy cost of more aggressive quantization (Table 12). With a continuous 1 000-frame input, asynchronous execution increases throughput to 68.5, 41.2, and 34.8 fps on the three platforms, respectively, corresponding to gains of 40.9%, 30.8%, and 21.7% over their serial configurations (Table 13). Conclusions The results support dynamic visual prompting as a parameter-efficient approach to semantic segmentation with RGB-Thermal and RGB-Depth inputs. Structured distillation reduces the inference cost of the large teacher while retaining 88.8% of its mIoU on MFNet. Results on NYUDv2 also show a trade-off between model capacity and segmentation accuracy. Platform-specific compilation and asynchronous scheduling further improve throughput, whereas quantization can substantially reduce segmentation accuracy. The framework thus addresses training adaptation, model compression, and runtime execution as complementary aspects of embedded deployment. Future work will investigate modality completion using generative models or cross-modal distillation to improve segmentation when auxiliary sensor information is unavailable.
- Semantic segmentation,
- Multi-modal fusion,
- Visual prompt,
- Knowledge distillation,
- Embedded platform

FullText(HTML)

References(32)

References

[1]	FENG Di, HAASE-SCHÜTZ C, ROSENBAUM L, et al. Deep multi-modal object detection and semantic segmentation for autonomous driving: Datasets, methods, and challenges[J]. IEEE Transactions on Intelligent Transportation Systems, 2021, 22(3): 1341–1360. doi: 10.1109/TITS.2020.2972974.
[2]	WANG Li, ZHANG Xinyu, SONG Ziying, et al. Multi-modal 3D object detection in autonomous driving: A survey and taxonomy[J]. IEEE Transactions on Intelligent Vehicles, 2023, 8(7): 3781–3798. doi: 10.1109/TIV.2023.3264658.
[3]	梁燕, 李俊范, 邵凯, 等. 空间信息引导扩散的遥感图像域自适应语义分割[J]. 电子与信息学报, 2026, 48(7): 2829–2842. doi: 10.11999/JEIT260031. LIANG Yan, LI Junfan, SHAO Kai, et al. Spatial information-guided diffusion for domain adaptation semantic segmentation of remote sensing images[J]. Journal of Electronics & Information Technology, 2026, 48(7): 2829–2842. doi: 10.11999/JEIT260031.
[4]	WANG Changshuo, WANG Chen, LI Weijun, et al. A brief survey on RGB-D semantic segmentation using deep learning[J]. Displays, 2021, 70: 102080. doi: 10.1016/j.displa.2021.102080.
[5]	HAZIRBAS C, MA Lingni, DOMOKOS C, et al. FuseNet: Incorporating depth into semantic segmentation via fusion-based CNN architecture[C]. 13th Asian Conference on Computer Vision, Taipei, China, 2016: 213–228. doi: 10.1007/978-3-319-54181-5_14.
[6]	ZHANG Jiaming, LIU Huayao, YANG Kailun, et al. CMX: Cross-modal fusion for RGB-X semantic segmentation with transformers[J]. IEEE Transactions on Intelligent Transportation Systems, 2023, 24(12): 14679–14694. doi: 10.1109/TITS.2023.3300537.
[7]	SUN Yuxiang, ZUO Weixun, and LIU Ming. RTFNet: RGB-thermal fusion network for semantic segmentation of urban scenes[J]. IEEE Robotics and Automation Letters, 2019, 4(3): 2576–2583. doi: 10.1109/LRA.2019.2904733.
[8]	CAPRA M, BUSSOLINO B, MARCHISIO A, et al. Hardware and software optimizations for accelerating deep neural networks: Survey of current trends, challenges, and the road ahead[J]. IEEE Access, 2020, 8: 225134–225180. doi: 10.1109/ACCESS.2020.3039858.
[9]	DONG Shaohua, FENG Yunhe, YANG Qing, et al. Efficient multimodal semantic segmentation via dual-prompt learning[C]. 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems, Abu Dhabi, United Arab Emirates, 2024: 14196–14203. doi: 10.1109/IROS58592.2024.10801872.
[10]	HAO Zhiwei, XIAO Zhongyu, GUO Jianyuan, et al. MixPrompt: Efficient mixed prompting for multimodal semantic segmentation[C]. The Thirty-ninth Annual Conference on Neural Information Processing Systems, San Diego, USA, 2025: 39255–39282. doi: 10.52202/085713-1173.
[11]	毛秀华, 张强, 阮航, 等. 多任务协同的多模态遥感目标分割算法[J]. 电子与信息学报, 2024, 46(8): 3363–3371. doi: 10.11999/JEIT231267. MAO Xiuhua, ZHANG Qiang, RUAN Hang, et al. Multitask collaborative multi-modal remote sensing target segmentation algorithm[J]. Journal of Electronics & Information Technology, 2024, 46(8): 3363–3371. doi: 10.11999/JEIT231267.
[12]	WANG Yikai, CHEN Xinghao, CAO Lele, et al. Multimodal token fusion for vision transformers[C]. The IEEE/CVF Conference on Computer Vision and Pattern Recognition, New Orleans, USA, 2022: 12176–12185. doi: 10.1109/cvpr52688.2022.01187.
[13]	ZHANG Yang, XIONG Chenyun, LIU Junjie, et al. Spatial information-guided adaptive context-aware network for efficient RGB-D semantic segmentation[J]. IEEE Sensors Journal, 2023, 23(19): 23512–23521. doi: 10.1109/JSEN.2023.3304637.
[14]	李晓玲, 陈后金, 李艳凤, 等. 多重关系感知的红外与可见光图像融合网络[J]. 电子与信息学报, 2024, 46(5): 2217–2227. doi: 10.11999/JEIT231062. LI Xiaoling, CHEN Houjin, LI Yanfeng, et al. Infrared and visible image fusion network with multi-relation perception[J]. Journal of Electronics & Information Technology, 2024, 46(5): 2217–2227. doi: 10.11999/JEIT231062.
[15]	DENG Fuqin, FENG Hua, LIANG Mingjian, et al. FEANet: Feature-enhanced attention network for RGB-thermal real-time semantic segmentation[C]. IEEE/RSJ International Conference on Intelligent Robots and Systems, Prague, Czech Republic, 2021: 4467–4473. doi: 10.1109/iros51168.2021.9636084.
[16]	HA Qishen, WATANABE K, KARASAWA T, et al. MFNet: Towards real-time semantic segmentation for autonomous vehicles with multi-spectral scenes[C]. 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems, Vancouver, Canada, 2017: 5108–5115. doi: 10.1109/iros.2017.8206396.
[17]	ZHOU Wujie, LIN Xinyang, LEI Jingsheng, et al. MFFENet: Multiscale feature fusion and enhancement network for RGB–thermal urban road scene parsing[J]. IEEE Transactions on Multimedia, 2022, 24: 2526–2538. doi: 10.1109/TMM.2021.3086618.
[18]	LIU Jinfu, ZHOU Wujie, CUI Yueli, et al. GCNet: Grid-like context-aware network for RGB-thermal semantic segmentation[J]. Neurocomputing, 2022, 506: 60–67. doi: 10.1016/j.neucom.2022.07.041.
[19]	ZHOU Wujie, DONG Shaohua, XU Caie, et al. Edge-aware guidance fusion network for RGB-thermal scene parsing[C]. The 36th AAAI Conference on Artificial Intelligence,Online, 2022: 3571–3579. doi: 10.1609/aaai.v36i3.20269. .
[20]	ZHANG Jiaming, LIU Ruiping, SHI Hao, et al. Delivering arbitrary-modal semantic segmentation[C]. The IEEE/CVF Conference on Computer Vision and Pattern Recognition, Vancouver, Canada, 2023: 1136–1147. doi: 10.1109/cvpr52729.2023.00116.
[21]	周国宇, 张菁, 闫伊, 等. 聚焦注意力与紧致特征融合Transformer的城市遥感影像语义分割[J]. 电子与信息学报, 2025, 47(12): 4790–4800. doi: 10.11999/JEIT250812. ZHOU Guoyu, ZHANG Jing, YAN Yi, et al. A focused attention and feature compact fusion transformer for semantic segmentation of urban remote sensing images[J]. Journal of Electronics & Information Technology, 2025, 47(12): 4790–4800. doi: 10.11999/JEIT250812.
[22]	HAO Zhiwei, XIAO Zhongyu, LUO Yong, et al. PrimKD: Primary modality guided multimodal fusion for RGB-D semantic segmentation[C]. The 32nd ACM International Conference on Multimedia, Melbourne, Australia, 2024: 1943–1951. doi: 10.1145/3664647.3681253.
[23]	FERROD R, DANTAS C F, DI CARO L, et al. Revisiting cross-modal knowledge distillation: A disentanglement approach for RGBD semantic segmentation[C]. Joint European Conference on Machine Learning and Knowledge Discovery in Databases, Porto, Portugal, 2025: 492–508. doi: 10.1007/978-3-032-06078-5_28.
[24]	ZHOU Wujie, XIAO Yuxiang, LIU Yuanyuan, et al. FIMKD: Feature-implicit mapping knowledge distillation for RGB-D indoor scene semantic segmentation[J]. IEEE Transactions on Artificial Intelligence, 2024, 5(12): 6488–6499. doi: 10.1109/TAI.2024.3452052.
[25]	GUO Xiaodong, ZHOU Wujie, and LIU Tong. Contrastive learning-based knowledge distillation for RGB-thermal urban scene semantic segmentation[J]. Knowledge-Based Systems, 2024, 292: 111588. doi: 10.1016/j.knosys.2024.111588.
[26]	XIE Enze, WANG Wenhai, YU Zhiding, et al. SegFormer: Simple and efficient design for semantic segmentation with transformers[C]. The 35th International Conference on Neural Information Processing Systems, Online,2021: 924. .
[27]	SZEGEDY C, VANHOUCKE V, IOFFE S, et al. Rethinking the inception architecture for computer vision[C]. The IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, USA, 2016: 2818–2826. doi: 10.1109/cvpr.2016.308.
[28]	MILLETARI F, NAVAB N, and AHMADI S A. V-Net: Fully convolutional neural networks for volumetric medical image segmentation[C]. 2016 4th International Conference on 3D Vision, Stanford, USA, 2016: 565–571. doi: 10.1109/3dv.2016.79.
[29]	DENG Jia, DONG Wei, SOCHER R, et al. ImageNet: A large-scale hierarchical image database[C]. 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, USA, 2009: 248–255. doi: 10.1109/cvpr.2009.5206848.
[30]	BOTTOU L. Large-scale machine learning with stochastic gradient descent[C]. 19th International Conference on Computational Statistics, Paris, France, 2010: 177–186. doi: 10.1007/978-3-7908-2604-3_16.
[31]	SHIN U, LEE K, KWEON I S, et al. Complementary random masking for RGB-thermal semantic segmentation[C]. 2024 IEEE International Conference on Robotics and Automation, Yokohama, Japan, 2024: 11110–11117. doi: 10.1109/icra57147.2024.10611200.
[32]	YIN Bowen, ZHANG Xuying, LI Zhongyu, et al. DFormer: Rethinking RGBD representation learning for semantic segmentation[C]. The 12th International Conference on Learning Representations, Vienna, Austria, 2024.