面向HSI与LiDAR融合分类的决策-学习-修正网络

王浩宇; 刘诺菲; 程玉虎; 刘晓敏; 王雪松

doi:10.11999/JEIT260362

面向HSI与LiDAR融合分类的决策-学习-修正网络

doi: 10.11999/JEIT260362 cstr: 32379.14.JEIT260362

中国矿业大学信息与控制工程学院，江苏徐州 221116

基金项目: 国家自然科学基金(62303468)，徐州科技项目(KC2025123)，国家自然科学基金(62373364, 62573416, 62303469)

详细信息

作者简介:
王浩宇：男，副教授，研究方向为高光谱图像分类、多模态融合、机器学习

刘诺菲：女，在读硕士生，研究方向为强化学习、多模态融合、深度学习

程玉虎：男，教授，研究方向为强化学习、图像处理、深度学习

刘晓敏：女，副教授，研究方向为强化学习、多模态融合、图像处理

王雪松：女，教授，研究方向为强化学习、图像处理、深度学习

通讯作者:
王雪松　wangxuesongcumt@163.com

中图分类号: TP18
计量
- 文章访问数: 26
- HTML全文浏览量: 11
- PDF下载量: 2
- 被引次数: 0
出版历程
- 收稿日期: 2026-03-30
- 修回日期: 2026-07-03
- 录用日期: 2026-07-03
- 网络出版日期: 2026-07-14

Decision Learning Correction Network for HSI and LiDAR Fusion Classification

School of Information and Control Engineering, China University of Mining and Technology, Xuzhou 221116, China

Funds: National Natural Science Foundation of China (62303468), Science and Technology Program of Xuzhou (KC2025123), National Natural Science Foundation of China (62373364, 62573416, 62303469)

摘要

摘要: 高光谱图像与LiDAR在光谱表征和空间结构刻画方面具有天然互补性，其有效融合被认为是提升遥感地物分类性能的重要途径。然而，现有方法多将融合过程建模为固定映射下的一次性静态聚合，隐含地假设同一融合策略能够适用于不同区域，因而难以应对遥感影像中普遍存在的空间异质性和样本复杂性。为此，本文提出一种决策-学习-修正网络，将传统静态融合重构为面向局部上下文的序列决策过程使模型能够在连续交互中学习上下文依赖的自适应融合策略。针对关键样本易被忽视的问题，构建关键样本导向采样模块，依据样本决策困难度提高复杂区域样本的训练参与度。针对融合动作可能破坏模态特性的问题，设计模态保真约束机制，对不合理动作进行修正，以保证融合特征质量。实验结果表明，所提方法在Houston2013、Trento和MUUFL三个基准数据集上的总体精度分别提升0.82%、0.38%和1.76%，验证了该方法在复杂场景下的有效性与优越性。
- 高光谱图像 /
- LiDAR /
- 多模态融合 /
- 序贯决策优化 /
- 分类
Abstract: Objective Hyperspectral images(HSI) and LiDAR data provide complementary information for land-cover classification. HSI contains rich spectral responses for material discrimination, while LiDAR supplies elevation and structural information for spatial perception. However, most existing fusion methods treat multimodal fusion as a static aggregation process, assuming that a fixed fusion rule is suitable for all pixels and regions. This assumption is difficult to satisfy in complex remote sensing scenes, where class boundaries and cross-modal heterogeneous regions show higher information density but occupy only a small proportion of samples (Fig.1). To address this problem, this paper proposes a Decision Learning Correction Network (DLCN), which transforms static HSI-LiDAR fusion into a context-dependent sequential decision-making process. Methods The proposed DLCN consists of feature extraction, fusion decision learning, and final classification. First, HSI and LiDAR are processed by two parallel branches to extract spectral-spatial features and elevation-structural features, respectively. Then, the extracted features are concatenated as the current fusion state and input into an Actor-Critic framework. The Actor network generates fusion actions to dynamically adjust modal contributions, while the Critic network evaluates the long-term value of each action for final classification. To improve the learning of difficult samples, a key-sample-oriented sampling module assigns higher sampling probabilities to samples with larger modal fidelity losses. Meanwhile, a modal fidelity constraint mechanism evaluates spectral fidelity, feature consistency, structural preservation, and resolution matching, and corrects destructive actions during fusion. Through this closed-loop structure, DLCN realizes dynamic fusion action generation, evaluation, and correction (Fig. 2). Results and Discussions Experiments are conducted on Houston2013, Trento, and MUUFL datasets. DLCN achieves the best OA values of 97.85%, 99.58%, and 94.38% on the three datasets, respectively, outperforming CHNet, DSymFuser, mPMCL, MEDFN, S3F2Net, and MSAF. The classification maps show that DLCN effectively reduces misclassification in class-boundary, mixed land-cover, and structurally complex regions, producing results closer to the ground-truth maps on the three datasets (Fig. 3-Fig. 5). Ablation results demonstrate that the value-guided policy optimization mechanism, key-sample-oriented sampling module, and modal fidelity constraint mechanism all contribute to performance improvement. Compared with the baseline model, the complete DLCN obtains consistent OA gains on Houston2013, Trento, and MUUFL, verifying the effectiveness of the proposed decision-learning-correction framework. The temporal analysis shows that DLCN gradually improves class accuracy while maintaining stable spectral-angle variation during sequential decision steps (Fig. 6). In addition, DLCN achieves inference times of 1.32 s, 0.86 s, and 2.23 s on the three datasets, respectively, ranking first among the compared methods. This indicates that the introduced Actor-Critic decision mechanism and modal fidelity constraints can be effectively converted into classification gains without causing excessive computational burden. Conclusions This paper proposes DLCN for HSI and LiDAR fusion classification. Different from static fusion methods, DLCN models multimodal fusion as a sequential decision-making process and dynamically adjusts fusion strategies according to local context. The closed-loop design enables the model to generate, evaluate, and correct fusion actions. Experimental results show that DLCN achieves more accurate classification maps in heterogeneous scenes, and the temporal analysis further confirms the stability of the sequential decision process. Future work will focus on more fine-grained feature representation and more robust policy optimization to improve generalization in complex remote sensing scenes.
- Hyperspectral images /
- LiDAR /
- Multimodal fusion /
- Sequential decision optimization /
- Classification

HTML全文

图 1 遥感数据的信息密度不均(以Houston2013数据集为例):采用7×7像素滑动窗口遍历遥感影像，计算每个局部区域的熵值；基于全部局部熵值开展统计分析并生成直方图，同时将局部熵值以伪彩色映射至影像对应空间位置，得到如下子图：(a)全局信息密度分布：为局部熵值的统计直方图。其分布呈左高右低的长尾特征：约80%窗口的熵值集中于1.5～2.5区间，熵大于3的高信息窗口占比约5%，此类区域对应类别边界像元、跨模态异质区域等关键样本。(b)局部信息熵:为局部熵的伪彩色渲染图(颜色亮度与熵值正相关，亮度越高代表区域纹理越复杂、信息越丰富)，图中四个区域局部信息熵直观呈现了遥感影像信息密度的空间不均匀性。

下载: 全尺寸图片幻灯片

图 2 决策-学习-修正网络

下载: 全尺寸图片幻灯片

图 3 分类图(Houston2013)

下载: 全尺寸图片幻灯片

图 4 分类图(MUUFL)

下载: 全尺寸图片幻灯片

图 5 分类图(Trento)

下载: 全尺寸图片幻灯片

图 6 DLCN在一段时间步长上

下载: 全尺寸图片幻灯片

表 1 Houston2013数据集的分类精度

类别(%)	CHNet	DSymFuser	mPMCL	MEDFN	S3F2Net	MSAF	DLCN
Healthy grass	100	88.98	100	84.25	98.39	98.29	97.15
Stressed grass	86.00	98.68	98.21	98.33	93.89	86.47	97.65
Synthetic grass	100	99.60	100	99.86	100	100	100
Trees	98.30	99.43	94.13	99.52	99.91	97.82	97.73
Soil	100	100	100	100	99.91	99.81	100
Water	100	100	100	92.62	100	99.30	100
Residential	98.13	99.81	98.69	92.82	99.91	98.41	99.72
Commercial	96.39	90.60	94.59	84.81	96.87	98.96	88.32
Road	99.81	87.82	89.24	68.69	43.34	94.33	98.68
Highway	90.44	94.21	100	84.51	100	97.97	100
Railway	96.58	97.15	92.13	91.58	100	98.96	99.43
Parking lot 1	100	96.06	98.27	81.02	99.52	97.31	97.60
Parking lot 2	100	92.98	95.08	95.95	100	99.65	96.14
Tennis court	100	98.79	100	99.77	100	100	100
Running track	100	100	100	100	100	99.36	99.79
OA	97.03	95.72	96.88	90.23	94.07	97.21	97.85
AA	97.71	96.28	97.36	91.58	95.44	97.78	98.15
Kappa	96.78	95.35	96.61	89.44	93.56	96.97	97.67

下载: 导出CSV

表 3 MUUFL数据集的分类精度

类别(%)	CHNet	DSymFuser	mPMCL	MEDFN	S3F2Net	MSAF	DLCN
Trees	91.70	93.99	92.20	79.11	91.71	92.87	96.62
Mostly grass	86.41	88.13	87.79	92.57	87.62	86.60	89.25
Mixed ground surface	84.85	87.12	81.61	53.30	83.02	79.93	90.54
Dirt and sand	96.36	96.84	97.37	96.12	95.40	96.90	95.05
Road	89.20	91.19	86.40	82.33	90.21	89.00	93.24
Water	100	99.68	100	99.68	99.37	99.68	99.68
Building shadow	93.13	96.74	93.33	96.45	95.54	93.52	97.36
Building	95.34	95.42	97.72	86.81	90.03	95.75	96.29
Sidewalk	84.21	90.36	85.10	65.91	77.48	84.78	79.51
Yellow curb	100	96.97	100	81.82	96.97	93.94	90.91
Cloth panels	99.16	99.16	100	100	100	99.16	99.16
OA	90.61	92.62	90.51	79.25	89.88	90.57	94.38
AA	92.76	94.15	92.87	84.92	91.58	92.01	93.42
Kappa	87.68	90.27	87.55	73.61	86.74	87.60	92.52

下载: 导出CSV

表 2 Trento数据集的分类精度

类别(%)	CHNet	DSymFuser	mPMCL	MEDFN	S3F2Net	MSAF	DLCN
Apple trees	99.46	96.72	94.55	96.12	100	99.90	99.28
Buildings	95.93	99.24	99.78	98.41	99.68	99.46	98.70
Ground	95.19	97.86	99.73	92.86	87.97	98.13	97.06
Woods	99.98	100	100	100	100	100	100
Vineyard	100	100	100	99.86	98.35	99.72	100
Roads	97.05	97.44	97.71	98.36	94.00	95.28	98.46
OA	99.17	99.20	99.01	99.01	98.62	99.32	99.58
AA	97.94	98.55	98.63	97.60	96.67	98.75	98.92
Kappa	98.89	98.93	98.68	98.67	98.15	99.09	99.44

下载: 导出CSV

表 4 消融模型

组件	Baseline-A	Baseline-B	Baseline-C	Baseline-D	DLCN
OMVGS	×	×	√	√	√
KSOSM	×	×	×	√	√
MFCM	×	×	×	×	√

下载: 导出CSV

表 5 不同组件对OA的影响(%)

数据集	Baseline-A	Baseline-B	Baseline-C	Baseline-D	DLCN
Houston2013	95.71	94.61	96.15	97.38	97.85
Trento	96.46	97.28	98.27	99.07	99.58
MUUFL	91.09	90.21	92.54	93.86	94.38

下载: 导出CSV

表 6 不同方法对每个数据集的计算代价

数据集	计算代价	CHNet	DSymFuser	mPMCL	MEDFN	S3F2Net	MSAF	DLCN
Houston 2013	Time(s)	3.07	6.42	15.36	2.34	1.99	1.61	1.32
	FLOPs(M)	334.28	75.43	7.59	165.76	57.50	3.92	161.42
	params(M)	23.78	0.85	0.17	1.45	0.28	0.05	0.79
Trento	Time(s)	7.83	5.35	25.96	2.14	0.93	7.167	0.86
	FLOPs(M)	334.18	41.36	27.69	164.98	10.89	3.76	160.25
	params(M)	23.73	0.47	0.18	1.45	0.27	0.05	0.79
MUUFL	Time(s)	14.31	9.40	34.36	3.56	2.56	14.75	2.23
	FLOPs(M)	334.23	41.85	7.57	164.54	8.68	3.77	160.40
	params(M)	23.75	0.48	0.17	1.45	0.28	0.05	0.79

下载: 导出CSV

参考文献(22)

[1]	ZHAO Xudong, ZHANG Mengmeng, TAO Ran, et al. Fractional Fourier image transformer for multimodal remote sensing data classification[J]. IEEE Transactions on Neural Networks and Learning Systems, 2024, 35(2): 2314–2326. doi: 10.1109/TNNLS.2022.3189994.
[2]	WU Xin, HONG Danfeng, and CHANUSSOT J. Convolutional neural networks for multimodal remote sensing data classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60: 5517010. doi: 10.1109/TGRS.2021.3124913.
[3]	VIVONE G, DENG Liangjian, DENG Shangqi, et al. Deep learning in remote sensing image fusion: Methods, protocols, data, and future perspectives[J]. IEEE Geoscience and Remote Sensing Magazine, 2025, 13(1): 269–310. doi: 10.1109/MGRS.2024.3495516.
[4]	LUO Fulin, HUA Yiyan, FU Chuan, et al. MMD-MLP: LiDAR-guided hyperspectral data classification using local-global directional-MLP with multiresolution multiscale representation[J]. IEEE Transactions on Geoscience and Remote Sensing, 2025, 63: 5508414. doi: 10.1109/TGRS.2025.3550370.
[5]	DUAN Puhong, LUO Yichen, KANG Xudong, et al. LaMamba: Linear attention mamba for hyperspectral image denoising[J]. IEEE Transactions on Geoscience and Remote Sensing, 2025, 63: 5527113. doi: 10.1109/TGRS.2025.3613739.
[6]	FU Chuan, DU Bo, and ZHANG Liangpei. ReSC-net: Hyperspectral image classification based on attention-enhanced residual module and spatial-channel attention[J]. IEEE Transactions on Geoscience and Remote Sensing, 2024, 62: 5518615. doi: 10.1109/TGRS.2024.3402364.
[7]	YU Chunyan, WANG Hande, SONG Meiping, et al. Interactive graph-based distillation integrated meta-learning network for hyperspectral image incremental classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2026, 64: 5500316. doi: 10.1109/TGRS.2025.3647656.
[8]	DONG Wenqian, YANG Teng, QU Jiahui, et al. Joint contextual representation model-informed interpretable network with dictionary aligning for hyperspectral and LiDAR classification[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2023, 33(11): 6804–6818. doi: 10.1109/TCSVT.2023.3268757.
[9]	YANG J X, ZHOU Jun, WANG Jing, et al. LiDAR-guided cross-attention fusion for hyperspectral band selection and image classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2024, 62: 5515815. doi: 10.1109/TGRS.2024.3389651.
[10]	YANG Bin, WANG Xuan, XING Ying, et al. Modality fusion vision transformer for hyperspectral and LiDAR data collaborative classification[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2024, 17: 17052–17065. doi: 10.1109/JSTARS.2024.3415729.
[11]	HE Ziping, ZHU Qianglin, WANG Wei, et al. Multilevel fusion network based on mix hybrid attention for hyperspectral and LiDAR image classification[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2026, 19: 470–483. doi: 10.1109/JSTARS.2025.3628896.
[12]	WANG Minhui, SUN Yaxiu, XIANG Jianhong, et al. Joint classification of hyperspectral and LiDAR data based on adaptive gating mechanism and learnable transformer[J]. Remote Sensing, 2024, 16(6): 1080. doi: 10.3390/rs16061080.
[13]	WANG Haoyu, CHENG Yuhu, LIU Xiaomin, et al. Reinforcement learning based Markov edge decoupled fusion network for fusion classification of hyperspectral and LiDAR[J]. IEEE Transactions on Multimedia, 2024, 26: 7174–7187. doi: 10.1109/TMM.2024.3360717.
[14]	SCHULMAN J, WOLSKI F, DHARIWAL P, et al. Proximal policy optimization algorithms[J]. arXiv preprint arXiv: 1707.06347, 2017. doi: 10.48550/arXiv.1707.06347.
[15]	DEBES C, MERENTITIS A, HEREMANS R, et al. Hyperspectral and LiDAR data fusion: Outcome of the 2013 GRSS data fusion contest[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2014, 7(6): 2405–2418. doi: 10.1109/JSTARS.2014.2305441.
[16]	RASTI B, GHAMISI P, and GLOAGUEN R. Hyperspectral and LiDAR fusion using extinction profiles and total variation component analysis[J]. IEEE Transactions on Geoscience and Remote Sensing, 2017, 55(7): 3997–4007. doi: 10.1109/TGRS.2017.2686450.
[17]	ZHANG Mengmeng, LI Wei, TAO Ran, et al. Information fusion for classification of hyperspectral and LiDAR data using IP-CNN[J]. IEEE Transactions on Geoscience and Remote Sensing, 2022, 60: 5506812. doi: 10.1109/TGRS.2021.3093334.
[18]	NI Kang, XIE Yunan, ZHAO Guofeng, et al. Coarse-to-fine high-order network for hyperspectral and LiDAR classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2025, 63: 5509716. doi: 10.1109/TGRS.2025.3554802.
[19]	CHANG Honghao, BI Haixia, LI Fan, et al. Deep symmetric fusion transformer for multimodal remote sensing data classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2024, 62: 5644115. doi: 10.1109/TGRS.2024.3476975.
[20]	LIU Hui, HUANG Chenjia, XIE Tao, et al. Positive matching benefits fusion: A novel contrastive learning framework for hyperspectral and LiDAR data classification[J]. IEEE Transactions on Geoscience and Remote Sensing, 2026, 64: 5502218. doi: 10.1109/TGRS.2026.3654168.
[21]	WANG Xianghai, SONG Liyang, FENG Yining, et al. S3F2Net: Spatial-spectral-structural feature fusion network for hyperspectral image and LiDAR data classification[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2025, 35(5): 4801–4815. doi: 10.1109/TCSVT.2025.3525734.
[22]	SHI Lulu, LI Chunchao, ZENG Zhengchao, et al. Masked self-attention fusion network for joint classification of hyperspectral and LiDAR data[J]. IEEE Transactions on Image Processing, 2026, 35: 346–360. doi: 10.1109/TIP.2025.3648926.