U-ST-M3: Uncertainty-Aware Spatio-Temporal Multi-Modal Multi-Hypothesis Beam Prediction for Low-Overhead mmWave V2I Communications

GU Peng; ZHOU Enzhi; FAN Zhiqiang; ZHENG Zhi; LAN Ping

doi:10.11999/JEIT260670

Article Contents

Article Navigation > Journal of Electronics & Information Technology > 2026 >

GU Peng, ZHOU Enzhi, FAN Zhiqiang, ZHENG Zhi, LAN Ping. U-ST-M3: Uncertainty-Aware Spatio-Temporal Multi-Modal Multi-Hypothesis Beam Prediction for Low-Overhead mmWave V2I Communications[J]. Journal of Electronics & Information Technology. doi: 10.11999/JEIT260670

Citation:

GU Peng, ZHOU Enzhi, FAN Zhiqiang, ZHENG Zhi, LAN Ping. U-ST-M3: Uncertainty-Aware Spatio-Temporal Multi-Modal Multi-Hypothesis Beam Prediction for Low-Overhead mmWave V2I Communications[J]. Journal of Electronics & Information Technology. doi: 10.11999/JEIT260670

Citation:

GU Peng, ZHOU Enzhi, FAN Zhiqiang, ZHENG Zhi, LAN Ping. U-ST-M3: Uncertainty-Aware Spatio-Temporal Multi-Modal Multi-Hypothesis Beam Prediction for Low-Overhead mmWave V2I Communications[J]. Journal of Electronics & Information Technology. doi: 10.11999/JEIT260670

PDF( 1064 KB)

U-ST-M3: Uncertainty-Aware Spatio-Temporal Multi-Modal Multi-Hypothesis Beam Prediction for Low-Overhead mmWave V2I Communications

doi: 10.11999/JEIT260670 cstr: 32379.14.JEIT260670

GU Peng^1
,,
ZHOU Enzhi^2
,,
FAN Zhiqiang^1
,,
ZHENG Zhi^3
,,
LAN Ping^{1, 4
,
,}

1.
Xizang University, Lhasa, 850000, China
2.
Xihua University, Chengdu, 610039, China
3.
University of Electronic Science and Technology of China, Chengdu, 611731, China
4.
Key Laboratory of Border Security and Intelligent Governance of the Xizang Autonomous Region, Lhasa, 850000, China

Funds: Key Research and Transformation Projects of the Science and Technology Program of the Xizang Autonomous Region(XZ202601ZY0163), Guangdong Basic and Applied Basic Research Fund Project(2024A1515140025), Xizang University Graduate High-Level Talent Training Program(2026-GSP-S089)

Accepted Date: 2026-11-02
Rev Recd Date: 2026-11-02

Available Online: 2026-11-09

Abstract

Abstract

Objective Beam management in millimeter-wave (mmWave) vehicular networks requires accurate beam selection under rapidly changing propagation conditions. Vehicle motion, blockage, and multipath changes can alter the preferred beam, while frequent channel measurements and exhaustive beam sweeping introduce training and search overhead. Environmental sensing offers complementary information about the communication scene, but heterogeneous observations differ in representation and reliability. A prediction framework therefore needs to integrate their temporal and cross-modal information while accounting for variations in their contributions. This study investigates environment-assisted beam prediction for vehicle-to-infrastructure (V2I) communications and formulates it as a candidate beam generation task. Instead of requiring a single prediction to identify the optimal beam, the task aims to generate a compact candidate set that covers the optimal beam and maintains small beam-index deviations. The objective is to improve candidate quality with a relatively small model and support subsequent beam management when historical multimodal inputs are already available. Methods An uncertainty-aware spatio-temporal multimodal multi-hypothesis Mamba framework, termed U-ST-M3, is proposed for the dynamic V2I scenario (Fig. 1). Red, Green, and Blue (RGB) images, Light Detection and Ranging (LiDAR), Radio Detection and Ranging (Radar), Global Positioning System (GPS) information, and historical Radio Frequency (RF) power measurements are organized into four feature branches: a joint RGB–LiDAR branch, a Radar branch, a GPS branch, and an RF branch (Fig. 2). In the joint branch, LiDAR-derived Bird’s-Eye View (BEV) features are used to gate visual features without assuming pixel-level geometric correspondence. GPS-conditioned modulation enhances the joint and Radar representations, while the RF branch captures energy and temporal-difference information. Future RF measurements are used only for supervision and training constraints. Sample-dependent uncertainty proxies determine branch fusion weights, and a Mamba encoder models the fused historical sequence. A shared decoder generates multiple hypotheses, each with beam logits and a hypothesis score. During inference, the three highest-ranked beams from the highest-scoring hypothesis form the candidate set. Training combines DBA-oriented classification, coverage, index-distance, RF-utility, and spectral-efficiency losses with hypothesis scoring, diversity, and usage-balance terms. Results and Discussions Experiments are conducted on Scenarios 32-34 of the DeepSense 6G dataset, with 80% of the data used for training, 10% for validation, and 10% for testing. U-ST-M3 achieves an Overall DBA Score of 0.9012, close to VLM’s 0.9016, and the highest reported Overall Top-3 Accuracy of 0.7842, exceeding ICMFE by 9.62 percentage points (Tables 1-2). Its nighttime DBA scores outperform the listed baselines, while daytime performance remains weaker. The model has 14.42 million parameters, the fewest among the listed methods, and requires 12.63 billion floating-point operations. Average forward-pass latency is measured on an NVIDIA RTX 4090 GPU with a batch size of one and 32-bit floating-point (FP32) precision. The measured latency is lower than those of TII, MMFF, and ICMFE but slower than Vision-Position. Baseline accuracy and complexity values come from the literature under their respective input configurations; available latencies are measured separately. Ablations show that additional modalities or fusion components do not consistently improve DBA (Table 3). Adding the full loss to the six-hypothesis configuration raises DBA from 0.8382 to 0.8590; the final four-hypothesis model reaches 0.9012 after further tuning, so this gain cannot be isolated to hypothesis count. On the communication side, an illustrative serial scanning configuration is considered with a 15 kHz subcarrier spacing and a normal cyclic prefix, yielding an average OFDM symbol duration of approximately 71.43 μs. Assuming four measurement symbols per beam, scanning all 64 beams requires approximately 18.29 ms of cumulative measurement time. The measured candidate generation latency is lower than this estimate under the specified assumptions. This comparison is limited to model inference latency and the estimated scanning time; end-to-end beam acquisition latency is not measured. Historical input acquisition, candidate confirmation, and reporting overhead require separate consideration in practical deployment. Conclusions U-ST-M3 integrates modality-specific encoding, GPS-conditioned modulation, uncertainty-aware fusion, temporal modeling, and multi-hypothesis prediction into a unified candidate beam generation framework. The results demonstrate high Top-3 candidate coverage and competitive DBA performance with a relatively small parameter count. The reported latency and the comparison with the specified serial scanning estimate support its potential for low-overhead beam management when historical inputs are available. The framework provides a practical basis for further investigation of candidate confirmation and recovery procedures. Future work will examine daytime beam-neighborhood consistency, reduce computational cost, and evaluate acquisition and feedback overhead in an explicitly defined communication process.
- Millimeter-wave Beam Prediction,
- Multimodal Learning,
- Uncertainty-aware Fusion,
- Spatio-temporal Mamba,
- Multi-hypothesis Prediction

FullText(HTML)

References(32)

References

[1]	XUE Qing, JI Chengwang, MA Shaodan, et al. A survey of beam management for mmWave and THz communications towards 6G[J]. IEEE Communications Surveys & Tutorials, 2024, 26(3): 1520–1559. doi: 10.1109/COMST.2024.3361991.
[2]	KHAN M Q, GABER A, SCHULZ P, et al. Machine learning for millimeter wave and terahertz beam management: A survey and open challenges[J]. IEEE Access, 2023, 11: 11880–11902. doi: 10.1109/ACCESS.2023.3242582.
[3]	LIU Fan, CUI Yuanhao, MASOUROS C, et al. Integrated sensing and communications: Toward dual-functional wireless networks for 6G and beyond[J]. IEEE Journal on Selected Areas in Communications, 2022, 40(6): 1728–1767. doi: 10.1109/JSAC.2022.3156632.
[4]	HENG Yuqiang and ANDREWS J G. Machine learning-assisted beam alignment for mmWave systems[J]. IEEE Transactions on Cognitive Communications and Networking, 2021, 7(4): 1142–1155. doi: 10.1109/TCCN.2021.3078147.
[5]	ZHANG Haotian, GAO Shijian, CHENG Xiang, et al. Integrated sensing and communications toward proactive beamforming in mmWave V2I via multi-modal feature fusion (MMFF)[J]. IEEE Transactions on Wireless Communications, 2024, 23(11): 15721–15735. doi: 10.1109/TWC.2024.3401686.
[6]	仲伟志, 何艺, 段洪涛, 等. 可重构智能表面辅助的V2I通信系统联合波束赋形算法[J]. 电子与信息学报, 2024, 46(8): 3117–3125. doi: 10.11999/JEIT231324. ZHONG Weizhi, HE Yi, DUAN Hongtao, et al. Joint beamforming algorithm for reconfigurable intelligent surface-aided V2I communication system[J]. Journal of Electronics & Information Technology, 2024, 46(8): 3117–3125. doi: 10.11999/JEIT231324.
[7]	MORAIS J, BCHBOODI A, PEZESHKI H, et al. Position-aided beam prediction in the real world: How useful GPS locations actually are?[C]. ICC 2023 - IEEE International Conference on Communications, Rome, Italy, 2023: 1824–1829. doi: 10.1109/ICC45041.2023.10278998.
[8]	ALKHATEEB A, CHARAN G, OSMAN T, et al. DeepSense 6G: A large-scale real-world multi-modal sensing and communication dataset[J]. IEEE Communications Magazine, 2023, 61(9): 122–128. doi: 10.1109/MCOM.006.2200730.
[9]	JIANG Shuaifeng and ALKHATEEB A. Computer vision aided beam tracking in a real-world millimeter wave deployment[C]. 2022 IEEE Globecom Workshops (GC Wkshps), Rio de Janeiro, Brazil, 2022: 142–147. doi: 10.1109/GCWkshps56602.2022.10008648.
[10]	JIANG Shuaifeng, CHARAN G, and ALKHATEEB A. LiDAR aided future beam prediction in real-world millimeter wave V2I communications[J]. IEEE Wireless Communications Letters, 2023, 12(2): 212–216. doi: 10.1109/LWC.2022.3219409.
[11]	DEMIRHAN U and ALKHATEEB A. Radar aided 6G beam prediction: Deep learning algorithms and real-world demonstration[C]. 2022 IEEE Wireless Communications and Networking Conference (WCNC), Austin, USA, 2022: 2655–2660. doi: 10.1109/WCNC51071.2022.9771564.
[12]	孙昆阳, 姚睿, 祝汉城, 等. 一种测试时间自适应的夜间图像辅助波束预测方法[J]. 电子与信息学报, 2025, 47(12): 5156–5165. doi: 10.11999/JEIT250530. SUN Kunyang, YAO Rui, ZHU Hancheng, et al. A test-time adaptive method for nighttime image-aided beam prediction[J]. Journal of Electronics & Information Technology, 2025, 47(12): 5156–5165. doi: 10.11999/JEIT250530.
[13]	CHARAN G, OSMAN T, HREDZAK A, et al. Vision-position multi-modal beam prediction using real millimeter wave datasets[C]. 2022 IEEE Wireless Communications and Networking Conference (WCNC), Austin, USA, 2022: 2727–2731. doi: 10.1109/WCNC51071.2022.9771835.
[14]	CHARAN G, HREDZAK A, STODDARD C, et al. Towards real-world 6G drone communication: Position and camera aided beam prediction[C]. 2022 IEEE Global Communications Conference (GLOBECOM), Rio de Janeiro, Brazil, 2022: 2951–2956. doi: 10.1109/GLOBECOM48099.2022.10000718.
[15]	CHARAN G, DEMIRHAN U, MORAIS J, et al. Multi-modal beam prediction challenge 2022: Towards generalization[EB/OL]. https://arxiv.org/abs/2209.07519, 2022.
[16]	TIAN Yu, ZHAO Qiyang, KHERROUBI Z E A, et al. Multimodal transformers for wireless communications: A case study in beam prediction[J]. ITU Journal on Future and Evolving Technologies, 2023, 4(3): 461–471. doi: 10.52953/JWRA8095.
[17]	SHI Binpu, LI Min, ZHAO Mingmin, et al. Multimodal deep learning empowered millimeter-wave beam prediction[C]. 2024 IEEE 99th Vehicular Technology Conference (VTC2024-Spring), Singapore, Singapore, 2024: 1–6. doi: 10.1109/VTC2024-Spring62846.2024.10683225.
[18]	TARIQ S, ARFETO B E, KHALID U, et al. Deep quantum-transformer networks for multimodal beam prediction in ISAC systems[J]. IEEE Internet of Things Journal, 2024, 11(18): 29387–29401. doi: 10.1109/JIOT.2024.3420455.
[19]	CHENG Lu, ZHANG Hongliang, DI Boya, et al. Large language models empower multimodal integrated sensing and communication[J]. IEEE Communications Magazine, 2025, 63(5): 190–197. doi: 10.1109/MCOM.004.2400281.
[20]	ZHENG Can, HE Jiguang, CAI Guofa, et al. BeamLLM: Vision-empowered mmWave beam prediction with large language models[C]. 2025 IEEE 102nd Vehicular Technology Conference (VTC2025-Fall), Chengdu, China, 2025: 1–6. doi: 10.1109/VTC2025-Fall65116.2025.11309918.
[21]	赵川斌, 许伟华, 林博, 等. 融合视觉的多模态通信感知一体化关键技术及原型验证[J]. 电子与信息学报, 2026, 48(2): 487–498. doi: 10.11999/JEIT250685. ZHAO Chuanbin, XU Weihua, LIN Bo, et al. Vision enabled multimodal integrated sensing and communications: Key technologies and prototype validation[J]. Journal of Electronics & Information Technology, 2026, 48(2): 487–498. doi: 10.11999/JEIT250685.
[22]	CHARAN G, ALRABEIAH M, OSMAN T, et al. Camera based mmWave beam prediction: Towards multi-candidate real-world scenarios[J]. IEEE Transactions on Vehicular Technology, 2025, 74(4): 5897–5913. doi: 10.1109/TVT.2024.3506948.
[23]	PARK J, AHN J H, SEO J, et al. Occlusion-aware vision-aided beam tracking for multi-user V2I mmWave networks[C]. ICC 2025 - IEEE International Conference on Communications, Montreal, Canada, 2025: 2210–2216. doi: 10.1109/ICC52391.2025.11161281.
[24]	ZHU Qihao, WANG Yu, LI Wenmei, et al. Advancing multi-modal beam prediction with cross-modal feature enhancement and dynamic fusion mechanism[J]. IEEE Transactions on Communications, 2025, 73(9): 7931–7940. doi: 10.1109/TCOMM.2025.3548021.
[25]	VUCKOVIC K, HOSSEINI S M, and RAHNAVARD N. Revisiting performance metrics for multimodal mmWave beam prediction using deep learning[C]. MILCOM 2024 - 2024 IEEE Military Communications Conference, Washington, USA, 2024: 881–887. doi: 10.1109/MILCOM61039.2024.10773747.
[26]	GU A and DAO T. Mamba: Linear-time sequence modeling with selective state spaces[C]. The First Conference on Language Modeling (COLM), Philadelphia, USA, 2024: 1–32.
[27]	ZHANG Rongsheng, ZHANG Ruichen, LU Yang, et al. Mamba for wireless communications and networking: Principles and opportunities[J]. IEEE Communications Magazine, 2026: 1–7. doi: 10.1109/MCOM.001.2500436.
[28]	廖帝灵, 赖涛, 黄海风, 等. LightMamba: 一种轻量级Mamba用于高光谱图形和激光雷达数据联合分类网络[J]. 电子与信息学报, 2025, 47(12): 4937–4947. doi: 10.11999/JEIT250981. LIAO Diling, LAI Tao, HUANG Haifeng, et al. LightMamba: A lightweight Mamba network for the joint classification of HSI and LiDAR data[J]. Journal of Electronics & Information Technology, 2025, 47(12): 4937–4947. doi: 10.11999/JEIT250981.
[29]	SHI Huaguang, JIN Kaibo, REN Xiaoquan, et al. ChannelMamba: A mamba-driven selective state-space model for channel prediction of high-mobility MIMO in 6G IoT[J]. IEEE Transactions on Wireless Communications, 2026, 25: 5291–5305. doi: 10.1109/TWC.2025.3617502.
[30]	NI Jinhong, BAI Yalong, ZHANG Wei, et al. Deep equilibrium multimodal fusion[EB/OL]. https://arxiv.org/abs/2306.16645, 2023.
[31]	LI Wei, GAO Yunhao, ZHANG Mengmeng, et al. Asymmetric feature fusion network for hyperspectral and SAR image classification[J]. IEEE Transactions on Neural Networks and Learning Systems, 2023, 34(10): 8057–8070. doi: 10.1109/TNNLS.2022.3149394.
[32]	WANG Ji, TANG Bin, XIAO Jian, et al. When vision-language model (VLM) meets beam prediction: A multimodal contrastive learning framework[J]. IEEE Transactions on Vehicular Technology, 2026, 75(7): 15113–15118. doi: 10.1109/TVT.2026.3663461.