A Reinforcement Learning Driven Power Allocation Algorithm for Collocated MIMO Radar
-
摘要: 集中式多输入多输出(Multiple-input Multiple-output, MIMO)雷达资源分配算法可有效提升多目标跟踪精度，而传统优化算法仅优化下一时刻跟踪性能，难以实现全时域多目标累积跟踪精度提升；同时，传统优化算法的计算复杂度较高，难以满足资源分配过程的实时性要求。针对该问题，本文提出一种强化学习(Reinforcement Learning, RL)驱动的集中式 MIMO 雷达功率分配算法。首先，利用后验克拉美罗界(Posterior Cramér-Rao Lower Bound, PCRLB)构建RL模型的状态与奖励函数，将功率资源分配过程建模为马尔可夫决策过程(Markov Decision Process, MDP)，建立RL驱动的资源分配优化模型；其次，结合工程实际与算法收敛性要求，采用对决双深度 Q 网络(Dueling Double Deep Q-Network, D3QN)算法对模型进行求解。仿真结果表明，相较于固定功率分配与传统优化分配算法，所提算法可实现功率资源的自适应合理分配，显著提升全时域累积跟踪精度；并且训练后的网络可以根据状态进行实时决策输出资源分配结果，提升了算法实时性。
-
关键词:
- 强化学习 /
- 集中式MIMO雷达 /
- 功率资源分配 /
- 对决双深度 Q 网络算法
Abstract:Objective Traditional optimization based on power allocation algorithm for collocated MIMO radar has two fundamental limitations. First, it optimizes tracking performance only for the next time step, lacking a global view across the entire time horizon. This myopic strategy cannot achieve optimal multiple target tracking accuracy over a long duration, especially when target trajectories vary significantly. Second, their iterative solving process involves nonlinear constrained optimization, which incurs high computational complexity. In dynamic battlefield environments where target states change rapidly, such algorithms fail to meet real-time requirements. To address these issues, this paper proposes a reinforcement learning (RL) driven power allocation algorithm. Unlike traditional methods, the proposed approach formulates the problem as a Markov decision process that maximizes long-term cumulative rewards. The algorithm adaptively allocates limited power resources among multiple beams based on the current system state, balancing immediate tracking performance and future gains. Methods The posterior Cramér-Rao lower bound (PCRLB) is employed to quantify the theoretical tracking error lower bound for each target. The state space is constructed by combining the motion states (position and velocity) of all targets and normalized PCRLB from the previous allocation. The action space consists of discrete transmit power levels for each beam, subject to the total power budget and beam power limits. All feasible power allocation vectors are enumerated and encoded to reduce dimensionality. The reward function is defined as the negative weighted sum of normalized PCRLB, encouraging the agent to minimize tracking errors. The power allocation process is formulated as a Markov decision process (MDP). The Dueling Double Deep Q-Network (D3QN) algorithm is adopted to solve this MDP. D3QN integrates three key enhancements: (1) a double network training framework (decision Q-network and target Q-network) to stabilize learning; (2) a dueling architecture that decomposes the Q-value into state value and action advantage functions, improving action discrimination; (3) off-policy learning with experience replay, enabling efficient use of historical trajectories. The $ \varepsilon \text{-greedy} $strategy is used for exploration, with epsilon decaying over episodes. After offline training, the learned network outputs power allocation decisions in real-time given the current system state, without any iterative optimization. Results and Discussions Simulations are conducted with three targets following constant velocity models. Fixed power allocation yields the lowest tracking accuracy due to inefficient resource utilization. The traditional optimization method, which minimizes the immediate tracking error, achieves moderate accuracy but remains myopic. For D3QN with discount factor $ \gamma =0 $, the performance is nearly identical to the traditional method. In contrast, D3QN with $ \gamma =0.99 $ achieves significantly better full time horizon accuracy. Power allocation patterns reveal that the D3QN with $ \gamma =0.99 $ preemptively allocates more power to distant and low signal to noise ratio (SNR) targets earlier, while reducing redundant power to close and high SNR targets. The training curves show that $ \gamma =0.99 $ achieves a higher steady state cumulative reward, although with more oscillations due to the complexity of estimating future returns. Moreover, the trained D3QN network outputs decisions instantaneously, whereas traditional optimization requires solving a constrained optimization problem at each time step, providing a clear real-time advantage. Conclusions This paper proposes a RL driven power allocation algorithm for collocated MIMO radar multiple target tracking that overcomes the myopic and computationally intensive limitations of traditional optimization methods. The algorithm uses PCRLB to construct state and reward functions, models the allocation process as an MDP, and solves it using the D3QN algorithm. Simulation results demonstrate that the approach based on D3QN with a suitable discount factor ($ \gamma =0.99 $) significantly improves full time horizon target tracking accuracy. The improvement stems from the agent’s ability to learn a long-term optimal policy that preemptively allocates resources to future challenging targets. Furthermore, the trained network enables real-time decision, substantially reducing computational latency from iterative solving to instantaneous forward propagation. This work provides a new approach for intelligent radar resource management in complex battlefield environments. -
1 D3QN算法
1 初始化雷达、目标、仿真参数 2 初始化智能体 Agent，环境 Env 3 初始化经验回放池 Replaybuffer 4 For $ \text { epieode }=1: E $ do 5 初始化状态$ s_1 $ 6 For $ k=1: T $ do 7 将状态$ \text { Ak } $输入智能体Agent 8 依据$ s \text { - greedy } $策略选取动作$ a_{k} $ 9 与环境Env交互 10 得到奖励$ \eta $以及下一时刻状态$ \Delta+1 $ 11 保存经验$ \left(s_{k}, a_{k}, \eta_{k}, s_{k+1}\right) $到经验回放池 12 If $ \text { aize }(\text { Replaybuffer }) \geq N $ then 13 从经验回放池中批量采样$ N $个样本 14 更新决策Q网络参数：$ \theta \leftarrow \theta-\alpha \nabla_{\theta} L(\theta) $ 15 End If 16 If $ [F+T \times(\text { epiaode }-1)] $mod $ N_{\text {upulate }} $== 0 then 17 更新目标Q网络参数：$ \theta \leftarrow \theta $ 18 End If 19 End For 20 End For 
下载: 导出CSV
表 1 目标初始运动状态
目标 初始位置(km) 速度(m/s) 1 (100, 24) (–220, –250) 2 (–20, 20) (200, –200) 3 (15, 10) (–200, –100)
下载: 导出CSV
-
[1] 何子述, 程子扬, 李军, 等. 集中式MIMO雷达研究综述[J]. 雷达学报, 2022, 11(5): 805–829. doi: 10.12000/JR22128.HE Zishu, CHENG Ziyang, LI Jun, et al. A survey of collocated MIMO radar[J]. Journal of Radars, 2022, 11(5): 805–829. doi: 10.12000/JR22128. [2] 严俊坤, 陈林, 刘宏伟, 等. 基于机会约束的MIMO雷达多波束稳健功率分配算法[J]. 电子学报, 2019, 47(6): 1230–1235. doi: 10.3969/j.issn.0372-2112.2019.06.007.YAN Junkun, CHEN Lin, LIU Hongwei, et al. Chance constrained based robust multibeam power allocation algorithm for MIMO radar[J]. Acta Electronica Sinica, 2019, 47(6): 1230–1235. doi: 10.3969/j.issn.0372-2112.2019.06.007. [3] 李正杰, 谢军伟, 张浩为, 等. 一种低截获背景下的集中式MIMO雷达快速功率分配算法[J]. 雷达学报, 2023, 12(3): 602–615. doi: 10.12000/JR22203.LI Zhengjie, XIE Junwei, ZHANG Haowei, et al. A fast power allocation algorithm in a collocated MIMO radar under low interception backgrounds[J]. Journal of Radars, 2023, 12(3): 602–615. doi: 10.12000/JR22203. [4] 时晨光, 丁琳涛, 汪飞, 等. 面向射频隐身的组网雷达多目标跟踪下射频辐射资源优化分配算法[J]. 电子与信息学报, 2021, 43(3): 539–546. doi: 10.11999/JEIT200636.SHI Chenguang, DING Lintao, WANG Fei, et al. Radio frequency stealth-based optimal radio frequency resource allocation algorithm for multiple-target tracking in radar network[J]. Journal of Electronics & Information Technology, 2021, 43(3): 539–546. doi: 10.11999/JEIT200636. [5] 黄洁瑜, 张浩为, 谢军伟, 等. 一种面向隐身目标跟踪的雷达组网系统资源优化分配算法[J]. 北京航空航天大学学报, 2026, 52(2): 470–481. doi: 10.13700/j.bh.1001-5965.2023.0782.HUANG Jieyu, ZHANG Haowei, XIE Junwei, et al. A resource optimization allocation algorithm for radar networked system for stealth target tracking[J]. Journal of Beijing University of Aeronautics and Astronautics, 2026, 52(2): 470–481. doi: 10.13700/j.bh.1001-5965.2023.0782. [6] YUAN Ye, YI Wei, HOSEINNEZHAD R, et al. Robust power allocation for resource-aware multi-target tracking with colocated MIMO radars[J]. IEEE Transactions on Signal Processing, 2021, 69: 443–458. doi: 10.1109/TSP.2020.3047519. [7] ZHANG Haowei, SHI Junpeng, ZHANG Qiliang, et al. Antenna selection for target tracking in collocated MIMO radar[J]. IEEE Transactions on Aerospace and Electronic Systems, 2021, 57(1): 423–436. doi: 10.1109/TAES.2020.3031767. [8] 丁梓航, 谢军伟, 齐铖. 基于强化学习的频控阵-多输入多输出雷达发射功率分配方法[J]. 电子与信息学报, 2023, 45(2): 550–557. doi: 10.11999/JEIT211555.DING Zihang, XIE Junwei, and QI Cheng. Transmit power allocation method of frequency diverse array-multi input and multi output radar based on reinforcement learning[J]. Journal of Electronics & Information Technology, 2023, 45(2): 550–557. doi: 10.11999/JEIT211555. [9] 张鹏, 严俊坤, 高畅, 等. 动态电磁环境下多功能雷达一体化发射资源管理方案[J]. 雷达学报, 2025, 14(2): 456–469. doi: 10.12000/JR24230.ZHANG Peng, YAN Junkun, GAO Chang, et al. Integrated transmission resource management scheme for multifunctional radars in dynamic electromagnetic environments[J]. Journal of Radars, 2025, 14(2): 456–469. doi: 10.12000/JR24230. [10] LU Ziyang and GURSOY M C. Resource allocation for multi-target radar tracking via constrained deep reinforcement learning[J]. IEEE Transactions on Cognitive Communications and Networking, 2023, 9(6): 1677–1690. doi: 10.1109/TCCN.2023.3304634. [11] 易伟, 袁野, 刘光宏, 等. 多雷达协同探测技术研究进展: 认知跟踪与资源调度算法[J]. 雷达学报, 2023, 12(3): 471–499. doi: 10.12000/JR23036.YI Wei, YUAN Ye, LIU Guanghong, et al. Recent advances in multi-radar collaborative surveillance: Cognitive tracking and resource scheduling algorithms[J]. Journal of Radars, 2023, 12(3): 471–499. doi: 10.12000/JR23036. [12] ZHANG Peng, YAN Junkun, PU Wenqiang, et al. Collaborative strategy learning based transmit resource management scheme for multiple target tracking in multi-cooperative jamming environment[J]. IEEE Transactions on Signal Processing, 2026, 74: 2053–2068. doi: 10.1109/TSP.2026.3693184. [13] 陈佳美, 孙慧雯, 李玉峰, 等. 基于双深度Q网络算法的无人机辅助密集网络资源优化策略[J]. 电子与信息学报, 2025, 47(8): 2621–2629. doi: 10.11999/JEIT250021.CHEN Jiamei, SUN Huiwen, LI Yufeng, et al. Double deep Q network algorithm-based unmanned aerial vehicle-assisted dense network resource optimization strategy[J]. Journal of Electronics & Information Technology, 2025, 47(8): 2621–2629. doi: 10.11999/JEIT250021. [14] 周黎鸣, 余汐, 范明虎, 等. 基于双深度Q网络的多目标遥感产品生产任务调度算法[J]. 电子与信息学报, 2025, 47(8): 2819–2829. doi: 10.11999/JEIT250089.ZHOU Liming, YU Xi, FAN Minghu, et al. Multi-objective remote sensing product production task scheduling algorithm based on double deep Q-network[J]. Journal of Electronics & Information Technology, 2025, 47(8): 2819–2829. doi: 10.11999/JEIT250089. [15] 李一兵, 孙柳晴, 戚昌龙. 基于改进秘书鸟算法的协同干扰资源分配方法[J]. 电子与信息学报, 2025, 47(5): 1494–1504. doi: 10.11999/JEIT240709.LI Yibing, SUN Liuqing, and QI Changlong. Collaborative interference resource allocation method based on improved secretary bird algorithm[J]. Journal of Electronics & Information Technology, 2025, 47(5): 1494–1504. doi: 10.11999/JEIT240709. -
下载:
图(8) / 表(3)
计量
- 文章访问数: 17
- HTML全文浏览量: 9
- PDF下载量: 1
- 被引次数: 0
