Double Deep Q-Network for Non-Uniform Position Optimization in Sparse Circular Arrays

CHEN Tao; LIANG Yaopeng; CHEN Xu; ZHAN Lei

doi:10.11999/JEIT250125

Article Contents

Article Navigation > Journal of Electronics & Information Technology > 2025 >

CHEN Tao, LIANG Yaopeng, CHEN Xu, ZHAN Lei. Double Deep Q-Network for Non-Uniform Position Optimization in Sparse Circular Arrays[J]. Journal of Electronics & Information Technology. doi: 10.11999/JEIT250125

Citation:

CHEN Tao, LIANG Yaopeng, CHEN Xu, ZHAN Lei. Double Deep Q-Network for Non-Uniform Position Optimization in Sparse Circular Arrays[J]. Journal of Electronics & Information Technology. doi: 10.11999/JEIT250125

Citation:

PDF( 5604 KB)

Double Deep Q-Network for Non-Uniform Position Optimization in Sparse Circular Arrays

doi: 10.11999/JEIT250125 cstr: 32379.14.JEIT250125

CHEN Tao^{1, 4},
LIANG Yaopeng^{1, 2},
CHEN Xu^{1, 4
,
,},
ZHAN Lei³

1.
College of Information and Communication Engineering, Harbin Engineering University, Harbin 150001, China
2.
Southwest China Institute of Electronic Technology, Chengdu 610036, China
3.
Chengdu LandTop Technology Co., Ltd, Chengdu 610045, China
4.
AVIC United Technology Center for Electromagnetic Spectrum Collaborative Detection and Intelligent Cognition, Aviation Industry Corporation of China, Harbin 150001, China

Funds: The National Natural Science Foundation of China (62071137)

Received Date: 2025-03-03
Rev Recd Date: 2025-08-28

Available Online: 2025-09-02

Abstract

Abstract

Objective To address sparse circular array deployment in practical engineering scenarios, where the number and positions of array elements are constrained, this study proposes an optimization algorithm based on Double Deep Q-Networks (DDQN) to maintain Direction-of-Arrival (DOA) estimation performance under limited channel conditions. This method enables flexible and efficient array design strategies and overcomes challenges that conventional optimization approaches are unable to resolve effectively. The sparse circular array design problem is formulated by minimizing the two-dimensional DOA estimation Ziv–Zakai Bound (ZZB) and Peak Sidelobe Level (PSL) as joint objectives to ensure both angular resolution and estimation accuracy. The state space, action space, and reward function are constructed accordingly, and the DDQN algorithm is employed to solve the optimization task. Experimental results demonstrate that the proposed method achieves stable convergence and robust DOA estimation performance under deployment constraints, and confirm its practical effectiveness. Methods To optimize sparse circular arrays under structural and channel limitations, a DDQN-based design approach is proposed. The method selects a subset of elements from a uniform circular array to maximize DOA estimation accuracy and angular resolution while satisfying constraints on the number of antennas and inter-element spacing. The array design task is cast as a constrained optimization problem with the two-dimensional DOA ZZB and PSL as the performance metrics. Within the reinforcement learning framework, the state space reflects potential array configurations, the action space corresponds to candidate element selections, and the reward function is derived from the optimization objectives. Once trained, the DDQN model outputs an optimized sparse array configuration that balances resolution and sidelobe suppression under the given constraints. Results and Discussions Simulation results show that the reward function of the proposed algorithm converges as the number of training episodes increases (Fig. 8). In contrast, traditional reinforcement learning algorithms exhibit slower convergence and yield suboptimal solutions, while genetic algorithms tend to suffer from premature convergence. The designed sparse circular array satisfies the optimization constraints, including the maximum inter-element spacing requirement (Fig. 7(a)). Under a three-source scenario, the array demonstrates robust DOA estimation capability, effectively resolving multiple incident signals DOA estimation problem (Fig. 9). In evaluations of DOA estimation with Root Mean Squared Error (RMSE) under varying Signal-to-Noise Ratio (SNR) conditions (Fig. 10), the proposed array achieves an estimation error below 0.5° when SNR is ≥ 0 dB. Compared with other sparse circular arrays, it achieves the lowest RMSE, indicating superior estimation performance. In angular resolution tests, the proposed array also exhibits lower PSL values (Table 2) and a higher angle estimation success rate. When the angular separation is ≥ 3°, the success rate exceeds 95% (Fig. 10), confirming the array’s high DOA estimation accuracy and strong angular resolution. Conclusions This study formulates sparse circular array optimization as a constrained problem with the maximum inter-element spacing as a design constraint. To enhance both DOA estimation accuracy and angular resolution, the two-dimensional DOA estimation ZZB and PSL are minimized as joint objectives function. A DDQN algorithm with a dual-network structure is employed to solve the optimization problem and generate the array configuration. Simulation experiments verify that, under channel limitations, the proposed array satisfies the imposed constraints and achieves the intended optimization goals. Compared with other sparse circular arrays, the design demonstrates superior overall DOA estimation performance.
- Array optimization design,
- Sparse circular array,
- Direction-of-Arrival (DOA) estimation,
- Deep Reinforcement Learning (DRL)

FullText(HTML)

References(20)

References

[1]	SINGH U and KAMAL T S. Synthesis of thinned planar concentric circular antenna arrays using biogeography-based optimisation[J]. IET Microwaves, Antennas & Propagation, 2012, 6(7): 822–829. doi: 10.1109/ET2ECN.2012.6470099.
[2]	GIL G T, LEE J Y, KIM H, et al. Comparison of UCA-OAM and UCA-MIMO systems for sub-THz band line-of-sight spatial multiplexing transmission[J]. Journal of Communications and Networks, 2021, 23(2): 83–90. doi: 10.23919/JCN.2021.000013.
[3]	HU Weiwei and WANG Qiang. DOA estimation for UCA in the presence of mutual coupling via error model equivalence[J]. IEEE Wireless Communications Letters, 2020, 9(1): 121–124. doi: 10.1109/LWC.2019.2944816.
[4]	LI Ping, LI Jianfeng, and ZHAO Gaofeng. Low complexity DOA estimation for massive UCA with single snapshot[J]. Journal of Systems Engineering and Electronics, 2022, 33(1): 22–27. doi: 10.23919/JSEE.2022.000003.
[5]	CAO Bin, XIONG Cong, GONG Linshu, et al. Two-dimensional direction of arrival estimation based on nested circular array[C]. Proceedings of 2022 IEEE USNC-URSI Radio Science Meeting, Denver, USA, 2022: 124–125. doi: 10.23919/USNC-URSI52669.2022.9887477.
[6]	ZHANG Xin, SUN Rongchen, GUO Kaifeng, et al. Coprime circular array DOA estimation method[C]. Proceedings of 2023 6th International Conference on Communication Engineering and Technology, Xi’an, China, 2023: 132–136. doi: 10.1109/ICCET58756.2023.00030.
[7]	ZHANG Zongyu, SHI Zhiguo, and GU Yujie. Ziv-zakai bound for DOAs estimation[J]. IEEE Transactions on Signal Processing, 2023, 71: 136–149. doi: 10.1109/TSP.2022.3229946.
[8]	ZHANG Zongyu, SHI Zhiguo, SHAO Cunqi, et al. Ziv–Zakai bound for 2D-DOAs estimation[J]. IEEE Transactions on Signal Processing, 2024, 72: 2483–2497. doi: 10.1109/TSP.2024.3375636.
[9]	PETKO J S and WERNER D H. Pareto optimization of thinned planar arrays with elliptical mainbeams and low sidelobe levels[J]. IEEE Transactions on Antennas and Propagation, 2011, 59(5): 1748–1751. doi: 10.1109/TAP.2011.2122212.
[10]	于波, 陈客松, 朱盼, 等. 稀布圆阵的降维优化方法[J]. 电子与信息学报, 2014, 36(2): 476–481. doi: 10.3724/SP.J.1146.2013.00526. YU Bo, CHEN Kesong, ZHU Pan, et al. An optimum method of sparse concentric rings array based on dimensionality reduction[J]. Journal of Electronics & Information Technology, 2014, 36(2): 476–481. doi: 10.3724/SP.J.1146.2013.00526.
[11]	JIANG Yi, ZHANG Shu, GUO Qiang, et al. Synthesis of uniformly excited concentric ring arrays using the improved integer GA[J]. IEEE Antennas and Wireless Propagation Letters, 2016, 15: 1124–1127. doi: 10.1109/LAWP.2015.2496173.
[12]	GUO Qiang, WANG Yani, YUAN Ding, et al. Optimization of sparse concentric ring arrays based on multiple constraints[J]. IEEE Antennas and Wireless Propagation Letters, 2020, 19(5): 781–785. doi: 10.1109/LAWP.2020.2980166.
[13]	LANG Rongling, XU Hao, GAO Fei, et al. Improving DOA estimation of GNSS interference through sparse non-uniform array reconfiguration[J]. Chinese Journal of Aeronautics, 2025, 38(8): 103384. doi: 10.1016/j.cja.2024.103384.
[14]	ZHANG Binchao, JIN Cheng, CAO Kaiqi, et al. Cognitive conformal antenna array exploiting deep reinforcement learning method[J]. IEEE Transactions on Antennas and Propagation, 2022, 70(7): 5094–5104. doi: 10.1109/TAP.2021.3096994.
[15]	FILIK T and TUNCER T E. Design and evaluation of V-shaped arrays for 2-D DOA estimation[C]. Proceedings of 2008 IEEE International Conference on Acoustics, Speech and Signal Processing, Las Vegas, USA, 2008: 2477–2480. doi: 10.1109/ICASSP.2008.4518150.
[16]	CHEN Zezong, HE Chao, ZHAO Chen, et al. Enhanced target detection for HFSWR by 2-D MUSIC based on sparse recovery[J]. IEEE Geoscience and Remote Sensing Letters, 2017, 14(11): 1983–1987. doi: 10.1109/LGRS.2017.2745048.
[17]	薛青, 来东, 徐勇军, 等. 基于分布式联邦学习的毫米波通信系统波束配置方法[J]. 电子与信息学报, 2024, 46(1): 138–145. doi: 10.11999/JEIT221536. XUE Qing, LAI Dong, XU Yongjun, et al. Beam configuration for millimeter wave communication systems based on distributed federated learning[J]. Journal of Electronics & Information Technology, 2024, 46(1): 138–145. doi: 10.11999/JEIT221536.
[18]	DING Yu, HAN Huimei, LU Weidang, et al. DDQN-based trajectory and resource optimization for UAV-Aided MEC secure communications[J]. IEEE Transactions on Vehicular Technology, 2024, 73(4): 6006–6011. doi: 10.1109/TVT.2023.3335210.
[19]	冯伟杨, 林思雨, 冯婧涛, 等. 基于Q学习的蜂窝车联网边缘计算系统PC-5/Uu接口联合卸载策略[J]. 电子学报, 2024, 52(2): 385–395. doi: 10.12263/DZXB.20220922. FENG Weiyang, LIN Siyu, FENG Jingtao, et al. Q-Learning based joint PC-5/Uu offloading strategy for C-V2X based vehicular edge computing system[J]. Acta Electronica Sinica, 2024, 52(2): 385–395. doi: 10.12263/DZXB.20220922.
[20]	CHEN Tao, WANG Xilin, SHI Lin, et al. Array sparse optimization method based on adaptive genetic algorithm[C]. IET International Radar Conference, 2020: 1271–1275. doi: 10.1049/icp.2021.0648.