Hierarchical reinforcement learning guidance with threat avoidance

doi:10.23919/JSEE.2022.000113

Journal of Systems Engineering and Electronics ›› 2022, Vol. 33 ›› Issue (5): 1173-1185.doi: 10.23919/JSEE.2022.000113

• CONTROL THEORY AND APPLICATION • Previous Articles Next Articles

Hierarchical reinforcement learning guidance with threat avoidance

Bohao LI^1,^2,³(), Yunjie WU^1,^2,³, Guofei LI^4,*()

¹ State Key Laboratory of Virtual Reality Technology and System, Beihang University, Beijing 100191, China
² School of Automation Science and Electrical Engineering, Beihang University, Beijing 100191, China
³ Science and Technology on Aircraft Control Laboratory, Beijing 100191, China
⁴ School of Astronautics, Northwestern Polytechnical University, Xi’an 710072, China

Received:2021-02-24 Online:2022-10-27 Published:2022-10-27
Contact: Guofei LI E-mail:libh08@buaa.edu.cn;liguofei1@126.com
About author:|LI Bohao was born in 1990. He received his B.E. degree from Lanzhou University, Lanzhou, China, in 2012, and M.S. degree in Lanzhou University of Technology, Lanzhou, China, in 2017. He is currently pursuing his Ph.D. degree in navigation, guidance and control with Beihang University, Beijing, China. His research interests include deep learning, deep reinforcement learning, and guidance. E-mail: libh08@buaa.edu.cn||WU Yunjie was born in 1969. She received her Ph.D. degree in navigation guidance and control from Beihang University in 2006. Now, she is a professor in the School of Automation Science and Electrical Engineering, Beihang University, Beijing, China. Her research interests include system simulation, intelligent control, servo control, aircraft guidance and control technology. E-mail: wyjmip@ buaa.edu.cn||LI Guofei was born in 1991. He received his Ph.D. degree from the School of Automation Science and Electrical Engineering, Beihang University, Beijing, China, in 2020. From 2020 to 2021, he was a postdoctoral fellow of Zhuoyue Program in the School of Cyber Science and Technology, Beihang University, Beijing, China. Now, he is an associate professor in the School of Astronautics, Northwestern Polytechnical University, Xi’an, China. His research interests include cooperative guidance, servo system control, and nonlinear control. E-mail: liguofei1@126.com
Supported by:
This work was supported by the National Natural Science Foundation of China (62003021; 91212304)

Abstract

Abstract:

The guidance strategy is an extremely critical factor in determining the striking effect of the missile operation. A novel guidance law is presented by exploiting the deep reinforcement learning (DRL) with the hierarchical deep deterministic policy gradient (DDPG) algorithm. The reward functions are constructed to minimize the line-of-sight (LOS) angle rate and avoid the threat caused by the opposed obstacles. To attenuate the chattering of the acceleration, a hierarchical reinforcement learning structure and an improved reward function with action penalty are put forward. The simulation results validate that the missile under the proposed method can hit the target successfully and keep away from the threatened areas effectively.

Key words: guidance law, deep reinforcement learning (DRL), threat avoidance, hierarchical reinforcement learning

Bohao LI, Yunjie WU, Guofei LI. Hierarchical reinforcement learning guidance with threat avoidance[J]. Journal of Systems Engineering and Electronics, 2022, 33(5): 1173-1185.

Figures/Tables 13

Fig 1

Fig 2

Fig 3

Fig 4

Fig 5

Fig 6

Fig 7

Fig 8

Fig 10

Fig 9

Fig 11

Fig 12

Fig 13

References 36

1	JI Y, LIN D F, WANG W, et al Three-dimensional terminal angle constrained robust guidance law with autopilot lag consideration. Aerospace Science and Technology, 2019, 86, 160- 176. doi: 10.1016/j.ast.2019.01.016
2	RYOO C K, CHO H, TAHK M J Time-to-go weighted optimal guidance with impact angle constraints. IEEE Trans. on Control Systems Technology, 2006, 14 (3): 483- 492. doi: 10.1109/TCST.2006.872525
3	JEON I S, LEE J I, TAHK M J Impact-time-control guidance law for anti-ship missiles. IEEE Trans. on Control Systems Technology, 2006, 14 (2): 260- 266. doi: 10.1109/TCST.2005.863655
4	DONG Y E, SHI M M, SUN Z W Satellite proximate interception vector guidance based on differential games. Chinese Journal of Aeronautics, 2018, 31 (6): 1352- 1361. doi: 10.1016/j.cja.2018.03.012
5	MARCHIDAN A, BAKOLAS E Collision avoidance for an unmanned aerial vehicle in the presence of static and moving obstacles. Journal of Guidance, Control, and Dynamics, 2020, 43 (1): 96- 110. doi: 10.2514/1.G004446
6	XU X G, WEI Z Y, REN Z, et al Time-varying fault-tolerant formation tracking based cooperative control and guidance for multiple cruise missile systems under actuator failures and directed topologies. Journal of Systems Engineering and Electronics, 2019, 30 (3): 587- 600. doi: 10.21629/JSEE.2019.03.16
7	DARSHAN D, ARCHANA C, DEBAJYOTI M Artificial intelligence based missile guidance system. Proc. of the 7th International Conference on Signal Processing and Integrated Networks, 2020, 873- 878.
8	JIE Z, LI H D, BIN X A joint mid-course and terminal course cooperative guidance law for multi-missile salvo attack. Chinese Journal of Aeronautics, 2018, 31 (6): 1311- 1326. doi: 10.1016/j.cja.2018.03.016
9	WANG P, ZHANG X B, ZHU J H Integrated missile guidance and control: a novel explicit reference governor using a disturbance observer. IEEE Trans. on Industrial Electronics, 2018, 66 (7): 5487- 5496.
10	FU S N, LIU X D, ZHANG W J, et al Multiconstraint adaptive three-dimensional guidance law using convex optimization. Journal of Systems Engineering and Electronics, 2020, 31 (4): 791- 803.
11	FANG M, GROEN F C A Collaborative multi-agent reinforcement learning based on experience propagation. Journal of Systems Engineering and Electronics, 2013, 24 (4): 683- 689. doi: 10.1109/JSEE.2013.00079
12	SHALUMOV V Cooperative online guide-launch-guide policy in a target-missile-defender engagement using deep reinforcement learning. Aerospace Science and Technology, 2020, 104, 105996. doi: 10.1016/j.ast.2020.105996
13	YOU S X, DIAO M, GAO L P, et al Target tracking strategy using deep deterministic policy gradient. Applied Soft Computing, 2020, 95, 106490. doi: 10.1016/j.asoc.2020.106490
14	GAUDET B, LINARES R, FURFARO R Deep reinforcement learning for six degree-of-freedom planetary landing. Advances in Space Research, 2020, 65 (7): 1723- 1741. doi: 10.1016/j.asr.2019.12.030
15	LI Y, QIU X H, LIU X D, et al Deep reinforcement learning and its application in autonomous fitting optimization for attack areas of UCAVs. Journal of Systems Engineering and Electronics, 2020, 31 (4): 734- 742. doi: 10.23919/JSEE.2020.000048
16	YAN C, XIANG X J, WANG C Towards real-time path planning through deep reinforcement learning for a UAV in dynamic environments. Journal of Intelligent & Robotic Systems, 2020, 98 (2): 297- 309.
17	WANG D W, FAN T X, HAN T, et al A two-stage reinforcement learning approach for multi-UAV collision avoidance under imperfect sensing. IEEE Robotics and Automation Letters, 2020, 5 (2): 3098- 3105. doi: 10.1109/LRA.2020.2974648
18	YUE W, GUAN X H, WANG L Y A novel searching method using reinforcement learning scheme for multi-UAVs in unknown environments. Applied Sciences, 2019, 9 (22): 4964. doi: 10.3390/app9224964
19	LI G F, WU Y, XU P Adaptive fault-tolerant cooperative guidance law for simultaneous arrival. Aerospace Science and Technology, 2018, 82, 243- 251.
20	LI G F, WU Y, XU P Fixed-time cooperative guidance law with input delay for simultaneous arrival. International Journal of Control, 2021, 94 (6): 1664- 1673. doi: 10.1080/00207179.2019.1662947
21	GAUDET B, LINARES R, FURFARO R Adaptive guidance and integrated navigation with reinforcement meta-learning. Acta Astronautica, 2020, 169, 180- 190. doi: 10.1016/j.actaastro.2020.01.007
22	LIANG C, WANG W H, LIU Z H, et al Range-aware impact angle guidance law with deep reinforcement meta-learning. IEEE Access, 2020, 8, 152093- 152104. doi: 10.1109/ACCESS.2020.3017480
23	HU Q L, HAN T, XIN M Sliding-mode impact time guidance law design for various target motions. Journal of Guidance, Control, and Dynamics, 2019, 42 (1): 136- 148. doi: 10.2514/1.G003620
24	ZHANG W J, FU S N, LI W, et al An impact angle constraint integral sliding mode guidance law for maneuvering targets interception. Journal of Systems Engineering and Electronics, 2020, 31 (1): 168- 184.
25	LI G F, LI Q, WU Y J, et al Leader-following cooperative guidance law with specified impact time. Science China: Technological Sciences, 2020, 63 (11): 2349- 2356. doi: 10.1007/s11431-020-1669-3
26	ZHANG W, SONG K, RONG X W, et al Coarse-to-fine UAV target tracking with deep reinforcement learning. IEEE Trans. on Automation Science and Engineering, 2018, 16 (4): 1522- 1530.
27	QIE H, SHI D X, SHEN T L, et al Joint optimization of multi-UAV target assignment and path planning based on multi-agent reinforcement learning. IEEE Access, 2019, 7, 146264- 146272. doi: 10.1109/ACCESS.2019.2943253
28	HONG D, KIM M, PARK S Study on reinforcement learning-based missile guidance law. Applied Sciences, 2020, 10 (18): 6567. doi: 10.3390/app10186567
29	CHENG L, LU H, LEI T, et al Path planning for anti-ship missile using tangent based dubins path. Proc. of the 2nd International Conference on Intelligent Autonomous Systems, 2019, 175- 180.
30	GUO H, FU W X, FU B, et al Smart homing guidance strategy with control saturation against a cooperative target-defender team. Journal of Systems Engineering and Electronics, 2019, 30 (2): 366- 383. doi: 10.21629/JSEE.2019.02.15
31	YU W B, CHEN W C Guidance law with circular no-fly zone constraint. Nonlinear Dynamics, 2014, 78 (3): 1953- 1971. doi: 10.1007/s11071-014-1571-2
32	WEISS M, SHIMA T Linear quadratic optimal control-based missile guidance law with obstacle avoidance. IEEE Trans. on Aerospace and Electronic Systems, 2018, 55 (1): 205- 214.
33	FAN S P, QI Q, LU K F, et al Autonomous collision avoidance technique of cruise missiles based on modified artificial potential method. Transaction of Beijing Institute of Technology, 2018, 38 (8): 828- 834.
34	CHAYSRI P, BLEKAS K, VLACHOS K Multiple mini-robots navigation using a collaborative multiagent reinforcement learning framework. Advanced Robotics, 2020, 34 (13): 902- 916. doi: 10.1080/01691864.2020.1757507
35	WANG C, WANG J, SHEN Y, et al Autonomous navigation of UAVs in large-scale complex environments: a deep reinforcement learning approach. IEEE Trans. on Vehicular Technology, 2019, 68 (3): 2124- 2136. doi: 10.1109/TVT.2018.2890773
36	LI B H, WU Y Path planning for UAV ground target tracking via deep reinforcement learning. IEEE Access, 2020, 8, 29064- 29074. doi: 10.1109/ACCESS.2020.2971780

[1]	Shengnan FU, Guanqun ZHOU, Qunli XIA. A trajectory shaping guidance law with field-of-view angle constraint and terminal limits [J]. Journal of Systems Engineering and Electronics, 2022, 33(2): 426-437.
[2]	Jia HUANG, Sijiang CHANG, Shengfu CHEN. A hybrid proportional navigation based two-stage impact time control guidance law [J]. Journal of Systems Engineering and Electronics, 2022, 33(2): 461-473.
[3]	Kaifang WAN, Bo LI, Xiaoguang GAO, Zijian HU, Zhipeng YANG. A learning-based flexible autonomous motion control method for UAV in dynamic unknown environments [J]. Journal of Systems Engineering and Electronics, 2021, 32(6): 1490-1508.
[4]	Wei LI, Qiuqiu WEN, Lei HE, Qunli XIA. Three-dimensional impact angle constrained distributed cooperative guidance law for anti-ship missiles [J]. Journal of Systems Engineering and Electronics, 2021, 32(2): 447-459.
[5]	Biao YANG, Wuxing JING, Changsheng GAO. Three-dimensional cooperative guidance law for multiple missiles with impact angle constraint [J]. Journal of Systems Engineering and Electronics, 2020, 31(6): 1286-1296.
[6]	Shengnan FU, Xiaodong LIU, Wenjie ZHANG, Qunli XIA. Multiconstraint adaptive three-dimensional guidance law using convex optimization [J]. Journal of Systems Engineering and Electronics, 2020, 31(4): 791-803.
[7]	Min ZHANG, Chenming ZHENG, Kun HUANG. Fixed-wing UAV guidance law for ground target over-flight tracking [J]. Journal of Systems Engineering and Electronics, 2019, 30(2): 384-392.
[8]	Xiaojian ZHANG, Mingyong LIU, Yang LI, Feihu ZHANG. Impact angle control over composite guidance law based on feedback linearization and finite time control [J]. Journal of Systems Engineering and Electronics, 2018, 29(5): 1036-1045.
[9]	Jingshuai HUANG, Hongbo ZHANG, Guojian TANG, Weimin BAO. Extended differential geometric guidance law for intercepting maneuvering targets [J]. Journal of Systems Engineering and Electronics, 2018, 29(5): 1046-1057.
[10]	Liang ZHANG, Yi LU, Shida XU, Han FENG. Multiple UAVs cooperative formation forming control based on back-stepping-like approach [J]. Journal of Systems Engineering and Electronics, 2018, 29(4): 816-822.
[11]	Qilong SUN, Naiming QI, Longxu XIAO, Haiqi LIN. Differential game strategy in three-player evasion and pursuit scenarios [J]. Journal of Systems Engineering and Electronics, 2018, 29(2): 352-366.
[12]	Ran LI, Qiuqiu WEN, Wangchun TAN, Yijie ZHANG. Adaptive weighting impact angle optimal guidance law considering seeker's FOV angle constraints [J]. Journal of Systems Engineering and Electronics, 2018, 29(1): 142-151.
[13]	Xiaojian Zhang, Mingyong Liu, and Yang Li. Sliding mode control and Lyapunov based guidance law with impact time constraints#br# [J]. Journal of Systems Engineering and Electronics, 2017, 28(6): 1186-1192.
[14]	Qingchun Li, Wensheng Zhang, Gang Han, and Yuan Xie. Fuzzy sliding mode control guidance law with terminal impact angle and acceleration constraints [J]. Systems Engineering and Electronics, 2016, 27(3): 664-679.
[15]	Ran Li, Qunli Xia, and Qiuqiu Wen. Extended optimal guidance law with impact angle and acceleration constriants [J]. Journal of Systems Engineering and Electronics, 2014, 25(5): 868-876.

Hierarchical reinforcement learning guidance with threat avoidance

RichHTML

PDF (PC)

Knowledge

Abstract

Cite this article

Share this article

Figures/Tables 13

References 36

Related Articles 15

Recommended Articles

Metrics

Comments