Recognition and interfere deceptive behavior based on inverse reinforcement learning and game theory

doi:10.23919/JSEE.2023.000012

Journal of Systems Engineering and Electronics ›› 2023, Vol. 34 ›› Issue (2): 270-288.doi: 10.23919/JSEE.2023.000012

• • 上一篇

收稿日期:2021-03-12 出版日期:2023-04-18 发布日期:2023-04-18

Recognition and interfere deceptive behavior based on inverse reinforcement learning and game theory

Yunxiu ZENG(), Kai XU()

¹ College of Systems Engineering, National University of Defense Technology, Changsha 410073, China

Received:2021-03-12 Online:2023-04-18 Published:2023-04-18
Contact: Yunxiu ZENG E-mail:yuunxiuzeng@hotmail.com;xukai09@nudt.edu.cn
About author:
ZENG Yunxiu was born in 1994. She received her M.S. degree from National University of Defense Technology (NUDT), Changsha, China, in 2018. She is currently a lecturer with NUDT. Her research interests include human behavior modeling and goal recognition. E-mail: yuunxiuzeng@hotmail.com

XU Kai was born in 1990. He received his M.S. and Ph.D. degrees from National University of Defense Technology (NUDT), Changsha, China, in 2016 and 2020 respectively. He is currently a lecturer with NUDT. His research interests include human behavior modeling and goal recognition. E-mail: xukai09@nudt.edu.cn

摘要/Abstract

Abstract:

In real-time strategy (RTS) games, the ability of recognizing other players’ goals is important for creating artifical intelligence (AI) players. However, most current goal recognition methods do not take the player ’s deceptive behavior into account which often occurs in RTS game scenarios, resulting in poor recognition results. In order to solve this problem, this paper proposes goal recognition for deceptive agent, which is an extended goal recognition method applying the deductive reason method (from general to special) to model the deceptive agent’s behavioral strategy. First of all, the general deceptive behavior model is proposed to abstract features of deception, and then these features are applied to construct a behavior strategy that best matches the deceiver’s historical behavior data by the inverse reinforcement learning (IRL) method. Final, to interfere with the deceptive behavior implementation, we construct a game model to describe the confrontation scenario and the most effective interference measures.

Key words: deceptive path planning, inverse reinforcement learning (IRL), game theory, goal recognition

. [J]. Journal of Systems Engineering and Electronics, 2023, 34(2): 270-288.

Yunxiu ZENG, Kai XU. Recognition and interfere deceptive behavior based on inverse reinforcement learning and game theory[J]. Journal of Systems Engineering and Electronics, 2023, 34(2): 270-288.

图/表 17

参考文献 26

1	MIRSKY R, STERN R, GAL Y, et al Goal and plan recognition design using plan libraries. ACM Transactions on Intelligent Systems and Technology, 2019, 10 (2): 1- 23.
2	LORTHIOIR G, INOUE K Design adaptive AI for RTS game by learning player's build order. Proc. of the 29th International Joint Conference on Artificial Intelligence and the 16th Pacific Rim International Conference on Artificial Intelligence, 2020, 5194- 5195.
3	MASTERS P, SARDINA S Deceptive path-planning. Proc. of the 26th International Joint Conference on Artificial Intelligence, 2017, 4368- 4375.
4	XU K, ZENG Y X, QIN L, et al Single real goal, magnitude-based deceptive path-planning. Entropy, 2020, 22 (1): 88. doi: 10.3390/e22010088
5	PAUWELS E, DETON S W Hybrid emerging threats and information warfare: the story of the Cyber-AI deception machine. Proc. of the 21st Century Prometheus, 2020, 107- 124.
6	KATAKWAR H, AGGARWAL P, MAQBOOL Z, et al. Influence of probing action costs on adversarial decision-making in a deception game. https://www.researchgate.net/publication/351928087.
7	OLIVEIRA E D, DONADONI L, BORIERO S, et al Deceptive actions to improve the attribution of rationality to playing robotic agents. International Journal of Social Robotics, 2021, 13, 391- 405. doi: 10.1007/s12369-020-00647-8
8	BOWYER J B. Cheating: deception in war & magic, games & sports, sex & religion, business & con games, politics & espionage, art & science. New York: St. Martin ’s Press, 1982.
9	KOTT A, MCENEANEY W M. Adversarial reasoning: computational approaches to reading the opponents mind. Boca Raton: Taylor of Francis Group, 2007.
10	XU K, HU Y, ZENG Y X, et al Improving the scalability of the magnitude-based deceptive path-planning using subgoal graphs. Entropy, 2020, 22 (2): 162. doi: 10.3390/e22020162
11	BONTRAGER P, KHALIFA A, ANDERSON D, et al Superstition in the network: deep reinforcement learning plays deceptive games. Proc. of the 15th AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, 2019, 10- 16.
12	CHAKRABORTI T, KULKARNI A, SREEDHARAN S, et al Explicability? Legibility? Predictability? Transparency? Privacy? Security? The emerging landscape of interpretable agent behavior. Proc. of the 29th International Conference on Automated Planning and Scheduling, 2019, 86- 96.
13	MASTERS P, SARDINA S Goal recognition for rational and irrational agents. Proc. of the 18th International Conference on Autonomous Agents and Multiagent Systems, 2019, 440- 448.
14	MASTERS P, KIRLEY M, SMITH W Extended goal recognition: a planning-based model for strategic deception. Proc. of the 20th International Conference on Autonomous Agents and Multiagent Systems, 2021, 871- 879.
15	YUE S G, XU K, QIN L et al A two-layer semi-Markov model for recognizing the destination of a moving agent. Proc. of the IEEE International Conference on Mechatronics and Automation, 2015, 918- 922.
16	SHAH S I H, PIETRO G D. An overview of inverse reinforcement learning techniques. Intelligent Environments, 2021. DOI: 10.3233/AISE210097.
17	ZIEBART B D, MAAS A L, BAGNELL J A, et al Maximum entropy inverse reinforcement learning. Proc. of the 23rd Association for the Advancement of Artificial Intelligence, 2008, 1433- 1438.
18	ZENG Y X, XU K, QIN L, et al A semi-Markov decision model with inverse reinforcement learning for recognizing the destination of a maneuvering agent in real time strategy games. IEEE Access, 2020, 8, 15392- 15409. doi: 10.1109/ACCESS.2020.2967642
19	AMADO L, AIRES J P, PEREIRA R F, et al. LSTM-based goal recognition in latent space. Proc. of the International Joint Conference on Neural Networks, 2018. DOI: 10.48550/arXiv.1808.05249.
20	MAYNARD M, DUHAMEL T, KABANZA F. Cost-based goal recognition meets deep learning. Proc. of the AAAI Workshops, 2019. DOI:arXiv:1911.10074.
21	BRAYNOV S Adversarial planning and plan recognition: two sides of the same coin. Proc. of the Secure Knowledge Management Workshop, 2006, 67- 70.
22	ANG S, CHAN H, JIANG A X, et al Game-theoretic goal recognition models with applications to security domains. Proc. of the International Conference on Decision and Game Theory for Security, 2017, 256- 272.
23	RAMIREZ M, GEFFNER H Plan recognition as planning. Proc. of the 21st International Joint Conference on Artifical Intelligence, 2009, 1778- 1783.
24	WANG X H, LIU X H, WANG Y C, et al Research on path planning of mobile robot based on improved A* algorithm. Proc. of the 3rd International Symposium on Autonomous Systems, 2019, 29- 31.
25	YIN Q J, YUE S G, ZHA Y B, et al A semi-Markov decision model for recognizing the destination of a maneuvering agent in real time strategy games. Mathematical Problems in Engineering, 2016, 2016, 1907971.
26	TASTAN B, CHANG Y, SUKTHANKAR G Learning to intercept opponents in first person shooter games. Proc. of the IEEE Conference on Computational Intelligence and Games, 2012, 100- 107.

Strategy	Goal	Training set sample	First feature	Second feature	Result/%
Dissimulation strategy	1	100	Feature (iii)	Feature (viii)	79.85
	2	100	Feature (iii)	Feature (viii)	98.26
	3	100	Feature (iii)	Feature (viii)	95.39
Simulation strategy	1	100	Feature (iii)	Feature (ix)	79.85
	2	100	Feature (iii)	Feature (ii)	98.26
	3	100	Feature (i)	Feature (ii)	95.39

Strategy	Goal	Training set samples	sim(j)	dissim(j)	Result/%
Dissimulation strategy	1	100	0.2	0.05	75.33
	2	100	0.1	0.3	60.87
	3	100	0.02	0.02	79.30
Simulation strategy	1	100	0.05	0.2	72.54
	2	100	0.3	0.01	78.75
	3	100	0.01	0.1	77.89

Strategy	Road network	Target 1		Target 2		Target 3
Strategy	Road network	Weighted path length	Path length	Weighted path length	Path length	Weighted path length	Path length
Simulation strategy	1	1110.9	1268	1085.2	1241	1555.8	1670
	2	1203.9	1370	1138.5	1303	1555.8	1670
	3	1203.9	1370	1138.5	1303	1555.8	1670
	4	1276.5	1460	1211	1393	1626	1769
	5	1276.5	1460	1211	1393	1555.8	1670
Dissimulation strategy	1	1260.7	1302	1172.9	1291	1762.2	1759
	2	1337.9	1399	1200.5	1332	1762.2	1759
	3	1337.9	1399	1200.5	1332	1762.2	1759
	4	1395.6	1459	1234.5	1353	1838	1849
	5	1366.7	1459	1197.5	1321	1787.7	1789

Deceptive strategy	Map	Goal recognition without deception/%	Goal recognition under deception/%	Goal recognition using IRL/%	Weighted length without deceptive modeling	Weighted length with deceptive modeling
Simulation strategy	Chicago sketch road network	85.5	33.7	82.4	1399.4	1347.7
	UT2004 map	84.6	42.5	74.6	16.2	18.1
	Moving-AI benchmarks	89.1	54.7	87.4	250.9	268.2
Dissimulation strategy	Chicago sketch road network	85.5	55.6	67.8	1433.5	1450.6
	UT2004 map	84.6	52.3	71.1	16.1	17.9
	Moving-AI benchmarks	89.1	67.5	79.1	178.2	197.1

Recognition and interfere deceptive behavior based on inverse reinforcement learning and game theory

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

图/表 17

参考文献 26

相关文章 0

编辑推荐

Metrics

本文评价