Scene image recognition with knowledge transfer for drone navigation

doi:10.23919/JSEE.2023.000096

Journal of Systems Engineering and Electronics ›› 2023, Vol. 34 ›› Issue (5): 1309-1318.doi: 10.23919/JSEE.2023.000096

• Control Theory and Application • Previous Articles Next Articles

Scene image recognition with knowledge transfer for drone navigation

Hao DU¹^,²(), Wei WANG²^,³(), Xuerao WANG¹(), Jingqiu ZUO²(), Yuanda WANG¹^,*()

¹ School of Automation, Southeast University, Nanjing 210096, China
² Autonomous Control Robot Laboratory, Jiangsu Zhongke Institute of Applied Research on Intelligent Science and Technology, Changzhou 213164, China
³ Jiangsu Collaborative Innovation Center of Atmospheric Environment and Equipment Technology, Nanjing University of Information Science and Technology, Nanjing 210044, China

Received:2022-02-16 Online:2023-10-18 Published:2023-10-30
Contact: Yuanda WANG E-mail:du-hao@seu.edu.cn;wwcb@nuist.edu.cn;wangxuerao@seu.edu.cn;zuojingqiu@arist.ac.cn;wangyd@seu.edu.cn
About author:
DU Hao was born in 1987. He received his B.E. and M.S. degrees in electronic information engineering and system analysis and integration from Nanjing University of Information Science and Technology, Nanjing, China, in 2009 and 2012, respectively. He is currently pursuing his Ph.D. degree in the School of Automation, Southeast University, Nanjing, China. His current research interests include multi-sensor fusion navigation for the drone, computer vision and visual simultaneous localization and mapping. E-mail: du-hao@seu.edu.cn

WANG Wei was born in 1972. He received his B.E., M.E., and Ph.D. degrees from Chiba University, Chiba, Japan, in 2004, 2006, and 2009, respectively. He is currently a professor at the School of Automation, Nanjing University of Information Science and Technology, Nanjing, China. His current research interests include nonlinear system control, intelligent control of the drone. E-mail: wwcb@nuist.edu.cn

WANG Xuerao was born in 1996. She received her B.S. degree in engineering from Qingdao University of Technology, Qingdao, China, in 2016, and M.S. degree in engineering from University of Science and Technology Beijing, Beijing, China, in 2019. She is currently pursuing her Ph.D. degree in control science and engineering with the School of Automation, Southeast University, Nanjing, China. Her research interests include intelligent control, nonlinear system control, and reinforcement learning. E-mail: wangxuerao@seu.edu.cn

ZUO Jingqiu was born in 1995. She received her B.S. degree in engineering from Jilin University, Changchun, China, in 2017, and M.S. degree in engineering from Osaka University, Suita, Japan, in 2020. She is currently an engineer in the Jiangsu Zhongke Institute of Applied Research on Intelligent Science & Technology, China. Her research interests include mobile robot and autonomous navigation. E-mail: zuojingqiu@arist.ac.cn

WANG Yuanda was born in 1993. He received his B.S. degree in automation from Nanjing University of Information Science and Technology, Nanjing, China in 2014, and Ph.D. degree in control science and engineering from Southeast University, Nanjing, China in 2020. Currently, he is working as a postdoctoral researcher with the School of Automation, Southeast University, Nanjing, China. He has been a visiting Ph.D. student with the Department of Electrical, Computer, and Biomedical Engineering, University of Rhode Island, Kingston, USA from 2016 to 2018. His current research interests include deep reinforcement learning, neural networks, and multi-agent systems. E-mail: wangyd@seu.edu.cn
Supported by:
This work was supported by the National Natural Science Foundation of China ( 62103104), the Natural Science Foundation of Jiangsu Province (BK20210215), and the China Postdoctoral Science Foundation (2021M690615)

Abstract

Abstract:

In this paper, we study scene image recognition with knowledge transfer for drone navigation. We divide navigation scenes into three macro-classes, namely outdoor special scenes (OSSs), the space from indoors to outdoors or from outdoors to indoors transitional scenes (TSs), and others. However, there are difficulties in how to recognize the TSs, to this end, we employ deep convolutional neural network (CNN) based on knowledge transfer, techniques for image augmentation, and fine tuning to solve the issue. Moreover, there is still a novelty detection problem in the classifier, and we use global navigation satellite systems (GNSS) to solve it in the prediction stage. Experiment results show our method, with a pre-trained model and fine tuning, can achieve 91.3196% top-1 accuracy on Scenes21 dataset, paving the way for drones to learn to understand the scenes around them autonomously.

Key words: scene recognition, convolutional neural network, knowledge transfer, global navigation satellite systems (GNSS)-aided

Hao DU, Wei WANG, Xuerao WANG, Jingqiu ZUO, Yuanda WANG. Scene image recognition with knowledge transfer for drone navigation[J]. Journal of Systems Engineering and Electronics, 2023, 34(5): 1309-1318.

Figures/Tables 11

Fig 1

Fig 2

Fig 3

Fig 4

Table 1

Fig 5

Fig 6

Fig 7

Table 2

Fig 8

Fig 9

References 28

1	LI B Q, HU X H Effective distributed convolutional neural network architecture for remote sensing images target classification with a pre-training approach. Journal of Systems Engineering and Electronics, 2019, 30 (2): 238- 244. doi: 10.21629/JSEE.2019.02.02
2	LU C W, TSOUGENIS E, TANG C K Improving object recognition with the l-channel. Pattern Recognition, 2016, 49, 187- 197. doi: 10.1016/j.patcog.2015.06.007
3	GIRSHICK R. Fast R-CNN. Proc. of the IEEE International Conference on Computer Vision, 2015: 1440−1448.
4	REN S Q, HE K M, GIRSHICK R, et al Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Trans. on Pattern Analysis and Machine Intelligence, 2017, 39 (6): 1137- 1149. doi: 10.1109/TPAMI.2016.2577031
5	REDMON J, FARHADI A. Yolov3: an incremental improvement. https://doi.org/10.48550/arXiv.1804.02767.
6	REDMON J, FARHADI A. Yolo9000: better, faster, stronger. Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2017: 6517−6525.
7	MA Y L, WANG C Y. Sdcnet for object recognition. Computer Vision and Image Understanding, 2022, 215: 103332.
8	WU X W, SAHOO D, HOI S Recent advances in deep learning for object detection. Neurocomputing, 2020, 396, 39- 64. doi: 10.1016/j.neucom.2020.01.085
9	ZOU Z X, CHEN K, SHI Z W, et al. Object detection in 20 years: a survey. Proceedings of the IEEE, 2023, 111(3): 257−276.
10	LI Y L, WANG S J, TIAN Q, et al Feature representation for statistical-learning-based object detection: a review. Pattern Recognition, 2015, 48 (11): 3542- 3559. doi: 10.1016/j.patcog.2015.04.018
11	ZHANG X G, MA D X, YU H, et al Scene perception guided crowd anomaly detection. Neurocomputing, 2020, 414, 291- 302. doi: 10.1016/j.neucom.2020.07.019
12	YANG J F, ZHANG S S, WANG G H, et al Scene and place recognition using a hierarchical latent topic model. Neurocomputing, 2015, 148, 578- 586. doi: 10.1016/j.neucom.2014.07.005
13	ESLAMI S M A, REZENDE D, BESSE F, et al Neural scene representation and rendering. Science, 2018, 360 (6394): 1204- 1210. doi: 10.1126/science.aar6170
14	SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition. https://doi.org/10.48550/arXiv.1409.1556.
15	XIE L, LEE F F, LIU L, et al Scene recognition: a comprehensive survey. Pattern Recognition, 2020, 102, 107205. doi: 10.1016/j.patcog.2020.107205
16	ZENG D L, LIAO M Y, TAVAKOLIAN M, et al. Deep learning for scene classification: a survey. https://arXiv.org/abs/2101.10531.
17	ZHANG X W, WANG L, SU Y Visual place recognition: a survey from deep learning perspective. Pattern Recognition, 2021, 113, 107760. doi: 10.1016/j.patcog.2020.107760
18	ZHOU B L, LAPEDRIZA A, KHOSLA, et, al Places: a 10 million image database for scene recognition. IEEE Trans. on Pattern Analysis and Machine Intelligence, 2018, 40 (6): 1452- 1464. doi: 10.1109/TPAMI.2017.2723009
19	NEYSHABUR B, SEDGHI H, ZHANG C. What is being transferred in transfer learning? Proc. of the 34th Conference on Neural Information Processing Systems, 2020: 512−523.
20	HE K M, ZHANG X Y, REN S Q, et, al. Deep residual learning for image recognition. Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2016: 770−778.
21	XIE S N, GIRSHICK R, DOLLAR P, et al. Aggregated residual transformations for deep neural networks. Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2017: 5987−5995.
22	STEGER C, ULRICH M, VIEDEMANN C. Machine vision algorithms and applications. New Jersey: Wiley-VCH GmbH, 2018.
23	THEODORIDIS S, KOUTROUMBAS K. Pattern recognition. 4th ed. Boston: Academic Press, 2009.
24	XIE P, PETOVELLO M G Measuring GNSS multipath distributions in urban canyon environments. IEEE Trans. on Instrumentation and Measurement, 2015, 64 (2): 366- 377. doi: 10.1109/TIM.2014.2342452
25	BRIDLE J S. Training stochastic model recognition algorithms as networks can lead to maximum mutual information estimation of parameters. Proc. of the Advances in Neural Information Processing Systems, 1990: 211−217.
26	QIAN N On the momentum term in gradient descent learning algorithms. Neural Networks, 1999, 12 (1): 145- 151. doi: 10.1016/S0893-6080(98)00116-6
27	SUTSKEVER I, MARTENS J, DAHL G, et al. On the importance of initialization and momentum in deep learning. Proc. of the 30th International Conference on Machine Learning, 2013: 1139−1147.
28	KINGMA D, BA J. Adam: a method for stochastic optimization. Proc. of the 3th International Conference on Learning Representations, 2015. https://doi.org/10.48550/arXiv.1412.6980.

M₁	M₂	M(OR)	Mode
0	0	0	GNSS (BDS/GPS/GLONASS)
0	1	1	LiDAR-SLAM or Vision-SLAM
1	0	1	LiDAR-SLAM or Vision-SLAM
1	1	1	LiDAR-SLAM or Vision-SLAM

Method	Batch-size	Learning rate	Error	Top-1 acc/%	Top-5 acc/%
ResNet-34 with ADAM	64	0.001	0.4333	85.1582	98.5694
ResNet-34 with SGD	64	0.001	0.3939	86.6048	98.8251
ResNet-34 with ADAM + SGD	64	ADAM:1e−4, SGD:1e−3	0.3498	88.0166	98.9125
ResNet-50 with ADAM	32	0.0001	0.3488	88.0570	98.9162
ResNeXt-50 with ADAM	32	0.0001	0.2567	91.3196	99.2593
ResNet-50 with SGD	32	0.01	0.3599	87.6350	99.9191
ResNet-50 with ADAM + SGD	32	ADAM:1e−4, SGD:1e−3	0.3913	86.4732	98.8608

[1]	Chaopeng YU, Wei XIONG, Xiaoqing LI, Lei DONG. Deep convolutional neural network for meteorology target detection in airborne weather radar images [J]. Journal of Systems Engineering and Electronics, 2023, 34(5): 1147-1157.
[2]	Qihai YAO, Yong WANG, Yixin YANG. Range estimation of few-shot underwater sound source in shallow water based on transfer learning and residual CNN [J]. Journal of Systems Engineering and Electronics, 2023, 34(4): 839-850.
[3]	Rui SUN, Zi YANG, Zhenghui ZHAO, Xudong ZHANG. Dual-stream coupling network with wavelet transform for cross-resolution person re-identification [J]. Journal of Systems Engineering and Electronics, 2023, 34(3): 682-695.
[4]	Hao DU, Wei WANG, Xuerao WANG, Yuanda WANG. Autonomous landing scene recognition based on transfer learning for drones [J]. Journal of Systems Engineering and Electronics, 2023, 34(1): 28-35.
[5]	Wenzhang LIU, Lu DONG, Jian LIU, Changyin SUN. Knowledge transfer in multi-agent reinforcement learning with incremental number of agents [J]. Journal of Systems Engineering and Electronics, 2022, 33(2): 447-460.
[6]	Zhengliang ZHU, Degui YANG, Junchao ZHANG, Feng TONG. Dataset of human motion status using IR-UWB through-wall radar [J]. Journal of Systems Engineering and Electronics, 2021, 32(5): 1083-1096.
[7]	Wantian WANG, Ziyue TANG, Yichang CHEN, Yongjian SUN. Parity recognition of blade number and manoeuvre intention classification algorithm of rotor target based on micro-Doppler features using CNN [J]. Journal of Systems Engineering and Electronics, 2020, 31(5): 884-889.
[8]	Chuan LIN, Qing CHANG, Xianxu LI. Uplink NOMA signal transmission with convolutional neural networks approach [J]. Journal of Systems Engineering and Electronics, 2020, 31(5): 890-898.
[9]	Weiwei WU, Qian MA, Yexin LIU, Yongjun KIM. A model for knowledge transfer in a multi-agent organization based on lattice kinetic model [J]. Journal of Systems Engineering and Electronics, 2020, 31(1): 156-167.
[10]	Binquan LI, Xiaohui HU. Effective distributed convolutional neural network architecture for remote sensing images target classification with a pre-training approach [J]. Journal of Systems Engineering and Electronics, 2019, 30(2): 238-244.
[11]	Baojun ZHAO, Boya ZHAO, Linbo TANG, Wenzheng WANG, Chen WU. Multi-scale object detection by top-down and bottom-up feature pyramid network [J]. Journal of Systems Engineering and Electronics, 2019, 30(1): 1-12.
[12]	Jinbo CHEN, Zhiheng WANG, Hengyu LI. Real-time object segmentation based on convolutional neural network with saliency optimization for picking [J]. Journal of Systems Engineering and Electronics, 2018, 29(6): 1300-1307.
[13]	Liangkui LIN, Shaoyou WANG, Zhongxing TANG. Using deep learning to detect small targets in infrared oversampling images [J]. Journal of Systems Engineering and Electronics, 2018, 29(5): 947-952.
[14]	Bendong Zhao, Huanzhang Lu, Shangfeng Chen, Junliang Liu, and Dongya Wu. Convolutional neural networks for time series classification [J]. Systems Engineering and Electronics, 2017, 28(1): 162-.

Scene image recognition with knowledge transfer for drone navigation

RichHTML

PDF (PC)

Knowledge

Abstract

Cite this article

Share this article

Figures/Tables 11

References 28

Related Articles 14

Recommended Articles

Metrics

Comments