Vision-based aerial image mosaicking algorithm with object detection

doi:10.23919/JSEE.2022.000026

Journal of Systems Engineering and Electronics ›› 2022, Vol. 33 ›› Issue (2): 259-268.doi: 10.23919/JSEE.2022.000026

收稿日期:2020-09-25 接受日期:2022-02-15 出版日期:2022-05-06 发布日期:2022-05-06

Vision-based aerial image mosaicking algorithm with object detection

Jun HAN(), Weixing LI*(), Kai FENG(), Feng PAN()

¹ School of Automation, Beijing Institute of Technology, Beijing 100081, China

Received:2020-09-25 Accepted:2022-02-15 Online:2022-05-06 Published:2022-05-06
Contact: Weixing LI E-mail:3120190878@bit.edu.cn;liweixing@bit.edu.cn;fengkai_bit@outlook.com;andropanfeng@126.com
About author:|HAN Jun was born in 1997. He received his B.S. degree from Beijing Institute of Technology (BIT) in 2019. He is a postgraduate student majoring in control science and engineering in BIT. His research interests include computer vision and multi-object tracking. E-mail: 3120190878@bit.edu.cn||LI Weixing was born in 1976. He is a senior experimentalist at the School of Automation in Beijing Institute of Technology (BIT). He received his M.S. degree in pattern recognition and intelligent control from BIT. His research interests include pattern recognition, video analysis, and machine learning. E-mail: liweixing@bit.edu.cn||FENG Kai was born in 1996. He is a postgraduate student majoring in control science and engineering in Beijing Institute of Technology (BIT). He received his B.S. degree from BIT in 2018. His research interests include computer vision and object detection. E-mail: fengkai_bit@outlook.com||PAN Feng was born in 1978. He is an associate professor at the School of Automation in Beijing Institute of Technology (BIT). He received his M.S. and Ph.D. degrees in pattern recognition and intelligent control from BIT. His research interests include intelligent computing and robotics and control. E-mail: andropanfeng@126.com
Supported by:
This work was supported by the National Natural Science Foundation of China (61603040; 61973036)

摘要/Abstract

Abstract:

Aerial image sequence mosaicking is one of the challenging research fields in computer vision. To obtain large-scale orthophoto maps with object detection information, we propose a vision-based image mosaicking algorithm without any extra location data. According to object detection results, we define a complexity factor to describe the importance of each input image and dynamically optimize the feature extraction process. The feature points extraction and matching processes are mainly guided by the speeded-up robust features (SURF) and the grid motion statistic (GMS) algorithm respectively. A robust reference frame selection method is proposed to eliminate the transformation distortion by searching for the center area based on overlaps. Besides, the sparse Levenberg-Marquardt (LM) algorithm and the heavy occluded frames removal method are applied to reduce accumulated errors and further improve the mosaicking performance. The proposed algorithm is performed by using multithreading and graphics processing unit (GPU) acceleration on several aerial image datasets. Extensive experiment results demonstrate that our algorithm outperforms most of the existing aerial image mosaicking methods in visual quality while guaranteeing a high calculation speed.

Key words: image mosaicking, object detection, grid motion statistic (GMS), mapping

. [J]. Journal of Systems Engineering and Electronics, 2022, 33(2): 259-268.

Jun HAN, Weixing LI, Kai FENG, Feng PAN. Vision-based aerial image mosaicking algorithm with object detection[J]. Journal of Systems Engineering and Electronics, 2022, 33(2): 259-268.

图/表 13

参考文献 28

1	CUI J G, LIU M, ZHANG Z T, et al Robust UAV thermal infrared remote sensing images stitching via overlap-prior-based global similarity prior model. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2021, 14, 270- 282. doi: 10.1109/JSTARS.2020.3032011
2	LIU C, ZHANG S H, AKBAR A Ground feature oriented path planning for unmanned aerial vehicle mapping. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019, 12 (4): 1175- 1187. doi: 10.1109/JSTARS.2019.2899369
3	BU S H, ZHAO Y, WAN G, et al Map2DFusion: real-time incremental UAV image mosaicing based on monocular SLAM. Proc. of the IEEE International Conference on Intelligent Robots and Systems, 2016, 4564- 4571.
4	ZHANG J M, ZHU H Q, WANG P Y, et al ATT squeeze U-Net: a lightweight network for forest fire detection and recognition. IEEE Access, 2021, 9, 10858- 10870. doi: 10.1109/ACCESS.2021.3050628
5	CAO Y C, YANG F, TANG Q F, et al An attention enhanced bidirectional LSTM for early forest fire smoke recognition. IEEE Access, 2019, 7, 154732- 154742. doi: 10.1109/ACCESS.2019.2946712
6	AVOLA D, CINQUE L, FORESTI G L, et al A UAV video dataset for mosaicking and change detection from low-altitude flights. IEEE Trans. on Systems, Man, and Cybernetics: Systems, 2020, 50 (6): 2139- 2149. doi: 10.1109/TSMC.2018.2804766
7	SZELISKI R Image alignment and stitching: a tutorial. Boston: Now Foundations and Trends, 2006.
8	ZITOVA B, FLUSSER J Image registration methods: a survey. Image and Vision Computing, 2003, 21 (11): 977- 1000. doi: 10.1016/S0262-8856(03)00137-9
9	LOWE D G Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 2004, 60 (2): 91- 110. doi: 10.1023/B:VISI.0000029664.99615.94
10	BAY H, ESS A, TUYTELAARS T, et al Speeded-up robust features (SURF). Computer Vision and Image Understanding, 2008, 110 (3): 346- 359. doi: 10.1016/j.cviu.2007.09.014
11	RUBLEE E, RABAUD V, KONOLIGE K, et al ORB: an efficient alternative to SIFT or SURF. Proc. of the IEEE International Conference on Computer Vision, 2011, 2564- 2571.
12	EDWARD R, REID B P, TOM D Faster and better: a machine learning approach to corner detection. IEEE Trans. on Pattern Analysis and Machine Intelligence, 2010, 32 (1): 105- 119.
13	BIAN J W, LIN W Y, LIU Y, et al GMS: grid-based motion statistics for fast, ultra-robust feature correspondence. Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, 2828- 2837.
14	AN M, JIANG Z G, ZHAO D P High speed robust image registration and localization using optimized algorithm and its performances evaluation. Journal of Systems Engineering and Electronics, 2010, 21 (3): 520- 526. doi: 10.3969/j.issn.1004-4132.2010.03.026
15	AMIRI A J, MORADI H Real-time video stabilization and mosaicking for monitoring and surveillance. Proc. of the 4th International Conference on Robotics and Mechatronics, 2016, 613- 618.
16	LI C, GUO B L, GUO X X, et al Real-time UAV imagery stitching based on grid-based motion statistics. Journal of Physics: Conference Series, 2018, (1069): 012163.
17	MCLAUCHLAN P F, JAENICKE A Image mosaicing using sequential bundle adjustment. Image and Vision Computing, 2002, 20 (9/10): 751- 759. doi: 10.1016/S0262-8856(02)00064-1
18	XU Q, CHEN J, LUO L B, et al UAV image mosaicking based on multiregion guided local projection deformation. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2020, 13, 3844- 3855. doi: 10.1109/JSTARS.2020.3006289
19	WU F L, GUAN H N, YAN D J, et al Precise geometric correction and robust mosaicking for airborne lightweight optical butting infrared imaging system. IEEE Access, 2019, 7, 93569- 93579. doi: 10.1109/ACCESS.2019.2928380
20	LI M, LI D R, GUO B X, et al Automatic seam-line detection in UAV remote sensing image mosaicking by use of graph cuts. International Journal of Geo-Information, 2018, 7 (9): 361. doi: 10.3390/ijgi7090361
21	YUAN Y T, FANG F M, ZHANG G X Superpixel-based seamless image stitching for UAV images. IEEE Trans. on Geoscience and Remote Sensing, 2021, 59 (2): 1565- 1576. doi: 10.1109/TGRS.2020.2999404
22	GE Y, WEN G J, YANG X L A fast mosaicking method for small UAV image sequence using a small number of ground control points. Proc. of the 8th International Conference on Intelligent Human-Machine Systems and Cybernetics, 2016, 90- 94.
23	REDMON J, FARHADI A. YOLOv3: an incremental improvement. arXiv preprint arXiv: 1804.02767, 2018.
24	XU Y H, OU J L, HE H, et al Mosaicking of unmanned aerial vehicle imagery in the absence of camera poses. Remote Sensing, 2016, 8 (3): 204. doi: 10.3390/rs8030204
25	HU J W, ZHOU Y H, ZHAO C H, et al An application of panoramic mosaic in UAV aerial image. Proc. of the IEEE 13th International Conference on Control and Automation, 2017, 1049- 1053.
26	QU Z, LI J, BAO K H An unordered image stitching method based on binary tree and estimated overlapping area. IEEE Trans. on Image Processing, 2020, 29, 6734- 6744. doi: 10.1109/TIP.2020.2993134
27	JI Y F, LI W X, FENG K, et al Automatic video mosaicking algorithm via dynamic key-frame. Journal of Systems Engineering and Electronics, 2020, 31 (2): 272- 278. doi: 10.23919/JSEE.2020.000005
28	HSIEH M R, LIN Y L, HSU W H Drone-based object counting by spatially regularized regional proposal network. Proc. of the IEEE International Conference on Computer Vision, 2017, 4165- 4173.

Image data	Sequence length	Mosaicking time/s
Image data	Sequence length	Pix4Dmapper	Ours
Dataset 1	71	169	14
Dataset 2	37	92	8
Dataset 3	40	113	7
Dataset 4	24	58	4

Vision-based aerial image mosaicking algorithm with object detection

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

图/表 13

参考文献 28

相关文章 0

编辑推荐

Metrics

本文评价