A deep learning-based binocular perception system

doi:10.23919/JSEE.2021.000002

Journal of Systems Engineering and Electronics ›› 2021, Vol. 32 ›› Issue (1): 7-20.doi: 10.23919/JSEE.2021.000002

• ELECTRONICS TECHNOLOGY • Previous Articles Next Articles

A deep learning-based binocular perception system

Zhao SUN¹(), Chao MA^1,*(), Liang WANG²(), Ran MENG³(), Shanshan PEI¹()

¹ Faculty of Information Technology, Macau University of Science and Technology, Macau 999078, China
² Baidu’s Intelligent Driving Group, Beijing 100193, China
³ Beijing Smarter Eye Technology Co., Ltd., Beijing 100023, China

Received:2020-02-08 Online:2021-02-25 Published:2021-02-25
Contact: Chao MA E-mail:sun.zhao.must@gmail.com;cma@must.edu.mo;wangliang18@baidu.com;ran.meng@smartereye.com;pei.shanshan.must@gmail.com
About author:|SUN Zhao was born in 1989. He received his master’s degree from Tianjin University of Science & Technology, China, in 2017. He is currently pursuing his Ph.D. degree in Faculty of Information Technology, Macau University of Science and Technology, Macau, China. His research interests include computer vision, pattern recognition, machine learning and deep learning. E-mail: sun.zhao.must@gmail.com||MA Chao was born in 1975. He received his Ph.D. degree in applied mathematics from Wuhan University, China, in 2005. He is currently an associate professor in Faculty of Information Technology, Macau University of Science and Technology, Macau, China. His research interests include Diophantine approximation, fractal geometry and applied mathematics. E-mail: cma@must.edu.mo||WANG Liang was born in 1981. He received his Ph.D. degree in computer science from University of Kentucky in 2012. He served as the area chair for International Conference on 3D Vision in 2015, 2016 and 2017. He is currently with Baidu ’s Intelligent Driving Group. His research interests include computer vision, graphics, and machine learning. E-mail: wangliang18@baidu.com||MENG Ran was born in 1980. He received his master’s degree from Tianjin University of Science and Technology, China, in 2006. He is currently with Beijing Smarter Eye Technology Co., Ltd. His research interests include computer vision, pattern recognition, and machine learning. E-mail: ran.meng@smartereye.com||PEI Shanshan was born in 1993. She received her master ’s degree from Tianjin University of Science & Technology, China, in 2018. She is currently pursuing her Ph.D. degree in Faculty of Information Technology, Macau University of Science and Technology, Macau, China. Her research interests include computer vision, pattern recognition, machine learning and deep learning. E-mail: pei.shanshan.must@gmail.com
Supported by:
This work was supported by the National Natural Science Foundation of China (61673381), the National Key R&D Program of China (2018AAA0103103), and the Science and Technology Development Fund (0024/2018/A1)

Abstract

Abstract:

An obstacle perception system for intelligent vehicle is proposed. The proposed system combines the stereo version technique and the deep learning network model, and is applied to obstacle perception tasks in complex environment. In this paper, we provide a complete system design project, which includes the hardware parameters, software framework, algorithm principle, and optimization method. In addition, special experiments are designed to demonstrate that the performance of the proposed system meets the requirements of actual application. The experiment results show that the proposed system is valid to both standard obstacles and non-standard obstacles, and suitable for different weather and lighting conditions in complex environment. It announces that the proposed system is flexible and robust to the intelligent vehicle.

Key words: intelligent vehicle, stereo matching, deep learning, environment perception

Zhao SUN, Chao MA, Liang WANG, Ran MENG, Shanshan PEI. A deep learning-based binocular perception system[J]. Journal of Systems Engineering and Electronics, 2021, 32(1): 7-20.

Figures/Tables 24

Fig 1

Fig 2

Table 1

Fig 3

Fig 4

Fig 5

Fig 6

Fig 7

Fig 8

Fig 9

Fig 10

Fig 11

Fig 12

Fig 13

Fig 14

Fig 15

Fig 16

Fig 17

Table 2

Table 3

Table 4

Table 5

Table 6

Table 7

References 40

1	IM J H, IM S H, JEE G I Vertical corner feature based precise vehicle localization using 3D LIDAR in urban area. Sensors, 2016, 16 (8): 1268. doi: 10.3390/s16081268
2	LI B, ZHANG T L, XIA T. Vehicle detection from 3D lidar using fully convolutional network. [2020-01-15]. https://arxiv.org/pdf/1608.07916.pdf.
3	HATA A Y, WOLF D F Feature detection for vehicle localization in urban environments using a multilayer lidar. IEEE Trans. on Intelligent Transportation Systems, 2016, 17 (2): 420- 429. doi: 10.1109/TITS.2015.2477817
4	ALESSANDRETTI G, BROGGI A, CERRI P Vehicle and guard rail detection using radar and vision data fusion. IEEE Trans. on Intelligent Transportation Systems, 2007, 8 (1): 95- 105. doi: 10.1109/TITS.2006.888597
5	KRSTANOVIC C, KELLER S, GROFT E Radar vehicle detection system. US Patent, 2012, 8279107.
6	LONG Q, XIE Q W, MITA S, et al A real-time dense stereo matching method for critical environment sensing in autonomous driving. Proc. of the IEEE International Conference on Intelligent Transportation Systems, 2014, 853- 860.
7	MOIZUMI H, SUGAYA Y, OMACHI M, et al Traffic light detection considering color saturation using in-vehicle stereo camera. Journal of Information Processing, 2016, 24 (2): 349- 357. doi: 10.2197/ipsjjip.24.349
8	WELLER A D, LARSON M L, MCCABE I A, et al Vision system for vehicle. US Patent, 2018, 10053013.
9	BLAKE R, WILSON H Binocular vision. Vision Research, 2011, 51 (7): 754- 770.
10	STROBL K H, HIRZINGER G More accurate pinhole camera calibration with imperfect planar target. Proc. of the IEEE International Conference on Computer Vision Workshops, 2011, 1068- 1075.
11	XIE Q W, LONG Q, MITA S Integration of optical flow and multi-path-viterbi algorithm for stereo vision. International Journal of Wavelets, Multiresolution and Information Processing, 2017, 15 (3): 1750022. doi: 10.1142/S0219691317500229
12	LUO J H, WU J X. An entropy-based pruning method for CNN compression. [2020-01-15]. https://arxiv.org/pdf/1706.05791.pdf.
13	WU J X, LENG C, WANG Y H, et al Quantized convolutional neural networks for mobile devices. Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, 4820- 4828.
14	GELADO I, STONE J E, CABEZAS J, et al. An asymmetric distributed shared memory model for heterogeneous parallel systems. Proc. of the 15th International Conference on Architectural Support for Programming Languages and Operating Systems, 2010: 347–358.
15	POWER J, BASU A, GU J, et al. Heterogeneous system coherence for integrated CPU-GPU systems. Proc. of the 46th Annual IEEE/ACM International Symposium on Microarchitecture, 2013: 457−467.
16	VINEET V, NARAYANAN P CUDA cuts: fast graph cuts on the GPU. Proc. of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, 2008, 1- 8.
17	KIRK D NVIDIA CUDA software and GPU parallel computing architecture. Proc. of the 6th International Symposium on Memory Management, 2007, 103- 104.
18	XIE Q W, LONG Q, ZHANG L M, et al. A robust real-time computing-based environment sensing system for intelligent vehicle. [2020-02-01]. https://arxiv.org/pdf/2001.09678.pdf.
19	ZHAO B J, ZHAO B Y, TANG L B, et al Multi-scale object detection by top-down and bottom-up feature pyramid network. Journal of Systems Engineering and Electronics, 2019, 30 (1): 1- 12. doi: 10.21629/JSEE.2019.01.01
20	WANG Z, BOVIK A C, SHEIKH H R, et al Image quality assessment: from error visibility to structural similarity. IEEE Trans. on Image Processing, 2004, 13 (4): 600- 612. doi: 10.1109/TIP.2003.819861
21	VIOLA P A, JONES M J. Rapid object detection using a boosted cascade of simple features. Proc. of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2001. DOI:10.1109/CVPR.2001.990517.
22	DATE K, NAGI R Level 2 reformulation linearization technique-based parallel algorithms for solving large quadratic assignment problems on graphics processing unit clusters. INFORMS Journal on Computing, 2019, 31 (4): 771- 789. doi: 10.1287/ijoc.2018.0866
23	LOWE D G Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 2004, 60 (2): 91- 110. doi: 10.1023/B:VISI.0000029664.99615.94
24	ALTUN M, CELENK M Road scene content analysis for driver assistance and autonomous driving. IEEE Trans. on Intelligent Transportation Systems, 2017, 18 (12): 3398- 3407. doi: 10.1109/TITS.2017.2688352
25	LIU W, ANGUELOV D, ERHAN D, et al. SSD: single shot multibox detector. [2020-01-15]. https://arxiv.org/pdf/1512.02325.pdf?ref=hackernoon.com.
26	REDMON J, FARHADI A. YOLOv3: an incremental improvement. [2020-01-15]. https://arxiv.org/pdf/1804.02767.pdf.
27	REN S Q, HE K M, GIRSHICK R, et al Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Trans. on Pattern Analysis and Machine Intelligence, 2017, 39 (6): 1137- 1149. doi: 10.1109/TPAMI.2016.2577031
28	SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition. [2020-01-15]. https://arxiv.org/pdf/1409.1556.pdf%20http://arxiv.org/abs/1409.1556.pdf.
29	SHIN H C, ROTH H, GAO M C, et al Deep convolutional neural networks for computer-aided detection: CNN architectures, dataset characteristics and transfer learning. IEEE Trans. on Medical Imaging, 2016, 35 (5): 1285- 1298. doi: 10.1109/TMI.2016.2528162
30	KRIZHEVSKY A, SUTSKEVER I, HINTON G E ImageNet classification with deep convolutional neural networks. Proc. of the 25th International Conference on Neural Information Processing Systems, 2012, 1097- 1105.
31	HINTON G E, SRIVASTAVA N, KRIZHEVSKY A, et al. Improving neural networks by preventing co-adaptation of feature detectors. [2020-01-15]. https://arxiv.org/pdf/1207.0580.pdf.
32	LIU H, WANG Y L, WANG X, et al Deep learning model of support vector machines with probability output networks. Systems Engineering— Theory and Practice, 2018, 38 (8): 2147- 2154.
33	IOFFE S, SZEGEDY C. Batch normalization: accelerating deep network training by reducing internal covariate shift. Proc. of the International Conference on Machine Learning, 2015: 448–456.
34	TAJBAKHSH N, SHIN J Y, GURUDU S R, et al. Convolutional neural networks for medical image analysis: full training or fine tuning? IEEE Trans. on Medical Imaging, 2016, 35(5):1299–1312.
35	SPACEY S, WIESEMANN W, KUHN D, et al Robust software partitioning with multiple instantiation. INFORMS Journal on Computing, 2012, 24 (3): 500- 515. doi: 10.1287/ijoc.1110.0467
36	BOSCHETTI M A, MANIEZZO V, STRAPPAVECCIA F Using GPU computing for solving the two-dimensional guillotine cutting problem. INFORMS Journal on Computing, 2016, 28 (3): 540- 552. doi: 10.1287/ijoc.2016.0693
37	ZHANG Y F, GAO Q X, GAO L X, et al Imapreduce: a distributed computing framework for iterative computation. Journal of Grid Computing, 2012, 10 (1): 47- 68. doi: 10.1007/s10723-012-9204-9
38	CHEN X, XIE Q W, SHEN L J, et al Morphology-retained non-linear image registration of serial electron microscopy sections. Proc. of the IEEE International Conference on Image Processing, 2018, 3833- 3837.
39	KSHEMKALYANI A D, SINGHAL M. Distributed computing: principles, algorithms, and systems. Cambridge: Cambridge University Press, 2011.
40	LI Z F, LI D, XU X L, et al New normalized LMS adaptive filter with a variable regularization factor. Journal of Systems Engineering and Electronics, 2019, 30 (2): 259- 269. doi: 10.21629/JSEE.2019.02.05

Item	Parameter
Baseline/mm	120
Focus length/mm	6
Resolution	640 × 320
DOF/m	3 ? 100
FOV/(°)	40
Dynamic range/dB	120
Data depth/bit	8
Service voltage/V	12

Result	Label
Result	Vehicle	Pedestrian	Bike	Others
Vehicle	351	0	0	27
Pedestrian	0	108	17	6
Bike	0	14	154	2
Other	36	8	11	253
Precision/%	92.86	82.44	90.59	82.14
Recall/%	90.70	83.08	84.62	87.85

Method	Vehicle		Pedestrian		Bike		Others
Method	Precision	Recall	Precision	Recall	Precision	Recall	Precision	Recall
SSD [25]	92.33	83.98	84.87	77.69	91.67	78.57	60.64	19.79
YOLO [26]	94.17	87.60	88.79	79.23	91.18	85.16	73.13	34.03
Faster R-CNN [27]	95.43	91.73	92.00	88.46	93.21	82.97	74.23	25.00
Ours	92.86	90.70	82.44	83.08	90.59	84.62	82.14	87.95

Method	Vehicle		Pedestrian		Bike
Method	Near	Far	Near	Far	Near	Far
SSD [25]	94.49	59.13	93.27	15.38	90.48	28.57
YOLO [26]	95.96	67.83	91.35	30.77	95.92	40.00
Faster R-CNN [27]	94.85	84.35	97.12	53.85	93.88	37.14
Ours	93.01	85.22	88.46	61.54	86.39	77.14

GPU Frequency/ MHz	Stereo matching module/ms		Obstacle extraction module/ms		Operation efficiency/fps
GPU Frequency/ MHz	Origin MPV algorithm	Our proposed improvement	Origin VGG model	Our proposed improvement	Origin MPV + VGG	Our proposed improvement
324	241	96	784	265	1	2.7
852	147	55	431	161	1.7	4.5

A deep learning-based binocular perception system

RichHTML

PDF (PC)

Knowledge

Abstract

Cite this article

Share this article

Figures/Tables 24

References 40

Related Articles 7

Recommended Articles

Metrics

Comments

Weather condition	Perception rate
Sunny	96.51
Light rain	91.04
Heavy rain	72.15
Snowy	89.34
Night	91.77
Backlight	84.53

Ambient temperature/ ℃	Chip temperature/ ℃	Power dissipation (full load)/W	Power dissipation (standby)/W
20	45?50	10	3
50	70?80	10	3
60	80?85	10	3
65	85?95	10	3

[1]	Haifen YANG, Hao ZHANG, Houjun WANG, Zhengyang GUO. A novel approach for unlabeled samples in radiation source identification [J]. Journal of Systems Engineering and Electronics, 2022, 33(2): 354-359.
[2]	Tao YE, Zongyang ZHAO, Jun ZHANG, Xinghua CHAI, Fuqiang ZHOU. Low-altitude small-sized object detection using lightweight feature-enhanced convolutional neural network [J]. Journal of Systems Engineering and Electronics, 2021, 32(4): 841-853.
[3]	Hongyin SHI, Yue LIU, Jianwen GUO, Mingxin LIU. ISAR autofocus imaging algorithm for maneuvering targets based on deep learning and keystone transform [J]. Journal of Systems Engineering and Electronics, 2020, 31(6): 1178-1185.
[4]	Chuan LIN, Qing CHANG, Xianxu LI. Uplink NOMA signal transmission with convolutional neural networks approach [J]. Journal of Systems Engineering and Electronics, 2020, 31(5): 890-898.
[5]	Liangkui LIN, Shaoyou WANG, Zhongxing TANG. Using deep learning to detect small targets in infrared oversampling images [J]. Journal of Systems Engineering and Electronics, 2018, 29(5): 947-952.
[6]	Chongsheng Zhang, Pengyou Wang, Ke Chen, and Joni-Kristian K¨am¨ ar¨ainen. Identity-aware convolutional neural networks for facial expression recognition [J]. Systems Engineering and Electronics, 2017, 28(4): 784-.
[7]	Tian Jinsi & Su Jianbo. Feature point matching of curved surface and robust uncertainty * [J]. Journal of Systems Engineering and Electronics, 2006, 17(2): 355-361.