Disparity estimation for multi-scale multi-sensor fusion

doi:10.23919/JSEE.2023.000101

Journal of Systems Engineering and Electronics ›› 2024, Vol. 35 ›› Issue (2): 259-274.doi: 10.23919/JSEE.2023.000101

• ELECTRONICS TECHNOLOGY •

Disparity estimation for multi-scale multi-sensor fusion

Guoliang SUN¹(), Shanshan PEI²(), Qian LONG³^,*(), Sifa ZHENG⁴(), Rui YANG⁵()

¹ Suzhou Automotive Research Institute, Tsinghua University, Suzhou 215000, China
² Beijing Smarter Eye Technology Co., Ltd., Beijing 100023, China
³ College of Artifical Intelligence, Tianjin University of Science and Technology, Tianjin 300457, China
⁴ State Key Laboratory of Automotive Safety and Energy, Tsinghua University, Beijing 100084, China
⁵ Department of Precision Instrument, Tsinghua University, Beijing 100084, China

Received:2022-12-08 Accepted:2023-07-24 Online:2024-04-18 Published:2024-04-18
Contact: Qian LONG E-mail:sunguoliang@tsari.tsinghua.edu.cn;pei.shanshan.must@gmail.com;longqian95@gmail.com;icsun@163.com;Yangrui19781230@163.com
About author:
SUN Guoliang was born in 1983. He received his M.S. degree from Beihang University, China, in 2007. He is pursuing his Ph.D. degree in the School of Vehicle and Mobility, Tsinghua University, China. His research interests include brain-like computing, neuromorphic computing, self-driving car, and automotive grade application specific integrated circuit design. E-mail: sunguoliang@tsari.tsinghua.edu.cn

PEI Shanshan was born in 1993. She received her M.S. degree from Tianjin University of Science & Technology, China, in 2018. She is pursuing her Ph.D. degree in the Faculty of Information Technology, Macau University of Science and Technology, Macau, China. Her research interests include computer vision, pattern recognition, machine learning, and deep learning. E-mail: pei.shanshan.must@gmail.com

LONG Qian was born in 1976. He received his B.S. and Ph.D. degrees from University of Science and Technology of China, Hefei, China, in 2000 and 2007 respectively. He was a faculty member at the Department of Precision Machinery and Precision Instrumentation of University of Science and Technology of China until 2015. He worked as a visiting scientist at Brookhaven National Laboratory, Upton, U.S. (2009), as a research associate at University of Texas at Brownsville, Brownsville, U.S. (2010), and as a postdoctoral fellow in the Research Center for Smart Vehicles, Toyota Technological Institute, Nagoya, Japan (2012−2015), and as a researcher in Nippon Soken, Inc., Nishio, Japan (2016). He is a professor in Yunnan Observatories, Chinese Academy of Sciences, Kunming, China. His research interests include computer vision, pattern recognition, machine learning, and artificial intelligence. E-mail: longqian95@gmail.com

ZHENG Sifa was born in 1970. He received his B.E. and Ph.D. degrees from Tsinghua University, Beijing, China, in 1993 and 1997, respectively. He is a professor in the School of Vehicle and Mobility and the State Key Laboratory of Automotive Safety and Energy, Tsinghua University. He is also the Deputy Director of Suzhou Automotive Research Institute, Tsinghua University. His current research interests include autonomous driving and vehicle dynamics and control. E-mail: icsun@163.com

YANG Rui was born in 1978. He received his M.S. degree from Tsinghua University, China, in 2007. He is pursuing his Ph.D. degree in the Department of Precision Instrument, Tsinghua University, China. His research interests include brain like computing, neuromorphic computing, intelligent transportation system, and integrated intelligent dispatching technology. E-mail: Yangrui19781230@163.com
Supported by:
This work was supported by the National Key R&D Program of China (2018AAA0103103).

Abstract

Abstract:

The perception module of advanced driver assistance systems plays a vital role. Perception schemes often use a single sensor for data processing and environmental perception or adopt the information processing results of various sensors for the fusion of the detection layer. This paper proposes a multi-scale and multi-sensor data fusion strategy in the front end of perception and accomplishes a multi-sensor function disparity map generation scheme. A binocular stereo vision sensor composed of two cameras and a light deterction and ranging (LiDAR) sensor is used to jointly perceive the environment, and a multi-scale fusion scheme is employed to improve the accuracy of the disparity map. This solution not only has the advantages of dense perception of binocular stereo vision sensors but also considers the perception accuracy of LiDAR sensors. Experiments demonstrate that the multi-scale multi-sensor scheme proposed in this paper significantly improves disparity map estimation.

Key words: stereo vision, light deterction and ranging (LiDAR), multi-sensor fusion, multi-scale fusion, disparity map

Guoliang SUN, Shanshan PEI, Qian LONG, Sifa ZHENG, Rui YANG. Disparity estimation for multi-scale multi-sensor fusion[J]. Journal of Systems Engineering and Electronics, 2024, 35(2): 259-274.

Figures/Tables 27

Fig 1

Fig 2

Fig 3

Fig 4

Fig 5

Fig 6

Fig 7

Fig 8

Fig 9

Table 1

Table 2

Fig 10

Fig 11

Table 3

Table 4

Fig 12

Fig 13

Table 5

Table 6

Fig 14

Fig 15

Table 7

Table 8

Fig 16

Fig 17

Table 9

Table 10

References 40

1	MA C, SUN Z, PEI S S, et al. A road environment prediction system for intelligent vehicle. Wireless Communications and Mobile Computing, 2021. DOI: 10.1155/2021/5569295.
2	XIE Q W, LIU R R, SUN Z, et al A flexible free-space detection system based on stereo vision. Neurocomputing, 2022, 485, 252- 262. doi: 10.1016/j.neucom.2021.05.115
3	XU Y Q, JOHN V, MITA S, et al 3D point cloud map based vehicle localization using stereo camera. Proc. of the IEEE Intelligent Vehicles Symposium, 2017, 487- 492.
4	CHEN L F, YANG Z, MA J J, et al Driving scene perception network: real-time joint detection, depth estimation and semantic segmentation. Proc. of the IEEE Winter Conference on Applications of Computer Vision, 2018, 1283- 1291.
5	GAO F, DUAN J L, HAN Z D, et al Automatic virtual test technology for intelligent driving systems considering both coverage and efficiency. IEEE Trans. on Vehicular Technology, 2020, 69 (12): 14365- 14376. doi: 10.1109/TVT.2020.3033565
6	JOHN V, MITA S Deep feature-level sensor fusion using skip connections for real-time object detection in autonomous driving. Electronics, 2021, 10 (4): 424. doi: 10.3390/electronics10040424
7	DURAND S, BENMOKHTAR R, PERROTTON X. 360 multisensor object fusion and sensor-based erroneous data management for autonomous vehicles. Proc. of the IEEE Sensors Applications Symposium, 2019. DOI: 10.1109/SAS.2019.8705970.
8	LONG Q, XIE Q W, MITA S, et al. A real-time dense stereo matching method for critical environment sensing in autonomous driving. Proc. of the IEEE International Conference on Intelligent Transportation Systems, 2014: 853−860.
9	XU Y Q, MITA S, TEHRANI H, et al Real-time stereo disparity quality improvement for challenging traffic environments. Proc. of the IEEE Intelligent Vehicles Symposium, 2018, 1749- 1754.
10	SUN Z, MA C, WANG L, et al A deep learning-based binocular perception system. Journal of Systems Engineering and Electronics, 2021, 32 (1): 7- 20. doi: 10.23919/JSEE.2021.000002
11	ACHARYYA M, KUNDU M K Extraction of noise tolerant, gray-scale transform and rotation invariant features for texture segmentation using wavelet frames. International Journal of Wavelets, Multiresolution and Information Processing, 2008, 6 (3): 391- 417. doi: 10.1142/S0219691308002252
12	XIE Q W, HU X Y, REN L, et al A binocular vision application in IoT: realtime trustworthy road condition detection system in passable area. IEEE Trans. on Industrial Informatics, 2022, 19 (1): 973- 983.
13	YANG L, WANG B Q, ZHANG R H, et al Analysis on location accuracy for the binocular stereo vision system. IEEE Photonics Journal, 2017, 10 (1): 1- 16.
14	PATOLE S M, TORLAK M, WANG D, et al Automotive radars: a review of signal processing techniques. IEEE Signal Processing Magazine, 2017, 34 (2): 22- 35. doi: 10.1109/MSP.2016.2628914
15	SKOLNIK M I. Introduction to radar. Radar Handbook. New York: McGraw-Hill, 1962.
16	YOSHIOKA M, SUGANUMA N, YONEDA K, et al Real-time object classification for autonomous vehicle using lidar. Proc. of the International Conference on Intelligent Informatics and Biomedical Sciences, 2017, 210- 211.
17	BHATTI A, NAHAVANDI S Depth estimation using multiwavelet analysis based stereo vision approach. International Journal of Wavelets, Multiresolution and Information Processing, 2008, 6 (3): 481- 497. doi: 10.1142/S0219691308002434
18	MONDAL M A M, ALI M H Self-guided stereo correspondence estimation algorithm. International Journal of Image and Graphics, 2021, 21 (3): 2150028. doi: 10.1142/S0219467821500285
19	CHENG C B, LI H, ZHANG L M An unsupervised stereo matching cost based on sparse representation. International Journal of Wavelets, Multiresolution and Information Processing, 2021, 19 (1): 2050060. doi: 10.1142/S0219691320500605
20	LIU H Y, BHATTACHARYA P Stereo matching using the discrete wavelet transform. International Journal of Wavelets, Multiresolution and Information Processing, 2007, 5 (4): 567- 588. doi: 10.1142/S0219691307001902
21	NIE Y, MA K K Adaptive rood pattern search for fast block-matching motion estimation. IEEE Trans. on Image Processing, 2002, 11 (12): 1442- 1449. doi: 10.1109/TIP.2002.806251
22	HIRSCHMULLER H Stereo vision in structured environments by consistent semiglobal matching. Proc. of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2006, 2, 2386- 2393.
23	MUKHERJEE S, GUDDETI R M R. A hybrid algorithm for disparity calculation from sparse disparity estimates based on stereo vision. Proc. of the International Conference on Signal Processing and Communications, 2014. DOI: 10.1109/SPCOM.2014.6983949.
24	YANG Q X, WANG L, AHUJA N A constant-space belief propagation algorithm for stereo matching. Proc. of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2010, 1458- 1465.
25	TANIAI T, MATSUSHITA Y, NAEMURA T Graph cut based continuous stereo matching using locally shared labels. Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2014, 1613- 1620.
26	HIRSCHMULLER H Stereo processing by semiglobal matching and mutual information. IEEE Trans. on Pattern Analysis and Machine Intelligence, 2007, 30 (2): 328- 341.
27	SLOAN C A, SHIPHERD J C An ethical imperative: effectively reducing SGM disparities utilizing a multi-level intervention approach. Cognitive and Behavioral Practice, 2019, 26 (2): 239- 242. doi: 10.1016/j.cbpra.2019.02.001
28	VANCEA C, NEDEVSHI S FPGA-based stereo vision hardware for generating dense disparity maps. Proc. of the IEEE 12th International Conference on Intelligent Computer Communication and Processing, 2016, 225- 232.
29	TIPPETTS B J, LEE D J, ARCHIBALD J K, et al Dense disparity real-time stereo vision algorithm for resource-limited systems. IEEE Trans. on Circuits and Systems for Video Technology, 2011, 21 (10): 1547- 1555. doi: 10.1109/TCSVT.2011.2163444
30	ALESSANDRETTI G, BROGGI A, CERRI P Vehicle and guard rail detection using radar and vision data fusion. IEEE Trans. on Intelligent Transportation Systems, 2007, 8 (1): 95- 105. doi: 10.1109/TITS.2006.888597
31	JOHN V, MITA S RVNet: deep sensor fusion of monocular camera and radar for image-based obstacle detection in challenging environments. Proc. of the Pacific-Rim Symposium on Image and Video Technology, 2019, 351- 364.
32	GANG C, JUN W Target detection method in passive bistatic radar. Journal of Systems Engineering and Electronics, 2020, 31 (3): 510- 519. doi: 10.23919/JSEE.2020.000021
33	FEI L, YAN L, CHEN C H, et al OSSIM: an object-based multiview stereo algorithm using SSIM index matching cost. IEEE Trans. on Geoscience and Remote Sensing, 2017, 55 (12): 6937- 6949. doi: 10.1109/TGRS.2017.2737033
34	NAHAR S, JOSHI M V Dense disparity estimation based on feature matching and IGMRF regularization. Proc. of the 23rd International Conference on Pattern Recognition, 2016, 3804- 3809.
35	XIE Q W, CHEN X, ZHANG L M, et al A robust and efficient video anti-shaking algorithm for low-end smartphone platforms. IEEE Trans. on Consumer Electronics, 2018, 65 (1): 1- 10. doi: 10.1109/TED.2017.2779685
36	TUNCEL T, HAMMITT J K A new meta-analysis on the WTP/WTA disparity. Journal of Environmental Economics and Management, 2014, 68 (1): 175- 187. doi: 10.1016/j.jeem.2014.06.001
37	XIE Q W, LONG Q, MITA S Integration of optical flow and Multi-Path-Viterbi algorithm for stereo vision. International Journal of Wavelets, Multiresolution and Information Processing, 2017, 15 (3): 1750022. doi: 10.1142/S0219691317500229
38	LONG Q, XIE Q W, MITA S, et al. Real-time dense disparity estimation based on Multi-Path-Viterbi for intelligent vehicle applications. Proc. of the British Machine Vision Conference, 2014. DOI:10.5244/C.28.127.
39	NISHIGUCHI H, NOMURA Y A study on SSD calculation between input image and subpixel-translated template images and its applications to a subpixel image matching problem. Proc. of the Intelligent Robots and Computer Vision XXVI: Algorithms and Techniques, 2009, 243- 250.
40	GEIGER A, LENZ P, URTASUN R Are we ready for autonomous driving? The KITTI vision benchmark suite. Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2012, 3354- 3361.

Method	MAE/px	RMSE/(1/px)	iMAE/px	iRMSE/(1/px)	logMAE/px	logRMSE/px	SILog/%	abs rel./%	sq. rel./%
SGM + MPV	1.2683	2.3481	0.0020	0.0057	0.0410	0.0752	0.0702	0.0409	0.0089
Proposed method	0.3988	0.9461	0.0008	0.0025	0.0140	0.0316	0.0313	0.0139	0.0015

Method	Accuracy (1 pixel)	Accuracy (3 pixels)	Accuracy (5 pixels)
SGM + MPV	0.5553	0.9503	0.9715
The proposed method	0.9436	0.9955	0.9978

Method	MAE/px	RMSE/(1/px)	iMAE/px	iRMSE/(1/px)	logMAE/px	logRMSE/px	SILog/%	abs rel./%	sq. rel./%
SGM	0.5682	1.3843	0.0010	0.0039	0.0196	0.0466	0.0453	0.0193	0.0031
Proposed method	0.3988	0.9461	0.0008	0.0025	0.0140	0.0316	0.0313	0.0139	0.0015

Method	Accuracy (1 pixel)	Accuracy (3 pixels)	Accuracy (5 pixels)
SGM	0.8640	0.9905	0.9954
Proposed method	0.9436	0.9955	0.9978

Percentage/%	MAE/px	RMSE/(1/px)	iMAE/px	iRMSE/(1/px)	logMAE/px	logRMSE/px	SILog/%	abs rel./%	sq. rel./%
25	0.6527	1.3681	0.0011	0.0036	0.0217	0.0452	0.0431	0.0216	0.0031
50	0.4879	1.1371	0.0009	0.0030	0.0166	0.0374	0.0366	0.0165	0.0021
100	0.3988	0.9461	0.0008	0.0025	0.0140	0.0316	0.0313	0.0139	0.0015

Disparity estimation for multi-scale multi-sensor fusion

RichHTML

PDF (PC)

Knowledge

Abstract

Cite this article

Share this article

Figures/Tables 27

References 40

Related Articles 0

Recommended Articles

Metrics

Comments

Percentage	Accuracy (1 pixel)	Accuracy (3 pixels)	Accuracy (5 pixels)
25%	0.8264	0.9872	0.9941
50%	0.9072	0.9927	0.9967
100%	0.9436	0.9955	0.9978

Size	MAE/px	RMSE/(1/px)	iMAE/px	iRMSE/(1/px)	logMAE/px	logRMSE/px	SILog/%	abs rel./%	sq. rel./%
3×3	0.3971	0.9277	0.0008	0.0025	0.0139	0.0314	0.0312	0.0139	0.0018
5×5	0.4879	1.1371	0.0009	0.0030	0.0166	0.0374	0.0366	0.0165	0.0021
7×7	0.3988	0.9461	0.0008	0.0025	0.0140	0.0316	0.0313	0.0139	0.0015

Algorithm	MAE/px	RMSE/(1/px)	iMAE/px	iRMSE/(1/px)	logMAE/px	logRMSE/px	SILog/%	abs rel./%	sq. rel./%
SSD	6.7050	10.2087	0.0159	0.0220	0.2872	0.3817	0.3213	0.2453	0.1199
SSD + LiDAR	0.4054	1.0733	0.0008	0.0030	0.0145	0.0389	0.0388	0.0147	0.0029
Census	1.4817	2.7550	0.0033	0.0725	0.0505	0.1012	0.0963	0.0496	0.0100
Census + LiDAR	0.5261	1.2520	0.0010	0.0070	0.0178	0.0419	0.8907	0.9853	0.9940
SSIM	1.2683	2.3481	0.0020	0.0057	0.0410	0.0752	0.0702	0.0409	0.0089
SSIM + LiDAR	0.3988	0.9461	0.0008	0.0025	0.0140	0.0316	0.0313	0.0139	0.0015