Real-time object segmentation based on convolutional neural network with saliency optimization for picking

doi:10.21629/JSEE.2018.06.17

Journal of Systems Engineering and Electronics ›› 2018, Vol. 29 ›› Issue (6): 1300-1307.doi: 10.21629/JSEE.2018.06.17

• Software Algorithm and Simulation • Previous Articles Next Articles

Real-time object segmentation based on convolutional neural network with saliency optimization for picking

Jinbo CHEN(), Zhiheng WANG(), Hengyu LI*()

School of Mechatronic Engineering and Automation, Shanghai University, Shanghai 200072, China

Received:2017-09-22 Online:2018-12-25 Published:2018-12-26
Contact: Hengyu LI E-mail:jbchen@shu.edu.cn;hengzz@i.shu.edu.cn;lihengyu@shu.edu.cn
About author:CHEN Jinbo was born in 1980. He received his M.S. and Ph.D. degrees, both in mechatronic engineering from Shanghai University, Shanghai, China, in 2005 and 2014 respectively. He is currently working as a lecturer in School of Mechatronic Engineering and Automation at Shanghai University. His research interests include computer vision and machine learning. E-mail: jbchen@shu.edu.cn|WANG Zhiheng was born in 1993. In 2015, he enrolled in Shanghai University with a major of mechatronic engineering. He is pursuing his academic master's degree in Shanghai University. He has experienced in some projects about control and vision inspection system. Especially in the machine vision, he has deep research experience. His research interests are in machine vision theory and technology. E-mail: hengzz@i.shu.edu.cn|LI Hengyu was born in 1983. He received his B.S. degree in mechanical engineering and automation from Henan Polytechnic University, China, in 2006, and M.S. and Ph.D. degrees in mechanical and electronic engineering from Shanghai University, China, in 2009 and 2012, respectively. He is currently an associate professor with the School of Mechatronic Engineering and Automation, Shanghai University. His research interests include mechatronics and robot bionic vision system autonomous cooperative control for multiple robots. E-mail: lihengyu@shu.edu.cn
Supported by:
the National Natural Science Foundation of China(61233010);the National Natural Science Foundation of China(61305106);the Shanghai Natural Science Foundation(17ZR1409700);the Shanghai Natural Science Foundation(18ZR1415300);the basic research project of Shanghai Municipal Science and Technology Commission(16JC1400900);This work was supported by the National Natural Science Foundation of China (61233010; 61305106), the Shanghai Natural Science Foundation (17ZR1409700; 18ZR1415300) and the basic research project of Shanghai Municipal Science and Technology Commission (16JC1400900)

Abstract

Abstract:

This paper concerns the problem of object segmentation in real-time for picking system. A region proposal method inspired by human glance based on the convolutional neural network is proposed to select promising regions, allowing more processing is reserved only for these regions. The speed of object segmentation is significantly improved by the region proposal method. By the combination of the region proposal method based on the convolutional neural network and superpixel method, the category and location information can be used to segment objects and image redundancy is significantly reduced. The processing time is reduced considerably by this to achieve the real time. Experiments show that the proposed method can segment the interested target object in real time on an ordinary laptop.

Key words: convolutional neural network, object detection, object segmentation, superpixel, saliency optimization

Jinbo CHEN, Zhiheng WANG, Hengyu LI. Real-time object segmentation based on convolutional neural network with saliency optimization for picking[J]. Journal of Systems Engineering and Electronics, 2018, 29(6): 1300-1307.

Figures/Tables 8

Fig 1

Fig 2

Fig 3

Table 1

Table 2

Fig 4

Fig 5

Table 3

References 26

1	CORRELL N, BEKRIS K E, BERENSON D, et al. Analysis and observations from the first amazon picking challenge. IEEE Trans. on Automation Science and Engineering, 2018, 15 (1): 172- 188. doi: 10.1109/TASE.2016.2600527
2	FELZENSZWALB P F, GIRSHICK R B, MCALLESTER D, et al. Object detection with discriminatively trained part-based models. IEEE Trans. on Pattern Analysis and Machine Intelligence, 2010, 32 (9): 1627- 1645. doi: 10.1109/TPAMI.2009.167
3	PARK D, RAMANAN D, FOWLKES C. Multi-resolution models for object detection. Proc. of the European Conference on Computer Vision, 2010: 241-254.
4	GIRSHICK R, DONAHUE J, DARRELL T, et al. Rich feature hierarchies for accurate object detection and semantic segmentation. Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2014: 580-587.
5	OUYANG W, WANG X. Joint deep learning for pedestrian detection. Proc. of the IEEE International Conference on Computer Vision, 2013: 2056-2063.
6	PAISITKRIANGKRAI S, SHEN C, HENGEL A V D. Strengthening the effectiveness of pedestrian detection with spatially pooled features. Proc. of the European Conference on Computer Vision, 2014: 546-561.
7	BENENSON R, MATHIAS M, TUYTELAARS T, et al. Seeking the strongest rigid detector. Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2013: 3666-3673.
8	HANNAT M, ZRIRA N, RAOUI Y, et al. A fast object recognition and categorization technique for robot grasping using the visual bag of words. Proc. of the International Conference on Multimedia Computing and Systems, 2016: 173-178.
9	NAM H, HAN B. Learning multi-domain convolutional neural networks for visual tracking. Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2016: 4293-4302.
10	KIRKEGAARD J, MOESLUND T B. Bin-picking based on harmonic shape contexts and graph-based matching. Proc. of the International Conference on Pattern Recognition, 2006: 581-584.
11	FANG J, DENG X, SUN C, et al. A vision based position system for robot picking. Proc. of the International Conference on Electrical and Control Engineering, 2010: 319-322.
12	YANG C, ZHANG L, LU H, et al. Saliency detection via graph-based manifold ranking. Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2013: 3166-3173.
13	SHEN X, WU Y. A unified approach to salient object detection via low rank matrix recovery. Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2012: 853-860.
14	JIANG Z, DAVIS L S. Submodular salient region detection. Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2013: 2043-2050.
15	NOH H, HONG S, HAN B. Learning deconvolution network for semantic segmentation. Proc. of the IEEE International Conference on Computer Vision, 2015: 1520-1528.
16	BADRINARAYANAN V, HANDA A, CIPOLLA R. SegNet: a deep convolutional encoder-decoder architecture for robust semantic pixel-wise labelling. IEEE Trans. on Pattern Analysis & Machine Intelligence, 2017, 39 (12): 2481- 2495.
17	CHEN L C, YANG Y, WANG J, et al. Attention to scale: scaleaware semantic image segmentation. Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2016: 3640-3649.
18	REDMON J, DIVVALA S, GIRSHICK R, et al. You only look once: unified, real-time object detection. Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2016: 779-788.
19	ACHANTA R, SHAJI A, SMITH K, et al. SLIC superpixels compared to state-of-the-art superpixel methods. IEEE Trans. on Pattern Analysis and Machine Intelligence, 2012, 34 (11): 2274- 2282. doi: 10.1109/TPAMI.2012.120
20	EVERINGHAM M, GOOL L V, WILLIAMS C K I, et al. The pascal visual object classes (VOC) challenge. International Journal of Computer Vision, 2010, 88 (2): 303- 338. doi: 10.1007/s11263-009-0275-4
21	ZHU W, LIANG S, WEI Y, et al. Saliency optimization from robust background detection. Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2014: 2814-2821.
22	JIA Y Q, SHELHAMER E. Caffe: convolutional architecture for fast feature embedding. 2014, arXiv: 1408.5093.
23	LONG J, SHELHAMER E, DARRELL T. Fully convolutional networks for semantic segmentation. IEEE Trans. on Pattern Analysis and Machine Intelligence, 2017, 39 (4): 640- 651. doi: 10.1109/TPAMI.2016.2572683
24	PERAZZI F, KRÄHENBÜHL P, PRITCH Y, et al. Saliency filters: contrast based filtering for salient region detection. Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2012: 733-740.
25	WEI Y, WEN F, ZHU W, et al. Geodesic saliency using background priors. Proc. of the European Conference on Computer Vision, 2017: 29-42.
26	HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition. Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2016: 770-778.

Algorithm	AP
YOLO	59.2
R-CNN	54.2
GlanceNet	53.3
DPM	43.2

Algorithm		Time/ms
YOLO		183.96
Our	GlanceNet	36.36
	Superpixel	0.78	54.27
	Saliency optimization	17.13

Method	SF	GS	FCN-32s
Time/ms	336.08	271.27	56.87

[1]	Jun HAN, Weixing LI, Kai FENG, Feng PAN. Vision-based aerial image mosaicking algorithm with object detection [J]. Journal of Systems Engineering and Electronics, 2022, 33(2): 259-268.
[2]	Zhengliang ZHU, Degui YANG, Junchao ZHANG, Feng TONG. Dataset of human motion status using IR-UWB through-wall radar [J]. Journal of Systems Engineering and Electronics, 2021, 32(5): 1083-1096.
[3]	Tao YE, Zongyang ZHAO, Jun ZHANG, Xinghua CHAI, Fuqiang ZHOU. Low-altitude small-sized object detection using lightweight feature-enhanced convolutional neural network [J]. Journal of Systems Engineering and Electronics, 2021, 32(4): 841-853.
[4]	Wantian WANG, Ziyue TANG, Yichang CHEN, Yongjian SUN. Parity recognition of blade number and manoeuvre intention classification algorithm of rotor target based on micro-Doppler features using CNN [J]. Journal of Systems Engineering and Electronics, 2020, 31(5): 884-889.
[5]	Chuan LIN, Qing CHANG, Xianxu LI. Uplink NOMA signal transmission with convolutional neural networks approach [J]. Journal of Systems Engineering and Electronics, 2020, 31(5): 890-898.
[6]	Guiling SONG, Aiwei YU, Xuejing KANG, Anlong MING. Monocular depth ordering with occlusion edges extraction and local depth inference [J]. Journal of Systems Engineering and Electronics, 2019, 30(6): 1081-1089.
[7]	Binquan LI, Xiaohui HU. Effective distributed convolutional neural network architecture for remote sensing images target classification with a pre-training approach [J]. Journal of Systems Engineering and Electronics, 2019, 30(2): 238-244.
[8]	Baojun ZHAO, Boya ZHAO, Linbo TANG, Wenzheng WANG, Chen WU. Multi-scale object detection by top-down and bottom-up feature pyramid network [J]. Journal of Systems Engineering and Electronics, 2019, 30(1): 1-12.
[9]	Liangkui LIN, Shaoyou WANG, Zhongxing TANG. Using deep learning to detect small targets in infrared oversampling images [J]. Journal of Systems Engineering and Electronics, 2018, 29(5): 947-952.
[10]	Bendong Zhao, Huanzhang Lu, Shangfeng Chen, Junliang Liu, and Dongya Wu. Convolutional neural networks for time series classification [J]. Systems Engineering and Electronics, 2017, 28(1): 162-.
[11]	Rui Yao and Yanning Zhang. Compressive sensing for small moving space object detection in astronomical images [J]. Journal of Systems Engineering and Electronics, 2012, 23(3): 378-384.
[12]	Qinkun Xiao, Nan Zhang, Fei Li, and Yue Gao. Object detection based on combination of local and spatial information [J]. Journal of Systems Engineering and Electronics, 2011, 22(4): 715-720.
[13]	Jing Li, Junzheng Wang, and Wei Shen. Moving object detection in framework of compressive sampling [J]. Journal of Systems Engineering and Electronics, 2010, 21(5): 740-745.
[14]	Wei Zhiqiang, Ji Xiaopeng & Wang Peng. Real-time moving object detection for video monitoring systems [J]. Journal of Systems Engineering and Electronics, 2006, 17(4): 731-736.

Real-time object segmentation based on convolutional neural network with saliency optimization for picking

RichHTML

PDF (PC)

Knowledge

Abstract

Cite this article

Share this article

Figures/Tables 8

References 26

Related Articles 14

Recommended Articles

Metrics

Comments