Journal of Systems Engineering and Electronics ›› 2024, Vol. 35 ›› Issue (6): 1357-1371.doi: 10.23919/JSEE.2023.000168
• • 上一篇
-
收稿日期:2022-08-16接受日期:2023-12-13出版日期:2024-12-18发布日期:2025-01-14
A content-aware correlation filter with multi-feature fusion for RGB-T tracking
Zihang FENG1(
), Liping YAN1,*(), Jinglan BAI1(), Yuanqing XIA1(), Bo XIAO2()
- 1 School of Automation, Beijing Institute of Technology, Beijing 100081, China
2 School of Artificial Intelligence, Beijing University of Posts and Telecommunications, Beijing 100876, China
-
Received:2022-08-16Accepted:2023-12-13Online:2024-12-18Published:2025-01-14 -
Contact:Liping YAN E-mail:3120185474@bit.edu.cn;ylp@bit.edu.cn;baijinglann@126.com;xia_yuanqing@bit.edu.cn;xiaobo@bupt.edu.cn -
About author:
FENG Zihang was born in 1997. He received his B.S. degree with the School of Automation from Beijing Institute of Technology, Beijing, China, in 2018, where he is currently pursuing his Ph.D. degree in control science and engineering with the School of Automation. His research interests include visual tracking and multimodality fusion. E-mail: 3120185474@bit.edu.cn
YAN Liping was born in 1979. She received her B.S. and M.S. degrees in mathematics from Henan University, Kaifeng, China, in 2000 and 2003, respectively, and Ph.D. degree in control science and engineering from Tsinghua University, Beijing, China, in 2007. From 2007 to 2009, she was a postdoctoral research associate with the Equipment Academy of Air Force, Beijing. Since 2009, she has been with the School of Automation, Beijing Institute of Technology, Beijing, first as assistant professor, from 2011 to 2021 as associate professor, then, since 2021, as a full professor. She has co-authored seven books and over 90 journal and conference papers. Her current research interests include multisensor data fusion, target tracking, fault detection, image registration, intelligent navigation, and integrated navigation. E-mail: ylp@bit.edu.cn
BAI Jinglan was born in 1996. She received her B.S. degree in automation from Minzu University of China in 2018, and M.S. degree in control science and engineering from Beijing Institute of Technology, Beijing, China, in 2021. Her research interests include image fusion and target tracking. E-mail: baijinglann@126.com
XIA Yuanqing was born in 1971. He received his Ph.D. degree in control theory and control engineering from Beijing University of Aeronautics and Astronautics, Beijing, China, in 2001. During January 2002 to November 2003, he was a postdoctoral research associate with the Institute of Systems Science, Academy of Mathematics and System Sciences, Chinese Academy of Sciences, Beijing, China. From November 2003 to February 2004, he was with the National University of Singapore as a research fellow, where he worked on variable structure control. Since 2004, he has been with the Beijing Institute of Technology (BIT), Beijing, where is currently a chair professor, as well as the dean of the School of Automation, BIT. Since October 2023, he has also benn with the Zhongyuan University of Technology (ZUT), zhengzhou, where he is currently the President of ZUT. His research interests include cloud control systems, networked control systems, robust control, signal processing, active disturbance rejection control, unmanned system control, and flight control. E-mail: xia_yuanqing@bit.edu.cn
XIAO Bo was born in 1975. He received his B.S. degree in image transmission and processing, M.S. degree in computer science, and Ph.D. degree in signal and information processing from Beijing University of Posts and Telecommunications, Beijing, China, in 1998, 2006, and 2009, respectively. He was with the School of Information and Communication Engineering, Beijing University of Posts and Telecommunications, from 1998 to 2020, where he has been an associate professor since 2010. From 2018 to 2019, he was a visiting scholar with the University of Windsor, Windsor, ON, Canada. He has been with the School of Artificial Intelligence since 2020. His current research interests include data mining, data fusion, and pattern recognition. E-mail: xiaobo@bupt.edu.cn -
Supported by:This work was supported by the National Natural Science Foundation of China (62073036;62076031) and Beijing Natural Science Foundation (4242049).
引用本文
. [J]. Journal of Systems Engineering and Electronics, 2024, 35(6): 1357-1371.
Zihang FENG, Liping YAN, Jinglan BAI, Yuanqing XIA, Bo XIAO. A content-aware correlation filter with multi-feature fusion for RGB-T tracking[J]. Journal of Systems Engineering and Electronics, 2024, 35(6): 1357-1371.
"
"
"
"
"
"
"
"
"
| PR/SR | ECO | SGT | MDNet | SRDCF | OursFF | OursCA | OursCAFF |
| No occlusion (NO) | 87.7/64.3 | 82.4/50.7 | 83.6/58.8 | 78.4/57.5 | 84.8/61.0 | 82.9/57.1 | 86.2/61.4 |
| Partial occlusion (PO) | 72.2/52.5 | 75.4/48.3 | 75.6/51.5 | 65.7/47.5 | 76.2/52.0 | 76.2/52.9 | 78.4/53.4 |
| Heavy occlusion (HO) | 58.3/41.3 | 53.1/34.1 | 58.8/39.8 | 51.2/35.5 | 52.5/36.6 | 55.1/37.3 | 58.4/40.0 |
| Low illumination (LI) | 66.6/45.6 | 71.6/44.7 | 67.0/45.5 | 54.8/38.7 | 74.3/48.0 | 72.4/48.4 | 76.5/49.9 |
| Low resolution (LR) | 64.1/38.1 | 65.8/37.5 | 66.1/42.9 | 48.9/29.3 | 67.4/39.3 | 65.3/39.4 | 68.3/37.5 |
| Thermal crossover (TC) | 82.1/58.8 | 64.9/40.7 | 71.8/47.5 | 60.3/42.3 | 65.7/48.1 | 61.5/42.7 | 68.2/48.8 |
| Deformation (DEF) | 61.2/45.0 | 65.3/45.9 | 66.2/47.4 | 55.2/40.0 | 60.8/45.5 | 63.6/44.5 | 63.4/46.6 |
| Fast motion (FM) | 58.2/39.2 | 58.0/33.1 | 58.1/35.5 | 54.0/34.5 | 59.0/39.0 | 58.3/37.6 | 64.4/41.3 |
| Scale variation (SV) | 74.5/55.4 | 67.4/41.7 | 73.0/50.0 | 70.7/51.4 | 70.8/51.0 | 71.5/50.1 | 74.6/53.7 |
| Motion blur (MB) | 67.8/49.9 | 58.6/39.6 | 60.4/42.5 | 51.9/38.4 | 57.5/40.8 | 56.8/38.9 | 64.7/44.5 |
| Camera moving (CM) | 61.7/45.0 | 59.0/40.7 | 60.6/43.1 | 55.6/39.6 | 61.2/43.4 | 61.2/43.1 | 66.3/46.7 |
| Background clutter (BC) | 52.9/35.2 | 58.6/35.5 | 61.0/40.6 | 46.4/30.9 | 60.5/38.7 | 59.2/39.2 | 60.9/37.3 |
| All videos | 69.0/49.8 | 67.5/43.0 | 70.2/48.0 | 61.9/44.2 | 68.0/47.2 | 68.7/47.3 | 71.5/49.2 |
"
| Dataset and evaluation | Tracker | ||||
| RT-MDNet | FANet | TFNet | OursCAFF | ||
| GTOT | PR | 0.839 | 0.891 | 0.886 | 0.852 |
| SR | 0.669 | 0.728 | 0.729 | 0.726 | |
| RGBT210 | PR | − | − | 0.777 | 0.715 |
| SR | − | − | 0.529 | 0.492 | |
| RGBT234 | PR | 0.737 | 0.787 | 0.806 | 0.740 |
| SR | 0.500 | 0.553 | 0.560 | 0.506 | |
| VOT- RGBT2019 | A | 0.482 | 0.472 | 0.461 | 0.569 |
| R | 0.624 | 0.492 | 0.406 | 0.529 | |
| EAO | 0.213 | 0.247 | 0.288 | 0.300 | |
| 1 |
ZHOU L, WEI S Y, CUI Z M, et al Lira-YOLO: a lightweight model for ship detection in radar images. Journal of Systems Engineering and Electronics, 2020, 31 (5): 950- 956.
doi: 10.23919/JSEE.2020.000063 |
| 2 |
YE T, ZHAO Z Y, ZHANG J, et al Low-altitude small-sized object detection using lightweight feature-enhanced convolutional neural network. Journal of Systems Engineering and Electronics, 2021, 32 (4): 841- 853.
doi: 10.23919/JSEE.2021.000073 |
| 3 |
HAN J, LI W X, FENG K, et al Vision-based aerial image mosaicking algorithm with object detection. Journal of Systems Engineering and Electronics, 2022, 33 (2): 259- 268.
doi: 10.23919/JSEE.2022.000026 |
| 4 |
KERIM A, CELIKCAN U, ERDEM E, et al Using synthetic data for person tracking under adverse weather conditions. Image and Vision Computing, 2021, 111, 104187.
doi: 10.1016/j.imavis.2021.104187 |
| 5 | SCHNELLE S R, CHAN A L. Enhanced target tracking through infrared-visible image fusion. Proc. of the 14th International Conference on Information Fusion, 2011: 1−8. |
| 6 |
GADE R, MOESLUND T B Thermal cameras and applications: a survey. Machine Vision and Applications, 2014, 25 (1): 245- 262.
doi: 10.1007/s00138-013-0570-5 |
| 7 | LI C L, XUE W L, JIA Y Q, et al LasHeR: a large-scale high-diversity benchmark for RGBT tracking. IEEE Trans. on Image Processing, 2021, 31, 392- 404. |
| 8 | ZHANG P Y, ZHAO J, WANG D, et al. Visible-thermal UAV tracking: a large-scale benchmark and new baseline. Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022: 8876−8885. |
| 9 |
ZHANG X C, YE P, LEUNG H, et al Object fusion tracking based on visible and infrared images: a comprehensive review. Information Fusion, 2020, 63 (1): 166- 187.
doi: 10.1016/j.inffus.2020.05.002 |
| 10 |
WANG J, QIN C X, ZHANG X F, et al A multi-source image fusion algorithm based on gradient regularized convolution sparse representation. Journal of Systems Engineering and Electronics, 2020, 31 (3): 447- 459.
doi: 10.23919/JSEE.2020.000027 |
| 11 |
CHEN C, XIN J X, WANG Y, et al A semisupervised classification approach for multidomain networks with domain selection. IEEE Trans. on Neural Networks and Learning Systems, 2019, 30 (1): 269- 283.
doi: 10.1109/TNNLS.2018.2837166 |
| 12 | LAN X Y, MA A J, YUEN P C. Multi-cue visual tracking using robust feature-level fusion based on joint sparse representation. Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2014: 1194−1201. |
| 13 |
LI C L, CHENG H, HU S Y, et al Learning collaborative sparse representation for grayscale-thermal tracking. IEEE Trans. on Image Processing, 2016, 25 (12): 5743- 5756.
doi: 10.1109/TIP.2016.2614135 |
| 14 | LAN X Y, YE M, ZHANG S P, et al. Robust collaborative discriminative learning for RGB-infrared tracking. Proc. of the AAAI Conference on Artificial Intelligence, 2018: 7008−7015. |
| 15 |
PENG J C, ZHAO H T, HU Z W Dynamic fusion network for RGBT tracking. IEEE Trans. on Intelligent Transportation Systems, 2023, 24 (4): 3822- 3832.
doi: 10.1109/TITS.2022.3229830 |
| 16 | ZHU Y B, LI C L, TANG J, et al RGBT tracking by trident fusion network. IEEE Trans. on Circuits and Systems for Video Technology, 2021, 32 (2): 579- 592. |
| 17 | WANG X, SHU X J, ZHANG S L, et al. MFGNet: dynamic modality-aware filter generation for RGB-T tracking. IEEE Trans. on Multimedia, 2022, 25: 4335−4348. |
| 18 | ZHANG T L, LIU X R, ZHANG Q, et al SiamCDA: complementarity-and distractor-aware RGB-T tracking based on siamese network. IEEE Trans. on Circuits and Systems for Video Technology, 2021, 32 (3): 1403- 1417. |
| 19 | BOLME D S, BEVERIDGE J R, DRAPER B A, et al. Visual object tracking using adaptive correlation filters. Proc. of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2010: 2544−2550. |
| 20 | DANELLJAN M, HAGER G, SHAHBAZ KHAN F , et al. Learning spatially regularized correlation filters for visual tracking. Proc. of the IEEE International Conference on Computer Vision, 2015: 4310−4318. |
| 21 |
LI X, LIU Q, HE Z Y, et al A multi-view model for visual tracking via correlation filters. Knowledge-Based Systems, 2016, 113, 88- 99.
doi: 10.1016/j.knosys.2016.09.014 |
| 22 | YUN X, SUN Y J, YANG X X, et al Discriminative fusion correlation learning for visible and infrared tracking. Mathematical Problems in Engineering, 2019, 2019, 2437521. |
| 23 | WANG Y L, LI C L, TANG J. Learning soft-consistent correlation filters for RBG-T object tracking. Proc. of the Chinese Conference on Pattern Recognition and Computer Vision, 2018: 295−306. |
| 24 |
ZHAI S L, SHAO P P, LIANG X Y, et al Fast RGB-T tracking via cross-modal correlation filters. Neurocomputing, 2019, 334, 172- 181.
doi: 10.1016/j.neucom.2019.01.022 |
| 25 | QIN Y, LU H C, XU Y Q, et al. Saliency detection via cellular automata. Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2015: 110−119. |
| 26 |
FENG W, HAN R Z, GUO Q, et al Dynamic saliency-aware regularization for correlation filter-based object tracking. IEEE Trans. on Image Processing, 2019, 28 (7): 3232- 3245.
doi: 10.1109/TIP.2019.2895411 |
| 27 | ZHAI Y, SHAH M. Visual attention detection in video sequences using spatiotemporal cues. Proc. of the 14th Annual ACM International Conference on Multimedia, 2006: 815−824. |
| 28 |
MA J L, ZHOU Z Q, WANG B, et al Infrared and visible image fusion based on visual saliency map and weighted least square optimization. Infrared Physics and Technology, 2017, 82, 8- 17.
doi: 10.1016/j.infrared.2017.02.005 |
| 29 | LI Y, ZHU J K. A scale adaptive kernel correlation filter tracker with feature integration. Proc. of the European Conference on Computer Vision, 2014: 254−265. |
| 30 | LI C L, ZHAO N, LU Y J, et al. Weighted sparse representation regularized graph learning for RGB-T object tracking. Proc. of the ACM Multimedia Conference, 2017: 1856−1864. |
| 31 |
LI C L, LIANG X Y, LU Y J, et al RGB-T object tracking: benchmark and baseline. Pattern Recognition, 2019, 96, 106977.
doi: 10.1016/j.patcog.2019.106977 |
| 32 | KRISTAN M, MATAS J, LEONARDIS A, et al. The seventh visual object tracking VOT2019 challenge results. Proc. of the IEEE/CVF International Conference on Computer Vision Workshop, 2019: 2206−2241. |
| 33 |
LIU H P, SUN F C Fusion tracking in color and infrared images using joint sparse representation. Science China: Information Sciences, 2012, 55 (3): 590- 599.
doi: 10.1007/s11432-011-4536-9 |
| 34 | WU Y, BLASCH E, CHEN G S, et al. Multiple source data fusion via sparse representation for robust visual tracking. Proc. of the International Conference on Information Fusion, 2011: 1−8. |
| 35 | LI C L, ZHU C L, HUANG Y, et al. Cross-modal ranking with soft consistency and noisy labels for robust RGB-T tracking. Proc. of the European Conference on Computer Vision, 2018: 831−847. |
| 36 | GALOOGAHI H, FAGG A, LUCEY S. Learning background-aware correlation filters for visual tracking. Proc. of the IEEE International Conference on Computer Vision, 2017: 1144−1152. |
| 37 | DANELLJAN M, BHAT G, SHAHBAZ K F, et al. ECO: efficient convolution operators for tracking. Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2017: 6638−6646. |
| 38 | NAM H, HAN B. Learning multi-domain convolutional neural networks for visual tracking. Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2016: 4293−4302. |
| 39 | SONG Y B, MA C, GONG L J, et al. CREST: convolutional residual learning for visual tracking. Proc. of the IEEE International Conference on Computer Vision, 2017: 2574−2583. |
| 40 | BERTINETTO L, VALMADRE J, HENRIQUES J F, et al. Fully-convolutional siamese networks for object tracking. Proc. of the European Conference on Computer Vision Workshops, 2016: 850−865. |
| 41 | KIM H U, LEE D Y, SIM J Y, et al. SOWP: spatially ordered and weighted patch descriptor for visual tracking. Proc. of the IEEE International Conference on Computer Vision, 2015: 3011−3019. |
| 42 | VALMADRE J, BERTINETTO L, HENRIQUES J F, et al. End-to-end representation learning for correlation filter based tracking. Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2017: 5000−5008. |
| 43 | JUNG I, SON J, BAEK M, et al. Real-time MDNet. Proc. of the European Conference on Computer Vision, 2018: 83−98. |
| 44 | ZHU Y B, LI C L, TANG J, et al Quality-aware feature aggregation network for robust RGBT tracking. IEEE Trans. on Intelligent Vehicles, 2020, 6 (1): 121- 130. |
| No related articles found! |
| 阅读次数 | ||||||
|
全文 |
|
|||||
|
摘要 |
|
|||||