A sparse algorithm for adaptive pruning least square support vector regression machine based on global representative point ranking

doi:10.23919/JSEE.2021.000014

Abstract

Abstract:

Least square support vector regression (LSSVR) is a method for function approximation, whose solutions are typically non-sparse, which limits its application especially in some occasions of fast prediction. In this paper, a sparse algorithm for adaptive pruning LSSVR algorithm based on global representative point ranking (GRPR-AP-LSSVR) is proposed. At first, the global representative point ranking (GRPR) algorithm is given, and relevant data analysis experiment is implemented which depicts the importance ranking of data points. Furthermore, the pruning strategy of removing two samples in the decremental learning procedure is designed to accelerate the training speed and ensure the sparsity. The removed data points are utilized to test the temporary learning model which ensures the regression accuracy. Finally, the proposed algorithm is verified on artificial datasets and UCI regression datasets, and experimental results indicate that, compared with several benchmark algorithms, the GRPR-AP-LSSVR algorithm has excellent sparsity and prediction speed without impairing the generalization performance.

Key words: least square support vector regression (LSSVR), global representative point ranking (GRPR), initial training dataset, pruning strategy, sparsity, regression accuracy

Lei HU, Guoxing YI, Chao HUANG. A sparse algorithm for adaptive pruning least square support vector regression machine based on global representative point ranking[J]. Journal of Systems Engineering and Electronics, 2021, 32(1): 151-162.

Figures/Tables 16

Fig 1

Fig 2

Table 1

Table 2

Table 3

Table 4

Table 5

Table 6

Table 7

Table 8

Table 9

Table 10

Table 11

Table 12

Fig 3

Fig 4

References 31

1	VAPNIK V N, CHERVONENKIS A Y Necessary and sufficient conditions for the uniform convergence of empirical means to their true values. Journal of Forecasting, 1981, 24 (7): 505- 521.
2	SHUAI Y, SONG T L, WANG J P Method on support vector machine prediction method considering the whole process optimization. System Engineering and Electronics, 2017, 39 (4): 931- 940.
3	SHUYU D, DONGXIAO N, YAN L Forecasting of energy consumption in China based on ensemble empirical mode decomposition and least squares support vector machine optimized by improved shuffled frog leaping algorithm. Applied Sciences, 2018, 8 (5): 678. doi: 10.3390/app8050678
4	DEO R C, KISI O, SINGH V P Drought forecasting in eastern Australia using multivariate adaptive regression spline least square support vector machine and M5Tree model. Atmospheric Research, 2017, 184 (16): 149- 175.
5	ZHANG W W, DING W R, LIU C H Prediction method of UAV data link interference effect in complex environment. System Engineering and Electronics, 2016, 38 (4): 760- 766.
6	LI B, LI X T, GAO X G, et al Air ground integrated attack task decision based on SVM and skyline query. System Engineering and Electronics, 2018, 40 (6): 1281- 1287.
7	WANG C Y, HUANG P P, LI X F, et al Radar HRRP target recognition based on AEPSO-SVM algorithm. System Engineering and Electronics, 2019, 41 (9): 1984- 1989.
8	HONG X, MITCHELL R, DI FATTA G Simplex basis function based sparse least squares support vector regression. Neurocomputing, 2019, 330 (22): 394- 402.
9	GAO R P, SAN Y Improved adaptive pruning algorithm for least squares support vector regression. Journal of System Engineering and Electronics, 2012, 23 (3): 438- 444. doi: 10.1109/JSEE.2012.00055
10	WU Q, ZANG B Y, QI Z X, et al Multi-kernal sparse support vector machine using compressive sensing. System Engineering and Electronics, 2019, 41 (9): 1930- 1936.
11	SUYKENS J A K, LUKAS L, VANDEWALLE J. Sparse approximation using least squares support vector machines. Proc. of the IEEE International Symposium on Circuits & Systems, 2002. DOI: 10.1109/ISCAS.2000.856439.
12	YANG J, BOUZERDOUM A, PHUNG S L. A training algorithm for sparse LS-SVM using compressive sampling. Proc. of the International Conference on Acoustics, Speech and Signal Processing, 2010: 2054–2057.
13	YANG L X, YANG S Y, ZHANG R Sparse least square support vector machine via coupled compressive pruning. Neurocomputing, 2014, 131 (5): 77- 86.
14	DE B K, DE B J, SUYKENS J A K Optimized fixed-size kernel models for large data sets. Computational Statistics & Data Analysis, 2010, 54 (6): 1484- 1504.
15	KARSMAKERS P, PELCKMANS K, BRABANTER K D Sparse conjugate directions pursuit with application to fixed-size kernel models. Machine Learning, 2011, 85 (1): 109- 148.
16	CAUWENBERGHS G, POGGIO T A. Incremental and decremental support vector machine learning. Proc. of the International Conference on Neural Information Processing Systems, 2000.
17	CHEN Y T, XIONG J, XU W H A novel online incremental and decremental learning algorithm based on variable support vector machine. Cluster Computing, 2019, 22 (3): 7435- 7445.
18	LEE W H, KO B J, WANG S. Exact incremental and decremental learning for LS-SVM. Proc. of the IEEE International Conference on Image Processing, 2019. DOI:10.1109/ICIP.2019.8803291.
19	JIN B, JING Z L, ZHAO H T Incremental and decremental extreme learning machine based on generalized inverse. IEEE Access, 2017, 5, 1- 14. doi: 10.1109/ACCESS.2017.2755738
20	ZHAO Y P, WANG K K, LI F A pruning method of refining recursive reduced least squares support vector regression. Information Sciences, 2015, 296, 160- 174. doi: 10.1016/j.ins.2014.10.058
21	JIAO L C, BO L F, WANG L Fast sparse approximation for least squares support vector machine. IEEE Trans. on Neural Network, 2007, 18 (3): 685- 697. doi: 10.1109/TNN.2006.889500
22	GUO G, ZHANG J S Reducing examples to accelerate support vector regression. Pattern Recognition Letters, 2007, 28 (16): 2173- 2183. doi: 10.1016/j.patrec.2007.04.017
23	ZHAO Y P, SUN J G Improved scheme to accelerate support vector regression. Journal of Systems Engineering and Electronics, 2009, 20 (5): 1086- 1090.
24	KRUIF B J D, VRIES T J A D Pruning error minimization in least squares support vector machines. IEEE Trans. on Neural Networks, 2003, 14 (3): 696- 702. doi: 10.1109/TNN.2003.810597
25	ZENG X Y, CHEN X W SMO-based pruning methods for sparse least squares support vector machines. IEEE Trans. on Neural Networks, 2006, 16 (6): 1541- 1546.
26	WU C G. Research on generalized chromosome genetic algorithm and iterative least squares support vector machine regression algorithm. Changchun: Jilin University, 2006.
27	MA Y F, LIANG X, ZHOU X P A fast least squares support vector machine sparse algorithm based on global representative points. Acta Automatica Sinica, 2017, 43 (1): 132- 141.
28	LIU J H, CHEN J P, CHENG J S Online LS-SVM for function estimation and classification. Journal of Beijing University of Science and Technology, 2003, 10 (5): 73- 77.
29	CAWLEY G C, TALBOT N L C Fast exact leave-one-out cross-validation of sparse least-squares support vector machines. Neural Networks, 2004, 17 (10): 1467- 1475. doi: 10.1016/j.neunet.2004.07.002
30	AN S, LIU W, VENKATESH S Fast cross-validation algorithms for least squares support vector machine and kernel ridge regression. Pattern Recognition, 2007, 40 (8): 2154- 2162. doi: 10.1016/j.patcog.2006.12.015
31	WANG W J, XU Z B A heuristic training for support vector regression. Neurocomputing, 2004, 61 (10): 259- 275.

Dataset name	Input dimension	Output dimension	Dataset size	Training size	Testing size
Heart_failure	12	1	299	270	29
Mpg	7	1	390	351	39
Real_estate	6	1	410	369	41
Housing	13	1	500	450	50
ENB2012	9	1	760	684	76
Qsar_fish_toxicity	6	1	908	828	90
Stock	9	1	940	846	94
ConcreteData	8	1	1030	927	103
Sin	1	1	1600	1440	160
Sample	1	1	3600	3240	360

Dataset	LSSVR		S-LSSVR		AI-LSSVR		IAP-LSSVR		PEM-LSSVR		DSAP-LSSVR
Dataset	$\gamma $	$\sigma $	$\gamma $	$\sigma $	$\gamma $	$\sigma $	$\gamma $	$\sigma $	$\gamma $	$\sigma $	$\gamma $	$\sigma $
Heart_failure	4	4	2	4	16	8	4	4	${2^{{\rm{ - }}3}}$	4	2	8
Mpg	512	16	16	4	16	4	16	4	16	4	16	4
Real_estate	32	2	16	1	64	8	8	4	16	1	0.5	4
Housing	128	8	16	4	${2^{10}}$	8	32	8	${2^{10}}$	8	256	8
ENB2012	${2^{10}}$	16	${2^{10}}$	16	128	16	${2^{10}}$	16	128	16	64	16
Qsar_fish_toxicity	16	0.5	2	1	32	8	32	8	32	8	${2^{10}}$	4
Stock	256	1	256	1	238	1	512	4	16	2	512	4
ConcreteData	32	2	64	2	512	8	512	8	16	4	512	8
Sin	64	${2^{{\rm{ - }}3}}$	64	${2^{{\rm{ - }}3}}$	32	0.25	64	${2^{{\rm{ - }}3}}$	32	0.25	32	0.25
Sample	${2^{10}}$	2	16	4	${2^{10}}$	1	${2^{10}}$	1	512	2	512	2

Algorithm	#SV	RMSE	WIA	Training time/s	Testing time/s
LSSVR	270	2.4×10^?4	0.91	2.8×10^?2	2.1×10^?3
S-LSSVR	257	2.4×10^?4	0.9	7.3×10^?2	2.3×10^?3
AI-LSSVR	141	2.6×10^?4	0.88	5.4×10^?2	1.3×10^?4
IAP-LSSVR	141	2.4×10^?4	0.89	5.6×10^?2	1.4×10^?4
PEM-LSSVR	141	2.5×10^?4	0.89	5.4×10^?2	1.4×10^?4
GRPR-AP-LSSVR	120	6.8×10^?4	0.88	5.8×10^?2	1.5×10^?4

Algorithm	#SV	RMSE	WIA	Training time/s	Testing time/s
LSSVR	351	1.4×10^?4	0.96	4.7×10^?2	3.3×10^?3
S-LSSVR	334	1.4×10^?4	0.96	0.12	3.4×10^?3
AI-LSSVR	120	1.5×10^?4	0.96	3.6×10^?2	1.4×10^?4
IAP-LSSVR	120	1.5×10^?4	0.96	3.6×10^?2	1.3×10^?4
PEM-LSSVR	119	1.5×10^?4	0.96	3.7×10^?2	1.3×10^?4
GRPR-AP-LSSVR	117	1.5×10^?4	0.96	4.2×10^?2	1.6×10^?4

Algorithm	#SV	RMSE	WIA	Training time/s	Testing time/s
LSSVR	369	1.3×10^?4	0.89	0.57	4.1×10^?3
S-LSSVR	351	1.4×10^?4	0.88	0.89	4×10^?3
AI-LSSVR	227	1.5×10^?4	0.86	0.15	2×10^?3
IAP-LSSVR	228	2.7×10^?4	0.83	0.16	2.1×10^?3
PEM-LSSVR	232	3.9×10^?4	0.81	0.16	2.8×10^?3
GRPR-AP-LSSVR	195	4.9×10^?4	0.82	0.15	1.9×10^?3