Construction and application of pre-classified smooth semi-supervised twin support vector machine

doi:10.21629/JSEE.2019.03.14

Abstract

Abstract:

In order to handle the semi-supervised problem quickly and efficiently in the twin support vector machine (TWSVM) field, a semi-supervised twin support vector machine (S²TSVM) is proposed by adding the original unlabeled samples. In S²TSVM, the addition of unlabeled samples can easily cause the classification hyper plane to deviate from the sample points. Then a centerdistance principle is proposed to pre-classify unlabeled samples, and a pre-classified S²TSVM (PS²TSVM) is proposed. Compared with S²TSVM, PS²TSVM not only improves the problem of the samples deviating from the classification hyper plane, but also improves the training speed. Then PS²TSVM is smoothed. After smoothing the model, the pre-classified smooth S²TSVM (PS³TSVM) is obtained, and its convergence is deduced. Finally, nine datasets are selected in the UCI machine learning database for comparison with other types of semi-supervised models. The experimental results show that the proposed PS³TSVM model has better classification results.

Key words: semi-supervised, twin support vector machine (TWSVM), pre-classified, center-distance, smooth

Xiaodan ZHANG, Hongye QI. Construction and application of pre-classified smooth semi-supervised twin support vector machine[J]. Journal of Systems Engineering and Electronics, 2019, 30(3): 564-572.

Figures/Tables 8

Fig 1

Table 1

Table 2

Table 3

Table 4

Fig 2

Fig 3

Table 5

References 32

1	DENG N Y, TIAN Y J. New method in data mining:support vector machine. Beijing: Science Press, 2004.
2	ZHANG X D, LI A, PAN R. Stock trend prediction based on a new status box method and AdaBoost probabilistic support vector machine. Applied Soft Computing, 2016, 49, 385- 398. doi: 10.1016/j.asoc.2016.08.026
3	JAYADEVA, KHEMCHANDANI R, CHANDRA S. Twin support vector machines for pattern classification. IEEE Trans. on Pattern Analysis and Machine Intelligence, 2007, 29 (5): 905- 910. doi: 10.1109/TPAMI.2007.1068
4	ARUN KUMAR M, GOPAL M. Least squares twin support vector machines for pattern classification. Expert Systems with Applications, 2009, 36 (4): 7535- 7543.
5	SHAO Y H, ZHANG C H, WANG X B, et al. Improvements on twin support vector machines. IEEE Trans. on Neural Networks, 2011, 22 (6): 962- 968. doi: 10.1109/TNN.2011.2130540
6	PENG X. TPMSVM:a novel twin parametric-margin support vector machine for pattern recognition. Pattern Recognition, 2011, 44 (10): 2678- 2692.
7	ZHANG Y X, MA Y. Application of supervised machine learning algorithms in the classification of sagittal gait patterns of cerebral palsy children with spastic diplegia. Computers in Biology and Medicine, 2019, 106, 33- 39. doi: 10.1016/j.compbiomed.2019.01.009
8	RAHMAN I, KUZLU M, RAHMAN S. Power disaggregation of combined HVAC loads using supervised machine learning algorithms. Energy and Buildings, 2018, 172, 57- 66. doi: 10.1016/j.enbuild.2018.03.074
9	BOSTIK O, KLECKA J. Recognition of CAPTCHA charac-ters by supervised machine learning algorithms. IFAC-Papers Online, 2018, 51 (6): 208- 213. doi: 10.1016/j.ifacol.2018.07.155
10	MIKHAIL B, PARTHA N, VIKAS S. Manifold regularization:a geometric framework for learning from labeled and unlabeled examples. Journal of Machine Learning Research, 2006, 7 (11): 2399- 2434.
11	YANG Z J, XU Y T. A safe screening rule for Laplacian support vector machine. Engineering Applications of Artificial Intelligence, 2018, 67, 309- 316. doi: 10.1016/j.engappai.2017.10.011
12	LI Z Y, TIAN Y, LI K, et al. Reject inference in credit scoring using semi-supervised support vector machines. Expert Systems with Applications, 2017, 74, 105- 114. doi: 10.1016/j.eswa.2017.01.011
13	QI Z Q, TIAN Y J, SHI Y. Laplacian twin support vector machine for semi-supervised classification. Neural Networks, 2012, 35, 46- 53. doi: 10.1016/j.neunet.2012.07.011
14	CHEN W J, SHAO Y H, DENG N Y, et al. Laplacian least squares twin support vector machine for semi-supervised classification. Neuro Computing, 2014, 145, 465- 476.
15	YANG Z J, XU Y T. Laplacian twin parametric-margin support vector machine for semi-supervised classification. Neuro Computing, 2016, 171, 325- 334.
16	RASTOGI R, SHARMA S. Fast Laplacian twin support vector machine with active learning for pattern classification. Applied Soft Computing Journal, 2019, 74, 424- 439. doi: 10.1016/j.asoc.2018.10.042
17	WANG Y Y, MENG Y, LI Y, et al. Semi-supervised manifold regularization with adaptive graph construction. Pattern Recognition Letters, 2017, 98, 90- 95. doi: 10.1016/j.patrec.2017.09.004
18	GAO S X, YU Z T, JIN T S, et al. Multi-view low-rank matrix factorization using multiple manifold regularization. Neuro Computing, 2019, 335, 143- 152.
19	LIU B, LI Y M, XU Z L. Manifold regularized matrix completion for multi-label learning with ADMM. Neural Networks, 2018, 101, 57- 67. doi: 10.1016/j.neunet.2018.01.011
20	ZHANG X D, MA J G, LI A H. Quintic spline smooth semisupervised support vector classification machine. Journal of Systems Engineering and Electronics, 2015, 26 (3): 626- 632. doi: 10.1109/JSEE.2015.00070
21	ZHANG X D, LI A, RAN P, et al. Construction and practice of the optimal smooth semi-supervised support vector machine. Journal of Systems Science and Information, 2015, 3 (5): 398- 410. doi: 10.1515/JSSI-2015-0398
22	ZHANG J, FAN X H, BAN D K. Smooth support vector machine based on circular tangent function. The Journal of China Universities of Posts and Telecommunications, 2016, 23 (1): 68- 72, 96. doi: 10.1016/S1005-8885(16)60010-9
23	KUMAR M A, GOPAL M. Application of smoothing technique on twin support vector machines. Pattern Recognition Letters, 2008, 29 (13): 1842- 1848. doi: 10.1016/j.patrec.2008.05.016
24	CHEN W J, SHAO Y H, HONG N. Laplacian smooth twin support vector machine for semi-supervised classification. International Journal of Machine Learning and Cybernetics, 2014, 5 (3): 459- 468. doi: 10.1007/s13042-013-0183-3
25	MALDONADO S, LOPEZ J. Dealing with high-dimensional class-imbalanced datasets:embedded feature selection for SVM classification. Applied Soft and Computing, 2018, 67, 94- 105. doi: 10.1016/j.asoc.2018.02.051
26	POVALEJ Z. Quasi-Newton's method for multi-objective optimization. Journal of Computational and Applied Mathematics, 2014, 255, 765- 777. doi: 10.1016/j.cam.2013.06.045
27	MANGASARIAN O L, WILD E W. Multi-surface proximal support vector machine classification via generalized eigenvalues. IEEE Trans. on Pattern Analysis and Machine Intelligence, 2006, 28 (1): 69- 74. doi: 10.1109/TPAMI.2006.17
28	ZHU L Y, LI W, GAO K, et al. The tourism-specific sentiment vector construction based on kernel optimization function. Procedia Computer Science, 2017, 122, 1162- 1167. doi: 10.1016/j.procs.2017.11.487
29	MA J G. Research and application of smooth semi-supervised support vector machine based on generalized spline function. Beijing: University of Science and Technology Beijing, 2013.
30	DENG X Y, LIU Q, DENG Y, et al. An improved method to construct basic probability assignment based on the confusion matrix for classification problem. Information Sciences, 2016, 340, 250- 261.
31	LUQUE A, CARRASCO A, MARTIN A, et al. The impact of class imbalance in classification performance metrics based on the binary confusion matrix. Pattern Recognition, 2019, 91, 216- 231. doi: 10.1016/j.patcog.2019.02.023
32	TRAJDOS P, KURZYNSKI M. Weighting scheme for a pairwise multi-label classifier based on the fuzzy confusion matrix. Pattern Recognition Letters, 2018, 103, 60- 67. doi: 10.1016/j.patrec.2018.01.012

Categories	Classifier results
Categories	Positive class	Negative class
Positive class	TP	FN
Negative class	FP	TN

Dataset	Number of features	Number of samples
Hepatitis	14	142
Glass 7	10	214
Heart-statlog	13	270
Ecoli12	8	336
BUPA	7	345
Monk_3	7	432
Haberman	3	306
Monk_2	7	432
CMC	9	1 473

Dataset	Performance	Lap-SVM	Lap-TSVM	6STSVM	6S³TSVM	6PS³TSVM
Hepatitis	Accuracy/%	70.24	70.24	69.05	72.62	98.81
Hepatitis	Time/s	0.063 101	0.045 293	0.088 500	0.236 362	0.209 835
Glass 7	Accuracy/%	96.09	100.00	84.38	97.66	91.41
Glass 7	Time/s	0.134 170	0.085 226	0.163 357	0.213 701	0.180 259
Heart-statlog	Accuracy/%	63.58	85.19	85.80	88.89	91.36
Heart-statlog	Time/s	0.227 010	0.118 954	0.545 303	0.517 258	0.417 952
Ecoli 12	Accuracy/%	65.67	65.67	62.69	84.08	84.08
Ecoli 12	Time/s	0.310 435	0.259 667	0.127 639	0.232 469	0.194 074
BUPA	Accuracy/%	69.57	73.43	70.05	71.98	90.34
BUPA	Time/s	0.357 661	0.178 551	0.193 258	0.506 778	0.300 107
Monk_3	Accuracy/%	85.66	87.98	78.68	86.82	97.29
Monk_3	Time/s	0.566 320	0.254 497	0.111 041	0.287 189	0.148 065
Haberman	Accuracy/%	73.77	74.86	61.75	78.14	68.85
Haberman	Time/s	0.274 577	0.128 810	0.297 550	0.534 159	0.383 161
Monk_2	Accuracy/%	67.18	67.18	64.48	69.11	76.83
Monk_2	Time/s	0.550 840	0.212 833	0.168 346	0.207 577	0.179 410
CMC	Accuracy/%	54.59	67.04	64.89	66.36	94.79
CMC	Time/s	7.936 766	2.509 618	1.045 089	1.380 702	0.992 013

Dataset	Performance	Lap-SVM	Lap-TSVM	6STSVM	6S³TSVM	6PS³TSVM
Hepatitis	Accuracy/%	70.24	55.95	63.10	73.81	98.81
Hepatitis	Time/s	0.110 433	0.094 775	1.279 270	0.345 278	0.048 696
Glass 7	Accuracy/%	13.28	86.72	90.63	97.66	99.22
Glass 7	Time/s	0.246 589	0.256 636	0.208 192	1.391 056	0.137 083
Heart-statlog	Accuracy/%	62.96	55.56	64.81	70.37	81.48
Heart-statlog	Time/s	0.378 107	0.452 089	1.453 128	4.818 488	1.666 953
Ecoli 12	Accuracy/%	65.67	65.67	60.20	65.67	81.59
Ecoli 12	Time/s	0.562 233	0.746 396	1.478 087	0.674 261	0.123 923
BUPA	Accuracy/%	58.94	57.97	60.39	72.46	80.19
BUPA	Time/s	0.604 278	0.816 913	4.910 599	7.665 614	1.944 274
Monk_3	Accuracy/%	77.91	52.71	97.29	83.33	87.21
Monk_3	Time/s	0.920 689	1.449 394	9.139 692	1.508 401	0.489 755
Haberman	Accuracy/%	26.78	73.77	73.77	78.14	75.41
Haberman	Time/s	0.517 161	0.604 987	3.869 920	1.881 832	0.781 050
Monk_2	Accuracy/%	67.18	67.18	57.53	67.18	79.15
Monk_2	Time/s	0.947 632	1.410 740	9.481 854	1.634 288	0.362 668
CMC	Accuracy/%	66.48	57.30	70.10	65.12	71.91
CMC	Time/s	11.894 136	38.772 600	233.004 053	14.691 282	9.590 188

Dataset	Proportion of unlabeled samples/%
Dataset	0	10	20	30	40	50	60	70	80	90	100
Hepatitis	66.90	69.72	71.83	70.42	69.01	70.42	70.42	69.72	70.42	68.31	69.72
Glass 7	68.69	97.66	97.66	97.66	93.93	95.79	92.06	92.06	92.06	92.06	92.06
Heart-statlog	82.22	82.22	82.59	82.22	82.59	82.59	82.59	82.22	82.22	82.22	82.22
Ecoli 12	71.43	70.54	80.06	79.46	81.55	80.65	82.14	80.06	84.23	77.08	79.17
BUPA	67.54	67.54	68.12	67.83	67.83	66.96	67.54	67.25	69.28	66.96	68.41
Monk_3	69.21	77.31	79.17	80.79	83.33	80.09	78.94	80.09	79.63	79.86	80.32
Haberman	66.99	72.88	74.18	71.90	71.90	71.90	71.90	71.90	71.90	72.22	71.90
Monk_2	67.13	68.52	68.75	69.21	68.52	68.75	68.75	69.21	68.98	68.98	68.75
CMC	66.26	66.53	66.80	66.80	66.80	66.80	66.80	66.46	66.46	66.33	66.53