A combined algorithm of K-means and MTRL for multi-class classification

doi:10.21629/JSEE.2019.05.06

Journal of Systems Engineering and Electronics ›› 2019, Vol. 30 ›› Issue (5): 875-885.doi: 10.21629/JSEE.2019.05.06

• Electronics Technology • Previous Articles Next Articles

A combined algorithm of K-means and MTRL for multi-class classification

Mengfan XUE(), NLei HA(), Dongliang PENG*()

School of Automation, Hangzhou Dianzi University, Hangzhou 310018, China

Received:2018-10-15 Online:2019-10-08 Published:2019-10-09
Contact: Dongliang PENG E-mail:xuemf@hdu.edu.cn;hanlei52001@qq.com;dlpeng@hdu.edu.cn
About author:XUE Mengfan was born in 1990. She received her Ph.D. degree from Xidian University in 2016. She is currently a lecturer in Hangzhou Dianzi University. Her research interests are signal processing and machine learning. E-mail: xuemf@hdu.edu.cn|HAN Lei was born in 1994. He received his M.S. degree from Hangzhou Dianzi University. His research interests are machine learning and automatic target recognition. E-mail: hanlei52001@qq.com|PENG Dongliang was born in 1977. He received his B.S. and M.S. degrees in flight vehicle design and engineering from the Harbin Institute of Technology, Harbin, China, in 1998 and 2000, respectively, and his Ph.D. degree in control science and engineering from Zhejiang University, Hangzhou, China, in 2013. In 2003, he joined Hangzhou Dianzi University, Hangzhou, China, where he is currently a professor in the School of Automation. His research interests include information fusion and estimated theory. E-mail: dlpeng@hdu.edu.cn
Supported by:
the National Natural Science Foundation of China(61703131);the National Natural Science Foundation of China(61703129);the National Natural Science Foundation of China(61701148);the National Natural Science Foundation of China(61703128);This work was supported by the National Natural Science Foundation of China (61703131; 61703129; 61701148; 61703128)

Abstract

Abstract:

The basic idea of multi-class classification is a disassembly method, which is to decompose a multi-class classification task into several binary classification tasks. In order to improve the accuracy of multi-class classification in the case of insufficient samples, this paper proposes a multi-class classification method combining K-means and multi-task relationship learning (MTRL). The method first uses the split method of One vs. Rest to disassemble the multi-class classification task into binary classification tasks. K-means is used to down sample the dataset of each task, which can prevent over-fitting of the model while reducing training costs. Finally, the sampled dataset is applied to the MTRL, and multiple binary classifiers are trained together. With the help of MTRL, this method can utilize the inter-task association to train the model, and achieve the purpose of improving the classification accuracy of each binary classifier. The effectiveness of the proposed approach is demonstrated by experimental results on the Iris dataset, Wine dataset, Multiple Features dataset, Wireless Indoor Localization dataset and Avila dataset.

Key words: machine learning, multi-class classification, K-means, multi-task relationship learning (MTRL), over-fitting

Mengfan XUE, NLei HA, Dongliang PENG. A combined algorithm of K-means and MTRL for multi-class classification[J]. Journal of Systems Engineering and Electronics, 2019, 30(5): 875-885.

Figures/Tables 14

Fig 1

Fig 2

Table 1

Fig 3

Fig 4

Fig 5

Fig 6

Table 2

Table 3

Table 4

Fig 7

Fig 8

Table 5

Fig 9

References 31

1	ZHOU Z H. Machine learning. Beijing: Tsinghua University Press, 2016.
2	KNERR S, PERSONNAZ L, DREYFUS G. Single-layer learning revisited:a stepwise procedure for building and training a neural network. Neurocomputing, 1990, 68 (11): 41- 50.
3	FRIEDMAN J H. Another approach to polychotomous classification. Technical Report, Departmet of Statistics, Stanford University, 1996.
4	KREBEL U H G. Pairwise classification and support vector machines. Advances in kernel methods: support vector learning. Cambridge, MA: MIT Press, 1999: 255-268.
5	DING S F, ZHANG X K, AN Y X, et al. Weighted linear loss multiple birth support vector machine based on information granulation for multi-class classification. Pattern Recognition, 2017, 67, 32- 46. doi: 10.1016/j.patcog.2017.02.011
6	ZHANG Y, LI B, LU H, et al. Sample-specific SVM learning for person re-identification. Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, 1278- 1287.
7	AN Y X, DING S F, SHI S H, et al. Discrete space reinforcement learning algorithm based on support vector machine classification. Pattern Recognition Letters, 2018, 111, 30- 35. doi: 10.1016/j.patrec.2018.04.012
8	DING S F, AN Y X, ZHANG X K, et al. Wavelet twin support vector machines based on glowworm swarm optimization. Neurocomputing, 2017, 225 (C): 157- 163.
9	DING S F, ZHANG X K, YU J Z. Twin support vector machines based on fruit fly optimization algorithm. International Journal of Machine Learning and Cybernetics, 2016, 7 (2): 193- 203. doi: 10.1007/s13042-015-0424-8
10	MUCHLINSKI D, SIROKY D, HE J R, et al. Comparing random forest with logistic regression for predicting classimbalanced civil war onset data. Political Analysis, 2017, 24 (1): 87- 103.
11	GUO H, LI Y, LI Y, et al. BPSO-adaboost-KNN ensemble learning algorithm for multi-class imbalanced data classification. Engineering Applications of Artificial Intelligence, 2016, 49 (C): 176- 193.
12	DONG G, KUANG G, WANG N, et al. SAR target recognition via joint sparse representation of monogenic signal. IEEE Journal of Selected Topics in Applied Earth Observations&Remote Sensing, 2015, 8 (7): 3316- 3328.
13	ZHENG H, GENG X, TAO D, et al. A multi-task model for simultaneous face identification and facial expression recognition. Neurocomputing, 2016, 171 (C): 515- 523.
14	LI C, BAO W, XU L, et al. Clustered multi-task learning for automatic radar target recognition. Sensors, 2017, 17 (10): 2218. doi: 10.3390/s17102218
15	BICKEL P J, RITOV Y, TSYBAKOV A B. Simultaneous analysis of Lasso and Dantzig selector. Annals of Statistics, 2009, 37 (4): 1705- 1732. doi: 10.1214/08-AOS620
16	ZU C, JIE B, LIU M, et al. Label-aligned multi-task feature learning for multimodal classification of Alzheimer's disease and mild cognitive impairment. Brain Imaging&Behavior, 2016, 10 (4): 1148- 1159.
17	LIU A, LU Y, NIE W, et al. HEp-2 cells classification via clustered multi-task learning. Neurocomputing, 2016, 195 (C): 195- 201.
18	ZHANG Y, YEUNG D Y. A regularization approach to learning task relationships in multitask learning. ACM Trans. on Knowledge Discovery from Data, 2014, 8 (3): 1- 31.
19	EVGENIOU T, PONTIL M. Regularized multi-task learning. Proc. of the 10th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2004, 109- 117.
20	ARGYRIOU A, EVGENIOU T, PONTIL M. Multi-task feature learning. Proc. of the 20th Annual Conference on Neural Information Processing Systems, 2007, 41- 48.
21	ARGYRIOU A, EVGENIOU T, PONTIL M. Convex multitask feature learning. Machine Learning, 2008, 73 (3): 243- 272. doi: 10.1007/s10994-007-5040-8
22	KEERTHI S S, SHEVADE S K. SMO algorithm for leastsquares SVM formulations. Neural Computation, 2003, 15 (2): 487- 507. doi: 10.1162/089976603762553013
23	PIATETSKY-SHAPIRO G. Advances in knowledge discovery and data mining. Menlo Park: AAAI Press, 1996.
24	DUDA R O, HART P E, STORK D G. Pattern classification and scene analysis. 2nd ed. New York: Wiley Interscience, 1995.
25	FELLER W. An introduction to probability theory and its applications. New York: Wiley, 1968.
26	JAIN A K. Data clustering:50 years beyond K-means. Pattern Recognition Letters, 2010, 31 (8): 651- 666. doi: 10.1016/j.patrec.2009.09.011
27	CHADHA A, KUMAR S. An improved K-means clustering algorithm: a step forward for removal of dependency on K. Proc. of the IEEE International Conference on Optimization, Reliabilty, and Information Technology, 2014: 136-140.
28	KAIE S S, BERE S S. An efficient K-means clustering algorithm. International Journal of Engineering Education and Technology, 2015, 3 (2): 1- 8.
29	EL AGHA M, ASHOUR W M. Efficient and fast initialization algorithm for k-means clustering. International Journal of Intelligent Systems and Applications, 2012, 4 (1): 21. doi: 10.5815/ijisa.2012.01.03
30	CHANG C C, LIN C J. LIBSVM: a library for support vector machines. ACM Trans. on Intelligent Systems and Technology, 2011, 2(3): 27.
31	GONG P, YE J, ZHANG C. Robust multi-task feature learning. Proc. of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2012: 895-903.

Class	task1	task2	task3
Positive class	class1	class2	class3
Negative class	class2, class3	class1, class3	class1, class2

Data set		Selected category		Feature dimension	Total number
Iris	setosa	versicolor	virginica	4	150
Wine	1	2	3	13	178
Multiple Features (Zernike moments)	0	1, 2, 3	4	47	1 000
WIL	1	2, 3	4	7	2 000
Avila	E	F, H, II	X	10	4 928(train) 931(test)

Reality	Prediction
Reality	Positive class	Negative dass
Positive class	TP	FN
Negative class	FP	TN

Method	Description
KNN	K-nearest neighbor classifier
SVM	Support vector machine
rMTFL	Select features based on related tasks and identify irrelevant tasks

Dataset	Without K-means/s	With K-means/s
Iris	0.342 9	0.129 0
Wine	0.624 0	0.061 7
WIL	244.083 4	19.982 7
Multiple Features	29.798 0	2.764 8
Avila	13 026	1 347.2

A combined algorithm of K-means and MTRL for multi-class classification

RichHTML

PDF (PC)

Knowledge

Abstract

Cite this article

Share this article

Figures/Tables 14

References 31

Related Articles 10

Recommended Articles

Metrics

Comments

[1]	Zhifei XI, An XU, Yingxin KOU, Zhanwu LI, Aiwu YANG. Target maneuver trajectory prediction based on RBF neural network optimized by hybrid algorithm [J]. Journal of Systems Engineering and Electronics, 2021, 32(2): 498-516.
[2]	Jian ZHAO, Chao ZHANG, Shunli ZHANG, Tingting LU, Weiwen SU, Jian JIA. Pre-detection and dual-dictionary sparse representation based face recognition algorithm in non-sufficient training samples [J]. Journal of Systems Engineering and Electronics, 2018, 29(1): 196-202.
[3]	Jing Yang and Jun Wang. Tag clustering algorithm LMMSK: improved K-means algorithm based on latent semantic analysis [J]. Systems Engineering and Electronics, 2017, 28(2): 374-384.
[4]	Jianfei Ding, Guangya Si, Guoqiang Yang, Yang Liu, and Xiao Liu. Visualization analysis of the capability of weapon system of systems for multi-dimensional indicators [J]. Systems Engineering and Electronics, 2017, 28(2): 292-300.
[5]	Moona Yakhchi, Javier Alonso,Mahdi Fazeli, Amir Akhavan Bitaraf,Ahmad Patooghy. Neural network based approach for time to crash prediction to cope with software aging [J]. Journal of Systems Engineering and Electronics, 2015, 26(2): 407-414.
[6]	Feng Pan, Xiaoting Li, Ting Long, Xiaohui Hu, Tingting Ren, and Junping Du. Novel magnetic field computation model in pattern classification [J]. Journal of Systems Engineering and Electronics, 2013, 24(5): 862-869.
[7]	Pei Wang, Gerhard Reinelt, and Yuejin Tan. Self-adaptive large neighborhood search algorithm for parallel machine scheduling problems [J]. Journal of Systems Engineering and Electronics, 2012, 23(2): 208-215.
[8]	Ling Wang?, Dongmei Fu, Qing Li, and Zhichun Mu. Modelling method with missing values based on clustering and support vector regression [J]. Journal of Systems Engineering and Electronics, 2010, 21(1): 142-147.
[9]	Yan Peng, Zheng Xuefeng, Zhu Jianyong & Xiao Yunhong. Lazy learner text categorization algorithm based on embedded feature selection [J]. Journal of Systems Engineering and Electronics, 2009, 20(3): 651-659.
[10]	Yi Qingming. Blind source separation by weighted K-means clustering [J]. Journal of Systems Engineering and Electronics, 2008, 19(5): 882-887.