DHSEGATs: distance and hop-wise structures encoding enhanced graph attention networks

doi:10.23919/JSEE.2023.000057

Abstract

Abstract:

Numerous works prove that existing neighbor-averaging graph neural networks (GNNs) cannot efficiently catch structure features, and many works show that injecting structure, distance, position, or spatial features can significantly improve the performance of GNNs, however, injecting high-level structure and distance into GNNs is an intuitive but untouched idea. This work sheds light on this issue and proposes a scheme to enhance graph attention networks (GATs) by encoding distance and hop-wise structure statistics. Firstly, the hop-wise structure and distributional distance information are extracted based on several hop-wise ego-nets of every target node. Secondly, the derived structure information, distance information, and intrinsic features are encoded into the same vector space and then added together to get initial embedding vectors. Thirdly, the derived embedding vectors are fed into GATs, such as GAT and adaptive graph diffusion network (AGDN) to get the soft labels. Fourthly, the soft labels are fed into correct and smooth (C&S) to conduct label propagation and get final predictions. Experiments show that the distance and hop-wise structures encoding enhanced graph attention networks (DHSEGATs) achieve a competitive result.

Key words: graph attention network (GAT), graph structure information, label propagation

Zhiguo HUANG. DHSEGATs: distance and hop-wise structures encoding enhanced graph attention networks[J]. Journal of Systems Engineering and Electronics, 2023, 34(2): 350-359.

Figures/Tables 12

Fig 1

Table 1

Table 2

Table 3

Table 4

Table 5

Table 6

Table 7

Table 8

Table 9

Table 10

Table 11

References 30

1	LI Q M, HAN Z C, WU X M. Deeper insights into graph convolutional networks for semi-supervised learning. Proc. of the 32nd AAAI Conference on Artificial Intelligence, 2018: 3538−3545.
2	OONO K, SUZUKI T. Graph neural networks exponentially lose expressive power for node classification. Proc. of the International Conference on Learning Representations, 2020: 1−37.
3	DEHMAMY N, BARABASI A L, YU R Understanding the representation power of graph neural networks in learning graph topology. Advances in Neural Information Processing Systems, 2019, 32, 15413- 15423.
4	MONTI F, OTNESS K, BRONSTEIN M M. Motifnet: a motif-based graph convolutional network for directed graphs. Proc. of the IEEE Data Science Workshop, 2018: 225−228.
5	BOURITSAS G, FRASCA F, ZAFEIRIOU S P, et al Improving graph neural network expressivity via subgraph isomorphism counting. IEEE Trans. on Pattern Analysis and Machine Intelligence, 2022, 45 (1): 657- 668.
6	YOU J X, GOMES-SELMAN J M, YING R, et al Identity-aware graph neural networks. Proc. of the AAAI Conference on Artificial Intelligence, 2021, 35, 10737- 10745. doi: 10.1609/aaai.v35i12.17283
7	YING C X, CAI T L, LUO S J, et al. Do transformers really perform badly for graph representation? Advances in Neural Information Processing Systems, 2021, 34: 28877−28888.
8	YOU J X, YING R, LESKOVEC J Position-aware graph neural networks. Proc. of the International Conference on Machine Learning, 2019, 97, 7134- 7143.
9	LI P, WANG Y B, WANG H W, et al Distance encoding: design provably more powerful neural networks for graph representation learning. Advances in Neural Information Processing Systems, 2020, 33, 4465- 4478.
10	ALSENTZER E, FINLAYSON S, LI M, et al. Subgraph neural networks. Advances in Neural Information Processing Systems, 2020, 33: 8017−8029.
11	DASOULAS G, SANTOS L D, SCAMAN K, et al. Coloring graph neural networks for node disambiguation. Proc. of the 29th International Joint Conference on Artificial Intelligence and the 17th Pacific Rim International Conference on Artificial Intelligence, 2021, 294: 2126−2132.
12	XU K, HU W H, LESKOVEC J, et al. How powerful are graph neural networks? Proc. of the International Conference on Learning Representations, 2019: 1−17.
13	ARVIND V, FUHLBRUCK F, KOBLER J, et al On Weisfeiler-Leman invariance: subgraph counts and related graph properties. Journal of Computer and System Sciences, 2020, 113, 42- 59.
14	CHEN Z D, CHEN L, VILLAR S, et al. Can graph neural networks count substructures? Advances in Neural Information Processing Systems, 2020, 33: 10383−10395.
15	VIGNAC C, LOUKAS A, FROSSARD P Building powerful and equivariant graph neural networks with structural message-passing. Advances in Neural Information Processing Systems, 2020, 33, 14143- 14155.
16	HOANG N, MAEHARA T, MURATA T. Revisiting graph neural networks: graph filtering perspective. Proc. of the 25th International Conference on Pattern Recognition, 2021: 8376−8383.
17	ROSSI E, FRASCA F, CHAMBERLAIN B, et al. Sign: scalable inception graph neural networks. https://arxiv.org/abs/2004.11198v1.
18	SUN C X, GU H M, HU J. Scalable and adaptive graph neural networks with self-label-enhanced training. https://arxiv.org/abs/2104.09376.
19	VELICKOVIC P, CUCURULL G, CASANOVA A, et al. Graph attention networks. Proc. of the International Conference on Learning Representations, 2018: 1−12.
20	YUN S J, JEONG M, KIM R, et al Graph transformer networks. Advances in Neural Information Processing Systems, 2019, 32, 11983- 11993.
21	HU Z N, DONG Y X, WANG K S, et al. GPT-GNN: generative pre-training of graph neural networks. Proc. of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2020: 1857−1867.
22	HU Z N, DONG Y X, WANG K S, et al. Heterogeneous graph transformer. Proc. of the Web Conference, 2020: 2704−2710.
23	SUN C X, HU J, GU H M, et al. Adaptive graph diffusion networks. https://arxiv.org/abs/2012.15024.
24	WANG H W, LESKOVEC J. Unifying graph convolutional neural networks and label propagation. http://arxiv.org/abs/2002.06755.
25	QU M, BENGIO Y, TANG J. Gmnn: graph Markov neural networks. Proc. of the International Conference on Machine Learning, 2019: 5241−5250.
26	GAO H C, PEI J, HUANG H. Conditional random field enhanced graph convolutional neural networks. Proc. of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2019: 276−284.
27	ZHENG Q K, LI H Y, ZHANG P, et al. GIPA: general information propagation algorithm for graph learning. https://arxiv.org/abs/2105.06035v2.
28	GASTEIGER J, BOJCHEVSKI A, GUNNEMANN S. Predict then propagate: graph neural networks meet personalized PageRank. Proc. of the International Conference on Learning Representations, 2019: 1−15.
29	HUANG Q, HE H, SINGH A, et al. Combining label propagation and simple models out-performs graph neural networks. Proc. of the International Conference on Learning Representations, 2021: 1−21.
30	SHI Y S, HUANG Z J, FENG S K, et al. Masked label prediction: unified message passing model for semi-supervised classification. Proc. of the 30th International Joint Conference on Artificial Intelligence, 2021: 1548−1554.

Hyperparameter	Value	Hyperparameter	Value
num_heads	3	scheduler	Cosine
learning_rate	0.002	num_layers	3
optimizer	Adam	num_hiddens	256

Hyperparameter	Value	Hyperparameter	Value
smooth_adj	DAD	correct_adj	DAD
smooth_alpha	0.756	correct_alpha	0.979
smooth_layers	50	correct_layers	50

Model	Valid accuracy	Test accuracy
UniMP_v2	0.7506 ± 0.0009	0.7397 ± 0.0015
GAT + C&S	0.7484 ± 0.0007	0.7386 ± 0.0014
AGDN (GAT-HA+3_heads)	0.7483 ± 0.0009	0.7375 ± 0.0021
AGDN+C&S	0.7471 ± 0.0007	0.7387 ± 0.0018
DHSEGAT	0.7441 ± 0.0012	0.7425 ± 0.0016
DHSEAGDN	0.7462 ± 0.0013	0.7439 ± 0.0019

Model	Valid accuracy	Test accuracy
DHSE+AGDN	0.7427 ± 0.0007	0.7283 ± 0.0030
DHSE+AGDN+C&S	0.7462 ± 0.0013	0.7439 ± 0.0019
AGDN	0.7429 ± 0.0010	0.7297 ± 0.0021
AGDN+C&S	0.7471 ± 0.0007	0.7387 ± 0.0018

Model	Valid accuracy	Test accuracy
DHSE+GAT	0.7412 ± 0.0008	0.7261 ± 0.0021
DHSE+GAT+C&S	0.7441 ± 0.0012	0.7425 ± 0.0016
GAT	0.7419 ± 0.0005	0.7273 ± 0.0012
GAT+C&S	0.7440 ± 0.0012	0.7371 ± 0.0023