面向图神经网络表征学习的类别知识探针

doi:10.6040/j.issn.1671-9352.1.2023.064

Abstract

Abstract:

In order to solve the problem that the graph neural network model lacks corresponding probes, a knowledge detection framework for graph neural network representation learning is proposed, and two kinds of class-aware knowledge probes are designed based on the category attributes of data in different domains, namely clustering probes and contrastive clustering probes. The two probe the characterization effect of different models and give corresponding scores. On 8 datasets in 3 neighborhoods, including reference networks, social networks and biological networks, the representation learning of 7 classical graph neural network models realizes systematic knowledge detection and evaluation experiments, and summarizes the detection and evaluation conclusions.

Key words: graph neural network, knowledge probing, model evaluation, representation learning

CLC Number:

TP391

Xingyu HUANG,Mingyu ZHAO,Ziyu LYU. Category-wise knowledge probers for representation learning of graph neural networks[J].JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2024, 59(7): 85-94.

Figures/Tables 7

Fig.1

Fig.2

Table 1

Table 2

Table 3

Table 4

Fig.3

References 28

1	GONG Liyu, CHENG Qiang. Exploiting edge features for graph neural networks[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach: IEEE, 2019: 9203-9211.
2	VELIČKOVIĆ P, CUCURULL G, CASANOVA A, et al. Graph attention networks[EB/OL]. (2017-12-20)[2023-11-24]. http://arxiv.org/abs/1710.10903.
3	FAN Wenqi, MA Yao, LI Qing, et al. Graph neural networks for social recommendation[C]//The World Wide Web Conference. San Francisco: ACM, 2019: 417-426.
4	CHAUDHARY A, MITTAL H, ARORA A. Anomalydetection using graph neural networks[C]//2019 International Conference on Machine Learning, Big Data, Cloud and Parallel Computing (COMITCon). Faridabad: IEEE, 2019: 346-350.
5	GUO Zhiwei , WANG Heng . A deep graph neural network-based mechanism for social recommendations[J]. IEEE Transactions on Industrial Informatics, 2021, 17 (4): 2776- 2783. doi: 10.1109/TII.2020.2986316
6	RATHI P C , LUDLOW R F , VERDONK M L . Practical high-quality electrostatic potential surfaces for drug discovery using a graph-convolutional deep neural network[J]. Journal of Medicinal Chemistry, 2020, 63 (16): 8778- 8790. doi: 10.1021/acs.jmedchem.9b01129
7	XIONG Zhaoping , WANG Dingyan , LIU Xiaohang , et al. Pushing the boundaries of molecular representation for drug discovery with the graph attention mechanism[J]. Journal of Medicinal Chemistry, 2019, 63 (16): 8749- 8760.
8	YE Xianbin , GUAN Quanlong , LUO Weiqi , et al. Molecular substructure graph attention network for molecular property identification in drug discovery[J]. Pattern Recognition, 2022, 128, 108659. doi: 10.1016/j.patcog.2022.108659
9	PETRONI F, ROCKTÄSCHEL T, LEWIS P, et al. Language models as knowledge bases?[EB/OL]. (2019-09-03)[2023-11-24]. http://arxiv.org/abs/1909.01066.
10	DEFFERRARD M, BRESSON X, VANDERGHEYNST P. Convolutional neural networks on graphs with fast localized spectral filtering[C]//Proceedings of the 30th International Conference on Neural Information Processing Systems. Barcelona: ACM, 2016: 3844-3852.
11	ZHU H, KONIUSZ P. Simple spectral graph convolution[C/OL]//International Conference on Learning Representations. Vienna, Austria, 2021: 1-15. https://openreview.net/pdf?id=CYO5T-YjWZV.
12	HE Xiangnan, DENG Kuan, WANG Xiang, et al. LightGCN: simplifying and powering graph convolution network for recommendation[C]//Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. [S. l. ]: ACM, 2020: 639-648.
13	HAMILTON W L, YING R, LESKOVEC J. Inductive representation learning on large graphs[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems. California: ACM, 2017: 1025-1035.
14	XU K, HU W H, LESKOVEC J, et al. How powerful are graph neural networks?[EB/OL]. (2018-12-26)[2023-11-24]. http://arxiv.org/abs/1810.00826.
15	BELINKOV Y, DURRANI N, DALVI F, et al. What do neural machine translation models learn about morphology?[EB/OL]. (2017-05-15)[2023-11-24]. http://arxiv.org/abs/1704.03471.
16	ADI Y, KERMANY E, BELINKOV Y, et al. Fine-grained analysis of sentence embeddings using auxiliary prediction tasks[EB/OL]. (2016-09-11)[203-11-24]. http://arxiv.org/abs/1608.04207.
17	CONNEAU A, KRUSZEWSKI G, LAMPLE G, et al. What you can cram into a single vector: probing sentence embeddings for linguistic properties[EB/OL]. (2018-05-03)[2023-11-24]. http://arxiv.org/abs/1805.01070.
18	HUPKES D , VELDHOEN S , ZUIDEMA W . Visualisation and 'diagnostic classifiers' reveal how recurrent and recursive neural networks process hierarchical structure[J]. Journal of Artificial Intelligence Research, 2018, 61, 907- 926. doi: 10.1613/jair.1.11196
19	PIMENTEL T, VALVODA J, MAUDSLAY R H, et al. Information-theoretic probing for linguistic structure[EB/OL]. (2020-04-07)[2023-11-24]. http://arxiv.org/abs/2004.03061.
20	WU Zhiyong, CHEN Yun, KAO Ben, et al. Perturbed masking: parameter-free probing for analyzing and interpreting BERT[EB/OL]. (2020-04-30)[2021-11-24]. http://arxiv.org/abs/2004.14786.
21	HEWITT J, LIANG P. Designing and interpreting probes with control tasks[EB/OL]. (2019-09-08)[2023-11-24]. http://arxiv.org/abs/1909.03368.
22	HEWITT J, MANNING C D. A structural probe for finding syntax in word representations[C]//Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Minnesota: ACL, 2019: 4129-4138.
23	YANG Z, COHEN W, SALAKHUDINOV R. Revisiting semi-supervised learning with graph embeddings[C]//International Conference on Machine Learning. New York: ACL, 2016: 40-48.
24	HARPER F M , KONSTAN J A . The MovieLens datasets: history and context[J]. ACM Transactions on Interactive Intelligent Systems, 2015, 5 (4): 1- 19.
25	KHOSLA P , TETERWAK P , WANG C , et al. Supervised contrastive learning[J]. Advances in Neural Information Processing Systems, 2020, 33, 18661- 18673.
26	ZITNIK M , LESKOVEC J . Predicting multicellular function through multi-layer tissue networks[J]. Bioinformatics, 2017, 33 (14): i190- i198. doi: 10.1093/bioinformatics/btx252
27	BORGWARDT K M , ONG C S , SCHÖNAUER S , et al. Protein function prediction via graph kernels[J]. Bioinformatics, 2005, 21 (Suppl 1): i47- i56. doi: 10.1093/bioinformatics/bti1007
28	DEBNATH A K , LOPEZ DE COMPADRE R L , DEBNATH G , et al. Structure-activity relationship of mutagenic aromatic and heteroaromatic nitro compounds. Correlation with molecular orbital energies and hydrophobicity[J]. Journal of Medicinal Chemistry, 1991, 34 (2): 786- 797. doi: 10.1021/jm00106a046

Metrics

Viewed

Full text

Abstract

Cited

Shared

Discussed

Comments

Recommended 10

[1]	WANG Ting-ming,LI Bo-tang . Proof of a class of matrix rank identities[J]. J4, 2007, 42(2): 43 -45 .
[2]	FU Yonghong ¹, YU Miaomiao ²*, TANG Yinghui ³, LI Cailiang ⁴. [J]. J4, 2009, 44(4): 72 -78 .
[3]	GUO Lan-lan1,2, GENG Jie1, SHI Shuo1,3, YUAN Fei1, LEI Li1, DU Guang-sheng1*. Computing research of the water hammer pressure in the process of #br# the variable speed closure of valve based on UDF method[J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2014, 49(03): 27 -30 .
[4]	SHI Kai-quan. P-information law intelligent fusion and soft information #br# image intelligent generation[J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2014, 49(04): 1 -17 .
[5]	ZHANG Ling ,ZHOU De-qun . Research on the relationships among the λ fuzzy measures, Mbius representation and interaction representation[J]. J4, 2007, 42(7): 33 -37 .
[6]	DING Chao1, 2, YUAN Chang-an1, 3, QIN Xiao1, 3. A prediction algorithm for multi-data streams based on GEP[J]. J4, 2010, 45(7): 50 -54 .
[7]	ZHANG De-yu,ZHAI Wen-guang . [J]. J4, 2006, 41(5): 4 -07 .
[8]	MENG Xiang-bo1, ZHANG Li-dong1, DU Zi-ping2. Investment and reinsurance strategy for insurers under #br# mean-variance criterion with jumps#br#[J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2014, 49(05): 36 -40 .
[9]	PENG Zhen-hua, XU Yi-hong*, TU Xiang-qiu. Optimality conditions for weakly efficient elements of nearly preinvex set-valued optimizaton#br#[J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2014, 49(05): 41 -44 .
[10]	YUAN Hun-ping . Schur factorization and normal matrices factorization of row (column) symmetric matrices[J]. J4, 2007, 42(10): 123 -126 .

图嵌入模型图	神经网络	特点	下游任务
图嵌入模型图	神经网络	特点	节点分类	边预测	图分类
谱域	Chebyshev	切比雪夫多项式近似拟合		√
谱域	SSGCN	马尔可夫扩散核	√
空域	GCN	拉普拉斯正则项	√
	LightGCN	推荐任务中的简化GCN		√
	GraphSAGE	随机定长采样策略	√
	GAT	自注意力机制	√
	GIN	可微分非线性函数			√

神经网络	节点分类直推式		节点分类归纳式		边预测		图分类
神经网络	Cora	Citeseer	PPI	Flickr	Yelp	MovieLens	MUTAG	ENZYMES
Chebyshev	20.1^*	19.9	45.8	44.0	5.10^*	5.75^*	71.7	20.0
SSGCN	43.6	29.1	38.8	44.8	8.87	7.09	61.5	18.3
GCN	35.8	26.3	46.1	43.9	8.72	6.55	66.7	17.5
LightGCN	21.6	18.1	39.6	45.2	5.27	6.62	61.5	15.0^*
GraphSAGE	39.3	21.7	38.7^*	42.8^*	7.27	7.78	74.4	15.8
GAT	47.8	35.0	43.3	45.4	7.18	6.20	61.4^*	20.8
GIN	40.2	25.0	45.7	45.3	7.46	7.82	71.8	20.8
MLP	15.8	16.4^*	39.6	45.3	5.14	7.05	71.8	17.5

特征初始化	神经网络	直推式						归纳式
		节点分类			边预测			节点分类	图分类
		Cora	Citeseer	Pubmed	Cora	Citeseer	Pubmed	Flickr	Mutag	Proteins	Enzymes
rand64	Chebyshev	12.009	20.165	72.238	31.191	42.054	91.277	39.503	0.973	1.840	2.690
	SSGCN	5.486	14.794	32.226	10.837	19.565	22.696	55.523	4.210^*	3.360	9.800
	GCN	5.768	12.439	37.887	13.860	25.343	36.635	48.492	1.330	2.140	3.280
	LightGCN	13.457	22.640	80.393	21.463	41.126	48.426	67.668^*	4.190	4.010^*	9.920^*
	GraphSAGE	4.861	16.684	31.617	40.150^*	72.680^*	226.829	20.293	1.030	1.710	3.180
	GAT	4.130	9.288	13.888	13.624	18.929	39.145	29.749	1.890	2.590	3.390
	GIN	4.685	8.543	11.721	8.953	17.695	15.999	25.363	2.380	3.600	6.370
	MLP	25.708^*	51.193^*	130.612^*	38.894	50.574	304.132^*	59.705	1.130	2.130	4.650
meta	Chebyshev	1.414	2.288	1.411	43.538^*	48.376^*	363.183^*	10.549	0.955	1.760	2.650
	SSGCN	1.156	2.126	1.486	4.705	5.079	40.599	15.900	1.020^*	1.790	11.100
	GCN	1.139	2.080	1.372	6.107	6.347	36.815	9.352	0.959	1.830^*	4.270
	LightGCN	1.404	2.202	1.559	11.948	13.196	5.316	11.857	1.020^*	1.790	13.000^*
	GraphSAGE	1.213	2.672	1.328	40.289	44.554	56.649	10.009	1.020^*	1.790	2.970
	GAT	1.116	2.077	1.370	6.985	5.356	23.751	11.535	0.977	1.800	6.250
	GIN	2.403	3.990	3.784^*	5.945	6.143	2.281	20.049^*	0.941	1.660	4.320
	MLP	3.644^*	4.808^*	3.674	16.824	37.931	33.164	8.206	0.929	1.790	2.590

特征初始化	神经网络	直推式						归纳式
		节点分类			边预测			节点分类	图分类
		Cora	Citeseer	Pubmed	Cora	Citeseer	Pubmed	Flickr	Mutag	Proteins	Enzymes
rand64	Chebyshev	9.792	28.594	128.167^*	31.398	60.836	135.442	23.610	0.736	0.873	1.558
	SSGCN	3.309	12.486	13.542	6.364	10.814	8.344	32.690	1.174	1.962	5.921
	GCN	4.696	16.306	100.546	8.339	29.539	19.237	74.333^*	0.905	0.853	1.571
	LightGCN	8.637	16.282	45.771	19.462	35.508	32.117	31.168	1.052	1.969	5.806
	GraphSAGE	4.394	23.678	32.052	38.923	86.968	41.095	18.306	0.447	0.828	1.619
	GAT	2.954	7.742	11.671	41.778^*	107.861^*	173.833^*	23.108	0.743	0.976	2.373
	GIN	2.927	5.268	4.172	8.644	15.958	10.836	20.066	2.137^*	2.117^*	8.019^*
	MLP	48.342^*	79.993^*	69.205	35.218	48.810	68.031	36.763	0.739	1.274	3.300
meta	Chebyshev	1.463	2.018	1.309	20.779	54.636^*	24.330	9.315	1.082^*	1.943	1.981
	SSGCN	1.189	2.181	1.293	2.193	2.528	19.252	9.310	1.009	1.839	7.900
	GCN	1.328	1.940	1.273	4.800	6.845	21.449	17.373	0.921	1.819	3.130
	LightGCN	1.506	1.953	1.241	6.965	12.028	5.045	11.320	0.867	1.929	10.163^*
	GraphSAGE	1.159	2.231	1.535	45.428^*	45.748	44.089^*	10.774	0.928	1.288	1.216
	GAT	1.162	2.041	1.233	6.178	8.451	38.720	9.696	0.850	1.850	7.037
	GIN	1.864	3.143	1.727	4.223	4.336	3.066	12.154	0.772	2.095^*	2.630
	MLP	2.602^*	3.897^*	1.846^*	11.797	16.880	13.051	19.922^*	0.859	1.799	1.889

Category-wise knowledge probers for representation learning of graph neural networks

RichHTML

PDF (PC)

Abstract

Cite this article

share this article

Figures/Tables 7

References 28

Related Articles 5

Metrics

Comments

Recommended 10

[1]	LUO Qi, GOU Gang. Multimodal conversation emotion recognition based on clustering and group normalization [J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2024, 59(7): 105-112.
[2]	Xinsheng WANG,Xiaofei ZHU,Chenghong LI. Label guided multi-scale graph neural network for protein-protein interactions prediction [J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2023, 58(12): 22-30.
[3]	ZHANG Bin-yan, ZHU Xiao-fei, XIAO Zhao-hui, HUANG Xian-ying, WU Jie. Short text classification based on semi-supervised graph neural network [J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2021, 56(5): 57-65.
[4]	YI Jie, ZHONG Mao-sheng, LIU Gen, WANG Ming-wen. Chinese word representation learning based on Gaussian distribution and Chinese character component characteristics [J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2021, 56(5): 85-91.
[5]	YU Chuan-ming, FENG Bo-lin, TIAN Xin, AN Lu. Deep representative learning based sentiment analysis in the cross-lingual environment [J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2018, 53(3): 13-23.