面向图神经网络表征学习的类别知识探针

doi:10.6040/j.issn.1671-9352.1.2023.064

摘要/Abstract

摘要：

针对图神经网络(graph neural network, GNN)模型缺乏相应的探针这一问题，提出面向图神经网络表征学习的知识探测框架，基于不同领域数据的类别属性设计2种类别感知的知识探针，分别为聚类探针和对比聚类探针。2种探针分别探测不同模型的表征效果并给出相应的分数。在引用网络、社交网络和生物网络等3个邻域的8个数据集上，对7个经典的图神经网络模型的表征学习实现了系统性地知识探测和评估实验，归纳出探测和评估结论。

关键词: 图神经网络, 知识探针, 模型评价, 表征学习

Abstract:

In order to solve the problem that the graph neural network model lacks corresponding probes, a knowledge detection framework for graph neural network representation learning is proposed, and two kinds of class-aware knowledge probes are designed based on the category attributes of data in different domains, namely clustering probes and contrastive clustering probes. The two probe the characterization effect of different models and give corresponding scores. On 8 datasets in 3 neighborhoods, including reference networks, social networks and biological networks, the representation learning of 7 classical graph neural network models realizes systematic knowledge detection and evaluation experiments, and summarizes the detection and evaluation conclusions.

Key words: graph neural network, knowledge probing, model evaluation, representation learning

中图分类号:

TP391

黄兴宇,赵明宇,吕子钰. 面向图神经网络表征学习的类别知识探针[J]. 《山东大学学报(理学版)》, 2024, 59(7): 85-94.

Xingyu HUANG,Mingyu ZHAO,Ziyu LYU. Category-wise knowledge probers for representation learning of graph neural networks[J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2024, 59(7): 85-94.

图/表 7

图1

图2

表1

表2

表3

表4

图3

参考文献 28

1	GONG Liyu, CHENG Qiang. Exploiting edge features for graph neural networks[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach: IEEE, 2019: 9203-9211.
2	VELIČKOVIĆ P, CUCURULL G, CASANOVA A, et al. Graph attention networks[EB/OL]. (2017-12-20)[2023-11-24]. http://arxiv.org/abs/1710.10903.
3	FAN Wenqi, MA Yao, LI Qing, et al. Graph neural networks for social recommendation[C]//The World Wide Web Conference. San Francisco: ACM, 2019: 417-426.
4	CHAUDHARY A, MITTAL H, ARORA A. Anomalydetection using graph neural networks[C]//2019 International Conference on Machine Learning, Big Data, Cloud and Parallel Computing (COMITCon). Faridabad: IEEE, 2019: 346-350.
5	GUO Zhiwei , WANG Heng . A deep graph neural network-based mechanism for social recommendations[J]. IEEE Transactions on Industrial Informatics, 2021, 17 (4): 2776- 2783. doi: 10.1109/TII.2020.2986316
6	RATHI P C , LUDLOW R F , VERDONK M L . Practical high-quality electrostatic potential surfaces for drug discovery using a graph-convolutional deep neural network[J]. Journal of Medicinal Chemistry, 2020, 63 (16): 8778- 8790. doi: 10.1021/acs.jmedchem.9b01129
7	XIONG Zhaoping , WANG Dingyan , LIU Xiaohang , et al. Pushing the boundaries of molecular representation for drug discovery with the graph attention mechanism[J]. Journal of Medicinal Chemistry, 2019, 63 (16): 8749- 8760.
8	YE Xianbin , GUAN Quanlong , LUO Weiqi , et al. Molecular substructure graph attention network for molecular property identification in drug discovery[J]. Pattern Recognition, 2022, 128, 108659. doi: 10.1016/j.patcog.2022.108659
9	PETRONI F, ROCKTÄSCHEL T, LEWIS P, et al. Language models as knowledge bases?[EB/OL]. (2019-09-03)[2023-11-24]. http://arxiv.org/abs/1909.01066.
10	DEFFERRARD M, BRESSON X, VANDERGHEYNST P. Convolutional neural networks on graphs with fast localized spectral filtering[C]//Proceedings of the 30th International Conference on Neural Information Processing Systems. Barcelona: ACM, 2016: 3844-3852.
11	ZHU H, KONIUSZ P. Simple spectral graph convolution[C/OL]//International Conference on Learning Representations. Vienna, Austria, 2021: 1-15. https://openreview.net/pdf?id=CYO5T-YjWZV.
12	HE Xiangnan, DENG Kuan, WANG Xiang, et al. LightGCN: simplifying and powering graph convolution network for recommendation[C]//Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. [S. l. ]: ACM, 2020: 639-648.
13	HAMILTON W L, YING R, LESKOVEC J. Inductive representation learning on large graphs[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems. California: ACM, 2017: 1025-1035.
14	XU K, HU W H, LESKOVEC J, et al. How powerful are graph neural networks?[EB/OL]. (2018-12-26)[2023-11-24]. http://arxiv.org/abs/1810.00826.
15	BELINKOV Y, DURRANI N, DALVI F, et al. What do neural machine translation models learn about morphology?[EB/OL]. (2017-05-15)[2023-11-24]. http://arxiv.org/abs/1704.03471.
16	ADI Y, KERMANY E, BELINKOV Y, et al. Fine-grained analysis of sentence embeddings using auxiliary prediction tasks[EB/OL]. (2016-09-11)[203-11-24]. http://arxiv.org/abs/1608.04207.
17	CONNEAU A, KRUSZEWSKI G, LAMPLE G, et al. What you can cram into a single vector: probing sentence embeddings for linguistic properties[EB/OL]. (2018-05-03)[2023-11-24]. http://arxiv.org/abs/1805.01070.
18	HUPKES D , VELDHOEN S , ZUIDEMA W . Visualisation and 'diagnostic classifiers' reveal how recurrent and recursive neural networks process hierarchical structure[J]. Journal of Artificial Intelligence Research, 2018, 61, 907- 926. doi: 10.1613/jair.1.11196
19	PIMENTEL T, VALVODA J, MAUDSLAY R H, et al. Information-theoretic probing for linguistic structure[EB/OL]. (2020-04-07)[2023-11-24]. http://arxiv.org/abs/2004.03061.
20	WU Zhiyong, CHEN Yun, KAO Ben, et al. Perturbed masking: parameter-free probing for analyzing and interpreting BERT[EB/OL]. (2020-04-30)[2021-11-24]. http://arxiv.org/abs/2004.14786.
21	HEWITT J, LIANG P. Designing and interpreting probes with control tasks[EB/OL]. (2019-09-08)[2023-11-24]. http://arxiv.org/abs/1909.03368.
22	HEWITT J, MANNING C D. A structural probe for finding syntax in word representations[C]//Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Minnesota: ACL, 2019: 4129-4138.
23	YANG Z, COHEN W, SALAKHUDINOV R. Revisiting semi-supervised learning with graph embeddings[C]//International Conference on Machine Learning. New York: ACL, 2016: 40-48.
24	HARPER F M , KONSTAN J A . The MovieLens datasets: history and context[J]. ACM Transactions on Interactive Intelligent Systems, 2015, 5 (4): 1- 19.
25	KHOSLA P , TETERWAK P , WANG C , et al. Supervised contrastive learning[J]. Advances in Neural Information Processing Systems, 2020, 33, 18661- 18673.
26	ZITNIK M , LESKOVEC J . Predicting multicellular function through multi-layer tissue networks[J]. Bioinformatics, 2017, 33 (14): i190- i198. doi: 10.1093/bioinformatics/btx252
27	BORGWARDT K M , ONG C S , SCHÖNAUER S , et al. Protein function prediction via graph kernels[J]. Bioinformatics, 2005, 21 (Suppl 1): i47- i56. doi: 10.1093/bioinformatics/bti1007
28	DEBNATH A K , LOPEZ DE COMPADRE R L , DEBNATH G , et al. Structure-activity relationship of mutagenic aromatic and heteroaromatic nitro compounds. Correlation with molecular orbital energies and hydrophobicity[J]. Journal of Medicinal Chemistry, 1991, 34 (2): 786- 797. doi: 10.1021/jm00106a046

多维度评价

Viewed

Full text

Abstract

Cited

Shared

Discussed

图嵌入模型图	神经网络	特点	下游任务
图嵌入模型图	神经网络	特点	节点分类	边预测	图分类
谱域	Chebyshev	切比雪夫多项式近似拟合		√
谱域	SSGCN	马尔可夫扩散核	√
空域	GCN	拉普拉斯正则项	√
	LightGCN	推荐任务中的简化GCN		√
	GraphSAGE	随机定长采样策略	√
	GAT	自注意力机制	√
	GIN	可微分非线性函数			√

神经网络	节点分类直推式		节点分类归纳式		边预测		图分类
神经网络	Cora	Citeseer	PPI	Flickr	Yelp	MovieLens	MUTAG	ENZYMES
Chebyshev	20.1^*	19.9	45.8	44.0	5.10^*	5.75^*	71.7	20.0
SSGCN	43.6	29.1	38.8	44.8	8.87	7.09	61.5	18.3
GCN	35.8	26.3	46.1	43.9	8.72	6.55	66.7	17.5
LightGCN	21.6	18.1	39.6	45.2	5.27	6.62	61.5	15.0^*
GraphSAGE	39.3	21.7	38.7^*	42.8^*	7.27	7.78	74.4	15.8
GAT	47.8	35.0	43.3	45.4	7.18	6.20	61.4^*	20.8
GIN	40.2	25.0	45.7	45.3	7.46	7.82	71.8	20.8
MLP	15.8	16.4^*	39.6	45.3	5.14	7.05	71.8	17.5

特征初始化	神经网络	直推式						归纳式
		节点分类			边预测			节点分类	图分类
		Cora	Citeseer	Pubmed	Cora	Citeseer	Pubmed	Flickr	Mutag	Proteins	Enzymes
rand64	Chebyshev	12.009	20.165	72.238	31.191	42.054	91.277	39.503	0.973	1.840	2.690
	SSGCN	5.486	14.794	32.226	10.837	19.565	22.696	55.523	4.210^*	3.360	9.800
	GCN	5.768	12.439	37.887	13.860	25.343	36.635	48.492	1.330	2.140	3.280
	LightGCN	13.457	22.640	80.393	21.463	41.126	48.426	67.668^*	4.190	4.010^*	9.920^*
	GraphSAGE	4.861	16.684	31.617	40.150^*	72.680^*	226.829	20.293	1.030	1.710	3.180
	GAT	4.130	9.288	13.888	13.624	18.929	39.145	29.749	1.890	2.590	3.390
	GIN	4.685	8.543	11.721	8.953	17.695	15.999	25.363	2.380	3.600	6.370
	MLP	25.708^*	51.193^*	130.612^*	38.894	50.574	304.132^*	59.705	1.130	2.130	4.650
meta	Chebyshev	1.414	2.288	1.411	43.538^*	48.376^*	363.183^*	10.549	0.955	1.760	2.650
	SSGCN	1.156	2.126	1.486	4.705	5.079	40.599	15.900	1.020^*	1.790	11.100
	GCN	1.139	2.080	1.372	6.107	6.347	36.815	9.352	0.959	1.830^*	4.270
	LightGCN	1.404	2.202	1.559	11.948	13.196	5.316	11.857	1.020^*	1.790	13.000^*
	GraphSAGE	1.213	2.672	1.328	40.289	44.554	56.649	10.009	1.020^*	1.790	2.970
	GAT	1.116	2.077	1.370	6.985	5.356	23.751	11.535	0.977	1.800	6.250
	GIN	2.403	3.990	3.784^*	5.945	6.143	2.281	20.049^*	0.941	1.660	4.320
	MLP	3.644^*	4.808^*	3.674	16.824	37.931	33.164	8.206	0.929	1.790	2.590

特征初始化	神经网络	直推式						归纳式
		节点分类			边预测			节点分类	图分类
		Cora	Citeseer	Pubmed	Cora	Citeseer	Pubmed	Flickr	Mutag	Proteins	Enzymes
rand64	Chebyshev	9.792	28.594	128.167^*	31.398	60.836	135.442	23.610	0.736	0.873	1.558
	SSGCN	3.309	12.486	13.542	6.364	10.814	8.344	32.690	1.174	1.962	5.921
	GCN	4.696	16.306	100.546	8.339	29.539	19.237	74.333^*	0.905	0.853	1.571
	LightGCN	8.637	16.282	45.771	19.462	35.508	32.117	31.168	1.052	1.969	5.806
	GraphSAGE	4.394	23.678	32.052	38.923	86.968	41.095	18.306	0.447	0.828	1.619
	GAT	2.954	7.742	11.671	41.778^*	107.861^*	173.833^*	23.108	0.743	0.976	2.373
	GIN	2.927	5.268	4.172	8.644	15.958	10.836	20.066	2.137^*	2.117^*	8.019^*
	MLP	48.342^*	79.993^*	69.205	35.218	48.810	68.031	36.763	0.739	1.274	3.300
meta	Chebyshev	1.463	2.018	1.309	20.779	54.636^*	24.330	9.315	1.082^*	1.943	1.981
	SSGCN	1.189	2.181	1.293	2.193	2.528	19.252	9.310	1.009	1.839	7.900
	GCN	1.328	1.940	1.273	4.800	6.845	21.449	17.373	0.921	1.819	3.130
	LightGCN	1.506	1.953	1.241	6.965	12.028	5.045	11.320	0.867	1.929	10.163^*
	GraphSAGE	1.159	2.231	1.535	45.428^*	45.748	44.089^*	10.774	0.928	1.288	1.216
	GAT	1.162	2.041	1.233	6.178	8.451	38.720	9.696	0.850	1.850	7.037
	GIN	1.864	3.143	1.727	4.223	4.336	3.066	12.154	0.772	2.095^*	2.630
	MLP	2.602^*	3.897^*	1.846^*	11.797	16.880	13.051	19.922^*	0.859	1.799	1.889

[1]	罗奇,苟刚. 基于聚类和群组归一化的多模态对话情绪识别[J]. 《山东大学学报(理学版)》, 2024, 59(7): 105-112.
[2]	王新生,朱小飞,李程鸿. 标签指导的多尺度图神经网络蛋白质作用关系预测方法[J]. 《山东大学学报(理学版)》, 2023, 58(12): 22-30.
[3]	张斌艳,朱小飞,肖朝晖,黄贤英,吴洁. 基于半监督图神经网络的短文本分类[J]. 《山东大学学报(理学版)》, 2021, 56(5): 57-65.