JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE) ›› 2021, Vol. 56 ›› Issue (5): 85-91.doi: 10.6040/j.issn.1671-9352.1.2020.032

Previous Articles    

Chinese word representation learning based on Gaussian distribution and Chinese character component characteristics

YI Jie, ZHONG Mao-sheng*, LIU Gen, WANG Ming-wen   

  1. Computer and Information Engineering College, Jiangxi Normal University, Nanchang 330022, Jiangxi, China
  • Published:2021-05-13

PDF (PC)

229

Abstract: We use a distributed embedded representation based on density, and give a method to learn the space representation of the Gaussian distribution, so as to better capture the uncertainty about the representation and its relationship, to express the asymmetry of the words more naturally than the dot product cosine similarity. At the same time, according to the characteristics of Chinese characters, the semantic information of the Chinese characters components is added to the word embedding training. Compared with existing methods, our model has better performance in terms of word similarity or downstream tasks, and can express the uncertainty of words.

Key words: word representation learning, Gaussian distribution, Chinese characters components, semantic uncertainty

CLC Number: 

  • TP391
[1] ZHANG Yue, YANG Jie. Chinese NER using lattice LSTM[C] //Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics. Melbourne: ACL, 2018: 1554-1564.
[2] XIAO Huiru, LIU Xin, SONG Yangqiu. Efficient path prediction for semi-supervised and weakly supervised hier-archical text classification[C] //Proceedings of the 2019 World Wide Web Conference.San Francisco: WWW, 2019: 3370-3376.
[3] LI Xin, BING Lidong, LI Piji, et al. A unified model for opinion target extraction and target sentiment prediction[EB/OL]. arXiv, 2018. arXiv: 1811.05082.
[4] KRISHNA K, IYYER M. Generating question-answer hierarchies[EB/OL]. arXiv, 2019. arXiv:1906.02622.
[5] MIKOLOV T, CHEN K, CORRADO G S, et al. Efficient estimation of word representations in vector space[EB/OL]. arXiv, 2013. arXiv:1301.3781.
[6] MIKOLOV T, CHEN K, CORRADO G S, et al. Distributed representations of words and phrases and their compositionality[C] //Proceedings of the 26th International Conference on Neural Information Processing Systems-Volume 2. New York: ACM, 2013: 3111-3119.
[7] VILNIS L, MCCALLUM A. Word representations via gaussian embedding[EB/OL]. arXiv, 2014. arXiv:1412.6623.
[8] ATHIWARATKUN B, WILSON A G. Multimodal word distributions[EB/OL]. arXiv, 2017. arXiv:1704.08424.
[9] PENNINGTON J, SOCHER R, MANNING C. Glove: global vectors for word representation[C] //Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing(EMNLP). Doha: ACL, 2014: 1532-1543.
[10] SUN Yaming, LIN Lei, YANG Nan, et al. Radical-enhanced chinese character embedding[C] //Proceedings of the International Conference on Neural Information Processing. Cham: Springer, 2014: 279-286.
[11] LI Yanran, LI Wenjie, SUN Fei, et al. Component-enhanced chinese character embeddings[EB/OL]. arXiv, 2015. arXiv:1508.06669.
[12] COLLOBERT R, WESTON J. A unified architecture for natural language processing: deep neural networks with multitask learning[C] //Proceedings of the 25th International Conference on Machine Learning. Helsinki: ICML, 2008: 160-167.
[13] CHEN Xinxiong, XU Lei, LIU Zhiyuan, et al. Joint learning of character and word embeddings[C] //Proceedings of the Twenty-fourth International Joint Conference on Artificial Intelligence. Buenos Aires: IJCAI, 2015.
[14] SU T R, LEE H Y. Learning chinese word repre-sentations from glyphs of characters[EB/OL]. arXiv, 2017. arXiv:1708.04755.
[15] YU Jinxing, JIAN Xun, XIN Hao, et al. Joint embeddings of chinese words, characters, and fine-grained subcharacter components[C] //Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Copenhagen: ACL, 2017: 286-291.
[16] CAO Shaosheng, LU Wei, ZHOU Jun, et al. cw2vec: Learning chinese word embeddings with stroke n-gram information[C] ///Proceedings of the Thirty-second AAAI Conference on Artificial Intelligence. Palo Alto: AAAI Press, 2018.
[17] LECUN Y, CHOPRA S, HADSELL R, et al. A tutorial on energy-based learning[M] //Predicting Structured Data. Cambridge: MIT Press, 2006.
[18] JOACHIMS T. Optimizing search engines using clickthrough data[C] //Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. New York: ACM, 2002: 133-142.
[19] WESTON J, BENGIO S, USUNIER N. Wsabie: scaling up to large vocabulary image annotation[C] //Proceedings of the Twenty-second International Joint Conference on Artificial Intelligence. Barcelona: IJCAI, 2011.
[20] DUCHI J, HAZAN E, SINGER Y. Adaptive subgradient methods for online learning and stochastic optimization[J]. Journal of Machine Learning Research, 2011, 12(7):257-269.
[21] 汪祥,贾焰,周斌,等. 基于中文维基百科链接结构与分类体系的语义相关度计算[J]. 小型微型计算机系统, 2011, 32(11):2237-2242. WANG Xiang, JIA Yan, ZHOU Bin, et al. Computing semantic relatedness using Chinese Wikipedia links and taxonom[J]. Journal of Chinese Computer Systems, 2011, 32(11):2237-2242.
[22] JIN P, WU Y F. SemEval-2012 Task 4: evaluating Chinese word similarity[M] //Proceedings of the First Joint Conference on Lexical and Computational Semantics. Montreal:[s.n.] , 2012: 374-377.
[23] 董强,董振东.基于知网的相关概念场的构建[M] //语言计算与基于内容的文本处理.北京: 清华大学出版社,2003:364-370. DONG Qiang, DONG Zhendong. Construction of hownet-based relevant concept field[C] //Language Computing and Content-based Text Processing Translation. Beijing: Tsinghua University Press, 2003: 364-370.
[1] Yi-ming ZHANG,Guo-yin WANG,Jun HU,Shun FU. Overlapping community detection based on density peaks and network embedding [J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2021, 56(1): 91-102.
[2] Kan XU,Rui-xin LIU,Hong-fei LIN,Hai-feng LIU,Jiao-jiao FENG,Jia-ping LI,Yuan LIN,Bo XU. Academic paper recommendation based on heterogeneous network embedding [J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2020, 55(11): 35-45.
[3] ZHANG Ling, REN Xue-fang. Intelligent data classification and intelligent retrieval-recognition of class [J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2020, 55(10): 7-14.
[4] Ming-xing LIN. Low-light image enhancement algorithm based on variational structure [J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2020, 55(9): 72-80.
[5] Jia-qi WANG,Mu-yun YANG,Tie-jun ZHAO,Zhen-yu ZHAO. Construction of retrieval dataset of procuratorate legal documents [J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2020, 55(7): 81-87.
[6] Ying YU,Xin-nian WU,Le-wei WANG,Ying-long ZHANG. A multi-label three-way classification algorithm based on label correlation [J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2020, 55(3): 81-88.
[7] Liu-ying WEN,Wei YUAN. Clustering method for multi-label symbolic value partition [J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2020, 55(3): 58-69.
[8] Min-qing ZHANG,Neng ZHOU,Meng-meng LIU,Han WANG,Yan KE. Reversible data hiding in homomorphic encrypted domain based on Paillier [J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2020, 55(3): 1-8,18.
[9] Xin-le WANG,Wen-feng YANG,Hua-ming LIAO,Yong-qing WANG,Yue LIU,Xiao-ming YU,Xue-qi CHENG. Topic tag popularity prediction based on multi-dimensional features [J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2020, 55(1): 94-101.
[10] Ni LI,Huan-mei GUAN,Piao YANG,Wen-yong DONG. BERT-IDCNN-CRF for named entity recognition in Chinese [J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2020, 55(1): 102-109.
[11] ZHANG Di, ZHA Dong-dong, LIU Hua-yong. Construction of the cubic λμ-α-DP curve with two kinds of shape parameters [J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2019, 54(9): 114-126.
[12] YANG Ya-ru, WANG Yong-qing, ZHANG Zhi-bin, LIU Yue, CHENG Xue-qi. Social network user identity linkage model based on comprehensive information [J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2019, 54(9): 105-113.
[13] Chang-ying HAO,Yan-yan LAN,Hai-nan ZHANG,Jia-feng GUO,Jun XU,Liang PANG,Xue-qi CHENG. Dialogue generation model based on extended keywords information [J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2019, 54(7): 68-76.
[14] Xiang-wen LIAO,Yang XU,Jing-jing WEI,Ding-da YANG,Guo-long CHEN. Review spam detection based on the two-level stacking classification model [J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2019, 54(7): 57-67.
[15] Yang XU,Jian-zhong SUN,Lei HUANG,Xiao-yao XIE. Trajectory model of area crowd based on WiFi positioning [J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2019, 54(5): 8-20.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
No Suggested Reading articles found!
Copyright 2018 © Editorial office of JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE)
Supported by Beijing Magtech Co.Ltd