JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE) ›› 2023, Vol. 58 ›› Issue (5): 36-45.doi: 10.6040/j.issn.1671-9352.0.2021.790

Previous Articles    

Text classification model based on dual-channel feature fusion based on XLNet

MENG Jinxu1, SHAN Hongtao1*, HUANG Runcai1, YAN Fengting3, LI Zhiwei1, ZHENG Guangyuan2, LIU Yiming1, SHI Changtong1   

  1. 1. School of Electronic and Electrical Engineering, Shanghai University of Engineering Science, Shanghai 201620, China;
    2. College of Information Technologhy, Shanghai Jianqiao University, Shanghai 201306, China;
    3. Shanghai Qingxi Intelligent Technology Co., Ltd., Shanghai 201800, China
  • Published:2023-05-15

PDF (PC)

277

Abstract: A two-channel feature-fusion text classification(XLNet-CNN-BiGRU, XLCBG)model based on XLNet is proposed. Compared with the single model channel, the XLCBG model can extract richer semantic features by integrating the feature information of XLNet+CNN and XLNet+BiGRU channels to diversify the methods of feature-fusion processing and increase the selectivity of the optimal classification model. The XLCBG model adopts Maxpooling, Avgpooling, and attention mechanism to extract the vector with the largest feature value in the global, the feature vector with the mean value in the global, and the key features of attention mechanism to replace the whole vector, respectively. Finally, the current popular text classification models are compared with the XLCBG model. The experimental results show that the XLCBG-S model exhibits better classification performance than other models on the Chinese THUCNews dataset. At the same time, the XLCBG-Ap model exhibits better classification performance than other models on the English AG News data set. In the English 20NewsGroups data set, the XLCBG-Att model is superior to other models in accuracy and recall rate, and the XLCBG-Mp model is superior to other models in accuracy rate and F1.

Key words: XLNet, dual-channel, text classification, BiGRU, CNN

CLC Number: 

  • TP391.1
[1] 汪岿,刘柏嵩.文本分类研究综述[J].数据通信,2019(3):37-47. WANG Kui, LIU Baisong. A summary of text classification research[J]. Data Communications, 2019(3):37-47.
[2] MANEK A S, SHENOY P D, MOHAN M C, et al. Aspect term extraction for sentiment analysis in large movie reviews using Gini Index feature selection method and SVM classifier[J]. World Wide Web, 2017, 20(2):135-154.
[3] TANHA J, VAN SOMEREN M, AFSARMANESH H. Semi-supervised self-training for decision tree classifiers[J]. International Journal of Machine Learning and Cybernetics, 2017, 8(1):355-370.
[4] TANG B, KAY S, HE H. Toward optimal feature selection in naive Bayes for text categorization[J]. IEEE Transactions on Knowledge and Data Engineering, 2016, 28(9):2508-2521.
[5] KIM Y. Convolutional neural networks for sentence classification[C] //Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing(EMNLP). Doha: Association for Computational Linguistics, 2014: 1746-1751.
[6] LIU P F, QIU X P, HUANG X J. Recurrent neural network for text classification with multi-task learning[C] //Proceedings of the Twenty-fifth International Joint Conference on Artificial Intelligence(IJCAI-16). New York: AAAI Press, 2016.
[7] 陈虹,杨燕,杜圣东. 用户评论方面级情感分析研究[J]. 计算机科学与探索, 2021, 15(3):478-485. CHEN Hong, YANG Yan, DU Shengdong. Research on aspect-level sentiment analysis of user reviews[J]. Journal of Frontiers of Computer Science & Technology, 2021, 15(3):478-485.
[8] 陶亮,刘宝宁,梁玮.基于CNN-LSTM 混合模型的心律失常自动检测[J].山东大学学报(工学版),2021,51(3):30-36. TAO Liang, LIU Baoning, LIANG Wei. Automatic detection research of arrhythmia based on CNN-LSTM hybrid model[J]. Journal of Shandong University(Engineering Science), 2021, 51(3):30-36.
[9] 吴汉瑜, 严江, 黄少滨,等.用于文本分类的CNN_BiLSTM_Attention混合模型[J].计算机科学,2020,47(S2):23-27. WU Hanyu, YAN Jiang, HUANG Shaobin, et al. CNN_BiLSTM_Attention hybrid model for text classification[J]. Computer Science, 2020, 47(S2):23-27.
[10] MIKOLOV T, CHEN K, CORRADO G, et al. Efficient estimation of word representations in vector space[J/OL]. arXiv, 2013. https://arxiv.org/pdf/1301.3781.pdf.
[11] DEVLIN J, CHANG M W, LEE K, et al. Bert: pre-training of deep bidirectional transformers for language understanding[J/OL]. arXiv, 2018. https://arxiv.org/abs/1810.04805.
[12] 陈德光,马金林,马自萍,等.自然语言处理预训练技术综述[J].计算机科学与探索,2015,15(8):1359-1388. CHEN Deguang, MA Jinlin, MA Ziping, et al. Review of pre-training techniques for natural language processing[J]. Journal of Frontiers of Computer Science and Technology, 2015, 15(8):1359-1388.
[13] 董彦如,刘培玉,刘文锋,等.基于双向长短期记忆网络和标签嵌入的文本分类模型[J].山东大学学报(理学版),2020,55(11):78-86. DONG Yanru, LIU Peiyu, LIU Wenfeng, et al. A text classification model based on BiLSTM and label embedding[J]. Journal of Shandong University(Natural Science), 2020, 55(11):78-86.
[14] YANG Z, DAI Z, YANG Y, et al. XLNet: generalized autoregressive pretraining for language understanding[J/OL]. arXiv, 2019. https://arxiv.org/abs/1906.08237.
[15] LAI S, XU L, LIU K, et al. Recurrent convolutional neural networks for text classification[C] //Twenty-ninth AAAI Conference on Artificial Intelligence. Austin: AAAI Press, 2015: 2267-2273.
[16] JOHNSON R, ZHANG T. Deep pyramid convolutional neural networks for text categorization[C] //Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics. Vancouve: Association for Computational Linguistics, 2017: 562-570.
[17] 郑诚,陈杰,董春阳.结合图卷积的深层神经网络用于文本分类[J].计算机工程与应用,2022,58(7):206-212. ZHENG Cheng, CHEN Jie, DONG Chunyang. Deep neural network combined with graph convolution for text classification[J].Computer Engineering and Applications, 2022, 58(7):206-212.
[18] 闫跃,霍其润,李天昊,等.融合多重注意力机制的卷积神经网络文本分类设计与实现[J].小型微型计算机系统,2021,42(2):362-367. YAN Yue, HUO Qirun, LI Tianhao, et al. Design and implementation of text classification based on convolutional neural network with multiple attention mechanisms[J]. Journal of Chinese Computer Systems, 2021, 42(2):362-367.
[19] 李启行,廖薇,孟静雯.基于注意力机制的双通道DAC-RNN文本分类模型[J].计算机工程与应用,2022,58(16):157-163. LI Qihang, LIAO Wei, MENG Jingwen. Dual-channel DAC-RNN text categorization model based on attention mechanism[J]. Computer Engineering and Applications, 2022, 58(16):157-163.
[1] XU Ming-xing, LI Xue-qin. Unified pricing and service effort strategy in a dual-channel supply chain with bidirectional free-riding [J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2022, 57(9): 55-70.
[2] ZHENG Cheng-yu, WANG Xin, WANG Ting, DENG Ya-ping, YIN Tian-tian. Multi-label classification for medical text based on ALBERT-TextCNN model [J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2022, 57(4): 21-29.
[3] ZHANG Bin-yan, ZHU Xiao-fei, XIAO Zhao-hui, HUANG Xian-ying, WU Jie. Short text classification based on semi-supervised graph neural network [J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2021, 56(5): 57-65.
[4] Yan JIAO,Chuan-xu WANG,Yuan-wei WANG,Lang XU. Pricing decision of dual-channel supply chain considering altruistic preferences under reference price effect [J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2021, 56(1): 103-110.
[5] DONG Yan-ru, LIU Pei-yu, LIU Wen-feng, ZHAO Hong-yan. A text classification model based on BiLSTM and label embedding [J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2020, 55(11): 78-86.
[6] Feng-shan SI,Jing WANG,Dao-ming DAI. Evolutionary game analysis of dual-channel supply chain considering service quality and delay decision [J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2020, 55(1): 86-93, 101.
[7] Ni LI,Huan-mei GUAN,Piao YANG,Wen-yong DONG. BERT-IDCNN-CRF for named entity recognition in Chinese [J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2020, 55(1): 102-109.
[8] LIU Yang, ZHAO Ke-jun, GE Lian-sheng, LIU Heng. A fast DGA domain detection algorithm based on deep learning [J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2019, 54(7): 106-112.
[9] Xiao-jie XIE,Ying LIANG,Xiang-xiang DONG. Sensitive attribute iterative inference method for social network users [J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2019, 54(3): 10-17, 27.
[10] . Reader emotion classification with news and comments [J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2018, 53(9): 35-39.
[11] CHEN Liang, HU Jin-song. Influence of risk aversion on different types of dual-channel supply chain decision [J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2018, 53(5): 30-40.
[12] YANG Yan, XU Bing, YANG Mu-yun, ZHAO Jing-jing. An emotional classification method based on joint deep learning model [J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2017, 52(9): 19-25.
[13] SUN Jian-dong, GU Xiu-sen, LI Yan, XU Wei-ran. Chinese entity relation extraction algorithms based on COAE2016 datasets [J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2017, 52(9): 7-12.
[14] WAN Zhong-ying, WANG Ming-wen, ZUO Jia-li, WAN Jian-yi. Feature selection combined with the global and local information(GLFS) [J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2016, 51(5): 87-93.
[15] MA Cheng-long, JIANG Ya-song, LI Yan-ling, ZHANG Yan, YAN Yong-hong. Short text classification based on word embedding similarity [J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2014, 49(12): 18-22.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
No Suggested Reading articles found!
Copyright 2018 © Editorial office of JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE)
Supported by Beijing Magtech Co.Ltd