基于ALBERT-TextCNN模型的多标签医疗文本分类方法

doi:10.6040/j.issn.1671-9352.7.2021.083

摘要/Abstract

摘要： 针对现有Word2Vec和Glove等静态词向量表征方法无法解决文本完整语义的问题,结合ALBERT预训练语言模型和TextCNN卷积神经网络,提出一种用于多标签医疗文本分类的深层神经网络模型ALBERT-TextCNN。该模型采用ALBERT预训练语言模型进行动态字向量表示,通过其内部多层双向的Transfomer结构获取更高效的文本向量表达,并引入TextCNN卷积神经网络模型构造多标签分类器进行训练,提取不同抽象层次的语义信息特征。在中文健康问句数据集上进行算法性能测试,实验结果表明,该模型分类的整体F1值达到了90.5%,能有效提升医疗文本的多标签分类效果。

关键词: ALBERT, TextCNN模型, 多标签分类, 医疗文本

Abstract: Aiming at the problem of existing static word vector representation methods such as Word2Vec and Glove cannot solve the problem of complete text semantics, combined with the ALBERT pre-trained language model and the TextCNN convolutional neural network, a deep neural network model for multi-label medical text classification named ALBERT-TextCNN is proposed. The model use the ALBERT pre-training language model for dynamic word vector representation to obtain a more efficient text vector representation through its internal multi-layer bidirectional Transfomer structure, and introduce the TextCNN convolutional neural network model to construct a multi-label classifier for training to extract semantic information features at different levels of abstraction. The performance of the algorithm is tested on the Chinese health question data set. The experimental results show that the overall F1 value of the model reaches 90.5%, which can effectively improve the multi-label classification effect of the medical text.

Key words: ALBERT, TextCNN model, multi-label classification, medical text

中图分类号:

TP391.1

郑承宇,王新,王婷,邓亚萍,尹甜甜. 基于ALBERT-TextCNN模型的多标签医疗文本分类方法[J]. 《山东大学学报(理学版)》, 2022, 57(4): 21-29.

ZHENG Cheng-yu, WANG Xin, WANG Ting, DENG Ya-ping, YIN Tian-tian. Multi-label classification for medical text based on ALBERT-TextCNN model[J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2022, 57(4): 21-29.

参考文献

[1] 廖开际, 邹珂欣, 席运江. 一种在线医疗社区问答文本实体识别方法:基于卷积神经网络和双向长短期记忆神经网络[J]. 科技管理研究, 2021, 41(8):173-179. LIAO Kaiji, ZOU Kexin, XI Yunjiang. An online medical community q&a text entity recognition method: based on CNN and BILSTM[J]. Science and Technology Management Research, 2021, 41(8):173-179.
[2] 熊回香, 杨梦婷, 李玉媛. 基于深度学习的信息组织与检索研究综述[J]. 情报科学,2020,38(3):3-10. XIONG Huixiang, YANG Mengting, LI Yuyuan. Information organization and retrieval research based on deep learning[J]. Information Science, 2020, 38(3):3-10.
[3] 程艳, 孙欢, 陈豪迈, 等. 融合卷积神经网络与双向GRU的文本情感分析胶囊模型[J]. 中文信息学报,2021,35(5):118-129. CHENG Yan, SUN Huan, CHEN Haomai, et al. Text sentiment analysis capsule model combining convolutional neural network and bidirectional GRU[J]. Journal of Chinese Information Processing, 2021, 35(5):118-129.
[4] 张鹏, 孙博文, 李唯实, 等. 基于LSTM的钓鱼邮件检测系统[J]. 北京理工大学学报,2020,40(12):1289-1294. ZHANG Peng, SUN Bowen, LI Weishi, et al. Phishing mail detection system based on LSTM neural network[J]. Transactions of Beijing Institute of Technology, 2020, 40(12):1289-1294.
[5] 林敏鸿, 蒙祖强. 基于注意力神经网络的多模态情感分析[J]. 计算机科学,2020,47(增刊2):508-514,548. LIN Minhong, MENG Zuqiang. Multimodal sentiment analysis based on attention neural network[J]. Computer Science, 2020, 47(Suppl.2):508-514, 548.
[6] CHEN Guibinm, YE Deheng, XING Zhenchang, et al. Ensemble application of convolutional and recurrent neural networks for multi-label text categorization[C] //2017 International Joint Conference on Neural Networks(IJCNN). Anchorage, USA:IEEE, 2017:2377-2383.
[7] ALY R, REMUS S, BIEMANN C. Hierarchical multi-label classification of text with capsule networks[C] //Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics:Student Research Workshop. Florence, Italy:ACL, 2019:323-330.
[8] DU Jingcheng, CHEN Qingyu, PENG Yifan, et al. Ml-net:multi-label classification of biomedical texts with deep neural networks[J]. Journal of the American Medical Informatics Association, 2019, 26(11):1279-1285.
[9] 赵亚欧, 张家重, 李贻斌, 等. 基于ELMo和Transformer混合模型的情感分析[J]. 中文信息学报,2021,35(3):115-124. ZHAO Yaou, ZHANG Jiazhong, LI Yibin, et al. Sentiment analysis based on hybrid model of ELMo and Transformer[J]. Journal of Chinese Information Processing, 2021, 35(3):115-124.
[10] 段丹丹, 唐加山, 温勇, 等. 基于BERT模型的中文短文本分类算法[J]. 计算机工程,2021,47(1):79-86. DUAN Dandan, TANG Jiashan, WEN Yong, et al. Chinese short text classification algorithm based on BERT model[J]. Computer Engineering, 2021, 47(1):79-86.
[11] LAN Z Z, CHEN M D, GOODMAN S, et al. Albert:a lite BERT for self-supervised learning of language representations [EB/OL].(2020-02-09)[2021-05-10]. https://arxiv.org/pdf/1909.11942.pdf.
[12] DEVLIN J, CHANG M W, LEE K, et al. Bert:pre-training of deep bidirectional transformers for language understanding[EB/OL].(2019-05-24)[2021-05-10]. https://arxiv.org/pdf/1810.04805.pdf.
[13] 温超东, 曾诚, 任俊伟, 等. 结合ALBERT和双向门控循环单元的专利文本分类[J]. 计算机应用,2021,41(2):407-412. WEN Chaodong, ZENG Cheng, REN Junwei, et al. Patent text classification based on ALBERT and bidirectional gated recurrent unit[J]. Journal of Computer Applications, 2021, 41(2):407-412.
[14] SUTSKEVER I, VINYALS O, LE Q V. Sequence to sequence learning with neural networks[C] //Advances in Neural Information Processing Systems. Montreal, Canada: NIPS, 2014: 3104-3112.
[15] BAHDANAU D, CHO K, BENGIO Y. Neural machine translation by jointly learning to align and translate[EB/OL].(2016-05-16)[2021-05-10]. https://arxiv.org/pdf/1409.0473.pdf.
[16] MIKOLOV T, CHEN K, CORRADO G, et al. Efficient estimation of word representations in vector space[EB/OL].(2013-09-07)[2021-05-10]. https://arxiv.org/pdf/1301.3781.pdf.
[17] KIM Y. Convolutional neural networks for sentence classification[EB/OL].(2014-09-03)[2021-05-10]. https://arxiv.org/pdf/1408.5882.pdf.
[18] 张志昌, 张治满, 张珍文. 融合局部语义和全局结构信息的健康问句分类[J]. 西安电子科技大学学报,2020,47(2):9-15. ZHANG Zhichang, ZHANG Zhiman, ZHANG Zhenwen. Classifying health questions with local semantic and global structural information[J]. Journal of Xidian University, 2020, 47(2):9-15.

多维度评价

Viewed

Full text

Abstract

Cited

Shared

Discussed