JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE) ›› 2022, Vol. 57 ›› Issue (4): 21-29.doi: 10.6040/j.issn.1671-9352.7.2021.083

Previous Articles    

Multi-label classification for medical text based on ALBERT-TextCNN model

ZHENG Cheng-yu, WANG Xin*, WANG Ting, DENG Ya-ping, YIN Tian-tian   

  1. School of Mathematics and Computer Science, Yunnan Minzu University, Kunming 650500, Yunnan, China
  • Published:2022-03-29

PDF (PC)

893

Abstract: Aiming at the problem of existing static word vector representation methods such as Word2Vec and Glove cannot solve the problem of complete text semantics, combined with the ALBERT pre-trained language model and the TextCNN convolutional neural network, a deep neural network model for multi-label medical text classification named ALBERT-TextCNN is proposed. The model use the ALBERT pre-training language model for dynamic word vector representation to obtain a more efficient text vector representation through its internal multi-layer bidirectional Transfomer structure, and introduce the TextCNN convolutional neural network model to construct a multi-label classifier for training to extract semantic information features at different levels of abstraction. The performance of the algorithm is tested on the Chinese health question data set. The experimental results show that the overall F1 value of the model reaches 90.5%, which can effectively improve the multi-label classification effect of the medical text.

Key words: ALBERT, TextCNN model, multi-label classification, medical text

CLC Number: 

  • TP391.1
[1] 廖开际, 邹珂欣, 席运江. 一种在线医疗社区问答文本实体识别方法:基于卷积神经网络和双向长短期记忆神经网络[J]. 科技管理研究, 2021, 41(8):173-179. LIAO Kaiji, ZOU Kexin, XI Yunjiang. An online medical community q&a text entity recognition method: based on CNN and BILSTM[J]. Science and Technology Management Research, 2021, 41(8):173-179.
[2] 熊回香, 杨梦婷, 李玉媛. 基于深度学习的信息组织与检索研究综述[J]. 情报科学,2020,38(3):3-10. XIONG Huixiang, YANG Mengting, LI Yuyuan. Information organization and retrieval research based on deep learning[J]. Information Science, 2020, 38(3):3-10.
[3] 程艳, 孙欢, 陈豪迈, 等. 融合卷积神经网络与双向GRU的文本情感分析胶囊模型[J]. 中文信息学报,2021,35(5):118-129. CHENG Yan, SUN Huan, CHEN Haomai, et al. Text sentiment analysis capsule model combining convolutional neural network and bidirectional GRU[J]. Journal of Chinese Information Processing, 2021, 35(5):118-129.
[4] 张鹏, 孙博文, 李唯实, 等. 基于LSTM的钓鱼邮件检测系统[J]. 北京理工大学学报,2020,40(12):1289-1294. ZHANG Peng, SUN Bowen, LI Weishi, et al. Phishing mail detection system based on LSTM neural network[J]. Transactions of Beijing Institute of Technology, 2020, 40(12):1289-1294.
[5] 林敏鸿, 蒙祖强. 基于注意力神经网络的多模态情感分析[J]. 计算机科学,2020,47(增刊2):508-514,548. LIN Minhong, MENG Zuqiang. Multimodal sentiment analysis based on attention neural network[J]. Computer Science, 2020, 47(Suppl.2):508-514, 548.
[6] CHEN Guibinm, YE Deheng, XING Zhenchang, et al. Ensemble application of convolutional and recurrent neural networks for multi-label text categorization[C] //2017 International Joint Conference on Neural Networks(IJCNN). Anchorage, USA:IEEE, 2017:2377-2383.
[7] ALY R, REMUS S, BIEMANN C. Hierarchical multi-label classification of text with capsule networks[C] //Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics:Student Research Workshop. Florence, Italy:ACL, 2019:323-330.
[8] DU Jingcheng, CHEN Qingyu, PENG Yifan, et al. Ml-net:multi-label classification of biomedical texts with deep neural networks[J]. Journal of the American Medical Informatics Association, 2019, 26(11):1279-1285.
[9] 赵亚欧, 张家重, 李贻斌, 等. 基于ELMo和Transformer混合模型的情感分析[J]. 中文信息学报,2021,35(3):115-124. ZHAO Yaou, ZHANG Jiazhong, LI Yibin, et al. Sentiment analysis based on hybrid model of ELMo and Transformer[J]. Journal of Chinese Information Processing, 2021, 35(3):115-124.
[10] 段丹丹, 唐加山, 温勇, 等. 基于BERT模型的中文短文本分类算法[J]. 计算机工程,2021,47(1):79-86. DUAN Dandan, TANG Jiashan, WEN Yong, et al. Chinese short text classification algorithm based on BERT model[J]. Computer Engineering, 2021, 47(1):79-86.
[11] LAN Z Z, CHEN M D, GOODMAN S, et al. Albert:a lite BERT for self-supervised learning of language representations [EB/OL].(2020-02-09)[2021-05-10]. https://arxiv.org/pdf/1909.11942.pdf.
[12] DEVLIN J, CHANG M W, LEE K, et al. Bert:pre-training of deep bidirectional transformers for language understanding[EB/OL].(2019-05-24)[2021-05-10]. https://arxiv.org/pdf/1810.04805.pdf.
[13] 温超东, 曾诚, 任俊伟, 等. 结合ALBERT和双向门控循环单元的专利文本分类[J]. 计算机应用,2021,41(2):407-412. WEN Chaodong, ZENG Cheng, REN Junwei, et al. Patent text classification based on ALBERT and bidirectional gated recurrent unit[J]. Journal of Computer Applications, 2021, 41(2):407-412.
[14] SUTSKEVER I, VINYALS O, LE Q V. Sequence to sequence learning with neural networks[C] //Advances in Neural Information Processing Systems. Montreal, Canada: NIPS, 2014: 3104-3112.
[15] BAHDANAU D, CHO K, BENGIO Y. Neural machine translation by jointly learning to align and translate[EB/OL].(2016-05-16)[2021-05-10]. https://arxiv.org/pdf/1409.0473.pdf.
[16] MIKOLOV T, CHEN K, CORRADO G, et al. Efficient estimation of word representations in vector space[EB/OL].(2013-09-07)[2021-05-10]. https://arxiv.org/pdf/1301.3781.pdf.
[17] KIM Y. Convolutional neural networks for sentence classification[EB/OL].(2014-09-03)[2021-05-10]. https://arxiv.org/pdf/1408.5882.pdf.
[18] 张志昌, 张治满, 张珍文. 融合局部语义和全局结构信息的健康问句分类[J]. 西安电子科技大学学报,2020,47(2):9-15. ZHANG Zhichang, ZHANG Zhiman, ZHANG Zhenwen. Classifying health questions with local semantic and global structural information[J]. Journal of Xidian University, 2020, 47(2):9-15.
[1] Kan XU,Rui-xin LIU,Hong-fei LIN,Hai-feng LIU,Jiao-jiao FENG,Jia-ping LI,Yuan LIN,Bo XU. Academic paper recommendation based on heterogeneous network embedding [J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2020, 55(11): 35-45.
[2] Jia-qi WANG,Mu-yun YANG,Tie-jun ZHAO,Zhen-yu ZHAO. Construction of retrieval dataset of procuratorate legal documents [J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2020, 55(7): 81-87.
[3] PANG Bo, LIU Yuan-chao. Fusion of pointwise and deep learning methods for passage ranking [J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2018, 53(3): 30-35.
[4] YANG Yan, XU Bing, YANG Mu-yun, ZHAO Jing-jing. An emotional classification method based on joint deep learning model [J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2017, 52(9): 19-25.
[5] HUANG Dong, XU Bo, XU Kan, LIN Hong-fei, YANG Zhi-hao. Short text clustering based on word embeddings and EMD [J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2017, 52(7): 66-72.
[6] DU Man, XU Xue-ke, DU Hui, WU Da-yong, LIU Yue, CHENG Xue-qi. Emotion-specific word embedding learning for emotion classification [J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2017, 52(7): 52-58.
[7] CAO Rong, HUANG Jin-zhu, YI Mian-zhu. Information retrieval: the final direction of human language technology research in DARPA [J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2016, 51(9): 11-17.
[8] FENG Guo-he, WANG Dan-di, LI Mei-chan. Text topic mining of archives research based on SVD [J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2016, 51(1): 95-100.
[9] TAN Jin-yuan, DIAO Yu-feng, YANG Liang, QI Rui-hua, LIN Hong-fei. Extractive-abstractive text automatic summary based on BERT-SUMOPN model [J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2021, 56(7): 82-90.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
No Suggested Reading articles found!
Copyright 2018 © Editorial office of JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE)
Supported by Beijing Magtech Co.Ltd