JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE) ›› 2023, Vol. 58 ›› Issue (1): 48-58.doi: 10.6040/j.issn.1671-9352.2.2021.035

Previous Articles    

Question generation method based on conditional variational autoencoder

LIU Dong, HONG Yu*, SU Yu-lan, ZHANG Min   

  1. School of Computer Science and Technology, Soochow University, Suzhou 215006, Jiangsu, China
  • Published:2023-02-12

PDF (PC)

100

Abstract: The conditional variational autoencoder, as an auxiliary module, is introduced into the encoding and decoding process of the pre-trained language model. It improves the robustness of the model through data augmentation(potential semantic expansion), and establishes a high-dimensional distribution connection between declarative sentences and interrogative sentences, which implements one-to-many question generation by sampling from the distribution. The results show that the fusion of conditional variational autoencoder can not only generate diverse questions, but also help to improve the performance of the question generation model. On the two answer-known question generation datasets Split1 and Split2, which are based on the SQuAD dataset, the BLEU-4 score is improved to 20.75% and 21.61%, respectively.

Key words: conditional variational autoencoder, question generation, pre-trained language model

CLC Number: 

  • TP391
[1] PAN L, LEI W, CHUA T S, et al. Recent advances in neural question generation[J/OL]. arXiv, 2019. https://arxiv.org/abs/1905.08949v3.
[2] QIU X, SUN T, XU Y, et al. Pre-trained models for natural language processing: a survey[J]. Science China Technological Sciences, 2020, 63(10):1-26.
[3] BAO H, DONG L, WEI F, et al. Unilmv2: pseudo-masked language models for unified language model pre-training[C] //International Conference on Machine Learning. New York: PMLR, 2020: 642-652.
[4] WANG T, WAN X. T-CVAE: transformer-based conditioned variational autoencoder for story completion[C] //International Joint Conferences on Artificial Intelligence.[S.l.] : IJCAI, 2019: 5233-5239.
[5] SERBAN I, SORDONI A, LOWE R, et al. A hierarchical latent variable encoder-decoder model for generating dialogues[C] //Proceedings of the AAAI Conference on Artificial Intelligence. Menlo Park: AAAI, 2017.
[6] RAJPURKAR P, ZHANG J, LOPYREVK, et al. Squad: 100,000+ questions for machine comprehension of text[J/OL]. arXiv, 2016. https://arxiv.org/abs/1606.05250v2.
[7] DU X, SHAO J, CARDIEC. Learning to ask: neural question generation for reading comprehension[J/OL]. arXiv, 2017. https://arxiv.org/abs/1705.00106v1.
[8] ZHOU Q, YANG N, WEI F, et al. Neural question generation from text: a preliminary study[C] //National CCF Conference on Natural Language Processing and Chinese Computing. Cham: Springer, 2017: 662-671.
[9] BOWMAN S R, VILNIS L, VINYALS O, et al. Generating sentences from a continuous space[J/OL]. arXiv, 2015. https://arxiv.org/abs/1511.06349v3.
[10] SUN X, LIU J, LYU Y, et al. Answer-focused and position-aware neural question generation[C] //Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Stroudsburg:Association for Computational Linguistics, 2018: 3930-3939.
[11] GU J, LU Z, LI H, et al. Incorporating copying mechanism in sequence-to-sequence learning[J/OL]. arXiv, 2016. https://arxiv.org/abs/1603.06393v3.
[12] VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C] //Advances in Neural Information Processing Systems. [S.l.] :Neural information processing systems foundation, 2017: 5998-6008.
[13] CHAN Y H, FAN Y C. A recurrent BERT-based model for question generation[C] //Proceedings of the 2nd Workshop on Machine Reading for Question Answering. Stroudsburg:Association for Computational Linguistics, 2019: 154-162.
[14] VARANASI S, AMIN S, NEUMANN G.CopyBERT: a unified approach to question generation with self-attention[C] //Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI. Stroudsburg:Association for Computational Linguistics, 2020: 25-31.
[15] DONG L, YANG N, WANG W, et al. Unified language model pre-training for natural language understanding and generation[J/OL]. arXiv, 2019. https://arxiv.org/abs/1905.03197.
[16] 黄民烈. 现代自然语言生成[M]. 北京:电子工业出版社,2021: 98-100. HUANG Minlie. Modern natural language generation[M]. Beijing:Publishing House of Electronics Industry of China, 2021: 98-100.
[17] SHARMA S, ASRI L E, SCHULZH, et al. Relevance of unsupervised metrics in task-oriented dialogue for evaluating natural language generation[J/OL]. arXiv, 2017. https://arxiv.org/abs/1706.09799.
[18] PAPINENI K, ROUKOS S, WARD T, et al. Bleu: a method for automatic evaluation of machine translation[C] //Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics. Stroudsburg: Association for Computational Linguistics, 2002: 311-318.
[19] DENKOWSKI M, LAVIEA. Meteor universal: language specific translation evaluation for any target language[C] //Proceedings of the Ninth Workshop on Statistical Machine Translation. Stroudsburg:Association for Computational Linguistics, 2014: 376-380.
[20] LIN C Y. Rouge: a package for automatic evaluation of summaries[C] //Proceedings of Workshop on Text Summarization of ACL. Stroudsburg:Association for Computational Linguistics, 2004.
[21] GU X D, CHO K H, HA J W, et al. Dialogwae: multimodal response generation with conditional wasserstein auto-encoder[J/OL]. arXiv, 2018. https://arxiv.org/pdf/1805.12352.pdf.
[22] ZHAO Y, NI X, DING Y, et al. Paragraph-level neural question generation with maxout pointer and gated self-attention networks[C] //Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Stroudsburg: Association for Computational Linguistics, 2018: 3901-3910.
[1] TAN Jin-yuan, DIAO Yu-feng, YANG Liang, QI Rui-hua, LIN Hong-fei. Extractive-abstractive text automatic summary based on BERT-SUMOPN model [J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2021, 56(7): 82-90.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
No Suggested Reading articles found!
Copyright 2018 © Editorial office of JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE)
Supported by Beijing Magtech Co.Ltd