JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE) ›› 2019, Vol. 54 ›› Issue (7): 68-76.doi: 10.6040/j.issn.1671-9352.1.2018.120

Previous Articles     Next Articles

Dialogue generation model based on extended keywords information

Chang-ying HAO1,2(),Yan-yan LAN1,2,*(),Hai-nan ZHANG1,2,Jia-feng GUO1,2,Jun XU1,2,Liang PANG1,2,Xue-qi CHENG1,2   

  1. 1. CAS Key Laboratory of Network Data Science and Technology, Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190, China
    2. University of Chinese Academy of Sciences, Beijing 100190, China
  • Received:2018-10-17 Online:2019-07-20 Published:2019-06-27
  • Contact: Yan-yan LAN E-mail:haochangying18s@ict.ac.cn;lanyanyan@ict.ac.cn
  • Supported by:
    西藏自治区科技计划项目(XZ201801-GB-17);国家重点研发计划资助项目(2016QY02D0405);国家自然科学基金资助项目(61425016);国家自然科学基金资助项目(61472401);国家自然科学基金资助项目(61722211);国家自然科学基金资助项目(61773362);国家自然科学基金资助项目(20180290);中国科学院青年创新促进会项目(20144310);中国科学院青年创新促进会项目(2016102)

RichHTML

16

PDF (PC)

473

Abstract:

Noting that people often reply to others based on the keywords in the post. In order to generate meaningful responses about the keywords, we proposed a dialogue generation model based on an attention mechanism with extended keywords information. First, we extract the keywords of the input, and then form the extended keywords set based on the words which are related to the input keywords using the cosine similarity. The words in the extended keywords set affect the response generation with the attention mechanism. Experiments on the Chinese Weibo dataset and the English Twitter dataset show that our model outperforms the traditional seq2seq model and its variations in both the metric-based evaluation and the human-evaluation.

Key words: dialogue generation, attention mechanism, Seq2Seq, deep learning

CLC Number: 

  • TP391

Fig.1

The S2S-K model"

Fig.2

The S2S-EKA model"

Fig.3

The text information alignment model (left) and the extended keywords information alignment model (right)"

Fig.4

Impact of different keyword numbers on quality of responses generated by S2SA-K Model"

Table 1

Metric-based evaluation result on Weibo text dataset"

模型PPLBLEUDistinct-1Distinct-2
S2S33.120.019 50.005 00.022 8
S2SA30.570.020 20.006 60.029 5
S2SA-MMI27.110.021 30.010 30.030 3
S2SA-K17.990.020 30.006 00.033 5
S2SA-EKA13.870.021 40.011 00.042 1

Table 2

Metric-based evaluation result on Twitter test dataset"

模型PPLBLEUDistinct-1Distinct-2
S2S4.390.049 80.002 00.010 4
S2SA4.260.055 30.002 60.014 9
S2SA-MMI4.220.058 40.004 10.029 5
S2SA-K3.740.057 20.002 90.026 7
S2SA-EKA3.560.059 30.004 50.032 4

Table 3

Human evaluation result on Weibo test dataset"

模型+2+10平均分Kappa
S2S30.7%34.5%34.8%0.9590.810 6
S2SA31.5%37.2%31.3%1.0020.664 2
S2SA-MMI34.2%34.7%31.1%1.0310.786 5
S2SA-K37.5%28.4%34.1%1.0340.831 5
S2SA-EKA45.2%24.6%30.2%1.1500.849 3

Table 4

Human evaluation result on Twitter test dataset"

模型+2+10平均分Kappa
S2S42.8%41.9%15.3%1.2750.654 2
S2SA46.2%39.7%14..1%1.3210.732 1
S2SA-MMI50.7%37.2%12.1%1.3860.625 9
S2SA-K52.5%34.2%13.3%1.3920.678 1
S2SA-EKA54.7%35.8%9.5%1.4520.649 9

Table 5

Cases of responses generated by different models on Weibo test dataset"

输入语句和原始回复模型生成的回复语句关键词及拓展关键词集合
输入:据说是二年级的数学题你会吗
回复:出题的就是外星人		S2S:我想说我是外星人
S2SA:我也会有这么个情况
S2SA-MMI:出题的老师是外星人
S2SA-K:我是小学二年级二年级的学生
S2SA-EKA:这题目是在黑我的智商吗		数学题:二年级 一年级 小学 智商  小学 压轴 题目 分钟…
二年级:数学题 会一年级 极限运动 耐克鞋 小学 班级 …
输入:刚录完节目,已到我生日,祝自己38岁快乐。
回复:生日快乐!永远爱你。		S2S:生日快乐,祝啊。
S2SA:生日快乐,生日快乐!
S2SA-MMI:生日快乐,永远支持你。
S2SA-K:生日快乐,快乐的每一天!!
S2SA-EKA:一切顺利吧,祝你生日快乐,身体健康		录:节目 点将 歌曲 间隙 盛典
王菲 录播 录像 顺利…
生日: 生日快乐 庆祝 岁 身体健康
祝 老妈 祝福 蛋糕 阴历…

Table 6

Cases of responses generated by different models on Twitter test dataset"

输入语句和原始回复模型生成的回复语句关键词及拓展关键词集合
输入: I have a problem with my iPhone
回复: We′re happy to help. What seems to be the problem		S2S: Hey there! We′d like to look into this for you.
S2SA: We′d love to help out. Check out these steps:
S2SA-MMI: We′d be happy to look into this with you. Send us a DM to continue.
S2SA-K: We′re here to help. Send us a DM, and we′ll take a closer look at this issue together.
S2SA-EKA: We′d like to help you solve it. DM us what ios version you′re on.
iphone: 5 s 6 s x plus 7 7plus ios 6 6 s+apple 8 ipod …
problem: solve issue solved facing fix having solution …
输入: Thank you, AmericanAir for having great flight attendants on my flight back home!
回复: We′re glad you got to kick back and enjoy a show while flying! Thanks for your kind words.		S2S: We′re glad to have you on board!
S2SA: We′re happy to have you with us. Please DM your record locator.
S2SA-MMI: We′re so glad that we could help! We′re so glad you′re enjoying your flight!
S2SA-K: We′re glad to hear how helpful is your flight and we appreciate you,
S2SA-EKA: We′re glad to hear our AATeam helped make your experience with a little smile.		AmericanAir: SouthwestAir Delta Vir-
ginAmerica AATeam
flight plane crew …
attendants: attendant flight pilots crew stewardess agents nicest passengers …
1 SUTSKEVER I, VINYALS O, LE Q V. Sequence to sequence learning with neural networks[C]// Advances in Neural Information Processing Systems. Cambridge: MIT Press, 2014: 3104-3112.
2 LI J W, GALLEY M, BROCKETT C, et al. A diversity-promoting objective function for neural conversation models C]// The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Stroudsburg: Association for Computational Linguistics, 2016: 110-119.
3 MIKOLOV T , SUTSKEVER I , CHEN K , et al. Distributed representations of words and phrases and their compositionality[J]. Advances in Neural Information Processing Systems, 2013, (26): 3111- 3119.
4 TOMAS M.Word2vec project[DB/OL].[2018-05-20]. http://code.google.com/p/word2vec/.
5 MIKOLOV T, CHEN K, CORRADO G, et al. Efficient estimation of word representations in vector space[EB/OL].[2013-09-07].https://arxiv.org/abs/1301.3781v3.
6 SHANG Lifeng, LU Zhengdong, LI Hang. Neural responding machine for short-text conversation[C]// Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics. Stroudsburg: Association for Computational Linguistics, 2015: 1577-1586.
7 BAHDANAU D, CHO K, BENGIO Y. Neural machine translation by jointly learning to align and translate[EB/OL].[2016-05-19]. https://arxiv.org/abs/1409.0473.
8 VINYALS O, LE Q. A neural conversational model[EB/OL].[2015-07-22]. https://arxiv.org/abs/1506.05869v3.
9 LI J W, MONROE W, RITTER A, et al. Deep reinforcement learning for dialogue generation[C]// Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Stroudsburg: Association for Computational Linguistics, 2016: 1192-1202.
10 LI J W, MONROE W, SHI T, et al. Adversarial learning for neural dialogue generation[C]// Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Stroudsburg: Association for Computational Linguistics, 2017: 2147-2159.
11 CHEN Xing, WEI Wu, YU Wu, et al. Topic aware neural response generation[C]// Proceedings of the AAAI Conference on Artificial Intelligence. Menlo Park: AAAI Press, 2017: 3351-3357.
12 CHO K, VAN MERRIËNBOER B, GULCEHRE C, et al. Learning phrase representations using RNN encoder-decoder for statistical machine translation[C]// Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing. Stroudsburg: Association for Computational Linguistics, 2014: 1724-1734.
13 PAPINENI K, ROUKOS S, WARD T, et al. BLEU: a method for automatic evaluation of machine translation[C]// Proceedings of the 40th Annual Meeting on Association for Computational Linguistics. Stroudsburg: Association for Computational Linguistics, 2002: 311-318.
14 MIHALCEA R, TARAU P. Textrank: bringing order into text[C]// Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing. Stroudsburg: Association for Computational Linguistics, 2004: 404-411.
15 ROSE S, ENGEL D, CRAMER N, et al. Automatic keyword extraction from individual documents[M]// Text Mining. Chichester: John Wiley & Sons, Ltd, 2010: 1-20.
[1] LIU Biao, LU Zhe, HUANG Yu-wei, JIAO Meng, LI Quan-qi, XUE Rui. Comparative study on neural network structures in power analysis [J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2019, 54(1): 60-66.
[2] PANG Bo, LIU Yuan-chao. Fusion of pointwise and deep learning methods for passage ranking [J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2018, 53(3): 30-35.
[3] LIU Ming-ming, ZHANG Min-qing, LIU Jia, GAO Pei-xian. Steganalysis method based on shallow convolution neural network [J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2018, 53(3): 63-70.
[4] LIU Ming, ZAN Hong-ying, YUAN Hui-bin. Key sentiment sentence prediction using SVM and RNN [J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2014, 49(11): 68-73.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
[1] TANG Xiao-hong1, HU Wen-xiao2*, WEI Yan-feng2, JIANG Xi-long2, ZHANG Jing-ying2, SHAO Xue-dong3. Screening and biological characteristics studies of wide wine-making yeasts[J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2014, 49(03): 12 -17 .
[2] ZENG Weng-fu1, HUANG Tian-qiang1,2, LI Kai1, YU YANG-qiang1, GUO Gong-de1,2. A local linear emedding agorithm based on harmonicmean geodesic kernel[J]. J4, 2010, 45(7): 55 -59 .
[3] GUO Wen-juan, YANG Gong-ping*, DONG Jin-li. A review of fingerprint image segmentation methods[J]. J4, 2010, 45(7): 94 -101 .
[4] HE Hai-lun， CHEN Xiu-lan* . Circular dichroism detection of the effects of denaturants and buffers on the conformation of cold-adapted protease MCP-01 and  mesophilic protease BP01[J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2013, 48(1): 23 -29 .
[5] WANG Bi-yu, CAO Xiao-hong*. The perturbation for the Browder’s theorem of operator matrix#br#[J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2014, 49(03): 90 -95 .
[6] DING Chao1, 2, YUAN Chang-an1, 3, QIN Xiao1, 3. A prediction algorithm for multi-data streams  based on GEP[J]. J4, 2010, 45(7): 50 -54 .
[7] YI Chao-qun, LI Jian-ping, ZHU Cheng-wen. A kind of feature selection based on classification accuracy of SVM[J]. J4, 2010, 45(7): 119 -121 .
[8] SUN Liang-ji,JI Guo-xing . Jordan(α,β)-derivations and generalized Jordan(α,β)-derivations on upper triangular matrix algebras[J]. J4, 2007, 42(10): 100 -105 .
[9] SONG Ying,ZHANG Xing-fang ,WANG Qing-ping . α-reverse triple I method in the parametric Kleene's system[J]. J4, 2007, 42(7): 45 -48 .
[10] MENG Xiang-bo1, ZHANG Li-dong1, DU Zi-ping2. Investment and reinsurance strategy for insurers under #br# mean-variance criterion with jumps#br#[J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2014, 49(05): 36 -40 .
Copyright 2018 © Editorial office of JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE)
Supported by Beijing Magtech Co.Ltd