基于拓展关键词信息的对话生成模型

doi:10.6040/j.issn.1671-9352.1.2018.120

《山东大学学报(理学版)》 ›› 2019, Vol. 54 ›› Issue (7): 68-76.doi: 10.6040/j.issn.1671-9352.1.2018.120

基于拓展关键词信息的对话生成模型

郝长盈^1,²(),兰艳艳^1,^2,*(),张海楠^1,²,郭嘉丰^1,²,徐君^1,²,庞亮^1,²,程学旗^1,²

1. 中国科学院网络数据科学与技术重点实验室, 中国科学院计算技术研究所, 北京 100190
2. 中国科学院大学, 北京 100190

收稿日期:2018-10-17 出版日期:2019-07-20 发布日期:2019-06-27
通讯作者: 兰艳艳 E-mail:haochangying18s@ict.ac.cn;lanyanyan@ict.ac.cn
作者简介:郝长盈(1995—),男,硕士研究生,研究方向为自然语言处理与对话生成. E-mail:haochangying18s@ict.ac.cn
基金资助:
西藏自治区科技计划项目(XZ201801-GB-17);国家重点研发计划资助项目(2016QY02D0405);国家自然科学基金资助项目(61425016);国家自然科学基金资助项目(61472401);国家自然科学基金资助项目(61722211);国家自然科学基金资助项目(61773362);国家自然科学基金资助项目(20180290);中国科学院青年创新促进会项目(20144310);中国科学院青年创新促进会项目(2016102)

Dialogue generation model based on extended keywords information

Chang-ying HAO^1,²(),Yan-yan LAN^1,^2,*(),Hai-nan ZHANG^1,²,Jia-feng GUO^1,²,Jun XU^1,²,Liang PANG^1,²,Xue-qi CHENG^1,²

1. CAS Key Laboratory of Network Data Science and Technology, Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190, China
2. University of Chinese Academy of Sciences, Beijing 100190, China

Received:2018-10-17 Online:2019-07-20 Published:2019-06-27
Contact: Yan-yan LAN E-mail:haochangying18s@ict.ac.cn;lanyanyan@ict.ac.cn
Supported by:
西藏自治区科技计划项目(XZ201801-GB-17);国家重点研发计划资助项目(2016QY02D0405);国家自然科学基金资助项目(61425016);国家自然科学基金资助项目(61472401);国家自然科学基金资助项目(61722211);国家自然科学基金资助项目(61773362);国家自然科学基金资助项目(20180290);中国科学院青年创新促进会项目(20144310);中国科学院青年创新促进会项目(2016102)

摘要/Abstract

摘要：

在对话过程中,人们通常根据对方上一句话的关键词做出相应的回复。为了生成与关键词含义相关的回复,提出了拓展关键词信息注意力机制的对话生成模型。首先从输入语句中提取关键词,然后根据关键词词向量余弦相似度找出与关键词相关的词语构成拓展关键词集合,将集合中词语的词向量通过注意力机制的方式加入解码过程来影响回复生成。在中文微博数据集及英文Twitter数据集上的实验表明,该模型在回复语句的相关性及多样性方面取得了优于其他模型的结果。

关键词: 对话生成, 注意力机制, 序列到序列模型, 深度学习

Abstract:

Noting that people often reply to others based on the keywords in the post. In order to generate meaningful responses about the keywords, we proposed a dialogue generation model based on an attention mechanism with extended keywords information. First, we extract the keywords of the input, and then form the extended keywords set based on the words which are related to the input keywords using the cosine similarity. The words in the extended keywords set affect the response generation with the attention mechanism. Experiments on the Chinese Weibo dataset and the English Twitter dataset show that our model outperforms the traditional seq2seq model and its variations in both the metric-based evaluation and the human-evaluation.

Key words: dialogue generation, attention mechanism, Seq2Seq, deep learning

中图分类号:

TP391

郝长盈,兰艳艳,张海楠,郭嘉丰,徐君,庞亮,程学旗. 基于拓展关键词信息的对话生成模型[J]. 《山东大学学报(理学版)》, 2019, 54(7): 68-76.

Chang-ying HAO,Yan-yan LAN,Hai-nan ZHANG,Jia-feng GUO,Jun XU,Liang PANG,Xue-qi CHENG. Dialogue generation model based on extended keywords information[J]. JOURNAL OF SHANDONG UNIVERSITY(NATURAL SCIENCE), 2019, 54(7): 68-76.

图/表 10

图1

图2

图3

图4

表1

表2

表3

表4

表5

表6

参考文献 15

1	SUTSKEVER I, VINYALS O, LE Q V. Sequence to sequence learning with neural networks[C]// Advances in Neural Information Processing Systems. Cambridge: MIT Press, 2014: 3104-3112.
2	LI J W, GALLEY M, BROCKETT C, et al. A diversity-promoting objective function for neural conversation models C]// The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Stroudsburg: Association for Computational Linguistics, 2016: 110-119.
3	MIKOLOV T , SUTSKEVER I , CHEN K , et al. Distributed representations of words and phrases and their compositionality[J]. Advances in Neural Information Processing Systems, 2013, (26): 3111- 3119.
4	TOMAS M.Word2vec project[DB/OL].[2018-05-20]. http://code.google.com/p/word2vec/.
5	MIKOLOV T, CHEN K, CORRADO G, et al. Efficient estimation of word representations in vector space[EB/OL].[2013-09-07].https://arxiv.org/abs/1301.3781v3.
6	SHANG Lifeng, LU Zhengdong, LI Hang. Neural responding machine for short-text conversation[C]// Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics. Stroudsburg: Association for Computational Linguistics, 2015: 1577-1586.
7	BAHDANAU D, CHO K, BENGIO Y. Neural machine translation by jointly learning to align and translate[EB/OL].[2016-05-19]. https://arxiv.org/abs/1409.0473.
8	VINYALS O, LE Q. A neural conversational model[EB/OL].[2015-07-22]. https://arxiv.org/abs/1506.05869v3.
9	LI J W, MONROE W, RITTER A, et al. Deep reinforcement learning for dialogue generation[C]// Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Stroudsburg: Association for Computational Linguistics, 2016: 1192-1202.
10	LI J W, MONROE W, SHI T, et al. Adversarial learning for neural dialogue generation[C]// Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Stroudsburg: Association for Computational Linguistics, 2017: 2147-2159.
11	CHEN Xing, WEI Wu, YU Wu, et al. Topic aware neural response generation[C]// Proceedings of the AAAI Conference on Artificial Intelligence. Menlo Park: AAAI Press, 2017: 3351-3357.
12	CHO K, VAN MERRIËNBOER B, GULCEHRE C, et al. Learning phrase representations using RNN encoder-decoder for statistical machine translation[C]// Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing. Stroudsburg: Association for Computational Linguistics, 2014: 1724-1734.
13	PAPINENI K, ROUKOS S, WARD T, et al. BLEU: a method for automatic evaluation of machine translation[C]// Proceedings of the 40th Annual Meeting on Association for Computational Linguistics. Stroudsburg: Association for Computational Linguistics, 2002: 311-318.
14	MIHALCEA R, TARAU P. Textrank: bringing order into text[C]// Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing. Stroudsburg: Association for Computational Linguistics, 2004: 404-411.
15	ROSE S, ENGEL D, CRAMER N, et al. Automatic keyword extraction from individual documents[M]// Text Mining. Chichester: John Wiley & Sons, Ltd, 2010: 1-20.

多维度评价

Viewed

Full text

Abstract

Cited

Shared

Discussed

模型	PPL	BLEU	Distinct-1	Distinct-2
S2S	33.12	0.019 5	0.005 0	0.022 8
S2SA	30.57	0.020 2	0.006 6	0.029 5
S2SA-MMI	27.11	0.021 3	0.010 3	0.030 3
S2SA-K	17.99	0.020 3	0.006 0	0.033 5
S2SA-EKA	13.87	0.021 4	0.011 0	0.042 1

模型	PPL	BLEU	Distinct-1	Distinct-2
S2S	4.39	0.049 8	0.002 0	0.010 4
S2SA	4.26	0.055 3	0.002 6	0.014 9
S2SA-MMI	4.22	0.058 4	0.004 1	0.029 5
S2SA-K	3.74	0.057 2	0.002 9	0.026 7
S2SA-EKA	3.56	0.059 3	0.004 5	0.032 4

模型	+2	+1	0	平均分	Kappa
S2S	30.7%	34.5%	34.8%	0.959	0.810 6
S2SA	31.5%	37.2%	31.3%	1.002	0.664 2
S2SA-MMI	34.2%	34.7%	31.1%	1.031	0.786 5
S2SA-K	37.5%	28.4%	34.1%	1.034	0.831 5
S2SA-EKA	45.2%	24.6%	30.2%	1.150	0.849 3

模型	+2	+1	0	平均分	Kappa
S2S	42.8%	41.9%	15.3%	1.275	0.654 2
S2SA	46.2%	39.7%	14..1%	1.321	0.732 1
S2SA-MMI	50.7%	37.2%	12.1%	1.386	0.625 9
S2SA-K	52.5%	34.2%	13.3%	1.392	0.678 1
S2SA-EKA	54.7%	35.8%	9.5%	1.452	0.649 9

输入语句和原始回复	模型生成的回复语句	关键词及拓展关键词集合
输入:据说是二年级的数学题你会吗回复:出题的就是外星人	S2S:我想说我是外星人 S2SA:我也会有这么个情况 S2SA-MMI:出题的老师是外星人 S2SA-K:我是小学二年级二年级的学生 S2SA-EKA:这题目是在黑我的智商吗	数学题:二年级一年级小学智商小学压轴题目分钟… 二年级:数学题会一年级极限运动耐克鞋小学班级 …
输入:刚录完节目,已到我生日,祝自己38岁快乐。回复:生日快乐!永远爱你。	S2S:生日快乐,祝啊。 S2SA:生日快乐,生日快乐! S2SA-MMI:生日快乐,永远支持你。 S2SA-K:生日快乐,快乐的每一天!! S2SA-EKA:一切顺利吧,祝你生日快乐,身体健康	录:节目点将歌曲间隙盛典王菲录播录像顺利… 生日: 生日快乐庆祝岁身体健康祝老妈祝福蛋糕阴历…

基于拓展关键词信息的对话生成模型

Dialogue generation model based on extended keywords information

RichHTML

PDF (PC)

摘要/Abstract

引用本文

使用本文

图/表 10

参考文献 15

相关文章 4

多维度评价

本文评价

推荐阅读 10

[1]	刘飚,路哲,黄雨薇,焦萌,李泉其,薛瑞. 神经网络结构在功耗分析中的性能对比[J]. 《山东大学学报(理学版)》, 2019, 54(1): 60-66.
[2]	庞博,刘远超. 融合pointwise及深度学习方法的篇章排序[J]. 山东大学学报（理学版）, 2018, 53(3): 30-35.
[3]	刘明明,张敏情,刘佳,高培贤. 一种基于浅层卷积神经网络的隐写分析方法[J]. 山东大学学报（理学版）, 2018, 53(3): 63-70.
[4]	刘铭, 昝红英, 原慧斌. 基于SVM与RNN的文本情感关键句判定与抽取[J]. 山东大学学报（理学版）, 2014, 49(11): 68-73.

输入语句和原始回复	模型生成的回复语句	关键词及拓展关键词集合
输入: I have a problem with my iPhone 回复: We′re happy to help. What seems to be the problem	S2S: Hey there! We′d like to look into this for you. S2SA: We′d love to help out. Check out these steps: S2SA-MMI: We′d be happy to look into this with you. Send us a DM to continue. S2SA-K: We′re here to help. Send us a DM, and we′ll take a closer look at this issue together. S2SA-EKA: We′d like to help you solve it. DM us what ios version you′re on.	iphone: 5 s 6 s x plus 7 7plus ios 6 6 s+apple 8 ipod … problem: solve issue solved facing fix having solution …
输入: Thank you, AmericanAir for having great flight attendants on my flight back home! 回复: We′re glad you got to kick back and enjoy a show while flying! Thanks for your kind words.	S2S: We′re glad to have you on board! S2SA: We′re happy to have you with us. Please DM your record locator. S2SA-MMI: We′re so glad that we could help! We′re so glad you′re enjoying your flight! S2SA-K: We′re glad to hear how helpful is your flight and we appreciate you, S2SA-EKA: We′re glad to hear our AATeam helped make your experience with a little smile.	AmericanAir: SouthwestAir Delta Vir- ginAmerica AATeam flight plane crew … attendants: attendant flight pilots crew stewardess agents nicest passengers …