改进判别式深度Dyna-Q的任务对话策略学习方法

戴彬; 曾碧; 魏鹏飞; 黄永健

doi:10.12052/gdutxb.220122

欢迎来到能源电力期刊网！

English Version 期刊登录

您当前的位置：

首页 >

文章列表页 >

改进判别式深度Dyna-Q的任务对话策略学习方法

计算机科学与技术 | 更新时间：2026-05-27

- 改进判别式深度Dyna-Q的任务对话策略学习方法
- A Task-oriented Dialogue Policy Learning Method of Improved Discriminative Deep Dyna-Q
- 广东工业大学学报 2023年40卷第4期页码：9-17, 23
- 作者机构：
  
  1. 广东工业大学计算机学院,广东,广州,510006
  2. 广州轩辕研究院有限公司,广东,广州,510000
- 作者简介：
  
  戴彬(1997–) ，男，硕士研究生，主要研究方向为自然语言处理、任务型对话系统和强化学习
  魏鹏飞(1991–) ，男，助理实验师，主要研究方向为任务型对话系统和强化学习，E-mail：wpf@mail2.gdut.edu.cn
- 基金信息：
  
  国家自然科学基金联合基金资助重点项目(U21A20478);广东省自然科学基金资助项目(2019A1515011056);顺德区核心技术攻关项目(2130218003002)
- DOI：10.12052/gdutxb.220122
  中图分类号： TP391
- 网络首发：2023-07-25，
  
  纸质出版：2023
- 稿件说明：
移动端阅览
戴彬, 曾碧, 魏鹏飞, 黄永健. 改进判别式深度Dyna-Q的任务对话策略学习方法[J]. 广东工业大学学报, 2023, 40(4): 9-17, 23.

Dai Bin, Zeng Bi, Wei Peng-fei, et al. A Task-oriented Dialogue Policy Learning Method of Improved Discriminative Deep Dyna-Q[J]. 2023, 40(4): 9-17, 23.
戴彬, 曾碧, 魏鹏飞, 黄永健. 改进判别式深度Dyna-Q的任务对话策略学习方法[J]. 广东工业大学学报, 2023, 40(4): 9-17, 23. DOI： 10.12052/gdutxb.220122.

Dai Bin, Zeng Bi, Wei Peng-fei, et al. A Task-oriented Dialogue Policy Learning Method of Improved Discriminative Deep Dyna-Q[J]. 2023, 40(4): 9-17, 23. DOI： 10.12052/gdutxb.220122.

摘要

作为任务型对话系统中的关键一环，对话策略可以通过判别式深度Dyna-Q框架训练得到。然而，该框架在直接强化学习阶段采用原始的深度Q网络方法学习对话策略，在世界模型方面采用多层感知机作为模型的基本结构，导致对话策略的训练效率、性能和稳定性降低。本文提出了一种改进判别式深度Dyna-Q的任务对话策略学习方法。在改进后的直接强化学习阶段，利用噪声网络改进了智能体的探索方式，同时将竞争网络的双流架构、双Q网络与

$ n $

img text_id='M1' class='formula-img' style='display:none;' src='gdgydxxb-40-4-9_M1.jpg'>

步自举法三者相结合，优化了

$ Q $

img text_id='M2' class='formula-img' style='display:none;' src='gdgydxxb-40-4-9_M2.jpg'>

值的计算过程。在世界模型方面，设计了一种基于软注意力的模型替代多层感知机结构。实验结果表明，本文提出的方法在对话成功率、平均对话轮数以及平均奖励3个指标上均优于现有的最佳结果，最后本文通过消融分析和鲁棒性分析，进一步验证了方法的有效性。

Abstract

As a pivotal part of the task-oriented dialogue system

dialogue policy can be trained by using the discriminative deep Dyna-Q framework. However

the framework uses vanilla deep Q-network method in the direct reinforcement learning phase and adopts MLPs as the basic network

of world model

which limits the efficiency and stability of the dialogue policy learning. In this paper

we purpose an improved discriminative deep Dyna-Q method for task-oriented dialogue policy learning. In the improved direct RL phase

we first employ a NoisyNet to improve the exploration method

and then combine the dual-stream architecture of Dueling Network

Double-Q Network and

-step bootstrapping to optimize the calculation of the

values. Moreover

we design a soft-attention-based model to replace the MLPs in the world model. The experimental results show that our proposed method achieves better results than other baseline models in terms of task success rate

average dialog turns and average reward. We further validate the effectiveness of proposed method by conducting both ablation and robustness analysis.

关键词

Keywords

references

ZHANG Z, TAKANOBU R, ZHU Q, et al . Recent advances and challenges in task-oriented dialog systems [J ] . Science China Technological Sciences, 2020, 63(10): 2011-2027.

LEVIN E, PIERACCINI R, ECKERT W. Learning dialogue strategies within the markov decision process framework[C]//1997 IEEE Workshop on Automatic Speech Recognition and Understanding Proceedings. Santa Barbara: IEEE, 1997: 72-79.

YOUNG S, GAŠIĆ M, THOMSON B, et al . POMDP-based statistical spoken dialog systems: a review [J ] . Proceedings of the IEEE, 2013, 101(5): 1160-1179.

WAN L P, LAN X G, ZHANG H B, et al . A review of deep reinforcement learning theory and applic ation [J ] . Pattern Recognition and Artificial Intelligence, 2019(1): 67-81.

MNIH V, KAVUKCUOGLU K, SILVER D, et al . Human-level control through deep reinforcement learning [J ] . Nature, 2015, 518(7540): 529-533.

HAARNOJA T, ZHOU A, ABBEEL P, et al. Soft actor-critic: off-policy maximum entropy deep reinforcement learning with a stochastic actor[C]// Proceedings of the 35th International Conference on Machine Learning. Stockholm: PMLR, 2018: 1861-1870.

SCHATZMANN J, THOMSON B, WEILHAMMER K, et al. Agenda-based user simulation for bootstrapping a POMDP dialogue system[C]// Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics. Rochester: Association for Computational Linguistics, 2007: 149-152.

PENG B, LI X, GAO J, et al. Deep Dyna-Q: integrating planning for task-completion dialogue policy learning[C]//Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics. Melbourne: Association for Computational Linguistics, 2018: 2182-2192.

SU S Y, LI X, GAO J, et al. Discriminative deep Dyna-Q: robust planning for dialogue policy learning[C]//Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Brussels: Association for Computational Linguistics, 2018: 3813-3823.

DHINGRA B, LI L, LI X, et al. Towards end-to-end reinforcement learning of dialogue agents for information access[C]//Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics. Vancouver: Association for Computational Linguistics, 2017: 484-495.

SUTTON R S. Integrated architectures for learning, planning, and reacting based on approximating dynamic programming[M]//Machine Learning Proceedings 1990. Burlington: Morgan Kaufmann, 1990: 216-224.

GOODFELLOW I J, POUGET-ABADIE J, MIRZA M, et al . Generative adversarial networks [J ] . Communications of the ACM, 2020, 63(11): 139-144.

HAKKANI-TR D, TR G, CELIKYILMAZ A, et al. Multi-domain joint semantic frame parsing using bi-directional RNN-LSTM[C]//Interspeech 2016. San Francisco: ISCA, 2016: 715-719.

MRKI N, SAGHDHA D , WEN T H, et al. Neural belief tracker: data-driven dialogue state tracking[C]//Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics. Vancouver: Association for Computational Linguistics, 2017: 1777-1788.

WEN T H, GASIC M, MRKI N, et al. Semantically conditioned LSTM-based natural language generation for spoken dialogue systems[C]//Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Lisbon: Association for Computational Linguistics, 2015: 1711-1721.

FORTUNATO M, AZAR M G, PIOT B, et al. Noisy networks for exploration[C]//International Conference on Learning Representations. Vancouver: ICLR, 2018: 1-21.

WANG Z, SCHAUL T, HESSEL M, et al. Dueling network architectures for deep reinforcement learning[C]// Proceedings of The 33rd International Conference on Machine Learning. New York: PMLR, 2016: 1995-2003.

VAN H H, GUEZ A, SILVER D. Deep reinforcement learning with double Q-learning[C]//Proceedings of the 30 th AAAI Conference on Artificial Intelligence. New York: AAAI, 2016, 2094-2100.

SUTTON R S. Learning to predict by the methods of temporal differences [J]. Machine Learning, 1988, 3(1): 9-44.

ZHANG Y, YU X, CUI Z, et al. Every document owns its structure: inductive text classification via graph neural networks[C]//Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Online: Association for Computational Linguistics, 2020: 334-339.

HOCHREITER S, SCHMIDHUBER J. Long short-term memory [J]. Neural Computation, 1997, 9(8): 1735-1780.

LI X, CHEN Y N, LI L, et al. End-to-end task-completion neural dialogue systems[C]//Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Taipei: IJCNLP, 2017: 733-743.

浏览量

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

基于深度强化学习的对话策略网络

基于深度强化学习的车联网频谱分配研究

基于强化学习的注塑工艺参数自动调优方法及应用

基于技能发现的元强化学习

基于多智能体注意力机制的自动巡检路线强化学习模型