A Task-oriented Dialogue Policy Learning Method of Improved Discriminative Deep Dyna-Q

Dai Bin; Zeng Bi; Wei Peng-fei; Huang Yong-jian

doi:10.12052/gdutxb.220122

English Version Journal Login

您当前的位置：

首页 >

文章列表页 >

A Task-oriented Dialogue Policy Learning Method of Improved Discriminative Deep Dyna-Q

更新时间：2026-05-27

- A Task-oriented Dialogue Policy Learning Method of Improved Discriminative Deep Dyna-Q
- Vol. 40, Issue 4, Pages: 9-17, 23(2023)
- 作者机构：
  
  1. 广东工业大学计算机学院,广东,广州,510006
  2. 广州轩辕研究院有限公司,广东,广州,510000
- 作者简介：
- 基金信息：
- DOI：10.12052/gdutxb.220122
  CLC： TP391
- Online First：25 July 2023，
  
  Published：2023
- 稿件说明：
移动端阅览
戴彬, 曾碧, 魏鹏飞, 黄永健. 改进判别式深度Dyna-Q的任务对话策略学习方法[J]. 广东工业大学学报, 2023, 40(4): 9-17, 23.

Dai Bin, Zeng Bi, Wei Peng-fei, et al. A Task-oriented Dialogue Policy Learning Method of Improved Discriminative Deep Dyna-Q[J]. 2023, 40(4): 9-17, 23.
戴彬, 曾碧, 魏鹏飞, 黄永健. 改进判别式深度Dyna-Q的任务对话策略学习方法[J]. 广东工业大学学报, 2023, 40(4): 9-17, 23. DOI： 10.12052/gdutxb.220122.

Dai Bin, Zeng Bi, Wei Peng-fei, et al. A Task-oriented Dialogue Policy Learning Method of Improved Discriminative Deep Dyna-Q[J]. 2023, 40(4): 9-17, 23. DOI： 10.12052/gdutxb.220122.

摘要

作为任务型对话系统中的关键一环，对话策略可以通过判别式深度Dyna-Q框架训练得到。然而，该框架在直接强化学习阶段采用原始的深度Q网络方法学习对话策略，在世界模型方面采用多层感知机作为模型的基本结构，导致对话策略的训练效率、性能和稳定性降低。本文提出了一种改进判别式深度Dyna-Q的任务对话策略学习方法。在改进后的直接强化学习阶段，利用噪声网络改进了智能体的探索方式，同时将竞争网络的双流架构、双Q网络与

$ n $

img text_id='M1' class='formula-img' style='display:none;' src='gdgydxxb-40-4-9_M1.jpg'>

步自举法三者相结合，优化了

$ Q $

img text_id='M2' class='formula-img' style='display:none;' src='gdgydxxb-40-4-9_M2.jpg'>

值的计算过程。在世界模型方面，设计了一种基于软注意力的模型替代多层感知机结构。实验结果表明，本文提出的方法在对话成功率、平均对话轮数以及平均奖励3个指标上均优于现有的最佳结果，最后本文通过消融分析和鲁棒性分析，进一步验证了方法的有效性。

Abstract

As a pivotal part of the task-oriented dialogue system

dialogue policy can be trained by using the discriminative deep Dyna-Q framework. However

the framework uses vanilla deep Q-network method in the direct reinforcement learning phase and adopts MLPs as the basic network

of world model

which limits the efficiency and stability of the dialogue policy learning. In this paper

we purpose an improved discriminative deep Dyna-Q method for task-oriented dialogue policy learning. In the improved direct RL phase

we first employ a NoisyNet to improve the exploration method

and then combine the dual-stream architecture of Dueling Network

Double-Q Network and

-step bootstrapping to optimize the calculation of the

values. Moreover

we design a soft-attention-based model to replace the MLPs in the world model. The experimental results show that our proposed method achieves better results than other baseline models in terms of task success rate

average dialog turns and average reward. We further validate the effectiveness of proposed method by conducting both ablation and robustness analysis.

关键词

Keywords

references

ZHANG Z, TAKANOBU R, ZHU Q, et al . Recent advances and challenges in task-oriented dialog systems [J ] . Science China Technological Sciences, 2020, 63(10): 2011-2027.

LEVIN E, PIERACCINI R, ECKERT W. Learning dialogue strategies within the markov decision process framework[C]//1997 IEEE Workshop on Automatic Speech Recognition and Understanding Proceedings. Santa Barbara: IEEE, 1997: 72-79.

YOUNG S, GAŠIĆ M, THOMSON B, et al . POMDP-based statistical spoken dialog systems: a review [J ] . Proceedings of the IEEE, 2013, 101(5): 1160-1179.

WAN L P, LAN X G, ZHANG H B, et al . A review of deep reinforcement learning theory and applic ation [J ] . Pattern Recognition and Artificial Intelligence, 2019(1): 67-81.

MNIH V, KAVUKCUOGLU K, SILVER D, et al . Human-level control through deep reinforcement learning [J ] . Nature, 2015, 518(7540): 529-533.

HAARNOJA T, ZHOU A, ABBEEL P, et al. Soft actor-critic: off-policy maximum entropy deep reinforcement learning with a stochastic actor[C]// Proceedings of the 35th International Conference on Machine Learning. Stockholm: PMLR, 2018: 1861-1870.

SCHATZMANN J, THOMSON B, WEILHAMMER K, et al. Agenda-based user simulation for bootstrapping a POMDP dialogue system[C]// Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics. Rochester: Association for Computational Linguistics, 2007: 149-152.

PENG B, LI X, GAO J, et al. Deep Dyna-Q: integrating planning for task-completion dialogue policy learning[C]//Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics. Melbourne: Association for Computational Linguistics, 2018: 2182-2192.

SU S Y, LI X, GAO J, et al. Discriminative deep Dyna-Q: robust planning for dialogue policy learning[C]//Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Brussels: Association for Computational Linguistics, 2018: 3813-3823.

DHINGRA B, LI L, LI X, et al. Towards end-to-end reinforcement learning of dialogue agents for information access[C]//Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics. Vancouver: Association for Computational Linguistics, 2017: 484-495.

SUTTON R S. Integrated architectures for learning, planning, and reacting based on approximating dynamic programming[M]//Machine Learning Proceedings 1990. Burlington: Morgan Kaufmann, 1990: 216-224.

GOODFELLOW I J, POUGET-ABADIE J, MIRZA M, et al . Generative adversarial networks [J ] . Communications of the ACM, 2020, 63(11): 139-144.

HAKKANI-TR D, TR G, CELIKYILMAZ A, et al. Multi-domain joint semantic frame parsing using bi-directional RNN-LSTM[C]//Interspeech 2016. San Francisco: ISCA, 2016: 715-719.

MRKI N, SAGHDHA D , WEN T H, et al. Neural belief tracker: data-driven dialogue state tracking[C]//Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics. Vancouver: Association for Computational Linguistics, 2017: 1777-1788.

WEN T H, GASIC M, MRKI N, et al. Semantically conditioned LSTM-based natural language generation for spoken dialogue systems[C]//Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Lisbon: Association for Computational Linguistics, 2015: 1711-1721.

FORTUNATO M, AZAR M G, PIOT B, et al. Noisy networks for exploration[C]//International Conference on Learning Representations. Vancouver: ICLR, 2018: 1-21.

WANG Z, SCHAUL T, HESSEL M, et al. Dueling network architectures for deep reinforcement learning[C]// Proceedings of The 33rd International Conference on Machine Learning. New York: PMLR, 2016: 1995-2003.

VAN H H, GUEZ A, SILVER D. Deep reinforcement learning with double Q-learning[C]//Proceedings of the 30 th AAAI Conference on Artificial Intelligence. New York: AAAI, 2016, 2094-2100.

SUTTON R S. Learning to predict by the methods of temporal differences [J]. Machine Learning, 1988, 3(1): 9-44.

ZHANG Y, YU X, CUI Z, et al. Every document owns its structure: inductive text classification via graph neural networks[C]//Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Online: Association for Computational Linguistics, 2020: 334-339.

HOCHREITER S, SCHMIDHUBER J. Long short-term memory [J]. Neural Computation, 1997, 9(8): 1735-1780.

LI X, CHEN Y N, LI L, et al. End-to-end task-completion neural dialogue systems[C]//Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Taipei: IJCNLP, 2017: 733-743.

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Deep Reinforcement Learning Based Dialogue Policy Network

Deep Reinforcement Learning-based Spectrum Allocation in Vehicular Networks

Reinforcement Learning-based Automatic Tuning Method and Application of Injection Molding Process Parameters

Meta Reinforcement Learning Based on Skill Discovery

Reinforcement Learning Model for Automatic Inspection Route Based on Multi-agent Attention Mechanism

Related Author

Xu Boyan

Hao Zhifeng

Zou Zhibin

Wan Pin

Wang Yonghua

Lin Zhicong

Ren Zhigang

Lin Jianghao

Related Institution

College of Engineering, Shantou University

School of Automation, Guangdong University of Technology

School of Intelligent Manufacturing and Electrical Engineering, Guangzhou Institute of Science and Technology

College of Science, Shantou University

School of Information Engineering, Guangdong University of Technology

AI问答

⁰