Deep Reinforcement Learning Based Dialogue Policy Network

Zou Zhibin; Hao Zhifeng; Xu Boyan

doi:10.12052/gdutxb.240052

English Version Journal Login

您当前的位置：

首页 >

文章列表页 >

Deep Reinforcement Learning Based Dialogue Policy Network

更新时间：2026-05-27

- Deep Reinforcement Learning Based Dialogue Policy Network
- Vol. 42, Issue 5, Pages: 105-111(2025)
- 作者机构：
  
  1. 广东工业大学计算机学院,广东,广州,510006
  2. 汕头大学工学院,广东,汕头,515063
- 作者简介：
- 基金信息：
- DOI：10.12052/gdutxb.240052
  CLC： TP391
- Published：2025
- 稿件说明：
移动端阅览
邹志彬, 郝志峰, 许柏炎. 基于深度强化学习的对话策略网络[J]. 广东工业大学学报, 2025, 42(5): 105–111.

Zou Zhibin, Hao Zhifeng, Xu Boyan. Deep Reinforcement Learning Based Dialogue Policy Network[J]. 2025, 42(5): 105-111.
邹志彬, 郝志峰, 许柏炎. 基于深度强化学习的对话策略网络[J]. 广东工业大学学报, 2025, 42(5): 105–111. DOI： 10.12052/gdutxb.240052

Zou Zhibin, Hao Zhifeng, Xu Boyan. Deep Reinforcement Learning Based Dialogue Policy Network[J]. 2025, 42(5): 105-111. DOI： 10.12052/gdutxb.240052.

摘要

对话系统是人机交互不可或缺的一部分，其目标是实现自然、智能和流畅的对话体验。然而，传统的对话系统在理解用户意图、生成合理回复和保持对话连贯性方面面临着挑战。其核心原因是对话系统策略网络的监督训练与最终评价指标不匹配，目标函数无法涵盖系统所需关键性指标。近来部分研究工作开始探索强化学习导向型训练，然而现有工作仍有依赖于人为定义模板、对话轮次过多等问题。为了解决上述问题，本文采用了基于Transformer的对话策略网络构建单词级深度强化学习训练过程，通过组合对话系统多方面指标构建奖励，改善对话系统交互性能。实验结果显示，相对于传统监督训练方法，本文方法在对话性能指标BLEU和Comb上有所提升，在其他指标上也能保持相当的性能。本文的贡献在于将强化学习技术应用于对话系统的策略网络训练过程，通过实验验证了其有效性。

Abstract

Dialogue systems are an integral part of human-computer interaction

and their goal is to achieve a natural

intelligent

and smooth dialog experience. However

traditional dialog systems face challenges in understanding user intent

generating reasonable responses

and maintaining dialog coherence. The core reason is the mismatch between the supervised training of the dialog system strategy network and the final evaluation metrics

and the objective function fails to cover the critical metrics required by the system. Some recent research work has begun to explore reinforcement learning-oriented training; however

the existing work still suffers from the problems of relying on artificially defined templates and too many dialog rounds. In order to solve the above problems

this study employs a Transformer-based dialog strategy network to construct a word-level deep reinforcement learning training process

which improves the interactive performance of the dialog system by combining the multifaceted metrics of the dialog system to construct rewards. The experimental results show that relative to the traditional supervised training method

the policy network for deep reinforcement learning has an improvement in the dialog performance metrics BLEU and Comb

and maintains comparable performance in other metrics. The contribution of this study is to apply reinforcement learning techniques to the training process of policy networks for dialog systems

and its effectiveness and potential are verified through experiments.

关键词

Keywords

references

LI Z, KISELEVA J, DE RIJKE M. Rethinking supervised learning and reinforcement learning in task-oriented dialogue systems[C]//Findings of the Association for Computational Linguistics: EMNLP2020. Online: Association for Computational Linguistics. 2020: 3537-3546.

WANG H, WONG K F. A collaborative multi-agent reinforcement learning framework for dialog action decomposition[C]//Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Punta Cana: Association for Computational Linguistics, 2021: 7882-7889.

GEISHAUSER C, VAN NIEKERK C, LIN H C, et al . Dynamic dialogue policy for continual reinforcement learning[C ] //Proceedings of the 29th International Conference on Computational Linguistics. Gyeongju: International Committee on Computational Linguistics, 2022: 266-284.

VASWANI A, SHAZEER N, PARMAR N, et al . Attention is all you need[C ] //Advances in Neural Information Processing Systems. Long Beach: Neural Information Processing Systems Foundation , 2017: 5998-6008.

BRANAVAN S R K, CHEN H, ZETTLEMOYER L, et al . Reinforcement learning for mapping instructions to actions[C ] //Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP. Suntec, Singapore: Association for Computational Linguistics, 2009: 82-90

ZHANG H, CAI J, XU J, et al . Pretraining-based natural language generation for text summarization[C ] //Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL) . Hong Kong, China: Association for Computational Linguistics, 2019: 789-797.

ZHANG Y, SUN S, GALLEY M, et al . DialoGPT: large-scale generative pre-training for conversational response generation[C ] //Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations. Online: Association for Computational Linguistics , 2020: 270-278.

ROLLER S, DINAN E, GOYAL N, et al . Recipes for building an open-domain chatbot[C ] //Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics. Online: Association for Computational Linguistics, 2021: 300-325.

BAO S, HE H, WANG F, et al . PLATO: Pre-trained dialogue generation model with discrete latent variable[C ] //P roceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Online: Association for Computational Linguistics, 2020: 85-96.

JIANG Z H, YU W, ZHOU D, et al . Convbert: improving bert with span-based dynamic convolution[J ] . Advances in Neural Information Processing Systems, 2020, 33: 12837-12848.

WU C S, HOI S C H, SOCHER R, et al . TOD-BERT: pre-trained natural language understanding for task-oriented dialogue[C ] //Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) . Online: Association for Computational Linguistics, 2020: 917-929.

PENG B, ZHU C, LI C, et al . Few-shot natural language generation for task-oriented dialog[C ] //Findings of the Association for Computational Linguistics: EMNLP 2020. Online: Association for Computational Linguistics, 2020: 172-182.

PENG B, LI C, LI J, et al . Soloist: few-shot task-oriented dialog with a single pretrained auto-regressive model[J ] . Transactions of the Association for Computational Linguistics, 2021, 9: 807-824.

HE W, DAI Y, ZHENG Y, et al . Galaxy: a generative pre-trained model for task-oriented dialog with semi-supervised learning and explicit policy injection[C ] //P roceedings of the AAAI Conference on Artificial Intelligence. New York: AAAI, 2022, 36(10): 10749-10757.

SU Y, SHU L, MANSIMOV E, et al . Multi-task pre-training for plug-and-play task-oriented dialogue system[C ] //Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics. Dublin: Association for Computational Linguistics, 2022: 4661-4676.

HE W, DAI Y, HUI B, et al . SPACE-2: tree-structured semi- supervised contrastive pre-training for task-oriented dialog understanding[C ] //Proceedings of the 29th International Conference on Computational Linguistics. Gyeongju: International Committee on Computational Linguistics, 2022: 553-569.

HE W, DAI Y, YANG M, et al . Unified dialog model pre-training for task-oriented dialog understanding and generation[C ] //Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval. New York: Association for Computing Machinery, 2022: 187-200.

PENG B, LI X, LI L, et al . Composite task-completion dialogue policy learning via hierarchical deep reinforcement learning[C ] //Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Copenhagen: Association for Computational Linguistics, 2017: 2231-2240.

WANG H, PENG B, WONG K F. Learning efficient dialogue policy from demonstrations through shaping[C]//Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Online: Association for Computational Linguistics, 2020: 6355-6365.

TAKANOBU R, LIANG R, HUANG M. Multi-agent task-oriented dialog policy learning with role-aware reward decomposition[C]//Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Online: Association for Computational Linguistics. 2020: 625-638.

DAI B, ZENG B, WEI P F, et al . The task dialogue strategy learning method of discriminative deep Dyna-Q was improved[J ] . Journal of Guangdong University of Technology, 2023, 40(4): 9-17.

ERIC M, GOEL R, PAUL S, et al . MultiWOZ 2.1: a consolidated multi-domain dialogue dataset with state corrections and state tracking baselines[C ] //Proceedings of the Twelfth Language Resources and Evaluation Conference. Marseille: European Language Resources Association, 2020: 422-428.

HOSSEINI-ASL E, MCCANN B, WU C S, et al . A simple language model for task-oriented dialogue[C ] //Advances in Neural Information Processing Systems (NeurIPS 2020) . Vancouver: Neural Information Processing Systems Foundation, 2020: 13.

YANG Y, LI Y, QUAN X. Ubar: Towards fully end-to-end task-oriented dialog system with gpt-2[J]. Proceedings of the AAAI Conference on Artificial Intelligence. 2021, 35(16): 14230-14238.

LEE Y. Improving end-to-end task-oriented dialog system with a simple auxiliary task[C]//Findings of the Association for Computational Linguistics: EMNLP 2021. Punta Cana: Association for Computational Linguistics, 2021: 1296-1303.

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

A Task-oriented Dialogue Policy Learning Method of Improved Discriminative Deep Dyna-Q

Deep Reinforcement Learning-based Spectrum Allocation in Vehicular Networks

Reinforcement Learning-based Automatic Tuning Method and Application of Injection Molding Process Parameters

Meta Reinforcement Learning Based on Skill Discovery

Reinforcement Learning Model for Automatic Inspection Route Based on Multi-agent Attention Mechanism

Related Author

Dai Bin

Zeng Bi

Wei Peng-fei

Huang Yong-jian

Wan Pin

Wang Yonghua

Lin Zhicong

Ren Zhigang

Related Institution

Guangzhou Xuanyuan Research Institute Co., Ltd.

School of Automation, Guangdong University of Technology

School of Intelligent Manufacturing and Electrical Engineering, Guangzhou Institute of Science and Technology

College of Science, Shantou University

School of Information Engineering, Guangdong University of Technology

AI问答

⁰