基于深度强化学习的对话策略网络

邹志彬; 郝志峰; 许柏炎

doi:10.12052/gdutxb.240052

欢迎来到能源电力期刊网！

English Version 期刊登录

您当前的位置：

首页 >

文章列表页 >

基于深度强化学习的对话策略网络

计算机科学与技术 | 更新时间：2026-05-27

- 基于深度强化学习的对话策略网络
- Deep Reinforcement Learning Based Dialogue Policy Network
- 广东工业大学学报 2025年42卷第5期页码：105-111
- 作者机构：
  
  1. 广东工业大学计算机学院,广东,广州,510006
  2. 汕头大学工学院,广东,汕头,515063
- 作者简介：
  
  邹志彬(1998–) ，男，硕士研究生，主要研究方向为强化学习、对话系统，E-mail：892356072@qq.com
  郝志峰(1968–) ，男，教授，博士，主要研究方向为数学建模、数据挖掘，E-mail：zfhao@gdut.edu.cn
- 基金信息：
  
  国家自然科学基金资助项目(61876043, 61976052, 62206064)
- DOI：10.12052/gdutxb.240052
  中图分类号： TP391
- 纸质出版：2025
- 稿件说明：
移动端阅览
邹志彬, 郝志峰, 许柏炎. 基于深度强化学习的对话策略网络[J]. 广东工业大学学报, 2025, 42(5): 105–111.

Zou Zhibin, Hao Zhifeng, Xu Boyan. Deep Reinforcement Learning Based Dialogue Policy Network[J]. 2025, 42(5): 105-111.
邹志彬, 郝志峰, 许柏炎. 基于深度强化学习的对话策略网络[J]. 广东工业大学学报, 2025, 42(5): 105–111. DOI： 10.12052/gdutxb.240052

Zou Zhibin, Hao Zhifeng, Xu Boyan. Deep Reinforcement Learning Based Dialogue Policy Network[J]. 2025, 42(5): 105-111. DOI： 10.12052/gdutxb.240052.

摘要

对话系统是人机交互不可或缺的一部分，其目标是实现自然、智能和流畅的对话体验。然而，传统的对话系统在理解用户意图、生成合理回复和保持对话连贯性方面面临着挑战。其核心原因是对话系统策略网络的监督训练与最终评价指标不匹配，目标函数无法涵盖系统所需关键性指标。近来部分研究工作开始探索强化学习导向型训练，然而现有工作仍有依赖于人为定义模板、对话轮次过多等问题。为了解决上述问题，本文采用了基于Transformer的对话策略网络构建单词级深度强化学习训练过程，通过组合对话系统多方面指标构建奖励，改善对话系统交互性能。实验结果显示，相对于传统监督训练方法，本文方法在对话性能指标BLEU和Comb上有所提升，在其他指标上也能保持相当的性能。本文的贡献在于将强化学习技术应用于对话系统的策略网络训练过程，通过实验验证了其有效性。

Abstract

Dialogue systems are an integral part of human-computer interaction

and their goal is to achieve a natural

intelligent

and smooth dialog experience. However

traditional dialog systems face challenges in understanding user intent

generating reasonable responses

and maintaining dialog coherence. The core reason is the mismatch between the supervised training of the dialog system strategy network and the final evaluation metrics

and the objective function fails to cover the critical metrics required by the system. Some recent research work has begun to explore reinforcement learning-oriented training; however

the existing work still suffers from the problems of relying on artificially defined templates and too many dialog rounds. In order to solve the above problems

this study employs a Transformer-based dialog strategy network to construct a word-level deep reinforcement learning training process

which improves the interactive performance of the dialog system by combining the multifaceted metrics of the dialog system to construct rewards. The experimental results show that relative to the traditional supervised training method

the policy network for deep reinforcement learning has an improvement in the dialog performance metrics BLEU and Comb

and maintains comparable performance in other metrics. The contribution of this study is to apply reinforcement learning techniques to the training process of policy networks for dialog systems

and its effectiveness and potential are verified through experiments.

关键词

Keywords

references

LI Z, KISELEVA J, DE RIJKE M. Rethinking supervised learning and reinforcement learning in task-oriented dialogue systems[C]//Findings of the Association for Computational Linguistics: EMNLP2020. Online: Association for Computational Linguistics. 2020: 3537-3546.

WANG H, WONG K F. A collaborative multi-agent reinforcement learning framework for dialog action decomposition[C]//Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Punta Cana: Association for Computational Linguistics, 2021: 7882-7889.

GEISHAUSER C, VAN NIEKERK C, LIN H C, et al . Dynamic dialogue policy for continual reinforcement learning[C ] //Proceedings of the 29th International Conference on Computational Linguistics. Gyeongju: International Committee on Computational Linguistics, 2022: 266-284.

VASWANI A, SHAZEER N, PARMAR N, et al . Attention is all you need[C ] //Advances in Neural Information Processing Systems. Long Beach: Neural Information Processing Systems Foundation , 2017: 5998-6008.

BRANAVAN S R K, CHEN H, ZETTLEMOYER L, et al . Reinforcement learning for mapping instructions to actions[C ] //Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP. Suntec, Singapore: Association for Computational Linguistics, 2009: 82-90

ZHANG H, CAI J, XU J, et al . Pretraining-based natural language generation for text summarization[C ] //Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL) . Hong Kong, China: Association for Computational Linguistics, 2019: 789-797.

ZHANG Y, SUN S, GALLEY M, et al . DialoGPT: large-scale generative pre-training for conversational response generation[C ] //Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations. Online: Association for Computational Linguistics , 2020: 270-278.

ROLLER S, DINAN E, GOYAL N, et al . Recipes for building an open-domain chatbot[C ] //Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics. Online: Association for Computational Linguistics, 2021: 300-325.

BAO S, HE H, WANG F, et al . PLATO: Pre-trained dialogue generation model with discrete latent variable[C ] //P roceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Online: Association for Computational Linguistics, 2020: 85-96.

JIANG Z H, YU W, ZHOU D, et al . Convbert: improving bert with span-based dynamic convolution[J ] . Advances in Neural Information Processing Systems, 2020, 33: 12837-12848.

WU C S, HOI S C H, SOCHER R, et al . TOD-BERT: pre-trained natural language understanding for task-oriented dialogue[C ] //Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) . Online: Association for Computational Linguistics, 2020: 917-929.

PENG B, ZHU C, LI C, et al . Few-shot natural language generation for task-oriented dialog[C ] //Findings of the Association for Computational Linguistics: EMNLP 2020. Online: Association for Computational Linguistics, 2020: 172-182.

PENG B, LI C, LI J, et al . Soloist: few-shot task-oriented dialog with a single pretrained auto-regressive model[J ] . Transactions of the Association for Computational Linguistics, 2021, 9: 807-824.

HE W, DAI Y, ZHENG Y, et al . Galaxy: a generative pre-trained model for task-oriented dialog with semi-supervised learning and explicit policy injection[C ] //P roceedings of the AAAI Conference on Artificial Intelligence. New York: AAAI, 2022, 36(10): 10749-10757.

SU Y, SHU L, MANSIMOV E, et al . Multi-task pre-training for plug-and-play task-oriented dialogue system[C ] //Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics. Dublin: Association for Computational Linguistics, 2022: 4661-4676.

HE W, DAI Y, HUI B, et al . SPACE-2: tree-structured semi- supervised contrastive pre-training for task-oriented dialog understanding[C ] //Proceedings of the 29th International Conference on Computational Linguistics. Gyeongju: International Committee on Computational Linguistics, 2022: 553-569.

HE W, DAI Y, YANG M, et al . Unified dialog model pre-training for task-oriented dialog understanding and generation[C ] //Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval. New York: Association for Computing Machinery, 2022: 187-200.

PENG B, LI X, LI L, et al . Composite task-completion dialogue policy learning via hierarchical deep reinforcement learning[C ] //Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Copenhagen: Association for Computational Linguistics, 2017: 2231-2240.

WANG H, PENG B, WONG K F. Learning efficient dialogue policy from demonstrations through shaping[C]//Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Online: Association for Computational Linguistics, 2020: 6355-6365.

TAKANOBU R, LIANG R, HUANG M. Multi-agent task-oriented dialog policy learning with role-aware reward decomposition[C]//Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Online: Association for Computational Linguistics. 2020: 625-638.

DAI B, ZENG B, WEI P F, et al . The task dialogue strategy learning method of discriminative deep Dyna-Q was improved[J ] . Journal of Guangdong University of Technology, 2023, 40(4): 9-17.

ERIC M, GOEL R, PAUL S, et al . MultiWOZ 2.1: a consolidated multi-domain dialogue dataset with state corrections and state tracking baselines[C ] //Proceedings of the Twelfth Language Resources and Evaluation Conference. Marseille: European Language Resources Association, 2020: 422-428.

HOSSEINI-ASL E, MCCANN B, WU C S, et al . A simple language model for task-oriented dialogue[C ] //Advances in Neural Information Processing Systems (NeurIPS 2020) . Vancouver: Neural Information Processing Systems Foundation, 2020: 13.

YANG Y, LI Y, QUAN X. Ubar: Towards fully end-to-end task-oriented dialog system with gpt-2[J]. Proceedings of the AAAI Conference on Artificial Intelligence. 2021, 35(16): 14230-14238.

LEE Y. Improving end-to-end task-oriented dialog system with a simple auxiliary task[C]//Findings of the Association for Computational Linguistics: EMNLP 2021. Punta Cana: Association for Computational Linguistics, 2021: 1296-1303.

浏览量

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

改进判别式深度Dyna-Q的任务对话策略学习方法

基于深度强化学习的车联网频谱分配研究

基于强化学习的注塑工艺参数自动调优方法及应用

基于技能发现的元强化学习

基于多智能体注意力机制的自动巡检路线强化学习模型