基于技能发现的元强化学习

伍家威; 郝志峰

doi:10.12052/gdutxb.240041

欢迎来到能源电力期刊网！

English Version 期刊登录

您当前的位置：

首页 >

文章列表页 >

基于技能发现的元强化学习

计算机科学与技术 | 更新时间：2026-05-27

- 基于技能发现的元强化学习
- Meta Reinforcement Learning Based on Skill Discovery
- 广东工业大学学报 2025年42卷第3期页码：52-61
- 作者机构：
  
  1. 广东工业大学计算机学院,广东,广州,510006
  2. 汕头大学理学院,广东,汕头,515063
- 作者简介：
  
  伍家威(1998–)，男，硕士研究生，主要研究方向为机器学习、强化学习，E-mail：289873439@qq.com
  郝志峰(1968–)，男，教授，博士，主要研究方向为机器学习、数据挖掘，E-mail：zfhao@gdut.edu.cn
- 基金信息：
  
  科技创新2030——“新一代人工智能”重大项目(2021ZD0111501);国家优秀青年科学基金资助项目 (62122022);国家自然科学基金资助项目(61876043，61976052，62206064)
- DOI：10.12052/gdutxb.240041
  中图分类号： TP181
- 纸质出版：2025
- 稿件说明：
移动端阅览
伍家威, 郝志峰. 基于技能发现的元强化学习[J]. 广东工业大学学报, 2025, 42(3): 52–61.

Wu Jiawei, Hao Zhifeng. Meta Reinforcement Learning Based on Skill Discovery[J]. 2025, 42(3): 52-61.
伍家威, 郝志峰. 基于技能发现的元强化学习[J]. 广东工业大学学报, 2025, 42(3): 52–61. DOI： 10.12052/gdutxb.240041

Wu Jiawei, Hao Zhifeng. Meta Reinforcement Learning Based on Skill Discovery[J]. 2025, 42(3): 52-61. DOI： 10.12052/gdutxb.240041.

摘要

在机器人控制任务的复杂环境中，元强化学习(Meta Reinforcement Learning

Meta-RL) 作为关键组成部分，它可以利用先前的经验来解决未见过的、长期的以及奖励稀疏的复杂任务。基于技能的元强化学习(Skill-based Meta-RL) 方法旨在利用任务上下文提取出有用的技能，这些技能可以帮助智能体在元测试阶段快速适应新环境。然而现有的方法所学到的技能缺乏泛用性与适应性，从而限制了其在元测试任务集中的表现。为了克服这一限制，本文提出了一种基于技能发现的元强化学习(Skill Discovery Meta-RL

SDMRL) 方法，通过此方法可以在没有奖励函数的情况下学习到更加有用的技能。本文将目标形式化为利用最大熵策略最大化信息论目标，这个简单而有效的探索目标能让智能体从非结构化数据中无监督地学习到有用的技能和技能先验。在迷宫导航(Maze Navigation) 连续控制任务中的实验表明，SDMRL方法比以往的元强化学习方法更有效，学习到的技能可以解决长期的复杂稀疏奖励任务。

Abstract

In the realm of complex environments for robot control tasks

Meta Reinforcement Learning has emerged as a pivotal component

leveraging prior experiences to tackle unseen

long-term

and sparsely rewarded intricate tasks. Skill-based Meta-RL methods aim to extract useful skills from task contexts

aiding agents in swiftly adapting to new environments during meta-testing. However

the skills learned by existing methods lack generality and adaptability

limiting their performance across meta-testing task sets. To address this

a Skill Discovery Meta-RL (SDMRL) approach is proposed in this paper

by learning more useful skills in the absence of reward functions. The objective of the SDMRL is formalized as maximizing an information-theoretic objective using maximum entropy policies

enabling agents to learn valuable skills and skill priors from unstructured data in an unsupervised manner. Experimental results in continuous control tasks such as Maze Navigation demonstrate the effectiveness of the SDMRL approach over previous meta reinforcement learning methods

and the learned skills proficiently address the long-term complex sparse reward tasks.

关键词

Keywords

references

FINN C, ABBEEL P, LEVINE S. Model-agnostic meta-learning for fast adaptation of deep networks[C]//International Conference on Machine Learnin. Sydney: PMLR, 2017: 1126-1135.

RAKELLY K, ZHOU A, FINN C, et al . Efficient off-policy meta-reinforcement learning via probabilistic context variables[C ] //International Conference on Machine Learning. California: PMLR, 2019: 5331-5340.

ZHAO T Z, LUO J, SUSHKOV O, et al . Offline meta-reinforcement learning for industrial insertion[C ] //2022 International Conference on Robotics and Automation (ICRA) . Philadelphia: IEEE, 2022: 6386-6393.

BAUMLI K, WARDE-FARLEY D, HANSEN S, et al . Relative variational intrinsic control[C ] //Proceedings of the AAAI Conference on Artificial Intelligence. Online: AAAI, 2021, 35(8): 6732-6740.

PERTSCH K, LEE Y, LIM J. Accelerating reinforcement learning with learned skill priors[C]//Conference on Robot Learning. Online: PMLR, 2021: 188-204.

NAM T, SUN S H, PERTSCH K, et al . Skill-based meta-reinforcement learning[C ] //International Conference on Learning Representations. Vancouver: ICLR, 2022: 1-22.

GUO Y, WU Q, LEE H. Learning action translator for meta reinforcement learning on sparse-reward tasks[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2022, 36(6): 6792-6800.

ZHANG J, WANG J, HU H, et al . Metacure: meta reinforcement learning with empowerment-driven exploration[C ] //International Conference on Machine Learning. Online: PMLR, 2021: 12600-12610.

CHRISTIANO P F, LEIKE J, BROWN T, et al . Deep reinforcement learning from human preferences[J ] . Advances in Neural Information Processing Systems, 2017, 30: 4302-4310.

DYLAN H, SMITHA M, PIETER A, et al . Inverse reward design[C ] //Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach: Curran Associates, Inc., 2017: 6768-6777.

CAMPOS V, TROTT A, XIONG C M, et al . Explore, Discover and learn: unsupervised discovery of state-covering skills[C ] //International Conference on Machine Learning. Long Beach: 2019: 1317-1327.

LIN S, WAN J L, XU T Y, et al . Model-based offline meta-reinforcement learning with regularization[C ] //International Conference on Learning Representations. Kigali: ICLR, 2022: 1-22.

SIEGEL N Y, SPRINGENBERG J T, BERKENKAMP F, et al . Keep doing what worked: behavioral modelling priors for offline reinforcement learning[C ] //International Conference on Learning Representations. Online: ICLR, 2020: 1-21.

SEO Y, LEE K, CLAVERA GILABERTE I, et al . Trajectory-wise multiple choice learning for dynamics generalization in reinforcement learning[J ] . Advances in Neural Information Processing Systems, 2020, 33: 12968-12979.

DORFMAN R, SHENFELD I, TAMAR A. Offline meta reinforcement learning—identifiability challenges and effective data collection strategies[J]. Advances in Neural Information Processing Systems, 2021, 34: 4607-4618.

Mitchell E, Rafailov R, Peng X B, et al . Offline meta-reinforcement learning with advantage weighting[C ] //International Conference on Machine Learning. Online: PMLR, 2021: 7780-7791.

PONG V H, NAIR A V, SMITH L M, et al . Offline meta-reinforcement learning with online self-supervision[C ] //International Conference on Machine Learning. Baltimore: PMLR, 2022: 17811-17829.

LIU E Z, RAGHUNATHAN A, LIANG P, et al . Decoupling exploration and exploitation for meta-reinforcement learning without sacrifices[C ] //International Conference on Machine Learning. Online: PMLR, 2021: 6925-6935.

LEE K, SEO Y, LEE S, et al . Context-aware dynamics model for generalization in model-based reinforcement learning[C ] //International Conference on Machine Learning. Online: PMLR, 2020: 5757-5766.

ZHAO T Z, ANUSHA N, KATE R, et al . MELD - meta-reinforcement learning from images via latent state models[C ] //Conference on Robot Learning. Online: PMLR, 2020: 1246-1261.

PERTSCH K, LEE Y, WU Y, et al . Guided reinforcement learning with learned skills[C ] //Conference on Robot Learning. Auckland: PMLR, 2022: 729-739.

MEREL J, TUNYASUVUNAKOOL S, AHUJA A, et al . Catch carry: reusable neural controllers for vision-guided whole-body tasks[J ] . ACM Transactions on Graphics (TOG), 2020, 39(4): 39: 1-39: 12.

AJAY A , KUMAR A , AGRAWAL P , et al . OPAL: offline primitive discovery for accelerating offline reinforcement learning[C ] //Internationa l Conference on Learning Representations. Online: ICLR, . 2021: 1-23.

PARK S, CHOI J, KIM J, et al . Lipschitz-constrained unsupervised skill discovery[C ] //International Conference on Learning Representations. Online: ICLR, 2021: 1-24.

CHEBOTAR Y, HAUSMAN K, LU Y, et al . Actionable models: unsupervised offline reinforcement learning of robotic skills[C ] //38th International Conference on Machine Learning, Online: ICML, 2021: 1518-1528.

BARBER D, AGAKOV F. The im algorithm: a variational approach to information maximization[J]. Advances in Neural Information Processing Systems, 2004, 16(320): 201.

HAARNOJA T, ZHOU A, ABBEEL P, et al . Soft actor-critic: off-policy maximum entropy deep reinforcement learning with a stochastic actor[C ] //International Conference on Machine Learning. stockholm: PMLR, 2018: 1861-1870.

YUAN H, LU Z. Robust task representations for offline meta-reinforcement learning via contrastive learning[C]//International Conference on Machine Learning. Baltimore: PMLR, 2022: 25747-25759.

LUISA Z, KYRIACOS S, MAXIMILI AN I, et al . VariBAD: a very good method for bayes-adaptive deep RL via meta-learning[C ] //Proceedings of the Eighth International Conference on Learning Representations. Online: ICLR, 2020: 1-20.

TEH Y, BAPST V, CZARNECKI W M, et al . Distral: robust multitask reinforcement learning[J ] . Conference on Neural Information Processing Systems, 2017, 30: 4496-4506.

浏览量

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

基于深度强化学习的对话策略网络

基于深度强化学习的车联网频谱分配研究

基于强化学习的注塑工艺参数自动调优方法及应用

基于多智能体注意力机制的自动巡检路线强化学习模型

改进判别式深度Dyna-Q的任务对话策略学习方法