Hierarchical Robust Reinforcement Learning Method for User-side Integrated Energy System Considering Flexibility Resources

XU Xuesong; ZENG Ziyang; HE Yuanxing; JIANG Hongtao; GUAN Min; XU Kai

doi:10.7500/AEPS20250626004

English Version Journal Login

您当前的位置：

首页 >

文章列表页 >

Hierarchical Robust Reinforcement Learning Method for User-side Integrated Energy System Considering Flexibility Resources

Basic Research | 更新时间：2026-10-10

- Hierarchical Robust Reinforcement Learning Method for User-side Integrated Energy System Considering Flexibility Resources
- Automation of Electric Power Systems Vol. 50, Issue 19, Pages: 189-198(2026)
- 作者机构：
  
  1.湖南工商大学智能工程与智能制造学院，湖南省长沙市 410205
  2.湘江实验室，湖南省长沙市 410205
  3.国网邵阳供电公司，湖南省邵阳市 422000
  4.多能源系统智慧互联技术湖南省重点实验室，湖南省长沙市 410073
- 作者简介：
- 基金信息：
  
  National Natural Science Foundation of China(72472049)
- DOI：10.7500/AEPS20250626004
  CLC：
- Received：26 June 2025，
  
  Revised：2026-02-26，
  
  Online First：17 August 2026，
  
  Published：10 October 2026
- 稿件说明：
移动端阅览
徐雪松,曾子洋,何远兴等.考虑灵活性资源的用户侧综合能源系统分层鲁棒强化学习方法[J].电力系统自动化,2026,50(19):189-198.

XU Xuesong,ZENG Ziyang,HE Yuanxing,et al.Hierarchical Robust Reinforcement Learning Method for User-side Integrated Energy System Considering Flexibility Resources[J].Automation of Electric Power Systems,2026,50(19):189-198.
徐雪松,曾子洋,何远兴等.考虑灵活性资源的用户侧综合能源系统分层鲁棒强化学习方法[J].电力系统自动化,2026,50(19):189-198. DOI： 10.7500/AEPS20250626004.

XU Xuesong,ZENG Ziyang,HE Yuanxing,et al.Hierarchical Robust Reinforcement Learning Method for User-side Integrated Energy System Considering Flexibility Resources[J].Automation of Electric Power Systems,2026,50(19):189-198. DOI： 10.7500/AEPS20250626004.

摘要

为解决综合能源系统中具有可调节能力的中小型灵活性资源未被有效利用的问题，文中构建用户侧综合能源系统(UIES)模型，并针对灵活性资源利用过程中可能引发的UIES求解不确定性问题，提出了一种分层鲁棒强化学习方法以提高求解结果的鲁棒性。首先，在模型搭建过程中为可调节设备预留一定灵活备用空间，以满足不同时段可能出现的调节需求。然后，为处理UIES模型中的不确定性，设计一种分层鲁棒强化学习方法。该方法上层为强化学习模型，旨在处理影响系统未来状态的长期动作；下层为鲁棒优化模型，专注于求解只影响系统即时运行的短期动作。将强化学习与鲁棒优化相结合，简化了智能体决策过程中的复杂度，同时充分考虑了求解结果的鲁棒性。算例分析表明，所提方法可有效调动用户侧灵活性资源，促使系统经济、低碳及稳定运行。

Abstract

To address the issue that small- and medium-sized flexibility resources with regulable capacity in integrated energy systems are not effectively utilized

this paper establishes a user-side integrated energy system (UIES) model. Aiming at the uncertainty in UIES solution that may arise during the utilization of flexibility resources

a hierarchical robust reinforcement learning method is proposed to enhance the robustness of solution results. First

certain flexible reserve margins are reserved for regulable devices during model construction to satisfy the regulation requirements that may occur during different time periods. Then

a hierarchical robust reinforcement learning method is designed to handle the uncertainties in the UIES model. In this method

the upper layer is a reinforcement learning model

which is intended to handle long‑term actions affecting the future states of the system. The lower layer is a robust optimization model

which focuses on solving short-term actions that only affect the real-time operation of the system. The combination of reinforcement learning and robust optimization simplifies the complexity of the agent's decision-making process

while fully taking into account the robustness of solution outcomes. Case studies demonstrate that the proposed method can effectively mobilize user-side flexibility resources and facilitate the economic

low-carbon and stable operation of the system.

关键词

Keywords

references

吴林林，陈璨，胡俊杰，等 . 支撑新能源电力系统灵活性需求的用户侧资源应用与关键技术［J］. 电网技术， 2024 ， 48 （ 4 ）： 1435 - 1444 .

Wu Linlin ， Chen Can ， Hu Junjie ， et al . User side resource application and key technologies for flexibility demand of renewable energy power system ［J］. Power System Technology ， 2024 ， 48 （ 4 ）： 1435 - 1444 .

尚文强，李广磊，丁月明，等 . 考虑源荷不确定性和新能源消纳的综合能源系统协同调度方法［J］. 电网技术， 2024 ， 48 （ 2 ）： 517 - 526 .

Shang Wenqiang ， Li Guanglei ， Ding Yueming ， et al . Collaborative scheduling for integrated energy system considering uncertainty of source load and absorption of new energy ［J］. Power System Technology ， 2024 ， 48 （ 2 ）： 517 - 526 .

崔全胜，白晓民，董伟杰，等 . 用户侧综合能源系统规划运行联合优化［J］. 中国电机工程学报， 2019 ， 39 （ 17 ）： 4967 - 4981 .

Cui Quansheng ， Bai Xiaomin ， Dong Weijie ， et al . Joint optimization of planning and operation in user-side multi-energy systems ［J］. Proceedings of the CSEE ， 2019 ， 39 （ 17 ）： 4967 - 4981 .

张沈习，王丹阳，程浩忠，等 . 双碳目标下低碳综合能源系统规划关键技术及挑战［J］. 电力系统自动化， 2022 ， 46 （ 8 ）： 189 - 207 .

Zhang Shenxi ， Wang Danyang ， Cheng Haozhong ， et al . Key technologies and challenges of low-carbon integrated energy system planning for carbon emission peak and carbon neutrality ［J］. Automation of Electric Power Systems ， 2022 ， 46 （ 8 ）： 189 - 207 .

汤翔鹰，胡炎，耿琪，等 . 考虑多能灵活性的综合能源系统多时间尺度优化调度［J］. 电力系统自动化， 2021 ， 45 （ 4 ）： 81 - 90 .

Tang Xiangying ， Hu Yan ， Geng Qi ， et al . Multi-time-scale optimal scheduling of integrated energy system considering multi-energy flexibility ［J］. Automation of Electric Power Systems ， 2021 ， 45 （ 4 ）： 81 - 90 .

童宇轩，胡俊杰，刘雪涛，等 . 新能源电力系统灵活性供需量化及分布鲁棒优化调度［J］. 电力系统自动化， 2023 ， 47 （ 15 ）： 80 - 90 .

Tong Yuxuan ， Hu Junjie ， Liu Xuetao ， et al . Quantification of flexibility supply and demand and distributionally robust optimal dispatch of renewable energy dominated power systems ［J］. Automation of Electric Power Systems ， 2023 ， 47 （ 15 ）： 80 - 90 .

Maitanova N ， SchlüTers S ， Hanke B ， et al . An analytical method for quantifying the flexibility potential of decentralised energy systems ［J］. Applied Energy ， 2024 ， 364 ： 123150 .

熊展，王素，王玲玲，等 . 考虑多能互补灵活性和用户低碳意愿的区域综合能源系统鲁棒优化调度［J］. 电网技术， 2024 ， 48 （ 7 ）： 2691 - 2701 .

Xiong Zhan ， Wang Su ， Wang Lingling ， et al . Robust optimal scheduling of regional integrated energy system considering multi-energy flexibility complementary and users’ low-carbon willingness ［J］. Power System Technology ， 2024 ， 48 （ 7 ）： 2691 - 2701 .

俞鸿飞，王韵楚，吕瑞扬，等 . 考虑灵活爬坡产品的虚拟电厂两阶段分布鲁棒优化运营策略［J］. 电力系统自动化， 2024 ， 48 （ 14 ）： 16 - 27 .

Yu Hongfei ， Wang Yunchu ， Lyu Ruiyang ， et al . Two-stage distributionally robust optimization operation strategy of virtual power plants considering flexible ramping products ［J］. Automation of Electric Power Systems ， 2024 ， 48 （ 14 ）： 16 - 27 .

Tan J ， Wu Q W ， Zhang M L . Strategic investment for district heating systems participating in energy and reserve markets using heat flexibility ［J］. International Journal of Electrical Power & Energy Systems ， 2022 ， 137 ： 107819 .

赵明欣，丁保迪，吴一恒，等 . 基于保守度自适应优化的综合能源系统鲁棒灵活性评估［J］. 电网技术， 2025 ， 49 （ 2 ）： 582 - 592 .

Zhao Mingxin ， Ding Baodi ， Wu Yiheng ， et al . Robust flexibility evaluation of integrated energy systems based on conservation adaptive optimization ［J］. Power System Technology ， 2025 ， 49 （ 2 ）： 582 - 592 .

周特，薛云飞，季节，等 . 考虑电力间接碳排放不确定性的电-冷-热综合能源系统两阶段鲁棒优化方法［J］. 电网技术， 2024 ， 48 （ 1 ）： 50 - 60 .

Zhou Te ， Xue Yunfei ， Ji Jie ， et al . Two-stage robust optimization for electricity-cooling-heat integrated energy system considering uncertainty of indirect carbon emissions of electricity ［J］. Power System Technology ， 2024 ， 48 （ 1 ）： 50 - 60 .

彭春华，娄钰杰，范国柱，等 . 基于节点碳势响应的新型电力系统鲁棒优化调度［J］. 电网技术， 2025 ， 49 （ 9 ）： 3677 - 3687 .

Peng Chunhua ， Lou Yujie ， Fan Guozhu ， et al . Robust optimal dispatch of new power systems based on node carbon intensity response ［J］. Power System Technology ， 2025 ， 49 （ 9 ）： 3677 - 3687 .

王灿，张雪菲，凌凯，等 . 基于区间概率不确定集的微电网两阶段自适应鲁棒优化调度［J］. 中国电机工程学报， 2024 ， 44 （ 5 ）： 1750 - 1763 .

Wang Can ， Zhang Xuefei ， Ling Kai ， et al . Two-stage adaptive robust optimal scheduling based on the interval probability uncertainty set for microgrids ［J］. Proceedings of the CSEE ， 2024 ， 44 （ 5 ）： 1750 - 1763 .

Hu R ， Zhou K L ， Yin H . Reinforcement learning model for incentive-based integrated demand response considering demand-side coupling ［J］. Energy ， 2024 ， 308 ： 132997 .

王磊，胡国，吴海，等 . 基于分层深度强化学习的分布式能源系统多能协同优化方法［J］. 电力系统自动化， 2024 ， 48 （ 1 ）： 67 - 76 .

Wang Lei ， Hu Guo ， Wu Hai ， et al . Multi-energy collaborative optimization method for distributed energy systems based on hierarchical deep reinforcement learning ［J］. Automation of Electric Power Systems ， 2024 ， 48 （ 1 ）： 67 - 76 .

Xu X S ， Xu K ， Zeng Z Y ， et al . Collaborative optimization of multi-energy multi-microgrid system： a hierarchical trust-region multi-agent reinforcement learning approach ［J］. Applied Energy ， 2024 ， 375 ： 123923 .

董雷，杨子民，乔骥，等 . 基于分层约束强化学习的综合能源多微网系统优化调度［J］. 电工技术学报， 2024 ， 39 （ 5 ）： 1436 - 1453 .

Dong Lei ， Yang Zimin ， Qiao Ji ， et al . Optimal scheduling of integrated energy multi-microgrid system based on hierarchical constraint reinforcement learning ［J］. Transactions of China Electrotechnical Society ， 2024 ， 39 （ 5 ）： 1436 - 1453 .

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Collaborative Robust Planning of Electric and Thermal Flexibility Resources Considering Reserve Optimization

Lightweight Large Model-driven Human-Computer Interactive Regulation Method for Virtual Power Plants

Coordinated Optimization Strategy for Bidding and Resource Scheduling of Industrial Park in Spot Market

Quantitative Evaluation Method for Time-series-Probabilistic Power and Energy Balance Oriented to High-dimensional Balance Region

Mechanism and Clearing Model of Regulatory Capacity Market for New Power System

Related Author

GAO Xueqian

LIU Chang

LIU Wenxia

ZHANG Bohan

YI Zhongkai

XU Ying

TU Zhenghong

WU Yuhang

Related Institution

State Key Laboratory of Alternate Electrical Power System with Renewable Energy Sources (North China Electric Power University)

Chengxi Power Supply Branch of State Grid Tianjin Electric Power Company

School of Electrical Engineering and Automation, Harbin Institute of Technology

School of Electric Power Engineering, South China University of Technology

Guangdong Key Laboratory of Clean Energy Technology (South China University of Technology)

AI问答

⁰