Energy Management Strategy for Microgrid Cluster Based on Improved Double Deep Q-Network

HE Jintao; WANG Can; WANG Mingchao; CHENG Bentao; LIU Yuzheng; CHANG Wenhan; WANG Rui; YU Han

doi:10.11930/j.issn.1004-9649.202503014

English Version Reader LoginJournal Login

您当前的位置：

首页 >

文章列表页 >

Energy Management Strategy for Microgrid Cluster Based on Improved Double Deep Q-Network

更新时间：2025-12-08

- Energy Management Strategy for Microgrid Cluster Based on Improved Double Deep Q-Network
- Vol. 58, Issue 10, Pages: 14-26(2025)
- 作者机构：
  
  1. 三峡大学电气与新能源学院,湖北,宜昌,443002
  2. 武汉长海高新技术有限公司,湖北,武汉,430223
  3. 湖北华中电力科技开发有限责任公司,湖北,武汉,430077
- 作者简介：
- 基金信息：
- DOI：10.11930/j.issn.1004-9649.202503014
  CLC： TM73
- Published Online：23 October 2025，
  
  Published：2025
- 稿件说明：
移动端阅览
何锦涛, 王灿, 王明超, 程本涛, 刘于正, 常文涵, 王锐, 余涵. 基于改进双深度Q网络的微电网群能量管理策略[J]. 中国电力, 2025, 58(10): 14-26.

HE Jintao, WANG Can, WANG Mingchao, et al. Energy Management Strategy for Microgrid Cluster Based on Improved Double Deep Q-Network[J]. 2025, 58(10): 14-26.
何锦涛, 王灿, 王明超, 程本涛, 刘于正, 常文涵, 王锐, 余涵. 基于改进双深度Q网络的微电网群能量管理策略[J]. 中国电力, 2025, 58(10): 14-26. DOI： 10.11930/j.issn.1004-9649.202503014.

HE Jintao, WANG Can, WANG Mingchao, et al. Energy Management Strategy for Microgrid Cluster Based on Improved Double Deep Q-Network[J]. 2025, 58(10): 14-26. DOI： 10.11930/j.issn.1004-9649.202503014.

摘要

针对传统微电网群能量管理方法存在的高估偏差与决策精度不足问题，提出一种基于改进双深度Q网络的能量管理策略。首先，构建基于裁剪双Q值思想的双目标价值网络框架，通过并行计算双价值网络的时序差分（temporal difference，TD）目标值并裁剪高TD目标值，抑制价值函数的高估偏差，提高决策精度。然后，采用动态贪婪策略，基于当前状态计算所有可能动作的值函数，避免频繁选择最大Q值动作，使智能体充分探索动作以防止过早收敛。最后，以包含3个子微网的微电网群进行算例验证。仿真结果表明，相较于基于模型预测控制和传统双深度Q网络的能量管理策略，本文所提方法具有更好的寻优效果和收敛性，同时将系统运行成本分别降低了44.62%和26.39%。

Abstract

To address the overestimation bias and poor decision accuracy of conventional microgrid cluster energy management methods

an energy management strategy based on improved double deep Q-network is proposed. Firstly

this study constructed a dual-objective value network framework based on clipped double Q-learning

which enhances decision-making precision by suppressing value overestimation bias through parallel computation of temporal difference (TD) targets for dual value networks and clipping high TD target values. And then

a dynamic greedy strategy was adopted to calculate the value function of all possible actions based on the current state

avoiding persistent exploitation of the greedy actions to ensure sufficient exploration and prevent premature convergence of the agent. Finally

a case study of a microgrid cluster with three sub-microgrids was conducted for verification. The simulation results show that compared to the energy management strategies based on model predictive control and conventional double deep Q-network

the proposed method achieves superior optimization performance and convergence characteristics

while reducing system operating costs by 44.62% and 26.39% respectively.

关键词

Keywords

references

1王灿, 张雪菲, 凌凯, 等. 基于区间概率不确定集的微电网两阶段自适应鲁棒优化调度[J]. 中国电机工程学报, 2024, 44 (5): 1750- 1764.

WANG Can, ZHANG Xuefei, LING Kai, et al. Two-stage adaptive robust optimal scheduling based on the interval probability uncertainty set for microgrids[J]. Proceedings of the CSEE, 2024, 44 (5): 1750- 1764.

2刘任, 刘洋, 许立雄, 等. 计及分布式需求响应的多微电网系统协同优化策略[J]. 电力建设, 2023, 44 (5): 72- 83.

LIU Ren, LIU Yang, XU Lixiong, et al. Multi-microgrid system collaborative optimization strategy considering distributed demand response[J]. Electric Power Construction, 2023, 44 (5): 72- 83.

3易文飞, 朱卫平, 郑明忠. 计及数据中心和风电不确定性的微电网经济调度[J]. 中国电力, 2024, 57 (2): 19- 26.

YI Wenfei, ZHU Weiping, ZHENG Mingzhong. Economic dispatch of microgrid considering data center and wind power uncertainty[J]. Electric Power, 2024, 57 (2): 19- 26.

4谭玲玲, 汤伟, 楚冬青, 等. 考虑电-氢一体化的微电网低碳-经济协同优化调度[J]. 中国电力, 2024, 57 (5): 137- 148.

TAN Lingling, TANG Wei, CHU Dongqing, et al. Low-carbon-economic collaborative optimal dispatching of microgrid considering electricity-hydrogen integration[J]. Electric Power, 2024, 57 (5): 137- 148.

5樊晓伟, 王瑞妙, 杨海峰, 等. 计及源荷不确定的综合能源微电网集群优化运行[J]. 电力建设, 2024, 45 (8): 128- 139.

FAN Xiaowei, WANG Ruimiao, YANG Haifeng, et al. Optimization operation of integrated energy microgrid cluster considering source-load uncertainty[J]. Electric Power Construction, 2024, 45 (8): 128- 139.

6凌凯, 王灿, 张高瑞, 等. 考虑路径损耗的热电联供型微网三层能量优化策略[J]. 中国电力, 2023, 56 (2): 102- 113.

LING Kai, WANG Can, ZHANG Gaorui, et al. Three-layer energy optimization strategy for CHP microgrid considering path loss[J]. Electric Power, 2023, 56 (2): 102- 113.

7姚文亮, 王成福, 赵雨菲, 等. 不确定性环境下基于合作博弈的综合能源系统分布式优化[J]. 电力系统自动化, 2022, 46 (20): 43- 53.

YAO Wenliang, WANG Chengfu, ZHAO Yufei, et al. Distributed optimization of integrated energy system based on cooperative game in uncertain environment[J]. Automation of Electric Power Systems, 2022, 46 (20): 43- 53.

8李建标, 陈建福, 高滢, 等. 基于RG-DDPG的直流微网能量管理策略[J]. 中国电力, 2023, 56 (7): 85- 94.

LI Jianbiao, CHEN Jianfu, GAO Ying, et al. Strategy for DC microgrid energy management based on RG-DDPG[J]. Electric Power, 2023, 56 (7): 85- 94.

9王勇, 吕华灿, 姚文亮, 等. 基于消费者耦合行为的综合需求响应建模与主从博弈运行策略研究[J]. 电网技术, 2024, 48 (7): 2873- 2883.

WANG Yong, LÜ Huacan, YAO Wenliang, et al. Integrated demand response modeling and master-slave game operation strategy based on consumer coupling behavior[J]. Power System Technology, 2024, 48 (7): 2873- 2883.

10肖白, 韩康琦, 张晓华. 含氢储能的独立微电网IGDT鲁棒规划[J]. 电力建设, 2024, 45 (4): 77- 88.

XIAO Bai, HAN Kangqi, ZHANG Xiaohua. Robust IGDT planning for stand-alone microgrid with hydrogen energy storage[J]. Electric Power Construction, 2024, 45 (4): 77- 88.

11AASLID P, KORPÅS M, BELSNES M M, et al. Stochastic optimization of microgrid operation with renewable generation and energy storages[J]. IEEE Transactions on Sustainable Energy, 2022, 13 (3): 1481- 1491. DOI

12WANG P B, TAN L L, ZHANG X C, et al. An energy management method for a microgrid group considering uncertainty models[C]//2019 22nd International Conference on Electrical Machines and Systems (ICEMS). Harbin, China. IEEE, 2019: 1–6.

13李扬, 马文捷, 卜凡金, 等. 多智能体深度强化学习驱动的跨园区能源交互优化调度[J]. 电力建设, 2024, 45 (5): 59- 70. DOI

LI Yang, MA Wenjie, BU Fanjin, et al. Deep reinforcement learning-driven cross-community energy interaction optimal scheduling[J]. Electric Power Construction, 2024, 45 (5): 59- 70. DOI

14张宏涛, 吴怡之, 邓开连, 等. 一种基于强化学习的微电网能量管理算法[J]. 物联网技术, 2022, 12 (12): 74- 78.

15YANG Y H, MA T F, LI H T, et al. Federated double DQN based multi-energy microgrid energy management strategy considering carbon emissions[J]. Global Energy Interconnection, 2023, 6 (6): 689- 699. DOI

16LIU D, ZANG C Z, ZENG P, et al. Deep reinforcement learning for real-time economic energy management of microgrid system considering uncertainties[J]. Frontiers in Energy Research, 2023, 11, 1163053. DOI

17FAN L Q, ZHANG J, HE Y, et al. Optimal scheduling of microgrid based on deep deterministic policy gradient and transfer learning[J]. Energies, 2021, 14 (3): 584. DOI

18雷嘉明, 姜爱华, 吴新飞, 等. 计及源荷不确定性的综合能源系统近端策略优化调度[J]. 电力科学与技术学报, 2023, 38 (5): 1- 11.

LEI Jiaming, JIANG Aihua, WU Xinfei, et al. Proximal policy optimization dispatch of integrated energy system considering source-load uncertainty[J]. Journal of Electric Power Science and Technology, 2023, 38 (5): 1- 11.

19刘向杰, 刘梓安, 孔小兵, 等. 基于深度Q网络的风光柴储微电网能量管理策略[J]. 控制工程, 2023, 30 (8): 1538- 1547.

LIU Xiangjie, LIU Zian, KONG Xiaobing, et al. Energy management strategy of wind-PV-diesel-battery microgrid based on deep Q-network[J]. Control Engineering of China, 2023, 30 (8): 1538- 1547.

20薛溟枫, 毛晓波, 肖浩, 等. 基于改进深度Q网络算法的多园区综合能源系统能量管理方法[J]. 电力建设, 2022, 43 (12): 83- 93.

XUE Mingfeng, MAO Xiaobo, XIAO Hao, et al. A novel energy management method based on modified deep Q network algorithm for multi-park integrated energy system[J]. Electric Power Construction, 2022, 43 (12): 83- 93.

21XIAO H, PU X W, PEI W, et al. A novel energy management method for networked multi-energy microgrids based on improved DQN[J]. IEEE Transactions on Smart Grid, 2023, 14 (6): 4912- 4926. DOI

22XIAO H, YANG Y H, ZHANG S, et al. Stochastic game based microgrid clusters energy management modeling and strategy[C]//2023 3rd Power System and Green Energy Conference (PSGEC). Shanghai, China. IEEE, 2023: 250–254.

23张冲标, 钱辰雯, 俞红燕, 等. 基于ADMM的多场景县域多微电网交互运行策略[J]. 中国电力, 2024, 57 (2): 9- 18.

ZHANG Chongbiao, QIAN Chenwen, YU Hongyan, et al. Interactive operation strategy for multi-scenario county-level multi-microgrid based on ADMM[J]. Electric Power, 2024, 57 (2): 9- 18.

24张晓佳, 王灿, 张佳恒, 等. 基于多能需求响应与改进BiLSTM的综合能源系统负荷预测[J]. 电力建设, 2025, 46 (4): 113- 125.

ZHANG Xiaojia, WANG Can, ZHANG Jiaheng, et al. Integrated energy system load forecasting based on multi-energy demand response and improved BiLSTM[J]. Electric Power Construction, 2025, 46 (4): 113- 125.

25崔永玲, 王成福, 牛远方, 等. 考虑综合需求响应不确定性的综合能源系统两阶段随机优化决策[J]. 电网技术, 2025, 49 (6): 2232- 2242.

CUI Yongling, WANG Chengfu, NIU Yuanfang, et al. Two-stage stochastic optimization decision of integrated energy system considering the uncertainty of integrated demand response[J]. Power System Technology, 2025, 49 (6): 2232- 2242.

26王灿, 张羽, 田福银, 等. 基于双向主从博弈的储能电站与综合能源系统经济运行策略[J]. 电工技术学报, 2023, 38 (13): 3436- 3446, 3472.

WANG Can, ZHANG Yu, TIAN Fuyin, et al. Economic operation of energy storage power stations and integrated energy systems based on bidirectional master-slave game[J]. Transactions of China Electrotechnical Society, 2023, 38 (13): 3436- 3446, 3472.

27姜智霖, 郝峰杰, 袁志昌, 等. 考虑SOC优化设定的电-氢混合储能系统的运行优化[J]. 电力系统保护与控制, 2024, 52 (8): 65- 76.

JIANG Zhilin, HAO Fengjie, YUAN Zhichang, et al. Optimal operation of an electro-hydrogen hybrid energy storage system considering SOC optimization setting[J]. Power System Protection and Control, 2024, 52 (8): 65- 76.

28万玲玲, 陈中, 王毅, 等. 考虑能量时空转移的城市规模化共享电动汽车充放电优化调度[J]. 电力建设, 2023, 44 (6): 135- 143. DOI

WAN Lingling, CHEN Zhong, WANG Yi, et al. Optimal charging and discharging scheduling of urban large-scale shared electric vehicles considering energy temporal and spatial transfer[J]. Electric Power Construction, 2023, 44 (6): 135- 143. DOI

29吕金玲, 王小君, 窦嘉铭, 等. 考虑运行状态信息的综合能源系统图强化学习优化调度[J]. 电力系统保护与控制, 2024, 52 (2): 1- 14.

LÜ Jinling, WANG Xiaojun, DOU Jiaming, et al. Optimal dispatch of an integrated energy system based on graph reinforcement learning considering operation state information[J]. Power System Protection and Control, 2024, 52 (2): 1- 14.

30谢敬东, 徐振邦. 基于三方博弈的多能交互微网群系统双层优化模型[J]. 太阳能学报, 2024, 45 (2): 384- 394.

XIE Jingdong, XU Zhenbang. Two-layer optimization model for multi-energy interactive microgrid cluster system based on tripartite game[J]. Acta Energiae Solaris Sinica, 2024, 45 (2): 384- 394.

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Strategy for DC Microgrid Energy Management Based on RG-DDPG

Multi-agent Peak Shaving and Valley Filling Strategy Considering the Flexibility of Electric-Thermal System with Optical Storage

An Energy Management Optimization Strategy for Regional Power Grid Energy Storage System Based on Improved Artificial Bee Colony Algorithm

An Energy Management Algorithm of PV-Assisted Smart Building Based on Offline Optimization and Online Decision

The Simulative Research on Energy Management Strategy for Microgrid Structure

Related Author

Jianfu CHEN

Jianbiao LI

Jie ZENG

Shaoxiong ZHOU

Zikai LU

Hongyuan WU

Xingyu PEI

Ying GAO

Related Institution

DC Power Distribution and Consumption Technology Research Centre of Guangdong Power Grid Co., Ltd.

China Southern Power Grid Technology Co.,Ltd.

Qingke Youneng (Shenzhen) Technology Co., Ltd.

State Grid Beijing Electric Power Research Institute

State Grid (Suzhou) City and Energy Research Institute Co., Ltd.

AI问答

⁰