YANG Xiangyan, YANG Shaobing, MENG Lingyu. A Reward Decoupling Approach Based on Hierarchical Reinforcement Learning for Integrated Energy System Scheduling[J]. 2025, 49(12): 5091-5100.
YANG Xiangyan, YANG Shaobing, MENG Lingyu. A Reward Decoupling Approach Based on Hierarchical Reinforcement Learning for Integrated Energy System Scheduling[J]. 2025, 49(12): 5091-5100. DOI: 10.13335/j.1000-3673.pst.2025.0643.
深度强化学习应用于综合能源系统(integrated energy system,IES)调度任务时,IES的多能耦合结构往往导致奖励函数由多分量耦合而成,而奖励的设计决定了智能体如何理解任务目标与动作价值,叠加过多的奖励项易使策略混乱,导致寻优效果不佳。对此,该文提出了一种基于分层强化学习的奖励解耦方法。首先根据IES经济调度中奖励函数的耦合机理提出电热机组的层级化决策方式,进而将优化调度问题表征为马尔可夫决策过程,并针对电力快速响应与热力惯性特征分别设计适用算法,最终形成分层强化学习模型以分离电力机组和热力机组的决策,从而实现奖励函数中电热分量的解耦。算例仿真结果表明,该模型能避免智能体跨能源动作奖励信号之间的互相干扰,在提高系统运行经济效益的同时能匹配电与热的不同调度时间尺度,验证了所提奖励解耦方法的有效性与优越性。
Abstract
When applying deep reinforcement learning to the scheduling tasks of integrated energy systems (IES)
the multi-energy coupling structure of IES often leads to reward functions composed of multiple coupled components. The design of reward functions fundamentally determines how agents perceive task objectives and action values; however
excessive reward terms can easily lead to policy confusion and suboptimal optimization. To address this
this paper proposes a reward decoupling method based on hierarchical reinforcement learning. Firstly
a hierarchical decision-making approach for electric-thermal units is developed according to the coupling mechanism of reward functions in IES economic scheduling. Subsequently
the optimal scheduling problem is formulated as a Markov decision process
with specifically designed algorithms for the rapid response characteristics of electrical systems and the thermal inertia features
respectively. Ultimately
a hierarchical reinforcement learning model is constructed to separate decision-making processes for electrical and thermal units
thereby achieving decoupling of electric-thermal components in the reward function. Case simulation results demonstrate that the proposed model effectively avoids mutual interference between cross-energy action reward signals
enhances operational economic benefits while aligning with different scheduling timescales for electricity and heat
verifying the effectiveness and superiority of the reward decoupling method.