Accurately forecasting sky imagery and its impact on solar power generation is essential for intelligent grid scheduling and the efficient management of renewable energy resources. However
capturing the dynamic evolution of cloud patterns and their complex
nonlinear relationship with photovoltaic (PV) power output remains a formidable challenge. To tackle this problem
we propose a multimodal deep learning framework that integrates Transformer and Mamba architectures
effectively leveraging both spatiotemporal features from sky images and infrared spectral information to improve forecasting precision. Within this framework
the Transformer is responsible for modeling the temporal dynamics of cloud motion
while Mamba focuses on extracting salient features from infrared spectra. A cross-modal feature fusion strategy is employed to learn the intricate nonlinear mapping between sky imagery and power output. We evaluate our model on the Stanford sky image and PV generation datasets
conducting extensive experiments that demonstrate superior performance in both sky image forecasting and PV power prediction tasks. Ablation studies reveal that the full model—incorporating infrared data
temporal modeling
and multi-scale feature extraction—achieves an RMSE of 13.04kW and an SSIM of 0.94653 for image prediction. Moreover
the proposed method achieves an MAE of 1.50 kW
approximately 4.5% lower than that of the second-best Tau model.Under typical cloudy weather conditions
the model is able to effectively track the trend of photovoltaic power fluctuations caused by cloud occlusion.
Zhong Y J, Wu Y K.Short-term solar power forecasts considering various weather variables[C]//2020 International Symposium on Computer, Consumer and Control (IS3C). Taichung City, Taiwan, China, 2020: 432-435.
刘兴冉, 张宏晔, 闫海明, 等. 区域复杂地形晴日太阳辐射估算研究[J]. 太阳能学报, 2022, 43(8): 174-180. Liu X R, Zhang H Y, Yan H M, et al. Estimation of regional solar radiation on clear days over complex terrain[J]. Acta Energiae Solaris Sinica, 2022, 43(8): 174-180.
徐姝影, 张哲, 袁佳伟, 等. 考虑光伏跟踪装置的混合可再生能源系统优化与评价[J]. 太阳能学报, 2025, 46(3): 123-131. Xu S Y, Zhang Z, Yuan J W, et al. Optimization and evaluation of hybrid renewable energy systems considering PV tracking devices[J]. Acta Energiae Solaris Sinica, 2025, 46(3): 123-131.
Vaswani A, Shazeer N, Parmar N, et al.Attention is all you need[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems. 2017: 6000-6010.
Jing T, Chen S L, Navarro-Alarcon D, et al.SolarFusionNet: enhanced solar irradiance forecasting via automated multi-modal feature selection and cross-modal fusion[J]. IEEE Transactions on Sustainable Energy, 2025, 16(2): 761-773.
Wei X S, Yue D, Hancke G P, et al.Ultra short-term solar irradiance forecast based on multimodal data fusion and fuzzification[J]. IEEE Transactions on Industrial Informatics, 2025, 21(4): 3256-3265.
Zhao X H, Fan H Y, Guo S, et al.Sky image-informed hybrid model for ultra-short-term prediction of photovoltaic power output[C]//4th Energy Conversion and Economics Annual Forum (ECE Forum 2024). Beijing, China, 2025: 375-379.
Meddahi A, Tuomiranta A, Guillon S.Skill-driven data sampling and deep learning framework for minute-scale solar forecasting with sky images[J]. Solar RRL, 2025, 9(4): 2400664.
王东风, 李青博, 张博洋, 等. 考虑季节性与趋势特征的光伏功率预测模型研究[J]. 太阳能学报, 2025, 46(3): 348-356. Wang D F, Li Q B, Zhang B Y, et al. Research on photovoltaic power prediction model considering seasonality and trend characteristics[J]. Acta Energiae Solaris Sinica, 2025, 46(3): 348-356.
Li H, Ma G, Wang B, et al.Multi-modal feature fusion model based on TimesNet and T2T-ViT for ultra-short-term solar irradiance forecasting[J]. Renewable Energy, 2025, 240: 122192.
Hategan S M, Stefu N, Petreus D, et al.Short-term forecasting of PV power based on aggregated machine learning and sky imagery approaches[J]. Energy, 2025, 316: 134595.
Nie Y H, Li X T, Scott A, et al.SKIPP’D: a SKy Images and Photovoltaic Power Generation Dataset for short-term solar forecasting[J]. Solar Energy, 2023, 255: 171-179.
Tan C, Gao Z Y, Wu L R, et al.Temporal attention unit: towards efficient spatiotemporal predictive learning[C]//2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Vancouver, BC, Canada, 2023: 18770-18782.
Yu G Z, Lu L, Tang B, et al.Ultra-short-term wind power subsection forecasting method based on extreme weather[J]. IEEE Transactions on Power Systems, 2023, 38(6): 5045-5056.
Yao T C, Wang J, Wu H Y, et al.A photovoltaic power output dataset: Multi-source photovoltaic power output dataset with Python toolkit[J]. Solar Energy, 2021, 230: 122-130.
Li Y Y, Song L D, Zhang S, et al.A TCN-based hybrid forecasting framework for hours-ahead utility-scale PV forecasting[J]. IEEE Transactions on Smart Grid, 2023, 14(5): 4073-4085.
LI Q Q, XU Y, CHEW B S H, et al. An integrated missing-data tolerant model for probabilistic PV power generation forecasting[J]. IEEE Transactions on Power Systems, 2022, 37(6): 4447-4459.