An intelligent power inspection defect classification method based on bidirectional cross-modal attention and adaptive image-text fusion

ZHOU Yuhao; YAN Yunfeng; GU Yanghao; QI Donglian

doi:10.19783/j.cnki.pspc.260086

English Version Journal Login

您当前的位置：

首页 >

文章列表页 >

An intelligent power inspection defect classification method based on bidirectional cross-modal attention and adaptive image-text fusion

更新时间：2026-09-28

- An intelligent power inspection defect classification method based on bidirectional cross-modal attention and adaptive image-text fusion
- Power System Protection and Control Vol. 54, Issue 19, (2026)
- 作者机构：
  
  浙江大学电气工程学院,浙江,杭州,310027
- 作者简介：
- 基金信息：
- DOI：10.19783/j.cnki.pspc.260086
  CLC：
- Published：2026
- 稿件说明：
移动端阅览
ZHOU Yuhao, YAN Yunfeng, GU Yanghao, et al. An intelligent power inspection defect classification method based on bidirectional cross-modal attention and adaptive image-text fusion[J]. Power System Protection and Control, 2026, 54(19).
DOI：

ZHOU Yuhao, YAN Yunfeng, GU Yanghao, et al. An intelligent power inspection defect classification method based on bidirectional cross-modal attention and adaptive image-text fusion[J]. Power System Protection and Control, 2026, 54(19). DOI： 10.19783/j.cnki.pspc.260086.

摘要

针对电力设备智能巡检中异构信息整合困难、语义对齐不精与模态缺失下表现不稳的挑战，提出一种基于双向跨模态注意力机制的自适应图文融合缺陷分类框架。该框架采用视觉Transformer(vision Transformer

ViT)与基于变换器的双向编码器表示(bidirectional encoder representation from transformers

BERT)分别构建高维视觉特征与深层语义表征，并引入基于对比学习的损失，在统一语义空间内实现图文嵌入的显式对齐。设计双向跨模态注意力机制，通过“文本→图像”与“图像→文本”的交互，自适应强化缺陷相关特征、抑制环境干扰。同时构建多分支分类策略，确保模型在单模态缺失等复杂工况下的鲁棒性。实验表明，在电力智能巡检场景12种典型缺陷任务上，该方法准确率达94.5%，F1值达93.9%。与现有主流单模态及多模态融合方法相比，所提方法在特征可分性与分类精度上均有显著提升，为电力巡检多模态处理提供了有效支撑。

Abstract

To address the challenges in intelligent power equipment inspection

including heterogeneous information integration

imprecise semantic alignment

and unstable performance under modality deficiencies

this paper proposes an adaptive image-text fusion defect classification framework based on a bidirectional cross-modal attention mechanism. This framework employs vision Transformer (ViT) and bidirectional encoder representation from transformers (BERT) to construct high-dimension visual features and deep semantic representations

respectively. Contrastive learning losses are then introduced to achieve explicit alignment of image and text embeddings within a unified semantic space. The bidirectional cross-modal attention mechanism is designed to adaptively enhance defect-related features and suppress environmental noise through “text→image” and “image→text” interactions. Concurrently

a multi-branch classification strategy is constructed to ensure model robustness under complex conditions such as single-modality deficiency. Experiments demonstrate that the proposed approach achieves an accuracy of 94.5% and an F1 score of 93.9% across 12 typical defect detection tasks in intelligent power inspection scenarios. Compared to existing mainstream unimodal and multimodal fusion methods

the proposed approach exhibits significant improvements in both feature separability and classification accuracy

providing effective support for multimodal processing in power inspection.

关键词

Keywords

references

LI Q, ZHAO F, XIANG H, et al. Research on multi- modal decision-making intelligence technology for complex power operation processes[C] // 2024 20th International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery (ICNC-FSKD), July 27-29, 2024, Guangzhou, China: 1-5.

DAI Y, YAN Z, CHENG J, et al. Analysis of multimodal data fusion from an information theory perspective[J]. Information Sciences, 2023, 623: 164-183.

LU H, ZHANG J, CHEN H, et al. Application of multimodal data fusion in panoramic monitoring and intelligent inspection of power distribution network[C] // 2024 International Conference on Telecommunications and Power Electronics (TELEPE), May 15-17, 2024, Changsha, China: 969-974.

BAYOURDH K. A survey of multimodal hybrid deep learning for computer vision: architectures, applications, trends, and challenges[J]. Information Fusion, 2024, 105.

SHAO J, LI X, LIU T, et al. Defect text recognition and condition assessment method of power equipment based on BERT-BiLSTM[J]. PeerJ Computer Science, 2025, 11.

CHEN K, MAHFOUD R J, SUN Y, et al. Defect texts mining of secondary device in smart substation with GloVe and attention-based bidirectional LSTM[J]. Energies, 2020, 13(17).

YE Z, GUO B, QI D. BERT-TriF: an inductive short text classification model for power equipment defect records[J]. Engineering Reports, 2023, 5(10).

DONG H, YUAN M, WANG S, et al. PHAM-YOLO: a parallel hybrid attention mechanism network for defect detection of meter in substation[J]. Sensors, 2023, 23(13).

CHEN B, ZHANG W, WU W, et al. ID-YOLOv7: an efficient method for insulator defect detection in power distribution network[J]. Frontiers in Neurorobotics, 2024, 17.

CHENG S, HOU R, LI M. The fusion strategy of multimodal learning in image and text recognition[C] // International Conference on Physics, Photonics, and Optical Engineering (ICPPOE 2024), November 22-24, 2024, Singapore, Singapore.

CHEN Z, ZHANG B. Application of multimodal deep learning algorithm in image and text fusion recognition[C] // 2025 IEEE 5th International Conference on Power, Electronics and Computer Applications (ICPECA), January 17-19, 2025, Shenyang, China: 599-604.

LI Q, SONG W, GAO M, et al. Towards text-refereed multi-modal image fusion by cross-modality interaction[J]. Signal Processing, 2025, 237.

WANG Q, ZHANG J, DU J, et al. A fine-tuned multimodal large model for power defect image-text question- answering[J]. Signal, Image and Video Processing, 2024, 18(12): 9191-9203.

LI P, ZENG G, HUANG B, et al. A short text classification model for electrical equipment defects based on contextual features[J]. Wuhan University Journal of Natural Sciences, 2022, 27(6): 465-475.

WANG Q, ZHANG J, DU J, et al. A fine-tuned multimodal large model for power defect image-text question- answering[J]. Signal, Image and Video Processing, 2024, 18(12): 9191-9203.

ZHANG Chunfeng, GONG Yu, ZHANG Yongjun, et al. An automatic multi-scale feature learner and a transformer algorithm for fault detection[J]. Protection and Control of Modern Power Systems, 2025, 10(5): 84-102.

ZHANG Q, YAO T, JIA Q Y, et al. IEVDET: a multi- granular visual event detection framework for power equipment based on cross-modal alignment[C] // 2025 5th International Conference on Artificial Intelligence, Big Data and Algorithms (CAIBDA), June 20-22, 2025, Beijing, China: 178-182.

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Transformer winding fault diagnosis method based on small sample extension of generative adversarial network and deep multimodal feature fusion

考虑局部纹理特征和全局温度分布的电力设备红外图像超分辨率重建方法

电气-环境耦合作用下的引流线可靠性评估方法

基于脉冲转换温度传感方法的电力设备温度监测系统研究

Minute-scale ultra-short-term multi-step power forecasting model for offshore wind farm based on real-time monitoring data

Related Author

XIONG Xiaofu

ZHOU Desheng

BAO Wei

ZENG Ping

SHEN Yuqing

DENG Xiangli

WANG Huidong

LÜ Tingyan

Related Institution

LÜ Tingyan(1. School of Electrical and Electronic Engineering, North China Electric Power University

State Grid Jibei Electric Power Research Institute

AI问答

⁰