Construction of a multimodal intelligent agent for live-line operation monitoring in distribution networks via first-person lightweight object detection fusion

LIU Zihao; LIU Youbo; GONG Haochen; WAN Li; YUAN Lin; LIU Junyong

doi:10.19783/j.cnki.pspc.251328

English Version Journal Login

您当前的位置：

首页 >

文章列表页 >

Construction of a multimodal intelligent agent for live-line operation monitoring in distribution networks via first-person lightweight object detection fusion

更新时间：2026-06-18

- Construction of a multimodal intelligent agent for live-line operation monitoring in distribution networks via first-person lightweight object detection fusion
- Power System Protection and Control Vol. 54, Issue 11, (2026)
- 作者机构：
- 作者简介：
- 基金信息：
- DOI：10.19783/j.cnki.pspc.251328
  CLC：
- Published：2026
- 稿件说明：
移动端阅览
LIU Zihao, LIU Youbo, GONG Haochen, et al. Construction of a multimodal intelligent agent for live-line operation monitoring in distribution networks via first-person lightweight object detection fusion[J]. Power System Protection and Control, 2026, 54(11).
DOI：

LIU Zihao, LIU Youbo, GONG Haochen, et al. Construction of a multimodal intelligent agent for live-line operation monitoring in distribution networks via first-person lightweight object detection fusion[J]. Power System Protection and Control, 2026, 54(11). DOI： 10.19783/j.cnki.pspc.251328.

摘要

针对配电网带电作业监护面临视角局限、边缘端算力不足，且缺乏专用轻量化智能方案支撑操作精准解析的问题，提出融合第一视角轻量目标检测的多模态智能体构建方法。首先，基于 YOLO11 模型引入改进多尺度注意力、风车形状卷积及跨视图交互模块，分别优化多尺度特征提取性能、低对比度特征轻量化提取性能与视图抗干扰性能，采用最小点距交并比损失函数优化边界框回归。然后，根据目标检测信息实现第一视角信息提取并生成结构化提示词。最后，结合目标检测模型与 DeepSeek-V3 构建多模态智能体，整合视觉检测结果，量化输出风险评估与操作分析。所提方法浮点运算次数为 7.3 G，其交并比阈值小于 0.5 的目标检测多类别平均精度显著优于主流单阶段模型。输出内容在状态判断准确率、结构化程度和内容可解释度上均优于主流开源多模态模型。所提方法为多场景配电网带电作业监护提供高效、可推广的方案。

Abstract

To address the limitations in live-line operation monitoring for distribution networks

including restricted field of view

insufficient edge computing capability

and the lack of dedicated lightweight intelligent solutions for accurate operation analysis

a construction method for multimodal intelligent agent integrating first-person lightweight object detection is proposed. First

built on the YOLO11 model

an improved architecture incorporating multi-scale attention mechanisms

windmill-shaped convolution

and cross-view interaction modules is introduced to enhance multi-scale feature extraction

lightweight low-contrast feature representation

and view robustness. The model adopts minimum point distance intersection over union (MPDIoU) loss function to refine bounding box regression. Then

first-person view information is extracted from detection results to generate structured prompts. Finally

a multimodal intelligent agent is constructed by integrating the optimized detection model with DeepSeek-V3

enabling the fusion of visual outputs for quantitative risk assessment and operation analysis. Experimental results show that the proposed method reduces computational complexity to 7.3 G FLOPs

and achieves up to a 9.44% improvement in mean precision for multi-class detection with IoU thresholds below 0.5 compared to mainstream single-stage models. The generated outputs outperform leading open-source multimodal models in state judgment accuracy

structuring degree

and interpretability

providing an efficient and scalable solution for multimodal monitoring of live-line operations in distribution networks.

关键词

Keywords

references

KOMAROMI P. Live working method comparisons: Rubber glove work vs. hotstick work vs. barehand work[C] // 2017 12th International Conference on Live Maintenance (ICOLIM), 2017: 1-6.

YANG Qi, ZOU Dehua, TONG Cheng, et al. The safety analysis of live working on 500/220 kV four-parallel lines on the same tower[C] // 2016 IEEE International Conference on Power and Renewable Energy (ICPRE), October 21-23, 2016, Shanghai, China: 15-18.

LOVRENČIĆ V, RIBIČ P, OGRIZEK P, et al. Safe work in the vicinity of live parts on high voltage double-circuit overhead lines[C] // 2022 13th International Conference on Live Maintenance (ICOLIM), June 15-17, 2022, Turin, Italy: 1-6.

ZHANG Haibo, ZHANG Yixuan, ZHONG Bin, et al. A comprehensive survey of vision-based human action recognition methods[J]. Sensors, 2019, 19(5).

YETGIN Ö E, BENLIGIRAY B, GEREK Ö N. Power line recognition from aerial images with deep learning[J]. IEEE Transactions on Aerospace and Electronic Systems, 2019, 55(5): 2241-2252.

GUO Hao, XU Liang, QIAO Ming, et al. A depth completion method used in live working safety monitoring tasks[C] // 2024 9th International Symposium on Computer and Information Processing Technology (ISCIPT), May 24-26, 2024, Xi’an, China: 427-432.

HU Wei, ZHENG Tingting, DU Puliang, et al. An intelligent assisted assessment method for distribution grid engineering quality based on large language modelling and ensemble learning[J]. Protection and Control of Modern Power Systems, 2025, 10(3): 83-97.

LIU Yuxuan, JIA Meng, ZHANG Yuxuan, et al. RePower: an LLM-driven autonomous platform for power system data-guided research[J]. Patterns, 2025, 6(4).

YANG Jiannan, LIU Shengjie, WU Jiayi, et al. Pinwheel- shaped convolution and scale-based dynamic loss for infrared small target detection[J]. arxiv preprint arxiv: 2412.16986, 2024.

WANG Yan, LI Ying, WANG Gang, et al. Multi-scale attention network for single image super-resolution[C] // Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 17-18, 2024, Seattle, WA, USA: 5950-5960.

LI Yijun, ZOU Wenbin, WEI Qiquan, et al. Multi-level feature fusion network for lightweight stereo image super-resolution[C] // Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 17-18, 2024, Seattle, WA, USA: 6489-6498.

MA Simao, XU Yong. MPDIoU: a loss for efficient and accurate bounding box regression[J]. arxiv preprint arxiv:2307.07662, 2023.

LIU Aixin, FENG Bei, XUE Bing, et al. Deepseek-v3 technical report[J]. arxiv preprint arxiv:2412.19437, 2024.

LIN Bin, YE Yang, ZHU Bin, et al. Video-llava: learning united visual representation by alignment before projection[J]. arxiv preprint arxiv:2311.10122, 2023.

BAI Shuai, CHEN Keqin, LIU Xian, et al. Qwen2.5-vl technical report[J]. arxiv preprint arxiv:2502.13923, 2025.

XU Jin, GUO Zhenru, HE Junyang, et al. Qwen2.5-omni technical report[J]. arxiv preprint arxiv: 2503.20215, 2025.

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

基于BERT模型的主设备缺陷诊断方法研究

Impact of asymmetry in non-transposed same-tower transmission lines on differential protection

Model identification pilot protection of dual-end weak-feed AC systems based on active detection

Transformer winding fault diagnosis method based on small sample extension of generative adversarial network and deep multimodal feature fusion

A section location method for compound single-phase open-circuit ground faults in distribution networks based on negative-sequence voltage amplitude difference

Related Author

YANG Hong

Related Institution

No data

AI问答

⁰