Currently

using crawler framework in internet scene has become an important way to get data efficiently. However

the existing crawler framework is designed for specific users

scenarios and uses of the internet

its direct application in power industry internet will cause problems such as inapplicability and insecurity. Therefore

this article proposes a crawler framework for power industry internet based on generic ideas

combining with the actual needs of the power industry internet

from the user

scene

module design

use and other aspects of generic design. The proposed frame not only has the ability of existing crawler frames

but also can meet the needs of the power industry internet data crawling. A verification test was carried out in a power industry internet production environment of a group

it shows that the proposed framework can meet the requirements of different users

different scenarios as well as the security requirements of the power industry internet. Moreover

it has the characteristics of the existing crawler framework

and the generic design achieves the desired results.