Jie Liu, Yijia Cao, Yong Li, 等. Big Data Cleaning Based on Improved CLOF and Random Forest for Distribution Networks[J]. 中国电机工程学会电力与能源系统学报(英文), 2024,10(6):2528-2538.
Jie Liu, Yijia Cao, Yong Li, et al. Big Data Cleaning Based on Improved CLOF and Random Forest for Distribution Networks[J]. CSEE Journal of Power and Energy Systems, 2024, 10(6): 2528-2538.
Jie Liu, Yijia Cao, Yong Li, 等. Big Data Cleaning Based on Improved CLOF and Random Forest for Distribution Networks[J]. 中国电机工程学会电力与能源系统学报(英文), 2024,10(6):2528-2538.DOI: 10.17775/CSEEJPES.2020.04080.
Jie Liu, Yijia Cao, Yong Li, et al. Big Data Cleaning Based on Improved CLOF and Random Forest for Distribution Networks[J]. CSEE Journal of Power and Energy Systems, 2024, 10(6): 2528-2538.DOI: 10.17775/CSEEJPES.2020.04080.
Big Data Cleaning Based on Improved CLOF and Random Forest for Distribution Networks
摘要
Abstract
In order to improve the data quality
the big data cleaning method for distribution networks is studied in this paper. First
the Local Outlier Factor (LOF) algorithm based on DBSCAN clustering is used to detect outliers. However
due to the difficulty in determining the LOF threshold
a method of dynamically calculating the threshold based on the transformer districts and time is proposed. In addition
the LOF algorithm combines the statistical distribution method to reduce the misjudgment rate. Aiming at the diversity and complexity of data missing forms in power big data
this paper has improved the Random Forest imputation algorithm
which can be applied to various forms of missing data
especially the blocked missing data and even some completely missing horizontal or vertical data. The data in this paper are from real data of 44 transformer districts of a certain 10 kV line in a distribution network. Experimental results show that outlier detection is accurate and suitable for any shape and multidimensional power big data. The improved Random Forest imputation algorithm is suitable for all missing forms
with higher imputation accuracy and better model stability. By comparing the network loss prediction between the data using this data cleaning method and the data removing outliers and missing values
it can be found that the accuracy of network loss prediction has improved by nearly 4 % using the data cleaning method identified in this paper. Additionally
as the proportion of bad data increased
the difference between the prediction accuracy of cleaned data and that of uncleaned data is more significant.
关键词
Keywords
references
28. L. Zhang, H. Y. Hu, Y. J. Zhang, J. G. Qiao, and J. M. Xu, “Feature selection and optimization of random forest modeling,” Applied Me-chanics and Materials , vol. 687–691, pp. 1416–419, Nov. 2014, doi: [Online ] . Available: https://doi.org/l0.4028/www.scientific.net/AMM.687-691.1416.