基于Log-sum正则化的稀疏神经网络

谢文杰; 刘玉兰

doi:10.12052/gdutxb.240107

欢迎来到能源电力期刊网！

English Version 期刊登录

您当前的位置：

首页 >

文章列表页 >

基于Log-sum正则化的稀疏神经网络

计算机科学与技术 | 更新时间：2026-05-27

- 基于Log-sum正则化的稀疏神经网络
- Sparse Neural Network Based on Log-sum Regularization
- 广东工业大学学报 2025年42卷第6期页码：78-85
- 作者机构：
  
  广东工业大学数学与统计学院,广东,广州,510520
- 作者简介：
  
  谢文杰(1999–)，男，硕士研究生，主要研究方向为最优化理论及应用，E-mail：1137656028@qq.com
  刘玉兰(1977–)，女，教授，博士，主要研究方向为最优化理论及应用，E-mail：ylliu@gdut.edu.cn
- 基金信息：
  
  广东省基础与应用基础研究基金资助项目(2023A1515012891)
- DOI：10.12052/gdutxb.240107
  中图分类号： TP301
- 纸质出版：2025
- 稿件说明：
移动端阅览
谢文杰, 刘玉兰. 基于Log-sum正则化的稀疏神经网络[J]. 广东工业大学学报, 2025, 42(6): 78–85.

Xie Wenjie, Liu Yulan. Sparse Neural Network Based on Log-sum Regularization[J]. 2025, 42(6): 78-85.
谢文杰, 刘玉兰. 基于Log-sum正则化的稀疏神经网络[J]. 广东工业大学学报, 2025, 42(6): 78–85. DOI： 10.12052/gdutxb.240107

Xie Wenjie, Liu Yulan. Sparse Neural Network Based on Log-sum Regularization[J]. 2025, 42(6): 78-85. DOI： 10.12052/gdutxb.240107.

摘要

为了训练出一个稀疏的神经网络，本文选用Log-sum函数作为正则项，交叉熵函数作为损失项建立模型，随后用邻近梯度法对它进行求解，其学习率用Meta-LR-Schedule-Net网络训练得到。在MNIST、Fashion-MNIST、CIFAR-10和CIFAR-100四个数据集上的数值实验结果表明：在同样的学习率规则下，用Log-sum函数作为正则项比用其他能诱导稀疏的函数，如向量1范数、截尾向量1范数或向量1/2范数等作为正则项建立的模型能把网络训练得更稀疏；在稀疏度近似的情况下，用Meta-LR-Schedule-Net训练得到的学习率，比使用固定规则训练得到的网络有更高的正确率。

Abstract

In order to train a sparse neural network

the Log-sum function is chosen as the regularization term and the cross entropy function as the loss function to establish the model and solved it by proximal gradient algorithm

whose learning rate was obtained by Meta-LR-Schedule-Net. The network is trained more sparsely by the model with the Log-sum regularized function than by other models with other induced sparse functions

such as the 1-norm function

transformed 1-norm function

and 1/2-norm function

according to numerical experiments conducted on four data sets: MNIST

Fashion-MNIST

CIFAR-10

and CIFAR-100. Additionally

the network trained by using the Meta-LR-Schedule-Net learning rule is more accurate than those trained by using a fixed learning rate.

关键词

Keywords

references

FUKUSHIMA K, MIYAKE S, ITO T. Neocognitron: a neural network model for a mechanism of visual pattern recognition[J]. IEEE Trans on System,Man,and Cybernetics, 1983, 13(5): 826-834.

COLLOBERT R, WESTON J. A unified architecture for natural language processing: deep neural networks with multitask learning[C]//Proceedings of the 25th International Conference on Machine Learning, 2008 (ICML 2008) . Helsinki, Finland: ACM, 2008 : 160-167.

ALVAREZ J M, SALZMANN M. Learning the number of neurons in deep networks[C]// Advances in Neural Information Processing Systems. Cambridge: MIT Press, 2016: 29.

SONG H, POOL J, TRAN J, et al . Learning both weights and connections for efficient neural networks[C ] //Advances in Neural Information Processing Systems. Cambridge: MIT Press, 2015: 28.

SRIVASTAVA N, HINTON G, KRIZHEVSKY A, et al . Dropout: a simple way to prevent neural networks from overfitting[J ] . Journal of Machine Learning Research, 2014, 15(1): 1929-1958.

MA R R, MIAO J Y, NIU L F, et al . Transformed ℓ1 regularization for learning sparse deep neural networks[J ] . Neural Networks, 2019, 119: 286-298.

QIAO C, SHI Y, DIAO Y X, et al . Log-sum enhanced sparse deep neural network[J ] . Neurocomputing, 2020, 407: 206-220.

CHEN C H, MANGASARIAN O L. Hybrid misclassification minimization[J]. Advances in Computational Mathematics, 1996, 5(1): 127-136.

AGHASI A, ABDI A, NGUYEN N, et al . Net-Trim: convex pruning of deep neural networks with performance guarantee[C ] //Advances in Neural Information Processing Systems. Cambridge: MIT Press, 2017: 30.

ZHANG J S, JI N N, LIU J M, et al . Enhancing performance of the backpropagation algorithm via sparse response regularization[J ] . Neurocomputing, 2015(153): 20-40.

FOUCART S, LAI M J. Sparsest solutions of underdetermined linear systems via ℓq-minimization for 0 q ≤1[J ] . Applied and Computational Harmonic Analysis, 2009, 26(3): 395-407.

WANG H, YANG X Y, SHI Y M, et al . A proximal iteratively reweighted approach for efficient network sparsification[J ] . IEEE Transactions on Computers, 2020(99): 185-196.

PRATER-BENNETTE A, SHEN L X, TRIPP E E. The proximity operator of the log-sum penalty[J]. Journal of Scientific Computing, 2022, 93(3): 67.

SHU J, ZHU Y W, ZHAO Q, et al . MLR-SNet: transferable LR schedules for heterogeneous tasks[J ] . IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 45(3): 3505-3521.

TSENG P. An incremental gradient(-projection) method with momentum term and adaptive stepsize rule[J]. SIAM Journal on Optimization, 1998, 8(2): 506-531.

KINGMA D, BA J. Adam: a method for stochastic optimization[C]// International Conference on Learning Representations (ICLR) . Washington DC‌‌: ACM, 2015: 7-9.

WARD L B. Reminiscence and rote learning[M]//Psychological Monographs. London: AP Association, 1937: 1-64.

ANDRYCHOWICZ M, DENIL M, GOMEZ S, et al . Learning to learn by gradient descent by gradient descent[C ] //Advances in Neural Information Processing Systems. Cambridge: MIT Press, 2016: 29.

CHEN Y T, HOFFMAN M W, GÓMEZ C S, et al . Learning to learn without gradient descent by gradient descent[C ] //International Conference on Machine Learning. Madison, Wis: International Machine Learning Society, 2017: 748-756.

BOTTOU L. Stochastic gradient learning in neural networks[C]//Proceedings of Neuro-Nimes. Oxford: Pergamon Press, 1991: 12.

BIAN W, CHEN X J. A smoothing proximal gradient algorithm for nonsmooth convex regression with cardinality penalty[J]. SIAM Journal on Numerical Analysis, 2020, 58(1): 858-883.

TANG A D, NIU L F, MIAO J Y, et al . Training compact DNNS with ℓ1/2 regularization[J ] . Pattern Recognitio n, 2023(136): 109206.

浏览量

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

融合沟道倾斜角的3D NAND存储单元建模及编程策略优化

基于高效时序建模的RGB-T跟踪

具守恒量波动方程的循环双子网物理信息神经网络

储能变流器电能质量治理策略研究

SCA-YOLO: 基于可变形卷积与上下文感知注意力的实时目标检测