Sparse Neural Network Based on Log-sum Regularization

Xie Wenjie; Liu Yulan

doi:10.12052/gdutxb.240107

English Version Journal Login

您当前的位置：

首页 >

文章列表页 >

Sparse Neural Network Based on Log-sum Regularization

更新时间：2026-05-27

- Sparse Neural Network Based on Log-sum Regularization
- Vol. 42, Issue 6, Pages: 78-85(2025)
- 作者机构：
  
  广东工业大学数学与统计学院,广东,广州,510520
- 作者简介：
- 基金信息：
- DOI：10.12052/gdutxb.240107
  CLC： TP301
- Published：2025
- 稿件说明：
移动端阅览
谢文杰, 刘玉兰. 基于Log-sum正则化的稀疏神经网络[J]. 广东工业大学学报, 2025, 42(6): 78–85.

Xie Wenjie, Liu Yulan. Sparse Neural Network Based on Log-sum Regularization[J]. 2025, 42(6): 78-85.
谢文杰, 刘玉兰. 基于Log-sum正则化的稀疏神经网络[J]. 广东工业大学学报, 2025, 42(6): 78–85. DOI： 10.12052/gdutxb.240107

Xie Wenjie, Liu Yulan. Sparse Neural Network Based on Log-sum Regularization[J]. 2025, 42(6): 78-85. DOI： 10.12052/gdutxb.240107.

摘要

为了训练出一个稀疏的神经网络，本文选用Log-sum函数作为正则项，交叉熵函数作为损失项建立模型，随后用邻近梯度法对它进行求解，其学习率用Meta-LR-Schedule-Net网络训练得到。在MNIST、Fashion-MNIST、CIFAR-10和CIFAR-100四个数据集上的数值实验结果表明：在同样的学习率规则下，用Log-sum函数作为正则项比用其他能诱导稀疏的函数，如向量1范数、截尾向量1范数或向量1/2范数等作为正则项建立的模型能把网络训练得更稀疏；在稀疏度近似的情况下，用Meta-LR-Schedule-Net训练得到的学习率，比使用固定规则训练得到的网络有更高的正确率。

Abstract

In order to train a sparse neural network

the Log-sum function is chosen as the regularization term and the cross entropy function as the loss function to establish the model and solved it by proximal gradient algorithm

whose learning rate was obtained by Meta-LR-Schedule-Net. The network is trained more sparsely by the model with the Log-sum regularized function than by other models with other induced sparse functions

such as the 1-norm function

transformed 1-norm function

and 1/2-norm function

according to numerical experiments conducted on four data sets: MNIST

Fashion-MNIST

CIFAR-10

and CIFAR-100. Additionally

the network trained by using the Meta-LR-Schedule-Net learning rule is more accurate than those trained by using a fixed learning rate.

关键词

Keywords

references

FUKUSHIMA K, MIYAKE S, ITO T. Neocognitron: a neural network model for a mechanism of visual pattern recognition[J]. IEEE Trans on System,Man,and Cybernetics, 1983, 13(5): 826-834.

COLLOBERT R, WESTON J. A unified architecture for natural language processing: deep neural networks with multitask learning[C]//Proceedings of the 25th International Conference on Machine Learning, 2008 (ICML 2008) . Helsinki, Finland: ACM, 2008 : 160-167.

ALVAREZ J M, SALZMANN M. Learning the number of neurons in deep networks[C]// Advances in Neural Information Processing Systems. Cambridge: MIT Press, 2016: 29.

SONG H, POOL J, TRAN J, et al . Learning both weights and connections for efficient neural networks[C ] //Advances in Neural Information Processing Systems. Cambridge: MIT Press, 2015: 28.

SRIVASTAVA N, HINTON G, KRIZHEVSKY A, et al . Dropout: a simple way to prevent neural networks from overfitting[J ] . Journal of Machine Learning Research, 2014, 15(1): 1929-1958.

MA R R, MIAO J Y, NIU L F, et al . Transformed ℓ1 regularization for learning sparse deep neural networks[J ] . Neural Networks, 2019, 119: 286-298.

QIAO C, SHI Y, DIAO Y X, et al . Log-sum enhanced sparse deep neural network[J ] . Neurocomputing, 2020, 407: 206-220.

CHEN C H, MANGASARIAN O L. Hybrid misclassification minimization[J]. Advances in Computational Mathematics, 1996, 5(1): 127-136.

AGHASI A, ABDI A, NGUYEN N, et al . Net-Trim: convex pruning of deep neural networks with performance guarantee[C ] //Advances in Neural Information Processing Systems. Cambridge: MIT Press, 2017: 30.

ZHANG J S, JI N N, LIU J M, et al . Enhancing performance of the backpropagation algorithm via sparse response regularization[J ] . Neurocomputing, 2015(153): 20-40.

FOUCART S, LAI M J. Sparsest solutions of underdetermined linear systems via ℓq-minimization for 0 q ≤1[J ] . Applied and Computational Harmonic Analysis, 2009, 26(3): 395-407.

WANG H, YANG X Y, SHI Y M, et al . A proximal iteratively reweighted approach for efficient network sparsification[J ] . IEEE Transactions on Computers, 2020(99): 185-196.

PRATER-BENNETTE A, SHEN L X, TRIPP E E. The proximity operator of the log-sum penalty[J]. Journal of Scientific Computing, 2022, 93(3): 67.

SHU J, ZHU Y W, ZHAO Q, et al . MLR-SNet: transferable LR schedules for heterogeneous tasks[J ] . IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 45(3): 3505-3521.

TSENG P. An incremental gradient(-projection) method with momentum term and adaptive stepsize rule[J]. SIAM Journal on Optimization, 1998, 8(2): 506-531.

KINGMA D, BA J. Adam: a method for stochastic optimization[C]// International Conference on Learning Representations (ICLR) . Washington DC‌‌: ACM, 2015: 7-9.

WARD L B. Reminiscence and rote learning[M]//Psychological Monographs. London: AP Association, 1937: 1-64.

ANDRYCHOWICZ M, DENIL M, GOMEZ S, et al . Learning to learn by gradient descent by gradient descent[C ] //Advances in Neural Information Processing Systems. Cambridge: MIT Press, 2016: 29.

CHEN Y T, HOFFMAN M W, GÓMEZ C S, et al . Learning to learn without gradient descent by gradient descent[C ] //International Conference on Machine Learning. Madison, Wis: International Machine Learning Society, 2017: 748-756.

BOTTOU L. Stochastic gradient learning in neural networks[C]//Proceedings of Neuro-Nimes. Oxford: Pergamon Press, 1991: 12.

BIAN W, CHEN X J. A smoothing proximal gradient algorithm for nonsmooth convex regression with cardinality penalty[J]. SIAM Journal on Numerical Analysis, 2020, 58(1): 858-883.

TANG A D, NIU L F, MIAO J Y, et al . Training compact DNNS with ℓ1/2 regularization[J ] . Pattern Recognitio n, 2023(136): 109206.

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

SCA-YOLO: Real-time Target Detection Based on Deformable Convolution and Context-aware Attention

Multi-Scale Contrastive Embedding Enhanced Adaptive Intrusion Detection Model

Ground Deformation Monitoring and Influencing Factors Analysis in Fuzhou City Based on PSI Technology

A Study of Power Quality Regulation Strategies of Energy Storage Converters

Robust Design for Artificial Noise-assisted UAV Terahertz Secure Communication

Related Author

No data

Related Institution

No data

AI问答

⁰