基于增量式类激活知识的蒸馏学习方法

张家悦; 张灵

doi:10.12052/gdutxb.220018

欢迎来到能源电力期刊网！

English Version 期刊登录

您当前的位置：

首页 >

文章列表页 >

基于增量式类激活知识的蒸馏学习方法

计算机科学与技术 | 更新时间：2026-05-27

- 基于增量式类激活知识的蒸馏学习方法
- Knowledge Distillation Method Based on Incremental Class Activation Knowledge
- 广东工业大学学报 2023年40卷第4期页码：24-30, 36
- 作者机构：
  
  广东工业大学计算机学院,广东,广州,510006
- 作者简介：
  
  张家悦(1998–)，男，硕士研究生，主要研究方向为深度学习
  张灵(1968–)，女，教授，主要研究方向为智能优化算法、深度学习及数字图像处理技术、分布式计算及大数据技术，E-mail：1252875930@qq.com
- 基金信息：
  
  广东省交通运输厅科技项目(科技-2016-02-030)
- DOI：10.12052/gdutxb.220018
  中图分类号： TP391
- 网络首发：2023-07-25，
  
  纸质出版：2023
- 稿件说明：
移动端阅览
张家悦, 张灵. 基于增量式类激活知识的蒸馏学习方法[J]. 广东工业大学学报, 2023, 40(4): 24-30, 36.

Zhang Jia-yue, Zhang Ling. Knowledge Distillation Method Based on Incremental Class Activation Knowledge[J]. 2023, 40(4): 24-30, 36.
张家悦, 张灵. 基于增量式类激活知识的蒸馏学习方法[J]. 广东工业大学学报, 2023, 40(4): 24-30, 36. DOI： 10.12052/gdutxb.220018.

Zhang Jia-yue, Zhang Ling. Knowledge Distillation Method Based on Incremental Class Activation Knowledge[J]. 2023, 40(4): 24-30, 36. DOI： 10.12052/gdutxb.220018.

摘要

由于样本特征缺乏类别判定性且设备资源不足以支持样本类别结构学习，现有的知识蒸馏方法往往忽略了样本的类别知识蒸馏。针对此问题，本文提出一种增量式类激活知识蒸馏方法(Incremental Class Activation Knowledge Distillation，ICAKD)。首先，利用类激活梯度图提取具备类别判定性的样本特征，并提出类激活图约束损失。然后，构建存储类别判定性特征的增量式记忆库，保存多个训练批次样本并迭代更新。最后，计算记忆库内每一类样本的类质中心

并构造类别结构关系，根据类激活图约束和类别结构关系实现类别知识蒸馏。在Cifar10、Cifar100、Tiny-ImageNet、ImageNet等数据集上进行对比实验，结果表明本文所提出的方法对比类别结构蒸馏方法(Category Structure Knowledge Distillation，CSKD)在准确率上有0.4%～1.21%的提升，说明了类别判定性特征和增量式方法对类别知识蒸馏起到促进作用。

Abstract

Due to the fact that features are not category-deterministic and the equipment resources are usually limit to support the category structure learning of samples

existing knowledge distillation methods possibly ignore the category knowledge distillation of samples. Therefore

this paper proposes a distillation method based on incremental class activation knowledge (ICAKD) . First

this paper uses the class activation gradient map to extract class-discriminative sample features and proposes a class-activation constraint loss. Then

an incremental memory bank is built to store class-deterministic features

and multiple training batch samples are saved and updated iteratively. Finally

our proposed method calculates the quasi-quality center of the samples in the memory bank to construct the category structure relationship

and further performs the category knowledge distillation according to the class-activation constraint and the category structure relationship. Experimental results on the Cifar10

Cifar100

Tiny-ImageNet

and ImageNet datasets show that the proposed method achieves a 0.4%～1.21% improvement in term of accuracy when compared with the Category Structure Knowledge Distillation(CSKD) methods

demonstrating the promising effectiveness of the characteristics and increment of category judgment for category knowledge distillation.

关键词

Keywords

references

CHEN Z, ZHENG X, SHEN H, et al. Improving knowledge distillation via category structure[C]//16th European Conference on Computer Vision. Glasgow: Springer, 2020: 205-219.

MÜLLER R, KORNBLITH S, HINTON G. When does label smoothing help?[C]//Annual Conference on Neural Information Processing Systems 2019. Vancouver: MIT Press, 2019: 4694-4703.

JANG Y, LEE H, HWANG S J, et al. Learning what and where to transfer[C]//Proceedings of the 36th International Conference on Machine Learning. Long Beach: PMLR, 2019: 3030-3039.

PARK W, KIM D, LU Y, et al. Relational knowledge distillation[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Long Beach: IEEE, 2019: 3967-3976.

PENG B, JIN X, LIU J, et al. Correlation congruence for knowledge distillation[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Long Beach: IEEE, 2019: 5007-5016.

LIU Y, CAO J, LI B, et al. Knowledge distillation via instance relationship graph[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Long Beach: IEEE, 2019: 7096-7104.

LI X, WU J, FANG H, et al. Local correlation consistency for knowledge distillation[C]//16th European Conference on Computer Vision. Glasgow: Springer, 2020: 18-33.

CHEN D, MEI J P, ZHANG Y, et al. Cross-layer distillation with semantic calibration[C]//Thirty-Fifth AAAI Conference on Artificial Intelligence. Virtual Event: AAAI, 2021, 35(8) : 7028-7036.

YUN S, PARK J, LEE K, et al. Regularizing class-wise predictions via self-knowledge distillation[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Seattle: IEEE, 2020: 13873-13882.

ZEILER M D, TAYLOR G W, FERGUS R. Adaptive deconvolutional networks for mid and high level feature learning[C]//2011 International Conference on Computer Vision. Barcelona: IEEE, 2011: 2018-2025.

ZHOU B, KHOSLA A, LAPEDRIZA A, et al. Learning deep features for discriminative localization[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . Las Vegas: IEEE, 2016: 2921-2929.

SELVARAJU R, COGSWELL M, DAS A, et al . Grad-CAM: visual explanations from deep networks via gradient-based localization [J ] . International Journal of Computer Vision, 2020, 128(2): 336-359.

LI K, WU Z, PENG K C, et al. Tell me where to look: guided attention inference network[C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 9215-9223.

VAN O A, KALCHBRENNER N, KAVUKCUOGLU K. Pixel recurrent neural networks[C]//Proceedings of the 33nd International Conference on Machine Learning. New York: JMLR. org, 2016: 1747-1756.

浏览量

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

多尺度对比嵌入增强的自适应入侵检测模型

人工噪声辅助无人机太赫兹安全通信的鲁棒设计

融合沟道倾斜角的3D NAND存储单元建模及编程策略优化

面向灾后救援的多无人机通感算一体化资源分配优化方法

基于高效时序建模的RGB-T跟踪