Knowledge Distillation Method Based on Incremental Class Activation Knowledge

Zhang Jia-yue; Zhang Ling

doi:10.12052/gdutxb.220018

English Version Journal Login

您当前的位置：

首页 >

文章列表页 >

Knowledge Distillation Method Based on Incremental Class Activation Knowledge

更新时间：2026-05-27

- Knowledge Distillation Method Based on Incremental Class Activation Knowledge
- Vol. 40, Issue 4, Pages: 24-30, 36(2023)
- 作者机构：
  
  广东工业大学计算机学院,广东,广州,510006
- 作者简介：
- 基金信息：
- DOI：10.12052/gdutxb.220018
  CLC： TP391
- Online First：25 July 2023，
  
  Published：2023
- 稿件说明：
移动端阅览
张家悦, 张灵. 基于增量式类激活知识的蒸馏学习方法[J]. 广东工业大学学报, 2023, 40(4): 24-30, 36.

Zhang Jia-yue, Zhang Ling. Knowledge Distillation Method Based on Incremental Class Activation Knowledge[J]. 2023, 40(4): 24-30, 36.
张家悦, 张灵. 基于增量式类激活知识的蒸馏学习方法[J]. 广东工业大学学报, 2023, 40(4): 24-30, 36. DOI： 10.12052/gdutxb.220018.

Zhang Jia-yue, Zhang Ling. Knowledge Distillation Method Based on Incremental Class Activation Knowledge[J]. 2023, 40(4): 24-30, 36. DOI： 10.12052/gdutxb.220018.

摘要

由于样本特征缺乏类别判定性且设备资源不足以支持样本类别结构学习，现有的知识蒸馏方法往往忽略了样本的类别知识蒸馏。针对此问题，本文提出一种增量式类激活知识蒸馏方法(Incremental Class Activation Knowledge Distillation，ICAKD)。首先，利用类激活梯度图提取具备类别判定性的样本特征，并提出类激活图约束损失。然后，构建存储类别判定性特征的增量式记忆库，保存多个训练批次样本并迭代更新。最后，计算记忆库内每一类样本的类质中心

并构造类别结构关系，根据类激活图约束和类别结构关系实现类别知识蒸馏。在Cifar10、Cifar100、Tiny-ImageNet、ImageNet等数据集上进行对比实验，结果表明本文所提出的方法对比类别结构蒸馏方法(Category Structure Knowledge Distillation，CSKD)在准确率上有0.4%～1.21%的提升，说明了类别判定性特征和增量式方法对类别知识蒸馏起到促进作用。

Abstract

Due to the fact that features are not category-deterministic and the equipment resources are usually limit to support the category structure learning of samples

existing knowledge distillation methods possibly ignore the category knowledge distillation of samples. Therefore

this paper proposes a distillation method based on incremental class activation knowledge (ICAKD) . First

this paper uses the class activation gradient map to extract class-discriminative sample features and proposes a class-activation constraint loss. Then

an incremental memory bank is built to store class-deterministic features

and multiple training batch samples are saved and updated iteratively. Finally

our proposed method calculates the quasi-quality center of the samples in the memory bank to construct the category structure relationship

and further performs the category knowledge distillation according to the class-activation constraint and the category structure relationship. Experimental results on the Cifar10

Cifar100

Tiny-ImageNet

and ImageNet datasets show that the proposed method achieves a 0.4%～1.21% improvement in term of accuracy when compared with the Category Structure Knowledge Distillation(CSKD) methods

demonstrating the promising effectiveness of the characteristics and increment of category judgment for category knowledge distillation.

关键词

Keywords

references

CHEN Z, ZHENG X, SHEN H, et al. Improving knowledge distillation via category structure[C]//16th European Conference on Computer Vision. Glasgow: Springer, 2020: 205-219.

MÜLLER R, KORNBLITH S, HINTON G. When does label smoothing help?[C]//Annual Conference on Neural Information Processing Systems 2019. Vancouver: MIT Press, 2019: 4694-4703.

JANG Y, LEE H, HWANG S J, et al. Learning what and where to transfer[C]//Proceedings of the 36th International Conference on Machine Learning. Long Beach: PMLR, 2019: 3030-3039.

PARK W, KIM D, LU Y, et al. Relational knowledge distillation[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Long Beach: IEEE, 2019: 3967-3976.

PENG B, JIN X, LIU J, et al. Correlation congruence for knowledge distillation[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Long Beach: IEEE, 2019: 5007-5016.

LIU Y, CAO J, LI B, et al. Knowledge distillation via instance relationship graph[C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Long Beach: IEEE, 2019: 7096-7104.

LI X, WU J, FANG H, et al. Local correlation consistency for knowledge distillation[C]//16th European Conference on Computer Vision. Glasgow: Springer, 2020: 18-33.

CHEN D, MEI J P, ZHANG Y, et al. Cross-layer distillation with semantic calibration[C]//Thirty-Fifth AAAI Conference on Artificial Intelligence. Virtual Event: AAAI, 2021, 35(8) : 7028-7036.

YUN S, PARK J, LEE K, et al. Regularizing class-wise predictions via self-knowledge distillation[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Seattle: IEEE, 2020: 13873-13882.

ZEILER M D, TAYLOR G W, FERGUS R. Adaptive deconvolutional networks for mid and high level feature learning[C]//2011 International Conference on Computer Vision. Barcelona: IEEE, 2011: 2018-2025.

ZHOU B, KHOSLA A, LAPEDRIZA A, et al. Learning deep features for discriminative localization[C]//2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . Las Vegas: IEEE, 2016: 2921-2929.

SELVARAJU R, COGSWELL M, DAS A, et al . Grad-CAM: visual explanations from deep networks via gradient-based localization [J ] . International Journal of Computer Vision, 2020, 128(2): 336-359.

LI K, WU Z, PENG K C, et al. Tell me where to look: guided attention inference network[C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 9215-9223.

VAN O A, KALCHBRENNER N, KAVUKCUOGLU K. Pixel recurrent neural networks[C]//Proceedings of the 33nd International Conference on Machine Learning. New York: JMLR. org, 2016: 1747-1756.

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Research and Implementation of a Pulsed Laser Ranging Method Based on Digital Mixing

Bimodal Iterative Cross-attention Fusion Ensemble Framework

Energy Management Strategy of Hybrid Locomotive Based on Road Condition Parameters

Modeling of 3D NAND Memory Cells with Taper Angle and Programming Strategy Optimization

Ground Deformation Monitoring and Influencing Factors Analysis in Fuzhou City Based on PSI Technology

Related Author

No data

Related Institution

No data

AI问答

⁰