基于Transformer的人脸深度伪造检测技术综述

赖志茂; 章云; 李东

doi:10.12052/gdutxb.230130

欢迎来到能源电力期刊网！

English Version 期刊登录

您当前的位置：

首页 >

文章列表页 >

基于Transformer的人脸深度伪造检测技术综述

人工智能 | 更新时间：2026-05-27

- 基于Transformer的人脸深度伪造检测技术综述
- A Survey of Deepfake Detection Techniques Based on Transformer
- 广东工业大学学报 2023年40卷第6期页码：155-167
- 作者机构：
  
  1. 广东工业大学自动化学院,广东,广州,510006
  2. 中国人民警察大学移民管理学院(广州),广东,广州,510663
- 作者简介：
  
  赖志茂(1987–) ，男，讲师，博士研究生，主要研究方向为人工智能、多媒体信息安全
  章云(1963–) ，男，教授，博士，博士生导师，主要研究方向为复杂系统建模与控制、图像处理与模式识别，E-mail：yun@gdut.edu.cn
- 基金信息：
  
  国家自然科学基金资助项目 (62271349);广东省基础与应用基础研究基金资助项目 (2021A1515011867);中国人民警察大学国家基金培育课题 (JJPY202402)
- DOI：10.12052/gdutxb.230130
  中图分类号： TP183
- 网络首发：2023-11-25，
  
  纸质出版：2023
- 稿件说明：
移动端阅览
赖志茂, 章云, 李东. 基于Transformer的人脸深度伪造检测技术综述[J]. 广东工业大学学报, 2023, 40(6): 155-167.

Lai Zhi-mao, Zhang Yun, Li Dong. A Survey of Deepfake Detection Techniques Based on Transformer[J]. 2023, 40(6): 155-167.
赖志茂, 章云, 李东. 基于Transformer的人脸深度伪造检测技术综述[J]. 广东工业大学学报, 2023, 40(6): 155-167. DOI： 10.12052/gdutxb.230130.

Lai Zhi-mao, Zhang Yun, Li Dong. A Survey of Deepfake Detection Techniques Based on Transformer[J]. 2023, 40(6): 155-167. DOI： 10.12052/gdutxb.230130.

摘要

人脸深度伪造检测旨在对人脸图像和视频进行真伪鉴别，能为肖像权保护、虚假消息鉴定、网络诈骗防范等提供理论和技术支撑。早期的检测技术主要基于卷积神经网络(Convolutional Neural Networks，CNNs) 实现，并取得了显著的效果，但普遍存在泛化性能不足的问题。为了进一步提高人脸深度伪造检测技术的泛化性，最新的研究工作开始引入一种基于自我注意力机制的深度神经网络Transformer，其具有长距离依赖建模能力和全局感受野，可用于捕捉到图像上下文关联和视频时序关系，有效提高了检测器的表征能力。本综述首先简要介绍了该领域研究背景，阐述了人脸深度伪造生成典型技术，然后对现有基于Transformer的检测技术进行总结和归纳，最后探讨人脸深度伪造检测技术面临的挑战和未来研究方向。

Abstract

Deepfake detection aims to authenticate facial images and videos

which can offer operational and technical support to safeguard personal portrait rights

prevent fake news

and curb online deceit. Early detection technologies are usually based on convolutional neural networks (CNNs) and have achieved promising detection performance. However

there exists a prevalent issue of mediocre generalisation performance. To enhance the overall generality of Deepfake detection

recent research has focused on a deep neural network Transformer by utilizing the self-attention mechanisms. The Transformer can better model long-distance dependency and global receptive fields to capture the image context association and video timing relationships

such that the representation ability of the detectors can be improved. This survey first provides an overview of the research background in this field

followed by an explanation of the common techniques used to generate Deepfake. Then

the existing Transformer-based detection methods are summarized and comparatively evaluated. Finally

the challenges and future research directions of Deepfake detection are discussed.

关键词

Keywords

references

ROSSLER A, COZZOLINO D, VERDOLIVA L, et al. Faceforensics++: learning to detect manipulated facial images[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. Seoul, South Korea: IEEE, 2019: 1-11.

QIAN Y, YIN G, SHENG L, et al. Thinking in frequency: face forgery detection by mining frequency-aware clues[C]//European Conference on Computer Vision. Online: Springer, 2020: 86-103.

MASI I, KILLEKAR A, MASCARENHAS R M, et al. Two-branch recurrent network for isolating deepfakes in videos[C] //European Conference on Computer Vision. Online: Springer, 2020: 667-684.

LI L, BAO J, ZHANG T, et al. Face x-ray for more general face forgery detection[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle, USA: IEEE, 2020: 5001-5010.

ZHAO H, ZHOU W, CHEN D, et al. Multi-attentional deepfake detection[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Online: IEEE, 2021: 2185-2194.

LIU H, LI X, ZHOU W, et al. Spatial-phase shallow learning: rethinking face forgery detection in frequency domain[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Online: IEEE, 2021: 772-781.

ZHAO T, XU X, XU M, et al. Learning self-consistency for deepfake detection[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. Montreal, Canada: IEEE, 2021: 15023-15033.

HALIASSOS A, VOUGIOUKAS K, PETRIDIS S, et al. Lips don't lie: a generalisable and robust approach to face forgery detection[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Online: IEEE, 2021: 5039-5049.

CHUGH K, GUPTA P, DHALL A, et al. Not made for each other-audio-visual dissonance-based deepfake detection and localization[C]//Proceedings of the 28th ACM International Conference on Multimedia. Seattle , USA: ACM, 2020: 439-447.

MITTAL T, BHATTACHARYA U, CHANDRA R, et al. Emotions don't lie: an audio-visual deepfake detection method using affective cues[C]//Proceedings of the 28th ACM International Conference on Multimedia. Seattle , USA : ACM, 2020: 2823-2832.

ZHOU Y, LIM S N. Joint audio-visual deepfake detection[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. Montreal, Canada: IEEE, 2021: 14800-14809.

PENG Z, GUO Z, HUANG W, et al . Conformer: local features coupling global representations for recognition and detection [J ] . IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023, 45(8): 9454-9468.

VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]//31st Conference on Neural Information Processing Systems. Long Beach, USA: MIT Press, 2017: 5998–6008.

DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An image is worth 16×16 words: transformers for image recognition at scale[C]//Proceedings of the 9th International Conference on Learning Representations. Online: ACM, 2021: 1-6.

LIU Z, LIN Y, CAO Y, et al. Swin transformer: hierarchical vision transformer using shifted windows[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. Montreal, Canada: IEEE, 2021: 10012-10022.

CHEN, RICHARD C F, FAN Q, et. al. Crossvit: cross-attention multi-scale vision transformer for image classification[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. Montreal, Canada: IEEE, 2021: 357-366.

KORSHUNOVA I, SHI W, DAMBRE J, et al. Fast face-swap using convolutional neural networks[C]//Proceedings of the IEEE International Conference on Computer Vision. Venice, Italy: IEEE, 2017: 3677-3685.

LI L, BAO J, YANG H, et al. Advancing high fidelity identity swapping for forgery detection[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle, USA: IEEE, 2020: 5074-5083.

CHEN R, CHEN X, NI B, et al. Simswap: an efficient framework for high fidelity face swapping[C]//Proceedings of the 28th ACM International Conference on Multimedia. Seattle, USA: ACM, 2020: 2003-2011.

THIES J, ZOLLHOFER M, STAMMINGER M, et al. Face2face: real-time face capture and reenactment of rgb videos[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, USA: IEEE, 2016: 2387-2395.

NIRKIN Y, KELLER Y, HASSNER T. FSGAN: subject agnostic face swapping and reenactment[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. Seoul, Korea (South) : IEEE, 2019: 7184-7193.

TOUVRON H, CORD M, DOUZE M, et al. Training data-efficient image transformers distillation through attention[C]//International Conference on Machine Learning. Online: PMLR, 2021: 10347-10357.

WU H, XIAO B, CODELLA N, et al. CvT: Introducing convolutions to vision transformers[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. Montreal, Canada: IEEE, 2021: 22-31.

D'ASCOLI S, TOUVRON H, LEAVITT M L, et al. Convit: Improving vision transformers with soft convolutional inductive biases[C]//Proceedings of the 38th International Conference on Machine Learning. Online: ACM, 2021: 2286-2296.

CHU X, TIAN Z, WANG Y, et al . Twins: revisiting the design of spatial attention in vision transformers [J ] . Advances in Neural Information Processing Systems, 2021, 34: 9355-9366.

CHEN R, PANDA R, FAN Q. RegionViT: regional-to-local attention for vision transformers[C]//International Conference on Learning Representations. Online: ACM , 2022.

WANG W, XIE E, LI X, et al. Pyramid vision transformer: a versatile backbone for dense prediction without convolutions[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. Montreal, Canada: IEEE, 2021: 568-578.

YUAN L, CHEN Y, WANG T, et al. Tokens-to-token ViT: Training vision transformers from scratch on imagenet[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. Montreal, Canada: IEEE, 2021: 558-567.

TOUVRON H, CORD M, SABLAYROLLES A, et al. Going deeper with image transformers[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. Montreal, Canada: IEEE, 2021: 32-42.

DONG X, BAO J, CHEN D, et al. Protecting celebrities from deepfake with identity consistency transformer[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans, USA: IEEE, 2022: 9468-9478.

CHEN H, LIN Y, LI B, et al . Learning features of intra-consistency and inter-diversity: keys toward generalizable deepfake detection [J ] . IEEE Transactions on Circuits and Systems for Video Technology, 2023, 33(3): 1468-1480.

WANG J, WU Z, OUYANG W, et al. M2tr: multi-modal multi-scale transformers for deepfake detection[C]//Proceedings of the 2022 International Conference on Multimedia Retrieval. Newark, USA: ACM, 2022: 615-623.

TAN Z, YANG Z, MIAO C, et al . Transformer-based feature compensation and aggregation for deepfake detection [J ] . IEEE Signal Processing Letters, 2022, 29: 2183-2187.

MIAO C, TAN Z, CHU Q, et al . Hierarchical frequency-assisted interactive networks for face manipulation detection [J ] . IEEE Transactions on Information Forensics and Security, 2022, 17: 3008-3021.

MIAO C, TAN Z, CHU Q, et al . F2Trans: high-frequency fine-grained transformer for face forgery detection [J ] . IEEE Transactions on Information Forensics and Security, 2023, 18: 1039-1051.

HOCHREITER S, SCHMIDHUBER J. Long short-term memory [J]. Neural computation, 1997, 9(8): 1735-1780.

ZHENG Y, BAO J, CHEN D, et al. Exploring temporal coherence for more general video face forgery detection[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. Montreal, Canada: IEEE, 2021: 15044-15054.

GUAN J, ZHOU H, HONG Z, et al . Delving into sequential patches for deepfake detection [J ] . Advances in Neural Information Processing Systems, 2022, 35: 4517-4530.

ZHAO C, WANG C, HU G, et al . ISTVT: interpretable spatial-temporal video transformer for deepfake detection [J ] . IEEE Transactions on Information Forensics and Security, 2023, 18: 1335-1348.

YU Y, NI R, ZHAO Y, et al . MSVT: multiple spatiotemporal views transformer for deepfake video detection [J ] . IEEE Transactions on Circuits and Systems for Video Technology, 2023, 33(9): 4462-4471.

ILYAS H, JAVED A, MALIK K M. AVFakeNet: A unified end-to-end dense swin transformer deep learning model for audio-visual deepfakes detection [J]. Applied Soft Computing, 2023, 136: 110124.

YANG W, ZHOU X, CHEN Z, et al . AVoiD-DF: audio-visual joint learning for detecting deepfake [J ] . IEEE Transactions on Information Forensics and Security, 2023, 18: 2015-2029.

FENG C, CHEN Z, OWENS A. Self-supervised video forensics by audio-visual anomaly detection[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Vancouver, Canada: IEEE, 2023: 10491-10503.

YU Y, LIU X, NI R, et al. PVASS-MDD: predictive visual-audio alignment self-supervision for multimodal deepfake detection[J ] . IEEE Transactions on Circuits and Systems for Video Technol ogy. https://ieeexplore.ieee.org/document/10233898

LI Y, YANG X, SUN P, et al. Celeb-DF: a large-scale challenging dataset for deepfake forensics[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle, USA : IEEE, 2020: 3207-3216.

ZI B, CHANG M, CHEN J, et al. WildDeepfake: a challenging real-world dataset for deepfake detection[C]//Proceedings of the 28th ACM International Conference on Multimedia. Seattle, USA: ACM, 2020: 2382-2390.

JIANG L, LI R, WU W, et al. Deeperforensics-1.0: a large-scale dataset for real-world face forgery detection[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle, USA: IEEE, 2020: 2889-2898.

ZHOU T, WANG W, LIANG Z, et al. Face forensics in the wild[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Online: IEEE, 2021: 5778-5788.

HE Y, GAN B, CHEN S, et al. ForgeryNet: a versatile benchmark for comprehensive forgery analysis[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Online: IEEE, 2021: 4360-4369.

KWON P, YOU J, NAM G, et al. KoDF: a large-scale Korean deepfake detection dataset[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. Montreal, Canada: IEEE, 2021: 10744-10753.

CAI Z, STEFANOV K, DHALL A, et al. Do you really mean that? content driven audio-visual deepfake dataset and multimodal method for temporal forgery localization[C]//2022 International Conference on Digital Image Computing: Techniques and Applications. Online: IEEE, 2022: 1-10.

浏览量

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

基于高效时序建模的RGB-T跟踪

基于时序对齐的风格控制语音合成算法

基于红外可见光融合的复杂环境下人脸识别方法

SCA-YOLO: 基于可变形卷积与上下文感知注意力的实时目标检测

多尺度对比嵌入增强的自适应入侵检测模型