A Survey of Deepfake Detection Techniques Based on Transformer

Lai Zhi-mao; Zhang Yun; Li Dong

doi:10.12052/gdutxb.230130

English Version Journal Login

您当前的位置：

首页 >

文章列表页 >

A Survey of Deepfake Detection Techniques Based on Transformer

更新时间：2026-05-27

- A Survey of Deepfake Detection Techniques Based on Transformer
- Vol. 40, Issue 6, Pages: 155-167(2023)
- 作者机构：
  
  1. 广东工业大学自动化学院,广东,广州,510006
  2. 中国人民警察大学移民管理学院(广州),广东,广州,510663
- 作者简介：
- 基金信息：
- DOI：10.12052/gdutxb.230130
  CLC： TP183
- Online First：25 November 2023，
  
  Published：2023
- 稿件说明：
移动端阅览
赖志茂, 章云, 李东. 基于Transformer的人脸深度伪造检测技术综述[J]. 广东工业大学学报, 2023, 40(6): 155-167.

Lai Zhi-mao, Zhang Yun, Li Dong. A Survey of Deepfake Detection Techniques Based on Transformer[J]. 2023, 40(6): 155-167.
赖志茂, 章云, 李东. 基于Transformer的人脸深度伪造检测技术综述[J]. 广东工业大学学报, 2023, 40(6): 155-167. DOI： 10.12052/gdutxb.230130.

Lai Zhi-mao, Zhang Yun, Li Dong. A Survey of Deepfake Detection Techniques Based on Transformer[J]. 2023, 40(6): 155-167. DOI： 10.12052/gdutxb.230130.

摘要

人脸深度伪造检测旨在对人脸图像和视频进行真伪鉴别，能为肖像权保护、虚假消息鉴定、网络诈骗防范等提供理论和技术支撑。早期的检测技术主要基于卷积神经网络(Convolutional Neural Networks，CNNs) 实现，并取得了显著的效果，但普遍存在泛化性能不足的问题。为了进一步提高人脸深度伪造检测技术的泛化性，最新的研究工作开始引入一种基于自我注意力机制的深度神经网络Transformer，其具有长距离依赖建模能力和全局感受野，可用于捕捉到图像上下文关联和视频时序关系，有效提高了检测器的表征能力。本综述首先简要介绍了该领域研究背景，阐述了人脸深度伪造生成典型技术，然后对现有基于Transformer的检测技术进行总结和归纳，最后探讨人脸深度伪造检测技术面临的挑战和未来研究方向。

Abstract

Deepfake detection aims to authenticate facial images and videos

which can offer operational and technical support to safeguard personal portrait rights

prevent fake news

and curb online deceit. Early detection technologies are usually based on convolutional neural networks (CNNs) and have achieved promising detection performance. However

there exists a prevalent issue of mediocre generalisation performance. To enhance the overall generality of Deepfake detection

recent research has focused on a deep neural network Transformer by utilizing the self-attention mechanisms. The Transformer can better model long-distance dependency and global receptive fields to capture the image context association and video timing relationships

such that the representation ability of the detectors can be improved. This survey first provides an overview of the research background in this field

followed by an explanation of the common techniques used to generate Deepfake. Then

the existing Transformer-based detection methods are summarized and comparatively evaluated. Finally

the challenges and future research directions of Deepfake detection are discussed.

关键词

Keywords

references

ROSSLER A, COZZOLINO D, VERDOLIVA L, et al. Faceforensics++: learning to detect manipulated facial images[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. Seoul, South Korea: IEEE, 2019: 1-11.

QIAN Y, YIN G, SHENG L, et al. Thinking in frequency: face forgery detection by mining frequency-aware clues[C]//European Conference on Computer Vision. Online: Springer, 2020: 86-103.

MASI I, KILLEKAR A, MASCARENHAS R M, et al. Two-branch recurrent network for isolating deepfakes in videos[C] //European Conference on Computer Vision. Online: Springer, 2020: 667-684.

LI L, BAO J, ZHANG T, et al. Face x-ray for more general face forgery detection[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle, USA: IEEE, 2020: 5001-5010.

ZHAO H, ZHOU W, CHEN D, et al. Multi-attentional deepfake detection[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Online: IEEE, 2021: 2185-2194.

LIU H, LI X, ZHOU W, et al. Spatial-phase shallow learning: rethinking face forgery detection in frequency domain[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Online: IEEE, 2021: 772-781.

ZHAO T, XU X, XU M, et al. Learning self-consistency for deepfake detection[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. Montreal, Canada: IEEE, 2021: 15023-15033.

HALIASSOS A, VOUGIOUKAS K, PETRIDIS S, et al. Lips don't lie: a generalisable and robust approach to face forgery detection[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Online: IEEE, 2021: 5039-5049.

CHUGH K, GUPTA P, DHALL A, et al. Not made for each other-audio-visual dissonance-based deepfake detection and localization[C]//Proceedings of the 28th ACM International Conference on Multimedia. Seattle , USA: ACM, 2020: 439-447.

MITTAL T, BHATTACHARYA U, CHANDRA R, et al. Emotions don't lie: an audio-visual deepfake detection method using affective cues[C]//Proceedings of the 28th ACM International Conference on Multimedia. Seattle , USA : ACM, 2020: 2823-2832.

ZHOU Y, LIM S N. Joint audio-visual deepfake detection[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. Montreal, Canada: IEEE, 2021: 14800-14809.

PENG Z, GUO Z, HUANG W, et al . Conformer: local features coupling global representations for recognition and detection [J ] . IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023, 45(8): 9454-9468.

VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]//31st Conference on Neural Information Processing Systems. Long Beach, USA: MIT Press, 2017: 5998–6008.

DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An image is worth 16×16 words: transformers for image recognition at scale[C]//Proceedings of the 9th International Conference on Learning Representations. Online: ACM, 2021: 1-6.

LIU Z, LIN Y, CAO Y, et al. Swin transformer: hierarchical vision transformer using shifted windows[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. Montreal, Canada: IEEE, 2021: 10012-10022.

CHEN, RICHARD C F, FAN Q, et. al. Crossvit: cross-attention multi-scale vision transformer for image classification[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. Montreal, Canada: IEEE, 2021: 357-366.

KORSHUNOVA I, SHI W, DAMBRE J, et al. Fast face-swap using convolutional neural networks[C]//Proceedings of the IEEE International Conference on Computer Vision. Venice, Italy: IEEE, 2017: 3677-3685.

LI L, BAO J, YANG H, et al. Advancing high fidelity identity swapping for forgery detection[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle, USA: IEEE, 2020: 5074-5083.

CHEN R, CHEN X, NI B, et al. Simswap: an efficient framework for high fidelity face swapping[C]//Proceedings of the 28th ACM International Conference on Multimedia. Seattle, USA: ACM, 2020: 2003-2011.

THIES J, ZOLLHOFER M, STAMMINGER M, et al. Face2face: real-time face capture and reenactment of rgb videos[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, USA: IEEE, 2016: 2387-2395.

NIRKIN Y, KELLER Y, HASSNER T. FSGAN: subject agnostic face swapping and reenactment[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. Seoul, Korea (South) : IEEE, 2019: 7184-7193.

TOUVRON H, CORD M, DOUZE M, et al. Training data-efficient image transformers distillation through attention[C]//International Conference on Machine Learning. Online: PMLR, 2021: 10347-10357.

WU H, XIAO B, CODELLA N, et al. CvT: Introducing convolutions to vision transformers[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. Montreal, Canada: IEEE, 2021: 22-31.

D'ASCOLI S, TOUVRON H, LEAVITT M L, et al. Convit: Improving vision transformers with soft convolutional inductive biases[C]//Proceedings of the 38th International Conference on Machine Learning. Online: ACM, 2021: 2286-2296.

CHU X, TIAN Z, WANG Y, et al . Twins: revisiting the design of spatial attention in vision transformers [J ] . Advances in Neural Information Processing Systems, 2021, 34: 9355-9366.

CHEN R, PANDA R, FAN Q. RegionViT: regional-to-local attention for vision transformers[C]//International Conference on Learning Representations. Online: ACM , 2022.

WANG W, XIE E, LI X, et al. Pyramid vision transformer: a versatile backbone for dense prediction without convolutions[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. Montreal, Canada: IEEE, 2021: 568-578.

YUAN L, CHEN Y, WANG T, et al. Tokens-to-token ViT: Training vision transformers from scratch on imagenet[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. Montreal, Canada: IEEE, 2021: 558-567.

TOUVRON H, CORD M, SABLAYROLLES A, et al. Going deeper with image transformers[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. Montreal, Canada: IEEE, 2021: 32-42.

DONG X, BAO J, CHEN D, et al. Protecting celebrities from deepfake with identity consistency transformer[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans, USA: IEEE, 2022: 9468-9478.

CHEN H, LIN Y, LI B, et al . Learning features of intra-consistency and inter-diversity: keys toward generalizable deepfake detection [J ] . IEEE Transactions on Circuits and Systems for Video Technology, 2023, 33(3): 1468-1480.

WANG J, WU Z, OUYANG W, et al. M2tr: multi-modal multi-scale transformers for deepfake detection[C]//Proceedings of the 2022 International Conference on Multimedia Retrieval. Newark, USA: ACM, 2022: 615-623.

TAN Z, YANG Z, MIAO C, et al . Transformer-based feature compensation and aggregation for deepfake detection [J ] . IEEE Signal Processing Letters, 2022, 29: 2183-2187.

MIAO C, TAN Z, CHU Q, et al . Hierarchical frequency-assisted interactive networks for face manipulation detection [J ] . IEEE Transactions on Information Forensics and Security, 2022, 17: 3008-3021.

MIAO C, TAN Z, CHU Q, et al . F2Trans: high-frequency fine-grained transformer for face forgery detection [J ] . IEEE Transactions on Information Forensics and Security, 2023, 18: 1039-1051.

HOCHREITER S, SCHMIDHUBER J. Long short-term memory [J]. Neural computation, 1997, 9(8): 1735-1780.

ZHENG Y, BAO J, CHEN D, et al. Exploring temporal coherence for more general video face forgery detection[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. Montreal, Canada: IEEE, 2021: 15044-15054.

GUAN J, ZHOU H, HONG Z, et al . Delving into sequential patches for deepfake detection [J ] . Advances in Neural Information Processing Systems, 2022, 35: 4517-4530.

ZHAO C, WANG C, HU G, et al . ISTVT: interpretable spatial-temporal video transformer for deepfake detection [J ] . IEEE Transactions on Information Forensics and Security, 2023, 18: 1335-1348.

YU Y, NI R, ZHAO Y, et al . MSVT: multiple spatiotemporal views transformer for deepfake video detection [J ] . IEEE Transactions on Circuits and Systems for Video Technology, 2023, 33(9): 4462-4471.

ILYAS H, JAVED A, MALIK K M. AVFakeNet: A unified end-to-end dense swin transformer deep learning model for audio-visual deepfakes detection [J]. Applied Soft Computing, 2023, 136: 110124.

YANG W, ZHOU X, CHEN Z, et al . AVoiD-DF: audio-visual joint learning for detecting deepfake [J ] . IEEE Transactions on Information Forensics and Security, 2023, 18: 2015-2029.

FENG C, CHEN Z, OWENS A. Self-supervised video forensics by audio-visual anomaly detection[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Vancouver, Canada: IEEE, 2023: 10491-10503.

YU Y, LIU X, NI R, et al. PVASS-MDD: predictive visual-audio alignment self-supervision for multimodal deepfake detection[J ] . IEEE Transactions on Circuits and Systems for Video Technol ogy. https://ieeexplore.ieee.org/document/10233898

LI Y, YANG X, SUN P, et al. Celeb-DF: a large-scale challenging dataset for deepfake forensics[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle, USA : IEEE, 2020: 3207-3216.

ZI B, CHANG M, CHEN J, et al. WildDeepfake: a challenging real-world dataset for deepfake detection[C]//Proceedings of the 28th ACM International Conference on Multimedia. Seattle, USA: ACM, 2020: 2382-2390.

JIANG L, LI R, WU W, et al. Deeperforensics-1.0: a large-scale dataset for real-world face forgery detection[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle, USA: IEEE, 2020: 2889-2898.

ZHOU T, WANG W, LIANG Z, et al. Face forensics in the wild[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Online: IEEE, 2021: 5778-5788.

HE Y, GAN B, CHEN S, et al. ForgeryNet: a versatile benchmark for comprehensive forgery analysis[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Online: IEEE, 2021: 4360-4369.

KWON P, YOU J, NAM G, et al. KoDF: a large-scale Korean deepfake detection dataset[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. Montreal, Canada: IEEE, 2021: 10744-10753.

CAI Z, STEFANOV K, DHALL A, et al. Do you really mean that? content driven audio-visual deepfake dataset and multimodal method for temporal forgery localization[C]//2022 International Conference on Digital Image Computing: Techniques and Applications. Online: IEEE, 2022: 1-10.

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Efficient Temporal Modeling for RGB-T Tracking

Temporal Alignment Style Control in Text-to-Speech Synthesis Algorithm

Face Recognition Method in Complex Environment Based on Infrared Visible Fusion

Bimodal Iterative Cross-attention Fusion Ensemble Framework

Research on Foreign Object Detection in Power Transmission Lines Based on the AAGV-YOLOX Model

Related Author

Ni Xinyu

Zeng Bi

Zhang Zhongxuan

Luo Chaofan

Liu Zhenyu

Guo Ao

Xu Bo-yan

Cai Rui-chu

Related Institution

School of Computer Science and Technology, Guangdong University of Technology

School of Information Engineering, Guangdong University of Technology

College of Science, Shantou University

AI问答

⁰