Deepfake Style Transfer Mixture: a First Forensic Ballistics Study on Synthetic Images

View on arXiv ← Back to list

Authors: Luca Guarnera, Oliver Giudice, Sebastiano Battiato

Published: 2022-03-18 13:11:54+00:00

AI Summary

This paper presents a novel approach to forensic image ballistics for deepfakes, focusing on detecting the number of style-transfer operations applied to an image. It investigates mathematical properties of style-transfer operations and proposes two methods: an analytical approach using DCT coefficients and a deep learning approach using ResNet-18.

Abstract

Most recent style-transfer techniques based on generative architectures are able to obtain synthetic multimedia contents, or commonly called deepfakes, with almost no artifacts. Researchers already demonstrated that synthetic images contain patterns that can determine not only if it is a deepfake but also the generative architecture employed to create the image data itself. These traces can be exploited to study problems that have never been addressed in the context of deepfakes. To this aim, in this paper a first approach to investigate the image ballistics on deepfake images subject to style-transfer manipulations is proposed. Specifically, this paper describes a study on detecting how many times a digital image has been processed by a generative architecture for style transfer. Moreover, in order to address and study accurately forensic ballistics on deepfake images, some mathematical properties of style-transfer operations were investigated.

Key findings

The deep learning approach (ResNet-18) achieved a significantly higher classification accuracy (92.75%) than the analytical method (81% using Random Forest). The study also showed that certain mathematical properties of style-transfer operations are not fully satisfied, offering insights for future forensic investigations.

Approach

The study uses two methods to detect the number of style-transfer operations on deepfake images. The first analyzes DCT coefficients of the image, while the second leverages a pre-trained ResNet-18 model for classification. Both methods are trained and tested on datasets of images processed by StarGAN-V2.

Datasets

Images generated by StarGAN-V2, specifically categorized into Deepfake-2 (one style transfer operation) and Deepfake-3 (two style transfer operations). A total of 2400 images were used for training and 400 for testing.

Model(s)

ResNet-18 (with added fully connected layer and softmax), k-NN, SVM, LDA, Decision Tree, Random Forest, GBoost.

Author countries

Italy

← Previous