IDRetracor: Towards Visual Forensics Against Malicious Face Swapping

View on arXiv ← Back to list

Authors: Jikang Cheng, Jiaxin Ai, Zhen Han, Chao Liang, Qin Zou, Zhongyuan Wang, Qian Wang

Published: 2024-08-13 04:53:48+00:00

AI Summary

This paper introduces a novel face retracing task for deepfake detection, aiming to reconstruct the original target face from a manipulated one. They propose IDRetracor, a framework using mapping-aware convolutions and a mapping resolver to dynamically retrace faces across multiple face-swapping methods and unseen target identities.

Abstract

The face swapping technique based on deepfake methods poses significant social risks to personal identity security. While numerous deepfake detection methods have been proposed as countermeasures against malicious face swapping, they can only output binary labels (Fake/Real) for distinguishing fake content without reliable and traceable evidence. To achieve visual forensics and target face attribution, we propose a novel task named face retracing, which considers retracing the original target face from the given fake one via inverse mapping. Toward this goal, we propose an IDRetracor that can retrace arbitrary original target identities from fake faces generated by multiple face swapping methods. Specifically, we first adopt a mapping resolver to perceive the possible solution space of the original target face for the inverse mappings. Then, we propose mapping-aware convolutions to retrace the original target face from the fake one. Such convolutions contain multiple kernels that can be combined under the control of the mapping resolver to tackle different face swapping mappings dynamically. Extensive experiments demonstrate that the IDRetracor exhibits promising retracing performance from both quantitative and qualitative perspectives.

Key findings

IDRetracor significantly outperforms baseline UNet models in retracing performance, achieving high identity similarity (Arcface similarity exceeding 0.65) across multiple face-swapping methods and unseen target identities. The method demonstrates promising visual forensics capabilities, even when target attributes are partially removed by sophisticated face-swapping techniques.

Approach

IDRetracor uses mapping-aware convolutions with multiple kernel groups to handle variations in face-swapping techniques. A mapping resolver guides the kernel recombination, dynamically adapting to different mappings and reconstructing the original target face.

Datasets

A custom dataset created using VGGFace2 and four face-swapping methods (SimSwap, InfoSwap, HifiFace, E4S); Celeb-DF-v2 was also used for supplementary experiments.

Model(s)

IDRetracor, a framework incorporating mapping-aware convolutions (inspired by Dynamic Convolution), a ResNet18-based mapping resolver, and a modified three-layer UNet for reconstruction. Vanilla UNet models (VU-S, VU-M) served as baselines.

Author countries

China

← Previous