AOT: Appearance Optimal Transport Based Identity Swapping for Forgery Detection

View on arXiv ← Back to list

Authors: Hao Zhu, Chaoyou Fu, Qianyi Wu, Wayne Wu, Chen Qian, Ran He

Published: 2020-11-05 06:17:04+00:00

AI Summary

This paper proposes a novel identity swapping algorithm, Appearance Optimal Transport (AOT), to generate diverse and realistic Deepfakes for improving face forgery detection. AOT formulates appearance mapping as an optimal transport problem, solving it in both latent and pixel space to minimize appearance gaps while preserving identity traits.

Abstract

Recent studies have shown that the performance of forgery detection can be improved with diverse and challenging Deepfakes datasets. However, due to the lack of Deepfakes datasets with large variance in appearance, which can be hardly produced by recent identity swapping methods, the detection algorithm may fail in this situation. In this work, we provide a new identity swapping algorithm with large differences in appearance for face forgery detection. The appearance gaps mainly arise from the large discrepancies in illuminations and skin colors that widely exist in real-world scenarios. However, due to the difficulties of modeling the complex appearance mapping, it is challenging to transfer fine-grained appearances adaptively while preserving identity traits. This paper formulates appearance mapping as an optimal transport problem and proposes an Appearance Optimal Transport model (AOT) to formulate it in both latent and pixel space. Specifically, a relighting generator is designed to simulate the optimal transport plan. It is solved via minimizing the Wasserstein distance of the learned features in the latent space, enabling better performance and less computation than conventional optimization. To further refine the solution of the optimal transport plan, we develop a segmentation game to minimize the Wasserstein distance in the pixel space. A discriminator is introduced to distinguish the fake parts from a mix of real and fake image patches. Extensive experiments reveal that the superiority of our method when compared with state-of-the-art methods and the ability of our generated data to improve the performance of face forgery detection.

Key findings

AOT outperforms state-of-the-art identity swapping methods in generating realistic Deepfakes with large appearance differences. The generated data improves the performance of face forgery detection algorithms on FF++ and DPF-1.0 datasets. Ablation studies highlight the importance of 3D features and the mix-and-segment discriminator.

Approach

AOT uses a relighting generator to simulate optimal transport in latent space, minimizing the Wasserstein distance of learned features. A segmentation game further refines the solution in pixel space by using a discriminator to distinguish real and fake image patches, forcing realistic synthesis.

Datasets

FaceForensics++ (FF++) and DeeperForensics-1.0 (DPF-1.0)

Model(s)

AOT model comprising a relighting generator and a mix-and-segment discriminator. The perceptual encoder uses a two-branch architecture. Experiments also use I3D and TSN for forgery detection.

Author countries

China

← Previous