Deepfake for the Good: Generating Avatars through Face-Swapping with Implicit Deepfake Generation

View on arXiv ← Back to list

Authors: Georgii Stanishevskii, Jakub Steczkiewicz, Tomasz Szczepanik, Sławomir Tadeja, Jacek Tabor, Przemysław Spurek

Published: 2024-02-09 13:11:57+00:00

AI Summary

This paper introduces ImplicitDeepfake, a novel method for generating 3D avatars by applying a 2D deepfake algorithm to training images before training them on Neural Radiance Fields (NeRF) or Gaussian Splatting (GS). This approach leverages existing deepfake technology to create realistic 3D deepfake avatars.

Abstract

Numerous emerging deep-learning techniques have had a substantial impact on computer graphics. Among the most promising breakthroughs are the rise of Neural Radiance Fields (NeRFs) and Gaussian Splatting (GS). NeRFs encode the object's shape and color in neural network weights using a handful of images with known camera positions to generate novel views. In contrast, GS provides accelerated training and inference without a decrease in rendering quality by encoding the object's characteristics in a collection of Gaussian distributions. These two techniques have found many use cases in spatial computing and other domains. On the other hand, the emergence of deepfake methods has sparked considerable controversy. Deepfakes refers to artificial intelligence-generated videos that closely mimic authentic footage. Using generative models, they can modify facial features, enabling the creation of altered identities or expressions that exhibit a remarkably realistic appearance to a real person. Despite these controversies, deepfake can offer a next-generation solution for avatar creation and gaming when of desirable quality. To that end, we show how to combine all these emerging technologies to obtain a more plausible outcome. Our ImplicitDeepfake uses the classical deepfake algorithm to modify all training images separately and then train NeRF and GS on modified faces. Such simple strategies can produce plausible 3D deepfake-based avatars.

Key findings

Gaussian Splatting produced more visually plausible results than NeRF. The approach successfully generated 3D avatars from a single image and text prompts. The method showed robustness across different 2D deepfake models.

Approach

ImplicitDeepfake uses a 2D deepfake algorithm (GHOST) to modify training images of a 3D face model. Then, it trains either NeRF or Gaussian Splatting on these modified images to generate novel views and create a 3D avatar. Diffusion models can further refine the avatar using text prompts.

Datasets

UNKNOWN. The abstract mentions using images of celebrities and 3D human avatar models, but doesn't specify any named datasets.

Model(s)

GHOST (for 2D deepfake), NeRF, Gaussian Splatting, Diffusion models (Stable Diffusion, ControlNet, EbSynth)

Author countries

Poland, UK

← Previous