Text-image guided Diffusion Model for generating Deepfake celebrity interactions

Authors: Yunzhuo Chen, Nur Al Hasan Haldar, Naveed Akhtar, Ajmal Mian

Published: 2023-09-26 08:24:37+00:00

Comment: 8 pages,8 figures, DICTA

AI Summary

This paper introduces Text-image Guided Diffusion Model (TIDM), a novel method that modifies the Stable Diffusion model to generate controllable, high-quality deepfake images of celebrity interactions. TIDM addresses the original model's limitations in multi-person image generation by incorporating an anchor image's latent representation and leveraging Dreambooth for enhanced realism. The results demonstrate the alarming potential of generating highly realistic fake visual content that could serve as believable evidence for spreading rumors.

Abstract

Deepfake images are fast becoming a serious concern due to their realism. Diffusion models have recently demonstrated highly realistic visual content generation, which makes them an excellent potential tool for Deepfake generation. To curb their exploitation for Deepfakes, it is imperative to first explore the extent to which diffusion models can be used to generate realistic content that is controllable with convenient prompts. This paper devises and explores a novel method in that regard. Our technique alters the popular stable diffusion model to generate a controllable high-quality Deepfake image with text and image prompts. In addition, the original stable model lacks severely in generating quality images that contain multiple persons. The modified diffusion model is able to address this problem, it add input anchor image's latent at the beginning of inferencing rather than Gaussian random latent as input. Hence, we focus on generating forged content for celebrity interactions, which may be used to spread rumors. We also apply Dreambooth to enhance the realism of our fake images. Dreambooth trains the pairing of center words and specific features to produce more refined and personalized output images. Our results show that with the devised scheme, it is possible to create fake visual content with alarming realism, such that the content can serve as believable evidence of meetings between powerful political figures.

Key findings

The proposed TIDM significantly improves the quality and realism of generated deepfake images, outperforming the original Stable Diffusion Model, especially in generating multi-person interactions and consistent backgrounds. Subjective human evaluations revealed that TIDM-generated images were rated significantly more convincing than those from SDM. This highlights the alarming potential of the method to create believable fake visual evidence for spreading misinformation.

Approach

The authors propose Text-image Guided Diffusion Model (TIDM) by altering the Stable Diffusion model. Their approach incorporates an additional Image Encoder to process the latent representation of an anchor image, which is then fed alongside text prompts into a two-stream denoising U-Net. This mechanism, combined with a Dreambooth-based fine-tuning method, allows for precise control over background and foreground elements, improved multi-person generation, and heightened realism.

Datasets

Training images of celebrities obtained from Google searches (3-5 pictures per individual).

Model(s)

A modified Stable Diffusion Model (SDM) utilizing a two-stream denoising U-Net architecture. It integrates an additional Image Encoder and is fine-tuned using a Dreambooth-based method.

Author countries

Australia

← Previous