Anonymization Prompt Learning for Facial Privacy-Preserving Text-to-Image Generation

View on arXiv ← Back to list

Authors: Liang Shi, Jie Zhang, Shiguang Shan

Published: 2024-05-27 07:38:26+00:00

AI Summary

This paper introduces Anonymization Prompt Learning (APL), a method to prevent text-to-image models from generating identifiable faces. APL trains a learnable prompt prefix that anonymizes faces without significantly impacting image quality or text fidelity for other prompts.

Abstract

Text-to-image diffusion models, such as Stable Diffusion, generate highly realistic images from text descriptions. However, the generation of certain content at such high quality raises concerns. A prominent issue is the accurate depiction of identifiable facial images, which could lead to malicious deepfake generation and privacy violations. In this paper, we propose Anonymization Prompt Learning (APL) to address this problem. Specifically, we train a learnable prompt prefix for text-to-image diffusion models, which forces the model to generate anonymized facial identities, even when prompted to produce images of specific individuals. Extensive quantitative and qualitative experiments demonstrate the successful anonymization performance of APL, which anonymizes any specific individuals without compromising the quality of non-identity-specific image generation. Furthermore, we reveal the plug-and-play property of the learned prompt prefix, enabling its effective application across different pretrained text-to-image models for transferrable privacy and security protection against the risks of deepfakes.

Key findings

APL successfully anonymizes faces across different text-to-image models, reducing identity accuracy by up to a tenfold decrease. The method maintains high image quality and text fidelity for non-identity-specific prompts. The approach also effectively anonymizes identities not seen during training, including those learned through fine-tuning methods like Dreambooth.

Approach

APL trains a learnable prompt prefix that is prepended to text prompts before feeding them to a text-to-image diffusion model. This prefix guides the model to generate anonymized facial identities when prompted with specific individuals, while maintaining generation quality for other prompts.

Datasets

A custom dataset SID containing images of public figures paired with their names (identity-specific prompts) and attribute descriptions (non-identity-specific prompts). The LAION-2B dataset was used for regularization. VGGFace-2 was used for testing anonymization on unseen identities.

Model(s)

Stable Diffusion v1.5, Stable Diffusion v1.4, Realistic Vision v5.1, Stable Diffusion XL. The main contribution is a method that can be used with these models.

Author countries

China

← Previous