Reverse Engineering of Generative Models: Inferring Model Hyperparameters from Generated Images

View on arXiv ← Back to list

Authors: Vishal Asnani, Xi Yin, Tal Hassner, Xiaoming Liu

Published: 2021-06-15 04:19:26+00:00

AI Summary

This paper introduces a novel framework for "model parsing," which infers generative model (GM) hyperparameters (network architecture and loss functions) from generated images. The framework uses a Fingerprint Estimation Network (FEN) to extract unique image patterns and a Parsing Network (PN) to predict hyperparameters, achieving state-of-the-art results on deepfake detection and image attribution benchmarks.

Abstract

State-of-the-art (SOTA) Generative Models (GMs) can synthesize photo-realistic images that are hard for humans to distinguish from genuine photos. Identifying and understanding manipulated media are crucial to mitigate the social concerns on the potential misuse of GMs. We propose to perform reverse engineering of GMs to infer model hyperparameters from the images generated by these models. We define a novel problem, ``model parsing, as estimating GM network architectures and training loss functions by examining their generated images -- a task seemingly impossible for human beings. To tackle this problem, we propose a framework with two components: a Fingerprint Estimation Network (FEN), which estimates a GM fingerprint from a generated image by training with four constraints to encourage the fingerprint to have desired properties, and a Parsing Network (PN), which predicts network architecture and loss functions from the estimated fingerprints. To evaluate our approach, we collect a fake image dataset with $100$K images generated by $116$ different GMs. Extensive experiments show encouraging results in parsing the hyperparameters of the unseen models. Finally, our fingerprint estimation can be leveraged for deepfake detection and image attribution, as we show by reporting SOTA results on both the deepfake detection (Celeb-DF) and image attribution benchmarks.

Key findings

The framework demonstrates promising results in parsing unseen model hyperparameters. Fingerprint estimation generalizes well to deepfake detection and image attribution, achieving state-of-the-art performance on respective benchmarks. The approach also shows potential for detecting coordinated misinformation attacks.

Approach

The proposed framework consists of a Fingerprint Estimation Network (FEN) that extracts unique patterns from generated images based on several defined constraints. A Parsing Network (PN) then uses these fingerprints to predict the generative model's hyperparameters, leveraging clustering to improve accuracy.

Datasets

A custom dataset of 100K images generated by 116 different generative models, including GANs, VAEs, Adversarial Attacks, Autoregressive, and Normalizing Flow models; Celeb-DF for deepfake detection; CelebA and LSUN for image attribution.

Model(s)

Fingerprint Estimation Network (FEN), Parsing Network (PN), and shallow networks for deepfake detection and image attribution.

Author countries

USA, Israel

← Previous