LaP-Forensics: Latent-Pixel Consistency Guided Multimodal Reasoning for Deepfake Detection

Authors: Can Wang, Yuhao Wang, Yushe Cao, Canran Xiao, Fei Shen

Published: 2026-07-28 16:45:52+00:00

Comment: Accepted at ACM Multimedia 2026 (ACM MM 2026)

AI Summary

LaP-Forensics is a multimodal deepfake detection framework that combines RGB semantics with reconstruction-based forensic evidence. It leverages a frozen Stable Diffusion DDIM inversion-reconstruction model to generate a residual map that measures local compatibility, which is then used alongside RGB information by a Where-What-Why model for textual analysis and artifact mask prediction. The framework is trained using supervised fine-tuning followed by Group Relative Policy Optimization, demonstrating competitive cross-generator detection and artifact localization.

Abstract

Recent generative models can produce images with few obvious visual artifacts, weakening detectors and explanations that rely only on surface appearance. We present LaP-Forensics, a multimodal framework that augments RGB semantics with reconstruction-based forensic evidence. A frozen Stable Diffusion DDIM inversion-reconstruction model provides a fixed reconstruction reference, and its residual map measures local compatibility with that reference. Independent projectors encode the RGB image and residual map before a structured Where-What-Why model predicts a textual analysis and an artifact mask.Supervised fine-tuning is followed by Group Relative Policy Optimization (GRPO), whose reward combines mask overlap with output-structure and evidence-reference terms. These text-side terms encourage the model to refer to the consistency map but do not constitute a verifier of free-form textual truth. A separate image-level head fuses RGB and DDIM-residual class features. Experiments show cross-generator detection on UniversalFakeDetect and competitive artifact localization on the official SynthScars benchmark. Controlled cue-construction, inversion-horizon, component, reward-term, and counterfactual analyses support the utility of the residual stream under the evaluated settings, while free-form textual faithfulness and reliability under post-processing remain open limitations.

Key findings

The reconstruction residual stream is a critical component for performance, with its removal leading to the largest performance drop. LaP-Forensics achieves strong cross-generator detection accuracy on UniversalFakeDetect, outperforming several baselines on specific generator families. The structured Where-What-Why reasoning and GRPO-based alignment contribute significantly to improved artifact localization and output coherence.

Approach

LaP-Forensics utilizes a dual-stream approach where one stream processes the RGB image and the other processes a residual map generated from a frozen Stable Diffusion DDIM inversion-reconstruction model. These two streams are encoded and fed into a multimodal large language model (LLM) which predicts both a textual analysis (Where-What-Why) and an artifact mask. The training involves supervised fine-tuning and Group Relative Policy Optimization (GRPO) that rewards mask overlap, output structure, and explicit references to the consistency map.

Datasets

SynthScars, UniversalFakeDetect, LOKI, RichHF, MSCOCOAI, OpenSDI, ProGAN

Model(s)

Stable Diffusion v1.5 (frozen DDIM inversion-reconstruction model), CLIP-ViT-L/14 (frozen encoder), LLaMA-2-7B (language backbone with LoRA adapters), Segment Anything Model (SAM) (decoder)

Author countries

Hong Kong, United Kingdom, China, Singapore

← Previous