Beyond Detection: Visual Realism Assessment of Deepfakes

View on arXiv ← Back to list

Authors: Luka Dragar, Peter Peer, Vitomir Štruc, Borut Batagelj

Published: 2023-06-09 15:53:01+00:00

AI Summary

This paper proposes a method for assessing the visual realism of deepfake videos using an ensemble of two CNN models, Eva and ConvNext, trained on the DeepFake Game Competition (DFGC) 2022 dataset to predict Mean Opinion Scores (MOS). This approach achieved third place in the DFGC 2023 Visual Realism Assessment competition.

Abstract

In the era of rapid digitalization and artificial intelligence advancements, the development of DeepFake technology has posed significant security and privacy concerns. This paper presents an effective measure to assess the visual realism of DeepFake videos. We utilize an ensemble of two Convolutional Neural Network (CNN) models: Eva and ConvNext. These models have been trained on the DeepFake Game Competition (DFGC) 2022 dataset and aim to predict Mean Opinion Scores (MOS) from DeepFake videos based on features extracted from sequences of frames. Our method secured the third place in the recent DFGC on Visual Realism Assessment held in conjunction with the 2023 International Joint Conference on Biometrics (IJCB 2023). We provide an over-view of the models, data preprocessing, and training procedures. We also report the performance of our models against the competition's baseline model and discuss the implications of our findings.

Key findings

The ensemble model achieved a third-place ranking in the DFGC-VRA 2023 competition. The Eva model, despite initially showing lower performance during training, exhibited superior generalization capabilities on test sets compared to ConvNext. The weighted ensemble of both models resulted in improved performance over either model alone.

Approach

The authors use an ensemble of two CNN models, Eva and ConvNext, trained on the DFGC 2022 dataset. These models predict Mean Opinion Scores (MOS) based on features extracted from sequences of five frames. A weighted average of the two models' predictions is used for the final result.

Datasets

DeepFake Game Competition (DFGC) 2022 dataset

Model(s)

Eva (Vision Transformer), ConvNext

Author countries

Slovenia

← Previous