The eyes know it: FakeET -- An Eye-tracking Database to Understand Deepfake Perception

View on arXiv ← Back to list

Authors: Parul Gupta, Komal Chugh, Abhinav Dhall, Ramanathan Subramanian

Published: 2020-06-12 06:14:49+00:00

AI Summary

The paper introduces FakeET, an eye-tracking database comprising eye movement and EEG data from 40 users viewing deepfake videos. This dataset reveals distinct viewing patterns between real and fake videos, demonstrating the potential of using human behavioral cues for deepfake detection.

Abstract

We present textbf{FakeET}-- an eye-tracking database to understand human visual perception of emph{deepfake} videos. Given that the principal purpose of deepfakes is to deceive human observers, FakeET is designed to understand and evaluate the ease with which viewers can detect synthetic video artifacts. FakeET contains viewing patterns compiled from 40 users via the emph{Tobii} desktop eye-tracker for 811 videos from the textit{Google Deepfake} dataset, with a minimum of two viewings per video. Additionally, EEG responses acquired via the emph{Emotiv} sensor are also available. The compiled data confirms (a) distinct eye movement characteristics for emph{real} vs emph{fake} videos; (b) utility of the eye-track saliency maps for spatial forgery localization and detection, and (c) Error Related Negativity (ERN) triggers in the EEG responses, and the ability of the emph{raw} EEG signal to distinguish between emph{real} and emph{fake} videos.

Key findings

Significant differences in eye movement patterns (more fixations and longer scan paths for real videos) and EEG responses (ERN triggers for fake videos) were found. Augmenting CNN input with eye-gaze maps improved deepfake detection accuracy. EEG-based classification achieved better-than-chance performance in detecting deepfakes.

Approach

The research uses eye-tracking and EEG data collected while participants viewed deepfake videos. They analyze differences in gaze patterns (fixations, scan paths, entropy) and EEG responses (ERN) between real and fake videos. A CNN is trained using gaze maps to improve deepfake detection performance.

Datasets

Google Deepfake Detection dataset (811 videos used in the study), FakeET (eye-tracking and EEG data from 40 users).

Model(s)

3D ResNet-based CNN, Naive Bayes, Logistic Regression, k-Nearest Neighbors, Decision Tree, Linear SVM, Time-series CNN.

Author countries

India, Australia

← Previous