From Pixels to Portraits: A Comprehensive Survey of Talking Head Generation Techniques and Applications

Authors: Shreyank N Gowda, Dheeraj Pandey, Shashank Narayana Gowda

Published: 2023-08-30 14:00:48+00:00

AI Summary

This survey provides a critical review of talking head generation techniques, classifying them into image-driven, audio-driven, video-driven, and 3D/neural-rendering-based approaches. It analyzes the gap between quantitative metrics and perceptual quality, compares publicly available models, and examines emerging trends and open challenges in the field. The paper also highlights the growing importance of provenance, watermarking, and deepfake detection.

Abstract

Talking head generation has progressed rapidly from landmark- and GAN-based facial animation to diffusion models, neural rendering, 3D-aware avatars, and foundation-model-assisted systems. This progress has enabled increasingly realistic audio-, image-, and video-driven talking heads, but it has also made the field difficult to navigate because methods differ substantially in their inputs, assumptions, controllability, temporal stability, computational cost, and risks of misuse. This survey provides a critical review of talking head generation techniques, organizing the literature into four broad families: image-driven, audio-driven, video-driven, and 3D/neural-rendering-based approaches. For each family, we discuss the underlying technical ideas, representative methods, strengths, limitations, datasets, and evaluation practices. Beyond cataloguing prior work, we analyse the persistent gap between commonly reported quantitative metrics and perceptual quality, and compare publicly available models in terms of inference time, memory requirements, and human-rated visual quality. We also examine emerging trends, including diffusion-based generation, 3D-aware representation learning, controllable emotional expression, real-time deployment, and the growing importance of provenance, watermarking, and deepfake detection. Finally, we identify open challenges around robust evaluation, identity preservation, lip synchronisation, temporal consistency, demographic fairness, computational efficiency, and responsible use. This review aims to provide researchers and practitioners with a structured and up-to-date map of the talking head generation landscape, while highlighting the technical and societal questions that should shape future work.

Key findings

The field has evolved rapidly, incorporating diffusion models, neural rendering, and 3D-aware avatars, often combining multiple input modalities. There is a persistent mismatch between commonly reported quantitative metrics and human perceptual quality, necessitating a multi-dimensional evaluation approach. Responsible development, including consent, provenance, watermarking, and deepfake detection, is crucial due to the potential for misuse of generated media.

Approach

This paper is a survey, so it provides a comprehensive review of existing talking head generation techniques rather than presenting a novel approach. It categorizes methods based on their driving signals (image, audio, video, 3D), internal representations, and generation paradigms, discussing technical ideas, models, datasets, and evaluation practices for each.

Datasets

GRID, CREMA-D, MSP-Improv, LRW, ObamaSet, VoxCeleb1, VoxCeleb2, LRW-1000, FaceForensics++, MEAD, HDTF, TalkingHead-1KH, VFHQ, CelebV-HQ, Multiface, CelebV-Text, TalkVid, SpeakerVid-5M

Model(s)

Face2Face, ReenactGAN, NeuralHead, MarioNETte, X2Face, PIRenderer, FD-GAN-LS, StyleMask, AVFR, FOMM, DAM, TPSM, CrossID-GAN, DaGAN/DaGAN++, LivePortrait, StyleHEAT, MegaPortrait, LipFormer, Synthesizing-Obama, Wav2Lip, TalkLip, ATVGNet, MakeItTalk, NeuralVoicePuppetry, LSP, Audio2Head, Speech2TalkingFace, SadTalker, EVP, GC-AVT, EAMM, EmoGen, VASA-1, Dimitra, Teller, READ, GaussianEmoTalker, EMO, Hallo, Hallo2, Hallo3, OmniHuman-1, MeshG, HeadGAN, HifiHead, CoRF, PECHead, IMavatar, ROME, Real3D-Portrait, NerFACE, HeadNeRF, HiDe-NeRF, GaussianAvatars, SplattingAvatar, LAM, GaussianSpeech, PGSTalker, UniGAHA, VASA-3D, Write-a-speaker, TalkCLIP, EditYourself

Author countries

United Kingdom, UNKNOWN

← Previous