Contrastive Learning of Global-Local Video Representations

Authors: Shuang Ma, Zhaoyang Zeng, Daniel McDuff, Yale Song

Published: 2021-04-07 07:35:08+00:00

AI Summary

This paper proposes a contrastive learning approach for learning video representations that generalize to both global and local tasks. It achieves this by optimizing two contrastive objectives that encourage the model to learn global-local visual information from audio signals, significantly outperforming models trained with disjoint objectives.

Abstract

Contrastive learning has delivered impressive results for various tasks in the self-supervised regime. However, existing approaches optimize for learning representations specific to downstream scenarios, i.e., textit{global} representations suitable for tasks such as classification or textit{local} representations for tasks such as detection and localization. While they produce satisfactory results in the intended downstream scenarios, they often fail to generalize to tasks that they were not originally designed for. In this work, we propose to learn video representations that generalize to both the tasks which require global semantic information (e.g., classification) and the tasks that require local fine-grained spatio-temporal information (e.g., localization). We achieve this by optimizing two contrastive objectives that together encourage our model to learn global-local visual information given audio signals. We show that the two objectives mutually improve the generalizability of the learned global-local representations, significantly outperforming their disjointly learned counterparts. We demonstrate our approach on various tasks including action/sound classification, lip reading, deepfake detection, event and sound localization (https://github.com/yunyikristy/global_local).