← Back to papers
2023 CVPR CVPR 2023

Non-Contrastive Learning Meets Language-Image Pre-Training

Abstract

Contrastive language-image pre-training (CLIP) serves as a de-facto standard to align images and texts. Nonetheless, the loose correlation between images and texts of web-crawled data renders the contrastive objective data inefficient and craving for a large training batch size. In this work, we explore the validity of non-contrastive language-image pre-training (nCLIP) and study whether nice properties exhibited in visual self-supervised models can emerge. We empirically observe that the non-contrastive objective nourishes representation learning while sufficiently underperforming under zero-shot recognition. Based on the above study, we further introduce xCLIP, a multi-tasking framework combining CLIP and nCLIP, and show that nCLIP aids CLIP in enhancing feature semantics. The synergy between two objectives lets xCLIP enjoy the best of both worlds: superior performance in both zero-shot transfer and representation learning. Systematic evaluation is conducted spanning a wide variety of downstream tasks including zero-shot classification, out-of-domain classification, retrieval, visual representation learning, and textual representation learning, showcasing a consistent performance gain and validating the effectiveness of xCLIP.

🌉 Interdisciplinary Bridge — Artificial Intelligence and Deep Learning and Machine Learning
🐝 Cross-Pollinator — Artificial Intelligence, Computer Science, Computer Vision, Data Science & Analytics, Deep Learning, Healthcare & Medicine, Interdisciplinary, Knowledge & Reasoning, Machine Learning, Mathematics & Optimization, Natural Language Processing, Reinforcement Learning, Robotics, Security & Privacy, Speech & Audio

Authors

Jinghao Zhou , Li Dong , Zhe Gan , Lijuan Wang , Furu Wei

Topics

Artificial Intelligence > Core AI > Multimodal Learning Machine Learning > Learning Types > Self-Supervised Learning Deep Learning > Learning Types > Self-Supervised Learning Deep Learning > Learning Types > Contrastive Learning Deep Learning > Learning Types > Multi-Modal Learning Deep Learning > Learning Types > Zero-Shot Learning

Keywords

representation learning zero-shot learning self-supervised learning multimodal learning non-contrastive learning zero-shot recognition image-text pre-training
Download PDF

Related papers

CORA: Adapting CLIP for Open-Vocabulary Detection With Region Prompting and Anchor Pre-Matching 2023
3DAvatarGAN: Bridging Domains for Personalized Editable Avatars 2023
Physics-Driven Diffusion Models for Impact Sound Synthesis From Videos 2023
Transductive Few-Shot Learning With Prototype-Based Label Propagation by Iterative Graph Refinement 2023
EXIF As Language: Learning Cross-Modal Associations Between Images and Camera Metadata 2023