← Back to papers
2023 CVPR CVPR 2023

Rethinking Video ViTs: Sparse Video Tubes for Joint Image and Video Learning

Abstract

We present a simple approach which can turn a ViT encoder into an efficient video model, which can seamlessly work with both image and video inputs. By sparsely sampling the inputs, the model is able to do training and inference from both inputs. The model is easily scalable and can be adapted to large-scale pre-trained ViTs without requiring full finetuning. The model achieves SOTA results.

🌉 Interdisciplinary Bridge — Computer Vision and Deep Learning
🐝 Cross-Pollinator — Artificial Intelligence, Computer Science, Computer Vision, Data Science & Analytics, Deep Learning, Healthcare & Medicine, Interdisciplinary, Knowledge & Reasoning, Machine Learning, Mathematics & Optimization, Natural Language Processing, Reinforcement Learning, Robotics, Security & Privacy, Speech & Audio

Authors

AJ Piergiovanni , Weicheng Kuo , Anelia Angelova

Topics

Deep Learning > Architectures > Transformers Computer Vision > Processing > Video Understanding Computer Vision > Core AI > Multimodal Learning Deep Learning > Learning Types > Self-Supervised Learning Deep Learning > Learning Types > Transfer Learning

Keywords

vision transformer transfer learning self-supervised learning multimodal learning video understanding video representation masked autoencoding sparse sampling
Download PDF

Related papers

CORA: Adapting CLIP for Open-Vocabulary Detection With Region Prompting and Anchor Pre-Matching 2023
3DAvatarGAN: Bridging Domains for Personalized Editable Avatars 2023
Physics-Driven Diffusion Models for Impact Sound Synthesis From Videos 2023
Transductive Few-Shot Learning With Prototype-Based Label Propagation by Iterative Graph Refinement 2023
EXIF As Language: Learning Cross-Modal Associations Between Images and Camera Metadata 2023