← Back to papers
2022 EMNLP EMNLP 2022

TRIPS: Efficient Vision-and-Language Pre-training with Text-Relevant Image Patch Selection

Abstract

AbstractVision Transformers (ViTs) have been widely used in large-scale Vision and Language Pre-training (VLP) models. Though previous VLP works have proved the effectiveness of ViTs, they still suffer from computational efficiency brought by the long visual sequence. To tackle this problem, in this paper, we propose an efficient vision-and-language pre-training model with Text-Relevant Image Patch Selection, namely TRIPS, which reduces the visual sequence progressively with a text-guided patch-selection layer in the visual backbone for efficient training and inference. The patch-selection layer can dynamically compute text-dependent visual attention to identify the attentive image tokens with text guidance and fuse inattentive ones in an end-to-end manner. Meanwhile, TRIPS does not introduce extra parameters to ViTs. Experimental results on a variety of popular benchmark datasets demonstrate that TRIPS gain a speedup of 40% over previous similar VLP models, yet with competitive or better downstream task performance.

🌉 Interdisciplinary Bridge — Artificial Intelligence and Computer Vision and Deep Learning and Machine Learning
📈 Trend Setter — Multimodal Learning
🧭 Keyword Pioneer — image patch selection
🐣 Hot Topic Early Bird — vision language
🐝 Cross-Pollinator — Artificial Intelligence, Computer Science, Computer Vision, Data Science & Analytics, Deep Learning, Healthcare & Medicine, Interdisciplinary, Knowledge & Reasoning, Machine Learning, Mathematics & Optimization, Natural Language Processing, Reinforcement Learning, Robotics, Security & Privacy, Speech & Audio

Authors

Chaoya Jiang , Haiyang Xu , Chenliang Li , Ming Yan , Wei Ye , Shikun Zhang , Bin Bi , Songfang Huang

Topics

Machine Learning > Application Areas > Efficient Computing Deep Learning > Architectures > Transformers Deep Learning > Techniques > Pretraining Artificial Intelligence > Core AI > Efficient Computing Deep Learning > Models > Transformers Machine Learning > Learning Types > Multimodal Learning Computer Vision > Core AI > Efficient Computing Machine Learning > Optimization & Theory > Efficient Computing Deep Learning > Optimization & Theory > Efficient Computing Deep Learning > Application Areas > Efficient Computing Deep Learning > Models > Vision-Language Models

Keywords

vision transformer multimodal learning efficient computing computational efficiency vision-language pre-training patch selection multimodal pretraining vision language text-guided attention image patch selection
Download PDF

Related papers

Bilingual Lexicon Induction for Low-Resource Languages using Graph Matching via Optimal Transport 2022
Generative Entity Typing with Curriculum Learning 2022
Towards Reinterpreting Neural Topic Models via Composite Activations 2022
Weakly Supervised Headline Dependency Parsing 2022
Cross-modal Transfer Between Vision and Language for Protest Detection 2022