2022
EMNLP
EMNLP 2022
Cross-modal Transfer Between Vision and Language for Protest Detection
Abstract
AbstractMost of today’s systems for socio-political event detection are text-based, while an increasing amount of information published on the web is multi-modal. We seek to bridge this gap by proposing a method that utilizes existing annotated unimodal data to perform event detection in another data modality, zero-shot. Specifically, we focus on protest detection in text and images, and show that a pretrained vision-and-language alignment model (CLIP) can be leveraged towards this end. In particular, our results suggest that annotated protest text data can act supplementarily for detecting protests in images, but significant transfer is demonstrated in the opposite direction as well.
🌉
Interdisciplinary Bridge
— Artificial Intelligence and Computer Vision and Deep Learning and Machine Learning
🐣
Hot Topic Early Bird
— vision language model
🐝
Cross-Pollinator
— Artificial Intelligence, Computer Science, Computer Vision, Data Science & Analytics, Deep Learning, Healthcare & Medicine, Interdisciplinary, Knowledge & Reasoning, Machine Learning, Mathematics & Optimization, Natural Language Processing, Reinforcement Learning, Robotics, Security & Privacy, Speech & Audio
Authors
Topics
Artificial Intelligence > Core AI > Multimodal Learning
Artificial Intelligence > Learning Paradigms > Transfer Learning
Machine Learning > Learning Types > Zero-Shot Learning
Computer Vision > Core AI > Multimodal Learning
Deep Learning > Learning Types > Multi-Modal Learning
Deep Learning > Learning Types > Zero-Shot Learning
Deep Learning > Models > Vision-Language Models