← Back to papers
2022 ACL ACL 2022

How does fake news use a thumbnail? CLIP-based Multimodal Detection on the Unrepresentative News Image

Abstract

AbstractThis study investigates how fake news use the thumbnail image for a news article. We aim at capturing the degree of semantic incongruity between news text and image by using the pretrained CLIP representation. Motivated by the stylistic distinctiveness in fake news text, we examine whether fake news tends to use an irrelevant image to the news content. Results show that fake news tends to have a high degree of semantic incongruity than general news. We further attempt to detect such image-text incongruity by training classification models on a newly generated dataset. A manual evaluation suggests our method can find news articles of which the thumbnail image is semantically irrelevant to news text with an accuracy of 0.8. We also release a new dataset of image and news text pairs with the incongruity label, facilitating future studies on the direction.

❓ The Questioner
🌉 Interdisciplinary Bridge — Artificial Intelligence and Deep Learning and Machine Learning
🧭 Keyword Pioneer — clip vision-language
🐣 Hot Topic Early Bird — image-text matching
🐝 Cross-Pollinator — Artificial Intelligence, Computer Science, Computer Vision, Data Science & Analytics, Deep Learning, Healthcare & Medicine, Interdisciplinary, Knowledge & Reasoning, Machine Learning, Mathematics & Optimization, Natural Language Processing, Reinforcement Learning, Robotics, Security & Privacy, Speech & Audio

Authors

Hyewon Choi , Yejun Yoon , Seunghyun Yoon , Kunwoo Park

Topics

Artificial Intelligence > Core AI > Multimodal Learning Artificial Intelligence > Learning Paradigms > Transfer Learning Machine Learning > Learning Types > Transfer Learning Deep Learning > Learning Types > Multimodal Learning Artificial Intelligence > Core AI > Multi-Modal Learning

Keywords

transfer learning multimodal learning semantic similarity fake news detection image-text matching multimodal detection clip vision-language semantic incongruity image-text incongruity
Download PDF

Related papers

HIE-SQL: History Information Enhanced Network for Context-Dependent Text-to-SQL Semantic Parsing 2022
KG-CRuSE: Recurrent Walks over Knowledge Graph for Explainable Conversation Reasoning using Semantic Embeddings 2022
Toward Knowledge-Enriched Conversational Recommendation Systems 2022
Investigating the Medical Coverage of a Translation System into Pictographs for Patients with an Intellectual Disability 2022
TableFormer: Robust Transformer Modeling for Table-Text Encoding 2022