← Back to papers
2023 EMNLP EMNLP 2023

ImageNetVC: Zero- and Few-Shot Visual Commonsense Evaluation on 1000 ImageNet Categories

Abstract

AbstractRecently, Large Language Models (LLMs) have been serving as general-purpose interfaces, posing a significant demand for comprehensive visual knowledge. However, it remains unclear how well current LLMs and their visually augmented counterparts (VaLMs) can master visual commonsense knowledge. To investigate this, we propose ImageNetVC, a human-annotated dataset specifically designed for zero- and few-shot visual commonsense evaluation across 1,000 ImageNet categories. Utilizing ImageNetVC, we benchmark the fundamental visual commonsense knowledge of both unimodal LLMs and VaLMs. Furthermore, we analyze the factors affecting the visual commonsense knowledge of large-scale models, providing insights into the development of language models enriched with visual commonsense knowledge. Our code and dataset are available at https://github.com/hemingkx/ImageNetVC.

🌉 Interdisciplinary Bridge — Artificial Intelligence and Deep Learning and Machine Learning
🐝 Cross-Pollinator — Artificial Intelligence, Computer Science, Computer Vision, Data Science & Analytics, Deep Learning, Healthcare & Medicine, Interdisciplinary, Knowledge & Reasoning, Machine Learning, Mathematics & Optimization, Natural Language Processing, Reinforcement Learning, Robotics, Security & Privacy, Speech & Audio

Authors

Heming Xia , Qingxiu Dong , Lei Li , Jingjing Xu , Tianyu Liu , Ziwei Qin , Zhifang Sui

Topics

Artificial Intelligence > Core AI > Foundation Models Artificial Intelligence > Core AI > Multimodal Learning Machine Learning > Learning Paradigms > Few-Shot Learning Deep Learning > Models > Large Language Models Machine Learning > Learning Paradigms > Zero-Shot Learning Deep Learning > Learning Types > Zero-Shot Learning Deep Learning > Learning Types > Few-Shot Learning Deep Learning > Learning Types > Multimodal Learning

Keywords

zero-shot learning few-shot learning multimodal learning visual commonsense large language model visual knowledge
Download PDF

Related papers

Exploring Linguistic Probes for Morphological Generalization 2023
NameGuess: Column Name Expansion for Tabular Data 2023
Vision-Enhanced Semantic Entity Recognition in Document Images via Visually-Asymmetric Consistency Learning 2023
Improving Conversational Recommendation Systems via Bias Analysis and Language-Model-Enhanced Data Augmentation 2023
On the Calibration of Large Language Models and Alignment 2023