← Back to papers
2022 ACL ACL 2022

Voxel-informed Language Grounding

Abstract

AbstractNatural language applied to natural 2D images describes a fundamentally 3D world. We present the Voxel-informed Language Grounder (VLG), a language grounding model that leverages 3D geometric information in the form of voxel maps derived from the visual input using a volumetric reconstruction model. We show that VLG significantly improves grounding accuracy on SNARE, an object reference game task. At the time of writing, VLG holds the top place on the SNARE leaderboard, achieving SOTA results with a 2.0% absolute improvement.

🌉 Interdisciplinary Bridge — Artificial Intelligence and Computer Vision and Deep Learning
🧭 Keyword Pioneer — voxel map
🐝 Cross-Pollinator — Artificial Intelligence, Computer Science, Computer Vision, Data Science & Analytics, Deep Learning, Healthcare & Medicine, Interdisciplinary, Knowledge & Reasoning, Machine Learning, Mathematics & Optimization, Natural Language Processing, Reinforcement Learning, Robotics, Security & Privacy, Speech & Audio

Authors

Rodolfo Corona , Shizhan Zhu , Dan Klein , Trevor Darrell

Topics

Artificial Intelligence > Core AI > Multimodal Learning Artificial Intelligence > Core AI > Multi-Modal Learning Computer Vision > Domain-Specific > Robotics Deep Learning > Models > Vision-Language Models

Keywords

3d reconstruction multimodal learning 3d vision visual reasoning language grounding volumetric reconstruction object reference voxel map
Download PDF

Related papers

HIE-SQL: History Information Enhanced Network for Context-Dependent Text-to-SQL Semantic Parsing 2022
KG-CRuSE: Recurrent Walks over Knowledge Graph for Explainable Conversation Reasoning using Semantic Embeddings 2022
Toward Knowledge-Enriched Conversational Recommendation Systems 2022
Investigating the Medical Coverage of a Translation System into Pictographs for Patients with an Intellectual Disability 2022
TableFormer: Robust Transformer Modeling for Table-Text Encoding 2022