← Back to papers
2018 EMNLP EMNLP 2018

TVQA: Localized, Compositional Video Question Answering

Abstract

AbstractRecent years have witnessed an increasing interest in image-based question-answering (QA) tasks. However, due to data limitations, there has been much less work on video-based QA. In this paper, we present TVQA, a large-scale video QA dataset based on 6 popular TV shows. TVQA consists of 152,545 QA pairs from 21,793 clips, spanning over 460 hours of video. Questions are designed to be compositional in nature, requiring systems to jointly localize relevant moments within a clip, comprehend subtitle-based dialogue, and recognize relevant visual concepts. We provide analyses of this new dataset as well as several baselines and a multi-stream end-to-end trainable neural network framework for the TVQA task. The dataset is publicly available at http://tvqa.cs.unc.edu.

🌉 Interdisciplinary Bridge — Artificial Intelligence and Computer Vision and Deep Learning and Natural Language Processing
📈 Trend Setter — Visual Question Answering
🧭 Keyword Pioneer — dialogue comprehension
🐣 Hot Topic Early Bird — temporal reasoning
🐝 Cross-Pollinator — Artificial Intelligence, Computer Science, Computer Vision, Data Science & Analytics, Deep Learning, Healthcare & Medicine, Interdisciplinary, Knowledge & Reasoning, Machine Learning, Mathematics & Optimization, Natural Language Processing, Reinforcement Learning, Robotics, Security & Privacy, Speech & Audio

Authors

Jie Lei , Licheng Yu , Mohit Bansal , Tamara Berg

Topics

Deep Learning > Architectures > Neural Networks Natural Language Processing > Applications > Question Answering Deep Learning > Learning Types > Multi-Modal Learning Artificial Intelligence > Core AI > Multi-Modal Learning Computer Vision > Applications > Visual Question Answering

Keywords

visual question answering temporal reasoning multimodal learning multi-modal learning video understanding video question answering dialogue understanding neural network dialogue comprehension
Download PDF

Related papers

Speeding Up Neural Machine Translation Decoding by Cube Pruning 2018
Limitations in learning an interpreted language with recurrent models 2018
Results of the sixth edition of the BioASQ Challenge 2018
Neural Segmental Hypergraphs for Overlapping Mention Recognition 2018
Hybrid Neural Attention for Agreement/Disagreement Inference in Online Debates 2018