← Back to papers
2017 CVPR CVPR 2017

Gaze Embeddings for Zero-Shot Image Classification

Abstract

Zero-shot image classification using auxiliary information, such as attributes describing discriminative object properties, requires time-consuming annotation by domain experts. We instead propose a method that relies on human gaze as auxiliary information, exploiting that even non-expert users have a natural ability to judge class membership. We present a data collection paradigm that involves a discrimination task to increase the information content obtained from gaze data. Our method extracts discriminative descriptors from the data and learns a compatibility function between image and gaze using three novel gaze embeddings: Gaze Histograms (GH), Gaze Features with Grid (GFG) and Gaze Features with Sequence (GFS). We introduce two new gaze-annotated datasets for fine-grained image classification and show that human gaze data is indeed class discriminative, provides a competitive alternative to expert-annotated attributes, and outperforms other baselines for zero-shot image classification.

🌉 Interdisciplinary Bridge — Artificial Intelligence and Computer Vision and Deep Learning and Machine Learning
📈 Trend Setter — Zero-Shot Learning
🧭 Keyword Pioneer — gaze embedding
🐣 Hot Topic Early Bird — zero-shot learning
🐝 Cross-Pollinator — Artificial Intelligence, Computer Science, Computer Vision, Data Science & Analytics, Deep Learning, Healthcare & Medicine, Interdisciplinary, Knowledge & Reasoning, Machine Learning, Mathematics & Optimization, Natural Language Processing, Reinforcement Learning, Robotics, Security & Privacy, Speech & Audio

Authors

Nour Karessli , Zeynep Akata , Bernt Schiele , Andreas Bulling

Topics

Machine Learning > Core Methods > Metric Learning Machine Learning > Core Methods > Embedding Learning Machine Learning > Learning Types > Zero-Shot Learning Artificial Intelligence > Learning Paradigms > Zero-Shot Learning Computer Vision > Core AI > Computer Vision Deep Learning > Learning Types > Zero-Shot Learning

Keywords

image classification zero-shot learning fine-grained classification human gaze attribute learning zero-shot classification compatibility function gaze embedding
Download PDF

Related papers

Deep Outdoor Illumination Estimation 2017
SRN: Side-output Residual Network for Object Symmetry Detection in the Wild 2017
Weakly Supervised Semantic Segmentation Using Web-Crawled Videos 2017
FASON: First and Second Order Information Fusion Network for Texture Recognition 2017
Recurrent Convolutional Neural Networks for Continuous Sign Language Recognition by Staged Optimization 2017