← Back to papers
2017 CVPR CVPR 2017

Multi-View Supervision for Single-View Reconstruction via Differentiable Ray Consistency

Abstract

We study the notion of consistency between a 3D shape and a 2D observation and propose a differentiable formulation which allows computing gradients of the 3D shape given an observation from an arbitrary view. We do so by reformulating view consistency using a differentiable ray consistency (DRC) term. We show that this formulation can be incorporated in a learning framework to leverage different types of multi-view observations e.g. foreground masks, depth, color images, semantics etc. as supervision for learning single-view 3D prediction. We present empirical analysis of our technique in a controlled setting. We also show that this approach allows us to improve over existing techniques for single-view reconstruction of objects from the PASCAL VOC dataset.

🌉 Interdisciplinary Bridge — Computer Vision and Deep Learning and Machine Learning
📈 Trend Setter — Multi-View Learning
🧭 Keyword Pioneer — multi-view supervision
🐣 Hot Topic Early Bird — multi-view learning
🐝 Cross-Pollinator — Artificial Intelligence, Computer Science, Computer Vision, Data Science & Analytics, Deep Learning, Healthcare & Medicine, Interdisciplinary, Knowledge & Reasoning, Machine Learning, Mathematics & Optimization, Natural Language Processing, Reinforcement Learning, Robotics, Security & Privacy, Speech & Audio

Authors

Shubham Tulsiani , Tinghui Zhou , Alexei A. Efros , Jitendra Malik

Topics

Machine Learning > Learning Types > Self-Supervised Learning Machine Learning > Learning Types > Semi-Supervised Learning Computer Vision > Analysis > 3D Vision Deep Learning > Optimization & Theory > Theory Machine Learning > Learning Types > Multi-View Learning

Keywords

3d reconstruction multi-view learning view consistency differentiable rendering geometric consistency 3d shape single-view reconstruction multi-view supervision single-view prediction differentiable ray consistency
Download PDF

Related papers

Deep Outdoor Illumination Estimation 2017
SRN: Side-output Residual Network for Object Symmetry Detection in the Wild 2017
Weakly Supervised Semantic Segmentation Using Web-Crawled Videos 2017
FASON: First and Second Order Information Fusion Network for Texture Recognition 2017
Recurrent Convolutional Neural Networks for Continuous Sign Language Recognition by Staged Optimization 2017