← Back to papers
2020 CVPR CVPR 2020

Optical Non-Line-of-Sight Physics-Based 3D Human Pose Estimation

Abstract

We describe a method for 3D human pose estimation from transient images (i.e., a 3D spatio-temporal histogram of photons) acquired by an optical non-line-of-sight (NLOS) imaging system. Our method can perceive 3D human pose by 'looking around corners' through the use of light indirectly reflected by the environment. We bring together a diverse set of technologies from NLOS imaging, human pose estimation and deep reinforcement learning to construct an end-to-end data processing pipeline that converts a raw stream of photon measurements into a full 3D human pose sequence estimate. Our contributions are the design of data representation process which includes (1) a learnable inverse point spread function (PSF) to convert raw transient images into a deep feature vector; (2) a neural humanoid control policy conditioned on the transient image feature and learned from interactions with a physics simulator; and (3) a data synthesis and augmentation strategy based on depth data that can be transferred to a real-world NLOS imaging system. Our preliminary experiments suggest that our method is able to generalize to real-world NLOS measurement to estimate physically-valid 3D human poses.

🌉 Interdisciplinary Bridge — Computer Vision and Deep Learning and Machine Learning and Reinforcement Learning
🧭 Keyword Pioneer — neural control policy
🐣 Hot Topic Early Bird — 3d human pose estimation
🐝 Cross-Pollinator — Artificial Intelligence, Computer Science, Computer Vision, Data Science & Analytics, Deep Learning, Healthcare & Medicine, Interdisciplinary, Knowledge & Reasoning, Machine Learning, Mathematics & Optimization, Natural Language Processing, Reinforcement Learning, Robotics, Security & Privacy, Speech & Audio

Authors

Mariko Isogawa , Ye Yuan , Matthew O'Toole , Kris M. Kitani

Topics

Machine Learning > Core Methods > Representation Learning Computer Vision > Analysis > 3D Vision Computer Vision > Analysis > Human Pose Estimation Reinforcement Learning > Methods > Deep RL Deep Learning > Learning Types > Reinforcement Learning Deep Learning > Learning Types > Representation Learning

Keywords

deep reinforcement learning reinforcement learning human pose estimation 3d human pose estimation physics simulation non-line-of-sight imaging neural network neural control policy transient image processing physics-based learning inverse point spread function neural humanoid control
Download PDF

Related papers

Deep Polarization Cues for Transparent Object Segmentation 2020
HRank: Filter Pruning Using High-Rank Feature Map 2020
Panoptic-Based Image Synthesis 2020
Select, Supplement and Focus for RGB-D Saliency Detection 2020
ClusterVO: Clustering Moving Instances and Estimating Visual Odometry for Self and Surroundings 2020