← Back to papers
2018 NIPS NeurIPS 2018

Self-Supervised Generation of Spatial Audio for 360° Video

Abstract

We introduce an approach to convert mono audio recorded by a 360° video camera into spatial audio, a representation of the distribution of sound over the full viewing sphere. Spatial audio is an important component of immersive 360° video viewing, but spatial audio microphones are still rare in current 360° video production. Our system consists of end-to-end trainable neural networks that separate individual sound sources and localize them on the viewing sphere, conditioned on multi-modal analysis from the audio and 360° video frames. We introduce several datasets, including one filmed ourselves, and one collected in-the-wild from YouTube, consisting of 360° videos uploaded with spatial audio. During training, ground truth spatial audio serves as self-supervision and a mixed down mono track forms the input to our network. Using our approach we show that it is possible to infer the spatial localization of sounds based only on a synchronized 360° video and the mono audio track.

🌉 Interdisciplinary Bridge — Artificial Intelligence and Deep Learning and Machine Learning and Speech & Audio
📈 Trend Setter — Speech Enhancement
🐣 Hot Topic Early Bird — source separation
🐝 Cross-Pollinator — Artificial Intelligence, Computer Science, Computer Vision, Data Science & Analytics, Deep Learning, Healthcare & Medicine, Interdisciplinary, Knowledge & Reasoning, Machine Learning, Mathematics & Optimization, Natural Language Processing, Reinforcement Learning, Robotics, Security & Privacy, Speech & Audio

Authors

Pedro Morgado , Nuno Nvasconcelos , Timothy Langlois , Oliver Wang

Topics

Artificial Intelligence > Core AI > Multimodal Learning Machine Learning > Learning Types > Self-Supervised Learning Deep Learning > Models > Generative Models Deep Learning > Learning Types > Self-Supervised Learning Deep Learning > Learning Types > Multi-Modal Learning Speech & Audio > Processing > Speech Enhancement

Keywords

source separation self-supervised learning multi-modal learning spatial audio neural network
Download PDF

Related papers

Maximum Causal Tsallis Entropy Imitation Learning 2018
Recurrent World Models Facilitate Policy Evolution 2018
Bandit Learning in Concave N-Person Games 2018
Algorithmic Assurance: An Active Approach to Algorithmic Testing using Bayesian Optimisation 2018
PAC-Bayes bounds for stable algorithms with instance-dependent priors 2018