← Back to papers
2023 INTERSPEECH INTERSPEECH 2023

HierVST: Hierarchical Adaptive Zero-shot Voice Style Transfer

Abstract

Despite rapid progress in the voice style transfer (VST) field, recent zero-shot VST systems still lack the ability to transfer the voice style of a novel speaker. In this paper, we present HierVST, a hierarchical adaptive end-to-end zero-shot VST model. Without any text transcripts, we only use the speech dataset to train the model by utilizing hierarchical variational inference and self-supervised representation. In addition, we adopt a hierarchical adaptive generator that generates the pitch representation and waveform audio sequentially. Moreover, we utilize unconditional generation to improve the speaker-relative acoustic capacity in the acoustic representation. With a hierarchical adaptive structure, the model can adapt to a novel voice style and convert speech progressively. The experimental results demonstrate that our method outperforms other VST models in zero-shot VST scenarios. Audio samples are available at https://hiervst.github.io/.

🌉 Interdisciplinary Bridge — Artificial Intelligence and Deep Learning and Machine Learning and Speech & Audio
🐝 Cross-Pollinator — Artificial Intelligence, Computer Science, Computer Vision, Data Science & Analytics, Deep Learning, Healthcare & Medicine, Interdisciplinary, Knowledge & Reasoning, Machine Learning, Mathematics & Optimization, Natural Language Processing, Reinforcement Learning, Robotics, Security & Privacy, Speech & Audio

Authors

Sang-Hoon Lee , Ha-Yeong Choi , Hyung-Seok Oh , Seong-Whan Lee

Topics

Machine Learning > Learning Types > Self-Supervised Learning Deep Learning > Models > Variational Inference Speech & Audio > Synthesis > Text-to-Speech Artificial Intelligence > Learning Paradigms > Zero-Shot Learning Machine Learning > Learning Paradigms > Zero-Shot Learning Deep Learning > Learning Types > Self-Supervised Learning

Keywords

zero-shot learning variational inference self-supervised learning hierarchical model self-supervised representation voice style transfer
Download PDF

Related papers

Multimodal Turn-Taking Model Using Visual Cues for End-of-Utterance Prediction in Spoken Dialogue Systems 2023
Audio-Visual Praise Estimation for Conversational Video based on Synchronization-Guided Multimodal Transformer 2023
Improving the response timing estimation for spoken dialogue systems by reducing the effect of speech recognition delay 2023
Improving Code-Switching and Name Entity Recognition in ASR with Speech Editing based Data Augmentation 2023
What are differences? Comparing DNN and Human by Their Performance and Characteristics in Speaker Age Estimation 2023