← Back to papers
2023 INTERSPEECH INTERSPEECH 2023

Comparison of Multilingual Self-Supervised and Weakly-Supervised Speech Pre-Training for Adaptation to Unseen Languages

Abstract

Recent models such as XLS-R and Whisper have made multilingual speech technologies more accessible by pre-training on audio from around 100 spoken languages each. However, there are thousands of spoken languages worldwide, and adapting to new languages is an important problem. In this work, we aim to understand which model adapts better to languages unseen during pre-training. We fine-tune both models on 13 unseen languages and 18 seen languages. Our results show that the number of hours seen per language and language family during pre-training is predictive of how the models compare, despite the significant differences in the pre-training methods.

🌉 Interdisciplinary Bridge — Artificial Intelligence and Machine Learning and Natural Language Processing and Speech & Audio
🐣 Hot Topic Early Bird — multilingual speech
🐝 Cross-Pollinator — Artificial Intelligence, Computer Science, Computer Vision, Data Science & Analytics, Deep Learning, Healthcare & Medicine, Interdisciplinary, Knowledge & Reasoning, Machine Learning, Mathematics & Optimization, Natural Language Processing, Reinforcement Learning, Robotics, Security & Privacy, Speech & Audio

Authors

Andrew Rouditchenko , Sameer Khurana , Samuel Thomas , Rogerio Feris , Leonid Karlinsky , Hilde Kuehne , David Harwath , Brian Kingsbury , James Glass

Topics

Artificial Intelligence > Learning Paradigms > Transfer Learning Machine Learning > Learning Types > Self-Supervised Learning Machine Learning > Learning Types > Weakly Supervised Learning Natural Language Processing > Resources & Methods > Multilingual NLP Speech & Audio > Recognition > Automatic Speech Recognition Machine Learning > Learning Types > Transfer Learning Machine Learning > Learning Paradigms > Self-Supervised Learning

Keywords

transfer learning self-supervised learning weakly supervised learning weakly-supervised learning multilingual speech language adaptation unseen language
Download PDF

Related papers

Multimodal Turn-Taking Model Using Visual Cues for End-of-Utterance Prediction in Spoken Dialogue Systems 2023
Audio-Visual Praise Estimation for Conversational Video based on Synchronization-Guided Multimodal Transformer 2023
Improving the response timing estimation for spoken dialogue systems by reducing the effect of speech recognition delay 2023
Improving Code-Switching and Name Entity Recognition in ASR with Speech Editing based Data Augmentation 2023
What are differences? Comparing DNN and Human by Their Performance and Characteristics in Speaker Age Estimation 2023