2021
EMNLP
EMNLP 2021
Knowledge Distillation with Noisy Labels for Natural Language Understanding
Abstract
AbstractKnowledge Distillation (KD) is extensively used to compress and deploy large pre-trained language models on edge devices for real-world applications. However, one neglected area of research is the impact of noisy (corrupted) labels on KD. We present, to the best of our knowledge, the first study on KD with noisy labels in Natural Language Understanding (NLU). We document the scope of the problem and present two methods to mitigate the impact of label noise. Experiments on the GLUE benchmark show that our methods are effective even under high noise levels. Nevertheless, our results indicate that more research is necessary to cope with label noise under the KD.
🌉
Interdisciplinary Bridge
— Deep Learning and Machine Learning and Natural Language Processing
📈
Trend Setter
— Large Language Models
🐝
Cross-Pollinator
— Artificial Intelligence, Computer Science, Computer Vision, Data Science & Analytics, Deep Learning, Healthcare & Medicine, Interdisciplinary, Knowledge & Reasoning, Machine Learning, Mathematics & Optimization, Natural Language Processing, Reinforcement Learning, Robotics, Security & Privacy, Speech & Audio
Authors
Topics
Machine Learning > Application Areas > Knowledge Distillation
Natural Language Processing > Resources & Methods > Large Language Models
Machine Learning > Learning Types > Supervised Learning
Machine Learning > Learning Types > Transfer Learning
Machine Learning > Learning Types > Knowledge Distillation
Deep Learning > Techniques > Knowledge Distillation
Machine Learning > Learning Types > Large Language Models