2020
ACL
ACL 2020
To Pretrain or Not to Pretrain: Examining the Benefits of Pretrainng on Resource Rich Tasks
Abstract
AbstractPretraining NLP models with variants of Masked Language Model (MLM) objectives has recently led to a significant improvements on many tasks. This paper examines the benefits of pretrained models as a function of the number of training samples used in the downstream task. On several text classification tasks, we show that as the number of training examples grow into the millions, the accuracy gap between finetuning BERT-based model and training vanilla LSTM from scratch narrows to within 1%. Our findings indicate that MLM-based models might reach a diminishing return point as the supervised data size increases significantly.
🌉
Interdisciplinary Bridge
— Machine Learning and Natural Language Processing
🐝
Cross-Pollinator
— Artificial Intelligence, Computer Science, Computer Vision, Data Science & Analytics, Deep Learning, Healthcare & Medicine, Interdisciplinary, Knowledge & Reasoning, Machine Learning, Mathematics & Optimization, Natural Language Processing, Reinforcement Learning, Robotics, Speech & Audio
Authors
Topics
Machine Learning > Learning Types > Semi-Supervised Learning
Natural Language Processing > Applications > Text Classification
Machine Learning > Learning Paradigms > Transfer Learning
Natural Language Processing > Resources & Methods > Transfer Learning
Deep Learning > Models > Transformers
Deep Learning > Learning Types > Transfer Learning
Deep Learning > Learning Types > Fine-Tuning
Deep Learning > Models > Language Models