2021
EMNLP
EMNLP 2021
Data and Parameter Scaling Laws for Neural Machine Translation
Abstract
AbstractWe observe that the development cross-entropy loss of supervised neural machine translation models scales like a power law with the amount of training data and the number of non-embedding parameters in the model. We discuss some practical implications of these results, such as predicting BLEU achieved by large scale models and predicting the ROI of labeling data in low-resource language pairs.
🌉
Interdisciplinary Bridge
— Artificial Intelligence and Deep Learning and Machine Learning and Natural Language Processing
🧭
Keyword Pioneer
— scaling law
🐣
Hot Topic Early Bird
— scaling law
🐝
Cross-Pollinator
— Artificial Intelligence, Computer Science, Computer Vision, Data Science & Analytics, Deep Learning, Healthcare & Medicine, Interdisciplinary, Knowledge & Reasoning, Machine Learning, Mathematics & Optimization, Natural Language Processing, Reinforcement Learning, Security & Privacy, Speech & Audio
Authors
Topics
Machine Learning > Optimization & Theory > Learning Theory
Machine Learning > Optimization & Theory > Statistical Learning
Natural Language Processing > Applications > Machine Translation
Natural Language Processing > Generation > Machine Translation
Deep Learning > Optimization & Theory > Theory
Artificial Intelligence > Core AI > Natural Language Processing