← Back to papers
2019 ACL ACL 2019

CUED@WMT19:EWC&LMs

Abstract

AbstractTwo techniques provide the fabric of the Cambridge University Engineering Department’s (CUED) entry to the WMT19 evaluation campaign: elastic weight consolidation (EWC) and different forms of language modelling (LMs). We report substantial gains by fine-tuning very strong baselines on former WMT test sets using a combination of checkpoint averaging and EWC. A sentence-level Transformer LM and a document-level LM based on a modified Transformer architecture yield further gains. As in previous years, we also extract n-gram probabilities from SMT lattices which can be seen as a source-conditioned n-gram LM.

🌉 Interdisciplinary Bridge — Deep Learning and Machine Learning and Natural Language Processing
📈 Trend Setter — Fine-Tuning
🧭 Keyword Pioneer — document-level language model
🐣 Hot Topic Early Bird — transformer language model
🐝 Cross-Pollinator — Artificial Intelligence, Computer Science, Computer Vision, Data Science & Analytics, Deep Learning, Healthcare & Medicine, Interdisciplinary, Knowledge & Reasoning, Machine Learning, Mathematics & Optimization, Natural Language Processing, Reinforcement Learning, Security & Privacy, Speech & Audio

Authors

Felix Stahlberg , Danielle Saunders , Adrià de Gispert , Bill Byrne

Topics

Deep Learning > Techniques > Pretraining Natural Language Processing > Generation > Language Modeling Natural Language Processing > Applications > Machine Translation Machine Learning > Learning Types > Representation Learning Machine Learning > Learning Types > Fine-Tuning

Keywords

neural machine translation language modeling elastic weight consolidation transformer language model checkpoint averaging document-level language model
Download PDF

Related papers

What do phone embeddings learn about Phonology? 2019
Unsupervised Morphological Segmentation for Low-Resource Polysynthetic Languages 2019
Understanding Undesirable Word Embedding Associations 2019
Inferential Machine Comprehension: Answering Questions by Recursively Deducing the Evidence Chain from Text 2019
Domain Adaptation of Neural Machine Translation by Lexicon Induction 2019