2020
EMNLP
EMNLP 2020
Which *BERT? A Survey Organizing Contextualized Encoders
Abstract
AbstractPretrained contextualized text encoders are now a staple of the NLP community. We present a survey on language representation learning with the aim of consolidating a series of shared lessons learned across a variety of recent efforts. While significant advancements continue at a rapid pace, we find that enough has now been discovered, in different directions, that we can begin to organize advances according to common themes. Through this organization, we highlight important considerations when interpreting recent contributions and choosing which model to use.
❓
The Questioner
🌉
Interdisciplinary Bridge
— Deep Learning and Machine Learning and Natural Language Processing
📈
Trend Setter
— Pretraining
🧭
Keyword Pioneer
— pretrained contextualized encoder
🐣
Hot Topic Early Bird
— text encoder
🐝
Cross-Pollinator
— Artificial Intelligence, Computer Science, Computer Vision, Data Science & Analytics, Deep Learning, Healthcare & Medicine, Interdisciplinary, Knowledge & Reasoning, Machine Learning, Mathematics & Optimization, Natural Language Processing, Reinforcement Learning, Robotics, Security & Privacy, Speech & Audio
Authors
Topics
Machine Learning > Core Methods > Representation Learning
Machine Learning > Learning Types > Self-Supervised Learning
Deep Learning > Techniques > Pretraining
Natural Language Processing > Resources & Methods > Language Modeling
Deep Learning > Models > Transformers
Deep Learning > Learning Types > Transfer Learning
Natural Language Processing > Resources & Methods > Pretraining